Skip to content

chore: [branch-17] backport #4760 #4829

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mbutrovich merged 2 commits into from
Jul 6, 2026
+423 −79
Merged

chore: [branch-17] backport #4760 #4829

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5c135d8
fix: size Iceberg delete files in the native scan to avoid dropping d…
mbutrovich Jun 30, 2026
f5a483f
run prettier
mbutrovich Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.4/aggregate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,5 @@ The following differences from Spark are always present and do not require any a
The following differences from Spark are always present and do not require any additional configuration:

- This function is not deterministic. Results may not match Spark.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.4/array.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,5 @@ The following incompatibilities cause `SortArray` to fall back to Spark by defau
The following cases are not supported by Comet:

- Nested arrays with `Struct` or `Null` child values are not supported natively and will fall back to Spark.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,4 +136,5 @@ The following cases are not supported by Comet:
The following cases are not supported by Comet:

- Only `TimestampType` and `DateType` inputs are supported. `TimestampNTZType` is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.4/map.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,5 @@ By default, Comet accelerates `MapFromEntries` using JVM codegen dispatch, which

- `BinaryType` is not supported as a map key in `map_from_entries`
- `BinaryType` is not supported as a map value in `map_from_entries`

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.4/math.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,5 @@ The following cases are not supported by Comet:
The following cases are not supported by Comet:

- The `seed` argument must be a literal value

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.4/misc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,5 @@ The following cases are not supported by Comet:
The following incompatibilities cause `SortOrder` to fall back to Spark by default. Set `spark.comet.expression.SortOrder.allowIncompatible=true` to enable Comet acceleration despite these differences.

- When `spark.comet.exec.strictFloatingPoint=true`, sorting on floating-point types is not 100% compatible with Spark

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.4/string.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,4 +94,5 @@ The following differences from Spark are always present and do not require any a
The following incompatibilities cause `StringTranslate` to fall back to Spark by default. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to enable Comet acceleration despite these differences.

- DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.4/struct.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,5 @@ The following incompatibilities cause `StructsToCsv` to fall back to Spark by de
The following cases are not supported by Comet:

- Complex types (arrays, maps, structs) in the schema are not supported

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.5/aggregate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,5 @@ The following differences from Spark are always present and do not require any a
The following differences from Spark are always present and do not require any additional configuration:

- This function is not deterministic. Results may not match Spark.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.5/array.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,5 @@ The following incompatibilities cause `SortArray` to fall back to Spark by defau
The following cases are not supported by Comet:

- Nested arrays with `Struct` or `Null` child values are not supported natively and will fall back to Spark.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,4 +136,5 @@ The following cases are not supported by Comet:
The following cases are not supported by Comet:

- Only `TimestampType` and `DateType` inputs are supported. `TimestampNTZType` is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.5/map.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,5 @@ By default, Comet accelerates `MapFromEntries` using JVM codegen dispatch, which

- `BinaryType` is not supported as a map key in `map_from_entries`
- `BinaryType` is not supported as a map value in `map_from_entries`

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.5/math.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,5 @@ The following cases are not supported by Comet:
The following cases are not supported by Comet:

- The `seed` argument must be a literal value

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.5/misc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,5 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
The following cases are not supported by Comet:

- Falls back to Spark when the input type cannot be cast to string.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.5/string.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,4 +94,5 @@ The following differences from Spark are always present and do not require any a
The following incompatibilities cause `StringTranslate` to fall back to Spark by default. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to enable Comet acceleration despite these differences.

- DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-3.5/struct.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,5 @@ The following incompatibilities cause `StructsToCsv` to fall back to Spark by de
The following cases are not supported by Comet:

- Complex types (arrays, maps, structs) in the schema are not supported

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.0/aggregate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,5 @@ The following differences from Spark are always present and do not require any a
The following differences from Spark are always present and do not require any additional configuration:

- This function is not deterministic. Results may not match Spark.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.0/array.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,5 @@ The following incompatibilities cause `SortArray` to fall back to Spark by defau
The following cases are not supported by Comet:

- Nested arrays with `Struct` or `Null` child values are not supported natively and will fall back to Spark.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,4 +136,5 @@ The following cases are not supported by Comet:
The following cases are not supported by Comet:

- Only `TimestampType` and `DateType` inputs are supported. `TimestampNTZType` is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.0/map.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,5 @@ The following incompatibilities cause `MapSort` to fall back to Spark by default
The following cases are not supported by Comet:

- MapSort is unsupported for non-scalar key types (struct, array, map, etc.).

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.0/math.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,5 @@ The following cases are not supported by Comet:
The following cases are not supported by Comet:

- The `seed` argument must be a literal value

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.0/misc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,5 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
The following cases are not supported by Comet:

- Falls back to Spark when the input type cannot be cast to string.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.0/string.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,4 +94,5 @@ The following differences from Spark are always present and do not require any a
The following incompatibilities cause `StringTranslate` to fall back to Spark by default. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to enable Comet acceleration despite these differences.

- DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.0/struct.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,5 @@ The following incompatibilities cause `StructsToCsv` to fall back to Spark by de
The following cases are not supported by Comet:

- Complex types (arrays, maps, structs) in the schema are not supported

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.1/aggregate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,5 @@ The following differences from Spark are always present and do not require any a
The following differences from Spark are always present and do not require any additional configuration:

- This function is not deterministic. Results may not match Spark.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.1/array.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,5 @@ The following incompatibilities cause `SortArray` to fall back to Spark by defau
The following cases are not supported by Comet:

- Nested arrays with `Struct` or `Null` child values are not supported natively and will fall back to Spark.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,4 +136,5 @@ The following cases are not supported by Comet:
The following cases are not supported by Comet:

- Only `TimestampType` and `DateType` inputs are supported. `TimestampNTZType` is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.1/map.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,5 @@ The following incompatibilities cause `MapSort` to fall back to Spark by default
The following cases are not supported by Comet:

- MapSort is unsupported for non-scalar key types (struct, array, map, etc.).

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.1/math.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,5 @@ The following cases are not supported by Comet:
The following cases are not supported by Comet:

- The `seed` argument must be a literal value

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.1/misc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,5 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
The following cases are not supported by Comet:

- Falls back to Spark when the input type cannot be cast to string.

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.1/string.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,4 +94,5 @@ The following differences from Spark are always present and do not require any a
The following incompatibilities cause `StringTranslate` to fall back to Spark by default. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to enable Comet acceleration despite these differences.

- DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/compatibility/expressions/spark-4.1/struct.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,5 @@ The following incompatibilities cause `StructsToCsv` to fall back to Spark by de
The following cases are not supported by Comet:

- Complex types (arrays, maps, structs) in the schema are not supported

<!--END:EXPR_COMPAT-->
1 change: 1 addition & 0 deletions docs/source/user-guide/latest/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ Here are the direct links for downloading the Comet $COMET_VERSION jar file.
- [Comet plugin for Spark 3.5 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.13/$COMET_VERSION/comet-spark-spark3.5_2.13-$COMET_VERSION.jar)
- [Comet plugin for Spark 4.0 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/$COMET_VERSION/comet-spark-spark4.0_2.13-$COMET_VERSION.jar)
- [Comet plugin for Spark 4.1 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.1_2.13/$COMET_VERSION/comet-spark-spark4.1_2.13-$COMET_VERSION.jar)

<!-- ENDIF -->

## Building from source
Expand Down
Loading
Loading