chore: [branch-17] backport #4795

docs/source/user-guide/latest/compatibility/expressions/spark-3.4/aggregate.md

@@ -44,4 +44,5 @@ The following differences from Spark are always present and do not require any a
 The following differences from Spark are always present and do not require any additional configuration:
 
 - This function is not deterministic. Results may not match Spark.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.4/array.md

@@ -70,4 +70,5 @@ The following incompatibilities cause `SortArray` to fall back to Spark by defau
 The following cases are not supported by Comet:
 
 - Nested arrays with `Struct` or `Null` child values are not supported natively and will fall back to Spark.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md

@@ -136,4 +136,5 @@ The following cases are not supported by Comet:
 The following cases are not supported by Comet:
 
 - Only `TimestampType` and `DateType` inputs are supported. `TimestampNTZType` is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.4/map.md

@@ -38,4 +38,5 @@ By default, Comet accelerates `MapFromEntries` using JVM codegen dispatch, which
 
 - `BinaryType` is not supported as a map key in `map_from_entries`
 - `BinaryType` is not supported as a map value in `map_from_entries`
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.4/math.md

@@ -38,4 +38,5 @@ The following cases are not supported by Comet:
 The following cases are not supported by Comet:
 
 - The `seed` argument must be a literal value
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.4/misc.md

@@ -50,4 +50,5 @@ The following cases are not supported by Comet:
 The following incompatibilities cause `SortOrder` to fall back to Spark by default. Set `spark.comet.expression.SortOrder.allowIncompatible=true` to enable Comet acceleration despite these differences.
 
 - When `spark.comet.exec.strictFloatingPoint=true`, sorting on floating-point types is not 100% compatible with Spark
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.4/string.md

@@ -94,4 +94,5 @@ The following differences from Spark are always present and do not require any a
 The following incompatibilities cause `StringTranslate` to fall back to Spark by default. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to enable Comet acceleration despite these differences.
 
 - DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.4/struct.md

@@ -36,4 +36,5 @@ The following incompatibilities cause `StructsToCsv` to fall back to Spark by de
 The following cases are not supported by Comet:
 
 - Complex types (arrays, maps, structs) in the schema are not supported
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.5/aggregate.md

@@ -44,4 +44,5 @@ The following differences from Spark are always present and do not require any a
 The following differences from Spark are always present and do not require any additional configuration:
 
 - This function is not deterministic. Results may not match Spark.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.5/array.md

@@ -70,4 +70,5 @@ The following incompatibilities cause `SortArray` to fall back to Spark by defau
 The following cases are not supported by Comet:
 
 - Nested arrays with `Struct` or `Null` child values are not supported natively and will fall back to Spark.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md

@@ -136,4 +136,5 @@ The following cases are not supported by Comet:
 The following cases are not supported by Comet:
 
 - Only `TimestampType` and `DateType` inputs are supported. `TimestampNTZType` is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.5/map.md

@@ -38,4 +38,5 @@ By default, Comet accelerates `MapFromEntries` using JVM codegen dispatch, which
 
 - `BinaryType` is not supported as a map key in `map_from_entries`
 - `BinaryType` is not supported as a map value in `map_from_entries`
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.5/math.md

@@ -38,4 +38,5 @@ The following cases are not supported by Comet:
 The following cases are not supported by Comet:
 
 - The `seed` argument must be a literal value
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.5/misc.md

@@ -56,4 +56,5 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
 The following cases are not supported by Comet:
 
 - Falls back to Spark when the input type cannot be cast to string.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.5/string.md

@@ -94,4 +94,5 @@ The following differences from Spark are always present and do not require any a
 The following incompatibilities cause `StringTranslate` to fall back to Spark by default. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to enable Comet acceleration despite these differences.
 
 - DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-3.5/struct.md

@@ -36,4 +36,5 @@ The following incompatibilities cause `StructsToCsv` to fall back to Spark by de
 The following cases are not supported by Comet:
 
 - Complex types (arrays, maps, structs) in the schema are not supported
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.0/aggregate.md

@@ -44,4 +44,5 @@ The following differences from Spark are always present and do not require any a
 The following differences from Spark are always present and do not require any additional configuration:
 
 - This function is not deterministic. Results may not match Spark.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.0/array.md

@@ -70,4 +70,5 @@ The following incompatibilities cause `SortArray` to fall back to Spark by defau
 The following cases are not supported by Comet:
 
 - Nested arrays with `Struct` or `Null` child values are not supported natively and will fall back to Spark.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md

@@ -136,4 +136,5 @@ The following cases are not supported by Comet:
 The following cases are not supported by Comet:
 
 - Only `TimestampType` and `DateType` inputs are supported. `TimestampNTZType` is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.0/map.md

@@ -48,4 +48,5 @@ The following incompatibilities cause `MapSort` to fall back to Spark by default
 The following cases are not supported by Comet:
 
 - MapSort is unsupported for non-scalar key types (struct, array, map, etc.).
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.0/math.md

@@ -38,4 +38,5 @@ The following cases are not supported by Comet:
 The following cases are not supported by Comet:
 
 - The `seed` argument must be a literal value
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.0/misc.md

@@ -56,4 +56,5 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
 The following cases are not supported by Comet:
 
 - Falls back to Spark when the input type cannot be cast to string.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.0/string.md

@@ -94,4 +94,5 @@ The following differences from Spark are always present and do not require any a
 The following incompatibilities cause `StringTranslate` to fall back to Spark by default. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to enable Comet acceleration despite these differences.
 
 - DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.0/struct.md

@@ -36,4 +36,5 @@ The following incompatibilities cause `StructsToCsv` to fall back to Spark by de
 The following cases are not supported by Comet:
 
 - Complex types (arrays, maps, structs) in the schema are not supported
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.1/aggregate.md

@@ -44,4 +44,5 @@ The following differences from Spark are always present and do not require any a
 The following differences from Spark are always present and do not require any additional configuration:
 
 - This function is not deterministic. Results may not match Spark.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.1/array.md

@@ -70,4 +70,5 @@ The following incompatibilities cause `SortArray` to fall back to Spark by defau
 The following cases are not supported by Comet:
 
 - Nested arrays with `Struct` or `Null` child values are not supported natively and will fall back to Spark.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md

@@ -136,4 +136,5 @@ The following cases are not supported by Comet:
 The following cases are not supported by Comet:
 
 - Only `TimestampType` and `DateType` inputs are supported. `TimestampNTZType` is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.1/map.md

@@ -48,4 +48,5 @@ The following incompatibilities cause `MapSort` to fall back to Spark by default
 The following cases are not supported by Comet:
 
 - MapSort is unsupported for non-scalar key types (struct, array, map, etc.).
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.1/math.md

@@ -38,4 +38,5 @@ The following cases are not supported by Comet:
 The following cases are not supported by Comet:
 
 - The `seed` argument must be a literal value
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.1/misc.md

@@ -56,4 +56,5 @@ The following incompatibilities cause `SortOrder` to fall back to Spark by defau
 The following cases are not supported by Comet:
 
 - Falls back to Spark when the input type cannot be cast to string.
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.1/string.md

@@ -94,4 +94,5 @@ The following differences from Spark are always present and do not require any a
 The following incompatibilities cause `StringTranslate` to fall back to Spark by default. Set `spark.comet.expression.StringTranslate.allowIncompatible=true` to enable Comet acceleration despite these differences.
 
 - DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/compatibility/expressions/spark-4.1/struct.md

@@ -36,4 +36,5 @@ The following incompatibilities cause `StructsToCsv` to fall back to Spark by de
 The following cases are not supported by Comet:
 
 - Complex types (arrays, maps, structs) in the schema are not supported
+
 <!--END:EXPR_COMPAT-->

docs/source/user-guide/latest/installation.md

@@ -92,6 +92,7 @@ Here are the direct links for downloading the Comet $COMET_VERSION jar file.
 - [Comet plugin for Spark 3.5 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark3.5_2.13/$COMET_VERSION/comet-spark-spark3.5_2.13-$COMET_VERSION.jar)
 - [Comet plugin for Spark 4.0 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.0_2.13/$COMET_VERSION/comet-spark-spark4.0_2.13-$COMET_VERSION.jar)
 - [Comet plugin for Spark 4.1 / Scala 2.13](https://repo1.maven.org/maven2/org/apache/datafusion/comet-spark-spark4.1_2.13/$COMET_VERSION/comet-spark-spark4.1_2.13-$COMET_VERSION.jar)
+
 <!-- ENDIF -->
 
 ## Building from source

spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/SparkColumnarArrowReader.scala

@@ -91,6 +91,11 @@ private[comet] class SparkColumnarArrowReader(
     rowsConsumedInCurrent += rowsToProduce
 
     writer.finish()
+    // ArrowWriter derives the root row count from its per-column writes, so a zero-column
+    // input batch (Spark's count-from-metadata scan: numRows > 0, numCols == 0) would otherwise
+    // produce a root with rowCount == 0 and silently drop the rows. Set rowCount explicitly so
+    // downstream aggregations (e.g. df.count()) see the correct value.
+    getVectorSchemaRoot.setRowCount(rowsToProduce)
     onConversionNs(System.nanoTime() - startNs)
     true
   }

spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala

@@ -3711,6 +3711,45 @@ class CometExecSuite extends CometTestBase {
     }
   }
 
+  test("SparkToColumnar preserves row count for zero-column input batches (df.count())") {
+    // Regression test: when spark.comet.scan is disabled but convert.parquet is enabled,
+    // Spark's count-from-metadata optimization emits ColumnarBatches with numRows > 0 and
+    // numCols == 0. SparkColumnarArrowReader used to derive the Arrow row count from
+    // per-column writes only, silently producing zero-row Arrow batches in this case and
+    // making df.count() return 0.
+    withSQLConf(
+      SQLConf.USE_V1_SOURCE_LIST.key -> "parquet",
+      CometConf.COMET_NATIVE_SCAN_ENABLED.key -> "false",
+      CometConf.COMET_SPARK_TO_ARROW_ENABLED.key -> "true",
+      CometConf.COMET_CONVERT_FROM_PARQUET_ENABLED.key -> "true") {
+      withTempPath { dir =>
+        val expected = 10000L
+        spark
+          .range(expected)
+          .selectExpr("id as key", "id % 8 as value")
+          .toDF("key", "value")
+          .write
+          .parquet(dir.toString)
+
+        val df = spark.read.parquet(dir.toString)
+        // Materialize the count query explicitly so we can inspect the plan
+        // that runs — Dataset.count() executes groupBy().count() internally
+        // but doesn't expose that plan
+        val countDf = df.groupBy().count()
+        assert(
+          countDf.collect().head.getLong(0) == expected,
+          "df.count() should match number of written rows")
+
+        // Ensure this test actually exercises the SparkColumnarArrowReader code path
+        // guarded by the fix, so future regressions are caught.
+        val sparkToColumnar = collect(countDf.queryExecution.executedPlan) {
+          case s: CometSparkToColumnarExec => s
+        }
+        assert(sparkToColumnar.nonEmpty, "Expected CometSparkToColumnarExec in the executed plan")
+      }
+    }
+  }
+
   test("read CSV file") {
     Seq("", "csv").foreach { v1List =>
       withSQLConf(