bench: improve Iceberg TPC workflow and plan capture

[Shekharrajak]

This PR improves the Iceberg benchmarking workflow for TPC‑H/TPC‑DS in Comet by:

• adding a Spark JVM Iceberg baseline engine (spark-iceberg.toml) to compare directly against Comet native Iceberg scans
• making Iceberg table creation accept both table.parquet files and table/ directories
• adding per‑query physical plan capture in tpcbench.py for Spark vs Comet plan comparison

Steps to run benchmark against TPC

mkdir -p /tmp/comet-bench/{spark,jars}
curl -L -o /tmp/comet-bench/spark/spark-3.5.8-bin-hadoop3.tgz \
  https://dlcdn.apache.org/spark/spark-3.5.8/spark-3.5.8-bin-hadoop3.tgz
tar xzf /tmp/comet-bench/spark/spark-3.5.8-bin-hadoop3.tgz -C /tmp/comet-bench/spark

curl -L -o /tmp/comet-bench/jars/iceberg-spark-runtime-3.5_2.12-1.8.1.jar \
  https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.8.1/iceberg-spark-runtime-3.5_2.12-1.8.1.jar

# Build Comet JAR (includes native library)
cd ~/Documents/apache/datafusion-comet
make release

# Generate TPC‑H Parquet (SF1)

cd ~/Documents/apache/datafusion-comet
make benchmark-org.apache.spark.sql.GenTPCHData -- \
  --location /tmp/comet-bench/tpch-data \
  --scaleFactor 1

# Generate using DuckDB
python3 -m venv /tmp/comet-bench/duckdb-venv
/tmp/comet-bench/duckdb-venv/bin/pip install duckdb

/tmp/comet-bench/duckdb-venv/bin/python - <<'PY'
import duckdb, os
out = "/tmp/tpcds/sf1_parquet"
os.makedirs(out, exist_ok=True)

con = duckdb.connect()
con.execute("INSTALL tpcds; LOAD tpcds;")
con.execute("CALL dsdgen(sf=1);")

tables = [r[0] for r in con.execute("SHOW TABLES").fetchall()]
for t in tables:
    con.execute(f"COPY {t} TO '{out}/{t}.parquet' (FORMAT PARQUET);")
PY

# Create Iceberg Tables

export SPARK_HOME=/tmp/comet-bench/spark/spark-3.5.8-bin-hadoop3

SPARK_LOCAL_IP=127.0.0.1 $SPARK_HOME/bin/spark-submit \
  --master "local[*]" \
  --driver-class-path /tmp/comet-bench/jars/iceberg-spark-runtime-3.5_2.12-1.8.1.jar \
  --conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \
  --conf "spark.driver.bindAddress=127.0.0.1" \
  --conf "spark.driver.host=127.0.0.1" \
  ~/Documents/apache/datafusion-comet/benchmarks/tpc/create-iceberg-tables.py \
  --benchmark tpch \
  --parquet-path /tmp/comet-bench/tpch-data/tpch/sf1_parquet \
  --warehouse /tmp/comet-bench/iceberg-warehouse

SPARK_LOCAL_IP=127.0.0.1 $SPARK_HOME/bin/spark-submit \
  --master "local[*]" \
  --driver-class-path /tmp/comet-bench/jars/iceberg-spark-runtime-3.5_2.12-1.8.1.jar \
  --conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \
  --conf "spark.driver.bindAddress=127.0.0.1" \
  --conf "spark.driver.host=127.0.0.1" \
  ~/Documents/apache/datafusion-comet/benchmarks/tpc/create-iceberg-tables.py \
  --benchmark tpcds \
  --parquet-path /tmp/tpcds/sf1_parquet \
  --warehouse /tmp/comet-bench/iceberg-warehouse

#  Run Spark JVM (Iceberg Baseline)

SPARK_LOCAL_IP=127.0.0.1 $SPARK_HOME/bin/spark-submit \
  --master "local[*]" \
  --driver-class-path /tmp/comet-bench/jars/iceberg-spark-runtime-3.5_2.12-1.8.1.jar \
  --conf "spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog" \
  --conf "spark.sql.catalog.local.type=hadoop" \
  --conf "spark.sql.catalog.local.warehouse=/tmp/comet-bench/iceberg-warehouse" \
  --conf "spark.sql.defaultCatalog=local" \
  ~/Documents/apache/datafusion-comet/benchmarks/tpc/tpcbench.py \
  --benchmark tpch --catalog local --database tpch \
  --iterations 1 --output /tmp/comet-bench/results \
  --name spark-iceberg \
  --plan-dir /tmp/comet-bench/plans/tpch-sf1

SPARK_LOCAL_IP=127.0.0.1 $SPARK_HOME/bin/spark-submit \
  --master "local[*]" \
  --driver-class-path /tmp/comet-bench/jars/iceberg-spark-runtime-3.5_2.12-1.8.1.jar \
  --conf "spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog" \
  --conf "spark.sql.catalog.local.type=hadoop" \
  --conf "spark.sql.catalog.local.warehouse=/tmp/comet-bench/iceberg-warehouse" \
  --conf "spark.sql.defaultCatalog=local" \
  ~/Documents/apache/datafusion-comet/benchmarks/tpc/tpcbench.py \
  --benchmark tpcds --catalog local --database tpcds \
  --iterations 1 --output /tmp/comet-bench/results \
  --name spark-iceberg \
  --plan-dir /tmp/comet-bench/plans/tpcds-sf1

# Run Comet Native (Iceberg Rust Scan)

export COMET_JAR=~/Documents/apache/datafusion-comet/spark/target/comet-spark-spark3.5_2.12-0.15.0-SNAPSHOT.jar

SPARK_LOCAL_IP=127.0.0.1 $SPARK_HOME/bin/spark-submit \
  --master "local[*]" \
  --driver-class-path "$COMET_JAR:/tmp/comet-bench/jars/iceberg-spark-runtime-3.5_2.12-1.8.1.jar" \
  --jars "$COMET_JAR,/tmp/comet-bench/jars/iceberg-spark-runtime-3.5_2.12-1.8.1.jar" \
  --conf "spark.plugins=org.apache.spark.CometPlugin" \
  --conf "spark.comet.enabled=true" \
  --conf "spark.comet.exec.enabled=true" \
  --conf "spark.comet.scan.icebergNative.enabled=true" \
  --conf "spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager" \
  --conf "spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog" \
  --conf "spark.sql.catalog.local.type=hadoop" \
  --conf "spark.sql.catalog.local.warehouse=/tmp/comet-bench/iceberg-warehouse" \
  --conf "spark.sql.defaultCatalog=local" \
  ~/Documents/apache/datafusion-comet/benchmarks/tpc/tpcbench.py \
  --benchmark tpch --catalog local --database tpch \
  --iterations 1 --output /tmp/comet-bench/results \
  --name comet-iceberg \
  --plan-dir /tmp/comet-bench/plans/tpch-sf1

# Generate Comparison Graphs
cd ~/Documents/apache/datafusion-comet/benchmarks/tpc

# TPC-H charts
python3 generate-comparison.py \
  --benchmark tpch \
  --labels "Spark Iceberg (JVM)" "Comet Iceberg (Native Rust)" \
  --title "TPC-H SF1: Iceberg JVM Scan vs Native Rust Scan" \
  /tmp/comet-bench/results/spark-iceberg-tpch-*.json \
  /tmp/comet-bench/results/comet-iceberg-tpch-*.json

# TPC-DS charts (Q1–Q22)
python3 generate-comparison.py \
  --benchmark tpcds \
  --labels "Spark Iceberg (JVM)" "Comet Iceberg (Native Rust)" \
  --title "TPC-DS SF1 (Q1–Q22): Iceberg JVM vs Native Rust" \
  /tmp/comet-bench/results/spark-iceberg-tpcds-q1to22-*.json \
  /tmp/comet-bench/results/comet-iceberg-tpcds-*.json

[Shekharrajak]

Results generated in Apple M4 Pro 48 GB memory:

[Shekharrajak]

Improvement results in TPC DS dataset:

[Shekharrajak]

Open to suggestions to reduce the commands and simple cli/bash script to have these benchmark graph generated.

[andygrove]

The changes here seem unrelated and should be in a separate PR?

[github-actions]

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.