Add TPC-DS SLT tests with plans and Parquet parity - #9804
Conversation
Merging this PR will degrade performance by 3.38%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | decompress[u64, (4000, 1024)] |
70.5 µs | 85.7 µs | -17.71% |
| ⚡ | Simulation | allocate_drop_arrow[0] |
456.9 ns | 402.7 ns | +13.45% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/tpcds-slt-plans (9e66771) with claude/duckdb-vortex-scan-fixes (06fa41d)2
Footnotes
-
206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
claude/duckdb-vortex-scan-fixes(385e8ec) during the generation of this report, so f7f6b87 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
Mirror the TPC-H and ClickBench SLT layout for TPC-DS at scale factor 0.1: `slt/tpcds/generate_data.sh` generates the 24 tables with DuckDB's tpcds extension, exports them to Parquet, converts each to Vortex, and runs `duckdb/parity.slt`, which reads every table in both formats through DuckDB and fails if any row differs. The wrapper `slt/generate_data.sh` accepts `tpcds` like the other datasets. Each engine gets `results/q1` to `results/q99` and `plans/q1` to `plans/q99` from the upstream queries in `vortex-bench/sql/tpcds`, run unchanged. `tpcds.slt` asserts results and EXPLAIN plans against Vortex, and `parquet.slt` reruns the same result records against Parquet so both formats must agree. All 99 queries run on both engines. The runner treats TPC-DS like the other generated-data suites: its tests are ignored until both fixture formats exist. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L57FHoJ15heJVNaqgRq7xB
a8ed1a2 to
9e66771
Compare
|
CI status on the previous head (
Generated by Claude Code |
Summary
Follow-up to #9794 (ClickBench): the same SLT layout for TPC-DS at scale factor 0.1. Stacked on the DuckDB/Vortex scan fixes PR, without which the DuckDB suite returns wrong results for q9 and q28 and stalls partway through the 99 queries.
Changes
slt/tpcds/generate_data.shgenerates the 24 tables with DuckDB'stpcdsextension throughuvx, exports them to Parquet, converts each to Vortex withvx convert, and then runsduckdb/parity.slt, which reads every table in both formats through DuckDB and fails if any row differs.slt/generate_data.shacceptstpcdslike the other datasets; CI already calls the wrapper.create.slt.no,drop.slt.no,results/q1..q99.slt.noandplans/q1..q99.slt.nofrom the upstream queries invortex-bench/sql/tpcds, run unchanged.tpcds.sltasserts results and EXPLAIN plans (EXPLAIN (FORMAT json)withexplain_output = 'all'on DuckDB) against Vortex;parquet.sltreruns the same result records against Parquet so both formats must agree. All 99 queries run on both engines with no errors, and no tie-breaking edits to the queries were needed.Fixture size: 14MB, of which 12MB is the DuckDB JSON plans (the format the TPC-H PR chose).
Checks run:
cargo nextest run -p vortex-sqllogictest -E 'test(tpcds)'twice (5 passed each), fullcargo nextest run -p vortex-sqllogictest(144 passed), clippy onvortex-sqllogictest.🤖 Generated with Claude Code
https://claude.ai/code/session_01L57FHoJ15heJVNaqgRq7xB
Generated by Claude Code