Skip to content

Add TPC-DS SLT tests with plans and Parquet parity - #9804

Open
joseph-isaacs wants to merge 1 commit into
claude/duckdb-vortex-scan-fixesfrom
claude/tpcds-slt-plans
Open

Add TPC-DS SLT tests with plans and Parquet parity#9804
joseph-isaacs wants to merge 1 commit into
claude/duckdb-vortex-scan-fixesfrom
claude/tpcds-slt-plans

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #9794 (ClickBench): the same SLT layout for TPC-DS at scale factor 0.1. Stacked on the DuckDB/Vortex scan fixes PR, without which the DuckDB suite returns wrong results for q9 and q28 and stalls partway through the 99 queries.

Changes

  • slt/tpcds/generate_data.sh generates the 24 tables with DuckDB's tpcds extension through uvx, exports them to Parquet, converts each to Vortex with vx convert, and then runs duckdb/parity.slt, which reads every table in both formats through DuckDB and fails if any row differs. slt/generate_data.sh accepts tpcds like the other datasets; CI already calls the wrapper.
  • Per engine: create.slt.no, drop.slt.no, results/q1..q99.slt.no and plans/q1..q99.slt.no from the upstream queries in vortex-bench/sql/tpcds, run unchanged. tpcds.slt asserts results and EXPLAIN plans (EXPLAIN (FORMAT json) with explain_output = 'all' on DuckDB) against Vortex; parquet.slt reruns the same result records against Parquet so both formats must agree. All 99 queries run on both engines with no errors, and no tie-breaking edits to the queries were needed.
  • The runner registers TPC-DS as a generated-data suite, so its tests are ignored until both fixture formats exist. README updated.

Fixture size: 14MB, of which 12MB is the DuckDB JSON plans (the format the TPC-H PR chose).

Checks run: cargo nextest run -p vortex-sqllogictest -E 'test(tpcds)' twice (5 passed each), full cargo nextest run -p vortex-sqllogictest (144 passed), clippy on vortex-sqllogictest.

🤖 Generated with Claude Code

https://claude.ai/code/session_01L57FHoJ15heJVNaqgRq7xB


Generated by Claude Code

@codspeed-hq

codspeed-hq Bot commented Sep 8, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 3.38%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 2197 untouched benchmarks
⏩ 206 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decompress[u64, (4000, 1024)] 70.5 µs 85.7 µs -17.71%
Simulation allocate_drop_arrow[0] 456.9 ns 402.7 ns +13.45%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/tpcds-slt-plans (9e66771) with claude/duckdb-vortex-scan-fixes (06fa41d)2

Open in CodSpeed

Footnotes

  1. 206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on claude/duckdb-vortex-scan-fixes (385e8ec) during the generation of this report, so f7f6b87 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Mirror the TPC-H and ClickBench SLT layout for TPC-DS at scale factor
0.1: `slt/tpcds/generate_data.sh` generates the 24 tables with DuckDB's
tpcds extension, exports them to Parquet, converts each to Vortex, and
runs `duckdb/parity.slt`, which reads every table in both formats through
DuckDB and fails if any row differs. The wrapper `slt/generate_data.sh`
accepts `tpcds` like the other datasets.

Each engine gets `results/q1` to `results/q99` and `plans/q1` to
`plans/q99` from the upstream queries in `vortex-bench/sql/tpcds`, run
unchanged. `tpcds.slt` asserts results and EXPLAIN plans against Vortex,
and `parquet.slt` reruns the same result records against Parquet so both
formats must agree. All 99 queries run on both engines.

The runner treats TPC-DS like the other generated-data suites: its tests
are ignored until both fixture formats exist.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L57FHoJ15heJVNaqgRq7xB

Copy link
Copy Markdown
Contributor Author

CI status on the previous head (a8ed1a2):


Generated by Claude Code

@joseph-isaacs joseph-isaacs added changelog/chore A trivial change ext/duckdb Relates to the DuckDB integration ext/datafusion Relates to the DataFusion integration labels Sep 8, 2026 — with Claude
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change ext/datafusion Relates to the DataFusion integration ext/duckdb Relates to the DuckDB integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants