Skip to content

Commit ac0940a

Browse files
committed
Refresh TPC-H comparisons with current measurements
1 parent fd5ac5f commit ac0940a

4 files changed

Lines changed: 1491 additions & 50 deletions

File tree

‎README.md‎

Lines changed: 45 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -65,55 +65,59 @@ and callback scan streaming.
6565

6666
## Performance
6767

68-
The latest recorded full **TPC-H Q1–Q22** comparison is a historical September 15,
69-
2026 run at **SF 0.03** (180,566 lineitems). CoreSQL completed and checked all 22
70-
queries. SQLite completed 20: CoreSQL was faster on 12, SQLite on eight; SQLite
71-
hit the 15-second timeout on Q19 and Q21. DuckDB was faster on all 22 in that run.
68+
The latest full **TPC-H Q1–Q22** comparison was measured September 23, 2026,
69+
at **SF 0.03** (180,566 lineitems), using CoreSQL commit `fd5ac5fe9c`.
70+
CoreSQL completed and checked all 22 queries. SQLite completed 20: CoreSQL was
71+
faster on 15, SQLite on five; SQLite hit the 15-second timeout on Q19 and Q21.
72+
DuckDB was faster than CoreSQL on all 22 queries.
7273

7374
These are single-thread, in-memory client timings on an Apple M1 with 16 GiB RAM:
7475
median of three measured runs after one warmup, excluding loading. Parsing,
7576
execution and fetching are included. **SQLite uses adapted SQL, TEXT dates and
76-
approximate REAL decimals; CoreSQL uses native DATE/DECIMAL.** Drivers also differ.
77-
This is an engineering comparison, not an official TPC-H result, a durability
78-
benchmark, or evidence of general superiority.
77+
approximate REAL decimals; CoreSQL and DuckDB use native DATE/DECIMAL.** Drivers
78+
also differ. This is an engineering comparison, not an official TPC-H result,
79+
a durability benchmark, or evidence of general superiority.
7980

8081
<details>
8182
<summary>All 22 query times — milliseconds, lower is better</summary>
8283

83-
| Query | CoreSQL ms | SQLite ms |
84-
| --- | ---: | ---: |
85-
| Q1 | 175.283 | 90.511 |
86-
| Q2 | 21.437 | 19.803 |
87-
| Q3 | 32.464 | 48.350 |
88-
| Q4 | 19.690 | 37.819 |
89-
| Q5 | 39.436 | 70.774 |
90-
| Q6 | 29.702 | 11.726 |
91-
| Q7 | 64.957 | 56.954 |
92-
| Q8 | 34.362 | 129.663 |
93-
| Q9 | 57.243 | 158.060 |
94-
| Q10 | 24.927 | 18.897 |
95-
| Q11 | 5.360 | 25.208 |
96-
| Q12 | 30.588 | 23.301 |
97-
| Q13 | 38.574 | 46.479 |
98-
| Q14 | 20.049 | 11.768 |
99-
| Q15 | 20.177 | 11.745 |
100-
| Q16 | 8.206 | 10.973 |
101-
| Q17 | 13.022 | 801.836 |
102-
| Q18 | 62.454 | 75.183 |
103-
| Q19 | 14.256 | Timeout (>15 s) |
104-
| Q20 | 13.201 | 1251.266 |
105-
| Q21 | 600.516 | Timeout (>15 s) |
106-
| Q22 | 8.364 | 336.354 |
107-
108-
No speed ratio is assigned to timeouts. SQLite is the pinned 3.54.0 development
109-
baseline, not a stable-release comparison. CoreSQL was a modified working tree
110-
based on `2d3342712b`; these measurements predate the current changes and are not
111-
a benchmark of the current checkout. Three trials do not establish a stable
112-
ranking for close results; Q8 also showed variability in a separate check.
113-
114-
[Full methodology and DuckDB results](benchmarks/tpch/results/OPTIMIZATION_ROUND2.md)
115-
· [Raw measurements and fingerprints](benchmarks/tpch/results/optimization-round2-sf0.03.json)
116-
· [Reproduction instructions](benchmarks/tpch/README.md#include-the-pinned-sqlite-reference)
84+
| Query | CoreSQL ms | SQLite ms | SQLite ms / CoreSQL ms | DuckDB ms |
85+
| --- | ---: | ---: | ---: | ---: |
86+
| Q1 | 106.300 | 90.900 | 0.86× | 6.312 |
87+
| Q2 | 23.542 | 19.524 | 0.83× | 1.965 |
88+
| Q3 | 23.870 | 47.900 | 2.01× | 1.524 |
89+
| Q4 | 17.598 | 37.976 | 2.16× | 1.857 |
90+
| Q5 | 30.974 | 65.812 | 2.12× | 2.131 |
91+
| Q6 | 18.648 | 11.648 | 0.62× | 0.512 |
92+
| Q7 | 38.794 | 57.164 | 1.47× | 2.793 |
93+
| Q8 | 21.514 | 126.984 | 5.90× | 2.787 |
94+
| Q9 | 47.590 | 171.599 | 3.61× | 5.170 |
95+
| Q10 | 21.979 | 19.184 | 0.87× | 4.291 |
96+
| Q11 | 4.902 | 24.453 | 4.99× | 2.245 |
97+
| Q12 | 20.895 | 23.133 | 1.11× | 3.616 |
98+
| Q13 | 38.959 | 46.179 | 1.19× | 4.924 |
99+
| Q14 | 10.443 | 11.533 | 1.10× | 0.917 |
100+
| Q15 | 10.258 | 11.595 | 1.13× | 0.947 |
101+
| Q16 | 7.826 | 10.840 | 1.39× | 1.672 |
102+
| Q17 | 14.819 | 803.753 | 54.24× | 1.431 |
103+
| Q18 | 81.326 | 73.726 | 0.91× | 3.517 |
104+
| Q19 | 16.919 | Timeout (>15 s) | — | 6.587 |
105+
| Q20 | 13.209 | 1270.208 | 96.17× | 1.760 |
106+
| Q21 | 513.586 | Timeout (>15 s) | — | 4.690 |
107+
| Q22 | 7.923 | 336.730 | 42.50× | 1.368 |
108+
109+
**SQLite ms / CoreSQL ms** is the ratio of the measured medians: above 1 means
110+
CoreSQL is faster; below 1 means SQLite is faster. No ratio is assigned to timeouts.
111+
SQLite is the pinned 3.54.0 development baseline, not a stable-release comparison;
112+
DuckDB is 1.5.5. CoreSQL used a clean,
113+
test-enabled Release build with AppleClang 21.0.0 on macOS 27.0. Three trials do not
114+
establish a stable ranking for close results. The OS differs from
115+
the historical September 15 comparison, so changes between those runs cannot be
116+
attributed solely to CoreSQL changes.
117+
118+
[Methodology and reproduction](benchmarks/tpch/README.md#bounded-performance-comparison)
119+
· [Raw measurements, dataset manifest and fingerprints](benchmarks/tpch/results/tpch-2026-09-23-sf0.03.json)
120+
· [Historical September 15 comparison](benchmarks/tpch/results/OPTIMIZATION_ROUND2.md)
117121

118122
</details>
119123

‎benchmarks/README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ performance claims. In-memory commits do not measure durable write performance.
77
- [Durable operations and landmark-memory example](operational.md)
88
- [Combined public SQL suite: speedtest1, DuckDB micro, H2O and TPC-H](suite/README.md)
99
- [TPC-H Q1–Q22: setup, validation and measurement](tpch/README.md)
10-
- [Retained September 15, 2026 comparison](tpch/results/OPTIMIZATION_ROUND2.md)
10+
- [September 23, 2026 comparison](../README.md#performance) and [raw evidence](tpch/results/tpch-2026-09-23-sf0.03.json)
11+
- [Historical September 15, 2026 comparison](tpch/results/OPTIMIZATION_ROUND2.md)
1112
- [speedtest1 adapter: coverage, semantics and reproduction](speedtest1.md)
1213
- [Checksum-pinned SQLite reference](reference/README.md)
1314

@@ -49,8 +50,8 @@ Run profiling separately from timed comparisons.
4950
Retain raw repetitions, correctness results, source/binary fingerprints, dataset
5051
hashes and environment details with any published table. Report failures and
5152
resource limits without assigning speed ratios. The retained TPC-H evidence
52-
supports the README's historical claim; rerun on an exact release revision before
53-
claiming release performance.
53+
identifies the exact measured source revision; rerun on an exact release revision
54+
before claiming release performance.
5455

5556
## General ON join profiling
5657

‎benchmarks/tpch/README.md‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,12 @@
11
# TPC-H workload comparison
22

33
Run the 22 checksum-pinned DuckDB queries against CoreSQL and optional reference
4-
engines. The [recorded comparison](results/OPTIMIZATION_ROUND2.md) supports the
5-
README table; it is a historical September 15, 2026 SF 0.03 run, not a measurement
6-
of the current checkout. This is an engineering comparison, not an official
4+
engines. The [README table](../../README.md#performance) uses the
5+
[September 23, 2026 measurements](results/tpch-2026-09-23-sf0.03.json) at
6+
SF 0.03 on CoreSQL `fd5ac5fe9c`. The report includes raw repetitions,
7+
answer checks, the dataset manifest and build fingerprints. The
8+
[September 15 comparison](results/OPTIMIZATION_ROUND2.md) is retained as historical
9+
evidence. This is an engineering comparison, not an official
710
TPC-H result or a durability benchmark. Query pins and origins are in
811
[manifest.json](manifest.json); reference dependencies are optional.
912

@@ -156,8 +159,10 @@ the native suite. It is a synthetic correctness fixture, not a benchmark run.
156159
## Bounded performance comparison
157160

158161
`measure.py` measures single-thread, in-memory **client latency** against the
159-
pinned DuckDB reference. See [the retained comparison](results/OPTIMIZATION_ROUND2.md)
160-
for the historical SF 0.03 results, including failures and resource limits.
162+
pinned DuckDB reference, with an optional pinned SQLite comparison. See the
163+
[latest table](../../README.md#performance) and
164+
[raw SF 0.03 results](results/tpch-2026-09-23-sf0.03.json), including failures
165+
and resource limits.
161166
This is an engineering comparison, not an official TPC-H score.
162167

163168
```sh
@@ -239,7 +244,7 @@ library/adapter fingerprints and explicit engine order. The raw field
239244
CoreSQL speedup, below 1 means CoreSQL is slower. The three drivers have different
240245
result-conversion overhead, so this remains a client-latency comparison.
241246

242-
See [the retained three-engine table](results/OPTIMIZATION_ROUND2.md).
247+
See [the latest three-engine table](../../README.md#performance).
243248

244249
### Inspect execution work
245250

0 commit comments

Comments
 (0)