Repository navigation
[Performance]: Standalone BVT ~1.77× slower on arm64-mo-shanghai-8c16g vs ubuntu-22.04 (median script ~2.5×) #25116
Description
Summary
CI A/B validation (ci-test run 28020049513) compared Standalone BVT Pessimistic on the same matrixorigin/matrixone@main commit, same mo-tester flags (-e optimistic -s resources), only runner/arch differs:
| ARM | x86 | |
|---|---|---|
| Runner | arm64-mo-shanghai-8c16g (8 CPU / 16 GiB K8s pod) |
ubuntu-22.04 GitHub hosted |
| Job | 82933895760 | 82933819100 |
| Start BVT Test wall time | 59.5 min | 33.7 min |
| Ratio | 1.77× |
Both jobs passed (functional correctness OK). Build on the same ARM runner was faster than x86 in the same run (~5.8 min vs ~7.0 min), so the gap is MO runtime / SQL execution, not compile time.
Artifacts: bvt-report-arm-pessimistic vs bvt-report-x86-pessimistic (1009 scripts each).
Environment
- MatrixOne:
matrixorigin/matrixonebranchmain(ci-test workflow inputs) - mo-tester:
main,./run.sh -n -g -o -p .../cases -s .../resources -e optimistic - ARM runner:
arm64-mo-shanghai-8c16g, Shanghai self-hosted K8s pod - x86 runner:
ubuntu-22.04GitHub Actions hosted - Validation workflow: ARM Runner Migration Test with
run_x86_baselines=true
mo-tester aggregate (from report.txt)
| Metric | ARM | x86 | Ratio |
|---|---|---|---|
| Total script time (1009 scripts) | 3549.8 s (~59.2 min) | 2010.1 s (~33.5 min) | 1.77× |
Report [SUMMARY] COST |
3175 s | 1710 s | 1.86× |
| Median per-script ratio | 2.53× | ||
| Mean per-script ratio | 2.61× |
Per-script ratio distribution (n=1003)
| Bucket | Count |
|---|---|
| ARM faster (<1.0×) | 35 |
| 1.0–1.5× | 124 |
| 1.5–2.0× | 179 |
| 2.0–3.0× | 277 |
| ≥3.0× | 388 |
Not a few outliers: ~746 scripts with x86 duration <1 s still have median 2.63× on ARM. Slowdown is broad-based on short/medium SQL cases.
Slowdown by x86 script duration
| x86 bucket | Scripts | Aggregate ARM/x86 | Median ratio |
|---|---|---|---|
| <1 s | 746 | 2.67× | 2.63× |
| 1–5 s | 170 | 2.39× | 2.38× |
| 5–30 s | 78 | 1.89× | 2.09× |
| ≥30 s | 9 | 0.94× | 1.02× |
Long fixed-wait cases (sleep/cron/restore polling) are ~1.0×; they dilute job-level average but do not explain the gap.
Examples (ratio ≈ 1.0×): function/func_sleep.sql, pessimistic_transaction/vector/vector_ivf_restore_idxcron.sql (~105 s both sides).
Top categories by ARM extra time
| Category | ARM total | x86 total | Aggregate ratio |
|---|---|---|---|
snapshot/ |
936 s | 321 s | 2.92× |
tenant/ |
96 s | 32 s | 3.02× |
join/ |
89 s | 30 s | 2.96× |
zz_accesscontrol/ |
123 s | 45 s | 2.75× |
optimizer/ |
170 s | 69 s | 2.46× |
pessimistic_transaction/ |
850 s | 718 s | 1.18× (many fixed-wait vector/fulltext cases) |
Top single-script contributors to ARM − x86 delta (~25.7 min total)
| Extra time | Ratio | Script |
|---|---|---|
| +65 s | 3.59× | optimizer/shuffle.test |
| +47 s | 2.99× | join/spill.sql |
| +29 s | 2.51× | pessimistic_transaction/vector/vector_hnsw_f64.sql |
| +25 s | 3.10× | snapshot/restore_dropped_account.sql |
| +24 s | 2.52× | zz_accesscontrol/create_account.sql |
ARM faster example: fulltext/gojieba.sql 60 s vs 102.8 s (0.58×).
Impact
- CI: Standalone / Upgrade BVT jobs are not suitable for
arm64-mo-shanghai-8c16gtoday (~2× wall time); plan keeps BVT onubuntu-22.04. - Product: If ARM64 is a delivery target, ~2.5× median SQL case slowdown needs investigation before perf parity with amd64.
Initial assessment
- Not observed as functional bug (BVT passed; fixed-wait cases match).
- Likely arm64 runtime / query-path performance gap (optimization debt), not mo-tester or CI framework overhead.
Suggested next steps
- Profile on ARM vs x86 for top delta scripts:
optimizer/shuffle.test,join/spill.sql,snapshot/restore_dropped_account.sql. - Check ARM runner CPU features (CRC32/AES etc.) affecting hash hot paths (
pkg/container/hashtable/hash_arm64.go). - Track arm64 BVT duration as a perf metric if ARM CI coverage expands later.
Activity
heni02 commentedon Jun 24, 2026
同一个 main 的 pr，对比 ubuntu（amd64）和 arm64 机器上 bvt运行的耗时
heni02 commentedon Jun 24, 2026
试了 3.0-dev 现象也是一样的
ci: parallelize coverage across two BVT groups (#26179)
VioletQwQ-0 commentedon Jul 31, 2026
Deferred after triage.
Reason: the original 1.77x ARM-vs-Ubuntu result does not isolate an ARM64 MatrixOne regression. That comparison also changed CPU model, self-hosted Kubernetes vs GitHub-hosted VM, storage/runner environment, and setup action (
setup-env@alter_runner_armvssetup-env@main). Production ARM performance parity is not currently established as a release blocker, and MatrixOne PR BVT remains on the Ubuntu runner by default.Current evidence: the later three-runner comparison run 28634290094 used the same MatrixOne SHA (
3b4a447fd1741810fd821fdd031da6616529817f), mo-tester SHA (b95f5c09930c42bdbe77828b6ae72d39f2be48ed), and setup-action SHA for ARM and GitHub Ubuntu. Both completed 95,617 test statements with 100% success; BVT summary cost was 2039s on ARM vs 1676s on Ubuntu, approximately 1.22x rather than 1.77x. The self-hosted AMD leg used a different MatrixOne SHA and is not accepted as an architecture comparator. PR #26179 reduced the CI critical path but did not fix or close this product-performance question.To promote later: first commit an ARM delivery/parity target and an acceptance threshold, then reproduce it with exact pinned MatrixOne/mo-tester/setup SHAs, comparable self-hosted ARM and AMD resources/storage, and at least 3-5 alternating runs. Capture CPU model/frequency, cgroup throttling, node load, storage latency, and CPU/block/mutex profiles for representative short-SQL, join/spill, and snapshot cases. Promote to active severity only if that controlled evidence repeatedly breaches the agreed threshold.
ci: parallelize coverage across two BVT groups (matrixorigin#26179)
ci: parallelize coverage across two BVT groups (matrixorigin#26179)
ci: parallelize coverage across two BVT groups (matrixorigin#26179)