Skip to content

[fix](aggregate) Preserve AVG accumulator width in distinct rewrite - #67740

Open
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/preserve-distinct-average-width
Open

[fix](aggregate) Preserve AVG accumulator width in distinct rewrite#67740
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/preserve-distinct-average-width

Conversation

@morrySnow

Copy link
Copy Markdown
Contributor

Problem

When multiple DISTINCT aggregates trigger the AVG decomposition rewrite,
AVG(DISTINCT BIGINT) can return an incorrect value. For example, averaging
9223372036854775807 and 9223372036854775806 produced -1.5; a predicate
such as avg_value > 0 could therefore discard a row that should match.

Root cause

The rewrite decomposed AVG(DISTINCT BIGINT) into
SUM(DISTINCT BIGINT) / COUNT(DISTINCT BIGINT). Native AVG uses a LARGEINT
accumulator for BIGINT input, while SUM keeps a BIGINT accumulator. The SUM
overflowed before the division result was converted to AVG's return type.

Reproduction

SELECT AVG(DISTINCT x)
FROM (
    SELECT CAST(9223372036854775807 AS BIGINT) AS x
    UNION ALL
    SELECT CAST(9223372036854775806 AS BIGINT) AS x
) t;

With the multi-distinct rewrite enabled, the result was -1.5 instead of
approximately 9.223372036854776e18.

Fix

Losslessly widen a BIGINT AVG argument to LARGEINT before constructing the
replacement SUM and COUNT. The same widened expression is reused by both
aggregates, preserving the shared DISTINCT argument required by the
multi-distinct rewrite while matching AVG's original accumulator width.

Tests

  • Added a focused rewrite unit test that verifies the SUM uses LARGEINT and
    the generated COUNT shares the same widened argument.
  • Added a regression case using the two BIGINT boundary values above together
    with another DISTINCT aggregate and an outer positive-value filter.
  • The focused FE unit test passed: 1 test, 0 failures.
  • The regression suite passed: 1 suite, 0 failed suites.
  • Sandbox verification changed the result from -1.5/0 matching rows to
    9.223372036854776e18/1 matching row.

Issue Number: None

Problem:
The multi-distinct rewrite decomposed AVG(DISTINCT BIGINT) into SUM and COUNT,
but SUM(BIGINT) can overflow before the result is converted to AVG's return type.

Fix:
Losslessly widen the shared distinct argument to LARGEINT before creating SUM
and COUNT so the rewritten expression preserves AVG's accumulator width and
the argument sharing required by multi-distinct aggregation.

Tests:
- add focused rewrite coverage for the widened shared argument
- add a boundary-value regression for AVG(DISTINCT BIGINT)

Issue Number: None
@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow morrySnow changed the title [fix](nereids) Preserve AVG accumulator width in distinct rewrite [fix](aggregate) Preserve AVG accumulator width in distinct rewrite Sep 9, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16882 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1c1e9c66b43654d49e15c8bcbb76e01e2dc0e343, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17587	3029	3003	3003
q2	2119	258	237	237
q3	10211	982	537	537
q4	4675	256	202	202
q5	7687	579	380	380
q6	132	114	93	93
q7	533	506	383	383
q8	9240	856	863	856
q9	3445	2394	2383	2383
q10	6520	839	715	715
q11	398	200	183	183
q12	613	251	205	205
q13	18136	1520	1157	1157
q14	163	151	141	141
q15	q16	439	387	372	372
q17	1400	904	813	813
q18	3133	2287	2238	2238
q19	1270	845	707	707
q20	365	286	198	198
q21	5602	1850	1890	1850
q22	331	270	229	229
Total cold run time: 93999 ms
Total hot run time: 16882 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3378	3294	3301	3294
q2	503	399	372	372
q3	2194	2253	2126	2126
q4	1205	1171	905	905
q5	2172	2117	2082	2082
q6	176	121	88	88
q7	1048	921	843	843
q8	1592	1378	1376	1376
q9	3113	3067	3054	3054
q10	1854	1791	1613	1613
q11	352	264	250	250
q12	458	426	339	339
q13	1489	1526	1163	1163
q14	179	172	158	158
q15	q16	394	390	348	348
q17	3599	3273	3157	3157
q18	4819	4439	4742	4439
q19	862	782	923	782
q20	1023	975	804	804
q21	3848	3117	3364	3117
q22	391	340	325	325
Total cold run time: 34649 ms
Total hot run time: 30635 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81687 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1c1e9c66b43654d49e15c8bcbb76e01e2dc0e343, data reload: false

query5	4247	409	340	340
query6	379	135	128	128
query7	4957	407	234	234
query8	301	131	118	118
query9	8702	2843	2884	2843
query10	376	221	193	193
query11	5372	1028	913	913
query12	121	72	68	68
query13	1193	413	331	331
query14	6094	2200	2085	2085
query14_1	1969	1954	1955	1954
query15	178	133	117	117
query16	914	384	383	383
query17	814	447	361	361
query18	2345	333	236	236
query19	166	141	111	111
query20	73	74	72	72
query21	198	100	88	88
query22	5368	5352	5367	5352
query23	6761	6189	6144	6144
query23_1	6009	6016	5956	5956
query24	7263	1088	738	738
query24_1	756	734	769	734
query25	404	282	232	232
query26	1232	223	130	130
query27	2794	431	248	248
query28	4690	1496	1470	1470
query29	912	430	333	333
query30	247	150	129	129
query31	831	402	324	324
query32	122	68	73	68
query33	458	225	172	172
query34	982	816	494	494
query35	395	404	348	348
query36	565	560	536	536
query37	118	84	72	72
query38	1002	847	814	814
query39	500	496	477	477
query39_1	461	457	472	457
query40	196	87	74	74
query41	55	51	53	51
query42	72	71	72	71
query43	239	238	215	215
query44	985	550	529	529
query45	107	103	99	99
query46	819	849	520	520
query47	767	762	707	707
query48	320	296	231	231
query49	527	246	189	189
query50	735	264	187	187
query51	8092	8041	7942	7942
query52	73	71	61	61
query53	192	205	156	156
query54	221	191	170	170
query55	88	61	59	59
query56	198	196	168	168
query57	685	679	646	646
query58	185	171	171	171
query59	1257	1219	1095	1095
query60	238	187	192	187
query61	129	173	103	103
query62	350	215	184	184
query63	182	160	136	136
query64	2715	691	609	609
query65	1587	1630	1626	1626
query66	1900	285	222	222
query67	10627	9604	9793	9604
query68	2989	1190	698	698
query69	349	213	192	192
query70	670	626	657	626
query71	254	179	172	172
query72	2273	1633	1513	1513
query73	667	584	323	323
query74	2010	1222	1144	1144
query75	1174	1101	970	970
query76	2377	699	515	515
query77	264	265	211	211
query78	4023	3617	3172	3172
query79	2727	807	585	585
query80	1613	329	277	277
query81	514	165	131	131
query82	610	123	95	95
query83	275	210	192	192
query84	296	108	88	88
query85	805	331	276	276
query86	461	170	174	170
query87	1022	971	890	890
query88	2804	2103	2116	2103
query89	278	195	175	175
query90	2012	130	129	129
query91	130	121	100	100
query92	94	72	71	71
query93	1442	1094	700	700
query94	629	254	223	223
query95	529	331	221	221
query96	849	545	270	270
query97	1049	1072	1020	1020
query98	167	133	132	132
query99	415	340	305	305
Total cold run time: 179002 ms
Total hot run time: 81687 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.74 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c1e9c66b43654d49e15c8bcbb76e01e2dc0e343, data reload: false

query1	0.00	0.01	0.00
query2	0.08	0.03	0.03
query3	0.25	0.11	0.11
query4	1.60	0.11	0.10
query5	0.17	0.15	0.16
query6	1.26	0.68	0.68
query7	0.03	0.01	0.00
query8	0.04	0.03	0.04
query9	0.28	0.21	0.22
query10	0.35	0.36	0.35
query11	0.17	0.11	0.10
query12	0.14	0.12	0.12
query13	0.31	0.31	0.30
query14	0.45	0.46	0.44
query15	0.37	0.33	0.35
query16	0.22	0.22	0.22
query17	0.69	0.67	0.67
query18	0.19	0.17	0.16
query19	1.13	1.17	1.16
query20	0.01	0.00	0.01
query21	15.45	0.16	0.11
query22	5.08	0.04	0.05
query23	16.18	0.24	0.11
query24	2.98	0.31	0.25
query25	0.10	0.04	0.03
query26	0.78	0.16	0.13
query27	0.03	0.04	0.03
query28	3.65	0.57	0.27
query29	12.45	3.18	2.59
query30	0.25	0.11	0.14
query31	2.76	0.38	0.17
query32	3.52	0.33	0.23
query33	1.64	1.49	1.46
query34	15.36	2.19	1.80
query35	1.78	1.74	1.74
query36	0.47	0.29	0.29
query37	0.07	0.04	0.04
query38	0.04	0.02	0.02
query39	0.03	0.03	0.02
query40	0.11	0.07	0.07
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 90.63 s
Total hot run time: 14.74 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants