Skip to content

[fix](nereids) Ignore lambda-local slots in aggregate validation - #67742

Open
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/ignore-lambda-local-slots-in-having
Open

[fix](nereids) Ignore lambda-local slots in aggregate validation#67742
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/ignore-lambda-local-slots-in-having

Conversation

@morrySnow

Copy link
Copy Markdown
Contributor

Problem

Valid HAVING predicates were rejected when a lambda function consumed an
aggregate result. Both map and array forms failed because the analyzer treated
the lambda's local parameters as ungrouped table columns.

Root cause

Lambda parameters are represented by ArrayItemSlot, which extends
SlotReference. FillUpMissingSlots.Resolver recursively visits HAVING
expressions and applied normal GROUP BY validation to every SlotReference,
including these lambda-local slots. They are bound by their
ArrayItemReference and are not inputs from the aggregate child; the general
expression input-slot collector already excludes them for the same reason.

Reproduction

SELECT id, COUNT(*) AS n
FROM (SELECT 1 id UNION ALL SELECT 1 id) t
GROUP BY id
HAVING map_exists((k, v) -> v > 1, map(1, COUNT(*)));

SELECT id, COUNT(*) AS n
FROM (SELECT 1 id UNION ALL SELECT 1 id) t
GROUP BY id
HAVING array_match_any(array_map(x -> x > 1, array(COUNT(*))));

The map query reported an internal map-entry parameter as ungrouped, and the
array query reported x as ungrouped. Both should return (1, 2).

Fix

Skip ArrayItemSlot at the missing-slot resolver entry point. The lambda
binder owns these local slots, so no aggregate output or GROUP BY validation
is needed for them. Ordinary SlotReference handling is unchanged, and a
real ungrouped input column inside the surrounding expression is still
rejected.

Tests

  • Added analyzer coverage for both map and array lambda parameters in HAVING.
  • Added a negative analyzer case proving an ordinary ungrouped input remains
    rejected.
  • Added execution-level regression coverage for both valid queries and the
    invalid-column boundary.
  • Focused FE tests passed: 13 tests, 0 failures.
  • Regression suite passed: 1 suite, 0 failed suites.
  • Sandbox verification returned (1, 2) for both valid queries and preserved
    the expected GROUP BY error for ungrouped_col.

Issue Number: None

Problem:
HAVING expressions that apply array or map lambda functions to aggregate
results were rejected because lambda-local parameters were treated as
ungrouped input columns.

Fix:
Skip ArrayItemSlot while resolving missing aggregate outputs. These slots are
bound inside their lambda and are not inputs from the aggregate child, while
ordinary SlotReference validation remains unchanged.

Tests:
- add analyzer coverage for map and array lambda parameters in HAVING
- verify a real ungrouped input column is still rejected
- add execution-level regression coverage for both lambda forms

Issue Number: None
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16853 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e562ff64735a0fc506f7cd574cd889e9e00fb609, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17577	3022	3007	3007
q2	2120	258	229	229
q3	10214	972	518	518
q4	4667	249	197	197
q5	7682	580	386	386
q6	133	116	93	93
q7	535	504	384	384
q8	9230	937	957	937
q9	3450	2439	2368	2368
q10	6517	878	714	714
q11	390	198	181	181
q12	619	261	199	199
q13	18128	1522	1168	1168
q14	158	152	138	138
q15	q16	441	400	368	368
q17	1394	929	801	801
q18	3037	2263	2284	2263
q19	1270	912	770	770
q20	363	285	197	197
q21	5577	1708	1919	1708
q22	335	272	227	227
Total cold run time: 93837 ms
Total hot run time: 16853 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3343	3289	3276	3276
q2	494	395	373	373
q3	2221	2312	2179	2179
q4	1187	1192	889	889
q5	2195	2112	2111	2111
q6	174	122	88	88
q7	1056	960	934	934
q8	1574	1390	1391	1390
q9	3112	3078	3063	3063
q10	1882	1782	1617	1617
q11	352	271	253	253
q12	451	426	339	339
q13	1490	1542	1165	1165
q14	171	165	155	155
q15	q16	395	391	352	352
q17	3621	3294	3158	3158
q18	4831	4441	4745	4441
q19	854	901	882	882
q20	1010	976	823	823
q21	3849	3101	3249	3101
q22	393	341	316	316
Total cold run time: 34655 ms
Total hot run time: 30905 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81929 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e562ff64735a0fc506f7cd574cd889e9e00fb609, data reload: false

query5	4290	402	331	331
query6	387	131	131	131
query7	4967	416	233	233
query8	297	129	116	116
query9	8692	2889	2879	2879
query10	402	215	183	183
query11	5395	1049	922	922
query12	121	74	74	74
query13	1206	413	303	303
query14	6113	2202	2102	2102
query14_1	1977	1953	1982	1953
query15	178	120	113	113
query16	936	360	334	334
query17	754	438	342	342
query18	2321	307	219	219
query19	155	129	101	101
query20	66	67	77	67
query21	204	102	83	83
query22	5499	5435	5367	5367
query23	6779	6074	6119	6074
query23_1	6013	6136	6047	6047
query24	7323	1060	755	755
query24_1	758	775	771	771
query25	401	275	237	237
query26	1221	235	127	127
query27	2772	431	258	258
query28	4696	1476	1483	1476
query29	898	417	328	328
query30	252	142	127	127
query31	809	388	322	322
query32	127	76	76	76
query33	451	216	177	177
query34	993	852	488	488
query35	407	419	360	360
query36	563	590	503	503
query37	121	81	70	70
query38	1002	842	836	836
query39	501	487	495	487
query39_1	466	443	474	443
query40	204	92	91	91
query41	58	58	56	56
query42	77	72	74	72
query43	235	241	210	210
query44	1033	530	525	525
query45	109	107	111	107
query46	777	806	516	516
query47	748	762	699	699
query48	317	304	217	217
query49	547	235	188	188
query50	738	266	193	193
query51	8016	8076	8094	8076
query52	74	66	58	58
query53	190	201	149	149
query54	225	155	161	155
query55	70	62	60	60
query56	196	172	156	156
query57	705	763	657	657
query58	199	159	159	159
query59	1178	1225	1078	1078
query60	252	191	168	168
query61	148	143	128	128
query62	354	217	184	184
query63	175	150	143	143
query64	2886	832	695	695
query65	1635	1611	1658	1611
query66	1980	271	216	216
query67	9824	9839	9649	9649
query68	3032	1168	704	704
query69	356	228	192	192
query70	679	636	620	620
query71	258	176	171	171
query72	2271	1661	1514	1514
query73	639	567	326	326
query74	1996	1223	1137	1137
query75	1168	1087	958	958
query76	2365	707	525	525
query77	260	264	216	216
query78	3979	3728	3231	3231
query79	2768	845	600	600
query80	1569	326	277	277
query81	514	153	131	131
query82	608	134	99	99
query83	271	209	189	189
query84	297	108	90	90
query85	838	331	281	281
query86	471	175	179	175
query87	1010	973	894	894
query88	3081	2108	2133	2108
query89	271	199	174	174
query90	2182	128	128	128
query91	132	115	97	97
query92	96	74	70	70
query93	2274	1069	665	665
query94	609	207	236	207
query95	532	334	226	226
query96	800	560	294	294
query97	1065	1086	1002	1002
query98	170	138	136	136
query99	418	350	308	308
Total cold run time: 179835 ms
Total hot run time: 81929 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e562ff64735a0fc506f7cd574cd889e9e00fb609, data reload: false

query1	0.01	0.00	0.01
query2	0.08	0.04	0.04
query3	0.25	0.11	0.11
query4	1.59	0.09	0.10
query5	0.17	0.16	0.15
query6	1.27	0.70	0.68
query7	0.04	0.01	0.00
query8	0.04	0.03	0.03
query9	0.29	0.22	0.21
query10	0.34	0.34	0.33
query11	0.16	0.13	0.12
query12	0.15	0.13	0.12
query13	0.30	0.31	0.31
query14	0.45	0.44	0.45
query15	0.36	0.34	0.35
query16	0.22	0.23	0.24
query17	0.68	0.69	0.65
query18	0.18	0.17	0.17
query19	1.21	1.22	1.20
query20	0.02	0.01	0.01
query21	15.48	0.18	0.13
query22	5.04	0.04	0.05
query23	16.17	0.25	0.11
query24	2.99	0.33	0.25
query25	0.09	0.05	0.03
query26	0.75	0.16	0.11
query27	0.03	0.02	0.04
query28	3.65	0.52	0.27
query29	12.49	3.17	2.60
query30	0.25	0.11	0.12
query31	2.78	0.37	0.17
query32	3.53	0.31	0.22
query33	1.32	1.48	1.50
query34	15.40	2.21	1.77
query35	1.74	1.72	1.69
query36	0.45	0.29	0.29
query37	0.06	0.04	0.03
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.12	0.07	0.08
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 90.36 s
Total hot run time: 14.73 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants