Skip to content

[fix](aggregate) Normalize projected count slots before null safety checks - #67732

Open
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/count-on-index-alias-null-check
Open

[fix](aggregate) Normalize projected count slots before null safety checks#67732
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/count-on-index-alias-null-check

Conversation

@morrySnow

Copy link
Copy Markdown
Contributor

Problem

Counting a projected alias over a nullable indexed column can return an incorrect non-zero result when the filter retains only null rows. Mixing the alias count with COUNT(*) or another count exposes the problem:

SELECT COUNT(x), COUNT(*)
FROM (SELECT k AS x FROM t WHERE k IS NULL) q;

For two matching null rows, the correct result is (0, 2), but the storage-layer index-count path can return (2, 2).

Root cause

The FE implementation rule validates IS NULL and OR predicates before pushing count aggregation to the storage layer. In the Project variant, this validation used the aggregate-side alias slot, while the filter below the Project refers to the source slot. Their expression IDs differ, so the null-safety guard did not recognize that the filter and COUNT referenced the same nullable value.

The rule normalized the aggregate argument to the source slot only later, after the safety decision had already been made.

Reproduction

CREATE TABLE t (
    id INT NOT NULL,
    k INT NULL,
    INDEX idx_k (k) USING INVERTED
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES ("replication_num" = "1");

INSERT INTO t VALUES (1, NULL), (2, NULL), (3, 1);

SELECT COUNT(x), COUNT(*)
FROM (SELECT k AS x FROM t WHERE k IS NULL) q;

SELECT COUNT(x), COUNT(id)
FROM (SELECT k AS x, id FROM t WHERE k IS NULL) q;

Before this change, both queries return (2, 2) and the plan contains pushAggOp=COUNT_ON_INDEX. Both queries should return (0, 2).

Fix

Normalize aggregate arguments through the Project before collecting the slots used by the predicate safety checks. The count slots and filter slots are now compared in the same source expression-ID domain. If IS NULL targets a counted source slot, the FE rejects the index-count pushdown and preserves the column's null values.

This change is limited to the FE planner.

Tests

  • Added a FE plan test for COUNT(projected_alias) + COUNT(*) above an IS NULL filter, verifying that the count-on-index implementation rule is rejected.
  • Ran PhysicalStorageLayerAggregateTest: 7 tests passed.
  • Deployed the FE to a local sandbox and reran both SQL reproductions. They return (0, 2), and the scan plan reports pushAggOp=NONE.

The count-on-index implementation rule checked filter safety using the
aggregate's projected slot IDs. A filter below the project uses source
slot IDs, so an IS NULL predicate on a counted alias could be missed.

Resolve aggregate arguments through the project before applying the
existing predicate checks. This keeps the checks and filters in the same
slot-ID domain and prevents an unsafe storage-layer count pushdown.

Issue Number: None

Tests:
- PhysicalStorageLayerAggregateTest
- sandbox SQL reproduction with nullable indexed data and a subquery alias
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@morrySnow morrySnow changed the title [fix](fe) Normalize projected count slots before null safety checks [fix](aggregate) Normalize projected count slots before null safety checks Sep 9, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16935 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d47c1c00b5189241ed2ac33d2182a8ea42503e24, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17636	3039	3036	3036
q2	2140	249	219	219
q3	10199	857	517	517
q4	4669	251	211	211
q5	7667	600	384	384
q6	135	114	95	95
q7	542	490	385	385
q8	9233	848	879	848
q9	3514	2424	2380	2380
q10	6534	875	723	723
q11	393	195	184	184
q12	613	268	198	198
q13	18136	1540	1176	1176
q14	161	155	136	136
q15	q16	436	394	361	361
q17	1374	800	811	800
q18	3182	2324	2286	2286
q19	1275	845	773	773
q20	380	294	200	200
q21	5555	1792	1898	1792
q22	339	268	231	231
Total cold run time: 94113 ms
Total hot run time: 16935 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3384	3334	3333	3333
q2	551	409	378	378
q3	2225	2310	2212	2212
q4	1205	1182	898	898
q5	2256	2191	2154	2154
q6	170	120	94	94
q7	1041	949	895	895
q8	1580	1408	1405	1405
q9	3200	3198	3219	3198
q10	1909	1834	1674	1674
q11	358	270	256	256
q12	453	429	342	342
q13	1472	1524	1153	1153
q14	172	170	164	164
q15	q16	390	393	354	354
q17	3704	3363	3325	3325
q18	4904	4521	4855	4521
q19	1007	853	868	853
q20	1018	985	843	843
q21	3866	3196	3177	3177
q22	406	348	343	343
Total cold run time: 35271 ms
Total hot run time: 31572 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82745 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d47c1c00b5189241ed2ac33d2182a8ea42503e24, data reload: false

query5	4246	407	335	335
query6	384	133	147	133
query7	4942	388	243	243
query8	287	126	118	118
query9	8696	2866	2895	2866
query10	388	217	180	180
query11	5398	1056	934	934
query12	117	71	68	68
query13	1189	459	316	316
query14	6056	2260	2151	2151
query14_1	2033	2020	2016	2016
query15	173	118	113	113
query16	925	377	365	365
query17	798	449	373	373
query18	2332	339	239	239
query19	197	134	99	99
query20	76	68	69	68
query21	209	105	84	84
query22	5570	5417	5377	5377
query23	6924	6292	6102	6102
query23_1	5984	6188	6317	6188
query24	7309	1085	768	768
query24_1	755	795	787	787
query25	398	282	236	236
query26	1226	236	129	129
query27	2785	427	246	246
query28	4680	1503	1496	1496
query29	894	415	330	330
query30	249	152	138	138
query31	831	394	343	343
query32	134	87	74	74
query33	441	214	178	178
query34	989	840	495	495
query35	396	400	346	346
query36	572	560	578	560
query37	119	80	69	69
query38	1005	865	810	810
query39	469	478	450	450
query39_1	453	491	462	462
query40	202	90	73	73
query41	55	54	54	54
query42	72	75	71	71
query43	244	240	214	214
query44	974	527	540	527
query45	113	104	101	101
query46	787	836	549	549
query47	778	769	705	705
query48	305	292	230	230
query49	538	249	203	203
query50	743	258	193	193
query51	8019	8054	8039	8039
query52	72	73	61	61
query53	188	197	149	149
query54	210	179	153	153
query55	72	60	55	55
query56	185	169	208	169
query57	742	666	671	666
query58	210	165	172	165
query59	1258	1282	1113	1113
query60	241	207	186	186
query61	173	122	126	122
query62	368	209	193	193
query63	171	137	137	137
query64	2735	714	587	587
query65	1682	1618	1647	1618
query66	1843	259	210	210
query67	10227	10089	9864	9864
query68	2934	1219	753	753
query69	349	233	178	178
query70	685	633	628	628
query71	257	172	163	163
query72	2249	1694	1490	1490
query73	647	630	345	345
query74	1983	1231	1144	1144
query75	1188	1113	972	972
query76	2296	707	511	511
query77	250	267	199	199
query78	4139	3801	3348	3348
query79	2608	842	561	561
query80	1559	316	283	283
query81	498	158	133	133
query82	621	132	99	99
query83	288	208	190	190
query84	288	109	87	87
query85	768	345	283	283
query86	392	171	169	169
query87	1029	978	901	901
query88	2774	2177	2080	2080
query89	286	195	173	173
query90	1978	133	126	126
query91	130	113	97	97
query92	80	66	70	66
query93	1460	1063	686	686
query94	615	251	209	209
query95	514	252	302	252
query96	794	600	269	269
query97	1064	1103	1014	1014
query98	153	133	131	131
query99	413	351	307	307
Total cold run time: 178551 ms
Total hot run time: 82745 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.84 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d47c1c00b5189241ed2ac33d2182a8ea42503e24, data reload: false

query1	0.00	0.01	0.00
query2	0.08	0.04	0.03
query3	0.25	0.11	0.10
query4	1.60	0.10	0.10
query5	0.17	0.15	0.17
query6	1.26	0.70	0.69
query7	0.03	0.01	0.00
query8	0.05	0.04	0.03
query9	0.28	0.21	0.22
query10	0.35	0.37	0.34
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.30	0.31	0.32
query14	0.46	0.46	0.46
query15	0.35	0.34	0.35
query16	0.21	0.24	0.21
query17	0.74	0.72	0.68
query18	0.18	0.16	0.16
query19	1.23	1.21	1.23
query20	0.01	0.01	0.00
query21	15.45	0.16	0.13
query22	5.05	0.05	0.05
query23	16.18	0.26	0.10
query24	2.99	0.31	0.27
query25	0.11	0.04	0.03
query26	0.83	0.16	0.12
query27	0.03	0.03	0.04
query28	3.67	0.54	0.28
query29	12.47	3.18	2.54
query30	0.25	0.11	0.12
query31	2.76	0.37	0.17
query32	3.52	0.32	0.23
query33	1.41	1.42	1.42
query34	15.36	2.21	1.82
query35	1.80	1.76	1.77
query36	0.46	0.29	0.30
query37	0.06	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.12	0.07	0.08
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 90.63 s
Total hot run time: 14.84 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants