Skip to content

[fix](expr opt) Preserve try-cast nullability in comparison simplification - #67730

Open
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/preserve-try-cast-nullability
Open

[fix](expr opt) Preserve try-cast nullability in comparison simplification#67730
morrySnow wants to merge 1 commit into
apache:masterfrom
morrySnow:fix/preserve-try-cast-nullability

Conversation

@morrySnow

Copy link
Copy Markdown
Contributor

Problem

A comparison whose left side is an overflowing TRY_CAST can be simplified to a non-null boolean constant even though the cast evaluates to NULL. For example, with a non-null BIGINT value of 128, this predicate must produce NULL:

TRY_CAST(k AS TINYINT) > 127

The incorrect simplification also makes (... > 127) IS NULL false and can filter out rows that should match.

Root cause

The type-range comparison simplifier unwraps a cast to inspect the source type and tighten the possible numeric range. It then reuses that unwrapped child to decide whether an always-true or always-false result can be null. A non-nullable child does not capture TRY_CAST semantics: a failed conversion can introduce NULL independently of child nullability.

Reproduction

CREATE TABLE t (
    id INT NOT NULL,
    k BIGINT NOT NULL
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES ("replication_num" = "1");

INSERT INTO t VALUES (1, 128), (2, 1);

SELECT id,
       TRY_CAST(k AS TINYINT),
       TRY_CAST(k AS TINYINT) > 127,
       (TRY_CAST(k AS TINYINT) > 127) IS NULL
FROM t
ORDER BY id;

Before this change, the comparison for id = 1 is simplified to FALSE. The correct result is NULL, and the final IS NULL expression is TRUE.

Fix

Separate the expression used for numeric range inference from the expression used for nullability. The cast child remains the range source, but an original TRY_CAST is retained when producing null-aware boolean constants. Regular CAST expressions keep their existing child-based normalization.

Tests

  • Extended SimplifyComparisonPredicateTest with out-of-range comparisons over a non-nullable source wrapped by TRY_CAST, covering both true-or-null and false-or-null simplifications.
  • Ran the complete test class: 14 tests passed.
  • Deployed the FE to a local sandbox and verified the SQL reproduction returns NULL for the overflow comparison and returns the row through an IS NULL filter.

The type-range comparison simplifier used a cast's child both to refine
the value range and to decide whether a constant result could be null.
That is incorrect for try-cast because conversion failures can produce
null even when its child is non-nullable.

Keep the original try-cast expression for null-aware constant results,
while retaining the existing child-based behavior for regular casts.

Issue Number: None

Tests:
- SimplifyComparisonPredicateTest
- sandbox SQL reproduction with an overflowing BIGINT-to-TINYINT try-cast
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

@morrySnow morrySnow changed the title [fix](fe) Preserve try-cast nullability in comparison simplification [fix](expr opt) Preserve try-cast nullability in comparison simplification Sep 9, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16764 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aac1613818128059233a0abb78dcb1380999accf, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17588	3011	3010	3010
q2	2080	260	227	227
q3	10246	901	521	521
q4	4671	247	204	204
q5	7667	550	392	392
q6	143	119	93	93
q7	529	501	381	381
q8	9227	853	887	853
q9	3429	2393	2390	2390
q10	6539	831	717	717
q11	391	197	183	183
q12	612	259	192	192
q13	18139	1548	1180	1180
q14	158	149	139	139
q15	q16	431	395	369	369
q17	1385	972	775	775
q18	3151	2228	2242	2228
q19	1258	884	808	808
q20	369	286	196	196
q21	5550	1678	1815	1678
q22	333	268	228	228
Total cold run time: 93896 ms
Total hot run time: 16764 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3367	3303	3299	3299
q2	487	392	383	383
q3	2195	2321	2210	2210
q4	1203	1163	888	888
q5	2182	2137	2104	2104
q6	166	118	89	89
q7	1030	901	815	815
q8	1579	1373	1371	1371
q9	3127	3075	3076	3075
q10	1862	1793	1636	1636
q11	349	274	252	252
q12	454	440	343	343
q13	1496	1516	1164	1164
q14	174	166	159	159
q15	q16	402	405	366	366
q17	3589	3272	3111	3111
q18	4811	4435	4767	4435
q19	885	834	908	834
q20	1028	967	829	829
q21	3837	3109	3286	3109
q22	382	339	331	331
Total cold run time: 34605 ms
Total hot run time: 30803 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82116 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aac1613818128059233a0abb78dcb1380999accf, data reload: false

query5	4261	411	346	346
query6	381	145	122	122
query7	4947	418	228	228
query8	293	120	110	110
query9	8706	2875	2879	2875
query10	405	222	176	176
query11	5379	1044	910	910
query12	115	72	78	72
query13	1200	431	325	325
query14	6098	2215	2090	2090
query14_1	1989	1988	1983	1983
query15	170	128	110	110
query16	911	379	365	365
query17	801	465	367	367
query18	2335	332	237	237
query19	163	136	115	115
query20	76	70	72	70
query21	208	102	91	91
query22	5530	5414	5417	5414
query23	6905	6291	6141	6141
query23_1	6134	6009	6108	6009
query24	7300	1116	780	780
query24_1	764	780	797	780
query25	438	314	258	258
query26	1235	250	128	128
query27	2778	445	257	257
query28	4647	1491	1507	1491
query29	925	429	334	334
query30	249	152	127	127
query31	805	402	332	332
query32	123	70	79	70
query33	471	221	172	172
query34	988	858	483	483
query35	420	423	349	349
query36	562	560	526	526
query37	118	80	72	72
query38	1001	840	809	809
query39	498	476	483	476
query39_1	445	469	467	467
query40	202	86	81	81
query41	54	51	51	51
query42	74	72	72	72
query43	235	242	213	213
query44	996	545	546	545
query45	111	102	98	98
query46	756	832	531	531
query47	764	742	704	704
query48	309	311	223	223
query49	539	239	198	198
query50	739	255	200	200
query51	8063	7905	8052	7905
query52	67	74	60	60
query53	198	200	149	149
query54	209	159	160	159
query55	74	60	55	55
query56	184	201	254	201
query57	699	676	659	659
query58	193	165	173	165
query59	1207	1235	1084	1084
query60	250	192	179	179
query61	160	142	138	138
query62	338	213	192	192
query63	175	152	143	143
query64	2831	723	586	586
query65	1698	1665	1734	1665
query66	1782	256	199	199
query67	10200	9674	9712	9674
query68	3034	1211	760	760
query69	356	227	190	190
query70	689	626	607	607
query71	260	172	171	171
query72	2261	1684	1479	1479
query73	678	635	337	337
query74	2008	1244	1140	1140
query75	1163	1103	978	978
query76	2410	730	520	520
query77	263	269	210	210
query78	3868	3745	3149	3149
query79	2341	816	599	599
query80	1579	316	274	274
query81	493	160	128	128
query82	624	119	106	106
query83	273	208	189	189
query84	294	110	83	83
query85	773	343	275	275
query86	382	177	180	177
query87	996	954	907	907
query88	2783	2121	2140	2121
query89	282	198	177	177
query90	2059	135	131	131
query91	135	120	96	96
query92	81	71	69	69
query93	1431	1000	684	684
query94	626	239	228	228
query95	522	258	298	258
query96	785	576	246	246
query97	1084	1110	1035	1035
query98	171	139	133	133
query99	423	348	305	305
Total cold run time: 178581 ms
Total hot run time: 82116 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.55 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aac1613818128059233a0abb78dcb1380999accf, data reload: false

query1	0.00	0.00	0.00
query2	0.07	0.03	0.04
query3	0.25	0.10	0.10
query4	1.60	0.09	0.11
query5	0.17	0.16	0.17
query6	1.27	0.69	0.69
query7	0.04	0.01	0.00
query8	0.05	0.03	0.02
query9	0.29	0.21	0.21
query10	0.34	0.35	0.33
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.31	0.32	0.31
query14	0.45	0.46	0.44
query15	0.36	0.36	0.34
query16	0.22	0.22	0.22
query17	0.71	0.74	0.66
query18	0.17	0.16	0.17
query19	1.21	1.25	1.14
query20	0.02	0.01	0.01
query21	15.46	0.16	0.13
query22	5.08	0.05	0.04
query23	16.18	0.26	0.10
query24	3.02	0.33	0.26
query25	0.09	0.04	0.03
query26	0.79	0.15	0.11
query27	0.02	0.03	0.04
query28	3.64	0.58	0.28
query29	12.43	3.17	2.55
query30	0.26	0.11	0.10
query31	2.76	0.37	0.17
query32	3.52	0.31	0.23
query33	1.48	1.40	1.40
query34	15.36	2.20	1.77
query35	1.76	1.73	1.72
query36	0.45	0.30	0.28
query37	0.06	0.04	0.03
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.12	0.07	0.07
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 90.53 s
Total hot run time: 14.55 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants