diff --git a/CHANGELOG.md b/CHANGELOG.md index a4d2199c..dd495ebf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,11 @@ All notable changes to this project will be documented in this file. It uses the `pg_clickhouse.session_settings` parameter, so that a ClickHouse server profile cannot silently change the `IN` semantics the pushdown rules rely on ([#315]). +* Added or corrected pushdown for `IN`-family operations on arrays. Previous + behavior was missing or incorrect, either structurally or due to behavioral + differences in the ClickHouse implementation of the operation. Corrected + the structurally incorrect pushdown ([#315]) and handled ClickHouse + behavioral changes using special case statements where needed ([#317]). * Added pushdown for more aggregate functions ([#290]): * [corr] * [covarpop] @@ -92,8 +97,7 @@ All notable changes to this project will be documented in this file. It uses the * Fixed wrong results from pushed-down `= ANY`/`<> ALL` expressions and `IN` expressions over constant lists and when a `NULL` could reach the comparison, because ClickHouse evaluates `IN` under two-valued logic while - PostgreSQL evaluates three-value `NULL` logic. These cases now ship only - where PostgreSQL's three-valued answer can be provably preserved ([#315]). + PostgreSQL evaluates three-value `NULL` logic ([#315], [#317]). * Fixed `<> ANY(array)` deparsing to ClickHouse SQL that computes `<> ALL`, which is wrong even with no `NULL`s involved: In Postgres, `1 <> ANY('{1,5}')` is `TRUE`. For now, do not push down ([#315]). @@ -149,6 +153,8 @@ All notable changes to this project will be documented in this file. It uses the "ClickHouse/pg_clickhouse#316 Support inserting Array(Nullable(T)) via binary driver" [#318]: https://github.com/ClickHouse/pg_clickhouse/pull/318 "ClickHouse/pg_clickhouse#318 Push down the re2 v0.4 `@~` operator" + [#317]: https://github.com/ClickHouse/pg_clickhouse/pull/317 + "ClickHouse/pg_clickhouse#317 Push down the array IN family unconditionally" ## [v0.3.2] — 2026-06-16 diff --git a/doc/pg_clickhouse.md b/doc/pg_clickhouse.md index fb0a0b15..8d2e560b 100644 --- a/doc/pg_clickhouse.md +++ b/doc/pg_clickhouse.md @@ -1319,18 +1319,25 @@ equivalents as follows: ClickHouse evaluates `IN` under two-valued logic: when the probe finds no match it returns `0` even if a NULL is involved, where PostgreSQL computes NULL. To preserve PostgreSQL semantics, pg_clickhouse pushes down the `IN` -family (`IN`, `NOT IN`, `= ANY`, `<> ALL`, and `IN (SELECT ...)`) only where -the two systems provably agree: in a plain filter condition, where a NULL -disqualifies a row exactly like `FALSE`, or when none of the probe, the list, -nor the subquery output can produce a NULL. A `NOT IN (SELECT ...)` filter -over nullable columns still pushes down, deparsed with compensating guards -that keep PostgreSQL's answer: a set containing a NULL disqualifies every -row, and a NULL probe passes only against an empty set. Each guard is -omitted when a `NOT NULL` declaration proves it unnecessary. The remaining -shapes (NULL-capable `IN` expressions in select lists, `GROUP BY`, or -`ORDER BY`, and `NOT IN` over arrays or grouped subqueries) are evaluated -locally. Declaring columns `NOT NULL` maximizes pushdown; [IMPORT FOREIGN -SCHEMA] does so automatically for non-`Nullable` ClickHouse columns. +family over a constant list or array (`IN`, `NOT IN`, `= ANY`, `= ALL`, +`<> ANY`, `<> ALL`) unconditionally: the native or cheap form where it can +prove neither the probe nor an array element can be NULL, or a guarded +`CASE` form otherwise that checks for NULL values at runtime instead, +computing PostgreSQL's exact three-valued answer (TRUE, FALSE, NULL) in +every context, including value positions like a `SELECT` list or +`GROUP BY`. + +A `NOT IN (SELECT ...)` filter over nullable columns also pushes down, +deparsed with compensating guards that keep PostgreSQL's behavior: a set +containing a NULL disqualifies every row, and a NULL probe passes only +against an empty set. Each guard is omitted when a `NOT NULL` declaration +proves it unnecessary. Unlike the array forms above, this guard only +applies in a plain filter condition (or under `NOT`); we still do not push +down `IN (SELECT ...)` (in a value position) nor grouped/aggregated subquery +bodies. Declaring columns `NOT NULL` maximizes pushdown by letting the cheaper +unguarded form ship instead; [IMPORT FOREIGN SCHEMA] does so automatically for +non-`Nullable` ClickHouse columns. The proof follows non-NULL constants, +`NOT NULL` columns, and basic arithmetic (`+`, `-`, `*`, unary `-`) over them. These rules assume ClickHouse's default `transform_null_in = 0`, which pg_clickhouse sets on every query through the default value of the diff --git a/src/deparse.c b/src/deparse.c index 5915ff23..66d50d0b 100644 --- a/src/deparse.c +++ b/src/deparse.c @@ -174,7 +174,7 @@ typedef struct deparse_expr_cxt { /* * How an expression's result is consumed, for the IN-family NULL-semantics - * rules (see the block comment for saop_null_semantics_ok). + * rules (see the block comment for deparseScalarArrayOpExpr). * * EXPR_CTX_TRUTH only qualifies rows: NULL and FALSE are interchangeable. * EXPR_CTX_NEGATED the direct operand of one NOT whose own result is in a @@ -185,6 +185,12 @@ typedef struct deparse_expr_cxt { * exactly which SubPlans get the guarded form. * EXPR_CTX_EXACT the value itself is observable: exact three-valued * equivalence required. + * + * NOTE: deparseExpr's own switch on nodeTag degrades its analogous + * deparse_expr_cxt.truth_ctx to false before dispatch for every node type + * except T_BoolExpr/T_CaseExpr/T_RelabelType/T_ScalarArrayOpExpr -- the same + * types this walker does something other than force EXPR_CTX_EXACT for. If + * you add a type here that preserves ctx for its children, add it there too. */ typedef enum ExprTruthCtx { EXPR_CTX_TRUTH, @@ -459,12 +465,11 @@ chfdw_is_foreign_expr( return true; } -/* 1: '=', 2: '<>', 0 - false */ -int +CHEqualOp chfdw_is_equal_op(Oid opno) { Form_pg_operator operform; HeapTuple opertup; - int res = 0; + CHEqualOp res = CH_OP_NONE; opertup = SearchSysCache1(OPEROID, ObjectIdGetDatum(opno)); if (!HeapTupleIsValid(opertup)) { @@ -474,11 +479,11 @@ chfdw_is_equal_op(Oid opno) { operform = (Form_pg_operator)GETSTRUCT(opertup); if (NameStr(operform->oprname)[0] == '=' && NameStr(operform->oprname)[1] == '\0') { - res = 1; + res = CH_OP_EQ; } else if ( NameStr(operform->oprname)[0] == '<' && NameStr(operform->oprname)[1] == '>' ) { - res = 2; + res = CH_OP_NE; } ReleaseSysCache(opertup); @@ -555,10 +560,14 @@ agg_partial_kind(Aggref* agg) { static bool is_ok_in_expr(Expr* expr); +static bool +op_expr_never_null(OpExpr* op, foreign_glob_cxt* glob_cxt); + /* - * True if `expr` provably cannot evaluate to NULL: a non-NULL constant, or a + * True if `expr` provably cannot evaluate to NULL: a non-NULL constant, a * column of a scanned relation that is declared NOT NULL and cannot have been - * null-extended by an outer join. Anything else is assumed nullable. + * null-extended by an outer join, or certain arithmetic operators over such + * inputs (see op_expr_never_null). Anything else is assumed nullable. * * PG 17+ precomputes the catalog half of this as * RelOptInfo->notnullattnums; the syscache lookup below keeps one code path @@ -624,9 +633,119 @@ expr_never_null(Expr* expr, foreign_glob_cxt* glob_cxt) { return notnull; } + if (IsA(expr, OpExpr)) { + return op_expr_never_null((OpExpr*)expr, glob_cxt); + } + return false; } +/* + * Operator implementation funcs that cannot return NULL when no input is NULL. + * Limited to basic arithmetic, which returns a value or raises an error. + * Being strict alone proves nothing here: a strict function may still compute + * NULL from non-NULL inputs (json's ->> on a missing key, for one), and no + * catalog flag distinguishes the two, so every entry below is hand-picked. + * Extend as pushdown-worthy shapes turn up; the list need not stay sorted here, + * since `sort_never_null_opfuncs` (below) sorts it in place once, at load time, + * for consistency across PG versions. + */ +static Oid never_null_opfuncs[] = { + /* + */ + F_INT2PL, + F_INT4PL, + F_INT8PL, + F_INT24PL, + F_INT42PL, + F_INT28PL, + F_INT82PL, + F_INT48PL, + F_INT84PL, + F_FLOAT4PL, + F_FLOAT8PL, + F_FLOAT48PL, + F_FLOAT84PL, + F_NUMERIC_ADD, + /* - */ + F_INT2MI, + F_INT4MI, + F_INT8MI, + F_INT24MI, + F_INT42MI, + F_INT28MI, + F_INT82MI, + F_INT48MI, + F_INT84MI, + F_FLOAT4MI, + F_FLOAT8MI, + F_FLOAT48MI, + F_FLOAT84MI, + F_NUMERIC_SUB, + /* * */ + F_INT2MUL, + F_INT4MUL, + F_INT8MUL, + F_INT24MUL, + F_INT42MUL, + F_INT28MUL, + F_INT82MUL, + F_INT48MUL, + F_INT84MUL, + F_FLOAT4MUL, + F_FLOAT8MUL, + F_FLOAT48MUL, + F_FLOAT84MUL, + F_NUMERIC_MUL, + /* unary - */ + F_INT2UM, + F_INT4UM, + F_INT8UM, + F_FLOAT4UM, + F_FLOAT8UM, + F_NUMERIC_UMINUS, +}; + +static void __attribute__((constructor)) +sort_never_null_opfuncs(void) { + qsort(never_null_opfuncs, lengthof(never_null_opfuncs), sizeof(Oid), oid_cmp); +} + +/* + * True if `opfuncid` is one of never_null_opfuncs. + */ +static bool +is_non_nullable_op(Oid opfuncid) { + return bsearch( + &opfuncid, + never_null_opfuncs, + lengthof(never_null_opfuncs), + sizeof(Oid), + oid_cmp + ) != NULL; +} + +/* + * Proves non-nullability of an OpExpr (in some cases). More cases = nicer SQL. + * (If we fail to prove something is non-nullable, we just emit runtime guards.) + * Returns true if `op` is a never-null operator (see `is_non_nullable_op`) and + * every argument is itself provably non-NULL. + */ +static bool +op_expr_never_null(OpExpr* op, foreign_glob_cxt* glob_cxt) { + ListCell* lc; + + set_opfuncid(op); + if (!is_non_nullable_op(op->opfuncid)) { + return false; + } + foreach (lc, op->args) { + if (!expr_never_null((Expr*)lfirst(lc), glob_cxt)) { + return false; + } + } + return true; +} + /* * What is known at plan time about NULL elements in a ScalarArrayOpExpr's * right-hand array. @@ -672,122 +791,6 @@ saop_array_nulls(Expr* arr, foreign_glob_cxt* glob_cxt) { return SAOP_ARR_UNKNOWN; } -/* - * Decide whether a ScalarArrayOpExpr keeps Postgres semantics on ClickHouse. - * This comment is the reference for the divergence rules used here and in - * the SubPlan gate (is_shippable_subplan) and deparse (deparseGuardedNotIn). - * - * ClickHouse evaluates IN under two-valued logic (given transform_null_in = - * 0, the ClickHouse default, which pg_clickhouse.session_settings sets - * per-query so a server profile cannot change it; overriding the GUC is at - * the user's own risk, like the join_use_nulls default the pushed outer - * joins rely on): a probe that finds no match yields 0 even when the searched set - * contains a NULL, and NOT IN is the exact complement, where Postgres - * yields NULL in both cases. The two systems agree whenever no NULL can - * reach the comparison; when one can, ClickHouse substitutes FALSE where - * Postgres computes NULL, and for the complemented forms (NOT IN, <> ALL) - * TRUE where Postgres computes NULL. The has()/countEqual() deparse forms - * additionally match a NULL probe against NULL elements as if they were - * ordinary values. - * - * A FALSE-for-NULL substitution is invisible exactly where a NULL qualifies - * a row the same way FALSE does: WHERE/JOIN/HAVING conditions, CASE WHEN - * branches, and aggregate FILTERs, composed through any number of AND/OR - * (EXPR_CTX_TRUTH). Under NOT, in a function/operator/aggregate argument, - * in a grouping or ordering expression — anywhere the boolean's value is - * consumed — the substitution is observable, so exact three-valued - * equivalence is demanded. A TRUE-for-NULL substitution is wrong in every - * context, so the complemented forms require a provably NULL-free - * right-hand side. - * - * The decision follows the same form deparseScalarArrayOpExpr will choose: - * the IN-list form for a non-empty constant array (is_ok_in_expr), - * has()/countEqual() otherwise. `optype` is chfdw_is_equal_op's - * classification: 1 for = ANY, 2 for <> ALL. - */ -static bool -saop_null_semantics_ok( - ScalarArrayOpExpr* saop, - int optype, - ExprTruthCtx ctx, - foreign_glob_cxt* glob_cxt -) { - /* Arrays have no guarded deparse form: negated means exact */ - bool exact_ctx = ctx != EXPR_CTX_TRUTH; - Expr* probe = linitial(saop->args); - Expr* arr = lsecond(saop->args); - SaopArrayNulls nulls = saop_array_nulls(arr, glob_cxt); - - /* - * A NULL array constant survives const-folding when the probe is not - * constant, and the operator's strictness makes the result NULL for - * every row. Keep it local: no deparse form models that, and - * is_ok_in_expr cannot inspect a NULL array datum. - */ - if (IsA(arr, Const) && ((Const*)arr)->constisnull) { - return false; - } - - /* - * <> ANY has no correct deparse form yet: the not has() ClickHouse SQL it - * would produce computes <> ALL (Postgres: 1 <> ANY of {1,5} is TRUE, one - * element differs; not has({1,5}, 1) is FALSE). Always evaluate locally. - * A truth-context spelling exists for a future follow-up — TRUE exactly - * when a non-NULL element differs from a non-NULL probe: - * x IS NOT NULL AND - * length(arr) - countEqual(arr, x) - countEqual(arr, NULL) > 0 - */ - if (saop->useOr && optype == 2) { - return false; - } - - /* - * = ALL deparses as countEqual() = length(), which counts a NULL element - * as equal to a NULL probe where Postgres never compares two NULLs equal. - * An empty array is exact by arithmetic (0 = 0 is TRUE on both systems, - * NULL probe included); otherwise ship only when no NULL can be involved. - */ - if (!saop->useOr && optype == 1) { - if (nulls == SAOP_ARR_EMPTY) { - return true; - } - return nulls == SAOP_ARR_NULL_FREE && expr_never_null(probe, glob_cxt); - } - - /* - * Both systems compare against nothing: = ANY is FALSE and <> ALL is TRUE - * for every probe, NULL included, in both deparse forms. - */ - if (nulls == SAOP_ARR_EMPTY) { - return true; - } - - if (is_ok_in_expr(arr)) { - /* - * IN-list form. ClickHouse's native IN yields NULL for a NULL probe, - * so a NULL-free list is exact for both = ANY and <> ALL. A list that - * may carry a NULL diverges only toward FALSE for = ANY (tolerable in - * truth context) and toward TRUE for <> ALL (never tolerable). - */ - if (nulls == SAOP_ARR_NULL_FREE) { - return true; - } - return optype == 1 && !exact_ctx; - } - - /* - * has()/countEqual() form: a NULL probe is matched like a value, so - * exactness additionally requires the probe to be provably non-NULL. - */ - if (optype == 1) { - if (nulls == SAOP_ARR_NULL_FREE) { - return !exact_ctx || expr_never_null(probe, glob_cxt); - } - return !exact_ctx && expr_never_null(probe, glob_cxt); - } - return nulls == SAOP_ARR_NULL_FREE && expr_never_null(probe, glob_cxt); -} - /* * How to ship a Postgres `NOT (x IN (SELECT ..))`. */ @@ -903,7 +906,7 @@ classify_notin_subplan(SubPlan* subplan, PlannerInfo* root, Relids relids) { * currently considered here. * * ctx tracks whether the expression's value is observable beyond qualifying - * rows (see ExprTruthCtx and saop_null_semantics_ok's block comment): + * rows (see ExprTruthCtx and deparseScalarArrayOpExpr's block comment): * EXPR_CTX_TRUTH only while every enclosing node treats NULL and FALSE * identically. */ @@ -1071,14 +1074,26 @@ foreign_expr_walker(Node* node, foreign_glob_cxt* glob_cxt, ExprTruthCtx ctx) { } break; case T_ScalarArrayOpExpr: { ScalarArrayOpExpr* oe = (ScalarArrayOpExpr*)node; - int optype = chfdw_is_equal_op(oe->opno); + CHEqualOp optype = chfdw_is_equal_op(oe->opno); + Expr* arr; - if (!optype) { + if (optype == CH_OP_NONE) { return false; } - /* IN under two-valued ClickHouse logic; see saop_null_semantics_ok */ - if (!saop_null_semantics_ok(oe, optype, ctx, glob_cxt)) { + /* + * ClickHouse doesn't support null arrays, and Postgres's planner won't + * optimize away `x in NULL::int[]` unless x is also NULL. Postgres will + * always evaluate this to null, but pushing the whole test down as NULL + * would remove the SQL in the probe (and any side-effects thereof). + * This is a pathological case and the most correct form to push would + * look like `tupleElement((, NULL), 2)`, which is dumb. + * Keep the pathological case local. Everything else ships; see + * deparseScalarArrayOpExpr's block comment for how it preserves + * Postgres's NULL semantics across the more difficult cases. + */ + arr = (Expr*)lsecond(oe->args); + if (IsA(arr, Const) && ((Const*)arr)->constisnull) { return false; } @@ -1590,7 +1605,7 @@ is_shippable_subplan(SubPlan* subplan, foreign_glob_cxt* glob_cxt, ExprTruthCtx if (list_length(op->args) != 2) { return false; } - if (chfdw_is_equal_op(op->opno) != 1) { + if (chfdw_is_equal_op(op->opno) != CH_OP_EQ) { return false; } if (!IsA(lsecond(op->args), Param)) { @@ -1668,7 +1683,7 @@ is_shippable_subplan(SubPlan* subplan, foreign_glob_cxt* glob_cxt, ExprTruthCtx * Postgres answer even when a NULL can reach the comparison, so only * grouped/aggregated bodies fall back; a genuine value position demands * exactness, which requires that no NULL can arrive from either side. - * See saop_null_semantics_ok's block comment for the divergence rules + * See deparseScalarArrayOpExpr's block comment for the divergence rules * and deparseGuardedNotIn for the guarded form. */ if (subplan->subLinkType == ANY_SUBLINK && ctx != EXPR_CTX_TRUTH) { @@ -5152,13 +5167,13 @@ deparseNullIfExpr(NullIfExpr* node, deparse_expr_cxt* context) { } static void -deparseAsIn(ScalarArrayOpExpr* node, deparse_expr_cxt* context, int optype) { +deparseAsIn(ScalarArrayOpExpr* node, deparse_expr_cxt* context, CHEqualOp optype) { StringInfo buf = context->buf; Expr* arg1 = linitial(node->args); Expr* arg2 = lsecond(node->args); deparseExpr(arg1, context); - if (optype == 1) { + if (optype == CH_OP_EQ) { appendStringInfoString(buf, " IN "); } else { appendStringInfoString(buf, " NOT IN "); @@ -5204,89 +5219,226 @@ is_ok_in_expr(Expr* expr) { } /* - * Deparse given ScalarArrayOpExpr expression. To avoid problems around - * priority of operations, we always parenthesize the arguments. + * Deparse `expr` into a freshly allocated string instead of context->buf, + * for callers that splice the same SQL into a template more than once. + * Slightly ups our memory footprint during planning, but safer than trying + * to reuse a buffer slice and more performant than deparsing multiple times. */ -static void -deparseScalarArrayOpExpr(ScalarArrayOpExpr* node, deparse_expr_cxt* context) { +static char* +deparseExprToString(Expr* expr, deparse_expr_cxt* context) { StringInfo buf = context->buf; - Expr* arg1; - Expr* arg2; + StringInfoData capture; - /* Retrieve information about the operator from system catalog. */ - int optype = chfdw_is_equal_op(node->opno); + initStringInfo(&capture); + context->buf = &capture; + deparseExpr(expr, context); + context->buf = buf; + return capture.data; +} - if (optype == 0) { +/* + * Emit a guarded CASE reproducing Postgres's exact three-valued answer for + * any combination of the IN family (useOr, optype), checking at runtime + * rather than requiring a plan-time proof: + * CASE WHEN IS NULL AND notEmpty() THEN NULL + * WHEN THEN + * WHEN THEN NULL + * ELSE + * END + * + * This is ExecEvalScalarArrayOp's order of business. Here, match_result is what + * is returned when an element of the array genuinely matches the probe, which + * depends on the operation (elaboration below). + * + * An empty array gives the complement of that (`no_match_result` in the code) + * for every probe, NULL included: Postgres actually resolves it before looking + * at the probe, and in our guarded SQL, every WHEN will be false thanks to the + * `notEmpty()` check in the first branch, so the ELSE answers. + * + * A NULL probe against a non-empty array is NULL on both systems. + * + * The actual predicate implementation for the form above must use `countEqual`, + * and there is no `countNotEqual`, which means that complemented forms must + * instead compute their count by subtracting from the full array length. + * Here, Postgres's `useOr` determines most of our logic, while optype logically + * determines which operation we apply to each element but in practice instead + * determines how we use the result of `countEqual()`. + * + * `= ANY`: optype=EQ, useOr=True, exhaustive_op=NE (match > 0) + * `= ALL`: optype=EQ, useOr=False, exhaustive_op=EQ (match + null < length) + * `<> ANY`: optype=NE, useOr=True, exhaustive_op=NE (match + null < length) + * `<> ALL`: optype=NE, useOr=False, exhaustive_op=EQ (match > 0) + */ +static void +deparseGuardedScalarArrayOp( + bool useOr, + CHEqualOp optype, + Expr* probe, + Expr* arr, + deparse_expr_cxt* context +) { + StringInfo buf = context->buf; + CHEqualOp exhaustive_op = useOr ? CH_OP_NE : CH_OP_EQ; + const char* match_result = useOr ? "true" : "false"; + const char* no_match_result = useOr ? "false" : "true"; + char* probesql = deparseExprToString(probe, context); + char* arrsql = deparseExprToString(arr, context); + + appendStringInfo( + buf, "(CASE WHEN %s IS NULL AND notEmpty(%s) THEN NULL WHEN ", probesql, arrsql + ); + if (optype != exhaustive_op) { + appendStringInfo(buf, "countEqual(%s, %s) > 0", arrsql, probesql); + } else { + appendStringInfo( + buf, + "(length(%s) - countEqual(%s, %s) - countEqual(%s, NULL)) > 0", + arrsql, + arrsql, + probesql, + arrsql + ); + } + appendStringInfo( + buf, + " THEN %s WHEN countEqual(%s, NULL) > 0 THEN NULL ELSE %s END)", + match_result, + arrsql, + no_match_result + ); +} + +/* + * Deparse given ScalarArrayOpExpr expression. Every branch below + * self-parenthesizes its own output rather than relying on a shared + * wrapper, so none of them depend on how a neighboring branch is spelled. + * + * ClickHouse's IN-family behavior diverges from Postgres, so we push down two + * forms (simple and guarded) below to account for it. This comment is the + * reference for the divergence rules used here and in the SubPlan gate + * (`is_shippable_subplan`) and deparse (`deparseGuardedNotIn`). + * + * ClickHouse evaluates IN under two-valued logic: + * 1 in {1, 2, 3}: 1 + * 10 in {1, 2, 3}: 0 + * 1 in {1, NULL}: 1 + * 10 in {1, NULL}: 0 + * NULL in {1, NULL}: 1 if transform_null_in else NULL + * NULL in {1, 2, 3}: 0 if transform_null_in else NULL + * + * Postgres/Standard SQL usually disagrees: + * 1 in {1, 2, 3}: 1 + * 10 in {1, 2, 3}: 0 + * 1 in {1, NULL}: 1 + * 10 in {1, NULL}: NULL <-- The problem case + * NULL in {1, NULL}: NULL + * NULL in {1, 2, 3}: NULL + * + * The two systems agree when NULL is not involved, but in standard SQL, NULL serves as + * "dunno, LOL" and wildcard matches anything, resulting in a final "dunno, LOL" + * answer, while ClickHouse just treats NULL as an ordinary value to match on. + * The two systems technically agree where FALSE/NULL are both falsey, such as + * WHERE/JOIN/HAVING conditions, CASE WHEN branches, and aggregate FILTERs. + * + * Under NOT, in a function/operator/aggregate argument, in a grouping or + * ordering expression, or anywhere else the boolean's value is observable, + * exact three-valued equivalence is required. And anywhere ClickHouse gives + * TRUE where Postgres gives NULL is guaranteed to be wrong, so the complemented + * forms require a provably NULL-free right-hand side or guardrails to handle + * the divergent cases. + * + * ClickHouse has a separate behavioral quirk for arrays, similar to its `IN` + * mechanics: `has()` and `countEqual()` will match NULL strictly against NULL + * in those contexts as well, and are unaffected by `transform_null_in` or + * similar settings. This means these routines cannot directly be used to + * implement `<> ALL()` / `= ANY()`, but it also gives us a correct (if uglier) + * pushdown spelling. + * + * We force `transform_null_in=0` to cover the cases it can, and in cases where + * values are nullable (or we cannot prove that they aren't), emit the extra guard + * branches below to ensure Postgres behavior is preserved after pushdown. + */ +static void +deparseScalarArrayOpExpr(ScalarArrayOpExpr* node, deparse_expr_cxt* context) { + StringInfo buf = context->buf; + Expr* arg1 = linitial(node->args); + Expr* arg2 = lsecond(node->args); + CHEqualOp optype = chfdw_is_equal_op(node->opno); + foreign_glob_cxt glob_cxt; + SaopArrayNulls nulls; + bool cheap_ok; + + if (optype == CH_OP_NONE) { ereport( ERROR, errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg( - "pg_clickhouse supports only equal (not equal) operations on ANY/ALL " + "pg_clickhouse supports only equal and not equal operations on ANY/ALL " "functions" ) ); } - /* Sanity check. */ Assert(list_length(node->args) == 2); - appendStringInfoChar(buf, '('); - if (node->useOr) { - arg2 = lsecond(node->args); - - /* very narrow case for IN() and = ANY(ARRAY) */ - if (optype == 1 && is_ok_in_expr(arg2)) { - deparseAsIn(node, context, optype); - } else { - if (optype == 1) { - appendStringInfoString(buf, "has("); - } else { - appendStringInfoString(buf, "not has("); - } + memset(&glob_cxt, 0, sizeof(glob_cxt)); + glob_cxt.root = context->root; + glob_cxt.relids = context->scanrel->relids; + nulls = saop_array_nulls(arg2, &glob_cxt); - /* Deparse right operand. */ - deparseExpr(arg2, context); - appendStringInfoChar(buf, ','); + /* + * Very narrow case for IN()/NOT IN(): the native form is exact once the + * array is a nice, non-empty, provably NULL-free constant, regardless + * of the probe (ClickHouse's native IN propagates a NULL probe + * correctly; see the block comment above) + */ + if (nulls == SAOP_ARR_NULL_FREE && is_ok_in_expr(arg2) && + ((node->useOr && optype == CH_OP_EQ) || (!node->useOr && optype == CH_OP_NE))) { + appendStringInfoChar(buf, '('); + deparseAsIn(node, context, optype); + appendStringInfoChar(buf, ')'); + return; + } - /* Deparse left operand. */ - arg1 = linitial(node->args); - deparseExpr(arg1, context); + /* + * An empty array is exact via the cheap form below regardless of the + * probe's nullability (has()/countEqual() degenerate correctly on an + * empty array by arithmetic), so it counts as "cheap ok" on its own; + * otherwise both the array and the probe must be provably non-NULL. + */ + cheap_ok = nulls == SAOP_ARR_EMPTY || + (nulls == SAOP_ARR_NULL_FREE && expr_never_null(arg1, &glob_cxt)); - /* Close function call */ - appendStringInfoChar(buf, ')'); - } - } else { - arg2 = lsecond(node->args); + if (cheap_ok) { + char* probesql = deparseExprToString(arg1, context); + char* arrsql = deparseExprToString(arg2, context); - /* very narrow case for NOT IN() and <> ANY(ARRAY) */ - if (optype == 2 && is_ok_in_expr(arg2)) { - deparseAsIn(node, context, optype); + if (node->useOr && optype == CH_OP_EQ) { + /* = ANY */ + appendStringInfo(buf, "(has(%s, %s))", arrsql, probesql); + } else if (node->useOr) { + /* + * <> ANY: TRUE exactly when an array element differs from the + * probe. `not has()` would compute <> ALL instead (see the + * block comment above); with the array proven NULL-free, "an + * element differs" is just "fewer matches than elements". + */ + appendStringInfo( + buf, "(countEqual(%s, %s) < length(%s))", arrsql, probesql, arrsql + ); + } else if (optype == CH_OP_EQ) { + /* = ALL */ + appendStringInfo( + buf, "(countEqual(%s, %s) = length(%s))", arrsql, probesql, arrsql + ); } else { - appendStringInfoString(buf, "countEqual("); - - /* Deparse right operand. */ - arg2 = lsecond(node->args); - deparseExpr(arg2, context); - appendStringInfoChar(buf, ','); - - /* Deparse left operand. */ - arg1 = linitial(node->args); - deparseExpr(arg1, context); - - /* Close function call */ - if (optype == 1) { - appendStringInfoString(buf, ") = length("); - - /* Deparse right operand again */ - deparseExpr(arg2, context); - appendStringInfoChar(buf, ')'); - } else { - appendStringInfoString(buf, ") = 0"); - } + /* <> ALL */ + appendStringInfo(buf, "(countEqual(%s, %s) = 0)", arrsql, probesql); } + return; } - appendStringInfoChar(buf, ')'); + deparseGuardedScalarArrayOp(node->useOr, optype, arg1, arg2, context); } /* @@ -5575,8 +5727,7 @@ aggref_on_aggregate_function(Aggref* node, deparse_expr_cxt* context) { */ static void deparsePartialStatArray(Aggref* node, AggPartialKind kind, deparse_expr_cxt* context) { - StringInfo buf = context->buf; - StringInfoData argbuf, condbuf; + StringInfo buf = context->buf; TargetEntry* tle = (TargetEntry*)linitial(node->args); const char* ifSuffix = node->aggfilter ? "If" : ""; char* arg; @@ -5585,21 +5736,15 @@ deparsePartialStatArray(Aggref* node, AggPartialKind kind, deparse_expr_cxt* con Assert(!tle->resjunk); /* Capture argument SQL so it can be referenced several times. */ - initStringInfo(&argbuf); - context->buf = &argbuf; - deparseExpr((Expr*)tle->expr, context); + arg = deparseExprToString((Expr*)tle->expr, context); /* Capture FILTER condition as each component's -If argument. */ if (node->aggfilter) { - initStringInfo(&condbuf); - context->buf = &condbuf; - deparseExpr((Expr*)node->aggfilter, context); - cf = psprintf(", (%s) > 0", condbuf.data); + cf = psprintf( + ", (%s) > 0", deparseExprToString((Expr*)node->aggfilter, context) + ); } - context->buf = buf; - arg = argbuf.data; - if (kind == AGG_PARTIAL_AVG_INT) { // https://github.com/postgres/postgres/blob/f5cc81719e6da4cbdb1f797c48b693e91018153a/src/backend/utils/adt/numeric.c#L6760 appendStringInfo( @@ -5645,9 +5790,8 @@ deparsePartialStatArray(Aggref* node, AggPartialKind kind, deparse_expr_cxt* con if (node->aggfilter) { pfree(cf); - pfree(condbuf.data); } - pfree(argbuf.data); + pfree(arg); } /* diff --git a/src/fdw.c b/src/fdw.c index c5d37a8b..ab4d1bca 100644 --- a/src/fdw.c +++ b/src/fdw.c @@ -1994,8 +1994,8 @@ is_simple_join_clause(Expr* expr) { if (IsA(expr, OpExpr)) { OpExpr* opexpr = (OpExpr*)expr; - if (chfdw_is_equal_op(opexpr->opno) == 1 && list_length(opexpr->args) == 2 && - IsA(list_nth(opexpr->args, 0), Var) && + if (chfdw_is_equal_op(opexpr->opno) == CH_OP_EQ && + list_length(opexpr->args) == 2 && IsA(list_nth(opexpr->args, 0), Var) && IsA(list_nth(opexpr->args, 1), Var)) { return true; } diff --git a/src/include/fdw.h b/src/include/fdw.h index 921a5972..221ccc5f 100644 --- a/src/include/fdw.h +++ b/src/include/fdw.h @@ -358,11 +358,19 @@ chfdw_get_jointype_name(JoinType jointype); char* chfdw_array_to_ch_literal(Datum arr); +/* chfdw_is_equal_op's classification of a comparison operator's name. */ +typedef enum CHEqualOp { + CH_OP_NONE, /* neither '=' nor '<>' */ + CH_OP_EQ, /* '=' */ + CH_OP_NE, /* '<>' */ +} CHEqualOp; + +extern CHEqualOp +chfdw_is_equal_op(Oid opno); + /* in shippable.c */ extern bool chfdw_is_builtin(Oid objectId); -extern int -chfdw_is_equal_op(Oid opno); /* * Connection cache hash table entry diff --git a/test/expected/binary_queries.out b/test/expected/binary_queries.out index 404d9f98..c375bc54 100644 --- a/test/expected/binary_queries.out +++ b/test/expected/binary_queries.out @@ -397,11 +397,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (4 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ---------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +---------------------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -521,11 +521,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------- + QUERY PLAN +------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/binary_queries_1.out b/test/expected/binary_queries_1.out index 30838498..30ff44c3 100644 --- a/test/expected/binary_queries_1.out +++ b/test/expected/binary_queries_1.out @@ -395,11 +395,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (3 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ---------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +---------------------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -519,11 +519,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------- + QUERY PLAN +------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/binary_queries_2.out b/test/expected/binary_queries_2.out index 27a766d0..961e25c5 100644 --- a/test/expected/binary_queries_2.out +++ b/test/expected/binary_queries_2.out @@ -395,11 +395,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (3 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ---------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +---------------------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -519,11 +519,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------- + QUERY PLAN +------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/binary_queries_3.out b/test/expected/binary_queries_3.out index 24ee0d16..2387cf71 100644 --- a/test/expected/binary_queries_3.out +++ b/test/expected/binary_queries_3.out @@ -395,11 +395,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (3 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ---------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +---------------------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -519,11 +519,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------- + QUERY PLAN +------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/binary_queries_4.out b/test/expected/binary_queries_4.out index 14209a63..93639e31 100644 --- a/test/expected/binary_queries_4.out +++ b/test/expected/binary_queries_4.out @@ -395,11 +395,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (3 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ---------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +---------------------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -519,11 +519,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------- + QUERY PLAN +------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/binary_queries_5.out b/test/expected/binary_queries_5.out index 29a221ae..d1f6e9f8 100644 --- a/test/expected/binary_queries_5.out +++ b/test/expected/binary_queries_5.out @@ -397,11 +397,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (4 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ---------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +---------------------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -521,11 +521,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------- + QUERY PLAN +------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/binary_queries_6.out b/test/expected/binary_queries_6.out index 966992b9..4781f2d6 100644 --- a/test/expected/binary_queries_6.out +++ b/test/expected/binary_queries_6.out @@ -397,11 +397,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (4 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ---------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +---------------------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -521,11 +521,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------- + QUERY PLAN +------------------------------------------------------------------------------------------------------- Foreign Scan on binary_queries_test.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM binary_queries_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/http.out b/test/expected/http.out index 562ae1fa..d97798b3 100644 --- a/test/expected/http.out +++ b/test/expected/http.out @@ -507,11 +507,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (4 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------------ + QUERY PLAN +------------------------------------------------------------------------------------------------------------ Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -631,11 +631,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN --------------------------------------------------------------------------------------------- + QUERY PLAN +--------------------------------------------------------------------------------------------- Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/http_1.out b/test/expected/http_1.out index d124d5e6..aa1769e6 100644 --- a/test/expected/http_1.out +++ b/test/expected/http_1.out @@ -505,11 +505,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (3 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------------ + QUERY PLAN +------------------------------------------------------------------------------------------------------------ Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -629,11 +629,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN --------------------------------------------------------------------------------------------- + QUERY PLAN +--------------------------------------------------------------------------------------------- Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/http_2.out b/test/expected/http_2.out index ab679e57..9d7cf836 100644 --- a/test/expected/http_2.out +++ b/test/expected/http_2.out @@ -505,11 +505,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (3 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------------ + QUERY PLAN +------------------------------------------------------------------------------------------------------------ Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -629,11 +629,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN --------------------------------------------------------------------------------------------- + QUERY PLAN +--------------------------------------------------------------------------------------------- Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/http_3.out b/test/expected/http_3.out index 31daee19..463ad947 100644 --- a/test/expected/http_3.out +++ b/test/expected/http_3.out @@ -505,11 +505,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (3 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------------ + QUERY PLAN +------------------------------------------------------------------------------------------------------------ Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -629,11 +629,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN --------------------------------------------------------------------------------------------- + QUERY PLAN +--------------------------------------------------------------------------------------------- Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/http_4.out b/test/expected/http_4.out index ebb0ddf2..85e671f2 100644 --- a/test/expected/http_4.out +++ b/test/expected/http_4.out @@ -505,11 +505,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (3 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------------ + QUERY PLAN +------------------------------------------------------------------------------------------------------------ Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -629,11 +629,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN --------------------------------------------------------------------------------------------- + QUERY PLAN +--------------------------------------------------------------------------------------------- Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/http_5.out b/test/expected/http_5.out index a5cf3fe9..ab3dfd5c 100644 --- a/test/expected/http_5.out +++ b/test/expected/http_5.out @@ -507,11 +507,11 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE (c1 IS NOT NULL) IS DIST (4 rows) EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]); -- ScalarArrayOpExpr - QUERY PLAN ------------------------------------------------------------------------------------------------------------ + QUERY PLAN +------------------------------------------------------------------------------------------------------------ Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([c2, 1, (c1 + 0)], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- ScalarArrayOpExpr @@ -631,11 +631,11 @@ SELECT * FROM ft1 t1 WHERE c1 = ANY(ARRAY[c2, 1, c1 + 0]) ORDER BY c1; -- Scalar -- Syntax error on ClickHouse 24 and earlier. EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr - QUERY PLAN --------------------------------------------------------------------------------------------- + QUERY PLAN +--------------------------------------------------------------------------------------------- Foreign Scan on public.ft1 t1 Output: c1, c2, c3, c4, c5, c6, c7, c8 - Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([],c1))) + Remote SQL: SELECT c1, c2, c3, c4, c5, c6, c7, c8 FROM http_test.t1 WHERE ((has([], c1))) (3 rows) SELECT * FROM ft1 t1 WHERE c1 = ANY('{}'); -- Empty ScalarArrayOpExpr diff --git a/test/expected/in_null_semantics.out b/test/expected/in_null_semantics.out index d5ad2df1..356b254f 100644 --- a/test/expected/in_null_semantics.out +++ b/test/expected/in_null_semantics.out @@ -2,12 +2,14 @@ -- -- ClickHouse evaluates IN under two-valued logic: a probe that finds no -- match yields 0 even when the searched set contains a NULL, where Postgres --- computes NULL; The ClickHouse has()/countEqual() functions also match a NULL probe as --- if it were an ordinary value. A FALSE-for-NULL substitution is invisible --- while the result only qualifies rows (WHERE/JOIN/HAVING through AND/OR), --- but observable under NOT and anywhere the boolean's value is consumed. --- These tests ensure the shapes deparse.c ships and that the rows returned --- match local (Postgres) semantics either way. +-- computes NULL, which changes negation semantics (NOT(NULL) -> NULL). +-- The has()/countEqual() functions also match a NULL probe against a NULL +-- element as if it were an ordinary value. Every shape below ships regardless: +-- deparse.c uses the cheap native form when it can prove neither side can be +-- NULL, and a guarded CASE otherwise that checks at runtime instead, computing +-- Postgres's exact three-valued answer (NULL, not FALSE, or worse, TRUE) +-- in every context. These tests ensure both forms deparse correctly and that +-- the rows returned match local (Postgres) semantics either way. CREATE SERVER in_null_svr FOREIGN DATA WRAPPER clickhouse_fdw OPTIONS(dbname 'in_null_test', driver 'binary'); CREATE USER MAPPING FOR CURRENT_USER SERVER in_null_svr; @@ -23,9 +25,10 @@ SELECT clickhouse_raw_query('CREATE DATABASE in_null_test'); (1 row) --- xn imports as a plain (nullable) int column, xp as int NOT NULL +-- xn imports as a plain (nullable) int column, xp as int NOT NULL, and arr +-- as int[]; its elements' nullability is never provable at plan time SELECT clickhouse_raw_query('CREATE TABLE in_null_test.tnull - (id Int32, xn Nullable(Int32), xp Int32) + (id Int32, xn Nullable(Int32), xp Int32, arr Array(Int32)) ENGINE = MergeTree ORDER BY id'); clickhouse_raw_query ---------------------- @@ -33,7 +36,8 @@ SELECT clickhouse_raw_query('CREATE TABLE in_null_test.tnull (1 row) SELECT clickhouse_raw_query($$ - INSERT INTO in_null_test.tnull VALUES (1, 1, 1), (2, 2, 2), (3, NULL, 3) + INSERT INTO in_null_test.tnull + VALUES (1, 1, 1, [1, 500]), (2, 2, 2, [500]), (3, NULL, 3, []) $$); clickhouse_raw_query ---------------------- @@ -46,19 +50,20 @@ SET SESSION search_path = in_null_test,public; -- The import must carry ClickHouse nullability into the declarations the -- never-null proofs rely on \d tnull - Foreign table "in_null_test.tnull" - Column | Type | Collation | Nullable | Default | FDW options ---------+---------+-----------+----------+---------+------------- - id | integer | | not null | | - xn | integer | | | | - xp | integer | | not null | | + Foreign table "in_null_test.tnull" + Column | Type | Collation | Nullable | Default | FDW options +--------+-----------+-----------+----------+---------+------------- + id | integer | | not null | | + xn | integer | | | | + xp | integer | | not null | | + arr | integer[] | | not null | | Server: in_null_svr FDW options: (database 'in_null_test', table_name 'tnull', engine 'MergeTree') -- ============================================================ --- 1. IN over a constant list: pushable in a WHERE condition +-- 1. IN over a constant list: always pushable in a WHERE condition -- ============================================================ --- NULL-free list is exactly equivalent +-- NULL-free list is exactly equivalent to the native IN EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xn IN (1, 500) ORDER BY id; QUERY PLAN @@ -74,15 +79,15 @@ SELECT id FROM tnull WHERE xn IN (1, 500) ORDER BY id; 1 (1 row) --- A NULL in the list only pulls results toward FALSE, which a WHERE treats --- like the NULL Postgres computes: still pushable +-- A NULL in the list can't be proven absent, so this ships via a guarded +-- CASE instead of the native IN, computing the real answer at runtime EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xn IN (1, NULL) ORDER BY id; - QUERY PLAN ------------------------------------------------------------------------------------------------------ + QUERY PLAN +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Foreign Scan on in_null_test.tnull Output: id - Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((xn IN (1,NULL))) ORDER BY id ASC NULLS LAST + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((CASE WHEN xn IS NULL AND notEmpty([1,NULL]) THEN NULL WHEN countEqual([1,NULL], xn) > 0 THEN true WHEN countEqual([1,NULL], NULL) > 0 THEN NULL ELSE false END)) ORDER BY id ASC NULLS LAST (3 rows) SELECT id FROM tnull WHERE xn IN (1, NULL) ORDER BY id; @@ -92,7 +97,7 @@ SELECT id FROM tnull WHERE xn IN (1, NULL) ORDER BY id; (1 row) -- ============================================================ --- 2. NOT IN over a constant list: pushable only when NULL-free +-- 2. NOT IN over a constant list: always pushable, native when NULL-free -- ============================================================ -- ClickHouse's native NOT IN yields NULL for a NULL probe, so a NULL-free -- list is exact: id 3 (xn IS NULL) is dropped on both systems @@ -111,17 +116,17 @@ SELECT id FROM tnull WHERE xn NOT IN (1, 500) ORDER BY id; 2 (1 row) --- With a NULL in the list Postgres returns no rows (every result is NULL --- or FALSE) where ClickHouse would return id 2: must stay local +-- With a NULL in the list, ClickHouse's native NOT IN would return id 2 +-- where Postgres returns no rows (every result is NULL or FALSE): the +-- guarded CASE computes Postgres's answer instead of using the native form EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xn NOT IN (1, NULL) ORDER BY id; - QUERY PLAN --------------------------------------------------------------------------------- + QUERY PLAN +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Foreign Scan on in_null_test.tnull Output: id - Filter: (tnull.xn <> ALL ('{1,NULL}'::integer[])) - Remote SQL: SELECT id, xn FROM in_null_test.tnull ORDER BY id ASC NULLS LAST -(4 rows) + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((CASE WHEN xn IS NULL AND notEmpty([1,NULL]) THEN NULL WHEN countEqual([1,NULL], xn) > 0 THEN false WHEN countEqual([1,NULL], NULL) > 0 THEN NULL ELSE true END)) ORDER BY id ASC NULLS LAST +(3 rows) SELECT id FROM tnull WHERE xn NOT IN (1, NULL) ORDER BY id; id @@ -131,13 +136,12 @@ SELECT id FROM tnull WHERE xn NOT IN (1, NULL) ORDER BY id; -- NOT over IN is the same thing arriving unrewritten EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE NOT (xn IN (1, NULL)) ORDER BY id; - QUERY PLAN --------------------------------------------------------------------------------- + QUERY PLAN +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Foreign Scan on in_null_test.tnull Output: id - Filter: (tnull.xn <> ALL ('{1,NULL}'::integer[])) - Remote SQL: SELECT id, xn FROM in_null_test.tnull ORDER BY id ASC NULLS LAST -(4 rows) + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((CASE WHEN xn IS NULL AND notEmpty([1,NULL]) THEN NULL WHEN countEqual([1,NULL], xn) > 0 THEN false WHEN countEqual([1,NULL], NULL) > 0 THEN NULL ELSE true END)) ORDER BY id ASC NULLS LAST +(3 rows) SELECT id FROM tnull WHERE NOT (xn IN (1, NULL)) ORDER BY id; id @@ -163,14 +167,15 @@ SELECT id FROM tnull WHERE NOT (xn IN (1, 500)) ORDER BY id; -- 3. Non-constant arrays: the has()/countEqual() deparse forms -- ============================================================ -- has() matches a NULL probe against a NULL element where Postgres computes --- NULL, so = ANY needs a provably non-NULL probe; xp is declared NOT NULL +-- NULL, so = ANY uses the cheap has() form only with a provably non-NULL +-- probe; xp is declared NOT NULL EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xp = ANY(ARRAY[xn, 5]) ORDER BY id; - QUERY PLAN ------------------------------------------------------------------------------------------------------- + QUERY PLAN +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Foreign Scan on in_null_test.tnull Output: id - Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((has([xn, 5],xp))) ORDER BY id ASC NULLS LAST + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((CASE WHEN xp IS NULL AND notEmpty([xn, 5]) THEN NULL WHEN countEqual([xn, 5], xp) > 0 THEN true WHEN countEqual([xn, 5], NULL) > 0 THEN NULL ELSE false END)) ORDER BY id ASC NULLS LAST (3 rows) SELECT id FROM tnull WHERE xp = ANY(ARRAY[xn, 5]) ORDER BY id; @@ -181,16 +186,16 @@ SELECT id FROM tnull WHERE xp = ANY(ARRAY[xn, 5]) ORDER BY id; (2 rows) -- A nullable probe against an array that may hold a NULL would turn --- Postgres's NULL (id 3: NULL = ANY({NULL,5})) into TRUE: must stay local +-- Postgres's NULL (id 3: NULL = ANY({NULL,5})) into TRUE via has(): the +-- guarded CASE ships instead, computing the real NULL at runtime EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xn = ANY(ARRAY[xn, 5]) ORDER BY id; - QUERY PLAN --------------------------------------------------------------------------------- + QUERY PLAN +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Foreign Scan on in_null_test.tnull Output: id - Filter: (tnull.xn = ANY (ARRAY[tnull.xn, 5])) - Remote SQL: SELECT id, xn FROM in_null_test.tnull ORDER BY id ASC NULLS LAST -(4 rows) + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((CASE WHEN xn IS NULL AND notEmpty([xn, 5]) THEN NULL WHEN countEqual([xn, 5], xn) > 0 THEN true WHEN countEqual([xn, 5], NULL) > 0 THEN NULL ELSE false END)) ORDER BY id ASC NULLS LAST +(3 rows) SELECT id FROM tnull WHERE xn = ANY(ARRAY[xn, 5]) ORDER BY id; id @@ -199,31 +204,69 @@ SELECT id FROM tnull WHERE xn = ANY(ARRAY[xn, 5]) ORDER BY id; 2 (2 rows) +-- Basic arithmetic over provably non-NULL inputs is itself provably +-- non-NULL (it returns a value or raises; see never_null_opfuncs), so an +-- expression element doesn't cost the cheap form... +EXPLAIN (VERBOSE, COSTS OFF) +SELECT id FROM tnull WHERE xp = ANY(ARRAY[xp + 0, 500]) ORDER BY id; + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Foreign Scan on in_null_test.tnull + Output: id + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((has([(xp + 0), 500], xp))) ORDER BY id ASC NULLS LAST +(3 rows) + +SELECT id FROM tnull WHERE xp = ANY(ARRAY[xp + 0, 500]) ORDER BY id; + id +---- + 1 + 2 + 3 +(3 rows) + +-- ...while the same arithmetic over a nullable input proves nothing: +-- guarded (id 3: 3 = ANY({NULL,500}) is NULL, dropped either way) +EXPLAIN (VERBOSE, COSTS OFF) +SELECT id FROM tnull WHERE xp = ANY(ARRAY[xn + 0, 500]) ORDER BY id; + QUERY PLAN +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + Foreign Scan on in_null_test.tnull + Output: id + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((CASE WHEN xp IS NULL AND notEmpty([(xn + 0), 500]) THEN NULL WHEN countEqual([(xn + 0), 500], xp) > 0 THEN true WHEN countEqual([(xn + 0), 500], NULL) > 0 THEN NULL ELSE false END)) ORDER BY id ASC NULLS LAST +(3 rows) + +SELECT id FROM tnull WHERE xp = ANY(ARRAY[xn + 0, 500]) ORDER BY id; + id +---- + 1 + 2 +(2 rows) + -- countEqual() = 0 turns any NULL involved into TRUE (id 3: 3 <> ALL of --- {NULL,5} is NULL on Postgres): <> ALL needs both sides provably non-NULL +-- {NULL,5} is NULL on Postgres): <> ALL uses the cheap countEqual() form +-- only with both sides provably non-NULL EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xp <> ALL(ARRAY[xn, 5]) ORDER BY id; - QUERY PLAN ------------------------------------------------------------------------------------- + QUERY PLAN +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Foreign Scan on in_null_test.tnull Output: id - Filter: (tnull.xp <> ALL (ARRAY[tnull.xn, 5])) - Remote SQL: SELECT id, xn, xp FROM in_null_test.tnull ORDER BY id ASC NULLS LAST -(4 rows) + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((CASE WHEN xp IS NULL AND notEmpty([xn, 5]) THEN NULL WHEN countEqual([xn, 5], xp) > 0 THEN false WHEN countEqual([xn, 5], NULL) > 0 THEN NULL ELSE true END)) ORDER BY id ASC NULLS LAST +(3 rows) SELECT id FROM tnull WHERE xp <> ALL(ARRAY[xn, 5]) ORDER BY id; id ---- (0 rows) --- ...and with both sides proven, it ships +-- ...and with both sides proven, the cheap form ships EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xp <> ALL(ARRAY[xp, 500]) ORDER BY id; - QUERY PLAN -------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +-------------------------------------------------------------------------------------------------------------------- Foreign Scan on in_null_test.tnull Output: id - Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((countEqual([xp, 500],xp) = 0)) ORDER BY id ASC NULLS LAST + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((countEqual([xp, 500], xp) = 0)) ORDER BY id ASC NULLS LAST (3 rows) SELECT id FROM tnull WHERE xp <> ALL(ARRAY[xp, 500]) ORDER BY id; @@ -231,17 +274,34 @@ SELECT id FROM tnull WHERE xp <> ALL(ARRAY[xp, 500]) ORDER BY id; ---- (0 rows) --- <> ANY has no correct deparse (not has() computes <> ALL: it would return --- no rows here, where one differing element makes every row qualify): local +-- `!=` is just `<>` by the time the parser hands it to us (transformed at +-- parse analysis, before any FDW code sees the tree); confirm it deparses +-- identically rather than falling through chfdw_is_equal_op's string match +EXPLAIN (VERBOSE, COSTS OFF) +SELECT id FROM tnull WHERE xp != ALL(ARRAY[xp, 500]) ORDER BY id; + QUERY PLAN +-------------------------------------------------------------------------------------------------------------------- + Foreign Scan on in_null_test.tnull + Output: id + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((countEqual([xp, 500], xp) = 0)) ORDER BY id ASC NULLS LAST +(3 rows) + +SELECT id FROM tnull WHERE xp != ALL(ARRAY[xp, 500]) ORDER BY id; + id +---- +(0 rows) + +-- <> ANY deparses by counting elements rather than has() (which would +-- compute <> ALL instead): the cheap form ships with both sides proven +-- non-NULL EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xp <> ANY(ARRAY[xp, 500]) ORDER BY id; - QUERY PLAN --------------------------------------------------------------------------------- + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------ Foreign Scan on in_null_test.tnull Output: id - Filter: (tnull.xp <> ANY (ARRAY[tnull.xp, 500])) - Remote SQL: SELECT id, xp FROM in_null_test.tnull ORDER BY id ASC NULLS LAST -(4 rows) + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((countEqual([xp, 500], xp) < length([xp, 500]))) ORDER BY id ASC NULLS LAST +(3 rows) SELECT id FROM tnull WHERE xp <> ANY(ARRAY[xp, 500]) ORDER BY id; id @@ -251,14 +311,94 @@ SELECT id FROM tnull WHERE xp <> ANY(ARRAY[xp, 500]) ORDER BY id; 3 (3 rows) --- = ALL ships with both sides proven non-NULL... +-- Neither side is proven non-NULL here, but this still ships: a guarded +-- CASE checks at runtime whether the probe or an array element is NULL and +-- returns a genuine NULL there instead of requiring a plan-time proof +EXPLAIN (VERBOSE, COSTS OFF) +SELECT id FROM tnull WHERE xn <> ANY(ARRAY[xn, 500]) ORDER BY id; + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ + Foreign Scan on in_null_test.tnull + Output: id + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((CASE WHEN xn IS NULL AND notEmpty([xn, 500]) THEN NULL WHEN (length([xn, 500]) - countEqual([xn, 500], xn) - countEqual([xn, 500], NULL)) > 0 THEN true WHEN countEqual([xn, 500], NULL) > 0 THEN NULL ELSE false END)) ORDER BY id ASC NULLS LAST +(3 rows) + +SELECT id FROM tnull WHERE xn <> ANY(ARRAY[xn, 500]) ORDER BY id; + id +---- + 1 + 2 +(2 rows) + +-- A provably empty array is exact via the cheap form regardless of the +-- probe's nullability (has()/countEqual() degenerate correctly on an +-- empty array by arithmetic): id 3's NULL probe joins ids 1/2 under the +-- same FALSE flag, since <> ANY(empty) is FALSE for every probe on both +-- systems, NULL probe included +EXPLAIN (VERBOSE, COSTS OFF) +SELECT xn <> ANY(ARRAY[]::int[]) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + Foreign Scan + Output: ((xn <> ANY ('{}'::integer[]))), (count(*)) + Relations: Aggregate on (tnull) + Remote SQL: SELECT (countEqual([], xn) < length([])), count(*) FROM in_null_test.tnull GROUP BY ((countEqual([], xn) < length([]))) ORDER BY (countEqual([], xn) < length([])) ASC NULLS LAST +(4 rows) + +SELECT xn <> ANY(ARRAY[]::int[]) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + flag | count +------+------- + f | 3 +(1 row) + +-- A non-constant array can be empty at *runtime*, and Postgres resolves an +-- empty array before even looking at the probe: id 3's NULL = ANY('{}') is +-- FALSE, not NULL, so it must group with id 2. The guarded CASE answers +-- NULL for a NULL probe only against a non-empty array and falls through +-- to the empty-array answer otherwise +EXPLAIN (VERBOSE, COSTS OFF) +SELECT xn = ANY(arr) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + Foreign Scan + Output: ((xn = ANY (arr))), (count(*)) + Relations: Aggregate on (tnull) + Remote SQL: SELECT (CASE WHEN xn IS NULL AND notEmpty(arr) THEN NULL WHEN countEqual(arr, xn) > 0 THEN true WHEN countEqual(arr, NULL) > 0 THEN NULL ELSE false END), count(*) FROM in_null_test.tnull GROUP BY ((CASE WHEN xn IS NULL AND notEmpty(arr) THEN NULL WHEN countEqual(arr, xn) > 0 THEN true WHEN countEqual(arr, NULL) > 0 THEN NULL ELSE false END)) ORDER BY (CASE WHEN xn IS NULL AND notEmpty(arr) THEN NULL WHEN countEqual(arr, xn) > 0 THEN true WHEN countEqual(arr, NULL) > 0 THEN NULL ELSE false END) ASC NULLS LAST +(4 rows) + +SELECT xn = ANY(arr) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + flag | count +------+------- + f | 2 + t | 1 +(2 rows) + +-- ...and the ALL side of the same coin: id 3's NULL <> ALL('{}') is TRUE +EXPLAIN (VERBOSE, COSTS OFF) +SELECT xn <> ALL(arr) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + Foreign Scan + Output: ((xn <> ALL (arr))), (count(*)) + Relations: Aggregate on (tnull) + Remote SQL: SELECT (CASE WHEN xn IS NULL AND notEmpty(arr) THEN NULL WHEN countEqual(arr, xn) > 0 THEN false WHEN countEqual(arr, NULL) > 0 THEN NULL ELSE true END), count(*) FROM in_null_test.tnull GROUP BY ((CASE WHEN xn IS NULL AND notEmpty(arr) THEN NULL WHEN countEqual(arr, xn) > 0 THEN false WHEN countEqual(arr, NULL) > 0 THEN NULL ELSE true END)) ORDER BY (CASE WHEN xn IS NULL AND notEmpty(arr) THEN NULL WHEN countEqual(arr, xn) > 0 THEN false WHEN countEqual(arr, NULL) > 0 THEN NULL ELSE true END) ASC NULLS LAST +(4 rows) + +SELECT xn <> ALL(arr) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + flag | count +------+------- + f | 1 + t | 2 +(2 rows) + +-- = ALL ships via the cheap form with both sides proven non-NULL... EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xp = ALL(ARRAY[xp]) ORDER BY id; - QUERY PLAN -------------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +-------------------------------------------------------------------------------------------------------------------------- Foreign Scan on in_null_test.tnull Output: id - Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((countEqual([xp],xp) = length([xp]))) ORDER BY id ASC NULLS LAST + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((countEqual([xp], xp) = length([xp]))) ORDER BY id ASC NULLS LAST (3 rows) SELECT id FROM tnull WHERE xp = ALL(ARRAY[xp]) ORDER BY id; @@ -270,16 +410,16 @@ SELECT id FROM tnull WHERE xp = ALL(ARRAY[xp]) ORDER BY id; (3 rows) -- ...but countEqual() counts NULL as equal to NULL (id 3 would come back --- where Postgres computes NULL = ALL({NULL}) as NULL): local +-- TRUE where Postgres computes NULL = ALL({NULL}) as NULL): the guarded +-- CASE ships instead EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xn = ALL(ARRAY[xn]) ORDER BY id; - QUERY PLAN --------------------------------------------------------------------------------- + QUERY PLAN +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Foreign Scan on in_null_test.tnull Output: id - Filter: (tnull.xn = ALL (ARRAY[tnull.xn])) - Remote SQL: SELECT id, xn FROM in_null_test.tnull ORDER BY id ASC NULLS LAST -(4 rows) + Remote SQL: SELECT id FROM in_null_test.tnull WHERE ((CASE WHEN xn IS NULL AND notEmpty([xn]) THEN NULL WHEN (length([xn]) - countEqual([xn], xn) - countEqual([xn], NULL)) > 0 THEN false WHEN countEqual([xn], NULL) > 0 THEN NULL ELSE true END)) ORDER BY id ASC NULLS LAST +(3 rows) SELECT id FROM tnull WHERE xn = ALL(ARRAY[xn]) ORDER BY id; id @@ -289,11 +429,9 @@ SELECT id FROM tnull WHERE xn = ALL(ARRAY[xn]) ORDER BY id; (2 rows) -- ============================================================ --- 4. Value contexts observe the difference even without NOT +-- 4. Value contexts no longer observe any divergence -- ============================================================ --- Grouping on the IN result exposes FALSE-vs-NULL directly: Postgres groups --- id 2 and id 3 together under NULL; ClickHouse would put id 2 under FALSE. --- A NULL-free list is exact and ships; a NULL in the list keeps it local. +-- A NULL-free list needs no guard: the native IN is already exact EXPLAIN (VERBOSE, COSTS OFF) SELECT xn IN (1, 500) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; QUERY PLAN @@ -312,20 +450,19 @@ SELECT xn IN (1, 500) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; | 1 (3 rows) +-- A NULL in the list previously forced this local, since the value +-- position demands exactness the native IN can't give; the guarded CASE +-- now ships instead, returning a genuine NULL so id 3 lands in the same +-- group Postgres would put it in EXPLAIN (VERBOSE, COSTS OFF) SELECT xn IN (1, NULL) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; - QUERY PLAN ----------------------------------------------------------------- - Sort + QUERY PLAN +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + Foreign Scan Output: ((xn = ANY ('{1,NULL}'::integer[]))), (count(*)) - Sort Key: ((tnull.xn = ANY ('{1,NULL}'::integer[]))) - -> HashAggregate - Output: ((xn = ANY ('{1,NULL}'::integer[]))), count(*) - Group Key: (tnull.xn = ANY ('{1,NULL}'::integer[])) - -> Foreign Scan on in_null_test.tnull - Output: (xn = ANY ('{1,NULL}'::integer[])) - Remote SQL: SELECT xn FROM in_null_test.tnull -(9 rows) + Relations: Aggregate on (tnull) + Remote SQL: SELECT (CASE WHEN xn IS NULL AND notEmpty([1,NULL]) THEN NULL WHEN countEqual([1,NULL], xn) > 0 THEN true WHEN countEqual([1,NULL], NULL) > 0 THEN NULL ELSE false END), count(*) FROM in_null_test.tnull GROUP BY ((CASE WHEN xn IS NULL AND notEmpty([1,NULL]) THEN NULL WHEN countEqual([1,NULL], xn) > 0 THEN true WHEN countEqual([1,NULL], NULL) > 0 THEN NULL ELSE false END)) ORDER BY (CASE WHEN xn IS NULL AND notEmpty([1,NULL]) THEN NULL WHEN countEqual([1,NULL], xn) > 0 THEN true WHEN countEqual([1,NULL], NULL) > 0 THEN NULL ELSE false END) ASC NULLS LAST +(4 rows) SELECT xn IN (1, NULL) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; flag | count @@ -334,15 +471,34 @@ SELECT xn IN (1, NULL) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; | 2 (2 rows) +-- <> ANY ships the same way, aggregation included, despite neither side +-- being proven non-NULL +EXPLAIN (VERBOSE, COSTS OFF) +SELECT xn <> ANY(ARRAY[xn, 500]) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + Foreign Scan + Output: ((xn <> ANY (ARRAY[xn, 500]))), (count(*)) + Relations: Aggregate on (tnull) + Remote SQL: SELECT (CASE WHEN xn IS NULL AND notEmpty([xn, 500]) THEN NULL WHEN (length([xn, 500]) - countEqual([xn, 500], xn) - countEqual([xn, 500], NULL)) > 0 THEN true WHEN countEqual([xn, 500], NULL) > 0 THEN NULL ELSE false END), count(*) FROM in_null_test.tnull GROUP BY ((CASE WHEN xn IS NULL AND notEmpty([xn, 500]) THEN NULL WHEN (length([xn, 500]) - countEqual([xn, 500], xn) - countEqual([xn, 500], NULL)) > 0 THEN true WHEN countEqual([xn, 500], NULL) > 0 THEN NULL ELSE false END)) ORDER BY (CASE WHEN xn IS NULL AND notEmpty([xn, 500]) THEN NULL WHEN (length([xn, 500]) - countEqual([xn, 500], xn) - countEqual([xn, 500], NULL)) > 0 THEN true WHEN countEqual([xn, 500], NULL) > 0 THEN NULL ELSE false END) ASC NULLS LAST +(4 rows) + +SELECT xn <> ANY(ARRAY[xn, 500]) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + flag | count +------+------- + t | 2 + | 1 +(2 rows) + -- An aggregate FILTER only qualifies rows, so it keeps truth-context rules EXPLAIN (VERBOSE, COSTS OFF) SELECT count(*) FILTER (WHERE xn IN (1, NULL)) FROM tnull; - QUERY PLAN --------------------------------------------------------------------------------- + QUERY PLAN +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Foreign Scan Output: (count(*) FILTER (WHERE (xn = ANY ('{1,NULL}'::integer[])))) Relations: Aggregate on (tnull) - Remote SQL: SELECT countIf((((xn IN (1,NULL))) > 0)) FROM in_null_test.tnull + Remote SQL: SELECT countIf((((CASE WHEN xn IS NULL AND notEmpty([1,NULL]) THEN NULL WHEN countEqual([1,NULL], xn) > 0 THEN true WHEN countEqual([1,NULL], NULL) > 0 THEN NULL ELSE false END)) > 0)) FROM in_null_test.tnull (4 rows) SELECT count(*) FILTER (WHERE xn IN (1, NULL)) FROM tnull; diff --git a/test/sql/in_null_semantics.sql b/test/sql/in_null_semantics.sql index d6444b90..ece252da 100644 --- a/test/sql/in_null_semantics.sql +++ b/test/sql/in_null_semantics.sql @@ -2,12 +2,14 @@ -- -- ClickHouse evaluates IN under two-valued logic: a probe that finds no -- match yields 0 even when the searched set contains a NULL, where Postgres --- computes NULL; The ClickHouse has()/countEqual() functions also match a NULL probe as --- if it were an ordinary value. A FALSE-for-NULL substitution is invisible --- while the result only qualifies rows (WHERE/JOIN/HAVING through AND/OR), --- but observable under NOT and anywhere the boolean's value is consumed. --- These tests ensure the shapes deparse.c ships and that the rows returned --- match local (Postgres) semantics either way. +-- computes NULL, which changes negation semantics (NOT(NULL) -> NULL). +-- The has()/countEqual() functions also match a NULL probe against a NULL +-- element as if it were an ordinary value. Every shape below ships regardless: +-- deparse.c uses the cheap native form when it can prove neither side can be +-- NULL, and a guarded CASE otherwise that checks at runtime instead, computing +-- Postgres's exact three-valued answer (NULL, not FALSE, or worse, TRUE) +-- in every context. These tests ensure both forms deparse correctly and that +-- the rows returned match local (Postgres) semantics either way. CREATE SERVER in_null_svr FOREIGN DATA WRAPPER clickhouse_fdw OPTIONS(dbname 'in_null_test', driver 'binary'); CREATE USER MAPPING FOR CURRENT_USER SERVER in_null_svr; @@ -15,12 +17,14 @@ CREATE USER MAPPING FOR CURRENT_USER SERVER in_null_svr; SELECT clickhouse_raw_query('DROP DATABASE IF EXISTS in_null_test'); SELECT clickhouse_raw_query('CREATE DATABASE in_null_test'); --- xn imports as a plain (nullable) int column, xp as int NOT NULL +-- xn imports as a plain (nullable) int column, xp as int NOT NULL, and arr +-- as int[]; its elements' nullability is never provable at plan time SELECT clickhouse_raw_query('CREATE TABLE in_null_test.tnull - (id Int32, xn Nullable(Int32), xp Int32) + (id Int32, xn Nullable(Int32), xp Int32, arr Array(Int32)) ENGINE = MergeTree ORDER BY id'); SELECT clickhouse_raw_query($$ - INSERT INTO in_null_test.tnull VALUES (1, 1, 1), (2, 2, 2), (3, NULL, 3) + INSERT INTO in_null_test.tnull + VALUES (1, 1, 1, [1, 500]), (2, 2, 2, [500]), (3, NULL, 3, []) $$); CREATE SCHEMA in_null_test; @@ -32,21 +36,21 @@ SET SESSION search_path = in_null_test,public; \d tnull -- ============================================================ --- 1. IN over a constant list: pushable in a WHERE condition +-- 1. IN over a constant list: always pushable in a WHERE condition -- ============================================================ --- NULL-free list is exactly equivalent +-- NULL-free list is exactly equivalent to the native IN EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xn IN (1, 500) ORDER BY id; SELECT id FROM tnull WHERE xn IN (1, 500) ORDER BY id; --- A NULL in the list only pulls results toward FALSE, which a WHERE treats --- like the NULL Postgres computes: still pushable +-- A NULL in the list can't be proven absent, so this ships via a guarded +-- CASE instead of the native IN, computing the real answer at runtime EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xn IN (1, NULL) ORDER BY id; SELECT id FROM tnull WHERE xn IN (1, NULL) ORDER BY id; -- ============================================================ --- 2. NOT IN over a constant list: pushable only when NULL-free +-- 2. NOT IN over a constant list: always pushable, native when NULL-free -- ============================================================ -- ClickHouse's native NOT IN yields NULL for a NULL probe, so a NULL-free -- list is exact: id 3 (xn IS NULL) is dropped on both systems @@ -54,8 +58,9 @@ EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xn NOT IN (1, 500) ORDER BY id; SELECT id FROM tnull WHERE xn NOT IN (1, 500) ORDER BY id; --- With a NULL in the list Postgres returns no rows (every result is NULL --- or FALSE) where ClickHouse would return id 2: must stay local +-- With a NULL in the list, ClickHouse's native NOT IN would return id 2 +-- where Postgres returns no rows (every result is NULL or FALSE): the +-- guarded CASE computes Postgres's answer instead of using the native form EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xn NOT IN (1, NULL) ORDER BY id; SELECT id FROM tnull WHERE xn NOT IN (1, NULL) ORDER BY id; @@ -73,59 +78,122 @@ SELECT id FROM tnull WHERE NOT (xn IN (1, 500)) ORDER BY id; -- 3. Non-constant arrays: the has()/countEqual() deparse forms -- ============================================================ -- has() matches a NULL probe against a NULL element where Postgres computes --- NULL, so = ANY needs a provably non-NULL probe; xp is declared NOT NULL +-- NULL, so = ANY uses the cheap has() form only with a provably non-NULL +-- probe; xp is declared NOT NULL EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xp = ANY(ARRAY[xn, 5]) ORDER BY id; SELECT id FROM tnull WHERE xp = ANY(ARRAY[xn, 5]) ORDER BY id; -- A nullable probe against an array that may hold a NULL would turn --- Postgres's NULL (id 3: NULL = ANY({NULL,5})) into TRUE: must stay local +-- Postgres's NULL (id 3: NULL = ANY({NULL,5})) into TRUE via has(): the +-- guarded CASE ships instead, computing the real NULL at runtime EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xn = ANY(ARRAY[xn, 5]) ORDER BY id; SELECT id FROM tnull WHERE xn = ANY(ARRAY[xn, 5]) ORDER BY id; +-- Basic arithmetic over provably non-NULL inputs is itself provably +-- non-NULL (it returns a value or raises; see never_null_opfuncs), so an +-- expression element doesn't cost the cheap form... +EXPLAIN (VERBOSE, COSTS OFF) +SELECT id FROM tnull WHERE xp = ANY(ARRAY[xp + 0, 500]) ORDER BY id; +SELECT id FROM tnull WHERE xp = ANY(ARRAY[xp + 0, 500]) ORDER BY id; + +-- ...while the same arithmetic over a nullable input proves nothing: +-- guarded (id 3: 3 = ANY({NULL,500}) is NULL, dropped either way) +EXPLAIN (VERBOSE, COSTS OFF) +SELECT id FROM tnull WHERE xp = ANY(ARRAY[xn + 0, 500]) ORDER BY id; +SELECT id FROM tnull WHERE xp = ANY(ARRAY[xn + 0, 500]) ORDER BY id; + -- countEqual() = 0 turns any NULL involved into TRUE (id 3: 3 <> ALL of --- {NULL,5} is NULL on Postgres): <> ALL needs both sides provably non-NULL +-- {NULL,5} is NULL on Postgres): <> ALL uses the cheap countEqual() form +-- only with both sides provably non-NULL EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xp <> ALL(ARRAY[xn, 5]) ORDER BY id; SELECT id FROM tnull WHERE xp <> ALL(ARRAY[xn, 5]) ORDER BY id; --- ...and with both sides proven, it ships +-- ...and with both sides proven, the cheap form ships EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xp <> ALL(ARRAY[xp, 500]) ORDER BY id; SELECT id FROM tnull WHERE xp <> ALL(ARRAY[xp, 500]) ORDER BY id; --- <> ANY has no correct deparse (not has() computes <> ALL: it would return --- no rows here, where one differing element makes every row qualify): local +-- `!=` is just `<>` by the time the parser hands it to us (transformed at +-- parse analysis, before any FDW code sees the tree); confirm it deparses +-- identically rather than falling through chfdw_is_equal_op's string match +EXPLAIN (VERBOSE, COSTS OFF) +SELECT id FROM tnull WHERE xp != ALL(ARRAY[xp, 500]) ORDER BY id; +SELECT id FROM tnull WHERE xp != ALL(ARRAY[xp, 500]) ORDER BY id; + +-- <> ANY deparses by counting elements rather than has() (which would +-- compute <> ALL instead): the cheap form ships with both sides proven +-- non-NULL EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xp <> ANY(ARRAY[xp, 500]) ORDER BY id; SELECT id FROM tnull WHERE xp <> ANY(ARRAY[xp, 500]) ORDER BY id; --- = ALL ships with both sides proven non-NULL... +-- Neither side is proven non-NULL here, but this still ships: a guarded +-- CASE checks at runtime whether the probe or an array element is NULL and +-- returns a genuine NULL there instead of requiring a plan-time proof +EXPLAIN (VERBOSE, COSTS OFF) +SELECT id FROM tnull WHERE xn <> ANY(ARRAY[xn, 500]) ORDER BY id; +SELECT id FROM tnull WHERE xn <> ANY(ARRAY[xn, 500]) ORDER BY id; + +-- A provably empty array is exact via the cheap form regardless of the +-- probe's nullability (has()/countEqual() degenerate correctly on an +-- empty array by arithmetic): id 3's NULL probe joins ids 1/2 under the +-- same FALSE flag, since <> ANY(empty) is FALSE for every probe on both +-- systems, NULL probe included +EXPLAIN (VERBOSE, COSTS OFF) +SELECT xn <> ANY(ARRAY[]::int[]) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; +SELECT xn <> ANY(ARRAY[]::int[]) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + +-- A non-constant array can be empty at *runtime*, and Postgres resolves an +-- empty array before even looking at the probe: id 3's NULL = ANY('{}') is +-- FALSE, not NULL, so it must group with id 2. The guarded CASE answers +-- NULL for a NULL probe only against a non-empty array and falls through +-- to the empty-array answer otherwise +EXPLAIN (VERBOSE, COSTS OFF) +SELECT xn = ANY(arr) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; +SELECT xn = ANY(arr) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + +-- ...and the ALL side of the same coin: id 3's NULL <> ALL('{}') is TRUE +EXPLAIN (VERBOSE, COSTS OFF) +SELECT xn <> ALL(arr) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; +SELECT xn <> ALL(arr) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + +-- = ALL ships via the cheap form with both sides proven non-NULL... EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xp = ALL(ARRAY[xp]) ORDER BY id; SELECT id FROM tnull WHERE xp = ALL(ARRAY[xp]) ORDER BY id; -- ...but countEqual() counts NULL as equal to NULL (id 3 would come back --- where Postgres computes NULL = ALL({NULL}) as NULL): local +-- TRUE where Postgres computes NULL = ALL({NULL}) as NULL): the guarded +-- CASE ships instead EXPLAIN (VERBOSE, COSTS OFF) SELECT id FROM tnull WHERE xn = ALL(ARRAY[xn]) ORDER BY id; SELECT id FROM tnull WHERE xn = ALL(ARRAY[xn]) ORDER BY id; -- ============================================================ --- 4. Value contexts observe the difference even without NOT +-- 4. Value contexts no longer observe any divergence -- ============================================================ --- Grouping on the IN result exposes FALSE-vs-NULL directly: Postgres groups --- id 2 and id 3 together under NULL; ClickHouse would put id 2 under FALSE. --- A NULL-free list is exact and ships; a NULL in the list keeps it local. +-- A NULL-free list needs no guard: the native IN is already exact EXPLAIN (VERBOSE, COSTS OFF) SELECT xn IN (1, 500) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; SELECT xn IN (1, 500) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; +-- A NULL in the list previously forced this local, since the value +-- position demands exactness the native IN can't give; the guarded CASE +-- now ships instead, returning a genuine NULL so id 3 lands in the same +-- group Postgres would put it in EXPLAIN (VERBOSE, COSTS OFF) SELECT xn IN (1, NULL) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; SELECT xn IN (1, NULL) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; +-- <> ANY ships the same way, aggregation included, despite neither side +-- being proven non-NULL +EXPLAIN (VERBOSE, COSTS OFF) +SELECT xn <> ANY(ARRAY[xn, 500]) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; +SELECT xn <> ANY(ARRAY[xn, 500]) AS flag, count(*) FROM tnull GROUP BY flag ORDER BY flag; + -- An aggregate FILTER only qualifies rows, so it keeps truth-context rules EXPLAIN (VERBOSE, COSTS OFF) SELECT count(*) FILTER (WHERE xn IN (1, NULL)) FROM tnull;