From 361c7fa79ba2bd1c49b75d8d1b61f0d02e00765a Mon Sep 17 00:00:00 2001 From: os-warren Date: Sat, 5 Sep 2026 19:16:40 +0000 Subject: [PATCH 1/3] =?UTF-8?q?fix(service-analytics):=20compile=20the=20$?= =?UTF-8?q?icontains=20ASCII=20fold=20per=20dialect=20=E2=80=94=20translat?= =?UTF-8?q?e()=20is=20not=20a=20SQLite=20function?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three of this package's SQL compilers spelled the #6520 fold as `translate(col, 'ABC…', 'abc…')` on EVERY dialect. `translate()` is PostgreSQL/Oracle; SQLite has none, so on a SQLite datasource an analytics `where` carrying `$icontains` — and an ADR-0021 D-C read scope carrying it — compiled a statement the engine refuses to parse. Measured on sql.js 1.14.1 (SQLite 3.49.1, the engine driver-sqlite-wasm runs): `SELECT translate('ABC','ABC','abc')` answers `no such function: translate`. `$icontains` now goes through `text-match-sql.ts`'s per-dialect construct table with one `fold` flag, set on that operator alone: - sqlite → `lower(col) GLOB lower(?)`, ASCII-only there (`lower('CAFÉ')` is `cafÉ`), which is the #4706 Q1 = A boundary rather than an approximation of it. - postgres → `translate()`, byte-identical to before. Never broken. - unknown → `translate()`, byte-identical to before. The residue keeps the shape it had; note this diverges from driver-sql, whose unknown arm folds with LOWER(), and neither face claims the other's. - mysql → the nested-REPLACE fold over CAST(… AS BINARY), matching driver-sql. TEXT ONLY — no MySQL server is provisionable here. The `sql` keyword field on `ObjectQLStrategy`'s LIKE_SQL_OPS lost its last reader in this move and is removed: a dead field named `sql` beside a compiler invites exactly the misreading this defect was. #15684's `$icontains` control asserted "the fold arm still emits translate() on every dialect". That was a PROXY for the property it protected — the two text families must not collapse onto one path — and this change makes the fold dialect-DEPENDENT by design, so the proxy no longer states the property. It is re-aimed rather than deleted or loosened: `$icontains` and `$contains` must now compile to DIFFERENT text on each dialect, and `$contains` must carry no fold in any of its three spellings. That discriminates against the collapse in both directions where dialect-invariance discriminated against one. Fixes #15780 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XpTx2tbq3pZRYAdoGt6E6Y --- .../analytics-icontains-per-dialect-fold.md | 15 + .../__tests__/icontains-dialect-sql.test.ts | 345 ++++++++++++++++++ .../text-operator-case-exactness.test.ts | 87 +++-- .../service-analytics/src/like-pattern.ts | 68 ++-- .../service-analytics/src/read-scope-sql.ts | 32 +- .../src/strategies/native-sql-strategy.ts | 57 +-- .../src/strategies/objectql-strategy.ts | 61 ++-- .../service-analytics/src/text-match-sql.ts | 117 +++++- 8 files changed, 644 insertions(+), 138 deletions(-) create mode 100644 .changeset/analytics-icontains-per-dialect-fold.md create mode 100644 packages/services/service-analytics/src/__tests__/icontains-dialect-sql.test.ts diff --git a/.changeset/analytics-icontains-per-dialect-fold.md b/.changeset/analytics-icontains-per-dialect-fold.md new file mode 100644 index 0000000000..e265cb04a6 --- /dev/null +++ b/.changeset/analytics-icontains-per-dialect-fold.md @@ -0,0 +1,15 @@ +--- +"@objectstack/service-analytics": patch +--- + +Analytics `$icontains` no longer compiles a `translate()` call on SQLite and MySQL, where that function does not exist and the statement failed to parse. + +`$icontains` folds ASCII case on both sides of the comparison (#4706 Q1 = A). All three of this package's SQL compilers — the query's own `where` (`NativeSQLStrategy.buildFilterClause`), the ADR-0021 D-C read scope (`compileScopedFilterToSql`) and the `ObjectQLStrategy` echo of that statement — spelled that fold as `translate(col, 'ABC…', 'abc…')` on **every** dialect. `translate()` is PostgreSQL/Oracle; SQLite has none. Measured on sql.js 1.14.1 (SQLite 3.49.1, the engine `driver-sqlite-wasm` runs), `SELECT translate('ABC','ABC','abc')` answers `no such function: translate` — so this was not a filter that returned the wrong rows, it was a statement the engine refused. On a SQLite datasource, an analytics `where` carrying `$icontains` and an **RLS read scope** carrying it were both unusable. + +The fold is now chosen per dialect, on the same construct table the case-exact text family already used, reached through one `fold` flag: + +- **SQLite** — `lower(col) GLOB lower(?)`. SQLite's `lower()` is ASCII-only (measured: `lower('CAFÉ')` is `cafÉ`), so this is the ruled fold rather than an approximation of it, and it runs. +- **PostgreSQL** and the `unknown` residue (a host that wires no dialect hook) — `translate()`, **unchanged**. These arms were never broken, so the emitted SQL and its bound parameters are byte-identical to before. +- **MySQL** — the nested-`REPLACE` fold over `CAST(… AS BINARY)`, matching what `driver-sql` emits for the same operator. Asserted as text only; no MySQL server is provisionable in the container that wrote this, so that cell is a declared skip, not a claimed pass. + +`$icontains` and the case-sensitive `$contains` family remain two separate constructs on every dialect — collapsing them would give `$contains` back the case fold #4706 Q2 = A took away from it. A host that answers no dialect keeps exactly the behaviour it had. diff --git a/packages/services/service-analytics/src/__tests__/icontains-dialect-sql.test.ts b/packages/services/service-analytics/src/__tests__/icontains-dialect-sql.test.ts new file mode 100644 index 0000000000..ccba607c47 --- /dev/null +++ b/packages/services/service-analytics/src/__tests__/icontains-dialect-sql.test.ts @@ -0,0 +1,345 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#15780] `$icontains` on this package's three SQL compilers, per DIALECT — + * `NativeSQLStrategy.buildFilterClause` (the query's own `where`), + * `compileScopedFilterToSql` (`read-scope-sql.ts`, the ADR-0021 D-C read scope) + * and the `ObjectQLStrategy` echo of that statement. + * + * ## The defect this closes + * + * All three compiled the #6520 fold as `translate(col, 'ABC…', 'abc…') LIKE + * translate(?, …) ESCAPE ?` on EVERY dialect. `translate()` is a + * PostgreSQL/Oracle function; SQLite has none. So this is not the #15684 + * failure mode restated — that family answered the WRONG ROWS. This one does + * not answer at all: measured on sql.js 1.14.1 (SQLite 3.49.1, the engine + * `driver-sqlite-wasm` runs), `SELECT translate('ABC','ABC','abc')` is `no such + * function: translate`, so the statement fails to PARSE. Asserted below as the + * `unknown`-dialect control rather than argued. + * + * `read-scope-sql.ts` is the compiler that makes this more than a broken chart: + * an RLS read scope carrying `$icontains` over a SQLite datasource compiles to + * a statement the engine refuses, so the scope cannot be evaluated at all. + * + * ## Why the pins are per dialect, and what each cell's evidence is + * + * - **sqlite → EXECUTED.** Every assertion below that names row ids ran on + * sql.js. The construct is `lower(col) GLOB lower(?)`, and SQLite's + * `lower()` is ASCII-only — measured, `lower('CAFÉ')` is `cafÉ` — which is + * exactly the #4706 Q1 = A boundary this operator is ruled to. + * - **postgres → byte-identical text.** `translate()` is correct there today, + * so the correct diff is no diff: the emitted SQL and params are asserted + * verbatim against the pre-#15780 bytes, for all three compilers. + * - **`unknown` (no hook wired) → byte-identical text, ALSO `translate()`.** + * The residue keeps the shape it has always had. It is not an endorsement: + * it is what still runs on the dialects nothing here models (mssql, oracle, + * which do have `translate()`), and falling back to `LOWER()` for it would + * silently restore the Unicode fold #4706 Q1 = A took away. Note this + * DIVERGES from `driver-sql`'s `unknown` arm, which folds with `LOWER()` — + * each side keeps its own pre-existing residue, and neither claims the + * other's. + * - **mysql → NOT MEASURED.** The nested-`REPLACE`-over-`CAST(… AS BINARY)` + * arm is asserted as TEXT only. No MySQL server is provisionable in this + * container — the same declared skip `driver-sql`'s #6518 suite and this + * package's #15684 suite both record, not a claimed pass. + * + * ## What holds this package's table to `driver-sql`'s + * + * The last describe runs the SAME `FILTER_TEXT_CASES` rows through a real + * `SqliteWasmDriver` (a devDependency, never a runtime one) and requires the + * same row sets from both faces — the anti-drift mechanism #15684 established, + * extended to the fold row. A third hand-copy of the table is the thing to + * refuse. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import type { Cube, DriverOptions, FilterCondition } from '@objectstack/spec/data'; +import { FILTER_TEXT_CASES, FILTER_TEXT_ROWS } from '@objectstack/spec/data'; +import type { AnalyticsQuery } from '@objectstack/spec/contracts'; +import { SqliteWasmDriver } from '@objectstack/driver-sqlite-wasm'; + +import { NativeSQLStrategy } from '../strategies/native-sql-strategy.js'; +import { ObjectQLStrategy } from '../strategies/objectql-strategy.js'; +import { compileScopedFilterToSql } from '../read-scope-sql.js'; +import type { DatasetScopedStrategyContext } from '../strategies/types.js'; + +/** + * The shared table's rows that aim `$icontains` at the TEXT column. + * + * Taken from `FILTER_TEXT_CASES` rather than restated, so this suite answers + * the same standard the five drivers answer, and the count is asserted below: a + * row added upstream must reach this face too. + */ +const NAME_ICONTAINS = FILTER_TEXT_CASES.filter( + (c): c is Extract => { + if (c.expectRejection === true) return false; + const entries = Object.entries(c.filter as Record); + if (entries.length !== 1) return false; + const [field, predicate] = entries[0]; + if (field !== 'name' || typeof predicate !== 'object' || predicate === null) return false; + return Object.keys(predicate as Record)[0] === '$icontains'; + }, +); + +const CUBE: Cube = { + name: 'texts', + title: 'Texts', + sql: 'rows', + measures: { total: { name: 'total', label: 'Total', type: 'count', sql: '*' } }, + dimensions: { + id: { name: 'id', label: 'Id', type: 'string', sql: 'id' }, + name: { name: 'name', label: 'Name', type: 'string', sql: 'name' }, + }, + public: false, +} as unknown as Cube; + +const query = (where: unknown): AnalyticsQuery => + ({ cube: 'texts', measures: ['total'], dimensions: ['id'], timezone: 'UTC', where }) as AnalyticsQuery; + +/** Point sql.js at the `.wasm` shipped inside its own package (Node-safe). */ +async function locateWasm(): Promise<((file: string) => string) | undefined> { + try { + const { createRequire } = await import('node:module'); + const require = createRequire(import.meta.url); + const pkgJsonPath = require.resolve('sql.js/package.json'); + const { dirname, join } = await import('node:path'); + return (file: string) => join(dirname(pkgJsonPath), 'dist', file); + } catch { + return undefined; + } +} + +const ctxFor = (dialect?: string): DatasetScopedStrategyContext => + ({ + getCube: (name: string) => (name === 'texts' ? CUBE : undefined), + queryCapabilities: () => ({ nativeSql: true, objectqlAggregate: true, inMemory: false }), + ...(dialect ? { sqlDialect: () => dialect } : {}), + }) as DatasetScopedStrategyContext; + +const TRANSLATE_FOLD = + "translate(name, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz')"; + +describe('[#15780] the compiled TEXT, per dialect — all three compilers', () => { + const nativeSql = async (where: unknown, dialect?: string) => + new NativeSQLStrategy().generateSql(query(where), ctxFor(dialect)); + const echoSql = async (where: unknown, dialect?: string) => + new ObjectQLStrategy().generateSql(query(where), ctxFor(dialect)); + + it('postgres and a host that wired NO hook keep the pre-#15780 bytes exactly', async () => { + // The non-regression half, for the two dialects that were already correct. + // Asserted verbatim, not by shape: `translate()` is right on Postgres, and + // the `unknown` residue must keep the only fold that still runs there. + for (const dialect of [undefined, 'postgres'] as const) { + const out = await nativeSql({ name: { $icontains: 'acme' } }, dialect); + expect(out.sql, String(dialect)).toContain( + `WHERE ${TRANSLATE_FOLD} LIKE translate($1, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') ESCAPE $2`, + ); + expect(out.params, String(dialect)).toEqual(['%acme%', '\\']); + + const echo = await echoSql({ name: { $icontains: 'acme' } }, dialect); + expect(echo.sql, `echo ${String(dialect)}`).toContain( + `${TRANSLATE_FOLD} LIKE translate($1, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') ESCAPE $2`, + ); + expect(echo.params, `echo ${String(dialect)}`).toEqual(['%acme%', '\\']); + } + + const noHook = compileScopedFilterToSql({ name: { $icontains: 'acme' } } as FilterCondition, 't'); + expect(noHook).toEqual({ + sql: + `translate("t"."name", 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') LIKE ` + + `translate(?, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') ESCAPE ?`, + params: ['%acme%', '\\'], + }); + expect( + compileScopedFilterToSql({ name: { $icontains: 'acme' } } as FilterCondition, 't', { dialect: 'postgres' }), + ).toEqual(noHook); + }); + + it('sqlite compiles lower() over GLOB — one bound value, no ESCAPE clause', async () => { + const out = await nativeSql({ name: { $icontains: 'acme' } }, 'sqlite'); + expect(out.sql).toContain('WHERE lower(name) GLOB lower($1)'); + expect(out.sql).not.toMatch(/translate\(/); + expect(out.sql).not.toMatch(/ESCAPE/); + expect(out.params).toEqual(['*acme*']); + + const echo = await echoSql({ name: { $icontains: 'acme' } }, 'sqlite'); + expect(echo.sql).toContain('lower(name) GLOB lower($1)'); + expect(echo.sql).not.toMatch(/translate\(/); + expect(echo.params).toEqual(['*acme*']); + + expect(compileScopedFilterToSql({ name: { $icontains: 'acme' } } as FilterCondition, 't', { dialect: 'sqlite' })) + .toEqual({ sql: 'lower("t"."name") GLOB lower(?)', params: ['*acme*'] }); + }); + + it('mysql compiles the nested-REPLACE binary fold — TEXT ONLY, NOT MEASURED on a server', async () => { + // Character for character `driver-sql`'s `mysqlAsciiLowerBinary`: 26 nested + // REPLACEs over `CAST(… AS BINARY)`, so the fold is ASCII-only and the + // comparison is byte-wise whatever the column's collation says. + const out = await nativeSql({ name: { $icontains: 'acme' } }, 'mysql'); + expect(out.sql).toContain("REPLACE(REPLACE(CAST(name AS BINARY), 'A', 'a')"); + expect(out.sql).toContain("REPLACE(REPLACE(CAST($1 AS BINARY), 'A', 'a')"); + expect(out.sql).toContain(' LIKE '); + expect(out.sql).toContain('ESCAPE $2'); + expect(out.sql).not.toMatch(/translate\(/); + expect(out.params).toEqual(['%acme%', '\\']); + // The last REPLACE of the chain is `Z` → `z`, on both sides. + expect(out.sql).toContain("'Z', 'z')"); + + const scoped = compileScopedFilterToSql( + { name: { $icontains: 'acme' } } as FilterCondition, + 't', + { dialect: 'mysql' }, + ); + expect(scoped.sql).toContain('CAST("t"."name" AS BINARY)'); + expect(scoped.sql).toContain('CAST(? AS BINARY)'); + expect(scoped.params).toEqual(['%acme%', '\\']); + }); + + it('the case-EXACT family is untouched by this change — #15684\'s constructs, unchanged', async () => { + // The mirror of the control #15684 wrote for `$icontains`: the two families + // must not collapse onto one path. If the fold ever leaks into these rows, + // `$contains` gets back the case-insensitivity #4706 Q2 = A took away. + const sqlite = await nativeSql({ name: { $contains: 'acme' } }, 'sqlite'); + expect(sqlite.sql).toContain('WHERE name GLOB $1'); + expect(sqlite.sql).not.toMatch(/lower\(/); + expect(sqlite.params).toEqual(['*acme*']); + const pg = await nativeSql({ name: { $contains: 'acme' } }, 'postgres'); + expect(pg.sql).toContain('WHERE name LIKE $1 ESCAPE $2'); + expect(pg.sql).not.toMatch(/translate\(/); + }); +}); + +describe('[#15780] the three compilers, EXECUTED on a real SQLite engine', () => { + let db: any; + let sqliteCtx: DatasetScopedStrategyContext; + /** The same host with no dialect hook — the pre-#15780 compiler, still reachable. */ + let unawareCtx: DatasetScopedStrategyContext; + + const run = (sql: string, params: unknown[]): string[] => { + const stmt = db.prepare(sql.replace(/\$\d+/g, '?')); + stmt.bind(params as any[]); + const rows: Record[] = []; + while (stmt.step()) rows.push(stmt.getAsObject()); + stmt.free(); + return rows.map((r) => String(r.id)).sort((a, b) => a.localeCompare(b)); + }; + + beforeAll(async () => { + const mod: any = await import('sql.js'); + const initSqlJs = mod.default ?? mod; + const locateFile = await locateWasm(); + const SQL = await initSqlJs(locateFile ? { locateFile } : undefined); + db = new SQL.Database(); + db.run(`CREATE TABLE "rows" ("id" TEXT PRIMARY KEY, "name" TEXT);`); + const insert = db.prepare(`INSERT INTO "rows" ("id","name") VALUES (?,?)`); + for (const r of FILTER_TEXT_ROWS) insert.run([r.id, r.name]); + insert.free(); + + const base = { + getCube: (name: string) => (name === 'texts' ? CUBE : undefined), + queryCapabilities: () => ({ nativeSql: true, objectqlAggregate: true, inMemory: false }), + }; + unawareCtx = { ...base } as DatasetScopedStrategyContext; + sqliteCtx = { ...base, sqlDialect: () => 'sqlite' } as DatasetScopedStrategyContext; + }); + + afterAll(() => { + db?.close(); + }); + + const executedIds = async (where: unknown, ctx: DatasetScopedStrategyContext): Promise => { + const { sql, params } = await new NativeSQLStrategy().generateSql(query(where), ctx); + return run(sql, params); + }; + + it('the engine has no translate(), and lower() there folds ASCII ONLY', () => { + // The card's three measurements, re-executed here so the arms below rest on + // this engine's answers rather than on a quoted table. + expect(() => db.exec(`SELECT translate('ABC','ABC','abc')`)).toThrow(/no such function: translate/); + expect(db.exec(`SELECT ('acme' GLOB 'ac*')`)[0].values[0][0]).toBe(1); + expect(db.exec(`SELECT lower('CAFÉ')`)[0].values[0][0]).toBe('cafÉ'); + }); + + it('the fixture is the shared nine rows, and $icontains is eight of the table\'s cases', () => { + expect(run('SELECT "id" FROM "rows"', [])).toEqual(['1', '2', '3', '4', '5', '6', '7', '8', '9']); + expect(NAME_ICONTAINS.map((c) => c.name)).toEqual([ + '$icontains matches an upper-case row from a lower-case comparand', + '$icontains matches a lower-case row from an upper-case comparand', + 'ASCII-only: a lower-case non-ASCII comparand does NOT match its upper-case row', + 'ASCII-only: an upper-case non-ASCII comparand does NOT match its lower-case row', + '$icontains treats % as a literal character, not a LIKE wildcard', + 'icontains (the infix/view spelling, #8934) lowers to $icontains — % stays a LITERAL through that door too', + '$icontains treats _ as a literal character, not a single-character wildcard', + '$icontains treats . as a literal character, not a regex metacharacter', + ]); + }); + + it('the defect, still reachable through a host that answers no dialect — so these pins discriminate', async () => { + // Before-red, stated as the engine's own refusal rather than argued: the + // `unknown` residue still emits `translate()`, and this engine cannot parse + // it. This is the control that keeps every green assertion above honest. + const { sql, params } = await new NativeSQLStrategy().generateSql( + query({ name: { $icontains: 'acme' } }), + unawareCtx, + ); + expect(sql).toContain('translate('); + expect(() => run(sql, params)).toThrow(/no such function: translate/); + }); + + it('NativeSQLStrategy answers the shared table\'s $icontains rows', async () => { + for (const c of NAME_ICONTAINS) { + expect(await executedIds(c.filter, sqliteCtx), c.name).toEqual([...c.expected]); + } + }); + + it('the READ SCOPE answers them too — the compiler where a wrong row set is over-reach', async () => { + for (const c of NAME_ICONTAINS) { + const scoped = { + ...sqliteCtx, + getReadScope: (object: string) => (object === 'rows' ? (c.filter as FilterCondition) : null), + } as DatasetScopedStrategyContext; + const { sql, params } = await new NativeSQLStrategy().generateSql(query(undefined), scoped); + expect(sql, c.name).toMatch(/GLOB/); + expect(sql, c.name).not.toMatch(/translate\(/); + expect(run(sql, params), c.name).toEqual([...c.expected]); + } + }); + + it('the ObjectQL echo prints the statement the native compiler runs', async () => { + for (const c of NAME_ICONTAINS) { + const echo = await new ObjectQLStrategy().generateSql(query(c.filter), sqliteCtx); + const native = await new NativeSQLStrategy().generateSql(query(c.filter), sqliteCtx); + expect(echo.sql, c.name).toMatch(/GLOB/); + expect(echo.sql, c.name).not.toMatch(/translate\(/); + expect(echo.params, c.name).toEqual(native.params); + expect(run(echo.sql, echo.params), `echo of ${c.name}`).toEqual([...c.expected]); + } + }); +}); + +/** + * The anti-drift pin: this package's construct table against the DRIVER's, run + * rather than compared — #15684's mechanism, extended to the fold row. + */ +describe('[#15780] this package and driver-sql answer the shared $icontains rows alike on SQLite', () => { + let driver: SqliteWasmDriver; + const BYPASS: DriverOptions = { bypassTenantAudit: true }; + + beforeAll(async () => { + driver = new SqliteWasmDriver({ filename: ':memory:' }); + await driver.initObjects([{ name: 'txt', fields: { name: { type: 'string' }, score: { type: 'number' } } }]); + for (const row of FILTER_TEXT_ROWS) await driver.create('txt', { ...row }, BYPASS); + }); + + afterAll(async () => { + await driver.disconnect(); + }); + + it('every $icontains row: same ids from the driver and from this package\'s compilers', async () => { + for (const c of NAME_ICONTAINS) { + const rows = await driver.find('txt', { where: c.filter as FilterCondition }, BYPASS); + const driverIds = rows.map((r) => String(r.id)).sort((a, b) => a.localeCompare(b)); + expect(driverIds, `driver: ${c.name}`).toEqual([...c.expected]); + } + }); +}); diff --git a/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts b/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts index 472530ee5b..27c09a5645 100644 --- a/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts +++ b/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts @@ -50,18 +50,40 @@ * — on the same engine, and requires the same row sets from both. A third * hand-copy of the table anywhere is the thing to refuse. * - * ## What is deliberately NOT changed + * ## What #15684 deliberately did not change, and what #15780 then did * - * `$icontains` (#6520) keeps its own construct on every dialect: it folds BOTH - * sides through `asciiLowerSqlExpr`, and the assertions below pin that the - * emitted text is untouched by the dialect. That fold is `translate()`, which - * SQLite does not have (measured: `no such function: translate` on sql.js - * 1.14.1) — so those statements are pinned as TEXT and are NOT executed here. - * That is a separate defect, filed as #15780, and this suite's `$icontains` - * assertions are the control that must stay unchanged while it is open. - * Escaping (#5567) is likewise unchanged for every `LIKE` arm; the GLOB arm - * brings its OWN escaped character class (`*`, `?`, `[`), which is why the - * second fixture below exists. + * As written, this suite pinned `$icontains` (#6520) as a CONTROL: it kept its + * own construct on every dialect, folding both sides through + * `asciiLowerSqlExpr`, and the assertion below pinned that its emitted text was + * untouched by the dialect. The property that control protected is the one that + * still matters: **the two text families must not collapse onto one path**, or + * `$contains` gets back the fold #4706 Q2 = A took away from it. The pin was + * "the fold arm still emits `translate()` on every dialect" only because, while + * #15684's scope was the case-exact four, dialect-invariance was a cheap + * PROXY for family-separation — the two happened to coincide. + * + * They stopped coinciding. That same `translate()` is a PostgreSQL/Oracle + * function SQLite does not have (measured: `no such function: translate` on + * sql.js 1.14.1), so `$icontains` did not merely go unpinned on SQLite — it + * failed to PARSE there. That was #15780, and closing it moved `$icontains` + * onto this same per-dialect table with a `fold` flag, which makes its emitted + * text dialect-DEPENDENT by design. The old assertion could then only be read + * two ways: as a defect it must go red for, or as a statement of the property, + * which it no longer is. + * + * ⛔ It was therefore neither deleted nor loosened — it was re-aimed at the + * property itself, which is now pinned DIRECTLY and more tightly than the proxy + * ever did (`$icontains and the case-EXACT family stay two constructs…` + * below): on each dialect, `$icontains` and `$contains` must compile to + * DIFFERENT text, and `$contains` must carry no fold. That discriminates + * against the collapse in both directions, where dialect-invariance only + * discriminated against one. The row sets that make it more than a text + * comparison are executed in `icontains-dialect-sql.test.ts`, which owns the + * `$icontains` half of the family from here on. + * + * Escaping (#5567) is unchanged for every `LIKE` arm; the GLOB arm brings its + * OWN escaped character class (`*`, `?`, `[`), which is why the second fixture + * below exists. */ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; @@ -234,22 +256,37 @@ describe('[#15684] the compiled TEXT, per dialect', () => { .toEqual({ sql: 'CAST("t"."name" AS BINARY) LIKE CAST(? AS BINARY) ESCAPE ?', params: ['%acme%', '\\'] }); }); - it('$icontains is untouched by the dialect — the fold arm still emits translate() on both sides', async () => { - // The control that must stay green. #6520's construct is case-INSENSITIVE - // by ruling, so it never wants the case-exact table; if a future edit routes - // it through `text-match-sql.ts`, the `$contains` family gets back the fold - // #4706 Q2 = A took away from it and this line reds first. + it('$icontains and the case-EXACT family stay two constructs on EVERY dialect', async () => { + // [#15684 → #15780] The control, re-aimed. This assertion used to read "the + // fold arm still emits translate() on every dialect", which was a PROXY for + // the property below and stopped being one when #15780 gave `$icontains` a + // per-dialect fold of its own. See this file's header for the full reading. + // + // The property is family SEPARATION: `$icontains` folds (#4706 Q1 = A) and + // `$contains` must not (#4706 Q2 = A). A collapse in EITHER direction reds + // here — the fold leaking onto `$contains`, or `$contains`' bare construct + // being handed to `$icontains`. for (const dialect of [undefined, 'sqlite', 'postgres', 'mysql'] as const) { - const out = await nativeSql({ name: { $icontains: 'acme' } }, dialect); - expect(out.sql, String(dialect)).toContain( - "WHERE translate(name, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') LIKE translate($1,", - ); - expect(out.sql, String(dialect)).toContain('ESCAPE $2'); - expect(out.params, String(dialect)).toEqual(['%acme%', '\\']); + const icontains = await nativeSql({ name: { $icontains: 'acme' } }, dialect); + const contains = await nativeSql({ name: { $contains: 'acme' } }, dialect); + expect(icontains.sql, String(dialect)).not.toBe(contains.sql); + // `$contains` carries NO fold, in any of the three spellings a fold has. + expect(contains.sql, String(dialect)).not.toMatch(/translate\(|lower\(|REPLACE\(/); + // …and `$icontains` carries exactly one of them, per dialect. + const FOLD_PER_DIALECT: Record = { + undefined: /translate\(name, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'/, + postgres: /translate\(name, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'/, + sqlite: /lower\(name\) GLOB lower\(\$1\)/, + mysql: /REPLACE\(CAST\(name AS BINARY\), 'A', 'a'\)/, + }; + expect(icontains.sql, String(dialect)).toMatch(FOLD_PER_DIALECT[String(dialect)]); } - expect( - compileScopedFilterToSql({ name: { $icontains: 'acme' } } as FilterCondition, 't', { dialect: 'sqlite' }).params, - ).toEqual(['%acme%', '\\']); + // The read scope, the compiler where the collapse would be an ADR-0021 + // over-reach rather than a wrong chart, holds the same separation. + const filterOf = (op: '$contains' | '$icontains') => + compileScopedFilterToSql({ name: { [op]: 'acme' } } as FilterCondition, 't', { dialect: 'sqlite' }); + expect(filterOf('$icontains').sql).not.toBe(filterOf('$contains').sql); + expect(filterOf('$contains').sql).not.toMatch(/lower\(/); }); }); diff --git a/packages/services/service-analytics/src/like-pattern.ts b/packages/services/service-analytics/src/like-pattern.ts index a18d8ecef7..bc44b26752 100644 --- a/packages/services/service-analytics/src/like-pattern.ts +++ b/packages/services/service-analytics/src/like-pattern.ts @@ -108,16 +108,18 @@ * * - {@link likePattern} / {@link escapeLikePattern} build the LIKE PATTERN, * and are used by the `LIKE` arms — Postgres, MySQL, and the `unknown` - * residue — plus `$icontains` on every dialect. + * residue — for BOTH text families. * - Which KEYWORD those patterns hang off, and whether a GLOB pattern with a * different escaped character class is built instead, is * `text-match-sql.ts`'s answer. ⛔ Do not re-derive it here. * - * `$icontains` IS implemented here since #6520, and it is a separate construct - * rather than a flag on the family above: it folds ASCII case on BOTH sides via - * {@link asciiLowerSqlExpr}, while the `$contains` family stays case-EXACT. The - * two must not be collapsed — a shared "case-insensitive" path would give the - * `$contains` family the fold the ruling took away from it. + * [#15780] `$icontains` (#6520) goes through that same table, and its fold is a + * per-dialect construct too — {@link asciiLowerSqlExpr} is only the Postgres / + * `unknown` arm of it. What the two families share is the TABLE and the + * escaping; what separates them is one `fold` flag set on the `$icontains` row + * alone. ⛔ The two must never be collapsed into one case-insensitive path — + * that would give the `$contains` family back the fold #4706 Q2 = A took away + * from it, which is the failure `text-operator-case-exactness.test.ts` guards. * * ## `String(value)` is safe here because nothing unrenderable reaches it (#5234) * @@ -177,9 +179,17 @@ export function likePattern(shape: LikeShape, value: unknown): string { return shape === 'starts' ? `${escaped}%` : shape === 'ends' ? `%${escaped}` : `%${escaped}%`; } -/** `A`..`Z` and the `a`..`z` they fold onto — the #4706 Q1 = A domain, as data. */ -const ASCII_UPPER_LETTERS = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'; -const ASCII_LOWER_LETTERS = 'abcdefghijklmnopqrstuvwxyz'; +/** + * `A`..`Z` and the `a`..`z` they fold onto — the #4706 Q1 = A domain, as data. + * + * [#15780] EXPORTED, because the fold is now chosen per dialect and two of the + * three arms are built from this domain rather than from `translate()`: + * `text-match-sql.ts`'s MySQL arm nests one `REPLACE` per letter. The domain + * itself stays here, in one copy — a second 26-character literal anywhere is + * how the Postgres arm and the MySQL arm start folding different alphabets. + */ +export const ASCII_UPPER_LETTERS = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'; +export const ASCII_LOWER_LETTERS = 'abcdefghijklmnopqrstuvwxyz'; /** * [#6520] Wrap a SQL expression in `$icontains`' ASCII-ONLY case fold. @@ -199,24 +209,28 @@ const ASCII_LOWER_LETTERS = 'abcdefghijklmnopqrstuvwxyz'; * alone, so it is the fold the ruling names rather than the one the database * happens to offer. * - * ## The dialect this assumes, stated so it can go red rather than stale - * - * Postgres. `translate()` is Postgres/Oracle; SQLite has no such function — and - * [#15684] MEASURED that these compilers' statements do reach SQLite, so the - * warning this paragraph used to write in the conditional is now a live defect, - * filed as #15780: on sql.js 1.14.1, `SELECT translate('ABC','ABC','abc')` - * answers `no such function: translate`, so an `$icontains` in an analytics - * `where` or in an RLS read scope over a SQLite datasource does not merely - * over-match — it fails to parse. - * - * ⛔ Do NOT close that by falling back to `LOWER()`, which would silently - * restore the Unicode fold this function exists to avoid. The remedy is one - * more arm on `text-match-sql.ts`'s per-dialect table, whose shapes - * `driver-sql`'s `textMatchPredicate` already carries: `lower(col) GLOB - * lower(?)` on SQLite (measured ASCII-only there — `lower('CAFÉ')` is - * `cafÉ`), nested `REPLACE` over `CAST(… AS BINARY)` on MySQL. #15684 - * deliberately did not build it: its scope was the case-EXACT four, and its - * suite pins THIS expression as the control that must stay unchanged. + * ## The dialect this arm is FOR — no longer the dialect it assumes + * + * Postgres, and the `unknown` residue. `translate()` is Postgres/Oracle and + * SQLite has no such function, so while this expression was emitted on EVERY + * dialect it did not merely over-match on SQLite — it failed to PARSE + * (`no such function: translate` on sql.js 1.14.1). That was #15780, and it is + * closed: this function is now ONE arm of `text-match-sql.ts`'s per-dialect + * table ({@link textMatchPredicateSql}, `fold: true`), reached on `postgres` + * and on `unknown`, while SQLite gets `lower(col) GLOB lower(?)` and MySQL the + * nested-`REPLACE` binary fold. + * + * ⛔ Do NOT "simplify" this to `LOWER()`, on any arm. Postgres' `LOWER()` is + * locale-aware and would silently restore the Unicode fold #4706 Q1 = A rules + * out; SQLite's `lower()` is ASCII-only, which is why the SQLite arm may use it + * and this one may not. That asymmetry is the whole reason the fold is chosen + * per dialect rather than written once. + * + * ⛔ Nor is `unknown` free to adopt `driver-sql`'s residue: that face folds + * `unknown` with `LOWER()` because `LOWER()` is the shape it emitted before + * #6518. This face's pre-existing shape is `translate()`, so `translate()` is + * what its residue keeps. Each side keeps its own, and neither claims the + * other's — an `unknown` dialect is by definition one nothing here measured. * * The caller must apply it to BOTH sides of the comparison. Folding only the * comparand compares a folded needle against a raw column and matches just the diff --git a/packages/services/service-analytics/src/read-scope-sql.ts b/packages/services/service-analytics/src/read-scope-sql.ts index 94f6bd3084..5c8c6787b8 100644 --- a/packages/services/service-analytics/src/read-scope-sql.ts +++ b/packages/services/service-analytics/src/read-scope-sql.ts @@ -2,7 +2,7 @@ import type { FilterCondition } from '@objectstack/spec/data'; import type { RegisteredErrorCode } from '@objectstack/spec/api'; -import { likePattern, LIKE_ESCAPE_CHAR, asciiLowerSqlExpr, type LikeShape } from './like-pattern.js'; +import { type LikeShape } from './like-pattern.js'; import { textMatchPredicateSql, normalizeSqlDialect } from './text-match-sql.js'; import { textOperatorPolarity } from './non-text-column.js'; import { @@ -800,6 +800,7 @@ function textMatch( negate: boolean, params: unknown[], opts: ReadScopeCompileOptions, + fold = false, ): string { return textMatchPredicateSql({ dialect: normalizeSqlDialect(opts.dialect), @@ -807,6 +808,7 @@ function textMatch( shape, value: val, negate, + fold, bind: (v) => bind(params, v), }); } @@ -1331,19 +1333,25 @@ function compileOperator( * the rows already lower-case — and on a read scope that is a row set the * policy author never wrote, in the narrowing direction here but in the * WIDENING direction under a `$not`. + * + * [#15780] …and WHICH fold is the DIALECT's answer, exactly as the keyword + * is for the case-exact arms. This line used to spell its own binds and + * emit `translate()` unconditionally, on the reasoning that a + * case-INSENSITIVE operator never wants the per-dialect case-EXACT + * construct. The first half of that was right and the second half hid the + * defect: `translate()` is Postgres/Oracle, so on a SQLite datasource this + * read scope compiled to a statement the engine could not PARSE — an RLS + * policy that cannot be evaluated at all. It goes through + * {@link textMatch} now with `fold` set, which keeps `translate()` on + * Postgres and the `unknown` residue, emits `lower(col) GLOB lower(?)` on + * SQLite and the nested-`REPLACE` binary fold on MySQL. The `ESCAPE` + * binding is still never folded — the construct table owns that, and the + * SQLite arm has no `ESCAPE` clause to bind at all. */ - case '$icontains': { + case '$icontains': assertRenderableText(op, field, val); - const gated = textOverNonTextColumn(op, field, opts); - if (gated) return gated; - // The two binds are spelled out rather than taken from {@link textMatch}, - // because only the PATTERN placeholder is folded, the `ESCAPE` one must - // not be, and this operator is case-INSENSITIVE by ruling so it never - // wants the per-dialect case-exact construct. Left-to-right, so the - // values land in `params` in placeholder order. - const patternRef = asciiLowerSqlExpr(bind(params, likePattern('contains', val))); - return `${asciiLowerSqlExpr(col)} LIKE ${patternRef} ESCAPE ${bind(params, LIKE_ESCAPE_CHAR)}`; - } + return textOverNonTextColumn(op, field, opts) + ?? textMatch(col, 'contains', val, false, params, opts, true); // [#5298] NULL-safe: `NOT LIKE` is UNKNOWN for a NULL column, and "does not // contain" is true of a value that is not there. case '$notContains': diff --git a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts index ba8b6a7e07..731a64c3c2 100644 --- a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts @@ -15,7 +15,7 @@ import { findCrossFieldComparand, findUninterpretableTemporalMember } from '../c import { assertReadScopeCannotVacate, compileScopedFilterToSql } from '../read-scope-sql.js'; import { nonTextColumnResolver, textOperatorPolarity } from '../non-text-column.js'; import { datasetInvalidError, invalidMemberError } from '../dataset-refusal.js'; -import { likePattern, LIKE_ESCAPE_CHAR, asciiLowerSqlExpr, type LikeShape } from '../like-pattern.js'; +import { type LikeShape } from '../like-pattern.js'; import { textMatchPredicateSql, sqlDialectFor } from '../text-match-sql.js'; import { nextUtcCalendarDay } from '@objectstack/core'; @@ -1078,13 +1078,14 @@ export class NativeSQLStrategy implements AnalyticsStrategy { ): string | null { const opMap: Record = { equals: '=', notEquals: '!=', gt: '>', gte: '>=', lt: '<', lte: '<=', - // [#15684] For the case-EXACT four these entries are the OPERATOR GATE, - // not the emitted keyword: `text-match-sql.ts` picks `LIKE` or `GLOB` - // per dialect below. `$icontains` still emits the `LIKE` written here. + // [#15684 / #15780] For every text operator these entries are the + // OPERATOR GATE, not the emitted keyword: `text-match-sql.ts` picks + // `LIKE` or `GLOB` per dialect below. ⛔ Reading these five as the + // emitted SQL is exactly the mistake #15780 was — `$icontains` was the + // last row still emitting the keyword written here, together with a + // `translate()` fold SQLite cannot parse. contains: 'LIKE', notContains: 'NOT LIKE', startsWith: 'LIKE', endsWith: 'LIKE', - // [#6520] `$icontains` — `LIKE` like its neighbours; what separates it is - // the ASCII fold applied below, not the keyword. icontains: 'LIKE', }; /** @@ -1137,34 +1138,36 @@ export class NativeSQLStrategy implements AnalyticsStrategy { if (polarity && nonTextColumnResolver(ctx, target.object)?.(target.field)) { return polarity === 'negative' ? SQL_CONST_TRUE : SQL_CONST_FALSE; } - // [#6520] `$icontains` folds ASCII case on BOTH sides. Only this operator - // folds: the rest of the family is case-EXACT by ruling (#4706 Q2 = A), - // and `objectql-strategy.ts`'s echo of this statement carries the same - // `fold` flag on the same single row so the two keep describing one query. + // [#15684] The case-EXACT family picks its construct per DIALECT, because + // a plain `LIKE` folds ASCII case on SQLite and follows the collation on + // MySQL — admitting rows #4706 Q2 = A excludes. `sqlOp` above still gates + // the operator into this branch; which KEYWORD it becomes is + // `text-match-sql.ts`'s answer, not this table's, and the escaping travels + // with it (the GLOB arm escapes a different character class and binds no + // `ESCAPE`). A host that wired no dialect hook answers `'unknown'` and + // keeps the `LIKE` this line always emitted. // - // [#5567] Escaped pattern AND an explicit `ESCAPE`, bound together: the - // escaping alone would search for a literal backslash on SQLite (no - // default escape character there), the clause alone would change nothing. - if (operator === 'icontains') { - params.push(likePattern(shape, values[0])); - const patternRef = `$${params.length}`; - params.push(LIKE_ESCAPE_CHAR); - return `${asciiLowerSqlExpr(rawCol)} ${sqlOp} ${asciiLowerSqlExpr(patternRef)} ESCAPE $${params.length}`; - } - // [#15684] …and the case-EXACT family picks its construct per DIALECT, - // because a plain `LIKE` folds ASCII case on SQLite and follows the - // collation on MySQL — admitting rows #4706 Q2 = A excludes. `sqlOp` - // above still gates the operator into this branch; which KEYWORD it - // becomes is `text-match-sql.ts`'s answer, not this table's, and the - // escaping travels with it (the GLOB arm escapes a different character - // class and binds no `ESCAPE`). A host that wired no dialect hook - // answers `'unknown'` and keeps the `LIKE` this line always emitted. + // [#15780] …and `$icontains` (#6520) comes through the SAME call, with + // `fold` set on that operator ALONE — the rest of the family is + // case-EXACT by ruling, and a fold reaching them is the #4706 Q2 = A + // regression `text-operator-case-exactness.test.ts` guards. It has to be + // per-dialect for a DIFFERENT reason than its neighbours: the fold this + // line used to emit unconditionally was `translate()`, which SQLite does + // not have, so the statement failed to PARSE there rather than answering + // wrong rows. `objectql-strategy.ts`'s echo carries the same flag on the + // same single row, so the two keep describing one query. + // + // [#5567] Escaped pattern AND an explicit `ESCAPE`, bound together on + // every LIKE arm: the escaping alone would search for a literal backslash + // on SQLite (no default escape character there), the clause alone would + // change nothing. The GLOB arm binds neither — see the construct table. return textMatchPredicateSql({ dialect: sqlDialectFor(ctx, target.object), column: rawCol, shape, value: values[0], negate: operator === 'notContains', + fold: operator === 'icontains', bind: (v) => { params.push(v); return `$${params.length}`; }, }); } diff --git a/packages/services/service-analytics/src/strategies/objectql-strategy.ts b/packages/services/service-analytics/src/strategies/objectql-strategy.ts index f30793f418..44cf5a97e9 100644 --- a/packages/services/service-analytics/src/strategies/objectql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/objectql-strategy.ts @@ -19,7 +19,7 @@ import { findCrossFieldComparand, isFieldReference } from '../comparand-shape.js import { assertReadScopeCannotVacate, compileScopedFilterToSql } from '../read-scope-sql.js'; import { nonTextColumnResolver, textOperatorPolarity } from '../non-text-column.js'; import { invalidMemberError } from '../dataset-refusal.js'; -import { likePattern, LIKE_ESCAPE_CHAR, asciiLowerSqlExpr, type LikeShape } from '../like-pattern.js'; +import { type LikeShape } from '../like-pattern.js'; import { textMatchPredicateSql, sqlDialectFor } from '../text-match-sql.js'; import { nextUtcCalendarDay } from '@objectstack/core'; import { @@ -82,22 +82,26 @@ const SCALAR_SQL_OPS: Record = { * than the query it claims to reproduce whenever the comparand carried a `_` or * `%` — the #3601 / #3602 / #3650 failure this render block exists to prevent. */ -const LIKE_SQL_OPS: Record = { - // [#15684] `sql` is the keyword for the FOLDING row only. The four - // case-EXACT rows take their construct from the DIALECT — `text-match-sql.ts` - // emits `GLOB` on SQLite and `LIKE` over `CAST(… AS BINARY)` on MySQL, since - // a plain `LIKE` is case-exact on Postgres alone. `negate` is what survives - // that move: which POLARITY the row is, spelled once, so the construct table - // and not this one decides how the negation is written. - contains: { sql: 'LIKE', shape: 'contains' }, - notContains: { sql: 'NOT LIKE', shape: 'contains', negate: true }, - startsWith: { sql: 'LIKE', shape: 'starts' }, - endsWith: { sql: 'LIKE', shape: 'ends' }, - // [#6520] `$icontains`: the same escaped pattern and bound `ESCAPE` as its - // four case-EXACT neighbours, with `fold` adding the ASCII-only case fold to - // both sides of the comparison. The flag is on this row alone — the family - // above it is case-sensitive by ruling (#4706 Q2 = A). - icontains: { sql: 'LIKE', shape: 'contains', fold: true }, +const LIKE_SQL_OPS: Record = { + // [#15684 / #15780] This table carries NO keyword, deliberately. Every row's + // construct comes from the DIALECT (`text-match-sql.ts`): `GLOB` on SQLite, + // `LIKE` over `CAST(… AS BINARY)` on MySQL, plain `LIKE` on Postgres, since a + // plain `LIKE` is case-exact on Postgres alone. #15684 kept a `sql` field + // here for the FOLDING row, which was the last row still emitting a keyword + // written locally — together with the `translate()` fold that could not parse + // on SQLite. #15780 moved that row onto the table too, so the field had no + // reader left, and a dead field named `sql` sitting beside a compiler is an + // invitation to read it as the emitted keyword. + // + // What survives here is only what the construct table cannot derive from the + // operator name: which POLARITY the row is (`negate`) and whether it FOLDS + // (`fold`), each spelled once. `fold` is on `icontains` ALONE — the four + // above it are case-SENSITIVE by ruling (#4706 Q2 = A). + contains: { shape: 'contains' }, + notContains: { shape: 'contains', negate: true }, + startsWith: { shape: 'starts' }, + endsWith: { shape: 'ends' }, + icontains: { shape: 'contains', fold: true }, }; /** One cross-object grouping dimension planned for FK-expand (#3654). */ @@ -1246,24 +1250,23 @@ export class ObjectQLStrategy implements AnalyticsStrategy { // folding only the comparand compares a folded needle against a raw column // and returns just the rows that were already lower-case — a wrong row set // that looks like a working predicate. - if (like.fold) { - params.push(likePattern(like.shape, values[0])); - const patternRef = `$${params.length}`; - params.push(LIKE_ESCAPE_CHAR); - return `${asciiLowerSqlExpr(col)} ${like.sql} ${asciiLowerSqlExpr(patternRef)} ESCAPE $${params.length}`; - } - // [#15684] The case-EXACT four print what the DIALECT will run. Asked of - // the same hook `NativeSQLStrategy` asks, on the same target, so the echo - // and the executed statement stay one description — the whole reason this - // render block exists (#5333). A caller that handed no target and no - // context cannot be asked (the four-argument shape the operator-coverage - // suite drives), and keeps the `LIKE` it always got. + // [#15684 / #15780] Both text families print what the DIALECT will run — + // the case-EXACT four because `LIKE` is case-exact on Postgres alone, and + // `$icontains` because its fold was `translate()`, which SQLite does not + // have at all. Asked of the same hook `NativeSQLStrategy` asks, on the + // same target, so the echo and the executed statement stay one + // description — the whole reason this render block exists (#5333). An + // echo that printed a parseable statement while the engine refused the + // real one is that failure in its loudest form. A caller that handed no + // target and no context cannot be asked (the four-argument shape the + // operator-coverage suite drives), and keeps the `LIKE` it always got. return textMatchPredicateSql({ dialect: target && ctx ? sqlDialectFor(ctx, target.object) : 'unknown', column: col, shape: like.shape, value: values[0], negate: like.negate === true, + fold: like.fold === true, bind: (v) => { params.push(v); return `$${params.length}`; }, }); } diff --git a/packages/services/service-analytics/src/text-match-sql.ts b/packages/services/service-analytics/src/text-match-sql.ts index de55a654a6..56cb378e6d 100644 --- a/packages/services/service-analytics/src/text-match-sql.ts +++ b/packages/services/service-analytics/src/text-match-sql.ts @@ -1,10 +1,11 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. /** - * [#15684] The case-EXACT text family — `$contains` / `$notContains` / - * `$startsWith` / `$endsWith` — compiled per DIALECT, for this package's three - * SQL compilers (`read-scope-sql.ts`, `NativeSQLStrategy.buildFilterClause`, - * the `ObjectQLStrategy` echo of that statement). + * [#15684 / #15780] The text operators — the case-EXACT family (`$contains` / + * `$notContains` / `$startsWith` / `$endsWith`) and the case-INSENSITIVE + * `$icontains` — compiled per DIALECT, for this package's three SQL compilers + * (`read-scope-sql.ts`, `NativeSQLStrategy.buildFilterClause`, the + * `ObjectQLStrategy` echo of that statement). * * ## The defect this closes * @@ -89,17 +90,55 @@ * endorsement: it is the only answer that still RUNS, and it is the residue * this file's own suite names. * - * ## What is NOT here + * ## [#15780] `$icontains` is here too — one FLAG on the same table * - * `$icontains` (#6520) keeps its own construct in `like-pattern.ts` and is - * untouched by this file: it folds BOTH sides through `asciiLowerSqlExpr`, and - * collapsing the two families onto one path would hand the case-EXACT family - * the fold #4706 Q2 = A took away from it. Escaping (#5567) is likewise - * unchanged — {@link likePattern} still builds every LIKE-arm pattern, and the - * GLOB arm's own escaped class is a DIFFERENT one, not a shared regex. + * It arrived second, and for a different failure mode. `$icontains` (#6520) + * folds both sides with `asciiLowerSqlExpr` — `translate(col, 'ABC…', 'abc…')` + * — which is a PostgreSQL/Oracle function. SQLite has none, so where the + * case-EXACT four answered the WRONG ROWS on SQLite, this one did not answer at + * all: measured on sql.js 1.14.1, `SELECT translate('ABC','ABC','abc')` is `no + * such function: translate`, so the statement failed to PARSE. On + * `read-scope-sql.ts` that meant an RLS read scope carrying `$icontains` over a + * SQLite datasource could not be evaluated. + * + * The remedy is one FLAG ({@link TextMatchRequest.fold}), not a second table: + * the dialect question, the escaping and the placeholder plumbing are identical + * for both families, and only the fold's spelling differs per arm: + * + * - **`sqlite` → `lower()` around the GLOB arm.** ASCII-only there — + * measured, `lower('CAFÉ')` is `cafÉ` — which is the #4706 Q1 = A boundary + * executed rather than argued. + * - **`postgres` / `unknown` → `translate()`, unchanged.** These two arms + * were never broken; the bytes emitted before #15780 are the bytes emitted + * now. ⚠️ This DIVERGES from `driver-sql`, whose `unknown` arm folds with + * `LOWER()`: each face keeps the residue it already had, and neither claims + * the other's. + * - **`mysql` → the nested-`REPLACE` binary fold** + * ({@link mysqlAsciiLowerBinarySql}). NOT MEASURED — no MySQL server is + * provisionable in this container, so this cell is a declared skip, exactly + * as its case-exact neighbour above. + * + * ⛔ What must NOT be done is give the case-EXACT four this flag: that hands + * them back the fold #4706 Q2 = A took away. One `fold: true`, on the + * `$icontains` row of each compiler's operator table, and nowhere else — + * `text-operator-case-exactness.test.ts` and + * `icontains-dialect-sql.test.ts` pin both halves of that boundary. + * + * Escaping (#5567) is unchanged — {@link likePattern} still builds every + * LIKE-arm pattern, and the GLOB arm's own escaped class is a DIFFERENT one, + * not a shared regex. The fold composes with it rather than replacing it: on + * the SQLite arm `lower()` wraps an already-GLOB-escaped pattern, and `[`, `]`, + * `*` and `?` are not letters, so the escape survives the fold untouched. */ -import { likePattern, LIKE_ESCAPE_CHAR, type LikeShape } from './like-pattern.js'; +import { + likePattern, + LIKE_ESCAPE_CHAR, + asciiLowerSqlExpr, + ASCII_UPPER_LETTERS, + ASCII_LOWER_LETTERS, + type LikeShape, +} from './like-pattern.js'; import type { StrategyContext } from '@objectstack/spec/contracts'; import type { DatasetScopedStrategyContext } from './strategies/types.js'; @@ -184,7 +223,30 @@ export function globPattern(shape: LikeShape, value: unknown): string { */ export type TextMatchBind = (value: unknown) => string; -/** One case-EXACT text predicate, ready to splice into a WHERE clause. */ +/** + * [#15780] MySQL's ASCII-ONLY case fold, byte-wise: one nested `REPLACE` per + * letter over `CAST(… AS BINARY)`. + * + * Character for character `driver-sql`'s `mysqlAsciiLowerBinary`, and built + * from the ONE copy of the domain ({@link ASCII_UPPER_LETTERS}) rather than a + * second 26-character literal. + * + * Why not `LOWER()`, which MySQL does have: `LOWER()` there follows the + * collation and folds well beyond ASCII, so it would answer the Unicode fold + * #4706 Q1 = A rules out. Why not `translate()`: MySQL has no such function — + * the same reason this whole card exists, one dialect over. The `CAST(… + * AS BINARY)` underneath is what makes the REPLACE chain the WHOLE fold rather + * than a fold on top of the collation's own. + */ +function mysqlAsciiLowerBinarySql(expr: string): string { + let out = `CAST(${expr} AS BINARY)`; + for (let i = 0; i < ASCII_UPPER_LETTERS.length; i++) { + out = `REPLACE(${out}, '${ASCII_UPPER_LETTERS[i]}', '${ASCII_LOWER_LETTERS[i]}')`; + } + return out; +} + +/** One text predicate, ready to splice into a WHERE clause. */ export interface TextMatchRequest { /** The dialect that will execute the statement. */ dialect: AnalyticsSqlDialect; @@ -196,22 +258,37 @@ export interface TextMatchRequest { value: unknown; /** `$notContains` — the negated keyword, on whichever construct the arm picks. */ negate?: boolean; + /** + * [#15780] `$icontains` — apply the ASCII-ONLY case fold (#4706 Q1 = A) to + * BOTH sides of the comparison, in whatever spelling this dialect has one. + * + * Set on the `$icontains` row ALONE. The case-EXACT four are case-sensitive + * by ruling (#4706 Q2 = A) and must never reach an arm with this true. + */ + fold?: boolean; /** The caller's placeholder plumbing. */ bind: TextMatchBind; } /** - * The one place a case-EXACT text predicate becomes SQL in this package. + * The one place a text predicate becomes SQL in this package — both families. * - * `$icontains` does NOT come through here — see this file's header. + * `fold` picks between them; every other input is shared. See this file's + * header for why each cell is the construct it is. */ export function textMatchPredicateSql(req: TextMatchRequest): string { const { dialect, column, shape, value, bind } = req; const negate = req.negate === true; + const fold = req.fold === true; if (dialect === 'sqlite') { // GLOB takes no ESCAPE clause, so this arm binds ONE value, not two. - return `${column} ${negate ? 'NOT GLOB' : 'GLOB'} ${bind(globPattern(shape, value))}`; + // [#15780] The fold is SQLite's own `lower()`, which is ASCII-only — + // measured, `lower('CAFÉ')` is `cafÉ` — so it is exactly the #4706 Q1 = A + // domain rather than an approximation of it. Applied to BOTH sides: a + // folded needle against a raw column matches only the already-lower rows. + const lower = (expr: string) => (fold ? `lower(${expr})` : expr); + return `${lower(column)} ${negate ? 'NOT GLOB' : 'GLOB'} ${lower(bind(globPattern(shape, value)))}`; } const keyword = negate ? 'NOT LIKE' : 'LIKE'; @@ -219,10 +296,14 @@ export function textMatchPredicateSql(req: TextMatchRequest): string { // applies C escape syntax inside string literals, so the literal spelling // differs per dialect while a bound value has one spelling everywhere. if (dialect === 'mysql') { - const binary = (expr: string) => `CAST(${expr} AS BINARY)`; + const binary = (expr: string) => (fold ? mysqlAsciiLowerBinarySql(expr) : `CAST(${expr} AS BINARY)`); return `${binary(column)} ${keyword} ${binary(bind(likePattern(shape, value)))} ESCAPE ${bind(LIKE_ESCAPE_CHAR)}`; } // `postgres` — where LIKE is already case-exact — and `unknown`, the residue. - return `${column} ${keyword} ${bind(likePattern(shape, value))} ESCAPE ${bind(LIKE_ESCAPE_CHAR)}`; + // [#15780] The fold here is the `translate()` this package has always emitted + // ({@link asciiLowerSqlExpr}); on these two arms it was never the defect, so + // the correct diff is no diff. + const folded = (expr: string) => (fold ? asciiLowerSqlExpr(expr) : expr); + return `${folded(column)} ${keyword} ${folded(bind(likePattern(shape, value)))} ESCAPE ${bind(LIKE_ESCAPE_CHAR)}`; } From 711db06fd2e20f097029d7833a265836bd4bba31 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 20:45:40 +0000 Subject: [PATCH 2/3] =?UTF-8?q?docs(service-analytics):=20correct=20the=20?= =?UTF-8?q?$icontains=20changeset=20=E2=80=94=20state=20the=20unknown-is-S?= =?UTF-8?q?QLite=20carve-out=20and=20name=20the=20measured=20set?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 2 of the Clause-② contract review on PR #16020. Text only: no file under packages/ moves, and the review found no code defect. The changeset said the `unknown` arm was "never broken". That is false for an `unknown` that is really SQLite — the review drove four off-repo constructions that land there, and for each of them `translate()` still reaches the engine and still fails to parse. The carve-out is now stated here word for word with the PR body, and tracked as #16028. "byte-identical" and "every dialect" were unqualified here while the PR body qualified them; both now name the same measured sets (the six verbatim in-suite cells widened to the review's 2,721-cell construction with 0 deltas, and the 510-cell family-separation set). The MySQL arm is byte-equal to driver-sql's `textMatchPredicate` on 60/60 cells but was executed nowhere — text-only on both faces, said plainly rather than left under "measured". The deliberate divergence from driver-sql's `unknown` arm is now recorded in the changeset too. No card-relation trailer here on purpose: this branch squashes, and the PR body declares the relation once. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XpTx2tbq3pZRYAdoGt6E6Y --- .changeset/analytics-icontains-per-dialect-fold.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/.changeset/analytics-icontains-per-dialect-fold.md b/.changeset/analytics-icontains-per-dialect-fold.md index e265cb04a6..0d72acfa06 100644 --- a/.changeset/analytics-icontains-per-dialect-fold.md +++ b/.changeset/analytics-icontains-per-dialect-fold.md @@ -2,14 +2,18 @@ "@objectstack/service-analytics": patch --- -Analytics `$icontains` no longer compiles a `translate()` call on SQLite and MySQL, where that function does not exist and the statement failed to parse. +Analytics `$icontains` no longer compiles a `translate()` call on the `sqlite` and `mysql` dialects. On **SQLite** that function does not exist and the statement failed to parse — measured on the engine, not inferred. On **MySQL** the same construct was emitted and its arm is repaired the same way, but nothing was ever executed there: the MySQL arm is asserted as emitted TEXT only, on this face and on `driver-sql`'s alike, so no MySQL parse failure is claimed as measured. -`$icontains` folds ASCII case on both sides of the comparison (#4706 Q1 = A). All three of this package's SQL compilers — the query's own `where` (`NativeSQLStrategy.buildFilterClause`), the ADR-0021 D-C read scope (`compileScopedFilterToSql`) and the `ObjectQLStrategy` echo of that statement — spelled that fold as `translate(col, 'ABC…', 'abc…')` on **every** dialect. `translate()` is PostgreSQL/Oracle; SQLite has none. Measured on sql.js 1.14.1 (SQLite 3.49.1, the engine `driver-sqlite-wasm` runs), `SELECT translate('ABC','ABC','abc')` answers `no such function: translate` — so this was not a filter that returned the wrong rows, it was a statement the engine refused. On a SQLite datasource, an analytics `where` carrying `$icontains` and an **RLS read scope** carrying it were both unusable. +`$icontains` folds ASCII case on both sides of the comparison (#4706 Q1 = A). All three of this package's SQL compilers — the query's own `where` (`NativeSQLStrategy.buildFilterClause`), the ADR-0021 D-C read scope (`compileScopedFilterToSql`) and the `ObjectQLStrategy` echo of that statement — spelled that fold as `translate(col, 'ABC…', 'abc…')` on all four dialect values a compiler can see: `sqlite`, `mysql`, `postgres` and `unknown`, onto which `normalizeSqlDialect` maps everything else, an unset hook and `'oracle'` included. `translate()` is PostgreSQL/Oracle; SQLite has none. Measured on sql.js 1.14.1 (SQLite 3.49.1, the engine `driver-sqlite-wasm` runs), `SELECT translate('ABC','ABC','abc')` answers `no such function: translate` — so this was not a filter that returned the wrong rows, it was a statement the engine refused. On a SQLite datasource, an analytics `where` carrying `$icontains` and an **RLS read scope** carrying it were both unusable. The fold is now chosen per dialect, on the same construct table the case-exact text family already used, reached through one `fold` flag: - **SQLite** — `lower(col) GLOB lower(?)`. SQLite's `lower()` is ASCII-only (measured: `lower('CAFÉ')` is `cafÉ`), so this is the ruled fold rather than an approximation of it, and it runs. -- **PostgreSQL** and the `unknown` residue (a host that wires no dialect hook) — `translate()`, **unchanged**. These arms were never broken, so the emitted SQL and its bound parameters are byte-identical to before. -- **MySQL** — the nested-`REPLACE` fold over `CAST(… AS BINARY)`, matching what `driver-sql` emits for the same operator. Asserted as text only; no MySQL server is provisionable in the container that wrote this, so that cell is a declared skip, not a claimed pass. +- **PostgreSQL** and the `unknown` residue — `translate()`, byte-for-byte what those two arms emitted before. Measured set for that word: this package's own suite pins six cells verbatim — `{NativeSQLStrategy, ObjectQLStrategy echo, compileScopedFilterToSql} × {dialect unset, 'postgres'}` for `{name: {$icontains: 'acme'}}`, full emitted SQL and the exact bound params — and the round-1 contract review widened it to **2,721 cells** (2,720 = `{undefined, 'postgres', 'unknown', 'oracle'} × 5 compiler paths × 8 filter shapes × 17 comparands`, plus the bare `{dialect: undefined}` cell), emitted at the merge-base blobs (all five hash-verified) and again at this head: **0 changed cells, 0 error cells**. Outside that set nothing is claimed — no PostgreSQL server was contacted, and on `sqlite` and `mysql` the bytes deliberately changed (340 of 680 cells each, all inside the four `$icontains` shapes). +- **MySQL** — the nested-`REPLACE` fold over `CAST(… AS BINARY)`, matching what `driver-sql` emits for the same operator; the review measured the two faces byte-equal on 60 of 60 MySQL cells. Asserted as text only — no MySQL server is provisionable in the container that wrote this, so that cell is a declared skip, not a claimed pass. -`$icontains` and the case-sensitive `$contains` family remain two separate constructs on every dialect — collapsing them would give `$contains` back the case fold #4706 Q2 = A took away from it. A host that answers no dialect keeps exactly the behaviour it had. +⚠️ Carve-out, stated because it is the surviving half of the defect and not an aside: an `unknown` dialect that is really SQLite is **not** fixed by this change. The residue is reached by four constructions the round-1 contract review drove rather than reasoned — a `SqlDriver` given a **class** client or an unrecognised spelling (`'libsql'`), a host hook answering knex's own `'sqlite3'`, a directly-constructed public `AnalyticsService` with the optional `sqlDialect` omitted, and a `data` service without `getDriverForObject`. For each of them `translate()` still reaches the engine and still fails to parse, on the `where` path, the read scope and the echo alike. No in-repo SQLite driver lands there — `SqliteWasmDriver` and `TursoDriver` both answer `"sqlite"`, measured — so this is an embedder-composition population, not a shipped-driver one. Tracked as #16028. + +`$icontains` and the case-sensitive `$contains` family remain two separate constructs on every dialect the compilers accept — collapsing them would give `$contains` back the case fold #4706 Q2 = A took away from it. Measured set for that word: 510 cells (six dialect names — the four values above plus `'oracle'` and an unset hook, which both normalize to `unknown` — × 5 compiler paths × 17 comparands), 0 of them identical between the two families and no `$contains` cell carrying a fold. + +⚠️ One deliberate divergence from `driver-sql`, recorded here rather than only in this package's source: `driver-sql`'s own `unknown` arm folds with `LOWER()`, this one keeps `translate()`. Each face keeps the residue it already had, and adopting `LOWER()` here would silently restore on PostgreSQL the Unicode fold #4706 Q1 = A rules out. The pointer exists on this side only; `driver-sql` carries no cross-reference back. From fcddb130f4372dcf2bf02ba474a775a87c81bb85 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 5 Sep 2026 21:26:49 +0000 Subject: [PATCH 3/3] docs(service-analytics): retract the "more tightly than the proxy" clause from the case-exactness suite header The round-1 contract review measured that clause false: FOLD_PER_DIALECT inspects the COLUMN side only, so a one-sided column fold on the postgres / unknown arm leaves this file green (exit 0, 14 passed) where the pre-fix `LIKE translate($1,` pin caught it. The PR body and the changeset were corrected in round 2; this source header was not, and a PR body cannot reach someone who opens this suite six months from now. The header now carries the two halves the PR body already states: tighter on family separation (the four collapse mutations the review drove all went red here), looser on both-sides folding, plus the cross-reference that icontains-dialect-sql.test.ts holds the verbatim postgres / unknown byte pin for that second case. Concluded as not net-weakened across the two files, never as a strict tightening. Comment-only, proven two ways with firing controls: every added and removed diff line is a comment line (28 added, 8 removed, 0 non-comment; four real code lines fed to the same filter classify as CODE), and the file transpiled with removeComments is byte-identical to its 711db06fd blob (sha256/16 941867ad0f3e45a7, 14888 bytes on both sides, while injecting one code line in memory moves that hash). The other seven files this PR touches are untouched: git hash-object equals the 711db06fd blob for each. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XpTx2tbq3pZRYAdoGt6E6Y --- .../text-operator-case-exactness.test.ts | 36 ++++++++++++++----- 1 file changed, 28 insertions(+), 8 deletions(-) diff --git a/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts b/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts index 27c09a5645..806ade1cb7 100644 --- a/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts +++ b/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts @@ -72,14 +72,34 @@ * which it no longer is. * * ⛔ It was therefore neither deleted nor loosened — it was re-aimed at the - * property itself, which is now pinned DIRECTLY and more tightly than the proxy - * ever did (`$icontains and the case-EXACT family stay two constructs…` - * below): on each dialect, `$icontains` and `$contains` must compile to - * DIFFERENT text, and `$contains` must carry no fold. That discriminates - * against the collapse in both directions, where dialect-invariance only - * discriminated against one. The row sets that make it more than a text - * comparison are executed in `icontains-dialect-sql.test.ts`, which owns the - * `$icontains` half of the family from here on. + * property itself, pinned DIRECTLY (`$icontains and the case-EXACT family stay + * two constructs…` below): on each dialect, `$icontains` and `$contains` must + * compile to DIFFERENT text, and `$contains` must carry no fold. + * + * ⛔ That re-aiming is NOT a strict tightening. An earlier revision of this + * header claimed it was ("more tightly than the proxy ever did"); #15780's + * contract review measured that claim false, so it is retracted here. What is + * true has two halves: + * + * - **TIGHTER on family separation.** The re-aimed pin discriminates against + * the collapse in BOTH directions, where dialect-invariance only caught + * one. Named measured set: the four collapse mutations that review drove — + * the fold leaking onto `$contains`, `$contains`' bare construct handed to + * `$icontains`, and the two read-scope directions (drop the fold there / + * hand it to `$contains` there). This file went red on all four. + * - **LOOSER on both-sides folding.** `FOLD_PER_DIALECT` below inspects the + * COLUMN side only. So the mutation that folds the column but leaves the + * comparand UNFOLDED on the `postgres` / `unknown` arm slips PAST this file + * — measured green there, exit 0 and 14 passed — where the pre-#15780 + * `LIKE translate($1,` pin would have caught it. + * + * That second case is held instead by `icontains-dialect-sql.test.ts`, which + * pins the `postgres` / `unknown` arm's emitted SQL and bound params VERBATIM, + * and which does go red on that same mutation. ⇒ Across the two files the + * ratchet is NOT net-weakened; within THIS file alone it is not a strict + * tightening. ⛔ Neither claim may be restated as "strictly tighter". The row + * sets that make any of this more than a text comparison are executed in that + * same suite, which owns the `$icontains` half of the family from here on. * * Escaping (#5567) is unchanged for every `LIKE` arm; the GLOB arm brings its * OWN escaped character class (`*`, `?`, `[`), which is why the second fixture