From b19656e923d6e9e6a5106aba9c42993935a70e8f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 05:26:45 +0000 Subject: [PATCH 1/2] fix(service-analytics): the `unknown` dialect arm folds `$icontains` with a portable construct, not `translate()` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `normalizeSqlDialect` routes EVERYTHING it cannot name onto `unknown` — an unset hook, `'oracle'`, `'libsql'`, a `SqlDriver` given a class client. #15780 left that arm folding with `translate()` on the reading that it "was never broken", which held only for the dialects the arm was pictured as (mssql, oracle). SQLite reaches it through four embedder compositions, and SQLite has no `translate()`, so all three of this package's compilers emitted a statement the engine refuses — a documented operator answering 500 because one OPTIONAL field was left out. The arm now folds with one nested `REPLACE` per ASCII letter: the same chain the MySQL arm already used, minus its `CAST(… AS BINARY)`, so there is one builder and the two cannot fold different alphabets. It parses on every SQL dialect and is ASCII-only BY CONSTRUCTION, so it serves both families `unknown` conflates — PostgreSQL/Oracle-like keep `translate()`'s exact result set, SQLite-like get an answer at all. - `postgres` is untouched and byte-identical; so is the case-EXACT family on every arm (`fold` false makes the fold function the identity on both). - ⛔ Not `LOWER()`, which `driver-sql`'s own `unknown` arm uses: it follows the collation and would trade this parse failure for silently wrong rows on PostgreSQL, the Unicode fold #4706 Q1 = A rules out. - The pins that moved are re-aimed at the property, not regenerated, and the pre-fix bytes are kept as a control that the engine still refuses them. - `SqliteWasmDriver`'s `isSqlite` override — 0 direct test hits, the sole reason no in-repo SQLite driver lands on this arm — gets a direct pin, with a control showing the base class answers `unknown` for that very config. Refs #16028 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XpTx2tbq3pZRYAdoGt6E6Y --- .../src/sqlite-wasm-dialect-identity.test.ts | 87 ++++++++ .../__tests__/icontains-dialect-sql.test.ts | 206 +++++++++++++++--- .../like-metacharacter-escape.test.ts | 28 ++- .../text-operator-case-exactness.test.ts | 9 +- .../service-analytics/src/like-pattern.ts | 21 +- .../service-analytics/src/read-scope-sql.ts | 8 +- .../src/strategies/native-sql-strategy.ts | 4 +- .../src/strategies/objectql-strategy.ts | 6 +- .../service-analytics/src/text-match-sql.ts | 94 +++++++- 9 files changed, 409 insertions(+), 54 deletions(-) create mode 100644 packages/drivers/driver-sqlite-wasm/src/sqlite-wasm-dialect-identity.test.ts diff --git a/packages/drivers/driver-sqlite-wasm/src/sqlite-wasm-dialect-identity.test.ts b/packages/drivers/driver-sqlite-wasm/src/sqlite-wasm-dialect-identity.test.ts new file mode 100644 index 0000000000..c0ce1280c0 --- /dev/null +++ b/packages/drivers/driver-sqlite-wasm/src/sqlite-wasm-dialect-identity.test.ts @@ -0,0 +1,87 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#16028] The `isSqlite` override, pinned DIRECTLY — the one thing that makes + * this transport answer `"sqlite"` when something outside the driver asks which + * SQL it speaks. + * + * ## Why this file exists + * + * `SqlDriver.dialectName` is public for exactly one consumer: + * `service-analytics` compiles its own statements (an analytics `where`, an + * ADR-0021 D-C read scope, the `/analytics/sql` echo) and needs the same + * per-dialect construct choices the driver makes. Answering `'sqlite'` is what + * routes `$icontains` onto `lower(col) GLOB lower(?)`; answering `'unknown'` + * routes it onto the residue arm instead. + * + * The base class derives that answer by STRING-MATCHING `config.client` against + * {@link SqlDriver}'s emission sets — and this transport passes a knex Client + * CLASS, which is no string at all. So the correct answer here is produced by + * one three-line override and by nothing else. + * + * ⚠️ #16028 measured that override at **0 direct test hits**: the only cover was + * an indirect row-set pin (#15684), which would keep passing if the override + * moved, because the ROWS come out the same either way — the driver runs its own + * SQL through its own SQLite. What changes silently is the answer handed to a + * package that compiles SQL for a DIFFERENT engine. That is the gap this file + * closes, and it is the reason the control below is not decoration: it shows the + * base class answering `'unknown'` for this exact config, so the pin above is a + * measurement of the override rather than of the class hierarchy. + */ + +import { describe, it, expect, afterEach } from 'vitest'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { SqlDriver } from '@objectstack/driver-sql'; + +import { SqliteWasmDriver } from '../src/index.js'; + +/** Nothing here connects — but every knex instance built is still torn down. */ +const opened: Array<{ disconnect(): Promise }> = []; +const dirs: string[] = []; + +afterEach(async () => { + await Promise.all(opened.splice(0).map((d) => d.disconnect().catch(() => {}))); + for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +const track = }>(d: T): T => { + opened.push(d); + return d; +}; + +describe('[#16028] SqliteWasmDriver names its dialect', () => { + it('answers "sqlite" — the answer service-analytics compiles against', () => { + // Read WITHOUT connecting, deliberately: `service-analytics` asks this + // while BUILDING a statement, so an answer that needed a live pool would + // arrive after the SQL it decides. + expect(track(new SqliteWasmDriver({ filename: ':memory:' })).dialectName).toBe('sqlite'); + }); + + it('…on a file-backed database too, and with persistence on', () => { + const dir = mkdtempSync(join(tmpdir(), 'wasm-dialect-')); + dirs.push(dir); + const file = join(dir, 'test.db'); + expect(track(new SqliteWasmDriver({ filename: file })).dialectName).toBe('sqlite'); + expect(track(new SqliteWasmDriver({ filename: file, persist: 'on-write' })).dialectName).toBe('sqlite'); + }); + + it('the client is a CLASS, so no string table could have answered it', () => { + // The override's premise, asserted rather than assumed: if this ever became + // a string knex spelling, the base class would answer on its own and the + // override would be dead code rather than the load-bearing line it is. + const client = (SqliteWasmDriver.toKnexConfig({ filename: ':memory:' }) as { client: unknown }).client; + expect(typeof client).toBe('function'); + expect(typeof client).not.toBe('string'); + }); + + it('CONTROL: the base class answers "unknown" for this very config', () => { + // Delete `isSqlite` from the subclass and this is what `service-analytics` + // would be told — #16028's residue arm, which for `$icontains` emitted a + // statement SQLite cannot parse at all until that card. This is what makes + // the pin above a measurement of the OVERRIDE rather than of the hierarchy. + const base = track(new SqlDriver(SqliteWasmDriver.toKnexConfig({ filename: ':memory:', pool: { min: 0, max: 1 } }))); + expect(base.dialectName).toBe('unknown'); + }); +}); diff --git a/packages/services/service-analytics/src/__tests__/icontains-dialect-sql.test.ts b/packages/services/service-analytics/src/__tests__/icontains-dialect-sql.test.ts index ccba607c47..5201993b91 100644 --- a/packages/services/service-analytics/src/__tests__/icontains-dialect-sql.test.ts +++ b/packages/services/service-analytics/src/__tests__/icontains-dialect-sql.test.ts @@ -30,14 +30,27 @@ * - **postgres → byte-identical text.** `translate()` is correct there today, * so the correct diff is no diff: the emitted SQL and params are asserted * verbatim against the pre-#15780 bytes, for all three compilers. - * - **`unknown` (no hook wired) → byte-identical text, ALSO `translate()`.** - * The residue keeps the shape it has always had. It is not an endorsement: - * it is what still runs on the dialects nothing here models (mssql, oracle, - * which do have `translate()`), and falling back to `LOWER()` for it would - * silently restore the Unicode fold #4706 Q1 = A took away. Note this - * DIVERGES from `driver-sql`'s `unknown` arm, which folds with `LOWER()` — - * each side keeps its own pre-existing residue, and neither claims the - * other's. + * - **[#16028] `unknown` (no hook wired) → the portable `REPLACE` chain, + * EXECUTED.** #15780 left this arm on `translate()` and called it + * unbroken, which held only for the dialects the arm was PICTURED as + * (mssql, oracle, which have `translate()`). `unknown` is not a dialect: + * `normalizeSqlDialect` routes EVERYTHING it cannot name here, SQLite + * included, so the residue arm carried the whole defect forward for the + * four embedder compositions #16028 lists — among them a public + * `AnalyticsService` constructed with its OPTIONAL `sqlDialect` left out. + * The arm now folds with one nested `REPLACE` per ASCII letter, which every + * SQL dialect parses, and the rows it answers on this engine are asserted + * below from the same shared table the `sqlite` arm answers. + * + * ⛔ NOT `LOWER()`, which is what `driver-sql`'s `unknown` arm folds with: + * `LOWER()` follows the collation, so adopting it would trade this parse + * failure for SILENTLY wrong rows on PostgreSQL — the Unicode fold #4706 + * Q1 = A rules out. ⚠️ And measuring `LOWER()` HERE proves nothing about + * that, because SQLite's `lower()` is ASCII-only and answers the shared + * table correctly: the trap is a green SQLite reading standing in for a + * PostgreSQL one, so the `REPLACE` chain is pinned as ASCII-only BY + * CONSTRUCTION (over every ASCII code point, below) rather than by a fold + * that happens to behave on the one engine in this container. * - **mysql → NOT MEASURED.** The nested-`REPLACE`-over-`CAST(… AS BINARY)` * arm is asserted as TEXT only. No MySQL server is provisionable in this * container — the same declared skip `driver-sql`'s #6518 suite and this @@ -119,40 +132,118 @@ const ctxFor = (dialect?: string): DatasetScopedStrategyContext => const TRANSLATE_FOLD = "translate(name, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz')"; +/** + * [#16028] The `unknown` arm's fold, rebuilt HERE from the ruled domain rather + * than imported from the implementation — a second construction of the expected + * text, so a change to the emitter's loop does not quietly re-bless itself. + * + * The endpoints are pinned verbatim beside every use (`REPLACE(name, 'A', 'a')` + * innermost, `'Z', 'z')` outermost), which is also what discriminates this arm + * from MySQL's: that one's innermost operand is `CAST(name AS BINARY)`. + */ +const ASCII_UP = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'; +const ASCII_LO = 'abcdefghijklmnopqrstuvwxyz'; +const replaceFold = (expr: string): string => { + let out = expr; + for (let i = 0; i < ASCII_UP.length; i++) out = `REPLACE(${out}, '${ASCII_UP[i]}', '${ASCII_LO[i]}')`; + return out; +}; + describe('[#15780] the compiled TEXT, per dialect — all three compilers', () => { const nativeSql = async (where: unknown, dialect?: string) => new NativeSQLStrategy().generateSql(query(where), ctxFor(dialect)); const echoSql = async (where: unknown, dialect?: string) => new ObjectQLStrategy().generateSql(query(where), ctxFor(dialect)); - it('postgres and a host that wired NO hook keep the pre-#15780 bytes exactly', async () => { - // The non-regression half, for the two dialects that were already correct. - // Asserted verbatim, not by shape: `translate()` is right on Postgres, and - // the `unknown` residue must keep the only fold that still runs there. - for (const dialect of [undefined, 'postgres'] as const) { + it('postgres keeps the pre-#15780 bytes exactly — the arm that was always right', async () => { + // The non-regression half. Asserted verbatim, not by shape: `translate()` + // is correct on Postgres, so the correct diff there is no diff — through + // #15780 and through #16028 alike. + const out = await nativeSql({ name: { $icontains: 'acme' } }, 'postgres'); + expect(out.sql).toContain( + `WHERE ${TRANSLATE_FOLD} LIKE translate($1, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') ESCAPE $2`, + ); + expect(out.params).toEqual(['%acme%', '\\']); + + const echo = await echoSql({ name: { $icontains: 'acme' } }, 'postgres'); + expect(echo.sql).toContain( + `${TRANSLATE_FOLD} LIKE translate($1, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') ESCAPE $2`, + ); + expect(echo.params).toEqual(['%acme%', '\\']); + + expect( + compileScopedFilterToSql({ name: { $icontains: 'acme' } } as FilterCondition, 't', { dialect: 'postgres' }), + ).toEqual({ + sql: + `translate("t"."name", 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') LIKE ` + + `translate(?, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') ESCAPE ?`, + params: ['%acme%', '\\'], + }); + }); + + it('[#16028] a host that wired NO hook folds with the PORTABLE chain, on all three compilers', async () => { + // The moved cells, stated as the pin rather than left to a regenerate: the + // `unknown` arm's emitted text CHANGED, from `translate()` — which SQLite + // and MySQL/MariaDB cannot parse — to a nested `REPLACE` per ASCII letter, + // which every SQL dialect can. The rows this text answers are executed + // further down; here it is the bytes and the bindings. + for (const dialect of [undefined, 'oracle', 'libsql'] as const) { + // All three normalize to `unknown`: an unset hook, and two spellings + // `normalizeSqlDialect` does not model. One arm, reached three ways. const out = await nativeSql({ name: { $icontains: 'acme' } }, dialect); expect(out.sql, String(dialect)).toContain( - `WHERE ${TRANSLATE_FOLD} LIKE translate($1, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') ESCAPE $2`, + `WHERE ${replaceFold('name')} LIKE ${replaceFold('$1')} ESCAPE $2`, ); + expect(out.sql, String(dialect)).not.toMatch(/translate\(/); + // The chain's endpoints, verbatim — and the innermost operand is the bare + // column, which is what tells this arm apart from MySQL's `CAST(… AS + // BINARY)` one. + expect(out.sql, String(dialect)).toContain("REPLACE(name, 'A', 'a')"); + expect(out.sql, String(dialect)).toContain("'Z', 'z')"); + expect(out.sql, String(dialect)).not.toMatch(/CAST\(/); expect(out.params, String(dialect)).toEqual(['%acme%', '\\']); const echo = await echoSql({ name: { $icontains: 'acme' } }, dialect); expect(echo.sql, `echo ${String(dialect)}`).toContain( - `${TRANSLATE_FOLD} LIKE translate($1, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') ESCAPE $2`, + `${replaceFold('name')} LIKE ${replaceFold('$1')} ESCAPE $2`, ); + expect(echo.sql, `echo ${String(dialect)}`).not.toMatch(/translate\(/); expect(echo.params, `echo ${String(dialect)}`).toEqual(['%acme%', '\\']); } + // The read scope — the compiler where an unevaluable statement is an + // ADR-0021 policy that cannot be applied at all. const noHook = compileScopedFilterToSql({ name: { $icontains: 'acme' } } as FilterCondition, 't'); expect(noHook).toEqual({ - sql: - `translate("t"."name", 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') LIKE ` + - `translate(?, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') ESCAPE ?`, + sql: `${replaceFold('"t"."name"')} LIKE ${replaceFold('?')} ESCAPE ?`, params: ['%acme%', '\\'], }); + expect(noHook.sql).not.toMatch(/translate\(/); expect( - compileScopedFilterToSql({ name: { $icontains: 'acme' } } as FilterCondition, 't', { dialect: 'postgres' }), + compileScopedFilterToSql({ name: { $icontains: 'acme' } } as FilterCondition, 't', { dialect: 'nonesuch' }), ).toEqual(noHook); + + // ⛔ And the arm did NOT become Postgres's: the two are now different text, + // which is the whole point of splitting them. + const pg = await nativeSql({ name: { $icontains: 'acme' } }, 'postgres'); + const unknown = await nativeSql({ name: { $icontains: 'acme' } }, undefined); + expect(unknown.sql).not.toBe(pg.sql); + expect(unknown.params).toEqual(pg.params); + }); + + it('[#16028] the case-EXACT four are byte-identical on `unknown` — only the FOLD moved', async () => { + // The blast-radius pin. `fold` is the only thing #16028 changed on this + // arm, so every case-exact operator must still emit the plain `LIKE` it + // emitted before — no `REPLACE` anywhere near them. + for (const op of ['$contains', '$notContains', '$startsWith', '$endsWith'] as const) { + const out = await nativeSql({ name: { [op]: 'acme' } }, undefined); + expect(out.sql, op).not.toMatch(/REPLACE\(|translate\(|lower\(/); + expect(out.sql, op).toMatch(/name (NOT )?LIKE \$1 ESCAPE \$2/); + } + expect(compileScopedFilterToSql({ name: { $contains: 'acme' } } as FilterCondition, 't')).toEqual({ + sql: '"t"."name" LIKE ? ESCAPE ?', + params: ['%acme%', '\\'], + }); }); it('sqlite compiles lower() over GLOB — one bound value, no ESCAPE clause', async () => { @@ -274,16 +365,81 @@ describe('[#15780] the three compilers, EXECUTED on a real SQLite engine', () => ]); }); - it('the defect, still reachable through a host that answers no dialect — so these pins discriminate', async () => { - // Before-red, stated as the engine's own refusal rather than argued: the - // `unknown` residue still emits `translate()`, and this engine cannot parse - // it. This is the control that keeps every green assertion above honest. + it('[#16028] the PRE-fix bytes are still refused by this engine — the control the greens rest on', () => { + // Before-red, kept as a control rather than deleted with the defect. This + // is the statement the `unknown` arm emitted until #16028, rebuilt here and + // handed to the engine: it does not parse. Every green below is therefore a + // measurement of the CHANGE, not of an engine that would have accepted + // anything. + const preFix = + `SELECT "id" FROM "rows" WHERE ${TRANSLATE_FOLD} LIKE ` + + `translate(?, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') ESCAPE ?`; + expect(() => run(preFix, ['%acme%', '\\'])).toThrow(/no such function: translate/); + }); + + it('[#16028] a host that answers NO dialect now compiles a statement this engine RUNS', async () => { + // The card's headline, executed: with no `sqlDialect` hook wired — the + // shape a directly-constructed public `AnalyticsService` has when its + // OPTIONAL field is omitted — the compiled `where` used to be refused by + // the engine. It now parses, and it answers the shared table's rows. const { sql, params } = await new NativeSQLStrategy().generateSql( query({ name: { $icontains: 'acme' } }), unawareCtx, ); - expect(sql).toContain('translate('); - expect(() => run(sql, params)).toThrow(/no such function: translate/); + expect(sql).not.toMatch(/translate\(/); + expect(run(sql, params)).toEqual(['1', '2']); + }); + + it('[#16028] the `unknown` arm answers the shared $icontains table — all three compilers', async () => { + // The same rows the `sqlite` arm is required to answer, required of the + // residue arm. Executed on sql.js 1.14.1, the engine `driver-sqlite-wasm` + // runs — which is what an `unknown` dialect that is really SQLite IS. + for (const c of NAME_ICONTAINS) { + expect(await executedIds(c.filter, unawareCtx), c.name).toEqual([...c.expected]); + + const scoped = { + ...unawareCtx, + getReadScope: (object: string) => (object === 'rows' ? (c.filter as FilterCondition) : null), + } as DatasetScopedStrategyContext; + const viaScope = await new NativeSQLStrategy().generateSql(query(undefined), scoped); + expect(viaScope.sql, `read scope: ${c.name}`).not.toMatch(/translate\(/); + expect(run(viaScope.sql, viaScope.params), `read scope: ${c.name}`).toEqual([...c.expected]); + + const echo = await new ObjectQLStrategy().generateSql(query(c.filter), unawareCtx); + expect(echo.sql, `echo: ${c.name}`).not.toMatch(/translate\(/); + expect(run(echo.sql, echo.params), `echo: ${c.name}`).toEqual([...c.expected]); + } + }); + + it('[#16028] the portable chain folds ASCII and NOTHING else — every code point, executed', () => { + // The property the arm rests on, measured instead of argued, because the + // one thing that must never happen here is a fold that reaches past `A`-`Z` + // (#4706 Q1 = A). Two halves: + // + // 1. the chain equals the simultaneous `A`-`Z` map — i.e. applying the 26 + // REPLACEs IN SEQUENCE cannot cascade, because every step writes a + // lower-case letter and every later step matches an upper-case one; + // 2. it touches nothing else — no accented letter, and none of the LIKE + // metacharacters the escaping depends on staying literal. + const asciiLower = (v: string) => + v.replace(/[A-Z]/g, (ch) => ASCII_LO[ASCII_UP.indexOf(ch)]); + const probes = [ + ...Array.from({ length: 128 }, (_, i) => String.fromCharCode(i)).filter((ch) => ch !== '\0'), + 'CAFÉ', 'café', 'ÀÉÎÕÜ', 'ÄÖÜ', 'ǍǏǑ', 'ΑΒΓ', 'АБВ', 'İIı', + 'ACME Corp', '100% match', 'a_b', 'A\\B', 'ZzAa', + ]; + for (const probe of probes) { + const literal = probe.replace(/'/g, "''"); + const got = db.exec(`SELECT ${replaceFold(`'${literal}'`)}`)[0].values[0][0]; + expect(got, JSON.stringify(probe)).toBe(asciiLower(probe)); + } + // ⚠️ And the control that says why `LOWER()` may not be adopted here even + // though it passes on THIS engine: SQLite's `lower()` is ASCII-only, so it + // agrees with the chain on every probe above. It is PostgreSQL's, which is + // collation-aware, that would fold `É` — and no PostgreSQL server is + // provisionable in this container. So this cell is why the arm is chosen by + // CONSTRUCTION and not by what happens to pass locally. + expect(db.exec(`SELECT lower('CAFÉ')`)[0].values[0][0]).toBe('cafÉ'); }); it('NativeSQLStrategy answers the shared table\'s $icontains rows', async () => { diff --git a/packages/services/service-analytics/src/__tests__/like-metacharacter-escape.test.ts b/packages/services/service-analytics/src/__tests__/like-metacharacter-escape.test.ts index e2f78fdc76..a41003eadf 100644 --- a/packages/services/service-analytics/src/__tests__/like-metacharacter-escape.test.ts +++ b/packages/services/service-analytics/src/__tests__/like-metacharacter-escape.test.ts @@ -609,6 +609,17 @@ describe('[#5567] analytics LIKE compilers escape their comparand', () => { * What it pins instead is the property the refusal was standing in for — * that the predicate is EMITTED, folded on BOTH sides, and folded with the * construct the ruling names. + * + * ⚠️ [#16028] The SPELLING of that construct moved here, and the assertions + * were re-aimed rather than regenerated. Neither door in this file answers + * a dialect — `compileScopedFilterToSql` is called with no `dialect` option + * and `nativeCtx` wires no `sqlDialect` hook — so both compile on the + * `unknown` arm, and that arm stopped folding with `translate()`. It had to: + * `unknown` is everything `normalizeSqlDialect` could not name, SQLite + * included, and `translate()` is a function SQLite does not have, so this + * file was pinning a fold that could not run on the very engine it opens a + * database on. The property is identical and is what is asserted below — + * ASCII-only, applied to BOTH sides — in the portable spelling. */ it('$icontains compiles at both doors, folding ASCII on both sides', async () => { const scoped = compileScopedFilterToSql( @@ -621,13 +632,18 @@ describe('[#5567] analytics LIKE compilers escape their comparand', () => { for (const [label, sql] of [['scoped', scoped.sql], ['native', native.sql]] as const) { // BOTH sides: folding only the comparand matches just the rows that were // already lower-case — a wrong row set that looks like a working filter. - expect(sql.match(/translate\(/g) ?? [], label).toHaveLength(2); + // 26 nested `REPLACE`s per side, one per letter of the ruled domain. + expect(sql.match(/REPLACE\(/g) ?? [], label).toHaveLength(52); expect(sql, label).toContain('LIKE'); - expect(sql, label).toContain("'ABCDEFGHIJKLMNOPQRSTUVWXYZ'"); - // NOT `LOWER()`: Postgres folds Unicode with it, and the contract is - // ASCII-only (#4706 Q1 = A). This is the same assertion the neighbouring - // Postgres-shape case makes, aimed at the one operator that folds. - expect(sql, label).not.toMatch(/LOWER\s*\(|ILIKE/i); + // The domain's endpoints, so a chain that stopped short of `Z` — an + // ASCII fold with holes in it — is not read as a fold. + expect(sql, label).toContain("'A', 'a')"); + expect(sql, label).toContain("'Z', 'z')"); + // ⛔ Still not `LOWER()`, and now not `translate()` either: the first + // folds Unicode on Postgres (#4706 Q1 = A forbids it), the second does + // not exist on the SQLite this arm can be. Also not MySQL's variant of + // the same chain — no `CAST(… AS BINARY)` on this arm. + expect(sql, label).not.toMatch(/LOWER\s*\(|ILIKE|translate\s*\(|CAST\s*\(/i); } // The comparand is still ESCAPED and its ESCAPE character still bound — // the fold rides ON TOP of the literal-comparand rule, it does not replace diff --git a/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts b/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts index 806ade1cb7..122b4232db 100644 --- a/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts +++ b/packages/services/service-analytics/src/__tests__/text-operator-case-exactness.test.ts @@ -293,8 +293,15 @@ describe('[#15684] the compiled TEXT, per dialect', () => { // `$contains` carries NO fold, in any of the three spellings a fold has. expect(contains.sql, String(dialect)).not.toMatch(/translate\(|lower\(|REPLACE\(/); // …and `$icontains` carries exactly one of them, per dialect. + // [#16028] `undefined` (the `unknown` arm) parted company with `postgres` + // here: it folds with the PORTABLE `REPLACE` chain, because `unknown` is + // every dialect nothing answered for and `translate()` does not exist on + // two of them. Each pattern below is discriminating against the other + // three arms — `REPLACE(name, 'A', 'a')` is the `unknown` chain's + // innermost call and cannot match MySQL's, whose innermost operand is + // `CAST(name AS BINARY)`. const FOLD_PER_DIALECT: Record = { - undefined: /translate\(name, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'/, + undefined: /REPLACE\(name, 'A', 'a'\)/, postgres: /translate\(name, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'/, sqlite: /lower\(name\) GLOB lower\(\$1\)/, mysql: /REPLACE\(CAST\(name AS BINARY\), 'A', 'a'\)/, diff --git a/packages/services/service-analytics/src/like-pattern.ts b/packages/services/service-analytics/src/like-pattern.ts index bc44b26752..79de324c53 100644 --- a/packages/services/service-analytics/src/like-pattern.ts +++ b/packages/services/service-analytics/src/like-pattern.ts @@ -184,7 +184,8 @@ export function likePattern(shape: LikeShape, value: unknown): string { * * [#15780] EXPORTED, because the fold is now chosen per dialect and two of the * three arms are built from this domain rather than from `translate()`: - * `text-match-sql.ts`'s MySQL arm nests one `REPLACE` per letter. The domain + * `text-match-sql.ts`'s MySQL arm nests one `REPLACE` per letter, and [#16028] + * its `unknown` arm nests the same chain without the binary cast. The domain * itself stays here, in one copy — a second 26-character literal anywhere is * how the Postgres arm and the MySQL arm start folding different alphabets. */ @@ -217,8 +218,9 @@ export const ASCII_LOWER_LETTERS = 'abcdefghijklmnopqrstuvwxyz'; * (`no such function: translate` on sql.js 1.14.1). That was #15780, and it is * closed: this function is now ONE arm of `text-match-sql.ts`'s per-dialect * table ({@link textMatchPredicateSql}, `fold: true`), reached on `postgres` - * and on `unknown`, while SQLite gets `lower(col) GLOB lower(?)` and MySQL the - * nested-`REPLACE` binary fold. + * ALONE, while SQLite gets `lower(col) GLOB lower(?)`, MySQL the nested- + * `REPLACE` binary fold and — since #16028 — `unknown` the same `REPLACE` chain + * without the binary cast. * * ⛔ Do NOT "simplify" this to `LOWER()`, on any arm. Postgres' `LOWER()` is * locale-aware and would silently restore the Unicode fold #4706 Q1 = A rules @@ -226,11 +228,14 @@ export const ASCII_LOWER_LETTERS = 'abcdefghijklmnopqrstuvwxyz'; * and this one may not. That asymmetry is the whole reason the fold is chosen * per dialect rather than written once. * - * ⛔ Nor is `unknown` free to adopt `driver-sql`'s residue: that face folds - * `unknown` with `LOWER()` because `LOWER()` is the shape it emitted before - * #6518. This face's pre-existing shape is `translate()`, so `translate()` is - * what its residue keeps. Each side keeps its own, and neither claims the - * other's — an `unknown` dialect is by definition one nothing here measured. + * ⛔ [#16028] Nor is this function the `unknown` arm's fold any more, and it may + * not be given back. `unknown` is not a dialect — it is everything + * `normalizeSqlDialect` could not name, SQLite and MariaDB included — so a + * PostgreSQL/Oracle function there is a statement those engines cannot PARSE. + * That arm folds with the portable `REPLACE` chain, which is ASCII-only by + * construction and therefore does not re-open the `LOWER()` question either. + * `driver-sql`'s `unknown` arm still folds with `LOWER()`, its own pre-#6518 + * shape; each face keeps its own answer and neither claims the other's. * * The caller must apply it to BOTH sides of the comparison. Folding only the * comparand compares a folded needle against a raw column and matches just the diff --git a/packages/services/service-analytics/src/read-scope-sql.ts b/packages/services/service-analytics/src/read-scope-sql.ts index 5c8c6787b8..3cc3580f67 100644 --- a/packages/services/service-analytics/src/read-scope-sql.ts +++ b/packages/services/service-analytics/src/read-scope-sql.ts @@ -1343,8 +1343,12 @@ function compileOperator( * read scope compiled to a statement the engine could not PARSE — an RLS * policy that cannot be evaluated at all. It goes through * {@link textMatch} now with `fold` set, which keeps `translate()` on - * Postgres and the `unknown` residue, emits `lower(col) GLOB lower(?)` on - * SQLite and the nested-`REPLACE` binary fold on MySQL. The `ESCAPE` + * Postgres, emits `lower(col) GLOB lower(?)` on SQLite, the nested- + * `REPLACE` binary fold on MySQL and — [#16028] — the same `REPLACE` chain + * without the cast on the `unknown` residue, because a datasource whose + * dialect nothing answered can BE SQLite and `translate()` failed to parse + * there just as loudly through this compiler as through the other two. The + * `ESCAPE` * binding is still never folded — the construct table owns that, and the * SQLite arm has no `ESCAPE` clause to bind at all. */ diff --git a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts index 731a64c3c2..1078b75b92 100644 --- a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts @@ -1154,7 +1154,9 @@ export class NativeSQLStrategy implements AnalyticsStrategy { // per-dialect for a DIFFERENT reason than its neighbours: the fold this // line used to emit unconditionally was `translate()`, which SQLite does // not have, so the statement failed to PARSE there rather than answering - // wrong rows. `objectql-strategy.ts`'s echo carries the same flag on the + // wrong rows — [#16028] including on the `unknown` arm, which is reached + // by an embedder that simply left the optional `sqlDialect` hook unwired. + // `objectql-strategy.ts`'s echo carries the same flag on the // same single row, so the two keep describing one query. // // [#5567] Escaped pattern AND an explicit `ESCAPE`, bound together on diff --git a/packages/services/service-analytics/src/strategies/objectql-strategy.ts b/packages/services/service-analytics/src/strategies/objectql-strategy.ts index 44cf5a97e9..b3773cff25 100644 --- a/packages/services/service-analytics/src/strategies/objectql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/objectql-strategy.ts @@ -1253,7 +1253,11 @@ export class ObjectQLStrategy implements AnalyticsStrategy { // [#15684 / #15780] Both text families print what the DIALECT will run — // the case-EXACT four because `LIKE` is case-exact on Postgres alone, and // `$icontains` because its fold was `translate()`, which SQLite does not - // have at all. Asked of the same hook `NativeSQLStrategy` asks, on the + // have at all — [#16028] on the `unknown` arm too, which this compiler + // reaches by its OWN extra door: a caller that hands no target and no + // context gets `'unknown'` written in below, so the echo's residue arm is + // not merely inherited from the hook. Asked of the same hook + // `NativeSQLStrategy` asks, on the // same target, so the echo and the executed statement stay one // description — the whole reason this render block exists (#5333). An // echo that printed a parseable statement while the engine refused the diff --git a/packages/services/service-analytics/src/text-match-sql.ts b/packages/services/service-analytics/src/text-match-sql.ts index 4c3e69e340..054316a886 100644 --- a/packages/services/service-analytics/src/text-match-sql.ts +++ b/packages/services/service-analytics/src/text-match-sql.ts @@ -116,11 +116,45 @@ * - **`sqlite` → `lower()` around the GLOB arm.** ASCII-only there — * measured, `lower('CAFÉ')` is `cafÉ` — which is the #4706 Q1 = A boundary * executed rather than argued. - * - **`postgres` / `unknown` → `translate()`, unchanged.** These two arms - * were never broken; the bytes emitted before #15780 are the bytes emitted - * now. ⚠️ This DIVERGES from `driver-sql`, whose `unknown` arm folds with - * `LOWER()`: each face keeps the residue it already had, and neither claims - * the other's. + * - **`postgres` → `translate()`, unchanged.** Correct there, so the bytes + * emitted before #15780 are the bytes emitted now. + * - **[#16028] `unknown` → the portable nested-`REPLACE` fold** + * ({@link asciiLowerReplaceSql}). #15780 left this arm on `translate()` on + * the reading that it "was never broken" — which was true of the dialects + * that arm was PICTURED as (mssql, oracle, which have `translate()`) and + * false of the ones {@link normalizeSqlDialect} actually routes here. + * `unknown` is not a dialect: it is everything nothing answered for, and + * SQLite reaches it through four embedder compositions (a `SqlDriver` given + * a CLASS client or an unrecognised spelling, a host hook answering knex's + * own `'sqlite3'`, a directly-constructed `AnalyticsService` with the + * OPTIONAL `sqlDialect` omitted, a `data` service with no + * `getDriverForObject`). On every one of them `translate()` reached the + * engine and the statement failed to PARSE — a documented operator + * answering 500 because one optional field was left out. + * + * ⛔ The fix is NOT `LOWER()`, even though `driver-sql`'s own `unknown` arm + * folds that way: `LOWER()` follows the collation, so it would trade a loud + * parse failure on SQLite for SILENTLY wrong rows on PostgreSQL — the + * Unicode fold #4706 Q1 = A rules out. The `REPLACE` chain is the third + * answer: ASCII-only BY CONSTRUCTION, and parsed by every SQL dialect, + * because `REPLACE` is the one string function all of them have. So the two + * families `unknown` conflates are both served — PostgreSQL/Oracle-like get + * `translate()`'s exact result set in different bytes, SQLite-like get an + * answer at all. + * + * ⚠️ The residue that REMAINS, stated because the arm is a residue and not + * a dialect: the fold is exact everywhere, but the COMPARISON is `LIKE`, + * which on a case-/accent-insensitive collation (MySQL/MariaDB reaching + * here through the `'mariadb'` spelling #11756 deliberately leaves + * unrecognised; SQL Server) over-matches beyond ASCII. That is the SAME + * residue this arm's case-EXACT neighbour above already carries and names, + * not a new one — and on those engines `translate()` did not run at all, + * so nothing that answered correctly before stops. + * + * ⚠️ This still DIVERGES from `driver-sql`, whose `unknown` arm folds with + * `LOWER()`. Each face keeps its own answer and neither claims the other's; + * what changed is that this one's is now portable rather than merely + * inherited. * - **`mysql` → the nested-`REPLACE` binary fold** * ({@link mysqlAsciiLowerBinarySql}). NOT MEASURED — no MySQL server is * provisionable in this container, so this cell is a declared skip, exactly @@ -247,7 +281,40 @@ export type TextMatchBind = (value: unknown) => string; * than a fold on top of the collation's own. */ function mysqlAsciiLowerBinarySql(expr: string): string { - let out = `CAST(${expr} AS BINARY)`; + return asciiLowerReplaceSql(`CAST(${expr} AS BINARY)`); +} + +/** + * [#16028] The ASCII-ONLY case fold in the ONE spelling every SQL dialect + * parses: one nested `REPLACE` per letter of {@link ASCII_UPPER_LETTERS}. + * + * This is the {@link mysqlAsciiLowerBinarySql} chain with its `CAST(… + * AS BINARY)` removed — one builder, two callers, so the MySQL arm's bytes and + * the `unknown` arm's fold cannot drift into two different alphabets. The + * `CAST` is the part that is MySQL's (byte-wise comparison whatever the + * collation says); the chain itself is nobody's dialect in particular, which is + * exactly why the residue arm can use it. + * + * ## Why the chain equals `translate()` rather than approximating it + * + * `translate(x, 'ABC…', 'abc…')` maps each `A`-`Z` occurrence SIMULTANEOUSLY; + * this applies the 26 maps in sequence. The two agree because no step can feed + * a later one: every replacement WRITES a lower-case letter and every later + * step MATCHES an upper-case one, so nothing a `REPLACE` produces is a target + * further down the chain. Measured rather than left as that argument — + * `icontains-dialect-sql.test.ts` runs the emitted chain over every ASCII code + * point on the engine and requires the ASCII-only map exactly. + * + * ⛔ Not `LOWER()`, which is what {@link asciiLowerSqlExpr}'s header refuses for + * the same arm and for the same reason: `LOWER()` follows the database's + * collation, so on PostgreSQL it folds `É` to `é` and silently restores the + * Unicode fold #4706 Q1 = A rules out. SQLite's `lower()` happens to be + * ASCII-only, which is why measuring `LOWER()` on THIS container's engine + * proves nothing about the arm — the trap is a green SQLite reading standing in + * for a PostgreSQL one. + */ +function asciiLowerReplaceSql(expr: string): string { + let out = expr; for (let i = 0; i < ASCII_UPPER_LETTERS.length; i++) { out = `REPLACE(${out}, '${ASCII_UPPER_LETTERS[i]}', '${ASCII_LOWER_LETTERS[i]}')`; } @@ -309,9 +376,16 @@ export function textMatchPredicateSql(req: TextMatchRequest): string { } // `postgres` — where LIKE is already case-exact — and `unknown`, the residue. - // [#15780] The fold here is the `translate()` this package has always emitted - // ({@link asciiLowerSqlExpr}); on these two arms it was never the defect, so - // the correct diff is no diff. - const folded = (expr: string) => (fold ? asciiLowerSqlExpr(expr) : expr); + // The CONSTRUCT is shared: `LIKE` over an escaped pattern with a bound + // `ESCAPE`. Only the FOLD's spelling differs, and only when there is a fold at + // all — for the case-EXACT four `folded` is the identity on both, so those + // arms emit one set of bytes here as they always have. + // + // [#15780 → #16028] `postgres` keeps `translate()`, which is correct there and + // whose bytes must not move. `unknown` cannot: it is every dialect nothing + // answered for — SQLite and MySQL/MariaDB included — and `translate()` exists + // on neither, so the residue arm's fold has to be the portable one. + const asciiLower = dialect === 'postgres' ? asciiLowerSqlExpr : asciiLowerReplaceSql; + const folded = (expr: string) => (fold ? asciiLower(expr) : expr); return `${folded(column)} ${keyword} ${folded(bind(likePattern(shape, value)))} ESCAPE ${bind(LIKE_ESCAPE_CHAR)}`; } From 98de36223b08a7dcdfd148065bb593072cda5cba Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 6 Sep 2026 05:36:03 +0000 Subject: [PATCH 2/2] docs(changeset): the `unknown` arm's portable `$icontains` fold, graded and with its moved cells enumerated Refs #16028 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01XpTx2tbq3pZRYAdoGt6E6Y --- ...unknown-dialect-icontains-portable-fold.md | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 .changeset/analytics-unknown-dialect-icontains-portable-fold.md diff --git a/.changeset/analytics-unknown-dialect-icontains-portable-fold.md b/.changeset/analytics-unknown-dialect-icontains-portable-fold.md new file mode 100644 index 0000000000..3429078ddc --- /dev/null +++ b/.changeset/analytics-unknown-dialect-icontains-portable-fold.md @@ -0,0 +1,21 @@ +--- +"@objectstack/service-analytics": patch +--- + +Analytics `$icontains` no longer compiles a `translate()` call on the `unknown` dialect arm, so a datasource whose dialect nothing answered — which includes SQLite — gets a statement its engine can parse. **Graded `patch`:** no exported type, signature or option changes; the package's own contract for the operator (#4706 Q1 = A, an ASCII-only fold on both sides) is unchanged, and this repairs an arm that could not run rather than adding or retiring behaviour. What moves is emitted SQL text on one arm, measured and enumerated below. + +`normalizeSqlDialect` maps **everything it cannot name** onto `unknown`: an unset `sqlDialect` hook, `'oracle'`, `'libsql'`, a `SqlDriver` handed a knex Client **class** rather than a spelling. #15780 left that arm folding with `translate()` and recorded it as "never broken", which was true of the dialects the arm was *pictured* as — mssql and oracle, which have `translate()` — and false of the ones actually routed there. Measured on sql.js 1.14.1 (SQLite 3.49.1, the engine `driver-sqlite-wasm` runs), `SELECT translate('ABC','ABC','abc')` answers `no such function: translate`, so on all three of this package's compilers — the query's own `where` (`NativeSQLStrategy.buildFilterClause`), the ADR-0021 D-C read scope (`compileScopedFilterToSql`) and the `ObjectQLStrategy` echo — the statement failed to **parse**. It reached the client as a 500, not an ADR-0112 refusal. One of the four constructions that land there is a directly-constructed public `AnalyticsService` with its **optional** `sqlDialect` omitted: leaving out an optional field turned a documented operator into a 500. + +The `unknown` arm now folds with one nested `REPLACE` per ASCII letter — the chain the MySQL arm already used, minus its `CAST(… AS BINARY)`, so there is one builder and the two arms cannot fold different alphabets. `REPLACE` is the one string function every SQL dialect has, and the domain is the same 26-letter constant, so the fold is ASCII-only **by construction**: + +- **PostgreSQL / Oracle-like** — same result set as `translate()`. The chain equals the simultaneous `A`-`Z` map because no step can feed a later one: every replacement writes a lower-case letter and every later step matches an upper-case one. Measured on the engine over **every ASCII code point** plus accented, Greek, Cyrillic and dotted-I probes, required equal to the ASCII-only map exactly. +- **SQLite-like** — it runs. Executed over the shared `FILTER_TEXT_CASES` `$icontains` rows through all three compilers on sql.js: the same row sets the `sqlite` arm is required to answer, including the `CAFÉ`/`café` pair that separates an ASCII fold from a Unicode one. +- ⛔ **Not `LOWER()`**, which is what `driver-sql`'s own `unknown` arm folds with. `LOWER()` follows the collation, so adopting it would trade this parse failure for **silently wrong rows** on PostgreSQL — the Unicode fold #4706 Q1 = A rules out. ⚠️ Measuring `LOWER()` in this container proves nothing about that: SQLite's `lower()` is ASCII-only and passes the same fixture, which is exactly the trap of letting a green SQLite reading stand in for a PostgreSQL one. No PostgreSQL server was contacted. + +**Which cells moved.** The emitted SQL and bound params of `{NativeSQLStrategy, ObjectQLStrategy echo, compileScopedFilterToSql} × {undefined, 'unknown', 'oracle', 'libsql', 'postgres', 'sqlite', 'mysql'} × 5 text operators × 17 comparands` = **1,785 cells**, generated at this head and again with the emitter reverted to its merge-base blob (both legs hash-verified on disk and rebuilt, the marker's presence and absence checked in `dist/`): **204 moved, 1,581 byte-identical, 0 error cells either side.** Every moved cell is `$icontains` on one of the four dialect inputs that normalize to `unknown` (51 each = 17 comparands × 3 compilers). **0 of the 204 changed their bound params** — only the fold's spelling moved, never the escaping or the `ESCAPE` binding. Nothing moved on `postgres`, `sqlite` or `mysql`, and no case-exact operator moved on any dialect input. + +⚠️ **The cost, stated rather than left to be found:** the predicate grows from 168 to 1,014 characters on the read scope (233 → 1,079 on the other two). Both constructs are non-sargable scalar expressions over the column, so the plan class is unchanged — what grows is statement text and per-row work, on the arm where the alternative was a statement that did not run. + +⚠️ **The residue that remains**, because this arm is a residue and not a dialect: the fold is exact everywhere, but the comparison is `LIKE`, which on a case- or accent-insensitive collation (MySQL/MariaDB arriving here through the `'mariadb'` spelling #11756 deliberately leaves unrecognised; SQL Server) over-matches beyond ASCII. That is the **same** residue this arm's case-exact neighbour already carries and names — not a new one — and on those engines `translate()` did not run at all, so nothing that answered correctly before stops answering. + +`SqliteWasmDriver.dialectName` gains a direct pin. It answers `"sqlite"` only through an `isSqlite` override (the base class string-matches `config.client`, and this transport passes a class), that override had **0 direct test hits**, and it is the sole reason no in-repo SQLite driver reaches the arm above. The new pin includes the control: the base class answers `'unknown'` for that very config.