diff --git a/.changeset/engine-text-operator-declared-type-door.md b/.changeset/engine-text-operator-declared-type-door.md new file mode 100644 index 0000000000..f5962643db --- /dev/null +++ b/.changeset/engine-text-operator-declared-type-door.md @@ -0,0 +1,66 @@ +--- +"@objectstack/objectql": minor +"@objectstack/spec": patch +--- + +feat(objectql)!: refuse a text operator aimed at a field whose DECLARED type can never store a string — `INVALID_FILTER` 400 at the engine's field-aware door (#15773) + + + +**BREAKING** for a caller that aims `$contains` / `$notContains` / `$startsWith` / `$endsWith` / `$icontains` / `$like` / `$ilike` at a numeric, boolean, temporal or structured-JSON field: the call used to be answered (with `[]`, with every row for `$notContains`, or with a dialect accident) and is now refused with `400 INVALID_FILTER`. Shipped as `minor` under the repo's launch-window convention. Execution lane (2) of the maintainer ruling on #15661 (decision batch #43, option C-deny); lane (1) is the contract it consults, `@objectstack/spec/data`'s `filter-text-operator-declared-type.ts` (#15804). + +## What was wrong + +Measured on `origin/main` `59db8a02cb` with a real `ObjectQL`, the lane-1 fixture registered and a recording driver beneath — the filter reached the driver verbatim every time: + +| filter | before | after | +|:--|:--|:--| +| `{ f_number: { $contains: '5' } }` | driver read, `[]` | `400 INVALID_FILTER` | +| `{ f_summary: { $contains: '5' } }` | driver read, `[]` | `400 INVALID_FILTER` | +| `{ f_json: { $contains: 'a' } }` | driver read, `[]` | `400 INVALID_FILTER` | +| `{ f_date: { $startsWith: '2026' } }` | `400 INVALID_FILTER` — from the #8690 TEMPORAL door, about the COMPARAND | `400 INVALID_FILTER`, naming the field's declared type | +| `{ f_text: { $contains: 'a' } }` | driver read | unchanged — driver read | + +What the driver then answered is #14079's option-A row: no row for a positive operator, EVERY row for `$notContains`. Neither answer is wrong beneath the door — it is the declared answer — and neither carries any signal that the field can never hold a string, which is the cell this closes. + +## What it does now + +- **One door, at the engine's single filter collection point** (`lowerWhereFilterArray`), third in the ladder: comparand shape (#5869) → materializable field (#8296 / #8371) → **declared type (this)** → temporal comparand (#8690). It runs before the temporal gate deliberately: a text operator over a `date` field was already refused there, with the same wire envelope but a message about the comparand, which sends the author to fix a value that could never have made the filter runnable. +- **The refused classes are DERIVED, never re-listed**: the verdict is `@objectstack/spec/data`'s `textOperatorDoorVerdict`, over `NUMERIC_VALUE_TYPES` ∪ `BOOLEAN_VALUE_TYPES` ∪ `CALENDAR_DATE_TYPES` ∪ `INSTANT_TYPES` ∪ `CLOCK_TIME_TYPES` ∪ `STRUCTURED_JSON_TYPES`. A type added to any of those sets is refused with no change in this package. String-valued classes pass unchanged — `STRING_VALUE_TYPES`, `autonumber`, option codes (single AND multi, so `tags` keeps its substring filter), reference ids and the file classes. +- **No vocabulary is minted.** `INVALID_FILTER` already exists (`StandardErrorCode`) and is this package's filter envelope; the refusal carries `code`, `status` and `httpStatus` per ADR-0112 D5, and names the field, its declared type and the operator. +- **Both filter forms and every verb**: the object form and the `FilterArray` sugar, on `find` / `findOne` / `count` / `aggregate` / `update` / `delete`, plus the per-aggregation `filter` position (#10576's second filter slot on `aggregate`) — a door that spoke on `where` alone would answer one mistake two ways within one verb. +- **Beneath the door nothing moves.** A direct driver call never passes this seam and keeps answering `FILTER_TEXT_CASES`' option-A row (#14079), as does `having` — both pinned. + +## Deliberately unjudged + +- **A dotted key** (`f_address.city`) — `filter-dotted-head`'s subject, whose structured-JSON heads are deliberately unjudged there (#8371). The door steps over it rather than re-closing that carve-out. +- **An unknown filter field** — the engine keeps its registry-less tolerance; this door adds no second opinion about a name. +- **A registry-less host** (`schema.fields` absent) — a door that cannot see the field map invents no verdict, the same early return both neighbours make. +- **`formula`** — judged one door earlier. `assertFilterIsMaterializable` (#8296) refuses every filter over a `formula` field with `INVALID_FIELD` 400, for the broader reason that no driver materialises a column for it, so a formula's declared `returnType` is never the deciding fact at this seam. Not reordered around: that would answer ONE condition with TWO wire codes chosen by `returnType`. The divergence from lane (1)'s formula rows is pinned by name in `engine-text-operator-declared-type-door.test.ts` rather than dropped. + +## The ADR-0087 ledger entry, and why this is `registered` rather than `not-required` + +`@objectstack/spec` carries one new semantic migration entry, `filter-text-operator-declared-type-refused` (protocol 18) — the `patch` bump above is that entry and nothing else; no schema, no export and no published set moved. + +It is a real registration because the refused shape has an AUTHORED, STORED surface, measured on the tree rather than assumed. Nothing rejects a stored filter at load — `FilterConditionSchema` constrains no field type, and `ViewFilterRuleSchema` takes `field: z.string()` with `contains` in its operator enum — so a filter body written before this change still parses, still loads, and answers `400` the next time it is executed. Carriers measured to reach this seam: + +| stored surface | how it reaches the door | +|:--|:--| +| `sys_saved_report.query_json.filter` | `report-service.ts` runs `engine.find(report.object_name, { where: q.filter })` verbatim; every `sys_report_schedule` row reaches the same body through `report_id` | +| `FieldSchema.summaryOperations[].filter` | `summary-aggregate.ts` ANDs it with the parent-FK match and calls `engine.aggregate` | +| `ListView.filter`, tab filters (`ViewFilterRuleSchema`) | `contains` / `not_contains` / `icontains` / `starts_with` / `ends_with` lower to the same operators through `AST_OPERATOR_MAP` | +| dashboard widget / `GlobalFilter`, dataset `filter`, report `runtimeFilter`, `FieldSchema.relatedListFilter` | `FilterConditionSchema` carriers, executed through the same engine seam | + +**Not** on that list, deliberately: an RLS / sharing / tenant predicate. Those are composed onto the AST by the middleware chain AFTER this door, so the door never judges one — a policy filter cannot become a 400 nobody can act on. + +No mechanical rewrite exists, which is exactly what a `semantic` entry is for: `{ amount: { $contains: '5' } }` may have meant `$eq: 5`, a range, or a different column, and `objectstack migrate meta` must not choose. The entry ships the repair procedure and its acceptance criteria instead. + +## FROM → TO + +| you wrote | write instead | +|:--|:--| +| `where: { amount: { $contains: '500' } }` | `where: { amount: { $eq: 500 } }` (or `$gte` / `$lte` for a range) | +| `where: { created_at: { $startsWith: '2026' } }` | `where: { created_at: { $gte: '2026-01-01', $lt: '2027-01-01' } }` | +| `where: { is_open: { $contains: 'true' } }` | `where: { is_open: true }` | +| `where: { address: { $contains: 'Berlin' } }` | filter a stored text field, or `where: { 'address.city': { $contains: 'Berlin' } }` (a dotted path stays unjudged) | +| `where: { tags: { $contains: 'urgent' } }` | unchanged — option codes are strings and still pass | diff --git a/content/docs/protocol/objectql/query-syntax.mdx b/content/docs/protocol/objectql/query-syntax.mdx index 72bc2ca06f..63d9431349 100644 --- a/content/docs/protocol/objectql/query-syntax.mdx +++ b/content/docs/protocol/objectql/query-syntax.mdx @@ -1093,11 +1093,21 @@ label→value mapping in the expansion rather than by the operator. subject — the divergence it warned about is closed: running your tests on the in-memory double no longer returns rows a SQL or MongoDB deployment would not because of a fold, and `FILTER_TEXT_CASES` holds all five drivers to that rule. - ⚠️ **One axis is NOT aligned, and it is not a case question.** Over a column whose - DECLARED type is temporal (`date` / `datetime` / `time`), the SQL family answers a - text operator with a declared no-match — `$contains` and its positive siblings - select nothing, `$notContains` selects every valued row — by the maintainer ruling - on [#15683](https://github.com/objectstack-ai/objectstack/issues/15683), while + ⚠️ **One axis is NOT aligned, it is not a case question, and which answer you see + depends on how you reach the data.** **Through the engine** — `ql.find()` and every + surface above it (REST `where` / `$filter`, saved reports, list views) — a text + operator aimed at a field whose DECLARED type can never store a string is REFUSED + with `400 INVALID_FILTER` before any driver dispatch, naming the field and its + declared type, by the maintainer ruling on + [#15661](https://github.com/objectstack-ai/objectstack/issues/15661): the refused + classes are the numeric, boolean, temporal (`date` / `datetime` / `time`) and + structured-JSON value classes, and there is no result set to compare. **Beneath that + door** — a DIRECT driver call, the path that ruling deliberately leaves answering — + each driver's own answer still stands, and they diverge: over a column whose DECLARED + type is temporal the SQL family answers a declared no-match — `$contains` and its + positive siblings select nothing, `$notContains` selects every valued row — by the + maintainer ruling on + [#15683](https://github.com/objectstack-ai/objectstack/issues/15683), while `driver-memory` canonicalises such a column to ISO text on write and so still MATCHES the substring. On that one axis the in-memory double does return rows a SQL deployment would not; it is tracked as diff --git a/packages/objectql/src/engine-text-operator-declared-type-door.test.ts b/packages/objectql/src/engine-text-operator-declared-type-door.test.ts new file mode 100644 index 0000000000..440cf85c86 --- /dev/null +++ b/packages/objectql/src/engine-text-operator-declared-type-door.test.ts @@ -0,0 +1,408 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#15773] The text-operator DECLARED-TYPE door at the engine's filter + * collection point — execution lane (2) of the #15661 C-deny ruling. + * + * The door's definition lives in `@objectstack/spec/data` + * (`filter-text-operator-declared-type.ts`, lane 1, where the ruling is quoted + * verbatim); this file is its CONSUMER, driven exactly the way that module's + * header prescribes: register {@link TEXT_OPERATOR_DOOR_FIXTURE} against a + * recording driver, run every case of {@link TEXT_OPERATOR_DOOR_CASES} through + * `find`, and assert per verdict — + * + * - `door-refusal`: rejects with `code` AND `status` (the ADR-0112 envelope — + * `toThrow()` alone is not a pin), the message carries every `mustMention` + * substring, and NO driver read ran. + * - `passes` / `deferred`: the driver read ran and received the filter + * UNCHANGED. + * + * Because the case table is DERIVED from the published value classes, a type + * added to `NUMERIC_VALUE_TYPES` (or to any other class) arrives here as a new + * case with no edit in this package — which is what lane (1)'s "⛔ no new set" + * buys at the consuming end. + * + * ## ⚠️ One NAMED DIVERGENCE, measured rather than dropped: `formula` + * + * The table's `formula` rows cannot be consumed as written AT THIS SEAM, and + * the reason is a neighbouring door rather than this one. `formula` is the one + * field type no driver materialises a column for, so + * `assertFilterIsMaterializable` (#8296) refuses EVERY filter over one, one + * step above this door, with `INVALID_FIELD` / 400. Measured on `origin/main` + * 59db8a02cb, before this card's door existed, for all three return-type + * shapes: + * + * ``` + * find(o, { where: { f_formula_number: { $contains: '5' } } }) -> INVALID_FIELD 400 (#8296) + * find(o, { where: { f_formula_text: { $contains: '5' } } }) -> INVALID_FIELD 400 (#8296) + * find(o, { where: { f_formula_untyped: { $contains: '5' } } }) -> INVALID_FIELD 400 (#8296) + * ``` + * + * So at this seam a `formula`'s declared return type is never the deciding + * fact, and the door was NOT reordered to make it one: doing so would answer + * ONE condition ("a formula field cannot be filtered") with TWO wire codes + * chosen by `returnType`. The rows are pinned below in the direction they + * actually answer — refused, `INVALID_FIELD`, no driver read — so that the day + * formula fields become filterable this file goes RED and the divergence is + * re-judged rather than silently inherited. + * + * ## Beneath the door, #14079's option-A row is untouched + * + * The ruling keeps that row for "every evaluator no door fronts". Two of them + * are pinned here: a DIRECT driver call (the door is at the engine seam, not + * inside a driver) and `having` (this package's own evaluator, which no filter + * door fronts), held to `FILTER_TEXT_CASES`' own numeric-column rows. + * + * @see https://github.com/objectstack-ai/objectstack/issues/15773 (this lane) + * @see https://github.com/objectstack-ai/objectstack/issues/15661 (the ruling) + * @see https://github.com/objectstack-ai/objectstack/issues/14079 (the row beneath) + */ + +import { describe, it, expect, beforeEach } from 'vitest'; +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; +import { + FILTER_TEXT_CASES, + FILTER_TEXT_ROWS, + TEXT_FILTER_OPERATORS, + TEXT_OPERATOR_DOOR_CASES, + TEXT_OPERATOR_DOOR_FIXTURE, + TEXT_OPERATOR_DOOR_FIXTURE_OBJECT, + TEXT_OPERATOR_DOOR_TYPE_CLASSES, + type FilterTextCase, + type FilterTextRowsCase, + type TextOperatorDoorCase, + type TextOperatorDoorRefusalCase, + type EngineAggregateOptions, + type EngineQueryOptions, +} from '@objectstack/spec/data'; +import { ObjectQL } from './engine.js'; +import { applyHaving } from './having-filter.js'; +import { findTextOperatorOverNonTextField } from './text-operator-declared-type-door.js'; + +const OBJECT = TEXT_OPERATOR_DOOR_FIXTURE_OBJECT; + +interface SeenRead { ast: any } + +/** Minimal recording driver — the same witness shape as the #7872 door suite. */ +function makeRecordingDriver() { + const rows = new Map>(); + const reads: SeenRead[] = []; + const writes: SeenRead[] = []; + const run = (_ast: any) => [...rows.values()]; + const driver: any = { + name: 'recording', version: '0.0.0', supports: {}, + async connect() {}, async disconnect() {}, async checkHealth() { return true; }, async execute() { return null; }, + async find(_o: string, ast: any) { reads.push({ ast }); return run(ast); }, + async findOne(_o: string, ast: any) { reads.push({ ast }); return run(ast)[0] ?? null; }, + async count(_o: string, ast: any) { reads.push({ ast }); return run(ast).length; }, + async create(_o: string, data: Record) { + const id = (data.id as string) ?? `r_${rows.size + 1}`; + const row = { ...data, id }; rows.set(id, row); return row; + }, + async update(_o: string, id: string, data: Record) { + const cur = rows.get(id) ?? {}; + const up = { ...cur, ...data, id }; rows.set(id, up); return up; + }, + async updateMany(_o: string, ast: any) { writes.push({ ast }); return 0; }, + async delete(_o: string, id: string) { return rows.delete(id); }, + async deleteMany(_o: string, ast: any) { writes.push({ ast }); return 0; }, + async bulkCreate(o: string, batch: Record[]) { + return Promise.all(batch.map((r) => this.create(o, r))); + }, + async beginTransaction() { return { commit: async () => {}, rollback: async () => {} }; }, + async commit() {}, async rollback() {}, + }; + return { driver, reads, writes, rows }; +} + +type Thrown = (Error & { code?: string; status?: number; httpStatus?: number }) | null; + +const refusalOf = async (p: Promise): Promise => + p.then(() => null, (e: any) => e as Error & { code?: string; status?: number }); + +/** A `formula` case — judged one door earlier, see the header's divergence note. */ +const isFormulaCase = (c: TextOperatorDoorCase): boolean => c.declaredType === 'formula'; + +describe('[#15773] the text-operator declared-type door at the engine collection point', () => { + let engine: ObjectQL; + let reads: SeenRead[]; + let writes: SeenRead[]; + let driver: any; + + beforeEach(async () => { + const rec = makeRecordingDriver(); + reads = rec.reads; + writes = rec.writes; + driver = rec.driver; + engine = new ObjectQL(); + engine.registerDriver(rec.driver, true); + await engine.init(); + engine.registry.registerObject(TEXT_OPERATOR_DOOR_FIXTURE as any, 'test'); + await engine.insert(OBJECT, { id: 'r1' }); + reads.length = 0; + writes.length = 0; + }); + + // ── the derived case table, driven end to end ──────────────────────────── + + const REFUSALS = TEXT_OPERATOR_DOOR_CASES.filter( + (c): c is TextOperatorDoorRefusalCase => c.verdict === 'door-refusal' && !isFormulaCase(c)); + const PASSES = TEXT_OPERATOR_DOOR_CASES.filter((c) => c.verdict === 'passes' && !isFormulaCase(c)); + const DEFERRED = TEXT_OPERATOR_DOOR_CASES.filter((c) => c.verdict === 'deferred' && !isFormulaCase(c)); + const FORMULA = TEXT_OPERATOR_DOOR_CASES.filter(isFormulaCase); + + it('GUARD the case table is non-empty in all four partitions — a shrunk table cannot read as coverage', () => { + expect(TEXT_OPERATOR_DOOR_CASES.length).toBe(REFUSALS.length + PASSES.length + DEFERRED.length + FORMULA.length); + expect(REFUSALS.length).toBeGreaterThan(0); + expect(PASSES.length).toBeGreaterThan(0); + expect(DEFERRED.length).toBeGreaterThan(0); + expect(FORMULA.length).toBeGreaterThan(0); + // Every text operator the ruling names is exercised, not just `$contains`. + expect(new Set(REFUSALS.map((c) => c.operator))).toEqual(new Set(TEXT_FILTER_OPERATORS)); + }); + + it('refuses every door-refusal case with the ADR-0112 envelope, and NO driver read runs', async () => { + for (const c of REFUSALS) { + reads.length = 0; + const err = await refusalOf(engine.find(OBJECT, { where: c.filter() })); + expect(err, `${c.name}: expected a refusal`).not.toBeNull(); + // `code` AND `status` — `toThrow()` alone would stay green for a bare Error. + expect({ code: err!.code, status: err!.status }, c.name).toEqual({ code: c.code, status: c.status }); + // ADR-0112 D5's second spelling, which the CLI's `--json` envelope reads. + expect(err!.httpStatus, c.name).toBe(400); + for (const substring of c.mustMention) { + expect(err!.message, `${c.name}: message must mention ${substring}`).toContain(substring); + } + // The engine's wording contract, shared with the three sibling gates. + expect(err!.message, c.name).toMatch(/^find\('text_door_probe'\): /); + expect(err!.message, c.name).toMatch(/NOT applied/); + // The whole point of a DOOR: refused before any driver dispatch. + expect(reads, `${c.name}: the driver must not have been read`).toHaveLength(0); + } + }); + + it('lets every passing case through UNCHANGED — the door rewrites nothing', async () => { + for (const c of PASSES) { + reads.length = 0; + const filter = c.filter(); + await expect(engine.find(OBJECT, { where: filter }), c.name).resolves.toBeDefined(); + expect(reads, `${c.name}: the driver must have been read`).toHaveLength(1); + expect(reads[0]?.ast?.where, `${c.name}: the filter must reach the driver unchanged`).toEqual(filter); + } + }); + + it('records NO verdict for a deferred case — a dotted path into structured JSON stays #8371\'s carve-out', async () => { + for (const c of DEFERRED) { + reads.length = 0; + const filter = c.filter(); + await expect(engine.find(OBJECT, { where: filter }), c.name).resolves.toBeDefined(); + expect(reads, `${c.name}: the driver must have been read`).toHaveLength(1); + expect(reads[0]?.ast?.where, `${c.name}: the filter must reach the driver unchanged`).toEqual(filter); + } + }); + + it('NAMED DIVERGENCE — every formula case is refused one door EARLIER, by #8296, with INVALID_FIELD', async () => { + // Not the table's own expectation for the refusal rows (`INVALID_FILTER`), + // and deliberately so — the header carries the measurement and the reason + // the ladder was not reordered around it. Pinned in the direction it + // answers so that a change to either door lands here first. + for (const c of FORMULA) { + reads.length = 0; + const err = await refusalOf(engine.find(OBJECT, { where: c.filter() })); + expect(err, `${c.name}: expected the #8296 refusal`).not.toBeNull(); + expect({ code: err!.code, status: err!.status }, c.name).toEqual({ code: 'INVALID_FIELD', status: 400 }); + expect(err!.message, c.name).toContain('virtual formula field'); + expect(reads, c.name).toHaveLength(0); + } + // …and the door's own verdict function still judges the class correctly, + // so the day that neighbour opens up, this door already answers. + const fields = (engine.registry.getObject(OBJECT) as any).fields; + expect(findTextOperatorOverNonTextField({ fields }, { f_formula_number: { $contains: '5' } })) + .toMatchObject({ field: 'f_formula_number', declaredType: 'formula', returnType: 'number', operator: '$contains' }); + expect(findTextOperatorOverNonTextField({ fields }, { f_formula_text: { $contains: '5' } })).toBeNull(); + expect(findTextOperatorOverNonTextField({ fields }, { f_formula_untyped: { $contains: '5' } })).toBeNull(); + }); + + it('pins EVERY class of the verdict matrix — refused and passing alike, one member at a time', async () => { + // Per-CLASS pins, not per-case: the case drive above could stay green with + // a class silently absent from the fixture, and "pins per refused class AND + // per passing class" is the card's own deliverable. + let refusedClasses = 0; + let passingClasses = 0; + for (const row of TEXT_OPERATOR_DOOR_TYPE_CLASSES) { + if (row.verdict === 'by-return-type') continue; // `formula` — the divergence above. + for (const type of row.types) { + reads.length = 0; + const field = `f_${type}`; + const err = await refusalOf(engine.find(OBJECT, { where: { [field]: { $contains: '5' } } })); + if (row.verdict === 'door-refusal') { + expect(err, `${row.name}/${type}: must be refused — ${row.note}`).not.toBeNull(); + expect({ code: err!.code, status: err!.status }, `${row.name}/${type}`) + .toEqual({ code: 'INVALID_FILTER', status: 400 }); + expect(err!.message, `${row.name}/${type}`).toContain(type); + expect(reads, `${row.name}/${type}`).toHaveLength(0); + } else { + expect(err, `${row.name}/${type}: must pass — ${row.note}`).toBeNull(); + expect(reads, `${row.name}/${type}`).toHaveLength(1); + } + } + if (row.verdict === 'door-refusal') refusedClasses += 1; else passingClasses += 1; + } + // Both halves were actually exercised — a matrix that lost one verdict + // entirely must not read as a green run. + expect(refusedClasses).toBeGreaterThan(0); + expect(passingClasses).toBeGreaterThan(0); + }); + + // ── the door's reach: every verb, both filter forms, nested structure ──── + + it('covers every engine verb that collects a filter — read and write sides', async () => { + const where = { f_number: { $contains: '5' } }; + for (const call of [ + () => engine.find(OBJECT, { where }), + () => engine.findOne(OBJECT, { where }), + () => engine.count(OBJECT, { where }), + () => engine.aggregate(OBJECT, { where, aggregations: [{ function: 'count', alias: 'n' }] } as EngineAggregateOptions), + () => engine.update(OBJECT, { id: 'r1' }, { where, multi: true }), + () => engine.delete(OBJECT, { where, multi: true }), + ]) { + const err = await refusalOf(call()); + expect(err).not.toBeNull(); + expect({ code: err!.code, status: err!.status }).toEqual({ code: 'INVALID_FILTER', status: 400 }); + } + expect(reads).toHaveLength(0); + expect(writes).toHaveLength(0); + }); + + it('refuses the same mistake arriving as FilterArray sugar — one answer per mistake, not per spelling', async () => { + // The cast names the contract being bypassed: `FilterArray` is INPUT-ONLY + // sugar `EngineQueryOptions.where` deliberately excludes (#5285). + const err = await refusalOf( + engine.find(OBJECT, { where: [['f_number', 'contains', '5']] } as unknown as EngineQueryOptions), + ); + expect({ code: err!.code, status: err!.status }).toEqual({ code: 'INVALID_FILTER', status: 400 }); + expect(err!.message).toContain('f_number'); + expect(reads).toHaveLength(0); + }); + + it('reaches inside $and / $or / $not — structure does not launder the predicate', async () => { + for (const where of [ + { $and: [{ f_text: { $contains: 'a' } }, { f_number: { $contains: '5' } }] }, + { $or: [{ f_text: { $contains: 'a' } }, { f_boolean: { $startsWith: 't' } }] }, + { $not: { f_date: { $endsWith: '01' } } }, + ]) { + reads.length = 0; + const err = await refusalOf(engine.find(OBJECT, { where })); + expect(err, JSON.stringify(where)).not.toBeNull(); + expect({ code: err!.code, status: err!.status }).toEqual({ code: 'INVALID_FILTER', status: 400 }); + expect(reads).toHaveLength(0); + } + }); + + it('refuses a text operator in ONE aggregation\'s own filter — the second filter position on the verb', async () => { + const err = await refusalOf(engine.aggregate(OBJECT, { + aggregations: [ + { function: 'count', alias: 'all' }, + { function: 'count', alias: 'bad', filter: { f_number: { $contains: '5' } } }, + ], + } as EngineAggregateOptions)); + expect(err).not.toBeNull(); + expect({ code: err!.code, status: err!.status }).toEqual({ code: 'INVALID_FILTER', status: 400 }); + expect(err!.message).toContain('f_number'); + expect(reads).toHaveLength(0); + }); + + // ── where the door deliberately has NO opinion ─────────────────────────── + + it('GUARD a registry-less host gets no verdict — a door that cannot see the field map invents none', () => { + expect(findTextOperatorOverNonTextField(undefined, { f_number: { $contains: '5' } })).toBeNull(); + expect(findTextOperatorOverNonTextField({}, { f_number: { $contains: '5' } })).toBeNull(); + expect(findTextOperatorOverNonTextField({ fields: {} }, { f_number: { $contains: '5' } })).toBeNull(); + }); + + it('GUARD an UNKNOWN filter field keeps the engine\'s registry-less tolerance — no second opinion about a name', async () => { + await expect(engine.find(OBJECT, { where: { not_a_field: { $contains: 'x' } } })).resolves.toBeDefined(); + expect(reads).toHaveLength(1); + }); + + it('GUARD a NON-text operator over the same refused field is untouched', async () => { + for (const where of [ + { f_number: { $gt: 5 } }, + { f_number: 5 }, + { f_boolean: { $eq: true } }, + { f_json: { $ne: null } }, + ]) { + reads.length = 0; + await expect(engine.find(OBJECT, { where }), JSON.stringify(where)).resolves.toBeDefined(); + expect(reads, JSON.stringify(where)).toHaveLength(1); + } + }); + + it('GUARD an unrecognised $ combinator leaves the fields beneath it ungated — a hole, never a false 400', () => { + expect(findTextOperatorOverNonTextField( + { fields: { f_number: { type: 'number' } } }, + { $nor: [{ f_number: { $contains: '5' } }] }, + )).toBeNull(); + }); + + // ── the REST doors that reach findData ────────────────────────────────── + + describe('the REST doors — one answer however the query arrived', () => { + let protocol: ObjectStackProtocolImplementation; + + beforeEach(() => { + protocol = new ObjectStackProtocolImplementation(engine); + }); + + const DOORS: ReadonlyArray<{ door: string; query: (field: string) => any }> = [ + // `where` object — `POST /data/:object/query` body. + { door: 'where object', query: (f) => ({ where: { [f]: { $contains: '5' } } }) }, + // `$filter` string — the OData spelling, JSON nested in a querystring value. + { door: '$filter string', query: (f) => ({ $filter: JSON.stringify({ [f]: { $contains: '5' } }) }) }, + // Filter AST — the sugar the ObjectUI client and FilterBuilder emit. + { door: 'filter AST', query: (f) => ({ filter: [[f, 'contains', '5']] }) }, + ]; + + it.each(DOORS)('the $door door refuses a text operator over a declared number field', async ({ query }) => { + const err = await refusalOf(protocol.findData({ object: OBJECT, query: query('f_number') })); + expect(err).not.toBeNull(); + expect({ code: err!.code, status: err!.status }).toEqual({ code: 'INVALID_FILTER', status: 400 }); + expect(err!.message).toContain('f_number'); + expect(reads).toHaveLength(0); + }); + + it.each(DOORS)('GUARD the $door door still applies a text operator over a text field', async ({ query }) => { + await expect(protocol.findData({ object: OBJECT, query: query('f_text') })).resolves.toBeDefined(); + expect(reads).toHaveLength(1); + }); + }); + + // ── beneath the door: #14079's option-A row, unmoved ───────────────────── + + describe('beneath the door — #14079\'s option-A row still answers', () => { + it('a DIRECT driver call is untouched: the door is at the engine seam, not inside a driver', async () => { + // The direct path the ruling protects — an embedder calling a driver it + // constructed, and this repo's own driver conformance suites. + await expect(driver.find(OBJECT, { where: { f_number: { $contains: '5' } } })).resolves.toBeDefined(); + expect(reads).toHaveLength(1); + expect(reads[0]?.ast?.where).toEqual({ f_number: { $contains: '5' } }); + }); + + it('`having` — an evaluator no filter door fronts — still answers FILTER_TEXT_CASES\' numeric-column rows', () => { + const rows = FILTER_TEXT_ROWS.map((r) => ({ ...r })); + const aRow = FILTER_TEXT_CASES.filter( + (c): c is FilterTextRowsCase => + c.expectRejection !== true && Object.keys(c.filter)[0] === 'score', + ); + // Derived from the table, so the row cannot quietly leave the standard. + expect(aRow.length).toBeGreaterThan(0); + for (const c of aRow) { + expect(applyHaving(rows, c.filter).map((r: any) => r.id), c.name).toEqual([...c.expected]); + } + // Both directions of the row are present in what was just asserted: + // the positive operators match nothing, `$notContains` matches every row. + expect(aRow.some((c: FilterTextCase & { expected?: readonly string[] }) => c.expected?.length === 0)).toBe(true); + expect(aRow.some((c: FilterTextCase & { expected?: readonly string[] }) => c.expected?.length === rows.length)).toBe(true); + }); + }); +}); diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index 66952d96d5..d14c056148 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -44,6 +44,7 @@ import { MAX_BULK_PER_ROW_HOOK_ROWS, resolveBulkPerRowHookBudget } from '@object import { ActionActivationProjection, type ActionActivationRow, type ActionActivationStore } from './action-activation.js'; import { assertListComparandShapes, assertFilterIsMaterializable } from './filter-comparand-shape.js'; import { assertTemporalComparandsInterpretable } from './temporal-comparand-door.js'; +import { assertTextOperatorTargetsAreStringCapable } from './text-operator-declared-type-door.js'; // Seek pagination for the walks that must read EVERY row — the autonumber seed // scan is one (#6249). Shared with `summary-backfill` rather than re-rolled: // the cursor merge is the part that is easy to get subtly wrong. @@ -778,10 +779,22 @@ function lowerWhereFilterArray( // run it against" — and because this seam is the one place EVERY // caller-supplied `where` passes through, whichever verb it arrived by. assertFilterIsMaterializable(object, operation, schema, where); - // [#8690] The TEMPORAL-comparand door, third on the same seam and third - // question about the same predicate: the shape gate asks "can this - // comparand run", the materializable gate asks "is there a column to run it - // against", and this asks "can that column's storage rule READ this value". + // [#15661] The DECLARED-TYPE door for the text operators, third on this + // seam and third question about the same predicate: the shape gate asks + // "can this comparand run", the materializable gate asks "is there a column + // to run it against", and this asks "can that column's declared type ever + // hold a string". It runs BEFORE the temporal gate below deliberately — a + // `$startsWith` over a `date` field is refused by that gate today, with the + // same `INVALID_FILTER` / 400 envelope but a message about the COMPARAND, + // which sends the author to fix a value that could never have made the + // filter runnable. Same wire envelope, the ruling's message. + assertTextOperatorTargetsAreStringCapable(object, operation, schema, where); + // [#8690] The TEMPORAL-comparand door, fourth on the same seam (#15661's + // declared-type door was inserted above it) and fourth question about the + // same predicate: the shape gate asks "can this comparand run", the + // materializable gate asks "is there a column to run it against", the + // declared-type gate asks "can that column ever hold a string", and this + // asks "can that column's storage rule READ this value". // It must run BEFORE `resolveWhereTokens` (which is downstream of every // caller of this function) because the refusal has to precede the driver — // hence the door steps around `{placeholder}` strings rather than judging @@ -856,6 +869,11 @@ function lowerWhereFilterArray( // the array sugar (`[['is_open','=',true]]`) names fields too, and a gate on // one branch would answer one mistake two ways depending on the spelling. assertFilterIsMaterializable(object, operation, schema, condition); + // [#15661] Same door as the object branch, on the LOWERED condition — the + // array sugar (`[['amount','contains','5']]`) names non-text fields too, and + // a gate on one branch would answer one mistake two ways depending on the + // spelling. + assertTextOperatorTargetsAreStringCapable(object, operation, schema, condition); // [#8690] Same door as the object branch, on the LOWERED condition — the // array sugar (`[['at','>=','last_30_days']]`) names temporal fields too, and // a gate on one branch would answer one mistake two ways depending on the @@ -14198,6 +14216,12 @@ export class ObjectQL implements IObjectQLEngine { if (aggFilter == null) continue; assertListComparandShapes(object, 'aggregate', aggFilter, `aggregations[${i}].filter`); assertFilterIsMaterializable(object, 'aggregate', this._registry.getObject(object), aggFilter); + // [#15661] …and the declared-type door for the text operators: a + // `$contains` over a numeric column in ONE aggregation's filter is + // the same silent zero at a second filter position, and a door that + // spoke on `where` alone would answer one mistake two ways within a + // single verb. + assertTextOperatorTargetsAreStringCapable(object, 'aggregate', this._registry.getObject(object), aggFilter); } const driver = this.getDriver(object); this.logger.debug(`Aggregate on ${object} using ${driver.name}`, query); diff --git a/packages/objectql/src/text-operator-declared-type-door.ts b/packages/objectql/src/text-operator-declared-type-door.ts new file mode 100644 index 0000000000..2cdfe493e2 --- /dev/null +++ b/packages/objectql/src/text-operator-declared-type-door.ts @@ -0,0 +1,273 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#15661] The TEXT-OPERATOR DECLARED-TYPE door, at the engine's single filter + * collection point — the fourth gate on the seam that already carries the + * #5869 comparand-shape gate, the #8296 unmaterializable-field gate and the + * #8690 temporal-comparand gate, answering a fourth question about the same + * predicate: *can the field this operator is aimed at ever hold a string.* + * + * ## The ruling this implements + * + * Maintainer ruling, 2026-09-05, recorded on #15661 (decision batch #43, + * option C-deny), quoted verbatim: + * + * > **C-deny, now**: a text operator (`$contains` / `$notContains` / + * > `$startsWith` / `$endsWith` / `$icontains` / `$like` / `$ilike`) over a + * > field whose DECLARED type can never store a string — `NUMERIC_VALUE_TYPES` + * > ∪ `BOOLEAN_VALUE_TYPES` ∪ `CALENDAR_DATE_TYPES` ∪ `INSTANT_TYPES` ∪ + * > `CLOCK_TIME_TYPES` ∪ `STRUCTURED_JSON_TYPES`, all existing sets in + * > `field-value.zod.ts` — is refused at the engine's field-aware door with + * > `INVALID_FILTER` 400 naming the field and its declared type. No new + * > vocabulary is minted. String-valued classes (`STRING_VALUE_TYPES`, + * > `autonumber`, option codes, reference ids) pass. `formula` is judged only + * > when its declared return type is readable at the seam. #14079's option-A + * > row stays beneath the door for every evaluator no door fronts. + * + * Execution lane (2) of that ruling is this file; lane (1) is the CONTRACT it + * consults — `@objectstack/spec/data`'s `filter-text-operator-declared-type.ts` + * (PR #15804), which owns the sets, the pure verdict + * ({@link textOperatorDoorVerdict}), the fixture and the case table. ⛔ Nothing + * here re-lists a class: a type added to `NUMERIC_VALUE_TYPES` tomorrow is + * refused by this door without a change in this package, which is what "⛔ no + * new set" means at the consuming end. + * + * ## What ran before this door existed, measured on `origin/main` 59db8a02cb + * + * A real {@link ObjectQL}, the lane-1 fixture object registered, a recording + * driver beneath — the filter reached the driver verbatim, every time: + * + * ``` + * find(o, { where: { f_number: { $contains: '5' } } }) -> driver read, where passed through + * find(o, { where: { f_summary: { $contains: '5' } } }) -> driver read, where passed through + * find(o, { where: { f_json: { $contains: 'a' } } }) -> driver read, where passed through + * find(o, { where: { f_date: { $startsWith: '2026' } } }) -> INVALID_FILTER, but from the + * #8690 TEMPORAL door, about the + * COMPARAND ("not a date value") + * ``` + * + * What the driver then answers is #14079's option-A row (`FILTER_TEXT_CASES`): + * no row for a positive operator, EVERY row for `$notContains`. Neither answer + * is wrong beneath the door — it is the declared answer — and neither carries + * any signal that the field can never hold a string. That silent cell is what + * the ruling closes here, above the evaluators rather than inside them. + * + * ## Where it sits in the ladder, and why exactly there + * + * Between the #8296 field gate and the #8690 comparand gate: + * + * 1. `assertListComparandShapes` — can this comparand run at all (#5869). + * 2. `assertFilterIsMaterializable` — is there a column to run it against + * (#8296 / #8371). + * 3. **this door** — can that column's DECLARED type ever hold a string. + * 4. `assertTemporalComparandsInterpretable` — can that column's storage rule + * read this VALUE (#8690). + * + * The declaration question precedes the value question because a text operator + * aimed at a `date` column is not a comparand mistake: measured above, + * `$startsWith: '2026'` over a `date` field was already refused — by the + * temporal door, with `INVALID_FILTER` 400 and a message reporting that + * `"2026"` is not a date value the platform can interpret. That is true and + * beside the point: no comparand would have made the filter runnable, and an + * author who "fixes" it to `$startsWith: '2026-01-01'` gets the silent cell + * back. Running BEFORE it keeps the wire envelope identical (`INVALID_FILTER` / + * 400 either way — no caller's error handling moves) and replaces the message + * with the one the ruling asked for: the field, its declared type, and the + * operator that cannot be aimed at it. + * + * ## `formula` is judged one door EARLIER, and this door never sees it + * + * The ruling judges a `formula` "only when its declared return type is readable + * at the seam", and lane (1) encodes that (`returnType: number|boolean|date` ⇒ + * refused, `text` ⇒ passes, absent ⇒ deferred). At THIS seam the question never + * arrives: `assertFilterIsMaterializable` (#8296) refuses EVERY filter over a + * `formula` field one step above, with `INVALID_FIELD` / 400, for the broader + * reason that no driver materialises a column for it — measured on the same + * tree, for all three return-type shapes: + * + * ``` + * find(o, { where: { f_formula_number: { $contains: '5' } } }) -> INVALID_FIELD 400 (#8296) + * find(o, { where: { f_formula_text: { $contains: '5' } } }) -> INVALID_FIELD 400 (#8296) + * find(o, { where: { f_formula_untyped: { $contains: '5' } } }) -> INVALID_FIELD 400 (#8296) + * ``` + * + * Deliberately NOT reordered around: overtaking that door would answer ONE + * condition ("a formula field cannot be filtered") with TWO wire codes decided + * by the formula's `returnType` — `INVALID_FILTER` for one returning `number` + * and `INVALID_FIELD` for one returning `text` — which is the split every door + * on this seam records its reasoning against. #8296's own note argues the code + * assignment (the verdict is about the NAME's type), and re-deciding it is not + * this card's. The formula rows of `TEXT_OPERATOR_DOOR_CASES` are therefore + * pinned in this package as a NAMED DIVERGENCE against the neighbouring door's + * envelope, not silently dropped — `engine-text-operator-declared-type-door.test.ts`. + * + * The verdict function is still consulted through its `formula` branch (this + * module hands it `returnType`), so the day that field class becomes filterable + * the door already answers it correctly. + * + * ## Scope — three boundaries + * + * - **UNDOTTED keys only.** A dotted key (`f_address.city`) is + * `filter-dotted-head`'s subject, and its structured-JSON heads are + * deliberately unjudged there (live on two of three backends, #8371). + * Reading the head's declared type here would re-close that carve-out, so a + * dotted key is stepped over — lane (1) declares the same `deferred`. + * - **A field the map does not declare is not judged.** The engine keeps its + * registry-less tolerance for unknown filter keys (the ingress door's first + * verdict, #7534); this door adds no second opinion about a name. + * - **The CALLER's own `where` only.** Like its two neighbours it runs before + * the middleware chain composes RLS / sharing / tenant predicates onto the + * AST: an injected read filter is the platform's own, not a declaration the + * caller can fix. + * + * @see textOperatorDoorVerdict — the pure verdict (lane 1, `@objectstack/spec`). + * @see FILTER_TEXT_CASES — #14079's row, which stays BENEATH this door. + * @see assertTemporalComparandsInterpretable — the value-half gate beside it. + * @see https://github.com/objectstack-ai/objectstack/issues/15661 (the ruling) + * @see https://github.com/objectstack-ai/objectstack/issues/15773 (this lane) + */ + +import { + isTextFilterOperator, + textOperatorDoorVerdict, + type TextFilterOperator, +} from '@objectstack/spec/data'; +import { invalidFilterError } from './filter-comparand-shape.js'; + +/** What the door found — the field, its declaration, and the operator aimed at it. */ +export interface TextOperatorOverNonTextField { + /** The filter key, which names a declared field of this object. */ + field: string; + /** Its declared `type` — a `FieldType` member. */ + declaredType: string; + /** `formula` only: its declared `returnType`, when the seam could read one. */ + returnType?: string; + /** The text operator aimed at it. */ + operator: TextFilterOperator; + /** The `where.…` key path the offending operator sits at. */ + path: string; +} + +/** + * A plain object — filter STRUCTURE rather than a comparand. The same + * classification the three sibling gates make: a `Date` is a comparand even + * though `typeof` calls it an object. + */ +function isFilterNode(value: unknown): value is Record { + return ( + typeof value === 'object' + && value !== null + && !Array.isArray(value) + && !(value instanceof Date) + ); +} + +/** The slice of a field declaration the verdict reads. */ +function fieldMetaOf(def: unknown): { type: string; returnType?: string } | null { + if (!isFilterNode(def)) return null; + const type = (def as { type?: unknown }).type; + if (typeof type !== 'string') return null; + const returnType = (def as { returnType?: unknown }).returnType; + return typeof returnType === 'string' ? { type, returnType } : { type }; +} + +/** + * Walk one `FilterCondition` and return the FIRST text operator aimed at a + * field whose declared type can never hold a string, or `null`. + * + * Exported for the same reason the temporal walk is: a consumer that needs to + * ask "would the engine door refuse this?" without provoking the refusal. + * + * Structure is discarded the same three conservative ways the sibling gates + * discard it: `$and` / `$or` / `$not` are descended, any OTHER `$` key at node + * level is skipped WITHOUT descending (an unrecognised combinator leaves the + * fields beneath it ungated — a hole, not a false 400, the right failure + * direction for a gate that exists to stop wrong answers), and a dotted key + * names a path this door does not judge (see the module note). + */ +export function findTextOperatorOverNonTextField( + schema: unknown, + where: unknown, + path = 'where', + depth = 0, +): TextOperatorOverNonTextField | null { + // A registry-less host must not invent a verdict about a field map it cannot + // see — the same early return both neighbours make. + const fields = (schema as { fields?: Record } | undefined)?.fields; + if (!fields || typeof fields !== 'object') return null; + if (depth > 32) return null; + if (!isFilterNode(where)) return null; + + for (const [key, value] of Object.entries(where)) { + const here = `${path}.${key}`; + if (key === '$and' || key === '$or') { + if (Array.isArray(value)) { + for (const [index, arm] of value.entries()) { + const hit = findTextOperatorOverNonTextField(schema, arm, `${here}[${index}]`, depth + 1); + if (hit) return hit; + } + } + continue; + } + if (key === '$not') { + const hit = findTextOperatorOverNonTextField(schema, value, here, depth + 1); + if (hit) return hit; + continue; + } + if (key.startsWith('$')) continue; + if (key.includes('.')) continue; + const meta = fieldMetaOf(fields[key]); + // A field whose `type` is unreadable is not judged: unresolvable is not + // wrong (ADR-0072 D1), the same rule `isVirtualSearchField` records. + if (!meta) continue; + // The verdict is the spec's, over the DECLARED type — never a list here. + if (textOperatorDoorVerdict(meta) !== 'door-refusal') continue; + // Only an operator bag can aim a text operator; an implicit-equality + // comparand (`{ f_number: 5 }`) names no operator and is not this door's. + if (!isFilterNode(value)) continue; + for (const op of Object.keys(value)) { + if (!isTextFilterOperator(op)) continue; + return { + field: key, + declaredType: meta.type, + ...(meta.returnType === undefined ? {} : { returnType: meta.returnType }), + operator: op, + path: `${here}.${op}`, + }; + } + } + return null; +} + +/** + * Refuse every text operator aimed at a field whose DECLARED type can never + * hold a string — `INVALID_FILTER` / 400, this package's existing filter + * envelope (#5869 / #7047), naming the field and its declared type as the + * ruling requires. No code is minted: `INVALID_FILTER` already exists + * (`StandardErrorCode`, `packages/spec/src/api/errors.zod.ts`). + */ +export function assertTextOperatorTargetsAreStringCapable( + object: string, + operation: string, + schema: unknown, + where: unknown, +): void { + const hit = findTextOperatorOverNonTextField(schema, where); + if (!hit) return; + const declared = hit.returnType === undefined + ? `${hit.declaredType} field` + : `${hit.declaredType} field returning ${hit.returnType}`; + throw invalidFilterError( + `${operation}('${object}'): filter on '${hit.field}' aims the text operator ` + + `${hit.operator} at a declared ${declared} at ${hit.path}. A ${hit.declaredType} value is ` + + 'never a string on any backend, so no record can be matched by reading one as text — the ' + + 'operator can only be aimed at this field by mistake. The filter was NOT applied: beneath ' + // The stored-value row is #14079's; the id stays in this comment rather than in + // the message, which reaches authors and operators who cannot resolve a tracker id. + + 'this door the predicate is answered by the declared no-match rule — a positive ' + + 'text operator matches NO row and $notContains matches EVERY row — an answer no caller can ' + + `tell apart from a real result set. Compare '${hit.field}' with an operator its declared ` + + 'type can answer ($eq / $ne / $in / $gt / $gte / $lt / $lte, a range for a temporal or ' + + 'numeric field), or aim the text operator at a text-valued field.', + ); +} diff --git a/packages/spec/src/migrations/entries/semantic/18.filter-text-operator-declared-type-refused.ts b/packages/spec/src/migrations/entries/semantic/18.filter-text-operator-declared-type-refused.ts new file mode 100644 index 0000000000..25335b947a --- /dev/null +++ b/packages/spec/src/migrations/entries/semantic/18.filter-text-operator-declared-type-refused.ts @@ -0,0 +1,59 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import type { SemanticMigration } from '../../types.js'; + +export const entry: SemanticMigration = { + id: 'filter-text-operator-declared-type-refused', + surface: 'a STORED filter body the engine executes, where a text operator names a ' + + 'field whose declared type can never store a string. Measured carriers: ' + + '`sys_saved_report.query_json.filter` (executed verbatim as `engine.find(object, ' + + '{ where: q.filter })`, and reached again by every `sys_report_schedule` row ' + + 'through its `report_id`), `FieldSchema.summaryOperations[].filter` (ANDed with ' + + 'the parent-FK match and handed to `engine.aggregate`), `ListView.filter` and tab ' + + 'filters (`ViewFilterRuleSchema`, whose `contains` / `not_contains` / `icontains` ' + + '/ `starts_with` / `ends_with` spellings lower to the same operators through ' + + '`AST_OPERATOR_MAP`), and the `FilterConditionSchema` carriers on dashboards ' + + '(widget `filter`, `GlobalFilter`), datasets and reports (`runtimeFilter`), plus ' + + '`FieldSchema.relatedListFilter`. NOT this surface: an RLS / sharing / tenant ' + + 'predicate, which the platform composes onto the AST AFTER this door and which ' + + 'the door therefore never judges.', + replacement: 'compare the field with an operator its declared type can answer — `$eq` ' + + '/ `$ne` / `$in`, or a range (`$gte` / `$lt`) for a temporal or numeric field — or ' + + 'aim the text operator at a text-valued field instead. A dotted path into a ' + + 'structured-JSON field (`address.city`) stays legal and is deliberately unjudged. ' + + 'NO rewrite is mechanical: the author\'s intent is not recoverable from the stored ' + + 'condition — `{ amount: { $contains: \'5\' } }` may have meant `$eq: 5`, a range, ' + + 'or a filter on a different column altogether — so the loader must not choose one.', + reason: + 'objectstack#15661, ruled 2026-09-05 (decision batch #43, option C-deny), landed at ' + + 'the engine seam as objectstack#15773. A text operator (`$contains` / ' + + '`$notContains` / `$startsWith` / `$endsWith` / `$icontains` / `$like` / `$ilike`) ' + + 'over a field whose DECLARED type can never store a string — `NUMERIC_VALUE_TYPES` ' + + '∪ `BOOLEAN_VALUE_TYPES` ∪ `CALENDAR_DATE_TYPES` ∪ `INSTANT_TYPES` ∪ ' + + '`CLOCK_TIME_TYPES` ∪ `STRUCTURED_JSON_TYPES` — is refused at the engine\'s ' + + 'field-aware door with `INVALID_FILTER` 400 instead of reaching a driver. It is a ' + + 'RUNTIME narrowing over an AUTHORED surface, which is why it is registered here ' + + 'rather than disposed of as needing no prescription: NO schema changed, so a ' + + 'stored filter carrying the refused shape still parses and still loads — ' + + '`FilterConditionSchema` constrains no field type, and `ViewFilterRuleSchema` ' + + 'takes `field: z.string()` with `contains` in its operator enum — and the first ' + + 'sign of it is a 400 on the read that executes it. Before the door those reads ' + + 'answered `[]` (or every row for `$notContains`, or a SQLite coercion accident) ' + + 'with no diagnostic, which is the silent cell the ruling closed. `objectstack ' + + 'migrate meta` cannot repair the stored bodies for the reason `replacement` ' + + 'records, so this is a structured TODO rather than a graduated conversion.', + acceptanceCriteria: + 'Every stored filter body listed under `surface` executes without an ' + + '`INVALID_FILTER` 400 naming a declared type: run each saved report, list view, ' + + 'dashboard widget, dataset and roll-up once after the upgrade and read the ' + + 'refusals — each message names the filter key, the field\'s declared type and the ' + + 'operator, which is the whole repair list. A filter re-authored onto a typed ' + + 'operator returns the rows its author meant; one left as written keeps answering ' + + '400, and NOTHING silently rewrites it. Filters over text-valued fields — ' + + 'including `select` / `radio` codes, `multiselect` / `checkboxes` / `tags`, lookup ' + + 'and `user` ids, `autonumber` and the file classes — are unaffected and must keep ' + + 'answering exactly as before; that is the control which proves a repair pass did ' + + 'not over-reach. A DIRECT driver call bypasses this door entirely and keeps ' + + 'answering the `FILTER_TEXT_CASES` stored-value row (objectstack#14079), so a ' + + 'driver-level test is not evidence about this migration in either direction.', +}; diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index 635b99bf4f..d37d12e48f 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -7616,6 +7616,61 @@ const step18: MigrationStep = { + 'door, 400 after), so re-check what the surface was supposed to show rather than ' + 'assuming the old result set was correct.', }, + { + id: 'filter-text-operator-declared-type-refused', + surface: 'a STORED filter body the engine executes, where a text operator names a ' + + 'field whose declared type can never store a string. Measured carriers: ' + + '`sys_saved_report.query_json.filter` (executed verbatim as `engine.find(object, ' + + '{ where: q.filter })`, and reached again by every `sys_report_schedule` row ' + + 'through its `report_id`), `FieldSchema.summaryOperations[].filter` (ANDed with ' + + 'the parent-FK match and handed to `engine.aggregate`), `ListView.filter` and tab ' + + 'filters (`ViewFilterRuleSchema`, whose `contains` / `not_contains` / `icontains` ' + + '/ `starts_with` / `ends_with` spellings lower to the same operators through ' + + '`AST_OPERATOR_MAP`), and the `FilterConditionSchema` carriers on dashboards ' + + '(widget `filter`, `GlobalFilter`), datasets and reports (`runtimeFilter`), plus ' + + '`FieldSchema.relatedListFilter`. NOT this surface: an RLS / sharing / tenant ' + + 'predicate, which the platform composes onto the AST AFTER this door and which ' + + 'the door therefore never judges.', + replacement: 'compare the field with an operator its declared type can answer — `$eq` ' + + '/ `$ne` / `$in`, or a range (`$gte` / `$lt`) for a temporal or numeric field — or ' + + 'aim the text operator at a text-valued field instead. A dotted path into a ' + + 'structured-JSON field (`address.city`) stays legal and is deliberately unjudged. ' + + 'NO rewrite is mechanical: the author\'s intent is not recoverable from the stored ' + + 'condition — `{ amount: { $contains: \'5\' } }` may have meant `$eq: 5`, a range, ' + + 'or a filter on a different column altogether — so the loader must not choose one.', + reason: + 'objectstack#15661, ruled 2026-09-05 (decision batch #43, option C-deny), landed at ' + + 'the engine seam as objectstack#15773. A text operator (`$contains` / ' + + '`$notContains` / `$startsWith` / `$endsWith` / `$icontains` / `$like` / `$ilike`) ' + + 'over a field whose DECLARED type can never store a string — `NUMERIC_VALUE_TYPES` ' + + '∪ `BOOLEAN_VALUE_TYPES` ∪ `CALENDAR_DATE_TYPES` ∪ `INSTANT_TYPES` ∪ ' + + '`CLOCK_TIME_TYPES` ∪ `STRUCTURED_JSON_TYPES` — is refused at the engine\'s ' + + 'field-aware door with `INVALID_FILTER` 400 instead of reaching a driver. It is a ' + + 'RUNTIME narrowing over an AUTHORED surface, which is why it is registered here ' + + 'rather than disposed of as needing no prescription: NO schema changed, so a ' + + 'stored filter carrying the refused shape still parses and still loads — ' + + '`FilterConditionSchema` constrains no field type, and `ViewFilterRuleSchema` ' + + 'takes `field: z.string()` with `contains` in its operator enum — and the first ' + + 'sign of it is a 400 on the read that executes it. Before the door those reads ' + + 'answered `[]` (or every row for `$notContains`, or a SQLite coercion accident) ' + + 'with no diagnostic, which is the silent cell the ruling closed. `objectstack ' + + 'migrate meta` cannot repair the stored bodies for the reason `replacement` ' + + 'records, so this is a structured TODO rather than a graduated conversion.', + acceptanceCriteria: + 'Every stored filter body listed under `surface` executes without an ' + + '`INVALID_FILTER` 400 naming a declared type: run each saved report, list view, ' + + 'dashboard widget, dataset and roll-up once after the upgrade and read the ' + + 'refusals — each message names the filter key, the field\'s declared type and the ' + + 'operator, which is the whole repair list. A filter re-authored onto a typed ' + + 'operator returns the rows its author meant; one left as written keeps answering ' + + '400, and NOTHING silently rewrites it. Filters over text-valued fields — ' + + 'including `select` / `radio` codes, `multiselect` / `checkboxes` / `tags`, lookup ' + + 'and `user` ids, `autonumber` and the file classes — are unaffected and must keep ' + + 'answering exactly as before; that is the control which proves a repair pass did ' + + 'not over-reach. A DIRECT driver call bypasses this door entirely and keeps ' + + 'answering the `FILTER_TEXT_CASES` stored-value row (objectstack#14079), so a ' + + 'driver-level test is not evidence about this migration in either direction.', + }, // No backticks in `surface` — build-upgrade-guide.ts renders it inside a code // span already, and a nested backtick would close it. {