diff --git a/.changeset/18670-project-operator-key-pattern.md b/.changeset/18670-project-operator-key-pattern.md new file mode 100644 index 0000000000..9be85cb099 --- /dev/null +++ b/.changeset/18670-project-operator-key-pattern.md @@ -0,0 +1,33 @@ +--- +"@objectstack/spec": minor +--- + +**BREAKING (published artifact narrows)** — `packages/spec/json-schema/**` now states the `$`-prefix key ban a normalized field condition enforces, so a validator reading the published files stops answering PASS on `{"$and":[{"$bogus":{"$eq":1}}]}` at `data/NormalizedFilter` — a document the runtime refuses by name (#18670 item 2, the fifth arm). + +Clause-②: yes (narrowing) + +One named pattern joins the closed list, and only one: + +- **`banned-key-pattern` — "no document may carry a key matching this pattern"**, emitted as `propertyNames` with a `not` over a `pattern`. `NormalizedFilter`'s `$and` / `$or` members and its `$not` operand each admit a field condition whose keys are field names (`amount`, `account.name`) and never `$`-prefixed operators. The published file now says so at all three nodes. + +**Scoped, and the scope is mechanical.** The ban is over an OPEN set of names, which is why the existing `banned-keys` arm cannot express it — a finite list that merely sampled the set would be wider than the rule. The pattern arm that can express it is bounded by a second closed list: `BannedKeyPattern` is a union of the pattern strings this package publishes, exactly one today (`^\$`), so a call site cannot invent a regex because there is no `string` to pass, and widening it is the same reviewed decision that adding an arm is. That is what answers the standing objection to a regex-shaped declaration — its over-reach cannot be read off the declaration the way a key list's can, so the bound is on how few declarations exist rather than on trusting the next caller. + +**The rows retired, by name.** `packages/spec/dropped-refinements.baseline.json`, entry `data/NormalizedFilter`: + +| row | before | after | +|:---|:---|:---| +| `lazy.$and.element.options[0]` | dropped | **deleted** — reads `projected`, arm `banned-key-pattern` | +| `lazy.$or.element.options[0]` | dropped | **deleted** — reads `projected`, arm `banned-key-pattern` | +| `lazy.$not.options[0]` | dropped | **deleted** — reads `projected`, arm `banned-key-pattern` | + +**⛔ Not a behaviour change, and no document the runtime accepts becomes refused.** The arm is EXACT rather than approximate. A JSON object's properties are exactly its own enumerable string-keyed ones and `propertyNames` judges exactly those names; JSON Schema specifies `pattern` as an ECMA-262 regular expression evaluated as a SEARCH, which is `RegExp.prototype.test` and nothing else — so the same source text decides the same set of names on both sides. It is presence and never value: a matching key present with a `null` value is present to both. Measured with ajv 8 (draft 2020-12) on the generated file, the verdict vector moves in one direction only: the three `$`-prefixed specimens go `true` → `false`, and every document the runtime accepts — `{}`, the empty combinators `{"$and":[]}` / `{"$or":[{}]}` / `{"$not":{}}`, a nested group, an ordinary field condition — is accepted before and after. Across the published tree, **1530 of 1535 files are byte-identical**; one file changes what it accepts, two change annotation only, and the remaining two are the bundle and the build-input hash. + +**The predicate and the keyword are ONE string.** `bannedKeyPattern` compiles its `RegExp` from the declared pattern, so the keyword the file publishes and the rule the runtime enforces cannot come to mean different things — the construction `requiredOneOf`, `dependentRequired` and `bannedKeys` already use, and the reason this arm needs no drift pin either. The `RegExp` carries no flags, which is part of the equality rather than a style choice: a JSON Schema `pattern` has none to carry, and `g` would make `test` stateful through `lastIndex` so a key's verdict would depend on which keys were judged before it. + +**A ratchet repair ships with it, and it is what made the rows exist to delete.** The detector decided `dropped` vs `projected` on a two-rung projection ladder while the generator publishes on a three-rung one — a node whose every io direction refuses over an unrepresentable member still reaches its file when that member sits in a union position, because the emit loop drops the branch and publishes the rest. Nine PUBLISHED sites therefore read `undecidable`, the one verdict the ledger does not count: they held no row, carried no `x-dropped-refinements`, and no repair of them could ever have deleted a row. The three nodes this arm closes were three of the nine. The detector now carries the generator's third rung and reports which rung answered, so a differential can never compare a pruned projection with an unpruned one; and a published site that still cannot be adjudicated fails the build by name, so the blind spot cannot reopen in silence. + +⚠️ **The ledger therefore GREW before it shrank, and the growth is the point.** Six sites became visible that were previously uncounted — `data/FieldOperators` and `data/RangeOperator` gained `$between.items[0]` / `[1]`, `data/NormalizedFilter` gained the same pair under `$not`, and `data/RangeOperator` entered the ledger as a published schema that had been holding no entry at all — then this arm deleted three. Net across the change: **204 entries / 560 sites → 205 / 566**, with the census at **566 dropped / 205 published schemas / 360 projected** (224 `non-blank-string`, 129 `required-one-of`, 3 `banned-key-pattern`, 2 `dependent-required`, 2 `banned-keys`) and **0 undecidable**, down from 9. Those two files gain annotation only: `x-` keywords are ignored by every validator, so the set of documents they accept is unchanged. + +⭐ **Superseding a sibling entry in this same release.** `18670-project-banned-keys.md` records that the `$`-prefix sites 「stay unprojected … they carry NO annotation and hold NO ledger row: published yet unratcheted」. That was a correct reading of its own tree and is no longer true of this one: the sites are projected, the blind spot is closed, and the population it described is empty. The earlier entry is left as the record of what it landed. + + diff --git a/packages/spec/api-surface-declarations/api.txt b/packages/spec/api-surface-declarations/api.txt index 9347bcb808..4fdb1f8965 100644 --- a/packages/spec/api-surface-declarations/api.txt +++ b/packages/spec/api-surface-declarations/api.txt @@ -6070,8 +6070,8 @@ declare const DiscoverySchema: z.ZodObject<{ scoping: z.ZodOptional; scoped: z.ZodBoolean; @@ -6110,8 +6110,8 @@ declare const DispatcherConfigSchema: z.ZodObject<{ }>; authRequired: z.ZodDefault; criticality: z.ZodDefault>; permissions: z.ZodOptional>; @@ -6181,8 +6181,8 @@ declare const DispatcherRouteSchema: z.ZodObject<{ }>; authRequired: z.ZodDefault; criticality: z.ZodDefault>; permissions: z.ZodOptional>; @@ -8209,8 +8209,8 @@ declare const GetDiscoveryResponseSchema: z.ZodObject<{ scoping: z.ZodOptional; scoped: z.ZodBoolean; @@ -26704,7 +26704,7 @@ declare const ObjectDefinitionResponseSchema: z.ZodObject<{ actions: { [I in keyof As]: As[I] extends infer T_1 ? T_1 extends As[I] ? T_1 extends { params: infer Ps extends readonly unknown[]; } ? T_1 & { - params: { [I_1 in keyof Ps]: Ps[I_1] & Record, never>; }; + params: { [I_1 in keyof Ps]: Ps[I_1] & Record, never>; }; } : T_1 : never : never; }; } : unknown)) => Omit & Pick; }; @@ -30331,8 +30331,8 @@ declare const RestApiConfig: z.ZodObject<{ enableSearch: z.ZodDefault; enableProjectScoping: z.ZodDefault; projectResolution: z.ZodDefault>; requireAuth: z.ZodOptional; @@ -30381,8 +30381,8 @@ declare const RestApiConfigSchema: z.ZodObject<{ enableSearch: z.ZodDefault; enableProjectScoping: z.ZodDefault; projectResolution: z.ZodDefault>; requireAuth: z.ZodOptional; @@ -31145,8 +31145,8 @@ declare const RestServerConfig: z.ZodObject<{ enableSearch: z.ZodDefault; enableProjectScoping: z.ZodDefault; projectResolution: z.ZodDefault>; requireAuth: z.ZodOptional; @@ -31240,8 +31240,8 @@ declare const RestServerConfigSchema: z.ZodObject<{ enableSearch: z.ZodDefault; enableProjectScoping: z.ZodDefault; projectResolution: z.ZodDefault>; requireAuth: z.ZodOptional; diff --git a/packages/spec/api-surface-declarations/system.txt b/packages/spec/api-surface-declarations/system.txt index a3618198c6..44549f400f 100644 --- a/packages/spec/api-surface-declarations/system.txt +++ b/packages/spec/api-surface-declarations/system.txt @@ -69266,8 +69266,8 @@ type ServiceCriticality = z.input; // ── ServiceCriticalitySchema (const) ── declare const ServiceCriticalitySchema: z.ZodEnum<{ - optional: "optional"; required: "required"; + optional: "optional"; core: "core"; }>; diff --git a/packages/spec/dropped-refinements.baseline.json b/packages/spec/dropped-refinements.baseline.json index 09732185ca..aa8809dace 100644 --- a/packages/spec/dropped-refinements.baseline.json +++ b/packages/spec/dropped-refinements.baseline.json @@ -2,10 +2,10 @@ "description": "Shrink-only ledger of every PUBLISHED JSON Schema that is STILL WIDER than the Zod type it was generated from, because a rule written as `.refine()` reaches the runtime and not the file (#18670). `z.toJSONSchema()` has no arm for a `custom` check: a plain record, the same record with a `.refine()`, and the same record with an ABORTING `.refine()` all project byte-identically (measured on zod 4.4.3, the version packages/spec resolves). So a document one of these files ACCEPTS can still be refused at parse time, and an author -- or an AI -- validating against packages/spec/json-schema/** finds out a release later. Each `sites` path is a position under that schema at which a refinement is dropped; the same paths are written onto the artifact itself as `x-dropped-refinements`. Item 2 closed the first patterns: a refinement DECLARED through the closed list in src/shared/refinement-projection.ts is emitted into the published file, reads `projected` rather than `dropped`, and its row LEAVES this ledger in the same PR -- which is why the ledger shrinks and never grows on a repair. Every refinement outside that closed list stays here, and adding an arm to the list is a public-contract decision, not a refactor. Hand-edited on purpose and with no `gen:` script: a generator would let a new gap be admitted by running a command instead of by a decision, which is the silence this ledger exists to end. Adding, removing or moving a site fails packages/spec/scripts/build-schemas.ts until the line moves with it, and the failure prints the corrected entry in full. ⛔ Do not delete or weaken a refinement to shorten this file -- the runtime rule is correct; it is the projection that is silent, and the remedy is to teach the closed list a NAMED pattern, never to drop the rule.", "measured": { "zod": "4.4.3", - "publishedSchemasWithDroppedRefinements": 204, - "droppedRefinementSites": 560, - "refinementSitesThatDidProject": 357, - "refinementSitesWithNoJsonFormToCompare": 9 + "publishedSchemasWithDroppedRefinements": 205, + "droppedRefinementSites": 566, + "refinementSitesThatDidProject": 360, + "refinementSitesWithNoJsonFormToCompare": 0 }, "entries": { "ai/BlueprintField": { @@ -605,6 +605,8 @@ }, "data/FieldOperators": { "sites": [ + "$between.items[0]", + "$between.items[1]", "$in", "$nin" ] @@ -668,6 +670,8 @@ }, "data/NormalizedFilter": { "sites": [ + "lazy.$not.options[0].valueType.$between.items[0]", + "lazy.$not.options[0].valueType.$between.items[1]", "lazy.$not.options[0].valueType.$in", "lazy.$not.options[0].valueType.$nin" ] @@ -736,6 +740,12 @@ "in.where.options[0].lazy.options[2].items[0]" ] }, + "data/RangeOperator": { + "sites": [ + "$between.items[0]", + "$between.items[1]" + ] + }, "data/ReferenceIdValue": { "sites": [ "" diff --git a/packages/spec/scripts/build-schemas.ts b/packages/spec/scripts/build-schemas.ts index e5f42c7917..c886e17783 100644 --- a/packages/spec/scripts/build-schemas.ts +++ b/packages/spec/scripts/build-schemas.ts @@ -3582,6 +3582,58 @@ if (hasDroppedRefinementProblems(droppedRefinementProblems)) { process.exit(1); } +// ⭐ #18670 item 2 — the ratchet's OWN blind spot, held at zero. +// +// A site reaches this census only from the successful-emit path above, so every +// entry here describes a PUBLISHED file. An `undecidable` site is therefore a +// node that is published and whose rule the comparison could not read: it holds +// no ledger row, it is named in no `x-dropped-refinements`, and no repair of it +// could ever delete a row. That is strictly worse than a declared gap — the gap +// is real and the instrument built to count it cannot see it — so it is refused +// rather than reported, on the same principle as every ratchet above: growth is +// a reviewed line in a diff, never a number that moved. +// +// This population was NINE sites when the item was ruled, all of them nodes the +// published file reaches through `projectByPruningUnionBranches` while the +// detector's ladder stopped at the two strict rungs. Giving the detector the +// generator's own third rung took it to ZERO and turned those nine into +// ordinary `dropped` rows, which is the reading this check now pins. +const undecidableEntries = refinementCensus.filter((entry) => entry.undecidable.length > 0); +if (undecidableEntries.length > 0) { + const undecidableTotal = undecidableEntries.reduce((sum, entry) => sum + entry.undecidable.length, 0); + console.error( + `\n❌ ${undecidableTotal} PUBLISHED refinement site(s) across ${undecidableEntries.length} schema(s) ` + + `carry a rule this build could not adjudicate (#18670 item 2):`, + ); + for (const entry of undecidableEntries) { + console.error(` + ${entry.defKey} (${entry.undecidable.length} site(s))`); + for (const site of entry.undecidable) { + console.error(` ${site.path === '' ? '' : site.path} (${site.nodeType})`); + } + } + console.error( + `\n These files ARE published, so the rule is enforced by the runtime and the file says\n` + + ` nothing about it — the same gap ${DROPPED_REFINEMENTS_BASELINE_FILE} exists to hold. What is\n` + + ` different, and worse, is that an undecidable site holds NO ledger row: it is invisible to\n` + + ` the ratchet, absent from \`x-dropped-refinements\`, and no repair of it can delete a row.\n` + + ` ⛔ Leaving it uncounted is the one outcome this check exists to refuse.\n\n` + + ` Two remedies, in order of preference:\n\n` + + ` 1. TEACH THE LADDER. \`projectOrNull\` in scripts/lib/dropped-refinements.ts must project\n` + + ` the way THIS generator does, or it answers a question about a file nobody publishes.\n` + + ` It carries three rungs — output, input, and the branch-pruning pass — because those\n` + + ` are the three attempts the emit loop above makes. If your export reaches its file by\n` + + ` a fourth route, the ladder owes that route too, and the site becomes an ordinary\n` + + ` \`dropped\` row you declare in the ledger.\n` + + ` 2. GIVE THE LEDGER AN \`undecidable\` ROW SHAPE. If a site genuinely cannot be judged,\n` + + ` the population stops being empty and needs recording rather than refusing — an entry\n` + + ` shape of its own, ratcheted in both directions like \`sites\`. That is a decision about\n` + + ` what the ledger counts, ⛔ not a refactor: take it, don't assume it.\n\n` + + ` ⛔ Do not delete or weaken the refinement to clear this line, and ⛔ do not stop\n` + + ` publishing the export — the rule is correct and the file is wanted.`, + ); + process.exit(1); +} + // The accepted population, reported in full on every run — the same discipline // as the never-published ledger above, and for the same reason: a population // that passes in silence is the silence this ratchet was built to end. @@ -3602,8 +3654,9 @@ if (droppedSiteTotal > 0) { ` the refinement, and ⛔ never an open-ended translator over the whole population.`, ); console.log( - ` Also measured this run: ${projectedSiteTotal} refinement site(s) DID reach the file, ` + - `${undecidableSiteTotal} had no JSON form on either side to compare.`, + ` Also measured this run: ${projectedSiteTotal} refinement site(s) DID reach the file, and ` + + `${undecidableSiteTotal} published site(s) could not be adjudicated — a population the check\n` + + ` above holds at zero, so this figure is a reading of it and not a tolerance.`, ); } diff --git a/packages/spec/scripts/lib/dropped-refinements.ts b/packages/spec/scripts/lib/dropped-refinements.ts index 54ed82dbe4..ca0d212171 100644 --- a/packages/spec/scripts/lib/dropped-refinements.ts +++ b/packages/spec/scripts/lib/dropped-refinements.ts @@ -94,6 +94,9 @@ import { projectableRefinementsOf, zodDefOf, } from './refinement-projection'; +// The generator's THIRD projection attempt. The detector owes the same ladder +// for the same reason it owes the same `override` — see `projectOrNull`. +import { projectByPruningUnionBranches } from './union-branch-projection'; /** File name of the committed ledger, resolved against the package root. */ export const DROPPED_REFINEMENTS_BASELINE_FILE = 'dropped-refinements.baseline.json'; @@ -134,8 +137,17 @@ export interface RefinementSite { /** * `dropped` — removing the refinements leaves the projection byte-identical. * `projected` — the projection changed, so the rule DID reach a reader. - * `undecidable` — the node has no JSON form in either io direction, so the - * comparison has no two sides. Reported, never counted as a gap. + * `undecidable` — no rung of the generator's own projection ladder gives this + * node a JSON form, so the comparison has no two sides. + * + * ⛔ `undecidable` is NOT a third way to be fine. A node can be undecidable + * here and PUBLISHED all the same — its export reaches the file through a + * route this comparison could not reproduce — and such a node states nothing + * about its rule while holding zero ledger rows, which is the ratchet's own + * blind spot rather than an absence of gap. Since #18670's second acceptance + * item the ladder carries the generator's third rung so far fewer nodes land + * here, and the ones that still do are counted: see + * {@link DroppedRefinementsEntry.undecidableSites}. */ readonly verdict: 'dropped' | 'projected' | 'undecidable'; /** @@ -229,8 +241,15 @@ function withoutCustomChecks(schema: z.ZodType): z.ZodType | null { return stripped; } +/** One side of the differential: which ladder rung answered, and what it said. */ +interface LadderReading { + /** `output` / `input` — a strict pass; `pruned` — the branch-dropping pass. */ + readonly rung: 'output' | 'input' | 'pruned'; + readonly text: string; +} + /** - * `toJSONSchema` in the generator's own io ladder, or `null` when neither side + * `toJSONSchema` in the generator's own io ladder, or `null` when no rung of it * has a JSON form. * * ⭐ It projects through `projectPublishedJsonSchema` — the SAME call the @@ -248,15 +267,27 @@ function withoutCustomChecks(schema: z.ZodType): z.ZodType | null { * `projected` behind a green ledger while the published file went wide in * silence. The helper's own docblock carries that measurement. */ -function projectOrNull(schema: z.ZodType): string | null { +function projectOrNull(schema: z.ZodType): LadderReading | null { for (const io of ['output', 'input'] as const) { try { - return JSON.stringify(projectPublishedJsonSchema(schema, { io })); + return { rung: io, text: JSON.stringify(projectPublishedJsonSchema(schema, { io })) }; } catch { // Try the other direction — the generator does the same, for the same reason. } } - return null; + // THIRD rung — the generator's own third attempt (#16431 (a)), owed here for + // the same reason the `override` is. A node whose every io direction refuses + // over one unrepresentable member still REACHES the published file when that + // member sits in a union position: `build-schemas.ts` drops the branch and + // publishes the rest. Stopping the ladder at two rungs therefore asked a + // different question than "what does the published file say about this rule" + // — it asked what a projection nobody publishes says — and answered + // `undecidable`, which is the one verdict the ledger does not count. Measured + // on `data/NormalizedFilter`: the three record nodes carrying the `$`-prefix + // ban published as bare objects, held ZERO ledger rows, and no repair of them + // could ever have deleted a row. + const pruned = projectByPruningUnionBranches(schema); + return pruned ? { rung: 'pruned', text: JSON.stringify(pruned.schema) } : null; } /** One node's raw differential and the verdict adjudicated from it. */ @@ -306,7 +337,16 @@ function readProjection(schema: z.ZodType): NodeProjectionReading { const before = projectOrNull(schema); const after = projectOrNull(stripped); if (before === null || after === null) return { verdict: 'undecidable', projectionMoved: false }; - if (before === after) return { verdict: 'dropped', projectionMoved: false }; + // ⛔ Two rungs are not two readings of one question. A differential whose + // sides were answered by different passes compares a pruned projection with + // an unpruned one, and the bytes then differ over the branches one side + // dropped — a `projected` verdict earned by the ladder rather than by the + // rule. Nothing observed reaches here (a `custom` check cannot change which + // types a subtree carries, so both sides refuse and survive alike), which is + // exactly why it is written down: the day it stops holding, this reads + // `undecidable` and is counted, instead of reading `projected` and vanishing. + if (before.rung !== after.rung) return { verdict: 'undecidable', projectionMoved: false }; + if (before.text === after.text) return { verdict: 'dropped', projectionMoved: false }; const stated = projectableRefinementsOf(schema).length; const total = customChecksOf(schema).length; return { verdict: total === stated ? 'projected' : 'dropped', projectionMoved: true }; diff --git a/packages/spec/scripts/lib/refinement-projection.ts b/packages/spec/scripts/lib/refinement-projection.ts index 0a21fb6f9b..4c5881bf5d 100644 --- a/packages/spec/scripts/lib/refinement-projection.ts +++ b/packages/spec/scripts/lib/refinement-projection.ts @@ -183,7 +183,38 @@ function emitDependentRequired( */ function emitBannedKeys(jsonSchema: JsonObject, keys: readonly string[]): void { if (keys.length === 0) return; - const rule = { not: { enum: [...keys] } }; + conjoinPropertyNames(jsonSchema, { not: { enum: [...keys] } }); +} + +/** + * `propertyNames` with a `not` over a `pattern` — the same keyword + * {@link emitBannedKeys} writes, carrying a rule about the SHAPE of a name + * where that one carries a list of names. + * + * No empty-value guard is owed here, and the reason is worth stating beside the + * one that IS owed above: `ProjectableRefinement`'s `keyPattern` is a closed + * union of literal pattern strings, so there is no empty pattern to emit and no + * caller that could supply one. `{ not: { pattern: '' } }` would be the + * hazard's analogue — a valid schema matching every string, i.e. a ban on every + * key — and it is unreachable rather than guarded. + */ +function emitBannedKeyPattern(jsonSchema: JsonObject, keyPattern: string): void { + conjoinPropertyNames(jsonSchema, { not: { pattern: keyPattern } }); +} + +/** + * Add one `propertyNames` rule to a node without taking away the one it has. + * + * Shared by both banned-key arms because the conjunction question is the same + * for both and has one right answer: a record emits `propertyNames: { type: + * 'string' }` of its own, so writing over it would trade the key-NAME rule + * being added for the key-TYPE rule already stated — a narrowing bought with a + * widening. An identical rule already present is left alone rather than + * duplicated, so each arm is idempotent, and two DIFFERENT rules both land: + * a node banning a finite list and a pattern states both bans, not the last + * one written. + */ +function conjoinPropertyNames(jsonSchema: JsonObject, rule: JsonObject): void { if (!('propertyNames' in jsonSchema)) { jsonSchema.propertyNames = rule; return; @@ -209,6 +240,9 @@ export function emitProjectableRefinement(jsonSchema: JsonObject, declared: Proj case 'banned-keys': emitBannedKeys(jsonSchema, declared.keys); return; + case 'banned-key-pattern': + emitBannedKeyPattern(jsonSchema, declared.keyPattern); + return; } } diff --git a/packages/spec/scripts/refinement-projection.test.ts b/packages/spec/scripts/refinement-projection.test.ts index 1c829f054f..4b45a53d25 100644 --- a/packages/spec/scripts/refinement-projection.test.ts +++ b/packages/spec/scripts/refinement-projection.test.ts @@ -23,6 +23,12 @@ * - `banned-keys` — over the whole key-presence lattice, with a * present-but-`null` value, and on a name `Object.prototype` carries, which * is the one JSON shape where "own property" and `in` come apart. + * - `banned-key-pattern` — the same lattice and the same own-property + * reading, plus the two things a regex adds: the emitted `pattern` is + * compared against the DECLARED string read off the predicate (one source, + * asserted rather than argued), and the predicate is shown STATELESS, which + * is what a flagless `RegExp` buys and what a JSON Schema `pattern` — which + * has no flags to carry — means. * * ## Why an equality pin and not a comment * @@ -46,13 +52,16 @@ import { z } from 'zod'; import { NON_BLANK_PATTERN, NON_BLANK_STRING, + OPERATOR_PREFIX_KEY_PATTERN, PROJECTABLE_REFINEMENT_PATTERNS, + bannedKeyPattern, bannedKeys, dependentRequired, projectableRefinementOf, requiredOneOf, } from '../src/shared/refinement-projection'; import { SSLConfigSchema } from '../src/data/driver-sql.zod'; +import { NormalizedFilterSchema } from '../src/data/filter.zod'; import { TraceSamplingConfigSchema } from '../src/system/tracing.zod'; import { emitProjectableRefinement, @@ -60,6 +69,7 @@ import { projectableRefinementsOf, } from './lib/refinement-projection'; import { collectDroppedRefinements } from './lib/dropped-refinements'; +import { projectByPruningUnionBranches } from './lib/union-branch-projection'; import { EvaluatedExpressionInputSchema, ExpressionSchema, @@ -114,6 +124,33 @@ const bannedKeysSatisfied = (node: Record, doc: Record, doc: Record): boolean => { + const clauses = [node, ...((node.allOf as Record[] | undefined) ?? [])]; + const patterns = clauses + .map((clause) => (clause.propertyNames as { not?: { pattern?: string } } | undefined)?.not?.pattern) + .filter((source): source is string => typeof source === 'string'); + if (patterns.length === 0) { + throw new Error('the node carries no propertyNames.not.pattern — nothing to evaluate'); + } + return patterns.every((source) => { + const matches = new RegExp(source); + return Object.keys(doc).every((name) => !matches.test(name)); + }); +}; + /** * ECMA-262 WhiteSpace ∪ LineTerminator, by code point so no control byte is * ever written into this file (`scripts/check-nul-bytes.mjs` is the authority @@ -137,9 +174,25 @@ describe('the list of projectable patterns is CLOSED', () => { 'non-blank-string', 'dependent-required', 'banned-keys', + 'banned-key-pattern', ]); }); + it('the set of publishable key PATTERNS is closed too, and holds exactly one', () => { + // ⛔ The second closed list, and the reason the regex arm is bounded at + // all: a call site cannot invent a pattern because `bannedKeyPattern` takes + // no `string`. Widening it is the same public-contract decision growing the + // roster above is, so the count is pinned rather than the mechanism trusted. + // ⚠️ If this line ever needs a second entry, that is the review, not a fix. + expect(OPERATOR_PREFIX_KEY_PATTERN).toBe('^\\$'); + // The predicate the ban is FOR: `^\$` and `key.startsWith('$')` name one + // set, which is what the call site in `filter.zod.ts` traded away. + const matches = new RegExp(OPERATOR_PREFIX_KEY_PATTERN); + for (const key of ['$and', '$', '$eq', 'amount', 'a$b', '', 'account.name', 'x$']) { + expect(matches.test(key), `disagreement on ${JSON.stringify(key)}`).toBe(key.startsWith('$')); + } + }); + it('a refinement nobody declared gets NO keyword', () => { const undeclared = z.string().refine((s) => s.startsWith('x'), 'must start with x'); expect(projectableRefinementsOf(undeclared)).toEqual([]); @@ -585,6 +638,123 @@ describe('banned-keys: one key list, read twice', () => { }); }); +describe('banned-key-pattern: one pattern string, read twice', () => { + it('declares the pattern it was given', () => { + const rule = bannedKeyPattern(OPERATOR_PREFIX_KEY_PATTERN); + expect(projectableRefinementOf(rule)).toEqual({ + pattern: 'banned-key-pattern', + keyPattern: '^\\$', + }); + }); + + it('⭐ the published keyword IS the declared string — one source, not two', () => { + // The ruling's own requirement for this arm. Not "they happen to be equal": + // the emitted `pattern` is read off the artifact and compared with the + // declaration read off the predicate, so an emitter that re-spelled the + // rule — or a declaration edited without its predicate — fails here. + const rule = bannedKeyPattern(OPERATOR_PREFIX_KEY_PATTERN); + const declared = projectableRefinementOf(rule) as { keyPattern: string }; + const node = publish(z.record(z.string(), z.unknown()).refine(rule)); + const emitted = (node.allOf as Array<{ propertyNames: { not: { pattern: string } } }>)[0]; + expect(emitted.propertyNames.not.pattern).toBe(declared.keyPattern); + }); + + it('emits `propertyNames` with a `not` over the pattern, when the node states none', () => { + const node: Record = { type: 'object' }; + emitProjectableRefinement(node, { pattern: 'banned-key-pattern', keyPattern: '^\\$' }); + expect(node).toEqual({ type: 'object', propertyNames: { not: { pattern: '^\\$' } } }); + }); + + it('conjoins through `allOf` rather than replacing the `propertyNames` a record already states', () => { + const node: Record = { type: 'object', propertyNames: { type: 'string' } }; + emitProjectableRefinement(node, { pattern: 'banned-key-pattern', keyPattern: '^\\$' }); + expect(node.propertyNames).toEqual({ type: 'string' }); + expect(node.allOf).toEqual([{ propertyNames: { not: { pattern: '^\\$' } } }]); + }); + + it('⛔ never writes a TOP-LEVEL `anyOf` or disturbs the node’s own shape', () => { + const node: Record = { type: 'object', properties: { a: { type: 'string' } } }; + emitProjectableRefinement(node, { pattern: 'banned-key-pattern', keyPattern: '^\\$' }); + expect(node.anyOf).toBeUndefined(); + expect(node.properties).toEqual({ a: { type: 'string' } }); + }); + + it('is idempotent — the same arm twice states one rule, not two', () => { + const node: Record = { type: 'object', propertyNames: { type: 'string' } }; + emitProjectableRefinement(node, { pattern: 'banned-key-pattern', keyPattern: '^\\$' }); + emitProjectableRefinement(node, { pattern: 'banned-key-pattern', keyPattern: '^\\$' }); + expect(node.allOf).toEqual([{ propertyNames: { not: { pattern: '^\\$' } } }]); + }); + + it('a LIST ban and a PATTERN ban on one node both land, neither replacing the other', () => { + // The two banned-key arms write the same keyword, so the conjunction they + // share has to keep both rules. A node stating only the last one written + // would be WIDER than its runtime in exactly the direction this card is + // about. + const node: Record = { type: 'object', propertyNames: { type: 'string' } }; + emitProjectableRefinement(node, { pattern: 'banned-keys', keys: ['dialect'] }); + emitProjectableRefinement(node, { pattern: 'banned-key-pattern', keyPattern: '^\\$' }); + expect(node.allOf).toEqual([ + { propertyNames: { not: { enum: ['dialect'] } } }, + { propertyNames: { not: { pattern: '^\\$' } } }, + ]); + }); + + it('the predicate and the keywords agree over the whole presence lattice', () => { + const rule = bannedKeyPattern(OPERATOR_PREFIX_KEY_PATTERN); + const node = publish(z.record(z.string(), z.unknown()).refine(rule)); + const keys = ['$eq', 'amount', 'a$b'] as const; + for (let mask = 0; mask < 8; mask += 1) { + const doc: Record = {}; + keys.forEach((key, i) => { + if (mask & (1 << i)) doc[key] = 'v'; + }); + const asJson = JSON.parse(JSON.stringify(doc)) as Record; + expect( + rule(asJson), + `runtime vs keywords disagree for ${JSON.stringify(asJson)}`, + ).toBe(bannedKeyPatternSatisfied(node, asJson)); + } + }); + + it('a matching key present with a `null` value is PRESENT on both sides', () => { + const rule = bannedKeyPattern(OPERATOR_PREFIX_KEY_PATTERN); + const node = publish(z.record(z.string(), z.unknown()).refine(rule)); + const doc = JSON.parse('{"$eq":null}') as Record; + expect(rule(doc)).toBe(false); + expect(bannedKeyPatternSatisfied(node, doc)).toBe(false); + }); + + it('⛔ the predicate is STATELESS — the same document answers the same twice', () => { + // This arm's own hazard, and the reason its `RegExp` carries no flags. With + // `g`, `test` advances `lastIndex` and alternates true/false down a key + // list, so a document's verdict would depend on which documents were judged + // before it — while a JSON Schema `pattern` has no flags to carry and would + // go on meaning the flagless rule. Two keys in one document, then the whole + // document twice: a stateful regex fails both halves. + const rule = bannedKeyPattern(OPERATOR_PREFIX_KEY_PATTERN); + const twoBanned = JSON.parse('{"$a":1,"$b":2}') as Record; + expect(rule(twoBanned)).toBe(false); + expect(rule(twoBanned)).toBe(false); + const clean = JSON.parse('{"amount":1,"total":2}') as Record; + expect(rule(clean)).toBe(true); + expect(rule(clean)).toBe(true); + }); + + it('⛔ judges OWN enumerable keys — an inherited name is not in the document', () => { + // The same reading `banned-keys` records: `Object.keys` is exactly what a + // JSON object's properties are and exactly what `propertyNames` judges, + // while `for…in` and `in` walk the prototype chain. Spelled with a name + // planted on the prototype so the two readings actually come apart. + const rule = bannedKeyPattern(OPERATOR_PREFIX_KEY_PATTERN); + const inherited = Object.create({ $planted: 'on the prototype' }) as Record; + expect('$planted' in inherited).toBe(true); + expect(rule(inherited)).toBe(true); + inherited.$own = 'on the document'; + expect(rule(inherited)).toBe(false); + }); +}); + describe("the LIVE seam: the card's own worked instance stops saying yes", () => { /** * The published `TraceSamplingConfig.composite[].condition` node. @@ -661,6 +831,168 @@ describe("the LIVE seam: the card's own worked instance stops saying yes", () => }); }); +describe('the LIVE seam: `data/NormalizedFilter` stops accepting `$`-prefixed field keys', () => { + /** + * The published `data/NormalizedFilter.json`, through the pass that actually + * writes it. + * + * ⭐ ⛔ NOT `publish()`. `FieldOperatorsSchema` carries `z.date()` members, so + * both strict io directions refuse this export outright and the generator + * reaches its file through the branch-pruning pass — which is the same reason + * these nodes were `undecidable` to the ledger until the detector's ladder + * grew that rung. A test projecting it any other way would be testing a file + * nobody publishes. + */ + const published = (): Record => { + const projected = projectByPruningUnionBranches(NormalizedFilterSchema); + if (!projected) { + throw new Error('NormalizedFilter no longer projects through the branch-pruning pass'); + } + return projected.schema as Record; + }; + + /** The three field-condition record nodes, by the path the artifact reads at. */ + const fieldConditionNodes = (): Record> => { + const props = published().properties as Record>; + const arrayMember = (key: '$and' | '$or'): Record => + ((props[key].items as Record).anyOf as Record[])[0]; + return { + 'properties.$and.items.anyOf[0]': arrayMember('$and'), + 'properties.$or.items.anyOf[0]': arrayMember('$or'), + 'properties.$not.anyOf[0]': (props.$not.anyOf as Record[])[0], + }; + }; + + /** A normalized filter that parses, with only the `$and` member varying. */ + const parsesAsMember = (member: unknown): boolean => + NormalizedFilterSchema.safeParse({ $and: [member] }).success; + + it('all THREE nodes state the ban, and keep the record shape they always stated', () => { + // The card's §6 measured all three publishing as a bare object with no ban + // at all. Named individually rather than counted: a loop that found two + // would still read "every node states it". + const nodes = fieldConditionNodes(); + expect(Object.keys(nodes)).toEqual([ + 'properties.$and.items.anyOf[0]', + 'properties.$or.items.anyOf[0]', + 'properties.$not.anyOf[0]', + ]); + for (const [where, node] of Object.entries(nodes)) { + expect(node.type, where).toBe('object'); + expect(node.propertyNames, where).toEqual({ type: 'string' }); + expect(node.allOf, where).toEqual([{ propertyNames: { not: { pattern: '^\\$' } } }]); + } + }); + + it("the thread's own specimen — a `$`-prefixed operator key — is refused by BOTH sides now", () => { + // Measured on the merged tree at filing: the file PASSed this document and + // the runtime refused it, naming the rule 「a field condition's keys are + // field names, never `$`-prefixed operators」. + const doc = JSON.parse('{"$and":[{"$bogus":{"$eq":1}}]}') as Record; + expect(NormalizedFilterSchema.safeParse(doc).success).toBe(false); + const member = (doc.$and as Record[])[0]; + expect(bannedKeyPatternSatisfied(fieldConditionNodes()['properties.$and.items.anyOf[0]'], member)).toBe(false); + }); + + it('⛔ the runtime and the emitted keywords agree on every FIELD-CONDITION member in the corpus', () => { + // ⚠️ Field conditions only, deliberately. A member may also be a GROUP — + // `{ "$and": [] }` is a `$`-keyed document the runtime accepts through the + // union's OTHER branch — so a corpus mixing the two would measure which + // branch answered, not whether this node's rule and its keywords agree. + // The group direction is the next case, and it is the one that proves + // nothing accepted became refused. + const node = fieldConditionNodes()['properties.$and.items.anyOf[0]']; + const corpus: Array> = [ + {}, + { amount: { $eq: 1 } }, + { 'account.name': { $eq: 'acme' } }, + { amount: { $eq: 1 }, total: { $gt: 2 } }, + { $bogus: { $eq: 1 } }, + { $eq: { $eq: 1 } }, + { amount: { $eq: 1 }, $x: { $eq: 2 } }, + { $and: { $eq: 1 } }, + ]; + for (const doc of corpus) { + const asJson = JSON.parse(JSON.stringify(doc)) as Record; + // Equality, not implication: the arm is exact, so a one-sided pin would + // pass a projection that had stopped narrowing at all. + expect( + bannedKeyPatternSatisfied(node, asJson), + `disagreement on ${JSON.stringify(asJson)}`, + ).toBe(parsesAsMember(asJson)); + } + }); + + it('⛔ no GROUP member the runtime accepts becomes refused — the other branch is untouched', () => { + // The direction the whole card turns on. A group member is `$`-keyed by + // construction, so if the ban had landed on the union instead of on the + // field-condition branch, every one of these would have been narrowed away. + const groups = [{}, { $and: [] }, { $or: [] }, { $or: [{}] }, { $not: {} }, { $and: [{ amount: { $eq: 1 } }] }]; + for (const group of groups) { + expect(parsesAsMember(group), `runtime refused ${JSON.stringify(group)}`).toBe(true); + } + const branches = (published().properties as Record>) + .$and.items as Record; + // The group branch is a bare `$ref` and gains nothing: the ban is on + // `anyOf[0]` and on nothing else. + expect((branches.anyOf as Record[])[1] && Object.keys((branches.anyOf as Record[])[1])) + .toEqual(['$ref']); + }); + + it('its three ledger rows are gone because the sites now read `projected`, naming the arm', () => { + const census = collectDroppedRefinements('data/NormalizedFilter', NormalizedFilterSchema); + const paths = ['lazy.$and.element.options[0]', 'lazy.$not.options[0]', 'lazy.$or.element.options[0]']; + for (const path of paths) { + const site = census.projected.find((s) => s.path === path); + expect(site?.declaredPatterns, path).toEqual(['banned-key-pattern']); + } + // ⛔ And they are not merely absent from `dropped`: `undecidable` was the + // verdict they used to carry, and it is the one that holds no ledger row. + expect(census.undecidable).toEqual([]); + expect(census.dropped.map((s) => s.path)).not.toContain('lazy.$and.element.options[0]'); + }); +}); + +describe('the detector projects through the generator`s OWN ladder, all three rungs', () => { + /** + * A node that no strict io direction can project — a `z.date()` member — but + * that the branch-pruning pass can, because the date sits in a UNION position. + * That is the shape of every site this rung was added for. + */ + const dateBearingRecord = (rule: (value: never) => boolean): z.ZodType => + z.record(z.string(), z.union([z.string(), z.date()])).refine(rule as (value: object) => boolean); + + it('a node only the pruning pass can project is ADJUDICATED, not shrugged at', () => { + // Before this rung the verdict was `undecidable`: the comparison had no two + // sides, the site held no ledger row, and no repair of it could delete one + // — while the published file carried the node all the same. + const undeclared = dateBearingRecord(((value: object) => Object.keys(value).length < 99) as never); + const census = collectDroppedRefinements('probe/OnlyPrunable', undeclared); + expect(census.undecidable).toEqual([]); + expect(census.dropped.map((s) => s.path)).toEqual(['']); + }); + + it('LIT CONTROL — a DECLARED rule on the same shape reads `projected` through that rung too', () => { + const declared = dateBearingRecord(bannedKeyPattern(OPERATOR_PREFIX_KEY_PATTERN) as never); + const census = collectDroppedRefinements('probe/OnlyPrunable', declared); + expect(census.undecidable).toEqual([]); + expect(census.dropped).toEqual([]); + expect(census.projected.map((s) => s.declaredPatterns)).toEqual([['banned-key-pattern']]); + }); + + it('a node NO rung can project is still `undecidable` — the rung is a ladder, not a blanket', () => { + // A live callable in a PROPERTY position: not droppable, because every + // document the declaration accepts carries that key. The verdict has to + // stay honest here, or the generator's zero-undecidable ratchet would be + // measuring a detector that simply stopped saying it. + const unprojectable = z.object({ handler: z.custom<() => void>((v) => typeof v === 'function') }) + .refine((value) => typeof value.handler === 'function'); + const census = collectDroppedRefinements('probe/NeverProjects', unprojectable); + expect(census.undecidable.map((s) => s.path)).toEqual(['']); + expect(census.dropped).toEqual([]); + }); +}); + describe('the verdict is adjudicated per NODE over every check on it', () => { const nonBlank = (): z.ZodString => z.string().refine(NON_BLANK_STRING, 'non-blank'); diff --git a/packages/spec/src/data/filter.zod.ts b/packages/spec/src/data/filter.zod.ts index d17f958abb..4dd3e7836e 100644 --- a/packages/spec/src/data/filter.zod.ts +++ b/packages/spec/src/data/filter.zod.ts @@ -4,6 +4,7 @@ import { z } from 'zod'; import { assertListComparandShapes } from './filter-comparand-shape'; import { normalizeFilterComparandTypes } from './filter-comparand-type'; import { bareDateRangePresetComparandMessage, isDateRangePresetName } from './date-range-presets'; +import { OPERATOR_PREFIX_KEY_PATTERN, bannedKeyPattern } from '../shared/refinement-projection'; /** * Unified Query DSL Specification @@ -1914,7 +1915,14 @@ function normalizedMemberMessage(position: string, input: unknown): string { */ const normalizedFieldConditionSchema = () => z.record(z.string(), FieldOperatorsSchema).refine( - (condition) => !Object.keys(condition).some((key) => key.startsWith('$')), + // DECLARED (#18670 item 2, the fifth arm), so the published + // `data/NormalizedFilter.json` states this ban instead of accepting the + // documents it refuses. `bannedKeyPattern` compiles its `RegExp` from the + // very string the file publishes, so the rule below and the keyword in the + // artifact cannot come to mean different things. It is the same set of keys + // the hand-written `key.startsWith('$')` named: `^\$` is a SEARCH from the + // start of input for a literal dollar, which is that predicate exactly. + bannedKeyPattern(OPERATOR_PREFIX_KEY_PATTERN), { message: 'A field condition\'s keys are field names, never $-prefixed operators.', // `abort` so this branch cannot become the union's spokesman. Measured on diff --git a/packages/spec/src/shared/refinement-projection.ts b/packages/spec/src/shared/refinement-projection.ts index ff40d660a0..6c3158d271 100644 --- a/packages/spec/src/shared/refinement-projection.ts +++ b/packages/spec/src/shared/refinement-projection.ts @@ -128,10 +128,39 @@ export type ProjectableRefinement = * * ⛔ A ban over an open set of names — every key starting with `$`, say — is * NOT this arm: its keys are a finite list, and a list that merely sampled an - * open set would be wider than the rule. Such a rule stays dropped and - * annotated until the list learns a pattern-shaped arm of its own. + * open set would be wider than the rule. That shape is + * {@link BannedKeyPattern}'s arm below, and the two are deliberately separate + * rather than one arm taking either — a finite list is readable off the + * declaration and a regex is not, so a reviewer must be able to see which of + * the two a site chose. */ - | { readonly pattern: 'banned-keys'; readonly keys: readonly string[] }; + | { readonly pattern: 'banned-keys'; readonly keys: readonly string[] } + /** + * "no document may carry a key MATCHING this pattern" — published as + * `propertyNames` with a `not` over a `pattern`, the spelling JSON Schema has + * for a rule about the SHAPE of a name where {@link ProjectableRefinement}'s + * `banned-keys` arm has one about a finite list of them. + * + * Exact in the JSON domain, and by the same reading `banned-keys` rests on + * from the other end. A JSON object's properties are exactly its own + * enumerable string-keyed ones and `propertyNames` judges exactly those + * names, so "no own property name matches" and "no property name matches" are + * one sentence. The two halves of the match agree as well: JSON Schema + * specifies `pattern` as an ECMA-262 regular expression evaluated as a + * SEARCH — unanchored, "does a match occur anywhere in the string" — which is + * `RegExp.prototype.test` and nothing else, so the same source text decides + * the same set of names on both sides. It is PRESENCE and never value: a + * matching key present with a `null` value is present to both. + * + * ⛔ The pattern is not free text. {@link BannedKeyPattern} is a CLOSED union + * of the patterns this repository publishes, exactly one today, and widening + * it is the same public-contract decision that adding an arm is — the reason + * it is a type and not a `string`. The objection this arm has to answer is + * that a regex's over-reach cannot be read off the declaration the way a key + * list's can; it is answered by keeping the set of patterns small enough to + * read, ⛔ never by trusting the next caller to pick a good one. + */ + | { readonly pattern: 'banned-key-pattern'; readonly keyPattern: BannedKeyPattern }; /** Every arm's `pattern` tag, for a reader that needs the list itself. */ export const PROJECTABLE_REFINEMENT_PATTERNS = [ @@ -139,8 +168,33 @@ export const PROJECTABLE_REFINEMENT_PATTERNS = [ 'non-blank-string', 'dependent-required', 'banned-keys', + 'banned-key-pattern', ] as const; +/** + * "a key naming a query operator rather than a field" — every name beginning + * with `$`. + * + * ECMA-262 source text, because that is what a JSON Schema `pattern` holds and + * what {@link bannedKeyPattern} builds its `RegExp` from: one string, read + * twice. The `$` is escaped because it is the end-of-input anchor unescaped, + * and `^\$` — start of input, then a literal dollar — is the rule the normalized + * filter's field-condition record enforces. + */ +export const OPERATOR_PREFIX_KEY_PATTERN = '^\\$'; + +/** + * The CLOSED set of key patterns the published JSON Schema may state. + * + * ⛔ Widening this union is a public-contract decision exactly as growing + * {@link ProjectableRefinement} is, and it is written as a type so the decision + * cannot be taken by a call site: a caller cannot invent a pattern, because + * there is no `string` to pass. That is the whole mechanism answering the + * objection to a regex-shaped arm — over-reach a reader cannot see in the + * declaration is instead bounded by how few declarations there are. + */ +export type BannedKeyPattern = typeof OPERATOR_PREFIX_KEY_PATTERN; + /** * The ECMA-262 pattern accepting exactly the strings {@link NON_BLANK_STRING} * accepts. Unanchored on purpose: a JSON Schema `pattern` is a SEARCH, so this @@ -281,3 +335,43 @@ export function bannedKeys( ); return declare(rule, declared); } + +/** + * "no key matches this pattern", as a `.refine()` predicate that also declares + * itself. + * + * The pattern is read once into the declaration and the `RegExp` the predicate + * tests with is COMPILED FROM IT, so the published `pattern` and the enforced + * match are one string used twice — the construction {@link requiredOneOf}, + * {@link dependentRequired} and {@link bannedKeys} share, and the reason this + * arm needs no drift pin either. There is no second spelling of the rule + * anywhere for a future edit to move independently. + * + * Spell the slot's own pattern at the call site, from the closed set: + * + * ```ts + * z.record(z.string(), FieldOperatorsSchema).refine( + * bannedKeyPattern(OPERATOR_PREFIX_KEY_PATTERN), + * { message: 'A field condition's keys are field names, never $-prefixed operators.', abort: true }, + * ) + * ``` + * + * ⛔ The `RegExp` carries NO flags, and that is part of the equality rather + * than a style choice. A JSON Schema `pattern` has no flags to carry, so a + * flagged `RegExp` would be enforcing something the keyword cannot state — and + * `g` in particular makes `test` stateful through `lastIndex`, which would make + * the verdict for a key depend on which keys were tested before it. It is also + * compiled ONCE per declaration rather than per call: same object, no + * per-parse construction cost on a hot validation path. + * + * ⛔ And the predicate reads OWN enumerable keys — `Object.keys` — never + * `for…in` and never `key in value`, for the reason {@link bannedKeys} records + * in full: the prototype chain carries names no JSON document has, and judging + * them would refuse documents `propertyNames` accepts. + */ +export function bannedKeyPattern(keyPattern: BannedKeyPattern): (value: object) => boolean { + const declared: ProjectableRefinement = { pattern: 'banned-key-pattern', keyPattern }; + const matches = new RegExp((declared as { keyPattern: string }).keyPattern); + const rule = (value: object): boolean => !Object.keys(value).some((key) => matches.test(key)); + return declare(rule, declared); +}