feat(spec)!: publish the $-prefix key ban the normalized filter enforces, and make the ratchet able to see it

.changeset/18670-project-operator-key-pattern.md

@@ -0,0 +1,33 @@
+---
+"@objectstack/spec": minor
+---
+
+**BREAKING (published artifact narrows)** — `packages/spec/json-schema/**` now states the `$`-prefix key ban a normalized field condition enforces, so a validator reading the published files stops answering PASS on `{"$and":[{"$bogus":{"$eq":1}}]}` at `data/NormalizedFilter` — a document the runtime refuses by name (#18670 item 2, the fifth arm).
+
+Clause-②: yes (narrowing)
+
+One named pattern joins the closed list, and only one:
+
+- **`banned-key-pattern` — "no document may carry a key matching this pattern"**, emitted as `propertyNames` with a `not` over a `pattern`. `NormalizedFilter`'s `$and` / `$or` members and its `$not` operand each admit a field condition whose keys are field names (`amount`, `account.name`) and never `$`-prefixed operators. The published file now says so at all three nodes.
+
+**Scoped, and the scope is mechanical.** The ban is over an OPEN set of names, which is why the existing `banned-keys` arm cannot express it — a finite list that merely sampled the set would be wider than the rule. The pattern arm that can express it is bounded by a second closed list: `BannedKeyPattern` is a union of the pattern strings this package publishes, exactly one today (`^\$`), so a call site cannot invent a regex because there is no `string` to pass, and widening it is the same reviewed decision that adding an arm is. That is what answers the standing objection to a regex-shaped declaration — its over-reach cannot be read off the declaration the way a key list's can, so the bound is on how few declarations exist rather than on trusting the next caller.
+
+**The rows retired, by name.** `packages/spec/dropped-refinements.baseline.json`, entry `data/NormalizedFilter`:
+
+| row | before | after |
+|:---|:---|:---|
+| `lazy.$and.element.options[0]` | dropped | **deleted** — reads `projected`, arm `banned-key-pattern` |
+| `lazy.$or.element.options[0]` | dropped | **deleted** — reads `projected`, arm `banned-key-pattern` |
+| `lazy.$not.options[0]` | dropped | **deleted** — reads `projected`, arm `banned-key-pattern` |
+
+**⛔ Not a behaviour change, and no document the runtime accepts becomes refused.** The arm is EXACT rather than approximate. A JSON object's properties are exactly its own enumerable string-keyed ones and `propertyNames` judges exactly those names; JSON Schema specifies `pattern` as an ECMA-262 regular expression evaluated as a SEARCH, which is `RegExp.prototype.test` and nothing else — so the same source text decides the same set of names on both sides. It is presence and never value: a matching key present with a `null` value is present to both. Measured with ajv 8 (draft 2020-12) on the generated file, the verdict vector moves in one direction only: the three `$`-prefixed specimens go `true` → `false`, and every document the runtime accepts — `{}`, the empty combinators `{"$and":[]}` / `{"$or":[{}]}` / `{"$not":{}}`, a nested group, an ordinary field condition — is accepted before and after. Across the published tree, **1530 of 1535 files are byte-identical**; one file changes what it accepts, two change annotation only, and the remaining two are the bundle and the build-input hash.
+
+**The predicate and the keyword are ONE string.** `bannedKeyPattern` compiles its `RegExp` from the declared pattern, so the keyword the file publishes and the rule the runtime enforces cannot come to mean different things — the construction `requiredOneOf`, `dependentRequired` and `bannedKeys` already use, and the reason this arm needs no drift pin either. The `RegExp` carries no flags, which is part of the equality rather than a style choice: a JSON Schema `pattern` has none to carry, and `g` would make `test` stateful through `lastIndex` so a key's verdict would depend on which keys were judged before it.
+
+**A ratchet repair ships with it, and it is what made the rows exist to delete.** The detector decided `dropped` vs `projected` on a two-rung projection ladder while the generator publishes on a three-rung one — a node whose every io direction refuses over an unrepresentable member still reaches its file when that member sits in a union position, because the emit loop drops the branch and publishes the rest. Nine PUBLISHED sites therefore read `undecidable`, the one verdict the ledger does not count: they held no row, carried no `x-dropped-refinements`, and no repair of them could ever have deleted a row. The three nodes this arm closes were three of the nine. The detector now carries the generator's third rung and reports which rung answered, so a differential can never compare a pruned projection with an unpruned one; and a published site that still cannot be adjudicated fails the build by name, so the blind spot cannot reopen in silence.
+
+⚠️ **The ledger therefore GREW before it shrank, and the growth is the point.** Six sites became visible that were previously uncounted — `data/FieldOperators` and `data/RangeOperator` gained `$between.items[0]` / `[1]`, `data/NormalizedFilter` gained the same pair under `$not`, and `data/RangeOperator` entered the ledger as a published schema that had been holding no entry at all — then this arm deleted three. Net across the change: **204 entries / 560 sites → 205 / 566**, with the census at **566 dropped / 205 published schemas / 360 projected** (224 `non-blank-string`, 129 `required-one-of`, 3 `banned-key-pattern`, 2 `dependent-required`, 2 `banned-keys`) and **0 undecidable**, down from 9. Those two files gain annotation only: `x-` keywords are ignored by every validator, so the set of documents they accept is unchanged.
+
+⭐ **Superseding a sibling entry in this same release.** `18670-project-banned-keys.md` records that the `$`-prefix sites 「stay unprojected … they carry NO annotation and hold NO ledger row: published yet unratcheted」. That was a correct reading of its own tree and is no longer true of this one: the sites are projected, the blind spot is closed, and the population it described is empty. The earlier entry is left as the record of what it landed.
+
+<!-- adr-0087: not-required (no-migration-prescription) Nothing an author can write is removed, renamed or re-spelled: no spec key, no export and no config field changes, and the accepted set of metadata documents is byte-for-byte what it was. What changed is a machine-readable DECLARATION catching up with the runtime it always described, so there is nothing for `objectstack migrate meta` to rewrite and no stored representation to convert. -->

packages/spec/api-surface-declarations/api.txt

@@ -6070,8 +6070,8 @@ declare const DiscoverySchema: z.ZodObject<{
     scoping: z.ZodOptional<z.ZodObject<{
         enabled: z.ZodBoolean;
         resolution: z.ZodEnum<{
-            optional: "optional";
             required: "required";
+            optional: "optional";
             auto: "auto";
         }>;
         scoped: z.ZodBoolean;
@@ -6110,8 +6110,8 @@ declare const DispatcherConfigSchema: z.ZodObject<{
         }>;
         authRequired: z.ZodDefault<z.ZodBoolean>;
         criticality: z.ZodDefault<z.ZodEnum<{
-            optional: "optional";
             required: "required";
+            optional: "optional";
             core: "core";
         }>>;
         permissions: z.ZodOptional<z.ZodArray<z.ZodString>>;
@@ -6181,8 +6181,8 @@ declare const DispatcherRouteSchema: z.ZodObject<{
     }>;
     authRequired: z.ZodDefault<z.ZodBoolean>;
     criticality: z.ZodDefault<z.ZodEnum<{
-        optional: "optional";
         required: "required";
+        optional: "optional";
         core: "core";
     }>>;
     permissions: z.ZodOptional<z.ZodArray<z.ZodString>>;
@@ -8209,8 +8209,8 @@ declare const GetDiscoveryResponseSchema: z.ZodObject<{
     scoping: z.ZodOptional<z.ZodOptional<z.ZodObject<{
         enabled: z.ZodBoolean;
         resolution: z.ZodEnum<{
-            optional: "optional";
             required: "required";
+            optional: "optional";
             auto: "auto";
         }>;
         scoped: z.ZodBoolean;
@@ -26704,7 +26704,7 @@ declare const ObjectDefinitionResponseSchema: z.ZodObject<{
             actions: { [I in keyof As]: As[I] extends infer T_1 ? T_1 extends As[I] ? T_1 extends {
                 params: infer Ps extends readonly unknown[];
             } ? T_1 & {
-                params: { [I_1 in keyof Ps]: Ps[I_1] & Record<Exclude<keyof Ps[I_1], "name" | "type" | "options" | "field" | "required" | "reference" | "label" | "visible" | "multiple" | "accept" | "defaultValue" | "maxSize" | "placeholder" | "helpText" | "requiresFeature" | "carryOver" | "objectOverride" | "defaultFromRow">, never>; };
+                params: { [I_1 in keyof Ps]: Ps[I_1] & Record<Exclude<keyof Ps[I_1], "name" | "required" | "type" | "options" | "field" | "reference" | "label" | "visible" | "multiple" | "accept" | "defaultValue" | "maxSize" | "placeholder" | "helpText" | "requiresFeature" | "carryOver" | "objectOverride" | "defaultFromRow">, never>; };
             } : T_1 : never : never; };
         } : unknown)) => Omit<ServiceObject, "fields"> & Pick<T, "fields">;
     };
@@ -30331,8 +30331,8 @@ declare const RestApiConfig: z.ZodObject<{
     enableSearch: z.ZodDefault<z.ZodBoolean>;
     enableProjectScoping: z.ZodDefault<z.ZodBoolean>;
     projectResolution: z.ZodDefault<z.ZodEnum<{
-        optional: "optional";
         required: "required";
+        optional: "optional";
         auto: "auto";
     }>>;
     requireAuth: z.ZodOptional<z.ZodNever>;
@@ -30381,8 +30381,8 @@ declare const RestApiConfigSchema: z.ZodObject<{
     enableSearch: z.ZodDefault<z.ZodBoolean>;
     enableProjectScoping: z.ZodDefault<z.ZodBoolean>;
     projectResolution: z.ZodDefault<z.ZodEnum<{
-        optional: "optional";
         required: "required";
+        optional: "optional";
         auto: "auto";
     }>>;
     requireAuth: z.ZodOptional<z.ZodNever>;
@@ -31145,8 +31145,8 @@ declare const RestServerConfig: z.ZodObject<{
         enableSearch: z.ZodDefault<z.ZodBoolean>;
         enableProjectScoping: z.ZodDefault<z.ZodBoolean>;
         projectResolution: z.ZodDefault<z.ZodEnum<{
-            optional: "optional";
             required: "required";
+            optional: "optional";
             auto: "auto";
         }>>;
         requireAuth: z.ZodOptional<z.ZodNever>;
@@ -31240,8 +31240,8 @@ declare const RestServerConfigSchema: z.ZodObject<{
         enableSearch: z.ZodDefault<z.ZodBoolean>;
         enableProjectScoping: z.ZodDefault<z.ZodBoolean>;
         projectResolution: z.ZodDefault<z.ZodEnum<{
-            optional: "optional";
             required: "required";
+            optional: "optional";
             auto: "auto";
         }>>;
         requireAuth: z.ZodOptional<z.ZodNever>;

packages/spec/api-surface-declarations/system.txt

@@ -69266,8 +69266,8 @@ type ServiceCriticality = z.input<typeof ServiceCriticalitySchema>;
 
 // ── ServiceCriticalitySchema (const) ──
 declare const ServiceCriticalitySchema: z.ZodEnum<{
-    optional: "optional";
     required: "required";
+    optional: "optional";
     core: "core";
 }>;
 

packages/spec/dropped-refinements.baseline.json

@@ -2,10 +2,10 @@
   "description": "Shrink-only ledger of every PUBLISHED JSON Schema that is STILL WIDER than the Zod type it was generated from, because a rule written as `.refine()` reaches the runtime and not the file (#18670). `z.toJSONSchema()` has no arm for a `custom` check: a plain record, the same record with a `.refine()`, and the same record with an ABORTING `.refine()` all project byte-identically (measured on zod 4.4.3, the version packages/spec resolves). So a document one of these files ACCEPTS can still be refused at parse time, and an author -- or an AI -- validating against packages/spec/json-schema/** finds out a release later. Each `sites` path is a position under that schema at which a refinement is dropped; the same paths are written onto the artifact itself as `x-dropped-refinements`. Item 2 closed the first patterns: a refinement DECLARED through the closed list in src/shared/refinement-projection.ts is emitted into the published file, reads `projected` rather than `dropped`, and its row LEAVES this ledger in the same PR -- which is why the ledger shrinks and never grows on a repair. Every refinement outside that closed list stays here, and adding an arm to the list is a public-contract decision, not a refactor. Hand-edited on purpose and with no `gen:` script: a generator would let a new gap be admitted by running a command instead of by a decision, which is the silence this ledger exists to end. Adding, removing or moving a site fails packages/spec/scripts/build-schemas.ts until the line moves with it, and the failure prints the corrected entry in full. ⛔ Do not delete or weaken a refinement to shorten this file -- the runtime rule is correct; it is the projection that is silent, and the remedy is to teach the closed list a NAMED pattern, never to drop the rule.",
   "measured": {
     "zod": "4.4.3",
-    "publishedSchemasWithDroppedRefinements": 204,
-    "droppedRefinementSites": 560,
-    "refinementSitesThatDidProject": 357,
-    "refinementSitesWithNoJsonFormToCompare": 9
+    "publishedSchemasWithDroppedRefinements": 205,
+    "droppedRefinementSites": 566,
+    "refinementSitesThatDidProject": 360,
+    "refinementSitesWithNoJsonFormToCompare": 0
   },
   "entries": {
     "ai/BlueprintField": {
@@ -605,6 +605,8 @@
     },
     "data/FieldOperators": {
       "sites": [
+        "$between.items[0]",
+        "$between.items[1]",
         "$in",
         "$nin"
       ]
@@ -668,6 +670,8 @@
     },
     "data/NormalizedFilter": {
       "sites": [
+        "lazy.$not.options[0].valueType.$between.items[0]",
+        "lazy.$not.options[0].valueType.$between.items[1]",
         "lazy.$not.options[0].valueType.$in",
         "lazy.$not.options[0].valueType.$nin"
       ]
@@ -736,6 +740,12 @@
         "in.where.options[0].lazy.options[2].items[0]"
       ]
     },
+    "data/RangeOperator": {
+      "sites": [
+        "$between.items[0]",
+        "$between.items[1]"
+      ]
+    },
     "data/ReferenceIdValue": {
       "sites": [
         ""

packages/spec/scripts/build-schemas.ts

@@ -3582,6 +3582,58 @@ if (hasDroppedRefinementProblems(droppedRefinementProblems)) {
   process.exit(1);
 }
 
+// ⭐ #18670 item 2 — the ratchet's OWN blind spot, held at zero.
+//
+// A site reaches this census only from the successful-emit path above, so every
+// entry here describes a PUBLISHED file. An `undecidable` site is therefore a
+// node that is published and whose rule the comparison could not read: it holds
+// no ledger row, it is named in no `x-dropped-refinements`, and no repair of it
+// could ever delete a row. That is strictly worse than a declared gap — the gap
+// is real and the instrument built to count it cannot see it — so it is refused
+// rather than reported, on the same principle as every ratchet above: growth is
+// a reviewed line in a diff, never a number that moved.
+//
+// This population was NINE sites when the item was ruled, all of them nodes the
+// published file reaches through `projectByPruningUnionBranches` while the
+// detector's ladder stopped at the two strict rungs. Giving the detector the
+// generator's own third rung took it to ZERO and turned those nine into
+// ordinary `dropped` rows, which is the reading this check now pins.
+const undecidableEntries = refinementCensus.filter((entry) => entry.undecidable.length > 0);
+if (undecidableEntries.length > 0) {
+  const undecidableTotal = undecidableEntries.reduce((sum, entry) => sum + entry.undecidable.length, 0);
+  console.error(
+    `\n❌ ${undecidableTotal} PUBLISHED refinement site(s) across ${undecidableEntries.length} schema(s) ` +
+      `carry a rule this build could not adjudicate (#18670 item 2):`,
+  );
+  for (const entry of undecidableEntries) {
+    console.error(`     + ${entry.defKey}  (${entry.undecidable.length} site(s))`);
+    for (const site of entry.undecidable) {
+      console.error(`         ${site.path === '' ? '<the export itself>' : site.path}  (${site.nodeType})`);
+    }
+  }
+  console.error(
+    `\n   These files ARE published, so the rule is enforced by the runtime and the file says\n` +
+      `   nothing about it — the same gap ${DROPPED_REFINEMENTS_BASELINE_FILE} exists to hold. What is\n` +
+      `   different, and worse, is that an undecidable site holds NO ledger row: it is invisible to\n` +
+      `   the ratchet, absent from \`x-dropped-refinements\`, and no repair of it can delete a row.\n` +
+      `   ⛔ Leaving it uncounted is the one outcome this check exists to refuse.\n\n` +
+      `   Two remedies, in order of preference:\n\n` +
+      `     1. TEACH THE LADDER. \`projectOrNull\` in scripts/lib/dropped-refinements.ts must project\n` +
+      `        the way THIS generator does, or it answers a question about a file nobody publishes.\n` +
+      `        It carries three rungs — output, input, and the branch-pruning pass — because those\n` +
+      `        are the three attempts the emit loop above makes. If your export reaches its file by\n` +
+      `        a fourth route, the ladder owes that route too, and the site becomes an ordinary\n` +
+      `        \`dropped\` row you declare in the ledger.\n` +
+      `     2. GIVE THE LEDGER AN \`undecidable\` ROW SHAPE. If a site genuinely cannot be judged,\n` +
+      `        the population stops being empty and needs recording rather than refusing — an entry\n` +
+      `        shape of its own, ratcheted in both directions like \`sites\`. That is a decision about\n` +
+      `        what the ledger counts, ⛔ not a refactor: take it, don't assume it.\n\n` +
+      `   ⛔ Do not delete or weaken the refinement to clear this line, and ⛔ do not stop\n` +
+      `   publishing the export — the rule is correct and the file is wanted.`,
+  );
+  process.exit(1);
+}
+
 // The accepted population, reported in full on every run — the same discipline
 // as the never-published ledger above, and for the same reason: a population
 // that passes in silence is the silence this ratchet was built to end.
@@ -3602,8 +3654,9 @@ if (droppedSiteTotal > 0) {
       `     the refinement, and ⛔ never an open-ended translator over the whole population.`,
   );
   console.log(
-    `     Also measured this run: ${projectedSiteTotal} refinement site(s) DID reach the file, ` +
-      `${undecidableSiteTotal} had no JSON form on either side to compare.`,
+    `     Also measured this run: ${projectedSiteTotal} refinement site(s) DID reach the file, and ` +
+      `${undecidableSiteTotal} published site(s) could not be adjudicated — a population the check\n` +
+      `     above holds at zero, so this figure is a reading of it and not a tolerance.`,
   );
 }