fix(spec): spec-changes.json's aggregate export diff declares the release pair it really spans

.changeset/18978-aggregate-surface-scope.md

@@ -0,0 +1,50 @@
+---
+'@objectstack/spec': minor
+---
+
+fix(spec): `spec-changes.json`'s aggregate export diff declares the release pair it really spans (#18978)
+
+Clause-②: yes (widening) — one new OPTIONAL key on a published artifact (`aggregate.surfaceScope`)
+and one new optional field on `SpecChangesSchema`. Nothing is renamed, retired or reshaped: the
+schema still ACCEPTS a record without it, every existing key keeps its spelling and meaning, and
+`perMajor` and the `release` section are byte-identical. Contract-review tier.
+
+`aggregate.added` / `aggregate.removed` are not registry-derived. A release-time api-surface diff
+fills them by comparing the artifact being published against the previously **published** one, so
+they span **one release** — while the record they sit in is keyed by protocol major (`from: 10,
+to: 17`) and every entry carries only `since: 17` / `removedIn: 17`, with
+`perMajor[16 → 17].added` at `0` beside it. Nothing in the file distinguished one minor's slice
+from the whole major-boundary delta.
+
+Measured on the published `@objectstack/spec@17.4.0` Release asset: `aggregate.added` = **225**,
+`aggregate.removed` = **51**, every entry `since`/`removedIn` = 17 — and set-identical to a
+recomputed `17.3.0 → 17.4.0` diff of the two tarballs' own `api-surface/` snapshots. It was the
+minor's delta wearing a major's label.
+
+**What ships now.** A record whose export arrays are non-empty carries the version pair they were
+diffed between:
+
+```bash
+jq '.aggregate | {from, to, surfaceScope, added: (.added | length), removed: (.removed | length)}' \
+  node_modules/@objectstack/spec/spec-changes.json
+```
+
+- `surfaceScope: { fromVersion, toVersion }` present ⇒ `added`/`removed` span exactly that
+  published-version pair. ⛔ They are **not** the `from` → `to` major delta, and never were.
+- `surfaceScope` absent ⇒ the record carries no export diff at all and `added`/`removed` are
+  empty. ⛔ Read that as "this record does not say", never as "nothing was added between `from`
+  and `to`" — the same rule the `release` section already states for itself.
+- `from` / `to` still answer the major-boundary question for `converted` / `migrated`, which are
+  registry-derived and unaffected.
+
+**Refused at the producer and at the publish gate, in both directions.** The generator reads the
+previous version off the previous artifact's own `package.json`, omits the arrays loudly when it
+cannot read one, and refuses outright to write a non-empty unlabelled array.
+`scripts/check-release-spec-changes.mjs` — which until now checked the `release` section and not
+the aggregate — recomputes the aggregate's claim from the two tarballs and refuses an absent,
+mislabelled or untrue scope. Its self-test roster grows from 15 batteries to 23.
+
+**Nothing previously honest moved.** The committed registry-only projection and every `perMajor`
+record carry no new key at all; the committed `spec-changes.json` changes on its `$comment` line
+and nowhere else. The published schema is deliberately not narrowed — every manifest published so
+far carries an unscoped diff and must keep parsing.

content/docs/upgrading.mdx

@@ -335,8 +335,27 @@ jq '.release | {fromVersion, toVersion,
 named — `"./ai: AgentSchema (const)"`, the entry point followed by the export
 and its kind. `converted` and `migrated` are the ADR-0087 conversions and
 semantic migrations first registered in that release. The same file's
-`aggregate` and `perMajor` records are unchanged and still answer the
-major-boundary question.
+`perMajor` records are unchanged and still answer the major-boundary question,
+and so does `aggregate` — for its `converted` and `migrated`, which are derived
+from the ADR-0087 registries across the whole `from` → `to` range.
+
+⛔ **But not for `aggregate.added` / `aggregate.removed`.** Those come from the
+same one-release export diff as the section above, not from the major range the
+record is keyed by, so when they are filled the `aggregate` record carries a
+`surfaceScope` naming the exact pair they span:
+
+```bash
+jq '.aggregate | {from, to, surfaceScope,
+                  added: (.added | length), removed: (.removed | length)}' \
+  node_modules/@objectstack/spec/spec-changes.json
+```
+
+`surfaceScope` absent means that record claims no export diff at all and its
+`added` / `removed` are empty — the registry-only shape. ⛔ Read that as "this
+record does not say", never as "nothing was added between `from` and `to`",
+which is the same rule the `release` section states for itself below. A release
+whose `aggregate` arrays disagree with the two published tarballs, or carry no
+`surfaceScope`, does not publish.
 
 The `os` CLI reads the same section, so a CI job does not have to know the file
 exists:

packages/spec/api-surface-declarations/root.txt

@@ -44929,6 +44929,10 @@ declare const SpecChangesSchema: z.ZodObject<{
         removedIn: z.ZodNumber;
         replacement: z.ZodOptional<z.ZodString>;
     }, z.core.$strip>>;
+    surfaceScope: z.ZodOptional<z.ZodObject<{
+        fromVersion: z.ZodString;
+        toVersion: z.ZodString;
+    }, z.core.$strip>>;
 }, z.core.$strip>;
 
 // ── SpecConverted (type) ──
@@ -45036,6 +45040,8 @@ type StoredConversionOptions = Omit<ApplyConversionsOptions, 'includeRetired'>;
 interface SurfaceDiff {
     added?: SpecSurfaceAdd[];
     removed?: SpecSurfaceRemove[];
+    /** The published-version pair `added`/`removed` were diffed between. */
+    scope?: SpecSurfaceScope;
 }
 
 // ── TemplateExpressionInputSchema (const) ──

packages/spec/scripts/build-spec-changes.ts

@@ -18,10 +18,19 @@
  * Release-time surface join: `--previous-surface <path>` diffs the current
  * committed export surface against a previously *published* one (both ship in
  * the npm artifact from protocol 15 on) and fills the `added[]`/`removed[]`
- * arrays of the aggregate record, attributed to the current major. The Release
- * workflow runs this against the last published spec tarball and attaches the
- * result to the GitHub Release; the committed copy keeps `added`/`removed`
- * empty (registry-derived content only) so it stays deterministic.
+ * arrays of the aggregate record. The Release workflow runs this against the
+ * last published spec tarball and attaches the result to the GitHub Release; the
+ * committed copy keeps `added`/`removed` empty (registry-derived content only)
+ * so it stays deterministic.
+ *
+ * ⚠️ That diff is ONE RELEASE wide while the aggregate record is keyed by
+ * protocol MAJOR (`from: 10, to: 17`), so the arrays ship with
+ * `surfaceScope: { fromVersion, toVersion }` naming the pair they really span.
+ * Without it a consumer read one minor's 225-export slice as the whole 10 → 17
+ * delta — with `perMajor[16 → 17].added` sitting at `0` beside it and no field
+ * distinguishing the two. The previous version is read off the previous
+ * artifact's own `package.json`; when it cannot be read the arrays are OMITTED,
+ * loudly, and a non-empty unlabelled array is refused outright.
  *
  * `<path>` is whichever shape that published tarball carried: the `api-surface/`
  * directory from #5837 on, or the single `api-surface.json` before it. Reading
@@ -68,9 +77,11 @@ import {
   composeSpecChanges,
   SpecChangesSchema,
   SpecReleaseChangesSchema,
+  surfaceScopeProblem,
   type SpecReleaseChanges,
   type SpecSurfaceAdd,
   type SpecSurfaceRemove,
+  type SpecSurfaceScope,
 } from '../src/migrations/spec-changes';
 import { API_SURFACE_DIR_NAME, readApiSurfaceFrom } from './lib/sharded-artifacts';
 
@@ -104,6 +115,39 @@ const PREV_SURFACE = PREV_PACKAGE
     ? process.argv[prevSurfaceIdx + 1]
     : undefined;
 
+/**
+ * The `version` of the unpacked published tarball an export snapshot came out
+ * of, or `null` when the snapshot's path does not sit inside one.
+ *
+ * `--previous-package` points at the `package/` root, so the manifest is right
+ * there; `--previous-surface` points at the snapshot itself (`api-surface/` or
+ * `api-surface.json`), whose parent is that same root in every shape the release
+ * lane has ever produced. Read, never transcribed — the same discipline
+ * {@link previousRelease} already applies to the registry ids.
+ */
+function publishedVersionAt(pkgDir: string): string | null {
+  const pkgPath = resolve(pkgDir, 'package.json');
+  if (!existsSync(pkgPath)) return null;
+  const version = (JSON.parse(readFileSync(pkgPath, 'utf8')) as { version?: string }).version;
+  return typeof version === 'string' && version.length > 0 ? version : null;
+}
+
+/**
+ * The version pair the aggregate's `added`/`removed` really span, or `null` when
+ * the previous release's version cannot be read off the inputs.
+ *
+ * ⛔ `null` is not "omit the label" — the caller then omits the ARRAYS, loudly.
+ * An unlabelled export diff under a major-keyed record is the defect this whole
+ * field exists to end, so producing it would be worse than producing nothing.
+ */
+function surfaceScope(): SpecSurfaceScope | null {
+  const root = PREV_PACKAGE ?? (PREV_SURFACE ? resolve(PREV_SURFACE, '..') : undefined);
+  if (!root) return null;
+  const fromVersion = publishedVersionAt(root);
+  if (!fromVersion) return null;
+  return { fromVersion, toVersion: THIS_VERSION };
+}
+
 /** Flatten an export surface ({ entry: ["name (kind)", …] }) into one set. */
 function flattenSurface(path: string): Set<string> {
   const doc = readApiSurfaceFrom(path);
@@ -186,7 +230,23 @@ function buildReleaseSection(current: ReturnType<typeof composeSpecChanges>): Sp
 }
 
 function build(): string {
-  const surfaceDiff = PREV_SURFACE ? diffSurfaces(PREV_SURFACE) : {};
+  // The export diff spans ONE RELEASE, so it ships only with the version pair
+  // that says so. No readable previous version ⇒ no arrays, and the reason is
+  // printed: an unlabelled slice under the MAJOR-keyed aggregate record is read
+  // as the whole major-boundary delta, which is strictly worse than an empty
+  // one — the same call `buildReleaseSection` makes for the same reason.
+  const scope = surfaceScope();
+  let surfaceDiff: ReturnType<typeof diffSurfaces> | { scope?: SpecSurfaceScope } = {};
+  if (PREV_SURFACE && scope) {
+    surfaceDiff = { ...diffSurfaces(PREV_SURFACE), scope };
+  } else if (PREV_SURFACE) {
+    console.error(
+      `No aggregate export diff: the previous artifact at ${PREV_PACKAGE ?? PREV_SURFACE} carries no ` +
+        'readable package.json, so the version pair the diff spans cannot be read. Omitting ' +
+        '`added`/`removed` — an unlabelled one-release slice under the major-keyed aggregate record ' +
+        'reads as the whole from → to delta.',
+    );
+  }
 
   // Per-major records compose (ADR-0087 D4): any tool can fold them into a
   // single from→to view. The aggregate is that fold, precomputed.
@@ -197,13 +257,24 @@ function build(): string {
   const aggregate = SpecChangesSchema.parse(
     composeSpecChanges(MIGRATION_SUPPORT_FLOOR, PROTOCOL_MAJOR, surfaceDiff),
   );
+  const problem = surfaceScopeProblem(aggregate);
+  if (problem) {
+    console.error(`Refusing to write ${SNAPSHOT}: ${problem}`);
+    process.exit(1);
+  }
   const release = buildReleaseSection(aggregate);
 
   const doc = {
     $comment:
       'GENERATED (ADR-0087 D4) — do not edit. Regenerate with: pnpm --filter @objectstack/spec gen:spec-changes. ' +
       'A projection of the D2 conversion table + D3 migration chain; the upgrade guide and the MCP spec_changes ' +
       'tool derive from this same data. ' +
+      'A record\'s `added`/`removed` are NOT at its `from` → `to` MAJOR resolution: they come from an ' +
+      'api-surface diff against the previously PUBLISHED artifact, so they span ONE RELEASE. When they are ' +
+      'non-empty the record carries `surfaceScope: { fromVersion, toVersion }` naming exactly that pair, and a ' +
+      'release whose arrays disagree with the two tarballs — or carry no `surfaceScope` — does not publish. ' +
+      'Absent `surfaceScope` means the record carries no export diff at all (`added`/`removed` empty), never ' +
+      '"nothing was added between from and to". ' +
       'When a `release` section is present, its four ADR-0087 D4 arrays report what that release ADDED: ' +
       '`added`/`removed` are the export-surface diff of the two published tarballs, and `converted`/`migrated` ' +
       'are the D2/D3 ids FIRST REGISTERED in it. An id that LEFT the published chain between the two releases ' +

packages/spec/spec-changes.json

@@ -1,5 +1,5 @@
 {
-  "$comment": "GENERATED (ADR-0087 D4) — do not edit. Regenerate with: pnpm --filter @objectstack/spec gen:spec-changes. A projection of the D2 conversion table + D3 migration chain; the upgrade guide and the MCP spec_changes tool derive from this same data. When a `release` section is present, its four ADR-0087 D4 arrays report what that release ADDED: `added`/`removed` are the export-surface diff of the two published tarballs, and `converted`/`migrated` are the D2/D3 ids FIRST REGISTERED in it. An id that LEFT the published chain between the two releases is reported in none of them — `converted: []` means \"this release registered none\", never \"none was withdrawn\"; a withdrawal is visible only by comparing two published manifests.",
+  "$comment": "GENERATED (ADR-0087 D4) — do not edit. Regenerate with: pnpm --filter @objectstack/spec gen:spec-changes. A projection of the D2 conversion table + D3 migration chain; the upgrade guide and the MCP spec_changes tool derive from this same data. A record's `added`/`removed` are NOT at its `from` → `to` MAJOR resolution: they come from an api-surface diff against the previously PUBLISHED artifact, so they span ONE RELEASE. When they are non-empty the record carries `surfaceScope: { fromVersion, toVersion }` naming exactly that pair, and a release whose arrays disagree with the two tarballs — or carry no `surfaceScope` — does not publish. Absent `surfaceScope` means the record carries no export diff at all (`added`/`removed` empty), never \"nothing was added between from and to\". When a `release` section is present, its four ADR-0087 D4 arrays report what that release ADDED: `added`/`removed` are the export-surface diff of the two published tarballs, and `converted`/`migrated` are the D2/D3 ids FIRST REGISTERED in it. An id that LEFT the published chain between the two releases is reported in none of them — `converted: []` means \"this release registered none\", never \"none was withdrawn\"; a withdrawal is visible only by comparing two published manifests.",
   "protocolVersion": "17.0.0",
   "supportFloor": 10,
   "migrateCommand": "objectstack migrate meta --from <N>  (N >= 10)",