fix(driver-memory): an analytics time dimension buckets by its declared granularity (#16178)

.changeset/analytics-time-dimension-granularity-buckets.md

@@ -0,0 +1,120 @@
+---
+"@objectstack/core": minor
+"@objectstack/objectql": minor
+"@objectstack/driver-memory": minor
+---
+
+fix(driver-memory)!: an analytics time dimension buckets by its declared `granularity`, and refuses a sub-day one instead of ignoring it (#16178)
+
+<!-- adr-0087: not-required (no-migration-prescription) Nothing authorable is renamed, retired or re-typed. `packages/spec` is untouched: `TimeUpdateInterval` still declares all eight intervals, `AnalyticsQuery.timeDimensions[].granularity` keeps its name, its type and its optionality, and every analytics request body parses byte-identically to before — so `objectstack migrate meta` has nothing to rewrite and this changeset carries no rewrite instructions. What narrows is one BACKEND's accept set at request time: `driver-memory`'s analytics face refuses the three sub-day granularities it cannot label, where it previously accepted them and produced an ungrouped answer. The remedy is a coarser granularity in the request itself, which is data a caller holds rather than an authored artifact with a stored representation; the spec-side narrowing of `TimeUpdateInterval` is filed separately as issue #17296, a `domain:spec` question under ADR-0049, and is deliberately not performed here. The other two packages add exports and relocate an implementation, both additive. -->
+
+**BREAKING** in three senses, all on `driver-memory`'s analytics face, landing in
+the launch window as `minor` under the lockstep convention this cluster's
+siblings already use:
+
+- an accepted request now answers **differently**: a time dimension carrying a
+  `granularity` folds its rows into calendar buckets instead of returning one
+  group per distinct timestamp. Every affected answer was wrong before;
+- a **trend query answers rows where it used to answer one total**: a
+  `granularity` on a member `dimensions` does not also list is now a group
+  column of its own, so `{measures, timeDimensions: [{dimension, granularity}]}`
+  — the canonical trend shape — comes back one row per bucket, carrying the
+  member and a `fields` entry for it, instead of a single ungrouped total with
+  no such column;
+- an accepted request is now **refused**: `granularity: 'second' | 'minute' |
+  'hour'` answers `NOT_IMPLEMENTED` / 501 instead of being silently dropped.
+
+## What was wrong
+
+`AnalyticsQuery.timeDimensions[].granularity` is declared by the spec and a cube
+dimension enumerates the granularities it offers (`granularities: ['day']`).
+`memory-analytics.ts` read neither. The `$group` stage keyed on the raw field
+path, so a time dimension bucketed **one group per distinct timestamp** — one bar
+per row in a "new accounts by month" chart, which is the symptom #3588
+catalogued and repaired for `service-analytics`.
+
+Measured through the public entry against the built package, two rows on one UTC
+calendar day (`2026-09-06T01:00:00Z` and `2026-09-06T23:00:00Z`) under
+`granularity: 'day'`:
+
+| | before | after |
+|:--|--:|--:|
+| `granularity: 'day'` | **2 groups**, keyed on the raw instants | 1 group, `2026-09-06` |
+| no granularity (control) | 2 groups | 2 groups, unchanged |
+| `granularity: 'hour'` | **2 groups**, silently | `NOT_IMPLEMENTED` / 501 |
+| same, but with no `dimensions` | **`{count: 2}`** — one total, no time column, and no `fields` entry naming it | `{'events.createdAt': '2026-09-06', count: 2}`, `fields` naming both |
+| `granularity: 'fortnight'` past the schema door | — | `INVALID_QUERY` / 400 |
+
+The emitted pipeline was byte-identical across all three, which is the whole
+finding: the request was accepted, no warning was emitted, and the key was inert.
+
+## What it does now
+
+- **One forward labeller, in `@objectstack/core`.** `bucketDateKey(value,
+  granularity, timezone)` sits beside the inverse `bucketKeyToCalendarRange` and
+  the `calendarPartsInTzOrUtc` primitive it builds on, and it is now the only
+  statement of the rule. `BUCKET_GRANULARITIES` and `isBucketGranularity` name
+  the five granularities that HAVE a canonical key, so a face that must refuse
+  the other three quotes the accepted set instead of hand-listing it.
+- **`@objectstack/objectql`'s `bucketDateValue` is a delegate**, export name and
+  signature unchanged, answers unchanged — pinned across granularity, timezone
+  and input form rather than asserted. A driver that pushes the bucket down into
+  SQL and this in-memory path must label one instant identically or a drill-down
+  breaks at the seam, and that is now one function rather than an agreement
+  between two.
+- **A granular time dimension is a group column, listed or not.** `dimensions`
+  no longer decides alone what `$group` keys on: every `timeDimensions` entry
+  carrying a `granularity` is grouped, projected and named in `fields`, deduped
+  against `dimensions` on the resolved member so two spellings of one member
+  stay one column. This is the rule the SQL/ObjectQL face already records
+  (`projectedDimensions`, #4033/#5688) — one set feeding grouping, row mapping
+  and field metadata, because rows carrying a bucket under a `fields` list that
+  never mentions it is a trend chart with no x-axis. ⛔ An entry carrying only a
+  `dateRange` is a predicate and is still **not** projected.
+- **`driver-memory` folds by granularity before its `$group`.** The pipeline is
+  cut at that stage: the `$match` half still runs in the driver, the bucket keys
+  are written onto the selected rows, and the grouping half runs over those. The
+  key travels under a synthetic field rather than overwriting the row's own, so a
+  member that is both a group key and a measure's aggregand still ranks instants
+  in `max()` while grouping on the label.
+- **The output vocabulary is the published one** — `2026`, `2026-Q3`, `2026-09`,
+  `2026-09-06`, `2026-W36`. The week label is `YYYY-Www`, never the Monday's
+  `YYYY-MM-DD`: `DriverCapabilitiesSchema.queryDateGranularity` calls this an
+  output contract, and a second spelling is what breaks a drill-down across a
+  backend seam.
+- **Bucketing honours `AnalyticsQuery.timezone`** — the same reference zone
+  #16042 threaded through the `dateRange` window resolver, so the window that
+  selects the rows and the bucket that folds them agree on where a calendar day
+  starts. The same two rows answer one group in UTC, two in `America/New_York`
+  and two in `Asia/Tokyo`. An absent zone buckets in UTC, the resolver's default.
+
+  ⚠️ That agreement is about the PRESET arm of `dateRange`, which the resolver
+  reads in the reference zone. An explicit `[start, end]` array is the caller's
+  own **instant** window and keeps its published reading (#16179), while the
+  bucket beside it is always a **calendar** label (ADR-0053) — so an array
+  window and a bucket can still disagree about where a day starts. That
+  combination is legitimate and is not refused; it is stated here rather than
+  left to be discovered.
+- **`second` / `minute` / `hour` are refused at compile**, in the ADR-0112
+  envelope this driver's other capability gaps speak (`NOT_IMPLEMENTED` / 501,
+  the class `refusePerAggregationFilter` uses for the same reason: the query is
+  spelled correctly, the spec declares the value, and it is this backend that
+  compiles nothing for it). The canonical key vocabulary defines no label for a
+  sub-day bucket, so there is no string another backend's pushed-down SQL would
+  agree with. Passing it through unbucketed is this card's own defect wearing a
+  new name.
+- **An undeclared granularity is a 400, not a 501.** A 501 says "this backend
+  cannot", which is only honest about a value the contract declares.
+  `TimeUpdateInterval` is checked first, so a spelling it never declared —
+  reachable past the schema door, where `POST /analytics/dataset/query` types
+  `selection.timeDimensions` without Zod-parsing them — answers `INVALID_QUERY`
+  / 400 rather than a 501 asserting the spec declared it. The same separation
+  the `dateRange` half of this face already draws (#16322 / #16041).
+
+## If a caller is refused
+
+A stored widget or a request asking for a sub-day granularity was never bucketed
+by this backend — it received one group per distinct timestamp under an ordinary
+200. Nothing that worked stops working. Ask for `day` or coarser and the answer
+is a real bucket; keep the raw timestamps deliberately by dropping the key, which
+is the behaviour that key used to produce by accident.

packages/core/src/utils/datetime.test.ts

@@ -11,6 +11,9 @@ import {
   zonedWallClockToUtcMs,
   calendarPartsInTz,
   nextUtcCalendarDay,
+  bucketDateKey,
+  BUCKET_GRANULARITIES,
+  isBucketGranularity,
 } from './datetime.js';
 
 const iso = (s: string) => Date.parse(s);
@@ -223,3 +226,77 @@ describe('nextUtcCalendarDay — re-exported from @objectstack/spec (ADR-0053 D-
     expect(nextUtcCalendarDay('2026-07-28T12:00:00Z')).toBeNull();
   });
 });
+
+// ── [#16178] `bucketDateKey` — the forward labeller ──────────────────────────
+//
+// Hoisted here so `driver-memory`'s analytics face and `@objectstack/objectql`'s
+// `bucketDateValue` label one instant identically without a driver depending on
+// objectql and without a third hand copy of the rule. These cells pin the OUTPUT
+// CONTRACT (`DriverCapabilitiesSchema.queryDateGranularity` calls it that), not
+// a display preference: a driver that pushes the bucket down into SQL has to
+// emit these exact strings.
+
+describe('bucketDateKey — canonical labels', () => {
+  it('labels each of the five granularities in its canonical vocabulary', () => {
+    expect(bucketDateKey('2024-05-15T00:00:00Z', 'year')).toBe('2024');
+    expect(bucketDateKey('2024-05-15T00:00:00Z', 'quarter')).toBe('2024-Q2');
+    expect(bucketDateKey('2024-05-15T00:00:00Z', 'month')).toBe('2024-05');
+    expect(bucketDateKey('2024-05-15T00:00:00Z', 'day')).toBe('2024-05-15');
+    expect(bucketDateKey('2024-05-15T00:00:00Z', 'week')).toBe('2024-W20');
+  });
+
+  it('numbers ISO weeks by the first-Thursday rule, across the year boundary', () => {
+    expect(bucketDateKey('2024-01-01T00:00:00Z', 'week')).toBe('2024-W01');
+    // 2024-12-30 is a Monday whose week's Thursday falls in 2025.
+    expect(bucketDateKey('2024-12-30T00:00:00Z', 'week')).toBe('2025-W01');
+  });
+
+  it('reads a finite number as epoch milliseconds (#3773)', () => {
+    const ms = Date.parse('2026-01-10T08:30:00Z');
+    expect(bucketDateKey(ms, 'day')).toBe('2026-01-10');
+    for (const g of BUCKET_GRANULARITIES) {
+      // The three input spellings a driver can hand back must agree.
+      expect(bucketDateKey(ms, g)).toBe(bucketDateKey(new Date(ms), g));
+      expect(bucketDateKey(ms, g)).toBe(bucketDateKey(new Date(ms).toISOString(), g));
+    }
+  });
+
+  it('answers null for an absent or unparseable instant (#3839)', () => {
+    expect(bucketDateKey(null, 'month')).toBeNull();
+    expect(bucketDateKey(undefined, 'month')).toBeNull();
+    expect(bucketDateKey('not-a-date', 'month')).toBeNull();
+  });
+
+  it('resolves the calendar day in the reference timezone (ADR-0053 Phase 2)', () => {
+    // 23:30 UTC on Feb 29 is still Feb 29 in UTC and already Mar 1 in Tokyo.
+    const nearMidnight = '2024-02-29T23:30:00Z';
+    expect(bucketDateKey(nearMidnight, 'day', 'UTC')).toBe('2024-02-29');
+    expect(bucketDateKey(nearMidnight, 'day', 'Asia/Tokyo')).toBe('2024-03-01');
+    expect(bucketDateKey(nearMidnight, 'month', 'Asia/Tokyo')).toBe('2024-03');
+    // The week math runs on the parts ALREADY shifted into the zone, so the
+    // week boundary moves with the day rather than re-applying the offset.
+    const mondayUtc = '2024-03-04T02:00:00Z'; // Monday 03-04 UTC, Sunday 03-03 in NY
+    expect(bucketDateKey(mondayUtc, 'week', 'UTC')).toBe('2024-W10');
+    expect(bucketDateKey(mondayUtc, 'week', 'America/New_York')).toBe('2024-W09');
+  });
+});
+
+describe('BUCKET_GRANULARITIES / isBucketGranularity', () => {
+  it('is exactly the set bucketDateKey can label', () => {
+    expect([...BUCKET_GRANULARITIES]).toEqual(['day', 'week', 'month', 'quarter', 'year']);
+    for (const g of BUCKET_GRANULARITIES) {
+      expect(isBucketGranularity(g)).toBe(true);
+      // Every accepted member really produces a label — the guard and the
+      // labeller cannot disagree about the set without this failing.
+      expect(bucketDateKey('2024-05-15T00:00:00Z', g)).toMatch(/^\d{4}/);
+    }
+  });
+
+  it('rejects the three sub-day intervals TimeUpdateInterval also declares', () => {
+    // These have no canonical key anywhere in the contract, which is why a face
+    // that receives one refuses rather than bucketing it (#16178).
+    for (const g of ['second', 'minute', 'hour']) expect(isBucketGranularity(g)).toBe(false);
+    expect(isBucketGranularity('decade')).toBe(false);
+    expect(isBucketGranularity(undefined)).toBe(false);
+  });
+});

packages/core/src/utils/datetime.ts

@@ -202,15 +202,112 @@ export { nextUtcCalendarDay, utcInstantMs } from '@objectstack/spec/data';
  */
 export type BucketGranularity = 'day' | 'week' | 'month' | 'quarter' | 'year';
 
+/**
+ * The granularities that HAVE a canonical bucket key — the accepted set of
+ * {@link bucketDateKey}, in ascending order.
+ *
+ * `@objectstack/spec`'s `TimeUpdateInterval` declares three more (`second`,
+ * `minute`, `hour`) for which the contract defines no canonical key vocabulary
+ * anywhere. Exported so a face that has to refuse one of those names the
+ * accepted set FROM HERE: a hand-listed copy in a refusal message agrees with
+ * this one on the day it is typed and never again.
+ */
+export const BUCKET_GRANULARITIES: readonly BucketGranularity[] = [
+  'day',
+  'week',
+  'month',
+  'quarter',
+  'year',
+];
+
+/**
+ * Is `value` one of the five granularities {@link bucketDateKey} can label?
+ *
+ * The guard a caller holding a wider vocabulary (`TimeUpdateInterval`) uses to
+ * split "bucket it" from "refuse it" without restating either set.
+ */
+export function isBucketGranularity(value: unknown): value is BucketGranularity {
+  return typeof value === 'string' && (BUCKET_GRANULARITIES as readonly string[]).includes(value);
+}
+
+/**
+ * The canonical date-bucket KEY an instant falls in, as seen in a reference
+ * timezone — the FORWARD direction of {@link bucketKeyToCalendarRange}, and the
+ * one labeller the in-memory bucketing faces delegate to.
+ *
+ * ⚠️ **The label vocabulary is an output contract, not a display choice.** A
+ * driver that advertises `supports.queryDateGranularity[g]` buckets that
+ * granularity in SQL instead and `engine.aggregate` picks between the two per
+ * query, so a label produced here must equal the label that driver's SQL
+ * produces for the same instant, or a drill-down breaks when it crosses the
+ * seam. `2026`, `2026-Q2`, `2026-06`, `2026-06-15`, `2026-W23` — editing them
+ * means editing every driver's bucket expression too. The seam is enforced by
+ * `checkDateBucketParity` (`@objectstack/verify`).
+ *
+ * `timezone` (ADR-0053 Phase 2) resolves the calendar day in a reference zone so
+ * an instant near a tz day-boundary buckets where a user in that zone would
+ * expect. An unset / `'UTC'` / invalid zone keeps UTC bucketing. The y/m/d are
+ * taken in the reference zone and the ISO-week math then runs on a UTC date
+ * built from those parts — the parts already carry the zone shift, so the week
+ * boundary lands correctly without re-applying any offset.
+ *
+ * A finite NUMBER is read as epoch milliseconds — the form SQLite stores a
+ * `Field.datetime` in, and what any driver that hands back raw storage values
+ * yields. `new Date(String(1767225600000))` is an Invalid Date, so without this
+ * branch such a row lands in the empty bucket while the pushed-down SQL buckets
+ * it correctly (#3773) — the two paths must label the same instant identically
+ * or a drill-down built on one breaks against the other.
+ *
+ * Returns `null` for a null/absent or unparseable instant — the same key the
+ * pushed-down SQL yields, where the bucket expression propagates NULL (#3839).
+ * Null and unparseable deliberately share one bucket: SQL cannot tell them apart
+ * either (`strftime('%Y-%m', 'not-a-date')` is NULL), and splitting them here
+ * would re-open the seam this function exists to close.
+ */
+export function bucketDateKey(
+  value: unknown,
+  granularity: BucketGranularity,
+  timezone?: string,
+): string | null {
+  if (value == null) return null;
+  const d =
+    value instanceof Date
+      ? value
+      : typeof value === 'number'
+        ? new Date(value)
+        : new Date(String(value));
+  if (Number.isNaN(d.getTime())) return null;
+  const { year: y, month: m, day } = calendarPartsInTzOrUtc(d, timezone);
+  switch (granularity) {
+    case 'year':
+      return String(y);
+    case 'quarter':
+      return `${y}-Q${Math.floor((m - 1) / 3) + 1}`;
+    case 'month':
+      return `${y}-${String(m).padStart(2, '0')}`;
+    case 'day':
+      return `${y}-${String(m).padStart(2, '0')}-${String(day).padStart(2, '0')}`;
+    case 'week':
+      return isoWeekLabelFromCalendarDay(y, m, day);
+    default:
+      // Unreachable through `BucketGranularity`. Kept as the same echo
+      // `@objectstack/objectql`'s `bucketDateValue` has always answered an
+      // off-type JS caller — this function is that one's delegate, so it must
+      // not change the answer for any input that already had one.
+      return String(value);
+  }
+}
+
 /**
  * ISO-8601 week label (Mon-start weeks, week 1 = the week of the first
- * Thursday) of a UTC calendar day. The forward-direction companion used to
- * *validate* a reconstructed week boundary; it mirrors the week branch of
- * `@objectstack/objectql`'s `bucketDateValue` (kept in lockstep by the
- * round-trip parity test in objectql).
+ * Thursday) of a calendar day given that day's parts (`month` is 1-12).
+ *
+ * The ONE statement of the week rule in this package: {@link bucketDateKey}'s
+ * `week` branch and {@link isoWeekLabelUtc} both call it, so the forward label
+ * and the round-trip validator that checks it cannot drift apart.
  */
-function isoWeekLabelUtc(d: Date): string {
-  const target = new Date(d.getTime());
+function isoWeekLabelFromCalendarDay(year: number, month: number, day: number): string {
+  const target = new Date(Date.UTC(year, month - 1, day));
   const dayNum = (target.getUTCDay() + 6) % 7; // Mon=0..Sun=6
   target.setUTCDate(target.getUTCDate() - dayNum + 3); // shift to that week's Thursday
   const firstThursday = new Date(Date.UTC(target.getUTCFullYear(), 0, 4));
@@ -225,6 +322,14 @@ function isoWeekLabelUtc(d: Date): string {
   return `${target.getUTCFullYear()}-W${String(weekNo).padStart(2, '0')}`;
 }
 
+/**
+ * ISO-8601 week label of a UTC calendar day — the forward-direction companion
+ * used to *validate* a reconstructed week boundary below.
+ */
+function isoWeekLabelUtc(d: Date): string {
+  return isoWeekLabelFromCalendarDay(d.getUTCFullYear(), d.getUTCMonth() + 1, d.getUTCDate());
+}
+
 /**
  * The half-open calendar span `[start, end)` of a canonical date-bucket KEY,
  * as `YYYY-MM-DD` strings (`start` inclusive, `end` exclusive — the next