diff --git a/.changeset/harden-load-subset-lifecycle.md b/.changeset/harden-load-subset-lifecycle.md new file mode 100644 index 0000000000..b71f2006a6 --- /dev/null +++ b/.changeset/harden-load-subset-lifecycle.md @@ -0,0 +1,42 @@ +--- +'@tanstack/db': minor +'@tanstack/db-ivm': patch +'@tanstack/db-sqlite-persistence-core': patch +'@tanstack/electric-db-collection': patch +'@tanstack/powersync-db-collection': patch +'@tanstack/query-db-collection': patch +--- + +Fix on-demand load settlement, ordered pagination, and replay to preserve coherent results across cancellation, failure, cleanup, and restart. Preserve subset results and ownership across Electric, PowerSync, Query, and SQLite persistence adapters. Correct live-query grouping, include projections, value identity, and indexed comparisons. D2 hashing now rejects structural cycles and excessive traversal depth or work with an explicit error; Collection handles retain object-reference identity without traversing their mutable contents. + +Remove the unused public subset-algebra helpers: `isWhereSubset`, `unionWherePredicates`, `minusWherePredicates`, `isOrderBySubset`, `isLimitSubset`, `isOffsetLimitSubset`, `isPredicateSubset`, and `isLoadSubsetRequestSubsumedBy`. Apps that import these helpers must remove those imports; normal queries and adapters are unaffected. `DeduplicatedLoadSubset` remains available and shares only exact demand identities. + +Reject compiled Collection-valued includes as `fn.select()` inputs, including nested descendants, before invoking the callback. Use `toArray()` or `materialize()` in the upstream `.select()` for child-value calculations. To keep live child Collections, use expression `.select()` or perform parent-only functional work before adding the includes. Ordinary Collection-valued includes remain supported. + +Remove proxy `DEBUG` logging and automatic index timing statistics to avoid diagnostic work on reads and writes. Remove `getStats()` and `IndexStats`; use `index.keyCount` for the current entry count, and instrument index methods externally when profiling. Custom index subclasses must remove calls to the retired `trackLookup()` and `updateTimestamp()` helpers. + +Fix mutation drafts so Map/Set `forEach` calls each callback once and read-only iteration reports no changes. Track nested Map `for...of` edits through `collection.update()`. Keep Set entries in place during nested edits, preserving live iteration and usable `has`, `delete`, and `add` handles without duplicate entries or repeated visits. Reverting one entry no longer discards another entry's pending changes. + +Keep rejected local-storage mutations out of later successful saves. Stage insert, update, delete, and manual transaction writes before persisting, then promote the shared cache only after storage succeeds. + +Deliver live-query observer publications to peer listeners even when one listener throws. Preserve queued publication order and stop delivery on disposal; report the first listener failure after delivery. + +Remove the live-query `utils.getRunCount()` diagnostic and its runtime counter. Apps that call this diagnostic must remove those calls. Query scheduling and results are unchanged. + +Remove test-only index inspection getters (`indexedKeysSet`, `valueMapData`, `orderedEntriesArray`, and `orderedEntriesArrayReversed`) and unused scheduler diagnostics. `ReverseIndex` now exposes only the `IndexReader` lookup, range, and forward traversal surface returned by `findIndexForField`; mutate the original index instead. Export `IndexReader` for callers that name this return type. Remove the unused subscription `releaseLoadSubset()` method; request owners use the release callback supplied by `onLoadSubsetResult`. Ordinary query and adapter APIs remain unchanged by these removals. + +Remove unused internal helpers and the unused public error classes `WhereClauseConversionError`, `SubscriptionNotFoundError`, and `AggregateNotSupportedError`. These classes have no remaining runtime throw sites; remove any imports of them. Keep the existing query and index behavior and exercise identity/evaluation tests through the production entry points. + +Ensure failed mutations roll back even when their rejection value cannot be converted to a string. Preserve ordinary Error instances; report unprintable rejection values as `Unknown error`. + +Make reentrant effect disposal share the active cleanup result, including calls from abort listeners or source release callbacks. A release failure reaches every waiting disposer while each source still receives one release attempt. + +Reject starting or preloading a collection from inside its active cleanup callbacks with a clear `CollectionStateError`. Nested cleanup cannot admit replacement work that the old teardown would discard. Restart after cleanup completes, or from its final `cleaned-up` status event, remains supported. + +Prevent older page or tie-boundary completions from clearing a newer full-source failure or starting redundant loading. Failed window moves retain their settled public snapshot; an explicit retry releases the failed acquisition once and publishes the completed replacement. + +Restrict direct subscription `requestLimitedSnapshot()` cursor inputs to one order term and one `minValues` entry. Composite and partial-composite cursor inputs now throw before local delivery or adapter work. Use normal live-query ordering and window APIs for multi-column pagination; those remain supported through prefix-and-tie loading. Existing adapters need no changes. + +Treat `LoadSubsetOptions` and their nested request data as immutable from submission onward. Core no longer copies expression trees or mutable constant payloads at the sync and deduplication boundaries. Create a new Date, byte array, membership array, or options object when changing a demand instead of mutating submitted data. Adapters must also leave request data unchanged. Use stable data properties rather than stateful getters. `AbortSignal` cancellation and subscription release remain live. + +Replace replay acquisitions sequentially: release the prior physical lease before starting its replacement. The logical demand and last complete public result remain retained. A failed release prevents replacement startup; failed startup leaves demand available for a later authoritative replay. Custom adapters must support a release/load gap and preserve resources still held by other owners; a sole underlying resource may stop and restart. diff --git a/docs/collections/query-collection.md b/docs/collections/query-collection.md index d80cbb1557..76343dcf90 100644 --- a/docs/collections/query-collection.md +++ b/docs/collections/query-collection.md @@ -411,6 +411,13 @@ Derived projections, such as `select: (response) => response.edges.map((edge) => The `meta` option allows you to pass additional metadata to your query function. By default, Query Collections automatically include `loadSubsetOptions` in the meta object, which contains filtering, sorting, and pagination options for on-demand queries. +Treat `ctx.meta.loadSubsetOptions` and its nested request data as read-only. +Do not edit expression nodes, ordering options, Dates, byte arrays, or membership +arrays. Build separate API parameters instead. Core retains request data without +cloning it; changing submitted data can make the request disagree with its cache +key. To change a query constant, supply a new value rather than mutating the old +one. Cancellation through the request's `AbortSignal` remains supported. + ### Type-Safe Meta Access The `ctx.meta.loadSubsetOptions` property is automatically typed as `LoadSubsetOptions` without requiring any additional imports or type assertions: diff --git a/docs/guides/error-handling.md b/docs/guides/error-handling.md index bff185c1a9..6f20d4aea1 100644 --- a/docs/guides/error-handling.md +++ b/docs/guides/error-handling.md @@ -157,11 +157,12 @@ their incremental result can no longer be kept complete. When a must-refetch truncate cannot reload every active subset, a subscription keeps its last successful snapshot and reports the subset error. It discards -the incomplete replay batch, then resumes publishing ordinary source changes. -The next truncate retries every active subset. Overlapping truncates form one -atomic replay: all in-flight requests settle, the newest attempt decides the -result, and subscribers receive the replacement only when that attempt -succeeds. +the incomplete replay batch and keeps later source changes private because they +cannot prove a complete replacement. The next truncate retries every active +subset. Overlapping truncates form one atomic replay: all in-flight requests +settle, the newest attempt decides the result, and subscribers receive the +replacement only when that attempt succeeds. Cleanup rejects window moves that +are waiting for replay with `AbortError`. ## Collection Status and Error States diff --git a/docs/guides/live-queries.md b/docs/guides/live-queries.md index f6f8557d51..2865edc018 100644 --- a/docs/guides/live-queries.md +++ b/docs/guides/live-queries.md @@ -2821,6 +2821,11 @@ The functional variant API provides an alternative to the standard API, offering ### Functional Select +> [!WARNING] +> `fn.select()` cannot consume Collection-valued includes, even when the callback ignores or passes through that field. This also applies to nested Collection-valued includes. Use `toArray()` or `materialize()` in the upstream `.select()` to provide inline child values. Keep these helpers outside the functional callback. + +Inline child updates rerun the functional projection. Arrays support JavaScript calculations, but do not expose Collection methods such as `get()`, `createIndex()`, or `subscribeChanges()`. To keep live child Collections, use standard `.select()`, or perform parent-only `.fn.select()` work before adding the child include. + > [!WARNING] > `fn.select()` cannot be used with `groupBy()`. The `groupBy` operator needs to statically analyze the `select` clause to discover which aggregate functions to compute, which is not possible with an opaque JavaScript function. Use the standard `.select()` API for grouped queries. diff --git a/docs/reference/classes/AggregateNotSupportedError.md b/docs/reference/classes/AggregateNotSupportedError.md deleted file mode 100644 index 2cc8d0db49..0000000000 --- a/docs/reference/classes/AggregateNotSupportedError.md +++ /dev/null @@ -1,216 +0,0 @@ ---- -id: AggregateNotSupportedError -title: AggregateNotSupportedError ---- - -# Class: AggregateNotSupportedError - -Defined in: [packages/db/src/errors.ts:785](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L785) - -Error thrown when aggregate expressions are used outside of a GROUP BY context. - -## Extends - -- [`QueryCompilationError`](QueryCompilationError.md) - -## Constructors - -### Constructor - -```ts -new AggregateNotSupportedError(): AggregateNotSupportedError; -``` - -Defined in: [packages/db/src/errors.ts:786](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L786) - -#### Returns - -`AggregateNotSupportedError` - -#### Overrides - -[`QueryCompilationError`](QueryCompilationError.md).[`constructor`](QueryCompilationError.md#constructor) - -## Properties - -### cause? - -```ts -optional cause: unknown; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es2022.error.d.ts:26 - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`cause`](QueryCompilationError.md#cause) - -*** - -### message - -```ts -message: string; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1077 - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`message`](QueryCompilationError.md#message) - -*** - -### name - -```ts -name: string; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1076 - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`name`](QueryCompilationError.md#name) - -*** - -### stack? - -```ts -optional stack: string; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1078 - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`stack`](QueryCompilationError.md#stack) - -*** - -### stackTraceLimit - -```ts -static stackTraceLimit: number; -``` - -Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:67 - -The `Error.stackTraceLimit` property specifies the number of stack frames -collected by a stack trace (whether generated by `new Error().stack` or -`Error.captureStackTrace(obj)`). - -The default value is `10` but may be set to any valid JavaScript number. Changes -will affect any stack trace captured _after_ the value has been changed. - -If set to a non-number value, or set to a negative number, stack traces will -not capture any frames. - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`stackTraceLimit`](QueryCompilationError.md#stacktracelimit) - -## Methods - -### captureStackTrace() - -```ts -static captureStackTrace(targetObject, constructorOpt?): void; -``` - -Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:51 - -Creates a `.stack` property on `targetObject`, which when accessed returns -a string representing the location in the code at which -`Error.captureStackTrace()` was called. - -```js -const myObject = {}; -Error.captureStackTrace(myObject); -myObject.stack; // Similar to `new Error().stack` -``` - -The first line of the trace will be prefixed with -`${myObject.name}: ${myObject.message}`. - -The optional `constructorOpt` argument accepts a function. If given, all frames -above `constructorOpt`, including `constructorOpt`, will be omitted from the -generated stack trace. - -The `constructorOpt` argument is useful for hiding implementation -details of error generation from the user. For instance: - -```js -function a() { - b(); -} - -function b() { - c(); -} - -function c() { - // Create an error without stack trace to avoid calculating the stack trace twice. - const { stackTraceLimit } = Error; - Error.stackTraceLimit = 0; - const error = new Error(); - Error.stackTraceLimit = stackTraceLimit; - - // Capture the stack trace above function b - Error.captureStackTrace(error, b); // Neither function c, nor b is included in the stack trace - throw error; -} - -a(); -``` - -#### Parameters - -##### targetObject - -`object` - -##### constructorOpt? - -`Function` - -#### Returns - -`void` - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`captureStackTrace`](QueryCompilationError.md#capturestacktrace) - -*** - -### prepareStackTrace() - -```ts -static prepareStackTrace(err, stackTraces): any; -``` - -Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:55 - -#### Parameters - -##### err - -`Error` - -##### stackTraces - -`CallSite`[] - -#### Returns - -`any` - -#### See - -https://v8.dev/docs/stack-trace-api#customizing-stack-traces - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`prepareStackTrace`](QueryCompilationError.md#preparestacktrace) diff --git a/docs/reference/classes/BTreeIndex.md b/docs/reference/classes/BTreeIndex.md index 1f82ddaef2..d8c6934963 100644 --- a/docs/reference/classes/BTreeIndex.md +++ b/docs/reference/classes/BTreeIndex.md @@ -121,33 +121,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:116](https://github.com/TanSt *** -### lastUpdated -```ts -protected lastUpdated: Date; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:123](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L123) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`lastUpdated`](BaseIndex.md#lastupdated) - -*** - -### lookupCount - -```ts -protected lookupCount: number = 0; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:121](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L121) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`lookupCount`](BaseIndex.md#lookupcount) - -*** ### name? @@ -177,17 +151,6 @@ Defined in: [packages/db/src/indexes/btree-index.ts:39](https://github.com/TanSt *** -### totalLookupTime - -```ts -protected totalLookupTime: number = 0; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:122](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L122) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`totalLookupTime`](BaseIndex.md#totallookuptime) ## Accessors @@ -445,23 +408,6 @@ Defined in: [packages/db/src/indexes/base-index.ts:246](https://github.com/TanSt *** -### getStats() - -```ts -getStats(): IndexStats; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:234](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L234) - -#### Returns - -[`IndexStats`](../interfaces/IndexStats.md) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`getStats`](BaseIndex.md#getstats) - -*** ### inArrayLookup() @@ -885,29 +831,6 @@ The last n items *** -### trackLookup() - -```ts -protected trackLookup(startTime): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:252](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L252) - -#### Parameters - -##### startTime - -`number` - -#### Returns - -`void` - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`trackLookup`](BaseIndex.md#tracklookup) - -*** ### update() @@ -945,19 +868,3 @@ Updates a value in the index [`BaseIndex`](BaseIndex.md).[`update`](BaseIndex.md#update) *** - -### updateTimestamp() - -```ts -protected updateTimestamp(): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:258](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L258) - -#### Returns - -`void` - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`updateTimestamp`](BaseIndex.md#updatetimestamp) diff --git a/docs/reference/classes/BaseIndex.md b/docs/reference/classes/BaseIndex.md index 5a282c2224..ef1e85fea7 100644 --- a/docs/reference/classes/BaseIndex.md +++ b/docs/reference/classes/BaseIndex.md @@ -105,25 +105,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:116](https://github.com/TanSt *** -### lastUpdated -```ts -protected lastUpdated: Date; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:123](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L123) - -*** - -### lookupCount - -```ts -protected lookupCount: number = 0; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:121](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L121) - -*** ### name? @@ -145,13 +127,6 @@ Defined in: [packages/db/src/indexes/base-index.ts:119](https://github.com/TanSt *** -### totalLookupTime - -```ts -protected totalLookupTime: number = 0; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:122](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L122) ## Accessors @@ -395,23 +370,6 @@ Defined in: [packages/db/src/indexes/base-index.ts:246](https://github.com/TanSt *** -### getStats() - -```ts -getStats(): IndexStats; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:234](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L234) - -#### Returns - -[`IndexStats`](../interfaces/IndexStats.md) - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`getStats`](../interfaces/IndexInterface.md#getstats) - -*** ### inArrayLookup() @@ -788,25 +746,6 @@ Defined in: [packages/db/src/indexes/base-index.ts:166](https://github.com/TanSt *** -### trackLookup() - -```ts -protected trackLookup(startTime): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:252](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L252) - -#### Parameters - -##### startTime - -`number` - -#### Returns - -`void` - -*** ### update() @@ -842,15 +781,3 @@ Defined in: [packages/db/src/indexes/base-index.ts:148](https://github.com/TanSt [`IndexInterface`](../interfaces/IndexInterface.md).[`update`](../interfaces/IndexInterface.md#update) *** - -### updateTimestamp() - -```ts -protected updateTimestamp(): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:258](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L258) - -#### Returns - -`void` diff --git a/docs/reference/classes/BaseQueryBuilder.md b/docs/reference/classes/BaseQueryBuilder.md index c143d7e189..dfa64f921a 100644 --- a/docs/reference/classes/BaseQueryBuilder.md +++ b/docs/reference/classes/BaseQueryBuilder.md @@ -145,6 +145,11 @@ toArray(), and materialize() cannot be returned from fn.select(). Use them as fields in select() so the compiler can add them to the query graph. +Compiled Collection-valued includes cannot be inputs to fn.select(), +including nested descendants. Use toArray() or materialize() in the +upstream select(), or do parent-only functional work before adding +live Collection includes with select(). + ###### where() ```ts diff --git a/docs/reference/classes/BasicIndex.md b/docs/reference/classes/BasicIndex.md index 5100dd6537..d5084aad03 100644 --- a/docs/reference/classes/BasicIndex.md +++ b/docs/reference/classes/BasicIndex.md @@ -127,33 +127,7 @@ Defined in: [packages/db/src/indexes/base-index.ts:116](https://github.com/TanSt *** -### lastUpdated -```ts -protected lastUpdated: Date; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:123](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L123) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`lastUpdated`](BaseIndex.md#lastupdated) - -*** - -### lookupCount - -```ts -protected lookupCount: number = 0; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:121](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L121) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`lookupCount`](BaseIndex.md#lookupcount) - -*** ### name? @@ -183,17 +157,6 @@ Defined in: [packages/db/src/indexes/basic-index.ts:46](https://github.com/TanSt *** -### totalLookupTime - -```ts -protected totalLookupTime: number = 0; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:122](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L122) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`totalLookupTime`](BaseIndex.md#totallookuptime) ## Accessors @@ -451,23 +414,6 @@ Defined in: [packages/db/src/indexes/base-index.ts:246](https://github.com/TanSt *** -### getStats() - -```ts -getStats(): IndexStats; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:234](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L234) - -#### Returns - -[`IndexStats`](../interfaces/IndexStats.md) - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`getStats`](BaseIndex.md#getstats) - -*** ### inArrayLookup() @@ -866,29 +812,6 @@ Returns the first n items in reverse sorted order (from the end) *** -### trackLookup() - -```ts -protected trackLookup(startTime): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:252](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L252) - -#### Parameters - -##### startTime - -`number` - -#### Returns - -`void` - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`trackLookup`](BaseIndex.md#tracklookup) - -*** ### update() @@ -926,19 +849,3 @@ Updates a value in the index [`BaseIndex`](BaseIndex.md).[`update`](BaseIndex.md#update) *** - -### updateTimestamp() - -```ts -protected updateTimestamp(): void; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:258](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L258) - -#### Returns - -`void` - -#### Inherited from - -[`BaseIndex`](BaseIndex.md).[`updateTimestamp`](BaseIndex.md#updatetimestamp) diff --git a/docs/reference/classes/QueryCompilationError.md b/docs/reference/classes/QueryCompilationError.md index e27ec52d32..2a5a9987de 100644 --- a/docs/reference/classes/QueryCompilationError.md +++ b/docs/reference/classes/QueryCompilationError.md @@ -27,8 +27,6 @@ Defined in: [packages/db/src/errors.ts:450](https://github.com/TanStack/db/blob/ - [`EmptyReferencePathError`](EmptyReferencePathError.md) - [`UnknownFunctionError`](UnknownFunctionError.md) - [`JoinCollectionNotFoundError`](JoinCollectionNotFoundError.md) -- [`SubscriptionNotFoundError`](SubscriptionNotFoundError.md) -- [`AggregateNotSupportedError`](AggregateNotSupportedError.md) - [`MissingAliasInputsError`](MissingAliasInputsError.md) - [`SetWindowRequiresOrderByError`](SetWindowRequiresOrderByError.md) diff --git a/docs/reference/classes/QueryOptimizerError.md b/docs/reference/classes/QueryOptimizerError.md index 90c7b08362..f0ff537461 100644 --- a/docs/reference/classes/QueryOptimizerError.md +++ b/docs/reference/classes/QueryOptimizerError.md @@ -14,7 +14,6 @@ Defined in: [packages/db/src/errors.ts:741](https://github.com/TanStack/db/blob/ ## Extended by - [`CannotCombineEmptyExpressionListError`](CannotCombineEmptyExpressionListError.md) -- [`WhereClauseConversionError`](WhereClauseConversionError.md) ## Constructors diff --git a/docs/reference/classes/ReverseIndex.md b/docs/reference/classes/ReverseIndex.md index 162c421317..abe9de0da9 100644 --- a/docs/reference/classes/ReverseIndex.md +++ b/docs/reference/classes/ReverseIndex.md @@ -259,23 +259,6 @@ Defined in: [packages/db/src/indexes/reverse-index.ts:120](https://github.com/Ta *** -### getStats() - -```ts -getStats(): IndexStats; -``` - -Defined in: [packages/db/src/indexes/reverse-index.ts:92](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/reverse-index.ts#L92) - -#### Returns - -[`IndexStats`](../interfaces/IndexStats.md) - -#### Implementation of - -[`IndexInterface`](../interfaces/IndexInterface.md).[`getStats`](../interfaces/IndexInterface.md#getstats) - -*** ### inArrayLookup() diff --git a/docs/reference/classes/SubscriptionNotFoundError.md b/docs/reference/classes/SubscriptionNotFoundError.md deleted file mode 100644 index a5184494f4..0000000000 --- a/docs/reference/classes/SubscriptionNotFoundError.md +++ /dev/null @@ -1,239 +0,0 @@ ---- -id: SubscriptionNotFoundError -title: SubscriptionNotFoundError ---- - -# Class: SubscriptionNotFoundError - -Defined in: [packages/db/src/errors.ts:769](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L769) - -Error when a subscription cannot be found during lazy join processing. -For subqueries, aliases may be remapped (e.g., 'activeUser' → 'user'). - -## Extends - -- [`QueryCompilationError`](QueryCompilationError.md) - -## Constructors - -### Constructor - -```ts -new SubscriptionNotFoundError( - resolvedAlias, - originalAlias, - collectionId, - availableAliases): SubscriptionNotFoundError; -``` - -Defined in: [packages/db/src/errors.ts:770](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L770) - -#### Parameters - -##### resolvedAlias - -`string` - -##### originalAlias - -`string` - -##### collectionId - -`string` - -##### availableAliases - -`string`[] - -#### Returns - -`SubscriptionNotFoundError` - -#### Overrides - -[`QueryCompilationError`](QueryCompilationError.md).[`constructor`](QueryCompilationError.md#constructor) - -## Properties - -### cause? - -```ts -optional cause: unknown; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es2022.error.d.ts:26 - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`cause`](QueryCompilationError.md#cause) - -*** - -### message - -```ts -message: string; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1077 - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`message`](QueryCompilationError.md#message) - -*** - -### name - -```ts -name: string; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1076 - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`name`](QueryCompilationError.md#name) - -*** - -### stack? - -```ts -optional stack: string; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1078 - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`stack`](QueryCompilationError.md#stack) - -*** - -### stackTraceLimit - -```ts -static stackTraceLimit: number; -``` - -Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:67 - -The `Error.stackTraceLimit` property specifies the number of stack frames -collected by a stack trace (whether generated by `new Error().stack` or -`Error.captureStackTrace(obj)`). - -The default value is `10` but may be set to any valid JavaScript number. Changes -will affect any stack trace captured _after_ the value has been changed. - -If set to a non-number value, or set to a negative number, stack traces will -not capture any frames. - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`stackTraceLimit`](QueryCompilationError.md#stacktracelimit) - -## Methods - -### captureStackTrace() - -```ts -static captureStackTrace(targetObject, constructorOpt?): void; -``` - -Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:51 - -Creates a `.stack` property on `targetObject`, which when accessed returns -a string representing the location in the code at which -`Error.captureStackTrace()` was called. - -```js -const myObject = {}; -Error.captureStackTrace(myObject); -myObject.stack; // Similar to `new Error().stack` -``` - -The first line of the trace will be prefixed with -`${myObject.name}: ${myObject.message}`. - -The optional `constructorOpt` argument accepts a function. If given, all frames -above `constructorOpt`, including `constructorOpt`, will be omitted from the -generated stack trace. - -The `constructorOpt` argument is useful for hiding implementation -details of error generation from the user. For instance: - -```js -function a() { - b(); -} - -function b() { - c(); -} - -function c() { - // Create an error without stack trace to avoid calculating the stack trace twice. - const { stackTraceLimit } = Error; - Error.stackTraceLimit = 0; - const error = new Error(); - Error.stackTraceLimit = stackTraceLimit; - - // Capture the stack trace above function b - Error.captureStackTrace(error, b); // Neither function c, nor b is included in the stack trace - throw error; -} - -a(); -``` - -#### Parameters - -##### targetObject - -`object` - -##### constructorOpt? - -`Function` - -#### Returns - -`void` - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`captureStackTrace`](QueryCompilationError.md#capturestacktrace) - -*** - -### prepareStackTrace() - -```ts -static prepareStackTrace(err, stackTraces): any; -``` - -Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:55 - -#### Parameters - -##### err - -`Error` - -##### stackTraces - -`CallSite`[] - -#### Returns - -`any` - -#### See - -https://v8.dev/docs/stack-trace-api#customizing-stack-traces - -#### Inherited from - -[`QueryCompilationError`](QueryCompilationError.md).[`prepareStackTrace`](QueryCompilationError.md#preparestacktrace) diff --git a/docs/reference/classes/WhereClauseConversionError.md b/docs/reference/classes/WhereClauseConversionError.md deleted file mode 100644 index 31fa15fd17..0000000000 --- a/docs/reference/classes/WhereClauseConversionError.md +++ /dev/null @@ -1,226 +0,0 @@ ---- -id: WhereClauseConversionError -title: WhereClauseConversionError ---- - -# Class: WhereClauseConversionError - -Defined in: [packages/db/src/errors.ts:757](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L757) - -Internal error when the query optimizer fails to convert a WHERE clause to a collection filter. - -## Extends - -- [`QueryOptimizerError`](QueryOptimizerError.md) - -## Constructors - -### Constructor - -```ts -new WhereClauseConversionError(collectionId, alias): WhereClauseConversionError; -``` - -Defined in: [packages/db/src/errors.ts:758](https://github.com/TanStack/db/blob/main/packages/db/src/errors.ts#L758) - -#### Parameters - -##### collectionId - -`string` - -##### alias - -`string` - -#### Returns - -`WhereClauseConversionError` - -#### Overrides - -[`QueryOptimizerError`](QueryOptimizerError.md).[`constructor`](QueryOptimizerError.md#constructor) - -## Properties - -### cause? - -```ts -optional cause: unknown; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es2022.error.d.ts:26 - -#### Inherited from - -[`QueryOptimizerError`](QueryOptimizerError.md).[`cause`](QueryOptimizerError.md#cause) - -*** - -### message - -```ts -message: string; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1077 - -#### Inherited from - -[`QueryOptimizerError`](QueryOptimizerError.md).[`message`](QueryOptimizerError.md#message) - -*** - -### name - -```ts -name: string; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1076 - -#### Inherited from - -[`QueryOptimizerError`](QueryOptimizerError.md).[`name`](QueryOptimizerError.md#name) - -*** - -### stack? - -```ts -optional stack: string; -``` - -Defined in: node\_modules/.pnpm/typescript@5.9.3/node\_modules/typescript/lib/lib.es5.d.ts:1078 - -#### Inherited from - -[`QueryOptimizerError`](QueryOptimizerError.md).[`stack`](QueryOptimizerError.md#stack) - -*** - -### stackTraceLimit - -```ts -static stackTraceLimit: number; -``` - -Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:67 - -The `Error.stackTraceLimit` property specifies the number of stack frames -collected by a stack trace (whether generated by `new Error().stack` or -`Error.captureStackTrace(obj)`). - -The default value is `10` but may be set to any valid JavaScript number. Changes -will affect any stack trace captured _after_ the value has been changed. - -If set to a non-number value, or set to a negative number, stack traces will -not capture any frames. - -#### Inherited from - -[`QueryOptimizerError`](QueryOptimizerError.md).[`stackTraceLimit`](QueryOptimizerError.md#stacktracelimit) - -## Methods - -### captureStackTrace() - -```ts -static captureStackTrace(targetObject, constructorOpt?): void; -``` - -Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:51 - -Creates a `.stack` property on `targetObject`, which when accessed returns -a string representing the location in the code at which -`Error.captureStackTrace()` was called. - -```js -const myObject = {}; -Error.captureStackTrace(myObject); -myObject.stack; // Similar to `new Error().stack` -``` - -The first line of the trace will be prefixed with -`${myObject.name}: ${myObject.message}`. - -The optional `constructorOpt` argument accepts a function. If given, all frames -above `constructorOpt`, including `constructorOpt`, will be omitted from the -generated stack trace. - -The `constructorOpt` argument is useful for hiding implementation -details of error generation from the user. For instance: - -```js -function a() { - b(); -} - -function b() { - c(); -} - -function c() { - // Create an error without stack trace to avoid calculating the stack trace twice. - const { stackTraceLimit } = Error; - Error.stackTraceLimit = 0; - const error = new Error(); - Error.stackTraceLimit = stackTraceLimit; - - // Capture the stack trace above function b - Error.captureStackTrace(error, b); // Neither function c, nor b is included in the stack trace - throw error; -} - -a(); -``` - -#### Parameters - -##### targetObject - -`object` - -##### constructorOpt? - -`Function` - -#### Returns - -`void` - -#### Inherited from - -[`QueryOptimizerError`](QueryOptimizerError.md).[`captureStackTrace`](QueryOptimizerError.md#capturestacktrace) - -*** - -### prepareStackTrace() - -```ts -static prepareStackTrace(err, stackTraces): any; -``` - -Defined in: node\_modules/.pnpm/@types+node@25.2.2/node\_modules/@types/node/globals.d.ts:55 - -#### Parameters - -##### err - -`Error` - -##### stackTraces - -`CallSite`[] - -#### Returns - -`any` - -#### See - -https://v8.dev/docs/stack-trace-api#customizing-stack-traces - -#### Inherited from - -[`QueryOptimizerError`](QueryOptimizerError.md).[`prepareStackTrace`](QueryOptimizerError.md#preparestacktrace) diff --git a/docs/reference/functions/isLimitSubset.md b/docs/reference/functions/isLimitSubset.md deleted file mode 100644 index 6cc0d35f05..0000000000 --- a/docs/reference/functions/isLimitSubset.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -id: isLimitSubset -title: isLimitSubset ---- - -# Function: isLimitSubset() - -```ts -function isLimitSubset(subset, superset): boolean; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:804](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L804) - -Check if one limit is a subset of another. -Returns true if the subset limit requirements are satisfied by the superset limit. - -Note: This function does NOT consider offset. For offset-aware subset checking, -use `isOffsetLimitSubset` instead. - -## Parameters - -### subset - -The limit requirement to check - -`number` | `undefined` - -### superset - -The limit that might satisfy the requirement - -`number` | `undefined` - -## Returns - -`boolean` - -true if subset is satisfied by superset - -## Example - -```ts -isLimitSubset(10, 20) // true (requesting 10 items when 20 are available) -isLimitSubset(20, 10) // false (requesting 20 items when only 10 are available) -isLimitSubset(10, undefined) // true (requesting 10 items when unlimited are available) -``` diff --git a/docs/reference/functions/isLoadSubsetRequestSubsumedBy.md b/docs/reference/functions/isLoadSubsetRequestSubsumedBy.md deleted file mode 100644 index 0770a0143c..0000000000 --- a/docs/reference/functions/isLoadSubsetRequestSubsumedBy.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -id: isLoadSubsetRequestSubsumedBy -title: isLoadSubsetRequestSubsumedBy ---- - -# Function: isLoadSubsetRequestSubsumedBy() - -```ts -function isLoadSubsetRequestSubsumedBy(demand, acquisitionRequest): boolean; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:953](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L953) - -Returns whether one acquisition request subsumes another demand. - -This is a directional relationship between request shapes, not proof of -applied or authoritative coverage. It must not be replaced with DemandKey -equality, which answers whether two exact requests are the same. - -## Parameters - -### demand - -[`LoadSubsetOptions`](../type-aliases/LoadSubsetOptions.md) - -### acquisitionRequest - -[`LoadSubsetOptions`](../type-aliases/LoadSubsetOptions.md) - -## Returns - -`boolean` diff --git a/docs/reference/functions/isOffsetLimitSubset.md b/docs/reference/functions/isOffsetLimitSubset.md deleted file mode 100644 index cd253d3b3d..0000000000 --- a/docs/reference/functions/isOffsetLimitSubset.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -id: isOffsetLimitSubset -title: isOffsetLimitSubset ---- - -# Function: isOffsetLimitSubset() - -```ts -function isOffsetLimitSubset(subset, superset): boolean; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:844](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L844) - -Check if one offset+limit range is a subset of another. -Returns true if the subset range is fully contained within the superset range. - -A query with `{limit: 10, offset: 0}` loads rows [0, 10). -A query with `{limit: 10, offset: 20}` loads rows [20, 30). - -For subset to be satisfied by superset: -- Superset must start at or before subset (superset.offset <= subset.offset) -- Superset must end at or after subset (superset.offset + superset.limit >= subset.offset + subset.limit) - -## Parameters - -### subset - -The offset+limit requirements to check - -#### limit? - -`number` - -#### offset? - -`number` - -### superset - -The offset+limit that might satisfy the requirements - -#### limit? - -`number` - -#### offset? - -`number` - -## Returns - -`boolean` - -true if subset range is fully contained within superset range - -## Example - -```ts -isOffsetLimitSubset({ offset: 0, limit: 5 }, { offset: 0, limit: 10 }) // true -isOffsetLimitSubset({ offset: 5, limit: 5 }, { offset: 0, limit: 10 }) // true (rows 5-9 within 0-9) -isOffsetLimitSubset({ offset: 5, limit: 10 }, { offset: 0, limit: 10 }) // false (rows 5-14 exceed 0-9) -isOffsetLimitSubset({ offset: 20, limit: 10 }, { offset: 0, limit: 10 }) // false (rows 20-29 outside 0-9) -``` diff --git a/docs/reference/functions/isOrderBySubset.md b/docs/reference/functions/isOrderBySubset.md deleted file mode 100644 index c09f6759ee..0000000000 --- a/docs/reference/functions/isOrderBySubset.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -id: isOrderBySubset -title: isOrderBySubset ---- - -# Function: isOrderBySubset() - -```ts -function isOrderBySubset(subset, superset): boolean; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:746](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L746) - -Check if one orderBy clause is a subset of another. -Returns true if the subset ordering requirements are satisfied by the superset ordering. - -## Parameters - -### subset - -The ordering requirements to check - -[`OrderBy`](../@tanstack/namespaces/IR/type-aliases/OrderBy.md) | `undefined` - -### superset - -The ordering that might satisfy the requirements - -[`OrderBy`](../@tanstack/namespaces/IR/type-aliases/OrderBy.md) | `undefined` - -## Returns - -`boolean` - -true if subset is satisfied by superset - -## Example - -```ts -// Subset is prefix of superset -isOrderBySubset([{expr: age, asc}], [{expr: age, asc}, {expr: name, desc}]) // true -``` diff --git a/docs/reference/functions/isPredicateSubset.md b/docs/reference/functions/isPredicateSubset.md deleted file mode 100644 index 8f1eae3a99..0000000000 --- a/docs/reference/functions/isPredicateSubset.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -id: isPredicateSubset -title: isPredicateSubset ---- - -# Function: isPredicateSubset() - -```ts -function isPredicateSubset(subset, superset): boolean; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:887](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L887) - -Check if one predicate (where + orderBy + limit + offset) is a subset of another. -Returns true if all aspects of the subset predicate are satisfied by the superset. - -## Parameters - -### subset - -[`LoadSubsetOptions`](../type-aliases/LoadSubsetOptions.md) - -The predicate requirements to check - -### superset - -[`LoadSubsetOptions`](../type-aliases/LoadSubsetOptions.md) - -The predicate that might satisfy the requirements - -## Returns - -`boolean` - -true if subset is satisfied by superset - -## Example - -```ts -isPredicateSubset( - { where: gt(ref('age'), val(20)), limit: 10 }, - { where: gt(ref('age'), val(10)), limit: 20 } -) // true -``` diff --git a/docs/reference/functions/isWhereSubset.md b/docs/reference/functions/isWhereSubset.md deleted file mode 100644 index f312acd9ae..0000000000 --- a/docs/reference/functions/isWhereSubset.md +++ /dev/null @@ -1,47 +0,0 @@ ---- -id: isWhereSubset -title: isWhereSubset ---- - -# Function: isWhereSubset() - -```ts -function isWhereSubset(subset, superset): boolean; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:27](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L27) - -Check if one where clause is a logical subset of another. -Returns true if the subset predicate is more restrictive than (or equal to) the superset predicate. - -## Parameters - -### subset - -The potentially more restrictive predicate - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> | `undefined` - -### superset - -The potentially less restrictive predicate - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> | `undefined` - -## Returns - -`boolean` - -true if subset logically implies superset - -## Examples - -```ts -// age > 20 is subset of age > 10 (more restrictive) -isWhereSubset(gt(ref('age'), val(20)), gt(ref('age'), val(10))) // true -``` - -```ts -// age > 10 AND name = 'X' is subset of age > 10 (more conditions) -isWhereSubset(and(gt(ref('age'), val(10)), eq(ref('name'), val('X'))), gt(ref('age'), val(10))) // true -``` diff --git a/docs/reference/functions/minusWherePredicates.md b/docs/reference/functions/minusWherePredicates.md deleted file mode 100644 index fda1448b35..0000000000 --- a/docs/reference/functions/minusWherePredicates.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -id: minusWherePredicates -title: minusWherePredicates ---- - -# Function: minusWherePredicates() - -```ts -function minusWherePredicates(fromPredicate, subtractPredicate): - | BasicExpression - | null; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:371](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L371) - -Compute the difference between two where predicates: `fromPredicate AND NOT(subtractPredicate)`. -Returns the simplified predicate, or null if the difference cannot be simplified -(in which case the caller should fetch the full fromPredicate). - -## Parameters - -### fromPredicate - -The predicate to subtract from - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> | `undefined` - -### subtractPredicate - -The predicate to subtract - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> | `undefined` - -## Returns - - \| [`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> - \| `null` - -The simplified difference, or null if cannot be simplified - -## Examples - -```ts -// Range difference -minusWherePredicates( - gt(ref('age'), val(10)), // age > 10 - gt(ref('age'), val(20)) // age > 20 -) // → age > 10 AND age <= 20 -``` - -```ts -// Set difference -minusWherePredicates( - inOp(ref('status'), ['A', 'B', 'C', 'D']), // status IN ['A','B','C','D'] - inOp(ref('status'), ['B', 'C']) // status IN ['B','C'] -) // → status IN ['A', 'D'] -``` - -```ts -// Common conditions -minusWherePredicates( - and(gt(ref('age'), val(10)), eq(ref('status'), val('active'))), // age > 10 AND status = 'active' - and(gt(ref('age'), val(20)), eq(ref('status'), val('active'))) // age > 20 AND status = 'active' -) // → age > 10 AND age <= 20 AND status = 'active' -``` - -```ts -// Complete overlap - empty result -minusWherePredicates( - gt(ref('age'), val(20)), // age > 20 - gt(ref('age'), val(10)) // age > 10 -) // → {type: 'val', value: false} (empty set) -``` diff --git a/docs/reference/functions/unionWherePredicates.md b/docs/reference/functions/unionWherePredicates.md deleted file mode 100644 index 1690f81c2b..0000000000 --- a/docs/reference/functions/unionWherePredicates.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -id: unionWherePredicates -title: unionWherePredicates ---- - -# Function: unionWherePredicates() - -```ts -function unionWherePredicates(predicates): BasicExpression; -``` - -Defined in: [packages/db/src/query/predicate-utils.ts:328](https://github.com/TanStack/db/blob/main/packages/db/src/query/predicate-utils.ts#L328) - -Combine multiple where predicates with OR logic (union). -Returns a predicate that is satisfied when any input predicate is satisfied. -Simplifies when possible (e.g., age > 10 OR age > 20 → age > 10). - -## Parameters - -### predicates - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\>[] - -Array of where predicates to union - -## Returns - -[`BasicExpression`](../@tanstack/namespaces/IR/type-aliases/BasicExpression.md)\<`boolean`\> - -Combined predicate representing the union - -## Examples - -```ts -// Take least restrictive -unionWherePredicates([gt(ref('age'), val(10)), gt(ref('age'), val(20))]) // age > 10 -``` - -```ts -// Combine equals into IN -unionWherePredicates([eq(ref('age'), val(5)), eq(ref('age'), val(10))]) // age IN [5, 10] -``` diff --git a/docs/reference/index.md b/docs/reference/index.md index dd0866a331..e288601b5d 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -12,7 +12,6 @@ title: "@tanstack/db" ## Classes - [AggregateFunctionNotInSelectError](classes/AggregateFunctionNotInSelectError.md) -- [AggregateNotSupportedError](classes/AggregateNotSupportedError.md) - [BaseIndex](classes/BaseIndex.md) - [BaseQueryBuilder](classes/BaseQueryBuilder.md) - [BasicIndex](classes/BasicIndex.md) @@ -89,7 +88,6 @@ title: "@tanstack/db" - [StorageError](classes/StorageError.md) - [StorageKeyRequiredError](classes/StorageKeyRequiredError.md) - [SubQueryMustHaveFromClauseError](classes/SubQueryMustHaveFromClauseError.md) -- [SubscriptionNotFoundError](classes/SubscriptionNotFoundError.md) - [SyncCleanupError](classes/SyncCleanupError.md) - [SyncTransactionAbortedError](classes/SyncTransactionAbortedError.md) - [SyncTransactionAlreadyCommittedError](classes/SyncTransactionAlreadyCommittedError.md) @@ -113,7 +111,6 @@ title: "@tanstack/db" - [UnsupportedJoinTypeError](classes/UnsupportedJoinTypeError.md) - [UnsupportedRootScalarSelectError](classes/UnsupportedRootScalarSelectError.md) - [UpdateKeyNotFoundError](classes/UpdateKeyNotFoundError.md) -- [WhereClauseConversionError](classes/WhereClauseConversionError.md) ## Interfaces @@ -139,7 +136,6 @@ title: "@tanstack/db" - [IndexDevModeConfig](interfaces/IndexDevModeConfig.md) - [IndexInterface](interfaces/IndexInterface.md) - [IndexOptions](interfaces/IndexOptions.md) -- [IndexStats](interfaces/IndexStats.md) - [IndexSuggestion](interfaces/IndexSuggestion.md) - [InsertConfig](interfaces/InsertConfig.md) - [LiveQueryCollectionConfig](interfaces/LiveQueryCollectionConfig.md) @@ -362,16 +358,10 @@ title: "@tanstack/db" - [isCollection](functions/isCollection.md) - [isCollectionOptions](functions/isCollectionOptions.md) - [isDevModeEnabled](functions/isDevModeEnabled.md) -- [isLimitSubset](functions/isLimitSubset.md) - [isLiveQueryWindowCollection](functions/isLiveQueryWindowCollection.md) -- [isLoadSubsetRequestSubsumedBy](functions/isLoadSubsetRequestSubsumedBy.md) - [isNull](functions/isNull.md) -- [isOffsetLimitSubset](functions/isOffsetLimitSubset.md) -- [isOrderBySubset](functions/isOrderBySubset.md) -- [isPredicateSubset](functions/isPredicateSubset.md) - [isSingleResultCollection](functions/isSingleResultCollection.md) - [isUndefined](functions/isUndefined.md) -- [isWhereSubset](functions/isWhereSubset.md) - [length](functions/length.md) - [like](functions/like.md) - [liveQueryCollectionOptions](functions/liveQueryCollectionOptions.md) @@ -383,7 +373,6 @@ title: "@tanstack/db" - [materialize](functions/materialize.md) - [max](functions/max.md) - [min](functions/min.md) -- [minusWherePredicates](functions/minusWherePredicates.md) - [multiply](functions/multiply.md) - [normalizeLiveQueryWindowPageSize](functions/normalizeLiveQueryWindowPageSize.md) - [not](functions/not.md) @@ -404,7 +393,6 @@ title: "@tanstack/db" - [toArray](functions/toArray.md) - [toBooleanPredicate](functions/toBooleanPredicate.md) - [trackQuery](functions/trackQuery.md) -- [unionWherePredicates](functions/unionWherePredicates.md) - [upper](functions/upper.md) - [walkExpression](functions/walkExpression.md) - [withArrayChangeTracking](functions/withArrayChangeTracking.md) diff --git a/docs/reference/interfaces/IndexInterface.md b/docs/reference/interfaces/IndexInterface.md index 415f739966..b5fd2d660c 100644 --- a/docs/reference/interfaces/IndexInterface.md +++ b/docs/reference/interfaces/IndexInterface.md @@ -93,19 +93,6 @@ Defined in: [packages/db/src/indexes/base-index.ts:63](https://github.com/TanSta *** -### getStats() - -```ts -getStats: () => IndexStats; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:107](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L107) - -#### Returns - -[`IndexStats`](IndexStats.md) - -*** ### inArrayLookup() diff --git a/docs/reference/interfaces/IndexStats.md b/docs/reference/interfaces/IndexStats.md deleted file mode 100644 index 2fec3b7709..0000000000 --- a/docs/reference/interfaces/IndexStats.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -id: IndexStats -title: IndexStats ---- - -# Interface: IndexStats - -Defined in: [packages/db/src/indexes/base-index.ts:44](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L44) - -Statistics about index usage and performance - -## Properties - -### averageLookupTime - -```ts -readonly averageLookupTime: number; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:47](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L47) - -*** - -### entryCount - -```ts -readonly entryCount: number; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:45](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L45) - -*** - -### lastUpdated - -```ts -readonly lastUpdated: Date; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:48](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L48) - -*** - -### lookupCount - -```ts -readonly lookupCount: number; -``` - -Defined in: [packages/db/src/indexes/base-index.ts:46](https://github.com/TanStack/db/blob/main/packages/db/src/indexes/base-index.ts#L46) diff --git a/docs/reference/type-aliases/CursorExpressions.md b/docs/reference/type-aliases/CursorExpressions.md index 2f14b5e515..6bd850026a 100644 --- a/docs/reference/type-aliases/CursorExpressions.md +++ b/docs/reference/type-aliases/CursorExpressions.md @@ -55,6 +55,5 @@ whereFrom: BasicExpression; Defined in: [packages/db/src/types.ts:290](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L290) Expression for rows greater than (after) the cursor value. -For multi-column orderBy, this is a composite cursor using OR of conditions. -Example for [col1 ASC, col2 DESC] with values [v1, v2]: - or(gt(col1, v1), and(eq(col1, v1), lt(col2, v2))) +Core emits cursors for a single order column. Multi-column queries use +prefix-and-tie loading instead of constructing a composite cursor. diff --git a/docs/reference/type-aliases/LiveQueryCollectionUtils.md b/docs/reference/type-aliases/LiveQueryCollectionUtils.md index 620333d20e..4e337c93e6 100644 --- a/docs/reference/type-aliases/LiveQueryCollectionUtils.md +++ b/docs/reference/type-aliases/LiveQueryCollectionUtils.md @@ -19,16 +19,6 @@ Defined in: [packages/db/src/query/live/collection-config-builder.ts:50](https:/ [LIVE_QUERY_INTERNAL]: LiveQueryInternalUtils; ``` -### getRunCount() - -```ts -getRunCount: () => number; -``` - -#### Returns - -`number` - ### getWindow() ```ts diff --git a/docs/reference/type-aliases/LoadSubsetOptions.md b/docs/reference/type-aliases/LoadSubsetOptions.md index a2f8153c03..54181a7b96 100644 --- a/docs/reference/type-aliases/LoadSubsetOptions.md +++ b/docs/reference/type-aliases/LoadSubsetOptions.md @@ -11,6 +11,13 @@ type LoadSubsetOptions = object; Defined in: [packages/db/src/types.ts:304](https://github.com/TanStack/db/blob/main/packages/db/src/types.ts#L304) +Request data is immutable from submission onward. Callers and adapters must not +mutate options, expression trees, comparison options, or constant payloads such +as Dates, byte arrays, and membership arrays. Create new request data to change +a demand. Core does not clone or freeze it. Use stable data properties, not +stateful getters. Signal and subscription references stay fixed, but aborting +the signal or releasing the subscription remains supported. + ## Properties ### cursor? diff --git a/package.json b/package.json index 93eae3610d..43675440d6 100644 --- a/package.json +++ b/package.json @@ -18,6 +18,7 @@ "lint-all": "eslint . --fix", "prepare": "husky", "test": "pnpm --filter \"./packages/**\" test", + "test:oracles": "pnpm --filter @tanstack/db test:oracles && pnpm --filter @tanstack/query-db-collection test:oracles", "test:docs": "node scripts/verify-links.ts", "test:sherif": "sherif -i zod -p offline-transactions-react-native -p shopping-list-react-native", "generate-docs": "node scripts/generate-docs.ts" diff --git a/packages/db-ivm/package.json b/packages/db-ivm/package.json index 5d400a632e..560ff822db 100644 --- a/packages/db-ivm/package.json +++ b/packages/db-ivm/package.json @@ -22,7 +22,8 @@ "build": "vite build", "dev": "vite build --watch", "lint": "eslint . --fix", - "test": "vitest --run" + "test": "vitest --run", + "bench:hash": "vitest bench --run tests/hash.bench.ts --coverage.enabled=false" }, "type": "module", "main": "dist/cjs/index.cjs", diff --git a/packages/db-ivm/src/hashing/hash.ts b/packages/db-ivm/src/hashing/hash.ts index 813e4ed35c..51c33b43f1 100644 --- a/packages/db-ivm/src/hashing/hash.ts +++ b/packages/db-ivm/src/hashing/hash.ts @@ -1,4 +1,4 @@ -import { MurmurHashStream, randomHash } from './murmur.js' +import { MurmurHashStream, getSymbolIdentity, randomHash } from './murmur.js' import type { Hasher } from './murmur.js' /* @@ -19,6 +19,10 @@ const MAP_MARKER = randomHash() const SET_MARKER = randomHash() const UINT8ARRAY_MARKER = randomHash() const TEMPORAL_MARKER = randomHash() +// Bound structural recursion and value visits. Shared acyclic subtrees are +// cached; cycles are rejected rather than given context-dependent hashes. +const MAX_STRUCTURAL_HASH_WORK = 1_000_000 +const MAX_STRUCTURAL_HASH_DEPTH = 768 const temporalTypes = new Set([ `Temporal.Duration`, @@ -48,63 +52,65 @@ const UINT8ARRAY_CONTENT_HASH_THRESHOLD = 128 const hashCache = new WeakMap() +/** @internal Register a mutable handle before it enters a structural value. */ +export function registerOpaqueHash(value: object): void { + cachedReferenceHash(value) +} + +type HashContext = { + activeObjects: Set + work: number + pendingHashes: Map +} + export function hash(input: any): number { const hasher = new MurmurHashStream() updateHasher(hasher, input) return hasher.digest() } -function hashObject(input: object): number { - const cachedHash = hashCache.get(input) - if (cachedHash !== undefined) { - return cachedHash +function hashObject(input: object, context: HashContext): number { + if (context.activeObjects.size >= MAX_STRUCTURAL_HASH_DEPTH) { + throw new RangeError( + `Value is too complex to hash safely: structural depth`, + ) } + context.activeObjects.add(input) + let valueHash: number | undefined - if (input instanceof Date) { - valueHash = hashDate(input) - } else if ( - // Check if input is a Uint8Array or Buffer - (typeof Buffer !== `undefined` && input instanceof Buffer) || - input instanceof Uint8Array - ) { - // For small Uint8Arrays/Buffers (e.g., ULIDs, UUIDs), hash by content - // to enable proper equality comparisons. For large arrays, hash by reference - // to avoid performance costs. - if (input.byteLength <= UINT8ARRAY_CONTENT_HASH_THRESHOLD) { + try { + if (input instanceof Date) { + valueHash = hashDate(input) + } else if (isBinaryValue(input)) { valueHash = hashUint8Array(input) + } else if (isTemporal(input)) { + valueHash = hashTemporal(input) } else { - // Deeply hashing large arrays would be too costly - // so we track them by reference and cache them in a weak map - return cachedReferenceHash(input) - } - } else if (input instanceof File) { - // Files are always hashed by reference due to their potentially large size - return cachedReferenceHash(input) - } else if (isTemporal(input)) { - valueHash = hashTemporal(input) - } else { - let plainObjectInput = input - let marker = OBJECT_MARKER - - if (input instanceof Array) { - marker = ARRAY_MARKER - } + let plainObjectInput = input + let marker = OBJECT_MARKER - if (input instanceof Map) { - marker = MAP_MARKER - plainObjectInput = [...input.entries()] - } + if (input instanceof Array) { + marker = ARRAY_MARKER + } - if (input instanceof Set) { - marker = SET_MARKER - plainObjectInput = [...input.entries()] - } + if (input instanceof Map) { + marker = MAP_MARKER + plainObjectInput = [...input.entries()] + } - valueHash = hashPlainObject(plainObjectInput, marker) + if (input instanceof Set) { + marker = SET_MARKER + plainObjectInput = [...input.entries()] + } + + valueHash = hashPlainObject(plainObjectInput, marker, context) + } + } finally { + context.activeObjects.delete(input) } - hashCache.set(input, valueHash) + context.pendingHashes.set(input, valueHash) return valueHash } @@ -135,7 +141,11 @@ function hashTemporal(input: TemporalLike): number { return hasher.digest() } -function hashPlainObject(input: object, marker: number): number { +function hashPlainObject( + input: object, + marker: number, + context: HashContext, +): number { const hasher = new MurmurHashStream() // Mark the type of the input @@ -145,13 +155,28 @@ function hashPlainObject(input: object, marker: number): number { for (const key of keys) { hasher.update(KEY) hasher.update(key) - updateHasher(hasher, input[key as keyof typeof input]) + updateHasher(hasher, input[key as keyof typeof input], context) + } + const symbolKeys = Object.getOwnPropertySymbols(input) + .filter((key) => Object.prototype.propertyIsEnumerable.call(input, key)) + .sort((left, right) => getSymbolIdentity(left) - getSymbolIdentity(right)) + for (const key of symbolKeys) { + hasher.update(KEY) + hasher.update(key) + updateHasher(hasher, input[key as keyof typeof input], context) } return hasher.digest() } -function updateHasher(hasher: Hasher, input: unknown): void { +function updateHasher( + hasher: Hasher, + input: unknown, + context?: HashContext, +): void { + if (context && ++context.work > MAX_STRUCTURAL_HASH_WORK) { + throw new RangeError(`Value is too complex to hash safely: structural work`) + } if (input === null) { hasher.update(NULL) return @@ -173,7 +198,7 @@ function updateHasher(hasher: Hasher, input: unknown): void { hasher.update(input) return case `object`: - hasher.update(getCachedHash(input)) + hasher.update(getCachedHash(input, context)) return case `function`: // Functions are assigned a globally unique ID @@ -187,12 +212,53 @@ function updateHasher(hasher: Hasher, input: unknown): void { } } -function getCachedHash(input: object): number { - let valueHash = hashCache.get(input) - if (valueHash === undefined) { - valueHash = hashObject(input) +function getCachedHash(input: object, context?: HashContext): number { + if (!context) { + const cached = hashCache.get(input) + if (cached !== undefined) return cached + if (isReferenceHashedObject(input)) return cachedReferenceHash(input) + + // Only an uncached structural root needs graph traversal state. Commit its + // cache entries after success so a failed traversal cannot poison retries. + context = { + activeObjects: new Set(), + work: 0, + pendingHashes: new Map(), + } + const result = hashObject(input, context) + for (const [object, valueHash] of context.pendingHashes) { + hashCache.set(object, valueHash) + } + return result } - return valueHash + + if (context.activeObjects.has(input)) { + throw new TypeError(`Cannot hash cyclic structural values`) + } + + // Opaque leaves cannot contain structural back-references. Resolve them + // before entering structural recursion, even when they have user properties. + if (isReferenceHashedObject(input)) return cachedReferenceHash(input) + + const valueHash = hashCache.get(input) ?? context.pendingHashes.get(input) + if (valueHash !== undefined) return valueHash + + return hashObject(input, context) +} + +function isReferenceHashedObject(input: object): boolean { + return ( + input instanceof File || + (isBinaryValue(input) && + input.byteLength > UINT8ARRAY_CONTENT_HASH_THRESHOLD) + ) +} + +function isBinaryValue(input: object): input is Uint8Array { + return ( + (typeof Buffer !== `undefined` && input instanceof Buffer) || + input instanceof Uint8Array + ) } let nextRefId = 1 diff --git a/packages/db-ivm/src/hashing/murmur.ts b/packages/db-ivm/src/hashing/murmur.ts index 9ce68be312..cd40030694 100644 --- a/packages/db-ivm/src/hashing/murmur.ts +++ b/packages/db-ivm/src/hashing/murmur.ts @@ -9,6 +9,49 @@ const BIG_INT_MARKER = randomHash() const NEG_BIG_INT_MARKER = randomHash() const SYMBOL_MARKER = randomHash() +type SymbolIdStore = { + get: (key: symbol) => number | undefined + set: (key: symbol, value: number) => unknown +} + +const symbolIds = createSymbolIdStore() +const registeredSymbolIds = new Map() +let nextSymbolId = 0 + +export function getSymbolIdentity(symbol: symbol): number { + const registeredKey = Symbol.keyFor(symbol) + if (registeredKey !== undefined) { + let id = registeredSymbolIds.get(registeredKey) + if (id === undefined) { + id = ++nextSymbolId + registeredSymbolIds.set(registeredKey, id) + } + return id + } + + let id = symbolIds.get(symbol) + if (id === undefined) { + id = ++nextSymbolId + symbolIds.set(symbol, id) + } + return id +} + +function createSymbolIdStore(): SymbolIdStore { + const weakIds = new WeakMap() as unknown as SymbolIdStore + const probe = Symbol() + + try { + weakIds.set(probe, 0) + if (weakIds.get(probe) === 0) return weakIds + } catch { + // Older runtimes reject symbols as weak keys. Retain them rather than + // merge distinct symbols and corrupt differential state. + } + + return new Map() +} + export type Hash = number export function randomHash() { @@ -67,16 +110,7 @@ export class MurmurHashStream implements Hasher { switch (typeof chunk) { case `symbol`: { this.update(SYMBOL_MARKER) - const description = chunk.description - if (!description) { - return - } - - for (let i = 0; i < description.length; i++) { - const code = description.charCodeAt(i) - this.writeByte(code & 0xff) - this.writeByte((code >>> 8) & 0xff) - } + this.update(getSymbolIdentity(chunk)) return } case `string`: diff --git a/packages/db-ivm/src/index.ts b/packages/db-ivm/src/index.ts index cae148b46e..441da1e484 100644 --- a/packages/db-ivm/src/index.ts +++ b/packages/db-ivm/src/index.ts @@ -3,3 +3,4 @@ export * from './multiset.js' export * from './operators/index.js' export * from './types.js' export { compareKeys, serializeValue } from './utils.js' +export { registerOpaqueHash } from './hashing/hash.js' diff --git a/packages/db-ivm/src/operators/groupBy.ts b/packages/db-ivm/src/operators/groupBy.ts index 9c2fe1e357..435156d25d 100644 --- a/packages/db-ivm/src/operators/groupBy.ts +++ b/packages/db-ivm/src/operators/groupBy.ts @@ -62,7 +62,7 @@ export function groupBy< stream: IStreamBuilder, ): IStreamBuilder> => { // Special key to store the original key object - const KEY_SENTINEL = `__original_key__` + const KEY_SENTINEL = Symbol(`original_group_key`) // First map to extract keys and pre-aggregate values const withKeysAndValues = stream.pipe( @@ -71,7 +71,7 @@ export function groupBy< const keyString = serializeValue(key) // Create values object with pre-aggregated values - const values: Record = {} + const values: Record = {} // Store the original key object values[KEY_SENTINEL] = key @@ -81,7 +81,10 @@ export function groupBy< values[name] = aggregate.preMap(data) } - return [keyString, values] as KeyValue> + return [keyString, values] as KeyValue< + string, + Record + > }), ) @@ -99,7 +102,7 @@ export function groupBy< return [] } - const result: Record = {} + const result: Record = {} // Get the original key from first value in group const originalKey = values[0]?.[0]?.[KEY_SENTINEL] diff --git a/packages/db-ivm/src/operators/orderByBTree.ts b/packages/db-ivm/src/operators/orderByBTree.ts deleted file mode 100644 index db95b2deac..0000000000 --- a/packages/db-ivm/src/operators/orderByBTree.ts +++ /dev/null @@ -1,20 +0,0 @@ -import { orderByWithFractionalIndexBase } from './orderBy.js' -import { topKWithFractionalIndexBTree } from './topKWithFractionalIndexBTree.js' -import type { KeyValue } from '../types.js' -import type { OrderByOptions } from './orderBy.js' - -export function orderByWithFractionalIndexBTree< - T extends KeyValue, - Ve = unknown, ->( - valueExtractor: ( - value: T extends KeyValue ? V : never, - ) => Ve, - options?: OrderByOptions, -) { - return orderByWithFractionalIndexBase( - topKWithFractionalIndexBTree, - valueExtractor, - options, - ) -} diff --git a/packages/db-ivm/src/utils.ts b/packages/db-ivm/src/utils.ts index 2c4170c897..d7569ba211 100644 --- a/packages/db-ivm/src/utils.ts +++ b/packages/db-ivm/src/utils.ts @@ -185,6 +185,14 @@ function range(start: number, end: number): Array { export function compareKeys(a: string | number, b: string | number): number { // Same type: compare directly if (typeof a === typeof b) { + if (typeof a === `number` && typeof b === `number`) { + const aIsNaN = Number.isNaN(a) + const bIsNaN = Number.isNaN(b) + if (aIsNaN || bIsNaN) { + if (aIsNaN && bIsNaN) return 0 + return aIsNaN ? 1 : -1 + } + } if (a < b) return -1 if (a > b) return 1 return 0 diff --git a/packages/db-ivm/tests/hash-graph.property.test.ts b/packages/db-ivm/tests/hash-graph.property.test.ts new file mode 100644 index 0000000000..c04dc7574f --- /dev/null +++ b/packages/db-ivm/tests/hash-graph.property.test.ts @@ -0,0 +1,119 @@ +import { describe, expect, it } from 'vitest' +import { fc } from '@fast-check/vitest' +import { hash } from '../src/hashing/hash' + +// Kahn's algorithm checks the reachable graph without using the hasher's +// recursive active-path algorithm. Unreachable cycles do not affect the root. +function isAcyclic(edges: Array>): boolean { + const reachable = new Set([0]) + for (const node of reachable) { + for (const target of edges[node]!) reachable.add(target) + } + const incoming = new Map([...reachable].map((node) => [node, 0])) + for (const node of reachable) { + for (const target of edges[node]!) { + incoming.set(target, incoming.get(target)! + 1) + } + } + const ready = [...reachable].filter((node) => incoming.get(node) === 0) + for (const node of ready) { + for (const target of edges[node]!) { + const remaining = incoming.get(target)! - 1 + incoming.set(target, remaining) + if (remaining === 0) ready.push(target) + } + } + return ready.length === reachable.size +} + +const graphArbitrary = fc + .array(fc.array(fc.nat({ max: 5 }), { maxLength: 3 }), { + minLength: 1, + maxLength: 6, + }) + .map((edges) => + edges.map((targets) => targets.map((target) => target % edges.length)), + ) + +describe(`structural hash graph boundary`, () => { + it.each([ + `object`, + `array`, + `map-key`, + `map-value`, + `set`, + `symbol`, + ] as const)(`rejects a cycle through %s on every attempt`, (kind) => { + const record: Record = {} + const array: Array = [] + const map = new Map() + const set = new Set() + const input = + kind === `array` + ? array + : kind.startsWith(`map`) + ? map + : kind === `set` + ? set + : record + if (kind === `object`) record.self = input + if (kind === `symbol`) record[Symbol(`self`)] = input + if (kind === `array`) array.push(input) + if (kind === `map-key`) map.set(input, 1) + if (kind === `map-value`) map.set(1, input) + if (kind === `set`) set.add(input) + for (let attempt = 0; attempt < 2; attempt++) { + expect(() => hash(input)).toThrow(`Cannot hash cyclic structural values`) + } + }) + + for (const seed of [1657019, undefined]) { + it(`matches reachable graph cycles and shared DAGs (${seed ?? `random`})`, () => { + fc.assert( + fc.property(graphArbitrary, (edges) => { + const nodes = edges.map((_, value) => ({ + value, + children: [] as Array, + })) + edges.forEach((targets, index) => { + nodes[index]!.children = targets.map((target) => nodes[target]) + }) + if (!isAcyclic(edges)) { + expect(() => hash(nodes[0])).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(nodes[0])).toThrow( + `Cannot hash cyclic structural values`, + ) + return + } + // Unfold sharing into equal but distinct subtrees. Hash identity must + // depend on values, not whether the graph reused an object reference. + const unfold = (node: number): unknown => ({ + value: node, + children: edges[node]!.map(unfold), + }) + expect(hash(nodes[0])).toBe(hash(unfold(0))) + }), + { numRuns: 300, ...(seed === undefined ? {} : { seed }) }, + ) + }) + } + + it(`leaves completed siblings uncached after a cycle rejects the root`, () => { + let reads = 0 + const sibling = { + get value() { + return ++reads + }, + } + const root: Record = { a: sibling } + root.z = root + expect(() => hash(root)).toThrow(`Cannot hash cyclic structural values`) + expect(() => hash(root)).toThrow(`Cannot hash cyclic structural values`) + expect(reads).toBe(2) + delete root.z + expect(hash(root)).toBe(hash({ a: { value: 3 } })) + expect(reads).toBe(3) + }) +}) diff --git a/packages/db-ivm/tests/hash-work.test.ts b/packages/db-ivm/tests/hash-work.test.ts new file mode 100644 index 0000000000..e0054c652f --- /dev/null +++ b/packages/db-ivm/tests/hash-work.test.ts @@ -0,0 +1,126 @@ +import { describe, expect, it, vi } from 'vitest' +import { hash, registerOpaqueHash } from '../src/hashing/hash' + +function countTraversalAllocations(run: () => void): number { + let allocations = 0 + for (const name of [`Map`, `Set`, `WeakMap`] as const) { + vi.stubGlobal( + name, + new Proxy(globalThis[name], { + construct(target, args) { + allocations++ + return Reflect.construct(target, args) + }, + }), + ) + } + try { + run() + } finally { + vi.unstubAllGlobals() + } + return allocations +} + +describe(`hash traversal work`, () => { + it.each([`object`, `array`] as const)( + `does not let a rejected %s traversal subsidize its own retry`, + (kind) => { + const left = Array.from({ length: 500_001 }, () => 0) + const right = Array.from({ length: 500_001 }, () => 0) + const root = kind === `object` ? { left, right } : [left, right] + // Either child fits, but this fresh root exceeds the combined work cap. + // Keeping the completed left child's cache after failure lets retry pass. + for (let attempt = 0; attempt < 2; attempt++) { + expect(() => hash(root)).toThrow( + `Value is too complex to hash safely: structural work`, + ) + } + }, + ) + + it(`does not allocate traversal collections for primitive and cached inputs`, () => { + const cached = { id: 1, title: `cached` } + hash(cached) + const inputs = [null, undefined, false, 0, 1n, `row`, Symbol(`key`), cached] + expect( + countTraversalAllocations(() => { + for (const input of inputs) hash(input) + }), + ).toBe(0) + }) + + it(`measures traversal collections for fresh structural inputs`, () => { + expect(countTraversalAllocations(() => hash({ id: 1 }))).toBeGreaterThan(0) + }) + + it(`uses identity for registered handles without traversing mutable internals`, () => { + const first: Record = {} + const second: Record = {} + for (const value of [first, second]) { + value.self = value + Object.defineProperty(value, `state`, { + enumerable: true, + get() { + throw new Error(`must not read handle state`) + }, + }) + registerOpaqueHash(value) + } + const before = hash({ handle: first }) + first.changed = true + expect(hash({ handle: first })).toBe(before) + expect(hash({ handle: second })).not.toBe(before) + expect( + countTraversalAllocations(() => { + hash(first) + hash(second) + }), + ).toBe(0) + }) + + it(`visits each shared acyclic subtree once`, () => { + let reads = 0 + let root: object = { value: 1 } + for (let depth = 0; depth < 200; depth++) { + const child = root + root = { + get left() { + reads++ + return child + }, + get right() { + reads++ + return child + }, + } + } + const result = hash(root) + expect(reads).toBe(400) + expect(hash(root)).toBe(result) + expect(reads).toBe(400) + }) + + it(`bounds value visits without publishing partial structural caches`, () => { + let reads = 0 + const shared = {} + const root = { + a: { + get value() { + reads++ + return 1 + }, + }, + // Repeated references must still count as work, even when their hashes + // are cached; no expanded tree is needed to reach the bound. + z: Array.from({ length: 1_000_001 }, () => shared), + } + for (let attempt = 1; attempt <= 2; attempt++) { + expect(() => hash(root)).toThrow( + `Value is too complex to hash safely: structural work`, + ) + expect(reads).toBe(attempt) + } + expect(hash({ value: 1 })).toBe(hash({ value: 1 })) + }) +}) diff --git a/packages/db-ivm/tests/hash.bench.ts b/packages/db-ivm/tests/hash.bench.ts new file mode 100644 index 0000000000..d870d968c3 --- /dev/null +++ b/packages/db-ivm/tests/hash.bench.ts @@ -0,0 +1,23 @@ +import { bench, describe } from 'vitest' +import { hash } from '../src/hashing/hash' + +const cached = { id: 1, title: `row`, active: true } +hash(cached) +let result = 0 + +describe(`hash input paths`, () => { + bench(`primitive`, () => { + result ^= hash(42) + }) + bench(`cached row`, () => { + result ^= hash(cached) + }) + bench(`fresh row`, () => { + result ^= hash({ id: 1, title: `row`, active: true }) + }) +}) + +// Keep benchmark results observable without adding work inside each sample. +export function getHashBenchmarkResult(): number { + return result +} diff --git a/packages/db-ivm/tests/operators/groupBy.test.ts b/packages/db-ivm/tests/operators/groupBy.test.ts index 615e31fdb3..52b0fac653 100644 --- a/packages/db-ivm/tests/operators/groupBy.test.ts +++ b/packages/db-ivm/tests/operators/groupBy.test.ts @@ -132,6 +132,39 @@ describe(`Operators`, () => { expect(result).toEqual(expectedResult) }) + test(`does not reserve an aggregate name for its original group key`, () => { + const graph = new D2() + const input = graph.newInput<{ category: string }>() + let latestMessage: MultiSet | undefined + + input.pipe( + groupBy((data) => ({ category: data.category }), { + __original_key__: count(), + }), + output((message) => { + latestMessage = message + }), + ) + graph.finalize() + input.sendData( + new MultiSet([ + [{ category: `A` }, 1], + [{ category: `A` }, 1], + ]), + ) + graph.run() + + expect(latestMessage?.getInner()).toEqual([ + [ + [ + serializeValue({ category: `A` }), + { category: `A`, __original_key__: 2 }, + ], + 1, + ], + ]) + }) + test(`with sum and count aggregates`, () => { const graph = new D2() const input = graph.newInput<{ diff --git a/packages/db-ivm/tests/operators/orderByWithFractionalIndex.test.ts b/packages/db-ivm/tests/operators/orderByWithFractionalIndex.test.ts index 4dc16430fb..e341b84d5e 100644 --- a/packages/db-ivm/tests/operators/orderByWithFractionalIndex.test.ts +++ b/packages/db-ivm/tests/operators/orderByWithFractionalIndex.test.ts @@ -5,11 +5,20 @@ import { orderByWithFractionalIndex, output, } from '../../src/operators/index.js' -import { orderByWithFractionalIndexBTree } from '../../src/operators/orderByBTree.js' -import { loadBTree } from '../../src/operators/topKWithFractionalIndexBTree.js' +import { orderByWithFractionalIndexBase } from '../../src/operators/orderBy.js' +import { + loadBTree, + topKWithFractionalIndexBTree, +} from '../../src/operators/topKWithFractionalIndexBTree.js' import { MessageTracker, compareFractionalIndex } from '../test-utils.js' import type { KeyValue } from '../../src/types.js' +const orderByWithBTree: typeof orderByWithFractionalIndex = ( + extract, + options, +) => + orderByWithFractionalIndexBase(topKWithFractionalIndexBTree, extract, options) + const stripFractionalIndex = ([[key, [value, _index]], multiplicity]: any) => [ key, value, @@ -27,7 +36,7 @@ beforeAll(async () => { describe(`Operators`, () => { describe.each([ [`with array`, { orderBy: orderByWithFractionalIndex }], - [`with B+ tree`, { orderBy: orderByWithFractionalIndexBTree }], + [`with B+ tree`, { orderBy: orderByWithBTree }], ])(`OrderByWithFractionalIndex operator %s`, (_, { orderBy }) => { test(`initial results with default comparator`, () => { const graph = new D2() diff --git a/packages/db-ivm/tests/utils.test.ts b/packages/db-ivm/tests/utils.test.ts index 3e6b17f4c1..c9e35c0ca2 100644 --- a/packages/db-ivm/tests/utils.test.ts +++ b/packages/db-ivm/tests/utils.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from 'vitest' import { Temporal } from 'temporal-polyfill' -import { DefaultMap, serializeValue } from '../src/utils.js' +import { DefaultMap, compareKeys, serializeValue } from '../src/utils.js' import { hash } from '../src/hashing/index.js' describe(`DefaultMap`, () => { @@ -30,6 +30,14 @@ describe(`DefaultMap`, () => { }) }) +describe(`compareKeys`, () => { + it(`orders finite numeric keys before NaN`, () => { + expect(compareKeys(1, Number.NaN)).toBeLessThan(0) + expect(compareKeys(Number.NaN, 1)).toBeGreaterThan(0) + expect(compareKeys(Number.NaN, Number.NaN)).toBe(0) + }) +}) + describe(`serializeValue`, () => { it(`preserves the established JSON form for ordinary keys`, () => { expect(serializeValue(`user1`)).toBe(`"user1"`) @@ -145,12 +153,21 @@ describe(`hash`, () => { expect(typeof result1).toBe(hashType) expect(typeof result2).toBe(hashType) expect(typeof result3).toBe(hashType) - // Note: Different symbol instances with same description have same hash - expect(result1).toBe(result2) + expect(result1).not.toBe(result2) expect(result1).not.toBe(result3) - expect(result4).toBe(result5) + expect(result4).not.toBe(result5) expect(result1).not.toBe(result4) }) + + it(`should hash registered symbols`, () => { + const first = Symbol.for(`tanstack-db-ivm-hash-first`) + const same = Symbol.for(`tanstack-db-ivm-hash-first`) + const second = Symbol.for(`tanstack-db-ivm-hash-second`) + + expect(hash(first)).toBe(hash(same)) + expect(hash(first)).not.toBe(hash(second)) + expect(hash({ [first]: 1 })).not.toBe(hash({ [second]: 1 })) + }) }) describe(`object types`, () => { @@ -166,6 +183,355 @@ describe(`hash`, () => { // Note: Different key orders might produce different hashes depending on JSON.stringify behavior }) + it(`includes enumerable symbol keys and values`, () => { + const key = Symbol(`key`) + + expect(hash({ [key]: `before` })).not.toBe(hash({ [key]: `after` })) + expect(hash({ [Symbol(`key`)]: `value` })).not.toBe( + hash({ [Symbol(`key`)]: `value` }), + ) + }) + + it(`rejects self and mutual cycles through symbol keys`, () => { + const key = Symbol(`cycle`) + const first: Record = {} + const second: Record = {} + first[key] = first + second[key] = second + + const firstPeer: Record = {} + const secondPeer: Record = {} + firstPeer[key] = secondPeer + secondPeer[key] = firstPeer + + for (const input of [first, second, firstPeer, secondPeer]) { + expect(() => hash(input)).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(input)).toThrow( + `Cannot hash cyclic structural values`, + ) + } + }) + + it.each([`object`, `map`] as const)( + `rejects shared cyclic branches through %s with bounded work`, + (container) => { + const size = 14 + let reads = 0 + const nodes: Array | Map> = + Array.from({ length: size }, (_, value) => + container === `object` + ? { value } + : new Map([[`value`, value]]), + ) + + for (let index = 0; index < size; index++) { + const node = nodes[index]! + const next = nodes[(index + 1) % size]! + for (const key of [`left`, `right`] as const) { + const wrapper = Object.defineProperty({}, `next`, { + enumerable: true, + get: () => { + reads++ + return next + }, + }) + if (node instanceof Map) node.set(key, wrapper) + else node[key] = wrapper + } + } + + expect(() => hash(nodes[0]!)).toThrow( + `Cannot hash cyclic structural values`, + ) + const firstReads = reads + const copy = structuredClone(nodes[0]!) + + expect(() => hash(copy)).toThrow(`Cannot hash cyclic structural values`) + expect(firstReads).toBeLessThanOrEqual(size * 2) + }, + ) + + it(`rejects a shared child that cycles to either ancestor`, () => { + const createGraph = (backBranch: `left` | `right`) => { + const root: Record = {} + const left: Record = {} + const right: Record = {} + const shared: Record = {} + root.left = left + root.right = right + left.next = shared + right.next = shared + shared.back = backBranch === `left` ? left : right + return root + } + + const left = createGraph(`left`) + const equalLeft = createGraph(`left`) + const right = createGraph(`right`) + + for (const input of [left, equalLeft, right]) { + expect(() => hash(input)).toThrow( + `Cannot hash cyclic structural values`, + ) + } + }) + + it(`rejects cyclic graphs with exponentially many ancestor contexts`, () => { + const depth = 11 + const shared = Array.from( + { length: depth + 1 }, + (_, level) => ({ level }) as Record, + ) + const left = Array.from({ length: depth }, (_, level) => ({ + side: `left`, + level, + next: shared[level + 1], + })) + const right = Array.from({ length: depth }, (_, level) => ({ + side: `right`, + level, + next: shared[level + 1], + })) + for (let level = 0; level < depth; level++) { + shared[level]!.left = left[level] + shared[level]!.right = right[level] + shared[depth]![`left${level}`] = left[level] + } + + expect(() => hash(shared[0])).toThrow(TypeError) + expect(() => hash(shared[0])).toThrow( + `Cannot hash cyclic structural values`, + ) + + const ring = Array.from( + { length: 600 }, + (_, value) => ({ value }) as { value: number; next?: unknown }, + ) + for (let index = 0; index < ring.length; index++) { + ring[index]!.next = ring[(index + 1) % ring.length] + } + expect(() => hash(structuredClone(ring[0]))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(ring[0])).toThrow( + `Cannot hash cyclic structural values`, + ) + + const independent: Record = {} + for (let index = 0; index < 600; index++) { + const cycle: { self?: unknown } = {} + cycle.self = cycle + independent[String(index)] = cycle + } + expect(() => hash(structuredClone(independent))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(independent)).toThrow( + `Cannot hash cyclic structural values`, + ) + + const independentDiamonds: Record = {} + for (let index = 0; index < 600; index++) { + const diamondCenter: Record = {} + const leftIngress = { next: diamondCenter } + const rightIngress = { next: diamondCenter } + diamondCenter.back = leftIngress + independentDiamonds[`left${index}`] = leftIngress + independentDiamonds[`right${index}`] = rightIngress + } + expect(() => hash(structuredClone(independentDiamonds))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(independentDiamonds)).toThrow( + `Cannot hash cyclic structural values`, + ) + + const small: { self?: unknown } = {} + small.self = small + expect(() => hash(structuredClone(small))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(small)).toThrow(`Cannot hash cyclic structural values`) + }) + + it(`rejects both small and large repeated cyclic traversals`, () => { + const createGraph = (size: number) => { + const nodes = Array.from( + { length: size }, + (_, value) => ({ value }) as Record, + ) + for (let index = 0; index < size; index++) { + const next = nodes[(index + 1) % size]! + nodes[index]!.left = { next } + nodes[index]!.right = { next } + } + return nodes[0] + } + + expect(() => hash(createGraph(20))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(createGraph(300))).toThrow( + `Cannot hash cyclic structural values`, + ) + }) + + it(`does not warm structural caches when a hash is rejected`, () => { + let reads = 0 + const sentinel = Object.defineProperty({}, `value`, { + enumerable: true, + get: () => ++reads, + }) + const shared: Record = { + payload: Array.from({ length: 66_000 }, (_, value) => ({ value })), + } + const left = { next: shared } + const right = { next: shared } + shared.back = left + const root = { aSentinel: sentinel, left, right } + + expect(() => hash(root)).toThrow(`Cannot hash cyclic structural values`) + expect(() => hash(root)).toThrow(`Cannot hash cyclic structural values`) + expect(reads).toBe(2) + }) + + it.each([ + [`Buffer`, () => Buffer.alloc(129)], + [`Uint8Array`, () => new Uint8Array(129)], + [`File`, () => new File([`opaque`], `opaque.bin`)], + ])( + `treats a large %s as an opaque leaf before structural work`, + (_name, createLeaf) => { + const leaves = Array.from({ length: 700 }, createLeaf) + for (const leaf of leaves) Object.assign(leaf, { self: leaf }) + const createChain = () => { + const ring = leaves.map((leaf, value) => ({ + value, + leaf, + next: undefined as unknown, + })) + for (let index = 0; index < ring.length; index++) { + ring[index]!.next = ring[index + 1] + } + return ring[0] + } + + const first = createChain() + const expectedHash = hash(first) + expect(hash(first)).toBe(expectedHash) + expect(hash(createChain())).toBe(expectedHash) + + let atDepthBoundary: unknown = createLeaf() + for (let index = 0; index < 768; index++) { + atDepthBoundary = { next: atDepthBoundary } + } + expect(() => hash(atDepthBoundary)).not.toThrow() + + const adoptionLeaves = Array.from({ length: 20 }, createLeaf) + const createAdoptionGraph = () => { + const nodes = adoptionLeaves.map((leaf, value) => ({ + value, + leaf, + })) as Array> + for (let index = 0; index < nodes.length; index++) { + const next = nodes[index + 1] + nodes[index]!.left = { next } + nodes[index]!.right = { next } + } + return nodes[0] + } + expect(hash(createAdoptionGraph())).toBe(hash(createAdoptionGraph())) + }, + ) + + it(`rejects deep structural recursion before the JavaScript stack overflows`, () => { + let reads = 0 + const sentinel = Object.defineProperty({}, `value`, { + enumerable: true, + get: () => ++reads, + }) + const ring = Array.from( + { length: 800 }, + (_, value) => ({ value }) as { value: number; next?: unknown }, + ) + for (let index = 0; index < ring.length; index++) { + ring[index]!.next = ring[(index + 1) % ring.length] + } + Object.defineProperty(ring[0]!, `aSentinel`, { + enumerable: true, + value: sentinel, + }) + + expect(() => hash(ring[0])).toThrow( + `Value is too complex to hash safely: structural depth`, + ) + expect(() => hash(ring[0])).toThrow( + `Value is too complex to hash safely: structural depth`, + ) + expect(reads).toBe(2) + + const createChain = (size: number) => { + const root: { next?: unknown } = {} + let tail = root + for (let index = 0; index < size; index++) { + const next: { next?: unknown } = {} + tail.next = next + tail = next + } + return root + } + const accepted = createChain(600) + expect(hash(structuredClone(accepted))).toBe(hash(accepted)) + + const root = createChain(800) + expect(() => hash(root)).toThrow( + `Value is too complex to hash safely: structural depth`, + ) + }) + + it(`rejects dense ancestor back-references without warming siblings`, () => { + const createGraph = (size: number) => { + const nodes: Array> = [] + for (let index = 0; index < size; index++) { + const node: Record = { index } + if (index > 0) nodes[index - 1]!.next = node + for (let ancestor = 0; ancestor < index; ancestor++) { + node[`ancestor${ancestor}`] = nodes[ancestor] + } + nodes.push(node) + } + return nodes[0]! + } + const accepted = createGraph(50) + expect(() => hash(structuredClone(accepted))).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(accepted)).toThrow( + `Cannot hash cyclic structural values`, + ) + + let reads = 0 + const sentinel = Object.defineProperty({}, `value`, { + enumerable: true, + get: () => ++reads, + }) + const rejected = createGraph(450) + Object.defineProperty(rejected, `aSentinel`, { + enumerable: true, + value: sentinel, + }) + + expect(() => hash(rejected)).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(() => hash(rejected)).toThrow( + `Cannot hash cyclic structural values`, + ) + expect(reads).toBe(2) + }) + it(`should hash arrays`, () => { const arr1 = [1, 2, 3] const arr2 = [1, 2, 3] diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 14358dd3e9..dc2084d39b 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -21,6 +21,7 @@ import type { CollectionIndexMetadata, DeleteMutationFnParams, InsertMutationFnParams, + LoadSubsetFn, LoadSubsetOptions, PendingMutation, SyncAppliedReceipt, @@ -1015,7 +1016,7 @@ class PersistedCollectionRuntime< async loadSubset( options: LoadSubsetOptions, - upstreamLoadSubset?: (options: LoadSubsetOptions) => true | Promise, + upstreamLoadSubset?: LoadSubsetFn, ): Promise { this.activeSubsets.set(this.getSubsetKey(options), options) @@ -1029,17 +1030,18 @@ class PersistedCollectionRuntime< if (upstreamLoadSubset) { try { - const maybePromise = upstreamLoadSubset(options) - if (maybePromise instanceof Promise) { - await maybePromise.catch((error) => { - console.warn( - `Failed to load remote subset in persisted wrapper:`, - error, - ) - this.queueRemoteSubsetEnsure(options) - }) - } + await upstreamLoadSubset(options) } catch (error) { + if ( + options.signal?.aborted || + (typeof error === `object` && + error !== null && + `name` in error && + error.name === `AbortError`) + ) { + this.pendingRemoteSubsetEnsures.delete(this.getSubsetKey(options)) + throw error + } console.warn(`Failed to trigger remote subset load:`, error) this.queueRemoteSubsetEnsure(options) } @@ -1051,6 +1053,7 @@ class PersistedCollectionRuntime< upstreamUnloadSubset?: (options: LoadSubsetOptions) => void, ): void { this.activeSubsets.delete(this.getSubsetKey(options)) + this.pendingRemoteSubsetEnsures.delete(this.getSubsetKey(options)) upstreamUnloadSubset?.(options) } @@ -1824,7 +1827,8 @@ class PersistedCollectionRuntime< private queueRemoteSubsetEnsure(options: LoadSubsetOptions): void { if ( this.mode !== `sync-present` || - !this.persistence.coordinator.requestEnsureRemoteSubset + !this.persistence.coordinator.requestEnsureRemoteSubset || + this.activeSubsets.get(this.getSubsetKey(options)) !== options ) { return } @@ -2276,24 +2280,7 @@ function createWrappedSyncConfig< const getOpenTransaction = () => transactionStack[transactionStack.length - 1] let fullStartPromise: Promise | null = null - const cancelledLoadKeys = new Set() - const loadSubscriptionIds = new WeakMap() - let nextLoadSubscriptionId = 0 - const getLoadKey = (options: LoadSubsetOptions) => { - const subscription = options.subscription as object | undefined - if (subscription && typeof subscription === `object`) { - const existingId = loadSubscriptionIds.get(subscription) - if (existingId) { - return `sub:${existingId}` - } - nextLoadSubscriptionId++ - const nextId = String(nextLoadSubscriptionId) - loadSubscriptionIds.set(subscription, nextId) - return `sub:${nextId}` - } - - return `opts:${stableSerialize(normalizeSubsetOptionsForKey(options))}` - } + const acquisitions = new Map() runtime.setSyncControls({ begin: params.begin, write: params.write as SyncControlFns[`write`], @@ -2590,23 +2577,48 @@ function createWrappedSyncConfig< return { cleanup: () => { startupState.cleanedUp = true + acquisitions.clear() sourceResult.cleanup?.() runtime.cleanup() runtime.clearSyncControls() }, loadSubset: async (options: LoadSubsetOptions) => { - const loadKey = getLoadKey(options) - cancelledLoadKeys.delete(loadKey) + const acquisition = { forwarded: false } + acquisitions.set(options, acquisition) await fullStartPromise const resolvedSourceResult = await sourceResultPromise - if (startupState.cleanedUp || cancelledLoadKeys.has(loadKey)) { + if ( + startupState.cleanedUp || + acquisitions.get(options) !== acquisition + ) { return } - await runtime.loadSubset(options, resolvedSourceResult.loadSubset) + return runtime.loadSubset(options, (loadOptions) => { + // Hydration is another async boundary. A release before this + // point owns no upstream lease and must not start one later. + if ( + startupState.cleanedUp || + acquisitions.get(options) !== acquisition + ) { + return true + } + if (!resolvedSourceResult.loadSubset) return true + acquisition.forwarded = true + try { + return resolvedSourceResult.loadSubset(loadOptions) + } catch (error) { + acquisition.forwarded = false + throw error + } + }) }, unloadSubset: (options: LoadSubsetOptions) => { - cancelledLoadKeys.add(getLoadKey(options)) - runtime.unloadSubset(options, sourceResult.unloadSubset) + const acquisition = acquisitions.get(options) + acquisitions.delete(options) + runtime.unloadSubset( + options, + acquisition?.forwarded ? sourceResult.unloadSubset : undefined, + ) }, } }, diff --git a/packages/db-sqlite-persistence-core/tests/persisted.test.ts b/packages/db-sqlite-persistence-core/tests/persisted.test.ts index 606f0d75e7..89f9fef44b 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' import { BasicIndex, DbClient, @@ -1744,6 +1744,137 @@ describe(`persistedCollectionOptions`, () => { expect(collection.get(`2`)).toBeUndefined() }) + it(`does not release or acquire an upstream lease cancelled during hydration`, async () => { + const adapter = createRecordingAdapter() + const hydrate = adapter.loadSubset + let blocked = false + let enterHydration!: () => void + let finishHydration!: () => void + const entered = new Promise((resolve) => { + enterHydration = resolve + }) + const gate = new Promise((resolve) => { + finishHydration = resolve + }) + adapter.loadSubset = async (...args) => { + if (blocked) { + enterHydration() + await gate + } + return hydrate(...args) + } + let leases = 0 + let loads = 0 + const collection = createCollection( + persistedCollectionOptions({ + id: `cancelled-hydration-lease`, + syncMode: `on-demand`, + getKey: (row) => row.id, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + leases++ + return true + }, + unloadSubset: () => { + leases-- + }, + } + }, + }, + persistence: { adapter }, + }), + ) + collection.startSyncImmediate() + const first: LoadSubsetOptions = { limit: 1 } + const second: LoadSubsetOptions = { limit: 1 } + try { + await collection._sync.loadSubset(first) + expect(leases).toBe(1) + blocked = true + const pending = collection._sync.loadSubset(second) + await entered + collection._sync.unloadSubset(second) + expect(leases).toBe(1) + finishHydration() + await pending + expect(loads).toBe(1) + collection._sync.unloadSubset(first) + expect(leases).toBe(0) + } finally { + finishHydration() + await collection.cleanup() + } + }) + + it.each([`abort`, `release`, `offline`] as const)( + `handles remote ensure after %s without resurrecting cancelled demand`, + async (action) => { + vi.useFakeTimers() + const warning = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const failure = Object.assign(new Error(action), { + name: action === `abort` ? `AbortError` : `Error`, + }) + const ensure = vi.fn(async () => { + throw new Error(`offline`) + }) + const coordinator: PersistedCollectionCoordinator = { + getNodeId: () => `cancel-ensure`, + subscribe: () => () => {}, + publish: () => {}, + isLeader: () => true, + ensureLeadership: async () => {}, + requestEnsurePersistedIndex: async () => {}, + requestEnsureRemoteSubset: ensure, + } + const collection = createCollection( + persistedCollectionOptions({ + id: `cancel-ensure-${action}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: async () => { + throw failure + }, + } + }, + }, + persistence: { adapter: createRecordingAdapter(), coordinator }, + }), + ) + const options = { limit: 1 } + try { + collection.startSyncImmediate() + const result = await Promise.resolve( + collection._sync.loadSubset(options), + ).then( + () => `ready`, + (error: unknown) => error, + ) + if (action === `release`) collection._sync.unloadSubset(options) + const callsBeforeRetry = ensure.mock.calls.length + await vi.advanceTimersByTimeAsync(200) + if (action === `offline`) { + expect(result).toBe(`ready`) + expect(ensure.mock.calls.length).toBeGreaterThan(callsBeforeRetry) + } else { + if (action === `abort`) expect(result).toBe(failure) + expect(ensure).toHaveBeenCalledTimes(callsBeforeRetry) + } + } finally { + await collection.cleanup() + warning.mockRestore() + vi.useRealTimers() + } + }, + ) + it(`retries queued remote subset ensure after transient failures`, async () => { const adapter = createRecordingAdapter() let ensureCalls = 0 diff --git a/packages/db/package.json b/packages/db/package.json index 6ebfd58d8b..452759b5a6 100644 --- a/packages/db/package.json +++ b/packages/db/package.json @@ -21,7 +21,8 @@ "dev": "vite build --watch", "lint": "eslint . --fix", "test": "vitest --run", - "test:oracles": "vitest --run tests/query/includes-oracle.property.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/includes-context-transport-oracle.test.ts" + "test:facade-retention": "node --expose-gc --import tsx tests/facade-retention.probe.ts", + "test:oracles": "vitest --run tests/collection-cleanup-restart-oracle.test.ts tests/effect-disposal-oracle.test.ts tests/collection-metadata-publication-oracle.property.test.ts tests/collection-state-retention-oracle.property.test.ts tests/collection-subscription-lifecycle-history.property.test.ts tests/collection-subscription-lifecycle-oracle.test.ts tests/collection-subscription-lifecycle-publication.property.test.ts tests/collection-subscription-replay-oracle.property.test.ts tests/d2-source-reconciliation-oracle.property.test.ts tests/query/includes-collection-oracle.property.test.ts tests/query/includes-functional-projection-oracle.test.ts tests/query/includes-functional-input-boundary.test.ts tests/query/includes-context-transport-oracle.test.ts tests/query/includes-cross-formulation-oracle.property.test.ts tests/query/includes-optimistic-oracle.property.test.ts tests/query/includes-oracle.property.test.ts tests/query/includes-publication-oracle.test.ts tests/query/includes-query-shape-oracle.test.ts tests/query/includes-temporal-oracle.test.ts tests/query/includes-work-counter-oracle.test.ts tests/query/load-subset-oracle.property.test.ts tests/query/load-subset-replay-refinement-oracle.test.ts tests/query/load-subset-source-readiness-refinement-oracle.test.ts tests/query/load-subset-transaction-refinement-oracle.test.ts tests/query/ordered-source-loader-state.test.ts tests/query/ordered-lifecycle-oracle.property.test.ts tests/query/ordered-work-oracle.property.test.ts tests/query/pagination-oracle.property.test.ts" }, "type": "module", "main": "dist/cjs/index.cjs", diff --git a/packages/db/src/collection/change-events.ts b/packages/db/src/collection/change-events.ts index e70e44903b..eca99275d9 100644 --- a/packages/db/src/collection/change-events.ts +++ b/packages/db/src/collection/change-events.ts @@ -1,7 +1,3 @@ -import { - createSingleRowRefProxy, - toExpression, -} from '../query/builder/ref-proxy' import { compileSingleRowExpression, toBooleanPredicate, @@ -20,7 +16,6 @@ import type { SubscribeChangesOptions, } from '../types' import type { CollectionImpl } from './index.js' -import type { SingleRowRefProxy } from '../query/builder/ref-proxy' import type { BasicExpression, OrderBy } from '../query/ir.js' import type { WithVirtualProps } from '../virtual-props.js' @@ -181,44 +176,6 @@ export function currentStateAsChanges< } } -/** - * Creates a filter function from a where callback - * @param whereCallback - The callback function that defines the filter condition - * @returns A function that takes an item and returns true if it matches the filter - */ -export function createFilterFunction( - whereCallback: (row: SingleRowRefProxy) => any, -): (item: T) => boolean { - return (item: T): boolean => { - try { - // First try the RefProxy approach for query builder functions - const singleRowRefProxy = createSingleRowRefProxy() - const whereExpression = whereCallback(singleRowRefProxy) - const expression = toExpression(whereExpression) - const evaluator = compileSingleRowExpression(expression) - const result = evaluator(item as Record) - // WHERE clauses should always evaluate to boolean predicates (Kevin's feedback) - return toBooleanPredicate(result) - } catch { - // If RefProxy approach fails (e.g., arithmetic operations), fall back to direct evaluation - try { - // Create a simple proxy that returns actual values for arithmetic operations - const simpleProxy = new Proxy(item as any, { - get(target, prop) { - return target[prop] - }, - }) as SingleRowRefProxy - - const result = whereCallback(simpleProxy) - return toBooleanPredicate(result) - } catch { - // If both approaches fail, exclude the item - return false - } - } - } -} - /** * Creates a filter function from a pre-compiled expression * @param expression - The pre-compiled expression to evaluate diff --git a/packages/db/src/collection/changes.ts b/packages/db/src/collection/changes.ts index 00523a2f48..4715f97f8a 100644 --- a/packages/db/src/collection/changes.ts +++ b/packages/db/src/collection/changes.ts @@ -1,5 +1,6 @@ import { NegativeActiveSubscribersError } from '../errors' -import { withPublicationContext } from '../scheduler.js' +import { recordPublicationError, withPublicationContext } from '../scheduler.js' +import { runAllCallbacks } from '../utils/callbacks.js' import { createSingleRowRefProxy, toExpression, @@ -37,6 +38,8 @@ export class CollectionChangesManager< public shouldBatchEvents = false private publicationDeferralDepth = 0 private discardDeferredPublications = false + private deferredStateRevision = 0 + private deferredLayoutRevision = 0 private deferredPublications: Array<{ changes: Array> layoutChanged: boolean @@ -83,8 +86,14 @@ export class CollectionChangesManager< */ public emitEmptyReadyEvent(): void { withPublicationContext(() => { - for (const subscription of this.changeSubscriptions) { - subscription.emitEvents([]) + try { + runAllCallbacks( + [...this.changeSubscriptions].map( + (subscription) => () => subscription.emitEvents([]), + ), + ) + } catch (error) { + recordPublicationError(error) } }) } @@ -127,7 +136,21 @@ export class CollectionChangesManager< // buffered optimistic events with the final changes so subscribers see the // whole picture, even if the sync diff is empty. if (this.batchedEvents.length > 0) { - rawEvents = [...this.batchedEvents, ...changes] + const combined = new Map( + this.batchedEvents.map((change) => [change.key, change]), + ) + for (const change of changes) { + const pending = combined.get(change.key) + // A buffered removal was never delivered. Re-insertion replaces the + // subscriber's old row rather than inserting an already-sent key. + combined.set( + change.key, + pending?.type === `delete` && change.type === `insert` + ? { ...change, type: `update`, previousValue: pending.value } + : change, + ) + } + rawEvents = [...combined.values()] } this.batchedEvents = [] this.shouldBatchEvents = false @@ -147,6 +170,10 @@ export class CollectionChangesManager< * normal transaction boundaries. */ public deferPublication(): PublicationDeferral { + if (this.publicationDeferralDepth === 0) { + this.deferredStateRevision = this.stateRevision + this.deferredLayoutRevision = this.layoutRevision + } this.publicationDeferralDepth++ let closed = false @@ -163,6 +190,8 @@ export class CollectionChangesManager< this.deferredPublications = [] if (this.discardDeferredPublications) { this.discardDeferredPublications = false + this.stateRevision = this.deferredStateRevision + this.layoutRevision = this.deferredLayoutRevision return } this.publishEvents( @@ -193,16 +222,19 @@ export class CollectionChangesManager< // Every subscriber sees one committed source batch before dependent query // graphs run. This keeps repeated aliases and sibling subqueries coherent. + const layoutListeners = [...this.layoutChangeListeners] + const subscriptions = [...this.changeSubscriptions] withPublicationContext(() => { - // Notify both internal layout consumers and the public subscription API. - // Public subscribers historically receive an empty batch for order-only - // moves because there is no row-value ChangeMessage to publish. + const callbacks: Array<() => void> = subscriptions.map( + (subscription) => () => subscription.emitEvents(enrichedEvents), + ) if (rawEvents.length === 0) { - for (const listener of this.layoutChangeListeners) listener() + callbacks.unshift(...layoutListeners) } - - for (const subscription of this.changeSubscriptions) { - subscription.emitEvents(enrichedEvents) + try { + runAllCallbacks(callbacks) + } catch (error) { + recordPublicationError(error) } }) } @@ -242,11 +274,13 @@ export class CollectionChangesManager< this.addSubscriber() let subscription: CollectionSubscription | undefined + const setupState = { closed: false } try { subscription = new CollectionSubscription(this.collection, callback, { ...opts, whereExpression, onUnsubscribe: () => { + setupState.closed = true this.removeSubscriber() if (subscription) this.changeSubscriptions.delete(subscription) }, @@ -273,7 +307,7 @@ export class CollectionChangesManager< } // Add to batched listeners - this.changeSubscriptions.add(subscription) + if (!setupState.closed) this.changeSubscriptions.add(subscription) } catch (error) { if (subscription) { try { diff --git a/packages/db/src/collection/cleanup-queue.ts b/packages/db/src/collection/cleanup-queue.ts index 1acf7751ae..db8a212b22 100644 --- a/packages/db/src/collection/cleanup-queue.ts +++ b/packages/db/src/collection/cleanup-queue.ts @@ -90,16 +90,4 @@ export class CleanupQueue { this.updateTimeout() } } - - /** - * Resets the singleton instance for tests. - */ - public static resetInstance(): void { - if (CleanupQueue.instance) { - if (CleanupQueue.instance.timeoutId !== null) { - clearTimeout(CleanupQueue.instance.timeoutId) - } - CleanupQueue.instance = null - } - } } diff --git a/packages/db/src/collection/events.ts b/packages/db/src/collection/events.ts index 8058e70b21..1846535737 100644 --- a/packages/db/src/collection/events.ts +++ b/packages/db/src/collection/events.ts @@ -107,15 +107,6 @@ export type AllCollectionEvents = { [K in CollectionStatus as `status:${K}`]: CollectionStatusEvent } -export type CollectionEvent = - | AllCollectionEvents[keyof AllCollectionEvents] - | CollectionStatusChangeEvent - | CollectionSubscribersChangeEvent - | CollectionLoadingSubsetChangeEvent - | CollectionTruncateEvent - | CollectionIndexAddedEvent - | CollectionIndexRemovedEvent - export type CollectionEventHandler = ( event: AllCollectionEvents[T], ) => void @@ -145,22 +136,32 @@ export class CollectionEventsManager extends EventEmitter { emitStatusChange( status: T, previousStatus: CollectionStatus, + isCurrent: () => boolean, ) { - this.emit(`status:change`, { - type: `status:change`, - collection: this.collection, - previousStatus, - status, - }) + this.emitInnerWhile( + `status:change`, + { + type: `status:change`, + collection: this.collection, + previousStatus, + status, + }, + isCurrent, + ) + if (!isCurrent()) return // Emit specific status event using type assertion const eventKey: `status:${T}` = `status:${status}` - this.emit(eventKey, { - type: eventKey, - collection: this.collection, - previousStatus, - status, - } as AllCollectionEvents[`status:${T}`]) + this.emitInnerWhile( + eventKey, + { + type: eventKey, + collection: this.collection, + previousStatus, + status, + } as AllCollectionEvents[`status:${T}`], + isCurrent, + ) } emitSubscribersChange( diff --git a/packages/db/src/collection/index.ts b/packages/db/src/collection/index.ts index 7b61a13503..0197182f27 100644 --- a/packages/db/src/collection/index.ts +++ b/packages/db/src/collection/index.ts @@ -1,3 +1,4 @@ +import { registerOpaqueHash } from '@tanstack/db-ivm' import { safeRandomUUID } from '../utils/uuid' import { CollectionConfigurationError, @@ -350,6 +351,9 @@ export class CollectionImpl< ) } + // Collections are mutable handles, not structural rows. Downstream queries + // must not hash their internal state or follow its ownership cycles. + registerOpaqueHash(this) this._changes = new CollectionChangesManager() this._events = new CollectionEventsManager() this._indexes = new CollectionIndexesManager() @@ -457,6 +461,11 @@ export class CollectionImpl< /** * Register a callback to be executed when the collection first becomes ready * Useful for preloading collections + * Every callback queued before the transition runs. Because ready state is + * established first, callbacks registered during or after delivery run + * immediately. If one throws, the collection remains ready. Direct sync + * startup rethrows the first failure; preload resolves from ready state. + * Cleanup discards pending callbacks without invoking them. * @param callback Function to call when the collection first becomes ready * @example * collection.onFirstReady(() => { @@ -495,6 +504,7 @@ export class CollectionImpl< /** * Start sync immediately - internal method for compiled queries * This bypasses lazy loading for special cases like live query results + * Throws during active cleanup; restart after cleanup completes instead. */ public startSyncImmediate(): void { this._sync.startSync() @@ -1039,6 +1049,8 @@ export class CollectionImpl< /** * Clean up the collection by stopping sync and clearing data * This can be called manually or automatically by garbage collection + * Cleanup callbacks must not restart this collection or call its preload(). + * Wait until cleanup completes before starting a new sync session. */ public async cleanup(): Promise { this._lifecycle.cleanup() diff --git a/packages/db/src/collection/lifecycle.ts b/packages/db/src/collection/lifecycle.ts index 661e8410d0..7afe258974 100644 --- a/packages/db/src/collection/lifecycle.ts +++ b/packages/db/src/collection/lifecycle.ts @@ -7,6 +7,7 @@ import { safeCancelIdleCallback, safeRequestIdleCallback, } from '../utils/browser-polyfills' +import { runAllCallbacks } from '../utils/callbacks' import { CleanupQueue } from './cleanup-queue' import type { IdleCallbackDeadline } from '../utils/browser-polyfills' import type { StandardSchemaV1 } from '@standard-schema/spec' @@ -37,6 +38,8 @@ export class CollectionLifecycleManager< public onFirstReadyCallbacks: Array<() => void> = [] private idleCallbackId: number | null = null private syncError: unknown + private statusRevision = 0 + private cleaningUp = false /** * Creates a new CollectionLifecycleManager instance @@ -104,11 +107,16 @@ export class CollectionLifecycleManager< ) } this.validateStatusTransition(this.status, newStatus) + const revision = ++this.statusRevision const previousStatus = this.status this.status = newStatus // Emit event - this.events.emitStatusChange(newStatus, previousStatus) + this.events.emitStatusChange( + newStatus, + previousStatus, + () => this.statusRevision === revision, + ) } /** @@ -133,12 +141,34 @@ export class CollectionLifecycleManager< * @private - Should only be called by sync implementations */ public markReady(): void { + const failure = this.applyReadyTransition() + if (failure) throw failure.error + } + + /** @internal Capture ready-effect failures while the sync entry completes. */ + public markReadyDuringSyncStart(): { error: unknown } | undefined { + return this.applyReadyTransition() + } + + private applyReadyTransition(): { error: unknown } | undefined { this.validateStatusTransition(this.status, `ready`) // A successful initial sync or recovery establishes a ready snapshot. if (this.status === `loading` || this.status === `error`) { this.syncError = undefined + const readyRevision = this.statusRevision + 1 this.setStatus(`ready`, true) + // A status listener can synchronously supersede this transition, even + // when it restarts the Collection back to ready before returning. + if ( + (this.status as CollectionStatus) !== `ready` || + this.statusRevision !== readyRevision + ) { + return undefined + } + + const readyEffects: Array<() => void> = [] + // Call any registered first ready callbacks (only on first time becoming ready) if (!this.hasBeenReady) { this.hasBeenReady = true @@ -148,16 +178,19 @@ export class CollectionLifecycleManager< this.hasReceivedFirstCommit = true } - const callbacks = [...this.onFirstReadyCallbacks] + readyEffects.push(...this.onFirstReadyCallbacks) this.onFirstReadyCallbacks = [] - callbacks.forEach((callback) => callback()) } // Notify dependents when markReady is called, after status is set // This ensures live queries get notified when their dependencies become ready - if (this.changes.changeSubscriptions.size > 0) { - this.changes.emitEmptyReadyEvent() + readyEffects.push(() => this.changes.emitEmptyReadyEvent()) + try { + runAllCallbacks(readyEffects) + } catch (error) { + return { error } } } + return undefined } /** Mark an asynchronous sync failure after sync has started. */ @@ -172,6 +205,14 @@ export class CollectionLifecycleManager< return this.syncError } + public assertCanStartSync(): void { + if (this.cleaningUp) { + throw new CollectionStateError( + `Cannot start collection "${this.id}" during cleanup. Restart after cleanup() completes.`, + ) + } + } + /** * Start the garbage collection timer * Called when the collection becomes inactive (no subscribers) @@ -242,37 +283,33 @@ export class CollectionLifecycleManager< * @returns true if cleanup was completed, false if it was rescheduled */ private performCleanup(deadline?: IdleCallbackDeadline): boolean { + // Nested cleanup belongs to this retirement, not a new lifecycle turn. + if (this.cleaningUp) return true // If we have a deadline, we can potentially split cleanup into chunks // For now, we'll do all cleanup at once but check if we have time const hasTime = !deadline || deadline.timeRemaining() > 0 || deadline.didTimeout if (hasTime) { - // Perform all cleanup operations except events - this.sync.cleanup() - this.state.cleanup() - this.changes.cleanup() - this.indexes.cleanup() + this.cleaningUp = true + try { + // Perform all cleanup operations except events + this.sync.cleanup() + this.state.cleanup() + this.changes.cleanup() + this.indexes.cleanup() - CleanupQueue.getInstance().cancel(this) + CleanupQueue.getInstance().cancel(this) - this.hasBeenReady = false - this.syncError = undefined + this.hasBeenReady = false + this.syncError = undefined - // Call any pending onFirstReady callbacks before clearing them. - // This ensures preload() promises resolve during cleanup instead of hanging. - const callbacks = [...this.onFirstReadyCallbacks] - this.onFirstReadyCallbacks = [] - callbacks.forEach((callback) => { - try { - callback() - } catch (error) { - console.error( - `${this.config.id ? `[${this.config.id}] ` : ``}Error in onFirstReady callback during cleanup:`, - error, - ) - } - }) + // Cleanup is not readiness. Sync cleanup rejects pending preload callers; + // first-ready listeners belong to the discarded run. + this.onFirstReadyCallbacks = [] + } finally { + this.cleaningUp = false + } // Set status to cleaned-up after everything is cleaned up // This fires the status:change event to notify listeners diff --git a/packages/db/src/collection/state.ts b/packages/db/src/collection/state.ts index 3783422013..f0702e5aab 100644 --- a/packages/db/src/collection/state.ts +++ b/packages/db/src/collection/state.ts @@ -127,11 +127,13 @@ export class CollectionStateManager< public size = 0 // State used for computing the change events - public syncedKeys = new Set() public preSyncVisibleState = new Map() + public preSyncVirtualState = new Map>() public recentlySyncedKeys = new Set() public hasReceivedFirstCommit = false public isCommittingSyncTransactions = false + private isDrainingSyncTransactions = false + private syncSessionGeneration = 0 public isLocalOnly = false /** @@ -829,6 +831,30 @@ export class CollectionStateManager< * This method processes operations from pending transactions and applies them to the synced data */ commitPendingTransactions = () => { + if (this.isDrainingSyncTransactions) return + this.isDrainingSyncTransactions = true + let failed = false + let firstError: unknown + try { + let result: { processed: boolean; failure?: { error: unknown } } + do { + result = this.commitNextPendingTransactionBatch() + if (result.failure && !failed) { + failed = true + firstError = result.failure.error + } + } while (result.processed) + } finally { + this.isDrainingSyncTransactions = false + } + if (failed) throw firstError + } + + private commitNextPendingTransactionBatch(): { + processed: boolean + failure?: { error: unknown } + } { + const syncSessionGeneration = this.syncSessionGeneration // Check if there are any persisting transaction let hasPersistingTransaction = false for (const transaction of this.transactions.values()) { @@ -875,6 +901,10 @@ export class CollectionStateManager< }, ) + if (committedSyncedTransactions.length === 0) { + return { processed: false } + } + // Process committed transactions if: // 1. No persisting user transaction (normal sync flow), OR // 2. There's a truncate operation (must be processed immediately), OR @@ -886,6 +916,9 @@ export class CollectionStateManager< // non-immediate transactions would be applied later and could overwrite newer state. // Processing all committed transactions together preserves causal ordering. if (!hasPersistingTransaction || hasTruncateSync || hasImmediateSync) { + const previousLayout = layoutChanged ? [...this.keys()] : undefined + this.pendingSyncedTransactions = uncommittedSyncedTransactions + // Application is now the point of no return. Event listeners run before // the receipts resolve, so a signal aborted from one of those listeners // must not cancel writes that are already becoming visible. @@ -926,6 +959,12 @@ export class CollectionStateManager< this.snapshotRowOriginsForKeys(virtualSnapshotKeys) const previousOptimisticUpserts = new Map(this.optimisticUpserts) const previousOptimisticDeletes = new Set(this.optimisticDeletes) + const completedDirectUpserts = new Set( + this.pendingOptimisticDirectUpserts, + ) + const completedDirectDeletes = new Set( + this.pendingOptimisticDirectDeletes, + ) // Use pre-captured state if available (from optimistic scenarios), // otherwise capture current state (for pure sync scenarios) @@ -993,7 +1032,6 @@ export class CollectionStateManager< truncatePendingLocalOrigins = new Set(this.pendingLocalOrigins) this.syncedData.clear() this.syncedMetadata.clear() - this.syncedKeys.clear() this.hydrationSeedKeys.clear() this.hydratedKeys.clear() this.clearOriginTrackingState() @@ -1014,15 +1052,18 @@ export class CollectionStateManager< for (const operation of transaction.operations) { const key = operation.key as TKey - this.syncedKeys.add(key) // Determine origin: 'local' for local-only collections or pending local changes + const retainedLocalOrigin = + (truncatePendingLocalChanges?.has(key) === true || + truncatePendingLocalOrigins?.has(key) === true) && + !completedDirectUpserts.has(key) && + !completedDirectDeletes.has(key) const origin: VirtualOrigin = this.isLocalOnly || this.pendingLocalChanges.has(key) || this.pendingLocalOrigins.has(key) || - truncatePendingLocalChanges?.has(key) === true || - truncatePendingLocalOrigins?.has(key) === true + retainedLocalOrigin ? 'local' : 'remote' @@ -1122,6 +1163,15 @@ export class CollectionStateManager< const reapplyDeletes = new Set( truncateOptimisticSnapshot!.deletes, ) + // A same-key authoritative row confirms a completed direct mutation. + // Keep active optimistic work, but do not restore a completed client + // value over the row that just replaced it. + for (const key of completedDirectUpserts) { + if (changedKeys.has(key)) reapplyUpserts.delete(key) + } + for (const key of completedDirectDeletes) { + if (changedKeys.has(key)) reapplyDeletes.delete(key) + } // Emit inserts for re-applied upserts, skipping any keys that have an optimistic delete. // If the server also inserted/updated the same key in this batch, override that value @@ -1178,9 +1228,11 @@ export class CollectionStateManager< // This includes items from transactions that may have completed during processing if (hasTruncateSync && truncateOptimisticSnapshot) { for (const [key, value] of truncateOptimisticSnapshot.upserts) { + if (completedDirectUpserts.has(key) && changedKeys.has(key)) continue this.optimisticUpserts.set(key, value) } for (const key of truncateOptimisticSnapshot.deletes) { + if (completedDirectDeletes.has(key) && changedKeys.has(key)) continue this.optimisticDeletes.add(key) } } @@ -1244,12 +1296,14 @@ export class CollectionStateManager< for (const key of changedKeys) { const previousVisibleValue = currentVisibleState.get(key) const newVisibleValue = this.get(key) // This returns the new derived state - const previousVirtualProps = this.getVirtualPropsSnapshotForState(key, { - rowOrigins: previousRowOrigins, - optimisticUpserts: previousOptimisticUpserts, - optimisticDeletes: previousOptimisticDeletes, - completedOptimisticKeys: completedOptimisticOps, - }) + const previousVirtualProps = + this.preSyncVirtualState.get(key) ?? + this.getVirtualPropsSnapshotForState(key, { + rowOrigins: previousRowOrigins, + optimisticUpserts: previousOptimisticUpserts, + optimisticDeletes: previousOptimisticDeletes, + completedOptimisticKeys: completedOptimisticOps, + }) const nextVirtualProps = this.getVirtualPropsSnapshotForState(key) const virtualChanged = previousVirtualProps.$synced !== nextVirtualProps.$synced || @@ -1356,27 +1410,38 @@ export class CollectionStateManager< } // End batching and emit all events (combines any batched events with sync events) - this.changes.emitEvents(events, true, layoutChanged) - - this.pendingSyncedTransactions = uncommittedSyncedTransactions - - // Clear the pre-sync state since sync operations are complete - this.preSyncVisibleState.clear() - - // Clear recently synced keys after a microtask to allow recomputeOptimisticState to see them - Promise.resolve().then(() => { - this.recentlySyncedKeys.clear() - }) + let failure: { error: unknown } | undefined + try { + const visibleLayoutChanged = + previousLayout !== undefined && + (previousLayout.length !== this.size || + [...this.keys()].some( + (key, index) => key !== previousLayout[index], + )) + this.changes.emitEvents(events, true, visibleLayoutChanged) + } catch (error) { + failure = { error } + } - // Mark that we've received the first commit (for tracking purposes) - if (!this.hasReceivedFirstCommit) { - this.hasReceivedFirstCommit = true + if (this.syncSessionGeneration === syncSessionGeneration) { + this.preSyncVisibleState.clear() + this.preSyncVirtualState.clear() + Promise.resolve().then(() => { + if (this.syncSessionGeneration === syncSessionGeneration) { + this.recentlySyncedKeys.clear() + } + }) + if (!this.hasReceivedFirstCommit) this.hasReceivedFirstCommit = true } for (const transaction of committedSyncedTransactions) { transaction.applied.resolve() } + + return { processed: true, failure } } + + return { processed: false } } /** Abandons one committed transaction before it becomes visible. */ @@ -1402,11 +1467,13 @@ export class CollectionStateManager< if (!remainingPendingKeys.has(key)) { this.recentlySyncedKeys.delete(key) this.preSyncVisibleState.delete(key) + this.preSyncVirtualState.delete(key) } } if (this.pendingSyncedTransactions.length === 0) { this.preSyncVisibleState.clear() + this.preSyncVirtualState.clear() this.recentlySyncedKeys.clear() this.changes.emitEvents([], true) } else { @@ -1467,6 +1534,10 @@ export class CollectionStateManager< const currentValue = this.get(key) if (currentValue !== undefined) { this.preSyncVisibleState.set(key, currentValue) + this.preSyncVirtualState.set( + key, + this.getVirtualPropsSnapshotForState(key), + ) } } } @@ -1492,6 +1563,7 @@ export class CollectionStateManager< * This can be called manually or automatically by garbage collection */ public cleanup(): void { + this.syncSessionGeneration++ for (const transaction of this.pendingSyncedTransactions) { transaction.applied.reject(new SyncTransactionAbortedError()) } @@ -1510,7 +1582,9 @@ export class CollectionStateManager< this.isLocalOnly = false this.size = 0 this.pendingSyncedTransactions = [] - this.syncedKeys.clear() + this.preSyncVisibleState.clear() + this.preSyncVirtualState.clear() + this.recentlySyncedKeys.clear() this.hasReceivedFirstCommit = false } } diff --git a/packages/db/src/collection/subscription.ts b/packages/db/src/collection/subscription.ts index 281e206c7b..8d4e0489ad 100644 --- a/packages/db/src/collection/subscription.ts +++ b/packages/db/src/collection/subscription.ts @@ -1,19 +1,24 @@ import { ensureIndexForExpression } from '../indexes/auto-index.js' -import { and, eq, gte, lt } from '../query/builder/functions.js' +import { and, eq } from '../query/builder/functions.js' import { PropRef, Value } from '../query/ir.js' import { EventEmitter } from '../event-emitter.js' import { compileExpression } from '../query/compiler/evaluators.js' -import { buildCursor } from '../utils/cursor.js' +import { buildCursor, buildCursorCurrent } from '../utils/cursor.js' import { deepEquals } from '../utils.js' +import { normalizeError } from '../utils/error.js' +import { runAllCallbacks } from '../utils/callbacks.js' +import { createDeferred } from '../deferred.js' +import { LoadSubsetOperationAbortedError } from '../errors.js' import { createFilterFunctionFromExpression, createFilteredCallback, } from './change-events.js' import type { BasicExpression, OrderBy } from '../query/ir.js' -import type { IndexInterface } from '../indexes/base-index.js' +import type { IndexReader } from '../indexes/base-index.js' import type { ChangeMessage, LoadSubsetOptions, + LoadSubsetRequestResult, Subscription, SubscriptionEvents, SubscriptionLoadSubsetErrorEvent, @@ -21,6 +26,7 @@ import type { SubscriptionUnsubscribedEvent, } from '../types.js' import type { CollectionImpl } from './index.js' +import type { Deferred } from '../deferred.js' type RequestSnapshotOptions = { where?: BasicExpression @@ -31,8 +37,8 @@ type RequestSnapshotOptions = { orderBy?: OrderBy /** Optional limit to pass to loadSubset for backend optimization */ limit?: number - /** Callback that receives the raw loadSubset result for external tracking */ - onLoadSubsetResult?: (result: Promise | true) => void + /** Callback that receives the normalized loadSubset result for internal tracking */ + onLoadSubsetResult?: SubsetResultObserver /** Called when the local snapshot must fall back from an index to a scan. */ onUnoptimized?: () => void } @@ -40,16 +46,24 @@ type RequestSnapshotOptions = { type RequestLimitedSnapshotOptions = { orderBy: OrderBy limit: number - /** All column values for cursor (first value used for local index, all values for sync layer) */ + /** A single cursor value; composite cursor inputs are rejected. */ minValues?: Array /** Row offset for offset-based pagination (passed to sync layer) */ offset?: number /** Whether to track the loadSubset promise on this subscription (default: true) */ trackLoadSubsetPromise?: boolean - /** Callback that receives the raw loadSubset result for external tracking */ - onLoadSubsetResult?: (result: Promise | true) => void + /** Callback that receives the normalized loadSubset result for internal tracking */ + onLoadSubsetResult?: SubsetResultObserver } +export type ReleaseLoadSubset = (primaryFailure?: { error: unknown }) => void + +type SubsetResultObserver = ( + result: LoadSubsetRequestResult, + options: LoadSubsetOptions, + release: ReleaseLoadSubset, +) => void + type CollectionSubscriptionOptions = { includeInitialState?: boolean /** Pre-compiled expression for filtering changes */ @@ -58,38 +72,62 @@ type CollectionSubscriptionOptions = { onUnsubscribe?: (event: SubscriptionUnsubscribedEvent) => void /** Callback for subset-load failures scoped to this subscription. */ onLoadSubsetError?: (event: SubscriptionLoadSubsetErrorEvent) => void + truncateReplayPublication?: TruncateReplayPublicationControl } +type TruncateReplayPublicationControl = Readonly<{ + start: () => void + succeed: () => void +}> + type TruncatePublicationState = { loadedInitialState: boolean snapshotSent: boolean - sentKeys: Set - publishedRows: Map limitedSnapshotRowCount: number lastSentKey: string | number | undefined } type SubsetAcquisition = { options: LoadSubsetOptions + loadSubsetSession: number abortController?: AbortController removeRequestAbortListener?: () => void + releaseAttempted?: true } -type SubsetDemand = SubsetAcquisition & { +type SubsetDemand = { requestOptions: LoadSubsetOptions + acquisition: SubsetAcquisition + acquisitionState: `starting` | `active` | `detached` + initialResult?: Deferred } type TruncateReplayAttempt = { - pending: Set<{ promise: Promise }> - failed: boolean + pendingCount: number setupComplete: boolean } type TruncateReplaySession = { + loadSubsetSession: number publicationState: TruncatePublicationState - buffer: Array>> - attempts: Set + /** Direct subscribers buffer the replacement here; delegated publication has no buffer. */ + privateRows: Map | undefined + pending: Set<{ demand: SubsetDemand; attempt: TruncateReplayAttempt }> + pendingSetups: number currentAttempt: TruncateReplayAttempt + failures: Map + completion: Deferred +} + +function createReplayCompletion(): Deferred { + const completion = createDeferred() + void completion.promise.catch(() => {}) + return completion +} + +function cancelAcquisition(acquisition: SubsetAcquisition): void { + acquisition.abortController?.abort() + acquisition.removeRequestAbortListener?.() } export class CollectionSubscription @@ -112,6 +150,7 @@ export class CollectionSubscription * We store the exact LoadSubsetOptions we passed to loadSubset to ensure symmetric unload. */ private subsetDemands: Array = [] + private primaryFailureDeliveryDepth = 0 private readonly requestedSubsetWhere = new WeakMap< LoadSubsetOptions, BasicExpression @@ -130,19 +169,31 @@ export class CollectionSubscription private filteredCallback: (changes: Array>) => boolean - private orderByIndex: IndexInterface | undefined + private orderByIndex: IndexReader | undefined // Status tracking private _status: SubscriptionStatus = `ready` + private statusRevision = 0 private _lastError: unknown | undefined - private pendingLoadSubsetPromises: Set> = new Set() + private pendingLoadSubsetParticipants = new Set<{ + demand: SubsetDemand + promise: Promise + }>() // Cleanup function for truncate event listener private truncateCleanup: (() => void) | undefined + private collectionCleanup: (() => void) | undefined + private collectionRestartCleanup: (() => void) | undefined // One replay session owns the publication baseline, overlapping attempts, // and buffered changes until every attempt settles. private truncateReplaySession: TruncateReplaySession | undefined + private readonly loadSubsetPromiseErrors = new WeakMap< + Promise, + Error + >() + private truncateReplacementPending = false + private unsubscribed = false public get status(): SubscriptionStatus { return this._status @@ -194,6 +245,96 @@ export class CollectionSubscription this.truncateCleanup = this.collection.on(`truncate`, () => { this.handleTruncate() }) + this.collectionCleanup = this.collection.on(`status:cleaned-up`, () => { + this.handleCollectionCleanup() + }) + this.collectionRestartCleanup = this.collection.on( + `status:change`, + ({ status }) => { + if (status !== `loading` && status !== `ready`) return + const loadSubsetSession = this.collection._sync.getLoadSubsetSession() + const replaySession = this.truncateReplaySession + if ( + this.subsetDemands.some( + (demand) => demand.acquisitionState === `detached`, + ) + ) { + this.setStatus(`loadingSubset`) + } + queueMicrotask(() => { + if (this.truncateReplaySession === replaySession) { + this.restartDetachedDemands(loadSubsetSession) + } + }) + }, + ) + } + + /** Detach logical demand from work owned by a discarded sync session. */ + private handleCollectionCleanup(): void { + this.discardTruncateReplay() + this.stalePublishedRows = new Map(this.publishedRows) + this.pendingLoadSubsetParticipants.clear() + + for (const demand of [...this.subsetDemands]) { + demand.initialResult?.reject(new LoadSubsetOperationAbortedError()) + cancelAcquisition(demand.acquisition) + if (demand.acquisitionState === `starting`) { + const index = this.subsetDemands.indexOf(demand) + if (index !== -1) this.subsetDemands.splice(index, 1) + } else { + demand.acquisitionState = `detached` + demand.acquisition = { + options: demand.requestOptions, + loadSubsetSession: demand.acquisition.loadSubsetSession, + } + } + } + this.setReadyIfIdle() + } + + /** Acquire detached demand after startup or initial-error recovery. */ + private restartDetachedDemands(loadSubsetSession: number): void { + if ( + this.unsubscribed || + !this.isLoadSubsetSessionCurrent(loadSubsetSession) + ) { + return + } + if ( + this.collection.status === `error` || + this.collection._sync.syncLoadSubsetFn === null + ) { + this.setReadyIfIdle() + return + } + const demands = this.subsetDemands.filter( + (demand) => + demand.acquisitionState === `detached` && + !demand.requestOptions.signal?.aborted, + ) + if (demands.length === 0) { + this.setReadyIfIdle() + return + } + + const session = this.createTruncateReplaySession(loadSubsetSession, () => { + const currentRows = this.collection.currentStateAsChanges({ + optimizedOnly: false, + }) + return new Map( + // The API returns void for unavailable snapshots, not just undefined. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + (currentRows ?? []) + .filter((change) => change.type !== `delete`) + .map((change) => [change.key, change.value]), + ) + }) + const attempt = session.currentAttempt + this.truncateReplaySession = session + this.setStatus(`loadingSubset`) + if (this.truncateReplaySession !== session) return + this.startTruncateReplayAttempt(session, attempt, demands) } /** @@ -202,244 +343,323 @@ export class CollectionSubscription * * To prevent a flash of missing content, we buffer all changes (deletes from truncate * and inserts from refetch) until all loadSubset calls succeed, then emit them together. - * A failed replay keeps the last published snapshot, resumes ordinary deltas, - * and retains subset ownership so a later truncate can retry the replay. + * A failed replay keeps the last published snapshot private until a later + * authoritative replay succeeds. */ private handleTruncate() { - const demandsToReload = [...this.subsetDemands] - - // Only buffer if there's an actual loadSubset handler that can do async work. - // Without a loadSubset handler, there's nothing to re-request and no reason to buffer. - // This prevents unnecessary buffering in eager sync mode or when loadSubset isn't implemented. + // Without a loader, replay only reconciles rows retained across cleanup. const hasLoadSubsetHandler = this.collection._sync.syncLoadSubsetFn !== null + const demandsToReload = hasLoadSubsetHandler ? [...this.subsetDemands] : [] - // If there are no subsets to reload OR no loadSubset handler, just reset state - if (demandsToReload.length === 0 || !hasLoadSubsetHandler) { - this.snapshotSent = false - this.loadedInitialState = false - this.limitedSnapshotRowCount = 0 - this.lastSentKey = undefined + // Retained rows still need the committed replacement even without demand. + if (demandsToReload.length === 0 && this.stalePublishedRows.size === 0) { + this.resetSnapshotTracking() return } - const attempt: TruncateReplayAttempt = { - pending: new Set(), - failed: false, - setupComplete: false, - } let session = this.truncateReplaySession - if (!session) { - session = { - publicationState: { - loadedInitialState: this.loadedInitialState, - snapshotSent: this.snapshotSent, - sentKeys: new Set(this.sentKeys), - publishedRows: new Map(this.publishedRows), - limitedSnapshotRowCount: this.limitedSnapshotRowCount, - lastSentKey: this.lastSentKey, - }, - buffer: [], - attempts: new Set(), - currentAttempt: attempt, + if (session) { + if (!session.completion.isPending()) { + session.completion = createReplayCompletion() } + // Setup itself holds publication: adapter/status callbacks may reenter + // before a request returns its promise and joins the pending set. + session.pendingSetups++ + session.failures.clear() + session.currentAttempt = { pendingCount: 0, setupComplete: false } + } else { + // Every overlapping attempt shares one publication baseline and buffer. + session = this.createTruncateReplaySession( + this.collection._sync.getLoadSubsetSession(), + () => new Map(this.publishedRows), + ) this.truncateReplaySession = session } - session.attempts.add(attempt) - session.currentAttempt = attempt + const attempt = session.currentAttempt + this.setStatus(`loadingSubset`) + + if (this.truncateReplaySession !== session) return + + if (this.options.truncateReplayPublication) { + this.truncateReplacementPending = true + this.options.truncateReplayPublication.start() + } // A newer replay replaces every prior acquisition for these demands. Abort // the old work before it can install rows into the new generation. for (const demand of demandsToReload) { - demand.abortController?.abort() + demand.acquisition.abortController?.abort() } - // Start buffering before the truncate commit publishes its deletes. Every - // overlapping attempt shares this one publication baseline and buffer. - // Retained rows from an earlier failed replay stay marked until this - // attempt either replaces them or proves they are absent. - - // Reset snapshot/pagination tracking state for the replacement snapshot. - this.snapshotSent = false - this.loadedInitialState = false - this.limitedSnapshotRowCount = 0 - this.lastSentKey = undefined + // Reset snapshot/pagination tracking for the replacement snapshot. Rows + // retained from an earlier failed replay stay marked until this attempt + // either replaces them or proves they are absent. + this.resetSnapshotTracking() // Defer the requests so the truncate commit's deletes enter the session // buffer before a synchronous adapter can publish replacement rows. queueMicrotask(() => { if (this.truncateReplaySession !== session) return + if (!this.isLoadSubsetSessionCurrent(session.loadSubsetSession)) { + this.retireStaleTruncateReplay(session) + return + } + // A newer truncate that arrived before this attempt began source work + // already captured the active demands. Starting them now would place the + // obsolete acquisition outside the newer abort sweep. + this.startTruncateReplayAttempt( + session, + attempt, + session.currentAttempt === attempt ? demandsToReload : [], + ) + }) + } - for (const demand of demandsToReload) { - if (!this.subsetDemands.includes(demand)) continue - - const isCurrentAttempt = () => - this.truncateReplaySession === session && - session.currentAttempt === attempt - const nextAcquisition = this.createSubsetAcquisition(demand) - let syncResult: Promise | true - try { - syncResult = this.loadSubset( - nextAcquisition.options, - isCurrentAttempt, - ) - } catch { - nextAcquisition.abortController.abort() - nextAcquisition.removeRequestAbortListener?.() - attempt.failed = true - continue - } + /** Make tentative replay ownership visible before adapter code can reenter. */ + private startTruncateReplayDemand( + session: TruncateReplaySession, + attempt: TruncateReplayAttempt, + demand: SubsetDemand, + ): void { + const isCurrentAttempt = () => + this.truncateReplaySession === session && + session.currentAttempt === attempt + const isCurrent = () => + isCurrentAttempt() && + this.isLoadSubsetSessionCurrent(session.loadSubsetSession) && + this.isDemandActive(demand) + const fail = (error: unknown) => { + if (isCurrent()) session.failures.set(demand, normalizeError(error)) + } + if (demand.initialResult) { + void session.completion.promise.then( + demand.initialResult.resolve, + demand.initialResult.reject, + ) + } - this.observeLoadSubsetResult( - syncResult, - nextAcquisition.options, - true, - () => isCurrentAttempt() && !nextAcquisition.options.signal?.aborted, - ) + // Sequential handoff: retire the old physical lease while retaining its + // logical demand. Callback reentry cannot release that lease twice. + const previous = demand.acquisition + const hadPreviousAcquisition = demand.acquisitionState === `active` + demand.acquisitionState = `detached` + if (hadPreviousAcquisition) { + try { + this.releaseAcquisition(previous) + } catch (error) { + fail(error) + return + } + } + if (!isCurrent() || demand.requestOptions.signal?.aborted) return - if (syncResult instanceof Promise) { - // A transport promise may be shared by several deduplicated logical - // demands. Track each demand separately so one settlement observer - // cannot complete the attempt before the others apply their result. - const pending = { promise: syncResult } - attempt.pending.add(pending) - void syncResult.then( - () => this.settleTruncateReplay(session, attempt, pending), - () => { - // A released demand no longer participates in the current - // replacement. Its cooperative AbortError must not discard the - // successful rows from demands that are still active. - if ( - this.subsetDemands.includes(demand) && - !nextAcquisition.options.signal?.aborted - ) { - attempt.failed = true - } - this.settleTruncateReplay(session, attempt, pending) - }, - ) - } + const next = this.createSubsetAcquisition(demand) + demand.acquisition = next + demand.acquisitionState = `starting` + let result: LoadSubsetRequestResult + try { + result = this.loadSubset(next.options, isCurrent) + } catch (error) { + if (demand.acquisition === next) demand.acquisitionState = `detached` + cancelAcquisition(next) + fail(error) + return + } - try { - this.replaceSubsetAcquisition(demand, nextAcquisition) - } catch (error) { - // The old lease is still owned because its release failed. Abort and - // release the new acquisition, but keep observing its work so rows - // from a non-cooperative adapter cannot escape the replay buffer. - nextAcquisition.abortController.abort() - nextAcquisition.removeRequestAbortListener?.() - try { - this.collection._sync.unloadSubset(nextAcquisition.options) - } catch { - // Preserve the first ownership error. The demand still retains the - // old acquisition so normal cleanup can retry that release. - } - this.recordLoadSubsetError(demand.options, error, true) - attempt.failed = true - } + if (!isCurrent()) { + if (demand.acquisition === next) demand.acquisitionState = `detached` + try { + this.releaseAcquisition(next) + } catch (error) { + fail(error) } + return + } - attempt.setupComplete = true - this.checkTruncateReplayComplete(session) - }) + demand.acquisitionState = `active` + this.trackTruncateReplayParticipant(session, attempt, demand, result) + this.observeLoadSubsetResult( + result, + demand, + next.options, + true, + () => isCurrent() && !next.options.signal?.aborted, + ) } private settleTruncateReplay( + session: TruncateReplaySession, + pending: { demand: SubsetDemand; attempt: TruncateReplayAttempt }, + ): void { + try { + if (this.truncateReplaySession !== session) return + if (!this.isLoadSubsetSessionCurrent(session.loadSubsetSession)) { + this.retireStaleTruncateReplay(session) + return + } + if (session.pending.delete(pending)) pending.attempt.pendingCount-- + this.checkTruncateReplayComplete(session) + } catch (error) { + // Replay settlement runs from a Promise callback, so throwing here would + // create an unobserved derived rejection. Surface subscriber errors like + // other async collection events instead. + queueMicrotask(() => { + throw error + }) + } + } + + /** Keep every acquisition begun during recovery inside its publication barrier. */ + private trackTruncateReplayParticipant( session: TruncateReplaySession, attempt: TruncateReplayAttempt, - pending: { promise: Promise }, + demand: SubsetDemand, + result: LoadSubsetRequestResult, ): void { - if (this.truncateReplaySession !== session) return - attempt.pending.delete(pending) - this.checkTruncateReplayComplete(session) + if ( + this.truncateReplaySession !== session || + (session.currentAttempt !== attempt && + attempt.setupComplete && + attempt.pendingCount === 0) || + !(result instanceof Promise) + ) { + return + } + + // An older attempt can still accept returning startup work while setup or + // another participant retains it. Once drained, it cannot reopen. Shared + // promises still get one participant per logical acquisition. + const pending = { demand, attempt } + attempt.pendingCount++ + session.pending.add(pending) + void result.then( + () => this.settleTruncateReplay(session, pending), + (error: unknown) => { + // A released demand no longer participates in this replacement. Its + // cooperative AbortError must not discard rows from active demands. + if ( + this.truncateReplaySession === session && + session.currentAttempt === attempt && + this.isLoadSubsetSessionCurrent(session.loadSubsetSession) && + this.subsetDemands.includes(demand) + ) { + const normalized = this.normalizeLoadSubsetPromiseError(result, error) + session.failures.set(demand, normalized) + } + this.settleTruncateReplay(session, pending) + }, + ) + } + + /** Stop obsolete logical demand from pinning a replay barrier. */ + private removeTruncateReplayParticipant(demand: SubsetDemand): void { + const session = this.truncateReplaySession + if (!session) return + session.failures.delete(demand) + for (const pending of session.pending) { + if (pending.demand === demand) { + session.pending.delete(pending) + pending.attempt.pendingCount-- + } + } } /** Publish only after every overlapping replay attempt has settled. */ private checkTruncateReplayComplete(session: TruncateReplaySession): void { if (this.truncateReplaySession !== session) return - for (const attempt of session.attempts) { - if (!attempt.setupComplete || attempt.pending.size > 0) return - } + if (session.pendingSetups > 0 || session.pending.size > 0) return - if (session.currentAttempt.failed) { - this.abandonTruncateReplay(session) - } else { - this.flushTruncateReplay(session) + const activeFailure = [...session.failures].find(([demand]) => + this.subsetDemands.includes(demand), + ) + try { + if (activeFailure) { + this.abandonTruncateReplay(session, activeFailure[1]) + } else { + this.flushTruncateReplay(session) + } + } finally { + this.setReadyIfIdle() } } /** - * Discard an incomplete current replay and restore the last publication. - * Rows in that publication remain stale until a later source delta or replay - * reconciles them with the source collection. + * Keep an incomplete replay private. The source no longer proves a complete + * state, so only a later successful truncate replay may reopen publication. */ - private abandonTruncateReplay(session: TruncateReplaySession): void { + private abandonTruncateReplay( + session: TruncateReplaySession, + failure: Error, + ): void { if (this.truncateReplaySession !== session) return + session.completion.reject(failure) + // Delegated publication already delivered its rows. Only a private buffer + // returns the caller's pagination position to the public snapshot; the + // private rows and their sent-key tracking stay together for a retry. + if (!session.privateRows) return const publicationState = session.publicationState this.loadedInitialState = publicationState.loadedInitialState this.snapshotSent = publicationState.snapshotSent - this.sentKeys = new Set(publicationState.sentKeys) - this.publishedRows = new Map(publicationState.publishedRows) - this.stalePublishedRows = new Map(publicationState.publishedRows) this.limitedSnapshotRowCount = publicationState.limitedSnapshotRowCount this.lastSentKey = publicationState.lastSentKey - this.truncateReplaySession = undefined } - /** Publish the complete buffered replacement as one subscriber batch. */ + /** Publish the buffered replacement as one batch, or release the delegate. */ private flushTruncateReplay(session: TruncateReplaySession): void { if (this.truncateReplaySession !== session) return this.truncateReplaySession = undefined + this.truncateReplacementPending = false - const retainedDeletes = [...this.stalePublishedRows].map( - ([key, value]): ChangeMessage => ({ - type: `delete`, - key, - value, - }), - ) + // Retained rows the source never re-delivered leave the replacement. + const { privateRows } = session + for (const key of this.stalePublishedRows.keys()) privateRows?.delete(key) this.stalePublishedRows.clear() + try { + if (privateRows) { + // Diff the retained public snapshot against the applied source replacement. + const replacement = this.createStateDiff( + this.publishedRows, + privateRows, + ) + if (replacement.length > 0) this.filteredCallback(replacement) + } + } finally { + // Restore tracking even when a subscriber rejects the replacement. + this.restorePublishedSnapshotTracking() + session.completion.resolve() + this.options.truncateReplayPublication?.succeed() + } + } - const merged = [...session.buffer.flat(), ...retainedDeletes] - const activeDemandFilters = this.subsetDemands.map((demand) => - demand.requestOptions.where - ? createFilterFunctionFromExpression(demand.requestOptions.where) - : undefined, - ) - const replacement = this.createPublicationDiff( - session.publicationState.publishedRows, - merged, - (value) => activeDemandFilters.some((filter) => filter?.(value) ?? true), - ) - if (replacement.length > 0) this.filteredCallback(replacement) - // Buffering records every source key before active-demand filtering. Reset - // the dedupe set to what the subscriber actually received so a later - // request can publish a row that belonged only to a released demand. + private restorePublishedSnapshotTracking(): void { this.sentKeys = new Set(this.publishedRows.keys()) - if (this.orderByIndex) { - this.limitedSnapshotRowCount = this.sentKeys.size - const orderedSentKeys = this.orderByIndex.takeFromStart( - this.sentKeys.size, - (key) => this.sentKeys.has(key), - ) - this.lastSentKey = orderedSentKeys.at(-1) - } + if (!this.orderByIndex) return + + this.limitedSnapshotRowCount = this.sentKeys.size + const orderedSentKeys = this.orderByIndex.takeFromStart( + this.sentKeys.size, + (key) => this.sentKeys.has(key), + ) + this.lastSentKey = orderedSentKeys.at(-1) } - /** Reduce a replay's raw delete/insert stream to one exact semantic delta. */ - private createPublicationDiff( - baseline: ReadonlyMap, + /** Fold changes into the private replacement; false when they publish now. */ + private bufferPrivately( changes: ReadonlyArray>, - isCoveredByActiveDemand: (value: object) => boolean, - ): Array> { - const finalRows = new Map(baseline) + ): boolean { + const privateRows = this.truncateReplaySession?.privateRows + if (!privateRows) return false for (const change of changes) { - if (change.type === `delete`) finalRows.delete(change.key) - else finalRows.set(change.key, change.value) - } - for (const [key, value] of finalRows) { - if (!isCoveredByActiveDemand(value)) finalRows.delete(key) + if (change.type === `delete`) privateRows.delete(change.key) + else privateRows.set(change.key, change.value) } + return true + } + private createStateDiff( + baseline: ReadonlyMap, + finalRows: ReadonlyMap, + ): Array> { const replacement: Array> = [] for (const [key, previousValue] of baseline) { const value = finalRows.get(key) @@ -468,84 +688,229 @@ export class CollectionSubscription return this.truncateReplaySession !== undefined } - setOrderByIndex(index: IndexInterface) { - this.orderByIndex = index + private setReadyIfIdle(): void { + const session = this.truncateReplaySession + const hasPendingReplayWork = + session && (session.pendingSetups > 0 || session.pending.size > 0) + if ( + this.pendingLoadSubsetParticipants.size === 0 && + !hasPendingReplayWork + ) { + this.setStatus(`ready`) + } } - /** - * Check if an orderBy index has been set for this subscription - */ - hasOrderByIndex(): boolean { - return this.orderByIndex !== undefined + private isLoadSubsetSessionCurrent(session: number): boolean { + return session === this.collection._sync.getLoadSubsetSession() + } + + private retireStaleTruncateReplay(session: TruncateReplaySession): void { + if (this.truncateReplaySession !== session) return + this.discardTruncateReplay() + this.stalePublishedRows.clear() + } + + /** Drop the replay without publishing; an unfinished wait rejects as aborted. */ + private discardTruncateReplay(): void { + const session = this.truncateReplaySession + if (session?.completion.isPending()) { + session.completion.reject(new LoadSubsetOperationAbortedError()) + } + this.truncateReplaySession = undefined + this.truncateReplacementPending = false + } + + private resetSnapshotTracking(): void { + this.snapshotSent = false + this.loadedInitialState = false + this.limitedSnapshotRowCount = 0 + this.lastSentKey = undefined + } + + /** One replay session; only direct subscribers buffer a private replacement. */ + private createTruncateReplaySession( + loadSubsetSession: number, + privateRows: () => Map, + ): TruncateReplaySession { + return { + loadSubsetSession, + publicationState: { + loadedInitialState: this.loadedInitialState, + snapshotSent: this.snapshotSent, + limitedSnapshotRowCount: this.limitedSnapshotRowCount, + lastSentKey: this.lastSentKey, + }, + privateRows: this.options.truncateReplayPublication + ? undefined + : privateRows(), + pending: new Set(), + // Setup itself holds publication: adapter/status callbacks may reenter + // before a request returns its promise and joins the pending set. + pendingSetups: 1, + currentAttempt: { pendingCount: 0, setupComplete: false }, + failures: new Map(), + completion: createReplayCompletion(), + } + } + + /** Start one attempt's demands, then release the setup hold on publication. */ + private startTruncateReplayAttempt( + session: TruncateReplaySession, + attempt: TruncateReplayAttempt, + demands: ReadonlyArray, + ): void { + for (const demand of demands) { + if (!this.subsetDemands.includes(demand)) continue + this.startTruncateReplayDemand(session, attempt, demand) + if ( + this.truncateReplaySession !== session || + session.currentAttempt !== attempt + ) { + break + } + } + attempt.setupComplete = true + session.pendingSetups-- + this.checkTruncateReplayComplete(session) + } + + public get hasPendingTruncateReplacement(): boolean { + return this.truncateReplacementPending + } + + public get pendingTruncateReplacement(): Promise | undefined { + const completion = this.truncateReplaySession?.completion + return completion?.isPending() ? completion.promise : undefined + } + + public get hasFailedTruncateReplacement(): boolean { + const completion = this.truncateReplaySession?.completion + return ( + this.truncateReplacementPending && + completion !== undefined && + !completion.isPending() + ) + } + + setOrderByIndex(index: IndexReader) { + this.orderByIndex = index } /** * Set subscription status and emit events if changed */ private setStatus(newStatus: SubscriptionStatus) { + if (this.unsubscribed) return if (this._status === newStatus) { return // No change } const previousStatus = this._status this._status = newStatus + const revision = ++this.statusRevision // Emit status:change event - this.emitInner(`status:change`, { - type: `status:change`, - subscription: this, - previousStatus, - status: newStatus, - }) + this.emitInnerWhile( + `status:change`, + { + type: `status:change`, + subscription: this, + previousStatus, + status: newStatus, + }, + () => this.statusRevision === revision, + ) + + // A listener may synchronously start or release demand. Do not follow that + // newer transition with a stale specific event. + if (this.statusRevision !== revision) return // Emit specific status event const eventKey: `status:${SubscriptionStatus}` = `status:${newStatus}` - this.emitInner(eventKey, { - type: eventKey, - subscription: this, - previousStatus, - status: newStatus, - } as SubscriptionEvents[typeof eventKey]) + this.emitInnerWhile( + eventKey, + { + type: eventKey, + subscription: this, + previousStatus, + status: newStatus, + } as SubscriptionEvents[typeof eventKey], + () => this.statusRevision === revision, + ) } /** Observe an asynchronous subset load and restore status on settlement. */ private observeLoadSubsetResult( - syncResult: Promise | true, + syncResult: LoadSubsetRequestResult, + demand: SubsetDemand, options: LoadSubsetOptions, trackStatus: boolean, shouldReportError: () => boolean = () => true, - ) { + ): void { if (!(syncResult instanceof Promise)) return + const loadSubsetSession = this.collection._sync.getLoadSubsetSession() + const participant = { demand, promise: syncResult } + if (trackStatus) { - this.pendingLoadSubsetPromises.add(syncResult) + this.pendingLoadSubsetParticipants.add(participant) this.setStatus(`loadingSubset`) } const finish = () => { if (trackStatus) { - this.pendingLoadSubsetPromises.delete(syncResult) - if (this.pendingLoadSubsetPromises.size === 0) { - this.setStatus(`ready`) + this.pendingLoadSubsetParticipants.delete(participant) + if (this.isLoadSubsetSessionCurrent(loadSubsetSession)) { + this.setReadyIfIdle() } } } void syncResult.then(finish, (error: unknown) => { - if (shouldReportError()) this.recordLoadSubsetError(options, error) + if ( + this.isLoadSubsetSessionCurrent(loadSubsetSession) && + shouldReportError() + ) { + this.recordLoadSubsetError( + options, + this.normalizeLoadSubsetPromiseError(syncResult, error), + ) + } finish() }) } + /** Give every logical observer of one transport rejection the same Error. */ + private normalizeLoadSubsetPromiseError( + promise: Promise, + error: unknown, + ): Error { + const existing = this.loadSubsetPromiseErrors.get(promise) + if (existing) return existing + const normalized = normalizeError(error) + this.loadSubsetPromiseErrors.set(promise, normalized) + return normalized + } + + private stopDemandStatusParticipants(demand: SubsetDemand): void { + for (const participant of this.pendingLoadSubsetParticipants) { + if (participant.demand === demand) { + this.pendingLoadSubsetParticipants.delete(participant) + } + } + this.setReadyIfIdle() + } + private loadSubset( options: LoadSubsetOptions, shouldReportError: () => boolean = () => true, - ): Promise | true { + ): LoadSubsetRequestResult { try { return this.collection._sync.loadSubset(options) } catch (error) { - if (shouldReportError()) this.recordLoadSubsetError(options, error) - throw error + const normalized = normalizeError(error) + if (shouldReportError()) this.recordLoadSubsetError(options, normalized) + throw normalized } } @@ -571,102 +936,182 @@ export class CollectionSubscription ...demand.requestOptions, signal: abortController.signal, }, + loadSubsetSession: this.collection._sync.getLoadSubsetSession(), abortController, removeRequestAbortListener, } } - /** Replace the adapter lease held for one logical subset demand. */ - private replaceSubsetAcquisition( - demand: SubsetDemand, - next: SubsetAcquisition & { abortController: AbortController }, + /** Retire an acquisition before user code; failed cleanup is not retryable. */ + private releaseAcquisition( + acquisition: SubsetAcquisition, + reportReleaseError = this.primaryFailureDeliveryDepth === 0, ): void { - const previousOptions = demand.options - const removePreviousAbortListener = demand.removeRequestAbortListener - this.collection._sync.unloadSubset(previousOptions) - removePreviousAbortListener?.() - demand.options = next.options - demand.abortController = next.abortController - demand.removeRequestAbortListener = next.removeRequestAbortListener - } - - /** Abort and release one current adapter acquisition. */ - private releaseSubsetDemand(demand: SubsetDemand): void { - demand.abortController?.abort() + if (acquisition.releaseAttempted) return + acquisition.releaseAttempted = true try { - this.collection._sync.unloadSubset(demand.options) + acquisition.abortController?.abort() + if (this.isLoadSubsetSessionCurrent(acquisition.loadSubsetSession)) { + this.collection._sync.unloadSubset(acquisition.options) + } + } catch (error) { + const normalized = reportReleaseError + ? this.recordLoadSubsetError( + acquisition.options, + normalizeError(error), + true, + ) + : normalizeError(error) + throw normalized } finally { - demand.removeRequestAbortListener?.() + acquisition.removeRequestAbortListener?.() } } /** Start and retain the first acquisition for one logical subset demand. */ private startSubsetDemand(requestOptions: LoadSubsetOptions): { demand: SubsetDemand - result: Promise | true + result: LoadSubsetRequestResult + started: boolean } { const demand: SubsetDemand = { requestOptions, - options: requestOptions, + acquisition: { + options: requestOptions, + loadSubsetSession: this.collection._sync.getLoadSubsetSession(), + }, + acquisitionState: `starting`, + } + if ( + this.collection.status === `cleaned-up` || + // Ready/error callbacks can run before sync returns its loader. Idle + // deferred starts still acquire through the sync manager's queue. + (this.collection.config.syncMode === `on-demand` && + (this.collection.status === `error` || + (this.collection.status !== `idle` && + this.collection._sync.syncLoadSubsetFn === null))) + ) { + demand.acquisitionState = `detached` + this.subsetDemands.push(demand) + const initialResult = createDeferred() + demand.initialResult = initialResult + const abort = () => + initialResult.reject(new LoadSubsetOperationAbortedError()) + requestOptions.signal?.addEventListener(`abort`, abort, { once: true }) + const finish = () => { + requestOptions.signal?.removeEventListener(`abort`, abort) + demand.initialResult = undefined + } + void initialResult.promise.then(finish, finish) + return { demand, result: initialResult.promise, started: false } } const acquisition = this.createSubsetAcquisition(demand) + demand.acquisition = acquisition + const replaySession = this.truncateReplaySession + const replayAttempt = replaySession?.currentAttempt + const loadSubsetSession = this.collection._sync.getLoadSubsetSession() + // Reentrant release must see the exact acquisition before adapter work + // starts. A genuine load throw removes this tentative logical owner below. + this.subsetDemands.push(demand) + let result: LoadSubsetRequestResult try { - const result = this.loadSubset(acquisition.options) - demand.options = acquisition.options - demand.abortController = acquisition.abortController - demand.removeRequestAbortListener = acquisition.removeRequestAbortListener - this.subsetDemands.push(demand) - return { demand, result } + result = this.loadSubset( + acquisition.options, + () => + this.isLoadSubsetSessionCurrent(loadSubsetSession) && + this.subsetDemands.includes(demand) && + (replaySession === undefined || + (this.truncateReplaySession === replaySession && + replaySession.currentAttempt === replayAttempt)), + ) } catch (error) { - acquisition.abortController.abort() - acquisition.removeRequestAbortListener?.() + const demandIndex = this.subsetDemands.indexOf(demand) + if (demandIndex !== -1) { + if ( + replaySession && + replayAttempt && + this.truncateReplaySession === replaySession && + replaySession.currentAttempt === replayAttempt + ) { + replaySession.failures.set(demand, normalizeError(error)) + } + this.subsetDemands.splice(demandIndex, 1) + } + cancelAcquisition(acquisition) throw error } + + if (!this.isLoadSubsetSessionCurrent(loadSubsetSession)) { + const demandIndex = this.subsetDemands.indexOf(demand) + if (demandIndex !== -1) this.subsetDemands.splice(demandIndex, 1) + cancelAcquisition(acquisition) + return { demand, result, started: true } + } + + demand.acquisitionState = `active` + if (!this.subsetDemands.includes(demand)) { + this.releaseAcquisition(acquisition) + return { demand, result, started: true } + } + + if (replaySession && replayAttempt) { + this.trackTruncateReplayParticipant( + replaySession, + replayAttempt, + demand, + result, + ) + } + return { demand, result, started: true } + } + + /** Re-check ownership after adapter and event callbacks that may reenter. */ + private isDemandActive(demand: SubsetDemand): boolean { + return !this.unsubscribed && this.subsetDemands.includes(demand) } private recordLoadSubsetError( options: LoadSubsetOptions, error: unknown, reportAborted = false, - ): void { + ): Error { + const normalized = normalizeError(error) // Aborted subset requests are obsolete demand, not load failures. The // request may reject after its route has already been released. - if (options.signal?.aborted && !reportAborted) return + if (options.signal?.aborted && !reportAborted) return normalized - this._lastError = error - this.emitInner(`loadSubset:error`, { - type: `loadSubset:error`, - subscription: this, - options, - error, - }) - } - - hasLoadedInitialState() { - return this.loadedInitialState - } - - hasSentAtLeastOneSnapshot() { - return this.snapshotSent + this._lastError = normalized + this.primaryFailureDeliveryDepth++ + try { + this.emitInner(`loadSubset:error`, { + type: `loadSubset:error`, + subscription: this, + options, + error: normalized, + }) + } finally { + this.primaryFailureDeliveryDepth-- + } + return normalized } emitEvents(changes: Array>): boolean { + if (this.unsubscribed) return false const newChanges = this.filterAndFlipChanges(changes) // Reconciliation can reduce a source delta to no visible change. Do not // wake subscribers for an empty semantic batch. if (changes.length > 0 && newChanges.length === 0) return false - if (this.isBufferingForTruncate) { - // Buffer the changes instead of emitting immediately - // This prevents a flash of missing content during truncate/refetch - if (newChanges.length > 0) { - this.truncateReplaySession!.buffer.push(newChanges) - } - return false - } else { - return this.filteredCallback(newChanges) - } + // A direct subscriber sees the replacement as one batch, not a flash of + // missing content. Delegated publication keeps its private D2 contributions. + if (this.bufferPrivately(newChanges)) return false + return this.filteredCallback(newChanges) + } + + /** Keep direct snapshot reads private while an authoritative replay is open. */ + private publishSnapshot(changes: Array>): void { + if (!this.bufferPrivately(changes)) this.callback(changes) } /** @@ -674,9 +1119,11 @@ export class CollectionSubscription * Returns a boolean indicating if it succeeded. * It can only fail if there is no index to fulfill the request * and the optimizedOnly option is set to true, - * or, the entire state was already loaded. + * or, the entire state was already loaded or the request was cancelled. */ requestSnapshot(opts?: RequestSnapshotOptions): boolean { + // Cancel before acquiring ownership or publishing a local snapshot. + if (this.unsubscribed || opts?.signal?.aborted) return false if (this.loadedInitialState) { // Subscription was deoptimized so we already sent the entire initial state return false @@ -715,17 +1162,31 @@ export class CollectionSubscription limit: opts?.limit, } - const { demand, result: syncResult } = this.startSubsetDemand(loadOptions) + const { + demand, + result: syncResult, + started, + } = this.startSubsetDemand(loadOptions) + if (!this.isDemandActive(demand)) return false if (opts?.where) this.requestedSubsetWhere.set(loadOptions, opts.where) - // Pass the raw loadSubset result to the caller for external tracking - opts?.onLoadSubsetResult?.(syncResult) - - this.observeLoadSubsetResult( + // Report the result synchronously, including a wait for an unavailable loader. + opts?.onLoadSubsetResult?.( syncResult, - demand.options, - opts?.trackLoadSubsetPromise ?? true, + demand.acquisition.options, + (primaryFailure) => this.releaseDemand(demand, primaryFailure), ) + if (!this.isDemandActive(demand)) return false + + if (started) { + this.observeLoadSubsetResult( + syncResult, + demand, + demand.acquisition.options, + opts?.trackLoadSubsetPromise ?? true, + ) + } + if (!this.isDemandActive(demand)) return false // Also load data immediately from the collection let snapshot: Array> | void @@ -736,6 +1197,9 @@ export class CollectionSubscription }) if (snapshot === undefined) { opts.onUnoptimized() + // The callback can unsubscribe; TypeScript retains the pre-call narrowing. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (this.unsubscribed) return false snapshot = this.collection.currentStateAsChanges({ ...stateOpts, optimizedOnly: false, @@ -744,15 +1208,24 @@ export class CollectionSubscription } else { snapshot = this.collection.currentStateAsChanges(stateOpts) } + // Snapshot evaluation may call user code that tears down the subscription. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (this.unsubscribed) return false if (snapshot === undefined) { // Couldn't load from indexes return false } - // Only send changes that have not been sent yet + // Skip known rows, except retained rows from an abandoned replay: a new + // snapshot must reconcile those with the source, not suppress their update. + const knownRows = + this.truncateReplaySession?.privateRows ?? this.publishedRows const filteredSnapshot = snapshot.filter( - (change) => !this.sentKeys.has(change.key), + (change) => + (!this.isBufferingForTruncate && + this.stalePublishedRows.has(change.key)) || + (!this.sentKeys.has(change.key) && !knownRows.has(change.key)), ) // Add keys to sentKeys BEFORE calling callback to prevent race condition. @@ -763,7 +1236,11 @@ export class CollectionSubscription } this.snapshotSent = true - this.callback(filteredSnapshot) + this.publishSnapshot( + this.isBufferingForTruncate + ? filteredSnapshot + : this.reconcileStalePublishedChanges(filteredSnapshot), + ) return true } @@ -776,8 +1253,90 @@ export class CollectionSubscription ) if (index === -1) return - const [demand] = this.subsetDemands.splice(index, 1) - if (demand) this.releaseSubsetDemand(demand) + this.releaseDemandAt(index) + } + + private releaseDemand( + demand: SubsetDemand, + primaryFailure?: { error: unknown }, + ): void { + if (!primaryFailure) { + const index = this.subsetDemands.indexOf(demand) + if (index !== -1) this.releaseDemandAt(index) + return + } + + try { + this.recordLoadSubsetError( + demand.acquisition.options, + primaryFailure.error, + true, + ) + } finally { + // The failed request remains the public error, even if cleanup also fails. + const index = this.subsetDemands.indexOf(demand) + if (index !== -1) this.releaseDemandAt(index, false) + } + } + + private releaseDemandAt( + index: number, + reportReleaseError = this.primaryFailureDeliveryDepth === 0, + ): void { + const demand = this.subsetDemands[index] + if (!demand) return + const replaySession = this.truncateReplaySession + const acquisition = demand.acquisition + this.subsetDemands.splice(index, 1) + demand.initialResult?.reject(new LoadSubsetOperationAbortedError()) + const releaseCallbacks = [ + () => this.removeTruncateReplayParticipant(demand), + ...(demand.acquisitionState === `active` + ? [ + // Adapter release is a supported reentrancy boundary. A demand + // started from unload joins this replacement before completion. + () => this.releaseAcquisition(acquisition, reportReleaseError), + ] + : []), + () => this.retireEmptyReplay(), + () => { + if (replaySession) this.checkTruncateReplayComplete(replaySession) + }, + // Ready follows replacement publication, never the delete half of it. + () => this.stopDemandStatusParticipants(demand), + ] + runAllCallbacks(releaseCallbacks) + } + + /** A replay with no remaining logical demand cannot establish more rows. */ + private retireEmptyReplay(): void { + if (this.subsetDemands.length !== 0 || !this.truncateReplaySession) { + return + } + this.discardTruncateReplay() + this.stalePublishedRows = new Map(this.publishedRows) + this.restorePublishedSnapshotTracking() + this.options.truncateReplayPublication?.succeed() + } + + /** Read the applied rows in an ordered acquisition without starting demand. */ + readOrderedSnapshot( + options: LoadSubsetOptions, + ): Array, string | number>> { + const predicates = [ + this.options.whereExpression, + options.where, + options.cursor?.whereFrom, + ].filter((where) => where !== undefined) + const snapshot = this.collection.currentStateAsChanges({ + orderBy: options.orderBy, + limit: options.limit, + where: + predicates.length > 0 + ? predicates.reduce((left, right) => and(left, right)) + : undefined, + }) + return Array.isArray(snapshot) ? snapshot : [] } /** @@ -785,9 +1344,8 @@ export class CollectionSubscription * Requires a range index to be set with `setOrderByIndex` prior to calling this method. * It uses that range index to load the items in the order of the index. * - * For multi-column orderBy: - * - Uses first value from `minValues` for LOCAL index operations (wide bounds, ensures no missed rows) - * - Uses all `minValues` to build a precise composite cursor for SYNC layer loadSubset + * Cursor requests support one order term and one minValue. Multi-column + * queries use the ordered loader's prefix-and-tie fallback instead. * * Note 1: it may load more rows than the provided LIMIT because it loads all values equal to the first cursor value + limit values greater. * This is needed to ensure that it does not accidentally skip duplicate values when the limit falls in the middle of some duplicated values. @@ -801,6 +1359,7 @@ export class CollectionSubscription trackLoadSubsetPromise: shouldTrackLoadSubsetPromise = true, onLoadSubsetResult, }: RequestLimitedSnapshotOptions) { + if (this.unsubscribed) return if (!limit) throw new Error(`limit is required`) if (!this.orderByIndex) { @@ -809,6 +1368,11 @@ export class CollectionSubscription ) } + // Validate cursor input before local delivery changes sent keys or calls user code. + const whereFromCursor = minValues + ? buildCursor(orderBy, minValues) + : undefined + // Check if minValues has a first element (regardless of its value) // This distinguishes between "no min value provided" vs "min value is undefined" const hasMinValue = minValues !== undefined && minValues.length > 0 @@ -845,9 +1409,6 @@ export class CollectionSubscription // so if minValue is 3 then the previous snapshot may not have included all 3s // e.g. if it was offset 0 and limit 3 it would only have loaded the first 3 // so we load all rows equal to minValue first, to be sure we don't skip any duplicate values - // - // For multi-column orderBy, we use the first column value for index operations (wide bounds) - // This may load some duplicates but ensures we never miss any rows. let keys: Array = [] if (hasMinValue) { // First, get all items with the same FIRST COLUMN value as minValue @@ -891,8 +1452,6 @@ export class CollectionSubscription : null while (valuesNeeded() > 0 && !collectionExhausted()) { - const insertedKeys = new Set() // Track keys we add to `changes` in this iteration - for (const key of keys) { const value = this.collection.get(key)! changes.push({ @@ -903,7 +1462,6 @@ export class CollectionSubscription // Extract the indexed value (e.g., salary) from the row, not the full row // This is needed for index.take() to work correctly with the BTree comparator biggestObservedValue = valueExtractor ? valueExtractor(value) : value - insertedKeys.add(key) // Track this key } keys = index.take(valuesNeeded(), biggestObservedValue!, filterFn) @@ -920,7 +1478,10 @@ export class CollectionSubscription this.sentKeys.add(change.key) } - this.callback(changes) + this.publishSnapshot(changes) + // A subscriber callback can synchronously tear down this subscription. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (this.unsubscribed) return // Update the row count and last key after sending (for next call's offset/cursor) this.limitedSnapshotRowCount = Math.max( @@ -942,27 +1503,9 @@ export class CollectionSubscription } | undefined - if (minValues !== undefined && minValues.length > 0) { - const whereFromCursor = buildCursor(orderBy, minValues) - - if (whereFromCursor) { - const { expression } = orderBy[0]! - const cursorMinValue = minValues[0] - - // Build the whereCurrent expression for the first orderBy column - // For Date values, we need to handle precision differences between JS (ms) and backends (μs) - // A JS Date represents a 1ms range, so we query for all values within that range - let whereCurrentCursor: BasicExpression - if (cursorMinValue instanceof Date) { - const cursorMinValuePlus1ms = new Date(cursorMinValue.getTime() + 1) - whereCurrentCursor = and( - gte(expression, new Value(cursorMinValue)), - lt(expression, new Value(cursorMinValuePlus1ms)), - ) - } else { - whereCurrentCursor = eq(expression, new Value(cursorMinValue)) - } - + if (whereFromCursor && minValues) { + const whereCurrentCursor = buildCursorCurrent(orderBy, minValues) + if (whereCurrentCursor) { cursorExpressions = { whereFrom: whereFromCursor, whereCurrent: whereCurrentCursor, @@ -984,15 +1527,29 @@ export class CollectionSubscription subscription: this, } - const { demand, result: syncResult } = this.startSubsetDemand(loadOptions) + const { + demand, + result: syncResult, + started, + } = this.startSubsetDemand(loadOptions) + if (!this.isDemandActive(demand)) return - // Pass the raw loadSubset result to the caller for external tracking - onLoadSubsetResult?.(syncResult) - this.observeLoadSubsetResult( + // Report the result synchronously, including a wait for an unavailable loader. + onLoadSubsetResult?.( syncResult, - demand.options, - shouldTrackLoadSubsetPromise, + demand.acquisition.options, + (primaryFailure) => this.releaseDemand(demand, primaryFailure), ) + if (!this.isDemandActive(demand)) return + if (started) { + this.observeLoadSubsetResult( + syncResult, + demand, + demand.acquisition.options, + shouldTrackLoadSubsetPromise, + ) + } + if (!this.isDemandActive(demand)) return } // TODO: also add similar test but that checks that it can also load it from the collection's loadSubset function @@ -1092,6 +1649,18 @@ export class CollectionSubscription }) } } + // Cleanup discards rows without publishing deletes. Eager sources publish + // their installed state; subset sources must first finish reacquisition. + if ( + this.collection.config.syncMode !== `on-demand` && + !this.isBufferingForTruncate + ) { + for (const [key, value] of this.stalePublishedRows) { + if (this.collection.has(key)) continue + this.stalePublishedRows.delete(key) + reconciled.push({ type: `delete`, key, value }) + } + } return reconciled } @@ -1143,42 +1712,52 @@ export class CollectionSubscription } unsubscribe() { - let firstCleanupError: unknown - - // Clean up truncate event listener - try { - this.truncateCleanup?.() - } catch (error) { - firstCleanupError = error - } + if (this.unsubscribed) return + this.unsubscribed = true + // Stop any status listener set already being iterated. Clearing the + // emitter's map cannot invalidate that captured Set by itself. + this.statusRevision++ + const sourceListenerCleanups = [ + this.truncateCleanup, + this.collectionCleanup, + this.collectionRestartCleanup, + ] this.truncateCleanup = undefined - - // Stop any buffered replay from publishing after unsubscription. - this.truncateReplaySession = undefined - this.stalePublishedRows.clear() - - // Release the current adapter acquisition for each logical subset demand. - for (const demand of this.subsetDemands) { - try { - this.releaseSubsetDemand(demand) - } catch (error) { - firstCleanupError ??= error - } - } - this.subsetDemands = [] - - try { - this.emitInner(`unsubscribed`, { - type: `unsubscribed`, - subscription: this, - }) - } catch (error) { - firstCleanupError ??= error - } finally { + this.collectionCleanup = undefined + this.collectionRestartCleanup = undefined + + runAllCallbacks([ + ...sourceListenerCleanups.map((cleanup) => () => cleanup?.()), + () => { + // Stop any buffered replay from publishing after unsubscription. + this.discardTruncateReplay() + this.stalePublishedRows.clear() + + // Retire every owner before an unload can reenter teardown. + const acquisitions = this.subsetDemands + .filter((demand) => demand.acquisitionState === `active`) + .map((demand) => demand.acquisition) + for (const demand of this.subsetDemands) { + demand.initialResult?.reject(new LoadSubsetOperationAbortedError()) + this.stopDemandStatusParticipants(demand) + if (demand.acquisitionState === `starting`) { + cancelAcquisition(demand.acquisition) + } + } + this.subsetDemands = [] + runAllCallbacks( + acquisitions.map( + (acquisition) => () => this.releaseAcquisition(acquisition), + ), + ) + }, + () => + this.emitInner(`unsubscribed`, { + type: `unsubscribed`, + subscription: this, + }), // Clear all event listeners to prevent memory leaks - this.clearListeners() - } - - if (firstCleanupError !== undefined) throw firstCleanupError + () => this.clearListeners(), + ]) } } diff --git a/packages/db/src/collection/sync.ts b/packages/db/src/collection/sync.ts index a14a357ddd..a5b02f7a6c 100644 --- a/packages/db/src/collection/sync.ts +++ b/packages/db/src/collection/sync.ts @@ -1,7 +1,9 @@ import { CollectionConfigurationError, CollectionIsInErrorStateError, + CollectionPreloadAbortedError, DuplicateKeySyncError, + LoadSubsetOperationAbortedError, NoPendingSyncTransactionCommitError, NoPendingSyncTransactionWriteError, SyncCleanupError, @@ -16,7 +18,9 @@ import type { ChangeMessageOrDeleteKeyMessage, CleanupFn, CollectionConfig, + LoadSubsetFn, LoadSubsetOptions, + LoadSubsetRequestResult, OptimisticChangeMessage, SyncConfigRes, SyncMetadataApi, @@ -34,7 +38,7 @@ type DeferredLoadSubset = { } type LoadSubsetOperation = { - pending: Set> + pending: Set> waiting: boolean completed: boolean hasError: boolean @@ -57,14 +61,13 @@ export class CollectionSyncManager< private syncMode: `eager` | `on-demand` public preloadPromise: Promise | null = null + private rejectPreload?: (error: unknown) => void public syncCleanupFn: (() => void) | null = null - public syncLoadSubsetFn: - | ((options: LoadSubsetOptions) => true | Promise) - | null = null + public syncLoadSubsetFn: LoadSubsetFn | null = null public syncUnloadSubsetFn: ((options: LoadSubsetOptions) => void) | null = null - private pendingLoadSubsetPromises: Set> = new Set() + private pendingLoadSubsetPromises: Set> = new Set() private activeLoadSubsetOperation: LoadSubsetOperation | undefined private loadSubsetOperations = new Set() private syncStartDeferred = false @@ -104,6 +107,7 @@ export class CollectionSyncManager< * This is called when the collection is first accessed or preloaded */ public startSync(): void { + this.lifecycle.assertCanStartSync() if ( this.lifecycle.status !== `idle` && this.lifecycle.status !== `cleaned-up` @@ -119,6 +123,9 @@ export class CollectionSyncManager< const syncEpoch = ++this.syncEpoch const isCurrentSync = () => syncEpoch === this.syncEpoch this.lifecycle.setStatus(`loading`) + if (!isCurrentSync()) return + let syncEntryActive = true + let readyEffectFailure: { error: unknown } | undefined try { const syncRes = normalizeSyncFnResult( @@ -279,7 +286,12 @@ export class CollectionSyncManager< return receipt }, markReady: () => { - if (isCurrentSync()) this.lifecycle.markReady() + if (!isCurrentSync()) return + if (syncEntryActive) { + readyEffectFailure ??= this.lifecycle.markReadyDuringSyncStart() + } else { + this.lifecycle.markReady() + } }, markError: (error?: unknown) => { if (isCurrentSync()) this.lifecycle.markError(error) @@ -323,6 +335,13 @@ export class CollectionSyncManager< metadata: this.createSyncMetadataApi(isCurrentSync), }), ) + syncEntryActive = false + + if (!isCurrentSync()) { + syncRes?.cleanup?.() + if (readyEffectFailure) throw readyEffectFailure.error + return + } // Store cleanup function if provided this.syncCleanupFn = syncRes?.cleanup ?? null @@ -341,9 +360,11 @@ export class CollectionSyncManager< ) } } catch (error) { - this.lifecycle.markError(error) + syncEntryActive = false + if (isCurrentSync()) this.lifecycle.markError(error) throw error } + if (readyEffectFailure) throw readyEffectFailure.error } public deferStart(): boolean { @@ -369,6 +390,7 @@ export class CollectionSyncManager< this.syncStartRequested = false const deferredLoadSubsets = this.deferredLoadSubsets this.deferredLoadSubsets = [] + const loadSubsetSession = this.loadSubsetSession try { if (shouldStart) { @@ -382,11 +404,18 @@ export class CollectionSyncManager< } for (const { options, deferred } of deferredLoadSubsets) { + const loadSubset = this.syncLoadSubsetFn try { - const result = this.syncLoadSubsetFn?.(options) ?? true + if ( + loadSubsetSession !== this.loadSubsetSession || + options.signal?.aborted + ) { + throw new LoadSubsetOperationAbortedError() + } + const result = loadSubset?.(options) ?? true if (result instanceof Promise) { void result.then( - () => deferred.resolve(undefined), + (sourceResult) => deferred.resolve(sourceResult), (error: unknown) => deferred.reject(error), ) } else { @@ -519,6 +548,11 @@ export class CollectionSyncManager< * Multiple concurrent calls will share the same promise */ public preload(): Promise { + try { + this.lifecycle.assertCanStartSync() + } catch (error) { + return Promise.reject(error) + } if (this.preloadPromise) { return this.preloadPromise } @@ -545,14 +579,19 @@ export class CollectionSyncManager< } let settled = false - let startingSync = false + const syncStartState = { active: false, ready: false } let unsubscribeError = () => {} let unsubscribeReady = () => {} const resolveReady = () => { + if (syncStartState.active) { + syncStartState.ready = true + return + } if (settled) return settled = true unsubscribeError() unsubscribeReady() + if (this.rejectPreload === rejectError) this.rejectPreload = undefined resolve() } const rejectError = (error: unknown) => { @@ -560,13 +599,15 @@ export class CollectionSyncManager< settled = true unsubscribeError() unsubscribeReady() + if (this.rejectPreload === rejectError) this.rejectPreload = undefined reject(error) } // Register callback BEFORE starting sync to avoid race condition + this.rejectPreload = rejectError unsubscribeReady = this.lifecycle.onFirstReady(resolveReady) unsubscribeError = this.collection.on(`status:error`, () => { - if (startingSync) { + if (syncStartState.active) { return } rejectError(this.getPreloadError()) @@ -577,17 +618,24 @@ export class CollectionSyncManager< this.lifecycle.status === `idle` || this.lifecycle.status === `cleaned-up` ) { - startingSync = true + syncStartState.active = true + let startFailure: { error: unknown } | undefined try { this.startSync() } catch (error) { - rejectError(error) - return + startFailure = { error } } finally { - startingSync = false + syncStartState.active = false } if (this.collection.status === `error`) { rejectError(this.getPreloadError()) + } else if (syncStartState.ready) { + // A first-ready listener can throw after readiness is established. + // That failure still escapes direct startSync(), but preload follows + // the final collection state after synchronous adapter entry. + resolveReady() + } else if (startFailure) { + rejectError(startFailure.error) } } }) @@ -615,17 +663,12 @@ export class CollectionSyncManager< return this.pendingLoadSubsetPromises.size > 0 } - /** Wait for the subset loads that are active during the current operation. */ - public waitForCurrentLoadSubset(): true | Promise { - if (this.pendingLoadSubsetPromises.size === 0) return true - return this.waitForPendingLoadSubset() - } - /** @internal Observe subset requests caused by one imperative operation. */ public beginLoadSubsetOperation(): { wait: () => true | Promise cancel: () => void } { + const previousOperation = this.activeLoadSubsetOperation const operation: LoadSubsetOperation = { pending: new Set(), waiting: false, @@ -643,7 +686,9 @@ export class CollectionSyncManager< operation.completed = true this.loadSubsetOperations.delete(operation) if (this.activeLoadSubsetOperation === operation) { - this.activeLoadSubsetOperation = undefined + this.activeLoadSubsetOperation = previousOperation?.completed + ? undefined + : previousOperation } }, } @@ -667,7 +712,7 @@ export class CollectionSyncManager< private settleLoadSubsetOperation( operation: LoadSubsetOperation, - promise: Promise, + promise: Promise, outcome: { ok: true } | { ok: false; error: unknown }, ): void { if (operation.completed) return @@ -697,7 +742,7 @@ export class CollectionSyncManager< } /** @internal Attach a relevant existing request to the active operation. */ - public trackLoadSubsetOperationPromise(promise: Promise): void { + public trackLoadSubsetOperationPromise(promise: Promise): void { const operation = this.activeLoadSubsetOperation if (!operation || operation.pending.has(promise)) return @@ -712,17 +757,11 @@ export class CollectionSyncManager< ) } - private async waitForPendingLoadSubset(): Promise { - do { - await Promise.all([...this.pendingLoadSubsetPromises]) - } while (this.pendingLoadSubsetPromises.size > 0) - } - /** * Tracks a load promise for isLoadingSubset state. * @internal This is for internal coordination (e.g., live-query glue code), not for general use. */ - public trackLoadPromise(promise: Promise): void { + public trackLoadPromise(promise: Promise): void { const loadSubsetSession = this.loadSubsetSession const loadingStarting = !this.isLoadingSubset this.pendingLoadSubsetPromises.add(promise) @@ -759,15 +798,20 @@ export class CollectionSyncManager< void promise.then(finish, finish) } + /** @internal Generation fence for subscription-owned async work. */ + public getLoadSubsetSession(): number { + return this.loadSubsetSession + } + /** * Requests the sync layer to load more data. * @param options Options to control what data is being loaded * @returns If data loading is asynchronous, this method returns a promise that resolves when the data is loaded. * Returns true if no sync function is configured, if syncMode is 'eager', or if there is no work to do. */ - public loadSubset(options: LoadSubsetOptions): Promise | true { + public loadSubset(options: LoadSubsetOptions): LoadSubsetRequestResult { if (options.signal?.aborted) { - return true + return Promise.reject(new LoadSubsetOperationAbortedError()) } // Bypass loadSubset when syncMode is 'eager' @@ -800,13 +844,16 @@ export class CollectionSyncManager< * @param options Options that identify what data is being unloaded */ public unloadSubset(options: LoadSubsetOptions): void { + // Eager loading bypasses subset acquisition, so there is no lease to release. + if (this.syncMode === `eager`) return + if (this.syncStartDeferred) { this.deferredLoadSubsets = this.deferredLoadSubsets.filter((request) => { if (request.options !== options) { return true } - request.deferred.resolve(undefined) + request.deferred.reject(new LoadSubsetOperationAbortedError()) return false }) return @@ -820,14 +867,19 @@ export class CollectionSyncManager< public cleanup(): void { // Invalidate callbacks retained by asynchronous work from this session // before invoking adapter cleanup or allowing a new session to start. - this.syncEpoch++ + const cleanupEpoch = ++this.syncEpoch this.loadSubsetSession++ + this.rejectPreload?.(new CollectionPreloadAbortedError()) + const cleanup = this.syncCleanupFn + this.syncCleanupFn = null + this.syncLoadSubsetFn = null + this.syncUnloadSubsetFn = null try { - if (this.syncCleanupFn) { - this.syncCleanupFn() - this.syncCleanupFn = null - } + cleanup?.() } catch (error) { + // Keep failed cleanup retryable, but never overwrite a replacement + // session installed by reentrant adapter code. + if (this.syncEpoch === cleanupEpoch) this.syncCleanupFn = cleanup // Re-throw in a microtask to surface the error after cleanup completes queueMicrotask(() => { if (error instanceof Error) { @@ -842,8 +894,6 @@ export class CollectionSyncManager< }) } this.preloadPromise = null - this.syncLoadSubsetFn = null - this.syncUnloadSubsetFn = null this.syncStartDeferred = false this.syncStartRequested = false const wasLoadingSubset = this.pendingLoadSubsetPromises.size > 0 @@ -862,14 +912,16 @@ export class CollectionSyncManager< if (!operation.completed) { operation.completed = true operation.pending.clear() - operation.deferred?.resolve() + operation.hasError = true + operation.error = new LoadSubsetOperationAbortedError() + operation.deferred?.reject(operation.error) } } this.loadSubsetOperations.clear() const deferredLoadSubsets = this.deferredLoadSubsets this.deferredLoadSubsets = [] - for (const { deferred } of deferredLoadSubsets) { - deferred.resolve(undefined) + for (const request of deferredLoadSubsets) { + request.deferred.reject(new LoadSubsetOperationAbortedError()) } } } diff --git a/packages/db/src/errors.ts b/packages/db/src/errors.ts index 12c6753d3b..a97cb117b7 100644 --- a/packages/db/src/errors.ts +++ b/packages/db/src/errors.ts @@ -737,56 +737,33 @@ export class SyncTransactionAbortedError extends Error { } } -// Query Optimizer Errors -export class QueryOptimizerError extends TanStackDBError { - constructor(message: string) { - super(message) - this.name = `QueryOptimizerError` - } -} - -export class CannotCombineEmptyExpressionListError extends QueryOptimizerError { +/** A collection was cleaned up before its initial preload became ready. */ +export class CollectionPreloadAbortedError extends Error { constructor() { - super(`Cannot combine empty expression list`) + super(`Collection preload was abandoned during cleanup`) + this.name = `AbortError` } } -/** - * Internal error when the query optimizer fails to convert a WHERE clause to a collection filter. - */ -export class WhereClauseConversionError extends QueryOptimizerError { - constructor(collectionId: string, alias: string) { - super( - `Failed to convert WHERE clause to collection filter for collection '${collectionId}' alias '${alias}'. This indicates a bug in the query optimization logic.`, - ) +/** A subset operation was canceled before its result became visible. */ +export class LoadSubsetOperationAbortedError extends Error { + constructor() { + super(`Load subset operation was aborted before its result became visible`) + this.name = `AbortError` } } -/** - * Error when a subscription cannot be found during lazy join processing. - * For subqueries, aliases may be remapped (e.g., 'activeUser' → 'user'). - */ -export class SubscriptionNotFoundError extends QueryCompilationError { - constructor( - resolvedAlias: string, - originalAlias: string, - collectionId: string, - availableAliases: Array, - ) { - super( - `Internal error: subscription for alias '${resolvedAlias}' (remapped from '${originalAlias}', collection '${collectionId}') is missing in join pipeline. Available aliases: ${availableAliases.join(`, `)}. This indicates a bug in alias tracking.`, - ) +// Query Optimizer Errors +export class QueryOptimizerError extends TanStackDBError { + constructor(message: string) { + super(message) + this.name = `QueryOptimizerError` } } -/** - * Error thrown when aggregate expressions are used outside of a GROUP BY context. - */ -export class AggregateNotSupportedError extends QueryCompilationError { +export class CannotCombineEmptyExpressionListError extends QueryOptimizerError { constructor() { - super( - `Aggregate expressions are not supported in this context. Use GROUP BY clause for aggregates.`, - ) + super(`Cannot combine empty expression list`) } } @@ -814,3 +791,13 @@ export class SetWindowRequiresOrderByError extends QueryCompilationError { ) } } + +/** Error thrown when setWindow is called from inside another setWindow call. */ +export class SetWindowReentrancyError extends TanStackDBError { + constructor() { + super( + `setWindow() cannot run reentrantly. Wait for the current window operation to return before starting another one.`, + ) + this.name = `SetWindowReentrancyError` + } +} diff --git a/packages/db/src/event-emitter.ts b/packages/db/src/event-emitter.ts index 6d7ad90aa2..06fec5adb0 100644 --- a/packages/db/src/event-emitter.ts +++ b/packages/db/src/event-emitter.ts @@ -5,7 +5,11 @@ export class EventEmitter> { private listeners = new Map< keyof TEvents, - Set<(event: TEvents[keyof TEvents]) => void> + Map<(event: TEvents[keyof TEvents]) => void, object> + >() + private onceCallbacks = new WeakMap< + (event: TEvents[keyof TEvents]) => void, + (event: TEvents[keyof TEvents]) => void >() /** @@ -19,12 +23,21 @@ export class EventEmitter> { callback: (event: TEvents[T]) => void, ): () => void { if (!this.listeners.has(event)) { - this.listeners.set(event, new Set()) + this.listeners.set(event, new Map()) + } + const listeners = this.listeners.get(event)! + const registered = callback as (event: any) => void + let registration = listeners.get(registered) + if (!registration) { + registration = {} + listeners.set(registered, registration) } - this.listeners.get(event)!.add(callback as (event: any) => void) return () => { - this.listeners.get(event)?.delete(callback as (event: any) => void) + const current = this.listeners.get(event) + if (current?.get(registered) === registration) { + current.delete(registered) + } } } @@ -38,10 +51,16 @@ export class EventEmitter> { event: T, callback: (event: TEvents[T]) => void, ): () => void { - const unsubscribe = this.on(event, (eventPayload) => { - callback(eventPayload) + let unsubscribe = () => {} + const listener = (eventPayload: TEvents[T]) => { unsubscribe() - }) + callback(eventPayload) + } + this.onceCallbacks.set( + listener as (event: TEvents[keyof TEvents]) => void, + callback as (event: TEvents[keyof TEvents]) => void, + ) + unsubscribe = this.on(event, listener) return unsubscribe } @@ -54,7 +73,16 @@ export class EventEmitter> { event: T, callback: (event: TEvents[T]) => void, ): void { - this.listeners.get(event)?.delete(callback as (event: any) => void) + const listeners = this.listeners.get(event) + if (!listeners) return + for (const listener of listeners.keys()) { + if ( + listener === callback || + this.onceCallbacks.get(listener) === callback + ) { + listeners.delete(listener) + } + } } /** @@ -97,7 +125,20 @@ export class EventEmitter> { event: T, eventPayload: TEvents[T], ): void { - this.listeners.get(event)?.forEach((listener) => { + this.emitInnerWhile(event, eventPayload, () => true) + } + + /** Emit until a reentrant callback invalidates the event being delivered. */ + protected emitInnerWhile( + event: T, + eventPayload: TEvents[T], + isCurrent: () => boolean, + ): void { + const listeners = this.listeners.get(event) + if (!listeners) return + for (const [listener, registration] of [...listeners]) { + if (!isCurrent()) break + if (this.listeners.get(event)?.get(listener) !== registration) continue try { listener(eventPayload) } catch (error) { @@ -106,7 +147,7 @@ export class EventEmitter> { throw error }) } - }) + } } /** diff --git a/packages/db/src/index.ts b/packages/db/src/index.ts index 8c4d7258e0..18a6bee3b6 100644 --- a/packages/db/src/index.ts +++ b/packages/db/src/index.ts @@ -38,8 +38,8 @@ export { BaseIndex } from './indexes/base-index.js' export type { IndexInterface, IndexConstructor, - IndexStats, IndexOperation, + IndexReader, } from './indexes/base-index.js' export { type IndexOptions } from './indexes/index-options.js' diff --git a/packages/db/src/indexes/auto-index.ts b/packages/db/src/indexes/auto-index.ts index 350b469a43..6e6f1896ed 100644 --- a/packages/db/src/indexes/auto-index.ts +++ b/packages/db/src/indexes/auto-index.ts @@ -5,10 +5,6 @@ import type { CompareOptions } from '../query/builder/types' import type { BasicExpression } from '../query/ir' import type { CollectionImpl } from '../collection/index.js' -export interface AutoIndexConfig { - autoIndex?: `off` | `eager` -} - function shouldAutoIndex(collection: CollectionImpl) { // Only proceed if auto-indexing is enabled // Note: autoIndex: 'eager' without defaultIndexType is caught at construction time diff --git a/packages/db/src/indexes/base-index.ts b/packages/db/src/indexes/base-index.ts index 9cb4216880..fb27afaf71 100644 --- a/packages/db/src/indexes/base-index.ts +++ b/packages/db/src/indexes/base-index.ts @@ -38,15 +38,18 @@ export const IndexOperation = comparisonFunctions */ export type IndexOperation = (typeof comparisonFunctions)[number] -/** - * Statistics about index usage and performance - */ -export interface IndexStats { - readonly entryCount: number - readonly lookupCount: number - readonly averageLookupTime: number - readonly lastUpdated: Date -} +/** The read-side surface consumers use on a resolved (possibly reversed) index. */ +export type IndexReader = Pick< + IndexInterface, + | `lookup` + | `rangeQuery` + | `take` + | `takeFromStart` + | `keyCount` + | `supports` + | `supportsRangeOptimization` + | `canOptimizeRangeFor` +> export interface IndexInterface< TKey extends string | number = string | number, @@ -68,13 +71,13 @@ export interface IndexInterface< take: ( n: number, - from: TKey, + from: unknown, filterFn?: (key: TKey) => boolean, ) => Array takeFromStart: (n: number, filterFn?: (key: TKey) => boolean) => Array takeReversed: ( n: number, - from: TKey, + from: unknown, filterFn?: (key: TKey) => boolean, ) => Array takeReversedFromEnd: ( @@ -83,12 +86,6 @@ export interface IndexInterface< ) => Array get keyCount(): number - get orderedEntriesArray(): Array<[any, Set]> - get orderedEntriesArrayReversed(): Array<[any, Set]> - - get indexedKeysSet(): Set - get valueMapData(): Map> - supports: (operation: IndexOperation) => boolean /** @@ -100,11 +97,16 @@ export interface IndexInterface< */ get supportsRangeOptimization(): boolean + /** + * Whether the live values in this index share the predicate operand's + * relational domain. Mixed domains can sort differently in the index and + * WHERE evaluator, which can make a range lookup omit matching rows. + */ + canOptimizeRangeFor?: (value: unknown) => boolean + matchesField: (fieldPath: Array) => boolean matchesCompareOptions: (compareOptions: CompareOptions) => boolean matchesDirection: (direction: OrderByDirection) => boolean - - getStats: () => IndexStats } /** @@ -117,10 +119,6 @@ export abstract class BaseIndex< public readonly name?: string public readonly expression: BasicExpression public abstract readonly supportedOperations: Set - - protected lookupCount = 0 - protected totalLookupTime = 0 - protected lastUpdated = new Date() protected compareOptions: CompareOptions private compiledIndexEvaluator: CompiledSingleRowExpression | undefined /** @@ -128,6 +126,7 @@ export abstract class BaseIndex< * ordering may not match the WHERE evaluator's relational operators. */ protected hasCustomComparator = false + private rangeValueDomains = new Map() constructor( id: number, @@ -151,7 +150,7 @@ export abstract class BaseIndex< abstract lookup(operation: IndexOperation, value: any): Set abstract take( n: number, - from: TKey, + from: unknown, filterFn?: (key: TKey) => boolean, ): Array abstract takeFromStart( @@ -160,7 +159,7 @@ export abstract class BaseIndex< ): Array abstract takeReversed( n: number, - from: TKey, + from: unknown, filterFn?: (key: TKey) => boolean, ): Array abstract takeReversedFromEnd( @@ -171,13 +170,22 @@ export abstract class BaseIndex< abstract equalityLookup(value: any): Set abstract inArrayLookup(values: Array): Set abstract rangeQuery(options: RangeQueryOptions): Set - abstract rangeQueryReversed(options: RangeQueryOptions): Set - abstract get orderedEntriesArray(): Array<[any, Set]> - abstract get orderedEntriesArrayReversed(): Array<[any, Set]> - abstract get indexedKeysSet(): Set - abstract get valueMapData(): Map> // Common methods + rangeQueryReversed(options: RangeQueryOptions = {}): Set { + const { from, to, fromInclusive = true, toInclusive = true } = options + const reversed: RangeQueryOptions = {} + if (`to` in options) { + reversed.from = to + reversed.fromInclusive = toInclusive + } + if (`from` in options) { + reversed.to = from + reversed.toInclusive = fromInclusive + } + return this.rangeQuery(reversed) + } + supports(operation: IndexOperation): boolean { return this.supportedOperations.has(operation) } @@ -186,6 +194,37 @@ export abstract class BaseIndex< return !this.hasCustomComparator } + protected addRangeValue(value: unknown): void { + const domain = rangeValueDomain(value) + if (domain === undefined) return + this.rangeValueDomains.set( + domain, + (this.rangeValueDomains.get(domain) ?? 0) + 1, + ) + } + + protected removeRangeValue(value: unknown): void { + const domain = rangeValueDomain(value) + if (domain === undefined) return + const count = this.rangeValueDomains.get(domain) + if (count === undefined) return + if (count === 1) this.rangeValueDomains.delete(domain) + else this.rangeValueDomains.set(domain, count - 1) + } + + protected clearRangeValues(): void { + this.rangeValueDomains.clear() + } + + canOptimizeRangeFor(value: unknown): boolean { + const domain = rangeValueDomain(value) + if (domain === undefined) return true + if (!isNativeRangeDomain(domain)) return false + return [...this.rangeValueDomains.keys()].every( + (storedDomain) => storedDomain === domain, + ) + } + matchesField(fieldPath: Array): boolean { return ( this.expression.type === `ref` && @@ -231,16 +270,6 @@ export abstract class BaseIndex< return this.compareOptions.direction === direction } - getStats(): IndexStats { - return { - entryCount: this.keyCount, - lookupCount: this.lookupCount, - averageLookupTime: - this.lookupCount > 0 ? this.totalLookupTime / this.lookupCount : 0, - lastUpdated: this.lastUpdated, - } - } - protected abstract initialize(options?: any): void protected evaluateIndexExpression(item: any): any { @@ -248,16 +277,22 @@ export abstract class BaseIndex< compileSingleRowExpression(this.expression)) return evaluator(item as Record) } +} - protected trackLookup(startTime: number): void { - const duration = performance.now() - startTime - this.lookupCount++ - this.totalLookupTime += duration - } +function rangeValueDomain(value: unknown): string | undefined { + if (value == null) return undefined + if (value instanceof Date) return `date` + return typeof value +} - protected updateTimestamp(): void { - this.lastUpdated = new Date() - } +function isNativeRangeDomain(domain: string): boolean { + return ( + domain === `number` || + domain === `bigint` || + domain === `boolean` || + domain === `string` || + domain === `date` + ) } /** diff --git a/packages/db/src/indexes/basic-index.ts b/packages/db/src/indexes/basic-index.ts index 8eac6f926a..8f47e1b665 100644 --- a/packages/db/src/indexes/basic-index.ts +++ b/packages/db/src/indexes/basic-index.ts @@ -1,10 +1,12 @@ +import { compareKeys } from '@tanstack/db-ivm' import { areSameValueZeroEqual, defaultComparator, + makeComparator, normalizeValue, } from '../utils/comparison.js' import { - deleteInSortedArray, + compareKeysReversed, findInsertPositionInArray, } from '../utils/array-utils.js' import { BaseIndex } from './base-index.js' @@ -68,11 +70,11 @@ export class BasicIndex< options?: any, ) { super(id, expression, name, options) - this.compareFn = options?.compareFn ?? defaultComparator - this.hasCustomComparator = options?.compareFn != null if (options?.compareOptions) { this.compareOptions = options!.compareOptions } + this.compareFn = options?.compareFn ?? makeComparator(this.compareOptions) + this.hasCustomComparator = options?.compareFn != null } protected initialize(_options?: BasicIndexOptions): void {} @@ -94,9 +96,9 @@ export class BasicIndex< const normalizedValue = normalizeValue(indexedValue) this.addToBucket(key, normalizedValue) + this.addRangeValue(indexedValue) this.indexedKeys.add(key) - this.updateTimestamp() } private addToBucket(key: TKey, normalizedValue: unknown): void { @@ -131,16 +133,15 @@ export class BasicIndex< error, ) this.indexedKeys.delete(key) - this.updateTimestamp() return } const normalizedValue = normalizeValue(indexedValue) this.removeFromBucket(key, normalizedValue) + this.removeRangeValue(indexedValue) this.indexedKeys.delete(key) - this.updateTimestamp() } private removeFromBucket(key: TKey, normalizedValue: unknown): void { @@ -151,7 +152,27 @@ export class BasicIndex< if (keySet.size === 0) { // No more keys for this value, remove from map and sorted array this.valueMap.delete(normalizedValue) - deleteInSortedArray(this.sortedValues, normalizedValue, this.compareFn) + let sortedIndex = findInsertPositionInArray( + this.sortedValues, + normalizedValue, + this.compareFn, + ) + // Distinct equality keys may share one comparator position. + while ( + sortedIndex < this.sortedValues.length && + this.compareFn(this.sortedValues[sortedIndex], normalizedValue) === 0 + ) { + if ( + areSameValueZeroEqual( + this.sortedValues[sortedIndex], + normalizedValue, + ) + ) { + this.sortedValues.splice(sortedIndex, 1) + break + } + sortedIndex++ + } } } } @@ -160,29 +181,34 @@ export class BasicIndex< * Updates a value in the index */ update(key: TKey, oldItem: any, newItem: any): void { - let oldValue: unknown - let newValue: unknown + let oldIndexedValue: unknown + let newIndexedValue: unknown try { - oldValue = normalizeValue(this.evaluateIndexExpression(oldItem)) - newValue = normalizeValue(this.evaluateIndexExpression(newItem)) + oldIndexedValue = this.evaluateIndexExpression(oldItem) + newIndexedValue = this.evaluateIndexExpression(newItem) } catch { this.remove(key, oldItem) this.add(key, newItem) return } + const oldValue = normalizeValue(oldIndexedValue) + const newValue = normalizeValue(newIndexedValue) if ( areSameValueZeroEqual(oldValue, newValue) && this.valueMap.get(newValue)?.has(key) && this.indexedKeys.has(key) ) { + this.removeRangeValue(oldIndexedValue) + this.addRangeValue(newIndexedValue) return } this.removeFromBucket(key, oldValue) + this.removeRangeValue(oldIndexedValue) this.addToBucket(key, newValue) + this.addRangeValue(newIndexedValue) this.indexedKeys.add(key) - this.updateTimestamp() } /** @@ -204,6 +230,7 @@ export class BasicIndex< ) } entriesArray.push({ key, value: normalizeValue(indexedValue) }) + this.addRangeValue(indexedValue) this.indexedKeys.add(key) } @@ -218,8 +245,6 @@ export class BasicIndex< // Build sorted array from unique values this.sortedValues = Array.from(this.valueMap.keys()).sort(this.compareFn) - - this.updateTimestamp() } /** @@ -229,15 +254,13 @@ export class BasicIndex< this.valueMap.clear() this.sortedValues = [] this.indexedKeys.clear() - this.updateTimestamp() + this.clearRangeValues() } /** * Performs a lookup operation */ lookup(operation: IndexOperation, value: any): Set { - const startTime = performance.now() - let result: Set switch (operation) { @@ -262,8 +285,6 @@ export class BasicIndex< default: throw new Error(`Operation ${operation} not supported by BasicIndex`) } - - this.trackLookup(startTime) return result } @@ -295,17 +316,20 @@ export class BasicIndex< const normalizedFrom = normalizeValue(from) const normalizedTo = normalizeValue(to) + const hasFrom = `from` in options + const hasTo = `to` in options // Find start index let startIdx = 0 - if (normalizedFrom !== undefined) { + if (hasFrom) { startIdx = findInsertPositionInArray( this.sortedValues, normalizedFrom, this.compareFn, ) - // If not inclusive and we found exact match, skip it - if ( + // Comparator-equal values form one range boundary even when they are + // distinct equality keys. + while ( !fromInclusive && startIdx < this.sortedValues.length && this.compareFn(this.sortedValues[startIdx], normalizedFrom) === 0 @@ -316,14 +340,14 @@ export class BasicIndex< // Find end index let endIdx = this.sortedValues.length - if (normalizedTo !== undefined) { + if (hasTo) { endIdx = findInsertPositionInArray( this.sortedValues, normalizedTo, this.compareFn, ) - // If inclusive and we found the value, include it - if ( + // Include the whole comparator group at an inclusive upper boundary. + while ( toInclusive && endIdx < this.sortedValues.length && this.compareFn(this.sortedValues[endIdx], normalizedTo) === 0 @@ -343,71 +367,25 @@ export class BasicIndex< return result } - /** - * Performs a reversed range query - */ - rangeQueryReversed(options: RangeQueryOptions = {}): Set { - const { from, to, fromInclusive = true, toInclusive = true } = options - - // Swap from/to and fromInclusive/toInclusive to handle reversed ranges - // If to is undefined, we want to start from the end (max value) - // If from is undefined, we want to end at the beginning (min value) - const swappedFrom = - to ?? - (this.sortedValues.length > 0 - ? this.sortedValues[this.sortedValues.length - 1] - : undefined) - const swappedTo = - from ?? (this.sortedValues.length > 0 ? this.sortedValues[0] : undefined) - - return this.rangeQuery({ - from: swappedFrom, - to: swappedTo, - fromInclusive: toInclusive, - toInclusive: fromInclusive, - }) - } - /** * Returns the next n items in sorted order */ - take(n: number, from?: any, filterFn?: (key: TKey) => boolean): Array { - const result: Array = [] - - let startIdx = 0 - if (from !== undefined) { - const normalizedFrom = normalizeValue(from) - startIdx = findInsertPositionInArray( - this.sortedValues, - normalizedFrom, - this.compareFn, - ) - // Skip past the 'from' value (exclusive) - while ( - startIdx < this.sortedValues.length && - this.compareFn(this.sortedValues[startIdx], normalizedFrom) <= 0 - ) { - startIdx++ - } - } - - for ( - let i = startIdx; - i < this.sortedValues.length && result.length < n; - i++ + take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { + const normalizedFrom = normalizeValue(from) + let startIdx = findInsertPositionInArray( + this.sortedValues, + normalizedFrom, + this.compareFn, + ) + // Skip past the 'from' value (exclusive) + while ( + startIdx < this.sortedValues.length && + this.compareFn(this.sortedValues[startIdx], normalizedFrom) <= 0 ) { - const keys = this.valueMap.get(this.sortedValues[i]) - if (keys) { - for (const key of keys) { - if (result.length >= n) break - if (!filterFn || filterFn(key)) { - result.push(key) - } - } - } + startIdx++ } - return result + return this.takeFromIndex(n, startIdx, 1, filterFn) } /** @@ -415,61 +393,32 @@ export class BasicIndex< */ takeReversed( n: number, - from?: any, + from: any, filterFn?: (key: TKey) => boolean, ): Array { - const result: Array = [] - - let startIdx = this.sortedValues.length - 1 - if (from !== undefined) { - const normalizedFrom = normalizeValue(from) - startIdx = - findInsertPositionInArray( - this.sortedValues, - normalizedFrom, - this.compareFn, - ) - 1 - // Skip past the 'from' value (exclusive) - while ( - startIdx >= 0 && - this.compareFn(this.sortedValues[startIdx], normalizedFrom) >= 0 - ) { - startIdx-- - } - } - - for (let i = startIdx; i >= 0 && result.length < n; i--) { - const keys = this.valueMap.get(this.sortedValues[i]) - if (keys) { - for (const key of keys) { - if (result.length >= n) break - if (!filterFn || filterFn(key)) { - result.push(key) - } - } - } + const normalizedFrom = normalizeValue(from) + let startIdx = + findInsertPositionInArray( + this.sortedValues, + normalizedFrom, + this.compareFn, + ) - 1 + // Skip past the 'from' value (exclusive) + while ( + startIdx >= 0 && + this.compareFn(this.sortedValues[startIdx], normalizedFrom) >= 0 + ) { + startIdx-- } - return result + return this.takeFromIndex(n, startIdx, -1, filterFn) } /** * Returns the first n items in sorted order (from the start) */ takeFromStart(n: number, filterFn?: (key: TKey) => boolean): Array { - const result: Array = [] - for (let i = 0; i < this.sortedValues.length && result.length < n; i++) { - const keys = this.valueMap.get(this.sortedValues[i]) - if (keys) { - for (const key of keys) { - if (result.length >= n) break - if (!filterFn || filterFn(key)) { - result.push(key) - } - } - } - } - return result + return this.takeFromIndex(n, 0, 1, filterFn) } /** @@ -478,21 +427,39 @@ export class BasicIndex< takeReversedFromEnd( n: number, filterFn?: (key: TKey) => boolean, + ): Array { + return this.takeFromIndex(n, this.sortedValues.length - 1, -1, filterFn) + } + + private takeFromIndex( + n: number, + startIndex: number, + step: 1 | -1, + filterFn?: (key: TKey) => boolean, ): Array { const result: Array = [] - for ( - let i = this.sortedValues.length - 1; - i >= 0 && result.length < n; - i-- + let index = startIndex + while ( + index >= 0 && + index < this.sortedValues.length && + result.length < n ) { - const keys = this.valueMap.get(this.sortedValues[i]) - if (keys) { - for (const key of keys) { - if (result.length >= n) break - if (!filterFn || filterFn(key)) { - result.push(key) - } + const groupValue = this.sortedValues[index] + const groupKeys: Array = [] + do { + for (const key of this.valueMap.get(this.sortedValues[index]) ?? []) { + groupKeys.push(key) } + index += step + } while ( + index >= 0 && + index < this.sortedValues.length && + this.compareFn(this.sortedValues[index], groupValue) === 0 + ) + groupKeys.sort(step === 1 ? compareKeys : compareKeysReversed) + for (const key of groupKeys) { + if (filterFn?.(key) ?? true) result.push(key) + if (result.length >= n) break } } return result @@ -514,29 +481,4 @@ export class BasicIndex< return result } - - // Getter methods for testing/compatibility - get indexedKeysSet(): Set { - return this.indexedKeys - } - - get orderedEntriesArray(): Array<[any, Set]> { - return this.sortedValues.map((value) => [ - value, - this.valueMap.get(value) ?? new Set(), - ]) - } - - get orderedEntriesArrayReversed(): Array<[any, Set]> { - const result: Array<[any, Set]> = [] - for (let i = this.sortedValues.length - 1; i >= 0; i--) { - const value = this.sortedValues[i] - result.push([value, this.valueMap.get(value) ?? new Set()]) - } - return result - } - - get valueMapData(): Map> { - return this.valueMap - } } diff --git a/packages/db/src/indexes/btree-index.ts b/packages/db/src/indexes/btree-index.ts index 6379b91b52..ecafef5dfe 100644 --- a/packages/db/src/indexes/btree-index.ts +++ b/packages/db/src/indexes/btree-index.ts @@ -1,9 +1,11 @@ import { compareKeys } from '@tanstack/db-ivm' +import { compareKeysReversed } from '../utils/array-utils.js' import { BTree } from '../utils/btree.js' import { areSameValueZeroEqual, defaultComparator, denormalizeUndefined, + makeComparator, normalizeForBTree, } from '../utils/comparison.js' import { BaseIndex } from './base-index.js' @@ -29,6 +31,12 @@ export interface RangeQueryOptions { toInclusive?: boolean } +type OrderedBucket = { + representative: unknown + exactValues: Set + keys: Set +} + /** * B+Tree index for sorted data with range queries * This maintains items in sorted order and provides efficient range operations @@ -46,10 +54,13 @@ export class BTreeIndex< ]) // Internal data structures - private to hide implementation details - // The `orderedEntries` B+ tree is used for efficient range queries - // The `valueMap` is used for O(1) lookups of PKs by indexed value - private orderedEntries: BTree // we don't associate values with the keys of the B+ tree (the keys are indexed values) - private valueMap = new Map>() // instead we store a mapping of indexed values to a set of PKs + // The `orderedEntries` B+ tree groups values that occupy the same comparator + // position. The `valueMap` keeps exact values separate for equality lookups. + private orderedEntries: BTree> + private valueMap = new Map< + unknown, + { keys: Set; ordered: OrderedBucket } + >() private indexedKeys = new Set() private compareFn: (a: any, b: any) => number = defaultComparator @@ -61,8 +72,13 @@ export class BTreeIndex< ) { super(id, expression, name, options) + if (options?.compareOptions) { + this.compareOptions = options!.compareOptions + } + // Get the base compare function - const baseCompareFn = options?.compareFn ?? defaultComparator + const baseCompareFn = + options?.compareFn ?? makeComparator(this.compareOptions) this.hasCustomComparator = options?.compareFn != null // Wrap it to denormalize sentinels before comparison @@ -71,9 +87,6 @@ export class BTreeIndex< this.compareFn = (a: any, b: any) => baseCompareFn(denormalizeUndefined(a), denormalizeUndefined(b)) - if (options?.compareOptions) { - this.compareOptions = options!.compareOptions - } this.orderedEntries = new BTree(this.compareFn) } @@ -96,22 +109,35 @@ export class BTreeIndex< const normalizedValue = normalizeForBTree(indexedValue) this.addToBucket(key, normalizedValue) + this.addRangeValue(indexedValue) this.indexedKeys.add(key) - this.updateTimestamp() } private addToBucket(key: TKey, normalizedValue: unknown): void { - const keySet = this.valueMap.get(normalizedValue) - if (keySet) { - // Add to existing set - keySet.add(key) + const exact = this.valueMap.get(normalizedValue) + if (exact) { + exact.keys.add(key) + exact.ordered.keys.add(key) + return + } + + let orderedBucket = this.orderedEntries.get(normalizedValue) + if (orderedBucket) { + orderedBucket.keys.add(key) + orderedBucket.exactValues.add(normalizedValue) } else { - // Create new set for this value - const newKeySet = new Set([key]) - this.valueMap.set(normalizedValue, newKeySet) - this.orderedEntries.set(normalizedValue, undefined) + orderedBucket = { + representative: normalizedValue, + exactValues: new Set([normalizedValue]), + keys: new Set([key]), + } + this.orderedEntries.set(normalizedValue, orderedBucket) } + this.valueMap.set(normalizedValue, { + keys: new Set([key]), + ordered: orderedBucket, + }) } /** @@ -133,23 +159,30 @@ export class BTreeIndex< const normalizedValue = normalizeForBTree(indexedValue) this.removeFromBucket(key, normalizedValue) + this.removeRangeValue(indexedValue) this.indexedKeys.delete(key) - this.updateTimestamp() } private removeFromBucket(key: TKey, normalizedValue: unknown): void { - const keySet = this.valueMap.get(normalizedValue) - if (keySet) { - keySet.delete(key) - - // If set is now empty, remove the entry entirely - if (keySet.size === 0) { - this.valueMap.delete(normalizedValue) - - // Remove from ordered entries - this.orderedEntries.delete(normalizedValue) - } + const exact = this.valueMap.get(normalizedValue) + if (!exact || !exact.keys.delete(key)) return + const removedExactValue = exact.keys.size === 0 + if (removedExactValue) this.valueMap.delete(normalizedValue) + const orderedBucket = exact.ordered + orderedBucket.keys.delete(key) + if (removedExactValue) orderedBucket.exactValues.delete(normalizedValue) + + if (orderedBucket.keys.size === 0) { + this.orderedEntries.delete(normalizedValue) + } else if ( + removedExactValue && + areSameValueZeroEqual(orderedBucket.representative, normalizedValue) + ) { + this.orderedEntries.delete(normalizedValue) + const representative = orderedBucket.exactValues.values().next().value + orderedBucket.representative = representative + this.orderedEntries.set(representative, orderedBucket) } } @@ -157,28 +190,33 @@ export class BTreeIndex< * Updates a value in the index */ update(key: TKey, oldItem: any, newItem: any): void { - let oldValue: unknown - let newValue: unknown + let oldIndexedValue: unknown + let newIndexedValue: unknown try { - oldValue = normalizeForBTree(this.evaluateIndexExpression(oldItem)) - newValue = normalizeForBTree(this.evaluateIndexExpression(newItem)) + oldIndexedValue = this.evaluateIndexExpression(oldItem) + newIndexedValue = this.evaluateIndexExpression(newItem) } catch { this.remove(key, oldItem) this.add(key, newItem) return } + const oldValue = normalizeForBTree(oldIndexedValue) + const newValue = normalizeForBTree(newIndexedValue) if ( areSameValueZeroEqual(oldValue, newValue) && - this.valueMap.get(newValue)?.has(key) + this.valueMap.get(newValue)?.keys.has(key) ) { + this.removeRangeValue(oldIndexedValue) + this.addRangeValue(newIndexedValue) return } this.removeFromBucket(key, oldValue) + this.removeRangeValue(oldIndexedValue) this.addToBucket(key, newValue) + this.addRangeValue(newIndexedValue) this.indexedKeys.add(key) - this.updateTimestamp() } /** @@ -199,15 +237,13 @@ export class BTreeIndex< this.orderedEntries.clear() this.valueMap.clear() this.indexedKeys.clear() - this.updateTimestamp() + this.clearRangeValues() } /** * Performs a lookup operation */ lookup(operation: IndexOperation, value: any): Set { - const startTime = performance.now() - let result: Set switch (operation) { @@ -232,8 +268,6 @@ export class BTreeIndex< default: throw new Error(`Operation ${operation} not supported by BTreeIndex`) } - - this.trackLookup(startTime) return result } @@ -251,7 +285,7 @@ export class BTreeIndex< */ equalityLookup(value: any): Set { const normalizedValue = normalizeForBTree(value) - return new Set(this.valueMap.get(normalizedValue) ?? []) + return new Set(this.valueMap.get(normalizedValue)?.keys ?? []) } /** @@ -276,7 +310,7 @@ export class BTreeIndex< fromKey, toKey, toInclusive, - (indexedValue, _) => { + (indexedValue, bucket) => { // Only exclude the boundary when an exclusive lower bound was // actually provided. Without a `from` bound, `fromKey` defaults to // the minimum key and must not be dropped. Compare against the @@ -292,33 +326,13 @@ export class BTreeIndex< return } - const keys = this.valueMap.get(indexedValue) - if (keys) { - keys.forEach((key) => result.add(key)) - } + bucket.keys.forEach((key) => result.add(key)) }, ) return result } - /** - * Performs a reversed range query - */ - rangeQueryReversed(options: RangeQueryOptions = {}): Set { - const { from, to, fromInclusive = true, toInclusive = true } = options - const hasFrom = `from` in options - const hasTo = `to` in options - - // Swap from/to for reversed query, respecting explicit undefined values - return this.rangeQuery({ - from: hasTo ? to : this.orderedEntries.maxKey(), - to: hasFrom ? from : this.orderedEntries.minKey(), - fromInclusive: toInclusive, - toInclusive: fromInclusive, - }) - } - /** * Internal method for taking items from the index. * @param n - The number of items to return @@ -329,32 +343,25 @@ export class BTreeIndex< */ private takeInternal( n: number, - nextPair: (k?: any) => [any, any] | undefined, + nextPair: (k?: any) => [any, OrderedBucket] | undefined, from: any, filterFn?: (key: TKey) => boolean, reversed: boolean = false, ): Array { - const keysInResult: Set = new Set() const result: Array = [] - let pair: [any, any] | undefined + let pair: [any, OrderedBucket] | undefined let key = from // Use as-is - it's already normalized by the caller + // Every key owns exactly one bucket, so the walk never repeats a key. while ((pair = nextPair(key)) !== undefined && result.length < n) { key = pair[0] - const keys = this.valueMap.get(key) as - | Set> - | undefined - if (keys && keys.size > 0) { - // Sort keys for deterministic order, reverse if needed - const sorted = Array.from(keys).sort(compareKeys) - if (reversed) sorted.reverse() - for (const ks of sorted) { - if (result.length >= n) break - if (!keysInResult.has(ks) && (filterFn?.(ks) ?? true)) { - result.push(ks) - keysInResult.add(ks) - } - } + // Sort keys for deterministic order within a comparator position. + const sorted = Array.from(pair[1].keys).sort( + reversed ? compareKeysReversed : compareKeys, + ) + for (const ks of sorted) { + if (result.length >= n) break + if (filterFn?.(ks) ?? true) result.push(ks) } } @@ -426,7 +433,7 @@ export class BTreeIndex< for (const value of values) { const normalizedValue = normalizeForBTree(value) - const keys = this.valueMap.get(normalizedValue) + const keys = this.valueMap.get(normalizedValue)?.keys if (keys) { keys.forEach((key) => result.add(key)) } @@ -434,34 +441,4 @@ export class BTreeIndex< return result } - - // Getter methods for testing compatibility - get indexedKeysSet(): Set { - return this.indexedKeys - } - - get orderedEntriesArray(): Array<[any, Set]> { - return this.orderedEntries - .keysArray() - .map((key) => [ - denormalizeUndefined(key), - this.valueMap.get(key) ?? new Set(), - ]) - } - - get orderedEntriesArrayReversed(): Array<[any, Set]> { - return this.takeReversedFromEnd(this.orderedEntries.size).map((key) => [ - denormalizeUndefined(key), - this.valueMap.get(key) ?? new Set(), - ]) - } - - get valueMapData(): Map> { - // Return a new Map with denormalized keys - const result = new Map>() - for (const [key, value] of this.valueMap) { - result.set(denormalizeUndefined(key), value) - } - return result - } } diff --git a/packages/db/src/indexes/reverse-index.ts b/packages/db/src/indexes/reverse-index.ts index 6ca61636e1..3cfda9f2e4 100644 --- a/packages/db/src/indexes/reverse-index.ts +++ b/packages/db/src/indexes/reverse-index.ts @@ -1,11 +1,9 @@ -import type { CompareOptions } from '../query/builder/types' -import type { OrderByDirection } from '../query/ir' -import type { IndexInterface, IndexOperation, IndexStats } from './base-index' +import type { IndexInterface, IndexOperation, IndexReader } from './base-index' import type { RangeQueryOptions } from './btree-index' export class ReverseIndex< TKey extends string | number, -> implements IndexInterface { +> implements IndexReader { private originalIndex: IndexInterface constructor(index: IndexInterface) { @@ -32,10 +30,6 @@ export class ReverseIndex< return this.originalIndex.rangeQueryReversed(options) } - rangeQueryReversed(options: RangeQueryOptions = {}): Set { - return this.originalIndex.rangeQuery(options) - } - take(n: number, from: any, filterFn?: (key: TKey) => boolean): Array { return this.originalIndex.takeReversed(n, from, filterFn) } @@ -44,29 +38,6 @@ export class ReverseIndex< return this.originalIndex.takeReversedFromEnd(n, filterFn) } - takeReversed( - n: number, - from: any, - filterFn?: (key: TKey) => boolean, - ): Array { - return this.originalIndex.take(n, from, filterFn) - } - - takeReversedFromEnd( - n: number, - filterFn?: (key: TKey) => boolean, - ): Array { - return this.originalIndex.takeFromStart(n, filterFn) - } - - get orderedEntriesArray(): Array<[any, Set]> { - return this.originalIndex.orderedEntriesArrayReversed - } - - get orderedEntriesArrayReversed(): Array<[any, Set]> { - return this.originalIndex.orderedEntriesArray - } - // All operations below delegate to the original index supports(operation: IndexOperation): boolean { @@ -77,59 +48,11 @@ export class ReverseIndex< return this.originalIndex.supportsRangeOptimization } - matchesField(fieldPath: Array): boolean { - return this.originalIndex.matchesField(fieldPath) - } - - matchesCompareOptions(compareOptions: CompareOptions): boolean { - return this.originalIndex.matchesCompareOptions(compareOptions) - } - - matchesDirection(direction: OrderByDirection): boolean { - return this.originalIndex.matchesDirection(direction) - } - - getStats(): IndexStats { - return this.originalIndex.getStats() - } - - add(key: TKey, item: any): void { - this.originalIndex.add(key, item) - } - - remove(key: TKey, item: any): void { - this.originalIndex.remove(key, item) - } - - update(key: TKey, oldItem: any, newItem: any): void { - this.originalIndex.update(key, oldItem, newItem) - } - - build(entries: Iterable<[TKey, any]>): void { - this.originalIndex.build(entries) - } - - clear(): void { - this.originalIndex.clear() + canOptimizeRangeFor(value: unknown): boolean { + return this.originalIndex.canOptimizeRangeFor?.(value) ?? true } get keyCount(): number { return this.originalIndex.keyCount } - - equalityLookup(value: any): Set { - return this.originalIndex.equalityLookup(value) - } - - inArrayLookup(values: Array): Set { - return this.originalIndex.inArrayLookup(values) - } - - get indexedKeysSet(): Set { - return this.originalIndex.indexedKeysSet - } - - get valueMapData(): Map> { - return this.originalIndex.valueMapData - } } diff --git a/packages/db/src/live-query-observer.ts b/packages/db/src/live-query-observer.ts index 735195816a..13d31c9e9e 100644 --- a/packages/db/src/live-query-observer.ts +++ b/packages/db/src/live-query-observer.ts @@ -766,6 +766,7 @@ class LiveQueryObserverImpl< private flushPublications(deliver = true): void { if (this.dispatching) return + let failure: { error: unknown } | undefined this.dispatching = true try { // A dispose() during dispatch empties the queue, ending this loop. @@ -793,14 +794,19 @@ class LiveQueryObserverImpl< // one added later does not. Late-subscriber seeds use the same queue. if (deliver) { for (const subRecord of publication.targets) { - if (this.disposed) return - subRecord.listener(publication.changes) + if (this.disposed) break + try { + subRecord.listener(publication.changes) + } catch (error) { + failure ??= { error } + } } } } } finally { this.dispatching = false } + if (failure) throw failure.error } preload(): Promise { diff --git a/packages/db/src/live-query-window-controller.ts b/packages/db/src/live-query-window-controller.ts index 0acd53a260..1736eaa901 100644 --- a/packages/db/src/live-query-window-controller.ts +++ b/packages/db/src/live-query-window-controller.ts @@ -189,10 +189,13 @@ class WindowCoordinator { }) } - isLeaseSatisfied(lease: symbol, minimumLimit: number): boolean { + getLeaseResult(lease: symbol, minimumLimit: number): WindowResult | false { const limit = this.leases.get(lease) if (limit === undefined || limit < minimumLimit) return false const desiredLimit = this.getDesiredLimit() + // getWindow reports settled state; the current lease may still be loading. + if (this.pending && this.pending.limit === desiredLimit) + return this.pending.promise const currentWindow = this.target.utils?.getWindow?.() return ( currentWindow === undefined || @@ -725,10 +728,12 @@ class LiveQueryWindowControllerImpl< fetchNextPage(): Promise { if (this.disposed) return Promise.resolve() - if (this.isFetchingNextPage && this.activeFetchPromise) { + if (this.activeFetchPromise) { return this.activeFetchPromise } - if (!this.getSnapshot().hasNextPage) return Promise.resolve() + const snapshot = this.getSnapshot() + const awaitingInitialLoad = snapshot.isLoading || snapshot.isIdle + if (!snapshot.hasNextPage && !awaitingInitialLoad) return Promise.resolve() let resolveFetch!: () => void let rejectFetch!: (error: unknown) => void @@ -740,7 +745,21 @@ class LiveQueryWindowControllerImpl< let request: Promise try { - request = this.requestPageCount(this.committedPageCount + 1, true) + const generation = this.windowGeneration + // An unpublished initial snapshot cannot establish that there is no next + // page. Keep one fetch pending, then decide from the settled first page. + request = awaitingInitialLoad + ? this.preload().then(() => { + if ( + this.disposed || + generation !== this.windowGeneration || + !this.getSnapshot().hasNextPage + ) { + return + } + return this.requestPageCount(this.committedPageCount + 1, true) + }) + : this.requestPageCount(this.committedPageCount + 1, true) } catch (error) { this.activeFetchPromise = null rejectFetch(error) @@ -769,10 +788,12 @@ class LiveQueryWindowControllerImpl< this.committedPageCount === 1 && !this.hasPaginationError && !this.isFetchingNextPage && + !this.activeFetchPromise && this.pendingWindowGeneration === undefined ) { return Promise.resolve() } + this.activeFetchPromise = null return this.requestPageCount(1, false) } @@ -909,11 +930,9 @@ class LiveQueryWindowControllerImpl< private ensureLeaseActive(pageCount: number): WindowResult { const minimumLimit = pageCount * this.pageSize + 1 - if ( - this.leaseActive && - this.coordinator?.isLeaseSatisfied(this.lease, minimumLimit) - ) { - return true + if (this.leaseActive) { + const result = this.coordinator?.getLeaseResult(this.lease, minimumLimit) + if (result) return result } return this.activateLease(pageCount) } diff --git a/packages/db/src/local-storage.ts b/packages/db/src/local-storage.ts index 7ab6b98449..be8e326a7f 100644 --- a/packages/db/src/local-storage.ts +++ b/packages/db/src/local-storage.ts @@ -433,6 +433,26 @@ export function localStorageCollectionOptions( return data ? new Blob([data]).size : 0 } + const persistMutations = ( + mutations: Array>>, + ): void => { + const staged = new Map(lastKnownData) + for (const mutation of mutations) { + if (mutation.type === `delete`) staged.delete(mutation.key) + else + staged.set(mutation.key, { + versionKey: generateUuid(), + data: mutation.modified, + }) + } + saveToStorage(staged) + // Sync and storage-event handling share this Map. Promote only after the + // write succeeds, so rejected mutations cannot contaminate a later save. + lastKnownData.clear() + for (const [key, value] of staged) lastKnownData.set(key, value) + sync.confirmOperationsSync(mutations) + } + /* * Create wrapper handlers for direct persistence operations that perform actual storage operations * Wraps the user's onInsert handler to also save changes to localStorage @@ -449,24 +469,7 @@ export function localStorageCollectionOptions( handlerResult = (await config.onInsert(params)) ?? {} } - // Always persist to storage - // Use lastKnownData (in-memory cache) instead of reading from storage - // Add new items with version keys - params.transaction.mutations.forEach((mutation) => { - // Use the engine's pre-computed key for consistency - const storedItem: StoredItem = { - versionKey: generateUuid(), - data: mutation.modified, - } - lastKnownData.set(mutation.key, storedItem) - }) - - // Save to storage - saveToStorage(lastKnownData) - - // Confirm mutations through sync interface (moves from optimistic to synced state) - // without reloading from storage - sync.confirmOperationsSync(params.transaction.mutations) + persistMutations(params.transaction.mutations) return handlerResult } @@ -483,24 +486,7 @@ export function localStorageCollectionOptions( handlerResult = (await config.onUpdate(params)) ?? {} } - // Always persist to storage - // Use lastKnownData (in-memory cache) instead of reading from storage - // Update items with new version keys - params.transaction.mutations.forEach((mutation) => { - // Use the engine's pre-computed key for consistency - const storedItem: StoredItem = { - versionKey: generateUuid(), - data: mutation.modified, - } - lastKnownData.set(mutation.key, storedItem) - }) - - // Save to storage - saveToStorage(lastKnownData) - - // Confirm mutations through sync interface (moves from optimistic to synced state) - // without reloading from storage - sync.confirmOperationsSync(params.transaction.mutations) + persistMutations(params.transaction.mutations) return handlerResult } @@ -512,20 +498,7 @@ export function localStorageCollectionOptions( handlerResult = (await config.onDelete(params)) ?? {} } - // Always persist to storage - // Use lastKnownData (in-memory cache) instead of reading from storage - // Remove items - params.transaction.mutations.forEach((mutation) => { - // Use the engine's pre-computed key for consistency - lastKnownData.delete(mutation.key) - }) - - // Save to storage - saveToStorage(lastKnownData) - - // Confirm mutations through sync interface (moves from optimistic to synced state) - // without reloading from storage - sync.confirmOperationsSync(params.transaction.mutations) + persistMutations(params.transaction.mutations) return handlerResult } @@ -579,33 +552,7 @@ export function localStorageCollectionOptions( } } - // Use lastKnownData (in-memory cache) instead of reading from storage - // Apply each mutation - for (const mutation of collectionMutations) { - // Use the engine's pre-computed key to avoid key derivation issues - switch (mutation.type) { - case `insert`: - case `update`: { - const storedItem: StoredItem> = { - versionKey: generateUuid(), - data: mutation.modified, - } - lastKnownData.set(mutation.key, storedItem) - break - } - case `delete`: { - lastKnownData.delete(mutation.key) - break - } - } - } - - // Save to storage - saveToStorage(lastKnownData) - - // Confirm the mutations in the collection to move them from optimistic to synced state - // This writes them through the sync interface to make them "synced" instead of "optimistic" - sync.confirmOperationsSync(collectionMutations) + persistMutations(collectionMutations) } const options = { diff --git a/packages/db/src/proxy.ts b/packages/db/src/proxy.ts index 57723e3cca..c1b43a0a9e 100644 --- a/packages/db/src/proxy.ts +++ b/packages/db/src/proxy.ts @@ -5,6 +5,14 @@ import { deepEquals, isTemporal } from './utils' +// Resolve draft handles before calling native Map/Set membership methods. +const draftCopies = new WeakMap() +function unwrapDraft(value: unknown): unknown { + return value !== null && typeof value === `object` + ? (draftCopies.get(value) ?? value) + : value +} + /** * Set of array methods that iterate with callbacks and may return elements. * Hoisted to module scope to avoid creating a new Set on every property access. @@ -39,11 +47,6 @@ const ARRAY_MODIFYING_METHODS = new Set([ `copyWithin`, ]) -/** - * Set of Map/Set methods that modify the collection in place. - */ -const MAP_SET_MODIFYING_METHODS = new Set([`set`, `delete`, `clear`, `add`]) - /** * Set of Map/Set iterator methods. */ @@ -245,234 +248,67 @@ function createModifyingMethodHandler( } /** - * Creates handlers for Map/Set iterator methods (entries, keys, values, forEach). - * Returns proxied values for iteration to enable change tracking. + * Use the native live iterator, but expose tracked values. Editing an entry + * changes its owned draft copy in place; it must not delete/reinsert a Set slot. */ function createMapSetIteratorHandler( methodName: string, prop: string | symbol, - methodFn: (...args: Array) => unknown, - target: Map | Set, changeTracker: ChangeTracker, + collectionProxy: unknown, memoizedCreateChangeProxy: ( obj: Record, - parent?: { - tracker: ChangeTracker> - prop: string | symbol - }, + parent?: ChangeParent, ) => { proxy: Record }, - markChanged: (tracker: ChangeTracker) => void, ): ((...args: Array) => unknown) | undefined { - const isIteratorMethod = - MAP_SET_ITERATOR_METHODS.has(methodName) || prop === Symbol.iterator - - if (!isIteratorMethod) { + if (!MAP_SET_ITERATOR_METHODS.has(methodName) && prop !== Symbol.iterator) { return undefined } - return function (this: unknown, ...args: Array) { - const result = methodFn.apply(changeTracker.copy_, args) + return (...args) => { + const copy = changeTracker.copy_ as Map | Set + const isMap = copy instanceof Map + if (isMap && methodName === `keys`) return copy.keys() + + const track = (value: unknown) => + isProxiableObject(value) + ? memoizedCreateChangeProxy(value, { + tracker: changeTracker as unknown as ChangeTracker< + Record + >, + prop: ``, + retainIdentity: true, + }).proxy + : value - // For forEach, wrap the callback to track changes if (methodName === `forEach`) { const callback = args[0] - if (typeof callback === `function`) { - const wrappedCallback = function ( - this: unknown, - value: unknown, - key: unknown, - collection: unknown, - ) { - const cbresult = callback.call(this, value, key, collection) - markChanged(changeTracker) - return cbresult - } - return methodFn.apply(target, [wrappedCallback, ...args.slice(1)]) - } + if (typeof callback !== `function`) + throw new TypeError(`forEach callback must be a function`) + return copy.forEach((value, key) => { + const tracked = track(value) + callback.call(args[1], tracked, isMap ? key : tracked, collectionProxy) + }) } - // For iterators (entries, keys, values, Symbol.iterator) - const isValueIterator = - methodName === `entries` || - methodName === `values` || - methodName === Symbol.iterator.toString() || - prop === Symbol.iterator - - if (isValueIterator) { - const originalIterator = result as Iterator - - // For values() iterator on Maps, create a value-to-key mapping - const valueToKeyMap = new Map() - if (methodName === `values` && target instanceof Map) { - for (const [key, mapValue] of ( - changeTracker.copy_ as unknown as Map - ).entries()) { - valueToKeyMap.set(mapValue, key) - } - } - - // For Set iterators, create an original-to-modified mapping - const originalToModifiedMap = new Map() - if (target instanceof Set) { - for (const setValue of ( - changeTracker.copy_ as unknown as Set - ).values()) { - originalToModifiedMap.set(setValue, setValue) + const entries = copy.entries() + const pairs = + methodName === `entries` || (isMap && prop === Symbol.iterator) + return { + next() { + const result = entries.next() + if (result.done) return result + const [key, value] = result.value + const tracked = track(value) + return { + done: false, + value: pairs ? [isMap ? key : tracked, tracked] : tracked, } - } - - // Return a wrapped iterator that proxies values - return { - next() { - const nextResult = originalIterator.next() - - if ( - !nextResult.done && - nextResult.value && - typeof nextResult.value === `object` - ) { - // For entries, the value is a [key, value] pair - if ( - methodName === `entries` && - Array.isArray(nextResult.value) && - nextResult.value.length === 2 - ) { - if ( - nextResult.value[1] && - typeof nextResult.value[1] === `object` - ) { - const mapKey = nextResult.value[0] - const mapParent = { - tracker: changeTracker as unknown as ChangeTracker< - Record - >, - prop: mapKey as string | symbol, - updateMap: (newValue: unknown) => { - if (changeTracker.copy_ instanceof Map) { - ;(changeTracker.copy_ as Map).set( - mapKey, - newValue, - ) - } - }, - } - const { proxy: valueProxy } = memoizedCreateChangeProxy( - nextResult.value[1] as Record, - mapParent as unknown as { - tracker: ChangeTracker> - prop: string | symbol - }, - ) - nextResult.value[1] = valueProxy - } - } else if ( - methodName === `values` || - methodName === Symbol.iterator.toString() || - prop === Symbol.iterator - ) { - // For Map values(), use the key mapping - if (methodName === `values` && target instanceof Map) { - const mapKey = valueToKeyMap.get(nextResult.value) - if (mapKey !== undefined) { - const mapParent = { - tracker: changeTracker as unknown as ChangeTracker< - Record - >, - prop: mapKey as string | symbol, - updateMap: (newValue: unknown) => { - if (changeTracker.copy_ instanceof Map) { - ;(changeTracker.copy_ as Map).set( - mapKey, - newValue, - ) - } - }, - } - const { proxy: valueProxy } = memoizedCreateChangeProxy( - nextResult.value as Record, - mapParent as unknown as { - tracker: ChangeTracker> - prop: string | symbol - }, - ) - nextResult.value = valueProxy - } - } else if (target instanceof Set) { - // For Set, track modifications - const setOriginalValue = nextResult.value - const setParent = { - tracker: changeTracker as unknown as ChangeTracker< - Record - >, - prop: setOriginalValue as unknown as string | symbol, - updateSet: (newValue: unknown) => { - if (changeTracker.copy_ instanceof Set) { - ;(changeTracker.copy_ as Set).delete( - setOriginalValue, - ) - ;(changeTracker.copy_ as Set).add(newValue) - originalToModifiedMap.set(setOriginalValue, newValue) - } - }, - } - const { proxy: valueProxy } = memoizedCreateChangeProxy( - nextResult.value as Record, - setParent as unknown as { - tracker: ChangeTracker> - prop: string | symbol - }, - ) - nextResult.value = valueProxy - } else { - // For other cases, use a symbol placeholder - const tempKey = Symbol(`iterator-value`) - const { proxy: valueProxy } = memoizedCreateChangeProxy( - nextResult.value as Record, - { - tracker: changeTracker as unknown as ChangeTracker< - Record - >, - prop: tempKey, - }, - ) - nextResult.value = valueProxy - } - } - } - - return nextResult - }, - [Symbol.iterator]() { - return this - }, - } + }, + [Symbol.iterator]() { + return this + }, } - - return result - } -} - -/** - * Simple debug utility that only logs when debug mode is enabled - * Set DEBUG to true in localStorage to enable debug logging - */ -function debugLog(...args: Array): void { - // Check if we're in a browser environment - const isBrowser = - typeof window !== `undefined` && typeof localStorage !== `undefined` - - // In browser, check localStorage for debug flag - if (isBrowser && localStorage.getItem(`DEBUG`) === `true`) { - console.log(`[proxy]`, ...args) - } - // In Node.js environment, check for environment variable (though this is primarily for browser) - else if ( - // true - !isBrowser && - typeof process !== `undefined` && - process.env.DEBUG === `true` - ) { - console.log(`[proxy]`, ...args) } } @@ -483,27 +319,19 @@ interface TypedArray { } // Update type for ChangeTracker +interface ChangeParent { + tracker: ChangeTracker> + prop: string | symbol + // Map/Set entries already belong to the parent's private copy. + retainIdentity?: boolean +} + interface ChangeTracker { originalObject: T modified: boolean copy_: T - proxyCount: number assigned_: Record - parent?: - | { - tracker: ChangeTracker> - prop: string | symbol - } - | { - tracker: ChangeTracker> - prop: string | symbol - updateMap: (newValue: unknown) => void - } - | { - tracker: ChangeTracker> - prop: unknown - updateSet: (newValue: unknown) => void - } + parent?: ChangeParent target: T } @@ -612,12 +440,6 @@ function deepClone( return clone as T } -let count = 0 -function getProxyCount() { - count += 1 - return count -} - /** * Creates a proxy that tracks changes to the target object * @@ -629,10 +451,7 @@ export function createChangeProxy< T extends Record, >( target: T, - parent?: { - tracker: ChangeTracker> - prop: string | symbol - }, + parent?: ChangeParent, ): { proxy: T @@ -644,15 +463,11 @@ export function createChangeProxy< TInner extends Record, >( innerTarget: TInner, - innerParent?: { - tracker: ChangeTracker> - prop: string | symbol - }, + innerParent?: ChangeParent, ): { proxy: TInner getChanges: () => Record } { - debugLog(`Object ID:`, innerTarget.constructor.name) if (changeProxyCache.has(innerTarget)) { return changeProxyCache.get(innerTarget) as { proxy: TInner @@ -670,21 +485,16 @@ export function createChangeProxy< const proxyCache = new Map() // Create a change tracker to track changes to the object + const valueCopies = new WeakMap() const changeTracker: ChangeTracker = { - copy_: deepClone(target), + copy_: parent?.retainIdentity ? target : deepClone(target, valueCopies), originalObject: deepClone(target), - proxyCount: getProxyCount(), modified: false, assigned_: {}, parent, target, // Store reference to the target object } - debugLog( - `createChangeProxy called for target`, - target, - changeTracker.proxyCount, - ) // Mark this object and all its ancestors as modified // Also propagate the actual changes up the chain function markChanged(state: ChangeTracker) { @@ -694,16 +504,7 @@ export function createChangeProxy< // Propagate the change up the parent chain if (state.parent) { - debugLog(`propagating change to parent`) - - // Check if this is a special Map parent with updateMap function - if (`updateMap` in state.parent) { - // Use the special updateMap function for Maps - state.parent.updateMap(state.copy_) - } else if (`updateSet` in state.parent) { - // Use the special updateSet function for Sets - state.parent.updateSet(state.copy_) - } else { + if (!state.parent.retainIdentity) { // Update parent's copy with this object's current state state.parent.tracker.copy_[state.parent.prop] = state.copy_ state.parent.tracker.assigned_[state.parent.prop] = true @@ -718,17 +519,22 @@ export function createChangeProxy< function checkIfReverted( state: ChangeTracker>, ): boolean { - debugLog( - `checkIfReverted called with assigned keys:`, - Object.keys(state.assigned_), - ) - + if (state.copy_ instanceof Map || state.copy_ instanceof Set) { + // Compare entry contents: these containers have no assigned properties. + return deepEquals( + Array.from(state.copy_), + Array.from( + state.originalObject as unknown as + | Map + | Set, + ), + ) + } // If there are no assigned properties, object is unchanged if ( Object.keys(state.assigned_).length === 0 && Object.getOwnPropertySymbols(state.assigned_).length === 0 ) { - debugLog(`No assigned properties, returning true`) return true } @@ -739,21 +545,12 @@ export function createChangeProxy< const currentValue = state.copy_[prop] const originalValue = (state.originalObject as any)[prop] - debugLog( - `Checking property ${String(prop)}, current:`, - currentValue, - `original:`, - originalValue, - ) - // If the value is not equal to original, something is still changed if (!deepEquals(currentValue, originalValue)) { - debugLog(`Property ${String(prop)} is different, returning false`) return false } } else if (state.assigned_[prop] === false) { // Property was deleted, so it's different from original - debugLog(`Property ${String(prop)} was deleted, returning false`) return false } } @@ -767,17 +564,14 @@ export function createChangeProxy< // If the value is not equal to original, something is still changed if (!deepEquals(currentValue, originalValue)) { - debugLog(`Symbol property is different, returning false`) return false } } else if (state.assigned_[sym] === false) { // Property was deleted, so it's different from original - debugLog(`Symbol property was deleted, returning false`) return false } } - debugLog(`All properties match original values, returning true`) // All assigned properties match their original values return true } @@ -785,49 +579,38 @@ export function createChangeProxy< // Update parent status based on child changes function checkParentStatus( parentState: ChangeTracker>, - childProp: string | symbol | unknown, ) { - debugLog(`checkParentStatus called for child prop:`, childProp) - // Check if all properties of the parent are reverted const isReverted = checkIfReverted(parentState) - debugLog(`Parent checkIfReverted returned:`, isReverted) if (isReverted) { - debugLog(`Parent is fully reverted, clearing tracking`) // If everything is reverted, clear the tracking parentState.modified = false parentState.assigned_ = {} // Continue up the chain if (parentState.parent) { - debugLog(`Continuing up the parent chain`) - checkParentStatus(parentState.parent.tracker, parentState.parent.prop) + checkParentStatus(parentState.parent.tracker) } } } // Create a proxy for the target object function createObjectProxy(obj: TObj): TObj { - debugLog(`createObjectProxy`, obj) // If we've already created a proxy for this object, return it if (proxyCache.has(obj)) { - debugLog(`proxyCache found match`) return proxyCache.get(obj) as TObj } // Create a proxy for the object const proxy = new Proxy(obj, { - get(ptarget, prop) { - debugLog(`get`, ptarget, prop) + get(ptarget, prop, receiver) { const value = changeTracker.copy_[prop as keyof T] ?? changeTracker.originalObject[prop as keyof T] const originalValue = changeTracker.originalObject[prop as keyof T] - debugLog(`value (at top of proxy get)`, value) - // If it's a getter, return the value directly const desc = Object.getOwnPropertyDescriptor(ptarget, prop) if (desc?.get) { @@ -872,7 +655,50 @@ export function createChangeProxy< if (ptarget instanceof Map || ptarget instanceof Set) { const methodName = prop.toString() - if (MAP_SET_MODIFYING_METHODS.has(methodName)) { + const resolveValue = (entry: unknown) => { + const raw = unwrapDraft(entry) + return raw !== null && typeof raw === `object` + ? (valueCopies.get(raw) ?? raw) + : raw + } + const copyValue = (entry: unknown) => { + const raw = unwrapDraft(entry) + return raw !== entry ? raw : deepClone(raw, valueCopies) + } + + if ( + methodName === `has` || + methodName === `delete` || + methodName === `add` || + methodName === `set` + ) { + return (...args: Array) => { + if (ptarget instanceof Set) + args[0] = + methodName === `add` + ? copyValue(args[0]) + : resolveValue(args[0]) + else if (methodName === `set`) args[1] = copyValue(args[1]) + const result = value.apply(ptarget, args) + if (methodName !== `has`) markChanged(changeTracker) + return result === ptarget ? receiver : result + } + } + + if (ptarget instanceof Map && methodName === `get`) { + return (key: unknown) => { + const entry = ptarget.get(key) + return isProxiableObject(entry) + ? memoizedCreateChangeProxy(entry, { + tracker: changeTracker, + prop: ``, + retainIdentity: true, + }).proxy + : entry + } + } + + if (methodName === `clear`) { return createModifyingMethodHandler( value, changeTracker, @@ -884,11 +710,9 @@ export function createChangeProxy< const iteratorHandler = createMapSetIteratorHandler( methodName, prop, - value, - ptarget, changeTracker, + receiver, memoizedCreateChangeProxy, - markChanged, ) if (iteratorHandler) { return iteratorHandler @@ -922,12 +746,6 @@ export function createChangeProxy< set(_sobj, prop, value) { const currentValue = changeTracker.copy_[prop as keyof T] - debugLog( - `set called for property ${String(prop)}, current:`, - currentValue, - `new:`, - value, - ) // Only track the change if the value is actually different if (!deepEquals(currentValue, value)) { @@ -935,48 +753,31 @@ export function createChangeProxy< // Important: Use the originalObject to get the true original value const originalValue = changeTracker.originalObject[prop as keyof T] const isRevertToOriginal = deepEquals(value, originalValue) - debugLog( - `value:`, - value, - `original:`, - originalValue, - `isRevertToOriginal:`, - isRevertToOriginal, - ) if (isRevertToOriginal) { - debugLog(`Reverting property ${String(prop)} to original value`) // If the value is reverted to its original state, remove it from changes delete changeTracker.assigned_[prop.toString()] // Make sure the copy is updated with the original value - debugLog(`Updating copy with original value for ${String(prop)}`) changeTracker.copy_[prop as keyof T] = deepClone(originalValue) // Check if all properties in this object have been reverted - debugLog(`Checking if all properties reverted`) const allReverted = checkIfReverted(changeTracker) - debugLog(`All reverted:`, allReverted) if (allReverted) { - debugLog(`All properties reverted, clearing tracking`) // If all have been reverted, clear tracking changeTracker.modified = false changeTracker.assigned_ = {} // If we're a nested object, check if the parent needs updating if (parent) { - debugLog(`Updating parent for property:`, parent.prop) - checkParentStatus(parent.tracker, parent.prop) + checkParentStatus(parent.tracker) } } else { // Some properties are still changed - debugLog(`Some properties still changed, keeping modified flag`) changeTracker.modified = true } } else { - debugLog(`Setting new value for property ${String(prop)}`) - // Set the value on the copy changeTracker.copy_[prop as keyof T] = value @@ -984,11 +785,8 @@ export function createChangeProxy< changeTracker.assigned_[prop.toString()] = true // Mark this object and its ancestors as modified - debugLog(`Marking object and ancestors as modified`, changeTracker) markChanged(changeTracker) } - } else { - debugLog(`Value unchanged, not tracking`) } return true @@ -1022,7 +820,6 @@ export function createChangeProxy< }, deleteProperty(dobj, prop) { - debugLog(`deleteProperty`, dobj, prop) const stringProp = typeof prop === `symbol` ? prop.toString() : prop if (stringProp in dobj) { @@ -1068,6 +865,7 @@ export function createChangeProxy< // Cache the proxy proxyCache.set(obj, proxy) + draftCopies.set(proxy, changeTracker.copy_) return proxy } @@ -1081,12 +879,8 @@ export function createChangeProxy< return { proxy, getChanges: () => { - debugLog(`getChanges called, modified:`, changeTracker.modified) - debugLog(changeTracker) - // First, check if the object is still considered modified if (!changeTracker.modified) { - debugLog(`Object not modified, returning empty object`) return {} } @@ -1115,7 +909,7 @@ export function createChangeProxy< result[key] = changeTracker.copy_[key] } } - debugLog(`Returning copy:`, result) + return result as unknown as Record }, } diff --git a/packages/db/src/query/builder/index.ts b/packages/db/src/query/builder/index.ts index 0cbfd4092d..c1097f9e9b 100644 --- a/packages/db/src/query/builder/index.ts +++ b/packages/db/src/query/builder/index.ts @@ -23,6 +23,7 @@ import { QueryMustHaveFromClauseError, SubQueryMustHaveFromClauseError, } from '../../errors.js' +import { getQueryIR } from './query-ir.js' import { createRefProxy, createRefProxyWithSelected, @@ -945,6 +946,11 @@ export class BaseQueryBuilder { * toArray(), and materialize() cannot be returned from fn.select(). Use * them as fields in select() so the compiler can add them to the query * graph. + * + * Compiled Collection-valued includes cannot be inputs to fn.select(), + * including nested descendants. Use toArray() or materialize() in the + * upstream select(), or do parent-only functional work before adding + * live Collection includes with select(). */ select( callback: (row: TContext[`schema`]) => TFuncSelectResult, @@ -1614,12 +1620,7 @@ export function buildQuery( return getQueryIR(result) } -// Internal function to get the QueryIR from a builder -export function getQueryIR( - builder: BaseQueryBuilder | QueryBuilder | InitialQueryBuilder, -): QueryIR { - return (builder as unknown as BaseQueryBuilder)._getQuery() -} +export { getQueryIR } // Type-only exports for the query builder export type InitialQueryBuilder = Pick< diff --git a/packages/db/src/query/builder/query-ir.ts b/packages/db/src/query/builder/query-ir.ts new file mode 100644 index 0000000000..2aefb0be91 --- /dev/null +++ b/packages/db/src/query/builder/query-ir.ts @@ -0,0 +1,13 @@ +import type { + BaseQueryBuilder, + InitialQueryBuilder, + QueryBuilder, +} from './index.js' +import type { QueryIR } from '../ir.js' + +// Keep IR access independent of Collection construction at runtime. +export function getQueryIR( + builder: BaseQueryBuilder | QueryBuilder | InitialQueryBuilder, +): QueryIR { + return (builder as unknown as BaseQueryBuilder)._getQuery() +} diff --git a/packages/db/src/query/builder/types.ts b/packages/db/src/query/builder/types.ts index 6286403e27..5adb14bc5b 100644 --- a/packages/db/src/query/builder/types.ts +++ b/packages/db/src/query/builder/types.ts @@ -481,11 +481,6 @@ export type ResultTypeFromSelect = }> > -export type SelectResult = - IsPlainObject extends true - ? ResultTypeFromSelect - : ResultTypeFromSelectValue - // Distribute over caseWhen branch unions so projection branches remain a union // of branch result shapes instead of being merged as one object type. type ResultTypeFromCaseWhen = T extends unknown diff --git a/packages/db/src/query/compiler/evaluators.ts b/packages/db/src/query/compiler/evaluators.ts index 25e7d5efb8..55b8c68f49 100644 --- a/packages/db/src/query/compiler/evaluators.ts +++ b/packages/db/src/query/compiler/evaluators.ts @@ -6,6 +6,7 @@ import { import { areValuesEqual, compareValues, + isUint8Array, isUnorderable, normalizeValue, } from '../../utils/comparison.js' @@ -19,6 +20,11 @@ function isUnknown(value: any): boolean { return value === null || value === undefined } +function normalizeEqualityOperand(value: unknown): unknown { + // Byte comparison needs no Map-key encoding, even for large binary values. + return isUint8Array(value) ? value : normalizeValue(value) +} + /** * Equality that follows PostgreSQL float semantics for `NaN`/invalid Dates: * such values are equal to one another and unequal to anything else. For all @@ -245,8 +251,8 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { const argA = compiledArgs[0]! const argB = compiledArgs[1]! return (data) => { - const a = normalizeValue(argA(data)) - const b = normalizeValue(argB(data)) + const a = normalizeEqualityOperand(argA(data)) + const b = normalizeEqualityOperand(argB(data)) // In 3-valued logic, any comparison with null/undefined returns UNKNOWN if (isUnknown(a) || isUnknown(b)) { return null @@ -392,7 +398,7 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { const valueEvaluator = compiledArgs[0]! const arrayEvaluator = compiledArgs[1]! return (data) => { - const value = normalizeValue(valueEvaluator(data)) + const value = normalizeEqualityOperand(valueEvaluator(data)) const array = arrayEvaluator(data) // In 3-valued logic, if the value is null/undefined, return UNKNOWN if (isUnknown(value)) { @@ -401,7 +407,9 @@ function compileFunction(func: Func, isSingleRow: boolean): (data: any) => any { if (!Array.isArray(array)) { return false } - return array.some((item) => valuesEqual(normalizeValue(item), value)) + return array.some((item) => + valuesEqual(normalizeEqualityOperand(item), value), + ) } } diff --git a/packages/db/src/query/compiler/expressions.ts b/packages/db/src/query/compiler/expressions.ts index f2856ed7eb..a52b8d11e5 100644 --- a/packages/db/src/query/compiler/expressions.ts +++ b/packages/db/src/query/compiler/expressions.ts @@ -1,6 +1,27 @@ import { Func, PropRef, Value } from '../ir.js' import type { BasicExpression, OrderBy } from '../ir.js' +/** Extracts the source aliases referenced by an expression. */ +export function getSourceAliasesFromExpression( + expr: BasicExpression, +): Set { + switch (expr.type) { + case `ref`: + return new Set(expr.path[0] ? [expr.path[0]] : []) + case `func`: { + const sourceAliases = new Set() + for (const arg of expr.args) { + for (const alias of getSourceAliasesFromExpression(arg)) { + sourceAliases.add(alias) + } + } + return sourceAliases + } + default: + return new Set() + } +} + /** * Normalizes a WHERE clause expression by removing table aliases from property references. * diff --git a/packages/db/src/query/compiler/group-by.ts b/packages/db/src/query/compiler/group-by.ts index 751d1e577e..9ce3b2f6a0 100644 --- a/packages/db/src/query/compiler/group-by.ts +++ b/packages/db/src/query/compiler/group-by.ts @@ -18,11 +18,24 @@ import { UnknownHavingExpressionTypeError, UnsupportedAggregateFunctionError, } from '../../errors.js' +import { + getEqualityValueIdentity, + getParentContextIdentity, + getParentContextValue, +} from '../equality-value-identity.js' import { compileExpression, isCaseWhenConditionTrue, toBooleanPredicate, } from './evaluators.js' +import { + INCLUDES_PUBLIC_KEY, + attachRouteMetadata, + getNamespacedRouteMetadata, + stripInternalCallbackMetadata, +} from './route-metadata.js' +import type { ValueIdentity } from '../equality-value-identity.js' +import type { RouteMetadata } from './route-metadata.js' import type { Aggregate, BasicExpression, @@ -34,55 +47,167 @@ import type { import type { NamespacedAndKeyedStream, NamespacedRow } from '../../types.js' import type { VirtualOrigin } from '../../virtual-props.js' -const VIRTUAL_SYNCED_KEY = `__virtual_synced__` -const VIRTUAL_HAS_LOCAL_KEY = `__virtual_has_local__` -const GROUP_KEY_REF_PREFIX = `__group_key_` +const RAW_REPRESENTATIVE = Symbol(`raw_group_representative`) + +type InternalGroupFields = ReturnType + +function createInternalGroupFields(groupCount: number, selectClause?: Select) { + const aliases = Object.keys(selectClause ?? {}) + let prefix = `__tanstack_group_` + while (aliases.some((alias) => alias.startsWith(prefix))) prefix += `_` + + return { + virtual: `${prefix}virtual`, + route: `${prefix}route`, + correlationIdentity: `${prefix}correlation_identity`, + parentContextIdentity: `${prefix}parent_context_identity`, + singleGroup: `${prefix}single_group`, + aggregatePrefix: `${prefix}aggregate_`, + groupKeys: Array.from( + { length: groupCount }, + (_, i) => `${prefix}key_${i}`, + ), + groupValues: Array.from( + { length: groupCount }, + (_, i) => `${prefix}value_${i}`, + ), + groupKeyRefs: Array.from( + { length: groupCount }, + (_, i) => `${prefix}key_ref_${i}`, + ), + } +} type RowVirtualMetadata = { synced: boolean hasLocal: boolean } -function addCorrelationRouteToGroupKey( +type Representative = { + key: string + [RAW_REPRESENTATIVE]: T +} + +function createPublicGroupKey(values: Array): unknown { + const identities = values.map(getEqualityValueIdentity) + if (identities.length === 1) { + const identity = identities[0] + if ( + identity == null || + (typeof identity !== `object` && + typeof identity !== `function` && + typeof identity !== `symbol`) + ) { + return identity + } + } + return serializeValue(identities) +} + +function attachPublicGroupKey( + row: Record, + publicKey: unknown, +): void { + const keyedRow = row as Record + keyedRow[INCLUDES_PUBLIC_KEY] = publicKey +} + +function createRepresentative( + rowKey: string, + value: T, + identity: unknown, +): Representative { + // Encode once per contribution, not once per member on every group change. + const representative = { + key: serializeValue([rowKey, identity]), + } as Representative + Object.defineProperty(representative, RAW_REPRESENTATIVE, { value }) + return representative +} + +function getRepresentative( + values: Array<[Representative, number]>, +): Representative | undefined { + let selected: Representative | undefined + for (const [candidate, multiplicity] of values) { + if (multiplicity <= 0) continue + if (selected === undefined || candidate.key < selected.key) { + selected = candidate + } + } + return selected +} + +function unwrapRepresentative( + value: Representative | undefined, +): T | undefined { + return value?.[RAW_REPRESENTATIVE] +} + +function addCorrelationRouteIdentityToGroupKey( key: Record, row: NamespacedRow, mainSource: string, + fields: InternalGroupFields, + valueIdentity: ValueIdentity, ): void { - const rowRecord = row as Record - const source = rowRecord[mainSource] as Record | undefined - key.__correlationKey = source?.__correlationKey - if (rowRecord.__parentContext != null) { - key.__parentContext = rowRecord.__parentContext + const route = getNamespacedRouteMetadata(row, mainSource) + key[fields.correlationIdentity] = valueIdentity.equality( + route?.correlationKey, + ) + if (route?.parentContext != null) { + key[fields.parentContextIdentity] = getParentContextIdentity( + route.parentContext, + ) } } -function getCorrelationRouteIdentity( +/** One representative carries the whole route so both parts come from one row. */ +function addCorrelationRouteAggregate( + aggregates: Record, + mainSource: string, + fields: InternalGroupFields, + valueIdentity: ValueIdentity, +): void { + aggregates[fields.route] = { + preMap: ([rowKey, row]: [string, NamespacedRow]) => { + const route = getNamespacedRouteMetadata(row, mainSource) + return createRepresentative(rowKey, route, [ + valueIdentity.exact(route?.correlationKey), + getParentContextIdentity(route?.parentContext), + ]) + }, + reduce: getRepresentative, + postMap: unwrapRepresentative, + } +} + +function getGroupRoute( aggregatedRow: Record, -): unknown { - return aggregatedRow.__parentContext == null - ? aggregatedRow.__correlationKey - : [aggregatedRow.__correlationKey, aggregatedRow.__parentContext] + fields: InternalGroupFields, +): RouteMetadata | undefined { + return aggregatedRow[fields.route] as RouteMetadata | undefined } -function getHavingEvaluationRow(row: Record): NamespacedRow { - const parentContext = row.__parentContext - return { - ...(parentContext !== null && typeof parentContext === `object` - ? (parentContext as NamespacedRow) - : {}), - $selected: row.$selected as Record, - } +function getCorrelationRouteIdentity( + aggregatedRow: Record, + fields: InternalGroupFields, +): unknown { + return getGroupRoute(aggregatedRow, fields)?.parentContext == null + ? aggregatedRow[fields.correlationIdentity] + : [ + aggregatedRow[fields.correlationIdentity], + aggregatedRow[fields.parentContextIdentity], + ] } -function getWrappedAggregateEvaluationRow( +function getGroupEvaluationRow( row: Record, - selected: Record, + fields: InternalGroupFields, + selected = row.$selected as Record, ): NamespacedRow { - const parentContext = row.__parentContext return { - ...(parentContext !== null && typeof parentContext === `object` - ? (parentContext as NamespacedRow) - : {}), + ...getParentContextValue(getGroupRoute(row, fields)?.parentContext), $selected: selected, } } @@ -118,14 +243,6 @@ function getRowVirtualMetadata(row: NamespacedRow): RowVirtualMetadata { const { sum, count, avg, min, max } = groupByOperators -/** - * Interface for caching the mapping between GROUP BY expressions and SELECT expressions - */ -interface GroupBySelectMapping { - selectToGroupByIndex: Map // Maps SELECT alias to GROUP BY expression index - groupByExpressions: Array // The GROUP BY expressions for reference -} - /** * Validates that all non-aggregate expressions in SELECT are present in GROUP BY * and creates a cached mapping for efficient lookup during processing @@ -133,12 +250,11 @@ interface GroupBySelectMapping { function validateAndCreateMapping( groupByClause: GroupBy, selectClause?: Select, -): GroupBySelectMapping { +): Map { const selectToGroupByIndex = new Map() - const groupByExpressions = [...groupByClause] if (!selectClause) { - return { selectToGroupByIndex, groupByExpressions } + return selectToGroupByIndex } // Validate each SELECT expression @@ -149,7 +265,7 @@ function validateAndCreateMapping( } // Non-aggregate expression must be in GROUP BY - const groupIndex = groupByExpressions.findIndex((groupExpr) => + const groupIndex = groupByClause.findIndex((groupExpr) => expressionsEqual(expr, groupExpr), ) @@ -161,7 +277,7 @@ function validateAndCreateMapping( selectToGroupByIndex.set(alias, groupIndex) } - return { selectToGroupByIndex, groupByExpressions } + return selectToGroupByIndex } /** @@ -171,204 +287,80 @@ function validateAndCreateMapping( export function processGroupBy( pipeline: NamespacedAndKeyedStream, groupByClause: GroupBy, + valueIdentity: ValueIdentity, havingClauses?: Array, selectClause?: Select, fnHavingClauses?: Array<(row: any) => any>, aggregateCollectionId?: string, mainSource?: string, + sanitizeCallbackRows = false, ): NamespacedAndKeyedStream { + const fields = createInternalGroupFields(groupByClause.length, selectClause) const virtualAggregates: Record = { - [VIRTUAL_SYNCED_KEY]: { - preMap: ([, row]: [string, NamespacedRow]) => - getRowVirtualMetadata(row).synced, - reduce: (values: Array<[boolean, number]>) => { - for (const [isSynced, multiplicity] of values) { - if (!isSynced && multiplicity > 0) { - return false - } - } - return true - }, - }, - [VIRTUAL_HAS_LOCAL_KEY]: { - preMap: ([, row]: [string, NamespacedRow]) => - getRowVirtualMetadata(row).hasLocal, - reduce: (values: Array<[boolean, number]>) => { - for (const [isLocal, multiplicity] of values) { - if (isLocal && multiplicity > 0) { - return true - } + [fields.virtual]: { + preMap: ([, row]: [string, NamespacedRow]) => getRowVirtualMetadata(row), + reduce: (values: Array<[RowVirtualMetadata, number]>) => { + const group: RowVirtualMetadata = { synced: true, hasLocal: false } + for (const [metadata, multiplicity] of values) { + if (multiplicity <= 0) continue + if (!metadata.synced) group.synced = false + if (metadata.hasLocal) group.hasLocal = true } - return false + return group }, }, } - // Handle empty GROUP BY (single-group aggregation) - if (groupByClause.length === 0) { - // For single-group aggregation, create a single group with all data - const aggregates: Record = virtualAggregates - - // Expressions that wrap aggregates (e.g. coalesce(count(...), 0)). - // Keys are the original SELECT aliases; values are pre-compiled evaluators - // over the transformed (aggregate-free) expression. - const wrappedAggExprs: Record any> = {} - const aggCounter = { value: 0 } - - if (selectClause) { - // Scan the SELECT clause for aggregate functions - for (const [alias, expr] of Object.entries(selectClause)) { - if (expr.type === `agg`) { - aggregates[alias] = getAggregateFunction(expr) - } else if (containsAggregate(expr)) { - const { transformed, extracted } = extractAndReplaceAggregates( - expr as SelectValueExpression, - aggCounter, - ) - for (const [syntheticAlias, aggExpr] of Object.entries(extracted)) { - aggregates[syntheticAlias] = getAggregateFunction(aggExpr) - } - wrappedAggExprs[alias] = compileGroupedSelectValue(transformed) - } - } - } - - // Use a constant key for single group. In includes mode, add the complete - // correlation route so parents with distinct projected inputs stay apart. - const keyExtractor = ([, row]: [string, NamespacedRow]) => { - const key: Record = { __singleGroup: true } - if (mainSource) addCorrelationRouteToGroupKey(key, row, mainSource) - return key - } - - // Apply the groupBy operator with single group - pipeline = pipeline.pipe( - groupBy(keyExtractor, aggregates), - ) as NamespacedAndKeyedStream - - // Update $selected to include aggregate values - pipeline = pipeline.pipe( - map(([, aggregatedRow]) => { - // Start with the existing $selected from early SELECT processing - const selectResults = (aggregatedRow as any).$selected || {} - const finalResults: Record = { ...selectResults } - - if (selectClause) { - // First pass: populate plain aggregate results and synthetic aliases - for (const [alias, expr] of Object.entries(selectClause)) { - if (expr.type === `agg`) { - finalResults[alias] = aggregatedRow[alias] - } - } - evaluateWrappedAggregates( - finalResults, - aggregatedRow as Record, - wrappedAggExprs, - ) - } - - // Use a single key for the result and update $selected. - // When in includes mode, restore the namespaced source structure with - // __correlationKey so output extraction can route results per-parent. - const correlationKey = mainSource - ? (aggregatedRow as any).__correlationKey - : undefined - const correlationRoute = mainSource - ? getCorrelationRouteIdentity(aggregatedRow) - : undefined - const resultKey = - correlationRoute !== undefined - ? `single_group_${serializeValue(correlationRoute)}` - : `single_group` - const resultRow: Record = { - ...(aggregatedRow as Record), - $selected: finalResults, - } - const groupSynced = (aggregatedRow as Record)[ - VIRTUAL_SYNCED_KEY - ] - const groupHasLocal = (aggregatedRow as Record)[ - VIRTUAL_HAS_LOCAL_KEY - ] - resultRow.$synced = groupSynced ?? true - resultRow.$origin = ( - groupHasLocal ? `local` : `remote` - ) satisfies VirtualOrigin - resultRow.$key = resultKey - resultRow.$collectionId = - aggregateCollectionId ?? resultRow.$collectionId - if (mainSource && correlationKey !== undefined) { - resultRow[mainSource] = { __correlationKey: correlationKey } - } - return [resultKey, resultRow] as [unknown, Record] - }), + if (mainSource) { + addCorrelationRouteAggregate( + virtualAggregates, + mainSource, + fields, + valueIdentity, ) - - // Apply HAVING clauses if present - if (havingClauses && havingClauses.length > 0) { - for (const havingClause of havingClauses) { - const havingExpression = getHavingExpression(havingClause) - const transformedHavingClause = replaceAggregatesByRefs( - havingExpression, - selectClause || {}, - `$selected`, - ) - const compiledHaving = compileExpression(transformedHavingClause) - - pipeline = pipeline.pipe( - filter(([, row]) => { - const namespacedRow = getHavingEvaluationRow(row) - return toBooleanPredicate(compiledHaving(namespacedRow)) - }), - ) - } - } - - // Apply functional HAVING clauses if present - if (fnHavingClauses && fnHavingClauses.length > 0) { - for (const fnHaving of fnHavingClauses) { - pipeline = pipeline.pipe( - filter(([, row]) => { - const namespacedRow = getHavingEvaluationRow(row) - return toBooleanPredicate(fnHaving(namespacedRow)) - }), - ) - } - } - - return pipeline } - // Multi-group aggregation logic... - // Validate and create mapping for non-aggregate expressions in SELECT - const mapping = validateAndCreateMapping(groupByClause, selectClause) + const singleGroup = groupByClause.length === 0 + // Single-group aggregation accepts selections without grouping validation. + const mapping = singleGroup + ? undefined + : validateAndCreateMapping(groupByClause, selectClause) // Pre-compile groupBy expressions const compiledGroupByExpressions = groupByClause.map((e) => compileExpression(e), ) - // Create a key extractor function using simple __key_X format. In includes - // mode, add the complete route so parents with distinct projected inputs do - // not aggregate together. + // Include the complete route so distinct parent inputs stay apart. const keyExtractor = ([, row]: [ string, NamespacedRow & { $selected?: any }, ]) => { // Use the original namespaced row for GROUP BY expressions, not $selected - const namespacedRow = { ...row } - delete (namespacedRow as any).$selected + const namespacedRow = singleGroup ? row : { ...row } + if (!singleGroup) delete namespacedRow.$selected - const key: Record = {} + const key: Record = singleGroup + ? { [fields.singleGroup]: true } + : {} - // Use simple __key_X format for each groupBy expression + // D2 must key groups by the same relation as the query evaluator. The raw + // representative is retained separately as an aggregate for projection. for (let i = 0; i < groupByClause.length; i++) { const compiledExpr = compiledGroupByExpressions[i]! const value = compiledExpr(namespacedRow) - key[`__key_${i}`] = value + key[fields.groupKeys[i]!] = valueIdentity.equality(value) } - if (mainSource) addCorrelationRouteToGroupKey(key, row, mainSource) + if (mainSource) { + addCorrelationRouteIdentityToGroupKey( + key, + row, + mainSource, + fields, + valueIdentity, + ) + } return key } @@ -378,6 +370,18 @@ export function processGroupBy( const wrappedAggExprs: Record any> = {} const aggCounter = { value: 0 } + for (let i = 0; i < compiledGroupByExpressions.length; i++) { + const compiledExpr = compiledGroupByExpressions[i]! + aggregates[fields.groupValues[i]!] = { + preMap: ([rowKey, row]: [string, NamespacedRow]) => { + const value = compiledExpr(row) + return createRepresentative(rowKey, value, valueIdentity.exact(value)) + }, + reduce: getRepresentative, + postMap: unwrapRepresentative, + } + } + if (selectClause) { // Scan the SELECT clause for aggregate functions for (const [alias, expr] of Object.entries(selectClause)) { @@ -387,12 +391,19 @@ export function processGroupBy( const { transformed, extracted } = extractAndReplaceAggregates( expr as SelectValueExpression, aggCounter, + fields.aggregatePrefix, ) for (const [syntheticAlias, aggExpr] of Object.entries(extracted)) { aggregates[syntheticAlias] = getAggregateFunction(aggExpr) } wrappedAggExprs[alias] = compileGroupedSelectValue( - replaceGroupByRefsInSelectValue(transformed, groupByClause), + singleGroup + ? transformed + : replaceGroupByRefsInSelectValue( + transformed, + groupByClause, + fields.groupKeyRefs, + ), ) } } @@ -406,18 +417,21 @@ export function processGroupBy( map(([, aggregatedRow]) => { // Start with the existing $selected from early SELECT processing const selectResults = (aggregatedRow as any).$selected || {} - const finalResults: Record = {} + const finalResults: Record = singleGroup + ? { ...selectResults } + : {} if (selectClause) { // First pass: populate group keys, plain aggregates, and synthetic aliases for (const [alias, expr] of Object.entries(selectClause)) { if (expr.type === `agg`) { finalResults[alias] = aggregatedRow[alias] - } else if (!wrappedAggExprs[alias]) { + } else if (!singleGroup && !wrappedAggExprs[alias]) { // Use cached mapping to get the corresponding __key_X for non-aggregates - const groupIndex = mapping.selectToGroupByIndex.get(alias) + const groupIndex = mapping?.get(alias) if (groupIndex !== undefined) { - finalResults[alias] = aggregatedRow[`__key_${groupIndex}`] + finalResults[alias] = + aggregatedRow[fields.groupValues[groupIndex]!] } else { // Fallback to original SELECT results finalResults[alias] = selectResults[alias] @@ -428,56 +442,71 @@ export function processGroupBy( finalResults, aggregatedRow as Record, wrappedAggExprs, - groupByClause.length, + fields, ) } else { // No SELECT clause - just use the group keys for (let i = 0; i < groupByClause.length; i++) { - finalResults[`__key_${i}`] = aggregatedRow[`__key_${i}`] + finalResults[`__key_${i}`] = aggregatedRow[fields.groupValues[i]!] } } // Generate a simple key for the live collection using group values. // In includes mode, add the complete route so correlated groups do not // collide. - const correlationKey = mainSource - ? (aggregatedRow as any).__correlationKey + const route = mainSource + ? getGroupRoute(aggregatedRow, fields) : undefined + const correlationKey = route?.correlationKey const correlationRoute = mainSource - ? getCorrelationRouteIdentity(aggregatedRow) + ? getCorrelationRouteIdentity(aggregatedRow, fields) : undefined const keyParts: Array = [] + const publicKeyParts: Array = [] for (let i = 0; i < groupByClause.length; i++) { - keyParts.push(aggregatedRow[`__key_${i}`]) + keyParts.push(aggregatedRow[fields.groupKeys[i]!]) + publicKeyParts.push(aggregatedRow[fields.groupValues[i]!]) } if (correlationRoute !== undefined) { keyParts.push(correlationRoute) } - const finalKey = - keyParts.length === 1 ? keyParts[0] : serializeValue(keyParts) - - // When in includes mode, restore the namespaced source structure with - // __correlationKey so output extraction can route results per-parent. + const finalKey = singleGroup + ? correlationRoute !== undefined + ? `single_group_${serializeValue(correlationRoute)}` + : `single_group` + : keyParts.length === 1 + ? keyParts[0] + : serializeValue(keyParts) + const publicKey = singleGroup + ? `single_group` + : createPublicGroupKey(publicKeyParts) + + // When in includes mode, restore route metadata for output routing. const resultRow: Record = { ...(aggregatedRow as Record), $selected: finalResults, } - const groupSynced = (aggregatedRow as Record)[ - VIRTUAL_SYNCED_KEY - ] - const groupHasLocal = (aggregatedRow as Record)[ - VIRTUAL_HAS_LOCAL_KEY - ] - resultRow.$synced = groupSynced ?? true + const virtual = (aggregatedRow as Record)[fields.virtual] as + | RowVirtualMetadata + | undefined + resultRow.$synced = virtual?.synced ?? true resultRow.$origin = ( - groupHasLocal ? `local` : `remote` + virtual?.hasLocal ? `local` : `remote` ) satisfies VirtualOrigin - resultRow.$key = finalKey + resultRow.$key = publicKey resultRow.$collectionId = aggregateCollectionId ?? resultRow.$collectionId if (mainSource && correlationKey !== undefined) { - resultRow[mainSource] = { __correlationKey: correlationKey } + attachPublicGroupKey(resultRow, publicKey) + attachRouteMetadata( + resultRow, + correlationKey, + route?.parentContext ?? null, + ) } - return [finalKey, resultRow] as [unknown, Record] + return [mainSource ? finalKey : publicKey, resultRow] as [ + unknown, + Record, + ] }), ) @@ -493,8 +522,10 @@ export function processGroupBy( pipeline = pipeline.pipe( filter(([, row]) => { - const namespacedRow = getHavingEvaluationRow(row) - return compiledHaving(namespacedRow) + const namespacedRow = getGroupEvaluationRow(row, fields) + const result = compiledHaving(namespacedRow) + // Preserve each path's coercion for unchecked nonboolean IR values. + return singleGroup ? toBooleanPredicate(result) : result }), ) } @@ -505,8 +536,11 @@ export function processGroupBy( for (const fnHaving of fnHavingClauses) { pipeline = pipeline.pipe( filter(([, row]) => { - const namespacedRow = getHavingEvaluationRow(row) - return toBooleanPredicate(fnHaving(namespacedRow)) + const namespacedRow = getGroupEvaluationRow(row, fields) + const callbackRow = sanitizeCallbackRows + ? stripInternalCallbackMetadata(namespacedRow) + : namespacedRow + return toBooleanPredicate(fnHaving(callbackRow)) }), ) } @@ -678,23 +712,27 @@ function evaluateWrappedAggregates( finalResults: Record, aggregatedRow: Record, wrappedAggExprs: Record any>, - groupKeyCount: number = 0, + fields: InternalGroupFields, ): void { for (const key of Object.keys(aggregatedRow)) { - if (key.startsWith(`__agg_`)) { + if (key.startsWith(fields.aggregatePrefix)) { finalResults[key] = aggregatedRow[key] } } - for (let i = 0; i < groupKeyCount; i++) { - finalResults[`${GROUP_KEY_REF_PREFIX}${i}`] = aggregatedRow[`__key_${i}`] + for (let i = 0; i < fields.groupKeyRefs.length; i++) { + finalResults[fields.groupKeyRefs[i]!] = + aggregatedRow[fields.groupValues[i]!] } for (const [alias, evaluator] of Object.entries(wrappedAggExprs)) { finalResults[alias] = evaluator( - getWrappedAggregateEvaluationRow(aggregatedRow, finalResults), + getGroupEvaluationRow(aggregatedRow, fields, finalResults), ) } for (const key of Object.keys(finalResults)) { - if (key.startsWith(`__agg_`) || key.startsWith(GROUP_KEY_REF_PREFIX)) { + if ( + key.startsWith(fields.aggregatePrefix) || + fields.groupKeyRefs.includes(key) + ) { delete finalResults[key] } } @@ -751,6 +789,7 @@ export function containsAggregate( function extractAndReplaceAggregates( expr: SelectValueExpression, counter: { value: number }, + aggregatePrefix: string, ): { transformed: SelectValueExpression extracted: Record @@ -760,7 +799,7 @@ function extractAndReplaceAggregates( } if (expr.type === `agg`) { - const alias = `__agg_${counter.value++}` + const alias = `${aggregatePrefix}${counter.value++}` return { transformed: new PropRef([`$selected`, alias]), extracted: { [alias]: expr }, @@ -770,7 +809,7 @@ function extractAndReplaceAggregates( if (expr.type === `func`) { const allExtracted: Record = {} const newArgs = expr.args.map((arg: BasicExpression | Aggregate) => { - const result = extractAndReplaceAggregates(arg, counter) + const result = extractAndReplaceAggregates(arg, counter, aggregatePrefix) Object.assign(allExtracted, result.extracted) return result.transformed as BasicExpression }) @@ -783,8 +822,16 @@ function extractAndReplaceAggregates( if (isConditionalSelect(expr)) { const allExtracted: Record = {} const branches = expr.branches.map((branch) => { - const condition = extractAndReplaceAggregates(branch.condition, counter) - const value = extractAndReplaceAggregates(branch.value, counter) + const condition = extractAndReplaceAggregates( + branch.condition, + counter, + aggregatePrefix, + ) + const value = extractAndReplaceAggregates( + branch.value, + counter, + aggregatePrefix, + ) Object.assign(allExtracted, condition.extracted, value.extracted) return { condition: condition.transformed as BasicExpression, @@ -794,7 +841,11 @@ function extractAndReplaceAggregates( const defaultValue = expr.defaultValue === undefined ? undefined - : extractAndReplaceAggregates(expr.defaultValue, counter) + : extractAndReplaceAggregates( + expr.defaultValue, + counter, + aggregatePrefix, + ) if (defaultValue) { Object.assign(allExtracted, defaultValue.extracted) @@ -814,6 +865,7 @@ function extractAndReplaceAggregates( const result = extractAndReplaceAggregates( value as SelectValueExpression, counter, + aggregatePrefix, ) Object.assign(allExtracted, result.extracted) transformed[key] = result.transformed @@ -829,6 +881,7 @@ function extractAndReplaceAggregates( function replaceGroupByRefsInSelectValue( value: SelectValueExpression, groupByClause: GroupBy, + groupKeyRefs: Array, ): SelectValueExpression { if (isConditionalSelect(value)) { return new ConditionalSelect( @@ -836,12 +889,21 @@ function replaceGroupByRefsInSelectValue( condition: replaceGroupByRefsInExpression( branch.condition, groupByClause, + groupKeyRefs, + ), + value: replaceGroupByRefsInSelectValue( + branch.value, + groupByClause, + groupKeyRefs, ), - value: replaceGroupByRefsInSelectValue(branch.value, groupByClause), })), value.defaultValue === undefined ? undefined - : replaceGroupByRefsInSelectValue(value.defaultValue, groupByClause), + : replaceGroupByRefsInSelectValue( + value.defaultValue, + groupByClause, + groupKeyRefs, + ), ) } @@ -851,6 +913,7 @@ function replaceGroupByRefsInSelectValue( transformed[key] = replaceGroupByRefsInSelectValue( entry as SelectValueExpression, groupByClause, + groupKeyRefs, ) } return transformed @@ -864,12 +927,13 @@ function replaceGroupByRefsInSelectValue( return value } - return replaceGroupByRefsInExpression(value, groupByClause) + return replaceGroupByRefsInExpression(value, groupByClause, groupKeyRefs) } function replaceGroupByRefsInExpression( expr: BasicExpression, groupByClause: GroupBy, + groupKeyRefs: Array, ): BasicExpression { if (expr.type === `ref`) { const groupIndex = groupByClause.findIndex((groupExpr) => @@ -877,14 +941,14 @@ function replaceGroupByRefsInExpression( ) return groupIndex === -1 ? expr - : new PropRef([`$selected`, `${GROUP_KEY_REF_PREFIX}${groupIndex}`]) + : new PropRef([`$selected`, groupKeyRefs[groupIndex]!]) } if (expr.type === `func`) { return new Func( expr.name, expr.args.map((arg) => - replaceGroupByRefsInExpression(arg, groupByClause), + replaceGroupByRefsInExpression(arg, groupByClause, groupKeyRefs), ), ) } diff --git a/packages/db/src/query/compiler/index.ts b/packages/db/src/query/compiler/index.ts index 3768016563..cc64fde363 100644 --- a/packages/db/src/query/compiler/index.ts +++ b/packages/db/src/query/compiler/index.ts @@ -8,7 +8,16 @@ import { serializeValue, tap, } from '@tanstack/db-ivm' +import { isPlainObject } from '../../utils/type-guards.js' +import { getOrCreate } from '../../utils/get-or-create.js' import { optimizeQuery } from '../optimizer.js' +import { materializeCompilation } from '../live/materialized-pipeline.js' +import { + createParentContext, + createValueIdentity, + getParentContextIdentity, + getParentContextValue, +} from '../equality-value-identity.js' import { CollectionInputNotFoundError, DistinctRequiresSelectError, @@ -33,6 +42,7 @@ import { PropRef, Value as ValClass, collectCollectionSources, + getFromSources, getWhereExpression, isExpressionLike, } from '../ir.js' @@ -51,10 +61,18 @@ import { processOrderBy } from './order-by.js' import { crossJoinParentRoutes } from './parent-routes.js' import { INCLUDES_PUBLIC_KEY, + INCLUDES_ROUTING, + attachRouteMetadata, attachRouteMetadataToResult, + getNamespacedRouteMetadata, + getRouteMetadata, getRoutedScalarMetadata, + stripInternalCallbackMetadata, + stripInternalRouteMetadata, + stripRouteMetadata, } from './route-metadata.js' import { processSelect } from './select.js' +import type { ValueIdentity } from '../equality-value-identity.js' import type { CollectionSubscription } from '../../collection/subscription.js' import type { OrderByOptimizationInfo } from './order-by.js' import type { @@ -77,11 +95,8 @@ import type { import type { QueryCache, QueryMapping, WindowOptions } from './types.js' export type { WindowOptions } from './types.js' -export { INCLUDES_PUBLIC_KEY } from './route-metadata.js' +export { INCLUDES_PUBLIC_KEY, INCLUDES_ROUTING } from './route-metadata.js' -/** Symbol used to tag parent $selected with routing metadata for includes */ -export const INCLUDES_ROUTING = Symbol(`includesRouting`) -export const FN_SELECT_STATE = Symbol(`fnSelectState`) const SKIP_INCLUDE = Symbol(`skipInclude`) function getUnsupportedFnSelectResultDescription( @@ -152,14 +167,23 @@ type CompiledParentProjection = { function projectParentContext( nsRow: NamespacedRow, projections: Array, + valueIdentity: ValueIdentity, ): Record { - const inherited = (nsRow as any).__parentContext + const inherited = getRouteMetadata(nsRow)?.parentContext + const inheritedValue = getParentContextValue(inherited) const parentContext: Record = - inherited != null && typeof inherited === `object` ? { ...inherited } : {} + inheritedValue === undefined ? {} : { ...inheritedValue } + const projectedIdentity: Array = [] for (const projection of projections) { + const projectedValue = projection.compiled(nsRow) + projectedIdentity.push([ + projection.alias, + projection.field, + valueIdentity.equality(projectedValue), + ]) if (projection.field.length === 0) { - const projectedAlias = projection.compiled(nsRow) + const projectedAlias = projectedValue parentContext[projection.alias] = projectedAlias != null && typeof projectedAlias === `object` ? { ...projectedAlias } @@ -185,17 +209,20 @@ function projectParentContext( target[segment] = nested target = nested } - target[projection.field[projection.field.length - 1]!] = - projection.compiled(nsRow) + target[projection.field[projection.field.length - 1]!] = projectedValue } - return parentContext + return createParentContext(parentContext, [ + getParentContextIdentity(inherited), + projectedIdentity, + ]) } function parameterizeByParentRoutes( pipeline: NamespacedAndKeyedStream, parentKeyStream: KeyedStream, mainSource: string, + valueIdentity: ValueIdentity, ): NamespacedAndKeyedStream { return crossJoinParentRoutes( pipeline, @@ -206,15 +233,19 @@ function parameterizeByParentRoutes( } as Record namespaced[mainSource] = { ...namespaced[mainSource], - __correlationKey: correlationKey, [INCLUDES_PUBLIC_KEY]: namespaced[mainSource]?.[INCLUDES_PUBLIC_KEY] ?? rowKey, } - if (parentContext != null) Object.assign(namespaced, parentContext) - namespaced.__correlationKey = correlationKey - namespaced.__parentContext = parentContext + if (parentContext != null) { + Object.assign(namespaced, getParentContextValue(parentContext)) + } + attachRouteMetadata(namespaced, correlationKey, parentContext) return [ - serializeValue([rowKey, correlationKey, parentContext]), + serializeValue([ + valueIdentity.equality(rowKey), + valueIdentity.equality(correlationKey), + getParentContextIdentity(parentContext), + ]), namespaced, ] as [string, NamespacedRow] }, @@ -222,9 +253,11 @@ function parameterizeByParentRoutes( } function getRowCorrelationKey(row: NamespacedRow, mainSource: string): unknown { - return ( - (row as any)[mainSource]?.__correlationKey ?? (row as any).__correlationKey - ) + return getNamespacedRouteMetadata(row, mainSource)?.correlationKey +} + +function getRowParentContext(row: NamespacedRow, mainSource: string): unknown { + return getNamespacedRouteMetadata(row, mainSource)?.parentContext ?? null } function correlationValuesEqual(left: unknown, right: unknown): boolean { @@ -277,6 +310,9 @@ export interface CompilationResult { /** The compiled query pipeline (D2 stream) */ pipeline: ResultStream + /** Runtime identity scope owned by this compiled graph. */ + valueIdentity: ValueIdentity + /** Map of opaque source IDs to their WHERE clauses for index optimization */ sourceWhereClauses: Map> @@ -306,6 +342,12 @@ export interface CompilationResult { includes?: Array } +const valueIdentitiesByCache = new WeakMap() + +function getCompilationValueIdentity(cache: QueryCache): ValueIdentity { + return getOrCreate(valueIdentitiesByCache, cache, createValueIdentity) +} + /** * Compiles a query IR into a D2 pipeline * @param rawQuery The query IR to compile @@ -340,6 +382,7 @@ export function compileQuery( if (cachedResult) { return cachedResult } + const valueIdentity = getCompilationValueIdentity(cache) // Validate the raw query BEFORE optimization to check user's original structure. // This must happen before optimization because the optimizer may create internal @@ -357,7 +400,8 @@ export function compileQuery( // Create a copy of the inputs map to avoid modifying the original const allInputs = { ...inputs } - bindSourceInputs(rawQuery, allInputs) + const rawSources = collectCollectionSources(rawQuery) + bindSourceInputs(rawSources, allInputs) // Track alias to collection id relationships discovered during compilation. // This includes all user-declared aliases plus inner aliases from subqueries. @@ -401,6 +445,10 @@ export function compileQuery( parentKeyStream, ) Object.assign(sources, fromSources) + const sourceCarriesInternalRouteState = + parentKeyStream !== undefined || + sourceIncludes.length > 0 || + directIncludes.length > 0 // If this is an includes child query, inner-join the raw input with parent keys. // This filters the child collection to only rows matching parents in the result set. @@ -417,37 +465,58 @@ export function compileQuery( if (parentKeyStream && childCorrelationField && joinsParentDirectly) { const mainInput = sources[mainSource]! let filteredMainInput = mainInput - // Re-key child input by correlation field: [correlationValue, [childKey, childRow]] + // Join on query equality rather than raw JavaScript identity. Keep the raw + // child value beside the row so result routing can still expose it. const childFieldPath = childCorrelationField.path.slice(1) // remove alias prefix const childRekeyed = mainInput.pipe( map(([key, row]: [unknown, any]) => { const correlationValue = getNestedValue(row, childFieldPath) - return [correlationValue, [key, row]] as [unknown, [unknown, any]] + return [ + valueIdentity.serializeEquality(correlationValue), + [key, row, correlationValue], + ] as [unknown, [unknown, any, unknown]] }), ) + const equalityParentKeys = parentKeyStream.pipe( + map(([correlationValue, parentContext]: [unknown, unknown]) => [ + valueIdentity.serializeEquality(correlationValue), + parentContext, + ]), + reduce((values: Array<[unknown, number]>) => + values.map(([value, multiplicity]) => [ + value, + multiplicity > 0 ? 1 : 0, + ]), + ), + ) + // Inner join: only children whose correlation key exists in parent keys pass through - const joined = childRekeyed.pipe(joinOperator(parentKeyStream, `inner`)) + const joined = childRekeyed.pipe(joinOperator(equalityParentKeys, `inner`)) // Extract: [correlationValue, [[childKey, childRow], parentContext]] → [childKey, childRow] - // Tag the row with __correlationKey for output routing - // If parentSide is non-null (parent context projected), attach as __parentContext + // Keep routing metadata outside the user-visible row namespace. filteredMainInput = joined.pipe( filter(([_correlationValue, [childSide]]: any) => { return childSide != null }), - map(([correlationValue, [childSide, parentSide]]: any) => { - const [childKey, childRow] = childSide - const tagged: any = { - ...childRow, - __correlationKey: correlationValue, - [INCLUDES_PUBLIC_KEY]: childKey, - } - if (parentSide != null) { - tagged.__parentContext = parentSide - } + map(([_correlationIdentity, [childSide, parentSide]]: any) => { + const [childKey, childRow, correlationValue] = childSide + const tagged: any = attachRouteMetadata( + { + ...childRow, + [INCLUDES_PUBLIC_KEY]: childKey, + }, + correlationValue, + parentSide, + ) const effectiveKey = - parentSide != null ? serializeValue([childKey, parentSide]) : childKey + parentSide != null + ? serializeValue([ + valueIdentity.equality(childKey), + getParentContextIdentity(parentSide), + ]) + : childKey return [effectiveKey, tagged] }), ) @@ -463,6 +532,7 @@ export function compileQuery( initialPipeline, parentKeyStream, mainSource, + valueIdentity, ) } @@ -489,6 +559,7 @@ export function compileQuery( aliasRemapping, sourceWhereClauses, parentKeyStream !== undefined, + valueIdentity, parentKeyStream, ) } @@ -527,7 +598,10 @@ export function compileQuery( for (const fnWhere of query.fnWhere) { pipeline = pipeline.pipe( filter(([_key, namespacedRow]) => { - return toBooleanPredicate(fnWhere(namespacedRow)) + const callbackRow = sourceCarriesInternalRouteState + ? (stripInternalCallbackMetadata(namespacedRow) as NamespacedRow) + : namespacedRow + return toBooleanPredicate(fnWhere(callbackRow)) }), ) } @@ -536,16 +610,17 @@ export function compileQuery( // Extract includes from SELECT, compile child pipelines, and replace with placeholders. // This must happen AFTER WHERE (so parent pipeline is filtered) but BEFORE processSelect // (so IncludesSubquery nodes are stripped before select compilation). - const includesResults: Array = !query.select + const inputIncludes = [ + ...directIncludes, + ...sourceIncludes.map(({ include }) => include), + ] + const materializeSelectInput = !!query.fnSelect && inputIncludes.length > 0 + let includesResults: Array = !query.select ? [...directIncludes] : [] - const includesRoutingFns: Array<{ + let includesRoutingFns: Array<{ fieldName: string - getRouting: (nsRow: any) => { - active: boolean - correlationKey: unknown - parentContext: Record | null - } + getRouting: (nsRow: any) => IncludeRouting }> = [] for (const { sourceAlias, include } of sourceIncludes) { const projectedPaths = @@ -555,7 +630,7 @@ export function compileQuery( sourceAlias, include.resultPath, ) - : query.fnSelect + : query.fnSelect && !materializeSelectInput ? [] : [ { @@ -573,30 +648,18 @@ export function compileQuery( `${sourceAlias}.${resultPath.join(`.`)}`, includesRoutingFns, ) - const compiledGuards = guards.map((guard) => ({ - condition: compileExpression(guard.condition), - expected: guard.expected, - })) includesResults.push({ ...include, fieldName, resultPath, }) - includesRoutingFns.push({ fieldName, - getRouting: (nsRow: any) => { - if (!matchesConditionalSelectGuards(compiledGuards, nsRow)) { - return { active: false, correlationKey: null, parentContext: null } - } - return ( - nsRow[sourceAlias]?.[INCLUDES_ROUTING]?.[include.fieldName] ?? { - active: false, - correlationKey: null, - parentContext: null, - } - ) - }, + getRouting: compileGuardedRouting( + guards, + (nsRow) => + nsRow[sourceAlias]?.[INCLUDES_ROUTING]?.[include.fieldName], + ), }) } } @@ -612,35 +675,17 @@ export function compileQuery( resultPath.join(`.`), includesRoutingFns, ) - const compiledGuards = guards.map((guard) => ({ - condition: compileExpression(guard.condition), - expected: guard.expected, - })) - includesResults.push({ ...include, fieldName, resultPath, }) - includesRoutingFns.push({ fieldName, - getRouting: (nsRow: any) => { - if (!matchesConditionalSelectGuards(compiledGuards, nsRow)) { - return { - active: false, - correlationKey: null, - parentContext: null, - } - } - return ( - nsRow[INCLUDES_ROUTING]?.[include.fieldName] ?? { - active: false, - correlationKey: null, - parentContext: null, - } - ) - }, + getRouting: compileGuardedRouting( + guards, + (nsRow) => nsRow[INCLUDES_ROUTING]?.[include.fieldName], + ), }) } } @@ -655,41 +700,31 @@ export function compileQuery( // Branch parent pipeline: map to [correlationValue, parentContext] // When parentProjection exists, project referenced parent fields; otherwise null (zero overhead) const compiledCorrelation = compileExpression(subquery.correlationField) - const compiledGuards = guards.map((guard) => ({ - condition: compileExpression(guard.condition), - expected: guard.expected, - })) const compiledProjections: Array = subquery.parentProjection?.map((ref) => ({ alias: ref.path[0]!, field: ref.path.slice(1), compiled: compileExpression(ref), })) ?? [] - let parentKeys: any - if (compiledProjections.length > 0) { - parentKeys = pipeline.pipe( - map(([_key, nsRow]: any) => { - if (!matchesConditionalSelectGuards(compiledGuards, nsRow)) { - return [SKIP_INCLUDE, null] as any - } - const parentContext = projectParentContext( - nsRow, - compiledProjections, - ) - return [compiledCorrelation(nsRow), parentContext] as any - }), - ) - } else { - parentKeys = pipeline.pipe( - map(([_key, nsRow]: any) => { - if (!matchesConditionalSelectGuards(compiledGuards, nsRow)) { - return [SKIP_INCLUDE, null] as any - } - return [compiledCorrelation(nsRow), null] as any - }), - ) - } - parentKeys = parentKeys.pipe( + // One routing function serves both the parent-key branch and the + // INCLUDES_ROUTING tag on $selected. + const getRouting = compileGuardedRouting(guards, (nsRow) => ({ + active: true, + correlationKey: compiledCorrelation(nsRow), + parentContext: + compiledProjections.length > 0 + ? projectParentContext(nsRow, compiledProjections, valueIdentity) + : null, + })) + let parentKeys: any = pipeline.pipe( + map(([_key, nsRow]: any) => { + const routing = getRouting(nsRow) + return ( + routing.active + ? [routing.correlationKey, routing.parentContext] + : [SKIP_INCLUDE, null] + ) as any + }), filter(([correlationValue]: any) => correlationValue !== SKIP_INCLUDE), ) @@ -751,7 +786,7 @@ export function compileQuery( tap((data: any) => { for (const [[correlationValue], weight] of data.getInner()) { if (correlationValue == null) continue - const encoded = serializeValue(correlationValue) + const encoded = valueIdentity.serializeEquality(correlationValue) const previous = demandWeights.get(encoded) const nextWeight = (previous?.weight ?? 0) + weight if (nextWeight === 0) { @@ -828,51 +863,7 @@ export function compileQuery( scalarField: subquery.scalarField, }) - // Capture routing function for INCLUDES_ROUTING tagging - if (compiledProjections.length > 0) { - const compiledCorr = compiledCorrelation - const compiledRoutingGuards = compiledGuards - includesRoutingFns.push({ - fieldName, - getRouting: (nsRow: any) => { - if (!matchesConditionalSelectGuards(compiledRoutingGuards, nsRow)) { - return { - active: false, - correlationKey: null, - parentContext: null, - } - } - const parentContext = projectParentContext( - nsRow, - compiledProjections, - ) - return { - active: true, - correlationKey: compiledCorr(nsRow), - parentContext, - } - }, - }) - } else { - const compiledRoutingGuards = compiledGuards - includesRoutingFns.push({ - fieldName, - getRouting: (nsRow: any) => { - if (!matchesConditionalSelectGuards(compiledRoutingGuards, nsRow)) { - return { - active: false, - correlationKey: null, - parentContext: null, - } - } - return { - active: true, - correlationKey: compiledCorrelation(nsRow), - parentContext: null, - } - }, - }) - } + includesRoutingFns.push({ fieldName, getRouting }) // Replace includes entry in select with a null placeholder query = { @@ -895,43 +886,98 @@ export function compileQuery( throw new FnSelectWithGroupByError() } + const selectHasAggregates = + query.select !== undefined && containsAggregate(query.select) + const routingFns = includesRoutingFns + const getRowIncludesRouting = (row: NamespacedRow) => + Object.fromEntries( + routingFns.map(({ fieldName, getRouting }) => [ + fieldName, + getRouting(row), + ]), + ) + if (materializeSelectInput) { + if (!inputIncludes.every(isInlineInclude)) { + throw new Error( + `fn.select() cannot consume Collection-valued includes. Use toArray() or materialize() in the upstream select(), or use an expression select() to keep live Collections.`, + ) + } + // Input paths belong before the callback: its arbitrary output may rename + // or discard them. Inline values need no public Collection boundary. + const inputPipeline = pipeline.pipe( + map( + ([key, row]: [ + unknown, + NamespacedRow & { [INCLUDES_ROUTING]?: object }, + ]) => [ + key, + [ + { + ...row, + [INCLUDES_ROUTING]: { + ...row[INCLUDES_ROUTING], + ...getRowIncludesRouting(row), + }, + }, + undefined, + ], + ], + ), + ) as ResultStream + const materializedInput = materializeCompilation({ + pipeline: inputPipeline, + includes: includesResults, + valueIdentity, + collectionId: mainCollectionId, + sourceWhereClauses, + aliasToCollectionId, + aliasRemapping, + }) + pipeline = materializedInput.pipeline.pipe( + map(([key, [value]]) => { + const row = { ...value } + delete row[INCLUDES_ROUTING] + return [key, row] + }), + ) as NamespacedAndKeyedStream + includesResults = [] + includesRoutingFns = [] + } + // Process the SELECT clause early - always create $selected // This eliminates duplication and allows for DISTINCT implementation if (query.fnSelect) { + const fnSelect = (row: NamespacedRow) => { + const selected = query.fnSelect!(row) + validateFnSelectResult(selected) + return selected + } // Handle functional select - apply the function to transform the row - pipeline = pipeline.pipe( - map(([key, namespacedRow]) => { - const selectResults = query.fnSelect!(namespacedRow) - validateFnSelectResult(selectResults) - let selected = selectResults - if (selectResults && typeof selectResults === `object`) { - selected = Array.isArray(selectResults) - ? [...selectResults] - : { ...selectResults } - const routing = (namespacedRow as any)[INCLUDES_ROUTING] - if (routing) { - selected[INCLUDES_ROUTING] = routing - } - if (directIncludes.length > 0) { - Object.defineProperty(selected, FN_SELECT_STATE, { - value: { - sourceRow: namespacedRow, - fnSelect: query.fnSelect!, - }, - enumerable: true, - configurable: true, - }) - } + const projectRow = (namespacedRow: NamespacedRow) => { + const callbackRow = sourceCarriesInternalRouteState + ? (stripInternalCallbackMetadata(namespacedRow) as NamespacedRow) + : namespacedRow + const selectResults = fnSelect(callbackRow) + let selected = selectResults + if ( + selectResults && + typeof selectResults === `object` && + (Array.isArray(selectResults) || isPlainObject(selectResults)) + ) { + selected = Array.isArray(selectResults) + ? [...selectResults] + : { ...selectResults } + const routing = (namespacedRow as any)[INCLUDES_ROUTING] + if (routing) { + selected[INCLUDES_ROUTING] = routing } - return [ - key, - { - ...namespacedRow, - $selected: selected, - }, - ] as [string, typeof namespacedRow & { $selected: any }] - }), - ) + } + return { + ...namespacedRow, + $selected: selected, + } + } + pipeline = pipeline.pipe(map(([key, row]) => [key, projectRow(row)])) } else if (query.select) { pipeline = processSelect(pipeline, query.select, allInputs) } else { @@ -962,21 +1008,10 @@ export function compileQuery( if (includesRoutingFns.length > 0) { pipeline = pipeline.pipe( map(([key, namespacedRow]: any) => { - const routing: Record< - string, - { - active: boolean - correlationKey: unknown - parentContext: Record | null - } - > = {} - for (const { fieldName, getRouting } of includesRoutingFns) { - routing[fieldName] = getRouting(namespacedRow) - } const selected = Array.isArray(namespacedRow.$selected) ? [...namespacedRow.$selected] : { ...namespacedRow.$selected } - selected[INCLUDES_ROUTING] = routing + selected[INCLUDES_ROUTING] = getRowIncludesRouting(namespacedRow) return [key, { ...namespacedRow, $selected: selected }] }), ) @@ -984,35 +1019,33 @@ export function compileQuery( // Process the GROUP BY clause if it exists. // When in includes mode (parentKeyStream), pass mainSource so that groupBy - // preserves __correlationKey for per-parent aggregation. + // preserves route metadata for per-parent aggregation. const groupByMainSource = parentKeyStream ? mainSource : undefined if (query.groupBy && query.groupBy.length > 0) { pipeline = processGroupBy( pipeline, query.groupBy, + valueIdentity, query.having, query.select, query.fnHaving, mainCollectionId, groupByMainSource, + sourceCarriesInternalRouteState || includesRoutingFns.length > 0, ) - } else if (query.select) { - // Check if SELECT contains aggregates but no GROUP BY (implicit single-group aggregation) - const hasAggregates = Object.values(query.select).some( - (expr) => expr.type === `agg` || containsAggregate(expr), + } else if (selectHasAggregates) { + // SELECT contains aggregates but no GROUP BY: implicit single-group aggregation + pipeline = processGroupBy( + pipeline, + [], // Empty group by means single group + valueIdentity, + query.having, + query.select, + query.fnHaving, + mainCollectionId, + groupByMainSource, + sourceCarriesInternalRouteState || includesRoutingFns.length > 0, ) - if (hasAggregates) { - // Handle implicit single-group aggregation - pipeline = processGroupBy( - pipeline, - [], // Empty group by means single group - query.having, - query.select, - query.fnHaving, - mainCollectionId, - groupByMainSource, - ) - } } // Process the HAVING clause if it exists (only applies after GROUP BY) @@ -1037,7 +1070,11 @@ export function compileQuery( for (const fnHaving of query.fnHaving) { pipeline = pipeline.pipe( filter(([_key, namespacedRow]) => { - return fnHaving(namespacedRow) + const callbackRow = + sourceCarriesInternalRouteState || includesRoutingFns.length > 0 + ? (stripInternalCallbackMetadata(namespacedRow) as NamespacedRow) + : namespacedRow + return fnHaving(callbackRow) }), ) } @@ -1048,7 +1085,7 @@ export function compileQuery( // the same key would otherwise keep the old value and hide route or order // changes. Joined contributors may differ in unselected namespaces; only // the public value and its route/order inputs must be congruent. - if (!query.select || !containsAggregate(query.select)) { + if (!selectHasAggregates) { pipeline = canonicalizeSelectedRows( pipeline, query, @@ -1058,7 +1095,7 @@ export function compileQuery( } const keyedSourceWhereClauses = keyWhereClausesBySource( - rawQuery, + rawSources, sourceWhereClauses, aliasRemapping, ) @@ -1068,7 +1105,35 @@ export function compileQuery( pipeline = pipeline.pipe(distinct(([_key, row]) => row.$selected)) } - // Process orderBy parameter if it exists + const finalizeRow = ( + key: unknown, + row: Record, + orderByIndex: string | undefined, + ) => { + const finalResults = attachVirtualPropsToSelected( + unwrapValue(row.$selected), + row, + ) + // When in includes mode, embed the correlation key and parentContext + if (parentKeyStream) { + return [ + key, + [ + stripInternalRouteMetadata(finalResults), + orderByIndex, + getRowCorrelationKey(row, mainSource), + getRowParentContext(row, mainSource), + getIncludesPublicKey(row, mainSource, key), + ], + ] as any + } + return [key, [finalResults, orderByIndex]] as [ + unknown, + [any, string | undefined], + ] + } + + let resultPipeline: ResultStream if (query.orderBy && query.orderBy.length > 0) { // When in includes mode with limit/offset, use grouped ordering so that // the limit is applied per parent (per correlation key), not globally. @@ -1076,18 +1141,25 @@ export function compileQuery( parentKeyStream && (query.limit !== undefined || query.offset !== undefined) ? (_key: unknown, row: unknown) => { - const correlationKey = - (row as any)?.[mainSource]?.__correlationKey ?? - (row as any)?.__correlationKey - const parentContext = (row as any)?.__parentContext + const correlationKey = getRowCorrelationKey( + row as NamespacedRow, + mainSource, + ) + const parentContext = getRowParentContext( + row as NamespacedRow, + mainSource, + ) if (parentContext != null) { - return serializeValue([correlationKey, parentContext]) + return serializeValue([ + valueIdentity.equality(correlationKey), + getParentContextIdentity(parentContext), + ]) } - return correlationKey + return valueIdentity.equality(correlationKey) } : undefined - const orderedPipeline = processOrderBy( + resultPipeline = processOrderBy( rawQuery, pipeline, query.orderBy, @@ -1098,90 +1170,22 @@ export function compileQuery( query.limit, query.offset, includesGroupKeyFn, - ) - - // Final step: extract the $selected and include orderBy index - const resultPipeline: ResultStream = orderedPipeline.pipe( - map(([key, [row, orderByIndex]]) => { - // Extract the final results from $selected and include orderBy index - const raw = (row as any).$selected - const finalResults = attachVirtualPropsToSelected( - unwrapValue(raw), - row as Record, - ) - // When in includes mode, embed the correlation key and parentContext - if (parentKeyStream) { - const correlationKey = - (row as any)[mainSource]?.__correlationKey ?? - (row as any).__correlationKey - const parentContext = (row as any).__parentContext ?? null - const publicKey = getIncludesPublicKey(row, mainSource, key) - const routedResults = stripInternalCorrelation(finalResults) - return [ - key, - [ - routedResults, - orderByIndex, - correlationKey, - parentContext, - publicKey, - ], - ] as any - } - return [key, [finalResults, orderByIndex]] as [unknown, [any, string]] - }), + ).pipe( + map(([key, [row, orderByIndex]]) => finalizeRow(key, row, orderByIndex)), ) as ResultStream - - // Cache the result before returning (use original query as key) - const compilationResult: CompilationResult = { - collectionId: mainCollectionId, - pipeline: resultPipeline, - sourceWhereClauses: keyedSourceWhereClauses, - aliasToCollectionId, - aliasRemapping, - includes: includesResults.length > 0 ? includesResults : undefined, - } - if (parentKeyStream === undefined) cache.set(rawQuery, compilationResult) - - return compilationResult } else if (query.limit !== undefined || query.offset !== undefined) { - // If there's a limit or offset without orderBy, throw an error throw new LimitOffsetRequireOrderByError() + } else { + resultPipeline = pipeline.pipe( + map(([key, row]) => finalizeRow(key, row, undefined)), + ) as ResultStream } - // Final step: extract the $selected and return tuple format (no orderBy) - const resultPipeline: ResultStream = pipeline.pipe( - map(([key, row]) => { - // Extract the final results from $selected and return [key, [results, undefined]] - const raw = (row as any).$selected - const finalResults = attachVirtualPropsToSelected( - unwrapValue(raw), - row as Record, - ) - // When in includes mode, embed the correlation key and parentContext - if (parentKeyStream) { - const correlationKey = - (row as any)[mainSource]?.__correlationKey ?? - (row as any).__correlationKey - const parentContext = (row as any).__parentContext ?? null - const publicKey = getIncludesPublicKey(row, mainSource, key) - const routedResults = stripInternalCorrelation(finalResults) - return [ - key, - [routedResults, undefined, correlationKey, parentContext, publicKey], - ] as any - } - return [key, [finalResults, undefined]] as [ - unknown, - [any, string | undefined], - ] - }), - ) - // Cache the result before returning (use original query as key) const compilationResult: CompilationResult = { collectionId: mainCollectionId, pipeline: resultPipeline, + valueIdentity, sourceWhereClauses: keyedSourceWhereClauses, aliasToCollectionId, aliasRemapping, @@ -1192,12 +1196,18 @@ export function compileQuery( return compilationResult } +function isInlineInclude(include: IncludesCompilationResult): boolean { + return ( + include.materialization !== `collection` && + (include.childCompilationResult.includes ?? []).every(isInlineInclude) + ) +} + function keyWhereClausesBySource( - query: QueryIR, + sources: Array, clauses: Map>, aliasRemapping: Record, ): Map> { - const sources = collectCollectionSources(query) const sourceIds = new Set(sources.map(({ sourceId }) => sourceId)) const result = new Map>() for (const [key, clause] of clauses) { @@ -1215,10 +1225,10 @@ function keyWhereClausesBySource( } function bindSourceInputs( - query: QueryIR, + sources: Array, inputs: Record, ): void { - for (const source of collectCollectionSources(query)) { + for (const source of sources) { const input = inputs[source.sourceId] ?? inputs[source.alias] if (!input) continue inputs[source.sourceId] = input @@ -1239,10 +1249,10 @@ function canonicalizeSelectedRows( value: row.$selected, routing: row.$selected?.[INCLUDES_ROUTING], outerCorrelation: isIncludedRelation - ? (row[mainSource]?.__correlationKey ?? row.__correlationKey) + ? getRowCorrelationKey(row, mainSource) : undefined, parentContext: isIncludedRelation - ? (row.__parentContext ?? row[mainSource]?.__parentContext ?? null) + ? getRowParentContext(row, mainSource) : undefined, order: compiledOrder.map((evaluate) => evaluate(row)), }) @@ -1391,6 +1401,7 @@ function processFromClause( isUnionFrom: boolean isParentRouted: boolean } { + const valueIdentity = getCompilationValueIdentity(cache) if (from.type === `unionAll`) { return processUnionAll( from, @@ -1488,6 +1499,7 @@ function processFromClause( wrapInputWithAlias(input, alias), parentKeyStream, alias, + valueIdentity, ) : wrapInputWithAlias(input, alias) const branch = routedBranch.pipe( @@ -1643,17 +1655,19 @@ function wrapInputWithAlias( const inputRow: unknown = row const scalar = getRoutedScalarMetadata(inputRow) if (scalar) { - const nsRow = { - [alias]: scalar.value, - __correlationKey: scalar.correlationKey, - __parentContext: scalar.parentContext, - [INCLUDES_PUBLIC_KEY]: scalar.publicKey, - } as unknown as NamespacedRow + const nsRow = attachRouteMetadata( + { + [alias]: scalar.value, + [INCLUDES_PUBLIC_KEY]: scalar.publicKey, + }, + scalar.correlationKey, + scalar.parentContext, + ) as unknown as NamespacedRow if ( scalar.parentContext != null && typeof scalar.parentContext === `object` ) { - Object.assign(nsRow, scalar.parentContext) + Object.assign(nsRow, getParentContextValue(scalar.parentContext)) } return [key, nsRow] as [unknown, NamespacedRow] } @@ -1662,14 +1676,18 @@ function wrapInputWithAlias( return [key, { [alias]: inputRow }] as [unknown, NamespacedRow] } - // Initialize the record with a nested structure. - // If __parentContext exists (from parent-referencing includes), merge parent - // aliases into the namespaced row so WHERE can resolve parent refs. - const { __parentContext, ...cleanRow } = row as any + // Initialize the record with a nested structure. Route metadata remains + // outside the user namespace while projected parent aliases stay visible. + const route = getRouteMetadata(inputRow) + const cleanRow = route + ? stripRouteMetadata(inputRow as Record) + : inputRow const nsRow: Record = { [alias]: cleanRow } - if (__parentContext) { - Object.assign(nsRow, __parentContext) - ;(nsRow as any).__parentContext = __parentContext + if (route?.parentContext != null) { + Object.assign(nsRow, getParentContextValue(route.parentContext)) + } + if (route) { + attachRouteMetadata(nsRow, route.correlationKey, route.parentContext) } return [key, nsRow] as [unknown, Record] }), @@ -1854,13 +1872,18 @@ function attachVirtualPropsToSelected( selected: any, row: Record, ): any { - if (!selected || typeof selected !== `object`) { + if ( + !selected || + typeof selected !== `object` || + (!Array.isArray(selected) && !isPlainObject(selected)) + ) { return selected } + const selectedRecord = selected as Record let needsMerge = false for (const prop of VIRTUAL_PROP_NAMES) { - if (selected[prop] == null && prop in row) { + if (selectedRecord[prop] == null && prop in row) { needsMerge = true break } @@ -1870,9 +1893,11 @@ function attachVirtualPropsToSelected( return selected } - const result = Array.isArray(selected) ? [...selected] : { ...selected } + const result = ( + Array.isArray(selected) ? [...selected] : { ...selected } + ) as Record for (const prop of VIRTUAL_PROP_NAMES) { - if (selected[prop] == null && prop in row) { + if (selectedRecord[prop] == null && prop in row) { result[prop] = row[prop] } } @@ -1880,24 +1905,6 @@ function attachVirtualPropsToSelected( return result } -function stripInternalCorrelation(selected: any): any { - if ( - !selected || - typeof selected !== `object` || - (!(`__correlationKey` in selected) && - !(`__parentContext` in selected) && - !(INCLUDES_PUBLIC_KEY in selected)) - ) { - return selected - } - - const result = Array.isArray(selected) ? [...selected] : { ...selected } - delete result.__correlationKey - delete result.__parentContext - delete result[INCLUDES_PUBLIC_KEY] - return result -} - function getIncludesPublicKey( row: Record, mainSource: string, @@ -1948,35 +1955,6 @@ function mapNestedQueries( } } -function getRefFromAlias( - query: QueryIR, - alias: string, -): CollectionRef | QueryRef | void { - for (const source of getFromSources(query.from)) { - if (source.alias === alias) { - return source - } - } - - for (const join of query.join || []) { - if (join.from.alias === alias) { - return join.from - } - } -} - -function getFromSources( - from: QueryIR[`from`], -): Array { - if (from.type === `unionFrom`) { - return from.sources - } - if (from.type === `unionAll`) { - return [] - } - return [from] -} - function getAllSources(query: QueryIR): Array { return [ ...getFromSources(query.from), @@ -2143,57 +2121,6 @@ function mapNestedFromQueries( } } -/** - * Follows the given reference in a query - * until its finds the root field the reference points to. - * @returns The collection, its alias, and the path to the root field in this collection - */ -export function followRef( - query: QueryIR, - ref: PropRef, - collection: Collection, -): { collection: Collection; path: Array } | void { - if (ref.path.length === 0) { - return - } - - if (ref.path.length === 1) { - // This field should be part of this collection - const field = ref.path[0]! - // is it part of the select clause? - if (query.select) { - const selectedField = query.select[field] - if (selectedField && selectedField.type === `ref`) { - return followRef(query, selectedField, collection) - } - } - - // Either this field is not part of the select clause - // and thus it must be part of the collection itself - // or it is part of the select but is not a reference - // so we can stop here and don't have to follow it - return { collection, path: [field] } - } - - if (ref.path.length > 1) { - // This is a nested field - const [alias, ...rest] = ref.path - const aliasRef = getRefFromAlias(query, alias!) - if (!aliasRef) { - return - } - - if (aliasRef.type === `queryRef`) { - return followRef(aliasRef.query, new PropRef(rest), collection) - } else { - // This is a reference to a collection - // we can't follow it further - // so the field must be on the collection itself - return { collection: aliasRef.collection, path: rest } - } - } -} - /** * Walks a Select object to find IncludesSubquery entries. * Plain nested objects still reject includes, but ConditionalSelect branches can @@ -2487,16 +2414,37 @@ function getNestedValue(obj: any, path: Array): any { return value } -function matchesConditionalSelectGuards( - guards: Array<{ - condition: (row: any) => any - expected: boolean - }>, - row: any, -): boolean { - return guards.every( - (guard) => isCaseWhenConditionTrue(guard.condition(row)) === guard.expected, - ) +type IncludeRouting = { + active: boolean + correlationKey: unknown + parentContext: Record | null +} + +/** + * Compiles a select-branch guard set once and resolves the include route only + * for rows whose guards hold. Every other row is routed as inactive. + */ +function compileGuardedRouting( + guards: Array, + resolve: (nsRow: any) => IncludeRouting | undefined, +): (nsRow: any) => IncludeRouting { + const compiledGuards = guards.map((guard) => ({ + condition: compileExpression(guard.condition), + expected: guard.expected, + })) + return (nsRow) => { + const active = compiledGuards.every( + (guard) => + isCaseWhenConditionTrue(guard.condition(nsRow)) === guard.expected, + ) + return ( + (active ? resolve(nsRow) : undefined) ?? { + active: false, + correlationKey: null, + parentContext: null, + } + ) + } } export type CompileQueryFn = typeof compileQuery diff --git a/packages/db/src/query/compiler/joins.ts b/packages/db/src/query/compiler/joins.ts index 45ce965878..4dfa935bc8 100644 --- a/packages/db/src/query/compiler/joins.ts +++ b/packages/db/src/query/compiler/joins.ts @@ -17,15 +17,26 @@ import { UnsupportedJoinTypeError, } from '../../errors.js' import { normalizeValue } from '../../utils/comparison.js' +import { + getParentContextIdentity, + getParentContextValue, +} from '../equality-value-identity.js' import { ensureIndexForField } from '../../indexes/auto-index.js' +import { getFromSources } from '../ir.js' import { compileExpression } from './evaluators.js' +import { getSourceAliasesFromExpression } from './expressions.js' import { getLazyLoadTargets } from './lazy-targets.js' import { crossJoinParentRoutes } from './parent-routes.js' import { INCLUDES_PUBLIC_KEY, + attachRouteMetadata, attachRouteMetadataToResult, + getNamespacedRouteMetadata, + getRouteMetadata, getRoutedScalarMetadata, + stripRouteMetadata, } from './route-metadata.js' +import type { ValueIdentity } from '../equality-value-identity.js' import type { CompileQueryFn } from './index.js' import type { OrderByOptimizationInfo } from './order-by.js' import type { @@ -63,18 +74,25 @@ let nextLazyDemandPlanId = 0 function parameterizeJoinInputByParentRoutes( input: KeyedStream, parentKeyStream: KeyedStream, + valueIdentity: ValueIdentity, ): KeyedStream { return crossJoinParentRoutes( input, parentKeyStream, (rowKey, row, correlationKey, parentContext) => { return [ - serializeValue([rowKey, correlationKey, parentContext]), - { - ...(row as Record), - __correlationKey: correlationKey, - __parentContext: parentContext, - }, + serializeValue([ + valueIdentity.equality(rowKey), + valueIdentity.equality(correlationKey), + getParentContextIdentity(parentContext), + ]), + attachRouteMetadata( + { + ...(row as Record), + }, + correlationKey, + parentContext, + ), ] }, ) @@ -83,17 +101,19 @@ function parameterizeJoinInputByParentRoutes( function wrapJoinedInputRow(alias: string, row: any): NamespacedRow { const scalar = getRoutedScalarMetadata(row) if (scalar) { - const namespaced = { - [alias]: scalar.value, - __correlationKey: scalar.correlationKey, - __parentContext: scalar.parentContext, - [INCLUDES_PUBLIC_KEY]: scalar.publicKey, - } as unknown as NamespacedRow + const namespaced = attachRouteMetadata( + { + [alias]: scalar.value, + [INCLUDES_PUBLIC_KEY]: scalar.publicKey, + }, + scalar.correlationKey, + scalar.parentContext, + ) as unknown as NamespacedRow if ( scalar.parentContext != null && typeof scalar.parentContext === `object` ) { - Object.assign(namespaced, scalar.parentContext) + Object.assign(namespaced, getParentContextValue(scalar.parentContext)) } return namespaced } @@ -102,11 +122,14 @@ function wrapJoinedInputRow(alias: string, row: any): NamespacedRow { return { [alias]: row } } - const { __parentContext, ...cleanRow } = row + const route = getRouteMetadata(row) + const cleanRow = route ? stripRouteMetadata(row) : row const namespaced: NamespacedRow = { [alias]: cleanRow } - if (__parentContext != null) { - Object.assign(namespaced, __parentContext) - namespaced.__parentContext = __parentContext + if (route?.parentContext != null) { + Object.assign(namespaced, getParentContextValue(route.parentContext)) + } + if (route) { + attachRouteMetadata(namespaced, route.correlationKey, route.parentContext) } return namespaced } @@ -115,11 +138,13 @@ function getRouteJoinKey( row: NamespacedRow, source: string, value: unknown, + valueIdentity: ValueIdentity, ): string { + const route = getNamespacedRouteMetadata(row, source) return serializeValue([ - row[source]?.__correlationKey ?? row.__correlationKey, - row.__parentContext ?? row[source]?.__parentContext ?? null, - value, + valueIdentity.equality(route?.correlationKey), + getParentContextIdentity(route?.parentContext ?? null), + valueIdentity.equality(value), ]) } @@ -164,6 +189,7 @@ export function processJoins( aliasRemapping: Record, sourceWhereClauses: Map>, mainSourceIsParentFiltered: boolean, + valueIdentity: ValueIdentity, parentKeyStream?: KeyedStream, ): NamespacedAndKeyedStream { let resultPipeline = pipeline @@ -190,6 +216,7 @@ export function processJoins( aliasRemapping, sourceWhereClauses, mainSourceIsParentFiltered, + valueIdentity, parentKeyStream, ) } @@ -222,6 +249,7 @@ function processJoin( aliasRemapping: Record, sourceWhereClauses: Map>, mainSourceIsParentFiltered: boolean, + valueIdentity: ValueIdentity, parentKeyStream?: KeyedStream, ): NamespacedAndKeyedStream { const isCollectionRef = joinClause.from.type === `collectionRef` @@ -263,6 +291,7 @@ function processJoin( aliasToCollectionId, aliasRemapping, sourceWhereClauses, + valueIdentity, routeJoinedSource ? parentKeyStream : undefined, ) @@ -310,7 +339,7 @@ function processJoin( // Extract the join key from the main source expression const value = normalizeValue(compiledMainExpr(namespacedRow)) const mainKey = routeJoinedSource - ? getRouteJoinKey(namespacedRow, mainSource, value) + ? getRouteJoinKey(namespacedRow, mainSource, value, valueIdentity) : value // Return [joinKey, [originalKey, namespacedRow]] @@ -330,7 +359,7 @@ function processJoin( // Extract the join key from the joined source expression const value = normalizeValue(compiledJoinedExpr(namespacedRow)) const joinedKey = routeJoinedSource - ? getRouteJoinKey(namespacedRow, joinedSource, value) + ? getRouteJoinKey(namespacedRow, joinedSource, value, valueIdentity) : value // Return [joinKey, [originalKey, namespacedRow]] @@ -407,7 +436,7 @@ function processJoin( tap((data) => { for (const [[joinKey], weight] of data.getInner()) { if (joinKey == null) continue - const encoded = serializeValue(joinKey) + const encoded = valueIdentity.serializeEquality(joinKey) const previous = demandWeights.get(encoded) const nextWeight = (previous?.weight ?? 0) + weight if (nextWeight === 0) { @@ -524,30 +553,6 @@ function analyzeJoinExpressions( throw new InvalidJoinCondition() } -/** - * Extracts the source alias from a join expression - */ -function getSourceAliasesFromExpression(expr: BasicExpression): Set { - switch (expr.type) { - case `ref`: - // PropRef path has the source alias as the first element - return new Set(expr.path[0] ? [expr.path[0]] : []) - case `func`: { - // For function expressions, we need to check if all arguments refer to the same source - const sourceAliases = new Set() - for (const arg of expr.args) { - for (const alias of getSourceAliasesFromExpression(arg)) { - sourceAliases.add(alias) - } - } - return sourceAliases - } - default: - // Values (type='val') don't reference any source - return new Set() - } -} - /** * Processes the join source (collection or sub-query) */ @@ -566,6 +571,7 @@ function processJoinSource( aliasToCollectionId: Record, aliasRemapping: Record, sourceWhereClauses: Map>, + valueIdentity: ValueIdentity, parentKeyStream?: KeyedStream, ): { alias: string; input: KeyedStream; collectionId: string } { switch (from.type) { @@ -582,7 +588,11 @@ function processJoinSource( return { alias: from.alias, input: parentKeyStream - ? parameterizeJoinInputByParentRoutes(input, parentKeyStream) + ? parameterizeJoinInputByParentRoutes( + input, + parentKeyStream, + valueIdentity, + ) : input, collectionId: from.collection.id, } @@ -692,15 +702,7 @@ function processJoinSource( } function getFirstFromAlias(query: QueryIR): string | undefined { - if (query.from.type === `unionFrom`) { - return query.from.sources[0]?.alias - } - - if (query.from.type === `unionAll`) { - return undefined - } - - return query.from.alias + return getFromSources(query.from)[0]?.alias } /** diff --git a/packages/db/src/query/compiler/lazy-targets.ts b/packages/db/src/query/compiler/lazy-targets.ts index cf4f5cfc4c..241ccd7764 100644 --- a/packages/db/src/query/compiler/lazy-targets.ts +++ b/packages/db/src/query/compiler/lazy-targets.ts @@ -1,4 +1,4 @@ -import { PropRef, followRef } from '../ir.js' +import { PropRef, followRef, getFromSources } from '../ir.js' import type { BasicExpression, CollectionRef, @@ -189,14 +189,7 @@ function getSourceFromAlias( } } - const from = query.from - const sources = - from.type === `unionFrom` - ? from.sources - : from.type === `unionAll` - ? [] - : [from] - return sources.find((source) => source.alias === alias) + return getFromSources(query.from).find((source) => source.alias === alias) } function resolveLazySource( @@ -227,11 +220,7 @@ function findCollectionSource( collection: Collection, ): CollectionRef | undefined { const sources = [ - ...(query.from.type === `unionFrom` - ? query.from.sources - : query.from.type === `unionAll` - ? [] - : [query.from]), + ...getFromSources(query.from), ...(query.join?.map((join) => join.from) ?? []), ] diff --git a/packages/db/src/query/compiler/order-by.ts b/packages/db/src/query/compiler/order-by.ts index 8595d0ff26..fb4843fbe0 100644 --- a/packages/db/src/query/compiler/order-by.ts +++ b/packages/db/src/query/compiler/order-by.ts @@ -3,10 +3,17 @@ import { orderByWithFractionalIndex, } from '@tanstack/db-ivm' import { defaultComparator, makeComparator } from '../../utils/comparison.js' -import { PropRef, collectCollectionSources, followRef } from '../ir.js' +import { + PropRef, + collectCollectionSources, + followRef, + getWhereExpression, + isResidualWhere, +} from '../ir.js' import { ensureIndexForField } from '../../indexes/auto-index.js' import { findIndexForField } from '../../utils/index-optimization.js' import { compileExpression } from './evaluators.js' +import { getSourceAliasesFromExpression } from './expressions.js' import { replaceAggregatesByRefs } from './group-by.js' import type { CompareOptions } from '../builder/types.js' import type { WindowOptions } from './types.js' @@ -18,7 +25,7 @@ import type { NamespacedRow, } from '../../types.js' import type { IStreamBuilder, KeyValue } from '@tanstack/db-ivm' -import type { IndexInterface } from '../../indexes/base-index.js' +import type { IndexReader } from '../../indexes/base-index.js' import type { Collection } from '../../collection/index.js' export type OrderByOptimizationInfo = { @@ -33,11 +40,13 @@ export type OrderByOptimizationInfo = { ) => number /** Extracts all orderBy column values from a raw row (array for multi-column) */ valueExtractorForRawRow: (row: Record) => unknown - /** Extracts only the first column value - used for index-based cursor */ - firstColumnValueExtractor: (row: Record) => unknown /** Index on the first orderBy column - used for lazy loading */ - index?: IndexInterface + index?: IndexReader dataNeeded?: () => number + /** Reads the source loader's synchronous request guard, when installed. */ + isRequesting?: () => boolean + /** Whether local operators can discard or reorder the provider's prefix. */ + requiresFullSource: boolean } /** @@ -70,7 +79,6 @@ export function processOrderBy( compareOptions: buildCompareOptions(clause, collection), } }) - // Create a value extractor function for the orderBy operator const valueExtractor = (row: NamespacedRow & { $selected?: any }) => { // The namespaced row contains: @@ -134,14 +142,13 @@ export function processOrderBy( // Skip this optimization when using grouped ordering (includes with limit), // because the limit is per-group, not global — the child collection needs all data loaded. if ( - limit && + limit !== undefined && !groupKeyFn && rawQuery.from.type !== `unionFrom` && rawQuery.from.type !== `unionAll` ) { - let index: IndexInterface | undefined + let index: IndexReader | undefined let followRefCollection: Collection | undefined - let firstColumnValueExtractor: CompiledSingleRowExpression | undefined let orderByAlias: string = rawQuery.from.alias let orderBySourceId: string | undefined @@ -149,170 +156,128 @@ export function processOrderBy( const firstClause = orderByClause[0]! const firstOrderByExpression = firstClause.expression - if (firstOrderByExpression.type === `ref`) { - const followRefResult = followRef( - rawQuery, - firstOrderByExpression, - collection, - ) - - if (followRefResult) { - followRefCollection = followRefResult.collection - orderBySourceId = followRefResult.sourceId - const fieldName = followRefResult.path[0] - const compareOpts = buildCompareOptions( - firstClause, - followRefCollection, - ) - - if (fieldName) { - // Use a single-column comparator for the index, not the - // multi-column `compare` function. The multi-column comparator - // expects array values [col1, col2, ...] but the index stores - // individual field values. Passing `compare` here causes the - // BTree to treat all single values as equal (since number[0] - // === undefined for both sides of the comparison). - const firstColumnCompareFn = makeComparator(compareOpts) - ensureIndexForField( - fieldName, - followRefResult.path, - followRefCollection, - compareOpts, - firstColumnCompareFn, - ) - } - - // First column value extractor - used for index cursor - firstColumnValueExtractor = compileExpression( - new PropRef(followRefResult.path), - true, - ) as CompiledSingleRowExpression - - index = findIndexForField( - followRefCollection, + const followRefResult = + firstOrderByExpression.type === `ref` + ? followRef(rawQuery, firstOrderByExpression, collection) + : undefined + if (firstOrderByExpression.type === `ref` && followRefResult) { + followRefCollection = followRefResult.collection + orderBySourceId = followRefResult.sourceId + const fieldName = followRefResult.path[0] + // The query's first source defines implicit string collation for the + // whole order. Build the source index with that same resolved term so + // provider admission cannot disagree with emitted query order. + const compareOpts = buildCompareOptions(firstClause, collection) + + if (fieldName) { + // Use a single-column comparator for the index, not the + // multi-column `compare` function. The multi-column comparator + // expects array values [col1, col2, ...] but the index stores + // individual field values. Passing `compare` here causes the + // BTree to treat all single values as equal (since number[0] + // === undefined for both sides of the comparison). + const firstColumnCompareFn = makeComparator(compareOpts) + ensureIndexForField( + fieldName, followRefResult.path, + followRefCollection, compareOpts, + firstColumnCompareFn, ) + } - // Only use the index if it supports range queries - if (!index?.supports(`gt`)) { - index = undefined - } + index = findIndexForField( + followRefCollection, + followRefResult.path, + compareOpts, + ) - if (!index) { - const collectionId = followRefCollection.id - const fieldPath = followRefResult.path.join(`.`) - console.warn( - `[TanStack DB]${collectionId ? ` [${collectionId}]` : ``} orderBy with limit requires an index on "${fieldPath}" for efficient lazy loading. ` + - `Falling back to loading all data. ` + - `Consider creating an index on the collection with collection.createIndex((row) => row.${fieldPath}) ` + - `or enable auto-indexing with autoIndex: 'eager' and a defaultIndexType.`, - ) - } + // Only use the index if it supports range queries + if (!index?.supports(`gt`)) { + index = undefined + } - orderByAlias = - firstOrderByExpression.path.length > 1 - ? String(firstOrderByExpression.path[0]) - : rawQuery.from.alias - orderBySourceId ??= collectCollectionSources(rawQuery).find( - (source) => - source.alias === orderByAlias && - source.collection === followRefCollection, - )?.sourceId + if (!index) { + const collectionId = followRefCollection.id + const fieldPath = followRefResult.path.join(`.`) + console.warn( + `[TanStack DB]${collectionId ? ` [${collectionId}]` : ``} orderBy with limit requires an index on "${fieldPath}" for efficient lazy loading. ` + + `Falling back to loading all data. ` + + `Consider creating an index on the collection with collection.createIndex((row) => row.${fieldPath}) ` + + `or enable auto-indexing with autoIndex: 'eager' and a defaultIndexType.`, + ) } + + orderByAlias = + firstOrderByExpression.path.length > 1 + ? String(firstOrderByExpression.path[0]) + : rawQuery.from.alias + orderBySourceId ??= collectCollectionSources(rawQuery).find( + (source) => + source.alias === orderByAlias && + source.collection === followRefCollection, + )?.sourceId } - // Only create comparator and value extractors if the first column is a ref expression - // For aggregate or computed expressions, we can't extract values from raw collection rows - if (!firstColumnValueExtractor) { - // Skip optimization for non-ref expressions (aggregates, computed values, etc.) - // The query will still work, but without lazy loading optimization - } else if (orderBySourceId) { - // Build value extractors for all columns (must all be ref expressions for multi-column) - // Check if all orderBy expressions are ref types (required for multi-column extraction) - const allColumnsAreRefs = orderByClause.every( - (clause) => clause.expression.type === `ref`, + if (orderBySourceId && followRefResult) { + const sourceOrderBy = resolveOrderBy( + orderByClause, + collection.compareOptions, ) - - // Create extractors for all columns if they're all refs - const allColumnExtractors: - | Array - | undefined = allColumnsAreRefs - ? orderByClause.map((clause) => { - // We know it's a ref since we checked allColumnsAreRefs - const refExpr = clause.expression as PropRef - const followResult = followRef(rawQuery, refExpr, collection) - if (followResult) { - return compileExpression( - new PropRef(followResult.path), - true, - ) as CompiledSingleRowExpression - } - // Fallback for refs that don't follow - return compileExpression( - clause.expression, - true, - ) as CompiledSingleRowExpression - }) - : undefined - - // Create a comparator for raw rows (used for tracking sent values) - // This compares ALL orderBy columns for proper ordering - const comparator = ( + const sourceOrderIsDirect = orderByClause.every(({ expression }) => { + if (expression.type !== `ref`) return false + return ( + followRef(rawQuery, expression, collection)?.sourceId === + orderBySourceId + ) + }) + const extract = compileExpression( + new PropRef(followRefResult.path), + true, + ) as CompiledSingleRowExpression + const compareTerm = makeComparator(sourceOrderBy[0]!.compareOptions) + const compareSourceRows = ( a: Record | null | undefined, b: Record | null | undefined, - ) => { - if (orderByClause.length === 1) { - // Single column: extract and compare - const extractedA = a ? firstColumnValueExtractor(a) : a - const extractedB = b ? firstColumnValueExtractor(b) : b - return compare(extractedA, extractedB) - } - if (allColumnExtractors) { - // Multi-column with all refs: extract all values and compare - const extractAll = ( - row: Record | null | undefined, - ) => { - if (!row) return row - return allColumnExtractors.map((extractor) => extractor(row)) - } - return compare(extractAll(a), extractAll(b)) - } - // Fallback: can't compare (shouldn't happen since we skip non-ref cases) - return 0 - } + ) => compareTerm(a ? extract(a) : a, b ? extract(b) : b) - // Create a value extractor for raw rows that extracts ALL orderBy column values - // This is used for tracking sent values and building composite cursors - const rawRowValueExtractor = (row: Record): unknown => { - if (orderByClause.length === 1) { - // Single column: return single value - return firstColumnValueExtractor(row) - } - if (allColumnExtractors) { - // Multi-column: return array of all values - return allColumnExtractors.map((extractor) => extractor(row)) - } - // Fallback (shouldn't happen) - return undefined - } - - orderByOptimizationInfo = { + const info: OrderByOptimizationInfo = { sourceId: orderBySourceId, alias: orderByAlias, offset: offset ?? 0, limit, - comparator, - valueExtractorForRawRow: rawRowValueExtractor, - firstColumnValueExtractor: firstColumnValueExtractor, + comparator: compareSourceRows, + valueExtractorForRawRow: extract, index, - orderBy: orderByClause, + orderBy: sourceOrderBy, + requiresFullSource: + !sourceOrderIsDirect || + rawQuery.from.type !== `collectionRef` || + rawQuery.from.sourceId !== orderBySourceId || + (rawQuery.join?.some( + ({ type }) => type === `inner` || type === `right`, + ) ?? + false) || + (rawQuery.where?.some( + (where) => + isResidualWhere(where) || + [ + ...getSourceAliasesFromExpression(getWhereExpression(where)), + ].some((alias) => alias !== orderByAlias), + ) ?? + false) || + (rawQuery.fnWhere?.length ?? 0) > 0 || + rawQuery.groupBy !== undefined || + rawQuery.having !== undefined || + rawQuery.fnHaving !== undefined || + rawQuery.distinct === true, } + orderByOptimizationInfo = info // Ordered loading is owned by one lexical source. A collection can occur // more than once in a query tree, so collection ID and alias are not // sufficient identities here. - optimizableOrderByCollections[orderBySourceId] = orderByOptimizationInfo + optimizableOrderByCollections[orderBySourceId] = info // Set up lazy loading callback to track how much more data is needed // This is used by loadMoreIfNeeded to determine if more data should be loaded @@ -323,7 +288,7 @@ export function processOrderBy( optimizableOrderByCollections[orderBySourceId]![`dataNeeded`] = () => { const size = getSize() - return Math.max(0, orderByOptimizationInfo!.limit - size) + return Math.max(0, info.limit - size) } } } @@ -397,13 +362,28 @@ export function buildCompareOptions( clause: OrderByClause, collection: CollectionLike, ): CompareOptions { - if (clause.compareOptions.stringSort !== undefined) { - return clause.compareOptions - } + return resolveCompareOptions(clause, collection.compareOptions) +} - return { - ...collection.compareOptions, - direction: clause.compareOptions.direction, - nulls: clause.compareOptions.nulls, - } +function resolveOrderBy( + orderBy: OrderBy, + defaults: CollectionLike[`compareOptions`], +): OrderBy { + return orderBy.map((clause) => ({ + expression: clause.expression, + compareOptions: resolveCompareOptions(clause, defaults), + })) +} + +function resolveCompareOptions( + clause: OrderByClause, + defaults: CollectionLike[`compareOptions`], +): CompareOptions { + return clause.compareOptions.stringSort === undefined + ? { + ...defaults, + direction: clause.compareOptions.direction, + nulls: clause.compareOptions.nulls, + } + : clause.compareOptions } diff --git a/packages/db/src/query/compiler/route-metadata.ts b/packages/db/src/query/compiler/route-metadata.ts index 703c7bd037..6d6c797330 100644 --- a/packages/db/src/query/compiler/route-metadata.ts +++ b/packages/db/src/query/compiler/route-metadata.ts @@ -1,13 +1,34 @@ +import { isPlainObject } from '../../utils/type-guards.js' + const ROUTED_SCALAR_VALUE = Symbol(`tanstack_db_routed_scalar_value`) +const ROUTE_METADATA = Symbol(`tanstack_db_route_metadata`) export const INCLUDES_PUBLIC_KEY = Symbol(`includesPublicKey`) +export const INCLUDES_ROUTING = Symbol(`includesRouting`) +const INTERNAL_ROUTE_KEYS = new Set([ + ROUTE_METADATA, + INCLUDES_PUBLIC_KEY, +]) +const INTERNAL_CALLBACK_KEYS = new Set([ + ...INTERNAL_ROUTE_KEYS, + INCLUDES_ROUTING, +]) -type RoutedScalarResult = { +type RoutedResult = { [ROUTED_SCALAR_VALUE]: unknown - __correlationKey: unknown - __parentContext: unknown + [ROUTE_METADATA]: RouteMetadata [INCLUDES_PUBLIC_KEY]: unknown } +type PublicContainerProperty = { + descriptor: PropertyDescriptor + value?: { original: unknown; replacement: unknown } +} + +export type RouteMetadata = { + correlationKey: unknown + parentContext: unknown +} + export type RoutedScalarMetadata = { value: unknown correlationKey: unknown @@ -15,6 +36,45 @@ export type RoutedScalarMetadata = { publicKey: unknown } +export function attachRouteMetadata( + value: T, + correlationKey: unknown, + parentContext: unknown, +): T { + return Object.assign(value, { + [ROUTE_METADATA]: { correlationKey, parentContext } satisfies RouteMetadata, + }) +} + +export function getRouteMetadata(value: unknown): RouteMetadata | undefined { + if ( + value == null || + typeof value !== `object` || + !(ROUTE_METADATA in value) + ) { + return undefined + } + return (value as { [ROUTE_METADATA]: RouteMetadata })[ROUTE_METADATA] +} + +export function getNamespacedRouteMetadata( + row: unknown, + source: string, +): RouteMetadata | undefined { + return ( + getRouteMetadata(row) ?? + (row != null && typeof row === `object` + ? getRouteMetadata((row as Record)[source]) + : undefined) + ) +} + +export function stripRouteMetadata(value: T): T { + const result = { ...value } as T & Record + delete result[ROUTE_METADATA] + return result +} + export function attachRouteMetadataToResult( value: unknown, correlationKey: unknown, @@ -29,21 +89,19 @@ export function attachRouteMetadataToResult( return value } - if (value != null && typeof value === `object`) { + if (isPlainObject(value)) { return { ...value, - __correlationKey: correlationKey, - __parentContext: parentContext, + [ROUTE_METADATA]: { correlationKey, parentContext }, [INCLUDES_PUBLIC_KEY]: publicKey, } } return { [ROUTED_SCALAR_VALUE]: value, - __correlationKey: correlationKey, - __parentContext: parentContext, + [ROUTE_METADATA]: { correlationKey, parentContext }, [INCLUDES_PUBLIC_KEY]: publicKey, - } satisfies RoutedScalarResult + } satisfies RoutedResult } export function getRoutedScalarMetadata( @@ -57,11 +115,117 @@ export function getRoutedScalarMetadata( return undefined } - const routed = value as RoutedScalarResult + const routed = value as RoutedResult + const route = routed[ROUTE_METADATA] return { value: routed[ROUTED_SCALAR_VALUE], - correlationKey: routed.__correlationKey, - parentContext: routed.__parentContext, + correlationKey: route.correlationKey, + parentContext: route.parentContext, publicKey: routed[INCLUDES_PUBLIC_KEY], } } + +/** Copy public containers while removing private route state at every depth. */ +export function stripInternalRouteMetadata(value: unknown): unknown { + return transformPublicContainers(value, (leaf) => leaf, INTERNAL_ROUTE_KEYS) +} + +/** Remove every compiler-owned key before invoking user code. */ +export function stripInternalCallbackMetadata(value: unknown): unknown { + return transformPublicContainers( + value, + (leaf) => leaf, + INTERNAL_CALLBACK_KEYS, + ) +} + +/** Copy only paths changed by a leaf transform or an omitted private key. */ +export function transformPublicContainers( + value: unknown, + transformLeaf: (value: unknown) => unknown, + omittedKeys: ReadonlySet, +): unknown { + const rootReplacement = transformLeaf(value) + if (rootReplacement !== value) return rootReplacement + if (!isPublicContainer(value)) return value + + const parents = new WeakMap>() + const properties = new WeakMap< + object, + Map + >() + const visited = new WeakSet() + const dirty = new Set() + const visit = (current: object): void => { + if (visited.has(current)) return + visited.add(current) + const currentProperties = new Map() + properties.set(current, currentProperties) + for (const key of Reflect.ownKeys(current)) { + if (omittedKeys.has(key)) { + dirty.add(current) + continue + } + const descriptor = Object.getOwnPropertyDescriptor(current, key) + if (!descriptor) continue + const property: PublicContainerProperty = { descriptor } + currentProperties.set(key, property) + if (!descriptor.enumerable || !(`value` in descriptor)) continue + const child = descriptor.value + const replacement = transformLeaf(child) + property.value = { original: child, replacement } + if (replacement !== child) { + dirty.add(current) + continue + } + if (!isPublicContainer(child)) continue + const childParents = parents.get(child) ?? new Set() + childParents.add(current) + parents.set(child, childParents) + visit(child) + } + } + visit(value) + + const queue = [...dirty] + for (const current of queue) { + for (const parent of parents.get(current) ?? []) { + if (dirty.has(parent)) continue + dirty.add(parent) + queue.push(parent) + } + } + if (!dirty.has(value)) return value + + const copies = new WeakMap() + const copy = (current: object): object => { + if (!dirty.has(current)) return current + const existing = copies.get(current) + if (existing) return existing + + const result = Array.isArray(current) + ? [] + : Object.create(Object.getPrototypeOf(current)) + copies.set(current, result) + for (const [key, property] of properties.get(current) ?? []) { + const descriptor = { ...property.descriptor } + if (property.value) { + const { original, replacement } = property.value + descriptor.value = + replacement !== original + ? replacement + : isPublicContainer(original) + ? copy(original) + : original + } + Object.defineProperty(result, key, descriptor) + } + return result + } + + return copy(value) +} + +function isPublicContainer(value: unknown): value is object { + return Array.isArray(value) || isPlainObject(value) +} diff --git a/packages/db/src/query/compiler/select.ts b/packages/db/src/query/compiler/select.ts index de428ff515..8b7fb4e127 100644 --- a/packages/db/src/query/compiler/select.ts +++ b/packages/db/src/query/compiler/select.ts @@ -5,10 +5,7 @@ import { Value as ValClass, isExpressionLike, } from '../ir.js' -import { - AggregateNotSupportedError, - UnsafeAliasPathError, -} from '../../errors.js' +import { UnsafeAliasPathError } from '../../errors.js' import { compileExpression, isCaseWhenConditionTrue } from './evaluators.js' import { containsAggregate } from './group-by.js' import type { @@ -266,24 +263,6 @@ function isAggregateExpression( return expr.type === `agg` } -/** - * Processes a single argument in a function context - */ -export function processArgument( - arg: BasicExpression | Aggregate, - namespacedRow: NamespacedRow, -): any { - if (isAggregateExpression(arg)) { - throw new AggregateNotSupportedError() - } - - // Pre-compile the expression and evaluate immediately - const compiledExpression = compileExpression(arg) - const value = compiledExpression(namespacedRow) - - return value -} - /** * Helper function to check if an object is a nested select object * diff --git a/packages/db/src/query/effect.ts b/packages/db/src/query/effect.ts index 9e77bbc9eb..bc35dec6a3 100644 --- a/packages/db/src/query/effect.ts +++ b/packages/db/src/query/effect.ts @@ -1,26 +1,25 @@ import { D2, output } from '@tanstack/db-ivm' +import { createDeferred } from '../deferred.js' import { getActivePublicationContext, transactionScopedScheduler, } from '../scheduler.js' import { getActiveTransaction } from '../transactions.js' +import { runAllCallbacks } from '../utils/callbacks.js' +import { normalizeError } from '../utils/error.js' import { compileQuery } from './compiler/index.js' -import { - normalizeExpressionPaths, - normalizeOrderByPaths, -} from './compiler/expressions.js' +import { normalizeExpressionPaths } from './compiler/expressions.js' import { getCollectionBuilder } from './live/collection-registry.js' import { SubsetDemandController } from './live/subset-demand-controller.js' +import { OrderedSourceLoader } from './live/ordered-source-loader.js' import { buildQueryFromConfig, - computeOrderedLoadCursor, computeSubscriptionOrderByHints, extractCollectionSources, extractCollectionsFromQuery, - filterDuplicateInserts, + reconcileChangesForD2, sendChangesToInput, splitUpdates, - trackBiggestSentValue, } from './live/utils.js' import type { RootStreamBuilder } from '@tanstack/db-ivm' import type { Collection } from '../collection/index.js' @@ -140,7 +139,10 @@ export interface EffectConfig< /** Handle returned by createEffect */ export interface Effect { - /** Dispose the effect. Returns a promise that resolves when in-flight handlers complete. */ + /** + * Dispose the effect and await in-flight handlers. Calls during one cleanup + * attempt, including calls from abort/release callbacks, share its outcome. + */ dispose: () => Promise /** Whether this effect has been disposed */ readonly disposed: boolean @@ -253,17 +255,24 @@ export function createEffect< let disposalPromise: Promise | undefined const dispose = (): Promise => { if (disposalPromise) return disposalPromise + // Abort and source-release callbacks may synchronously call dispose again. + // Publish the shared result before entering either user callback boundary. + const completion = createDeferred() + const attempt = completion.promise + disposalPromise = attempt disposed = true // Abort signal for in-flight handlers abortController.abort() - disposalPromise = (async () => { + void (async () => { // Tear down the pipeline (unsubscribe from sources, etc.) + let cleanupFailed = false let cleanupError: unknown try { runner.dispose() } catch (error) { + cleanupFailed = true cleanupError = error } @@ -272,9 +281,15 @@ export function createEffect< await Promise.allSettled([...inFlightHandlers]) } - if (cleanupError !== undefined) throw cleanupError - })() - return disposalPromise + if (cleanupFailed) throw cleanupError + })().then(completion.resolve, completion.reject) + void attempt.then( + () => {}, + () => { + if (disposalPromise === attempt) disposalPromise = undefined + }, + ) + return attempt } // Create and start the pipeline @@ -382,16 +397,14 @@ class EffectPipelineRunner { > = {} // Ordered subscription state for cursor-based loading - private readonly biggestSentValue = new Map() - private readonly lastLoadRequestKey = new Map() - private pendingOrderedLoadPromise: Promise | undefined + private readonly orderedLoaders = new Map() // Subscription management private readonly unsubscribeCallbacks = new Set<() => void>() - // Duplicate insert prevention per lexical source - private readonly sentToD2KeysBySource = new Map< + // Exact row last contributed to D2 per lexical source key. + private readonly sentToD2RowsBySource = new Map< string, - Set + Map> >() // Output accumulator @@ -404,14 +417,11 @@ class EffectPipelineRunner { // Scheduler integration private subscribedToAllCollections = false private readonly builderDependencies = new Set() - private readonly sourceDependencies: Record> = {} // Reentrance guard private isGraphRunning = false private starting = false private disposed = false - // When dispose() is called mid-graph-run, defer heavy cleanup until the run completes - private deferredCleanup = false private readonly onBatchProcessed: ( events: Array>, @@ -513,17 +523,13 @@ class EffectPipelineRunner { const { sourceId, alias, collection } = source const collectionId = collection.id - // Initialise per-source duplicate tracking - this.sentToD2KeysBySource.set(sourceId, new Set()) + this.sentToD2RowsBySource.set(sourceId, new Map()) // Discover dependencies: if source collection is itself a live query // collection, its builder must run first during transaction flushes. const dependencyBuilder = getCollectionBuilder(collection) if (dependencyBuilder) { - this.sourceDependencies[sourceId] = [dependencyBuilder] this.builderDependencies.add(dependencyBuilder) - } else { - this.sourceDependencies[sourceId] = [] } // Get where clause for this alias (for predicate push-down) @@ -546,13 +552,18 @@ class EffectPipelineRunner { const orderByInfo = this.getOrderByInfoForSource(sourceId) // Build the change callback — for ordered aliases, split updates into - // delete+insert and track the biggest sent value for cursor positioning. + // delete+insert and invalidate loading state from changed contributions. const changeCallback = orderByInfo ? (changes: Array>) => { if (pendingBuffers.has(sourceId)) { pendingBuffers.get(sourceId)!.push(changes) } else { - this.trackSentValues(sourceId, changes, orderByInfo.comparator) + this.orderedLoaders + .get(sourceId) + ?.onSourceChanges( + changes, + this.sentToD2RowsBySource.get(sourceId), + ) const split = [...splitUpdates(changes)] this.handleSourceChanges(sourceId, split) } @@ -574,7 +585,7 @@ class EffectPipelineRunner { whereExpression, ), onLoadSubsetError: ({ error }) => { - this.onSourceError(normaliseError(error)) + this.onSourceError(normalizeError(error)) }, }) @@ -582,8 +593,8 @@ class EffectPipelineRunner { this.subscriptions[sourceId] = subscription const unsubscribe = () => { - subscription.unsubscribe() delete this.subscriptions[sourceId] + subscription.unsubscribe() } // subscribeChanges can synchronously report a source error and dispose @@ -613,7 +624,9 @@ class EffectPipelineRunner { // For ordered aliases with an index, trigger the initial limited snapshot. // This loads only the top N rows rather than the entire collection. if (orderByInfo) { - this.requestInitialOrderedSnapshot(alias, orderByInfo, subscription) + const loader = new OrderedSourceLoader(orderByInfo, subscription, alias) + this.orderedLoaders.set(sourceId, loader) + loader.start() } // Listen for status changes on source collections @@ -675,7 +688,9 @@ class EffectPipelineRunner { // through handleSourceChanges directly (not back into this buffer). for (const changes of buffer) { if (orderByInfo) { - this.trackSentValues(sourceId, changes, orderByInfo.comparator) + this.orderedLoaders + .get(sourceId) + ?.onSourceChanges(changes, this.sentToD2RowsBySource.get(sourceId)) const split = [...splitUpdates(changes)] this.sendChangesToD2(sourceId, split) } else { @@ -704,7 +719,7 @@ class EffectPipelineRunner { changes: Array>, ): void { this.sendChangesToD2(sourceId, changes) - this.scheduleGraphRun(sourceId) + this.scheduleGraphRun() } private setDemand( @@ -719,7 +734,7 @@ class EffectPipelineRunner { // The subscription error event already reports adapter failures and // disposes this effect. Do not let that query-local failure escape the // source commit, but keep unrelated graph errors visible. - if (subscription.lastError !== error) throw error + if (!Object.is(subscription.lastError, error)) throw error if (this.starting) throw error return } @@ -741,20 +756,12 @@ class EffectPipelineRunner { * Dependencies are discovered from source collections that are themselves * live query collections, ensuring parent queries run before effects. */ - private scheduleGraphRun(sourceId?: string): void { + private scheduleGraphRun(): void { const contextId = getActiveTransaction()?.id ?? getActivePublicationContext() - // Collect dependencies for this schedule call - const deps = new Set(this.builderDependencies) - if (sourceId) { - const sourceDeps = this.sourceDependencies[sourceId] - if (sourceDeps) { - for (const dep of sourceDeps) { - deps.add(dep) - } - } - } + // Snapshot before scheduling parents, which can reenter source setup. + const deps = [...this.builderDependencies] // Ensure dependent builders are scheduled in this context so that // dependency edges always point to a real job. @@ -801,11 +808,10 @@ class EffectPipelineRunner { const input = this.inputs[sourceId] if (!input) return 0 - // Filter duplicates per lexical source - const sentKeys = this.sentToD2KeysBySource.get(sourceId)! - const filtered = filterDuplicateInserts(changes, sentKeys) + const sentRows = this.sentToD2RowsBySource.get(sourceId)! + const reconciled = reconcileChangesForD2(changes, sentRows) - return sendChangesToInput(input, filtered) + return sendChangesToInput(input, reconciled) } /** @@ -820,7 +826,8 @@ class EffectPipelineRunner { this.isGraphRunning = true try { - while (this.graph.pendingWork()) { + // Ordered refill can also dispose the runner between graph steps. + while (!this.isDisposed() && this.graph.pendingWork()) { this.graph.run() // A handler (via onBatchProcessed) or source error callback may have // called dispose() during graph.run(). Stop early to avoid operating @@ -837,13 +844,6 @@ class EffectPipelineRunner { this.flushPendingChanges() } finally { this.isGraphRunning = false - // If dispose() was called during this graph run, it deferred the heavy - // cleanup (clearing graph/inputs/pipeline) to avoid nulling references - // mid-loop. Complete that cleanup now. - if (this.deferredCleanup) { - this.deferredCleanup = false - this.finalCleanup() - } } } @@ -895,6 +895,10 @@ class EffectPipelineRunner { orderBy?: any limit?: number } { + if (this.query.limit === 0) { + return { includeInitialState: false, whereExpression } + } + // Ordered aliases explicitly disable initial state — data is loaded // via requestLimitedSnapshot/requestSnapshot after subscription setup. if (orderByInfo) { @@ -915,35 +919,6 @@ class EffectPipelineRunner { } } - /** - * Request the initial ordered snapshot for an alias. - * Uses requestLimitedSnapshot (index-based cursor) or requestSnapshot - * (full load with limit) depending on whether an index is available. - */ - private requestInitialOrderedSnapshot( - alias: string, - orderByInfo: OrderByOptimizationInfo, - subscription: CollectionSubscription, - ): void { - const { orderBy, offset, limit, index } = orderByInfo - const normalizedOrderBy = normalizeOrderByPaths(orderBy, alias) - - if (index) { - subscription.setOrderByIndex(index) - subscription.requestLimitedSnapshot({ - limit: offset + limit, - orderBy: normalizedOrderBy, - trackLoadSubsetPromise: false, - }) - } else { - subscription.requestSnapshot({ - orderBy: normalizedOrderBy, - limit: offset + limit, - trackLoadSubsetPromise: false, - }) - } - } - /** Get orderBy optimization info for one lexical source. */ private getOrderByInfoForSource( sourceId: string, @@ -960,148 +935,57 @@ class EffectPipelineRunner { * needs more data. If so, load more rows via requestLimitedSnapshot. */ private loadMoreIfNeeded(): void { - for (const [, orderByInfo] of Object.entries( - this.optimizableOrderByCollections, - )) { - if (!orderByInfo.dataNeeded || !orderByInfo.index) continue - - if (this.pendingOrderedLoadPromise) { - // Wait for in-flight loads to complete before requesting more - continue - } - - const n = orderByInfo.dataNeeded() - if (n > 0) { - this.loadNextItems(orderByInfo, n) - } - } - } - - /** - * Load n more items from the source collection, starting from the cursor - * position (the biggest value sent so far). - */ - private loadNextItems(orderByInfo: OrderByOptimizationInfo, n: number): void { - const { alias, sourceId } = orderByInfo - const source = this.collectionSources.find( - (candidate) => candidate.sourceId === sourceId, - ) - if (!source) return - const subscription = this.subscriptions[sourceId] - if (!subscription) return - - const cursor = computeOrderedLoadCursor( - orderByInfo, - this.biggestSentValue.get(sourceId), - this.lastLoadRequestKey.get(sourceId), - alias, - n, - ) - if (!cursor) return // Duplicate request — skip - - this.lastLoadRequestKey.set(sourceId, cursor.loadRequestKey) - - try { - subscription.requestLimitedSnapshot({ - orderBy: cursor.normalizedOrderBy, - limit: n, - minValues: cursor.minValues, - trackLoadSubsetPromise: false, - onLoadSubsetResult: (loadResult: Promise | true) => { - // Track in-flight load to prevent redundant concurrent requests - if (loadResult instanceof Promise) { - this.pendingOrderedLoadPromise = loadResult - const finish = () => { - if (this.pendingOrderedLoadPromise === loadResult) { - this.pendingOrderedLoadPromise = undefined - } - } - void loadResult.then(finish, finish) - } - }, - }) - } catch (error) { - if (subscription.lastError !== error) throw error - // subscribeChanges already routed the error through onSourceError. Do - // not let an automatic refill fail the source transaction that exposed - // the missing row. - if (this.lastLoadRequestKey.get(sourceId) === cursor.loadRequestKey) { - this.lastLoadRequestKey.delete(sourceId) + for (const loader of this.orderedLoaders.values()) { + try { + loader.loadMore() + } catch (error) { + if ( + !this.disposed && + !Object.values(this.subscriptions).some((subscription) => + Object.is(subscription.lastError, error), + ) + ) + throw error } } } - /** - * Track the biggest value sent for a given ordered alias. - * Used for cursor-based pagination in loadNextItems. - */ - private trackSentValues( - sourceId: string, - changes: Array>, - comparator: (a: any, b: any) => number, - ): void { - const sentKeys = this.sentToD2KeysBySource.get(sourceId) ?? new Set() - const result = trackBiggestSentValue( - changes, - this.biggestSentValue.get(sourceId), - sentKeys, - comparator, - ) - this.biggestSentValue.set(sourceId, result.biggest) - if (result.shouldResetLoadKey) { - this.lastLoadRequestKey.delete(sourceId) - } - } - /** Tear down subscriptions and clear state */ dispose(): void { if (this.disposed) return this.disposed = true this.subscribedToAllCollections = false - // Immediately unsubscribe from every source, even if one release fails. - let firstCleanupError: unknown - for (const unsubscribe of this.unsubscribeCallbacks) { - try { - unsubscribe() - } catch (error) { - firstCleanupError ??= error - } + // Release every source in one attempt; the first failure wins after the + // peers finish. A reentrant dispose returns at the guard above, so this + // call still owns each release exactly once. + try { + runAllCallbacks(this.unsubscribeCallbacks) + } finally { + this.unsubscribeCallbacks.clear() + this.clearPipelineState() } - this.unsubscribeCallbacks.clear() - this.sentToD2KeysBySource.clear() + } + + private clearPipelineState(): void { + this.sentToD2RowsBySource.clear() this.pendingChanges.clear() this.lazySources.clear() this.demand.clear() this.builderDependencies.clear() - this.biggestSentValue.clear() - this.lastLoadRequestKey.clear() - this.pendingOrderedLoadPromise = undefined + for (const loader of this.orderedLoaders.values()) loader.dispose() + this.orderedLoaders.clear() // Clear mutable objects for (const key of Object.keys(this.lazySourcesCallbacks)) { delete this.lazySourcesCallbacks[key] } - for (const key of Object.keys(this.sourceDependencies)) { - delete this.sourceDependencies[key] - } for (const key of Object.keys(this.optimizableOrderByCollections)) { delete this.optimizableOrderByCollections[key] } - // If the graph is currently running, defer clearing graph/inputs/pipeline - // until runGraph() completes — otherwise we'd null references mid-loop. - if (this.isGraphRunning) { - this.deferredCleanup = true - } else { - this.finalCleanup() - } - - if (firstCleanupError !== undefined) throw firstCleanupError - } - - /** Clear graph references — called after graph run completes or immediately from dispose */ - private finalCleanup(): void { + // graph.run() keeps its own stack reference. The disposed guard prevents + // another step or new input; clearing our references does not destroy it. this.graph = undefined this.inputs = undefined this.pipeline = undefined @@ -1210,7 +1094,7 @@ function reportError( event: DeltaEvent, onError?: (error: Error, event: DeltaEvent) => void, ): void { - const normalised = normaliseError(error) + const normalised = normalizeError(error) if (onError) { try { onError(normalised, event) @@ -1223,7 +1107,3 @@ function reportError( console.error(`[Effect] Unhandled error in handler:`, normalised) } } - -function normaliseError(error: unknown): Error { - return error instanceof Error ? error : new Error(String(error)) -} diff --git a/packages/db/src/query/equality-value-identity.ts b/packages/db/src/query/equality-value-identity.ts new file mode 100644 index 0000000000..5e397f9e0f --- /dev/null +++ b/packages/db/src/query/equality-value-identity.ts @@ -0,0 +1,104 @@ +import { serializeValue } from '@tanstack/db-ivm' +import { normalizeValue } from '../utils/comparison.js' +import { + createRuntimeReferenceIdentityFactory, + getRuntimeReferenceIdentity, +} from './runtime-reference-identity.js' + +const PARENT_CONTEXT = Symbol(`tanstack_db_parent_context`) + +type ParentContext = { + [PARENT_CONTEXT]: true + value: Record + identity: unknown +} + +type ReferenceIdentity = typeof getRuntimeReferenceIdentity + +export type ValueIdentity = { + equality: (value: unknown) => unknown + exact: (value: unknown) => unknown + serializeEquality: (value: unknown) => string +} + +function equalityIdentity( + value: unknown, + referenceIdentity: ReferenceIdentity, +): unknown { + const normalized = normalizeValue(value) + if ( + (typeof normalized === `object` && normalized !== null) || + typeof normalized === `function` || + typeof normalized === `symbol` + ) { + return referenceIdentity(normalized as object | symbol) + } + return normalized +} + +function exactIdentity( + value: unknown, + referenceIdentity: ReferenceIdentity, +): unknown { + if ( + (typeof value === `object` && value !== null) || + typeof value === `function` || + typeof value === `symbol` + ) { + return referenceIdentity(value) + } + if (typeof value === `number`) { + if (Object.is(value, -0)) return [`number`, `-0`] + if (Number.isNaN(value)) return [`number`, `NaN`] + } + return value +} + +export function createValueIdentity(): ValueIdentity { + const referenceIdentity = createRuntimeReferenceIdentityFactory() + const equality = (value: unknown) => + equalityIdentity(value, referenceIdentity) + return { + equality, + exact: (value) => exactIdentity(value, referenceIdentity), + serializeEquality: (value) => serializeValue(equality(value)), + } +} + +/** Preserve the value relation used by equality predicates in keyed state. */ +export function getEqualityValueIdentity(value: unknown): unknown { + return equalityIdentity(value, getRuntimeReferenceIdentity) +} + +/** Keep compiler identity outside the namespace that holds user aliases. */ +export function createParentContext( + value: Record, + identity: unknown, +): ParentContext { + return { [PARENT_CONTEXT]: true, value, identity } +} + +function isParentContext(context: unknown): context is ParentContext { + return ( + typeof context === `object` && context !== null && PARENT_CONTEXT in context + ) +} + +export function getParentContextValue( + context: unknown, +): Record | undefined { + if (isParentContext(context)) return context.value + if (typeof context === `object` && context !== null) { + return context as Record + } + return undefined +} + +/** + * The envelope is structural D2 state, but its value keeps the user's alias + * namespace separate from compiler identity. A later insert or retract can + * therefore rebuild the same route without reserving a user-visible key. + */ +export function getParentContextIdentity(context: unknown): unknown { + return isParentContext(context) ? context.identity : context +} diff --git a/packages/db/src/query/index.ts b/packages/db/src/query/index.ts index 75c020758e..3c104b813c 100644 --- a/packages/db/src/query/index.ts +++ b/packages/db/src/query/index.ts @@ -107,16 +107,4 @@ export { type QueryIdentity, } from './ir-stable-identity.js' -// Predicate utilities for predicate push-down -export { - isWhereSubset, - unionWherePredicates, - minusWherePredicates, - isOrderBySubset, - isLimitSubset, - isOffsetLimitSubset, - isPredicateSubset, - isLoadSubsetRequestSubsumedBy, -} from './predicate-utils.js' - export { DeduplicatedLoadSubset } from './subset-dedupe.js' diff --git a/packages/db/src/query/ir-stable-identity.ts b/packages/db/src/query/ir-stable-identity.ts index 1c07d0b748..7139942ce2 100644 --- a/packages/db/src/query/ir-stable-identity.ts +++ b/packages/db/src/query/ir-stable-identity.ts @@ -1,6 +1,7 @@ +import { isPlainObject } from '../utils/type-guards.js' import { normalizeValue } from '../utils/comparison.js' import { isRefProxy, toExpression } from './builder/ref-proxy.js' -import { getQueryIR } from './builder/index.js' +import { getQueryIR } from './builder/query-ir.js' import { getRuntimeReferenceIdentity } from './runtime-reference-identity.js' import type { Aggregate, @@ -86,18 +87,6 @@ export function getQueryIdentity(query: QueryIR): QueryIdentity { return JSON.stringify(canonicalizeQueryIR(query)) as QueryIdentity } -/** Returns the semantic identity of one structured expression. */ -export function getStableExpressionHash(expression: BasicExpression): string { - return JSON.stringify( - canonicalizeExpression( - expression, - `expression`, - new WeakSet(), - `exact-output`, - ), - ) -} - /** * Returns the exact semantic identity of a loadSubset request. * @@ -999,7 +988,11 @@ function canonicalizeExactOutputRuntimeValue( path: string, seen: WeakSet, ): StableIdentityValue { - if (typeof value === `object` && value !== null) { + if ( + (typeof value === `object` && value !== null) || + typeof value === `function` || + typeof value === `symbol` + ) { return getRuntimeReferenceIdentity(value) } @@ -1040,7 +1033,11 @@ function canonicalizeEqualityRuntimeValue( return canonicalizeRuntimeValue(normalized, path, seen) } - if (typeof value === `object` && value !== null) { + if ( + (typeof value === `object` && value !== null) || + typeof value === `function` || + typeof value === `symbol` + ) { return getRuntimeReferenceIdentity(value) } @@ -1149,10 +1146,3 @@ function isExpression( expressionType === `includesSubquery` ) } - -function isPlainObject(value: unknown): value is Record { - if (value === null || typeof value !== `object`) return false - - const prototype = Object.getPrototypeOf(value) - return prototype === Object.prototype || prototype === null -} diff --git a/packages/db/src/query/ir.ts b/packages/db/src/query/ir.ts index d551831afd..a04370a90a 100644 --- a/packages/db/src/query/ir.ts +++ b/packages/db/src/query/ir.ts @@ -356,18 +356,21 @@ export function createResidualWhere( return { expression, residual: true } } +/** Sources declared by a FROM clause. UnionAll branches own their sources. */ +export function getFromSources(from: From): Array { + if (from.type === `unionFrom`) return from.sources + if (from.type === `unionAll`) return [] + return [from] +} + function getRefFromAlias( query: QueryIR, alias: string, ): CollectionRef | QueryRef | void { - if (query.from.type === `unionFrom`) { - for (const source of query.from.sources) { - if (source.alias === alias) { - return source - } + for (const source of getFromSources(query.from)) { + if (source.alias === alias) { + return source } - } else if (query.from.type !== `unionAll` && query.from.alias === alias) { - return query.from } for (const join of query.join || []) { diff --git a/packages/db/src/query/live/ARCHITECTURE.md b/packages/db/src/query/live/ARCHITECTURE.md index 794bfcc4a5..d09d590ccc 100644 --- a/packages/db/src/query/live/ARCHITECTURE.md +++ b/packages/db/src/query/live/ARCHITECTURE.md @@ -10,9 +10,11 @@ The central rule is simple: > one D2 graph. Use custom state only at asynchronous source and public > Collection boundaries. -The correlated-materialization oracle suites listed below are green behavioral -contracts for this design. Suites for adjacent planner and query-db ownership -boundaries may also contain exact classifiers for defects outside this graph. +The correlated-materialization oracle suites listed below are behavioral +contracts for this design. Functional projections accept inline include values, +not compiled Collection-valued inputs. Suites for +adjacent planner and query-db ownership boundaries may also contain exact +classifiers for defects outside this graph. ## Scope @@ -84,19 +86,43 @@ model. They are not a second set of runtime objects, nor does every name need a matching TypeScript type. The implementation maps this model onto existing D2 operators and a few boundary adapters: -| Architectural role | Concrete implementation | -| ------------------------------------- | ------------------------------------------------------------------------------------------------------ | -| Compile relation IDs and demand plans | `packages/db/src/query/compiler/index.ts`, `packages/db/src/query/compiler/joins.ts` | -| Reduce public keys and build routes | `packages/db/src/query/live/materialized-pipeline.ts` | -| Run the graph and publish root rows | `packages/db/src/query/live/collection-config-builder.ts` | -| Publish Collection-valued buckets | `packages/db/src/query/live/bucket-facade-adapter.ts` | -| Start and release asynchronous demand | `packages/db/src/query/live/subset-demand-controller.ts`, `packages/db/src/collection/subscription.ts` | +| Architectural role | Concrete implementation | +| ----------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| Compile relation IDs and demand plans | `packages/db/src/query/compiler/index.ts`, `packages/db/src/query/compiler/joins.ts` | +| Reduce public keys and build routes | `packages/db/src/query/live/materialized-pipeline.ts` | +| Run the graph and publish root rows | `packages/db/src/query/live/collection-config-builder.ts` | +| Publish Collection-valued buckets | `packages/db/src/query/live/bucket-facade-adapter.ts` | +| Start and release asynchronous demand | `packages/db/src/query/live/subset-demand-controller.ts`, `packages/db/src/collection/subscription.ts` | +| Ordered provider loading and continuation | `packages/db/src/query/live/ordered-source-loader.ts` | Queries without includes keep the original compiled pipeline and do not pay for facade state. The one exception is a joined query with a custom public-key function: its possible duplicate contributors still pass through the keyed reduction that enforces public-key congruence and multiplicity. +### Loading handoffs + +These owners cooperate; they are not phases of one exclusive state machine. +The detailed loading and publication laws below still apply. + +| Owner | Accepts / retires | Does not establish | +| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | +| Subscription acquisition | Retires the old physical lease before replay acquisition; installs tentative ownership before adapter callbacks; each lease gets one cleanup attempt | Replay completion or permission to publish | +| OrderedSourceLoader | Tracks request settlement, safe continuation and repair debt; reset discards the cursor, disposal ignores late settlement | Provider exhaustion or acceptance of an imperative window | +| Subscription replay | Counts setup and logical acquisition participants; checks completion after reentrant release callbacks; success releases the source replacement hold | Success of a previously failed window operation | +| Query builder | Tracks ordered publication participants in one sync session and accepts a window only for its operation generation | Physical adapter ownership or cancellation | +| D2 and public Collection boundary | D2 accumulates private result changes; the builder flushes root and child changes when the existing gates allow it | Source completeness merely because graph work drained | + +Session and participant checks precede changes to the builder's ordered failure +state, not just scheduling. An obsolete rejection cannot close a replacement +session's publication gate. Loader-local stale-result guards are separate. + +`hasPendingTruncateReplacement` means publication is still withheld, including +after replay failure. `pendingTruncateReplacement` exposes only an unsettled +completion promise. Neither is a general readiness flag. A direct subscriber +buffers and diffs its own replacement rows; a query subscription delegates +publication to the builder while the graph keeps its private contributions. + ## Identity Aliases are lexical query-language names rather than source runtime identities. @@ -134,7 +160,44 @@ type BucketKey = readonly [ Correlation equality must use the same value semantics as query predicates. Implementations use canonical values, interned handles, or nested maps; they do not reconstruct array or object keys and expect JavaScript `Map` identity to -match. +match. Equality tokens collapse `-0` with `0`, compare Date, Temporal, and +binary values by the same normalized value as `eq`/`in`, and retain runtime +reference identity for other objects, functions, and symbols. These tokens are +valid only for equality-keyed routing, grouping, and demand. Output values and +arbitrary function arguments keep their exact runtime identity and value. +Tree indexes give symbols a stable runtime-local order because JavaScript +relational comparison throws for them; comparator equality still holds only +for the same symbol. That order is a physical index detail: symbol range +predicates fall back to the evaluator instead of treating it as query +semantics. Range predicates also fall back when the live indexed values do not +share the bound's relational domain. An index's advertised comparison options +also define its executable comparator; metadata cannot claim an order that the +index does not use. Explicit `undefined` range and cursor bounds denote the +indexed nullish comparator group, while an absent bound denotes the start or +end of the index. An ordered index groups exact value +buckets that compare at the same position and keeps a live representative for +each group, so range traversal and ordered limits cannot drop rows whose +distinct values are comparator-equal. +Compiler tokens belong to one compiled graph. This keeps every operator in the +graph on the same identity relation. Objects, functions, and local symbols are +weakly keyed where the runtime supports weak symbol keys. Older runtimes retain +local symbols strongly within the scope rather than collapse distinct symbols +and corrupt equality. Registered symbols use their registry key because the +runtime registry already retains them. A demand controller owns a separate +scope and discards it when the controller is cleared. Process-wide query +identity and opaque public group keys keep their own runtime scope because +equivalent query plans and retained public keys must survive graph replacement. +For grouping, the equality token is the D2 group key. The group retains a raw +value from a currently positive contributor only as the projected +representative. The representative is chosen by stable source-row identity, so +restoring the same source state restores the same value regardless of update +history. D2 sees only safe exact-value identity for that representative, not +the raw value itself. A separate public group key preserves primitive keys and +serializes opaque equality identity; graph-local identity tokens never cross +the Collection boundary. Compiler group fields use a query-local namespace +disjoint from every selected alias. Direct correlated joins canonicalize both +sides before the first D2 join; normalizing only the later group key is too +late. ### Route-context transport @@ -178,18 +241,96 @@ The executable oracle factors that product into valid compiler sub-grammars: - recursive source boundary by evaluation phase; and - join-key side by correlation attachment point; - union form and public-key identity; and -- derived-result boundary by selection mode and scalar nullability. - -Objects carry route metadata as hidden fields while the compiler moves them -through recursive sources. Scalars, including `null`, cannot carry fields, so -the compiler uses an internal envelope at those same edges. Namespacing and -join adapters unwrap the value, keep the route beside it, and never expose the -envelope in the public query result. - -Every valid plan is checked as a Collection, `toArray`, and `materialize` -include at initial load, after a parent-route update, and after a child update. -The grammar declarations generate the cases; individual reported defects do -not get one-off tests outside that product. +- derived-result boundary by selection mode and scalar nullability; and +- user namespace collision by parent alias and selected child field, crossed + with direct, `QueryRef`, join, and group boundaries; and +- public-surface shape across opaque atomic values, opaque wrappers, nested + reference identity, user symbol keys, adversarial property keys, functional + spreads, and implicit joins. + +Plain record results carry route metadata under a private symbol while the +compiler moves them through recursive sources. Primitives and opaque objects, +such as `Date`, use an internal envelope at those same edges. Namespacing and +join adapters unwrap the value and keep its route beside it. Every functional +callback whose source can carry route state receives a clean copy of only the +paths that contain private state; this includes recursive and union sources, +not only directly correlated child queries. The callback boundary removes all +compiler-owned fields before invoking user code. The publication boundary +applies the same copy-on-write walk while resolving facade references. Both +paths preserve property descriptors, clean nested references, cycles, +adversarial keys, and user-owned symbols. Discovery reads data descriptors +directly and never invokes an accessor merely to find private state. + +This walker also strips metadata from a correlated subquery's output before +its parent query consumes it. Clean object and array references at that internal +boundary are equality operands, not just render identities. Eagerly cloning +them can make a later `eq(projected.key, parent.key)` lose a matching row. +Relaxing cross-publication reference stability does not permit changing these +internal matches. The public-container copy matrix crosses reference-key type, +ordered and unordered subqueries, materialization form, and parent/child updates. + +D2 hashes enumerable symbol keys and uses exact local-symbol identity plus +registry keys for registered symbols. D2 rejects structural cycles with a clear +error, including cycles through arrays, Maps, Sets, and enumerable symbol keys. +Shared acyclic subtrees remain supported and are hashed once per traversal. Structural hashing +limits recursion depth and value visits; it rejects values that exceed these +limits instead of expanding a shared graph or overflowing the JavaScript stack. +This does not bound the cost of arbitrary user getters or key sorting. +A failed hash does not publish partial +structural cache entries, so retrying the same value cannot bypass a guard. +A graph-run failure marks the current live query as errored and preserves the +thrown error. It must not continue publishing from a partly advanced graph; +recovery requires a fresh query session. +Opaque reference-hashed leaves are resolved before structural recursion; their +own properties, including self-references, are not traversed. Hash inputs must +remain immutable once successfully cached, as with other retained D2 values. +Collections register as opaque handles at construction using the existing hash +cache. Their identity, not their mutable internal state, is visible to hashing +operators in a downstream query. This does not add child-row dependencies to a +functional projection that reads a Collection-valued field. +The descriptor-preserving boundary walkers may still encounter cycles, but that +does not make cyclic structural results valid input to a hashing operator. +Symbol-only changes cannot disappear before publication, and unsupported cycles +fail rather than silently merge. Neither +boundary mutates values retained by D2. Compiler-created +parent contexts use a separate internal +envelope that keeps projected user aliases apart from the equality identity +derived from their leaves. The whole parent-context envelope is structural D2 +state. This avoids reserving user aliases or selected field names while keeping +the context stable across D2 operators without collapsing two +reference-sensitive leaf values that happen to have the same object shape. + +A functional projection consumes fully materialized inline input before +downstream operators run. Compiled Collection-valued includes are not supported +as `fn.select()` inputs, including nested descendants. The compiler rejects +the plan before invoking the callback, even if the callback would ignore or +pass through the Collection. This keeps callbacks inside the ordinary D2 +pipeline without temporary Collection views or graph continuations. + +Use `toArray()` or `materialize()` in the upstream expression `select()` to +make child values available to a functional callback. Child changes then +update the inline value and rerun the projection. To keep live child +Collections, use expression projections, or do parent-only functional work +before adding the Collection-valued include. This restriction concerns compiled +include inputs; it does not inspect arbitrary source-row fields or captured +Collections. Reading an already published Collection from a callback does not +add a child-row dependency. + +Include paths describe a functional projection's input, not its arbitrary +output. A callback may drop or rename a field, or return a scalar. Its input +paths must not be attached to that output by a downstream QueryRef consumer. +The compiler consumes those descriptors through the existing D2 materializer; +downstream keys, distinct, ordering, and QueryRef consumers see the callback's +actual output. Queries without includes keep their original pipeline. + +The projection matrices keep Collection-input cases as rejection checks and +exercise supported inline forms across route changes, child updates, recursive +sources, unions, and chained callbacks. Expression controls retain ordinary +Collection reads, indexes, subscriptions, rollback, pending loads, and +cleanup/restart coverage. Work counters check repeated reads on public facades. +A manual forced-GC probe checks retained public handles and captured methods +after cleanup, with live facades as a positive retention control; it is not a +whole-application heap or throughput measurement. A materialization cell identifies one include field on one parent-row occurrence: @@ -401,8 +542,17 @@ compose( ): MaterializedRow ``` +When a parent result changes, unchanged inline include arrays may receive new +object identities. Cross-publication `===` equality for those arrays is not a +contract. Their values and prior snapshots must remain correct; a downstream +query whose selected result is unchanged must not emit a spurious update. This +does not guarantee that a UI component using shallow prop comparison skips a +render, nor does it relax the stable public Collection facade contract above. + ## Demand plane +### Demand grouping and ownership + Demand is derived from data, but it performs asynchronous side effects outside D2: @@ -415,7 +565,7 @@ ActiveBucket(bucket, demand parameters) -> source deltas return to D2 inputs ``` -The adapter treats demand as coverage, not as one request per bucket: +The adapter groups demand into shared source work, not one request per bucket: ```ts type DemandPlanId = Brand @@ -426,18 +576,121 @@ type DemandSet = readonly [ ] ``` -One request may cover many buckets, and the adapter may coalesce or reuse -requests according to the compiled demand plan. A coalesced request has one -shared abort lease. If one owner releases its lease, the source request remains -active while another owner still needs its coverage. The source signal aborts -only after every attached owner has released it. - -Its semantic contract is: - -> Every active, satisfiable bucket must be covered by a settled current demand +One request may serve many buckets according to the compiled demand plan. +This does not imply transport sharing between independent subscriptions. +The exact-request deduper reuses completed requests and shares in-flight work +only when callers supply no abort signal. Independently cancelable requests +use separate transports, trading duplicate concurrent fetches for simpler +ownership. An adapter may share its own resources, but releasing one owner +must not cancel work or remove rows still owned by another. + +Request data is immutable from submission onward, including the options, +expression trees, comparison options, and constant payloads such as Dates, +byte arrays, and membership arrays. Core and adapters retain that data without +cloning or freezing it. Changed demand needs new request data, not edits to an +old constant, even after its first load settles: deduplication and query state +may retain its identity. Request data uses stable data properties, not stateful +getters. The signal and subscription references do not change, but their +lifecycle remains live. Cancellation and release are not data mutations. +The immutable-demand boundary matrix checks direct and deferred sync startup, +adapter return and asynchronous settlement, cancellation, and release identity. + +A Collection subscription installs each logical subset owner before it calls +the source adapter. Reentrant release during `loadSubset` therefore retires the +logical owner at once, but physical release waits until the adapter returns and +proves that it established an acquisition. A synchronous `loadSubset` throw +rolls the tentative owner back without calling `unloadSubset`. Logical demand +retires even when `unloadSubset` fails. Each physical acquisition gets one release +attempt, marked before calling adapter or error-listener code. Reentrant and +repeated teardown cannot repeat it. Other acquisitions still receive cleanup, +and a cleanup failure cannot replace an earlier request failure. Core reports +the error but retains no retry debt: a broken adapter can leak external resources +if it throws before freeing them. Adapters must make their own cleanup reliable. +Replay replaces physical leases sequentially: detach and release the old lease, +then acquire a fresh one only if the logical demand and replay are still current. +A release failure fails that replay without starting a replacement. A load +throw leaves the logical demand detached; a later authoritative replay can +reacquire it. Neither path restores an already released lease. A sole adapter +resource may stop and restart in this gap; adapters must not tear down resources +held by another owner. The public replacement barrier remains closed throughout +the gap and through failed startup, so visible results do not flicker. Once a +new load returns successfully, its lease is active before status callbacks run. +Reentrant callbacks therefore see either detached demand, tentative startup, +or one active lease, not an old and new lease being transferred together. + +Request predicates describe acquisition, not row ownership. Releasing a demand +does not delete matching rows from either the public snapshot or an unfinished +replacement. The source controls retention through actual row writes; a +successful authoritative replacement reconciles the retained public snapshot. +This rule also applies when another demand overlaps the released predicate or +an independent source write happens to match it. Source deletions during replay +stay private until successful publication; failure preserves the last complete +snapshot. Query filters and routes, not request release, decide which retained +source rows belong in a query result. + +### Cleanup, restart, and detached waiters + +Restart is not allowed inside an active cleanup callback. `startSyncImmediate()` +throws `CollectionStateError` and `preload()` rejects with it before acquiring +new work. Nested cleanup does not open a new lifecycle turn. The Collection +holds this guard until sync, state, subscriptions, and indexes finish retiring; +it releases the guard even if teardown throws. Restart after cleanup completes, +including from its final `cleaned-up` status event, remains supported. This +avoids letting old teardown clear a replacement graph or its source ownership. + +Collection cleanup detaches surviving logical demand from the discarded sync +session. It aborts that session's physical work and rejects its replay barrier, +and rejects an unfinished initial preload with `AbortError`. Cleanup never +invokes first-ready callbacks; those callbacks belong to the discarded run. +Physical acquisitions belong to the sync session that created them; cleanup +retires them instead of sending an old release to a replacement adapter. +Unlike individual subset releases, a failed sync adapter cleanup callback +remains retryable only while that +retirement is current; it cannot replace a newer session's cleanup callback. +Demand requested while the Collection is cleaned up remains detached +rather than pretending that a physical acquisition succeeded. When the +Collection starts a new sync session, the subscription enters `loadingSubset` +before it queues reacquisition, then reacquires all detached demand through a +fresh private publication barrier. Settlements from the old session cannot +publish rows, report errors, or change readiness in the new session. + +This is the direct subscription's restart contract, not automatic recovery of +a dependent live query. Manually cleaning up a source puts its live queries in +a terminal error state. Restarting that source alone does not revive their +graphs or publish replacement results; callers must restart or recreate the +live query itself. This differs from a source truncate, which keeps the live +query active behind its replay publication barrier. + +An initial sync error also leaves newly requested demand detached, even when +the adapter has installed a loader. Same-session `markReady()` resumes that +demand; releasing it before recovery creates no physical acquisition or unload. +Queued reacquisition must not retry a failed attempt merely because both +loading and ready notifications scheduled it. + +Requests waiting for a loader report one pending promise synchronously through +`onLoadSubsetResult`, including requests made during initial error or after +cleanup. The callback is not delayed until acquisition: query callers capture +its result before the snapshot request returns. This promise waits for the +recovery's publication barrier, not just adapter return. Failure rejects it with +the replay error; release, external abort, unsubscribe, or another cleanup +rejects it with `AbortError`. Later transport settlement cannot change that +outcome. Cleanup may retain logical demand for the next session, but it does +not retain the old caller's unfinished wait. + +Eager collections have no subset reacquisition barrier. After cleanup, their +next public batch reconciles retained subscriber rows against the installed +state, including deletions for keys that do not return. An empty ready batch +also reconciles an empty replacement. On-demand sources cannot infer absence +from their partial installed state; their replay barrier owns replacement. + +### Source cancellation and applied settlement + +The initial-demand contract is: + +> Every active, satisfiable bucket must be served by a settled current demand > request before initial preload completes. -A request may remain in flight after some covered buckets become inactive. +A request may remain in flight after some served buckets become inactive. Those buckets no longer participate in readiness and cannot receive rows through routes that no longer exist. Sharing source work never merges the route rows themselves. @@ -445,12 +698,15 @@ rows themselves. The source contract stays abstract: a demand request eventually establishes one coherent baseline and identifies when that baseline is complete. Each request receives an `AbortSignal`. Cancellation is cooperative at this source -boundary. Core guarantees that an obsolete request cannot settle current -readiness; the source must honor the signal immediately before installing a -baseline or later request-scoped result. Core cannot prevent an arbitrary -adapter from writing after it ignores that signal. Buffering, snapshot tokens, -shape offsets, Collection transactions, and local indexes are source-specific -ways to satisfy that contract; they are not materializer state. +boundary. An obsolete request cannot satisfy current demand. Its settlement +may release a replay wait, but never substitutes for completion of the current +acquisition. A source that can cancel request-scoped work must honor the signal +before installing more rows. A source that cannot cancel an in-flight baseline +must settle that work; core keeps overlapping replay private until then. Core +cannot prevent an arbitrary adapter from writing after it ignores both parts +of that contract. Buffering, snapshot tokens, shape offsets, Collection +transactions, and local indexes are source-specific ways to satisfy it; they +are not materializer state. Every sync `commit()` returns an applied receipt: `true` when that transaction's writes and events are already visible, or a promise when the @@ -465,8 +721,198 @@ priority merely to make a subset load settle. Existing immediate bootstrap and persistence-hydration paths, plus truncate, retain their queue-bypass contract; if one applies a parked subset transaction as part of that prefix, the subset receipt settles only after the writes are -visible. Rejected, canceled, and obsolete acquisitions establish no coverage. -Sources must honor cancellation before publishing request-scoped rows. +visible. Rejected acquisitions establish no result. Canceled or obsolete +acquisitions either stop before publishing more request-scoped rows or settle +behind the active replay barrier. + +### Ordered requests, continuation, and recovery + +Core constructs cursors only for one order column. A direct +`requestLimitedSnapshot()` call with a nonempty `minValues` must supply one +value and one order term; composite or partial-composite inputs throw before +local delivery or source acquisition. Multi-column queries remain supported +through the ordered loader's prefix-and-tie fallback. Its first-column equality +request closes a tie group; it is not a composite continuation cursor. + +Successful settlement proves only that the exact request finished and that its +writes were applied. It does not prove source exhaustion or broader coverage. +Ordered loading reaches a fixed point from public rows and exact request +identity; it must not invent source extent from a requested limit. A local row +seen before the first ordered source request proves neither a continuation +boundary nor a remote offset. This matters when a zero-sized window admits live +source changes before it opens: the first nonzero window must still request its +prefix from the start. Starting that request proves nothing until it succeeds; +if it rejects, an explicit retry must also start at offset zero without a +cursor. The same holds after any later ordered request fails or is canceled: +the adapter may already have written only part of its response, so those rows +cannot establish a continuation boundary. The next explicit retry starts from +the source as one authoritative filtered full-source request. Core cannot know +which rows a failed request wrote, and a successful limited request proves +neither how many authoritative rows it applied nor source exhaustion. Recovery +therefore does not infer a safe finite prefix from local row count or boundary +values. This rare error path trades bandwidth for a small, sound rule and keeps +the last settled public snapshot visible until recovery succeeds. It also lets +multi-column windows revalidate after a non-boundary row leaves. If the provider +predicate cannot express the local order relation, such as locale string order, +ordinary refinement likewise loads the full source instead of treating boundary +equality as an ordered continuation. An asynchronous failure of that +full-source acquisition does not start duplicate recovery work. It keeps the +logical demand so a later truncate replay can retry one authoritative +replacement, and clears the loader's completion marker so an explicit retry +of the window can issue the request again. That explicit retry retires and +releases the earlier failed acquisition before installing its replacement, so +a later truncate replays one logical demand rather than both attempts. A +successful authoritative replay clears its source-recovery gate, but it does +not clear an unrelated failed window operation. A later explicit window move +revalidates that physical window before publishing it. + +A finite page or tie-boundary request can remain in flight when full-source +repair starts. Its later success still settles its publication participant, +but cannot clear a recorded failure, change the repair's state, or start more +finite work. The full-source request owns that repair outcome. Explicit retry +releases a failed full-source acquisition once before replacing it. + +An ordered request cannot start another ordered request through its own +synchronous writes. If the adapter then throws, graph callbacks scheduled by +those writes still belong to the failed window operation and cannot retry it. +A public `setWindow()` call made from inside that synchronous operation throws +`SetWindowReentrancyError`; it must not claim that a nested window settled after +the loader suppressed its work. +The guard reads the loader's existing synchronous request state for initial +and later refinement requests, including requests after an asynchronous page. +It also rejects window changes during graph publication, before mutating top-K. +A synchronous result callback is provisional until the whole snapshot request +returns: a later local read or publication throw fails and retires that +acquisition instead of letting its queued success erase the failure. +A later explicit window operation has a new generation and may retry from the +safe source boundary. + +The ordered loader retains one settled loading boundary, independently of +live rows sent to D2. It derives invalidation from the existing contribution +rows rather than tracking a second largest-row cursor. New keys may reopen +refinement, while duplicate delivery and order-equal updates do not. After a +successful finite acquisition, it reads at +most the requested limit within that request's filtered, ordered range. That +range's last available row can advance the boundary; an unrelated live outlier +cannot advance it merely by entering D2. This relies on the adapter fulfilling +the exact ordered request, not just resolving after an arbitrary partial write. +An empty range does not invent a boundary or prove source exhaustion. + +An explicit window move counts current rows at or before that boundary in the +requested prefix. It acquires only the missing portion, with both cursor and +offset derived from that confirmed range, not from all observed rows. These +reads reuse the Collection's indexed snapshot code; they retain no page list +or second row index. Transfer checks and local-read work are separate costs: +counting a long prefix can still revisit its rows. Boundary-read failures use +the same authoritative recovery path as failed acquisitions. Deletes and +source-order changes invalidate finite coverage as described below. Cleanup +and truncate discard the boundary; replay establishes an authoritative source +replacement instead of reviving a stale cursor. + +### Atomic window publication + +An initial ordered load or imperative window move includes every page, +tie-boundary request, and forward refill needed to reach its fixed point. Its +preload or window promise cannot settle before that chain, and a failure in any +required step belongs to the same operation. Rows may enter the private D2 +result while the chain runs, but the public Collection publishes the completed +window once. If refinement fails, the operation rejects and leaves the last +settled public snapshot visible. The private source and D2 state may already +have advanced, so core does not try to reconstruct the old window over that +new state. A later successful retry publishes the coherent replacement. A +superseding window also waits for older source work that still gates +publication; it does not report success until its own chosen window is visible. +Window controllers treat `getWindow()` as settled state, not the current lease +request. An overlapping preload joins its lease's pending window promise rather +than replacing it with the smaller committed page count. Lease release may also +settle asynchronously; completion, not the release call, establishes its window. +Partial window options inherit omitted fields from the active requested window, +or from the last settled window when no move is active. Collection cleanup +rejects a pending window operation with `AbortError`; it cannot report success +after discarding the graph and requested window. +That error belongs to the operation even if cleanup precedes registration of +its waiter. Cleanup does not retroactively cancel an already completed operation. +Window-operation generations stay monotonic across cleanup and restart, so a +late rejection from an abandoned session cannot reset the replacement +session's requested window. +A window move started during an active source replay waits for that replay and +applies only after its replacement is complete. A failed replay rejects the +move without advancing the reported window. Replay completion callbacks carry +their sync-session identity and become no-ops after cleanup or restart. +Cleanup rejects the replay barrier, and therefore every window move waiting on +it, with `AbortError`; no waiter may outlive the discarded subscription. +Subscription-owned Promise observers carry the Collection's load-session +generation. Cleanup invalidates that generation before adapter teardown, so an +obsolete replay cannot publish its private rows, report a late error, or emit a +late `ready` transition even when the transport ignores cancellation. +Ordinary source mutations stay synchronous except while an initial ordered +load, imperative window move, or asynchronous repair of invalid finite source +coverage owns this publication barrier. A visible delete or a change to a +visible row's source-order value can invalidate a provider prefix because a +hidden row may now belong in the window. That repair loads the authoritative +source and keeps the last complete public snapshot until it settles; an update +that compares equal under the source order does not broaden demand. Mutations +that arrive during a barrier join the private state and publish with the +completed replacement; a failed operation keeps them private until retry or +restart. Queued ordered-repair startup joins this barrier before invoking the +adapter, so a synchronous throw cannot publish a partial replacement merely +because it returned no acquisition promise. The queued task belongs to the +loader that scheduled it, not a replacement created after cleanup. +The loader tracks each sequential request as a bounded participant, +not every recursive suffix of a long refinement chain. + +### Replay participants and failure + +A truncate replay is one publication barrier. Every acquisition started while +that replay is active, including ordered full-source recovery, belongs to the +barrier. Success publishes only after all current acquisitions settle. A +released demand stops participating even if its canceled transport promise +never settles. A newer truncate aborts prior acquisitions, but publication +still waits for overlapping work that had already started because some sources +cannot cancel an in-flight snapshot. Such work must settle and must not install +rows after observing cancellation. Settled historical attempts are discarded. +Replacing an acquisition does not release its logical owner. A delayed +cancellation therefore remains pending; prompt cancellation settles that wait. +Releasing the owner removes both its current and older work from readiness. +An ordinary acquisition started before replay may still hold subscription +readiness after its replacement publishes. It is not a replay publication +participant: its canceled writes must stop at the source boundary. Work started +inside replay, including an older overlapping replay, does hold publication. +Core installs each tentative acquisition and binds it to the current replay +attempt before calling adapter code. A reentrant release or newer truncate can +therefore see and retire the exact work it supersedes; work returned after that +reentrancy cannot attach itself to a newer attempt. Once reentrancy supersedes +an attempt, core starts none of that attempt's remaining demands. A demand that +releases itself during adapter or status callbacks cannot join readiness or +poison the replay with a later synchronous failure. Successful replacement +publication happens before the subscription emits `ready`. Cleanup runs every +ownership step even when replacement publication throws. Subscriber errors +raised by an asynchronous replacement do not turn source success into replay +failure: core finishes its internal state and surfaces the exact callback error +in a host microtask. Status callbacks may synchronously change demand. Generic +and specific status delivery capture the transition revision and stop before a +later listener when reentry supersedes it, including an ABA transition back to +the same status label. Subscription teardown is a one-shot logical transition: +it stops the listener set already being walked, emits no later status, and +removes subscriber ownership once. A later `unsubscribe()` is a no-op, including +after a physical subset release failed. +Failure keeps the last complete result visible and partly replayed source state +private for both direct subscribers and query graphs. Ordinary source deltas or +snapshot requests do not reopen that gate because they cannot prove the source +complete; only a later successful truncate replay provides the authoritative +replacement. Replay failure is scoped to the logical demand that failed. If +that demand retires, its failure cannot poison a successful replacement for the +remaining demand. If the last logical demand retires, the now-unreachable source +replay rejects its completion with `AbortError` and stops gating the shared +graph; unrelated parent or sibling changes may then publish. If release +publication or adapter unload synchronously acquires new demand, core checks +completion after that callback: the new demand joins the private replacement, +while the retired transport can no longer gate it. A genuine replay failure is +normalized once by the subscription. +The `loadSubset:error` event, `lastSubsetError`, and any window move waiting on +that replay expose the same `Error` object. + +### Mutation boundaries and initial readiness A transaction `mutationFn` must not start or await collection or live-query preloads. User persistence owns the causal queue while that function runs, so a @@ -559,16 +1005,21 @@ create recursive Collection machinery. equal current materialization-cell values. 5. **Total materialization:** every active inline cell has exactly one value, including its mode's empty value when its bucket has no rows. -6. **Stale demand:** an obsolete graph or demand generation cannot settle - current readiness, and a conforming source cannot publish its request-scoped - rows after cancellation. +6. **Stale demand:** an obsolete graph cannot settle current readiness, and an + obsolete acquisition cannot satisfy current demand. A conforming source + cannot publish its request-scoped rows after cancellation. 7. **Applied settlement:** a successful subset load settles only after its establishing sync transactions are visible; a source must not add queue - priority merely to force the load to settle. + priority merely to force the load to settle. Settlement proves no broader + source extent than the exact request. 8. **Nested propagation:** every materialized relation consumes the fully materialized output relation of its children. 9. **Publication:** reads, events, and downstream queries observe the same - complete graph result. + complete graph result. A truncate replacement stays private until all work + started by its active replay demands settles; failure keeps the prior public + result and later partial source changes private until an authoritative replay + succeeds. A failed replay with no remaining logical demand cannot gate other + graph work. 10. **Initial demand:** preload completes when every initially reachable demand is covered; obsolete demand does not block it. 11. **Ownership:** a query-db row exists exactly while an explicit owner @@ -576,7 +1027,9 @@ create recursive Collection machinery. 12. **Work:** irrelevant rows do not cause unrelated scans or activate unrelated routes when an applicable index exists. 13. **Space:** state scales with retained D2 relation/index rows, active demands, - materialization cells, visible rows, and required Collection facades. + materialization cells, visible rows, the current private replay state, and + required Collection facades—not with settled historical replay attempts or + raw delta history. ## Glossary @@ -606,18 +1059,24 @@ create recursive Collection machinery. ## Executable contracts -| Contract | Test suite | -| --------------------------------------------------------------------------- | ------------------------------------------------------------------------- | -| State equivalence, route lifecycle, transition history, and batch partition | `packages/db/tests/query/includes-oracle.property.test.ts` | -| Joined multiplicity, alias identity, and null-key normalization | `packages/db/tests/query/includes-query-shape-oracle.test.ts` | -| Demand, cancellation, and progressive timing | `packages/db/tests/query/includes-temporal-oracle.test.ts` | -| Optimistic confirmation, rollback, and later reactivity | `packages/db/tests/query/includes-optimistic-oracle.property.test.ts` | -| Coherent layered publication | `packages/db/tests/query/includes-publication-oracle.test.ts` | -| Collection facades, event coherence, and route activation | `packages/db/tests/query/includes-collection-oracle.property.test.ts` | -| Correlated physical work | `packages/db/tests/query/includes-work-counter-oracle.test.ts` | -| Route-context discovery and transport across recursive and join boundaries | `packages/db/tests/query/includes-context-transport-oracle.test.ts` | -| Query-db ownership | `packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts` | -| Reachable nested shape | `packages/query-db-collection/tests/includes-work-counter-oracle.test.ts` | +| Contract | Test suite | +| ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | +| State equivalence, route lifecycle, transition history, and batch partition | `packages/db/tests/query/includes-oracle.property.test.ts` | +| Joined multiplicity, alias identity, and null-key normalization | `packages/db/tests/query/includes-query-shape-oracle.test.ts` | +| Demand, cancellation, and progressive timing | `packages/db/tests/query/includes-temporal-oracle.test.ts` | +| Optimistic confirmation, rollback, and later reactivity | `packages/db/tests/query/includes-optimistic-oracle.property.test.ts` | +| Coherent layered publication | `packages/db/tests/query/includes-publication-oracle.test.ts` | +| Collection facades, event coherence, and route activation | `packages/db/tests/query/includes-collection-oracle.property.test.ts` | +| Correlated physical work | `packages/db/tests/query/includes-work-counter-oracle.test.ts` | +| Route-context discovery and transport across recursive and join boundaries | `packages/db/tests/query/includes-context-transport-oracle.test.ts` | +| Functional projection input boundaries, timing, and output preservation | `packages/db/tests/query/includes-functional-projection-oracle.test.ts` | +| Functional input rejection and inline alternatives | `packages/db/tests/query/includes-functional-input-boundary.test.ts` | +| Public-container descriptors and reference-key matches across internal query stages | `packages/db/tests/query/public-container-copy.test.ts` | +| Cross-formulation equivalence and reference-sensitive route identity | `packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts` | +| Query-db ownership | `packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts` | +| Failed replay retention, peer isolation, and explicit consumer-only recovery | `packages/db/tests/query/replay-failure-boundary.test.ts` | +| Replay lease balance, reference-counted peers, and failed-start recovery | `packages/db/tests/replay-adapter-ownership.test.ts` | +| Reachable nested shape | `packages/query-db-collection/tests/includes-work-counter-oracle.test.ts` | Each oracle identifies the first divergent checkpoint and compares either the whole result or one exact structural difference. Correlated-materialization diff --git a/packages/db/src/query/live/bucket-facade-adapter.ts b/packages/db/src/query/live/bucket-facade-adapter.ts index f350e88850..2648416809 100644 --- a/packages/db/src/query/live/bucket-facade-adapter.ts +++ b/packages/db/src/query/live/bucket-facade-adapter.ts @@ -1,6 +1,11 @@ import { output, serializeValue } from '@tanstack/db-ivm' +import { isPlainObject } from '../../utils/type-guards.js' +import { getOrCreate } from '../../utils/get-or-create.js' import { createCollection } from '../../collection/index.js' -import { FN_SELECT_STATE, INCLUDES_ROUTING } from '../compiler/index.js' +import { + INCLUDES_ROUTING, + transformPublicContainers, +} from '../compiler/route-metadata.js' import { BUCKET_FACADE_REF } from './materialized-pipeline.js' import type { Collection } from '../../collection/index.js' import type { SyncConfig } from '../../types.js' @@ -11,6 +16,8 @@ import type { BucketRow, } from './materialized-pipeline.js' +const PRIVATE_RESULT_KEYS = new Set([INCLUDES_ROUTING]) + type FacadeSync = Parameters[`sync`]>[0] type PendingRow = { @@ -41,6 +48,7 @@ type FacadeSnapshot = { } export type FacadePublication = { + prepare: () => void publish: () => void rollback: () => void } @@ -101,6 +109,7 @@ export class BucketFacadeAdapter { deferredEntries.add(entry) publications.push(entry.collection._deferPublication()) } + const newBaselines: Array = [] // Compilations are child-first, so nested facade references resolve before // their containing rows are written to the next facade. @@ -108,7 +117,6 @@ export class BucketFacadeAdapter { for (const compilation of this.compilations) { const activity = this.pendingActivity.get(compilation.edgeId) const active = this.getActiveBuckets(compilation.edgeId) - const newBaselines: Array = [] for (const [bucketKey, multiplicity] of activity ?? []) { if (multiplicity > 0 && !active.has(bucketKey)) { active.add(bucketKey) @@ -134,8 +142,6 @@ export class BucketFacadeAdapter { } sync.commit() } - for (const entry of newBaselines) entry.sync?.markReady() - for (const [bucketKey, multiplicity] of activity ?? []) { if (multiplicity >= 0) continue active.delete(bucketKey) @@ -152,9 +158,17 @@ export class BucketFacadeAdapter { this.pendingActivity.clear() let closed = false + let prepared = false + const prepare = () => { + if (closed || prepared) return + prepared = true + for (const entry of newBaselines) entry.sync?.markReady() + } return { + prepare, publish: () => { if (closed) return + prepare() closed = true for (const publication of publications) publication.publish() // Drop only the adapter's strong reference. External holders keep an @@ -194,16 +208,8 @@ export class BucketFacadeAdapter { row: BucketRow, multiplicity: number, ): void { - let buckets = this.pending.get(edgeId) - if (!buckets) { - buckets = new Map() - this.pending.set(edgeId, buckets) - } - let rows = buckets.get(bucketKey) - if (!rows) { - rows = new Map() - buckets.set(bucketKey, rows) - } + const buckets = getOrCreate(this.pending, edgeId, () => new Map()) + const rows = getOrCreate(buckets, bucketKey, () => new Map()) const key = serializeValue(row.publicKey) const change = rows.get(key) ?? { @@ -275,14 +281,21 @@ export class BucketFacadeAdapter { if (!previousEntries.has(entry)) continue const sync = entry.sync if (!sync) continue + const rows = snapshot.rows.get(entry) ?? [] + const restoredKeys = new Set(rows.map((row) => row.key)) sync.begin() - sync.truncate() + for (const key of entry.collection.keys()) { + if (!restoredKeys.has(key)) sync.write({ type: `delete`, key }) + } entry.currentOrder.clear() - for (const row of snapshot.rows.get(entry) ?? []) { + for (const row of rows) { entry.keys.set(row.value, row.key) if (row.order !== undefined) entry.order.set(row.value, row.order) entry.currentOrder.set(row.key, row.order) - sync.write({ type: `insert`, value: row.value }) + sync.write({ + type: entry.collection.has(row.key) ? `update` : `insert`, + value: row.value, + }) } sync.commit() } @@ -307,21 +320,12 @@ export class BucketFacadeAdapter { bucketKey: string, multiplicity: number, ): void { - let activity = this.pendingActivity.get(edgeId) - if (!activity) { - activity = new Map() - this.pendingActivity.set(edgeId, activity) - } + const activity = getOrCreate(this.pendingActivity, edgeId, () => new Map()) activity.set(bucketKey, (activity.get(bucketKey) ?? 0) + multiplicity) } private getActiveBuckets(edgeId: string): Set { - let active = this.activeBuckets.get(edgeId) - if (!active) { - active = new Set() - this.activeBuckets.set(edgeId, active) - } - return active + return getOrCreate(this.activeBuckets, edgeId, () => new Set()) } private retireEntry( @@ -343,20 +347,12 @@ export class BucketFacadeAdapter { } byBucket!.delete(bucketKey) if (byBucket!.size === 0) this.entries.delete(edgeId) - let retired = this.retiredEntries.get(edgeId) - if (!retired) { - retired = new Map() - this.retiredEntries.set(edgeId, retired) - } + const retired = getOrCreate(this.retiredEntries, edgeId, () => new Map()) retired.set(bucketKey, entry) } private getEntry(edgeId: string, bucketKey: string): FacadeEntry { - let byBucket = this.entries.get(edgeId) - if (!byBucket) { - byBucket = new Map() - this.entries.set(edgeId, byBucket) - } + const byBucket = getOrCreate(this.entries, edgeId, () => new Map()) const existing = byBucket.get(bucketKey) if (existing) return existing @@ -416,10 +412,7 @@ export class BucketFacadeAdapter { const nextOrder = change.value.order const orderChanged = sync.collection.has(key) && previousOrder !== nextOrder const resolvedRow = this.resolve(change.value.value) - const row = - orderChanged && sync.collection.get(key) === resolvedRow - ? { ...resolvedRow } - : resolvedRow + const row = orderChanged ? { ...resolvedRow } : resolvedRow entry.keys.set(row, key) if (nextOrder !== undefined) { entry.order.set(row, nextOrder) @@ -451,10 +444,9 @@ export class BucketFacadeAdapter { } private resolveValue(value: unknown): unknown { - if (value !== null && typeof value === `object`) { - const cached = this.resolvedValues.get(value) - if (cached !== undefined) return cached - } + if (value === null || typeof value !== `object`) return value + const cached = this.resolvedValues.get(value) + if (cached !== undefined) return cached if (isBucketFacadeRef(value)) { const { edgeId, bucketKey } = value[BUCKET_FACADE_REF] const facade = @@ -464,21 +456,16 @@ export class BucketFacadeAdapter { this.resolvedValues.set(value, facade) return facade } - if (Array.isArray(value)) { - const result: Array = [] + if (Array.isArray(value) || isPlainObject(value)) { + const result = transformPublicContainers( + value, + (leaf) => (isBucketFacadeRef(leaf) ? this.resolveValue(leaf) : leaf), + PRIVATE_RESULT_KEYS, + ) this.resolvedValues.set(value, result) - result.push(...value.map((item) => this.resolveValue(item))) return result } - if (!isPlainObject(value)) return value - - const result: Record = {} - this.resolvedValues.set(value, result) - for (const key of Reflect.ownKeys(value)) { - if (key === INCLUDES_ROUTING || key === FN_SELECT_STATE) continue - result[key] = this.resolveValue(value[key]) - } - return result + return value } private cleanupRetiredEntries(): void { @@ -496,9 +483,3 @@ function isBucketFacadeRef(value: unknown): value is BucketFacadeRef { value !== null && typeof value === `object` && BUCKET_FACADE_REF in value ) } - -function isPlainObject(value: unknown): value is Record { - if (value === null || typeof value !== `object`) return false - const prototype = Object.getPrototypeOf(value) - return prototype === Object.prototype || prototype === null -} diff --git a/packages/db/src/query/live/collection-config-builder.ts b/packages/db/src/query/live/collection-config-builder.ts index be53fed522..11966c14ba 100644 --- a/packages/db/src/query/live/collection-config-builder.ts +++ b/packages/db/src/query/live/collection-config-builder.ts @@ -2,14 +2,18 @@ import { D2, output } from '@tanstack/db-ivm' import { compileQuery } from '../compiler/index.js' import { MissingAliasInputsError, + SetWindowReentrancyError, SetWindowRequiresOrderByError, } from '../../errors.js' import { getActivePublicationContext, transactionScopedScheduler, + withPublicationContext, } from '../../scheduler.js' import { getActiveTransaction } from '../../transactions.js' import { deepEquals } from '../../utils.js' +import { runAllCallbacks } from '../../utils/callbacks.js' +import { normalizeError } from '../../utils/error.js' import { CollectionSubscriber } from './collection-subscriber.js' import { getCollectionBuilder } from './collection-registry.js' import { LIVE_QUERY_INTERNAL } from './internal.js' @@ -48,7 +52,6 @@ import type { import type { AllCollectionEvents } from '../../collection/events.js' export type LiveQueryCollectionUtils = UtilsRecord & { - getRunCount: () => number /** Most recent subset-load failure observed by this live query. */ readonly lastSubsetError: unknown | undefined /** @@ -68,7 +71,8 @@ export type LiveQueryCollectionUtils = UtilsRecord & { } type PendingGraphRun = { - loadCallbacks: Set<() => boolean> + syncSession: number + loadCallbacks: Set<() => void> } // Global counter for auto-generated collection IDs @@ -88,9 +92,6 @@ export class CollectionConfigBuilder< private readonly collectionSources: ReturnType< typeof extractCollectionSources > - private readonly collectionByAlias: Record> - // Populated during compilation with all aliases (including subquery inner aliases) - private compiledAliasToCollectionId: Record = {} // WeakMap to store the keys of the results // so that we can retrieve them in the getKey function @@ -103,7 +104,6 @@ export class CollectionConfigBuilder< private readonly compareOptions?: StringCollationConfig private isGraphRunning = false - private runCount = 0 // Current sync session state (set when sync starts, cleared when it stops) // Public for testing purposes (CollectionConfigBuilder is internal, not public API) @@ -124,17 +124,12 @@ export class CollectionConfigBuilder< private windowFn: ((options: WindowOptions) => void) | undefined private readonly initialWindow: WindowOptions | undefined private currentWindow: WindowOptions | undefined + private settledWindow: WindowOptions | undefined private activeWindowOperation: - | { failed: boolean; error?: unknown } + | { generation: number; failed: boolean; error?: unknown } | undefined private maybeRunGraphFn: (() => void) | undefined - - private readonly sourceDependencies: Record< - string, - Array> - > = {} - private readonly builderDependencies = new Set< CollectionConfigBuilder >() @@ -170,9 +165,18 @@ export class CollectionConfigBuilder< readonly lazySources = new Set() private readonly activeDemands = new Map< string, - { generation: number; settled: boolean } + { + generation: number + settled: boolean + } >() private readonly demandGenerations = new Map() + private readonly pendingOrderedLoads = new Set>() + private orderedLoadFailed = false + // Source replay cannot settle a failed imperative window operation. + private windowFailed = false + private syncSession = 0 + private windowOperationGeneration = 0 // Map of lexical source IDs to optimizable ORDER BY state optimizableOrderByCollections: Record = {} @@ -192,14 +196,9 @@ export class CollectionConfigBuilder< limit: this.query.limit ?? Infinity, } : undefined + this.settledWindow = this.initialWindow this.collections = extractCollectionsFromQuery(this.query) this.collectionSources = extractCollectionSources(this.query) - this.collectionByAlias = Object.fromEntries( - this.collectionSources.map(({ alias, collection }) => [ - alias, - collection, - ]), - ) // Create compare function for ordering if the query has orderBy if (this.query.orderBy && this.query.orderBy.length > 0) { @@ -268,7 +267,6 @@ export class CollectionConfigBuilder< startSync: this.config.startSync, singleResult: this.query.singleResult, utils: { - getRunCount: this.getRunCount.bind(this), get lastSubsetError() { return builder.lastSubsetError }, @@ -285,30 +283,66 @@ export class CollectionConfigBuilder< } setWindow(options: WindowOptions): true | Promise { - if (!this.windowFn) { + const windowFn = this.windowFn + if (!windowFn) { throw new SetWindowRequiresOrderByError() } + if ( + this.activeWindowOperation || + this.isGraphRunning || + Object.values(this.optimizableOrderByCollections).some((info) => + info.isRequesting?.(), + ) + ) { + throw new SetWindowReentrancyError() + } + // Keep caller-owned objects out of the long-lived query state. A caller may + // reuse and mutate its options object after this operation settles. + const baseWindow = + this.currentWindow ?? this.settledWindow ?? this.initialWindow + const requestedWindow: WindowOptions = { + offset: options.offset ?? baseWindow?.offset, + limit: options.limit ?? baseWindow?.limit, + } + const sourceRecovery = this.pendingSourceRecovery() + if (sourceRecovery) { + return sourceRecovery.then(async () => { + const settlement = this.setWindow(requestedWindow) + if (settlement !== true) await settlement + }) + } + if (this.hasFailedSourceRecovery()) { + return Promise.reject( + this.lastSubsetError ?? new Error(`Source recovery failed`), + ) + } + const windowOperationGeneration = ++this.windowOperationGeneration const loadOperation = this.liveQueryCollection?._sync.beginLoadSubsetOperation() - const previousWindow = this.currentWindow ?? this.initialWindow const previousOperation = this.activeWindowOperation - const operation: { failed: boolean; error?: unknown } = { failed: false } + const operation: { + generation: number + failed: boolean + error?: unknown + } = { generation: windowOperationGeneration, failed: false } this.activeWindowOperation = operation + this.windowFailed = false + if (this.pendingOrderedLoads.size === 0) this.orderedLoadFailed = false try { - this.windowFn(options) - this.maybeRunGraphFn?.() + // The window and all source work it causes form one synchronous + // publication. This makes operation tracking see requests scheduled by + // the graph rather than declaring the window settled too early. + this.currentWindow = requestedWindow + withPublicationContext(() => { + windowFn(requestedWindow) + this.maybeRunGraphFn?.() + }) if (operation.failed) throw operation.error - this.currentWindow = options } catch (error) { - if (previousWindow) { - try { - this.windowFn(previousWindow) - this.maybeRunGraphFn?.() - } catch { - // Recovery is best-effort; preserve the error from the requested - // window rather than replacing it with a rollback failure. - } + if (windowOperationGeneration === this.windowOperationGeneration) { + this.windowFailed = true + this.currentWindow = this.settledWindow } loadOperation?.cancel() throw error @@ -316,12 +350,30 @@ export class CollectionConfigBuilder< this.activeWindowOperation = previousOperation } - return loadOperation?.wait() ?? true + const settlement = loadOperation?.wait() ?? true + if (settlement === true) { + this.settledWindow = requestedWindow + return true + } + return settlement.then( + () => { + if (windowOperationGeneration === this.windowOperationGeneration) { + this.settledWindow = requestedWindow + } + }, + (error) => { + if (windowOperationGeneration === this.windowOperationGeneration) { + this.windowFailed = true + this.currentWindow = this.settledWindow + } + throw error + }, + ) } getWindow(): { offset: number; limit: number } | undefined { // Only return window if this is a windowed query (has orderBy and windowFn) - const window = this.currentWindow ?? this.initialWindow + const window = this.settledWindow ?? this.initialWindow if (!this.windowFn || !window) { return undefined } @@ -331,29 +383,6 @@ export class CollectionConfigBuilder< } } - /** - * Resolves a collection alias to its collection ID. - * - * Uses a two-tier lookup strategy: - * 1. First checks compiled aliases (includes subquery inner aliases) - * 2. Falls back to declared aliases from the query's from/join clauses - * - * @param alias - The alias to resolve (e.g., "employee", "manager") - * @returns The collection ID that the alias references - * @throws {Error} If the alias is not found in either lookup - */ - getCollectionIdForAlias(alias: string): string { - const compiled = this.compiledAliasToCollectionId[alias] - if (compiled) { - return compiled - } - const collection = this.collectionByAlias[alias] - if (collection) { - return collection.id - } - throw new Error(`Unknown source alias "${alias}"`) - } - isLazySource(sourceId: string): boolean { return this.lazySources.has(sourceId) } @@ -361,7 +390,10 @@ export class CollectionConfigBuilder< beginDemand(planId: string): number { const generation = (this.demandGenerations.get(planId) ?? 0) + 1 this.demandGenerations.set(planId, generation) - this.activeDemands.set(planId, { generation, settled: false }) + this.activeDemands.set(planId, { + generation, + settled: false, + }) return generation } @@ -375,42 +407,129 @@ export class CollectionConfigBuilder< failDemand(planId: string, generation: number, error: unknown): void { const demand = this.activeDemands.get(planId) if (!demand || demand.generation !== generation) return - this.recordSubsetError(error) - if (this.activeWindowOperation) { - this.activeWindowOperation.failed = true - this.activeWindowOperation.error = error - } - const message = error instanceof Error ? error.message : String(error) + const normalized = this.recordSubsetError(error) this.transitionToError( - `Subset demand '${planId}' failed: ${message}`, - error, + `Subset demand '${planId}' failed: ${normalized.message}`, + normalized, ) } - recordSubsetError(error: unknown, fatalBeforeReady = false): void { - this.lastSubsetError = error + recordSubsetError(error: unknown, fatalBeforeReady = false): Error { + const normalized = normalizeError(error) + this.lastSubsetError = normalized if (this.activeWindowOperation) { this.activeWindowOperation.failed = true - this.activeWindowOperation.error = error + this.activeWindowOperation.error = normalized + // A synchronous adapter failure can arrive before it returns a promise + // for the ordered-load tracker. Keep any private graph changes hidden. + this.orderedLoadFailed = true } if (fatalBeforeReady) { - const message = error instanceof Error ? error.message : String(error) - this.transitionToError(`Initial subset load failed: ${message}`, error) + this.transitionToError( + `Initial subset load failed: ${normalized.message}`, + normalized, + ) } + return normalized } - trackSubsetLoadPromise(promise: Promise): void { + trackSubsetLoadPromise(promise: Promise): void { this.liveQueryCollection!._sync.trackLoadPromise(promise) } - trackSubsetLoadOperationPromise(promise: Promise): void { + trackSubsetLoadOperationPromise(promise: Promise): void { this.liveQueryCollection!._sync.trackLoadSubsetOperationPromise(promise) } + hasActiveWindowOperation(): boolean { + return this.activeWindowOperation !== undefined + } + + getActiveWindowOperationGeneration(): number | undefined { + return this.activeWindowOperation?.generation + } + + scheduleGraphRunForSession(syncSession: number): void { + if ( + syncSession !== this.syncSession || + !this.currentSyncConfig || + !this.currentSyncState + ) { + return + } + this.scheduleGraphRun() + } + + trackOrderedLoadPromise( + promise: Promise, + holdPublication = false, + ): void { + // Hold the last complete public snapshot during an initial load or an + // imperative window move. Source changes that arrive during the move join + // its private graph state and publish with the completed replacement. + if ( + !holdPublication && + !this.activeWindowOperation && + this.liveQueryCollection?.status !== `loading` && + this.pendingOrderedLoads.size === 0 + ) { + return + } + const syncSession = this.syncSession + if (this.pendingOrderedLoads.size === 0) this.orderedLoadFailed = false + this.pendingOrderedLoads.add(promise) + const finish = (succeeded: boolean) => { + // Admission precedes mutation: cleanup retires this session's participants. + if ( + syncSession !== this.syncSession || + !this.pendingOrderedLoads.delete(promise) + ) { + return + } + if (!succeeded) this.orderedLoadFailed = true + if (!this.orderedLoadFailed && this.pendingOrderedLoads.size === 0) { + // The ordered chain already drove its source graph to quiescence. + // Flush the retained result without invoking the source loaders again. + this.scheduleGraphRun() + } + } + void promise.then( + () => finish(true), + () => finish(false), + ) + } + retireDemand(planId: string): void { this.activeDemands.delete(planId) } + hasPendingSourceRecovery(): boolean { + return Object.values(this.subscriptions).some( + (subscription) => subscription.hasPendingTruncateReplacement, + ) + } + + private pendingSourceRecovery(): Promise | undefined { + const pending = Object.values(this.subscriptions).flatMap((subscription) => + subscription.pendingTruncateReplacement + ? [subscription.pendingTruncateReplacement] + : [], + ) + return pending.length > 0 + ? Promise.all(pending).then(() => undefined) + : undefined + } + + private hasFailedSourceRecovery(): boolean { + return Object.values(this.subscriptions).some( + (subscription) => subscription.hasFailedTruncateReplacement, + ) + } + + getSyncSession(): number { + return this.syncSession + } + // The callback function is called after the graph has run. // This gives the callback a chance to load more data if needed, // that's used to optimize orderBy operators that set a limit, @@ -418,8 +537,8 @@ export class CollectionConfigBuilder< // That can happen because even though we load N rows, the pipeline might filter some of these rows out // causing the orderBy operator to receive less than N rows or even no rows at all. // So this callback would notice that it doesn't have enough rows and load some more. - // The callback returns a boolean, when it's true it's done loading data and we can mark the collection as ready. - maybeRunGraph(callback?: () => boolean) { + // Readiness follows source/demand state, not the callback's return value. + maybeRunGraph(callback?: () => void) { if (this.isGraphRunning) { // no nested runs of the graph // which is possible if the `callback` @@ -437,8 +556,14 @@ export class CollectionConfigBuilder< this.isGraphRunning = true try { - const { begin, commit } = this.currentSyncConfig + const syncSession = this.syncSession + const config = this.currentSyncConfig + const { begin, commit } = config const syncState = this.currentSyncState + const isCurrentSession = () => + syncSession === this.syncSession && + this.currentSyncConfig === config && + this.currentSyncState === syncState // Don't run if the live query is in an error state if (this.isInErrorState) { @@ -448,24 +573,47 @@ export class CollectionConfigBuilder< // Always run the graph if subscribed (eager execution) if (syncState.subscribedToAllCollections) { let callbackCalled = false - while (syncState.graph.pendingWork()) { - syncState.graph.run() - callback?.() - callbackCalled = true + const drainGraph = () => { + while (syncState.graph.pendingWork()) { + try { + syncState.graph.run() + } catch (error) { + if (isCurrentSession()) { + this.transitionToError(`Live query graph failed`, error) + } + throw error + } + if (!isCurrentSession()) return false + callback?.() + if (!isCurrentSession()) return false + callbackCalled = true + } + return true } - // Publish only after every operator has reached quiescence. A source - // change can reach sibling materializations in different graph steps; - // flushing between those steps would expose a mixed root snapshot. - syncState.flushPendingChanges?.() + if (!drainGraph()) return // Ensure the callback runs at least once even when the graph has no pending work. // This handles lazy loading scenarios where setWindow() increases the limit or // an async loadSubset completes and we need to re-check if more data is needed. + // drainGraph changes this flag inside its closure. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition if (!callbackCalled) { callback?.() + if (!isCurrentSession()) return } + // A synchronous loader can write while this graph run is active. Its + // nested schedule is intentionally coalesced, so drain that new input + // here before publishing the transaction. + if (!drainGraph()) return + + // Publish only after every operator has reached quiescence. A source + // change can reach sibling materializations in different graph steps; + // flushing between those steps would expose a mixed root snapshot. + syncState.flushPendingChanges?.() + if (!isCurrentSession()) return + // On the initial run, we may need to do an empty commit to ensure that // the collection is initialized if (syncState.messagesCount === 0) { @@ -478,7 +626,7 @@ export class CollectionConfigBuilder< // 1. All data has been processed through the graph // 2. All source collections have had a chance to send their initial data // This prevents marking ready before data is processed (fixes isReady=true with empty data) - this.updateLiveQueryStatus(this.currentSyncConfig) + this.updateLiveQueryStatus(config) } } finally { this.isGraphRunning = false @@ -496,19 +644,17 @@ export class CollectionConfigBuilder< * * Uses the current sync session's config and syncState from instance properties. * - * @param callback - Optional callback to load more data if needed (returns true when done) + * @param callback - Optional callback to load more data if needed * @param options - Optional scheduling configuration * @param options.contextId - Transaction ID to group work; defaults to active transaction * @param options.jobId - Unique identifier for this job; defaults to this builder instance - * @param options.sourceId - Source that triggered this schedule; adds its dependencies * @param options.dependencies - Explicit dependency list; overrides auto-discovered dependencies */ scheduleGraphRun( - callback?: () => boolean, + callback?: () => void, options?: { contextId?: SchedulerContextId jobId?: unknown - sourceId?: string dependencies?: Array> }, ) { @@ -519,25 +665,10 @@ export class CollectionConfigBuilder< // Use the builder instance as the job ID for deduplication. This is memory-safe // because the scheduler's context Map is deleted after flushing (no long-term retention). const jobId = options?.jobId ?? this - const dependentBuilders = (() => { - if (options?.dependencies) { - return options.dependencies - } - - const deps = new Set(this.builderDependencies) - if (options?.sourceId) { - const sourceDeps = this.sourceDependencies[options.sourceId] - if (sourceDeps) { - for (const dep of sourceDeps) { - deps.add(dep) - } - } - } - - deps.delete(this) - - return Array.from(deps) - })() + // Snapshot before scheduling parents, which can reenter source setup. + const dependentBuilders = options?.dependencies ?? [ + ...this.builderDependencies, + ] // Ensure dependent builders are actually scheduled in this context so that // dependency edges always point to a real job (or a deduped no-op if already scheduled). @@ -561,8 +692,9 @@ export class CollectionConfigBuilder< // Manage our own state - get or create pending callbacks for this context let pending = contextId ? this.pendingGraphRuns.get(contextId) : undefined - if (!pending) { + if (!pending || pending.syncSession !== this.syncSession) { pending = { + syncSession: this.syncSession, loadCallbacks: new Set(), } if (contextId) { @@ -630,31 +762,15 @@ export class CollectionConfigBuilder< } // If sync session has ended, don't execute (graph is finalized, subscriptions cleared) - if (!this.currentSyncConfig || !this.currentSyncState) { + if ( + pending.syncSession !== this.syncSession || + !this.currentSyncConfig || + !this.currentSyncState + ) { return } - this.incrementRunCount() - - const combinedLoader = () => { - let allDone = true - let firstError: unknown - pending.loadCallbacks.forEach((loader) => { - try { - allDone = loader() && allDone - } catch (error) { - allDone = false - firstError ??= error - } - }) - if (firstError) { - throw firstError - } - // Returning false signals that callers should schedule another pass. - return allDone - } - - this.maybeRunGraph(combinedLoader) + this.maybeRunGraph(() => runAllCallbacks(pending.loadCallbacks)) } private getSyncConfig(): SyncConfig { @@ -664,15 +780,8 @@ export class CollectionConfigBuilder< } } - incrementRunCount() { - this.runCount++ - } - - getRunCount() { - return this.runCount - } - private syncFn(config: SyncMethods) { + const syncSession = ++this.syncSession // Store reference to the live query collection for error state transitions this.liveQueryCollection = config.collection // Reset error state from any previous sync session so a restarted sync can become ready again. @@ -693,58 +802,17 @@ export class CollectionConfigBuilder< const teardown = () => { if (tornDown) return tornDown = true + if (this.syncSession === syncSession) this.syncSession++ - let firstCleanupError: unknown - for (const unsubscribe of syncState.unsubscribeCallbacks) { - try { - unsubscribe() - } catch (error) { - firstCleanupError ??= error - } + // Release every source in one attempt; the first failure wins after the + // peers finish. Each subscription release is itself one-shot, so the + // Collection's cleanup retry has nothing left to repeat here. + try { + runAllCallbacks(syncState.unsubscribeCallbacks) + } finally { + syncState.unsubscribeCallbacks.clear() + this.clearSyncSessionState() } - syncState.unsubscribeCallbacks.clear() - - // Clear current sync session state - this.currentSyncConfig = undefined - this.currentSyncState = undefined - this.maybeRunGraphFn = undefined - this.currentWindow = undefined - this.isInErrorState = false - this.fatalQueryError = false - this.erroredSourceIds.clear() - - // Clear all pending graph runs to prevent memory leaks from in-flight transactions - // that may flush after the sync session ends - this.pendingGraphRuns.clear() - - // Reset caches so a fresh graph/pipeline is compiled on next start - // This avoids reusing a finalized D2 graph across GC restarts - this.graphCache = undefined - this.inputsCache = undefined - this.pipelineCache = undefined - this.sourceWhereClausesCache = undefined - this.bucketFacadesCache = undefined - - // Reset lazy source alias state - this.lazySources.clear() - this.demandGenerations.clear() - this.activeDemands.clear() - this.optimizableOrderByCollections = {} - this.lazySourcesCallbacks = {} - - // Clear subscription references to prevent memory leaks - // Note: Individual subscriptions are already unsubscribed via unsubscribeCallbacks - Object.keys(this.subscriptions).forEach( - (key) => delete this.subscriptions[key], - ) - this.compiledAliasToCollectionId = {} - - // Unregister from scheduler's onClear listener to prevent memory leaks - // The scheduler's listener Set would otherwise keep a strong reference to this builder - this.unsubscribeFromSchedulerClears?.() - this.unsubscribeFromSchedulerClears = undefined - - if (firstCleanupError !== undefined) throw firstCleanupError } try { @@ -773,6 +841,7 @@ export class CollectionConfigBuilder< if (!event.isLoadingSubset) { // Subset loading finished, check if we can now mark ready this.updateLiveQueryStatus(config) + if (this.hasPendingSourceRecovery()) this.maybeRunGraphFn?.() } }, ) @@ -800,6 +869,53 @@ export class CollectionConfigBuilder< return teardown } + private clearSyncSessionState(): void { + // Late window settlement belongs to the discarded graph, not its restart. + this.windowOperationGeneration++ + // Clear current sync session state + this.currentSyncConfig = undefined + this.currentSyncState = undefined + this.maybeRunGraphFn = undefined + this.currentWindow = undefined + this.settledWindow = this.initialWindow + this.isInErrorState = false + this.fatalQueryError = false + this.erroredSourceIds.clear() + + // Clear all pending graph runs to prevent memory leaks from in-flight transactions + // that may flush after the sync session ends + this.pendingGraphRuns.clear() + + // Reset caches so a fresh graph/pipeline is compiled on next start + // This avoids reusing a finalized D2 graph across GC restarts + this.graphCache = undefined + this.inputsCache = undefined + this.pipelineCache = undefined + this.sourceWhereClausesCache = undefined + this.bucketFacadesCache = undefined + + // Reset lazy source alias state + this.lazySources.clear() + this.demandGenerations.clear() + this.activeDemands.clear() + this.pendingOrderedLoads.clear() + this.orderedLoadFailed = false + this.windowFailed = false + this.optimizableOrderByCollections = {} + this.lazySourcesCallbacks = {} + + // Clear subscription references to prevent memory leaks + // Note: Individual subscriptions are already unsubscribed via unsubscribeCallbacks + Object.keys(this.subscriptions).forEach( + (key) => delete this.subscriptions[key], + ) + + // Unregister from scheduler's onClear listener to prevent memory leaks + // The scheduler's listener Set would otherwise keep a strong reference to this builder + this.unsubscribeFromSchedulerClears?.() + this.unsubscribeFromSchedulerClears = undefined + } + /** * Compiles the query pipeline with all declared aliases. */ @@ -838,7 +954,6 @@ export class CollectionConfigBuilder< ) this.pipelineCache = materialized.pipeline this.sourceWhereClausesCache = compilation.sourceWhereClauses - this.compiledAliasToCollectionId = compilation.aliasToCollectionId this.bucketFacadesCache = materialized.facades const missingSources = this.collectionSources @@ -903,6 +1018,15 @@ export class CollectionConfigBuilder< return } + if ( + this.windowFailed || + this.orderedLoadFailed || + this.hasPendingSourceRecovery() || + this.pendingOrderedLoads.size > 0 + ) { + return + } + let facadePublication: | ReturnType | undefined @@ -928,7 +1052,10 @@ export class CollectionConfigBuilder< return [key, resolved] }), ) - + // New facades are not reachable until their root row is installed, so + // make them ready first. A facade failure then leaves the root intact, + // and the root commit is the final state change before publication. + facadePublication.prepare() if (hasParentChanges) { begin() changesToApply.forEach(this.applyChanges.bind(this, config)) @@ -938,7 +1065,6 @@ export class CollectionConfigBuilder< commit() } } catch (error) { - pendingChanges = new Map() rootPublication?.discard() facadePublication?.rollback() throw error @@ -959,7 +1085,6 @@ export class CollectionConfigBuilder< } if (publicationError !== undefined) throw publicationError } - graph.finalize() // Extend the sync state with the graph, inputs, and pipeline @@ -1147,10 +1272,7 @@ export class CollectionConfigBuilder< const dependencyBuilder = getCollectionBuilder(collection) if (dependencyBuilder && dependencyBuilder !== this) { - this.sourceDependencies[sourceId] = [dependencyBuilder] this.builderDependencies.add(dependencyBuilder) - } else { - this.sourceDependencies[sourceId] = [] } // CollectionSubscriber handles the actual subscription to the source collection @@ -1203,14 +1325,6 @@ export class CollectionConfigBuilder< return loadMore }) - // Combine all loaders into a single callback that initiates loading more data - // from any source that needs it. Returns true once all loaders have been called, - // but the actual async loading may still be in progress. - const loadSubsetDataCallbacks = () => { - loaders.map((loader) => loader()) - return true - } - // Mark as subscribed so the graph can start running // (graph only runs when all collections are subscribed) syncState.subscribedToAllCollections = true @@ -1220,7 +1334,7 @@ export class CollectionConfigBuilder< // The canonical place to mark ready is after the graph processes data // in maybeRunGraph(), which ensures data has been processed first. - return loadSubsetDataCallbacks + return () => runAllCallbacks(loaders) } } diff --git a/packages/db/src/query/live/collection-subscriber.ts b/packages/db/src/query/live/collection-subscriber.ts index 24e973662b..a4e35b8af6 100644 --- a/packages/db/src/query/live/collection-subscriber.ts +++ b/packages/db/src/query/live/collection-subscriber.ts @@ -1,19 +1,17 @@ +import { normalizeExpressionPaths } from '../compiler/expressions.js' +import { OrderedSourceLoader } from './ordered-source-loader.js' import { - normalizeExpressionPaths, - normalizeOrderByPaths, -} from '../compiler/expressions.js' -import { - computeOrderedLoadCursor, computeSubscriptionOrderByHints, - filterDuplicateInserts, + reconcileChangesForD2, sendChangesToInput, splitUpdates, - trackBiggestSentValue, } from './utils.js' import { SubsetDemandController } from './subset-demand-controller.js' import type { Collection } from '../../collection/index.js' import type { ChangeMessage, + LoadSubsetRequestResult, + SubscribeChangesOptions, SubscriptionLoadSubsetErrorEvent, SubscriptionStatusChangeEvent, } from '../../types.js' @@ -28,33 +26,26 @@ const loadMoreCallbackSymbol = Symbol.for( `@tanstack/db.collection-config-builder`, ) +type TruncateReplayPublicationControl = NonNullable< + SubscribeChangesOptions[`truncateReplayPublication`] +> + export class CollectionSubscriber< TContext extends Context, TResult extends object = GetResult, > { - // Keep track of the biggest value we've sent so far (needed for orderBy optimization) - private biggest: any = undefined - - // Track the most recent ordered load request key (cursor + window). - // This avoids infinite loops from cached data re-writes while still allowing - // window moves or new keys at the same cursor value to trigger new requests. - private lastLoadRequestKey: string | undefined - // Track deferred promises for subscription loading states private subscriptionLoadingPromises = new Map< CollectionSubscription, { resolve: () => void } >() - // Track keys that have been sent to the D2 pipeline to prevent duplicate inserts - // This is necessary because different code paths (initial load, change events) - // can potentially send the same item to D2 multiple times. - private sentToD2Keys = new Set() + // Exact row last contributed to D2 for each source key. + private sentToD2Rows = new Map>() // Direct load tracking callback for ordered path (set during subscribeToOrderedChanges, // used by loadNextItems for subsequent requestLimitedSnapshot calls) - private orderedLoadSubsetResult?: (result: Promise | true) => void - private pendingOrderedLoadPromise: Promise | undefined + private orderedLoader: OrderedSourceLoader | undefined private readonly demand = new SubsetDemandController() constructor( @@ -77,14 +68,11 @@ export class CollectionSubscriber< private subscribeToChanges(whereExpression?: BasicExpression) { const orderByInfo = this.getOrderByInfo() - let initialSubsetPending = !this.collectionConfigBuilder.isLazySource( - this.sourceId, - ) // Direct load promise tracking: pipes loadSubset results straight to the // live query collection, avoiding the multi-hop deferred promise chain that // can break under microtask timing (e.g., queueMicrotask in TanStack Query). - const trackLoadResult = (result: Promise | true) => { + const trackLoadResult = (result: LoadSubsetRequestResult) => { if (result instanceof Promise) { // Defer the tracked rejection by one microtask so the subscription's // error event can put an initial live query in error before loading @@ -94,16 +82,6 @@ export class CollectionSubscriber< throw error }) this.collectionConfigBuilder.trackSubsetLoadPromise(trackedResult) - if (initialSubsetPending) { - void result.then( - () => { - initialSubsetPending = false - }, - () => {}, - ) - } - } else { - initialSubsetPending = false } } @@ -128,7 +106,11 @@ export class CollectionSubscriber< const onLoadSubsetError = (event: SubscriptionLoadSubsetErrorEvent) => { this.collectionConfigBuilder.recordSubsetError( event.error, - initialSubsetPending, + // Lazy demand owns its fatal-error path. For eager sources, one + // successful page does not finish initial ordered refinement. + !this.collectionConfigBuilder.isLazySource(this.sourceId) && + this.collectionConfigBuilder.liveQueryCollection?.status === + `loading`, ) } @@ -144,9 +126,10 @@ export class CollectionSubscriber< ) } else { // Lazy sources load only the subsets demanded by the compiled graph. - const includeInitialState = !this.collectionConfigBuilder.isLazySource( - this.sourceId, - ) + const includeInitialState = + (this.collection.config.syncMode !== `on-demand` || + this.collectionConfigBuilder.query.limit !== 0) && + !this.collectionConfigBuilder.isLazySource(this.sourceId) subscription = this.subscribeToMatchingChanges( whereExpression, @@ -181,8 +164,11 @@ export class CollectionSubscriber< deferred.resolve() } - this.demand.clear() - subscription.unsubscribe() + try { + this.demand.clear() + } finally { + subscription.unsubscribe() + } } // currentSyncState is always defined when subscribe() is called // (called during sync session setup) @@ -204,7 +190,7 @@ export class CollectionSubscriber< // Convert that synchronous form to the same query-local fatal demand // state as a rejected load, without letting it escape the source commit. // Preserve unrelated graph/programming errors as throws. - if (subscription.lastError !== error) throw error + if (!Object.is(subscription.lastError, error)) throw error const isInitialSync = this.collectionConfigBuilder.liveQueryCollection?.status === `loading` const generation = this.collectionConfigBuilder.beginDemand(plan.id) @@ -234,19 +220,18 @@ export class CollectionSubscriber< private sendChangesToPipeline( changes: Iterable>, - callback?: () => boolean, + callback?: () => void, ) { const changesArray = Array.isArray(changes) ? changes : [...changes] - const filteredChanges = filterDuplicateInserts( + const reconciledChanges = reconcileChangesForD2( changesArray, - this.sentToD2Keys, + this.sentToD2Rows, ) - // currentSyncState and input are always defined when this method is called // (only called from active subscriptions during a sync session) const input = this.collectionConfigBuilder.currentSyncState!.inputs[this.sourceId]! - const sentChanges = sendChangesToInput(input, filteredChanges) + const sentChanges = sendChangesToInput(input, reconciledChanges) // Do not provide the callback that loads more data // if there's no more data to load @@ -256,16 +241,14 @@ export class CollectionSubscriber< // We need to schedule a graph run even if there's no data to load // because we need to mark the collection as ready if it's not already // and that's only done in `scheduleGraphRun` - this.collectionConfigBuilder.scheduleGraphRun(dataLoader, { - sourceId: this.sourceId, - }) + this.collectionConfigBuilder.scheduleGraphRun(dataLoader) } private subscribeToMatchingChanges( whereExpression: BasicExpression | undefined, includeInitialState: boolean, onStatusChange: (event: SubscriptionStatusChangeEvent) => void, - onLoadSubsetResult: (result: Promise | true) => void, + onLoadSubsetResult: (result: LoadSubsetRequestResult) => void, onLoadSubsetError: (event: SubscriptionLoadSubsetErrorEvent) => void, ): CollectionSubscription { const sendChanges = ( @@ -288,6 +271,7 @@ export class CollectionSubscriber< whereExpression, onStatusChange, onLoadSubsetError, + truncateReplayPublication: this.truncateReplayPublicationControl(), orderBy: hints.orderBy, limit: hints.limit, onLoadSubsetResult: includeInitialState ? onLoadSubsetResult : undefined, @@ -300,45 +284,24 @@ export class CollectionSubscriber< whereExpression: BasicExpression | undefined, orderByInfo: OrderByOptimizationInfo, onStatusChange: (event: SubscriptionStatusChangeEvent) => void, - onLoadSubsetResult: (result: Promise | true) => void, + onLoadSubsetResult: (result: LoadSubsetRequestResult) => void, onLoadSubsetError: (event: SubscriptionLoadSubsetErrorEvent) => void, ): CollectionSubscription { - const { orderBy, offset, limit, index } = orderByInfo - - // Store the callback so loadNextItems can also use direct tracking. - // Track in-flight ordered loads to avoid issuing redundant requests while - // a previous snapshot is still pending. - const handleLoadSubsetResult = (result: Promise | true) => { - if (result instanceof Promise) { - this.pendingOrderedLoadPromise = result - const finish = () => { - if (this.pendingOrderedLoadPromise === result) { - this.pendingOrderedLoadPromise = undefined - } - } - void result.then(finish, finish) - } - onLoadSubsetResult(result) - } - - this.orderedLoadSubsetResult = handleLoadSubsetResult - // Use a holder to forward-reference subscription in the callback const subscriptionHolder: { current?: CollectionSubscription } = {} const sendChangesInRange = ( changes: Iterable>, ) => { + const subscription = subscriptionHolder.current + if (!subscription) return const changesArray = Array.isArray(changes) ? changes : [...changes] - this.trackSentValues(changesArray, orderByInfo.comparator) + this.orderedLoader?.onSourceChanges(changesArray, this.sentToD2Rows) // Split live updates into a delete of the old value and an insert of the new value const splittedChanges = splitUpdates(changesArray) - this.sendChangesToPipelineWithTracking( - splittedChanges, - subscriptionHolder.current!, - ) + this.sendChangesToPipelineWithTracking(splittedChanges, subscription) } // Subscribe to changes with onStatusChange - listener is registered before any snapshot @@ -347,98 +310,102 @@ export class CollectionSubscriber< whereExpression, onStatusChange, onLoadSubsetError, + truncateReplayPublication: this.truncateReplayPublicationControl(() => { + // Recovery favors a simple, authoritative rebuild over resuming a + // fragile cursor. The retained full-source demand is replayed on later + // truncates, so this adds at most one demand per subscription. + // Queue startup inside the publication barrier too: a synchronous + // throw establishes no acquisition for the replay to wait on. + const loader = this.orderedLoader + this.collectionConfigBuilder.trackOrderedLoadPromise( + Promise.resolve().then(() => loader?.loadFullSource()), + true, + ) + }), }) subscriptionHolder.current = subscription this.registerSubscriptionCleanup(subscription) - // Listen for truncate events to reset cursor tracking state and sentToD2Keys - // This ensures that after a must-refetch/truncate, we don't use stale cursor data - // and allow re-inserts of previously sent keys + // Reset ordered-load state on truncate. Keep exact D2 rows until the + // replacement publication retracts or replaces them. const truncateUnsubscribe = this.collection.on(`truncate`, () => { - this.biggest = undefined - this.lastLoadRequestKey = undefined - this.pendingOrderedLoadPromise = undefined - this.sentToD2Keys.clear() + this.orderedLoader?.resetCursor() }) // Clean up truncate listener when subscription is unsubscribed subscription.on(`unsubscribed`, () => { truncateUnsubscribe() + subscriptionHolder.current = undefined + this.orderedLoader?.dispose() + this.orderedLoader = undefined }) - // Normalize the orderBy clauses such that the references are relative to the collection - const normalizedOrderBy = normalizeOrderByPaths(orderBy, this.alias) - - // Trigger the snapshot request — use direct load tracking (trackLoadSubsetPromise: false) - // to pipe the loadSubset result straight to the live query collection. This bypasses - // the subscription status → onStatusChange → deferred promise chain which is fragile - // under microtask timing (e.g., queueMicrotask delays in TanStack Query observers). - if (index) { - // We have an index on the first orderBy column - use lazy loading optimization - subscription.setOrderByIndex(index) - - subscription.requestLimitedSnapshot({ - limit: offset + limit, - orderBy: normalizedOrderBy, - trackLoadSubsetPromise: false, - onLoadSubsetResult: handleLoadSubsetResult, - }) - } else { - // No index available (e.g., non-ref expression): pass orderBy/limit to loadSubset - subscription.requestSnapshot({ - orderBy: normalizedOrderBy, - limit: offset + limit, - trackLoadSubsetPromise: false, - onLoadSubsetResult: handleLoadSubsetResult, - }) - } + this.orderedLoader = new OrderedSourceLoader( + orderByInfo, + subscription, + this.alias, + (result, holdPublication) => { + if (result instanceof Promise) { + this.collectionConfigBuilder.trackOrderedLoadPromise( + result, + holdPublication && !subscription.hasPendingTruncateReplacement, + ) + } + onLoadSubsetResult(result) + }, + ) + this.orderedLoader.start() return subscription } + private truncateReplayPublicationControl( + onStart?: () => void, + ): TruncateReplayPublicationControl { + const syncSession = this.collectionConfigBuilder.getSyncSession() + return { + start: () => { + onStart?.() + }, + succeed: () => { + if (syncSession !== this.collectionConfigBuilder.getSyncSession()) { + return + } + this.orderedLoader?.settleFullSourceReplay() + this.collectionConfigBuilder.scheduleGraphRunForSession(syncSession) + }, + } + } + // This function is called by maybeRunGraph // after each iteration of the query pipeline // to ensure that the orderBy operator has enough data to work with - loadMoreIfNeeded(subscription: CollectionSubscription) { + loadMoreIfNeeded(subscription: CollectionSubscription): void { + if ( + subscription.hasPendingTruncateReplacement && + !this.collectionConfigBuilder.hasActiveWindowOperation() + ) { + return + } + const orderByInfo = this.getOrderByInfo() if (!orderByInfo) { // This query has no orderBy operator // so there's no data to load - return true - } - - const { dataNeeded, index } = orderByInfo - - if (!dataNeeded || !index) { - // dataNeeded is not set when there's no index (e.g., non-ref expression - // or auto-indexing is disabled). Without an index, lazy loading can't work — - // all data was already loaded eagerly via requestSnapshot. - return true + return } - // `dataNeeded` probes the orderBy operator to see if it needs more data - // if it needs more data, it returns the number of items it needs - const n = dataNeeded() - if (n > 0) { - if (this.pendingOrderedLoadPromise) { - // The current window still needs the in-flight coverage. Attach it to - // this operation without making an unrelated or superseded request a - // dependency of every window change. - this.collectionConfigBuilder.trackSubsetLoadOperationPromise( - this.pendingOrderedLoadPromise, - ) - return true - } - try { - this.loadNextItems(n, subscription) - } catch (error) { - if (subscription.lastError !== error) throw error - // The subscription already reported the failure. Automatic refills - // must not make the source transaction that exposed the gap fail. + try { + const pending = this.orderedLoader?.loadMore( + this.collectionConfigBuilder.getActiveWindowOperationGeneration(), + ) + if (pending) { + this.collectionConfigBuilder.trackSubsetLoadOperationPromise(pending) } + } catch (error) { + if (!Object.is(subscription.lastError, error)) throw error } - return true } private sendChangesToPipelineWithTracking( @@ -455,7 +422,7 @@ export class CollectionSubscriber< // This ensures we pass the same function instance to the scheduler each time, // allowing it to deduplicate callbacks when multiple changes arrive during a transaction. type SubscriptionWithLoader = CollectionSubscription & { - [loadMoreCallbackSymbol]?: () => boolean + [loadMoreCallbackSymbol]?: () => void } const subscriptionWithLoader = subscription as SubscriptionWithLoader @@ -469,54 +436,6 @@ export class CollectionSubscriber< ) } - // Loads the next `n` items from the collection - // starting from the biggest item it has sent - private loadNextItems(n: number, subscription: CollectionSubscription) { - const orderByInfo = this.getOrderByInfo() - if (!orderByInfo) { - return - } - - const cursor = computeOrderedLoadCursor( - orderByInfo, - this.biggest, - this.lastLoadRequestKey, - this.alias, - n, - ) - if (!cursor) return // Duplicate request — skip - - const loadRequestKey = cursor.loadRequestKey - this.lastLoadRequestKey = loadRequestKey - - // Take the `n` items after the biggest sent value - // Omit offset so requestLimitedSnapshot can advance based on - // the number of rows already loaded (supports offset-based backends). - try { - subscription.requestLimitedSnapshot({ - orderBy: cursor.normalizedOrderBy, - limit: n, - minValues: cursor.minValues, - trackLoadSubsetPromise: false, - onLoadSubsetResult: (result) => { - if (result instanceof Promise) { - void result.then(undefined, () => { - if (this.lastLoadRequestKey === loadRequestKey) { - this.lastLoadRequestKey = undefined - } - }) - } - this.orderedLoadSubsetResult?.(result) - }, - }) - } catch (error) { - if (this.lastLoadRequestKey === loadRequestKey) { - this.lastLoadRequestKey = undefined - } - throw error - } - } - private getWhereClause(): BasicExpression | undefined { const sourceWhereClausesCache = this.collectionConfigBuilder.sourceWhereClausesCache @@ -535,22 +454,6 @@ export class CollectionSubscriber< return undefined } - private trackSentValues( - changes: Array>, - comparator: (a: any, b: any) => number, - ): void { - const result = trackBiggestSentValue( - changes, - this.biggest, - this.sentToD2Keys, - comparator, - ) - this.biggest = result.biggest - if (result.shouldResetLoadKey) { - this.lastLoadRequestKey = undefined - } - } - private ensureLoadingPromise(subscription: CollectionSubscription) { if (this.subscriptionLoadingPromises.has(subscription)) { return diff --git a/packages/db/src/query/live/materialized-pipeline.ts b/packages/db/src/query/live/materialized-pipeline.ts index c05de5639a..d7f76e79c5 100644 --- a/packages/db/src/query/live/materialized-pipeline.ts +++ b/packages/db/src/query/live/materialized-pipeline.ts @@ -7,13 +7,10 @@ import { reduce, serializeValue, } from '@tanstack/db-ivm' -import { - FN_SELECT_STATE, - INCLUDES_ROUTING, - validateFnSelectResult, -} from '../compiler/index.js' -import { VIRTUAL_PROP_NAMES } from '../../virtual-props.js' import { deepEquals } from '../../utils.js' +import { getParentContextIdentity } from '../equality-value-identity.js' +import { INCLUDES_ROUTING } from '../compiler/route-metadata.js' +import type { ValueIdentity } from '../equality-value-identity.js' import type { CompilationResult, IncludesCompilationResult, @@ -39,11 +36,6 @@ type IncludeRoute = { parentContext: Record | null } -type FnSelectState = { - sourceRow: Record - fnSelect: (row: any) => unknown -} - type CanonicalResult = { publicKey: unknown tuple: ResultTuple @@ -128,6 +120,7 @@ function materializeRelation( exposeRouting(compilation.pipeline), getKey, scope, + compilation.valueIdentity, ) const facades: Array = [] @@ -140,10 +133,17 @@ function materializeRelation( ) facades.push(...child.facades) - const bucketRows = createBucketRows(child.pipeline) + const bucketRows = createBucketRows( + child.pipeline, + include.childCompilationResult.valueIdentity, + ) if (include.materialization === `collection`) { const edgeId = `bucket-facade-${++nextBucketFacadeEdgeId}` - const activeBuckets = createActiveBuckets(pipeline, include) + const activeBuckets = createActiveBuckets( + pipeline, + include, + compilation.valueIdentity, + ) const activeBucketRows = activeBuckets.pipe( join(bucketRows), map(([bucketKey, [, row]]) => [bucketKey, row]), @@ -154,9 +154,21 @@ function materializeRelation( activeBuckets, hasOrderBy: include.hasOrderBy, }) - pipeline = attachCollectionInclude(pipeline, include, edgeId, scope) + pipeline = attachCollectionInclude( + pipeline, + include, + edgeId, + scope, + compilation.valueIdentity, + ) } else { - pipeline = attachInlineInclude(pipeline, bucketRows, include, scope) + pipeline = attachInlineInclude( + pipeline, + bucketRows, + include, + scope, + compilation.valueIdentity, + ) } } @@ -194,6 +206,7 @@ function canonicalizeByPublicKey( pipeline: ResultStream, getKey: ((row: any) => unknown) | undefined, scope: RelationScope, + valueIdentity: ValueIdentity, ): ResultStream { return pipeline.pipe( map(([internalKey, rawTuple]) => { @@ -202,7 +215,10 @@ function canonicalizeByPublicKey( const relationKey = scope === `root` ? serializeValue([`root`, publicKey]) - : serializeValue([routeKey(tuple[2], tuple[3]), publicKey]) + : serializeValue([ + routeKey(tuple[2], tuple[3], valueIdentity), + publicKey, + ]) return [relationKey, { publicKey, tuple }] as [string, CanonicalResult] }), reduce((values: Array<[CanonicalResult, number]>) => { @@ -261,6 +277,7 @@ function attachInlineInclude( bucketRows: IStreamBuilder<[string, BucketRow]>, include: IncludesCompilationResult, scope: RelationScope, + valueIdentity: ValueIdentity, ): ResultStream { const bucketValues = bucketRows.pipe( reduce((values: Array<[BucketRow, number]>) => { @@ -286,7 +303,11 @@ function attachInlineInclude( return [ routing?.active !== true ? `inactive:${serializeValue(parentKey)}` - : routeKey(routing.correlationKey, routing.parentContext), + : routeKey( + routing.correlationKey, + routing.parentContext, + valueIdentity, + ), { parentKey, tuple }, ] as [string, { parentKey: unknown; tuple: ResultTuple }] }), @@ -308,7 +329,7 @@ function attachInlineInclude( return [ parent!.parentKey, [ - setMaterializedInclude(value, include.resultPath, materialized), + setNestedValue(value, include.resultPath, materialized), order, correlationKey, parentContext, @@ -325,18 +346,20 @@ function attachInlineInclude( routedParents as ResultStream, undefined, scope, + valueIdentity, ) } function createBucketRows( childPipeline: ResultStream, + valueIdentity: ValueIdentity, ): IStreamBuilder<[string, BucketRow]> { return childPipeline.pipe( map(([internalKey, rawTuple]) => { const [value, order, correlationKey, parentContext, , publicKey] = rawTuple as ResultTuple return [ - routeKey(correlationKey, parentContext), + routeKey(correlationKey, parentContext, valueIdentity), { publicKey: publicKey ?? internalKey, value, order }, ] as [string, BucketRow] }), @@ -348,6 +371,7 @@ function attachCollectionInclude( include: IncludesCompilationResult, edgeId: string, scope: RelationScope, + valueIdentity: ValueIdentity, ): ResultStream { const routedParents = parentPipeline.pipe( map(([parentKey, rawTuple]) => { @@ -356,12 +380,12 @@ function attachCollectionInclude( if (routing?.active !== true) return [parentKey, tuple] const facade = createBucketFacadeRef( edgeId, - routeKey(routing.correlationKey, routing.parentContext), + routeKey(routing.correlationKey, routing.parentContext, valueIdentity), ) return [ parentKey, [ - setMaterializedInclude(tuple[0], include.resultPath, facade), + setNestedValue(tuple[0], include.resultPath, facade), tuple[1], tuple[2], tuple[3], @@ -375,12 +399,14 @@ function attachCollectionInclude( routedParents as ResultStream, undefined, scope, + valueIdentity, ) } function createActiveBuckets( parentPipeline: ResultStream, include: IncludesCompilationResult, + valueIdentity: ValueIdentity, ): IStreamBuilder<[string, true]> { return parentPipeline.pipe( map(([parentKey, rawTuple]) => { @@ -388,7 +414,11 @@ function createActiveBuckets( const routing = getIncludeRoute(tuple, include.fieldName) const bucketKey = routing?.active === true - ? routeKey(routing.correlationKey, routing.parentContext) + ? routeKey( + routing.correlationKey, + routing.parentContext, + valueIdentity, + ) : undefined return [parentKey, bucketKey] as [unknown, string | undefined] }), @@ -415,8 +445,12 @@ function getIncludeRoute( function routeKey( correlationKey: unknown, parentContext: Record | null | undefined, + valueIdentity: ValueIdentity, ): string { - return serializeValue([correlationKey ?? null, parentContext ?? null]) + return serializeValue([ + valueIdentity.equality(correlationKey ?? null), + getParentContextIdentity(parentContext ?? null), + ]) } function compareBucketRows(left: BucketRow, right: BucketRow): number { @@ -485,35 +519,3 @@ function setNestedValue( target[path[path.length - 1]!] = value return root } - -function setMaterializedInclude( - value: Record, - path: Array, - materialized: unknown, -): Record { - const state = value[FN_SELECT_STATE] as FnSelectState | undefined - if (!state) return setNestedValue(value, path, materialized) - - const sourceRow = setNestedValue(state.sourceRow, path, materialized) - const selectedValue = state.fnSelect(sourceRow) - validateFnSelectResult(selectedValue) - if (!selectedValue || typeof selectedValue !== `object`) { - throw new Error(`fn.select must return an object when it projects includes`) - } - - const selected: Record = Array.isArray(selectedValue) - ? [...selectedValue] - : { ...selectedValue } - for (const property of VIRTUAL_PROP_NAMES) { - if (property in value && !(property in selected)) { - selected[property] = value[property] - } - } - selected[INCLUDES_ROUTING] = value[INCLUDES_ROUTING] - Object.defineProperty(selected, FN_SELECT_STATE, { - value: { sourceRow, fnSelect: state.fnSelect }, - enumerable: true, - configurable: true, - }) - return selected -} diff --git a/packages/db/src/query/live/ordered-source-loader.ts b/packages/db/src/query/live/ordered-source-loader.ts new file mode 100644 index 0000000000..2ce16d7d27 --- /dev/null +++ b/packages/db/src/query/live/ordered-source-loader.ts @@ -0,0 +1,493 @@ +import { + buildCursorCurrent, + canExpressCursorOrder, +} from '../../utils/cursor.js' +import { normalizeError } from '../../utils/error.js' +import { normalizeOrderByPaths } from '../compiler/expressions.js' +import type { + CollectionSubscription, + ReleaseLoadSubset, +} from '../../collection/subscription.js' +import type { + ChangeMessage, + LoadSubsetOptions, + LoadSubsetRequestResult, +} from '../../types.js' +import type { OrderByOptimizationInfo } from '../compiler/order-by.js' + +type OrderedRequestKind = `ordered` | `boundary` | `full-source` + +/** Owns the conservative provider-loading policy for one ordered source. */ +export class OrderedSourceLoader { + private pending: Promise | undefined + // Exact request settlement is not provider extent. This latch only records + // that some request once completed; reset may discard the boundary, and an + // empty page retains it. A failure never reads it before a full-source + // completion sets it again, so it never needs clearing. + private hasSettledSourceRequest = false + private settledSourceBoundary: Record | undefined + // Independent of finite success: only full-source success repairs ordering. + private needsFullSourceRecovery = false + private requesting = false + // Retaining a demand does not prove it succeeded. Async failure retains it + // (`failed`) for replay; a synchronous startup failure retains nothing. + private fullSource: `none` | `held` | `failed` = `none` + // The record's presence blocks automatic retry, including initial requests + // that have no explicit window-operation generation. + private failedRequest: + | { windowOperationGeneration: number | undefined } + | undefined + private releaseFailedAcquisition: ReleaseLoadSubset | undefined + private active = true + private generation = 0 + private lastPage: { count: number; boundary: unknown } | undefined + private lastPrefixCount: number | undefined + private lastBoundary: unknown + + constructor( + private readonly info: OrderByOptimizationInfo, + private readonly subscription: CollectionSubscription, + private readonly alias: string, + private readonly onResult: ( + result: LoadSubsetRequestResult, + holdPublication: boolean, + ) => void = () => {}, + ) { + this.info.isRequesting = () => this.requesting + } + + /** Derive invalidation from actual contributions, not a second cursor. */ + onSourceChanges( + changes: Array, string | number>>, + sentRows: ReadonlyMap> | undefined, + ): void { + let hasNewRows = false + for (const change of changes) { + const previous = sentRows?.get(change.key) + if ( + change.type !== `insert` && + previous !== undefined && + (change.type === `delete` || + this.info.comparator(previous, change.value) !== 0) + ) { + this.invalidateSourceOrdering() + return + } + if (change.type !== `delete` && previous === undefined) hasNewRows = true + } + // New keys, including ties, may need another page. Duplicate delivery or + // an order-equal update cannot invalidate an already attempted request. + if (hasNewRows) this.invalidateCursor() + } + + start(): void { + const { index, limit, offset, orderBy, requiresFullSource } = this.info + if (index) this.subscription.setOrderByIndex(index) + if (limit === 0) return + if (requiresFullSource) { + this.loadFullSource() + return + } + if (!index || orderBy.length !== 1) { + this.loadPrefix(offset + limit) + return + } + this.loadPage(offset + limit) + } + + loadMore(windowOperationGeneration?: number): Promise | undefined { + if (!this.active || this.info.limit === 0 || this.requesting) return + const mayRetryFailure = + this.failedRequest === undefined || + (windowOperationGeneration !== undefined && + windowOperationGeneration !== + this.failedRequest.windowOperationGeneration) + if (!mayRetryFailure) return this.pending + if ( + (this.failedRequest || this.releaseFailedAcquisition) && + windowOperationGeneration !== undefined + ) { + // Move ownership to the explicit replacement before releasing the old + // lease. Adapter cleanup may reenter the loader. + if (this.failedRequest) { + this.failedRequest.windowOperationGeneration = windowOperationGeneration + } + const releaseFailedAcquisition = this.releaseFailedAcquisition + this.releaseFailedAcquisition = undefined + if (releaseFailedAcquisition) { + this.requesting = true + try { + releaseFailedAcquisition() + } finally { + this.requesting = false + } + // Adapter cleanup can synchronously tear down this loader. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + if (!this.active) return + } + } + if (this.fullSource === `failed`) this.fullSource = `none` + else if (this.fullSource === `held`) return this.pending + if (this.needsFullSourceRecovery || this.info.requiresFullSource) { + this.loadFullSource(windowOperationGeneration) + return this.pending + } + if (!this.info.index || this.info.orderBy.length !== 1) { + this.loadPrefix( + this.info.offset + this.info.limit, + windowOperationGeneration, + ) + return this.pending + } + if (!this.info.dataNeeded || this.pending) return this.pending + // A recorded failure always carries recovery debt, so it cannot reach this + // finite path; only the first request needs the whole prefix here. + let count = Math.max( + this.info.dataNeeded(), + this.hasSettledSourceRequest ? 0 : this.info.offset + this.info.limit, + ) + if ( + windowOperationGeneration !== undefined && + this.settledSourceBoundary !== undefined + ) { + const needed = this.info.offset + this.info.limit + count = Math.max(count, needed - this.countAcquiredRows()) + } + if (count > 0) { + this.loadPage(count, windowOperationGeneration) + } + return this.pending + } + + loadFullSource(windowOperationGeneration?: number): void { + if (!this.active || this.fullSource !== `none`) return + this.fullSource = `held` + this.requestAndObserve( + (onLoadSubsetResult) => { + this.subscription.requestSnapshot({ + trackLoadSubsetPromise: false, + onLoadSubsetResult, + }) + }, + `full-source`, + windowOperationGeneration, + ) + } + + private loadPrefix(count: number, windowOperationGeneration?: number): void { + if (!this.active || this.pending) return + if (this.lastPrefixCount === count) { + if ((this.info.dataNeeded?.() ?? 0) > 0) { + this.loadFullSource(windowOperationGeneration) + } + return + } + this.requestAndObserve( + (onLoadSubsetResult) => { + this.subscription.requestSnapshot({ + orderBy: normalizeOrderByPaths(this.info.orderBy, this.alias), + limit: count, + trackLoadSubsetPromise: false, + onLoadSubsetResult, + }) + }, + `ordered`, + windowOperationGeneration, + ) + this.lastPrefixCount = count + } + + resetCursor(): void { + this.generation++ + this.pending = undefined + this.lastBoundary = undefined + this.settledSourceBoundary = undefined + this.invalidateCursor() + } + + settleFullSourceReplay(): void { + // Replay repaired the retained logical acquisition. A later window retry + // must not release that now-successful source demand. A failed finite + // page is still obsolete and must be released by that retry. + if (this.fullSource === `failed`) { + this.releaseFailedAcquisition = undefined + this.fullSource = `held` + } + } + + invalidateCursor(): void { + this.lastPage = undefined + this.lastPrefixCount = undefined + } + + invalidateSourceOrdering(): void { + this.invalidateCursor() + this.requireFullSourceRecovery() + } + + dispose(): void { + this.active = false + this.resetCursor() + } + + private countAcquiredRows(): number { + return this.subscription + .readOrderedSnapshot({ + orderBy: normalizeOrderByPaths(this.info.orderBy, this.alias), + limit: this.info.offset + this.info.limit, + }) + .filter( + ({ value }) => + this.info.comparator(value, this.settledSourceBoundary) <= 0, + ).length + } + + private loadPage(count: number, windowOperationGeneration?: number): void { + if (!this.active || this.pending) return + // Rows observed before the first provider request do not prove ordered + // source coverage. In particular, a row inserted while limit is zero must + // not become the cursor when that window first opens. + const startsFromSourcePrefix = this.settledSourceBoundary === undefined + const biggest = this.settledSourceBoundary + let minValues: Array | undefined + if (biggest !== undefined) { + const value = this.info.valueExtractorForRawRow(biggest) + if (!canExpressCursorOrder(this.info.orderBy, [value])) { + this.loadPrefix( + this.info.offset + this.info.limit, + windowOperationGeneration, + ) + return + } + minValues = [value] + } + const boundary = minValues?.[0] + if ( + this.lastPage?.count === count && + Object.is(this.lastPage.boundary, boundary) + ) { + return + } + this.lastPage = { count, boundary } + this.requestAndObserve( + (onLoadSubsetResult) => { + this.subscription.requestLimitedSnapshot({ + orderBy: normalizeOrderByPaths(this.info.orderBy, this.alias), + limit: count, + minValues, + // Local rows seen before the first provider request prove neither + // a cursor nor a remote offset. Start the first acquisition at zero. + offset: startsFromSourcePrefix ? 0 : this.countAcquiredRows(), + trackLoadSubsetPromise: false, + onLoadSubsetResult, + }) + }, + `ordered`, + windowOperationGeneration, + ) + } + + private observe( + result: LoadSubsetRequestResult, + releaseAcquisition: ReleaseLoadSubset, + kind: OrderedRequestKind, + windowOperationGeneration?: number, + options?: LoadSubsetOptions, + ): Promise { + const isFullSource = kind === `full-source` + const generation = this.generation + const complete = (): void => { + if (this.pending === tracked) this.pending = undefined + if (!this.active || generation !== this.generation) return + // A finite request may finish behind an authoritative repair. It cannot + // clear that repair's failure or resume finite refinement around it. + if (!isFullSource && (this.failedRequest || this.fullSource !== `none`)) + return + this.failedRequest = undefined + if (kind !== `boundary`) { + this.hasSettledSourceRequest = true + // Source delivery can invalidate the in-flight prefix marker. + if (options?.orderBy && !options.cursor) { + this.lastPrefixCount = options.limit + } + if (!isFullSource && options?.orderBy) { + try { + this.settledSourceBoundary = + this.subscription.readOrderedSnapshot(options).at(-1)?.value ?? + this.settledSourceBoundary + } catch (error) { + fail(error) + } + } + } + if (isFullSource) this.needsFullSourceRecovery = false + if (kind === `ordered`) { + this.loadBoundary(windowOperationGeneration) + return + } + // A boundary request may add tied rows without filling the query's + // window. Resume forward loading once it settles. + this.loadMore() + } + const settlesAsync = result instanceof Promise + const request = settlesAsync ? result : Promise.resolve() + const fail = (error: unknown) => { + if (this.pending === tracked) this.pending = undefined + if (!this.active) return + // A failed request may already have written only part of its result. + // None of those rows is a safe continuation boundary. + this.requireFullSourceRecovery() + if (generation !== this.generation) return + // A failed request proves no full-source coverage. An explicit window + // move or later replay may retry it, but an ordinary graph pass must + // not start an eager retry loop. + if (isFullSource) this.fullSource = `failed` + this.recordRequestFailure(windowOperationGeneration) + this.releaseFailedAcquisition = releaseAcquisition + throw error + } + const tracked = request.then(complete, fail) + this.pending = tracked + void tracked.catch(() => {}) + // Register each request separately. The operation tracker observes the + // next request before this promise settles, so the logical chain remains + // pending without retaining every ancestor promise until the final page. + this.onResult( + tracked, + settlesAsync && isFullSource && this.needsFullSourceRecovery, + ) + return tracked + } + + private loadBoundary( + windowOperationGeneration?: number, + ): Promise | undefined { + const biggest = this.settledSourceBoundary + if (biggest === undefined) return + const value = this.info.valueExtractorForRawRow(biggest) + const orderBy = normalizeOrderByPaths(this.info.orderBy, this.alias) + if (!canExpressCursorOrder(orderBy.slice(0, 1), [value])) { + this.loadFullSource(windowOperationGeneration) + return this.pending + } + // Undefined is not an expressible cursor boundary, so it denotes that no + // tie request has been attempted. Other falsy values remain valid keys. + if (Object.is(this.lastBoundary, value)) { + return this.loadMore() + } + const where = buildCursorCurrent(orderBy, [value]) + if (!where) { + this.loadFullSource(windowOperationGeneration) + return this.pending + } + this.lastBoundary = value + return this.requestAndObserve( + (onLoadSubsetResult) => { + this.subscription.requestSnapshot({ + where, + trackLoadSubsetPromise: false, + onLoadSubsetResult, + }) + }, + `boundary`, + windowOperationGeneration, + ) + } + + private requireFullSourceRecovery(): void { + this.settledSourceBoundary = undefined + this.needsFullSourceRecovery = true + } + + private failRequest( + observed: + | { + result: LoadSubsetRequestResult + options: LoadSubsetOptions + release: ReleaseLoadSubset + } + | undefined, + error: Error, + isFullSource: boolean, + windowOperationGeneration?: number, + cancelObservedSettlement = false, + ): Error { + if (cancelObservedSettlement) { + this.generation++ + this.pending = undefined + } + this.requireFullSourceRecovery() + this.recordRequestFailure(windowOperationGeneration) + if (isFullSource) this.fullSource = `none` + try { + observed?.release({ error }) + } catch { + // Cleanup is attempted once and must not replace the request failure. + } + return error + } + + /** A failed request blocks ordinary refinement until a new operation. */ + private recordRequestFailure(windowOperationGeneration?: number): void { + this.failedRequest = { windowOperationGeneration } + this.invalidateCursor() + this.lastBoundary = undefined + } + + /** Observe settlement only after all synchronous request work succeeds. */ + private requestAndObserve( + request: ( + onResult: ( + result: LoadSubsetRequestResult, + options: LoadSubsetOptions, + release: ReleaseLoadSubset, + ) => void, + ) => void, + kind: OrderedRequestKind, + windowOperationGeneration?: number, + ): Promise | undefined { + const isFullSource = kind === `full-source` + let observed: + | { + result: LoadSubsetRequestResult + options: LoadSubsetOptions + release: ReleaseLoadSubset + } + | undefined + this.requesting = true + let observing = false + try { + try { + request((result, options, release) => { + observed = { result, options, release } + }) + } finally { + this.requesting = false + } + if (!observed) return + observing = true + return this.observe( + observed.result, + observed.release, + kind, + windowOperationGeneration, + observed.options, + ) + } catch (error) { + // Both request and settlement callbacks may reenter through cleanup. + // Keep refinement blocked until failure and release finish unwinding. + this.requesting = true + try { + throw this.failRequest( + observed, + normalizeError(error), + isFullSource, + windowOperationGeneration, + observing, + ) + } finally { + this.requesting = false + } + } + } +} diff --git a/packages/db/src/query/live/subset-demand-controller.ts b/packages/db/src/query/live/subset-demand-controller.ts index 8ccda3d1f9..223f9c96de 100644 --- a/packages/db/src/query/live/subset-demand-controller.ts +++ b/packages/db/src/query/live/subset-demand-controller.ts @@ -1,15 +1,17 @@ -import { serializeValue } from '@tanstack/db-ivm' import { inArray } from '../builder/functions.js' import { PropRef } from '../ir.js' +import { createValueIdentity } from '../equality-value-identity.js' +import type { ValueIdentity } from '../equality-value-identity.js' import type { CollectionSubscription } from '../../collection/subscription.js' import type { LazyDemandPlan } from '../compiler/joins.js' import type { BasicExpression } from '../ir.js' +import type { LoadSubsetRequestResult } from '../../types.js' type DemandSegment = { keys: Map where: BasicExpression abortController: AbortController - ready: Promise | true + ready: LoadSubsetRequestResult state: `pending` | `settled` | `failed` } @@ -21,7 +23,7 @@ type DemandState = { export type DemandUpdate = { changed: boolean empty: boolean - ready: Promise | true + ready: Promise> | true } /** @@ -32,13 +34,14 @@ export type DemandUpdate = { export class SubsetDemandController { private readonly states = new Map() private readonly warnedPlans = new Set() + private valueIdentity = createValueIdentity() setDemand( subscription: CollectionSubscription, plan: LazyDemandPlan, keys: Set, ): DemandUpdate { - const nextKeys = canonicalizeKeys(keys) + const nextKeys = canonicalizeKeys(keys, this.valueIdentity) const previous = this.states.get(plan.id) const hasFailedCoverage = previous?.segments.some( (segment) => @@ -61,7 +64,13 @@ export class SubsetDemandController { } segment.abortController.abort() - subscription.releaseSnapshot(segment.where) + try { + subscription.releaseSnapshot(segment.where) + } catch { + // The subscription reports adapter cleanup failures and keeps the + // physical acquisition for a later unsubscribe retry. Demand changes + // must still reach the graph instead of escaping the source commit. + } } const coveredKeys = new Set( @@ -93,8 +102,7 @@ export class SubsetDemandController { return { changed: true, empty: nextKeys.size === 0, - ready: - pending.length > 0 ? Promise.all(pending).then(() => undefined) : true, + ready: pending.length > 0 ? Promise.all(pending) : true, } } @@ -104,6 +112,7 @@ export class SubsetDemandController { } this.states.clear() this.warnedPlans.clear() + this.valueIdentity = createValueIdentity() } private warnUnoptimized(plan: LazyDemandPlan): void { @@ -119,8 +128,13 @@ export class SubsetDemandController { } } -function canonicalizeKeys(keys: Set): Map { - return new Map([...keys].map((key) => [serializeValue(key), key])) +function canonicalizeKeys( + keys: Set, + valueIdentity: ValueIdentity, +): Map { + return new Map( + [...keys].map((key) => [valueIdentity.serializeEquality(key), key]), + ) } function equalKeySets( @@ -147,7 +161,7 @@ function requestSegment( ): DemandSegment { const where = inArray(new PropRef(plan.path), [...keys.values()]) const abortController = new AbortController() - const load = { ready: true as Promise | true } + const load = { ready: true as LoadSubsetRequestResult } subscription.requestSnapshot({ where, signal: abortController.signal, diff --git a/packages/db/src/query/live/utils.ts b/packages/db/src/query/live/utils.ts index f45177a322..a1823b7bf0 100644 --- a/packages/db/src/query/live/utils.ts +++ b/packages/db/src/query/live/utils.ts @@ -1,4 +1,4 @@ -import { MultiSet, serializeValue } from '@tanstack/db-ivm' +import { MultiSet } from '@tanstack/db-ivm' import { UnsupportedRootScalarSelectError } from '../../errors.js' import { normalizeOrderByPaths } from '../compiler/expressions.js' import { buildQuery, getQueryIR } from '../builder/index.js' @@ -9,7 +9,6 @@ import type { ChangeMessage } from '../../types.js' import type { InitialQueryBuilder, QueryBuilder } from '../builder/index.js' import type { Context } from '../builder/types.js' import type { OrderBy, QueryIR } from '../ir.js' -import type { OrderByOptimizationInfo } from '../compiler/order-by.js' /** * Helper function to extract collections from a compiled query. @@ -139,71 +138,35 @@ export function* splitUpdates< } } -/** - * Filter changes to prevent duplicate inserts to a D2 pipeline. - * Maintains D2 multiplicity at 1 for visible items so that deletes - * properly reduce multiplicity to 0. - * - * Mutates `sentKeys` in place: adds keys on insert, removes on delete. - */ -export function filterDuplicateInserts( - changes: Array>, - sentKeys: Set, -): Array> { - const filtered: Array> = [] +/** Keep each source key at one exact D2 contribution. */ +export function reconcileChangesForD2< + T extends object, + TKey extends string | number, +>( + changes: Array>, + sentRows: Map, +): Array> { + const reconciled: Array> = [] for (const change of changes) { + const previousValue = sentRows.get(change.key) if (change.type === `insert`) { - if (sentKeys.has(change.key)) { - continue // Skip duplicate - } - sentKeys.add(change.key) + if (previousValue !== undefined) continue + sentRows.set(change.key, change.value) + reconciled.push(change) } else if (change.type === `delete`) { - sentKeys.delete(change.key) - } - filtered.push(change) - } - return filtered -} - -/** - * Track the biggest value seen in a stream of changes, used for cursor-based - * pagination in ordered subscriptions. Returns whether the load request key - * should be reset (allowing another load). - * - * @param changes - changes to process (deletes are skipped) - * @param current - the current biggest value (or undefined if none) - * @param sentKeys - set of keys already sent to D2 (for new-key detection) - * @param comparator - orderBy comparator - * @returns `{ biggest, shouldResetLoadKey }` — the new biggest value and - * whether the caller should clear its last-load-request-key - */ -export function trackBiggestSentValue( - changes: Array>, - current: unknown | undefined, - sentKeys: Set, - comparator: (a: any, b: any) => number, -): { biggest: unknown; shouldResetLoadKey: boolean } { - let biggest = current - let shouldResetLoadKey = false - - for (const change of changes) { - if (change.type === `delete`) continue - - const isNewKey = !sentKeys.has(change.key) - - if (biggest === undefined) { - biggest = change.value - shouldResetLoadKey = true - } else if (comparator(biggest, change.value) < 0) { - biggest = change.value - shouldResetLoadKey = true - } else if (isNewKey) { - // New key at same sort position — allow another load if needed - shouldResetLoadKey = true + if (previousValue === undefined) continue + sentRows.delete(change.key) + reconciled.push({ ...change, value: previousValue }) + } else { + sentRows.set(change.key, change.value) + reconciled.push( + previousValue === undefined + ? { type: `insert`, key: change.key, value: change.value } + : { ...change, previousValue }, + ) } } - - return { biggest, shouldResetLoadKey } + return reconciled } /** @@ -239,58 +202,3 @@ export function computeSubscriptionOrderByHints( limit: canPassOrderBy ? effectiveLimit : undefined, } } - -/** - * Compute the cursor for loading the next batch of ordered data. - * Extracts values from the biggest sent row and builds the `minValues` - * array and a deduplication key. - * - * @returns `undefined` if the load should be skipped (duplicate request), - * otherwise `{ minValues, normalizedOrderBy, loadRequestKey }`. - */ -export function computeOrderedLoadCursor( - orderByInfo: Pick< - OrderByOptimizationInfo, - 'orderBy' | 'valueExtractorForRawRow' | 'offset' - >, - biggestSentRow: unknown | undefined, - lastLoadRequestKey: string | undefined, - alias: string, - limit: number, -): - | { - minValues: Array | undefined - normalizedOrderBy: OrderBy - loadRequestKey: string - } - | undefined { - const { orderBy, valueExtractorForRawRow, offset } = orderByInfo - - // Extract all orderBy column values from the biggest sent row - // For single-column: returns single value, for multi-column: returns array - const extractedValues = biggestSentRow - ? valueExtractorForRawRow(biggestSentRow as Record) - : undefined - - // Normalize to array format for minValues - let minValues: Array | undefined - if (extractedValues !== undefined) { - minValues = Array.isArray(extractedValues) - ? extractedValues - : [extractedValues] - } - - // Deduplicate: skip if we already issued an identical load request - const loadRequestKey = serializeValue({ - minValues: minValues ?? null, - offset, - limit, - }) - if (lastLoadRequestKey === loadRequestKey) { - return undefined - } - - const normalizedOrderBy = normalizeOrderByPaths(orderBy, alias) - - return { minValues, normalizedOrderBy, loadRequestKey } -} diff --git a/packages/db/src/query/optimizer.ts b/packages/db/src/query/optimizer.ts index fbc50661da..19ce43b40f 100644 --- a/packages/db/src/query/optimizer.ts +++ b/packages/db/src/query/optimizer.ts @@ -130,6 +130,7 @@ import { UnionAll as UnionAllClass, UnionFrom as UnionFromClass, createResidualWhere, + getFromSources, getWhereExpression, isResidualWhere, } from './ir.js' @@ -900,16 +901,6 @@ function optimizeNestedFrom(from: From): From { return from } -function getFromSources(from: From): Array { - if (from.type === `unionFrom`) { - return from.sources - } - if (from.type === `unionAll`) { - return [] - } - return [from] -} - function getFirstFromAlias(query: QueryIR): string | undefined { return getFromSources(query.from)[0]?.alias } diff --git a/packages/db/src/query/predicate-utils.ts b/packages/db/src/query/predicate-utils.ts deleted file mode 100644 index 3241f9e55d..0000000000 --- a/packages/db/src/query/predicate-utils.ts +++ /dev/null @@ -1,1641 +0,0 @@ -import { Func, Value } from './ir.js' -import { - UnhashableQueryIRError, - getStableExpressionHash, - getStableValueHash, -} from './ir-stable-identity.js' -import type { BasicExpression, OrderBy, PropRef } from './ir.js' -import type { LoadSubsetOptions } from '../types.js' -import type { CompareOptions } from './builder/types.js' - -/** - * Check if one where clause is a logical subset of another. - * Returns true if the subset predicate is more restrictive than (or equal to) the superset predicate. - * - * @example - * // age > 20 is subset of age > 10 (more restrictive) - * isWhereSubset(gt(ref('age'), val(20)), gt(ref('age'), val(10))) // true - * - * @example - * // age > 10 AND name = 'X' is subset of age > 10 (more conditions) - * isWhereSubset(and(gt(ref('age'), val(10)), eq(ref('name'), val('X'))), gt(ref('age'), val(10))) // true - * - * @param subset - The potentially more restrictive predicate - * @param superset - The potentially less restrictive predicate - * @returns true if subset logically implies superset - */ -export function isWhereSubset( - subset: BasicExpression | undefined, - superset: BasicExpression | undefined, -): boolean { - // undefined/missing where clause means "no filter" (all data) - // Both undefined means subset relationship holds (all data ⊆ all data) - if (subset === undefined && superset === undefined) { - return true - } - - // If subset is undefined but superset is not, we're requesting ALL data - // but have only loaded SOME data - subset relationship does NOT hold - if (subset === undefined && superset !== undefined) { - return false - } - - // If superset is undefined (no filter = all data loaded), - // then any constrained subset is contained - if (superset === undefined && subset !== undefined) { - return true - } - - return isWhereSubsetInternal(subset!, superset!, new WeakMap()) -} - -function makeDisjunction( - preds: Array>, -): BasicExpression { - if (preds.length === 0) { - return new Value(false) - } - if (preds.length === 1) { - return preds[0]! - } - return new Func(`or`, preds) -} - -function convertInToOr(inField: InField) { - const equalities = inField.values.map( - (value) => new Func(`eq`, [inField.ref, new Value(value)]), - ) - return makeDisjunction(equalities) -} - -function isWhereSubsetInternal( - subset: BasicExpression, - superset: BasicExpression, - expressionHashes: ExpressionHashCache, -): boolean { - // If subset is false it is requesting no data, - // thus the result set is empty - // and the empty set is a subset of any set - if (subset.type === `val` && subset.value === false) { - return true - } - - // If expressions are structurally equal, subset relationship holds - if (areExpressionsEqual(subset, superset, expressionHashes)) { - return true - } - - // Handle superset being an AND: subset must imply ALL conjuncts - // If superset is (A AND B), then subset ⊆ (A AND B) only if subset ⊆ A AND subset ⊆ B - // Example: (age > 20) ⊆ (age > 10 AND status = 'active') is false (doesn't imply status condition) - if (superset.type === `func` && superset.name === `and`) { - return superset.args.every((arg) => - isWhereSubsetInternal( - subset, - arg as BasicExpression, - expressionHashes, - ), - ) - } - - // Handle OR in subset: (A OR B) ⊆ C only if both A ⊆ C and B ⊆ C. - // Must be checked before OR superset so that or(A, B) ⊆ or(C, D) - // decomposes the subset first: A ⊆ or(C, D) AND B ⊆ or(C, D). - if (subset.type === `func` && subset.name === `or`) { - return subset.args.every((arg) => - isWhereSubsetInternal( - arg as BasicExpression, - superset, - expressionHashes, - ), - ) - } - - // Handle OR in superset: subset ⊆ (A OR B) if subset ⊆ A or subset ⊆ B. - // Must be checked before decomposing AND subsets so that and(A, B) can - // match a structurally equal disjunct via areExpressionsEqual. - if (superset.type === `func` && superset.name === `or`) { - return superset.args.some((arg) => - isWhereSubsetInternal( - subset, - arg as BasicExpression, - expressionHashes, - ), - ) - } - - // Handle subset being an AND: (A AND B) implies both A and B - if (subset.type === `func` && subset.name === `and`) { - // For (A AND B) ⊆ C, since (A AND B) implies A, we check if any conjunct implies C - return subset.args.some((arg) => - isWhereSubsetInternal( - arg as BasicExpression, - superset, - expressionHashes, - ), - ) - } - - // Turn x IN [A, B, C] into x = A OR x = B OR x = C - // for unified handling of IN and OR - if (subset.type === `func` && subset.name === `in`) { - const inField = extractInField(subset) - if (inField) { - return isWhereSubsetInternal( - convertInToOr(inField), - superset, - expressionHashes, - ) - } - } - - if (superset.type === `func` && superset.name === `in`) { - const inField = extractInField(superset) - if (inField) { - return isWhereSubsetInternal( - subset, - convertInToOr(inField), - expressionHashes, - ) - } - } - - // Handle comparison operators on the same field - if (subset.type === `func` && superset.type === `func`) { - const subsetFunc = subset as Func - const supersetFunc = superset as Func - - // Check if both are comparisons on the same field - const subsetField = extractComparisonField(subsetFunc) - const supersetField = extractComparisonField(supersetFunc) - - if ( - subsetField && - supersetField && - areRefsEqual(subsetField.ref, supersetField.ref) - ) { - return isComparisonSubset( - subsetFunc, - subsetField.value, - supersetFunc, - supersetField.value, - ) - } - - /* - // Handle eq vs in - if (subsetFunc.name === `eq` && supersetFunc.name === `in`) { - const subsetFieldEq = extractEqualityField(subsetFunc) - const supersetFieldIn = extractInField(supersetFunc) - if ( - subsetFieldEq && - supersetFieldIn && - areRefsEqual(subsetFieldEq.ref, supersetFieldIn.ref) - ) { - // field = X is subset of field IN [X, Y, Z] if X is in the array - // Use cached primitive set and metadata from extraction - return arrayIncludesWithSet( - supersetFieldIn.values, - subsetFieldEq.value, - supersetFieldIn.primitiveSet ?? null, - supersetFieldIn.areAllPrimitives - ) - } - } - - // Handle in vs in - if (subsetFunc.name === `in` && supersetFunc.name === `in`) { - const subsetFieldIn = extractInField(subsetFunc) - const supersetFieldIn = extractInField(supersetFunc) - if ( - subsetFieldIn && - supersetFieldIn && - areRefsEqual(subsetFieldIn.ref, supersetFieldIn.ref) - ) { - // field IN [A, B] is subset of field IN [A, B, C] if all values in subset are in superset - // Use cached primitive set and metadata from extraction - return subsetFieldIn.values.every((subVal) => - arrayIncludesWithSet( - supersetFieldIn.values, - subVal, - supersetFieldIn.primitiveSet ?? null, - supersetFieldIn.areAllPrimitives - ) - ) - } - } - */ - } - - // Conservative: if we can't determine, return false - return false -} - -/** - * Helper to combine where predicates with common logic for AND/OR operations - */ -function combineWherePredicates( - predicates: Array>, - operation: `and` | `or`, - simplifyFn: ( - preds: Array>, - ) => BasicExpression | null, -): BasicExpression { - const emptyValue = operation === `and` ? true : false - const identityValue = operation === `and` ? true : false - - if (predicates.length === 0) { - return { type: `val`, value: emptyValue } as BasicExpression - } - - if (predicates.length === 1) { - return predicates[0]! - } - - // Flatten nested expressions of the same operation - const flatPredicates: Array> = [] - for (const pred of predicates) { - if (pred.type === `func` && pred.name === operation) { - flatPredicates.push(...pred.args) - } else { - flatPredicates.push(pred) - } - } - - // Group predicates by field for simplification - const grouped = groupPredicatesByField(flatPredicates) - - // Simplify each group - const simplified: Array> = [] - for (const [field, preds] of grouped.entries()) { - if (field === null) { - // Complex predicates that we can't group by field - simplified.push(...preds) - } else { - // Try to simplify same-field predicates - const result = simplifyFn(preds) - - // For intersection: check for empty set (contradiction) - if ( - operation === `and` && - result && - result.type === `val` && - result.value === false - ) { - // Intersection is empty (conflicting constraints) - entire AND is false - return { type: `val`, value: false } as BasicExpression - } - - // For union: result may be null if simplification failed - if (result) { - simplified.push(result) - } - } - } - - if (simplified.length === 0) { - return { type: `val`, value: identityValue } as BasicExpression - } - - if (simplified.length === 1) { - return simplified[0]! - } - - // Return combined predicate - return { - type: `func`, - name: operation, - args: simplified, - } as BasicExpression -} - -/** - * Combine multiple where predicates with OR logic (union). - * Returns a predicate that is satisfied when any input predicate is satisfied. - * Simplifies when possible (e.g., age > 10 OR age > 20 → age > 10). - * - * @example - * // Take least restrictive - * unionWherePredicates([gt(ref('age'), val(10)), gt(ref('age'), val(20))]) // age > 10 - * - * @example - * // Combine equals into IN - * unionWherePredicates([eq(ref('age'), val(5)), eq(ref('age'), val(10))]) // age IN [5, 10] - * - * @param predicates - Array of where predicates to union - * @returns Combined predicate representing the union - */ -export function unionWherePredicates( - predicates: Array>, -): BasicExpression { - return combineWherePredicates(predicates, `or`, unionSameFieldPredicates) -} - -/** - * Compute the difference between two where predicates: `fromPredicate AND NOT(subtractPredicate)`. - * Returns the simplified predicate, or null if the difference cannot be simplified - * (in which case the caller should fetch the full fromPredicate). - * - * @example - * // Range difference - * minusWherePredicates( - * gt(ref('age'), val(10)), // age > 10 - * gt(ref('age'), val(20)) // age > 20 - * ) // → age > 10 AND age <= 20 - * - * @example - * // Set difference - * minusWherePredicates( - * inOp(ref('status'), ['A', 'B', 'C', 'D']), // status IN ['A','B','C','D'] - * inOp(ref('status'), ['B', 'C']) // status IN ['B','C'] - * ) // → status IN ['A', 'D'] - * - * @example - * // Common conditions - * minusWherePredicates( - * and(gt(ref('age'), val(10)), eq(ref('status'), val('active'))), // age > 10 AND status = 'active' - * and(gt(ref('age'), val(20)), eq(ref('status'), val('active'))) // age > 20 AND status = 'active' - * ) // → age > 10 AND age <= 20 AND status = 'active' - * - * @example - * // Complete overlap - empty result - * minusWherePredicates( - * gt(ref('age'), val(20)), // age > 20 - * gt(ref('age'), val(10)) // age > 10 - * ) // → {type: 'val', value: false} (empty set) - * - * @param fromPredicate - The predicate to subtract from - * @param subtractPredicate - The predicate to subtract - * @returns The simplified difference, or null if cannot be simplified - */ -export function minusWherePredicates( - fromPredicate: BasicExpression | undefined, - subtractPredicate: BasicExpression | undefined, -): BasicExpression | null { - // If nothing to subtract, return the original - if (subtractPredicate === undefined) { - return ( - fromPredicate ?? - ({ type: `val`, value: true } as BasicExpression) - ) - } - - // If from is undefined then we are asking for all data - // so we need to load all data minus what we already loaded - // i.e. we need to load NOT(subtractPredicate) - if (fromPredicate === undefined) { - return { - type: `func`, - name: `not`, - args: [subtractPredicate], - } as BasicExpression - } - - // Check if fromPredicate is entirely contained in subtractPredicate - // In that case, fromPredicate AND NOT(subtractPredicate) = empty set - if (isWhereSubset(fromPredicate, subtractPredicate)) { - return { type: `val`, value: false } as BasicExpression - } - - // Try to detect and handle common conditions - const commonConditions = findCommonConditions( - fromPredicate, - subtractPredicate, - ) - if (commonConditions.length > 0) { - // Extract predicates without common conditions - const fromWithoutCommon = removeConditions(fromPredicate, commonConditions) - const subtractWithoutCommon = removeConditions( - subtractPredicate, - commonConditions, - ) - - // Recursively compute difference on simplified predicates - const simplifiedDifference = minusWherePredicates( - fromWithoutCommon, - subtractWithoutCommon, - ) - - if (simplifiedDifference !== null) { - // Combine the simplified difference with common conditions - return combineConditions([...commonConditions, simplifiedDifference]) - } - } - - // Check if they are on the same field - if so, we can try to simplify - if (fromPredicate.type === `func` && subtractPredicate.type === `func`) { - const result = minusSameFieldPredicates(fromPredicate, subtractPredicate) - if (result !== null) { - return result - } - } - - // Can't simplify - return null to indicate caller should fetch full fromPredicate - return null -} - -/** - * Helper function to compute difference for same-field predicates - */ -function minusSameFieldPredicates( - fromPred: Func, - subtractPred: Func, -): BasicExpression | null { - // Extract field information - const fromField = - extractComparisonField(fromPred) || - extractEqualityField(fromPred) || - extractInField(fromPred) - const subtractField = - extractComparisonField(subtractPred) || - extractEqualityField(subtractPred) || - extractInField(subtractPred) - - // Must be on the same field - if ( - !fromField || - !subtractField || - !areRefsEqual(fromField.ref, subtractField.ref) - ) { - return null - } - - // Handle IN minus IN: status IN [A,B,C,D] - status IN [B,C] = status IN [A,D] - if (fromPred.name === `in` && subtractPred.name === `in`) { - const fromInField = fromField as InField - const subtractInField = subtractField as InField - - // Filter out values that are in the subtract set - const remainingValues = fromInField.values.filter( - (v) => - !arrayIncludesWithSet( - subtractInField.values, - v, - subtractInField.primitiveSet ?? null, - subtractInField.areAllPrimitives, - ), - ) - - if (remainingValues.length === 0) { - return { type: `val`, value: false } as BasicExpression - } - - if (remainingValues.length === 1) { - return { - type: `func`, - name: `eq`, - args: [fromField.ref, { type: `val`, value: remainingValues[0] }], - } as BasicExpression - } - - return { - type: `func`, - name: `in`, - args: [fromField.ref, { type: `val`, value: remainingValues }], - } as BasicExpression - } - - // Handle IN minus equality: status IN [A,B,C] - status = B = status IN [A,C] - if (fromPred.name === `in` && subtractPred.name === `eq`) { - const fromInField = fromField as InField - const subtractValue = (subtractField as { ref: PropRef; value: any }).value - - const remainingValues = fromInField.values.filter( - (v) => !areValuesEqual(v, subtractValue), - ) - - if (remainingValues.length === 0) { - return { type: `val`, value: false } as BasicExpression - } - - if (remainingValues.length === 1) { - return { - type: `func`, - name: `eq`, - args: [fromField.ref, { type: `val`, value: remainingValues[0] }], - } as BasicExpression - } - - return { - type: `func`, - name: `in`, - args: [fromField.ref, { type: `val`, value: remainingValues }], - } as BasicExpression - } - - // Handle equality minus equality: age = 15 - age = 15 = empty, age = 15 - age = 20 = age = 15 - if (fromPred.name === `eq` && subtractPred.name === `eq`) { - const fromValue = (fromField as { ref: PropRef; value: any }).value - const subtractValue = (subtractField as { ref: PropRef; value: any }).value - - if (areValuesEqual(fromValue, subtractValue)) { - return { type: `val`, value: false } as BasicExpression - } - - // No overlap - return original - return fromPred as BasicExpression - } - - // Handle range minus range: age > 10 - age > 20 = age > 10 AND age <= 20 - const fromComp = extractComparisonField(fromPred) - const subtractComp = extractComparisonField(subtractPred) - - if ( - fromComp && - subtractComp && - areRefsEqual(fromComp.ref, subtractComp.ref) - ) { - // Try to compute the difference using range logic - const result = minusRangePredicates( - fromPred, - fromComp.value, - subtractPred, - subtractComp.value, - ) - return result - } - - // Can't simplify - return null -} - -/** - * Helper to compute difference between range predicates - */ -function minusRangePredicates( - fromFunc: Func, - fromValue: any, - subtractFunc: Func, - subtractValue: any, -): BasicExpression | null { - const fromOp = fromFunc.name as `gt` | `gte` | `lt` | `lte` | `eq` - const subtractOp = subtractFunc.name as `gt` | `gte` | `lt` | `lte` | `eq` - const ref = (extractComparisonField(fromFunc) || - extractEqualityField(fromFunc))!.ref - - // age > 10 - age > 20 = (age > 10 AND age <= 20) - if (fromOp === `gt` && subtractOp === `gt`) { - if (fromValue < subtractValue) { - // Result is: fromValue < field <= subtractValue - return { - type: `func`, - name: `and`, - args: [ - fromFunc as BasicExpression, - { - type: `func`, - name: `lte`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - ], - } as BasicExpression - } - // fromValue >= subtractValue means no overlap - return fromFunc as BasicExpression - } - - // age >= 10 - age >= 20 = (age >= 10 AND age < 20) - if (fromOp === `gte` && subtractOp === `gte`) { - if (fromValue < subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - fromFunc as BasicExpression, - { - type: `func`, - name: `lt`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age > 10 - age >= 20 = (age > 10 AND age < 20) - if (fromOp === `gt` && subtractOp === `gte`) { - if (fromValue < subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - fromFunc as BasicExpression, - { - type: `func`, - name: `lt`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age >= 10 - age > 20 = (age >= 10 AND age <= 20) - if (fromOp === `gte` && subtractOp === `gt`) { - if (fromValue <= subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - fromFunc as BasicExpression, - { - type: `func`, - name: `lte`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age < 30 - age < 20 = (age >= 20 AND age < 30) - if (fromOp === `lt` && subtractOp === `lt`) { - if (fromValue > subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - { - type: `func`, - name: `gte`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - fromFunc as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age <= 30 - age <= 20 = (age > 20 AND age <= 30) - if (fromOp === `lte` && subtractOp === `lte`) { - if (fromValue > subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - { - type: `func`, - name: `gt`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - fromFunc as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age < 30 - age <= 20 = (age > 20 AND age < 30) - if (fromOp === `lt` && subtractOp === `lte`) { - if (fromValue > subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - { - type: `func`, - name: `gt`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - fromFunc as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // age <= 30 - age < 20 = (age >= 20 AND age <= 30) - if (fromOp === `lte` && subtractOp === `lt`) { - if (fromValue >= subtractValue) { - return { - type: `func`, - name: `and`, - args: [ - { - type: `func`, - name: `gte`, - args: [ref, { type: `val`, value: subtractValue }], - } as BasicExpression, - fromFunc as BasicExpression, - ], - } as BasicExpression - } - return fromFunc as BasicExpression - } - - // Can't simplify other combinations - return null -} - -/** - * Check if one orderBy clause is a subset of another. - * Returns true if the subset ordering requirements are satisfied by the superset ordering. - * - * @example - * // Subset is prefix of superset - * isOrderBySubset([{expr: age, asc}], [{expr: age, asc}, {expr: name, desc}]) // true - * - * @param subset - The ordering requirements to check - * @param superset - The ordering that might satisfy the requirements - * @returns true if subset is satisfied by superset - */ -export function isOrderBySubset( - subset: OrderBy | undefined, - superset: OrderBy | undefined, -): boolean { - // No ordering requirement is always satisfied - if (!subset || subset.length === 0) { - return true - } - - // If there's no superset ordering but subset requires ordering, not satisfied - if (!superset || superset.length === 0) { - return false - } - - // Check if subset is a prefix of superset with matching expressions and compare options - if (subset.length > superset.length) { - return false - } - - for (let i = 0; i < subset.length; i++) { - const subClause = subset[i]! - const superClause = superset[i]! - - // Check if expressions match - if (!areExpressionsEqual(subClause.expression, superClause.expression)) { - return false - } - - // Check if compare options match - if ( - !areCompareOptionsEqual( - subClause.compareOptions, - superClause.compareOptions, - ) - ) { - return false - } - } - - return true -} - -/** - * Check if one limit is a subset of another. - * Returns true if the subset limit requirements are satisfied by the superset limit. - * - * Note: This function does NOT consider offset. For offset-aware subset checking, - * use `isOffsetLimitSubset` instead. - * - * @example - * isLimitSubset(10, 20) // true (requesting 10 items when 20 are available) - * isLimitSubset(20, 10) // false (requesting 20 items when only 10 are available) - * isLimitSubset(10, undefined) // true (requesting 10 items when unlimited are available) - * - * @param subset - The limit requirement to check - * @param superset - The limit that might satisfy the requirement - * @returns true if subset is satisfied by superset - */ -export function isLimitSubset( - subset: number | undefined, - superset: number | undefined, -): boolean { - // Unlimited superset satisfies any limit requirement - if (superset === undefined) { - return true - } - - // If requesting all data (no limit), we need unlimited data to satisfy it - // But we know superset is not unlimited so we return false - if (subset === undefined) { - return false - } - - // Otherwise, subset must be less than or equal to superset - return subset <= superset -} - -/** - * Check if one offset+limit range is a subset of another. - * Returns true if the subset range is fully contained within the superset range. - * - * A query with `{limit: 10, offset: 0}` loads rows [0, 10). - * A query with `{limit: 10, offset: 20}` loads rows [20, 30). - * - * For subset to be satisfied by superset: - * - Superset must start at or before subset (superset.offset <= subset.offset) - * - Superset must end at or after subset (superset.offset + superset.limit >= subset.offset + subset.limit) - * - * @example - * isOffsetLimitSubset({ offset: 0, limit: 5 }, { offset: 0, limit: 10 }) // true - * isOffsetLimitSubset({ offset: 5, limit: 5 }, { offset: 0, limit: 10 }) // true (rows 5-9 within 0-9) - * isOffsetLimitSubset({ offset: 5, limit: 10 }, { offset: 0, limit: 10 }) // false (rows 5-14 exceed 0-9) - * isOffsetLimitSubset({ offset: 20, limit: 10 }, { offset: 0, limit: 10 }) // false (rows 20-29 outside 0-9) - * - * @param subset - The offset+limit requirements to check - * @param superset - The offset+limit that might satisfy the requirements - * @returns true if subset range is fully contained within superset range - */ -export function isOffsetLimitSubset( - subset: { offset?: number; limit?: number }, - superset: { offset?: number; limit?: number }, -): boolean { - const subsetOffset = subset.offset ?? 0 - const supersetOffset = superset.offset ?? 0 - - // Superset must start at or before subset - if (supersetOffset > subsetOffset) { - return false - } - - // If superset is unlimited, it covers everything from its offset onwards - if (superset.limit === undefined) { - return true - } - - // If subset is unlimited but superset has a limit, subset can't be satisfied - if (subset.limit === undefined) { - return false - } - - // Both have limits - check if subset range is within superset range - const subsetEnd = subsetOffset + subset.limit - const supersetEnd = supersetOffset + superset.limit - - return subsetEnd <= supersetEnd -} - -/** - * Check if one predicate (where + orderBy + limit + offset) is a subset of another. - * Returns true if all aspects of the subset predicate are satisfied by the superset. - * - * @example - * isPredicateSubset( - * { where: gt(ref('age'), val(20)), limit: 10 }, - * { where: gt(ref('age'), val(10)), limit: 20 } - * ) // true - * - * @param subset - The predicate requirements to check - * @param superset - The predicate that might satisfy the requirements - * @returns true if subset is satisfied by superset - */ -export function isPredicateSubset( - subset: LoadSubsetOptions, - superset: LoadSubsetOptions, -): boolean { - // When the superset has a limit, we can only determine subset relationship - // if the where clauses are equal (not just subset relationship). - // - // This is because a limited query only loads a portion of the matching rows. - // A more restrictive where clause might require rows outside that portion. - // - // Example: superset = {where: undefined, limit: 10, orderBy: desc} - // subset = {where: LIKE 'search%', limit: 10, orderBy: desc} - // The top 10 items matching 'search%' might include items outside the overall top 10. - // - // However, if the where clauses are equal, then the subset relationship can - // be determined by orderBy, limit, and offset: - // Example: superset = {where: status='active', limit: 10, offset: 0, orderBy: desc} - // subset = {where: status='active', limit: 5, offset: 0, orderBy: desc} - // The top 5 active items ARE contained in the top 10 active items. - if (superset.limit !== undefined || superset.cursor !== undefined) { - // A cursor page only covers another request for the same page, whether or - // not the adapter also uses a numeric limit. - // Adapters may use the cursor expressions instead of offset, so matching - // offsets alone do not prove that two requests load the same rows. - if (!areCursorExpressionsEqual(subset.cursor, superset.cursor)) { - return false - } - } - - // A cursor-relative request is also a finite window. Even when it has no - // numeric limit, a different predicate can select rows outside that window. - if ( - superset.cursor !== undefined && - !areWhereClausesEqual(subset.where, superset.where) - ) { - return false - } - - if (superset.limit !== undefined) { - // For limited supersets, where clauses must be equal - if (!areWhereClausesEqual(subset.where, superset.where)) { - return false - } - return ( - isOrderBySubset(subset.orderBy, superset.orderBy) && - isOffsetLimitSubset(subset, superset) - ) - } - - // For unlimited supersets, use the normal subset logic - // Still need to consider offset - an unlimited query with offset only covers - // rows from that offset onwards - return ( - isWhereSubset(subset.where, superset.where) && - isOrderBySubset(subset.orderBy, superset.orderBy) && - isOffsetLimitSubset(subset, superset) - ) -} - -/** - * Returns whether one acquisition request subsumes another demand. - * - * This is a directional relationship between request shapes, not proof of - * applied or authoritative coverage. It must not be replaced with DemandKey - * equality, which answers whether two exact requests are the same. - */ -export function isLoadSubsetRequestSubsumedBy( - demand: LoadSubsetOptions, - acquisitionRequest: LoadSubsetOptions, -): boolean { - return isPredicateSubset(demand, acquisitionRequest) -} - -function areCursorExpressionsEqual( - a: LoadSubsetOptions[`cursor`], - b: LoadSubsetOptions[`cursor`], -): boolean { - if (a === undefined || b === undefined) return a === b - return ( - Object.is(a.lastKey, b.lastKey) && - areExpressionsEqual(a.whereFrom, b.whereFrom) && - areExpressionsEqual(a.whereCurrent, b.whereCurrent) - ) -} - -/** - * Check if two where clauses are structurally equal. - * Used for limited query subset checks where subset relationship isn't sufficient. - */ -function areWhereClausesEqual( - a: BasicExpression | undefined, - b: BasicExpression | undefined, -): boolean { - if (a === undefined && b === undefined) { - return true - } - if (a === undefined || b === undefined) { - return false - } - return areExpressionsEqual(a, b) -} - -// ============================================================================ -// Helper functions -// ============================================================================ - -/** - * Find common conditions between two predicates. - * Returns an array of conditions that appear in both predicates. - */ -function findCommonConditions( - predicate1: BasicExpression, - predicate2: BasicExpression, -): Array> { - const conditions1 = extractAllConditions(predicate1) - const conditions2 = extractAllConditions(predicate2) - - const common: Array> = [] - - for (const cond1 of conditions1) { - for (const cond2 of conditions2) { - if (areExpressionsEqual(cond1, cond2)) { - // Avoid duplicates - if (!common.some((c) => areExpressionsEqual(c, cond1))) { - common.push(cond1) - } - break - } - } - } - - return common -} - -/** - * Extract all individual conditions from a predicate, flattening AND operations. - */ -function extractAllConditions( - predicate: BasicExpression, -): Array> { - if (predicate.type === `func` && predicate.name === `and`) { - const conditions: Array> = [] - for (const arg of predicate.args) { - conditions.push(...extractAllConditions(arg as BasicExpression)) - } - return conditions - } - - return [predicate] -} - -/** - * Remove specified conditions from a predicate. - * Returns the predicate with the specified conditions removed, or undefined if all conditions are removed. - */ -function removeConditions( - predicate: BasicExpression, - conditionsToRemove: Array>, -): BasicExpression | undefined { - if (predicate.type === `func` && predicate.name === `and`) { - const remainingArgs = predicate.args.filter( - (arg) => - !conditionsToRemove.some((cond) => - areExpressionsEqual(arg as BasicExpression, cond), - ), - ) - - if (remainingArgs.length === 0) { - return undefined - } else if (remainingArgs.length === 1) { - return remainingArgs[0]! - } else { - return { - type: `func`, - name: `and`, - args: remainingArgs, - } as BasicExpression - } - } - - // For non-AND predicates, don't remove anything - return predicate -} - -/** - * Combine multiple conditions into a single predicate using AND logic. - * Flattens nested AND operations to avoid unnecessary nesting. - */ -function combineConditions( - conditions: Array>, -): BasicExpression { - if (conditions.length === 0) { - return { type: `val`, value: true } as BasicExpression - } else if (conditions.length === 1) { - return conditions[0]! - } else { - // Flatten all conditions, including those that are already AND operations - const flattenedConditions: Array> = [] - - for (const condition of conditions) { - if (condition.type === `func` && condition.name === `and`) { - // Flatten nested AND operations - flattenedConditions.push(...condition.args) - } else { - flattenedConditions.push(condition) - } - } - - if (flattenedConditions.length === 1) { - return flattenedConditions[0]! - } else { - return { - type: `func`, - name: `and`, - args: flattenedConditions, - } as BasicExpression - } - } -} - -/** - * Find a predicate with a specific operator and value - */ -function findPredicateWithOperator( - predicates: Array>, - operator: string, - value: any, -): BasicExpression | undefined { - return predicates.find((p) => { - if (p.type === `func`) { - const f = p as Func - const field = extractComparisonField(f) - return f.name === operator && field && areValuesEqual(field.value, value) - } - return false - }) -} - -const unhashableExpression = Symbol(`unhashableExpression`) -type ExpressionHashCache = WeakMap< - BasicExpression, - string | typeof unhashableExpression -> - -function getCachedExpressionHash( - expression: BasicExpression, - expressionHashes: ExpressionHashCache, -): string | typeof unhashableExpression { - const cachedHash = expressionHashes.get(expression) - if (cachedHash !== undefined) return cachedHash - - try { - const hash = getStableExpressionHash(expression) - expressionHashes.set(expression, hash) - return hash - } catch (error) { - if (!(error instanceof UnhashableQueryIRError)) throw error - expressionHashes.set(expression, unhashableExpression) - return unhashableExpression - } -} - -function areExpressionsEqual( - a: BasicExpression, - b: BasicExpression, - expressionHashes: ExpressionHashCache = new WeakMap(), -): boolean { - const aHash = getCachedExpressionHash(a, expressionHashes) - const bHash = getCachedExpressionHash(b, expressionHashes) - if (aHash === unhashableExpression || bHash === unhashableExpression) { - return areExpressionsStructurallyEqual(a, b) - } - return aHash === bHash -} - -function areExpressionsStructurallyEqual( - a: BasicExpression, - b: BasicExpression, -): boolean { - if (a.type !== b.type) return false - if (a.type === `val` && b.type === `val`) { - return areValuesEqual(a.value, b.value) - } - if (a.type === `ref` && b.type === `ref`) { - return areRefsEqual(a, b) - } - if (a.type === `func` && b.type === `func`) { - return ( - a.name === b.name && - a.args.length === b.args.length && - a.args.every((arg, index) => - areExpressionsStructurallyEqual(arg, b.args[index]!), - ) - ) - } - return false -} - -function areValuesEqual(a: any, b: any): boolean { - // Simple equality check - could be enhanced for deep object comparison - if (a === b) { - return true - } - - // Handle NaN - if (typeof a === `number` && typeof b === `number` && isNaN(a) && isNaN(b)) { - return true - } - - // Handle Date objects - if (a instanceof Date && b instanceof Date) { - return a.getTime() === b.getTime() - } - - // For arrays and objects, use reference equality - // (In practice, we don't need deep equality for these cases - - // same object reference means same value for our use case) - if ( - typeof a === `object` && - typeof b === `object` && - a !== null && - b !== null - ) { - return a === b - } - - return false -} - -function areRefsEqual(a: PropRef, b: PropRef): boolean { - if (a.path.length !== b.path.length) { - return false - } - return a.path.every((segment, i) => segment === b.path[i]) -} - -/** - * Check if a value is a primitive (string, number, boolean, null, undefined) - * Primitives can use Set for fast lookups - */ -function isPrimitive(value: any): boolean { - return ( - value === null || - value === undefined || - typeof value === `string` || - typeof value === `number` || - typeof value === `boolean` - ) -} - -/** - * Check if all values in an array are primitives - */ -function areAllPrimitives(values: Array): boolean { - return values.every(isPrimitive) -} - -/** - * Check if a value is in an array, with optional pre-built Set for optimization. - * The primitiveSet is cached in InField during extraction and reused for all lookups. - */ -function arrayIncludesWithSet( - array: Array, - value: any, - primitiveSet: Set | null, - arrayIsAllPrimitives?: boolean, -): boolean { - // Fast path: use pre-built Set for O(1) lookup - if (primitiveSet) { - // Skip isPrimitive check if we know the value must be primitive for a match - // (if array is all primitives, only primitives can match) - if (arrayIsAllPrimitives || isPrimitive(value)) { - return primitiveSet.has(value) - } - return false // Non-primitive can't be in primitive-only set - } - - // Fallback: use areValuesEqual for Dates and objects - return array.some((v) => areValuesEqual(v, value)) -} - -/** - * Get the maximum of two values, handling both numbers and Dates - */ -function maxValue(a: any, b: any): any { - if (a instanceof Date && b instanceof Date) { - return a.getTime() > b.getTime() ? a : b - } - return Math.max(a, b) -} - -/** - * Get the minimum of two values, handling both numbers and Dates - */ -function minValue(a: any, b: any): any { - if (a instanceof Date && b instanceof Date) { - return a.getTime() < b.getTime() ? a : b - } - return Math.min(a, b) -} - -function areCompareOptionsEqual(a: CompareOptions, b: CompareOptions): boolean { - if ( - a.direction !== b.direction || - a.nulls !== b.nulls || - a.stringSort !== b.stringSort - ) { - return false - } - - if (a.stringSort !== `locale` || b.stringSort !== `locale`) { - return true - } - - if (a.locale !== b.locale) return false - if (Object.is(a.localeOptions, b.localeOptions)) return true - - try { - return ( - getStableValueHash(a.localeOptions) === - getStableValueHash(b.localeOptions) - ) - } catch (error) { - if (!(error instanceof UnhashableQueryIRError)) throw error - return false - } -} - -interface ComparisonField { - ref: PropRef - value: any -} - -function extractComparisonField(func: Func): ComparisonField | null { - // Handle comparison operators: eq, gt, gte, lt, lte - if ([`eq`, `gt`, `gte`, `lt`, `lte`].includes(func.name)) { - // Assume first arg is ref, second is value - const firstArg = func.args[0] - const secondArg = func.args[1] - - if (firstArg?.type === `ref` && secondArg?.type === `val`) { - return { - ref: firstArg, - value: secondArg.value, - } - } - } - - return null -} - -function extractEqualityField(func: Func): ComparisonField | null { - if (func.name === `eq`) { - const firstArg = func.args[0] - const secondArg = func.args[1] - - if (firstArg?.type === `ref` && secondArg?.type === `val`) { - return { - ref: firstArg, - value: secondArg.value, - } - } - } - return null -} - -interface InField { - ref: PropRef - values: Array - // Cached optimization data (computed once, reused many times) - areAllPrimitives?: boolean - primitiveSet?: Set | null -} - -function extractInField(func: Func): InField | null { - if (func.name === `in`) { - const firstArg = func.args[0] - const secondArg = func.args[1] - - if ( - firstArg?.type === `ref` && - secondArg?.type === `val` && - Array.isArray(secondArg.value) - ) { - let values = secondArg.value - // Precompute optimization metadata once - const allPrimitives = areAllPrimitives(values) - let primitiveSet: Set | null = null - - if (allPrimitives && values.length > 10) { - // Build Set and dedupe values at the same time - primitiveSet = new Set(values) - // If we found duplicates, use the deduped array going forward - if (primitiveSet.size < values.length) { - values = Array.from(primitiveSet) - } - } - - return { - ref: firstArg, - values, - areAllPrimitives: allPrimitives, - primitiveSet, - } - } - } - return null -} - -function isComparisonSubset( - subsetFunc: Func, - subsetValue: any, - supersetFunc: Func, - supersetValue: any, -): boolean { - const subOp = subsetFunc.name - const superOp = supersetFunc.name - - // Handle same operator - if (subOp === superOp) { - if (subOp === `eq`) { - // field = X is subset of field = X only - // Fast path: primitives can use strict equality - if (isPrimitive(subsetValue) && isPrimitive(supersetValue)) { - return subsetValue === supersetValue - } - return areValuesEqual(subsetValue, supersetValue) - } else if (subOp === `gt`) { - // field > 20 is subset of field > 10 if 20 > 10 - return subsetValue >= supersetValue - } else if (subOp === `gte`) { - // field >= 20 is subset of field >= 10 if 20 >= 10 - return subsetValue >= supersetValue - } else if (subOp === `lt`) { - // field < 10 is subset of field < 20 if 10 <= 20 - return subsetValue <= supersetValue - } else if (subOp === `lte`) { - // field <= 10 is subset of field <= 20 if 10 <= 20 - return subsetValue <= supersetValue - } - } - - // Handle different operators on same field - // eq vs gt/gte: field = 15 is subset of field > 10 if 15 > 10 - if (subOp === `eq` && superOp === `gt`) { - return subsetValue > supersetValue - } - if (subOp === `eq` && superOp === `gte`) { - return subsetValue >= supersetValue - } - if (subOp === `eq` && superOp === `lt`) { - return subsetValue < supersetValue - } - if (subOp === `eq` && superOp === `lte`) { - return subsetValue <= supersetValue - } - - // gt/gte vs gte/gt - if (subOp === `gt` && superOp === `gte`) { - // field > 10 is subset of field >= 10 if 10 >= 10 (always true for same value) - return subsetValue >= supersetValue - } - if (subOp === `gte` && superOp === `gt`) { - // field >= 11 is subset of field > 10 if 11 > 10 - return subsetValue > supersetValue - } - - // lt/lte vs lte/lt - if (subOp === `lt` && superOp === `lte`) { - // field < 10 is subset of field <= 10 if 10 <= 10 - return subsetValue <= supersetValue - } - if (subOp === `lte` && superOp === `lt`) { - // field <= 9 is subset of field < 10 if 9 < 10 - return subsetValue < supersetValue - } - - return false -} - -function groupPredicatesByField( - predicates: Array>, -): Map>> { - const groups = new Map>>() - - for (const pred of predicates) { - let fieldKey: string | null = null - - if (pred.type === `func`) { - const func = pred as Func - const field = - extractComparisonField(func) || - extractEqualityField(func) || - extractInField(func) - if (field) { - fieldKey = field.ref.path.join(`.`) - } - } - - const group = groups.get(fieldKey) || [] - group.push(pred) - groups.set(fieldKey, group) - } - - return groups -} - -function unionSameFieldPredicates( - predicates: Array>, -): BasicExpression | null { - if (predicates.length === 1) { - return predicates[0]! - } - - // Try to extract range constraints - let maxGt: number | null = null - let maxGte: number | null = null - let minLt: number | null = null - let minLte: number | null = null - const eqValues: Set = new Set() - const inValues: Set = new Set() - const otherPredicates: Array> = [] - - for (const pred of predicates) { - if (pred.type === `func`) { - const func = pred as Func - const field = extractComparisonField(func) - - if (field) { - const value = field.value - if (func.name === `gt`) { - maxGt = maxGt === null ? value : minValue(maxGt, value) - } else if (func.name === `gte`) { - maxGte = maxGte === null ? value : minValue(maxGte, value) - } else if (func.name === `lt`) { - minLt = minLt === null ? value : maxValue(minLt, value) - } else if (func.name === `lte`) { - minLte = minLte === null ? value : maxValue(minLte, value) - } else if (func.name === `eq`) { - eqValues.add(value) - } else { - otherPredicates.push(pred) - } - } else { - const inField = extractInField(func) - if (inField) { - for (const val of inField.values) { - inValues.add(val) - } - } else { - otherPredicates.push(pred) - } - } - } else { - otherPredicates.push(pred) - } - } - - // If we have multiple equality values, combine into IN - if (eqValues.size > 1 || (eqValues.size > 0 && inValues.size > 0)) { - const allValues = [...eqValues, ...inValues] - const ref = predicates.find((p) => { - if (p.type === `func`) { - const field = - extractComparisonField(p as Func) || extractInField(p as Func) - return field !== null - } - return false - }) - - if (ref && ref.type === `func`) { - const field = - extractComparisonField(ref as Func) || extractInField(ref as Func) - if (field) { - return { - type: `func`, - name: `in`, - args: [ - field.ref, - { type: `val`, value: allValues } as BasicExpression, - ], - } as BasicExpression - } - } - } - - // Build the least restrictive range - const result: Array> = [] - - // Choose the least restrictive lower bound - if (maxGt !== null && maxGte !== null) { - // Take the smaller one (less restrictive) - const pred = - maxGte <= maxGt - ? findPredicateWithOperator(predicates, `gte`, maxGte) - : findPredicateWithOperator(predicates, `gt`, maxGt) - if (pred) result.push(pred) - } else if (maxGt !== null) { - const pred = findPredicateWithOperator(predicates, `gt`, maxGt) - if (pred) result.push(pred) - } else if (maxGte !== null) { - const pred = findPredicateWithOperator(predicates, `gte`, maxGte) - if (pred) result.push(pred) - } - - // Choose the least restrictive upper bound - if (minLt !== null && minLte !== null) { - const pred = - minLte >= minLt - ? findPredicateWithOperator(predicates, `lte`, minLte) - : findPredicateWithOperator(predicates, `lt`, minLt) - if (pred) result.push(pred) - } else if (minLt !== null) { - const pred = findPredicateWithOperator(predicates, `lt`, minLt) - if (pred) result.push(pred) - } else if (minLte !== null) { - const pred = findPredicateWithOperator(predicates, `lte`, minLte) - if (pred) result.push(pred) - } - - // Add single eq value - if (eqValues.size === 1 && inValues.size === 0) { - const pred = findPredicateWithOperator(predicates, `eq`, [...eqValues][0]) - if (pred) result.push(pred) - } - - // Add IN if only IN values - if (eqValues.size === 0 && inValues.size > 0) { - result.push( - predicates.find((p) => { - if (p.type === `func`) { - return (p as Func).name === `in` - } - return false - })!, - ) - } - - // Add other predicates - result.push(...otherPredicates) - - if (result.length === 0) { - return { type: `val`, value: true } as BasicExpression - } - - if (result.length === 1) { - return result[0]! - } - - return { - type: `func`, - name: `or`, - args: result, - } as BasicExpression -} diff --git a/packages/db/src/query/runtime-reference-identity.ts b/packages/db/src/query/runtime-reference-identity.ts index e41aeaf67a..03162dab29 100644 --- a/packages/db/src/query/runtime-reference-identity.ts +++ b/packages/db/src/query/runtime-reference-identity.ts @@ -4,29 +4,75 @@ export type RuntimeReferenceIdentity = [ sequence: number, ] +type ReferenceIdStore = { + get: (key: TKey) => number | undefined + set: (key: TKey, value: number) => unknown +} + export function createRuntimeReferenceIdentityFactory(): ( - value: object, + value: object | symbol, ) => RuntimeReferenceIdentity { - const namespace = createRuntimeReferenceNamespace() const referenceIds = new WeakMap() + let localSymbolIds: ReferenceIdStore | undefined + let registeredSymbolIds: Map | undefined + let namespace: string | undefined let sequence = 0 - return (value) => { - let referenceId = referenceIds.get(value) + const getReferenceId = ( + ids: ReferenceIdStore, + key: TKey, + ): number => { + let referenceId = ids.get(key) if (referenceId === undefined) { referenceId = ++sequence - referenceIds.set(value, referenceId) + ids.set(key, referenceId) + } + return referenceId + } + + return (value) => { + namespace ??= createRuntimeReferenceNamespace() + let referenceId: number + if (typeof value === `symbol`) { + const registeredKey = Symbol.keyFor(value) + if (registeredKey === undefined) { + localSymbolIds ??= createLocalSymbolIdStore() + referenceId = getReferenceId(localSymbolIds, value) + } else { + registeredSymbolIds ??= new Map() + referenceId = getReferenceId(registeredSymbolIds, registeredKey) + } + } else { + referenceId = getReferenceId(referenceIds, value) } return [`runtimeReference`, namespace, referenceId] } } +function createLocalSymbolIdStore(): ReferenceIdStore { + const weakIds = new WeakMap< + object, + number + >() as unknown as ReferenceIdStore + const probe = Symbol() + + try { + weakIds.set(probe, 0) + if (weakIds.get(probe) === 0) return weakIds + } catch { + // Older runtimes reject symbols as weak keys. Retain them rather than + // collapse distinct symbols and corrupt equality. + } + + return new Map() +} + let runtimeReferenceIdentityFactory: | ReturnType | undefined export function getRuntimeReferenceIdentity( - value: object, + value: object | symbol, ): RuntimeReferenceIdentity { runtimeReferenceIdentityFactory ??= createRuntimeReferenceIdentityFactory() diff --git a/packages/db/src/query/subset-dedupe.ts b/packages/db/src/query/subset-dedupe.ts index f5b98b2531..8b819888c5 100644 --- a/packages/db/src/query/subset-dedupe.ts +++ b/packages/db/src/query/subset-dedupe.ts @@ -1,418 +1,69 @@ -import { - isLoadSubsetRequestSubsumedBy, - isWhereSubset, - minusWherePredicates, - unionWherePredicates, -} from './predicate-utils.js' -import { Func, PropRef, Value } from './ir.js' -import type { BasicExpression } from './ir.js' -import type { LoadSubsetOptions } from '../types.js' - -type SharedAbortLease = { - signal: AbortSignal | undefined - aborted: boolean - attach: (signal: AbortSignal | undefined) => void - dispose: () => void -} - -type InflightCall = { - options: LoadSubsetOptions - promise: Promise - lease: SharedAbortLease -} +import { getLoadSubsetDemandKey } from './ir-stable-identity.js' +import type { LoadSubsetFn, LoadSubsetOptions } from '../types.js' /** - * Deduplicated wrapper for a loadSubset function. - * Tracks what data has been loaded and avoids redundant calls by applying - * subset logic to predicates. - * - * @param opts - The options for the DeduplicatedLoadSubset - * @param opts.loadSubset - The underlying loadSubset function to wrap - * @param opts.onDeduplicate - An optional callback function that is invoked when a loadSubset call is deduplicated. - * If the call is deduplicated because the requested data is being loaded by an inflight request, - * then this callback is invoked when the inflight request completes successfully and the data is fully loaded. - * This callback is useful if you need to track rows per query, in which case you can't ignore deduplicated calls - * because you need to know which rows were loaded for each query. - * @example - * const dedupe = new DeduplicatedLoadSubset({ loadSubset: myLoadSubset, onDeduplicate: (opts) => console.log(`Call was deduplicated:`, opts) }) - * - * // First call - fetches data - * await dedupe.loadSubset({ where: gt(ref('age'), val(10)) }) - * - * // Second call - subset of first, returns true immediately - * await dedupe.loadSubset({ where: gt(ref('age'), val(20)) }) - * - * // Clear state to start fresh - * dedupe.reset() + * Deduplicates exact canonical demands without inferring broader coverage. + * Requests follow the immutable LoadSubsetOptions contract; no copies are made. */ export class DeduplicatedLoadSubset { - // The underlying loadSubset function to wrap - private readonly _loadSubset: ( - options: LoadSubsetOptions, - ) => true | Promise - - // An optional callback function that is invoked when a loadSubset call is deduplicated. - private readonly onDeduplicate: - | ((options: LoadSubsetOptions) => void) - | undefined - - // Combined where predicate for all unlimited calls (no limit) - private unlimitedWhere: BasicExpression | undefined = undefined - - // Flag to track if we've loaded all data (unlimited call with no where clause) - private hasLoadedAllData = false - - // List of calls with a finite or cursor-relative result window. - // We clone options before storing to prevent mutation of stored predicates - private limitedCalls: Array = [] - - // Track in-flight calls to prevent concurrent duplicate requests - // Each entry also owns the shared cancellation lease for its requesters. - private inflightCalls: Array = [] - - // Generation counter to invalidate in-flight requests after reset() - // When reset() is called, this increments, and any in-flight completion handlers - // check if their captured generation matches before updating tracking state + private readonly completed = new Set() + private readonly inflight = new Map>() private generation = 0 - constructor(opts: { - loadSubset: (options: LoadSubsetOptions) => true | Promise - onDeduplicate?: (options: LoadSubsetOptions) => void - }) { - this._loadSubset = opts.loadSubset - this.onDeduplicate = opts.onDeduplicate - } + constructor( + private readonly options: { + loadSubset: LoadSubsetFn + onDeduplicate?: (options: LoadSubsetOptions) => void + }, + ) {} - /** - * Load a subset of data, with automatic deduplication based on previously - * loaded predicates and in-flight requests. - * - * This method is auto-bound, so it can be safely passed as a callback without - * losing its `this` context (e.g., `loadSubset: dedupe.loadSubset` in a sync config). - * - * @param options - The predicate options (where, orderBy, limit) - * @returns true if data is already loaded, or a Promise that resolves when data is loaded - */ loadSubset = (options: LoadSubsetOptions): true | Promise => { - // If we've loaded all data, everything is covered - if (this.hasLoadedAllData) { - this.onDeduplicate?.(options) + const key = getLoadSubsetDemandKey(options) + if (this.completed.has(key)) { + this.options.onDeduplicate?.(options) return true } - // Check against unlimited combined predicate - // If we've loaded all data matching a where clause, we don't need to refetch subsets - if (this.unlimitedWhere !== undefined && options.where !== undefined) { - if (isWhereSubset(options.where, this.unlimitedWhere)) { - this.onDeduplicate?.(options) - return true // Data already loaded via unlimited call - } + // Requests with independent cancellation own independent transports. + // Unabortable requests can share without an ownership protocol. + const existing = options.signal ? undefined : this.inflight.get(key) + if (existing) { + // Observer failures must not reject a detached promise after success. + void existing + .then(() => this.options.onDeduplicate?.(options)) + .catch(() => {}) + return existing } - // Check against limited calls - if (options.limit !== undefined || options.cursor !== undefined) { - const alreadyLoaded = this.limitedCalls.some((loaded) => - isLoadSubsetRequestSubsumedBy(options, loaded), - ) + const generation = this.generation + const result = this.options.loadSubset(options) - if (alreadyLoaded) { - this.onDeduplicate?.(options) - return true // Already loaded + if (result === true) { + if (generation === this.generation && !options.signal?.aborted) { + this.completed.add(key) } - } - - // Check against in-flight calls using the same subset logic as resolved calls - // This prevents duplicate requests when concurrent calls have subset relationships - const matchingInflight = this.inflightCalls.find( - (inflight) => - !inflight.lease.aborted && - isLoadSubsetRequestSubsumedBy(options, inflight.options), - ) - - if (matchingInflight !== undefined) { - matchingInflight.lease.attach(options.signal) - // An in-flight call will load data that covers this request - // Return the same promise so this caller waits for the data to load - // The in-flight promise already handles tracking updates when it completes - const prom = matchingInflight.promise - // Call `onDeduplicate` when the inflight request has loaded the data - void prom - .then(() => this.onDeduplicate?.(options)) - .catch(() => { - // The original caller owns the transport failure. This observer only - // waits to publish successful deduplication. - }) - return prom - } - - // Preserve the original request for tracking and in-flight dedupe, but allow - // the backend request to be narrowed to only the missing subset. - const lease = createSharedAbortLease(options.signal) - const trackingOptions = cloneOptions({ ...options, signal: lease.signal }) - const loadOptions = cloneOptions({ ...options, signal: lease.signal }) - if ( - this.unlimitedWhere !== undefined && - options.limit === undefined && - options.cursor === undefined - ) { - // Compute difference to get only the missing data - // We can only do this for unlimited queries - // and we can only remove data that was loaded from unlimited queries - // because with limited queries we have no way to express that we already loaded part of the matching data - loadOptions.where = - minusWherePredicates(loadOptions.where, this.unlimitedWhere) ?? - loadOptions.where - } - - // Call underlying loadSubset to load the missing data - let resultPromise: true | Promise - try { - resultPromise = this._loadSubset(loadOptions) - } catch (error) { - lease.dispose() - throw error - } - - // Handle both sync (true) and async (Promise) return values - if (resultPromise === true) { - if (!lease.aborted) this.updateTracking(trackingOptions) - lease.dispose() return true - } else { - // Async return - track the promise and update tracking after it resolves - - // Capture the current generation - this lets us detect if reset() was called - // while this request was in-flight, so we can skip updating tracking state - const capturedGeneration = this.generation - - // We need to create a reference to the in-flight entry so we can remove it later - const inflightEntry = { - options: trackingOptions, - lease, - promise: resultPromise - .then((result) => { - // Only update tracking if this request is still from the current generation - // If reset() was called, the generation will have incremented and we should - // not repopulate the state that was just cleared - if (capturedGeneration === this.generation && !lease.aborted) { - this.updateTracking(trackingOptions) - } - return result - }) - .finally(() => { - // Always remove from in-flight array on completion OR rejection - // This ensures failed requests can be retried instead of being cached forever - const index = this.inflightCalls.indexOf(inflightEntry) - if (index !== -1) { - this.inflightCalls.splice(index, 1) - } - lease.dispose() - }), - } + } - // Store the in-flight entry so concurrent subset calls can wait for it - this.inflightCalls.push(inflightEntry) - return inflightEntry.promise + const promise = result + .then((value) => { + if (generation === this.generation && !options.signal?.aborted) { + this.completed.add(key) + } + return value + }) + .finally(() => { + if (this.inflight.get(key) === promise) this.inflight.delete(key) + }) + if (!options.signal && generation === this.generation) { + this.inflight.set(key, promise) } + return promise } - /** - * Reset all tracking state. - * Clears the history of loaded predicates and in-flight calls. - * Use this when you want to start fresh, for example after clearing the underlying data store. - * - * Note: Any in-flight requests will still complete, but they will not update the tracking - * state after the reset. This prevents old requests from repopulating cleared state. - */ reset(): void { - this.unlimitedWhere = undefined - this.hasLoadedAllData = false - this.limitedCalls = [] - this.inflightCalls = [] - // Increment generation to invalidate any in-flight completion handlers - // This ensures requests that were started before reset() don't repopulate the state + this.completed.clear() + this.inflight.clear() this.generation++ } - - private updateTracking(options: LoadSubsetOptions): void { - // Update tracking based on whether this was a limited or unlimited call - if (options.limit === undefined && options.cursor === undefined) { - // Unlimited call - update combined where predicate - // We ignore orderBy for unlimited calls as mentioned in requirements - if (options.where === undefined) { - // No where clause = all data loaded - this.hasLoadedAllData = true - this.unlimitedWhere = undefined - this.limitedCalls = [] - this.inflightCalls = [] - } else if (this.unlimitedWhere === undefined) { - this.unlimitedWhere = options.where - } else { - this.unlimitedWhere = unionWherePredicates([ - this.unlimitedWhere, - options.where, - ]) - } - } else { - // Limited call - add to list for future subset checks - // Options are already cloned by caller to prevent mutation issues - this.limitedCalls.push(options) - } - } -} - -function createSharedAbortLease( - initialSignal: AbortSignal | undefined, -): SharedAbortLease { - const controller = initialSignal ? new AbortController() : undefined - const listeners = new Map void>() - let hasUnabortableOwner = initialSignal === undefined - let activeAbortableOwners = 0 - - const abortIfUnused = (reason?: unknown) => { - if ( - !hasUnabortableOwner && - listeners.size > 0 && - activeAbortableOwners === 0 - ) { - controller?.abort(reason) - } - } - - const attach = (signal: AbortSignal | undefined) => { - if (!signal) { - hasUnabortableOwner = true - return - } - if (listeners.has(signal)) return - - const onAbort = () => { - activeAbortableOwners -= 1 - abortIfUnused(signal.reason) - } - listeners.set(signal, onAbort) - if (signal.aborted) { - abortIfUnused(signal.reason) - } else { - activeAbortableOwners += 1 - signal.addEventListener(`abort`, onAbort, { once: true }) - } - } - - attach(initialSignal) - - return { - signal: controller?.signal, - get aborted() { - return controller?.signal.aborted ?? false - }, - attach, - dispose: () => { - for (const [signal, listener] of listeners) { - signal.removeEventListener(`abort`, listener) - } - listeners.clear() - }, - } -} - -/** - * Clones a LoadSubsetOptions object to prevent mutation of stored predicates. - * This is crucial because callers often reuse the same options object and mutate - * properties like limit or where between calls. Without cloning, our stored history - * would reflect the mutated values rather than what was actually loaded. - */ -export function cloneOptions(options: LoadSubsetOptions): LoadSubsetOptions { - return { - ...options, - where: options.where - ? cloneBasicExpression(options.where, `predicate`) - : undefined, - orderBy: options.orderBy?.map((clause) => ({ - ...clause, - expression: cloneBasicExpression(clause.expression), - compareOptions: { ...clause.compareOptions }, - })), - cursor: options.cursor - ? { - ...options.cursor, - whereFrom: cloneBasicExpression( - options.cursor.whereFrom, - `predicate`, - ), - whereCurrent: cloneBasicExpression( - options.cursor.whereCurrent, - `predicate`, - ), - } - : undefined, - } -} - -type ExpressionCloneContext = `exact` | `predicate` | `comparison` - -function cloneBasicExpression( - expression: BasicExpression, - context: ExpressionCloneContext = `exact`, -): BasicExpression { - switch (expression.type) { - case `ref`: - return new PropRef([...expression.path]) - case `val`: - return new Value( - context === `comparison` - ? snapshotComparisonValue(expression.value) - : expression.value, - ) - case `func`: - return new Func( - expression.name, - expression.args.map((arg, index) => { - if ( - context === `predicate` && - expression.name === `in` && - index === 1 && - arg.type === `val` && - Array.isArray(arg.value) - ) { - return new Value( - arg.value.map((value) => snapshotComparisonValue(value)), - ) - } - - const argumentContext = - context === `predicate` && isComparisonFunction(expression.name) - ? `comparison` - : context - return cloneBasicExpression(arg, argumentContext) - }), - ) - } -} - -function isComparisonFunction(name: string): boolean { - return ( - name === `eq` || - name === `gt` || - name === `gte` || - name === `lt` || - name === `lte` - ) -} - -function snapshotComparisonValue(value: T): T { - if (value instanceof Date) { - return new Date(value.getTime()) as T - } - - if (typeof Buffer !== `undefined` && value instanceof Buffer) { - return Buffer.from(value) as T - } - - if (value instanceof Uint8Array) { - return value.slice() as T - } - - // Other objects use reference equality in predicate identity and comparison. - return value } diff --git a/packages/db/src/scheduler.ts b/packages/db/src/scheduler.ts index d87ac05319..d5faa684e5 100644 --- a/packages/db/src/scheduler.ts +++ b/packages/db/src/scheduler.ts @@ -1,3 +1,5 @@ +import { runAllCallbacks } from './utils/callbacks.js' + /** * Identifier used to scope scheduled work. Maps to a transaction id for live queries. */ @@ -16,13 +18,13 @@ interface ScheduleOptions { /** * State per context. Queue preserves order, jobs hold run functions, dependencies track - * prerequisites, and completed records which jobs have run during the current flush. + * prerequisites. A job leaves the pending map before its callback runs, so work + * queued by that callback is a new pending dependency. */ interface SchedulerContextState { queue: Array jobs: Map void> dependencies: Map> - completed: Set } interface PendingAwareJob { @@ -62,7 +64,6 @@ export class Scheduler { queue: [], jobs: new Map(), dependencies: new Map(), - completed: new Set(), } this.contexts.set(contextId, context) } @@ -98,9 +99,6 @@ export class Scheduler { } else if (!context.dependencies.has(jobId)) { context.dependencies.set(jobId, new Set()) } - - // Clear completion status since we're rescheduling - context.completed.delete(jobId) } /** @@ -111,7 +109,7 @@ export class Scheduler { const context = this.contexts.get(contextId) if (!context) return - const { queue, jobs, dependencies, completed } = context + const { queue, jobs, dependencies } = context while (queue.length > 0) { let ranThisPass = false @@ -122,7 +120,6 @@ export class Scheduler { const run = jobs.get(jobId) if (!run) { dependencies.delete(jobId) - completed.delete(jobId) continue } @@ -137,13 +134,10 @@ export class Scheduler { isPendingAwareJob(dep) && dep.hasPendingGraphRun(contextId) // Treat dependencies as blocking if the dep has a pending run in this - // context or if it's enqueued and not yet complete. If the dep is + // context or if it's enqueued. If the dep is // neither pending nor enqueued, consider it satisfied to avoid deadlocks // on lazy sources that never schedule work. - if ( - (jobs.has(dep) && !completed.has(dep)) || - (!jobs.has(dep) && depHasPending) - ) { + if (jobs.has(dep) || depHasPending) { ready = false break } @@ -153,10 +147,9 @@ export class Scheduler { if (ready) { jobs.delete(jobId) dependencies.delete(jobId) - // Run the job. If it throws, we don't mark it complete, allowing the - // error to propagate while maintaining scheduler state consistency. + // A reentrant schedule now owns a fresh pending job; finishing this + // callback must not mark that replacement as complete. run() - completed.add(jobId) ranThisPass = true } else { queue.push(jobId) @@ -175,20 +168,12 @@ export class Scheduler { this.contexts.delete(contextId) } - /** - * Flush all contexts with pending work. Useful during tear-down. - */ - flushAll(): void { - for (const contextId of Array.from(this.contexts.keys())) { - this.flush(contextId) - } - } - /** Clear all scheduled jobs for a context. */ clear(contextId: SchedulerContextId): void { this.contexts.delete(contextId) - // Notify listeners that this context was cleared - this.clearListeners.forEach((listener) => listener(contextId)) + runAllCallbacks( + [...this.clearListeners].map((listener) => () => listener(contextId)), + ) } /** Register a listener to be notified when a context is cleared. */ @@ -196,32 +181,16 @@ export class Scheduler { this.clearListeners.add(listener) return () => this.clearListeners.delete(listener) } - - /** Check if a context has pending jobs. */ - hasPendingJobs(contextId: SchedulerContextId): boolean { - const context = this.contexts.get(contextId) - return !!context && context.jobs.size > 0 - } - - /** Remove a single job from a context and clean up its dependencies. */ - clearJob(contextId: SchedulerContextId, jobId: unknown): void { - const context = this.contexts.get(contextId) - if (!context) return - - context.jobs.delete(jobId) - context.dependencies.delete(jobId) - context.completed.delete(jobId) - context.queue = context.queue.filter((id) => id !== jobId) - - if (context.jobs.size === 0) { - this.contexts.delete(contextId) - } - } } export const transactionScopedScheduler = new Scheduler() let activePublicationContext: SchedulerContextId | undefined +let activePublicationFailure: { error: unknown } | undefined + +function getActivePublicationFailure(): { error: unknown } | undefined { + return activePublicationFailure +} /** * Returns the Collection publication that currently owns synchronous change @@ -232,6 +201,12 @@ export function getActivePublicationContext(): SchedulerContextId | undefined { return activePublicationContext } +/** Report a listener failure after the whole publication graph has drained. */ +export function recordPublicationError(error: unknown): void { + if (activePublicationContext === undefined) throw error + activePublicationFailure ??= { error } +} + /** * Runs one synchronous Collection publication inside a scheduler context. * Nested publications share the outer context, so downstream live queries run @@ -242,14 +217,29 @@ export function withPublicationContext(publish: () => T): T { const contextId = Symbol(`collection-publication`) activePublicationContext = contextId + activePublicationFailure = undefined + let result!: T + let listenerFailure: { error: unknown } | undefined try { - const result = publish() + result = publish() transactionScopedScheduler.flush(contextId) - return result + listenerFailure = getActivePublicationFailure() } catch (error) { - transactionScopedScheduler.clear(contextId) + try { + transactionScopedScheduler.clear(contextId) + } catch { + // Keep the earlier publication or graph failure. + } + // Keep the first reported failure, including one from an earlier listener. + const publicationFailure = getActivePublicationFailure() + if (publicationFailure) { + throw publicationFailure.error + } throw error } finally { activePublicationContext = undefined + activePublicationFailure = undefined } + if (listenerFailure) throw listenerFailure.error + return result } diff --git a/packages/db/src/transactions.ts b/packages/db/src/transactions.ts index c30ee78f05..34461569a0 100644 --- a/packages/db/src/transactions.ts +++ b/packages/db/src/transactions.ts @@ -1,5 +1,6 @@ import { createDeferred } from './deferred' import { safeRandomUUID } from './utils/uuid' +import { normalizeError } from './utils/error.js' import './duplicate-instance-check' import { MissingMutationFunctionError, @@ -535,6 +536,7 @@ class Transaction> { if (this.state === `completed`) { throw new TransactionAlreadyCompletedRollbackError() } + if (this.state === `failed`) return this this.setState(`failed`) @@ -636,14 +638,17 @@ class Transaction> { transaction: this as unknown as TransactionWithMutations, }) + if ((this.state as TransactionState) !== `persisting`) return this + this.setState(`completed`) this.touchCollection() this.isPersisted.resolve(this) } catch (error) { + if ((this.state as TransactionState) !== `persisting`) return this + // Preserve the original error for rethrowing - const originalError = - error instanceof Error ? error : new Error(String(error)) + const originalError = normalizeError(error) // Update transaction with error information this.error = { diff --git a/packages/db/src/types.ts b/packages/db/src/types.ts index 29db572da0..a7795f85f1 100644 --- a/packages/db/src/types.ts +++ b/packages/db/src/types.ts @@ -283,9 +283,8 @@ export interface Subscription extends EventEmitter { export type CursorExpressions = { /** * Expression for rows greater than (after) the cursor value. - * For multi-column orderBy, this is a composite cursor using OR of conditions. - * Example for [col1 ASC, col2 DESC] with values [v1, v2]: - * or(gt(col1, v1), and(eq(col1, v1), lt(col2, v2))) + * Core emits cursors for a single order column. Multi-column queries use + * prefix-and-tie loading instead of constructing a composite cursor. */ whereFrom: BasicExpression /** @@ -301,6 +300,15 @@ export type CursorExpressions = { lastKey?: string | number } +/** + * Immutable request data. From submission onward, callers and adapters must + * not mutate these options, their expression trees, comparison options, or + * constant payloads (including Dates, byte arrays, and membership arrays). + * Create new request data to change a demand; core does not clone or freeze it. + * Use stable data properties, not stateful getters, for request data. + * Signal and subscription references stay fixed, but their lifecycle remains + * live: aborting the signal or releasing the subscription is supported. + */ export type LoadSubsetOptions = { /** The where expression to filter the data (does NOT include cursor expressions) */ where?: BasicExpression @@ -321,8 +329,10 @@ export type LoadSubsetOptions = { offset?: number /** * Aborted when this exact subset request is no longer current. Cancellation - * is cooperative: async sync adapters must check the signal immediately - * before installing a baseline or later request-scoped rows. + * is cooperative: async adapters should stop before installing more + * request-scoped rows. If an in-flight baseline cannot be canceled, the + * returned load promise must settle after those writes become visible so + * core can keep overlapping replay private until then. */ signal?: AbortSignal /** @@ -336,12 +346,16 @@ export type LoadSubsetOptions = { subscription?: Subscription } +/** @internal Result returned by the collection's normalized subset boundary. */ +export type LoadSubsetRequestResult = true | Promise + /** * Loads one subset and transfers its ongoing resource ownership only after * returning `true` or a promise. An implementation that throws synchronously * must release any partially acquired resource before throwing. A successful * implementation must await or return every applied receipt from the sync - * `commit()` calls that establish the loaded subset. + * `commit()` calls that establish the loaded subset. A result describes only + * the exact `options` passed to this call. */ export type LoadSubsetFn = (options: LoadSubsetOptions) => true | Promise @@ -353,6 +367,14 @@ export type LoadSubsetFn = (options: LoadSubsetOptions) => true | Promise */ export type SyncAppliedReceipt = true | Promise +/** + * Releases the exact acquisition created for `options`. + * + * Implementations must be idempotent and must not throw. An adapter owns any + * remote unsubscribe retry needed to make release reliable. Core preserves a + * failed release defensively so a later cleanup attempt can retry the same + * acquisition identity. + */ export type UnloadSubsetFn = (options: LoadSubsetOptions) => void export type CleanupFn = () => void @@ -933,9 +955,14 @@ export interface SubscribeChangesOptions< * Allows the caller to directly track the loading promise for isReady status. * @internal */ - onLoadSubsetResult?: (result: Promise | true) => void + onLoadSubsetResult?: (result: LoadSubsetRequestResult) => void /** Receives subset-load failures scoped to this subscription. @internal */ onLoadSubsetError?: (event: SubscriptionLoadSubsetErrorEvent) => void + /** Lets a live-query graph retain its last publication during replay. @internal */ + truncateReplayPublication?: { + readonly start: () => void + readonly succeed: () => void + } } export interface SubscribeChangesSnapshotOptions< diff --git a/packages/db/src/utils.ts b/packages/db/src/utils.ts index e652087419..c5eb9aef3e 100644 --- a/packages/db/src/utils.ts +++ b/packages/db/src/utils.ts @@ -30,6 +30,14 @@ export function deepEquals(a: any, b: any): boolean { return deepEqualsInternal(a, b, new Map()) } +function enumerableOwnKeys(value: object): Array { + const keys: Array = Object.keys(value) + for (const key of Object.getOwnPropertySymbols(value)) { + if (Object.prototype.propertyIsEnumerable.call(value, key)) keys.push(key) + } + return keys +} + /** * Internal implementation with cycle detection to prevent infinite recursion */ @@ -188,9 +196,10 @@ function deepEqualsInternal( } visited.set(a, b) - // Get all keys from both objects - const keysA = Object.keys(a) - const keysB = Object.keys(b) + // Compare enumerable symbol keys as well as string keys. Query results may + // use user-owned symbols, and a symbol-only update is still a value change. + const keysA = enumerableOwnKeys(a) + const keysB = enumerableOwnKeys(b) // Check if they have the same number of keys if (keysA.length !== keysB.length) { @@ -200,7 +209,9 @@ function deepEqualsInternal( // Check if all keys exist in both objects and their values are equal const result = keysA.every( - (key) => key in b && deepEqualsInternal(a[key], b[key], visited), + (key) => + Object.prototype.propertyIsEnumerable.call(b, key) && + deepEqualsInternal(a[key], b[key], visited), ) visited.delete(a) diff --git a/packages/db/src/utils/array-utils.ts b/packages/db/src/utils/array-utils.ts index 47569cacb7..67d505d4e6 100644 --- a/packages/db/src/utils/array-utils.ts +++ b/packages/db/src/utils/array-utils.ts @@ -1,3 +1,13 @@ +import { compareKeys } from '@tanstack/db-ivm' + +/** Key order for descending pages, so no page needs a separate reverse pass. */ +export function compareKeysReversed( + a: string | number, + b: string | number, +): number { + return compareKeys(b, a) +} + /** * Finds the correct insert position for a value in a sorted array using binary search * @param sortedArray The sorted array to search in @@ -26,52 +36,3 @@ export function findInsertPositionInArray( return left } - -/** - * Finds the correct insert position for a value in a sorted tuple array using binary search - * @param sortedArray The sorted tuple array to search in - * @param value The value to find the position for - * @param compareFn Comparison function to use for ordering - * @returns The index where the value should be inserted to maintain order - */ -export function findInsertPosition( - sortedArray: Array<[T, any]>, - value: T, - compareFn: (a: T, b: T) => number, -): number { - let left = 0 - let right = sortedArray.length - - while (left < right) { - const mid = Math.floor((left + right) / 2) - const comparison = compareFn(sortedArray[mid]![0], value) - - if (comparison < 0) { - left = mid + 1 - } else { - right = mid - } - } - - return left -} - -/** - * Deletes a value from a sorted array while maintaining sort order - * @param sortedArray The sorted array to delete from - * @param value The value to delete - * @param compareFn Comparison function to use for ordering - * @returns True if the value was found and deleted, false otherwise - */ -export function deleteInSortedArray( - sortedArray: Array, - value: T, - compareFn: (a: T, b: T) => number, -): boolean { - const idx = findInsertPositionInArray(sortedArray, value, compareFn) - if (idx < sortedArray.length && compareFn(sortedArray[idx]!, value) === 0) { - sortedArray.splice(idx, 1) - return true - } - return false -} diff --git a/packages/db/src/utils/btree.ts b/packages/db/src/utils/btree.ts index 0d35cf7a5b..39a7a0a38c 100644 --- a/packages/db/src/utils/btree.ts +++ b/packages/db/src/utils/btree.ts @@ -32,71 +32,14 @@ type index = number // - V8 source (NewElementsCapacity in src/objects.h): arrays grow by 50% + 16 elements /** - * A reasonably fast collection of key-value pairs with a powerful API. - * Largely compatible with the standard Map. BTree is a B+ tree data structure, - * so the collection is sorted by key. - * - * B+ trees tend to use memory more efficiently than hashtables such as the - * standard Map, especially when the collection contains a large number of - * items. However, maintaining the sort order makes them modestly slower: - * O(log size) rather than O(1). This B+ tree implementation supports O(1) - * fast cloning. It also supports freeze(), which can be used to ensure that - * a BTree is not changed accidentally. - * - * Confusingly, the ES6 Map.forEach(c) method calls c(value,key) instead of - * c(key,value), in contrast to other methods such as set() and entries() - * which put the key first. I can only assume that the order was reversed on - * the theory that users would usually want to examine values and ignore keys. - * BTree's forEach() therefore works the same way, but a second method - * `.forEachPair((key,value)=>{...})` is provided which sends you the key - * first and the value second; this method is slightly faster because it is - * the "native" for-each method for this class. - * - * Out of the box, BTree supports keys that are numbers, strings, arrays of - * numbers/strings, Date, and objects that have a valueOf() method returning a - * number or string. Other data types, such as arrays of Date or custom - * objects, require a custom comparator, which you must pass as the second - * argument to the constructor (the first argument is an optional list of - * initial items). Symbols cannot be used as keys because they are unordered - * (one Symbol is never "greater" or "less" than another). - * - * @example - * Given a {name: string, age: number} object, you can create a tree sorted by - * name and then by age like this: - * - * var tree = new BTree(undefined, (a, b) => { - * if (a.name > b.name) - * return 1; // Return a number >0 when a > b - * else if (a.name < b.name) - * return -1; // Return a number <0 when a < b - * else // names are equal (or incomparable) - * return a.age - b.age; // Return >0 when a.age > b.age - * }); - * - * tree.set({name:"Bill", age:17}, "happy"); - * tree.set({name:"Fran", age:40}, "busy & stressed"); - * tree.set({name:"Bill", age:55}, "recently laid off"); - * tree.forEachPair((k, v) => { - * console.log(`Name: ${k.name} Age: ${k.age} Status: ${v}`); - * }); - * - * @description - * The "range" methods (`forEach, forRange, editRange`) will return the number - * of elements that were scanned. In addition, the callback can return {break:R} - * to stop early and return R from the outer function. - * - * - TODO: Test performance of preallocating values array at max size - * - TODO: Add fast initialization when a sorted array is provided to constructor - * - * For more documentation see https://github.com/qwertie/btree-typescript - * - * Are you a C# developer? You might like the similar data structures I made for C#: - * BDictionary, BList, etc. See http://core.loyc.net/collections/ - * + * Mutable B+ tree used by BTreeIndex for sorted value buckets. Keys use the + * supplied comparator; point operations cost O(log size). This local fork has + * no copy-on-write sharing, cloning, or optional-value storage. + * Range callbacks may return { break: result } to stop traversal early. * @author David Piepgrass */ export class BTree { - private _root: BNode = EmptyLeaf as BNode + private _root: BNode = new BNode() _size = 0 _maxNodeSize: number @@ -109,19 +52,12 @@ export class BTree { /** * Initializes an empty B+ tree. * @param compare Custom function to compare pairs of elements in the tree. - * If not specified, defaultComparator will be used which is valid as long as K extends DefaultComparable. - * @param entries A set of key-value pairs to initialize the tree * @param maxNodeSize Branching factor (maximum items or children per node) * Must be in range 4..256. If undefined or <4 then default is used; if >256 then 256. */ - public constructor( - compare: (a: K, b: K) => number, - entries?: Array<[K, V]>, - maxNodeSize?: number, - ) { + public constructor(compare: (a: K, b: K) => number, maxNodeSize?: number) { this._maxNodeSize = maxNodeSize! >= 4 ? Math.min(maxNodeSize!, 256) : 32 this._compare = compare - if (entries) this.setPairs(entries) } // /////////////////////////////////////////////////////////////////////////// @@ -131,18 +67,10 @@ export class BTree { get size() { return this._size } - /** Gets the number of key-value pairs in the tree. */ - get length() { - return this._size - } - /** Returns true iff the tree contains no key-value pairs. */ - get isEmpty() { - return this._size === 0 - } /** Releases the tree so that its size is 0. */ clear() { - this._root = EmptyLeaf as BNode + this._root = new BNode() this._size = 0 } @@ -160,7 +88,7 @@ export class BTree { * Adds or overwrites a key-value pair in the B+ tree. * @param key the key is used to determine the sort order of * data in the tree. - * @param value data to associate with the key (optional) + * @param value data to associate with the key * @param overwrite Whether to overwrite an existing key-value pair * (default: true). If this is false and there is an existing * key-value pair then this method has no effect. @@ -171,7 +99,6 @@ export class BTree { * has data that does not affect its sort order. */ set(key: K, value: V, overwrite?: boolean): boolean { - if (this._root.isShared) this._root = this._root.clone() const result = this._root.set(key, value, overwrite, this) if (result === true || result === false) return result // Root node has split, so create a new root node. @@ -203,11 +130,6 @@ export class BTree { // /////////////////////////////////////////////////////////////////////////// // Additional methods /////////////////////////////////////////////////////// - /** Returns the maximum number of children/values before nodes will split. */ - get maxNodeSize() { - return this._maxNodeSize - } - /** Gets the lowest key in the tree. Complexity: O(log size) */ minKey(): K | undefined { return this._root.minKey() @@ -218,23 +140,6 @@ export class BTree { return this._root.maxKey() } - /** Gets an array of all keys, sorted */ - keysArray() { - const results: Array = [] - this._root.forRange( - this.minKey()!, - this.maxKey()!, - true, - false, - this, - 0, - (k, _v) => { - results.push(k) - }, - ) - return results - } - /** Returns the next pair whose key is larger than the specified key (or undefined if there is none). * If key === undefined, this function returns the lowest pair. * @param key The key to search for. @@ -254,14 +159,6 @@ export class BTree { ) } - /** Returns the next key larger than the specified key, or undefined if there is none. - * Also, nextHigherKey(undefined) returns the lowest key. - */ - nextHigherKey(key: K | undefined): K | undefined { - const p = this.nextHigherPair(key, ReusedArray as [K, V]) - return p && p[0] - } - /** Returns the next pair whose key is smaller than the specified key (or undefined if there is none). * If key === undefined, this function returns the highest pair. * @param key The key to search for. @@ -276,31 +173,6 @@ export class BTree { return this._root.getPairOrNextLower(key, this._compare, false, reusedArray) } - /** Returns the next key smaller than the specified key, or undefined if there is none. - * Also, nextLowerKey(undefined) returns the highest key. - */ - nextLowerKey(key: K | undefined): K | undefined { - const p = this.nextLowerPair(key, ReusedArray as [K, V]) - return p && p[0] - } - - /** Adds all pairs from a list of key-value pairs. - * @param pairs Pairs to add to this tree. If there are duplicate keys, - * later pairs currently overwrite earlier ones (e.g. [[0,1],[0,7]] - * associates 0 with 7.) - * @param overwrite Whether to overwrite pairs that already exist (if false, - * pairs[i] is ignored when the key pairs[i][0] already exists.) - * @returns The number of pairs added to the collection. - * @description Computational complexity: O(pairs.length * log(size + pairs.length)) - */ - setPairs(pairs: Array<[K, V]>, overwrite?: boolean): number { - let added = 0 - for (const pair of pairs) { - if (this.set(pair[0], pair[1], overwrite)) added++ - } - return added - } - forRange( low: K, high: K, @@ -348,12 +220,10 @@ export class BTree { /** * Scans and potentially modifies values for a subsequence of keys. * Note: the callback `onFound` should ideally be a pure function. - * Specfically, it must not insert items, call clone(), or change - * the collection except via return value; out-of-band editing may - * cause an exception or may cause incorrect data to be sent to - * the callback (duplicate or missed items). It must not cause a - * clone() of the collection, otherwise the clone could be modified - * by changes requested by the callback. + * Specfically, it must not insert items or change the collection + * except via return value; out-of-band editing may cause an + * exception or may cause incorrect data to be sent to the callback + * (duplicate or missed items). * @param low The first key scanned will be greater than or equal to `low`. * @param high Scanning stops when a key larger than this is reached. * @param includeHigh If the `high` key is present, `onFound` is called for @@ -370,9 +240,6 @@ export class BTree { * `{break:R}` to stop early. * @description * Computational complexity: O(number of items scanned + log size) - * Note: if the tree has been cloned with clone(), any shared - * nodes are copied before `onFound` is called. This takes O(n) time - * where n is proportional to the amount of shared data scanned. */ editRange( low: K, @@ -382,7 +249,6 @@ export class BTree { initialCounter?: number, ): R | number { let root = this._root - if (root.isShared) this._root = root = root.clone() try { const r = root.forRange( low, @@ -395,18 +261,12 @@ export class BTree { ) return typeof r === `number` ? r : r.break! } finally { - let isShared while (root.keys.length <= 1 && !root.isLeaf) { - isShared ||= root.isShared this._root = root = root.keys.length === 0 - ? EmptyLeaf + ? new BNode() : (root as any as BNodeInternal).children[0]! } - // If any ancestor of the new root was shared, the new root must also be shared - if (isShared) { - root.isShared = true - } } } } @@ -416,19 +276,13 @@ class BNode { // If this is an internal node, _keys[i] is the highest key in children[i]. keys: Array values: Array - // True if this node might be within multiple `BTree`s (or have multiple parents). - // If so, it must be cloned before being mutated to avoid changing an unrelated tree. - // This is transitive: if it's true, children are also shared even if `isShared!=true` - // in those children. (Certain operations will propagate isShared=true to children.) - isShared: true | undefined get isLeaf() { return (this as any).children === undefined } - constructor(keys: Array = [], values?: Array) { + constructor(keys: Array = [], values: Array = []) { this.keys = keys - this.values = values || undefVals - this.isShared = undefined + this.values = values } // ///////////////////////////////////////////////////////////////////////// @@ -486,11 +340,6 @@ class BNode { return reusedArray } - clone(): BNode { - const v = this.values - return new BNode(this.keys.slice(0), v === undefVals ? v : v.slice(0)) - } - get(key: K, defaultValue: V | undefined, tree: BTree): V | undefined { const i = this.indexOf(key, -1, tree._compare) return i < 0 ? defaultValue : this.values[i] @@ -545,7 +394,7 @@ class BNode { tree._size++ if (this.keys.length < tree._maxNodeSize) { - return this.insertInLeaf(i, key, value, tree) + return this.insertInLeaf(i, key, value) } else { // This leaf node is full and must split const newRightSibling = this.splitOffRightSide() @@ -554,13 +403,12 @@ class BNode { i -= this.keys.length target = newRightSibling } - target.insertInLeaf(i, key, value, tree) + target.insertInLeaf(i, key, value) return newRightSibling } } else { // Key already exists if (overwrite !== false) { - if (value !== undefined) this.reifyValues() // usually this is a no-op, but some users may wish to edit the key this.keys[i] = key this.values[i] = value @@ -569,61 +417,30 @@ class BNode { } } - reifyValues() { - if (this.values === undefVals) - return (this.values = this.values.slice(0, this.keys.length)) - return this.values - } - - insertInLeaf(i: index, key: K, value: V, tree: BTree) { + insertInLeaf(i: index, key: K, value: V) { this.keys.splice(i, 0, key) - if (this.values === undefVals) { - while (undefVals.length < tree._maxNodeSize) undefVals.push(undefined) - if (value === undefined) { - return true - } else { - this.values = undefVals.slice(0, this.keys.length - 1) - } - } this.values.splice(i, 0, value) return true } takeFromRight(rhs: BNode) { // Reminder: parent node must update its copy of key for this node - // assert: neither node is shared // assert rhs.keys.length > (maxNodeSize/2 && this.keys.length) { // Reminder: parent node must update its copy of key for this node - // assert: neither node is shared // assert rhs.keys.length > (maxNodeSize/2 && this.keys.length { // Reminder: parent node must update its copy of key for this node - const half = this.keys.length >> 1, - keys = this.keys.splice(half) - const values = - this.values === undefVals ? undefVals : this.values.splice(half) - return new BNode(keys, values) + const half = this.keys.length >> 1 + return new BNode(this.keys.splice(half), this.values.splice(half)) } // /////////////////////////////////////////////////////////////////////////// @@ -658,11 +475,11 @@ class BNode { const result = onFound(key, values[i]!, count++) if (result !== undefined) { if (editMode === true) { - if (key !== keys[i] || this.isShared === true) - throw new Error(`BTree illegally changed or cloned in editRange`) + if (key !== keys[i]) + throw new Error(`BTree illegally changed in editRange`) if (result.delete) { this.keys.splice(i, 1) - if (this.values !== undefVals) this.values.splice(i, 1) + this.values.splice(i, 1) tree._size-- i-- iHigh-- @@ -680,11 +497,7 @@ class BNode { /** Adds entire contents of right-hand sibling (rhs is left unchanged) */ mergeSibling(rhs: BNode, _: number) { this.keys.push.apply(this.keys, rhs.keys) - if (this.values === undefVals) { - if (rhs.values === undefVals) return - this.values = this.values.slice(0, this.keys.length) - } - this.values.push.apply(this.values, rhs.reifyValues()) + this.values.push.apply(this.values, rhs.values) } } @@ -695,10 +508,6 @@ class BNodeInternal extends BNode { // keys[i] caches the value of children[i].maxKey(). children: Array> - /** - * This does not mark `children` as shared, so it is the responsibility of the caller - * to ensure children are either marked shared, or aren't included in another tree. - */ constructor(children: Array>, keys?: Array) { if (!keys) { keys = [] @@ -783,10 +592,9 @@ class BNodeInternal extends BNode { const c = this.children, max = tree._maxNodeSize, cmp = tree._compare - let i = Math.min(this.indexOf(key, 0, cmp), c.length - 1), - child = c[i]! + let i = Math.min(this.indexOf(key, 0, cmp), c.length - 1) + const child = c[i]! - if (child.isShared) c[i] = child = child.clone() if (child.keys.length >= max) { // child is full; inserting anything else will cause a split. // Shifting an item to the left or right sibling may avoid a split. @@ -798,7 +606,6 @@ class BNodeInternal extends BNode { (other = c[i - 1]!).keys.length < max && cmp(child.keys[0]!, key) < 0 ) { - if (other.isShared) c[i - 1] = other = other.clone() other.takeFromRight(child) this.keys[i - 1] = other.maxKey()! } else if ( @@ -806,7 +613,6 @@ class BNodeInternal extends BNode { other.keys.length < max && cmp(child.maxKey()!, key) < 0 ) { - if (other.isShared) c[i + 1] = other = other.clone() other.takeFromLeft(child) this.keys[i] = c[i]!.maxKey()! } @@ -835,11 +641,7 @@ class BNodeInternal extends BNode { } } - /** - * Inserts `child` at index `i`. - * This does not mark `child` as shared, so it is the responsibility of the caller - * to ensure that either child is marked shared, or it is not included in another tree. - */ + /** Inserts `child` at index `i`. */ insert(i: index, child: BNode) { this.children.splice(i, 0, child) this.keys.splice(i, 0, child.maxKey()!) @@ -850,7 +652,6 @@ class BNodeInternal extends BNode { * Modifies this to remove the second half of the items, returning a separate node containing them. */ splitOffRightSide() { - // assert !this.isShared; const half = this.children.length >> 1 return new BNodeInternal( this.children.splice(half), @@ -860,7 +661,6 @@ class BNodeInternal extends BNode { takeFromRight(rhs: BNode) { // Reminder: parent node must update its copy of key for this node - // assert: neither node is shared // assert rhs.keys.length > (maxNodeSize/2 && this.keys.length).children.shift()!) @@ -868,7 +668,6 @@ class BNodeInternal extends BNode { takeFromLeft(lhs: BNode) { // Reminder: parent node must update its copy of key for this node - // assert: neither node is shared // assert rhs.keys.length > (maxNodeSize/2 && this.keys.length).children.pop()!) @@ -916,7 +715,6 @@ class BNodeInternal extends BNode { } else if (i <= iHigh) { try { for (; i <= iHigh; i++) { - if (children[i]!.isShared) children[i] = children[i]!.clone() const result = children[i]!.forRange( low, high, @@ -959,9 +757,6 @@ class BNodeInternal extends BNode { const children = this.children if (i >= 0 && i + 1 < children.length) { if (children[i]!.keys.length + children[i + 1]!.keys.length <= maxSize) { - if (children[i]!.isShared) - // cloned already UNLESS i is outside scan range - children[i] = children[i]!.clone() children[i]!.mergeSibling(children[i + 1]!, maxSize) children.splice(i + 1, 1) this.keys.splice(i + 1, 1) @@ -974,22 +769,14 @@ class BNodeInternal extends BNode { /** * Move children from `rhs` into this. - * `rhs` must be part of this tree, and be removed from it after this call - * (otherwise isShared for its children could be incorrect). + * `rhs` must be part of this tree, and be removed from it after this call. */ mergeSibling(rhs: BNode, maxNodeSize: number) { - // assert !this.isShared; const oldLength = this.keys.length this.keys.push.apply(this.keys, rhs.keys) const rhsChildren = (rhs as any as BNodeInternal).children this.children.push.apply(this.children, rhsChildren) - if (rhs.isShared && !this.isShared) { - // All children of a shared node are implicitly shared, and since their new - // parent is not shared, they must now be explicitly marked as shared. - for (const child of rhsChildren) child.isShared = true - } - // If our children are themselves almost empty due to a mass-delete, // they may need to be merged too (but only the oldLength-1 and its // right sibling should need this). @@ -997,27 +784,8 @@ class BNodeInternal extends BNode { } } -// Optimization: this array of `undefined`s is used instead of a normal -// array of values in nodes where `undefined` is the only value. -// Its length is extended to max node size on first use; since it can -// be shared between trees with different maximums, its length can only -// increase, never decrease. Its type should be undefined[] but strangely -// TypeScript won't allow the comparison V[] === undefined[]. To prevent -// users from making this array too large, BTree has a maximum node size. -// -// FAQ: undefVals[i] is already undefined, so why increase the array size? -// Reading outside the bounds of an array is relatively slow because it -// has the side effect of scanning the prototype chain. -const undefVals: Array = [] - const Delete = { delete: true }, DeleteRange = () => Delete -const EmptyLeaf = (function () { - const n = new BNode() - n.isShared = true - return n -})() -const ReusedArray: Array = [] // assumed thread-local function check(fact: boolean, ...args: Array) { if (!fact) { diff --git a/packages/db/src/utils/callbacks.ts b/packages/db/src/utils/callbacks.ts new file mode 100644 index 0000000000..1b0aba4859 --- /dev/null +++ b/packages/db/src/utils/callbacks.ts @@ -0,0 +1,12 @@ +/** Attempt every callback, then rethrow the first exact failure value. */ +export function runAllCallbacks(callbacks: Iterable<() => void>): void { + let firstFailure: { error: unknown } | undefined + for (const callback of callbacks) { + try { + callback() + } catch (error) { + firstFailure ??= { error } + } + } + if (firstFailure) throw firstFailure.error +} diff --git a/packages/db/src/utils/comparison.ts b/packages/db/src/utils/comparison.ts index 2d76f699b4..4c01a49873 100644 --- a/packages/db/src/utils/comparison.ts +++ b/packages/db/src/utils/comparison.ts @@ -1,4 +1,5 @@ import { isTemporal } from '../utils' +import { getRuntimeReferenceIdentity } from '../query/runtime-reference-identity' import type { CompareOptions } from '../query/builder/types' // WeakMap to store stable IDs for objects @@ -85,6 +86,18 @@ export const ascComparator = (a: any, b: any, opts: CompareOptions): number => { return compareTemporalValues(a, b) } + // Symbols have identity but no built-in order: relational comparison throws. + // A stable runtime ID gives tree indexes a total order while preserving + // equality only for the same symbol. + const aIsSymbol = typeof a === `symbol` + const bIsSymbol = typeof b === `symbol` + if (aIsSymbol && bIsSymbol) { + if (a === b) return 0 + return getRuntimeReferenceIdentity(a)[2] - getRuntimeReferenceIdentity(b)[2] + } + if (aIsSymbol) return 1 + if (bIsSymbol) return -1 + // If at least one of the values is an object, use stable IDs for comparison const aIsObject = typeof a === `object` const bIsObject = typeof b === `object` @@ -142,9 +155,15 @@ export const defaultComparator = makeComparator({ stringSort: `locale`, }) -/** - * Compare two Uint8Arrays for content equality - */ +/** Include host Buffers when the current realm has a different Uint8Array. */ +export function isUint8Array(value: unknown): value is Uint8Array { + return ( + value instanceof Uint8Array || + (typeof Buffer !== `undefined` && value instanceof Buffer) + ) +} + +/** Compare two Uint8Arrays for content equality. */ function areUint8ArraysEqual(a: Uint8Array, b: Uint8Array): boolean { if (a.byteLength !== b.byteLength) { return false @@ -157,20 +176,26 @@ function areUint8ArraysEqual(a: Uint8Array, b: Uint8Array): boolean { return true } -/** - * Threshold for normalizing Uint8Arrays to string representations. - * Arrays larger than this will use reference equality to avoid memory overhead. - * 128 bytes is enough for common ID formats (ULIDs are 16 bytes, UUIDs are 16 bytes) - * while avoiding excessive string allocation for large binary data. - */ -const UINT8ARRAY_NORMALIZE_THRESHOLD = 128 +const NORMALIZED_KEY_PREFIX = `\u0000tanstack-db:` + +function normalizedKey(kind: string, value: string): string { + return `${NORMALIZED_KEY_PREFIX}${kind}:${value}` +} + +function normalizeBinary(value: Uint8Array): string { + let bytes = `` + for (let index = 0; index < value.byteLength; index++) { + bytes += String.fromCharCode(value[index]!) + } + return normalizedKey(`binary`, bytes) +} /** * Sentinel value representing undefined in normalized form. * This allows distinguishing between "start from beginning" (undefined parameter) * and "start from the key undefined" (actual undefined value in the tree). */ -export const UNDEFINED_SENTINEL = `__TS_DB_BTREE_UNDEFINED_VALUE__` +export const UNDEFINED_SENTINEL = normalizedKey(`undefined`, ``) /** * Normalize a value for comparison and Map key usage @@ -181,6 +206,12 @@ export const UNDEFINED_SENTINEL = `__TS_DB_BTREE_UNDEFINED_VALUE__` * for BTree index operations that need to distinguish undefined values. */ export function normalizeValue(value: any): any { + if (typeof value === `string`) { + return value.startsWith(NORMALIZED_KEY_PREFIX) + ? normalizedKey(`string`, value) + : value + } + if (typeof value !== `object` || value === null) { return value } @@ -190,24 +221,16 @@ export function normalizeValue(value: any): any { } if (isTemporal(value)) { - return `__temporal__${value[Symbol.toStringTag]}__${value.toString()}` + return normalizedKey( + `temporal`, + `${value[Symbol.toStringTag]}:${value.toString()}`, + ) } // Normalize Uint8Arrays/Buffers to a string representation for Map key usage // This enables content-based equality for binary data like ULIDs - const isUint8Array = - (typeof Buffer !== `undefined` && value instanceof Buffer) || - value instanceof Uint8Array - - if (isUint8Array) { - // Only normalize small arrays to avoid memory overhead for large binary data - if (value.byteLength <= UINT8ARRAY_NORMALIZE_THRESHOLD) { - // Convert to a string representation that can be used as a Map key - // Use a special prefix to avoid collisions with user strings - return `__u8__${Array.from(value).join(`,`)}` - } - // For large arrays, fall back to reference equality - // Users working with large binary data should use a derived key if needed + if (isUint8Array(value)) { + return normalizeBinary(value) } return value @@ -316,15 +339,8 @@ export function areValuesEqual(a: any, b: any): boolean { } // Check for Uint8Array/Buffer comparison - const aIsUint8Array = - (typeof Buffer !== `undefined` && a instanceof Buffer) || - a instanceof Uint8Array - const bIsUint8Array = - (typeof Buffer !== `undefined` && b instanceof Buffer) || - b instanceof Uint8Array - // If both are Uint8Arrays, compare by content - if (aIsUint8Array && bIsUint8Array) { + if (isUint8Array(a) && isUint8Array(b)) { return areUint8ArraysEqual(a, b) } diff --git a/packages/db/src/utils/cursor.ts b/packages/db/src/utils/cursor.ts index 322a374703..b2aca0a994 100644 --- a/packages/db/src/utils/cursor.ts +++ b/packages/db/src/utils/cursor.ts @@ -1,78 +1,92 @@ -import { and, eq, gt, lt, or } from '../query/builder/functions.js' +import { + and, + eq, + gt, + gte, + isNull, + isUndefined, + lt, + not, + or, +} from '../query/builder/functions.js' import { Value } from '../query/ir.js' -import type { BasicExpression, OrderBy } from '../query/ir.js' +import type { BasicExpression, OrderBy, OrderByClause } from '../query/ir.js' -/** - * Builds a cursor expression for paginating through ordered results. - * For multi-column orderBy, creates a composite cursor that respects all columns. - * - * For [col1 ASC, col2 DESC] with values [v1, v2], produces: - * or( - * gt(col1, v1), // col1 > v1 - * and(eq(col1, v1), lt(col2, v2)) // col1 = v1 AND col2 < v2 (DESC) - * ) - * - * This creates a precise cursor that works with composite indexes on the backend. - * - * @param orderBy - The order-by clauses defining sort columns and directions - * @param values - The cursor values corresponding to each order-by column - * @returns A filter expression for rows after the cursor position, or undefined if empty - */ +function isNullish( + expression: OrderByClause[`expression`], +): BasicExpression { + return or(isNull(expression), isUndefined(expression)) +} + +function followsBoundary( + clause: OrderByClause, + value: unknown, +): BasicExpression { + const nullish = isNullish(clause.expression) + if (value == null) { + return clause.compareOptions.nulls === `first` + ? not(nullish) + : new Value(false) + } + + const operator = clause.compareOptions.direction === `asc` ? gt : lt + const comparison = operator(clause.expression, new Value(value)) + return clause.compareOptions.nulls === `last` + ? or(comparison, nullish) + : comparison +} + +/** Build a single-column cursor; multi-column queries use prefix loading. */ export function buildCursor( orderBy: OrderBy, values: Array, ): BasicExpression | undefined { - if (values.length === 0 || orderBy.length === 0) { - return undefined + if (values.length === 0) return undefined + if (orderBy.length !== 1 || values.length !== 1) { + throw new Error(`Only single-column cursors are supported`) } + return followsBoundary(orderBy[0]!, values[0]) +} - // For single column, just use simple gt/lt - if (orderBy.length === 1) { - const { expression, compareOptions } = orderBy[0]! - const operator = compareOptions.direction === `asc` ? gt : lt - return operator(expression, new Value(values[0])) - } - - // For multi-column, build the composite cursor: - // or( - // gt(col1, v1), - // and(eq(col1, v1), gt(col2, v2)), - // and(eq(col1, v1), eq(col2, v2), gt(col3, v3)), - // ... - // ) - const clauses: Array> = [] - - for (let i = 0; i < orderBy.length && i < values.length; i++) { - const clause = orderBy[i]! - const value = values[i] - - // Build equality conditions for all previous columns - const eqConditions: Array> = [] - for (let j = 0; j < i; j++) { - const prevClause = orderBy[j]! - const prevValue = values[j] - eqConditions.push(eq(prevClause.expression, new Value(prevValue))) - } - - // Add the comparison for the current column (respecting direction) - const operator = clause.compareOptions.direction === `asc` ? gt : lt - const comparison = operator(clause.expression, new Value(value)) - - if (eqConditions.length === 0) { - // First column: just the comparison - clauses.push(comparison) - } else { - // Subsequent columns: and(eq(prev...), comparison) - // We need to spread into and() which expects at least 2 args - const allConditions = [...eqConditions, comparison] - clauses.push(allConditions.reduce((acc, cond) => and(acc, cond))) - } +/** Build the equality range that closes the first ordered boundary term. */ +export function buildCursorCurrent( + orderBy: OrderBy, + values: ReadonlyArray, +): BasicExpression | undefined { + const { expression } = orderBy[0] ?? {} + if (!expression || values.length === 0) return undefined + const value = values[0] + if (value == null) return isNullish(expression) + if (value instanceof Date) { + if (!Number.isFinite(value.getTime())) return undefined + return and( + gte(expression, new Value(value)), + lt(expression, new Value(new Date(value.getTime() + 1))), + ) } + if (typeof value === `object`) return undefined + return eq(expression, new Value(value)) +} - // Combine all clauses with OR - if (clauses.length === 1) { - return clauses[0]! +/** + * Whether the public predicate IR can express this boundary's comparison. + * Unsupported values must use an unbounded fetch rather than a provider order + * that may differ from the local comparator. + */ +export function canExpressCursorOrder( + orderBy: OrderBy, + values: ReadonlyArray, +): boolean { + if (orderBy.length !== 1 || values.length !== 1) return false + const value = values[0] + if (value == null) return false + if (value instanceof Date) return Number.isFinite(value.getTime()) + if (typeof value === `string`) { + return orderBy[0]!.compareOptions.stringSort === `lexical` } - // Use reduce to combine with or() which expects exactly 2 args - return clauses.reduce((acc, clause) => or(acc, clause)) + return ( + (typeof value === `number` && Number.isFinite(value)) || + typeof value === `bigint` || + typeof value === `boolean` + ) } diff --git a/packages/db/src/utils/error.ts b/packages/db/src/utils/error.ts new file mode 100644 index 0000000000..17842241a5 --- /dev/null +++ b/packages/db/src/utils/error.ts @@ -0,0 +1,8 @@ +export const normalizeError = (error: unknown): Error => { + try { + if (error instanceof Error) return error + return new Error(String(error)) + } catch { + return new Error(`Unknown error`) + } +} diff --git a/packages/db/src/utils/get-or-create.ts b/packages/db/src/utils/get-or-create.ts new file mode 100644 index 0000000000..313a2683ba --- /dev/null +++ b/packages/db/src/utils/get-or-create.ts @@ -0,0 +1,16 @@ +/** Lazily initialize a map entry; undefined denotes an absent value. */ +export function getOrCreate( + entries: { + get: (key: K) => V | undefined + set: (key: K, value: V) => unknown + }, + key: K, + create: () => V, +): V { + let value = entries.get(key) + if (value === undefined) { + value = create() + entries.set(key, value) + } + return value +} diff --git a/packages/db/src/utils/index-optimization.ts b/packages/db/src/utils/index-optimization.ts index 5a52a5ec54..92eee5a376 100644 --- a/packages/db/src/utils/index-optimization.ts +++ b/packages/db/src/utils/index-optimization.ts @@ -20,7 +20,7 @@ import { ReverseIndex } from '../indexes/reverse-index.js' import { hasVirtualPropPath } from '../virtual-props.js' import { makeComparator } from './comparison.js' import type { CompareOptions } from '../query/builder/types.js' -import type { IndexInterface, IndexOperation } from '../indexes/base-index.js' +import type { IndexOperation, IndexReader } from '../indexes/base-index.js' import type { BasicExpression } from '../query/ir.js' import type { CollectionLike } from '../types.js' @@ -46,7 +46,7 @@ export function findIndexForField( collection: CollectionLike, fieldPath: Array, compareOptions?: CompareOptions, -): IndexInterface | undefined { +): IndexReader | undefined { if (hasVirtualPropPath(fieldPath)) { return undefined } @@ -162,6 +162,8 @@ function isRangeOrderingDivergent( return false case `string`: return usesLocaleStringSort(collection) + case `symbol`: + return true case `object`: { if (value === null) return false // Dates order consistently with the evaluator: valid Dates by time, and @@ -181,12 +183,13 @@ function isRangeOrderingDivergent( */ function canRangeOptimize( value: unknown, - index: IndexInterface, + index: IndexReader, collection: CollectionLike, ): boolean { return ( !isRangeOrderingDivergent(value, collection) && - index.supportsRangeOptimization + index.supportsRangeOptimization && + (index.canOptimizeRangeFor?.(value) ?? true) ) } diff --git a/packages/db/src/utils/type-guards.ts b/packages/db/src/utils/type-guards.ts index 4c54d80773..a6cc0d803a 100644 --- a/packages/db/src/utils/type-guards.ts +++ b/packages/db/src/utils/type-guards.ts @@ -1,3 +1,11 @@ +export function isPlainObject( + value: unknown, +): value is Record { + if (value === null || typeof value !== `object`) return false + const prototype = Object.getPrototypeOf(value) + return prototype === Object.prototype || prototype === null +} + /** * Type guard to check if a value is promise-like (has a `.then` method) * @param value - The value to check diff --git a/packages/db/src/virtual-props.ts b/packages/db/src/virtual-props.ts index ef285821fa..3f600a5008 100644 --- a/packages/db/src/virtual-props.ts +++ b/packages/db/src/virtual-props.ts @@ -157,34 +157,6 @@ export function hasVirtualProps( ) } -/** - * Creates virtual properties for a row in a source collection. - * - * This is the internal function used by collections to add virtual properties - * to rows when emitting change messages. - * - * @param key - The row's key - * @param collectionId - The collection's ID - * @param isSynced - Whether the row is synced (not optimistic) - * @param origin - Whether the change was local or remote - * @returns Virtual properties object to merge with the row - * - * @internal - */ -export function createVirtualProps( - key: TKey, - collectionId: string, - isSynced: boolean, - origin: VirtualOrigin, -): VirtualRowProps { - return { - $synced: isSynced, - $origin: origin, - $key: key, - $collectionId: collectionId, - } -} - /** * Enriches a row with virtual properties using the "add-if-missing" pattern. * @@ -226,39 +198,6 @@ export function enrichRowWithVirtualProps< } as WithVirtualProps } -/** - * Computes aggregate virtual properties for a group of rows. - * - * For aggregates: - * - `$synced`: true if ALL rows in the group are synced; false if ANY row is optimistic - * - `$origin`: 'local' if ANY row in the group is local; otherwise 'remote' - * - * @param rows - The rows in the group - * @param groupKey - The group key - * @param collectionId - The collection ID - * @returns Virtual properties for the aggregate row - * - * @internal - */ -export function computeAggregateVirtualProps( - rows: Array>>, - groupKey: TKey, - collectionId: string, -): VirtualRowProps { - // $synced = true only if ALL rows are synced (false if ANY is optimistic) - const allSynced = rows.every((row) => row.$synced ?? true) - - // $origin = 'local' if ANY row is local (consistent with "local influence" semantics) - const hasLocal = rows.some((row) => row.$origin === 'local') - - return { - $synced: allSynced, - $origin: hasLocal ? 'local' : 'remote', - $key: groupKey, - $collectionId: collectionId, - } -} - /** * List of virtual property names for iteration and checking. * @internal diff --git a/packages/db/tests/basic-index-work.test.ts b/packages/db/tests/basic-index-work.test.ts new file mode 100644 index 0000000000..438a9bfede --- /dev/null +++ b/packages/db/tests/basic-index-work.test.ts @@ -0,0 +1,97 @@ +import { describe, expect, it, vi } from 'vitest' +import { BasicIndex } from '../src/indexes/basic-index' +import { PropRef } from '../src/query/ir' + +describe(`BasicIndex removal work`, () => { + it.each([1, 4])( + `searches only the comparator group of size %s`, + (groupSize) => { + const size = 1024 + let comparisons = 0 + let scanned = 0 + const index = new BasicIndex( + 1, + new PropRef([`value`]), + undefined, + { + compareFn: (a: number, b: number) => { + comparisons++ + return Math.floor(a / groupSize) - Math.floor(b / groupSize) + }, + }, + ) + for (let value = 0; value < size; value++) index.add(value, { value }) + const target = size - groupSize + const findIndex = Array.prototype.findIndex + const spy = vi + .spyOn(Array.prototype, `findIndex`) + .mockImplementation(function ( + this: Array, + predicate, + thisArg, + ) { + return findIndex.call(this, (value, position, array) => { + scanned++ + return predicate.call(thisArg, value, position, array) + }) + }) + comparisons = 0 + try { + index.remove(target, { value: target }) + } finally { + spy.mockRestore() + } + expect(scanned + comparisons).toBeLessThanOrEqual( + Math.ceil(Math.log2(size)) + groupSize + 1, + ) + expect(index.lookup(`eq`, target).size).toBe(0) + for (let value = target + 1; value < size; value++) { + expect(index.lookup(`eq`, value)).toEqual(new Set([value])) + } + }, + ) +}) + +describe(`BasicIndex page filtering work`, () => { + it.each( + [30, 3000, 100000].flatMap((size) => + [false, true].flatMap((reverse) => + [1, 3].map((stride) => ({ size, reverse, stride })), + ), + ), + )( + `filters only visited keys: $size rows, reverse=$reverse, stride=$stride`, + ({ size, reverse, stride }) => { + const index = new BasicIndex(1, new PropRef([`value`])) + const rows = Array.from({ length: size }, (_, id) => ({ + id, + value: id % 3, + })) + // Deliberately insert backwards; insertion order is not key order. + for (const row of [...rows].reverse()) index.add(row.id, row) + const ordered = rows + .slice() + .sort((a, b) => a.value - b.value || a.id - b.id) + if (reverse) ordered.reverse() + let calls = 0 + const accept = (key: number) => Math.floor(key / 3) % stride === 0 + const expected = ordered + .filter((row) => accept(row.id)) + .slice(0, 10) + .map((row) => row.id) + const filter = (key: number) => { + calls++ + return accept(key) + } + const actual = reverse + ? index.takeReversedFromEnd(10, filter) + : index.takeFromStart(10, filter) + expect(actual).toEqual(expected) + const visits = + expected.length === 10 + ? ordered.findIndex((row) => row.id === expected[9]) + 1 + : size + expect(calls).toBe(visits) + }, + ) +}) diff --git a/packages/db/tests/btree-index-undefined-values.test.ts b/packages/db/tests/btree-index-undefined-values.test.ts index 1510c02e3c..252737e0ab 100644 --- a/packages/db/tests/btree-index-undefined-values.test.ts +++ b/packages/db/tests/btree-index-undefined-values.test.ts @@ -16,6 +16,7 @@ import { createLiveQueryCollection } from '../src/query/live-query-collection.js import { eq } from '../src/query/builder/functions.js' import { BTreeIndex } from '../src/indexes/btree-index.js' import { PropRef } from '../src/query/ir.js' +import { orderedEntriesArray, valueMapData } from './utils' import type { Collection } from '../src/collection/index.js' interface TaskItem { @@ -195,7 +196,7 @@ describe(`BTreeIndex - undefined value handling`, () => { index.add(`num2`, { value: 2 }) index.add(`num0`, { value: 0 }) - const ordered = index.orderedEntriesArray + const ordered = orderedEntriesArray(index) expect(ordered[0]![0]).toBe(undefined) expect(ordered[0]![1]).toContain(`undef`) }) @@ -206,7 +207,7 @@ describe(`BTreeIndex - undefined value handling`, () => { index.add(`undef`, { name: undefined }) index.add(`str2`, { name: `banana` }) - expect(index.orderedEntriesArray[0]![0]).toBe(undefined) + expect(orderedEntriesArray(index)[0]![0]).toBe(undefined) }) it(`should handle mixed undefined and null values`, () => { @@ -312,7 +313,7 @@ describe(`BTreeIndex - undefined value handling`, () => { ) expect(undefinedComparisons.length).toBeGreaterThan(0) - const ordered = index.orderedEntriesArray + const ordered = orderedEntriesArray(index) expect(ordered[0]![0]).toBe(1) expect(ordered[1]![0]).toBe(undefined) }) @@ -408,7 +409,7 @@ describe(`BTreeIndex - undefined value handling`, () => { index.add(`a`, { value: undefined }) index.add(`b`, { value: 1 }) - const mapData = index.valueMapData + const mapData = valueMapData(index) expect(mapData.has(undefined)).toBe(true) expect(mapData.has(`__TS_DB_BTREE_UNDEFINED_VALUE__`)).toBe(false) diff --git a/packages/db/tests/btree-index-work.test.ts b/packages/db/tests/btree-index-work.test.ts new file mode 100644 index 0000000000..d907df5694 --- /dev/null +++ b/packages/db/tests/btree-index-work.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from 'vitest' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { PropRef } from '../src/query/ir.js' + +describe(`BTree exact-bucket ownership work`, () => { + it.each( + [300, 100000].flatMap((size) => + [1, 2].map((groupSize) => ({ size, groupSize })), + ), + )( + `reuses comparator buckets for $size keys with $groupSize exact values per position`, + ({ size, groupSize }) => { + let comparisons = 0 + const index = new BTreeIndex( + 1, + new PropRef([`value`]), + undefined, + { + compareFn: (a: number, b: number) => { + comparisons++ + return Math.floor(a / groupSize) - Math.floor(b / groupSize) + }, + }, + ) + const distinct = 3 * groupSize + // Keep one owner of every exact value throughout the measured batch. + for (let value = 0; value < distinct; value++) + index.add(-value - 1, { value }) + comparisons = 0 + for (let key = 0; key < size; key++) + index.add(key, { value: key % distinct }) + const insertComparisons = comparisons + comparisons = 0 + for (let key = 0; key < size; key++) + index.remove(key, { value: key % distinct }) + const removeComparisons = comparisons + expect(index.keyCount).toBe(distinct) + for (let value = 0; value < distinct; value++) { + expect(index.lookup(`eq`, value)).toEqual(new Set([-value - 1])) + } + expect(insertComparisons).toBe(0) + expect(removeComparisons).toBe(0) + }, + ) +}) diff --git a/packages/db/tests/btree-map-oracle.test.ts b/packages/db/tests/btree-map-oracle.test.ts new file mode 100644 index 0000000000..351b23f491 --- /dev/null +++ b/packages/db/tests/btree-map-oracle.test.ts @@ -0,0 +1,63 @@ +import { describe, expect, it } from 'vitest' +import { BTree } from '../src/utils/btree.js' + +describe(`BTree Map oracle`, () => { + it(`matches a Map oracle under random insert/delete/overwrite with small nodes`, () => { + let seed = 12345 + const rnd = () => + (seed = (seed * 1103515245 + 12345) & 0x7fffffff) / 0x7fffffff + for (let round = 0; round < 40; round++) { + const tree = new BTree( + (a, b) => a - b, + 4 + Math.floor(rnd() * 5), + ) + const oracle = new Map() + for (let step = 0; step < 3000; step++) { + const key = Math.floor(rnd() * 200) + const op = rnd() + if (op < 0.5) { + const val = { v: step } + const added = tree.set(key, val) + expect(added).toBe(!oracle.has(key)) + oracle.set(key, val) + } else if (op < 0.85) { + const deleted = tree.delete(key) + expect(deleted).toBe(oracle.delete(key)) + } else if (op < 0.9) { + tree.clear() + oracle.clear() + } else { + expect(tree.get(key)).toBe(oracle.get(key)) + expect(tree.has(key)).toBe(oracle.has(key)) + } + if (step % 97 === 0) { + const sorted = [...oracle.keys()].sort((a, b) => a - b) + expect(tree.size).toBe(oracle.size) + expect(tree.minKey()).toBe(sorted[0]) + expect(tree.maxKey()).toBe(sorted[sorted.length - 1]) + const seen: Array = [] + if (sorted.length) + tree.forRange( + sorted[0]!, + sorted[sorted.length - 1]!, + true, + (k, v) => { + seen.push(k) + expect(v).toBe(oracle.get(k)) + }, + ) + expect(seen).toEqual(sorted) + const probe = Math.floor(rnd() * 200) + const higher = sorted.find((k) => k > probe) + const lower = [...sorted].reverse().find((k) => k < probe) + expect(tree.nextHigherPair(probe)?.[0]).toBe(higher) + expect(tree.nextLowerPair(probe)?.[0]).toBe(lower) + expect(tree.nextHigherPair(undefined)?.[0]).toBe(sorted[0]) + expect(tree.nextLowerPair(undefined)?.[0]).toBe( + sorted[sorted.length - 1], + ) + } + } + } + }) +}) diff --git a/packages/db/tests/cleanup-queue.test.ts b/packages/db/tests/cleanup-queue.test.ts index d95ddc9f77..8aab26047d 100644 --- a/packages/db/tests/cleanup-queue.test.ts +++ b/packages/db/tests/cleanup-queue.test.ts @@ -1,15 +1,16 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { CleanupQueue } from '../src/collection/cleanup-queue' +import { resetCleanupQueue } from './utils' describe('CleanupQueue', () => { beforeEach(() => { vi.useFakeTimers() - CleanupQueue.resetInstance() + resetCleanupQueue() }) afterEach(() => { vi.useRealTimers() - CleanupQueue.resetInstance() + resetCleanupQueue() }) it('batches setTimeout creations across multiple synchronous schedules', async () => { diff --git a/packages/db/tests/collection-auto-index.test.ts b/packages/db/tests/collection-auto-index.test.ts index b4e25fdd8f..c295222c6b 100644 --- a/packages/db/tests/collection-auto-index.test.ts +++ b/packages/db/tests/collection-auto-index.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' import { CollectionConfigurationError } from '../src/errors' import { createCollection } from '../src/collection/index.js' import { @@ -250,6 +250,49 @@ describe(`Collection Auto-Indexing`, () => { subscription.unsubscribe() }) + it(`indexes symbol-valued equality fields without falling back to a scan`, async () => { + type SymbolItem = { id: string; group: symbol } + const firstGroup = Symbol(`first`) + const secondGroup = Symbol(`second`) + const symbolRow = createSingleRowRefProxy() + const warning = vi.spyOn(console, `warn`).mockImplementation(() => {}) + const collection = createCollection({ + getKey: (item) => item.id, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `one`, group: firstGroup } }) + write({ type: `insert`, value: { id: `two`, group: secondGroup } }) + commit() + markReady() + }, + }, + }) + + try { + await collection.stateWhenReady() + const changes: Array = [] + const subscription = collection.subscribeChanges( + (items) => changes.push(...items), + { + includeInitialState: true, + whereExpression: eq(symbolRow.group, firstGroup), + }, + ) + + expect(collection.indexes.size).toBe(1) + expect(changes.map(({ value }) => value.id)).toEqual([`one`]) + expect(warning).not.toHaveBeenCalled() + subscription.unsubscribe() + } finally { + warning.mockRestore() + await collection.cleanup() + } + }) + it(`should create auto-indexes for transformed fields of subqueries when autoIndex is "eager"`, async () => {}) it(`should not create duplicate auto-indexes when locale options are omitted`, async () => { diff --git a/packages/db/tests/collection-change-events.test.ts b/packages/db/tests/collection-change-events.test.ts index 085af31f08..dee88c6d59 100644 --- a/packages/db/tests/collection-change-events.test.ts +++ b/packages/db/tests/collection-change-events.test.ts @@ -1,6 +1,9 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' -import { currentStateAsChanges } from '../src/collection/change-events.js' +import { + createFilterFunctionFromExpression, + currentStateAsChanges, +} from '../src/collection/change-events.js' import { Func, PropRef, Value } from '../src/query/ir.js' import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' import { BTreeIndex } from '../src/indexes/btree-index.js' @@ -13,6 +16,26 @@ interface TestUser { status: `active` | `inactive` } +it(`treats predicate evaluation failures as nonmatches`, () => { + const filter = createFilterFunctionFromExpression( + new Func(`eq`, [new PropRef([`status`]), new Value(`active`)]), + ) + const row = { + id: `1`, + name: `Ada`, + age: 36, + score: 100, + status: `active`, + } as TestUser + Object.defineProperty(row, `status`, { + get: () => { + throw new Error(`predicate evaluation failed`) + }, + }) + + expect(filter(row)).toBe(false) +}) + describe(`currentStateAsChanges`, () => { let mockSync: ReturnType diff --git a/packages/db/tests/collection-cleanup-restart-oracle.test.ts b/packages/db/tests/collection-cleanup-restart-oracle.test.ts new file mode 100644 index 0000000000..b131d67c0f --- /dev/null +++ b/packages/db/tests/collection-cleanup-restart-oracle.test.ts @@ -0,0 +1,208 @@ +import { describe, expect, it } from 'vitest' +import { createCollection, createLiveQueryCollection } from '../src' +import type { SyncConfig } from '../src/types' + +type Row = { id: number; rank: number } +const cleanupError = { + name: `CollectionStateError`, + message: expect.stringContaining(`after cleanup() completes`), +} + +const scenarios = ([`abort`, `release`] as const).flatMap((boundary) => + [false, true].flatMap((nestedCleanup) => + [1, 2].map((attempts) => ({ boundary, nestedCleanup, attempts })), + ), +) + +describe(`Collection cleanup admission oracle`, () => { + it.each(scenarios)( + `rejects restart without creating replacement ownership: %j`, + async ({ boundary, nestedCleanup, attempts }) => { + let ops!: Parameters[`sync`]>[0] + let loads = 0 + let releases = 0 + let armed = false + const errors: Array = [] + const cleanups: Array> = [] + const reenter = () => { + if (!armed) return + armed = false + for (let i = 0; i < attempts; i++) { + if (nestedCleanup) cleanups.push(live.cleanup()) + try { + live.startSyncImmediate() + errors.push(undefined) + } catch (error) { + errors.push(error) + } + } + } + const source = createCollection({ + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (methods) => { + ops = methods + methods.begin() + methods.write({ type: `insert`, value: { id: 1, rank: 1 } }) + methods.commit() + methods.markReady() + return { + loadSubset: ({ signal }) => { + loads++ + signal?.addEventListener(`abort`, () => { + if (boundary === `abort`) reenter() + }) + return true + }, + unloadSubset: () => { + releases++ + if (boundary === `release`) reenter() + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => q.from({ row: source })) + try { + await live.preload() + armed = true + await live.cleanup() + await Promise.all(cleanups) + expect(armed).toBe(false) + expect(errors).toHaveLength(attempts) + for (const error of errors) expect(error).toMatchObject(cleanupError) + expect(loads).toBe(1) + expect(releases).toBe(1) + expect(source.subscriberCount).toBe(0) + expect(live.status).toBe(`cleaned-up`) + + // The rejected calls must not poison a later, ordinary restart. + await live.preload() + expect(loads).toBe(2) + expect(source.subscriberCount).toBe(1) + ops.begin() + ops.write({ type: `update`, value: { id: 1, rank: 2 } }) + ops.commit() + expect(live.status).toBe(`ready`) + expect(live.get(1)?.rank).toBe(2) + } finally { + armed = false + await live.cleanup() + await source.cleanup() + } + }, + ) + + it.each([`start`, `preload`] as const)( + `rejects %s from adapter cleanup but allows another collection to start`, + async (method) => { + let starts = 0 + let armed = false + let observed: Promise | undefined + const peer = createCollection({ + getKey: (row) => row.id, + sync: { sync: ({ markReady }) => markReady() }, + }) + const source = createCollection({ + getKey: (row) => row.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + starts++ + begin() + write({ type: `insert`, value: { id: 1, rank: starts } }) + commit() + markReady() + return () => { + if (!armed) return + armed = false + void source.cleanup() + try { + const result = + method === `start` + ? source.startSyncImmediate() + : source.preload() + observed = Promise.resolve(result).then( + () => undefined, + (error: unknown) => error, + ) + } catch (error) { + observed = Promise.resolve(error) + } + peer.startSyncImmediate() + } + }, + }, + }) + try { + await source.preload() + armed = true + await source.cleanup() + expect(await observed).toMatchObject(cleanupError) + expect(starts).toBe(1) + expect(peer.status).toBe(`ready`) + await source.preload() + expect(starts).toBe(2) + expect(source.get(1)?.rank).toBe(2) + } finally { + armed = false + await source.cleanup() + await peer.cleanup() + } + }, + ) + + it.each( + ([`event`, `await`] as const).flatMap((boundary) => + [false, true].map((liveQuery) => ({ boundary, liveQuery })), + ), + )( + `admits restart at the completed cleanup boundary: %j`, + async ({ boundary, liveQuery }) => { + let starts = 0 + let armed = false + let ops!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: (row) => row.id, + sync: { + sync: (methods) => { + ops = methods + const { begin, write, commit, markReady } = methods + starts++ + begin() + write({ type: `insert`, value: { id: 1, rank: starts } }) + commit() + markReady() + }, + }, + }) + const collection = liveQuery + ? createLiveQueryCollection((q) => q.from({ row: source })) + : source + const off = collection.on(`status:change`, ({ status }) => { + if (armed && boundary === `event` && status === `cleaned-up`) { + armed = false + collection.startSyncImmediate() + } + }) + try { + await collection.preload() + armed = true + await collection.cleanup() + if (boundary === `await`) collection.startSyncImmediate() + expect(starts).toBe(liveQuery ? 1 : 2) + expect(collection.status).toBe(`ready`) + expect(collection.get(1)?.rank).toBe(liveQuery ? 1 : 2) + ops.begin() + ops.write({ type: `update`, value: { id: 1, rank: 3 } }) + ops.commit() + expect(collection.get(1)?.rank).toBe(3) + } finally { + armed = false + off() + if (liveQuery) await collection.cleanup() + await source.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/collection-errors.test.ts b/packages/db/tests/collection-errors.test.ts index e0d67963c5..1422e89f9b 100644 --- a/packages/db/tests/collection-errors.test.ts +++ b/packages/db/tests/collection-errors.test.ts @@ -28,6 +28,107 @@ describe(`Collection Error Handling`, () => { }) describe(`Cleanup Error Handling`, () => { + it.each([false, true])( + `finishes adapter resource cleanup after a failure, already released=%s`, + async (releaseBeforeThrow) => { + const resources = new Set() + const failure = new Error(`adapter cleanup interrupted`) + let attempts = 0 + const collection = createCollection<{ id: string }>({ + getKey: (row) => row.id, + sync: { + sync: ({ markReady }) => { + const resource = {} + resources.add(resource) + markReady() + return () => { + attempts++ + if (attempts === 1) { + if (releaseBeforeThrow) resources.delete(resource) + throw failure + } + resources.delete(resource) + } + }, + }, + }) + collection.startSyncImmediate() + try { + expect(resources.size).toBe(1) + await collection.cleanup() + expect(collection.status).toBe(`cleaned-up`) + expect(resources.size).toBe(releaseBeforeThrow ? 0 : 1) + expect(mockQueueMicrotask).toHaveBeenCalledTimes(1) + expect(() => mockQueueMicrotask.mock.calls[0]![0]()).toThrow( + SyncCleanupError, + ) + + // The Collection's public status alone does not prove resource release. + await collection.cleanup() + expect(resources.size).toBe(0) + expect(attempts).toBe(2) + expect(mockQueueMicrotask).toHaveBeenCalledTimes(1) + } finally { + await collection.cleanup() + } + }, + ) + + it.each([false, true])( + `retries failed cleanup only before replacement, restart after rejection=%s`, + async (restart) => { + const failure = new Error(`cleanup failed`) + const cleanups: Array = [] + let session = 0 + const collection = createCollection<{ id: string }>({ + id: `failed-cleanup-session-${restart}`, + getKey: ({ id }) => id, + sync: { + sync: ({ markReady }) => { + const currentSession = session++ + markReady() + return () => { + cleanups.push(currentSession) + if (cleanups.length !== 1) return + if (restart) { + void collection.cleanup() + expect(() => collection.startSyncImmediate()).toThrow( + `after cleanup() completes`, + ) + } + throw failure + } + }, + }, + }) + + collection.startSyncImmediate() + try { + await collection.cleanup() + expect(cleanups).toEqual([0]) + expect(mockQueueMicrotask).toHaveBeenCalledTimes(1) + let reportedError: unknown + try { + mockQueueMicrotask.mock.calls[0]![0]() + } catch (error) { + reportedError = error + } + expect(reportedError).toBeInstanceOf(SyncCleanupError) + expect((reportedError as Error).cause).toBe(failure) + + expect(session).toBe(1) + if (restart) collection.startSyncImmediate() + await collection.cleanup() + expect(cleanups).toEqual(restart ? [0, 1] : [0, 0]) + await collection.cleanup() + expect(cleanups).toHaveLength(2) + expect(mockQueueMicrotask).toHaveBeenCalledTimes(1) + } finally { + await collection.cleanup() + } + }, + ) + it(`should complete cleanup successfully even when sync cleanup function throws an Error`, async () => { const collection = createCollection<{ id: string; name: string }>({ id: `error-test-collection`, @@ -319,9 +420,11 @@ describe(`Collection Error Handling`, () => { }, }) const preload = collection.preload() - + const cancelled = expect(preload).rejects.toMatchObject({ + name: `AbortError`, + }) await collection.cleanup() - await preload + await cancelled markError() expect(collection.status).toBe(`cleaned-up`) @@ -347,9 +450,11 @@ describe(`Collection Error Handling`, () => { const preload = collection.preload() expect(sessions).toHaveLength(1) const first = sessions[0]! - + const cancelled = expect(preload).rejects.toMatchObject({ + name: `AbortError`, + }) await collection.cleanup() - await preload + await cancelled const restartedPreload = collection.preload() expect(sessions).toHaveLength(2) const second = sessions[1]! @@ -379,8 +484,11 @@ describe(`Collection Error Handling`, () => { const firstPreload = collection.preload() const first = sessions[0]! + const cancelled = expect(firstPreload).rejects.toMatchObject({ + name: `AbortError`, + }) await collection.cleanup() - await firstPreload + await cancelled const secondPreload = collection.preload() const second = sessions[1]! diff --git a/packages/db/tests/collection-events.test.ts b/packages/db/tests/collection-events.test.ts index 3e3221b43a..4f55482e2e 100644 --- a/packages/db/tests/collection-events.test.ts +++ b/packages/db/tests/collection-events.test.ts @@ -1,8 +1,19 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' +import { EventEmitter } from '../src/event-emitter.js' import { BTreeIndex } from '../src/indexes/btree-index.js' import type { Collection } from '../src/collection/index.js' +class TestEventEmitter extends EventEmitter<{ event: { id: number } }> { + emit(id: number): void { + this.emitInner(`event`, { id }) + } + + clear(): void { + this.clearListeners() + } +} + describe(`Collection Events System`, () => { let collection: Collection let mockSync: ReturnType @@ -47,6 +58,88 @@ describe(`Collection Events System`, () => { status: `loading`, }) }) + + it(`stops an obsolete status event after a listener changes status`, () => { + const genericEvents: Array<{ + previousStatus: string + status: string + current: string + }> = [] + const loadingEvents: Array = [] + collection.on(`status:change`, ({ status }) => { + if (status === `loading`) collection._lifecycle.markReady() + }) + collection.on(`status:change`, ({ previousStatus, status }) => { + genericEvents.push({ + previousStatus, + status, + current: collection.status, + }) + }) + collection.on(`status:loading`, ({ status }) => { + loadingEvents.push(status) + }) + + collection.startSyncImmediate() + + expect(genericEvents).toEqual([ + { + previousStatus: `loading`, + status: `ready`, + current: `ready`, + }, + ]) + expect(loadingEvents).toEqual([]) + }) + + it.each([`generic`, `specific`] as const)( + `keeps cross-channel order under %s-listener ABA reentry`, + (reentryEvent) => { + const trace: Array = [] + let reentered = false + const reenter = () => { + if (reentered) return + reentered = true + collection._lifecycle.setStatus(`error`) + collection._lifecycle.setStatus(`idle`) + collection._lifecycle.setStatus(`loading`) + } + if (reentryEvent === `generic`) { + collection.on(`status:change`, ({ status }) => { + if (status === `loading`) reenter() + }) + } else { + collection.on(`status:loading`, reenter) + } + collection.on(`status:change`, ({ previousStatus, status }) => { + trace.push( + `generic:${previousStatus}->${status}:${collection.status}`, + ) + }) + collection.on(`status:loading`, () => { + trace.push(`specific:loading:${collection.status}`) + }) + + collection.startSyncImmediate() + + expect(trace).toEqual( + reentryEvent === `generic` + ? [ + `generic:loading->error:error`, + `generic:error->idle:idle`, + `generic:idle->loading:loading`, + `specific:loading:loading`, + ] + : [ + `generic:idle->loading:loading`, + `generic:loading->error:error`, + `generic:error->idle:idle`, + `generic:idle->loading:loading`, + `specific:loading:loading`, + ], + ) + }, + ) }) describe(`Subscriber Count Change Events`, () => { @@ -256,6 +349,169 @@ describe(`Collection Events System`, () => { unsubscribe() }) + + it(`removes a once listener before invoking a throwing callback`, () => { + const emitter = new TestEventEmitter() + const failure = new Error(`once listener failed`) + const deferredMicrotasks: Array = [] + const queueMicrotaskSpy = vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => deferredMicrotasks.push(callback)) + const listener = vi.fn(() => { + throw failure + }) + + try { + emitter.once(`event`, listener) + emitter.emit(1) + emitter.emit(2) + + expect(listener).toHaveBeenCalledTimes(1) + expect(deferredMicrotasks).toHaveLength(1) + expect(() => deferredMicrotasks[0]!()).toThrow(failure) + } finally { + queueMicrotaskSpy.mockRestore() + } + }) + + it(`removes a pending once listener through off`, () => { + const emitter = new TestEventEmitter() + const calls: Array = [] + const onceListener = vi.fn(() => calls.push(`once`)) + emitter.on(`event`, () => { + calls.push(`off`) + emitter.off(`event`, onceListener) + }) + emitter.once(`event`, onceListener) + + emitter.emit(1) + + expect(calls).toEqual([`off`]) + expect(onceListener).not.toHaveBeenCalled() + }) + + it(`removes a pending once listener through its returned unsubscribe`, () => { + const emitter = new TestEventEmitter() + const onceListener = vi.fn() + const unsubscribe = emitter.once(`event`, onceListener) + + unsubscribe() + emitter.emit(1) + + expect(onceListener).not.toHaveBeenCalled() + }) + + it(`removes every pending once registration for the same callback`, () => { + const emitter = new TestEventEmitter() + const onceListener = vi.fn() + emitter.once(`event`, onceListener) + emitter.once(`event`, onceListener) + + emitter.off(`event`, onceListener) + emitter.emit(1) + + expect(onceListener).not.toHaveBeenCalled() + }) + + it(`does not treat an ordinary callback property as a once registration`, () => { + const emitter = new TestEventEmitter() + const claimedOnceCallback = vi.fn() + const ordinaryListener = Object.assign(vi.fn(), { + onceCallback: claimedOnceCallback, + }) + emitter.on(`event`, ordinaryListener) + + emitter.off(`event`, claimedOnceCallback) + emitter.emit(1) + + expect(ordinaryListener).toHaveBeenCalledOnce() + }) + + it(`removes a once listener before a reentrant emission`, () => { + const emitter = new TestEventEmitter() + const observed: Array = [] + emitter.once(`event`, ({ id }) => { + observed.push(id) + emitter.emit(2) + }) + + emitter.emit(1) + + expect(observed).toEqual([1]) + }) + + it(`visits a listener once when it removes and re-adds itself`, () => { + const emitter = new TestEventEmitter() + const observed: Array = [] + let readded = false + let unsubscribe = () => {} + const listener = ({ id }: { id: number }) => { + observed.push(id) + unsubscribe() + if (!readded) { + readded = true + unsubscribe = emitter.on(`event`, listener) + } + } + unsubscribe = emitter.on(`event`, listener) + + emitter.emit(1) + + expect(observed).toEqual([1]) + }) + + it(`defers a pending listener that is removed and re-added`, () => { + const emitter = new TestEventEmitter() + const observed: Array = [] + let replaced = false + const pending = ({ id }: { id: number }) => { + observed.push(`pending:${id}`) + } + let unsubscribePending = () => {} + emitter.on(`event`, ({ id }) => { + observed.push(`first:${id}`) + if (replaced) return + replaced = true + unsubscribePending() + unsubscribePending = emitter.on(`event`, pending) + }) + unsubscribePending = emitter.on(`event`, pending) + + emitter.emit(1) + expect(observed).toEqual([`first:1`]) + + emitter.emit(2) + expect(observed).toEqual([`first:1`, `first:2`, `pending:2`]) + }) + + it(`clears ordinary and once listeners together`, () => { + const emitter = new TestEventEmitter() + const ordinaryListener = vi.fn() + const onceListener = vi.fn() + emitter.on(`event`, ordinaryListener) + emitter.once(`event`, onceListener) + + emitter.clear() + emitter.emit(1) + + expect(ordinaryListener).not.toHaveBeenCalled() + expect(onceListener).not.toHaveBeenCalled() + }) + + it(`exposes the same pending-once removal law through Collection`, () => { + const calls: Array = [] + const onceListener = vi.fn(() => calls.push(`once`)) + collection.on(`status:change`, () => { + calls.push(`off`) + collection.off(`status:change`, onceListener) + }) + collection.once(`status:change`, onceListener) + + collection.startSyncImmediate() + + expect(calls).toEqual([`off`]) + expect(onceListener).not.toHaveBeenCalled() + }) }) describe(`Event Structure`, () => { diff --git a/packages/db/tests/collection-indexes.test.ts b/packages/db/tests/collection-indexes.test.ts index a453b8fb76..0d8062d628 100644 --- a/packages/db/tests/collection-indexes.test.ts +++ b/packages/db/tests/collection-indexes.test.ts @@ -14,11 +14,20 @@ import { or, } from '../src/query/builder/functions' import { PropRef } from '../src/query/ir' +import { BasicIndex } from '../src/indexes/basic-index.js' import { BTreeIndex } from '../src/indexes/btree-index.js' import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' import { findIndexForField } from '../src/utils/index-optimization.js' import { makeComparator } from '../src/utils/comparison.js' -import { expectIndexUsage, stripVirtualProps, withIndexTracking } from './utils' +import { + expectIndexUsage, + indexedKeysSet, + orderedEntriesArray, + orderedEntriesArrayReversed, + stripVirtualProps, + valueMapData, + withIndexTracking, +} from './utils' import type { Collection } from '../src/collection/index.js' import type { MutationFn, PendingMutation } from '../src/types' @@ -152,7 +161,7 @@ describe(`Collection Indexes`, () => { expect(index.id).toBeGreaterThan(0) expect(index.name).toBeUndefined() expect(index.expression.type).toBe(`ref`) - expect(index.indexedKeysSet.size).toBe(5) + expect(indexedKeysSet(index).size).toBe(5) }) it(`should create a named index`, () => { @@ -161,7 +170,7 @@ describe(`Collection Indexes`, () => { }) expect(index.name).toBe(`ageIndex`) - expect(index.indexedKeysSet.size).toBe(5) + expect(indexedKeysSet(index).size).toBe(5) }) it(`should match compare options by collation semantics`, () => { @@ -237,15 +246,15 @@ describe(`Collection Indexes`, () => { const ageIndex = collection.createIndex((row) => row.age) expect(statusIndex.id).not.toBe(ageIndex.id) - expect(statusIndex.indexedKeysSet.size).toBe(5) - expect(ageIndex.indexedKeysSet.size).toBe(5) + expect(indexedKeysSet(statusIndex).size).toBe(5) + expect(indexedKeysSet(ageIndex).size).toBe(5) }) it(`should maintain ordered entries`, () => { const ageIndex = collection.createIndex((row) => row.age) // Ages should be ordered: 22, 25, 28, 30, 35 - const orderedAges = ageIndex.orderedEntriesArray.map(([age]) => age) + const orderedAges = orderedEntriesArray(ageIndex).map(([age]) => age) expect(orderedAges).toEqual([22, 25, 28, 30, 35]) }) @@ -253,10 +262,10 @@ describe(`Collection Indexes`, () => { const statusIndex = collection.createIndex((row) => row.status) // Should have 3 unique status values - expect(statusIndex.orderedEntriesArray.length).toBe(3) + expect(orderedEntriesArray(statusIndex).length).toBe(3) // "active" status should have 3 items - const activeKeys = statusIndex.valueMapData.get(`active`) + const activeKeys = valueMapData(statusIndex).get(`active`) expect(activeKeys?.size).toBe(3) }) @@ -264,10 +273,10 @@ describe(`Collection Indexes`, () => { const scoreIndex = collection.createIndex((row) => row.score) // Should include the item with undefined score - expect(scoreIndex.indexedKeysSet.size).toBe(5) + expect(indexedKeysSet(scoreIndex).size).toBe(5) // undefined should be first in ordered entries - const firstValue = scoreIndex.orderedEntriesArray[0]?.[0] + const firstValue = orderedEntriesArray(scoreIndex)[0]?.[0] expect(firstValue).toBeUndefined() }) }) @@ -1574,6 +1583,143 @@ describe(`Collection Indexes`, () => { expect(ids).toEqual([`1`]) }) + it(`should match symbol range predicates consistently with a full scan`, async () => { + const boundary = Symbol(`boundary`) + const symbolCollection = createCollection< + { id: string; group: symbol }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { id: `1`, group: Symbol(`first`) }, + }) + write({ + type: `insert`, + value: { id: `2`, group: Symbol(`second`) }, + }) + commit() + markReady() + }, + }, + }) + await symbolCollection.stateWhenReady() + + const where = gt(new PropRef([`group`]), boundary) + const scanned = symbolCollection.currentStateAsChanges({ where })! + + symbolCollection.createIndex((row) => row.group) + withIndexTracking(symbolCollection, (tracker) => { + const indexed = symbolCollection.currentStateAsChanges({ where })! + + expect(indexed.map((change) => change.key).sort()).toEqual( + scanned.map((change) => change.key).sort(), + ) + expectIndexUsage(tracker.stats, { + shouldUseIndex: false, + shouldUseFullScan: true, + }) + }) + }) + + it.each( + [BasicIndex, BTreeIndex].flatMap((IndexType) => [ + { + name: `a symbol row under a numeric lower bound`, + IndexType, + rows: [ + { id: `number`, value: 1 as unknown }, + { id: `other`, value: Symbol(`other`) as unknown }, + ], + where: gt(new PropRef([`value`]), 0), + }, + { + name: `an array row under a numeric upper bound`, + IndexType, + rows: [ + { id: `number`, value: 50 as unknown }, + { id: `other`, value: [20] as unknown }, + ], + where: lt(new PropRef([`value`]), 100), + }, + ]), + )( + `should scan mixed domains for $name with $IndexType.name`, + async ({ rows, where, IndexType }) => { + const mixedCollection = createCollection< + { id: string; value: unknown }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: IndexType, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const value of rows) write({ type: `insert`, value }) + commit() + markReady() + }, + }, + }) + await mixedCollection.stateWhenReady() + + const scanned = mixedCollection.currentStateAsChanges({ where })! + mixedCollection.createIndex((row) => row.value) + + withIndexTracking(mixedCollection, (tracker) => { + const indexed = mixedCollection.currentStateAsChanges({ where })! + expect(indexed.map((change) => change.key).sort()).toEqual( + scanned.map((change) => change.key).sort(), + ) + expectIndexUsage(tracker.stats, { + shouldUseIndex: false, + shouldUseFullScan: true, + }) + }) + }, + ) + + it(`should retain every row whose index values share one comparator position`, async () => { + const shared = Symbol(`shared`) + const groupedCollection = createCollection< + { id: string; value: Array }, + string + >({ + getKey: (row) => row.id, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `first`, value: [shared] } }) + write({ type: `insert`, value: { id: `second`, value: [shared] } }) + commit() + markReady() + }, + }, + }) + await groupedCollection.stateWhenReady() + + const index = groupedCollection.createIndex((row) => row.value) + + expect(index.takeFromStart(2)).toEqual([`first`, `second`]) + expect(orderedEntriesArray(index)[0]?.[1]).toEqual( + new Set([`first`, `second`]), + ) + expect(orderedEntriesArrayReversed(index)[0]?.[1]).toEqual( + new Set([`first`, `second`]), + ) + }) + it(`should return all matching rows for a range predicate on a custom-comparator index`, async () => { // A range predicate must return every row that satisfies it regardless // of the comparator the index was created with. With scores 5 and 20, @@ -2103,11 +2249,11 @@ describe(`Collection Indexes`, () => { const ageIndex = specialCollection.createIndex((row) => row.age) // Verify index contains all items including special values - expect(ageIndex.indexedKeysSet.size).toBe(8) // Original 5 + 3 special - expect(ageIndex.orderedEntriesArray).toHaveLength(8) // 8 unique age values (including null) + expect(indexedKeysSet(ageIndex).size).toBe(8) // Original 5 + 3 special + expect(orderedEntriesArray(ageIndex)).toHaveLength(8) // 8 unique age values (including null) // Null/undefined should be ordered first - const firstValue = ageIndex.orderedEntriesArray[0]?.[0] + const firstValue = orderedEntriesArray(ageIndex)[0]?.[0] expect(firstValue == null).toBe(true) // Test that queries with special values use indexes correctly @@ -2151,17 +2297,17 @@ describe(`Collection Indexes`, () => { const index = emptyCollection.createIndex((row) => row.age) - expect(index.indexedKeysSet.size).toBe(0) - expect(index.orderedEntriesArray).toHaveLength(0) - expect(index.valueMapData.size).toBe(0) + expect(indexedKeysSet(index).size).toBe(0) + expect(orderedEntriesArray(index)).toHaveLength(0) + expect(valueMapData(index).size).toBe(0) }) it(`should handle index updates when data changes through sync`, async () => { const ageIndex = collection.createIndex((row) => row.age) // Original index should have 5 items - expect(ageIndex.indexedKeysSet.size).toBe(5) - expect(ageIndex.orderedEntriesArray).toHaveLength(5) + expect(indexedKeysSet(ageIndex).size).toBe(5) + expect(orderedEntriesArray(ageIndex)).toHaveLength(5) // Perform mutations that will sync back and update indexes const tx1 = createTransaction({ mutationFn }) @@ -2195,7 +2341,7 @@ describe(`Collection Indexes`, () => { await new Promise((resolve) => setTimeout(resolve, 10)) // Verify that indexes are updated after sync - expect(ageIndex.indexedKeysSet.size).toBe(5) // 5 original - 1 deleted + 1 inserted + expect(indexedKeysSet(ageIndex).size).toBe(5) // 5 original - 1 deleted + 1 inserted // Test that index-optimized queries work with the updated data withIndexTracking(collection, (tracker) => { diff --git a/packages/db/tests/collection-lifecycle.test.ts b/packages/db/tests/collection-lifecycle.test.ts index a5cf03f19e..7779c08e75 100644 --- a/packages/db/tests/collection-lifecycle.test.ts +++ b/packages/db/tests/collection-lifecycle.test.ts @@ -1,12 +1,151 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' import { CleanupQueue } from '../src/collection/cleanup-queue.js' +import { InvalidCollectionStatusTransitionError } from '../src/errors.js' +import { + getActivePublicationContext, + transactionScopedScheduler, + withPublicationContext, +} from '../src/scheduler.js' +import { resetCleanupQueue } from './utils' // Mock setTimeout and clearTimeout for testing GC behavior const originalSetTimeout = global.setTimeout const originalClearTimeout = global.clearTimeout +function getChangesManager(collection: object): { + emitEmptyReadyEvent: () => void +} { + return ( + collection as unknown as { + _changes: { emitEmptyReadyEvent: () => void } + } + )._changes +} + describe(`Collection Lifecycle Management`, () => { + it.each( + ([`same`, `missing`, `changed`, `empty`] as const).flatMap((shape) => + ([`atomic`, `split`] as const).map((delivery) => ({ shape, delivery })), + ), + )( + `keeps eager restart messages coherent for $shape keys with $delivery commits`, + async ({ shape, delivery }) => { + type Row = { id: string; version: number } + let rows: Array = [ + { id: `a`, version: 1 }, + { id: `b`, version: 1 }, + ] + const collection = createCollection({ + getKey: ({ id }) => id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + const batches = + delivery === `atomic` ? [rows] : rows.map((row) => [row]) + for (const batch of batches) { + begin() + for (const value of batch) write({ type: `insert`, value }) + commit() + } + markReady() + }, + }, + }) + await collection.preload() + const delivered = new Map() + const read = () => + collection.toArray.map(({ id, version }) => ({ id, version })) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) { + expect(delivered.get(change.key)).toEqual({ + id: change.value.id, + version: change.value.version, + }) + delivered.delete(change.key) + } else { + if (change.type === `insert`) + expect(delivered.has(change.key)).toBe(false) + else + expect(change.previousValue).toMatchObject( + delivered.get(change.key)!, + ) + delivered.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + expect([...delivered.values()]).toEqual(read()) + }, + { includeInitialState: true }, + ) + try { + expect([...delivered.values()]).toEqual(rows) + await collection.cleanup() + rows = + shape === `empty` + ? [] + : shape === `changed` + ? [ + { id: `c`, version: 2 }, + { id: `d`, version: 2 }, + ] + : shape === `missing` + ? [{ id: `a`, version: 2 }] + : [ + { id: `a`, version: 2 }, + { id: `b`, version: 2 }, + ] + await collection.preload() + expect(collection.status).toBe(`ready`) + expect([...delivered.values()]).toEqual(rows) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each([`pending`, `starting`, `ready`, `failed`] as const)( + `cleanup settles a %s preload without inventing first readiness`, + async (phase) => { + let starts = 0 + const failure = new Error(`initial failure`) + const ready = vi.fn() + const collection = createCollection<{ id: string }>({ + getKey: ({ id }) => id, + sync: { + sync: ({ collection: source, markReady, markError }) => { + starts++ + if (starts > 1 || phase === `ready`) markReady() + else if (phase === `failed`) markError(failure) + else if (phase === `starting`) void source.cleanup() + }, + }, + }) + collection.onFirstReady(ready) + const preload = collection.preload().then( + () => undefined, + (error: unknown) => error, + ) + if (phase !== `starting`) await collection.cleanup() + const result = await preload + if (phase === `ready`) expect(result).toBeUndefined() + else if (phase === `failed`) expect(result).toBe(failure) + else expect(result).toMatchObject({ name: `AbortError` }) + expect(ready).toHaveBeenCalledTimes(phase === `ready` ? 1 : 0) + const restartedReady = vi.fn() + collection.onFirstReady(restartedReady) + await collection.preload() + expect(starts).toBe(2) + expect(restartedReady).toHaveBeenCalledOnce() + expect(ready).toHaveBeenCalledTimes(phase === `ready` ? 1 : 0) + await collection.cleanup() + }, + ) + let mockSetTimeout: ReturnType let mockClearTimeout: ReturnType let timeoutCallbacks: Map void> @@ -47,7 +186,7 @@ describe(`Collection Lifecycle Management`, () => { global.setTimeout = originalSetTimeout global.clearTimeout = originalClearTimeout vi.clearAllMocks() - CleanupQueue.resetInstance() + resetCleanupQueue() }) const triggerAllTimeouts = () => { @@ -133,6 +272,38 @@ describe(`Collection Lifecycle Management`, () => { expect(collection.status).toBe(`cleaned-up`) }) + it(`clears terminal state without publishing one delete per row`, async () => { + const collection = createCollection<{ id: number; name: string }>({ + id: `cleanup-without-row-publication`, + getKey: (item) => item.id, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (let id = 0; id < 100; id++) { + write({ type: `insert`, value: { id, name: `row-${id}` } }) + } + commit() + markReady() + }, + }, + }) + const onChanges = vi.fn() + const subscription = collection.subscribeChanges(onChanges, { + includeInitialState: false, + }) + + try { + await collection.cleanup() + + expect(collection.toArray).toEqual([]) + expect(onChanges).not.toHaveBeenCalled() + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + it(`should transition when subscribing to changes`, () => { let beginCallback: (() => void) | undefined let commitCallback: (() => void) | undefined @@ -511,6 +682,919 @@ describe(`Collection Lifecycle Management`, () => { subscription.unsubscribe() }) + it(`freezes first-ready callback membership before delivery`, async () => { + let markReadyCallback: (() => void) | undefined + const calls: Array = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `first-ready-membership-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}) + let removeLater = () => {} + + collection.onFirstReady(() => { + calls.push(`first`) + removeLater() + collection.onFirstReady(() => calls.push(`nested`)) + }) + removeLater = collection.onFirstReady(() => calls.push(`later`)) + + try { + markReadyCallback!() + + expect(calls).toEqual([`first`, `nested`, `later`]) + + collection.onFirstReady(() => calls.push(`after`)) + expect(calls).toEqual([`first`, `nested`, `later`, `after`]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + expect(calls).toEqual([`first`, `nested`, `later`, `after`]) + } + }) + + it.each([ + { + from: `ready`, + expectedStatus: `ready`, + expectedFirstReadyCalls: 1, + invalid: false, + }, + { + from: `error`, + expectedStatus: `ready`, + expectedFirstReadyCalls: 1, + invalid: false, + }, + { + from: `idle`, + expectedStatus: `idle`, + expectedFirstReadyCalls: 0, + invalid: true, + }, + { + from: `cleaned-up`, + expectedStatus: `cleaned-up`, + expectedFirstReadyCalls: 0, + invalid: true, + }, + ] as const)( + `defines the $from -> ready transition`, + async ({ from, expectedStatus, expectedFirstReadyCalls, invalid }) => { + const syncFailure = new Error(`sync failed before recovery`) + let firstReadyCalls = 0 + let recoveryFirstReadyCalls = 0 + const collection = createCollection<{ id: string; name: string }>({ + id: `mark-ready-from-${from}`, + getKey: (item) => item.id, + startSync: false, + sync: { sync: () => {} }, + }) + collection.onFirstReady(() => { + firstReadyCalls++ + }) + + if (from === `ready` || from === `error`) { + collection._lifecycle.setStatus(`loading`) + collection._lifecycle.markReady() + } + if (from === `error`) { + collection._lifecycle.markError(syncFailure) + expect(collection._lifecycle.getSyncError()).toBe(syncFailure) + } else if (from === `cleaned-up`) { + collection._lifecycle.setStatus(`cleaned-up`) + } + expect(collection.status).toBe(from) + + if (from === `error`) { + collection.onFirstReady(() => { + recoveryFirstReadyCalls++ + }) + expect(recoveryFirstReadyCalls).toBe(1) + } + + const transitionTrace: Array< + | { + kind: `status` + previousStatus: string + status: string + syncError: unknown + } + | { + kind: `dependent-ready` + status: string + syncError: unknown + } + > = [] + collection.on(`status:change`, ({ previousStatus, status }) => { + transitionTrace.push({ + kind: `status`, + previousStatus, + status, + syncError: collection._lifecycle.getSyncError(), + }) + }) + const changes = getChangesManager(collection) + const originalEmitEmptyReadyEvent = + changes.emitEmptyReadyEvent.bind(changes) + vi.spyOn(changes, `emitEmptyReadyEvent`).mockImplementation(() => { + transitionTrace.push({ + kind: `dependent-ready`, + status: collection.status, + syncError: collection._lifecycle.getSyncError(), + }) + originalEmitEmptyReadyEvent() + }) + + let didThrow = false + let thrown: unknown + try { + collection._lifecycle.markReady() + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(invalid) + if (invalid) { + expect(thrown).toBeInstanceOf(InvalidCollectionStatusTransitionError) + expect((thrown as Error).message).toBe( + `Invalid collection status transition from "${from}" to "ready" for collection "mark-ready-from-${from}"`, + ) + } + expect(collection.status).toBe(expectedStatus) + expect(firstReadyCalls).toBe(expectedFirstReadyCalls) + expect(recoveryFirstReadyCalls).toBe(from === `error` ? 1 : 0) + expect(transitionTrace).toEqual( + from === `error` + ? [ + { + kind: `status`, + previousStatus: `error`, + status: `ready`, + syncError: undefined, + }, + { + kind: `dependent-ready`, + status: `ready`, + syncError: undefined, + }, + ] + : [], + ) + expect(collection._lifecycle.getSyncError()).toBeUndefined() + + await collection.cleanup() + }, + ) + + it(`does not resume ready effects after a status listener cleans up`, () => { + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-listener-cleanup-test`, + getKey: (item) => item.id, + startSync: false, + sync: { sync: () => {} }, + }) + const readyEvent = vi.spyOn( + getChangesManager(collection), + `emitEmptyReadyEvent`, + ) + const firstReadyStatuses: Array = [] + collection.onFirstReady(() => { + firstReadyStatuses.push(collection.status) + }) + collection.on(`status:ready`, () => { + void collection.cleanup() + }) + collection._lifecycle.setStatus(`loading`) + + collection._lifecycle.markReady() + + expect(collection.status).toBe(`cleaned-up`) + expect(collection._lifecycle.hasBeenReady).toBe(false) + expect(firstReadyStatuses).toEqual([]) + expect(readyEvent).not.toHaveBeenCalled() + + const laterFirstReady = vi.fn() + const removeLater = collection.onFirstReady(laterFirstReady) + expect(laterFirstReady).not.toHaveBeenCalled() + removeLater() + }) + + it(`does not resume ready effects after a status listener enters error`, async () => { + const failure = new Error(`ready listener failed the sync`) + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-listener-error-test`, + getKey: (item) => item.id, + startSync: false, + sync: { sync: () => {} }, + }) + const readyEvent = vi.spyOn( + getChangesManager(collection), + `emitEmptyReadyEvent`, + ) + const firstReady = vi.fn() + collection.onFirstReady(firstReady) + collection.on(`status:ready`, () => { + collection._lifecycle.markError(failure) + }) + collection._lifecycle.setStatus(`loading`) + + collection._lifecycle.markReady() + + expect(collection.status).toBe(`error`) + expect(collection._lifecycle.getSyncError()).toBe(failure) + expect(collection._lifecycle.hasBeenReady).toBe(false) + expect(firstReady).not.toHaveBeenCalled() + expect(readyEvent).not.toHaveBeenCalled() + await collection.cleanup() + }) + + it(`does not resume an outer ready transition after a synchronous restart`, async () => { + let syncStarts = 0 + let restartedPreload: Promise | undefined + let restartOnce = true + let lateSubscription: { unsubscribe: () => void } | undefined + const lateReadyBatches: Array> = [] + const firstReadyStatuses: Array = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-listener-aba-test`, + getKey: (item) => item.id, + startSync: false, + sync: { + sync: ({ markReady }) => { + syncStarts++ + markReady() + }, + }, + }) + const readyEvent = vi.spyOn( + getChangesManager(collection), + `emitEmptyReadyEvent`, + ) + collection.onFirstReady(() => { + firstReadyStatuses.push(collection.status) + }) + collection.on(`status:ready`, () => { + if (!restartOnce) return + restartOnce = false + void collection.cleanup() + restartedPreload = collection.preload() + lateSubscription = collection.subscribeChanges((batch) => { + lateReadyBatches.push(batch) + }) + }) + collection._lifecycle.setStatus(`loading`) + + collection._lifecycle.markReady() + await restartedPreload + + expect(syncStarts).toBe(1) + expect(collection.status).toBe(`ready`) + expect(firstReadyStatuses).toEqual([]) + expect(lateReadyBatches).toEqual([]) + expect(readyEvent).toHaveBeenCalledOnce() + lateSubscription!.unsubscribe() + await collection.cleanup() + }) + + it(`starts a fresh first-ready cycle after cleanup of a failed ready effect`, async () => { + const readyCallbacks: Array<() => void> = [] + const firstFailure = new Error(`first ready cycle failed exactly`) + const trace: Array = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-effect-restart-test`, + getKey: (item) => item.id, + startSync: false, + sync: { + sync: ({ markReady }) => { + readyCallbacks.push(markReady) + }, + }, + }) + const readyEvent = vi.spyOn( + getChangesManager(collection), + `emitEmptyReadyEvent`, + ) + collection.onFirstReady(() => { + trace.push(`first failure:${collection.status}`) + throw firstFailure + }) + collection.onFirstReady(() => { + trace.push(`first later:${collection.status}`) + }) + const firstPreload = collection.preload() + let firstPreloadSettled = false + void firstPreload.then(() => { + firstPreloadSettled = true + }) + await Promise.resolve() + expect(collection.status).toBe(`loading`) + expect(firstPreloadSettled).toBe(false) + + let thrown: unknown + try { + readyCallbacks[0]!() + } catch (error) { + thrown = error + } + expect(thrown).toBe(firstFailure) + await expect(firstPreload).resolves.toBeUndefined() + expect(trace).toEqual([`first failure:ready`, `first later:ready`]) + expect(readyEvent).toHaveBeenCalledOnce() + + await collection.cleanup() + expect(collection.status).toBe(`cleaned-up`) + expect(collection._lifecycle.hasBeenReady).toBe(false) + + collection.onFirstReady(() => { + trace.push(`second:${collection.status}`) + }) + expect(trace).toEqual([`first failure:ready`, `first later:ready`]) + + const secondPreload = collection.preload() + let secondPreloadSettled = false + void secondPreload.then(() => { + secondPreloadSettled = true + }) + expect(secondPreload).not.toBe(firstPreload) + expect(readyCallbacks).toHaveLength(2) + await Promise.resolve() + expect(collection.status).toBe(`loading`) + expect(secondPreloadSettled).toBe(false) + + readyCallbacks[0]!() + await Promise.resolve() + expect(collection.status).toBe(`loading`) + expect(secondPreloadSettled).toBe(false) + expect(trace).toEqual([`first failure:ready`, `first later:ready`]) + expect(readyEvent).toHaveBeenCalledOnce() + + readyCallbacks[1]!() + await expect(secondPreload).resolves.toBeUndefined() + expect(secondPreloadSettled).toBe(true) + + expect(trace).toEqual([ + `first failure:ready`, + `first later:ready`, + `second:ready`, + ]) + expect(readyEvent).toHaveBeenCalledTimes(2) + + await collection.cleanup() + }) + + it(`attempts every first-ready effect before rethrowing the first failure`, async () => { + let markReadyCallback: (() => void) | undefined + const readyBatches: Array> = [] + const readyTrace: Array = [] + const laterFailure = new Error(`later first-ready failure`) + const laterCallback = vi.fn(() => { + readyTrace.push(`later:${collection.status}`) + throw laterFailure + }) + let preloadSettled = false + + const collection = createCollection<{ id: string; name: string }>({ + id: `first-ready-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady as () => void + }, + }, + }) + const subscription = collection.subscribeChanges((batch) => { + readyTrace.push(`dependent:${collection.status}`) + readyBatches.push(batch) + }) + collection.onFirstReady(() => { + readyTrace.push(`first:${collection.status}`) + throw undefined + }) + collection.onFirstReady(laterCallback) + void collection.preload().then(() => { + preloadSettled = true + }) + + try { + let didThrow = false + let thrown: unknown + try { + markReadyCallback!() + } catch (error) { + didThrow = true + thrown = error + } + await Promise.resolve() + + expect(didThrow).toBe(true) + expect(thrown).toBeUndefined() + expect(laterCallback).toHaveBeenCalledOnce() + expect(preloadSettled).toBe(true) + expect(readyBatches).toEqual([[]]) + expect(readyTrace).toEqual([ + `first:ready`, + `later:ready`, + `dependent:ready`, + ]) + expect(collection.status).toBe(`ready`) + + expect(() => markReadyCallback!()).not.toThrow() + expect(laterCallback).toHaveBeenCalledOnce() + expect(readyBatches).toEqual([[]]) + expect(readyTrace).toEqual([ + `first:ready`, + `later:ready`, + `dependent:ready`, + ]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not classify synchronous first-ready callback failures as sync failures`, async () => { + const laterFailure = new Error(`later synchronous first-ready failure`) + const callbackTrace: Array = [] + let syncContinued = false + + const collection = createCollection<{ id: string; name: string }>({ + id: `synchronous-first-ready-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReady() + syncContinued = true + }, + }, + }) + collection.onFirstReady(() => { + callbackTrace.push(`first`) + throw undefined + }) + collection.onFirstReady(() => { + callbackTrace.push(`later`) + throw laterFailure + }) + + try { + let didThrow = false + let thrown: unknown + try { + collection._sync.startSync() + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(thrown).toBeUndefined() + expect(syncContinued).toBe(true) + expect(callbackTrace).toEqual([`first`, `later`]) + expect(collection.status).toBe(`ready`) + await expect(collection.preload()).resolves.toBeUndefined() + } finally { + await collection.cleanup() + } + }) + + it(`rejects a pending preload when the adapter fails after marking ready`, async () => { + const adapterFailure = new Error(`adapter failed after ready`) + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-then-adapter-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReady() + throw adapterFailure + }, + }, + }) + collection.onFirstReady(() => { + throw undefined + }) + + try { + await expect(collection.preload()).rejects.toBe(adapterFailure) + expect(collection.status).toBe(`error`) + } finally { + await collection.cleanup() + } + }) + + it(`ends the synchronous sync-entry boundary after an adapter failure`, async () => { + const adapterFailure = new Error(`adapter entry failed`) + let markReadyCallback: (() => void) | undefined + const collection = createCollection<{ id: string; name: string }>({ + id: `failed-sync-entry-boundary-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + throw adapterFailure + }, + }, + }) + collection.onFirstReady(() => { + throw undefined + }) + + try { + expect(() => collection._sync.startSync()).toThrow(adapterFailure) + expect(collection.status).toBe(`error`) + + let didThrow = false + let thrown: unknown + try { + markReadyCallback!() + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(thrown).toBeUndefined() + expect(collection.status).toBe(`ready`) + } finally { + await collection.cleanup() + } + }) + + it(`attempts every dependent ready listener before rethrowing`, async () => { + let markReadyCallback: (() => void) | undefined + const firstFailure = new Error(`first dependent failed`) + const firstBatches: Array> = [] + const secondBatches: Array> = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady as () => void + }, + }, + }) + const first = collection.subscribeChanges((batch) => { + firstBatches.push(batch) + throw firstFailure + }) + const second = collection.subscribeChanges((batch) => { + secondBatches.push(batch) + }) + + try { + let thrown: unknown + try { + markReadyCallback!() + } catch (error) { + thrown = error + } + + expect(thrown).toBe(firstFailure) + expect(firstBatches).toEqual([[]]) + expect(secondBatches).toEqual([[]]) + expect(collection.status).toBe(`ready`) + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`flushes work queued by a ready listener when a sibling throws`, async () => { + let markReadyCallback: (() => void) | undefined + const firstFailure = new Error(`dependent failed after sibling queued`) + const scheduledJob = vi.fn() + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-scheduler-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + const contextId = getActivePublicationContext() + expect(contextId).toBeDefined() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + const second = collection.subscribeChanges(() => { + throw firstFailure + }) + + try { + expect(() => markReadyCallback!()).toThrow(firstFailure) + expect(scheduledJob).toHaveBeenCalledOnce() + expect(collection.status).toBe(`ready`) + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`flushes ready work before rethrowing at an outer publication boundary`, async () => { + let markReadyCallback: (() => void) | undefined + const listenerFailure = new Error(`nested dependent failed`) + const scheduledJob = vi.fn() + const collection = createCollection<{ id: string; name: string }>({ + id: `nested-dependent-ready-scheduler-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + const second = collection.subscribeChanges(() => { + throw listenerFailure + }) + + try { + expect(() => + withPublicationContext(() => markReadyCallback!()), + ).toThrow(listenerFailure) + expect(scheduledJob).toHaveBeenCalledOnce() + expect(collection.status).toBe(`ready`) + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`preserves a falsy ready failure through a nested publication`, async () => { + let markReadyCallback: (() => void) | undefined + const scheduledJob = vi.fn() + const collection = createCollection<{ id: string; name: string }>({ + id: `nested-falsy-ready-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + const second = collection.subscribeChanges(() => { + throw undefined + }) + + try { + let didThrow = false + let thrown: unknown + try { + withPublicationContext(() => markReadyCallback!()) + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(thrown).toBeUndefined() + expect(scheduledJob).toHaveBeenCalledOnce() + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`surfaces a ready graph failure after running its job`, async () => { + let markReadyCallback: (() => void) | undefined + const graphFailure = new Error(`ready graph failed`) + const scheduledJob = vi.fn(() => { + throw graphFailure + }) + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-graph-failure-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const subscription = collection.subscribeChanges(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + + try { + expect(() => markReadyCallback!()).toThrow(graphFailure) + expect(scheduledJob).toHaveBeenCalledOnce() + expect(collection.status).toBe(`ready`) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`keeps the ready listener failure when its queued graph job also fails`, async () => { + let markReadyCallback: (() => void) | undefined + const listenerFailure = new Error(`ready listener failed first`) + const graphFailure = new Error(`ready graph also failed`) + const scheduledJob = vi.fn(() => { + throw graphFailure + }) + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-failure-priority-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + const second = collection.subscribeChanges(() => { + throw listenerFailure + }) + + try { + expect(() => markReadyCallback!()).toThrow(listenerFailure) + expect(scheduledJob).toHaveBeenCalledOnce() + expect(collection.status).toBe(`ready`) + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`resolves a pending preload after a ready callback failure alone`, async () => { + let syncContinued = false + const collection = createCollection<{ id: string; name: string }>({ + id: `ready-callback-preload-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReady() + syncContinued = true + }, + }, + }) + collection.onFirstReady(() => { + throw undefined + }) + + try { + await expect(collection.preload()).resolves.toBeUndefined() + expect(syncContinued).toBe(true) + expect(collection.status).toBe(`ready`) + } finally { + await collection.cleanup() + } + }) + + it(`skips a ready listener unsubscribed during the same delivery`, async () => { + let markReadyCallback: (() => void) | undefined + const calls: Array = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-membership-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + calls.push(`first`) + second.unsubscribe() + }) + const second = collection.subscribeChanges(() => { + calls.push(`second`) + }) + + try { + markReadyCallback!() + expect(calls).toEqual([`first`]) + } finally { + first.unsubscribe() + second.unsubscribe() + await collection.cleanup() + } + }) + + it(`excludes a dependent added during ready delivery until the next batch`, async () => { + let beginCallback: (() => void) | undefined + let writeCallback: + | ((message: { + type: `insert` + value: { id: string; name: string } + }) => void) + | undefined + let commitCallback: (() => void) | undefined + let markReadyCallback: (() => void) | undefined + let added: { unsubscribe: () => void } | undefined + const calls: Array = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `dependent-ready-addition-test`, + getKey: (item) => item.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + beginCallback = begin + writeCallback = write + commitCallback = () => { + commit() + } + markReadyCallback = markReady + }, + }, + }) + const first = collection.subscribeChanges(() => { + calls.push(`first`) + added ??= collection.subscribeChanges(() => calls.push(`added`)) + }) + const second = collection.subscribeChanges(() => calls.push(`second`)) + + try { + markReadyCallback!() + expect(calls).toEqual([`first`, `second`]) + + beginCallback!() + writeCallback!({ + type: `insert`, + value: { id: `one`, name: `One` }, + }) + commitCallback!() + expect(calls).toEqual([`first`, `second`, `first`, `second`, `added`]) + } finally { + first.unsubscribe() + second.unsubscribe() + added?.unsubscribe() + await collection.cleanup() + } + }) + + it(`notifies a dependent added during the first-ready fan-out`, async () => { + let markReadyCallback: (() => void) | undefined + let dependent: { unsubscribe: () => void } | undefined + const readyBatches: Array> = [] + const collection = createCollection<{ id: string; name: string }>({ + id: `nested-dependent-ready-test`, + getKey: (item) => item.id, + sync: { + sync: ({ markReady }) => { + markReadyCallback = markReady as () => void + }, + }, + }) + collection.onFirstReady(() => { + dependent = collection.subscribeChanges((batch) => { + readyBatches.push(batch) + }) + }) + const preload = collection.preload() + + try { + markReadyCallback!() + await preload + expect(readyBatches).toEqual([[]]) + } finally { + dependent?.unsubscribe() + await collection.cleanup() + } + }) + it(`should fire status:change event with 'cleaned-up' status before clearing event handlers`, () => { const collection = createCollection<{ id: string; name: string }>({ id: `cleanup-event-test`, diff --git a/packages/db/tests/collection-metadata-publication-oracle.property.test.ts b/packages/db/tests/collection-metadata-publication-oracle.property.test.ts new file mode 100644 index 0000000000..2da3765008 --- /dev/null +++ b/packages/db/tests/collection-metadata-publication-oracle.property.test.ts @@ -0,0 +1,535 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { SyncTransactionAbortedError } from '../src/errors.js' +import { createLiveQueryCollection } from '../src/query/index.js' +import { createTransaction } from '../src/transactions.js' +import { oraclePropertyOptions } from './oracle-config.js' +import type { Collection } from '../src/collection/index.js' +import type { ChangeMessage, SyncConfig } from '../src/types.js' + +type PublicationRow = { + id: number + position: number +} + +type SyncActions = Parameters[`sync`]>[0] + +type MetadataOperation = { type: `set`; value: unknown } | { type: `delete` } + +type MetadataEntryState = { present: false } | { present: true; value: unknown } + +type MetadataWrite = { key: number } & MetadataOperation + +type PublicationRound = { + key: number + delta: number + metadata: ReadonlyArray + outcome: `commit` | `abort` +} + +type ReadablePublicationCollection = { + values: () => IterableIterator + cleanup: () => Promise +} + +type PublicationHarness = { + rows: Collection + liveRows: ReadablePublicationCollection + batches: Array>> + unsubscribe: () => void + getSync: () => SyncActions +} + +type PublishedPublicationRow = PublicationRow & { + $collectionId: string + $key: number + $origin: `local` | `remote` + $synced: boolean +} + +const metadataValueArbitrary = fc.oneof( + fc.constant(undefined), + fc.constant(null), + fc.constant(false), + fc.constant(true), + fc.constant(0), + fc.constant(Number.NaN), + fc.constant(``), + fc.integer(), + fc.string(), + fc.record({ nested: fc.integer() }), +) + +const metadataOperationArbitrary: fc.Arbitrary = fc.oneof( + metadataValueArbitrary.map((value) => ({ type: `set` as const, value })), + fc.constant({ type: `delete` as const }), +) + +const metadataEntryStateArbitrary: fc.Arbitrary = fc.oneof( + fc.constant({ present: false as const }), + metadataValueArbitrary.map((value) => ({ + present: true as const, + value, + })), +) + +const metadataWriteArbitrary = fc + .tuple(fc.integer({ min: 0, max: 2 }), metadataOperationArbitrary) + .map(([key, operation]) => ({ key, ...operation })) + +const publicationRoundArbitrary: fc.Arbitrary = fc + .record({ + key: fc.integer({ min: 0, max: 2 }), + delta: fc.constantFrom(-2, -1, 1, 2), + extraMetadata: fc.array(metadataWriteArbitrary, { maxLength: 2 }), + outcome: fc.constantFrom(`commit` as const, `abort` as const), + primaryMetadata: metadataOperationArbitrary, + }) + .map(({ key, delta, extraMetadata, outcome, primaryMetadata }) => ({ + key, + delta, + outcome, + metadata: [{ key, ...primaryMetadata }, ...extraMetadata], + })) + +const metadataCancellationArbitrary = fc.record({ + canceledKeys: fc.uniqueArray(fc.integer({ min: 0, max: 2 }), { + minLength: 1, + maxLength: 3, + }), + retainedKeys: fc.uniqueArray(fc.integer({ min: 0, max: 2 }), { + minLength: 1, + maxLength: 3, + }), + canceledOperation: metadataOperationArbitrary, + retainedOperation: metadataOperationArbitrary, + canceledFirst: fc.boolean(), + initialMetadata: fc.array(metadataEntryStateArbitrary, { + minLength: 3, + maxLength: 3, + }), +}) + +async function createPublicationHarness(): Promise { + let sync!: SyncActions + const rows = createCollection({ + id: `metadata-publication-source`, + getKey: (row) => row.id, + startSync: true, + sync: { + sync: (actions) => { + sync = actions + actions.begin() + for (let id = 0; id < 3; id++) { + actions.write({ type: `insert`, value: { id, position: id } }) + } + actions.commit() + actions.markReady() + }, + }, + }) + const liveRows = createLiveQueryCollection((query) => + query.from({ row: rows }), + ) + await liveRows.preload() + + const batches: Array>> = + [] + const subscription = rows.subscribeChanges((changes) => { + batches.push(changes) + }) + return { + rows, + liveRows, + batches, + unsubscribe: () => subscription.unsubscribe(), + getSync: () => sync, + } +} + +function expectUniqueBatchKeys( + batches: ReadonlyArray< + ReadonlyArray> + >, +): void { + for (const batch of batches) { + const keys = batch.map((change) => change.key) + expect(keys).toEqual([...new Set(keys)]) + } +} + +function selectPublishedRow( + row: PublicationRow | undefined, +): PublishedPublicationRow | undefined { + if (row === undefined) return undefined + const published = row as PublishedPublicationRow + return { + id: published.id, + position: published.position, + $collectionId: published.$collectionId, + $key: published.$key, + $origin: published.$origin, + $synced: published.$synced, + } +} + +function selectPublishedChange( + change: ChangeMessage, +) { + return { + type: change.type, + key: change.key, + value: selectPublishedRow(change.value), + previousValue: selectPublishedRow(change.previousValue), + } +} + +function expectPublishedRows( + harness: PublicationHarness, + model: ReadonlyMap, +): void { + const expected = [...model.values()].sort((a, b) => a.id - b.id) + const selectBaseRows = (collection: ReadablePublicationCollection) => + [...collection.values()] + .map((row) => ({ id: row.id, position: row.position })) + .sort((a, b) => a.id - b.id) + + expect(selectBaseRows(harness.rows)).toEqual(expected) + expect(selectBaseRows(harness.liveRows)).toEqual(expected) +} + +function readMetadata( + harness: PublicationHarness, + keys: Iterable, +): Map { + const metadata = harness.getSync().metadata!.row + return new Map([...keys].map((key) => [key, metadata.get(key)])) +} + +function observableMetadata( + model: ReadonlyMap, + keys: Iterable, +): Map { + return new Map([...keys].map((key) => [key, model.get(key)])) +} + +async function applyRound( + harness: PublicationHarness, + round: PublicationRound, + model: Map, + metadataModel: Map, +): Promise { + const previous = model.get(round.key)! + const next = { ...previous, position: previous.position + round.delta } + const batchCountBefore = harness.batches.length + const keyWasPreviouslyPublished = harness.batches.some((batch) => + batch.some((change) => change.key === round.key), + ) + const sync = harness.getSync() + const transaction = createTransaction({ + mutationFn: async () => { + sync.begin({ immediate: true }) + sync.write({ type: `update`, value: next }) + sync.commit() + + sync.begin() + for (const write of round.metadata) { + if (write.type === `set`) { + sync.metadata!.row.set(write.key, write.value) + } else { + sync.metadata!.row.delete(write.key) + } + } + if (round.outcome === `commit`) { + sync.commit() + } else { + const controller = new AbortController() + const receipt = sync.commit(controller.signal) + controller.abort() + if (receipt !== true) { + await receipt.catch((error: unknown) => { + if (!(error instanceof SyncTransactionAbortedError)) throw error + }) + } + } + }, + }) + transaction.mutate(() => { + harness.rows.update(round.key, (draft) => { + draft.position = next.position + }) + }) + await transaction.isPersisted.promise + + model.set(round.key, next) + if (round.outcome === `commit`) { + for (const write of round.metadata) { + if (write.type === `set`) { + metadataModel.set(write.key, write.value) + } else { + metadataModel.delete(write.key) + } + } + } + await Promise.resolve() + const virtualRow = ( + row: PublicationRow, + synced: boolean, + ): PublishedPublicationRow => ({ + ...row, + $collectionId: harness.rows.id, + $key: row.id, + $origin: `local`, + $synced: synced, + }) + const expectedOptimisticChange = keyWasPreviouslyPublished + ? { + type: `update`, + key: round.key, + value: virtualRow(next, false), + previousValue: virtualRow(previous, true), + } + : { + type: `insert`, + key: round.key, + value: virtualRow(next, false), + previousValue: undefined, + } + expect( + harness.batches + .slice(batchCountBefore) + .map((batch) => batch.map(selectPublishedChange)), + ).toEqual([ + [expectedOptimisticChange], + [ + { + type: `update`, + key: round.key, + value: virtualRow(next, true), + previousValue: virtualRow(next, false), + }, + ], + ]) + expectUniqueBatchKeys(harness.batches) + expectPublishedRows(harness, model) + expect(readMetadata(harness, [0, 1, 2])).toEqual( + observableMetadata(metadataModel, [0, 1, 2]), + ) + expect(harness.rows._state.preSyncVisibleState.size).toBe(0) + expect(harness.rows._state.preSyncVirtualState.size).toBe(0) + expect(harness.rows._state.recentlySyncedKeys.size).toBe(0) +} + +async function runPublicationHistory( + rounds: ReadonlyArray, +): Promise { + const harness = await createPublicationHarness() + const model = new Map( + [0, 1, 2].map((id) => [id, { id, position: id }] as const), + ) + const metadataModel = new Map() + try { + for (const round of rounds) { + await applyRound(harness, round, model, metadataModel) + } + } finally { + harness.unsubscribe() + await Promise.all([harness.liveRows.cleanup(), harness.rows.cleanup()]) + } +} + +async function expectMetadataCancellationOwnership( + canceledKeys: ReadonlyArray, + retainedKeys: ReadonlyArray, + canceledOperation: MetadataOperation, + retainedOperation: MetadataOperation, + canceledFirst: boolean, + initialMetadataState: ReadonlyArray, +): Promise { + const harness = await createPublicationHarness() + const initialMetadata = new Map() + for (const [key, state] of initialMetadataState.entries()) { + if (state.present) initialMetadata.set(key, state.value) + } + const initialSync = harness.getSync() + initialSync.begin() + for (const [key, value] of initialMetadata) { + initialSync.metadata!.row.set(key, value) + } + initialSync.commit() + await Promise.resolve() + + const persistence = createDeferred() + const heldTransaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + heldTransaction.mutate(() => { + harness.rows.insert({ id: 99, position: 99 }) + }) + expect(heldTransaction.state).toBe(`persisting`) + + const stageMetadata = ( + keys: ReadonlyArray, + operation: MetadataOperation, + signal?: AbortSignal, + ) => { + const sync = harness.getSync() + sync.begin() + for (const key of keys) { + if (operation.type === `set`) { + sync.metadata!.row.set(key, operation.value) + } else { + sync.metadata!.row.delete(key) + } + } + const receipt = sync.commit(signal) + if (receipt === true) { + throw new Error(`Persisting optimistic work did not hold metadata sync`) + } + void receipt.catch(() => undefined) + return receipt + } + + const canceledController = new AbortController() + const first = canceledFirst + ? stageMetadata(canceledKeys, canceledOperation, canceledController.signal) + : stageMetadata(retainedKeys, retainedOperation) + const second = canceledFirst + ? stageMetadata(retainedKeys, retainedOperation) + : stageMetadata(canceledKeys, canceledOperation, canceledController.signal) + const canceled = canceledFirst ? first : second + const retained = canceledFirst ? second : first + const expectedMetadata = new Map(initialMetadata) + for (const key of retainedKeys) { + if (retainedOperation.type === `set`) { + expectedMetadata.set(key, retainedOperation.value) + } else { + expectedMetadata.delete(key) + } + } + + try { + const batchCountBefore = harness.batches.length + const rowsBefore = [...harness.rows.values()] + + canceledController.abort() + + await expect(canceled).rejects.toBeInstanceOf(SyncTransactionAbortedError) + expect(harness.batches).toHaveLength(batchCountBefore) + expect([...harness.rows.values()]).toEqual(rowsBefore) + expect(readMetadata(harness, [0, 1, 2])).toEqual( + observableMetadata(expectedMetadata, [0, 1, 2]), + ) + + persistence.resolve() + await heldTransaction.isPersisted.promise + await expect(retained).resolves.toBeUndefined() + expect(readMetadata(harness, [0, 1, 2])).toEqual( + observableMetadata(expectedMetadata, [0, 1, 2]), + ) + expectPublishedRows( + harness, + new Map([0, 1, 2].map((id) => [id, { id, position: id }] as const)), + ) + } finally { + persistence.resolve() + await heldTransaction.isPersisted.promise.catch(() => undefined) + await Promise.all([ + canceled.catch(() => undefined), + retained.catch(() => undefined), + ]) + harness.unsubscribe() + await Promise.all([harness.liveRows.cleanup(), harness.rows.cleanup()]) + } +} + +it(`publishes one event per key when metadata-only sync retires optimistic work`, async () => { + await runPublicationHistory([ + { + key: 1, + delta: 1, + metadata: [{ key: 1, type: `set`, value: false }], + outcome: `commit`, + }, + { + key: 1, + delta: 1, + metadata: [{ key: 1, type: `delete` }], + outcome: `commit`, + }, + ]) +}) + +it(`releases only canceled metadata keys while another sync remains pending`, async () => { + await expectMetadataCancellationOwnership( + [0, 1], + [1, 2], + { type: `delete` }, + { type: `set`, value: false }, + true, + [ + { present: true, value: undefined }, + { present: true, value: false }, + { present: true, value: null }, + ], + ) +}) + +it(`does not apply canceled metadata to an absent base key`, async () => { + await expectMetadataCancellationOwnership( + [0, 1], + [1, 2], + { type: `set`, value: `canceled` }, + { type: `set`, value: `retained` }, + true, + [{ present: false }, { present: false }, { present: false }], + ) +}) + +it(`settles an older metadata owner after canceling the newer owner`, async () => { + await expectMetadataCancellationOwnership( + [0, 1], + [1, 2], + { type: `delete` }, + { type: `set`, value: `retained` }, + false, + [ + { present: true, value: undefined }, + { present: false }, + { present: true, value: false }, + ], + ) +}) + +fcTest.prop( + [fc.array(publicationRoundArbitrary, { minLength: 1, maxLength: 8 })], + oraclePropertyOptions(50, `collection-publication.metadata-only`), +)( + `keeps metadata-only optimistic settlement a valid keyed diff across histories`, + runPublicationHistory, +) + +fcTest.prop( + [metadataCancellationArbitrary], + oraclePropertyOptions(50, `collection-publication.metadata-cancellation`), +)( + `keeps metadata suppression owned by the remaining pending transactions`, + ({ + canceledKeys, + retainedKeys, + canceledOperation, + retainedOperation, + canceledFirst, + initialMetadata, + }) => + expectMetadataCancellationOwnership( + canceledKeys, + retainedKeys, + canceledOperation, + retainedOperation, + canceledFirst, + initialMetadata, + ), +) diff --git a/packages/db/tests/collection-query-publication-boundaries.test.ts b/packages/db/tests/collection-query-publication-boundaries.test.ts new file mode 100644 index 0000000000..f5197fc4e1 --- /dev/null +++ b/packages/db/tests/collection-query-publication-boundaries.test.ts @@ -0,0 +1,198 @@ +import { expect, it } from 'vitest' +import { MultiSet } from '@tanstack/db-ivm' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { createLiveQueryCollection, eq } from '../src/query/index.js' +import { createTransaction } from '../src/transactions.js' +import { flushPromises } from './utils.js' +import type { SyncConfig } from '../src/types.js' + +type Row = { id: number; value: number } +type Actions = Parameters[`sync`]>[0] + +it(`consolidates Collection handles by instance, not their id or mutable state`, async () => { + const makeCollection = () => + createCollection({ + id: `shared-definition-id`, + getKey: (row) => row.id, + sync: { + sync: ({ markReady }) => { + markReady() + }, + }, + }) + const first = makeCollection() + const second = makeCollection() + try { + await Promise.all([first.preload(), second.preload()]) + const before = new MultiSet([[{ handle: first }, 1]]) + expect( + new MultiSet([ + [{ handle: first }, 1], + [{ handle: second }, -1], + ]) + .consolidate() + .getInner(), + ).toHaveLength(2) + await first.cleanup() + expect( + before + .concat(new MultiSet([[{ handle: first }, -1]])) + .consolidate() + .getInner(), + ).toEqual([]) + } finally { + await Promise.all([first.cleanup(), second.cleanup()]) + } +}) + +it.each([0, 2])(`opens an inner-join window from limit %s`, async (limit) => { + const makeSource = (collectionId: string) => + createCollection({ + id: collectionId, + getKey: (row) => row.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (let id = 1; id <= 5; id++) + write({ type: `insert`, value: { id, value: id } }) + commit() + markReady() + }, + }, + }) + const parent = makeSource(`window-parent-${limit}`) + const child = makeSource(`window-child-${limit}`) + const live = createLiveQueryCollection({ + query: (q) => + q + .from({ a: parent }) + .join({ b: child }, ({ a, b }) => eq(a.id, b.id), `inner`) + .orderBy(({ a }) => a.value) + .limit(limit) + .select(({ a }) => ({ id: a.id, value: a.value })), + }) + try { + await live.preload() + expect(live.size).toBe(limit) + await live.utils.setWindow({ limit: 10 }) + expect(live.toArray.map((row) => row.id)).toEqual([1, 2, 3, 4, 5]) + } finally { + await live.cleanup() + await parent.cleanup() + await child.cleanup() + } +}) + +it.each([1, 99])( + `publishes server echo value %s after two optimistic inserts hold a sync batch`, + async (echoValue) => { + let sync!: Actions + const source = createCollection({ + getKey: (row) => row.id, + sync: { + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + const observed = new Map() + const subscription = source.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) observed.delete(change.key) + else observed.set(change.key, change.value.value) + } + }, + { includeInitialState: true }, + ) + const live = createLiveQueryCollection({ query: (q) => q.from({ source }) }) + await live.preload() + const first = createDeferred() + const second = createDeferred() + const tx1 = createTransaction({ mutationFn: () => first.promise }) + const tx2 = createTransaction({ mutationFn: () => second.promise }) + try { + tx1.mutate(() => source.insert({ id: 1, value: 1 })) + tx2.mutate(() => source.insert({ id: 2, value: 2 })) + sync.begin() + sync.write({ type: `insert`, value: { id: 3, value: 3 } }) + const blocked = sync.commit() + first.resolve() + await tx1.isPersisted.promise + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: echoValue } }) + const echo = sync.commit() + second.resolve() + await tx2.isPersisted.promise + await blocked + await echo + await flushPromises() + expect(source.get(1)?.value).toBe(echoValue) + expect(observed.get(1)).toBe(echoValue) + expect(live.toArray.find((row) => row.id === 1)?.value).toBe(echoValue) + } finally { + first.resolve() + second.resolve() + subscription.unsubscribe() + await live.cleanup() + await source.cleanup() + } + }, +) + +it.each([`depth`, `cycle`] as const)( + `makes a graph hashing $failure failure visible without publishing it`, + async (failure) => { + type DeepRow = { id: number; nested: object } + let sync!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: (row) => row.id, + sync: { + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + const live = createLiveQueryCollection({ + query: (q) => + q + .from({ source }) + .select(({ source: row }) => ({ id: row.id, nested: row.nested })) + .distinct(), + }) + try { + await live.preload() + sync.begin() + sync.write({ type: `insert`, value: { id: 0, nested: { safe: true } } }) + sync.commit() + const before = [...live.toArray] + expect(before).toHaveLength(1) + let nested: object = {} + if (failure === `depth`) { + for (let depth = 0; depth < 800; depth++) nested = { child: nested } + } else { + Object.assign(nested, { self: nested }) + } + sync.begin() + sync.write({ type: `insert`, value: { id: 1, nested } }) + expect(() => sync.commit()).toThrow( + failure === `depth` + ? RangeError + : `Cannot hash cyclic structural values`, + ) + expect(source.has(1)).toBe(true) + expect(live.status).toBe(`error`) + sync.begin() + sync.write({ type: `insert`, value: { id: 2, nested: {} } }) + sync.commit() + expect(live.status).toBe(`error`) + expect(live.toArray).toEqual(before) + } finally { + await live.cleanup() + await source.cleanup() + } + }, +) diff --git a/packages/db/tests/collection-state-retention-oracle.property.test.ts b/packages/db/tests/collection-state-retention-oracle.property.test.ts new file mode 100644 index 0000000000..587def1a7e --- /dev/null +++ b/packages/db/tests/collection-state-retention-oracle.property.test.ts @@ -0,0 +1,735 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { DuplicateKeySyncError } from '../src/errors.js' +import { createTransaction } from '../src/transactions.js' +import { oraclePropertyOptions } from './oracle-config.js' +import type { Collection } from '../src/collection/index.js' +import type { SyncConfig, TransactionState } from '../src/types.js' + +type RetainedRow = { + id: number + value: number +} + +type SyncActions = Parameters[`sync`]>[0] + +type RetentionAction = + | { type: `insert`; row: RetainedRow } + | { type: `update`; row: RetainedRow } + | { type: `delete`; key: number } + | { type: `replace`; rows: ReadonlyArray } + | { type: `restart` } + | { + type: `reentrantRestart` + row: RetainedRow + commitPhase: `insideListener` | `afterOldReturn` + } + +type RetentionHarness = { + collection: Collection + sync: SyncActions +} + +const retainedRowArbitrary = fc.record({ + id: fc.integer({ min: 0, max: 3 }), + value: fc.integer({ min: -2, max: 2 }), +}) + +function snapshotRetainedRow(row: RetainedRow): RetainedRow { + return { id: row.id, value: row.value } +} + +const retentionActionArbitrary: fc.Arbitrary = fc.oneof( + { + weight: 4, + arbitrary: retainedRowArbitrary.map((row) => ({ + type: `insert` as const, + row, + })), + }, + { + weight: 4, + arbitrary: retainedRowArbitrary.map((row) => ({ + type: `update` as const, + row, + })), + }, + { + weight: 4, + arbitrary: fc + .integer({ min: 0, max: 3 }) + .map((key) => ({ type: `delete` as const, key })), + }, + { + weight: 2, + arbitrary: fc + .uniqueArray(retainedRowArbitrary, { + selector: (row) => row.id, + maxLength: 4, + }) + .map((rows) => ({ type: `replace` as const, rows })), + }, + { weight: 1, arbitrary: fc.constant({ type: `restart` as const }) }, + { + // Keep each phase at least as likely as the original unsplit restart arm. + weight: 3, + arbitrary: fc + .tuple( + retainedRowArbitrary, + fc.constantFrom(`insideListener` as const, `afterOldReturn` as const), + ) + .map(([row, commitPhase]) => ({ + type: `reentrantRestart` as const, + row, + commitPhase, + })), + }, +) + +function createRetentionHarness(): RetentionHarness { + let sync!: SyncActions + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + rowUpdateMode: `full`, + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + return { + collection, + get sync() { + return sync + }, + } +} + +function applyAction( + action: RetentionAction, + model: Map, + sync: SyncActions, +): void { + sync.begin() + switch (action.type) { + case `insert`: { + const previous = model.get(action.row.id) + if (previous !== undefined && previous.value !== action.row.value) { + expect(() => + sync.write({ + type: `insert`, + value: snapshotRetainedRow(action.row), + }), + ).toThrow(DuplicateKeySyncError) + break + } + const expectedRow = snapshotRetainedRow(action.row) + sync.write({ type: `insert`, value: snapshotRetainedRow(action.row) }) + model.set(expectedRow.id, expectedRow) + break + } + case `update`: { + const expectedRow = snapshotRetainedRow(action.row) + sync.write({ type: action.type, value: snapshotRetainedRow(action.row) }) + model.set(expectedRow.id, expectedRow) + break + } + case `delete`: + sync.write({ type: `delete`, key: action.key }) + model.delete(action.key) + break + case `replace`: + sync.truncate() + model.clear() + for (const row of action.rows) { + const expectedRow = snapshotRetainedRow(row) + sync.write({ type: `insert`, value: snapshotRetainedRow(row) }) + model.set(expectedRow.id, expectedRow) + } + break + case `restart`: + case `reentrantRestart`: + throw new Error(`Restart actions require the lifecycle driver`) + } + expect(sync.commit()).toBe(true) +} + +function expectRetainedState( + collection: Collection, + model: ReadonlyMap, +): void { + const expectedRows = [...model.entries()].sort(([a], [b]) => a - b) + const retainedRows = [...collection._state.syncedData.entries()].sort( + ([a], [b]) => a - b, + ) + + expect(retainedRows).toEqual(expectedRows) + expect( + [...collection._state.rowOrigins.keys()] + .filter((key) => !model.has(key)) + .sort((a, b) => a - b), + ).toEqual([]) + expect( + [...collection.state.entries()] + .map(([key, row]) => [key, { id: row.id, value: row.value }] as const) + .sort(([a], [b]) => a - b), + ).toEqual(expectedRows) +} + +async function runRetentionHistory( + actions: ReadonlyArray, +): Promise { + const harness = createRetentionHarness() + const { collection } = harness + const model = new Map() + try { + expectRetainedState(collection, model) + for (const action of actions) { + if (action.type === `restart`) { + await collection.cleanup() + collection.startSyncImmediate() + model.clear() + } else if (action.type === `reentrantRestart`) { + const oldSync = harness.sync + const triggerType = model.has(action.row.id) ? `update` : `insert` + const triggerRow = { + id: action.row.id, + value: (model.get(action.row.id)?.value ?? action.row.value) + 1, + } + const expectedTriggerRow = snapshotRetainedRow(triggerRow) + const restartedRow = { + id: (action.row.id + 1) % 4, + value: action.row.value + 1, + } + const expectedRestartedRow = snapshotRetainedRow(restartedRow) + let cleanup: Promise | undefined + let restarted = false + let restartedSync: SyncActions | undefined + let restartedReceipt: true | Promise | undefined + const batches: Array<{ + changes: Array<{ + type: string + key: string | number + row: RetainedRow + previousRow: RetainedRow | undefined + }> + rows: Array + }> = [] + const subscription = collection.subscribeChanges( + (changes) => { + batches.push({ + changes: changes.map(({ type, key, value, previousValue }) => ({ + type, + key, + row: { id: value.id, value: value.value }, + previousRow: + previousValue === undefined + ? undefined + : { + id: previousValue.id, + value: previousValue.value, + }, + })), + rows: [...collection.values()] + .map(({ id, value }) => ({ id, value })) + .sort((left, right) => left.id - right.id), + }) + if (restarted) return + restarted = true + cleanup = collection.cleanup() + collection.startSyncImmediate() + restartedSync = harness.sync + restartedSync.begin() + restartedSync.write({ + type: `insert`, + value: snapshotRetainedRow(restartedRow), + }) + if (action.commitPhase === `insideListener`) { + restartedReceipt = restartedSync.commit() + } + }, + { includeInitialState: false }, + ) + + oldSync.begin() + oldSync.write({ + type: `update`, + value: snapshotRetainedRow(triggerRow), + }) + expect(oldSync.commit()).toBe(true) + expect(restarted).toBe(true) + expect(restartedSync).toBeDefined() + if (restartedSync === undefined) { + throw new Error(`restarted sync session was not captured`) + } + if (action.commitPhase === `insideListener`) { + expect(restartedReceipt).toBeDefined() + if (restartedReceipt !== true) await restartedReceipt + } else { + expect(restartedSync.commit()).toBe(true) + } + const triggerRows = new Map(model) + triggerRows.set(expectedTriggerRow.id, expectedTriggerRow) + expect(batches).toEqual([ + { + changes: [ + { + type: triggerType, + key: expectedTriggerRow.id, + row: expectedTriggerRow, + previousRow: model.get(expectedTriggerRow.id), + }, + ], + rows: [...triggerRows.values()].sort( + (left, right) => left.id - right.id, + ), + }, + { + // This subscriber observed the trigger, but did not request the + // earlier initial state. Restart retracts its known old-session row. + changes: [ + { + type: `delete`, + key: expectedTriggerRow.id, + row: expectedTriggerRow, + previousRow: undefined, + }, + ], + rows: [], + }, + { + changes: [ + { + type: `insert`, + key: expectedRestartedRow.id, + row: expectedRestartedRow, + previousRow: undefined, + }, + ], + rows: [expectedRestartedRow], + }, + ]) + subscription.unsubscribe() + + await cleanup + model.clear() + model.set(expectedRestartedRow.id, expectedRestartedRow) + } else { + applyAction(action, model, harness.sync) + } + expectRetainedState(collection, model) + } + } finally { + await collection.cleanup() + } +} + +it(`retains only keys in the authoritative synced state`, async () => { + await runRetentionHistory([ + { type: `insert`, row: { id: 1, value: 1 } }, + { type: `insert`, row: { id: 2, value: 2 } }, + { type: `delete`, key: 1 }, + { type: `update`, row: { id: 1, value: -1 } }, + { type: `replace`, rows: [{ id: 3, value: 0 }] }, + { type: `delete`, key: 3 }, + ]) +}) + +it(`retains a missing row introduced by a sync update`, async () => { + await runRetentionHistory([{ type: `update`, row: { id: 1, value: 1 } }]) +}) + +it.each( + ([`insert`, `update`] as const).flatMap((triggerType) => + ([`insideListener`, `afterOldReturn`] as const).map( + (commitPhase) => [triggerType, commitPhase] as const, + ), + ), +)( + `retains an old-session %s and a restarted row committed %s`, + async (triggerType, commitPhase) => { + await runRetentionHistory([ + ...(triggerType === `update` + ? ([{ type: `insert`, row: { id: 1, value: 1 } }] as const) + : []), + { + type: `reentrantRestart`, + row: { id: 1, value: 1 }, + commitPhase, + }, + ]) + }, +) + +it(`releases retained keys after long unique-key churn`, async () => { + const keyCount = 1_000 + const actions: Array = [] + for (let key = 0; key < keyCount; key++) { + actions.push({ type: `insert`, row: { id: key, value: key } }) + actions.push({ type: `delete`, key }) + } + + await runRetentionHistory(actions) +}) + +it(`starts a new sync session without retained publication state`, async () => { + let sync!: SyncActions + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + rowUpdateMode: `full`, + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + const events: Array<{ type: string; key: string | number }> = [] + let subscription: ReturnType | undefined + + try { + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: 1 } }) + expect(sync.commit()).toBe(true) + + subscription = collection.subscribeChanges( + (changes) => { + events.push( + ...changes.map((change) => ({ + type: change.type, + key: change.key, + })), + ) + }, + { includeInitialState: false }, + ) + + sync.begin() + sync.write({ type: `update`, value: { id: 1, value: 2 } }) + collection._state.capturePreSyncVisibleState() + expect(collection._state.preSyncVisibleState.size).toBe(1) + expect(collection._state.recentlySyncedKeys).toEqual(new Set([1])) + + const cleanup = collection.cleanup() + const retainedAfterCleanup = { + visibleRows: collection._state.preSyncVisibleState.size, + virtualRows: collection._state.preSyncVirtualState.size, + recentKeys: collection._state.recentlySyncedKeys.size, + } + await cleanup + + events.length = 0 + collection.startSyncImmediate() + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: 3 } }) + expect(sync.commit()).toBe(true) + + expect({ retainedAfterCleanup, events }).toEqual({ + retainedAfterCleanup: { visibleRows: 0, virtualRows: 0, recentKeys: 0 }, + events: [{ type: `insert`, key: 1 }], + }) + } finally { + subscription?.unsubscribe() + await collection.cleanup() + } +}) + +it(`keeps a restarted session's publication state after the old listener returns`, async () => { + let sync!: SyncActions + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + rowUpdateMode: `full`, + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + let cleanup: Promise | undefined + let restarted = false + const subscription = collection.subscribeChanges( + () => { + if (restarted) return + restarted = true + cleanup = collection.cleanup() + collection.startSyncImmediate() + collection._state.preSyncVisibleState.set(2, { id: 2, value: 2 }) + collection._state.recentlySyncedKeys.add(2) + }, + { includeInitialState: false }, + ) + + try { + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: 1 } }) + expect(sync.commit()).toBe(true) + + expect(restarted).toBe(true) + expect(collection._state.preSyncVisibleState).toEqual( + new Map([[2, { id: 2, value: 2 }]]), + ) + expect(collection._state.recentlySyncedKeys).toEqual(new Set([2])) + expect(collection._state.hasReceivedFirstCommit).toBe(false) + + sync.begin() + sync.write({ type: `insert`, value: { id: 3, value: 3 } }) + expect(sync.commit()).toBe(true) + expect(collection._state.preSyncVisibleState.size).toBe(0) + expect(collection._state.preSyncVirtualState.size).toBe(0) + expect(collection._state.hasReceivedFirstCommit).toBe(true) + await Promise.resolve() + expect(collection._state.recentlySyncedKeys.size).toBe(0) + await cleanup + } finally { + subscription.unsubscribe() + await collection.cleanup() + } +}) + +it(`does not let an old publication microtask clear restarted sync state`, async () => { + let sync!: SyncActions + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + rowUpdateMode: `full`, + sync: (actions) => { + sync = actions + actions.markReady() + }, + }, + }) + + try { + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: 1 } }) + expect(sync.commit()).toBe(true) + + const cleanup = collection.cleanup() + collection.startSyncImmediate() + sync.begin() + sync.write({ type: `insert`, value: { id: 2, value: 2 } }) + collection._state.capturePreSyncVisibleState() + expect(collection._state.recentlySyncedKeys).toEqual(new Set([2])) + + await Promise.resolve() + + expect(collection._state.recentlySyncedKeys).toEqual(new Set([2])) + + expect(sync.commit()).toBe(true) + expect(collection._state.hasReceivedFirstCommit).toBe(true) + await Promise.resolve() + expect(collection._state.preSyncVisibleState.size).toBe(0) + expect(collection._state.preSyncVirtualState.size).toBe(0) + expect(collection._state.recentlySyncedKeys.size).toBe(0) + await cleanup + } finally { + await collection.cleanup() + } +}) + +it(`publishes a virtual-state update when a restarted optimistic row is confirmed`, async () => { + let sync!: SyncActions + let syncSession = 0 + let releaseMutation!: () => void + const mutationHold = new Promise((resolve) => { + releaseMutation = resolve + }) + const collection = createCollection({ + getKey: (row) => row.id, + startSync: true, + sync: { + rowUpdateMode: `full`, + sync: (actions) => { + sync = actions + syncSession++ + if (syncSession === 1) actions.markReady() + }, + }, + }) + type ObservedRow = RetainedRow & { + $collectionId: string + $key: number + $origin: `local` | `remote` + $synced: boolean + } + type ObservedChange = { + type: string + key: string | number + value: ObservedRow + previousValue?: ObservedRow + } + const snapshotRow = (row: ObservedRow): ObservedRow => ({ + id: row.id, + value: row.value, + $collectionId: row.$collectionId, + $key: row.$key, + $origin: row.$origin, + $synced: row.$synced, + }) + const publications: Array<{ + changes: Array + rows: Array + }> = [] + const restartStatuses: Array = [] + const settlementTimeline: Array<`publication` | `receipt`> = [] + let restarted = false + let readMutationState: (() => TransactionState) | undefined + let rollbackMutation: (() => void) | undefined + let mutationCommit: Promise | undefined + let syncReceipt: ReturnType | undefined + let syncReceiptOutcome: Promise | undefined + let syncReceiptSettled = false + const subscription = collection.subscribeChanges( + (changes) => { + publications.push({ + changes: changes.map(({ type, key, value, previousValue }) => ({ + type, + key, + value: snapshotRow(value), + ...(previousValue === undefined + ? {} + : { previousValue: snapshotRow(previousValue) }), + })), + rows: [...collection.state.values()].map(snapshotRow), + }) + if (changes.some(({ type, key }) => type === `update` && key === 2)) { + queueMicrotask(() => settlementTimeline.push(`publication`)) + } + if (restarted || !changes.some(({ key }) => key === 1)) return + + restarted = true + restartStatuses.push(collection.status) + void collection.cleanup() + restartStatuses.push(collection.status) + collection.startSyncImmediate() + restartStatuses.push(collection.status) + sync.markReady() + restartStatuses.push(collection.status) + + const transaction = createTransaction({ + autoCommit: false, + mutationFn: () => mutationHold, + }) + readMutationState = () => transaction.state + rollbackMutation = () => transaction.rollback() + void transaction.isPersisted.promise.catch(() => undefined) + transaction.mutate(() => collection.insert({ id: 2, value: 2 })) + mutationCommit = transaction.commit() + + sync.begin() + sync.write({ type: `insert`, value: { id: 2, value: 2 } }) + syncReceipt = sync.commit() + if (syncReceipt !== true) { + syncReceiptOutcome = syncReceipt.then((value) => { + settlementTimeline.push(`receipt`) + syncReceiptSettled = true + return value + }) + } + }, + { includeInitialState: false }, + ) + + try { + sync.begin() + sync.write({ type: `insert`, value: { id: 1, value: 1 } }) + expect(sync.commit()).toBe(true) + + const remoteRow = (id: number): ObservedRow => ({ + id, + value: id, + $collectionId: collection.id, + $key: id, + $origin: `remote`, + $synced: true, + }) + const localRow = (id: number): ObservedRow => ({ + id, + value: id, + $collectionId: collection.id, + $key: id, + $origin: `local`, + $synced: false, + }) + const expectedPublications = [ + { + changes: [{ type: `insert`, key: 1, value: remoteRow(1) }], + rows: [remoteRow(1)], + }, + { changes: [{ type: `delete`, key: 1, value: remoteRow(1) }], rows: [] }, + { + changes: [{ type: `insert`, key: 2, value: localRow(2) }], + rows: [localRow(2)], + }, + { + changes: [ + { + type: `update`, + key: 2, + value: remoteRow(2), + previousValue: localRow(2), + }, + ], + rows: [remoteRow(2)], + }, + ] + expect(publications).toEqual(expectedPublications.slice(0, 3)) + expect([...collection.state.keys()]).toEqual([2]) + expect(restartStatuses).toEqual([`ready`, `cleaned-up`, `loading`, `ready`]) + expect(collection.status).toBe(`ready`) + + expect(syncReceipt).toBeDefined() + expect(syncReceipt).not.toBe(true) + expect(syncReceiptSettled).toBe(false) + if (syncReceipt === undefined || syncReceipt === true) { + throw new Error(`restarted sync receipt was not parked`) + } + expect(syncReceipt).toBeInstanceOf(Promise) + expect(syncReceiptOutcome).toBeDefined() + expect(rollbackMutation).toBeDefined() + await Promise.resolve() + expect(syncReceiptSettled).toBe(false) + expect(settlementTimeline).toEqual([]) + + rollbackMutation?.() + expect(publications).toEqual(expectedPublications) + expect(syncReceiptSettled).toBe(false) + await expect(syncReceiptOutcome).resolves.toBeUndefined() + expect(syncReceiptSettled).toBe(true) + expect(settlementTimeline).toEqual([`publication`, `receipt`]) + expect(publications).toEqual(expectedPublications) + expect([...collection.state.values()].map(snapshotRow)).toEqual([ + remoteRow(2), + ]) + + releaseMutation() + await mutationCommit + expect(readMutationState?.()).toBe(`failed`) + expect(publications).toEqual(expectedPublications) + expect([...collection.state.values()].map(snapshotRow)).toEqual([ + remoteRow(2), + ]) + } finally { + releaseMutation() + await mutationCommit + subscription.unsubscribe() + await collection.cleanup() + } +}) + +fcTest.prop( + [fc.array(retentionActionArbitrary, { minLength: 1, maxLength: 20 })], + oraclePropertyOptions(100, `collection-state.retention`), +)( + `matches retained authoritative state without optimistic overlays after every committed sync history`, + async (actions) => { + await runRetentionHistory(actions) + }, +) diff --git a/packages/db/tests/collection-subscribe-changes.test.ts b/packages/db/tests/collection-subscribe-changes.test.ts index 08dce91992..000e025555 100644 --- a/packages/db/tests/collection-subscribe-changes.test.ts +++ b/packages/db/tests/collection-subscribe-changes.test.ts @@ -2671,6 +2671,58 @@ describe(`Virtual properties`, () => { expect(collection.state.get(`row-1`)?.$origin).toBe(`remote`) }) + it(`replaces a completed direct mutation with an authoritative truncate row`, async () => { + let syncFns: + | { + begin: () => void + write: (change: { + type: `insert` + value: { id: string; value: string } + }) => void + commit: () => true | Promise + truncate: () => void + } + | undefined + + const collection = createCollection<{ id: string; value: string }, string>({ + id: `truncate-replaces-completed-direct-mutation`, + getKey: (item) => item.id, + startSync: true, + sync: { + sync: ({ begin, write, commit, truncate, markReady }) => { + syncFns = { begin, write, commit, truncate } + markReady() + }, + }, + onInsert: () => Promise.resolve(), + }) + + await collection.stateWhenReady() + const transaction = collection.insert({ id: `row-1`, value: `client` }) + await transaction.isPersisted.promise + expect(collection.get(`row-1`)).toMatchObject({ + id: `row-1`, + value: `client`, + }) + + if (!syncFns) throw new Error(`Sync not ready`) + syncFns.begin() + syncFns.truncate() + syncFns.write({ + type: `insert`, + value: { id: `row-1`, value: `server` }, + }) + const applied = syncFns.commit() + if (applied !== true) await applied + await waitForChanges() + + expect(collection.get(`row-1`)).toMatchObject({ + id: `row-1`, + value: `server`, + }) + expect(collection.state.get(`row-1`)?.$origin).toBe(`remote`) + }) + it(`should preserve local origin for rows confirmed in the same truncate batch`, async () => { let syncFns: | { diff --git a/packages/db/tests/collection-subscriber-duplicate-inserts.test.ts b/packages/db/tests/collection-subscriber-duplicate-inserts.test.ts index 8b9d6be57c..b53f2bcf34 100644 --- a/packages/db/tests/collection-subscriber-duplicate-inserts.test.ts +++ b/packages/db/tests/collection-subscriber-duplicate-inserts.test.ts @@ -15,8 +15,9 @@ import type { ChangeMessage } from '../src/types.js' * If duplicate inserts reach D2, multiplicity becomes > 1, and deletes won't * properly remove items (multiplicity goes from 2 to 1, not triggering removal). * - * The fix: CollectionSubscriber tracks keys sent to D2 (sentToD2Keys) and - * filters out duplicate inserts before they reach the pipeline. + * The source boundary tracks the exact row sent for each key. It filters + * duplicate inserts and uses the stored row for later D2 retractions. The + * generated reconciliation oracle covers that stateful boundary directly. * * Additionally, for JOIN queries with lazy sources: * - The includeInitialState fix ensures internal lazy-loading subscriptions diff --git a/packages/db/tests/collection-subscription-lifecycle-grammar.ts b/packages/db/tests/collection-subscription-lifecycle-grammar.ts new file mode 100644 index 0000000000..ae3d4312ef --- /dev/null +++ b/packages/db/tests/collection-subscription-lifecycle-grammar.ts @@ -0,0 +1,697 @@ +import { fc } from '@fast-check/vitest' + +export type DemandName = `a` | `b` +export type AttemptScope = `current` | `obsolete` +export type AttemptAge = `oldest` | `newest` +export type LifecycleCommand = + | { type: `request`; demand: DemandName } + | { type: `abort`; demand: DemandName } + | { type: `release`; demand: DemandName } + | { + type: `settle` + demand: DemandName + scope: AttemptScope + age: AttemptAge + outcome: `resolve` | `reject` + } + | { type: `truncate` } + | { type: `cleanup` } + | { type: `restart` } + | { type: `unsubscribe` } + +export const lifecycleCommandArbitrary: fc.Arbitrary = + fc.oneof( + fc.record({ + type: fc.constant(`request` as const), + demand: fc.constantFrom(`a` as const, `b` as const), + }), + fc.record({ + type: fc.constant(`abort` as const), + demand: fc.constantFrom(`a` as const, `b` as const), + }), + fc.record({ + type: fc.constant(`release` as const), + demand: fc.constantFrom(`a` as const, `b` as const), + }), + fc.record({ + type: fc.constant(`settle` as const), + demand: fc.constantFrom(`a` as const, `b` as const), + scope: fc.constantFrom(`current` as const, `obsolete` as const), + age: fc.constantFrom(`oldest` as const, `newest` as const), + outcome: fc.constantFrom(`resolve` as const, `reject` as const), + }), + fc.constant({ type: `truncate` as const }), + fc.constant({ type: `cleanup` as const }), + fc.constant({ type: `restart` as const }), + fc.constant({ type: `unsubscribe` as const }), + ) + +export type LifecycleOwner = { + id: number + demand: DemandName + aborted: boolean + attemptId?: number +} + +export type LifecycleAttempt = { + id: number + ownerId: number + demand: DemandName + session: number + replay: number + settled: boolean + outcome?: `resolve` | `reject` + gating: boolean + inReplacement: boolean + reportable: boolean + aborted: boolean + failure: Error +} + +export type LifecycleLoadEvent = Pick< + LifecycleAttempt, + `id` | `demand` | `session` | `replay` +> +export type LifecycleUnloadEvent = { + attemptId: number + handlerSession: number +} +export type LifecycleErrorEvent = { attemptId: number; error: Error } +export type LifecycleResultKind = `promise` | `true` +export type LifecycleResultEvent = { + attemptId: number | `unacquired` + resultKind: LifecycleResultKind +} +export type LifecycleTraceEvent = + | ({ type: `load` } & LifecycleLoadEvent) + | ({ type: `unload` } & LifecycleUnloadEvent) + | { type: `error`; attemptId: number } + | ({ type: `result` } & LifecycleResultEvent) + | { type: `status`; status: string } + | { type: `publication` } + +export type LifecycleModel = { + acquisitionMode: `async-pending` | `sync-success` + cancellation: `manual` | `reject` + active: boolean + unsubscribed: boolean + session: number + replay: number + publicationBarrierOpen: boolean + nextOwnerId: number + nextAttemptId: number + owners: Array + attempts: Array + loads: Array + unloads: Array + errors: Array + results: Array + publications: number + statuses: Array + status: string + collectionStatus: `ready` | `cleaned-up` + lastError?: Error + failureForAttempt: (attemptId: number) => Error + reach: Set + trace: Array +} + +export type LifecycleEffect = { + ownerId?: number + attemptId?: number + requestResult?: boolean +} + +export function createLifecycleModel( + acquisitionMode: LifecycleModel[`acquisitionMode`] = `async-pending`, + failureForAttempt: (attemptId: number) => Error = (attemptId) => + new Error(`attempt ${attemptId} failed`), + cancellation: LifecycleModel[`cancellation`] = `manual`, +): LifecycleModel { + return { + acquisitionMode, + cancellation, + active: true, + unsubscribed: false, + session: 0, + replay: 0, + publicationBarrierOpen: false, + nextOwnerId: 0, + nextAttemptId: 0, + owners: [], + attempts: [], + loads: [], + unloads: [], + errors: [], + results: [], + publications: 0, + statuses: [], + status: `ready`, + collectionStatus: `ready`, + reach: new Set(), + trace: [], + failureForAttempt, + } +} + +function setStatus(model: LifecycleModel, queuedReplay = false): void { + if (model.unsubscribed) return + const status = + model.active && + (queuedReplay || model.attempts.some(({ gating }) => gating)) + ? `loadingSubset` + : `ready` + if (status !== model.status) { + model.status = status + model.statuses.push(status) + model.trace.push({ type: `status`, status }) + } +} + +// Settled failure is not an authoritative replacement. Keep subsequent reads +// private until the failed owner retires or a new replay succeeds. +function replacementSucceeded(model: LifecycleModel): boolean { + return ( + !model.attempts.some( + ({ gating, inReplacement }) => gating && inReplacement, + ) && + model.owners.every( + ({ attemptId }) => + attemptId === undefined || + model.attempts[attemptId]!.outcome === `resolve`, + ) + ) +} + +function startAttempt( + model: LifecycleModel, + owner: LifecycleOwner, + trace = true, +): LifecycleAttempt { + const id = model.nextAttemptId++ + const attempt: LifecycleAttempt = { + id, + ownerId: owner.id, + demand: owner.demand, + session: model.session, + replay: model.replay, + settled: model.acquisitionMode === `sync-success`, + ...(model.acquisitionMode === `sync-success` + ? { outcome: `resolve` as const } + : {}), + gating: model.acquisitionMode === `async-pending`, + // Initial/progressive acquisition can hold readiness without joining the + // authoritative replacement's publication boundary. + inReplacement: model.publicationBarrierOpen, + reportable: true, + aborted: false, + failure: model.failureForAttempt(id), + } + model.attempts.push(attempt) + model.reach.add( + `attempt-session:${attempt.session === 0 ? `initial` : `restarted`}`, + ) + model.reach.add( + `attempt-replay:${attempt.replay === 0 ? `initial` : `replayed`}`, + ) + model.reach.add( + `attempt-location:${attempt.session === 0 ? `initial` : `restarted`}:${attempt.replay === 0 ? `initial` : `replayed`}`, + ) + model.loads.push({ + id, + demand: attempt.demand, + session: attempt.session, + replay: attempt.replay, + }) + if (trace) { + model.trace.push({ + type: `load`, + id, + demand: attempt.demand, + session: attempt.session, + replay: attempt.replay, + }) + } + owner.attemptId = id + return attempt +} + +function retireAttempt( + model: LifecycleModel, + owner: LifecycleOwner, + options: { unload: boolean; trace?: boolean; keepPending?: boolean }, +): void { + if (owner.attemptId === undefined) return + const attempt = model.attempts[owner.attemptId] + owner.attemptId = undefined + if (!attempt) throw new Error(`model lost attempt`) + attempt.gating = options.keepPending === true && !attempt.settled + attempt.reportable = false + abortAttempt(model, attempt) + if (options.unload) { + model.unloads.push({ attemptId: attempt.id, handlerSession: model.session }) + if (options.trace !== false) { + model.trace.push({ + type: `unload`, + attemptId: attempt.id, + handlerSession: model.session, + }) + } + } +} + +function abortAttempt(model: LifecycleModel, attempt: LifecycleAttempt): void { + attempt.aborted = true + if (model.cancellation === `reject` && !attempt.settled) { + attempt.settled = true + attempt.outcome = `reject` + attempt.gating = false + } +} + +function selectAttempt( + model: LifecycleModel, + command: Extract, +): LifecycleAttempt | undefined { + const currentAttemptIds = new Set( + model.owners.flatMap(({ attemptId }) => + attemptId === undefined ? [] : [attemptId], + ), + ) + const candidates = model.attempts.filter( + (attempt) => + !attempt.settled && + attempt.demand === command.demand && + (command.scope === `current` + ? currentAttemptIds.has(attempt.id) + : !currentAttemptIds.has(attempt.id)), + ) + return command.age === `oldest` ? candidates[0] : candidates.at(-1) +} + +/** Pure reference transition. It never reads adapter callbacks or SUT state. */ +export function reduceLifecycle( + model: LifecycleModel, + command: LifecycleCommand, +): LifecycleEffect { + model.reach.add(`command:${command.type}`) + if (model.unsubscribed) { + if (command.type === `cleanup` && model.active) { + model.reach.add(`effective:cleanup`) + model.active = false + model.collectionStatus = `cleaned-up` + } else if (command.type === `restart` && !model.active) { + model.reach.add(`effective:restart`) + model.active = true + model.session++ + model.replay = 0 + model.publicationBarrierOpen = false + model.collectionStatus = `ready` + } else { + model.reach.add(`noop:${command.type}`) + } + return { requestResult: false } + } + + if (command.type === `request`) { + model.reach.add(`effective:request`) + if (model.owners.some(({ demand }) => demand === command.demand)) { + model.reach.add(`duplicate-owner`) + } + if (!model.active) model.reach.add(`request-while-cleaned`) + const owner: LifecycleOwner = { + id: model.nextOwnerId++, + demand: command.demand, + aborted: false, + } + model.owners.push(owner) + if (model.active) { + const attemptId = startAttempt(model, owner).id + const result = { + attemptId, + resultKind: + model.acquisitionMode === `async-pending` + ? (`promise` as const) + : (`true` as const), + } + model.results.push(result) + model.trace.push({ type: `result`, ...result }) + } else { + // A waiting owner gets a promise now, without claiming an acquisition. + const result = { attemptId: `unacquired`, resultKind: `promise` } as const + model.results.push(result) + model.trace.push({ type: `result`, ...result }) + } + setStatus(model) + if (!model.publicationBarrierOpen) { + model.publications++ + model.trace.push({ type: `publication` }) + } + return { ownerId: owner.id, requestResult: true } + } + + if (command.type === `abort`) { + const owner = model.owners.find( + ({ demand, aborted }) => demand === command.demand && !aborted, + ) + if (!owner) { + model.reach.add(`noop:abort`) + return {} + } + model.reach.add(`effective:abort`) + owner.aborted = true + if (owner.attemptId !== undefined) { + const attempt = model.attempts[owner.attemptId]! + abortAttempt(model, attempt) + attempt.reportable = false + } + setStatus(model) + return { ownerId: owner.id } + } + + if (command.type === `release`) { + const index = model.owners.findIndex( + ({ demand }) => demand === command.demand, + ) + if (index === -1) { + model.reach.add(`noop:release`) + return {} + } + model.reach.add(`effective:release`) + const [owner] = model.owners.splice(index, 1) + retireAttempt(model, owner!, { unload: true }) + // Retirement removes the logical owner, including its older transports. + for (const attempt of model.attempts) { + if (attempt.ownerId === owner!.id) attempt.gating = false + } + if (model.publicationBarrierOpen && replacementSucceeded(model)) { + model.publicationBarrierOpen = false + } + setStatus(model) + return { ownerId: owner!.id } + } + + if (command.type === `settle`) { + const attempt = selectAttempt(model, command) + if (!attempt) { + model.reach.add(`noop:settle`) + return {} + } + model.reach.add(`effective:settle`) + model.reach.add(`settle-scope:${command.scope}`) + model.reach.add(`settle-age:${command.age}`) + model.reach.add(`settle-outcome:${command.outcome}`) + model.reach.add(`settle:${command.scope}:${command.age}:${command.outcome}`) + attempt.settled = true + attempt.outcome = command.outcome + attempt.gating = false + if ( + command.outcome === `reject` && + attempt.reportable && + !attempt.aborted + ) { + model.lastError = attempt.failure + model.errors.push({ attemptId: attempt.id, error: attempt.failure }) + model.trace.push({ type: `error`, attemptId: attempt.id }) + } + if (model.publicationBarrierOpen && replacementSucceeded(model)) { + model.publicationBarrierOpen = false + } + setStatus(model) + return { attemptId: attempt.id } + } + + if (command.type === `truncate`) { + if (!model.active) { + model.reach.add(`noop:truncate`) + return {} + } + model.reach.add(`effective:truncate`) + if ( + model.replay > 0 && + model.attempts.some( + ({ session, settled }) => session === model.session && !settled, + ) + ) { + model.reach.add(`overlapping-replay`) + } + model.replay++ + // Replay setup is asynchronous even when every acquisition is synchronous + // or canceled. Logical owners queue setup; live owners start acquisitions. + setStatus(model, model.owners.length > 0) + // A canceled-only truncate starts no new work, but cannot end a prior + // replay's publication wait while that owner still owes settlement. + model.publicationBarrierOpen = + model.owners.some(({ aborted }) => !aborted) || + model.attempts.some( + ({ gating, inReplacement }) => gating && inReplacement, + ) + for (const owner of model.owners) { + // Replacing an acquisition is not releasing its logical owner. Delayed + // cancellation still holds readiness; replay work also holds publication. + retireAttempt(model, owner, { + unload: true, + keepPending: true, + }) + if (!owner.aborted) { + startAttempt(model, owner) + } + } + if (model.publicationBarrierOpen && replacementSucceeded(model)) { + model.publicationBarrierOpen = false + } + setStatus(model) + return {} + } + + if (command.type === `cleanup`) { + if (!model.active) { + model.reach.add(`noop:cleanup`) + return {} + } + model.reach.add(`effective:cleanup`) + const current = model.attempts.filter( + ({ session }) => session === model.session, + ) + if ( + current.some(({ settled }) => settled) && + current.some(({ settled }) => !settled) + ) { + model.reach.add(`partial-generation-supersession`) + } + for (const owner of model.owners) + retireAttempt(model, owner, { unload: false }) + for (const attempt of model.attempts) attempt.gating = false + model.active = false + model.publicationBarrierOpen = false + model.collectionStatus = `cleaned-up` + setStatus(model) + return {} + } + + if (command.type === `restart`) { + if (model.active) { + model.reach.add(`noop:restart`) + return {} + } + model.reach.add(`effective:restart`) + model.active = true + model.session++ + model.replay = 0 + model.publicationBarrierOpen = model.owners.some(({ aborted }) => !aborted) + model.collectionStatus = `ready` + setStatus(model, model.owners.length > 0) + const replayLoads: Array = [] + for (const owner of model.owners) { + if (!owner.aborted) replayLoads.push(startAttempt(model, owner, false)) + } + if (model.publicationBarrierOpen && replacementSucceeded(model)) { + model.publicationBarrierOpen = false + } + model.publications++ + model.trace.push({ type: `publication` }) + for (const load of replayLoads) { + model.trace.push({ + type: `load`, + id: load.id, + demand: load.demand, + session: load.session, + replay: load.replay, + }) + } + setStatus(model) + return {} + } + + model.reach.add(`effective:unsubscribe`) + for (const owner of model.owners) + retireAttempt(model, owner, { unload: true }) + model.owners.length = 0 + model.unsubscribed = true + return {} +} + +export const greenLifecycleHistoryArbitrary = fc.array( + lifecycleCommandArbitrary, + { minLength: 1, maxLength: 20 }, +) + +export const syncLifecycleHistoryArbitrary = fc.array( + lifecycleCommandArbitrary, + { minLength: 1, maxLength: 20 }, +) + +export const settle = ( + demand: DemandName, + scope: AttemptScope, + age: AttemptAge, + outcome: `resolve` | `reject`, +): LifecycleCommand => ({ type: `settle`, demand, scope, age, outcome }) + +const compoundSettlementHistories = ([`current`, `obsolete`] as const).flatMap( + (scope) => + ([`oldest`, `newest`] as const).flatMap((age) => + ([`resolve`, `reject`] as const).map((outcome) => [ + { type: `request`, demand: `a` } as const, + { type: `request`, demand: `a` } as const, + ...(scope === `obsolete` + ? ([ + { type: `release`, demand: `a` }, + { type: `release`, demand: `a` }, + ] as const) + : []), + settle(`a`, scope, age, outcome), + ]), + ), +) + +export const compoundLifecycleCoverageHistories: ReadonlyArray< + ReadonlyArray +> = [ + ...compoundSettlementHistories, + [ + { type: `request`, demand: `a` }, + settle(`a`, `current`, `oldest`, `resolve`), + { type: `truncate` }, + settle(`a`, `current`, `oldest`, `resolve`), + ], +] + +export const greenLifecycleHistories: ReadonlyArray< + ReadonlyArray +> = [ + [ + { type: `request`, demand: `a` }, + { type: `abort`, demand: `a` }, + settle(`a`, `current`, `oldest`, `reject`), + { type: `release`, demand: `a` }, + ], + [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `a` }, + settle(`a`, `current`, `newest`, `resolve`), + settle(`a`, `current`, `oldest`, `resolve`), + { type: `release`, demand: `a` }, + { type: `release`, demand: `a` }, + ], + [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `b` }, + settle(`a`, `current`, `oldest`, `resolve`), + { type: `cleanup` }, + { type: `restart` }, + settle(`b`, `obsolete`, `oldest`, `reject`), + settle(`a`, `current`, `oldest`, `resolve`), + settle(`b`, `current`, `oldest`, `reject`), + ], + [ + { type: `cleanup` }, + { type: `request`, demand: `a` }, + { type: `restart` }, + settle(`a`, `current`, `oldest`, `resolve`), + { type: `truncate` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ], + [ + { type: `request`, demand: `a` }, + settle(`a`, `current`, `oldest`, `reject`), + { type: `cleanup` }, + { type: `restart` }, + settle(`a`, `current`, `oldest`, `resolve`), + ], + [ + { type: `abort`, demand: `a` }, + { type: `release`, demand: `a` }, + settle(`a`, `current`, `oldest`, `resolve`), + { type: `truncate` }, + { type: `cleanup` }, + { type: `truncate` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `restart` }, + { type: `unsubscribe` }, + { type: `request`, demand: `b` }, + { type: `unsubscribe` }, + ], +] + +export const syncLifecycleHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `a` }, + { type: `release`, demand: `a` }, + { type: `request`, demand: `b` }, + { type: `truncate` }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, +] + +export const pendingSupersessionHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `a` }, + { type: `truncate` }, + { type: `truncate` }, + settle(`a`, `current`, `newest`, `resolve`), + settle(`a`, `current`, `oldest`, `reject`), + { type: `release`, demand: `a` }, +] + +export const abortReplayHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `abort`, demand: `a` }, + settle(`a`, `current`, `oldest`, `reject`), + { type: `truncate` }, + { type: `release`, demand: `a` }, +] + +export const abortedRestartHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `abort`, demand: `a` }, + { type: `restart` }, + { type: `release`, demand: `a` }, +] + +export const mixedAbortedRestartHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `b` }, + { type: `abort`, demand: `a` }, + settle(`a`, `current`, `oldest`, `reject`), + { type: `cleanup` }, + { type: `restart` }, + settle(`b`, `current`, `oldest`, `resolve`), + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, +] + +export const releasedObsoleteResolveHistory: ReadonlyArray = [ + { type: `request`, demand: `a` }, + { type: `release`, demand: `a` }, + settle(`a`, `obsolete`, `oldest`, `resolve`), +] diff --git a/packages/db/tests/collection-subscription-lifecycle-history.property.test.ts b/packages/db/tests/collection-subscription-lifecycle-history.property.test.ts new file mode 100644 index 0000000000..fa861a3250 --- /dev/null +++ b/packages/db/tests/collection-subscription-lifecycle-history.property.test.ts @@ -0,0 +1,798 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { + abortReplayHistory, + abortedRestartHistory, + compoundLifecycleCoverageHistories, + createLifecycleModel, + greenLifecycleHistories, + greenLifecycleHistoryArbitrary, + mixedAbortedRestartHistory, + pendingSupersessionHistory, + reduceLifecycle, + syncLifecycleHistory, + syncLifecycleHistoryArbitrary, +} from './collection-subscription-lifecycle-grammar.js' +import { flushPromises } from './utils.js' +import { + oraclePropertyOptions, + oracleRandomParameters, + readOracleRunConfig, +} from './oracle-config.js' +import type { LoadSubsetOptions, SyncConfig } from '../src/types.js' +import type { + DemandName, + LifecycleCommand, + LifecycleLoadEvent, + LifecycleResultEvent, + LifecycleTraceEvent, + LifecycleUnloadEvent, +} from './collection-subscription-lifecycle-grammar.js' + +type RuntimeAttempt = { + id: number + ownerId: number + demand: DemandName + options: LoadSubsetOptions + deferred?: ReturnType> + failure: Error + settled: boolean + current: boolean +} +type RuntimeOwner = { + id: number + demand: DemandName + controller: AbortController + aborted: boolean + attemptId?: number +} + +async function runHistory( + history: ReadonlyArray, + options: { + acquisitionMode?: `async-pending` | `sync-success` + cancellation?: `manual` | `reject` + continueAfterMismatch?: boolean + } = {}, +): Promise> { + const acquisitionMode = options.acquisitionMode ?? `async-pending` + const check = options.continueAfterMismatch ? expect.soft : expect + const failures = new Map() + const failureForAttempt = (attemptId: number): Error => { + const existing = failures.get(attemptId) + if (existing) return existing + const failure = new Error(`attempt ${attemptId} failed`) + failures.set(attemptId, failure) + return failure + } + const cancellation = options.cancellation ?? `manual` + const model = createLifecycleModel( + acquisitionMode, + failureForAttempt, + cancellation, + ) + const where = { + a: new Func(`eq`, [new PropRef([`id`]), new Value(`a`)]), + b: new Func(`eq`, [new PropRef([`id`]), new Value(`b`)]), + } + const demandForWhere = new Map([ + [where.a, `a`], + [where.b, `b`], + ]) + const runtimeAttempts = new Map() + const runtimeOwners: Array = [] + const attemptByOptions = new Map() + const observedLoads: Array = [] + const observedUnloads: Array< + LifecycleUnloadEvent | { attemptId: `unacquired`; handlerSession: number } + > = [] + const observedErrors: Array<{ + attemptId: number | `unacquired` + error: unknown + }> = [] + const observedResults: Array< + LifecycleResultEvent | { attemptId: `unacquired`; resultKind: string } + > = [] + const observedStatuses: Array = [] + const observedTrace: Array< + | LifecycleTraceEvent + | { type: `unload`; attemptId: `unacquired`; handlerSession: number } + | { type: `error`; attemptId: `unacquired` } + | { type: `result`; attemptId: `unacquired`; resultKind: string } + > = [] + let nextObservedAttemptId = 0 + let nextObservedOwnerId = 0 + let observedReplay = 0 + let observedSession = -1 + let observedActive = true + let observedUnsubscribed = false + let syncOps: + | Parameters[`sync`]>[0] + | undefined + + const collection = createCollection<{ id: string }, string>({ + id: `generated-async-demand-lifecycle`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + const handlerSession = ++observedSession + syncOps = operations + operations.markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`adapter load lost its demand`) + const owner = runtimeOwners.find( + (candidate) => + candidate.demand === demand && + !candidate.aborted && + candidate.attemptId === undefined, + ) + if (!owner) throw new Error(`adapter load has no runtime owner`) + const observed: LifecycleLoadEvent = { + id: nextObservedAttemptId++, + demand, + session: handlerSession, + replay: observedReplay, + } + const deferred = + acquisitionMode === `async-pending` + ? createDeferred() + : undefined + void deferred?.promise.catch(() => undefined) + runtimeAttempts.set(observed.id, { + id: observed.id, + ownerId: owner.id, + demand, + options, + deferred, + failure: failureForAttempt(observed.id), + settled: acquisitionMode === `sync-success`, + current: true, + }) + if (deferred && cancellation === `reject`) { + options.signal?.addEventListener( + `abort`, + () => { + const attempt = runtimeAttempts.get(observed.id)! + if (attempt.settled) return + attempt.settled = true + deferred.reject( + new DOMException(`acquisition aborted`, `AbortError`), + ) + }, + { once: true }, + ) + } + owner.attemptId = observed.id + attemptByOptions.set(options, observed.id) + observedLoads.push(observed) + observedTrace.push({ type: `load`, ...observed }) + return deferred?.promise ?? true + }, + unloadSubset: (options) => { + const unload = { + attemptId: attemptByOptions.get(options) ?? `unacquired`, + handlerSession, + } as const + observedUnloads.push(unload) + observedTrace.push({ type: `unload`, ...unload }) + }, + } + }, + }, + }) + const publications: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + publications.push(changes) + observedTrace.push({ type: `publication` }) + }, + { includeInitialState: false }, + ) + subscription.on(`status:change`, ({ status }) => { + observedStatuses.push(status) + observedTrace.push({ type: `status`, status }) + }) + subscription.on(`loadSubset:error`, ({ options, error }) => { + const attemptId = attemptByOptions.get(options) ?? `unacquired` + observedErrors.push({ attemptId, error }) + observedTrace.push({ type: `error`, attemptId }) + }) + + const assertState = (command: LifecycleCommand) => { + const context = JSON.stringify({ + history, + command, + observedTrace, + expectedTrace: model.trace, + }) + check(observedLoads, context).toEqual(model.loads) + check(observedUnloads, context).toEqual(model.unloads) + check( + observedErrors.map(({ attemptId }) => attemptId), + context, + ).toEqual(model.errors.map(({ attemptId }) => attemptId)) + for (const [index, { error }] of observedErrors.entries()) { + check(error, context).toBe(model.errors[index]?.error) + } + check(observedResults, context).toEqual(model.results) + check(observedStatuses, context).toEqual(model.statuses) + check(subscription.status, context).toBe(model.status) + check(subscription.lastError, context).toBe(model.lastError) + check(collection.status, context).toBe(model.collectionStatus) + check(publications, context).toEqual( + Array.from({ length: model.publications }, () => []), + ) + check(observedTrace, context).toEqual(model.trace) + for (const attempt of model.attempts) { + check( + runtimeAttempts.get(attempt.id)?.options.signal?.aborted, + context, + ).toBe(attempt.aborted) + } + } + + const selectRuntimeAttempt = ( + command: Extract, + ): RuntimeAttempt | undefined => { + if (observedUnsubscribed) return undefined + const candidates = [...runtimeAttempts.values()].filter( + (attempt) => + !attempt.settled && + attempt.demand === command.demand && + attempt.current === (command.scope === `current`), + ) + return command.age === `oldest` ? candidates[0] : candidates.at(-1) + } + + try { + for (const command of history) { + const runtimeOwner = + command.type === `request` + ? { + id: nextObservedOwnerId++, + demand: command.demand, + controller: new AbortController(), + aborted: false, + } + : command.type === `abort` + ? runtimeOwners.find( + ({ demand, aborted }) => demand === command.demand && !aborted, + ) + : command.type === `release` + ? runtimeOwners.find(({ demand }) => demand === command.demand) + : undefined + if (command.type === `request` && !model.unsubscribed) { + runtimeOwners.push(runtimeOwner!) + } + const runtimeAttempt = + command.type === `settle` ? selectRuntimeAttempt(command) : undefined + const effect = reduceLifecycle(model, command) + if (command.type === `request`) { + check(effect.ownerId).toBe( + model.unsubscribed ? undefined : runtimeOwner?.id, + ) + const result = subscription.requestSnapshot({ + where: where[command.demand], + signal: runtimeOwner?.controller.signal, + onLoadSubsetResult: (result, requestOptions) => { + const attemptId = + attemptByOptions.get(requestOptions) ?? `unacquired` + const resultKind = result === true ? `true` : `promise` + observedResults.push({ attemptId, resultKind }) + observedTrace.push({ type: `result`, attemptId, resultKind }) + }, + }) + check(result).toBe(effect.requestResult) + } else if (command.type === `abort`) { + check(effect.ownerId).toBe(runtimeOwner?.id) + if (runtimeOwner) { + runtimeOwner.aborted = true + runtimeOwner.controller.abort() + } + } else if (command.type === `release`) { + check(effect.ownerId).toBe(runtimeOwner?.id) + if (runtimeOwner) { + if (runtimeOwner.attemptId !== undefined) { + runtimeAttempts.get(runtimeOwner.attemptId)!.current = false + } + runtimeOwners.splice(runtimeOwners.indexOf(runtimeOwner), 1) + } + subscription.releaseSnapshot(where[command.demand]) + } else if (command.type === `settle`) { + check(effect.attemptId).toBe(runtimeAttempt?.id) + if (effect.attemptId === undefined) { + // Neither model found an effective settlement. + } else if (!runtimeAttempt?.deferred) { + throw new Error(`model selected an already settled acquisition`) + } else { + runtimeAttempt.settled = true + if (command.outcome === `resolve`) runtimeAttempt.deferred.resolve() + else runtimeAttempt.deferred.reject(runtimeAttempt.failure) + } + } else if (command.type === `truncate`) { + if (observedActive) { + observedReplay++ + for (const owner of runtimeOwners) { + if (owner.attemptId !== undefined) { + runtimeAttempts.get(owner.attemptId)!.current = false + } + owner.attemptId = undefined + } + } + syncOps?.begin() + syncOps?.truncate() + const receipt = syncOps?.commit() + if (observedActive && !observedUnsubscribed && model.owners.length) { + check(subscription.status).toBe(`loadingSubset`) + } + if (receipt !== true) await receipt + } else if (command.type === `cleanup`) { + if (observedActive) { + for (const owner of runtimeOwners) { + if (owner.attemptId !== undefined) { + runtimeAttempts.get(owner.attemptId)!.current = false + } + owner.attemptId = undefined + } + } + await collection.cleanup() + observedActive = false + } else if (command.type === `restart`) { + const queuesReplay = + !observedActive && !observedUnsubscribed && model.owners.length > 0 + if (!observedActive) { + observedReplay = 0 + observedActive = true + } + collection.startSyncImmediate() + if (queuesReplay) check(subscription.status).toBe(`loadingSubset`) + } else if (command.type === `unsubscribe`) { + for (const attempt of runtimeAttempts.values()) attempt.current = false + subscription.unsubscribe() + observedUnsubscribed = true + runtimeOwners.length = 0 + } + await flushPromises() + assertState(command) + } + } finally { + for (const { deferred } of runtimeAttempts.values()) deferred?.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + return model.reach +} + +if (process.env.TANSTACK_DB_ORACLE_STATISTICS === `1`) { + fc.statistics( + greenLifecycleHistoryArbitrary, + (history) => { + const model = createLifecycleModel() + for (const command of history) reduceLifecycle(model, command) + return [...model.reach] + }, + oraclePropertyOptions(1_000, `subscription-lifecycle.history-statistics`), + ) +} + +describe(`CollectionSubscription async lifecycle history oracle`, () => { + it(`covers every required command and cross-phase transition`, async () => { + const reach = new Set() + for (const history of [ + ...greenLifecycleHistories, + ...compoundLifecycleCoverageHistories, + ]) { + for (const label of await runHistory(history)) reach.add(label) + } + const commands = [ + `request`, + `abort`, + `release`, + `settle`, + `truncate`, + `cleanup`, + `restart`, + `unsubscribe`, + ] + const required = new Set([ + ...commands.map((type) => `command:${type}`), + ...commands.map((type) => `effective:${type}`), + ...commands.map((type) => `noop:${type}`), + `settle-scope:current`, + `settle-scope:obsolete`, + `settle-age:oldest`, + `settle-age:newest`, + `settle-outcome:resolve`, + `settle-outcome:reject`, + ...([`current`, `obsolete`] as const).flatMap((scope) => + ([`oldest`, `newest`] as const).flatMap((age) => + ([`resolve`, `reject`] as const).map( + (outcome) => `settle:${scope}:${age}:${outcome}`, + ), + ), + ), + `attempt-session:initial`, + `attempt-session:restarted`, + `attempt-replay:initial`, + `attempt-replay:replayed`, + `attempt-location:initial:initial`, + `attempt-location:initial:replayed`, + `attempt-location:restarted:initial`, + `attempt-location:restarted:replayed`, + `duplicate-owner`, + `request-while-cleaned`, + `partial-generation-supersession`, + ]) + expect([...required].filter((label) => !reach.has(label))).toEqual([]) + }) + + it(`names the overlapping replay transition in the model`, () => { + const model = createLifecycleModel() + for (const command of pendingSupersessionHistory) { + reduceLifecycle(model, command) + } + expect(model.reach).toContain(`overlapping-replay`) + }) + + it.each([ + { + name: `one demand after one replay`, + history: [ + { type: `request`, demand: `a` }, + { type: `truncate` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + ] satisfies Array, + }, + { + name: `duplicate owners after overlapping replay`, + history: pendingSupersessionHistory, + }, + ])( + `waits for delayed cancellation settlement for $name`, + async ({ history }) => { + await runHistory(history) + }, + ) + + it(`releases exact ownership while older replay work is pending`, async () => { + await runHistory( + [ + ...pendingSupersessionHistory, + { type: `cleanup` }, + { type: `restart` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it.each( + ([`manual`, `reject`] as const).flatMap((cancellation) => + ([1, 2] as const).flatMap((replays) => + ([`resolve`, `reject`] as const).map((outcome) => ({ + cancellation, + replays, + outcome, + })), + ), + ), + )( + `tracks $cancellation cancellation across $replays replay(s) ending in $outcome`, + async ({ cancellation, replays, outcome }) => { + await runHistory( + [ + { type: `request`, demand: `a` }, + ...Array.from( + { length: replays }, + (): LifecycleCommand => ({ type: `truncate` }), + ), + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome, + }, + ...Array.from( + { length: replays }, + (_, index): LifecycleCommand => ({ + type: `settle`, + demand: `a`, + scope: `obsolete`, + age: `oldest`, + outcome: index % 2 === 0 ? `reject` : `resolve`, + }), + ), + { type: `release`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { cancellation }, + ) + }, + ) + + it.each([ + { name: `truncate replay`, history: abortReplayHistory }, + { name: `cleanup restart`, history: abortedRestartHistory }, + ])( + `queues replay without reacquiring an aborted demand on $name`, + async ({ history }) => { + await runHistory(history) + }, + ) + + it(`does not release an unacquired replacement after an aborted demand replays`, async () => { + await runHistory( + [ + ...abortReplayHistory, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`replays a live peer without reacquiring an aborted demand`, async () => { + await runHistory([ + { type: `request`, demand: `a` }, + { type: `request`, demand: `b` }, + { type: `abort`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `truncate` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + ]) + }) + + it(`restarts a live peer without reacquiring an aborted demand and completes teardown`, async () => { + await runHistory(mixedAbortedRestartHistory, { + continueAfterMismatch: true, + }) + }) + + const syncReplayScenarios = ([`truncate`, `restart`] as const).flatMap( + (transition) => + ([1, 2] as const).flatMap((ownerCount) => + ([false, true] as const).map((abortFirst) => { + const history: Array = [ + { type: `request`, demand: `a` }, + ...(ownerCount === 2 + ? ([{ type: `request`, demand: `b` }] as const) + : []), + ...(abortFirst ? ([{ type: `abort`, demand: `a` }] as const) : []), + ...(transition === `truncate` + ? ([{ type: `truncate` }] as const) + : ([{ type: `cleanup` }, { type: `restart` }] as const)), + ] + return { transition, ownerCount, abortFirst, history } + }), + ), + ) + + it.each(syncReplayScenarios)( + `settles queued synchronous $transition with $ownerCount owner(s), abort=$abortFirst`, + async ({ history }) => { + await runHistory(history, { acquisitionMode: `sync-success` }) + }, + ) + + it(`preserves physical ownership across queued synchronous replay`, async () => { + await runHistory(syncLifecycleHistory, { + acquisitionMode: `sync-success`, + continueAfterMismatch: true, + }) + }) + + it.each([ + { + name: `same-key owners across truncate`, + history: [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `a` }, + { type: `truncate` }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + }, + { + name: `same-key owners across restart`, + history: [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ], + }, + { + name: `aborted owner across repeated truncate`, + history: [ + { type: `request`, demand: `a` }, + { type: `abort`, demand: `a` }, + { type: `truncate` }, + { type: `truncate` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ], + }, + { + name: `detached last-owner abort across restart`, + history: [ + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `abort`, demand: `a` }, + { type: `restart` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ], + }, + ] satisfies ReadonlyArray<{ + name: string + history: ReadonlyArray + }>)( + `continues through the full synchronous replay suffix for $name`, + async ({ history }) => { + await runHistory(history, { + acquisitionMode: `sync-success`, + continueAfterMismatch: true, + }) + }, + ) + + it(`publishes a new snapshot while canceled initial work still holds readiness`, async () => { + // Seed 1413322355, path 757:13:15:15:9:9:9. This checks an empty snapshot + // notification: initial readiness is not a replacement publication gate. + await runHistory( + [ + { type: `request`, demand: `b` }, + { type: `truncate` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { + type: `settle`, + demand: `b`, + scope: `obsolete`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`keeps a new snapshot private after failed restart`, async () => { + // Minimized from seed 317005625 at 100×. The mismatch is an empty + // notification, not lost rows: failed replacement must keep reads private. + await runHistory( + [ + { type: `request`, demand: `b` }, + { type: `cleanup` }, + { type: `restart` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + const { multiplier, ...replay } = readOracleRunConfig() + const runs = 80 * multiplier + const cancellationArbitrary = fc.constantFrom( + `manual` as const, + `reject` as const, + ) + + fcTest.prop([greenLifecycleHistoryArbitrary, cancellationArbitrary], { + numRuns: runs, + seed: 1_657_003, + })( + `matches the pure lifecycle model for a fixed seed`, + async (history, cancellation) => { + await runHistory(history, { cancellation }) + }, + 120_000, + ) + fcTest.prop( + [greenLifecycleHistoryArbitrary, cancellationArbitrary], + oracleRandomParameters( + runs, + replay, + `subscription-lifecycle.async-history`, + ), + )( + `matches the pure lifecycle model for a random or replayed seed`, + async (history, cancellation) => { + await runHistory(history, { cancellation }) + }, + 120_000, + ) + fcTest.prop([syncLifecycleHistoryArbitrary], { + numRuns: runs, + seed: 1_657_004, + })( + `matches synchronous success histories for a fixed seed`, + async (history) => { + await runHistory(history, { acquisitionMode: `sync-success` }) + }, + 120_000, + ) + fcTest.prop( + [syncLifecycleHistoryArbitrary], + oracleRandomParameters(runs, replay, `subscription-lifecycle.sync-history`), + )( + `matches synchronous success histories for a random or replayed seed`, + async (history) => { + await runHistory(history, { acquisitionMode: `sync-success` }) + }, + 120_000, + ) +}) diff --git a/packages/db/tests/collection-subscription-lifecycle-oracle.test.ts b/packages/db/tests/collection-subscription-lifecycle-oracle.test.ts new file mode 100644 index 0000000000..31eb18be51 --- /dev/null +++ b/packages/db/tests/collection-subscription-lifecycle-oracle.test.ts @@ -0,0 +1,4111 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { afterAll, describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { createOnDemandCollection, flushPromises } from './utils.js' +import { + oraclePropertyOptions, + oracleRandomParameters, + readOracleRunConfig, +} from './oracle-config.js' +import type { CollectionSubscription } from '../src/collection/subscription.js' +import type { LoadSubsetOptions, SyncConfig } from '../src/types.js' + +type StartOutcome = `return` | `throw` | `resolve` | `reject` +type StartReentry = + | `none` + | `abort-self` + | `truncate` + | `release-self` + | `release-peer` + | `unsubscribe` + | `cleanup` +type RestartReentry = + | `none` + | `release-self` + | `release-peer` + | `unsubscribe` + | `cleanup` + +const acquisitionPhases = [ + `deferred`, + `starting`, + `on-demand`, + `eager`, + `retiring`, + `unavailable`, +] as const +const acquisitionEntries = [ + `request`, + `resume`, + `markReady`, + `markError`, + `syncReturn`, +] as const +type AcquisitionPhase = (typeof acquisitionPhases)[number] +type AcquisitionEntry = (typeof acquisitionEntries)[number] +type AcquisitionCell = `${AcquisitionPhase}:${AcquisitionEntry}` + +type AcquisitionCellDefinition = + | { kind: `covered` } + | { kind: `excluded`; reason: string } + +const acquisitionCellDefinitions = { + 'deferred:request': { kind: `covered` }, + 'deferred:resume': { kind: `covered` }, + 'deferred:markReady': { + kind: `excluded`, + reason: `the sync callback has not started and cannot mark ready`, + }, + 'deferred:markError': { + kind: `excluded`, + reason: `the sync callback has not started and cannot mark error`, + }, + 'deferred:syncReturn': { + kind: `excluded`, + reason: `the deferred sync callback has no result to return`, + }, + 'starting:request': { kind: `covered` }, + 'starting:resume': { + kind: `excluded`, + reason: `resuming the deferred gate enters this phase only once`, + }, + 'starting:markReady': { kind: `covered` }, + 'starting:markError': { kind: `covered` }, + 'starting:syncReturn': { kind: `covered` }, + 'on-demand:request': { kind: `covered` }, + 'on-demand:resume': { + kind: `excluded`, + reason: `an installed loader is no longer behind the deferred gate`, + }, + 'on-demand:markReady': { kind: `covered` }, + 'on-demand:markError': { kind: `covered` }, + 'on-demand:syncReturn': { + kind: `excluded`, + reason: `the sync callback already returned the installed loader`, + }, + 'eager:request': { kind: `covered` }, + 'eager:resume': { + kind: `excluded`, + reason: `eager sync is not a deferred subset acquisition`, + }, + 'eager:markReady': { + kind: `excluded`, + reason: `eager readiness does not install a subset loader`, + }, + 'eager:markError': { + kind: `excluded`, + reason: `eager errors do not change subset acquisition availability`, + }, + 'eager:syncReturn': { + kind: `excluded`, + reason: `eager sync results own no subset loader contract`, + }, + 'retiring:request': { kind: `covered` }, + 'retiring:resume': { + kind: `excluded`, + reason: `retirement is outside the deferred-start gate`, + }, + 'retiring:markReady': { + kind: `excluded`, + reason: `callbacks from a retiring session cannot restore availability`, + }, + 'retiring:markError': { + kind: `excluded`, + reason: `callbacks from a retiring session are obsolete`, + }, + 'retiring:syncReturn': { + kind: `excluded`, + reason: `obsolete returned resources use the resource-installation axis`, + }, + 'unavailable:request': { kind: `covered` }, + 'unavailable:resume': { + kind: `excluded`, + reason: `same-session recovery uses markReady rather than defer resume`, + }, + 'unavailable:markReady': { kind: `covered` }, + 'unavailable:markError': { + kind: `excluded`, + reason: `a repeated error leaves acquisition unavailable`, + }, + 'unavailable:syncReturn': { + kind: `excluded`, + reason: `handler-less return is the transition into unavailable`, + }, +} satisfies Record + +const legalAcquisitionCells = new Set( + Object.entries(acquisitionCellDefinitions) + .filter(([, definition]) => definition.kind === `covered`) + .map(([cell]) => cell as AcquisitionCell), +) +const excludedAcquisitionCells = new Map( + Object.entries(acquisitionCellDefinitions).flatMap(([cell, definition]) => + definition.kind === `excluded` + ? [[cell as AcquisitionCell, definition.reason]] + : [], + ), +) +const observedAcquisitionCells = new Set() + +function acquisitionCase( + cells: ReadonlyArray, + name: string, + run: (reach: (cell: AcquisitionCell) => void) => void | Promise, +): void { + const declaredCells = new Set(cells) + it(name, () => + run((cell) => { + if (!declaredCells.has(cell)) { + throw new Error(`${name} reached undeclared acquisition cell ${cell}`) + } + observedAcquisitionCells.add(cell) + }), + ) +} + +const physicalAcquisitionStates = [ + `none`, + `starting`, + `active`, + `obsolete`, + `failed-release`, +] as const +const physicalInteractionCauses = [ + `release`, + `abort`, + `truncate`, + `cleanup`, + `unsubscribe`, +] as const +type PhysicalAcquisitionState = (typeof physicalAcquisitionStates)[number] +type PhysicalInteractionCause = (typeof physicalInteractionCauses)[number] +type PhysicalInteractionCell = + `${PhysicalAcquisitionState}:${PhysicalInteractionCause}` +type PhysicalInteraction = + | `no-acquisition` + | `abort-only` + | `retire` + | `no-repeat-on-truncate` + | `no-repeat-on-cleanup` + | `no-repeat-on-unsubscribe` +type PhysicalInteractionCellDefinition = + | { kind: `covered`; interaction: PhysicalInteraction } + | { kind: `excluded`; reason: string } + +const physicalInteractionCellDefinitions = { + 'none:release': { + kind: `covered`, + interaction: `no-acquisition`, + }, + 'none:abort': { + kind: `covered`, + interaction: `no-acquisition`, + }, + 'none:truncate': { + kind: `covered`, + interaction: `no-acquisition`, + }, + 'none:cleanup': { + kind: `covered`, + interaction: `no-acquisition`, + }, + 'none:unsubscribe': { + kind: `covered`, + interaction: `no-acquisition`, + }, + 'starting:release': { kind: `covered`, interaction: `retire` }, + 'starting:abort': { kind: `covered`, interaction: `abort-only` }, + 'starting:truncate': { kind: `covered`, interaction: `retire` }, + 'starting:cleanup': { kind: `covered`, interaction: `retire` }, + 'starting:unsubscribe': { kind: `covered`, interaction: `retire` }, + 'active:release': { kind: `covered`, interaction: `retire` }, + 'active:abort': { + kind: `covered`, + interaction: `abort-only`, + }, + 'active:truncate': { kind: `covered`, interaction: `retire` }, + 'active:cleanup': { kind: `covered`, interaction: `retire` }, + 'active:unsubscribe': { kind: `covered`, interaction: `retire` }, + 'obsolete:release': { + kind: `excluded`, + reason: `the replacement owns later release; obsolete work was retired once`, + }, + 'obsolete:abort': { + kind: `excluded`, + reason: `obsolete work was already signaled and retired`, + }, + 'obsolete:truncate': { + kind: `excluded`, + reason: `another truncate retires the current replacement, not already-obsolete work`, + }, + 'obsolete:cleanup': { + kind: `excluded`, + reason: `source cleanup retires the current session; obsolete work was retired once`, + }, + 'obsolete:unsubscribe': { + kind: `excluded`, + reason: `unsubscribe retires current ownership; obsolete work was retired once`, + }, + 'failed-release:release': { + kind: `excluded`, + reason: `logical release already happened; the physical attempt is final`, + }, + 'failed-release:abort': { + kind: `excluded`, + reason: `the failed physical release is already aborted`, + }, + 'failed-release:truncate': { + kind: `covered`, + interaction: `no-repeat-on-truncate`, + }, + 'failed-release:cleanup': { + kind: `covered`, + interaction: `no-repeat-on-cleanup`, + }, + 'failed-release:unsubscribe': { + kind: `covered`, + interaction: `no-repeat-on-unsubscribe`, + }, +} satisfies Record + +const requiredPhysicalInteractions = new Map< + PhysicalInteractionCell, + PhysicalInteraction +>( + Object.entries(physicalInteractionCellDefinitions).flatMap( + ([cell, definition]) => + definition.kind === `covered` + ? [[cell as PhysicalInteractionCell, definition.interaction]] + : [], + ), +) +const observedPhysicalInteractions = new Map< + PhysicalInteractionCell, + PhysicalInteraction +>() + +function observePhysicalInteraction( + cell: PhysicalInteractionCell, + interaction: PhysicalInteraction, +): void { + observedPhysicalInteractions.set(cell, interaction) +} + +const requiredSourceSessionBoundaries = new Set([ + `active-cleanup`, + `restart-installed`, + `cleanup-callback-reentry`, + `obsolete-resource-return`, +] as const) +type SourceSessionBoundary = + typeof requiredSourceSessionBoundaries extends Set ? T : never +const observedSourceSessionBoundaries = new Set() + +function observeSourceSessionBoundary(boundary: SourceSessionBoundary): void { + observedSourceSessionBoundaries.add(boundary) +} + +const startOutcomes = [`return`, `throw`, `resolve`, `reject`] as const +const startReentries = [ + `none`, + `abort-self`, + `truncate`, + `release-self`, + `release-peer`, + `unsubscribe`, + `cleanup`, +] as const + +type StartScenario = { + outcome: StartOutcome + reentry: StartReentry +} + +const startScenarios: ReadonlyArray = startOutcomes.flatMap( + (outcome) => startReentries.map((reentry) => ({ outcome, reentry })), +) + +const failureScenarios = ([`throw`, `reject`] as const).flatMap((outcome) => + startReentries.map((reentry) => ({ outcome, reentry })), +) +type FailureDeliverySuffix = `${`throw` | `reject`}:${StartReentry}` +const requiredFailureDeliverySuffixes = new Set( + failureScenarios.map( + ({ outcome, reentry }) => `${outcome}:${reentry}` as const, + ), +) +const observedFailureDeliverySuffixes = new Set() + +const releaseScenarios = ([`return`, `throw`] as const).flatMap((outcome) => + ([`none`, `reacquire-self`, `release-peer`, `unsubscribe`] as const).map( + (reentry) => ({ outcome, reentry }), + ), +) + +const restartScenarios = startOutcomes.flatMap((outcome) => + ( + [`none`, `release-self`, `release-peer`, `unsubscribe`, `cleanup`] as const + ).map((reentry: RestartReentry) => ({ outcome, reentry })), +) + +const threeGenerationScenarios = ([`resolve`, `reject`] as const).flatMap( + (obsoleteOutcome) => + ([`resolve`, `reject`] as const).flatMap((currentOutcome) => + ([`obsolete-first`, `current-first`] as const).map((settlementOrder) => ({ + obsoleteOutcome, + currentOutcome, + settlementOrder, + })), + ), +) + +type AsyncRestartScenario = { + demands: ReadonlyArray<`a` | `b`> + generationOutcomes: ReadonlyArray> + settlementOrder: `obsolete-first` | `current-first` | `interleaved` +} + +const asyncRestartCoverageScenarios = [ + { + demands: [`a`], + generationOutcomes: [[`reject`]], + settlementOrder: `current-first`, + }, + { + demands: [`a`], + generationOutcomes: [[`resolve`], [`resolve`]], + settlementOrder: `obsolete-first`, + }, + { + demands: [`a`, `b`], + generationOutcomes: [ + [`reject`, `resolve`], + [`resolve`, `reject`], + ], + settlementOrder: `current-first`, + }, + { + demands: [`a`, `b`], + generationOutcomes: [ + [`resolve`, `resolve`], + [`reject`, `reject`], + [`resolve`, `resolve`], + ], + settlementOrder: `interleaved`, + }, +] as const satisfies ReadonlyArray + +const asyncRestartScenarioArbitrary: fc.Arbitrary = fc + .uniqueArray(fc.constantFrom(`a` as const, `b` as const), { + minLength: 1, + maxLength: 2, + }) + .chain((demands) => + fc.record({ + demands: fc.constant(demands), + generationOutcomes: fc.array( + fc.array(fc.constantFrom(`resolve` as const, `reject` as const), { + minLength: demands.length, + maxLength: demands.length, + }), + { minLength: 1, maxLength: 3 }, + ), + settlementOrder: fc.constantFrom( + `obsolete-first` as const, + `current-first` as const, + `interleaved` as const, + ), + }), + ) + +if (process.env.TANSTACK_DB_ORACLE_STATISTICS === `1`) { + fc.statistics( + asyncRestartScenarioArbitrary, + ({ demands, generationOutcomes, settlementOrder }) => { + const realizesInterleaving = + settlementOrder === `interleaved` && + demands.length > 1 && + generationOutcomes.length > 1 + return [ + `demands=${demands.length}`, + `generations=${generationOutcomes.length + 1}`, + `current=${generationOutcomes.at(-1)?.join(`+`)}`, + `mixed-current=${new Set(generationOutcomes.at(-1)).size > 1}`, + `obsolete-reject=${generationOutcomes + .slice(0, -1) + .some((outcomes) => outcomes.includes(`reject`))}`, + `requested-order=${ + settlementOrder === `interleaved` && !realizesInterleaving + ? `degenerate-interleaved` + : settlementOrder + }`, + ] + }, + oraclePropertyOptions(1_000, `subscription-lifecycle.async-statistics`), + ) +} + +async function runAsyncRestartScenario( + scenario: AsyncRestartScenario, +): Promise> { + type DemandName = `a` | `b` + type Row = { id: DemandName; version: number } + type Attempt = { + session: number + demand: DemandName + options: LoadSubsetOptions + deferred: ReturnType> + } + type SettlementEvent = { + attempt: Attempt + session: number + demand: DemandName + outcome: `resolve` | `reject` + activeSession: number + } + const where = { + a: new Func(`eq`, [new PropRef([`id`]), new Value(`a`)]), + b: new Func(`eq`, [new PropRef([`id`]), new Value(`b`)]), + } + const demandForWhere = new Map([ + [where.a, `a`], + [where.b, `b`], + ]) + const attempts: Array = [] + const errors: Array<{ demand: DemandName; error: unknown }> = [] + const publications: Array> = [] + const statuses: Array = [] + const visible = new Map() + const unloads: Array<{ session: number; demand: DemandName }> = [] + const settlements: Array = [] + const settledAttempts = new Set() + const publishedBeforeRetirement = new Set() + const failures = scenario.generationOutcomes.map((_, generation) => + scenario.demands.map( + (demand) => new Error(`session ${generation + 1} ${demand} failed`), + ), + ) + let session = -1 + + const outcomeFor = (attempt: Attempt) => + scenario.generationOutcomes[attempt.session - 1]![ + scenario.demands.indexOf(attempt.demand) + ]! + const failureFor = (attempt: Attempt) => + failures[attempt.session - 1]![scenario.demands.indexOf(attempt.demand)]! + + const settleAttempt = async (attempt: Attempt): Promise => { + const outcome = outcomeFor(attempt) + if (outcome === `resolve`) attempt.deferred.resolve() + else attempt.deferred.reject(failureFor(attempt)) + await flushPromises() + settlements.push({ + attempt, + session: attempt.session, + demand: attempt.demand, + outcome, + activeSession: session, + }) + settledAttempts.add(attempt) + } + + const collection = createOnDemandCollection({ + id: `async-restart-lifecycle`, + sync: { + sync: (operations) => { + session++ + const ownSession = session + operations.markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown async demand`) + const deferred = createDeferred() + void deferred.promise.catch(() => {}) + attempts.push({ + session: ownSession, + demand, + options, + deferred, + }) + return deferred.promise.then(() => { + operations.begin() + operations.write({ + type: `insert`, + value: { id: demand, version: ownSession + 1 }, + }) + const receipt = operations.commit() + if (receipt !== true) return receipt + return undefined + }) + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown async demand`) + unloads.push({ session: ownSession, demand }) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + publications.push( + [...visible.values()].sort((a, b) => a.id.localeCompare(b.id)), + ) + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ options, error }) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown errored demand`) + errors.push({ demand, error }) + }) + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + + try { + for (const demand of scenario.demands) { + subscription.requestSnapshot({ where: where[demand] }) + } + for (const attempt of attempts.filter( + ({ session: value }) => value === 0, + )) { + attempt.deferred.resolve() + } + await flushPromises() + expect( + [...visible.values()].sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual( + [...scenario.demands] + .sort((a, b) => a.localeCompare(b)) + .map((id) => ({ id, version: 1 })), + ) + + for ( + let generation = 0; + generation < scenario.generationOutcomes.length; + generation++ + ) { + const discardedSession = session + await collection.cleanup() + for (const attempt of attempts.filter( + ({ session: attemptSession }) => attemptSession === discardedSession, + )) { + expect(attempt.options.signal?.aborted).toBe(true) + } + collection.startSyncImmediate() + await flushPromises() + const expectedSession = generation + 1 + expect( + attempts + .filter( + ({ session: attemptSession }) => attemptSession <= expectedSession, + ) + .map(({ session: attemptSession, demand }) => ({ + session: attemptSession, + demand, + })), + ).toEqual( + Array.from({ length: expectedSession + 1 }, (_, attemptSession) => + scenario.demands.map((demand) => ({ + session: attemptSession, + demand, + })), + ).flat(), + ) + + const publishesBeforeLaterRestart = + scenario.settlementOrder === `interleaved` && + generation === 0 && + scenario.generationOutcomes.length > 1 && + scenario.generationOutcomes[generation]!.every( + (outcome) => outcome === `resolve`, + ) + if (publishesBeforeLaterRestart) { + const publicationCount = publications.length + for (const attempt of attempts.filter( + ({ session: attemptSession }) => attemptSession === expectedSession, + )) { + await settleAttempt(attempt) + } + const expectedRows = [...scenario.demands] + .sort((a, b) => a.localeCompare(b)) + .map((id) => ({ id, version: expectedSession + 1 })) + expect( + [...visible.values()].sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual(expectedRows) + expect(publications.slice(publicationCount)).toEqual([expectedRows]) + publishedBeforeRetirement.add(expectedSession) + } + } + + const currentSession = scenario.generationOutcomes.length + expect( + attempts.map(({ session: attemptSession, demand }) => ({ + session: attemptSession, + demand, + })), + ).toEqual( + Array.from({ length: currentSession + 1 }, (_, attemptSession) => + scenario.demands.map((demand) => ({ + session: attemptSession, + demand, + })), + ).flat(), + ) + const obsolete = attempts.filter( + (attempt) => + attempt.session > 0 && + attempt.session < currentSession && + !settledAttempts.has(attempt), + ) + const current = attempts.filter( + ({ session: value }) => value === currentSession, + ) + const orderedAttempts = + scenario.settlementOrder === `obsolete-first` + ? [...obsolete, ...current] + : scenario.settlementOrder === `current-first` + ? [...current, ...obsolete] + : attempts + .filter(({ session: value }) => value > 0) + .filter((attempt) => !settledAttempts.has(attempt)) + .sort((left, right) => + left.demand === right.demand + ? right.session - left.session + : left.demand.localeCompare(right.demand), + ) + + const settledCurrent: Array = [] + const publicationTraceStart = publications.length + const statusTraceStart = statuses.length + const retainedVersion = publishedBeforeRetirement.size + ? Math.max(...publishedBeforeRetirement) + 1 + : 1 + const assertObservableState = () => { + const currentComplete = settledCurrent.length === current.length + const currentSucceeded = current.every( + (attempt) => outcomeFor(attempt) === `resolve`, + ) + const visibleVersion = + currentComplete && currentSucceeded + ? currentSession + 1 + : retainedVersion + const expectedRows = [...scenario.demands] + .sort((a, b) => a.localeCompare(b)) + .map((id) => ({ id, version: visibleVersion })) + expect( + [...visible.values()].sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual(expectedRows) + const expectedFailedAttempts = settledCurrent.filter( + (attempt) => outcomeFor(attempt) === `reject`, + ) + expect(errors.map(({ demand }) => demand)).toEqual( + expectedFailedAttempts.map(({ demand }) => demand), + ) + for (const [index, { error }] of errors.entries()) { + expect(error).toBe(failureFor(expectedFailedAttempts[index]!)) + } + expect(subscription.lastError).toBe( + expectedFailedAttempts.length + ? failureFor(expectedFailedAttempts.at(-1)!) + : undefined, + ) + expect(subscription.status).toBe( + currentComplete ? `ready` : `loadingSubset`, + ) + expect(publications.slice(publicationTraceStart)).toEqual( + currentComplete && currentSucceeded ? [expectedRows] : [], + ) + expect(statuses.slice(statusTraceStart)).toEqual( + currentComplete ? [`ready`] : [], + ) + } + + for (const attempt of orderedAttempts) { + await settleAttempt(attempt) + if (attempt.session === currentSession) settledCurrent.push(attempt) + assertObservableState() + } + + const currentSucceeded = current.every( + (attempt) => outcomeFor(attempt) === `resolve`, + ) + const expectedVersion = currentSucceeded + ? currentSession + 1 + : retainedVersion + expect( + [...visible.values()].sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual( + [...scenario.demands] + .sort((a, b) => a.localeCompare(b)) + .map((id) => ({ id, version: expectedVersion })), + ) + if (currentSucceeded) { + expect(errors).toEqual([]) + expect(subscription.lastError).toBeUndefined() + } else { + const expectedFailedAttempts = settledCurrent.filter( + (attempt) => outcomeFor(attempt) === `reject`, + ) + expect(errors.map(({ demand }) => demand)).toEqual( + expectedFailedAttempts.map(({ demand }) => demand), + ) + for (const [index, { error }] of errors.entries()) { + expect(error).toBe(failureFor(expectedFailedAttempts[index]!)) + } + expect(subscription.lastError).toBe( + failureFor(expectedFailedAttempts.at(-1)!), + ) + } + expect(subscription.status).toBe(`ready`) + for (const attempt of current) { + expect(attempt.options.signal?.aborted).toBe(false) + } + + const finalScopes = settlements.map(({ session: attemptSession }) => + attemptSession === currentSession ? `current` : `obsolete`, + ) + const firstCurrent = finalScopes.indexOf(`current`) + const lastCurrent = finalScopes.lastIndexOf(`current`) + const firstObsolete = finalScopes.indexOf(`obsolete`) + const lastObsolete = finalScopes.lastIndexOf(`obsolete`) + const observedOrder = + firstCurrent === -1 || firstObsolete === -1 + ? undefined + : lastObsolete < firstCurrent + ? `obsolete-first` + : lastCurrent < firstObsolete + ? `current-first` + : `interleaved` + const currentOutcomes = settlements + .filter( + ({ session: attemptSession }) => attemptSession === currentSession, + ) + .map(({ outcome }) => outcome) + expect(new Set(settlements.map(({ attempt }) => attempt)).size).toBe( + settlements.length, + ) + expect(settlements).toHaveLength( + attempts.filter(({ session: attemptSession }) => attemptSession > 0) + .length, + ) + const reach = new Set([ + `demands:${new Set(attempts.map(({ demand }) => demand)).size}`, + `sessions:${new Set(attempts.map(({ session: attemptSession }) => attemptSession)).size}`, + ...[...new Set(currentOutcomes)].map((outcome) => `current:${outcome}`), + `mixed-current:${new Set(currentOutcomes).size > 1}`, + `obsolete-reject:${settlements.some( + ({ session: attemptSession, outcome }) => + attemptSession < currentSession && outcome === `reject`, + )}`, + ...(observedOrder ? [`order:${observedOrder}`] : []), + `real-interleaving:${settlements.some( + ({ session: attemptSession, activeSession }) => + attemptSession < currentSession && + activeSession === attemptSession && + publishedBeforeRetirement.has(attemptSession), + )}`, + ]) + + subscription.unsubscribe() + for (const attempt of current) { + expect(attempt.options.signal?.aborted).toBe(true) + } + expect(unloads).toEqual( + scenario.demands.map((demand) => ({ + session: currentSession, + demand, + })), + ) + return reach + } finally { + subscription.unsubscribe() + await collection.cleanup() + } +} + +/** + * Exhaust the synchronous adapter-start boundary before adding more runtime + * special cases. Logical demand is visible during this callback, but a + * physical lease exists only if the callback returns. + */ +describe(`CollectionSubscription demand lifecycle oracle`, () => { + it(`executes every required async restart regime`, async () => { + const reach = new Set() + for (const scenario of asyncRestartCoverageScenarios) { + for (const label of await runAsyncRestartScenario(scenario)) { + reach.add(label) + } + } + const required = [ + `demands:1`, + `demands:2`, + `sessions:2`, + `sessions:3`, + `sessions:4`, + `current:resolve`, + `current:reject`, + `mixed-current:true`, + `obsolete-reject:true`, + `order:obsolete-first`, + `order:current-first`, + `order:interleaved`, + `real-interleaving:true`, + ] + expect(required.filter((label) => !reach.has(label))).toEqual([]) + }) + + it(`covers every finite start, failure-delivery, and release cell`, () => { + expect( + new Set( + startScenarios.map(({ outcome, reentry }) => `${outcome}:${reentry}`), + ), + ).toHaveLength(startOutcomes.length * startReentries.length) + expect( + new Set( + failureScenarios.map(({ outcome, reentry }) => `${outcome}:${reentry}`), + ), + ).toHaveLength(2 * startReentries.length) + expect( + new Set( + releaseScenarios.map(({ outcome, reentry }) => `${outcome}:${reentry}`), + ), + ).toHaveLength(2 * 4) + expect( + new Set( + restartScenarios.map(({ outcome, reentry }) => `${outcome}:${reentry}`), + ), + ).toHaveLength(4 * 5) + }) + + it(`accounts for every acquisition phase and entry pair`, () => { + const allCells = new Set( + acquisitionPhases.flatMap((phase) => + acquisitionEntries.map((entry) => `${phase}:${entry}` as const), + ), + ) + expect( + new Set([...legalAcquisitionCells, ...excludedAcquisitionCells.keys()]), + ).toEqual(allCells) + }) + + it(`accounts for every physical acquisition state and interaction cause`, () => { + const allCells = new Set( + physicalAcquisitionStates.flatMap((state) => + physicalInteractionCauses.map((cause) => `${state}:${cause}` as const), + ), + ) + expect(new Set(Object.keys(physicalInteractionCellDefinitions))).toEqual( + allCells, + ) + }) + + afterAll(() => { + expect(observedAcquisitionCells).toEqual(legalAcquisitionCells) + expect(observedPhysicalInteractions).toEqual(requiredPhysicalInteractions) + expect(observedSourceSessionBoundaries).toEqual( + requiredSourceSessionBoundaries, + ) + expect(observedFailureDeliverySuffixes).toEqual( + requiredFailureDeliverySuffixes, + ) + }) + + it.each(startScenarios)( + `keeps logical and physical ownership aligned for $outcome × $reentry`, + async ({ outcome, reentry }) => { + const targetWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`target`), + ]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`peer`)]) + const failure = new Error(`target load failed`) + const pending = createDeferred() + // A reentrant release can make the subscription stop observing the + // adapter Promise. Keep the test process deterministic while separately + // asserting the subscription's public error trace below. + void pending.promise.catch(() => {}) + const loads: Array = [] + const unloads: Array = [] + const errors: Array = [] + const statuses: Array = [] + const controller = new AbortController() + let didReenter = false + let statusAtTruncate: string | undefined + let truncate!: () => void + let runReentry = () => {} + + const collection = createOnDemandCollection<{ id: string }>({ + id: `demand-start-${outcome}-${reentry}`, + sync: { + sync: (operations) => { + const { markReady } = operations + truncate = () => { + operations.begin() + operations.truncate() + operations.commit() + } + markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (options.where === peerWhere) return true + if (didReenter) return true + didReenter = true + runReentry() + if (outcome === `throw`) throw failure + if (outcome === `return`) return true + return pending.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + + if (reentry === `release-peer`) { + subscription.requestSnapshot({ where: peerWhere }) + } + runReentry = () => { + if (reentry === `abort-self`) { + controller.abort() + } else if (reentry === `truncate`) { + truncate() + statusAtTruncate = subscription.status + } else if (reentry === `release-self`) { + subscription.releaseSnapshot(targetWhere) + } else if (reentry === `release-peer`) { + subscription.releaseSnapshot(peerWhere) + } else if (reentry === `unsubscribe`) { + subscription.unsubscribe() + } else if (reentry === `cleanup`) { + void collection.cleanup() + } + } + + let thrown: unknown + try { + subscription.requestSnapshot({ + where: targetWhere, + signal: controller.signal, + }) + } catch (error) { + thrown = error + } + + if (reentry === `truncate`) { + // The old acquisition is obsolete, but the queued replacement still + // owns a loading interval until its setup and work finish. + expect(statusAtTruncate).toBe(`loadingSubset`) + expect(subscription.status).toBe(`loadingSubset`) + expect(loads).toHaveLength(1) + } + const targetLoad = loads.find(({ where }) => where === targetWhere)! + const peerLoad = loads.find(({ where }) => where === peerWhere) + const targetWasReleased = + reentry === `release-self` || + reentry === `truncate` || + reentry === `unsubscribe` || + reentry === `cleanup` + const targetStarted = outcome !== `throw` && reentry !== `cleanup` + + if (outcome === `resolve`) pending.resolve() + if (outcome === `reject`) pending.reject(failure) + await flushPromises() + + const interaction = + reentry === `abort-self` + ? `starting:abort` + : reentry === `truncate` + ? `starting:truncate` + : reentry === `release-self` + ? `starting:release` + : reentry === `release-peer` + ? `active:release` + : reentry === `unsubscribe` + ? `starting:unsubscribe` + : reentry === `cleanup` + ? `starting:cleanup` + : undefined + if (interaction) { + observePhysicalInteraction( + interaction, + reentry === `abort-self` ? `abort-only` : `retire`, + ) + } + + expect(thrown).toBe(outcome === `throw` ? failure : undefined) + expect(targetLoad.signal?.aborted).toBe( + outcome === `throw` || targetWasReleased || reentry === `abort-self`, + ) + expect(unloads.filter((options) => options === targetLoad)).toHaveLength( + Number(targetStarted && targetWasReleased), + ) + expect(unloads.filter((options) => options === peerLoad)).toHaveLength( + Number(reentry === `release-peer`), + ) + expect(errors).toEqual( + (outcome === `throw` || outcome === `reject`) && + !targetWasReleased && + reentry !== `abort-self` + ? [failure] + : [], + ) + expect(statuses).toEqual( + reentry === `truncate` || + ((outcome === `resolve` || outcome === `reject`) && + !targetWasReleased) + ? [`loadingSubset`, `ready`] + : [], + ) + if (reentry === `truncate`) { + // A synchronous throw never acquired an owner to replay. Returned work + // retains logical demand, even when its first transport later rejects. + expect(loads).toHaveLength(outcome === `throw` ? 1 : 2) + if (outcome !== `throw`) { + expect(loads[1]).not.toBe(targetLoad) + expect(loads[1]?.where).toBe(targetWhere) + expect(loads[1]?.signal?.aborted).toBe(false) + } + } + + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + it.each(failureScenarios)( + `keeps a $outcome failure primary during $reentry error delivery`, + async ({ outcome, reentry }) => { + const targetWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`target`), + ]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`peer`)]) + const failure = new Error(`target load failed`) + const pending = createDeferred() + const loads: Array = [] + const attempts: Array<{ + session: number + options: LoadSubsetOptions + result: `peer-return` | `throw` | `pending` | `replay-return` + }> = [] + const unloads: Array = [] + const sourceCleanupSessions: Array = [] + const errors: Array = [] + const statuses: Array = [] + const controller = new AbortController() + let truncateCount = 0 + let truncate = () => {} + let targetLoadCount = 0 + + const collection = createOnDemandCollection<{ id: string }>({ + id: `demand-failure-${outcome}-${reentry}`, + sync: { + sync: (operations) => { + truncate = () => { + truncateCount++ + operations.begin() + operations.truncate() + operations.commit() + } + const { markReady } = operations + markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (options.where === peerWhere) { + attempts.push({ + session: 0, + options, + result: `peer-return`, + }) + return true + } + targetLoadCount++ + if (targetLoadCount > 1) { + attempts.push({ + session: 0, + options, + result: `replay-return`, + }) + return true + } + if (outcome === `throw`) { + attempts.push({ session: 0, options, result: `throw` }) + throw failure + } + attempts.push({ session: 0, options, result: `pending` }) + return pending.promise + }, + unloadSubset: (options) => unloads.push(options), + cleanup: () => sourceCleanupSessions.push(0), + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + subscription.on(`loadSubset:error`, ({ error }) => { + errors.push(error) + if (reentry === `abort-self`) { + controller.abort() + } else if (reentry === `truncate`) { + truncate() + } else if (reentry === `release-self`) { + subscription.releaseSnapshot(targetWhere) + } else if (reentry === `release-peer`) { + subscription.releaseSnapshot(peerWhere) + } else if (reentry === `unsubscribe`) { + subscription.unsubscribe() + } else if (reentry === `cleanup`) { + void collection.cleanup() + } + }) + + subscription.requestSnapshot({ where: peerWhere }) + let thrown: unknown + try { + subscription.requestSnapshot({ + where: targetWhere, + signal: controller.signal, + }) + } catch (error) { + thrown = error + } + if (outcome === `reject`) { + pending.reject(failure) + } + await flushPromises() + + const targetLoad = loads.find(({ where }) => where === targetWhere)! + const peerLoad = loads.find(({ where }) => where === peerWhere)! + const targetAttempts = attempts.filter( + ({ options }) => options.where === targetWhere, + ) + const peerAttempts = attempts.filter( + ({ options }) => options.where === peerWhere, + ) + const tearsDownTarget = + reentry === `release-self` || reentry === `unsubscribe` + + expect.soft(thrown).toBe(outcome === `throw` ? failure : undefined) + expect.soft(errors).toEqual([failure]) + expect.soft(subscription.lastError).toBe(failure) + expect.soft(targetAttempts[0]?.options).toBe(targetLoad) + expect.soft(targetAttempts[0]?.session).toBe(0) + expect + .soft(targetAttempts[0]?.result) + .toBe(outcome === `throw` ? `throw` : `pending`) + expect.soft(controller.signal.aborted).toBe(reentry === `abort-self`) + expect.soft(truncateCount).toBe(Number(reentry === `truncate`)) + expect + .soft(unloads.filter((options) => options === targetLoad)) + .toHaveLength( + Number( + outcome === `reject` && (tearsDownTarget || reentry === `truncate`), + ), + ) + expect + .soft(unloads.filter((options) => options === peerLoad)) + .toHaveLength( + Number( + reentry === `release-peer` || + reentry === `unsubscribe` || + reentry === `truncate`, + ), + ) + expect + .soft(statuses) + .toEqual( + reentry === `truncate` + ? [`loadingSubset`, `ready`] + : outcome === `reject` + ? reentry === `unsubscribe` + ? [`loadingSubset`] + : [`loadingSubset`, `ready`] + : [], + ) + if (reentry === `abort-self`) { + expect.soft(targetLoad.signal?.aborted).toBe(true) + expect.soft(targetAttempts).toHaveLength(1) + expect.soft(peerAttempts).toHaveLength(1) + } + const replacement = targetAttempts[1]?.options + const peerReplacement = peerAttempts[1]?.options + if (reentry === `truncate`) { + expect.soft(targetLoad.signal?.aborted).toBe(true) + // Error delivery may request replay, but it cannot turn a failed + // synchronous start into an owned acquisition. Only the peer survives. + expect.soft(targetAttempts).toHaveLength(outcome === `reject` ? 2 : 1) + if (outcome === `reject`) { + expect.soft(replacement).not.toBe(targetLoad) + expect.soft(replacement?.where).toBe(targetWhere) + expect.soft(targetAttempts[1]?.session).toBe(0) + expect.soft(targetAttempts[1]?.result).toBe(`replay-return`) + } else { + expect.soft(replacement).toBeUndefined() + } + expect.soft(peerLoad.signal?.aborted).toBe(true) + expect.soft(peerAttempts).toHaveLength(2) + expect.soft(peerReplacement).not.toBe(peerLoad) + expect.soft(peerReplacement?.where).toBe(peerWhere) + expect.soft(peerAttempts[1]?.session).toBe(0) + expect.soft(peerAttempts[1]?.result).toBe(`peer-return`) + expect.soft(unloads).toHaveLength(outcome === `reject` ? 2 : 1) + } + if (reentry === `cleanup`) { + expect.soft(collection.status).toBe(`cleaned-up`) + expect.soft(peerLoad.signal?.aborted).toBe(true) + expect.soft(targetLoad.signal?.aborted).toBe(true) + expect.soft(subscription.status).toBe(`ready`) + } + + subscription.unsubscribe() + const replays = reentry === `truncate` + expect + .soft(targetAttempts) + .toHaveLength(replays && outcome === `reject` ? 2 : 1) + expect.soft(peerAttempts).toHaveLength(replays ? 2 : 1) + expect + .soft(unloads.filter((options) => options === targetLoad)) + .toHaveLength(Number(outcome === `reject` && reentry !== `cleanup`)) + expect + .soft(unloads.filter((options) => options === peerLoad)) + .toHaveLength(Number(reentry !== `cleanup`)) + if (replacement) { + expect + .soft(unloads.filter((options) => options === replacement)) + .toHaveLength(Number(replays)) + expect.soft(replacement.signal?.aborted).toBe(true) + } + if (peerReplacement) { + expect + .soft(unloads.filter((options) => options === peerReplacement)) + .toHaveLength(Number(replays)) + expect.soft(peerReplacement.signal?.aborted).toBe(true) + } + const expectedUnloads = + reentry === `cleanup` + ? 0 + : (outcome === `reject` ? 2 : 1) * (replays ? 2 : 1) + expect.soft(unloads).toHaveLength(expectedUnloads) + expect.soft(peerLoad.signal?.aborted).toBe(true) + expect.soft(targetLoad.signal?.aborted).toBe(true) + const terminalAttempts = [...attempts] + const terminalUnloads = [...unloads] + const terminalStatuses = [...statuses] + await collection.cleanup() + expect.soft(sourceCleanupSessions).toEqual([0]) + expect.soft(collection.status).toBe(`cleaned-up`) + expect.soft(errors).toEqual([failure]) + expect.soft(subscription.lastError).toBe(failure) + expect.soft(attempts).toEqual(terminalAttempts) + expect.soft(unloads).toEqual(terminalUnloads) + expect.soft(statuses).toEqual(terminalStatuses) + observedFailureDeliverySuffixes.add(`${outcome}:${reentry}`) + }, + ) + + it.each(releaseScenarios)( + `retires logical ownership once for unload $outcome × $reentry`, + async ({ outcome, reentry }) => { + const targetWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`target`), + ]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`peer`)]) + const releaseFailure = new Error(`target release failed`) + const loads: Array = [] + const unloads: Array = [] + const errors: Array = [] + let allowRelease = outcome === `return` + let runReentry = () => {} + + const collection = createOnDemandCollection<{ id: string }>({ + id: `demand-release-${outcome}-${reentry}`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[1]) { + runReentry() + if (!allowRelease) throw releaseFailure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.requestSnapshot({ where: peerWhere }) + subscription.requestSnapshot({ where: targetWhere }) + const peerLoad = loads[0]! + const oldTargetLoad = loads[1]! + runReentry = () => { + runReentry = () => {} + if (reentry === `reacquire-self`) { + subscription.requestSnapshot({ where: targetWhere }) + } else if (reentry === `release-peer`) { + subscription.releaseSnapshot(peerWhere) + } else if (reentry === `unsubscribe`) { + subscription.unsubscribe() + } + } + + let thrown: unknown + try { + subscription.releaseSnapshot(targetWhere) + } catch (error) { + thrown = error + } + observePhysicalInteraction(`active:release`, `retire`) + if (reentry === `unsubscribe`) { + observePhysicalInteraction(`active:unsubscribe`, `retire`) + } + + expect(thrown).toBe(outcome === `throw` ? releaseFailure : undefined) + expect(oldTargetLoad.signal?.aborted).toBe(true) + expect( + unloads.filter((options) => options === oldTargetLoad), + ).toHaveLength(1) + expect(unloads.filter((options) => options === peerLoad)).toHaveLength( + Number(reentry === `release-peer` || reentry === `unsubscribe`), + ) + expect(errors).toEqual( + outcome === `throw` && reentry !== `unsubscribe` + ? [releaseFailure] + : [], + ) + expect(subscription.lastError).toBe( + outcome === `throw` ? releaseFailure : undefined, + ) + + allowRelease = true + subscription.unsubscribe() + expect( + unloads.filter((options) => options === oldTargetLoad), + ).toHaveLength(1) + const replacement = loads[2] + expect( + replacement === undefined + ? [] + : unloads.filter((options) => options === replacement), + ).toHaveLength(Number(reentry === `reacquire-self`)) + expect(unloads.filter((options) => options === peerLoad)).toHaveLength(1) + if (outcome === `throw` && reentry === `unsubscribe`) { + observePhysicalInteraction( + `failed-release:unsubscribe`, + `no-repeat-on-unsubscribe`, + ) + } + await collection.cleanup() + }, + ) + + it.each([`resolve`, `reject`] as const)( + `retires a pending replay on cleanup before an obsolete %s`, + async (outcome) => { + type Row = { id: string; version: number } + const replay = createDeferred() + const replayFailure = new Error(`obsolete replay failed`) + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let syncSession = 0 + let loadCount = 0 + const visible = new Map() + const errors: Array = [] + const statuses: Array = [] + + const collection = createOnDemandCollection({ + id: `cleanup-pending-replay-${outcome}`, + sync: { + sync: (operations) => { + syncSession++ + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + if (syncSession > 1) { + begin() + write({ type: `insert`, value: { id: `row`, version: 3 } }) + commit() + } + operations.markReady() + return { + loadSubset: () => { + loadCount++ + begin() + write({ + type: `insert`, + value: { id: `row`, version: loadCount }, + }) + commit() + return loadCount === 1 || syncSession > 1 + ? true + : replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + + subscription.requestSnapshot() + expect([...visible.values()]).toEqual([{ id: `row`, version: 1 }]) + begin() + truncate() + commit() + await flushPromises() + expect(subscription.status).toBe(`loadingSubset`) + + await collection.cleanup() + collection.startSyncImmediate() + expect(syncSession).toBe(2) + expect([...visible.values()]).toEqual([{ id: `row`, version: 3 }]) + expect(subscription.status).toBe(`loadingSubset`) + await flushPromises() + expect(subscription.status).toBe(`ready`) + + if (outcome === `resolve`) replay.resolve() + else replay.reject(replayFailure) + await flushPromises() + + expect([...visible.values()]).toEqual([{ id: `row`, version: 3 }]) + expect(errors).toEqual([]) + expect(subscription.lastError).toBeUndefined() + expect(statuses.at(-1)).toBe(`ready`) + + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + it(`reacquires surviving on-demand demand after collection restart`, async () => { + type Row = { id: string; version: number } + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let syncSession = 0 + let loadCount = 0 + const loads: Array = [] + const unloads: Array = [] + const visible = new Map() + const collection = createOnDemandCollection({ + id: `restart-surviving-demand`, + sync: { + sync: (operations) => { + syncSession++ + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + loadCount++ + begin() + write({ + type: `insert`, + value: { id: `row`, version: loadCount }, + }) + commit() + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + }, + { includeInitialState: false }, + ) + subscription.requestSnapshot() + expect([...visible.values()]).toEqual([{ id: `row`, version: 1 }]) + + await collection.cleanup() + collection.startSyncImmediate() + expect(subscription.status).toBe(`loadingSubset`) + expect(loads).toHaveLength(1) + await flushPromises() + + expect(syncSession).toBe(2) + expect(loads).toHaveLength(2) + expect(unloads).toEqual([]) + expect([...visible.values()]).toEqual([{ id: `row`, version: 2 }]) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + expect(unloads).toEqual([loads[1]]) + await collection.cleanup() + }) + + it(`reacquires demand requested while the collection is cleaned up`, async () => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + let syncSession = 0 + const loads: Array<{ session: number; options: LoadSubsetOptions }> = [] + const unloads: Array<{ session: number; options: LoadSubsetOptions }> = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `request-while-cleaned-up`, + sync: { + sync: ({ markReady }) => { + const session = syncSession++ + markReady() + return { + loadSubset: (options) => { + loads.push({ session, options }) + return true + }, + unloadSubset: (options) => unloads.push({ session, options }), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where: oldWhere }) + + await collection.cleanup() + subscription.requestSnapshot({ where: newWhere }) + collection.startSyncImmediate() + await flushPromises() + + expect(loads.map(({ session }) => session)).toEqual([0, 1, 1]) + expect(loads.slice(1).map(({ options }) => options.where)).toEqual([ + oldWhere, + newWhere, + ]) + + subscription.unsubscribe() + expect(unloads.map(({ session }) => session)).toEqual([1, 1]) + expect(unloads.map(({ options }) => options.where)).toEqual([ + oldWhere, + newWhere, + ]) + await collection.cleanup() + }) + + it(`does not report a detached demand as physically settled`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const observed: Array = [] + let loads = 0 + const collection = createOnDemandCollection<{ id: string }>({ + id: `detached-demand-settlement`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + await collection.cleanup() + subscription.requestSnapshot({ + where, + onLoadSubsetResult: (result) => observed.push(result), + }) + + expect(loads).toBe(0) + expect(observed).toEqual([expect.any(Promise)]) + + collection.startSyncImmediate() + await flushPromises() + expect(loads).toBe(1) + expect(observed).toEqual([expect.any(Promise)]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + acquisitionCase( + [`starting:request`], + `includes demand created by the synchronous restart status callback`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const demandForWhere = new Map([ + [oldWhere, `old`], + [newWhere, `new`], + ]) + const loads: Array<{ session: number; demand: `old` | `new` }> = [] + const unloads: Array<{ session: number; demand: `old` | `new` }> = [] + let session = -1 + let requestOnRestart = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `restart-status-reentry`, + sync: { + sync: ({ markReady }) => { + session++ + const adapterSession = session + markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown restart demand`) + loads.push({ session: adapterSession, demand }) + return true + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown restart demand`) + unloads.push({ session: adapterSession, demand }) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`status:change`, ({ status }) => { + if (!requestOnRestart || status !== `loadingSubset`) return + requestOnRestart = false + subscription.requestSnapshot({ where: newWhere }) + reach(`starting:request`) + }) + subscription.requestSnapshot({ where: oldWhere }) + + await collection.cleanup() + requestOnRestart = true + collection.startSyncImmediate() + await flushPromises() + + expect(loads).toEqual([ + { session: 0, demand: `old` }, + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + expect(unloads).toEqual([ + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + await collection.cleanup() + }, + ) + + acquisitionCase( + [`starting:markReady`], + `does not settle demand reentered before the restart loader is installed`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const demandForWhere = new Map([ + [oldWhere, `old`], + [newWhere, `new`], + ]) + const loads: Array<{ session: number; demand: `old` | `new` }> = [] + const unloads: Array<{ session: number; demand: `old` | `new` }> = [] + const observed: Array = [] + let session = -1 + let requestOnReady = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `restart-ready-reentry`, + sync: { + sync: ({ markReady }) => { + session++ + const adapterSession = session + markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown ready demand`) + loads.push({ session: adapterSession, demand }) + return true + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown ready unload`) + unloads.push({ session: adapterSession, demand }) + }, + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + const removeReadyListener = collection.on(`status:ready`, () => { + if (!requestOnReady) return + requestOnReady = false + subscription.requestSnapshot({ + where: newWhere, + onLoadSubsetResult: (result) => observed.push(result), + }) + reach(`starting:markReady`) + }) + subscription.requestSnapshot({ where: oldWhere }) + + await collection.cleanup() + requestOnReady = true + collection.startSyncImmediate() + await flushPromises() + + expect(loads).toEqual([ + { session: 0, demand: `old` }, + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + expect(observed).toEqual([expect.any(Promise)]) + expect(subscription.status).toBe(`ready`) + + removeReadyListener() + subscription.unsubscribe() + expect(unloads).toEqual([ + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + await collection.cleanup() + }, + ) + + acquisitionCase( + [`unavailable:request`], + `does not settle demand reentered before a failed restart installs a loader`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const syncFailure = new Error(`replacement sync failed`) + const demandForWhere = new Map([ + [oldWhere, `old`], + [newWhere, `new`], + ]) + const loads: Array<{ session: number; demand: `old` | `new` }> = [] + const unloads: Array<{ session: number; demand: `old` | `new` }> = [] + const observed: Array = [] + let session = -1 + let requestOnError = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `restart-error-reentry`, + sync: { + sync: ({ markReady }) => { + session++ + const adapterSession = session + if (session === 1) throw syncFailure + markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown error demand`) + loads.push({ session: adapterSession, demand }) + return true + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown error unload`) + unloads.push({ session: adapterSession, demand }) + }, + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + const removeErrorListener = collection.on(`status:error`, () => { + if (!requestOnError) return + requestOnError = false + subscription.requestSnapshot({ + where: newWhere, + onLoadSubsetResult: (result) => observed.push(result), + }) + reach(`unavailable:request`) + }) + subscription.requestSnapshot({ where: oldWhere }) + + await collection.cleanup() + requestOnError = true + expect(() => collection.startSyncImmediate()).toThrow(syncFailure) + expect(observed).toEqual([expect.any(Promise)]) + expect(collection.status).toBe(`error`) + expect(loads).toEqual([{ session: 0, demand: `old` }]) + + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + expect(loads).toEqual([ + { session: 0, demand: `old` }, + { session: 2, demand: `old` }, + { session: 2, demand: `new` }, + ]) + expect(subscription.status).toBe(`ready`) + + removeErrorListener() + subscription.unsubscribe() + expect(unloads).toEqual([ + { session: 2, demand: `old` }, + { session: 2, demand: `new` }, + ]) + await collection.cleanup() + }, + ) + + acquisitionCase( + [`retiring:request`], + `does not acquire through a retiring adapter cleanup callback`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const demandForWhere = new Map([ + [oldWhere, `old`], + [newWhere, `new`], + ]) + const loads: Array<{ session: number; demand: `old` | `new` }> = [] + const unloads: Array<{ session: number; demand: `old` | `new` }> = [] + const observed: Array = [] + let session = -1 + let requestDuringCleanup = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `adapter-cleanup-reentry`, + sync: { + sync: ({ markReady }) => { + session++ + const adapterSession = session + markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown cleanup demand`) + loads.push({ session: adapterSession, demand }) + return true + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown cleanup unload`) + unloads.push({ session: adapterSession, demand }) + }, + cleanup: () => { + if (!requestDuringCleanup) return + requestDuringCleanup = false + subscription.requestSnapshot({ + where: newWhere, + onLoadSubsetResult: (result) => observed.push(result), + }) + reach(`retiring:request`) + observeSourceSessionBoundary(`cleanup-callback-reentry`) + }, + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + subscription.requestSnapshot({ where: oldWhere }) + + requestDuringCleanup = true + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + + expect(loads).toEqual([ + { session: 0, demand: `old` }, + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + expect(observed).toEqual([expect.any(Promise)]) + + subscription.unsubscribe() + expect(unloads).toEqual([ + { session: 1, demand: `old` }, + { session: 1, demand: `new` }, + ]) + await collection.cleanup() + }, + ) + + acquisitionCase( + [`eager:request`], + `does not release a physical subset acquisition in eager mode`, + async (reach) => { + let loads = 0 + let unloads = 0 + const collection = createCollection<{ id: string }>({ + id: `eager-subset-ownership`, + getKey: ({ id }) => id, + syncMode: `eager`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + unloadSubset: () => { + unloads++ + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + subscription.requestSnapshot() + reach(`eager:request`) + subscription.unsubscribe() + observePhysicalInteraction(`none:unsubscribe`, `no-acquisition`) + + expect(loads).toBe(0) + expect(unloads).toBe(0) + await collection.cleanup() + }, + ) + + it(`directly releases eager demand without a physical acquisition`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + let loads = 0 + let unloads = 0 + const collection = createCollection<{ id: string }>({ + id: `eager-direct-release`, + getKey: ({ id }) => id, + syncMode: `eager`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + unloadSubset: () => { + unloads++ + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + subscription.requestSnapshot({ where }) + subscription.releaseSnapshot(where) + observePhysicalInteraction(`none:release`, `no-acquisition`) + + expect(loads).toBe(0) + expect(unloads).toBe(0) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`truncates eager demand without creating or releasing a physical acquisition`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const loads: Array = [] + const unloads: Array = [] + const collection = createCollection<{ id: string }>({ + id: `eager-truncate-without-acquisition`, + getKey: ({ id }) => id, + syncMode: `eager`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + + begin() + truncate() + commit() + await flushPromises() + observePhysicalInteraction(`none:truncate`, `no-acquisition`) + + expect.soft(loads).toEqual([]) + expect.soft(unloads).toEqual([]) + + subscription.unsubscribe() + expect(unloads).toEqual([]) + await collection.cleanup() + }) + + acquisitionCase( + [`on-demand:request`], + `does not release a subset request aborted before adapter acquisition`, + async (reach) => { + const controller = new AbortController() + controller.abort() + let loads = 0 + let unloads = 0 + const errors: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `pre-aborted-subset-ownership`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + unloadSubset: () => { + unloads++ + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + + subscription.requestSnapshot({ signal: controller.signal }) + await flushPromises() + reach(`on-demand:request`) + subscription.unsubscribe() + observePhysicalInteraction(`none:unsubscribe`, `no-acquisition`) + + expect(loads).toBe(0) + expect(unloads).toBe(0) + expect(errors).toEqual([]) + await collection.cleanup() + }, + ) + + it(`directly releases pre-aborted demand without a physical acquisition`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const controller = new AbortController() + controller.abort() + let loads = 0 + let unloads = 0 + const errors: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `pre-aborted-direct-release`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + unloadSubset: () => { + unloads++ + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + + subscription.requestSnapshot({ + where, + signal: controller.signal, + }) + await flushPromises() + subscription.releaseSnapshot(where) + observePhysicalInteraction(`none:release`, `no-acquisition`) + + expect(loads).toBe(0) + expect(unloads).toBe(0) + expect(errors).toEqual([]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it.each([false, true])( + `ignores a pre-aborted snapshot without changing an existing demand: %s`, + async (existingDemand) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const loads: Array = [] + const unloads: Array = [] + let publications = 0 + let results = 0 + const collection = createOnDemandCollection<{ id: string }>({ + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row` } }) + commit() + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => publications++, { + includeInitialState: false, + }) + try { + if (existingDemand) subscription.requestSnapshot({ where }) + const previousPublications = publications + const previousLoads = [...loads] + const controller = new AbortController() + controller.abort() + + expect( + subscription.requestSnapshot({ + where, + signal: controller.signal, + onLoadSubsetResult: () => results++, + }), + ).toBe(false) + await flushPromises() + expect(publications).toBe(previousPublications) + expect(results).toBe(0) + expect(loads).toEqual(previousLoads) + expect(unloads).toEqual([]) + if (existingDemand) expect(loads[0]!.signal?.aborted).toBe(false) + + subscription.releaseSnapshot(where) + expect(unloads).toEqual(previousLoads) + subscription.unsubscribe() + expect(unloads).toEqual(previousLoads) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`aborts detached demand without creating a physical acquisition`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const controller = new AbortController() + const loads: Array = [] + const unloads: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `detached-abort-without-acquisition`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + await collection.cleanup() + subscription.requestSnapshot({ where, signal: controller.signal }) + controller.abort() + await flushPromises() + observePhysicalInteraction(`none:abort`, `no-acquisition`) + + expect(loads).toEqual([]) + expect(unloads).toEqual([]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`keeps an aborted active acquisition until its owner retires`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const controller = new AbortController() + const pending = createDeferred() + const loads: Array = [] + const unloads: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `active-abort-before-release`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return pending.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + subscription.requestSnapshot({ where, signal: controller.signal }) + const acquisition = loads[0]! + controller.abort() + await flushPromises() + observePhysicalInteraction(`active:abort`, `abort-only`) + + expect(loads).toEqual([acquisition]) + expect(acquisition.signal?.aborted).toBe(true) + expect(unloads).toEqual([]) + + pending.resolve() + await flushPromises() + subscription.unsubscribe() + expect(unloads).toEqual([acquisition]) + await collection.cleanup() + }) + + acquisitionCase( + [`on-demand:request`, `on-demand:markReady`], + `acquires before and after ready once the on-demand loader is installed`, + async (reach) => { + const beforeReady = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`before`), + ]) + const afterReady = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`after`), + ]) + const loads: Array = [] + const unloads: Array = [] + let markReady!: () => void + const collection = createOnDemandCollection<{ id: string }>({ + id: `installed-loader-before-ready`, + startSync: false, + sync: { + sync: (operations) => { + markReady = operations.markReady + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + collection.startSyncImmediate() + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const removeReadyListener = collection.on(`status:ready`, () => { + subscription.requestSnapshot({ where: afterReady }) + reach(`on-demand:markReady`) + }) + + subscription.requestSnapshot({ where: beforeReady }) + reach(`on-demand:request`) + expect(collection.status).toBe(`loading`) + expect(loads.map(({ where }) => where)).toEqual([beforeReady]) + + markReady() + await flushPromises() + expect(loads.map(({ where }) => where)).toEqual([beforeReady, afterReady]) + + removeReadyListener() + subscription.unsubscribe() + expect(unloads).toEqual(loads) + await collection.cleanup() + }, + ) + + acquisitionCase( + [`deferred:request`, `deferred:resume`], + `owns deferred-start acquisition only when it reaches the adapter`, + async (reach) => { + for (const action of [`resume`, `release-before-resume`] as const) { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const loads: Array = [] + const unloads: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `deferred-start-${action}`, + startSync: false, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + reach(`deferred:request`) + expect(loads).toEqual([]) + + if (action === `release-before-resume`) { + subscription.releaseSnapshot(where) + } + collection._resumeSyncStart() + await flushPromises() + if (action === `resume`) reach(`deferred:resume`) + + expect(loads).toHaveLength(action === `resume` ? 1 : 0) + subscription.unsubscribe() + expect(unloads).toHaveLength(action === `resume` ? 1 : 0) + if (action === `resume`) expect(unloads).toEqual(loads) + await collection.cleanup() + } + }, + ) + + it.each([`cleanup`, `release`, `unsubscribe`, `resume`] as const)( + `settles queued demand according to whether it starts: %s`, + async (action) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const observed: Array> = [] + let loads = 0 + const collection = createOnDemandCollection<{ id: string }>({ + id: `deferred-start-cleanup-before-resume`, + startSync: false, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ + where, + onLoadSubsetResult: (result) => observed.push(result), + }) + + if (action === `cleanup`) await collection.cleanup() + else if (action === `release`) subscription.releaseSnapshot(where) + else if (action === `unsubscribe`) subscription.unsubscribe() + else collection._resumeSyncStart() + await flushPromises() + if (action === `cleanup`) { + observePhysicalInteraction(`none:cleanup`, `no-acquisition`) + } + + expect(loads).toBe(action === `resume` ? 1 : 0) + expect(observed).toHaveLength(1) + const deferredResult = observed[0] + expect(deferredResult).toBeInstanceOf(Promise) + if (!(deferredResult instanceof Promise)) { + throw new Error(`deferred acquisition did not return a promise`) + } + if (action === `resume`) + await expect(deferredResult).resolves.toBeUndefined() + else + await expect(deferredResult).rejects.toMatchObject({ + name: `AbortError`, + }) + + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + acquisitionCase( + [`starting:syncReturn`], + `does not settle ready-callback demand when on-demand sync returns no loader`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const loads: Array = [] + const observed: Array = [] + let session = 0 + let requestOnReady = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `ready-before-invalid-on-demand-return`, + sync: { + sync: ({ markReady }) => { + const ownSession = session++ + markReady() + if (ownSession === 1) return + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + const removeReadyListener = collection.on(`status:ready`, () => { + if (!requestOnReady) return + requestOnReady = false + subscription.requestSnapshot({ + where: newWhere, + onLoadSubsetResult: (result) => observed.push(result), + }) + }) + subscription.requestSnapshot({ where: oldWhere }) + + await collection.cleanup() + requestOnReady = true + expect(() => collection.startSyncImmediate()).toThrow( + /did not return a loadSubset handler/, + ) + reach(`starting:syncReturn`) + + expect(observed).toEqual([expect.any(Promise)]) + expect(collection.status).toBe(`error`) + expect(loads.map(({ where }) => where)).toEqual([oldWhere]) + + removeReadyListener() + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + it(`retires resources returned after ready-callback cleanup invalidates sync`, async () => { + const cleanupSessions: Array = [] + let session = 0 + let cleanOnReady = false + const collection = createOnDemandCollection<{ id: string }>({ + id: `obsolete-sync-return`, + sync: { + sync: ({ markReady }) => { + const ownSession = session++ + markReady() + return { + loadSubset: () => true, + unloadSubset: () => {}, + cleanup: () => cleanupSessions.push(ownSession), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const removeReadyListener = collection.on(`status:ready`, () => { + if (!cleanOnReady) return + cleanOnReady = false + void collection.cleanup() + }) + + await collection.cleanup() + cleanOnReady = true + collection.startSyncImmediate() + observeSourceSessionBoundary(`obsolete-resource-return`) + + expect(collection.status).toBe(`cleaned-up`) + expect(cleanupSessions).toEqual([0, 1]) + + removeReadyListener() + subscription.unsubscribe() + await collection.cleanup() + }) + + it.each( + ([false, true] as const).flatMap((restart) => + ( + [`loading`, `ready`, `adapter-throw`, `ready-effect-throw`] as const + ).map((entry) => ({ restart, entry })), + ), + )( + `retires startup at $entry with nested restart=$restart`, + async ({ entry, restart }) => { + const failure = new Error(`obsolete startup failed`) + const cleanups: Array = [] + const loads: Array = [] + const unloads: Array = [] + let session = -1 + let retire = false + const collection = createOnDemandCollection<{ id: string }>({ + sync: { + sync: ({ markReady }) => { + const ownSession = ++session + markReady() + if (ownSession === 1 && entry === `adapter-throw`) throw failure + return { + loadSubset: () => { + loads.push(ownSession) + return true + }, + unloadSubset: () => unloads.push(ownSession), + cleanup: () => cleanups.push(ownSession), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + let removeListener = () => {} + try { + await collection.cleanup() + const retireSession = () => { + if (!retire) return + retire = false + void collection.cleanup() + if (restart) collection.startSyncImmediate() + if (entry === `ready-effect-throw`) throw failure + } + removeListener = + entry === `ready-effect-throw` + ? collection.onFirstReady(retireSession) + : collection.on( + entry === `loading` ? `status:loading` : `status:ready`, + retireSession, + ) + retire = true + if (entry === `adapter-throw` || entry === `ready-effect-throw`) { + expect(() => collection.startSyncImmediate()).toThrow(failure) + } else { + collection.startSyncImmediate() + } + await flushPromises() + + expect(collection.status).toBe(restart ? `ready` : `cleaned-up`) + const returnsObsoleteCleanup = + entry === `ready` || entry === `ready-effect-throw` + expect(cleanups).toEqual(returnsObsoleteCleanup ? [0, 1] : [0]) + expect(session).toBe( + entry === `loading` ? (restart ? 1 : 0) : restart ? 2 : 1, + ) + + if (restart) { + subscription.requestSnapshot({ + where: new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]), + }) + expect(loads).toEqual([session]) + subscription.unsubscribe() + expect(unloads).toEqual([session]) + } else { + expect(loads).toEqual([]) + expect(unloads).toEqual([]) + } + } finally { + removeListener() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + acquisitionCase( + [`starting:markError`, `unavailable:markReady`], + `retains demand requested during initial error for same-session recovery`, + async (reach) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const loads: Array = [] + const observed: Array = [] + let syncSession = 0 + let recover!: () => void + const collection = createOnDemandCollection<{ id: string }>({ + id: `sync-entry-error-ready-recovery`, + startSync: false, + sync: { + sync: ({ markError, markReady }) => { + if (syncSession++ === 0) { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: () => {}, + } + } + recover = markReady + markError(new Error(`initial sync failed`)) + reach(`starting:markError`) + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + await collection.cleanup() + const removeErrorListener = collection.on(`status:error`, () => { + subscription.requestSnapshot({ + where, + onLoadSubsetResult: (result) => observed.push(result), + }) + }) + + collection.startSyncImmediate() + await flushPromises() + expect.soft(collection.status).toBe(`error`) + expect.soft(loads).toEqual([]) + expect.soft(observed).toEqual([expect.any(Promise)]) + + recover() + reach(`unavailable:markReady`) + await flushPromises() + + expect(collection.status).toBe(`ready`) + expect(loads.map(({ where: loadedWhere }) => loadedWhere)).toEqual([ + where, + ]) + expect(observed).toEqual([expect.any(Promise)]) + await expect(observed[0]).resolves.toBeUndefined() + + removeErrorListener() + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + it.each( + ([`error`, `cleaned-up`] as const).flatMap((unavailable) => + ( + [ + `return`, + `resolve`, + `reject`, + `throw`, + `release`, + `unsubscribe`, + `cleanup`, + `abort`, + ] as const + ).flatMap((outcome) => + (outcome === `release` || + outcome === `unsubscribe` || + outcome === `cleanup` || + outcome === `abort` + ? ([`before`, `during`] as const) + : ([`during`] as const) + ).flatMap((phase) => + // Ordered snapshots do not accept an external AbortSignal. + (outcome === `abort` + ? ([`snapshot`] as const) + : ([`snapshot`, `limited`] as const) + ).map((entry) => ({ unavailable, outcome, phase, entry })), + ), + ), + ), + )( + `observes unavailable demand synchronously: $entry / $unavailable / $outcome / $phase`, + async ({ unavailable, outcome, phase, entry }) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const transport = createDeferred() + const failure = new Error(`recovery acquisition failed`) + const signal = new AbortController() + const loads: Array = [] + const unloads: Array = [] + const rows = new Map() + let operations!: Parameters[`sync`]>[0] + const collection = createOnDemandCollection<{ id: string }>({ + sync: { + sync: (next) => { + operations = next + return { + loadSubset: (options) => { + loads.push(options) + if (outcome === `throw`) throw failure + operations.begin() + operations.write({ type: `insert`, value: { id: `row` } }) + operations.commit() + return outcome === `return` ? true : transport.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) rows.delete(change.key) + else rows.set(change.key, change.value) + } + }, + { includeInitialState: false }, + ) + let result: true | Promise | undefined + let release: (() => void) | undefined + const settlements: Array = [] + const visibleOnSuccess: Array> = [] + let callbacks = 0 + try { + if (entry === `limited`) { + subscription.setOrderByIndex( + collection.createIndex((row) => row.id, { indexType: BTreeIndex }), + ) + } + if (unavailable === `error`) + operations.markError(new Error(`initial error`)) + else await collection.cleanup() + const onLoadSubsetResult = ( + value: true | Promise, + _options: LoadSubsetOptions, + releaseDemand?: () => void, + ) => { + callbacks++ + result = value + release = releaseDemand + if (value instanceof Promise) + void value.then( + () => { + visibleOnSuccess.push([...rows.values()].map(({ id }) => id)) + settlements.push(`success`) + }, + (error: unknown) => settlements.push(error), + ) + } + if (entry === `snapshot`) { + subscription.requestSnapshot({ + where, + signal: signal.signal, + onLoadSubsetResult, + }) + } else { + subscription.requestLimitedSnapshot({ + orderBy: [ + { + expression: new PropRef([`id`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + onLoadSubsetResult, + }) + } + // Production callers copy the result as soon as requestSnapshot returns. + expect(callbacks).toBe(1) + expect(result).toBeInstanceOf(Promise) + await flushPromises() + expect(settlements).toEqual([]) + expect(loads).toEqual([]) + const recover = () => { + if (unavailable === `cleaned-up`) collection.startSyncImmediate() + operations.markReady() + } + if (phase === `during`) { + recover() + await flushPromises() + expect(loads).toHaveLength(1) + if (outcome !== `return` && outcome !== `throw`) { + expect(settlements).toEqual([]) + expect([...rows.values()]).toEqual([]) + } + } + if (outcome === `release`) release!() + else if (outcome === `unsubscribe`) subscription.unsubscribe() + else if (outcome === `cleanup`) await collection.cleanup() + else if (outcome === `abort`) signal.abort() + else if (outcome === `reject`) transport.reject(failure) + else if (outcome === `resolve`) transport.resolve() + await flushPromises() + if (outcome === `return` || outcome === `resolve`) { + expect(settlements).toEqual([`success`]) + expect(visibleOnSuccess).toEqual([[`row`]]) + expect([...rows.values()].map(({ id }) => id)).toEqual([`row`]) + } else if (outcome === `throw` || outcome === `reject`) { + expect(settlements).toHaveLength(1) + expect(settlements[0]).toBe(failure) + expect([...rows.values()]).toEqual([]) + } else { + expect(settlements).toEqual([ + expect.objectContaining({ name: `AbortError` }), + ]) + expect(unloads).toEqual( + phase === `during` && + (outcome === `release` || outcome === `unsubscribe`) + ? loads + : [], + ) + if (phase === `before` && outcome !== `cleanup`) { + recover() + await flushPromises() + expect(loads).toEqual([]) + } + if (outcome === `cleanup`) { + // Cleanup ends this wait, not the surviving subscription's demand. + collection.startSyncImmediate() + operations.markReady() + await flushPromises() + expect(loads).toHaveLength(phase === `before` ? 1 : 2) + } + // Non-cooperative late settlement cannot rewrite the observed outcome. + transport.resolve() + await flushPromises() + expect(settlements).toEqual([ + expect.objectContaining({ name: `AbortError` }), + ]) + } + expect(callbacks).toBe(1) + } finally { + transport.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`re-enables an installed loader after same-session initial recovery`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const loads: Array = [] + let markError!: (error: unknown) => void + let markReady!: () => void + const collection = createOnDemandCollection<{ id: string }>({ + id: `installed-loader-error-ready-recovery`, + startSync: false, + sync: { + sync: (operations) => { + markError = operations.markError + markReady = operations.markReady + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + collection.startSyncImmediate() + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + markError(new Error(`initial sync failed`)) + markReady() + subscription.requestSnapshot({ where }) + + expect(collection.status).toBe(`ready`) + expect(loads.map(({ where: loadedWhere }) => loadedWhere)).toEqual([where]) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`releases unavailable demand without creating a physical acquisition`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const loads: Array = [] + const unloads: Array = [] + let markError!: (error: unknown) => void + let markReady!: () => void + const collection = createOnDemandCollection<{ id: string }>({ + id: `release-unavailable-demand`, + startSync: false, + sync: { + sync: (operations) => { + markError = operations.markError + markReady = operations.markReady + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + collection.startSyncImmediate() + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + markError(new Error(`initial sync failed`)) + subscription.requestSnapshot({ where }) + subscription.releaseSnapshot(where) + markReady() + await flushPromises() + + expect(loads).toHaveLength(0) + expect(unloads).toHaveLength(0) + + subscription.unsubscribe() + await collection.cleanup() + }) + + acquisitionCase( + [`on-demand:markError`], + `defers demand while an installed loader is in initial error`, + async (reach) => { + const oldWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`old`)]) + const newWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`new`)]) + const loads: Array = [] + let markError!: (error: unknown) => void + let markReady!: () => void + const collection = createOnDemandCollection<{ id: string }>({ + id: `installed-loader-initial-error`, + startSync: false, + sync: { + sync: (operations) => { + markError = operations.markError + markReady = operations.markReady + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + collection.startSyncImmediate() + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where: oldWhere }) + const removeErrorListener = collection.on(`status:error`, () => { + subscription.requestSnapshot({ where: newWhere }) + reach(`on-demand:markError`) + }) + + markError(new Error(`initial sync failed`)) + + await flushPromises() + expect.soft(collection.status).toBe(`error`) + expect.soft(loads.map(({ where }) => where)).toEqual([oldWhere]) + + markReady() + await flushPromises() + expect(loads.map(({ where }) => where)).toEqual([oldWhere, newWhere]) + + removeErrorListener() + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + it.each( + ([`loading`, `ready`] as const).flatMap((phase) => + ([`cleanup`, `release`, `unsubscribe`] as const).map((action) => ({ + phase, + action, + })), + ), + )( + `cancels queued acquisition during $phase via $action`, + async ({ phase, action }) => { + const loads: Array = [] + const unloads: Array = [] + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + let cancelOnEntry = false + const observed: Array> = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `deferred-resume-cleanup`, + startSync: false, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const removeReadyListener = collection.on(`status:${phase}`, () => { + if (!cancelOnEntry) return + cancelOnEntry = false + if (action === `cleanup`) void collection.cleanup() + else if (action === `release`) subscription.releaseSnapshot(where) + else subscription.unsubscribe() + }) + subscription.requestSnapshot({ + where, + onLoadSubsetResult: (result) => observed.push(result), + }) + + try { + cancelOnEntry = true + collection._resumeSyncStart() + await flushPromises() + + expect(collection.status).toBe( + action === `cleanup` ? `cleaned-up` : `ready`, + ) + expect(loads).toHaveLength(0) + expect(unloads).toHaveLength(0) + expect(observed).toHaveLength(1) + await expect(observed[0]).rejects.toMatchObject({ name: `AbortError` }) + } finally { + removeReadyListener() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`keeps an eager subscription ready after collection restart`, async () => { + const collection = createCollection<{ id: string }>({ + id: `eager-subscription-restart`, + getKey: ({ id }) => id, + syncMode: `eager`, + sync: { sync: ({ markReady }) => markReady() }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot() + + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + + expect(collection.status).toBe(`ready`) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`retires restart loading when the replacement sync fails`, async () => { + const syncFailure = new Error(`replacement sync failed`) + let session = 0 + const collection = createOnDemandCollection<{ id: string }>({ + id: `failed-sync-restart`, + sync: { + sync: ({ markReady }) => { + if (session++ > 0) throw syncFailure + markReady() + return { loadSubset: () => true } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot() + + await collection.cleanup() + expect(() => collection.startSyncImmediate()).toThrow(syncFailure) + await flushPromises() + + expect(collection.status).toBe(`error`) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`retires failed physical release with its source session cleanup`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const releaseFailure = new Error(`release failed`) + let unloads = 0 + let sourceCleanups = 0 + const collection = createOnDemandCollection<{ id: string }>({ + id: `cleanup-failed-release`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloads++ + if (unloads === 1) throw releaseFailure + }, + cleanup: () => { + sourceCleanups++ + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + expect(() => subscription.releaseSnapshot(where)).toThrow(releaseFailure) + + await collection.cleanup() + expect(unloads).toBe(1) + expect(sourceCleanups).toBe(1) + observeSourceSessionBoundary(`active-cleanup`) + + await collection.cleanup() + expect(unloads).toBe(1) + expect(sourceCleanups).toBe(1) + observePhysicalInteraction(`failed-release:cleanup`, `no-repeat-on-cleanup`) + subscription.unsubscribe() + }) + + it.each( + ([`return`, `throw`] as const).flatMap((outcome) => + [false, true].map((releaseSelf) => ({ outcome, releaseSelf })), + ), + )( + `retires only the acquired lease for aborted replay with unload=$outcome, releaseSelf=$releaseSelf`, + async ({ outcome, releaseSelf }) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const controller = new AbortController() + const failure = new Error(`release failed`) + const loads: Array = [] + const unloads: Array = [] + const errors: Array = [] + let releaseOwner = () => {} + let operations!: Parameters[`sync`]>[0] + const collection = createCollection<{ id: string }, string>({ + id: `aborted-replay-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (nextOperations) => { + operations = nextOperations + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (unloads.length === 1) { + if (releaseSelf) releaseOwner() + if (outcome === `throw`) throw failure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + releaseOwner = () => subscription.releaseSnapshot(where) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + try { + subscription.requestSnapshot({ where, signal: controller.signal }) + controller.abort() + operations.begin() + operations.truncate() + const receipt = operations.commit() + if (receipt !== true) await receipt + await flushPromises() + + expect(loads).toHaveLength(1) + expect(unloads).toHaveLength(1) + expect(unloads[0]).toBe(loads[0]) + expect(loads[0]!.signal!.aborted).toBe(true) + expect(subscription.status).toBe(`ready`) + expect(errors).toHaveLength(outcome === `throw` ? 1 : 0) + if (outcome === `throw`) expect(errors[0]).toBe(failure) + + releaseOwner() + expect(unloads).toHaveLength(1) + subscription.unsubscribe() + expect(unloads).toHaveLength(1) + for (const options of unloads) expect(options).toBe(loads[0]) + expect(loads).toHaveLength(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each( + ([`adapter`, `error-listener`] as const).flatMap((reentry) => + [1, 2].map((failures) => ({ reentry, failures })), + ), + )( + `attempts release once across $reentry reentry with $failures configured failures`, + async ({ reentry, failures }) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const failure = new Error(`physical release failed`) + const loads: Array = [] + const unloads: Array = [] + const errors: Array = [] + const nestedFailures: Array = [] + let releaseOwner = () => {} + const collection = createOnDemandCollection<{ id: string }>({ + id: `release-reentry-${reentry}-${failures}`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (unloads.length === 1 && reentry === `adapter`) { + releaseOwner() + } + if (unloads.length <= failures) throw failure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + releaseOwner = () => { + try { + subscription.unsubscribe() + } catch (error) { + nestedFailures.push(error) + } + } + subscription.on(`loadSubset:error`, ({ error }) => { + errors.push(error) + if (errors.length === 1 && reentry === `error-listener`) releaseOwner() + }) + try { + subscription.requestSnapshot({ where }) + let releaseError: unknown + try { + subscription.releaseSnapshot(where) + } catch (error) { + releaseError = error + } + expect(releaseError).toBe(failure) + // Both callback paths see a retired acquisition, including after throw. + expect(unloads).toHaveLength(1) + expect(collection.subscriberCount).toBe(0) + if (reentry === `error-listener`) expect(errors[0]).toBe(failure) + expect(nestedFailures).toEqual([]) + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads).toHaveLength(1) + subscription.unsubscribe() + expect(unloads).toHaveLength(1) + expect(loads).toHaveLength(1) + for (const options of unloads) expect(options).toBe(loads[0]) + } finally { + await collection.cleanup() + subscription.unsubscribe() + } + }, + ) + + it(`keeps failed releases out of truncate replay`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const releaseFailure = new Error(`release failed`) + let unloads = 0 + let operations!: Parameters[`sync`]>[0] + const collection = createCollection<{ id: string }, string>({ + id: `truncate-failed-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (nextOperations) => { + operations = nextOperations + operations.markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloads++ + if (unloads === 1) throw releaseFailure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + expect(() => subscription.releaseSnapshot(where)).toThrow(releaseFailure) + + operations.begin() + operations.truncate() + const receipt = operations.commit() + if (receipt !== true) await receipt + expect(unloads).toBe(1) + + subscription.unsubscribe() + expect(unloads).toBe(1) + observePhysicalInteraction( + `failed-release:truncate`, + `no-repeat-on-truncate`, + ) + await collection.cleanup() + }) + + it(`does not repeat failed cleanup through a replacement adapter session`, async () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + let syncSession = 0 + const unloadSessions: Array = [] + const releaseFailure = new Error(`old session release failed`) + const collection = createOnDemandCollection<{ id: string }>({ + id: `cleanup-debt-session`, + sync: { + sync: ({ markReady }) => { + const session = syncSession++ + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloadSessions.push(session) + if (session === 0) throw releaseFailure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + expect(() => subscription.releaseSnapshot(where)).toThrow(releaseFailure) + + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + subscription.unsubscribe() + + expect(unloadSessions).toEqual([0]) + await collection.cleanup() + }) + + it.each(restartScenarios)( + `keeps restart ownership aligned for $outcome × $reentry`, + async ({ outcome, reentry }) => { + type DemandName = `target` | `peer` + const targetWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`target`), + ]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`peer`)]) + const demandForWhere = new Map([ + [targetWhere, `target`], + [peerWhere, `peer`], + ]) + const failure = new Error(`restart acquisition failed`) + const pending = createDeferred() + void pending.promise.catch(() => {}) + const loads: Array<{ session: number; demand: DemandName }> = [] + const unloads: Array<{ session: number; demand: DemandName }> = [] + const sourceCleanups: Array = [] + const errors: Array = [] + let session = -1 + let ranReentry = false + + const collection = createOnDemandCollection<{ id: string }>({ + id: `restart-${outcome}-${reentry}`, + sync: { + sync: ({ markReady }) => { + session++ + const adapterSession = session + markReady() + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown restart demand`) + loads.push({ session: adapterSession, demand }) + if (adapterSession === 0 || demand === `peer`) return true + if (!ranReentry) { + ranReentry = true + if (reentry === `release-self`) { + subscription.releaseSnapshot(targetWhere) + } else if (reentry === `release-peer`) { + subscription.releaseSnapshot(peerWhere) + } else if (reentry === `unsubscribe`) { + subscription.unsubscribe() + } else if (reentry === `cleanup`) { + void collection.cleanup() + } + } + if (outcome === `throw`) throw failure + if (outcome === `return`) return true + return pending.promise + }, + unloadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`unknown restart demand`) + unloads.push({ session: adapterSession, demand }) + }, + cleanup: () => sourceCleanups.push(adapterSession), + } + }, + }, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.requestSnapshot({ where: targetWhere }) + subscription.requestSnapshot({ where: peerWhere }) + + await collection.cleanup() + expect(sourceCleanups).toEqual([0]) + observePhysicalInteraction(`active:cleanup`, `retire`) + collection.startSyncImmediate() + observeSourceSessionBoundary(`restart-installed`) + await flushPromises() + if (outcome === `resolve`) pending.resolve() + if (outcome === `reject`) pending.reject(failure) + await flushPromises() + + const targetEstablished = outcome !== `throw` && reentry !== `cleanup` + const targetSurvives = reentry === `none` || reentry === `release-peer` + const peerStarts = + reentry !== `release-peer` && + reentry !== `unsubscribe` && + reentry !== `cleanup` + expect(loads).toEqual([ + { session: 0, demand: `target` }, + { session: 0, demand: `peer` }, + { session: 1, demand: `target` }, + ...(peerStarts ? [{ session: 1, demand: `peer` as const }] : []), + ]) + expect(errors).toEqual( + (outcome === `throw` || outcome === `reject`) && targetSurvives + ? [failure] + : [], + ) + + if (reentry !== `unsubscribe`) subscription.unsubscribe() + expect(unloads).toEqual([ + ...(targetEstablished && !targetSurvives + ? [{ session: 1, demand: `target` as const }] + : []), + ...(targetEstablished && targetSurvives + ? [{ session: 1, demand: `target` as const }] + : []), + ...(peerStarts ? [{ session: 1, demand: `peer` as const }] : []), + ]) + await collection.cleanup() + expect(sourceCleanups).toEqual([0, 1]) + }, + ) + + it.each( + ([`initial`, `replay`] as const).flatMap((origin) => + ([`resolve`, `reject`] as const).flatMap((oldOutcome) => + ([`old-first`, `current-first`] as const).map((order) => ({ + origin, + oldOutcome, + order, + })), + ), + ), + )( + `separates publication from readiness for pending $origin work, $oldOutcome, $order`, + async ({ origin, oldOutcome, order }) => { + type Row = { id: string; version: number } + const where = { + a: new Func(`eq`, [new PropRef([`id`]), new Value(`a`)]), + b: new Func(`eq`, [new PropRef([`id`]), new Value(`b`)]), + } + const loads: Array<{ + options: LoadSubsetOptions + deferred: ReturnType> + }> = [] + const unloads: Array = [] + const errors: Array = [] + const visible = new Map() + let emptyBatches = 0 + let replacementChanges = 0 + let operations!: Parameters[`sync`]>[0] + const collection = createCollection({ + id: `publication-readiness-boundary`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (nextOperations) => { + operations = nextOperations + operations.markReady() + return { + loadSubset: (options) => { + const deferred = createDeferred() + void deferred.promise.catch(() => {}) + loads.push({ options, deferred }) + return deferred.promise + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + if (changes.length === 0) emptyBatches++ + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + if (change.value.id === `b` && change.value.version === 2) + replacementChanges++ + } + } + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + const write = async (id: string, version: number) => { + operations.begin() + operations.write({ + type: collection.has(id) ? `update` : `insert`, + value: { id, version }, + }) + const receipt = operations.commit() + if (receipt !== true) await receipt + } + const truncate = async () => { + operations.begin() + operations.truncate() + const receipt = operations.commit() + if (receipt !== true) await receipt + await flushPromises() + } + const settle = async ( + attempt: (typeof loads)[number], + outcome: `resolve` | `reject`, + id = `b`, + ) => { + // The source cannot cancel transport promptly, but must suppress its + // canceled writes. Late settlement does not install an obsolete row. + if (outcome === `resolve`) { + if (!attempt.options.signal?.aborted) await write(id, 2) + attempt.deferred.resolve() + } else attempt.deferred.reject(new Error(`obsolete source failed`)) + await flushPromises() + } + try { + subscription.requestSnapshot({ where: where.b }) + await write(`b`, 0) + if (origin === `replay`) { + loads[0]!.deferred.resolve() + await flushPromises() + await truncate() + } + const old = loads.at(-1)! + await truncate() + const current = loads.at(-1)! + expect(old.options.signal?.aborted).toBe(true) + expect([...visible.values()]).toEqual([{ id: `b`, version: 0 }]) + if (order === `old-first`) { + await settle(old, oldOutcome) + expect(subscription.status).toBe(`loadingSubset`) + expect([...visible.values()]).toEqual([{ id: `b`, version: 0 }]) + } + await settle(current, `resolve`) + const privateReplay = origin === `replay` && order === `current-first` + expect([...visible.values()]).toEqual([ + { id: `b`, version: privateReplay ? 0 : 2 }, + ]) + expect(subscription.status).toBe( + order === `old-first` ? `ready` : `loadingSubset`, + ) + const emptyBeforeRequest = emptyBatches + subscription.requestSnapshot({ where: where.a }) + expect(emptyBatches - emptyBeforeRequest).toBe(privateReplay ? 0 : 1) + await settle(loads.at(-1)!, `resolve`, `a`) + expect([...visible.values()]).toEqual( + privateReplay + ? [{ id: `b`, version: 0 }] + : [ + { id: `b`, version: 2 }, + { id: `a`, version: 2 }, + ], + ) + if (order === `current-first`) await settle(old, oldOutcome) + expect([...visible.values()]).toEqual([ + { id: `b`, version: 2 }, + { id: `a`, version: 2 }, + ]) + expect(subscription.status).toBe(`ready`) + expect(errors).toEqual([]) + expect(replacementChanges).toBe(1) + subscription.unsubscribe() + expect(unloads).toHaveLength(loads.length) + for (const { options } of loads) + expect(unloads.filter((value) => value === options)).toHaveLength(1) + } finally { + for (const { deferred } of loads) deferred.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each(threeGenerationScenarios)( + `fences three generations for $obsoleteOutcome/$currentOutcome settled $settlementOrder`, + async ({ obsoleteOutcome, currentOutcome, settlementOrder }) => { + type Row = { id: string; version: number } + const obsolete = createDeferred() + const current = createDeferred() + void obsolete.promise.catch(() => {}) + void current.promise.catch(() => {}) + const obsoleteFailure = new Error(`obsolete generation failed`) + const currentFailure = new Error(`current generation failed`) + const visible = new Map() + const errors: Array = [] + const unloadSessions: Array = [] + let session = -1 + + const collection = createOnDemandCollection({ + id: `three-generation-${obsoleteOutcome}-${currentOutcome}-${settlementOrder}`, + sync: { + sync: (operations) => { + session++ + const ownSession = session + operations.markReady() + return { + loadSubset: (options) => { + if (ownSession === 0) { + operations.begin() + operations.write({ + type: `insert`, + value: { id: `row`, version: 1 }, + }) + operations.commit(options.signal) + return true + } + const gate = ownSession === 1 ? obsolete : current + const outcome = + ownSession === 1 ? obsoleteOutcome : currentOutcome + const failure = + ownSession === 1 ? obsoleteFailure : currentFailure + return gate.promise.then(() => { + if (outcome === `reject`) throw failure + operations.begin() + operations.write({ + type: `insert`, + value: { id: `row`, version: ownSession + 1 }, + }) + const receipt = operations.commit(options.signal) + if (receipt !== true) return receipt + return undefined + }) + }, + unloadSubset: () => unloadSessions.push(ownSession), + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.requestSnapshot() + expect([...visible.values()]).toEqual([{ id: `row`, version: 1 }]) + + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + await collection.cleanup() + collection.startSyncImmediate() + await flushPromises() + expect(subscription.status).toBe(`loadingSubset`) + + const settleObsolete = () => + obsoleteOutcome === `resolve` + ? obsolete.resolve() + : obsolete.reject(obsoleteFailure) + const settleCurrent = () => + currentOutcome === `resolve` + ? current.resolve() + : current.reject(currentFailure) + if (settlementOrder === `obsolete-first`) { + settleObsolete() + await flushPromises() + expect(subscription.status).toBe(`loadingSubset`) + settleCurrent() + } else { + settleCurrent() + await flushPromises() + settleObsolete() + } + await flushPromises() + + expect([...visible.values()]).toEqual([ + currentOutcome === `resolve` + ? { id: `row`, version: 3 } + : { id: `row`, version: 1 }, + ]) + expect(errors).toEqual( + currentOutcome === `reject` ? [currentFailure] : [], + ) + expect(subscription.lastError).toBe( + currentOutcome === `reject` ? currentFailure : undefined, + ) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + expect(unloadSessions).toEqual([2]) + await collection.cleanup() + }, + ) + + it(`treats an externally aborted replay as failed without publishing partial rows`, async () => { + type Row = { id: string; value: string } + const abort = new AbortController() + const replay = createDeferred() + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const visible = new Map() + const collection = createOnDemandCollection({ + id: `externally-aborted-replay`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount++ + begin() + write({ + type: `insert`, + value: { + id: `row`, + value: loadCount === 1 ? `old` : `partial`, + }, + }) + commit() + return loadCount === 1 ? true : replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + value: change.value.value, + }) + } + } + }, + { includeInitialState: false }, + ) + + subscription.requestSnapshot({ signal: abort.signal }) + expect([...visible.values()]).toEqual([{ id: `row`, value: `old` }]) + begin() + truncate() + commit() + await flushPromises() + abort.abort() + replay.reject(new DOMException(`aborted`, `AbortError`)) + await flushPromises() + + expect([...visible.values()]).toEqual([{ id: `row`, value: `old` }]) + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`enters loading status when a truncate queues replay work`, async () => { + const replay = createDeferred() + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const loads: Array = [] + const unloads: Array = [] + const collection = createOnDemandCollection<{ id: string }>({ + id: `queued-replay-status`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return loads.length === 1 ? true : replay.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where }) + const original = loads[0]! + + begin() + truncate() + commit() + observePhysicalInteraction(`active:truncate`, `retire`) + + expect(loads).toHaveLength(1) + expect(subscription.status).toBe(`loadingSubset`) + + await flushPromises() + const replacement = loads[1]! + expect(loads).toHaveLength(2) + expect(original.signal?.aborted).toBe(true) + expect(unloads).toEqual([original]) + expect(replacement).not.toBe(original) + expect(replacement.where).toBe(where) + expect(replacement.signal?.aborted).toBe(false) + replay.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + + subscription.unsubscribe() + expect(unloads).toEqual([original, replacement]) + await collection.cleanup() + }) + + it.each(startOutcomes)( + `retires replay setup when its adapter cleans up before %s`, + async (outcome) => { + type Row = { id: string; version: number } + const pending = createDeferred() + void pending.promise.catch(() => {}) + const failure = new Error(`obsolete replay failed`) + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const visible = new Map() + const errors: Array = [] + const statuses: Array = [] + + const collection = createOnDemandCollection({ + id: `reentrant-cleanup-${outcome}`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount++ + begin() + write({ + type: `insert`, + value: { id: `row`, version: loadCount }, + }) + commit() + if (loadCount === 1) return true + void collection.cleanup() + if (outcome === `throw`) throw failure + if (outcome === `return`) return true + return pending.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + visible.set(change.key, { + id: change.value.id, + version: change.value.version, + }) + } + } + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => errors.push(error)) + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + subscription.requestSnapshot() + + begin() + truncate() + commit() + await flushPromises() + if (outcome === `resolve`) pending.resolve() + if (outcome === `reject`) pending.reject(failure) + await flushPromises() + + expect(collection.status).toBe(`cleaned-up`) + expect([...visible.values()]).toEqual([{ id: `row`, version: 1 }]) + expect(errors).toEqual([]) + expect(subscription.lastError).toBeUndefined() + expect(subscription.status).toBe(`ready`) + expect(statuses.at(-1)).not.toBe(`loadingSubset`) + + subscription.unsubscribe() + await collection.cleanup() + }, + ) + + const { multiplier, ...replay } = readOracleRunConfig() + + fcTest.prop([asyncRestartScenarioArbitrary], { + numRuns: 30 * multiplier, + seed: 1_657_002, + })( + `fences async demand settlements across restart generations for a fixed seed`, + async (scenario) => { + await runAsyncRestartScenario(scenario) + }, + 120_000, + ) + + fcTest.prop( + [asyncRestartScenarioArbitrary], + oracleRandomParameters( + 30 * multiplier, + replay, + `subscription-lifecycle.async-restart`, + ), + )( + `fences async demand settlements across restart generations for a random or replayed seed`, + async (scenario) => { + await runAsyncRestartScenario(scenario) + }, + 120_000, + ) +}) diff --git a/packages/db/tests/collection-subscription-lifecycle-publication.property.test.ts b/packages/db/tests/collection-subscription-lifecycle-publication.property.test.ts new file mode 100644 index 0000000000..b2755b2e5a --- /dev/null +++ b/packages/db/tests/collection-subscription-lifecycle-publication.property.test.ts @@ -0,0 +1,1878 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { + createLifecycleModel, + greenLifecycleHistories, + greenLifecycleHistoryArbitrary, + reduceLifecycle, +} from './collection-subscription-lifecycle-grammar.js' +import { oracleRandomParameters, readOracleRunConfig } from './oracle-config.js' +import { flushPromises } from './utils.js' +import type { SyncConfig } from '../src/types.js' +import type { + DemandName, + LifecycleAttempt, + LifecycleCommand, + LifecycleEffect, + LifecycleModel, +} from './collection-subscription-lifecycle-grammar.js' + +type RowKey = DemandName | `c` | `d` +type Row = { id: RowKey; value: number } +type PublicationChange = { + type: `insert` | `update` | `delete` + key: RowKey + value: Row + previousValue?: Row +} +type SourceMutation = { + type: `source` + key: RowKey + action: `upsert` | `delete` + value: number +} +type PublicationCommand = + | Exclude + | { type: `truncate`; replacement?: Row } + | SourceMutation +type SyncOperations = Parameters[`sync`]>[0] +type RuntimeAttempt = { + id: number + ownerId: number + demand: DemandName + session: number + operations: SyncOperations + deferred: ReturnType> + signal: AbortSignal | undefined + settled: boolean + current: boolean +} +type RuntimeOwner = { + id: number + demand: DemandName + controller: AbortController + aborted: boolean + attemptId?: number +} +type Replacement = { + session: number + replay: number + rows: Map + failed: boolean +} +type PublicationModel = { + source: Map + visible: Map + // Public rows carried across a discarded source/replay, not yet refreshed. + retainedKeys: Set + replacement?: Replacement + batches: Array> + sentKeys: Set +} +type PublicationPhase = + | `public` + | `private-pending` + | `private-settling` + | `private-failed` +type SourceEffect = `insert` | `update` | `delete` +type PublicationObservation = { + index: number + command: PublicationCommand[`type`] + phaseBefore: PublicationPhase + pendingAttemptsBefore: number + executed: boolean + sourceEffect?: SourceEffect + settlement?: `resolve` | `reject` + publications: number + unloads: number + sessions: number + collectionStatus: string +} +type PublicationMismatch = { + history: string + commandIndex: number + command: PublicationCommand + expected: Array> + observed: Array> +} +type PublicationRunOptions = { + withoutLoader?: boolean + continueAfterMismatch?: boolean + historyName?: string + mismatches?: Array +} + +function recordSourceWrite(publication: PublicationModel, row: Row): void { + // This driver requests raw future changes (includeInitialState: false). + // After retiring private work, an unseen source row can still be updated. + // Retained public rows take precedence when reconciling a stale snapshot. + const previous = + publication.visible.get(row.id) ?? publication.source.get(row.id) + publication.source.set(row.id, cloneRow(row)) + publication.retainedKeys.delete(row.id) + if (previous?.value === row.value) return + publication.visible.set(row.id, cloneRow(row)) + publication.sentKeys.add(row.id) + publication.batches.push([ + previous + ? { + type: `update`, + key: row.id, + value: cloneRow(row), + previousValue: cloneRow(previous), + } + : { type: `insert`, key: row.id, value: cloneRow(row) }, + ]) +} + +const mapsEqual = ( + left: ReadonlyMap, + right: ReadonlyMap, +): boolean => + left.size === right.size && + [...left].every(([id, row]) => right.get(id)?.value === row.value) + +function cloneRow(row: Row): Row { + return { id: row.id, value: row.value } +} + +function clonePublicationBatches( + batches: ReadonlyArray>, +): Array> { + return batches.map((batch) => + batch.map((change) => ({ + ...change, + value: cloneRow(change.value), + ...(change.previousValue + ? { previousValue: cloneRow(change.previousValue) } + : {}), + })), + ) +} + +function normalizePublicationOrder( + batches: ReadonlyArray>, +): Array> { + // Distinct keys have no canonical delivery order within one callback. + // Keep callback boundaries and stable order among changes to the same key. + return clonePublicationBatches(batches).map((batch) => + batch.sort((left, right) => left.key.localeCompare(right.key)), + ) +} + +function publicationPhase( + publication: PublicationModel, + lifecycle: LifecycleModel, +): PublicationPhase { + if (!publication.replacement) return `public` + if (publication.replacement.failed) return `private-failed` + const currentAttemptIds = new Set( + lifecycle.owners.flatMap(({ aborted, attemptId }) => + aborted || attemptId === undefined ? [] : [attemptId], + ), + ) + return lifecycle.attempts.some( + ({ id, settled }) => currentAttemptIds.has(id) && settled, + ) + ? `private-settling` + : `private-pending` +} + +function publicationDiff( + previous: ReadonlyMap, + next: ReadonlyMap, +): Array { + const changes: Array = [] + for (const [key, previousValue] of [...previous].sort(([left], [right]) => + left.localeCompare(right), + )) { + const value = next.get(key) + if (!value) { + changes.push({ type: `delete`, key, value: cloneRow(previousValue) }) + } else if (value.value !== previousValue.value) { + changes.push({ + type: `update`, + key, + value: cloneRow(value), + previousValue: cloneRow(previousValue), + }) + } + } + for (const [key, value] of [...next].sort(([left], [right]) => + left.localeCompare(right), + )) { + if (!previous.has(key)) { + changes.push({ type: `insert`, key, value: cloneRow(value) }) + } + } + return changes +} + +function publishIfChanged( + publication: PublicationModel, + next: Map, +): void { + if (mapsEqual(publication.visible, next)) return + publication.batches.push(publicationDiff(publication.visible, next)) + publication.visible = next + publication.sentKeys = new Set(next.keys()) +} + +function finishReplacement( + publication: PublicationModel, + lifecycle: LifecycleModel, +): void { + const replacement = publication.replacement + if (!replacement) return + const currentAttempts = lifecycle.owners.flatMap(({ aborted, attemptId }) => + aborted || attemptId === undefined ? [] : [lifecycle.attempts[attemptId]!], + ) + replacement.failed = currentAttempts.some( + ({ outcome }) => outcome === `reject`, + ) + if (lifecycle.publicationBarrierOpen) return + if (replacement.failed) { + replacement.failed = true + if (currentAttempts.length === 0) { + publication.replacement = undefined + } + } else if ( + // A canceled-only reset can establish an empty replacement once older + // transports settle. Releasing every owner instead retires the work. + lifecycle.owners.length > 0 && + currentAttempts.every(({ outcome }) => outcome === `resolve`) + ) { + publishIfChanged(publication, new Map(replacement.rows)) + publication.retainedKeys.clear() + publication.replacement = undefined + } else { + // Retire private publication work, not the source's independently applied state. + publication.replacement = undefined + publication.retainedKeys = new Set(publication.visible.keys()) + } +} + +function projectPublication( + publication: PublicationModel, + lifecycle: LifecycleModel, + command: PublicationCommand, + effect: LifecycleEffect, + priorPublicationCount: number, + eagerRestart: boolean, +): void { + if ( + command.type === `truncate` && + lifecycle.active && + !lifecycle.unsubscribed + ) { + const previousSource = new Map(publication.source) + publication.source.clear() + if (command.replacement) { + publication.source.set( + command.replacement.id, + cloneRow(command.replacement), + ) + } + if (lifecycle.publicationBarrierOpen) { + publication.replacement = { + session: lifecycle.session, + replay: lifecycle.replay, + rows: new Map(publication.source), + failed: false, + } + } else { + publication.replacement = undefined + if ( + publication.retainedKeys.size === 0 && + (eagerRestart || lifecycle.owners.length === 0) + ) { + // Without held publications or replay demand this is a raw source + // transaction: all old source rows are deleted, even if never shown. + // A same-key replacement keeps its delete/insert pair in one callback. + const changes: Array = [ + ...[...previousSource].map(([key, value]) => ({ + type: `delete` as const, + key, + value: cloneRow(value), + })), + ...[...publication.source].map(([key, value]) => ({ + type: `insert` as const, + key, + value: cloneRow(value), + })), + ] + if (changes.length > 0) publication.batches.push(changes) + publication.visible = new Map(publication.source) + publication.sentKeys = new Set(publication.source.keys()) + } else { + // Held publications need a replacement diff, not raw source deletes. + publishIfChanged(publication, new Map(publication.source)) + } + publication.retainedKeys.clear() + } + } else if ( + command.type === `restart` && + lifecycle.publications > priorPublicationCount && + lifecycle.publicationBarrierOpen + ) { + publication.replacement = { + session: lifecycle.session, + replay: lifecycle.replay, + rows: new Map(), + failed: false, + } + } else if ( + command.type === `source` && + lifecycle.active && + !lifecycle.unsubscribed + ) { + const previousValue = publication.source.get(command.key) + if (command.action === `delete`) { + publication.source.delete(command.key) + publication.replacement?.rows.delete(command.key) + if (!publication.replacement && previousValue) { + const deletedValue = publication.retainedKeys.has(command.key) + ? (publication.visible.get(command.key) ?? previousValue) + : previousValue + publication.visible.delete(command.key) + publication.retainedKeys.delete(command.key) + publication.sentKeys.delete(command.key) + publication.batches.push([ + { + type: `delete`, + key: command.key, + value: cloneRow(deletedValue), + }, + ]) + } + } else { + const row = { + id: command.key, + value: command.value, + } + if (publication.replacement) { + publication.source.set(command.key, row) + publication.replacement.rows.set(command.key, row) + } else { + recordSourceWrite(publication, row) + } + } + } else if (command.type === `cleanup`) { + publication.retainedKeys = new Set(publication.visible.keys()) + publication.source.clear() + publication.replacement = undefined + } else if (command.type === `release`) { + // Release changes demand, not source retention. Only source writes or a + // successful replacement can change the subscriber's rows. + finishReplacement(publication, lifecycle) + } else if (command.type === `settle` && effect.attemptId !== undefined) { + const attempt = lifecycle.attempts[effect.attemptId]! + const isCurrent = lifecycle.owners.some( + ({ attemptId }) => attemptId === attempt.id, + ) + if (command.outcome === `resolve` && isCurrent && !attempt.aborted) { + const row = { id: attempt.demand, value: attempt.id } + const replacement = publication.replacement + if ( + replacement && + replacement.session === attempt.session && + replacement.replay === attempt.replay + ) { + publication.source.set(row.id, row) + replacement.rows.set(row.id, row) + } else { + recordSourceWrite(publication, row) + } + } + finishReplacement(publication, lifecycle) + } + + // This fixture marks an eager restart ready with its complete (empty) source. + // Unlike on-demand restart, that is authority to retire the retained snapshot. + if ( + eagerRestart && + command.type === `restart` && + lifecycle.publications > priorPublicationCount && + !mapsEqual(publication.visible, publication.source) + ) { + publishIfChanged(publication, new Map(publication.source)) + publication.retainedKeys.clear() + priorPublicationCount++ + } + + for ( + let index = priorPublicationCount; + index < lifecycle.publications; + index++ + ) { + const row = + command.type === `request` && + lifecycle.active && + lifecycle.owners.filter(({ demand }) => demand === command.demand) + .length === 1 && + !publication.sentKeys.has(command.demand) + ? publication.source.get(command.demand) + : undefined + publication.batches.push( + row + ? [ + { + type: `insert`, + key: row.id, + value: cloneRow(row), + }, + ] + : [], + ) + if (row) { + publication.visible.set(row.id, cloneRow(row)) + publication.retainedKeys.delete(row.id) + publication.sentKeys.add(row.id) + } + } +} + +const sourceMutationArbitrary: fc.Arbitrary = fc.record({ + type: fc.constant(`source` as const), + key: fc.constantFrom(`a` as const, `b` as const, `c` as const, `d` as const), + action: fc.constantFrom(`upsert` as const, `delete` as const), + value: fc.integer({ min: 0, max: 5 }), +}) + +const publicationCommandHistoryArbitrary: fc.Arbitrary< + Array +> = greenLifecycleHistoryArbitrary.chain((history) => + fc + .array( + fc.record({ + position: fc.integer({ min: 0, max: history.length }), + command: sourceMutationArbitrary, + }), + { minLength: 1, maxLength: 5 }, + ) + .map((insertions) => { + const commands: Array = [...history] + for (const { position, command } of insertions.sort( + (left, right) => right.position - left.position, + )) { + commands.splice(position, 0, command) + } + return commands + }), +) + +async function runPublicationHistory( + history: ReadonlyArray, + options: PublicationRunOptions = {}, +): Promise> { + const check = options.continueAfterMismatch ? expect.soft : expect + const observations: Array = [] + const lifecycle = createLifecycleModel() + const publication: PublicationModel = { + source: new Map(), + visible: new Map(), + retainedKeys: new Set(), + batches: [], + sentKeys: new Set(), + } + const where = { + a: new Func(`eq`, [new PropRef([`id`]), new Value(`a`)]), + b: new Func(`eq`, [new PropRef([`id`]), new Value(`b`)]), + } + const demandForWhere = new Map([ + [where.a, `a`], + [where.b, `b`], + ]) + const attempts = new Map() + const attemptForOptions = new Map() + const unloads: Array = [] + const owners: Array = [] + const sourceRows = new Map>() + const operationsBySession = new Map() + let nextAttemptId = 0 + let nextOwnerId = 0 + let session = -1 + let active = true + let unsubscribed = false + + const collection = createCollection({ + id: `generated-lifecycle-publication`, + getKey: ({ id }) => id, + syncMode: options.withoutLoader ? `eager` : `on-demand`, + sync: { + sync: (operations) => { + const ownSession = ++session + operationsBySession.set(ownSession, operations) + sourceRows.set(ownSession, new Map()) + operations.markReady() + if (options.withoutLoader) return + return { + loadSubset: (options) => { + const demand = demandForWhere.get(options.where) + if (!demand) throw new Error(`publication load lost its demand`) + const owner = owners.find( + (candidate) => + candidate.demand === demand && + !candidate.aborted && + candidate.attemptId === undefined, + ) + if (!owner) throw new Error(`publication load has no runtime owner`) + const id = nextAttemptId++ + const deferred = createDeferred() + void deferred.promise.catch(() => undefined) + attempts.set(id, { + id, + ownerId: owner.id, + demand, + session: ownSession, + operations, + deferred, + signal: options.signal, + settled: false, + current: true, + }) + attemptForOptions.set(options, id) + owner.attemptId = id + return deferred.promise + }, + unloadSubset: (options) => { + const attemptId = attemptForOptions.get(options) + if (attemptId === undefined) { + throw new Error(`publication unload lost its acquisition`) + } + unloads.push(attemptId) + }, + } + }, + }, + }) + + const visible = new Map() + const observedBatches: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => { + const batch = changes.map((change): PublicationChange => { + const key = change.key + if (key !== `a` && key !== `b` && key !== `c` && key !== `d`) { + throw new Error(`publication used an unknown row key`) + } + return { + type: change.type, + key, + value: cloneRow(change.value), + ...(change.previousValue === undefined + ? {} + : { previousValue: cloneRow(change.previousValue) }), + } + }) + for (const change of batch) { + const id = change.key + if (change.type === `delete`) visible.delete(id) + else visible.set(id, { id, value: change.value.value }) + } + observedBatches.push(batch) + }, + { includeInitialState: false }, + ) + + const writeAttempt = async (attempt: RuntimeAttempt): Promise => { + // Cancellation fences request-scoped writes at the adapter boundary. + // Transport may settle later; it must not publish canceled snapshot rows. + if (attempt.signal?.aborted) return + const rows = sourceRows.get(attempt.session) + const previous = rows?.get(attempt.demand) + const value = { id: attempt.demand, value: attempt.id } + attempt.operations.begin() + attempt.operations.write({ + type: previous ? `update` : `insert`, + value, + ...(previous ? { previousValue: previous } : {}), + }) + const receipt = attempt.operations.commit() + if (receipt !== true) await receipt + rows?.set(attempt.demand, value) + } + + const assertPublications = ( + command: PublicationCommand, + commandIndex: number, + expectedStart: number, + observedStart: number, + ): void => { + const expectedBatches = publication.batches.slice(expectedStart) + const observed = observedBatches.slice(observedStart) + const expected = normalizePublicationOrder(expectedBatches) + const normalizedObserved = normalizePublicationOrder(observed) + const context = JSON.stringify({ + history, + command, + commandIndex, + observed, + expected: expectedBatches, + }) + if ( + options.mismatches && + JSON.stringify(normalizedObserved) !== JSON.stringify(expected) + ) { + const historyName = options.historyName ?? JSON.stringify(history) + options.mismatches.push({ + history: historyName, + commandIndex, + command, + expected: clonePublicationBatches(expectedBatches), + observed: clonePublicationBatches(observed), + }) + return + } + check(normalizedObserved, context).toEqual(expected) + } + + const selectRuntimeAttempt = ( + command: Extract, + ): RuntimeAttempt | undefined => { + if (unsubscribed) return undefined + const candidates = [...attempts.values()].filter( + (attempt) => + !attempt.settled && + attempt.demand === command.demand && + attempt.current === (command.scope === `current`), + ) + return command.age === `oldest` ? candidates[0] : candidates.at(-1) + } + + try { + for (const [index, command] of history.entries()) { + const priorPublicationCount = lifecycle.publications + const phaseBefore = publicationPhase(publication, lifecycle) + const pendingAttemptsBefore = [...attempts.values()].filter( + ({ current, settled }) => current && !settled, + ).length + const observedPublicationCount = observedBatches.length + const expectedPublicationCount = publication.batches.length + const unloadCount = unloads.length + const sessionCount = operationsBySession.size + let executed = false + let sourceEffect: SourceEffect | undefined + let settlement: `resolve` | `reject` | undefined + const runtimeOwner = + command.type === `request` + ? { + id: nextOwnerId++, + demand: command.demand, + controller: new AbortController(), + aborted: false, + } + : command.type === `abort` + ? owners.find( + ({ demand, aborted }) => demand === command.demand && !aborted, + ) + : command.type === `release` + ? owners.find(({ demand }) => demand === command.demand) + : undefined + if (command.type === `request` && !unsubscribed) + owners.push(runtimeOwner!) + const runtimeAttempt = + command.type === `settle` ? selectRuntimeAttempt(command) : undefined + const effect = + command.type === `source` + ? ({} satisfies LifecycleEffect) + : reduceLifecycle(lifecycle, command) + + if (command.type === `source` && active) { + const operations = operationsBySession.get(session) + const rows = sourceRows.get(session) + const previous = rows?.get(command.key) + executed = operations !== undefined && rows !== undefined + sourceEffect = + command.action === `delete` + ? previous + ? `delete` + : undefined + : previous + ? `update` + : `insert` + operations?.begin() + if (command.action === `delete`) { + operations?.write({ type: `delete`, key: command.key }) + rows?.delete(command.key) + } else { + const value = { id: command.key, value: command.value } + operations?.write({ + type: previous ? `update` : `insert`, + value, + ...(previous ? { previousValue: previous } : {}), + }) + rows?.set(command.key, value) + } + const receipt = operations?.commit() + if (receipt !== true) await receipt + } else if (command.type === `request`) { + check(effect.ownerId).toBe(unsubscribed ? undefined : runtimeOwner?.id) + executed = runtimeOwner !== undefined && !unsubscribed + subscription.requestSnapshot({ + where: where[command.demand], + signal: runtimeOwner?.controller.signal, + }) + } else if (command.type === `abort`) { + check(effect.ownerId).toBe(runtimeOwner?.id) + executed = runtimeOwner !== undefined + if (runtimeOwner) { + runtimeOwner.aborted = true + runtimeOwner.controller.abort() + } + } else if (command.type === `release`) { + check(effect.ownerId).toBe(runtimeOwner?.id) + executed = runtimeOwner !== undefined + if (runtimeOwner) { + if (runtimeOwner.attemptId !== undefined) { + attempts.get(runtimeOwner.attemptId)!.current = false + } + owners.splice(owners.indexOf(runtimeOwner), 1) + } + subscription.releaseSnapshot(where[command.demand]) + } else if (command.type === `settle`) { + check(effect.attemptId).toBe(runtimeAttempt?.id) + executed = runtimeAttempt !== undefined + if (runtimeAttempt) settlement = command.outcome + if (effect.attemptId !== undefined && runtimeAttempt) { + runtimeAttempt.settled = true + const expected = lifecycle.attempts[ + effect.attemptId + ] as LifecycleAttempt + if (command.outcome === `resolve`) { + await writeAttempt(runtimeAttempt) + runtimeAttempt.deferred.resolve() + } else { + runtimeAttempt.deferred.reject(expected.failure) + } + } + } else if (command.type === `truncate` && active) { + executed = true + for (const owner of owners) { + if (owner.attemptId !== undefined) { + attempts.get(owner.attemptId)!.current = false + } + owner.attemptId = undefined + } + const operations = operationsBySession.get(session) + operations?.begin() + operations?.truncate() + if (command.replacement) { + operations?.write({ + type: `insert`, + value: cloneRow(command.replacement), + }) + } + const receipt = operations?.commit() + if (receipt !== true) await receipt + sourceRows.get(session)?.clear() + if (command.replacement) { + sourceRows + .get(session) + ?.set(command.replacement.id, cloneRow(command.replacement)) + } + } else if (command.type === `cleanup` && active) { + executed = true + for (const owner of owners) { + if (owner.attemptId !== undefined) { + attempts.get(owner.attemptId)!.current = false + } + owner.attemptId = undefined + } + await collection.cleanup() + active = false + } else if (command.type === `restart` && !active) { + executed = true + collection.startSyncImmediate() + active = true + } else if (command.type === `unsubscribe`) { + executed = !unsubscribed + for (const attempt of attempts.values()) attempt.current = false + subscription.unsubscribe() + unsubscribed = true + owners.length = 0 + } + + await flushPromises() + projectPublication( + publication, + lifecycle, + command, + effect, + priorPublicationCount, + options.withoutLoader ?? false, + ) + // Public retention never rewrites the independently installed source. + // The publication model stops tracking source commands after unsubscribe; + // its callback-silence assertions below still cover that suffix. + if (!lifecycle.unsubscribed) + check( + [...(active ? collection.values() : [])] + .map(cloneRow) + .sort((left, right) => left.id.localeCompare(right.id)), + `source state after command ${index}: ${JSON.stringify(command)}`, + ).toEqual( + [...publication.source.values()].sort((left, right) => + left.id.localeCompare(right.id), + ), + ) + assertPublications( + command, + index, + expectedPublicationCount, + observedPublicationCount, + ) + check( + [...visible.values()].sort((left, right) => + left.id.localeCompare(right.id), + ), + `consumer state after command ${index}: ${JSON.stringify(command)}`, + ).toEqual( + [...publication.visible.values()].sort((left, right) => + left.id.localeCompare(right.id), + ), + ) + observations.push({ + index, + command: command.type, + phaseBefore, + pendingAttemptsBefore, + executed, + ...(sourceEffect ? { sourceEffect } : {}), + ...(settlement ? { settlement } : {}), + publications: observedBatches.length - observedPublicationCount, + unloads: unloads.length - unloadCount, + sessions: operationsBySession.size - sessionCount, + collectionStatus: collection.status, + }) + } + } finally { + for (const attempt of attempts.values()) attempt.deferred.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + return observations +} + +type ProductSettlement = `none` | `resolve` | `reject` +type ProductSuffix = `release` | `cleanup` | `restart` | `unsubscribe` +type PriorIndependentRow = `absent` | `present` +type PublicationProductCase = { + name: string + phase: PublicationPhase + sourceEffect: SourceEffect + settlement: ProductSettlement + suffix: ProductSuffix + priorIndependentRow: PriorIndependentRow + history: Array + focalSourceIndex: number + settlementIndex?: number + pendingProbeIndex: number + suffixIndex: number + postUnsubscribeProbeIndex: number +} + +const settleCurrent = ( + demand: DemandName, + outcome: `resolve` | `reject`, +): LifecycleCommand => ({ + type: `settle`, + demand, + scope: `current`, + age: `oldest`, + outcome, +}) + +function createPublicationProductCase( + phase: PublicationPhase, + sourceEffect: SourceEffect, + settlement: ProductSettlement, + suffix: ProductSuffix, + priorIndependentRow: PriorIndependentRow, +): PublicationProductCase { + const history: Array = [] + const push = (command: PublicationCommand): number => + history.push(command) - 1 + const hasPeer = phase === `private-settling` || phase === `private-failed` + + if (priorIndependentRow === `present`) { + push({ type: `source`, key: `d`, action: `upsert`, value: 30 }) + } + push({ type: `request`, demand: `a` }) + if (hasPeer) push({ type: `request`, demand: `b` }) + if (phase !== `public`) { + push(settleCurrent(`a`, `resolve`)) + if (hasPeer) push(settleCurrent(`b`, `resolve`)) + push({ type: `truncate` }) + if (phase === `private-settling`) { + push(settleCurrent(`b`, `resolve`)) + } else if (phase === `private-failed`) { + push(settleCurrent(`b`, `reject`)) + } + } + + if (sourceEffect !== `insert`) { + push({ type: `source`, key: `c`, action: `upsert`, value: 40 }) + } + const focalSourceIndex = push({ + type: `source`, + key: `c`, + action: sourceEffect === `delete` ? `delete` : `upsert`, + value: 41, + }) + const settlementIndex = + settlement === `none` ? undefined : push(settleCurrent(`a`, settlement)) + const pendingProbeIndex = history.length + + let suffixIndex: number + if (suffix === `release`) { + suffixIndex = push({ type: `release`, demand: `a` }) + if (hasPeer) suffixIndex = push({ type: `release`, demand: `b` }) + push({ type: `unsubscribe` }) + } else if (suffix === `cleanup`) { + suffixIndex = push({ type: `cleanup` }) + push({ type: `unsubscribe` }) + } else if (suffix === `restart`) { + push({ type: `cleanup` }) + suffixIndex = push({ type: `restart` }) + push({ type: `unsubscribe` }) + } else { + suffixIndex = push({ type: `unsubscribe` }) + } + if (suffix === `cleanup`) push({ type: `restart` }) + const postUnsubscribeProbeIndex = push({ + type: `source`, + key: `d`, + action: `upsert`, + value: 99, + }) + + return { + name: `${phase}:${sourceEffect}:${settlement}:${suffix}:prior-${priorIndependentRow}`, + phase, + sourceEffect, + settlement, + suffix, + priorIndependentRow, + history, + focalSourceIndex, + ...(settlementIndex === undefined ? {} : { settlementIndex }), + pendingProbeIndex, + suffixIndex, + postUnsubscribeProbeIndex, + } +} + +const publicationPhases = [ + `public`, + `private-pending`, + `private-settling`, + `private-failed`, +] as const +const sourceEffects = [`insert`, `update`, `delete`] as const +const productSettlements = [`none`, `resolve`, `reject`] as const +const productSuffixes = [ + `release`, + `cleanup`, + `restart`, + `unsubscribe`, +] as const +const priorIndependentRows = [`absent`, `present`] as const + +const publicationProductCases = publicationPhases.flatMap((phase) => + sourceEffects.flatMap((sourceEffect) => + productSettlements.flatMap((settlement) => + productSuffixes.flatMap((suffix) => + priorIndependentRows.map((priorIndependentRow) => + createPublicationProductCase( + phase, + sourceEffect, + settlement, + suffix, + priorIndependentRow, + ), + ), + ), + ), + ), +) + +const successfulReplacementCases = publicationProductCases.filter( + ({ phase, sourceEffect, settlement }) => + (phase === `private-pending` || phase === `private-settling`) && + sourceEffect !== `delete` && + settlement === `resolve`, +) +const replacementOrderingCases = publicationProductCases.filter( + ({ phase, sourceEffect, settlement, suffix, priorIndependentRow }) => + (phase === `private-pending` || phase === `private-settling`) && + sourceEffect === `delete` && + settlement === `resolve` && + suffix !== `release` && + priorIndependentRow === `present`, +) +const replacementRetirementCases = publicationProductCases.filter( + (scenario) => + scenario.phase !== `public` && + scenario.suffix === `release` && + !successfulReplacementCases.includes(scenario) && + !replacementOrderingCases.includes(scenario), +) +const publicationControlCases = publicationProductCases.filter( + (scenario) => + !successfulReplacementCases.includes(scenario) && + !replacementOrderingCases.includes(scenario) && + !replacementRetirementCases.includes(scenario), +) + +async function runPublicationProduct( + scenarios: ReadonlyArray, +): Promise> { + const mismatches: Array = [] + const reached = new Set() + + for (const scenario of scenarios) { + const observations = await runPublicationHistory(scenario.history, { + historyName: scenario.name, + mismatches, + }) + expect(observations).toHaveLength(scenario.history.length) + expect(observations[scenario.focalSourceIndex]).toMatchObject({ + command: `source`, + phaseBefore: scenario.phase, + executed: true, + sourceEffect: scenario.sourceEffect, + }) + if (scenario.settlementIndex === undefined) { + expect( + observations[scenario.pendingProbeIndex]!.pendingAttemptsBefore, + scenario.name, + ).toBe(1) + } else { + expect(observations[scenario.settlementIndex]).toMatchObject({ + command: `settle`, + executed: true, + settlement: scenario.settlement, + publications: + scenario.settlement === `resolve` && + scenario.phase !== `private-failed` + ? 1 + : 0, + }) + } + + const suffix = observations[scenario.suffixIndex]! + expect(suffix).toMatchObject({ + command: scenario.suffix, + executed: true, + }) + if (scenario.suffix === `release`) { + expect(suffix.unloads, scenario.name).toBe(1) + } else if (scenario.suffix === `cleanup`) { + expect(suffix.collectionStatus, scenario.name).toBe(`cleaned-up`) + } else if (scenario.suffix === `restart`) { + expect(suffix.sessions, scenario.name).toBe(1) + } else { + const ownerCount = + scenario.phase === `private-settling` || + scenario.phase === `private-failed` + ? 2 + : 1 + expect(suffix.unloads, scenario.name).toBe(ownerCount) + } + + expect(observations[scenario.postUnsubscribeProbeIndex]).toMatchObject({ + command: `source`, + executed: true, + publications: 0, + }) + reached.add(scenario.name) + } + + expect(reached).toEqual(new Set(scenarios.map(({ name }) => name))) + return mismatches +} + +function expectNoPublicationMismatches( + mismatches: ReadonlyArray, +): void { + const summary = mismatches.map( + ({ history, commandIndex, command, expected, observed }) => ({ + history, + commandIndex, + command, + expected, + observed, + }), + ) + expect( + summary, + `publication product mismatches: ${JSON.stringify(summary)}`, + ).toEqual([]) +} + +describe(`CollectionSubscription lifecycle publication oracle`, () => { + it.each( + ([undefined, false] as const).flatMap((includeInitialState) => + ([`update`, `delete`, `truncate`] as const).map((operation) => ({ + includeInitialState, + operation, + })), + ), + )( + `distinguishes unseen-row $operation with includeInitialState=$includeInitialState`, + async ({ includeInitialState, operation }) => { + let operations!: SyncOperations + const collection = createCollection({ + getKey: ({ id }) => id, + startSync: true, + sync: { + sync: (sync) => { + operations = sync + sync.begin() + sync.write({ type: `insert`, value: { id: `d`, value: 0 } }) + sync.commit() + sync.markReady() + }, + }, + }) + const changes: Array = [] + const subscription = collection.subscribeChanges( + (batch) => { + for (const change of batch) { + changes.push({ + type: change.type, + key: change.value.id, + value: cloneRow(change.value), + ...(change.previousValue + ? { previousValue: cloneRow(change.previousValue) } + : {}), + }) + } + }, + { includeInitialState }, + ) + try { + expect(changes).toEqual([]) + operations.begin() + if (operation === `truncate`) operations.truncate() + else if (operation === `delete`) + operations.write({ type: `delete`, key: `d` }) + else operations.write({ type: `update`, value: { id: `d`, value: 4 } }) + await operations.commit() + expect(changes).toEqual( + operation === `update` + ? [ + { + type: includeInitialState === false ? `update` : `insert`, + key: `d`, + value: { id: `d`, value: 4 }, + ...(includeInitialState === false + ? { previousValue: { id: `d`, value: 0 } } + : {}), + }, + ] + : includeInitialState === false + ? [{ type: `delete`, key: `d`, value: { id: `d`, value: 0 } }] + : [], + ) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`keeps raw source updates after retiring the final replay owner`, async () => { + await runPublicationHistory([ + { type: `cleanup` }, + { type: `release`, demand: `b` }, + { type: `request`, demand: `b` }, + { type: `restart` }, + { type: `source`, key: `d`, action: `upsert`, value: 0 }, + { type: `abort`, demand: `b` }, + { type: `release`, demand: `b` }, + { type: `abort`, demand: `b` }, + { type: `source`, key: `d`, action: `upsert`, value: 4 }, + { type: `release`, demand: `b` }, + { type: `restart` }, + { type: `unsubscribe` }, + ]) + }) + + it(`keeps raw truncate deletes after retiring the final replay owner`, async () => { + await runPublicationHistory([ + { + type: `settle`, + demand: `a`, + scope: `obsolete`, + age: `oldest`, + outcome: `reject`, + }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `source`, key: `a`, action: `upsert`, value: 5 }, + { type: `release`, demand: `a` }, + { type: `restart` }, + { type: `truncate` }, + ]) + }) + + it(`records requested snapshot rows before a canceled reset`, async () => { + await runPublicationHistory([ + { type: `request`, demand: `b` }, + { type: `truncate` }, + { type: `source`, key: `b`, action: `upsert`, value: 0 }, + { type: `release`, demand: `b` }, + { type: `request`, demand: `b` }, + { type: `restart` }, + { type: `abort`, demand: `a` }, + { type: `abort`, demand: `b` }, + { type: `restart` }, + { type: `truncate` }, + { type: `restart` }, + { type: `unsubscribe` }, + { type: `release`, demand: `b` }, + ]) + }) + + it(`does not invent a publication for an unchanged private row`, async () => { + await runPublicationHistory([ + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `source`, key: `a`, action: `upsert`, value: 5 }, + { type: `release`, demand: `a` }, + { type: `source`, key: `a`, action: `upsert`, value: 5 }, + { type: `unsubscribe` }, + ]) + }) + + it.each( + ([`none`, `retained`, `refreshed`, `new`] as const).flatMap((baseline) => + ([`delete`, `empty`, `same`, `other`] as const).map((reset) => ({ + baseline, + reset, + })), + ), + )( + `distinguishes raw source resets from retained replacements: $baseline/$reset`, + async ({ baseline, reset }) => { + await runPublicationHistory([ + ...(baseline === `retained` || baseline === `refreshed` + ? [{ type: `source`, key: `d`, action: `upsert`, value: 0 } as const] + : []), + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `source`, key: `a`, action: `upsert`, value: 5 }, + { type: `release`, demand: `a` }, + ...(baseline === `refreshed` || baseline === `new` + ? [{ type: `source`, key: `d`, action: `upsert`, value: 0 } as const] + : []), + reset === `delete` + ? { type: `source`, key: `a`, action: `delete`, value: 0 } + : { + type: `truncate`, + ...(reset === `same` + ? { replacement: { id: `a`, value: 5 } as const } + : reset === `other` + ? { replacement: { id: `c`, value: 6 } as const } + : {}), + }, + { type: `unsubscribe` }, + ]) + }, + ) + + it(`reconciles a repeated reset after the last replay owner aborts`, async () => { + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `request`, demand: `a` }, + { type: `truncate` }, + { type: `abort`, demand: `a` }, + { type: `truncate` }, + { + type: `settle`, + demand: `a`, + scope: `obsolete`, + age: `newest`, + outcome: `resolve`, + }, + { type: `cleanup` }, + ]) + }) + + it(`keeps independent source rows when a replay settles after redundant restart calls`, async () => { + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `truncate` }, + { type: `source`, key: `b`, action: `upsert`, value: 1 }, + { type: `restart` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `newest`, + outcome: `reject`, + }, + { type: `restart` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `truncate` }, + { type: `request`, demand: `b` }, + { type: `cleanup` }, + ]) + }) + + it.each([ + `none`, + `missing`, + `duplicate`, + `value`, + `previous-value`, + `split`, + `merge`, + `same-key-order`, + ] as const)( + `normalizes only independent change order with corruption: %s`, + (corruption) => { + const baseline: Array> = [ + [ + { + type: `update`, + key: `a`, + value: { id: `a`, value: 1 }, + previousValue: { id: `a`, value: 0 }, + }, + { type: `delete`, key: `b`, value: { id: `b`, value: 0 } }, + { type: `insert`, key: `c`, value: { id: `c`, value: 1 } }, + ], + [ + { + type: `update`, + key: `a`, + value: { id: `a`, value: 2 }, + previousValue: { id: `a`, value: 1 }, + }, + { type: `delete`, key: `a`, value: { id: `a`, value: 2 } }, + ], + ] + const permutations = [ + [0, 1, 2], + [0, 2, 1], + [1, 0, 2], + [1, 2, 0], + [2, 0, 1], + [2, 1, 0], + ] + for (const permutation of permutations) { + const candidate = clonePublicationBatches(baseline) + const changes = candidate[0]! + if (corruption === `value`) changes[0]!.value.value++ + if (corruption === `previous-value`) changes[0]!.previousValue!.value++ + candidate[0] = permutation.map((index) => changes[index]!) + if (corruption === `missing`) candidate[0].pop() + if (corruption === `duplicate`) candidate[0].push(changes[0]!) + if (corruption === `split`) + candidate.splice(1, 0, candidate[0].splice(1)) + if (corruption === `merge`) candidate.splice(0, 2, candidate.flat()) + if (corruption === `same-key-order`) candidate[1]!.reverse() + const actual = normalizePublicationOrder(candidate) + const expected = normalizePublicationOrder(baseline) + if (corruption === `none`) expect(actual).toEqual(expected) + else expect(actual).not.toEqual(expected) + } + }, + ) + + it(`defines all 288 unique row-publication lifecycle cells`, () => { + expect(publicationProductCases).toHaveLength(288) + expect(new Set(publicationProductCases.map(({ name }) => name)).size).toBe( + 288, + ) + expect(successfulReplacementCases).toHaveLength(32) + expect(replacementOrderingCases).toHaveLength(6) + expect(replacementRetirementCases).toHaveLength(46) + expect(publicationControlCases).toHaveLength(204) + }) + + it(`matches row publications for lifecycle control cells`, async () => { + expectNoPublicationMismatches( + await runPublicationProduct(publicationControlCases), + ) + }) + + it(`preserves independent source work when a successful replay publishes`, async () => { + expectNoPublicationMismatches( + await runPublicationProduct(successfulReplacementCases), + ) + }) + + it(`publishes complete replacement batches regardless of independent key order`, async () => { + expectNoPublicationMismatches( + await runPublicationProduct(replacementOrderingCases), + ) + }) + + it(`retires incomplete or failed replacement without changing independent public rows`, async () => { + expectNoPublicationMismatches( + await runPublicationProduct(replacementRetirementCases), + ) + }) + + it(`maps every canonical green lifecycle history to public rows`, async () => { + for (const history of greenLifecycleHistories) { + await runPublicationHistory(history) + } + }) + + it(`suppresses canceled source writes when a released acquisition settles`, async () => { + await runPublicationHistory( + [ + { type: `request`, demand: `a` }, + { type: `request`, demand: `b` }, + { type: `release`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `obsolete`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `b` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`publishes an authoritative truncate after the final demand is released`, async () => { + await runPublicationHistory([ + { type: `request`, demand: `b` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `b` }, + { type: `truncate` }, + { type: `unsubscribe` }, + ]) + }) + + it(`publishes independent source changes with a successful replay`, async () => { + await runPublicationHistory( + [ + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `truncate` }, + { type: `source`, key: `b`, action: `upsert`, value: 50 }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`does not republish a row already delivered by a live change`, async () => { + await runPublicationHistory( + [ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `request`, demand: `b` }, + { type: `source`, key: `b`, action: `upsert`, value: 1 }, + { type: `request`, demand: `b` }, + { type: `release`, demand: `b` }, + { type: `release`, demand: `b` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`suppresses canceled source writes when an aborted acquisition settles`, async () => { + await runPublicationHistory( + [ + { type: `request`, demand: `a` }, + { type: `abort`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`keeps later source changes private after a failed replay`, async () => { + await runPublicationHistory([ + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `truncate` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `source`, key: `b`, action: `upsert`, value: 51 }, + ]) + }) + + it(`retires failed private replacement rows when its final owner releases`, async () => { + await runPublicationHistory( + [ + { type: `source`, key: `b`, action: `upsert`, value: 7 }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `truncate` }, + { type: `source`, key: `b`, action: `upsert`, value: 51 }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `reject`, + }, + { type: `release`, demand: `a` }, + { type: `source`, key: `b`, action: `upsert`, value: 8 }, + { type: `cleanup` }, + { type: `restart` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`does not delete an independent public row when releasing a restarted demand`, async () => { + await runPublicationHistory( + [ + { type: `source`, key: `b`, action: `upsert`, value: 0 }, + { type: `request`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + { type: `source`, key: `b`, action: `upsert`, value: 1 }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`keeps a restarted snapshot private while canceled replay work settles`, async () => { + // Seed 2018803696, path 65:10:2:10:13:12:12:0:0:0. A canceled-only + // truncate does not discharge the earlier replay's publication wait. + await runPublicationHistory( + [ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `cleanup` }, + { type: `request`, demand: `b` }, + { type: `restart` }, + { type: `abort`, demand: `b` }, + { type: `truncate` }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { + type: `settle`, + demand: `b`, + scope: `obsolete`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it(`matches retained rows across an empty restart and canceled-only truncates`, async () => { + // Seed 2018803696, path 65:29:0:0:0 exposed the model's missing retained-row + // deletion: an authoritative empty reset is not limited to resident rows. + await runPublicationHistory( + [ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `release`, demand: `a` }, + { type: `cleanup` }, + { type: `restart` }, + { type: `request`, demand: `b` }, + { type: `restart` }, + { type: `abort`, demand: `b` }, + { type: `abort`, demand: `b` }, + { type: `truncate` }, + { type: `truncate` }, + { type: `request`, demand: `b` }, + { + type: `settle`, + demand: `b`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { + type: `settle`, + demand: `b`, + scope: `obsolete`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `b` }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, + ], + { continueAfterMismatch: true }, + ) + }) + + it.each( + [false, true].flatMap((restart) => + [false, true].map((canceledOwner) => ({ restart, canceledOwner })), + ), + )( + `publishes an authoritative empty reset: %j`, + async ({ restart, canceledOwner }) => { + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + ...(restart + ? [{ type: `cleanup` } as const, { type: `restart` } as const] + : []), + ...(canceledOwner + ? [ + { type: `request`, demand: `b` } as const, + { type: `abort`, demand: `b` } as const, + ] + : []), + { type: `truncate` }, + { type: `truncate` }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, + ]) + }, + ) + + it.each( + [false, true].flatMap((canceledOwner) => + ( + [ + { id: `a`, value: 0 }, + { id: `a`, value: 1 }, + { id: `c`, value: 2 }, + ] satisfies Array + ).map((replacement) => ({ canceledOwner, replacement })), + ), + )( + `installs a retained-row replacement atomically: %j`, + async ({ canceledOwner, replacement }) => { + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `cleanup` }, + { type: `restart` }, + ...(canceledOwner + ? [ + { type: `request`, demand: `b` } as const, + { type: `abort`, demand: `b` } as const, + ] + : []), + { type: `truncate`, replacement }, + { type: `source`, key: replacement.id, action: `upsert`, value: 3 }, + { type: `release`, demand: `b` }, + { type: `unsubscribe` }, + ]) + }, + ) + + it.each([ + undefined, + { id: `a`, value: 0 }, + { id: `a`, value: 1 }, + { id: `c`, value: 2 }, + ] satisfies Array)( + `publishes eager restart before a later source reset: %j`, + async (replacement) => { + await runPublicationHistory( + [ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `cleanup` }, + { type: `restart` }, + { type: `truncate`, replacement }, + { type: `source`, key: `a`, action: `upsert`, value: 3 }, + { type: `unsubscribe` }, + ], + { withoutLoader: true }, + ) + }, + ) + + it(`resets retained rows after no-op cleanup and release commands`, async () => { + // Seed 1657005, path 164:18 after removing the visible-row request omission. + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 1 }, + { type: `cleanup` }, + { type: `cleanup` }, + { type: `release`, demand: `b` }, + { type: `restart` }, + { type: `truncate` }, + { type: `request`, demand: `a` }, + { type: `release`, demand: `a` }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ]) + }) + + it(`resets retained rows after the last replay owner retires`, async () => { + // Seed 333468655, path 59:13:0:0:0. + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `request`, demand: `b` }, + { type: `truncate` }, + { type: `release`, demand: `b` }, + { type: `truncate` }, + { type: `release`, demand: `a` }, + { type: `truncate` }, + { type: `request`, demand: `a` }, + { + type: `settle`, + demand: `a`, + scope: `current`, + age: `oldest`, + outcome: `resolve`, + }, + { type: `release`, demand: `a` }, + { type: `unsubscribe` }, + ]) + }) + + it(`does not restore source rows when the final replay owner retires`, async () => { + // Seed 1337491191, path 591:20:1:8:8:8:7:7. + await runPublicationHistory([ + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `restart` }, + { type: `truncate` }, + { type: `source`, key: `a`, action: `upsert`, value: 0 }, + { type: `request`, demand: `b` }, + { type: `truncate` }, + { type: `release`, demand: `b` }, + { type: `source`, key: `a`, action: `delete`, value: 0 }, + { type: `source`, key: `a`, action: `upsert`, value: 1 }, + { type: `unsubscribe` }, + ]) + }) + + const { multiplier, ...replay } = readOracleRunConfig() + const runs = 60 * multiplier + + fcTest.prop([publicationCommandHistoryArbitrary], { + numRuns: runs, + seed: 1_657_005, + })( + `matches row publications for a fixed seed`, + async (history) => { + await runPublicationHistory(history) + }, + 120_000, + ) + fcTest.prop( + [publicationCommandHistoryArbitrary], + oracleRandomParameters( + runs, + replay, + `subscription-lifecycle.publication-history`, + ), + )( + `matches row publications for a random or replayed seed`, + async (history) => { + await runPublicationHistory(history) + }, + 120_000, + ) +}) diff --git a/packages/db/tests/collection-subscription-replay-oracle.property.test.ts b/packages/db/tests/collection-subscription-replay-oracle.property.test.ts index eabf80b23e..9f9a114fea 100644 --- a/packages/db/tests/collection-subscription-replay-oracle.property.test.ts +++ b/packages/db/tests/collection-subscription-replay-oracle.property.test.ts @@ -1,5 +1,5 @@ import { fc, test as fcTest } from '@fast-check/vitest' -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' import { createCollection } from '../src/collection/index.js' import { createDeferred } from '../src/deferred.js' import { BTreeIndex } from '../src/indexes/btree-index.js' @@ -10,11 +10,14 @@ import { createTransaction } from '../src/transactions.js' import { oracleRandomParameters, readOracleRunConfig } from './oracle-config.js' import { flushPromises } from './utils.js' import type { Collection } from '../src/collection/index.js' +import type { CollectionSubscription } from '../src/collection/subscription.js' import type { OrderBy } from '../src/query/ir.js' import type { ChangeMessageOrDeleteKeyMessage, LoadSubsetOptions, + SyncConfig, } from '../src/types.js' +import type { Scheduler } from 'fast-check' type ReplayRow = { id: `one` | `two` @@ -184,7 +187,18 @@ const replayScenarioArbitrary: fc.Arbitrary = fc minLength: 0, maxLength: 3, }) - : fc.constant>([]), + : fc + .tuple( + fc.constant({ + type: `request`, + demandId: releaseOnLastAttempt, + }), + fc.array(sourceActionArbitrary(demandIds), { + minLength: 0, + maxLength: 2, + }), + ) + .map(([request, actions]) => [request, ...actions]), }) .map(({ settlementOrder, rawSettlementPhases, afterSettlement }) => ({ initialRows, @@ -599,9 +613,10 @@ async function runReplayScenario(scenario: ReplayScenario): Promise { const settleReplay = async (replayIndex: number) => { const pending = pendingReplays[replayIndex]! - const session = modelSession! + const session = modelSession const load = pending.load const isCurrent = + session !== undefined && pending.attemptIndex === session.currentAttemptIndex && activeDemandIds.has(load.demandId) pending.settled = true @@ -616,11 +631,18 @@ async function runReplayScenario(scenario: ReplayScenario): Promise { } pending.deferred.reject(pending.error) } - session.pending.delete(replayIndex) + session?.pending.delete(replayIndex) await flushPromises() assertSource() - const hasPendingReplay = pendingReplays.some(({ settled }) => !settled) + if (!session) { + expect(subscription.status).toBe(`ready`) + assertPublished(expectedPublished) + expect(subscription.lastError).toBe(lastReportedError) + return + } + + const hasPendingReplay = session.pending.size > 0 expect(subscription.status).toBe( hasPendingReplay ? `loadingSubset` : `ready`, ) @@ -633,13 +655,7 @@ async function runReplayScenario(scenario: ReplayScenario): Promise { ) const previousPublication = new Map(expectedPublished) expectedPublished.clear() - const nextRows = currentAttemptSucceeds - ? rowsById( - currentAttempt.loads.flatMap(({ demandId, rows }) => - activeDemandIds.has(demandId) ? rows : [], - ), - ) - : session.baseline + const nextRows = currentAttemptSucceeds ? sourceRows : session.baseline for (const [id, row] of nextRows) { expectedPublished.set(id, { ...row }) } @@ -657,11 +673,11 @@ async function runReplayScenario(scenario: ReplayScenario): Promise { sortedChanges(expectedBatch), ) } + modelSession = undefined } else { expect(publicationCount).toBe(session.publicationCount) } expectedPublicationCount = publicationCount - modelSession = undefined } else { expect(publicationCount).toBe(session.publicationCount) } @@ -704,16 +720,32 @@ async function runReplayScenario(scenario: ReplayScenario): Promise { attemptIndex === scenario.attempts.length - 1 && scenario.releaseOnLastAttempt !== undefined ) { - subscription.releaseSnapshot( - demandWheres.get(scenario.releaseOnLastAttempt)!, - ) - activeDemandIds.delete(scenario.releaseOnLastAttempt) + const releasedDemand = scenario.releaseOnLastAttempt + subscription.releaseSnapshot(demandWheres.get(releasedDemand)!) + activeDemandIds.delete(releasedDemand) + for (const replayIndex of modelSession.pending) { + if (pendingReplays[replayIndex]?.load.demandId === releasedDemand) { + modelSession.pending.delete(replayIndex) + } + } + // A released request does not retract rows already applied by the + // source, nor change the retained baseline of an unfinished replay. + if (modelSession.pending.size === 0 && activeDemandIds.size === 0) { + expectedPublicationCount = publicationCount + modelSession = undefined + } } assertSource() assertPublished(expectedPublished) - expect(publicationCount).toBe(modelSession.publicationCount) + expect(publicationCount).toBe( + modelSession?.publicationCount ?? expectedPublicationCount, + ) expect(subscription.lastError).toBe(lastReportedError) - expect(subscription.status).toBe(`loadingSubset`) + expect(subscription.status).toBe( + modelSession && modelSession.pending.size > 0 + ? `loadingSubset` + : `ready`, + ) for (const replayIndex of scenario.settlementOrder) { const replay = pendingReplays[replayIndex] @@ -727,7 +759,7 @@ async function runReplayScenario(scenario: ReplayScenario): Promise { } } - expect(modelSession).toBeUndefined() + expect(modelSession?.pending.size ?? 0).toBe(0) for (const action of scenario.afterSettlement) { const countBeforeAction = publicationCount @@ -740,15 +772,19 @@ async function runReplayScenario(scenario: ReplayScenario): Promise { where: demandWheres.get(action.demandId), }) const row = sourceRows.get(action.demandId) - if (row) expectedPublished.set(action.demandId, { ...row }) + if (!modelSession && row) { + expectedPublished.set(action.demandId, { ...row }) + } } const applied = applySourceAction(action) if (applied && action.type === `delete`) { - expectedPublished.delete(action.id) + if (!modelSession) expectedPublished.delete(action.id) } else if (applied && action.type === `put`) { recordExpectedSourceWrite([action.row], { type: `ordinary` }, true) assertSourceWrites() - expectedPublished.set(action.row.id, { ...action.row }) + if (!modelSession) { + expectedPublished.set(action.row.id, { ...action.row }) + } } assertSource() assertPublished(expectedPublished) @@ -756,10 +792,12 @@ async function runReplayScenario(scenario: ReplayScenario): Promise { previousPublication, expectedPublished, ) + const expectsPublication = + !modelSession && (action.type === `request` || expectedBatch.length > 0) expect(publicationCount).toBe( - countBeforeAction + Number(expectedBatch.length > 0), + countBeforeAction + Number(expectsPublication), ) - if (expectedBatch.length > 0) { + if (expectsPublication) { expect(sortedChanges(publicationBatches.at(-1)!)).toEqual( sortedChanges(expectedBatch), ) @@ -1044,6 +1082,82 @@ async function runCleanupRestartScenario( } } +async function expectScheduledReplaySettlementIsGenerationSafe( + scheduler: Scheduler, +): Promise { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const loads: Array<{ + signal: AbortSignal | undefined + outcome: Promise + }> = [] + const collection = createCollection({ + id: `scheduled-replay-settlement`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (actions) => { + begin = actions.begin + write = actions.write + commit = actions.commit + truncate = actions.truncate + actions.markReady() + return { + loadSubset: ({ signal }) => { + const generation = loads.length + 1 + const outcome = scheduler + .schedule(Promise.resolve(), `generation-${generation}`) + .then(() => { + if (signal?.aborted) return + begin() + write({ + type: `insert`, + value: { id: `one`, value: generation }, + }) + commit() + }) + loads.push({ signal, outcome }) + return outcome + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes) + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + + expect(loads).toHaveLength(2) + expect(loads[0]!.signal?.aborted).toBe(true) + + await scheduler.waitAll() + await Promise.all(loads.map(({ outcome }) => outcome)) + await flushPromises() + + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + expect(subscription.status).toBe(`ready`) + expect(subscription.lastError).toBeUndefined() + } finally { + if (scheduler.count() > 0) await scheduler.waitAll() + await Promise.allSettled(loads.map(({ outcome }) => outcome)) + subscription.unsubscribe() + await collection.cleanup() + } +} + async function runSharedSubscriptionScenario( scenario: SharedSubscriptionScenario, ): Promise { @@ -1052,14 +1166,14 @@ async function runSharedSubscriptionScenario( message: ChangeMessageOrDeleteKeyMessage, ) => void let commit!: () => void - const transport = createDeferred() - let transportOptions: LoadSubsetOptions | undefined - let transportCalls = 0 + const transports = [createDeferred(), createDeferred()] as const + const transportOptions: Array = [] const unloads: Array = [] const dedupe = new DeduplicatedLoadSubset({ loadSubset: (options) => { - transportCalls++ - transportOptions = options + const transport = transports[transportOptions.length] + if (!transport) throw new Error(`unexpected transport`) + transportOptions.push(options) return transport.promise }, }) @@ -1104,34 +1218,37 @@ async function runSharedSubscriptionScenario( try { subscriptions[0].requestSnapshot({ where }) subscriptions[1].requestSnapshot({ where }) - expect(transportCalls).toBe(1) + expect(transportOptions).toHaveLength(2) expect(subscriptions[0].status).toBe(`loadingSubset`) expect(subscriptions[1].status).toBe(`loadingSubset`) if (scenario.releaseCountBeforeSettlement >= 1) { subscriptions[0].unsubscribe() firstUnsubscribed = true - expect(transportOptions?.signal?.aborted).toBe(false) + expect(transportOptions[0]?.signal?.aborted).toBe(true) + expect(transportOptions[1]?.signal?.aborted).toBe(false) } if (scenario.releaseCountBeforeSettlement === 2) { subscriptions[1].unsubscribe() secondUnsubscribed = true - expect(transportOptions?.signal?.aborted).toBe(true) + expect(transportOptions[1]?.signal?.aborted).toBe(true) } const failure = new Error(`shared transport failed`) if (scenario.outcome === `resolve`) { - if (!transportOptions?.signal?.aborted) { + if (transportOptions.some(({ signal }) => !signal?.aborted)) { begin() write({ type: `insert`, value: { id: `one`, value: 1 } }) commit() } - transport.resolve() + for (const transport of transports) transport.resolve() } else { - transport.reject( - transportOptions?.signal?.aborted - ? new DOMException(`obsolete`, `AbortError`) - : failure, + transports.forEach((transport, index) => + transport.reject( + transportOptions[index]?.signal?.aborted + ? new DOMException(`obsolete`, `AbortError`) + : failure, + ), ) } await flushPromises() @@ -1167,7 +1284,7 @@ async function runSharedSubscriptionScenario( expect(unloads).toHaveLength(2) expect(new Set(unloads).size).toBe(2) } finally { - transport.resolve() + for (const transport of transports) transport.resolve() await flushPromises() if (!firstUnsubscribed) subscriptions[0].unsubscribe() if (!secondUnsubscribed) subscriptions[1].unsubscribe() @@ -1324,10 +1441,166 @@ async function runOptimisticReplayScenario( } } -const { multiplier, replaySeed } = readOracleRunConfig() +const { multiplier, ...replay } = readOracleRunConfig() const generatedRuns = 30 * multiplier describe(`CollectionSubscription replay oracle`, () => { + it.each([`resolve`, `reject`] as const)( + `starts a replacement that lets canceled replay %s`, + async (outcome) => { + const oldReplay = createDeferred() + const newReplay = createDeferred() + const aborted = new DOMException(`superseded`, `AbortError`) + const loads: Array = [] + const unloads: Array = [] + const events: Array = [] + let operations!: Parameters[`sync`]>[0] + const collection = createCollection({ + id: `replacement-start-dependency-${outcome}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (sync) => { + operations = sync + sync.markReady() + return { + loadSubset: (options) => { + loads.push(options) + events.push(`load:${loads.length}`) + if (loads.length === 1) { + sync.begin() + sync.write({ type: `insert`, value: { id: `one`, value: 0 } }) + sync.commit() + return true + } + if (loads.length === 2) return oldReplay.promise + // This provider has stopped old request-scoped writes on abort. + // Its shared refresh protocol completes the old waiter only + // when a replacement acquisition registers. Completion does + // not require new result publication or a callback from core. + expect(loads[1]?.signal?.aborted).toBe(true) + if (outcome === `resolve`) oldReplay.resolve() + else oldReplay.reject(aborted) + events.push(`old:settled`) + return newReplay.promise + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else { + const { id, value } = change.value + visible.set(change.key, { id, value }) + } + } + }) + const truncate = () => { + operations.begin() + operations.truncate() + operations.commit() + } + try { + subscription.requestSnapshot({ optimizedOnly: false }) + expect([...visible.values()]).toEqual([{ id: `one`, value: 0 }]) + truncate() + const completion = subscription.pendingTruncateReplacement + expect(completion).toBeDefined() + let completed = false + const completionErrors: Array = [] + void completion?.then( + () => { + completed = true + }, + (error: unknown) => completionErrors.push(error), + ) + await flushPromises() + expect(loads).toHaveLength(2) + expect(oldReplay.isPending()).toBe(true) + expect([...visible.values()]).toEqual([{ id: `one`, value: 0 }]) + + truncate() + await flushPromises() + expect(events).toEqual([`load:1`, `load:2`, `load:3`, `old:settled`]) + expect(oldReplay.isPending()).toBe(false) + expect([...visible.values()]).toEqual([{ id: `one`, value: 0 }]) + expect(subscription.status).toBe(`loadingSubset`) + expect(subscription.lastError).toBeUndefined() + expect(completed).toBe(false) + expect(completionErrors).toEqual([]) + + operations.begin() + operations.write({ type: `insert`, value: { id: `one`, value: 2 } }) + await operations.commit() + newReplay.resolve() + await flushPromises() + expect([...visible.values()]).toEqual([{ id: `one`, value: 2 }]) + expect(subscription.status).toBe(`ready`) + expect(subscription.lastError).toBeUndefined() + expect(completed).toBe(true) + expect(completionErrors).toEqual([]) + } finally { + oldReplay.resolve() + newReplay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(loads.length) + for (const load of loads) { + expect(unloads.filter((unload) => unload === load)).toHaveLength(1) + } + }, + ) + + it(`generates shared, failed, stale, released, and post-replay histories`, () => { + const scenarios = fc.sample(replayScenarioArbitrary, { + seed: 1755, + numRuns: 300, + }) + + expect(scenarios.some(({ demandIds }) => demandIds.length > 1)).toBe(true) + expect(scenarios.some(({ attempts }) => attempts.length > 1)).toBe(true) + expect( + scenarios.some(({ attempts }) => + attempts.some(({ loads }) => + loads.some(({ outcome }) => outcome === `reject`), + ), + ), + ).toBe(true) + expect( + scenarios.some(({ attempts }) => + attempts.some(({ loads }) => + loads.some(({ writeBeforeSettlement }) => writeBeforeSettlement), + ), + ), + ).toBe(true) + expect( + scenarios.some(({ settlementOrder }) => + settlementOrder.some((value, index) => value !== index), + ), + ).toBe(true) + expect( + scenarios.some(({ releaseOnLastAttempt }) => + Boolean(releaseOnLastAttempt), + ), + ).toBe(true) + expect( + scenarios.some(({ afterSettlement }) => afterSettlement.length > 0), + ).toBe(true) + expect( + scenarios.some(({ afterSettlement }) => + afterSettlement.some(({ type }) => type === `request`), + ), + ).toBe(true) + }) + it(`aborts an in-flight initial acquisition before its replay replaces it`, async () => { let begin!: () => void let write!: ( @@ -1404,7 +1677,7 @@ describe(`CollectionSubscription replay oracle`, () => { } }) - it(`uses the published replacement as the baseline of a reentrant replay`, async () => { + it(`keeps the published replacement after a reentrant replay fails`, async () => { let begin!: () => void let write!: ( message: ChangeMessageOrDeleteKeyMessage, @@ -1483,7 +1756,7 @@ describe(`CollectionSubscription replay oracle`, () => { write({ type: `insert`, value: { id: `one`, value: 1 } }) commit() - expect(sortedRows(visible)).toEqual([{ id: `one`, value: 1 }]) + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) } finally { subscription.unsubscribe() await collection.cleanup() @@ -1559,7 +1832,6 @@ describe(`CollectionSubscription replay oracle`, () => { subscription.releaseSnapshot(demandOne) expect(replays[0]?.options.signal?.aborted).toBe(true) - replays[0]?.deferred.reject(new DOMException(`obsolete`, `AbortError`)) begin() write({ type: `insert`, value: { id: `two`, value: 2 } }) commit() @@ -1567,6 +1839,7 @@ describe(`CollectionSubscription replay oracle`, () => { await flushPromises() expect(sortedRows(visible)).toEqual([{ id: `two`, value: 2 }]) + expect(subscription.status).toBe(`ready`) expect(subscription.lastError).toBeUndefined() } finally { subscription.unsubscribe() @@ -1733,6 +2006,7 @@ describe(`CollectionSubscription replay oracle`, () => { succeeds ? [...expectedIds].sort() : [], ) + const batchCount = batches.length subscription.requestLimitedSnapshot({ orderBy, limit: 1, @@ -1744,7 +2018,8 @@ describe(`CollectionSubscription replay oracle`, () => { lastKey: succeeds ? expectedIds[1] : initialIds[1], }, }) - expect(batches.at(-1)).toEqual([]) + if (succeeds) expect(batches.at(-1)).toEqual([]) + else expect(batches).toHaveLength(batchCount) } finally { subscription.unsubscribe() await collection.cleanup() @@ -1752,7 +2027,107 @@ describe(`CollectionSubscription replay oracle`, () => { }, ) - it(`publishes a same-key replacement after a failed replay`, async () => { + it(`keeps private row tracking through consecutive failed replays`, async () => { + await runReplayScenario({ + initialRows: [ + { id: `one`, value: -2 }, + { id: `two`, value: 0 }, + ], + demandIds: [`two`], + attempts: [ + { + loads: [ + { + demandId: `two`, + rows: [], + outcome: `reject`, + writeBeforeSettlement: true, + }, + ], + }, + { + loads: [ + { + demandId: `two`, + rows: [], + outcome: `reject`, + writeBeforeSettlement: true, + }, + ], + }, + { + loads: [ + { + demandId: `two`, + rows: [{ id: `two`, value: -2 }], + outcome: `resolve`, + writeBeforeSettlement: true, + }, + ], + }, + ], + settlementOrder: [0, 2, 1], + settlementPhases: [0, 1, 2], + afterSettlement: [], + }) + }) + + it(`retains a successful retry after retiring its failed peer`, async () => { + await runReplayScenario({ + initialRows: [{ id: `two`, value: 0 }], + demandIds: [`one`, `two`], + attempts: [ + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 1 }], + outcome: `reject`, + writeBeforeSettlement: false, + }, + { + demandId: `two`, + rows: [{ id: `two`, value: -1 }], + outcome: `reject`, + writeBeforeSettlement: false, + }, + ], + }, + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 2 }], + outcome: `reject`, + writeBeforeSettlement: true, + }, + { + demandId: `two`, + rows: [{ id: `two`, value: 1 }], + outcome: `resolve`, + writeBeforeSettlement: false, + }, + ], + }, + ], + settlementOrder: [3, 1, 0, 2], + settlementPhases: [0, 0, 1, 1], + releaseOnLastAttempt: `one`, + afterSettlement: [{ type: `request`, demandId: `one` }], + }) + }) + + it(`does not republish an identical snapshot after a synchronous replay failure`, async () => { + await runSequentialReplayScenario({ + initialRows: [{ id: `one`, value: 0 }], + loads: [ + { rows: [], outcome: `throw` }, + { rows: [{ id: `one`, value: 0 }], outcome: `return` }, + ], + }) + }) + + it(`keeps a same-key source replacement private after a failed replay`, async () => { await runReplayScenario({ initialRows: [{ id: `one`, value: 1 }], demandIds: [`one`], @@ -1907,7 +2282,7 @@ describe(`CollectionSubscription replay oracle`, () => { }) }) - it(`excludes rows written before their replay demand is released`, async () => { + it(`retains applied rows after their replay demand is released`, async () => { await runReplayScenario({ initialRows: [{ id: `two`, value: 0 }], demandIds: [`one`, `two`], @@ -1935,6 +2310,30 @@ describe(`CollectionSubscription replay oracle`, () => { }) }) + it(`refreshes a retained row when its final released demand is reacquired`, async () => { + // Reduced from the fixed replay corpus after removing release-time pruning. + await runReplayScenario({ + initialRows: [{ id: `one`, value: 0 }], + demandIds: [`one`], + attempts: [ + { + loads: [ + { + demandId: `one`, + rows: [{ id: `one`, value: 1 }], + outcome: `resolve`, + writeBeforeSettlement: true, + }, + ], + }, + ], + settlementOrder: [0], + settlementPhases: [0], + releaseOnLastAttempt: `one`, + afterSettlement: [{ type: `request`, demandId: `one` }], + }) + }) + it(`replaces a retained snapshot with a later empty replay`, async () => { await runReplayScenario({ initialRows: [{ id: `one`, value: 1 }], @@ -1953,26 +2352,2145 @@ describe(`CollectionSubscription replay oracle`, () => { }) }) - fcTest.prop([replayScenarioArbitrary], { - numRuns: generatedRuns, - seed: 1756, - })(`matches replay and ownership laws for a fixed seed`, runReplayScenario) - - fcTest.prop( - [replayScenarioArbitrary], - oracleRandomParameters(generatedRuns, replaySeed), - )( - `matches replay and ownership laws for a random or replayed seed`, - runReplayScenario, - ) - - fcTest.prop( - [sequentialReplayScenarioArbitrary], - oracleRandomParameters(generatedRuns, replaySeed), - )( - `matches synchronous, asynchronous, and partial-failure replay laws`, - runSequentialReplayScenario, - ) + it.each([ + ...[`current`, `pending`].flatMap((scope) => + [`throw`, `reject`].map((failureMode) => ({ scope, failureMode })), + ), + // An async rejection runs after setup; only a sync failure can be held + // by an attempt whose setup stack has not returned yet. + { scope: `setup`, failureMode: `throw` }, + ])( + `drops released failure references: $scope, $failureMode`, + async ({ scope, failureMode }) => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const pendingPeer = createDeferred() + const replacement = createDeferred() + const failure = new Error(`failed owner`) + let loads = 0 + const collection = createCollection({ + id: `released-replay-failure-${scope}-${failureMode}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loads++ + if (loads <= 2) return true + if (loads === 3) { + if (failureMode === `throw`) throw failure + return Promise.reject(failure) + } + if (scope === `setup` && loads === 4) { + expect(retainedFailures()).toEqual([failure]) + subscription.requestSnapshot({ + where: peerWhere, + optimizedOnly: false, + }) + return pendingPeer.promise + } + if (scope === `setup` && loads === 5) { + begin() + truncate() + commit() + subscription.releaseSnapshot(failedWhere) + } + return loads === 4 ? pendingPeer.promise : replacement.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const failedWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`one`), + ]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const retainedFailures = () => { + // Narrow retention witness for old and new representations. Follow + // stored replay frames, not a captured map that the source discarded. + type Frame = { failures?: Map } + const session = ( + subscription as unknown as { + truncateReplaySession: Frame & { + currentAttempt: Frame + attempts?: Set + pending?: Set<{ attempt: Frame }> + } + } + ).truncateReplaySession + const frames = new Set([ + session, + session.currentAttempt, + ...(session.attempts ?? []), + ...[...(session.pending ?? [])].map(({ attempt }) => attempt), + ]) + return [...frames].flatMap((frame) => [ + ...(frame.failures?.values() ?? []), + ]) + } + const replaySource = async () => { + begin() + truncate() + commit() + await flushPromises() + } + + try { + subscription.requestSnapshot({ + where: failedWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ where: peerWhere, optimizedOnly: false }) + await replaySource() + // This is a retained-state witness, not a row oracle or GC benchmark. + // Public rows cannot reveal a released owner held by an old error map. + // Adapt this witness if the replay representation changes again. + if (scope !== `setup`) expect(retainedFailures()).toEqual([failure]) + if (scope === `pending`) await replaySource() + subscription.releaseSnapshot(failedWhere) + expect(retainedFailures()).toEqual([]) + expect(subscription.status).toBe(`loadingSubset`) + replacement.resolve() + pendingPeer.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + } finally { + replacement.resolve() + pendingPeer.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`does not start a queued replay after a newer truncate supersedes it`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const loadSignals: Array = [] + const collection = createCollection({ + id: `superseded-before-replay-setup`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: ({ signal }) => { + loadSignals.push(signal) + return loadSignals.length === 1 ? true : replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + + begin() + truncate() + commit() + begin() + truncate() + commit() + await flushPromises() + + expect(loadSignals).toHaveLength(2) + expect(loadSignals[0]?.aborted).toBe(true) + expect(loadSignals[1]?.aborted).toBe(false) + } finally { + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it.each([ + `replay`, + `additional demand`, + `additional pending demand`, + ] as const)( + `aborts a %s acquisition before a reentrant newer truncate starts`, + async (start) => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const olderReplay = createDeferred() + const newerReplay = createDeferred() + const predecessor = createDeferred() + const replaySignals: Array = [] + let loadCount = 0 + const collection = createCollection({ + id: `reentrant-newer-replay`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: ({ signal }) => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 0 } }) + commit() + return true + } + + if (loadCount === 2 && start !== `replay`) { + if (start === `additional pending demand`) { + return predecessor.promise + } + // A failed replay retains its public baseline after setup and + // all participants finish. Start the extra demand in that gap. + throw new Error(`retain the failed replay`) + } + replaySignals.push(signal) + if (loadCount === (start === `replay` ? 2 : 3)) { + begin() + truncate() + commit() + return olderReplay.promise + } + return newerReplay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + }) + + const install = (value: number) => { + begin() + write({ + type: collection.has(`one`) ? `update` : `insert`, + value: { id: `one`, value }, + }) + commit() + } + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 0 }]) + + begin() + truncate() + commit() + await flushPromises() + + if (start !== `replay`) { + if (start === `additional demand`) { + expect(subscription.lastError).toEqual( + new Error(`retain the failed replay`), + ) + } + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 0 }]) + subscription.requestSnapshot({ + where: new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]), + optimizedOnly: false, + }) + await flushPromises() + } + + expect(replaySignals).toHaveLength(start === `replay` ? 2 : 3) + expect(replaySignals[0]?.aborted).toBe(true) + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 0 }]) + + install(2) + newerReplay.resolve() + await flushPromises() + if (start === `additional pending demand`) { + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 0 }]) + predecessor.resolve() + await flushPromises() + // The returning extra demand joined the retained old attempt. Its + // transport still holds publication after that attempt's prior work. + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 0 }]) + expect(subscription.status).toBe(`loadingSubset`) + } else { + // A startup superseded before return does not hold publication. An + // ordinary demand still owns its separate readiness participant. + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + expect(subscription.status).toBe( + start === `replay` ? `ready` : `loadingSubset`, + ) + } + if (!replaySignals[0]?.aborted) install(1) + olderReplay.resolve() + await flushPromises() + + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + expect(subscription.status).toBe(`ready`) + } finally { + predecessor.resolve() + olderReplay.resolve() + newerReplay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`does not retain replay work registered after its demand is released`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const releasedReplay = createDeferred() + let loadCount = 0 + const collection = createCollection({ + id: `released-during-replay-start`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + write({ type: `insert`, value: { id: `two`, value: 1 } }) + commit() + operations.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount <= 2) return true + if (loadCount === 3) { + subscription.releaseSnapshot(firstWhere) + return releasedReplay.promise + } + begin() + write({ type: `insert`, value: { id: `two`, value: 2 } }) + commit() + return Promise.resolve() + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + }) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + + begin() + truncate() + commit() + await flushPromises() + + expect(subscription.status).toBe(`ready`) + expect(sortedRows(visible)).toEqual([{ id: `two`, value: 2 }]) + } finally { + releasedReplay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`finishes replay state and surfaces an async subscriber failure`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const listenerFailure = new Error(`replay subscriber failed`) + const queuedMicrotasks: Array = [] + let loadCount = 0 + let rejectReplacement = false + const collection = createCollection({ + id: `async-replay-subscriber-failure`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + return true + } + return replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + if (rejectReplacement) throw listenerFailure + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + + begin() + write({ type: `insert`, value: { id: `one`, value: 2 } }) + commit() + rejectReplacement = true + const queueMicrotaskSpy = vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => queuedMicrotasks.push(callback)) + try { + replay.resolve() + await flushPromises() + + expect(subscription.status).toBe(`ready`) + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + expect(queuedMicrotasks).toHaveLength(1) + expect(() => queuedMicrotasks[0]!()).toThrow(listenerFailure) + } finally { + queueMicrotaskSpy.mockRestore() + } + } finally { + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not start another demand for a replay superseded by reentrancy`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + let loadCount = 0 + let inSupersededSetup = false + let staleSecondDemandStarted = false + const collection = createCollection({ + id: `reentrant-multi-demand-replay`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + begin() + write({ type: `insert`, value: { id: `one`, value: 0 } }) + write({ type: `insert`, value: { id: `two`, value: 0 } }) + commit() + operations.markReady() + return { + loadSubset: (options) => { + loadCount++ + if (loadCount <= 2) return true + if (loadCount === 3) { + inSupersededSetup = true + queueMicrotask(() => { + inSupersededSetup = false + }) + begin() + truncate() + commit() + return true + } + if (inSupersededSetup) { + staleSecondDemandStarted = true + begin() + write({ type: `insert`, value: { id: `two`, value: 1 } }) + commit() + return true + } + if (options.where === firstWhere) { + begin() + write({ type: `insert`, value: { id: `one`, value: 2 } }) + commit() + } + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + }) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + begin() + truncate() + commit() + await flushPromises() + + expect(staleSecondDemandStarted).toBe(false) + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not start replay work retired by the loading transition`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const loads: Array = [] + const unloads: Array = [] + const collection = createCollection({ + id: `reentrant-status-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return loads.length === 1 ? true : replay.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + let releaseOnLoading = false + subscription.on(`status:loadingSubset`, () => { + if (releaseOnLoading) subscription.releaseSnapshot(where) + }) + + try { + subscription.requestSnapshot({ where, optimizedOnly: false }) + releaseOnLoading = true + begin() + truncate() + commit() + await flushPromises() + + expect(loads).toHaveLength(1) + expect(unloads).toEqual([loads[0]]) + expect(subscription.status).toBe(`ready`) + } finally { + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`releases replay work when its final publication callback throws`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const loads: Array = [] + const unloads: Array = [] + const listenerFailure = new Error(`release publication failed`) + let rejectReplacement = false + const collection = createCollection({ + id: `replay-release-callback-cleanup`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (loads.length === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + return true + } + return replay.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + truncateReplayPublication: { + start: () => {}, + succeed: () => { + if (rejectReplacement) throw listenerFailure + }, + }, + }) + + try { + subscription.requestSnapshot({ where, optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + begin() + write({ type: `insert`, value: { id: `one`, value: 2 } }) + commit() + rejectReplacement = true + + expect(() => subscription.releaseSnapshot(where)).toThrow(listenerFailure) + expect(loads[1]?.signal?.aborted).toBe(true) + expect(unloads.map((options) => loads.indexOf(options))).toEqual([0, 1]) + expect(subscription.status).toBe(`ready`) + } finally { + rejectReplacement = false + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not retain a demand released during adapter startup`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const releasedLoad = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const unloads: Array = [] + const collection = createCollection({ + id: `reentrant-new-demand-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (loads.length === 1) return true + if (loads.length === 2) return replay.promise + subscription.releaseSnapshot(secondWhere) + return releasedLoad.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + begin() + truncate() + commit() + await flushPromises() + + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + replay.resolve() + await flushPromises() + + expect(loads).toHaveLength(3) + expect(loads[2]?.signal?.aborted).toBe(true) + expect(unloads).toContain(loads[2]) + expect(subscription.status).toBe(`ready`) + expect(subscription.pendingTruncateReplacement).toBeUndefined() + } finally { + releasedLoad.resolve() + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`publishes a replay replacement before reporting ready`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + let loadCount = 0 + const collection = createCollection({ + id: `replay-ready-after-publication`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + return true + } + return replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const readyValues: Array = [] + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + }) + subscription.on(`status:ready`, () => { + readyValues.push(visible.get(`one`)?.value) + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + begin() + write({ type: `insert`, value: { id: `one`, value: 2 } }) + commit() + + replay.resolve() + await flushPromises() + + expect(readyValues).toEqual([2]) + expect(visible.get(`one`)?.value).toBe(2) + expect(subscription.status).toBe(`ready`) + } finally { + replay.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`ignores a synchronous replay failure after its demand releases itself`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const releasedFailure = new Error(`released replay load failed`) + const reportedErrors: Array = [] + let loadCount = 0 + const collection = createCollection({ + id: `released-sync-failure`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + write({ type: `insert`, value: { id: `two`, value: 1 } }) + commit() + operations.markReady() + return { + loadSubset: () => { + loadCount++ + if (loadCount <= 2) return true + if (loadCount === 3) { + subscription.releaseSnapshot(firstWhere) + throw releasedFailure + } + begin() + write({ type: `insert`, value: { id: `two`, value: 2 } }) + commit() + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const visible = new Map() + const subscription = collection.subscribeChanges((changes) => { + recordPublishedChanges(visible, changes as Array) + }) + subscription.on(`loadSubset:error`, ({ error }) => { + reportedErrors.push(error) + }) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + begin() + truncate() + commit() + await flushPromises() + + expect(reportedErrors).not.toContain(releasedFailure) + expect(sortedRows(visible)).toEqual([{ id: `two`, value: 2 }]) + expect(subscription.status).toBe(`ready`) + expect(subscription.pendingTruncateReplacement).toBeUndefined() + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it.each( + [false, true].flatMap((releaseDemand) => + [false, true].map((failRelease) => ({ releaseDemand, failRelease })), + ), + )( + `releases before reacquisition with releaseDemand=$releaseDemand and failRelease=$failRelease`, + async ({ releaseDemand, failRelease }) => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const loads: Array = [] + const unloads: Array = [] + let reentered = false + const releaseFailure = new Error(`old replay lease release failed`) + const collection = createCollection({ + id: `reentrant-replay-lease-replacement`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[0] && !reentered) { + reentered = true + if (releaseDemand) subscription.releaseSnapshot(where) + if (failRelease) throw releaseFailure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + try { + subscription.requestSnapshot({ where, optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + + const reacquires = !releaseDemand && !failRelease + expect(loads).toHaveLength(reacquires ? 2 : 1) + // indexOf checks the exact options object, not a structurally equal copy. + expect(unloads.map((options) => loads.indexOf(options))).toEqual([0]) + if (reacquires) expect(loads[1]!.signal?.aborted).toBe(false) + if (failRelease) expect(subscription.lastError).toBe(releaseFailure) + subscription.unsubscribe() + // A failed release or retired logical demand never starts a replacement. + expect(unloads.map((options) => loads.indexOf(options))).toEqual( + reacquires ? [0, 1] : [0], + ) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`rejects replay completion with the exact reported adapter error`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replayLoad = createDeferred() + const failure = new Error(`exact replay failure`) + const reportedErrors: Array = [] + let loadCount = 0 + const collection = createCollection({ + id: `exact-replay-completion-error`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => (++loadCount === 1 ? true : replayLoad.promise), + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { + start: () => {}, + succeed: () => {}, + }, + }) + subscription.on(`loadSubset:error`, ({ error }) => { + reportedErrors.push(error) + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + begin() + truncate() + commit() + await flushPromises() + const replacement = subscription.pendingTruncateReplacement + expect(replacement).toBeInstanceOf(Promise) + + replayLoad.reject(failure) + + await expect(replacement).rejects.toBe(failure) + expect(subscription.lastError).toBe(failure) + expect(reportedErrors).toEqual([failure]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`normalizes one primitive rejection for every observer of a shared replay`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replayLoad = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const reportedErrors: Array<{ + options: LoadSubsetOptions + error: unknown + }> = [] + let loadCount = 0 + const collection = createCollection({ + id: `shared-primitive-replay-error`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + loadCount++ + return loadCount <= 2 ? true : replayLoad.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { + start: () => {}, + succeed: () => {}, + }, + }) + subscription.on(`loadSubset:error`, ({ options, error }) => { + reportedErrors.push({ options, error }) + }) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + begin() + truncate() + commit() + await flushPromises() + const replacement = subscription.pendingTruncateReplacement + expect(replacement).toBeInstanceOf(Promise) + expect(loads.map(({ where }) => where)).toEqual([ + firstWhere, + secondWhere, + firstWhere, + secondWhere, + ]) + + replayLoad.reject(undefined) + + let replacementError: unknown + try { + await replacement + } catch (error) { + replacementError = error + } + expect(reportedErrors).toHaveLength(2) + expect(reportedErrors.map(({ options }) => options)).toEqual([ + loads[2], + loads[3], + ]) + expect(reportedErrors[0]?.error).toBeInstanceOf(Error) + expect(reportedErrors[1]?.error).toBe(reportedErrors[0]?.error) + expect(subscription.lastError).toBe(reportedErrors[0]?.error) + expect(replacementError).toBe(reportedErrors[0]?.error) + } finally { + replayLoad.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it.each([`none`, `first`, `second`, `both`] as const)( + `normalizes one primitive rejection for ordinary shared loads after releasing %s demand`, + async (released) => { + const sharedLoad = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`two`), + ]) + const loads: Array = [] + const unloads: Array = [] + const reportedErrors: Array<{ + options: LoadSubsetOptions + error: unknown + }> = [] + const collection = createCollection({ + id: `shared-primitive-ordinary-error-${released}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return sharedLoad.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ options, error }) => { + reportedErrors.push({ options, error }) + }) + + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + expect(loads.map(({ where }) => where)).toEqual([ + firstWhere, + secondWhere, + ]) + + if (released === `first` || released === `both`) { + subscription.releaseSnapshot(firstWhere) + } + if (released === `second` || released === `both`) { + subscription.releaseSnapshot(secondWhere) + } + expect(loads[0]?.signal?.aborted).toBe( + released === `first` || released === `both`, + ) + expect(loads[1]?.signal?.aborted).toBe( + released === `second` || released === `both`, + ) + + sharedLoad.reject(undefined) + await flushPromises() + + const activeLoads = loads.filter((load) => !load.signal?.aborted) + expect(reportedErrors.map(({ options }) => options)).toEqual( + activeLoads, + ) + if (activeLoads.length > 0) { + expect(reportedErrors[0]?.error).toBeInstanceOf(Error) + for (const { error } of reportedErrors) { + expect(error).toBe(reportedErrors[0]?.error) + } + expect(subscription.lastError).toBe(reportedErrors[0]?.error) + } else { + expect(subscription.lastError).toBeUndefined() + } + expect(subscription.status).toBe(`ready`) + } finally { + sharedLoad.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(loads.length) + for (const load of loads) { + expect(unloads.filter((options) => options === load)).toHaveLength(1) + } + }, + ) + + it.each([`first`, `second`, `both`] as const)( + `settles a shared replay rejection after releasing %s demand`, + async (released) => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const replayLoad = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`two`), + ]) + const loads: Array = [] + const unloads: Array = [] + const reportedErrors: Array<{ + options: LoadSubsetOptions + error: unknown + }> = [] + let loadCount = 0 + let replayStarts = 0 + let replaySuccesses = 0 + const collection = createCollection({ + id: `released-shared-primitive-replay-error-${released}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + loadCount++ + return loadCount <= 2 ? true : replayLoad.promise + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { + start: () => replayStarts++, + succeed: () => replaySuccesses++, + }, + }) + subscription.on(`loadSubset:error`, ({ options, error }) => { + reportedErrors.push({ options, error }) + }) + + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + begin() + truncate() + commit() + await flushPromises() + const replacement = subscription.pendingTruncateReplacement + expect(replacement).toBeInstanceOf(Promise) + expect(loads.map(({ where }) => where)).toEqual([ + firstWhere, + secondWhere, + firstWhere, + secondWhere, + ]) + const settlement = replacement!.then( + () => ({ status: `resolved` as const }), + (error: unknown) => ({ status: `rejected` as const, error }), + ) + expect({ replayStarts, replaySuccesses }).toEqual({ + replayStarts: 1, + replaySuccesses: 0, + }) + + if (released === `first` || released === `both`) { + subscription.releaseSnapshot(firstWhere) + } + if (released === `second` || released === `both`) { + subscription.releaseSnapshot(secondWhere) + } + expect(loads[2]?.signal?.aborted).toBe( + released === `first` || released === `both`, + ) + expect(loads[3]?.signal?.aborted).toBe( + released === `second` || released === `both`, + ) + + if (released === `both`) { + const result = await settlement + expect(result).toMatchObject({ + status: `rejected`, + error: { name: `AbortError` }, + }) + expect(reportedErrors).toEqual([]) + expect(subscription.lastError).toBeUndefined() + expect(subscription.status).toBe(`ready`) + expect(replaySuccesses).toBe(1) + } else { + expect(subscription.pendingTruncateReplacement).toBe(replacement) + expect(subscription.status).toBe(`loadingSubset`) + replayLoad.reject(undefined) + const result = await settlement + expect(result.status).toBe(`rejected`) + + const activeIndex = released === `first` ? 3 : 2 + expect(reportedErrors).toHaveLength(1) + expect(reportedErrors[0]?.options).toBe(loads[activeIndex]) + expect(reportedErrors[0]?.error).toBeInstanceOf(Error) + expect(subscription.lastError).toBe(reportedErrors[0]?.error) + expect(result).toMatchObject({ + status: `rejected`, + error: reportedErrors[0]?.error, + }) + } + } finally { + replayLoad.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(loads.length) + for (const load of loads) { + expect(unloads.filter((options) => options === load)).toHaveLength(1) + } + expect(replayStarts).toBe(1) + expect(replaySuccesses).toBe(released === `both` ? 1 : 0) + }, + ) + + it(`ignores a retired demand's replay failure once surviving demand succeeds`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const firstReplay = createDeferred() + const secondReplay = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const failure = new Error(`retired demand failed`) + let replaySuccesses = 0 + const collection = createCollection({ + id: `retired-replay-failure`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (loads.length <= 2) return true + return loads.length === 3 + ? firstReplay.promise + : secondReplay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { + start: () => {}, + succeed: () => replaySuccesses++, + }, + }) + + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + begin() + truncate() + commit() + await flushPromises() + const replacement = subscription.pendingTruncateReplacement + expect(replacement).toBeInstanceOf(Promise) + + firstReplay.reject(failure) + await flushPromises() + subscription.releaseSnapshot(firstWhere) + secondReplay.resolve() + + await expect(replacement).resolves.toBeUndefined() + expect(replaySuccesses).toBe(1) + expect(subscription.status).toBe(`ready`) + } finally { + firstReplay.resolve() + secondReplay.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`retains successful peer rows after failed replay demand retires`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const failed = createDeferred() + const successful = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const unloads: Array = [] + const visible = new Map() + const batches: Array> = [] + let survivingRows: Array = [] + const collection = createCollection({ + id: `settled-replay-peer`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + const id = options.where === firstWhere ? `one` : `two` + begin() + write({ + type: `insert`, + value: { id, value: loads.length <= 2 ? 1 : 2 }, + }) + commit() + if (loads.length <= 2) return true + return id === `one` ? failed.promise : successful.promise + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges((changes) => { + batches.push(recordPublishedChanges(visible, changes)) + }) + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + expect(sortedRows(visible)).toEqual([ + { id: `one`, value: 1 }, + { id: `two`, value: 1 }, + ]) + begin() + truncate() + commit() + await flushPromises() + expect(loads).toHaveLength(4) + failed.reject(new Error(`first demand replay failed`)) + successful.resolve() + await flushPromises() + expect(sortedRows(visible)).toEqual([ + { id: `one`, value: 1 }, + { id: `two`, value: 1 }, + ]) + subscription.releaseSnapshot(firstWhere) + await flushPromises() + // Retirement leaves only the successful demand. Its replacement rows + // must survive failure handling for the now-retired peer. + survivingRows = sortedRows(visible) + subscription.releaseSnapshot(secondWhere) + // Outside replay, release ends acquisition ownership; this adapter does + // not evict its cached rows. The still-live subscriber observes deletion + // when the source actually removes the row. + expect(sortedRows(visible)).toEqual([ + { id: `one`, value: 2 }, + { id: `two`, value: 2 }, + ]) + begin() + write({ type: `delete`, key: `two` }) + commit() + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + } finally { + failed.resolve() + successful.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(4) + for (const load of loads) { + expect(unloads.filter((options) => options === load)).toHaveLength(1) + } + expect(survivingRows).toEqual([ + { id: `one`, value: 2 }, + { id: `two`, value: 2 }, + ]) + }) + + it(`keeps replay completion failure separate from a peer release failure`, async () => { + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const failed = createDeferred() + const peer = createDeferred() + const replayFailure = new Error(`replay failed`) + const releaseFailure = new Error(`peer release failed`) + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const peerWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const unloadAttempts: Array = [] + const unloaded: Array = [] + let failRelease = false + const succeeded = vi.fn() + const collection = createCollection({ + id: `replay-error-ownership`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (loads.length <= 2) return true + return options.where === firstWhere + ? failed.promise + : peer.promise + }, + unloadSubset: (options) => { + unloadAttempts.push(options) + if (failRelease && options === loads[3]) { + failRelease = false + throw releaseFailure + } + unloaded.push(options) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { start: () => {}, succeed: succeeded }, + }) + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: peerWhere }) + begin() + truncate() + commit() + await flushPromises() + expect(loads).toHaveLength(4) + const completion = subscription.pendingTruncateReplacement + expect(completion).toBeInstanceOf(Promise) + const observed = completion!.then( + () => ({ status: `resolved` as const }), + (error: unknown) => ({ status: `rejected` as const, error }), + ) + failed.reject(replayFailure) + await flushPromises() + failRelease = true + expect(() => subscription.releaseSnapshot(peerWhere)).toThrow( + releaseFailure, + ) + expect(succeeded).not.toHaveBeenCalled() + await expect(observed).resolves.toEqual({ + status: `rejected`, + error: replayFailure, + }) + const result = await observed + expect(`error` in result ? result.error : undefined).toBe(replayFailure) + peer.resolve() + await flushPromises() + await expect(observed).resolves.toEqual({ + status: `rejected`, + error: replayFailure, + }) + } finally { + failed.resolve() + peer.resolve() + failRelease = false + subscription.unsubscribe() + await collection.cleanup() + } + expect( + unloadAttempts.filter((options) => options === loads[3]), + ).toHaveLength(1) + expect(unloaded).toHaveLength(3) + for (const load of loads) { + expect(unloaded.filter((options) => options === load)).toHaveLength( + load === loads[3] ? 0 : 1, + ) + } + }) + + it(`waits for a new async demand acquired while unloading a replay lease`, async () => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const replay = createDeferred() + const nested = createDeferred() + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const nestedWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const loads: Array = [] + const unloads: Array = [] + const ready = vi.fn() + const visible = new Map() + const publications: Array> = [] + const collection = createCollection({ + id: `replay-new-demand-readiness`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + begin() + write({ + type: `insert`, + value: { + id: options.where === nestedWhere ? `two` : `one`, + value: loads.length === 1 ? 1 : 2, + }, + }) + commit() + if (loads.length === 1) return true + return options.where === nestedWhere + ? nested.promise + : replay.promise + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[0]) { + subscription.requestSnapshot({ where: nestedWhere }) + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + recordPublishedChanges(visible, changes) + publications.push(sortedRows(visible)) + }, + { includeInitialState: false }, + ) + try { + subscription.requestSnapshot({ where: firstWhere }) + publications.length = 0 + subscription.on(`status:ready`, ready) + begin() + truncate() + commit() + await flushPromises() + expect(loads.map(({ where }) => where)).toEqual([ + firstWhere, + nestedWhere, + firstWhere, + ]) + expect(unloads).toEqual([loads[0]]) + const completion = subscription.pendingTruncateReplacement + expect(completion).toBeInstanceOf(Promise) + const settled = vi.fn() + void completion!.then(settled, settled) + replay.resolve() + await flushPromises() + expect(subscription.status).not.toBe(`ready`) + expect(ready).not.toHaveBeenCalled() + expect(settled).not.toHaveBeenCalled() + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 1 }]) + expect(publications).toEqual([]) + nested.resolve() + await completion + await flushPromises() + expect(subscription.status).toBe(`ready`) + expect(ready).toHaveBeenCalledTimes(1) + expect(settled).toHaveBeenCalledTimes(1) + expect(sortedRows(visible)).toEqual([ + { id: `one`, value: 2 }, + { id: `two`, value: 2 }, + ]) + expect(publications).toEqual([ + [ + { id: `one`, value: 2 }, + { id: `two`, value: 2 }, + ], + ]) + } finally { + replay.resolve() + nested.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(3) + for (const load of loads) { + expect(unloads.filter((options) => options === load)).toHaveLength(1) + } + }) + + it.each([`after-release`, `during-ready`, `during-unload`] as const)( + `reacquires a final released replay demand %s without waiting for obsolete work`, + async (reacquireTiming) => { + let begin!: () => void + let write!: ( + message: ChangeMessageOrDeleteKeyMessage, + ) => void + let commit!: () => void + let truncate!: () => void + const replayLoad = createDeferred() + const reacquiredLoad = createDeferred() + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const loads: Array = [] + const unloads: Array = [] + let reacquireOnReady = false + let reacquireInUnload = false + const collection = createCollection({ + id: `final-replay-reacquire-${reacquireTiming}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (loads.length === 1) { + begin() + write({ type: `insert`, value: { id: `one`, value: 1 } }) + commit() + return true + } + if (loads.length === 2) { + begin() + write({ type: `insert`, value: { id: `one`, value: 2 } }) + commit() + return replayLoad.promise + } + return reacquireTiming === `during-unload` + ? reacquiredLoad.promise + : true + }, + unloadSubset: (options) => { + unloads.push(options) + if (reacquireInUnload && options === loads[1]) { + reacquireInUnload = false + subscription.requestSnapshot({ where }) + } + }, + } + }, + }, + }) + const visible = new Map() + const batches: Array> = [] + const subscription: CollectionSubscription = collection.subscribeChanges( + (changes) => { + batches.push(recordPublishedChanges(visible, changes)) + }, + ) + const readyRows: Array> = [] + subscription.on(`status:ready`, () => { + readyRows.push(sortedRows(visible)) + if (reacquireOnReady) { + reacquireOnReady = false + subscription.requestSnapshot({ where }) + } + }) + + try { + subscription.requestSnapshot({ where }) + begin() + truncate() + commit() + await flushPromises() + const replacement = subscription.pendingTruncateReplacement + expect(replacement).toBeInstanceOf(Promise) + const settlement = replacement!.then( + () => ({ status: `resolved` as const }), + (error: unknown) => ({ status: `rejected` as const, error }), + ) + + // Release has no synthetic delete callback. Reenter from its actual + // ready notification instead; the old replay is already retired then. + reacquireOnReady = reacquireTiming === `during-ready` + reacquireInUnload = reacquireTiming === `during-unload` + subscription.releaseSnapshot(where) + if (reacquireTiming === `after-release`) { + await expect(settlement).resolves.toMatchObject({ + status: `rejected`, + error: { name: `AbortError` }, + }) + subscription.requestSnapshot({ where }) + } else if (reacquireTiming === `during-unload`) { + let settled = false + void settlement.then(() => { + settled = true + }) + const batchesBeforePendingFlush = batches.length + await flushPromises() + expect(settled).toBe(false) + expect(subscription.status).not.toBe(`ready`) + expect(readyRows).toEqual([]) + expect(batches).toHaveLength(batchesBeforePendingFlush) + const batchesBeforeObsoleteSettlement = batches.length + replayLoad.resolve() + await flushPromises() + expect(settled).toBe(false) + expect(subscription.status).not.toBe(`ready`) + expect(readyRows).toEqual([]) + expect(batches).toHaveLength(batchesBeforeObsoleteSettlement) + reacquiredLoad.resolve() + await expect(settlement).resolves.toEqual({ status: `resolved` }) + } else { + expect(subscription.pendingTruncateReplacement).toBeUndefined() + await expect(settlement).resolves.toMatchObject({ + status: `rejected`, + error: { name: `AbortError` }, + }) + } + + expect(subscription.pendingTruncateReplacement).toBeUndefined() + expect(sortedRows(visible)).toEqual([{ id: `one`, value: 2 }]) + expect(sortedChanges(batches[0]!)).toEqual([ + { type: `insert`, key: `one`, value: { id: `one`, value: 1 } }, + ]) + expect(sortedChanges(batches.at(-1)!)).toEqual([ + { + type: `update`, + key: `one`, + value: { id: `one`, value: 2 }, + previousValue: { id: `one`, value: 1 }, + }, + ]) + expect(batches.filter((batch) => batch.length > 0)).toHaveLength(2) + expect(loads).toHaveLength(3) + expect(loads.map(({ where: requestWhere }) => requestWhere)).toEqual([ + where, + where, + where, + ]) + if (reacquireTiming === `during-unload`) { + expect(readyRows).toEqual([[{ id: `one`, value: 2 }]]) + } + } finally { + replayLoad.resolve() + reacquiredLoad.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + expect(unloads).toHaveLength(loads.length) + for (const load of loads) { + expect(unloads.filter((options) => options === load)).toHaveLength(1) + } + }, + ) + + it(`does not start replacement work after release unsubscribes`, () => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const loads: Array = [] + const unloads: Array = [] + const collection = createCollection({ + id: `reentrant-replacement-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + subscription.unsubscribe() + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + let release: (() => void) | undefined + subscription.requestSnapshot({ + where, + optimizedOnly: false, + onLoadSubsetResult: (_result, _options, releaseDemand) => { + release = releaseDemand + }, + }) + expect(release).toBeTypeOf(`function`) + release!() + expect( + subscription.requestSnapshot({ + where, + optimizedOnly: false, + }), + ).toBe(false) + + expect(loads).toHaveLength(1) + expect(unloads).toEqual([loads[0]]) + }) + + it.each([`generic`, `specific`] as const)( + `does not emit a stale specific status after reentrant release from a %s listener`, + async (reentryEvent) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const load = createDeferred() + const collection = createCollection({ + id: `reentrant-specific-status`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + operations.markReady() + return { + loadSubset: () => load.promise, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const observed: Array<{ event: string; current: string }> = [] + if (reentryEvent === `generic`) { + subscription.on(`status:change`, ({ status }) => { + if (status === `loadingSubset`) subscription.releaseSnapshot(where) + }) + } else { + subscription.on(`status:loadingSubset`, () => { + subscription.releaseSnapshot(where) + }) + } + subscription.on(`status:loadingSubset`, ({ status }) => { + observed.push({ event: status, current: subscription.status }) + }) + subscription.on(`status:ready`, ({ status }) => { + observed.push({ event: status, current: subscription.status }) + }) + + try { + subscription.requestSnapshot({ where, optimizedOnly: false }) + expect(observed).toEqual([{ event: `ready`, current: `ready` }]) + } finally { + load.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each([`generic`, `specific`] as const)( + `does not resume an obsolete status transition after %s-listener ABA reentry`, + async (reentryEvent) => { + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`two`), + ]) + const load = createDeferred() + const collection = createCollection({ + id: `reentrant-status-aba-${reentryEvent}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + operations.markReady() + return { + loadSubset: () => load.promise, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const trace: Array = [] + let reentered = false + const reenter = () => { + if (reentered) return + reentered = true + subscription.releaseSnapshot(firstWhere) + subscription.requestSnapshot({ where: secondWhere }) + } + if (reentryEvent === `generic`) { + subscription.on(`status:change`, ({ status }) => { + if (status === `loadingSubset`) reenter() + }) + } else { + subscription.on(`status:loadingSubset`, reenter) + } + subscription.on(`status:change`, ({ previousStatus, status }) => { + trace.push( + `generic:${previousStatus}->${status}:${subscription.status}`, + ) + }) + subscription.on(`status:loadingSubset`, () => { + trace.push(`specific:loadingSubset:${subscription.status}`) + }) + + try { + subscription.requestSnapshot({ where: firstWhere }) + + expect(trace).toEqual( + reentryEvent === `generic` + ? [ + `generic:loadingSubset->ready:ready`, + `generic:ready->loadingSubset:loadingSubset`, + `specific:loadingSubset:loadingSubset`, + ] + : [ + `generic:ready->loadingSubset:loadingSubset`, + `generic:loadingSubset->ready:ready`, + `generic:ready->loadingSubset:loadingSubset`, + `specific:loadingSubset:loadingSubset`, + ], + ) + } finally { + load.resolve() + await flushPromises() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each([`generic`, `specific`] as const)( + `stops %s status delivery when an earlier ready listener unsubscribes`, + async (reentryEvent) => { + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const load = createDeferred() + const collection = createCollection({ + id: `reentrant-status-unsubscribe-${reentryEvent}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + operations.markReady() + return { + loadSubset: () => load.promise, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const trace: Array = [] + const unsubscribe = () => { + trace.push(`unsubscribe`) + subscription.unsubscribe() + } + if (reentryEvent === `generic`) { + subscription.on(`status:change`, ({ status }) => { + if (status === `ready`) unsubscribe() + }) + } else { + subscription.on(`status:ready`, unsubscribe) + } + subscription.on(`status:change`, ({ status }) => { + if (status === `ready`) trace.push(`late-generic`) + }) + subscription.on(`status:ready`, () => trace.push(`late-specific`)) + subscription.on(`unsubscribed`, () => trace.push(`unsubscribed`)) + + try { + subscription.requestSnapshot({ where }) + load.resolve() + await flushPromises() + + expect(trace).toEqual( + reentryEvent === `generic` + ? [`unsubscribe`, `unsubscribed`] + : [`late-generic`, `unsubscribe`, `unsubscribed`], + ) + } finally { + load.resolve() + await collection.cleanup() + } + }, + ) + + fcTest.prop([replayScenarioArbitrary], { + numRuns: generatedRuns, + seed: 1756, + })(`matches replay and ownership laws for a fixed seed`, runReplayScenario) + + fcTest.prop( + [replayScenarioArbitrary], + oracleRandomParameters( + generatedRuns, + replay, + `subscription-replay.completion`, + ), + )( + `matches replay and ownership laws for a random or replayed seed`, + runReplayScenario, + ) + + fcTest.prop( + [sequentialReplayScenarioArbitrary], + oracleRandomParameters( + generatedRuns, + replay, + `subscription-replay.sequential`, + ), + )( + `matches synchronous, asynchronous, and partial-failure replay laws`, + runSequentialReplayScenario, + ) fcTest.prop([cleanupRestartScenarioArbitrary], { numRuns: generatedRuns, @@ -1984,25 +4502,46 @@ describe(`CollectionSubscription replay oracle`, () => { fcTest.prop( [cleanupRestartScenarioArbitrary], - oracleRandomParameters(generatedRuns, replaySeed), + oracleRandomParameters( + generatedRuns, + replay, + `subscription-replay.restart`, + ), )( `isolates cleanup and restart sessions for a random or replayed seed`, runCleanupRestartScenario, ) + fcTest.prop([fc.scheduler()], { numRuns: generatedRuns, seed: 1760 })( + `keeps same-tick obsolete and current replay settlements generation-safe`, + expectScheduledReplaySettlementIsGenerationSafe, + ) + + fcTest.prop( + [fc.scheduler()], + oracleRandomParameters( + generatedRuns, + replay, + `subscription-replay.same-tick`, + ), + )( + `keeps same-tick replay settlements generation-safe for a random or replayed seed`, + expectScheduledReplaySettlementIsGenerationSafe, + ) + fcTest.prop([sharedSubscriptionScenarioArbitrary], { numRuns: generatedRuns, seed: 1758, })( - `keeps shared transport and logical ownership distinct for a fixed seed`, + `keeps independent transport and logical ownership aligned for a fixed seed`, runSharedSubscriptionScenario, ) fcTest.prop( [sharedSubscriptionScenarioArbitrary], - oracleRandomParameters(generatedRuns, replaySeed), + oracleRandomParameters(generatedRuns, replay, `subscription-replay.shared`), )( - `keeps shared transport and logical ownership distinct for a random or replayed seed`, + `keeps independent transport and logical ownership aligned for a random or replayed seed`, runSharedSubscriptionScenario, ) @@ -2016,7 +4555,11 @@ describe(`CollectionSubscription replay oracle`, () => { fcTest.prop( [optimisticReplayScenarioArbitrary], - oracleRandomParameters(generatedRuns, replaySeed), + oracleRandomParameters( + generatedRuns, + replay, + `subscription-replay.optimistic`, + ), )( `preserves optimistic overlays across replay outcomes for a random or replayed seed`, runOptimisticReplayScenario, diff --git a/packages/db/tests/collection-subscription-retention.test.ts b/packages/db/tests/collection-subscription-retention.test.ts new file mode 100644 index 0000000000..b83c5fd67e --- /dev/null +++ b/packages/db/tests/collection-subscription-retention.test.ts @@ -0,0 +1,142 @@ +import { expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import { flushPromises } from './utils.js' +import type { LoadSubsetOptions } from '../src/types.js' + +const cases = ([`none`, `success`, `failure`] as const).flatMap((replay) => + [`a`, `c`].flatMap((group) => + [false, true].flatMap((overlap) => + [false, true].map((evict) => ({ replay, group, overlap, evict })), + ), + ), +) + +it.each(cases)( + `source retention controls release: replay=$replay group=$group overlap=$overlap evict=$evict`, + async ({ group, replay, overlap, evict }) => { + type Row = { id: number; group: string } + const loads: Array<{ + options: LoadSubsetOptions + deferred: ReturnType> + }> = [] + const unloads: Array = [] + const a = new Func(`eq`, [new PropRef([`group`]), new Value(`a`)]) + const b = overlap + ? new Func(`in`, [new PropRef([`group`]), new Value([`a`, `b`])]) + : new Func(`eq`, [new PropRef([`group`]), new Value(`b`)]) + const rows = [ + { id: 1, group }, + { id: 2, group: `b` }, + ] + let replaceRows!: (truncate: boolean) => void + let releaseTarget = false + const collection = createCollection({ + id: `source-retention`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync(operations) { + replaceRows = (truncate) => { + operations.begin() + if (truncate) operations.truncate() + for (const row of rows) + operations.write({ type: `insert`, value: row }) + operations.commit() + } + operations.markReady() + return { + loadSubset(options) { + const deferred = createDeferred() + void deferred.promise.catch(() => undefined) + options.signal?.addEventListener( + `abort`, + () => + deferred.reject(new DOMException(`Aborted`, `AbortError`)), + { once: true }, + ) + loads.push({ options, deferred }) + return deferred.promise + }, + unloadSubset(options) { + unloads.push(options) + if (releaseTarget && options.where === a && evict) { + // The source, not predicate membership, decides retention. + // This also covers a row unrelated to the released predicate. + operations.begin() + operations.write({ type: `delete`, key: 1 }) + operations.commit() + } + }, + } + }, + }, + }) + const visible = new Map() + let publications = 0 + const subscription = collection.subscribeChanges( + (changes) => { + publications++ + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else + visible.set(change.key, { + id: change.value.id, + group: change.value.group, + }) + } + }, + { includeInitialState: false }, + ) + try { + // These rows are independent source data, not request-scoped writes. + replaceRows(false) + subscription.requestSnapshot({ where: a }) + subscription.requestSnapshot({ where: b }) + loads.forEach(({ deferred }) => deferred.resolve()) + await flushPromises() + expect([...visible.values()]).toEqual(rows) + if (replay !== `none`) { + replaceRows(true) + await flushPromises() + expect(loads).toHaveLength(4) + } + const beforeRelease = publications + releaseTarget = true + subscription.releaseSnapshot(a) + releaseTarget = false + const released = loads[replay === `none` ? 0 : 2]! + expect(released.options.signal?.aborted).toBe(true) + expect( + unloads.filter((options) => options === released.options), + ).toHaveLength(1) + expect(collection.has(1)).toBe(!evict) + if (replay !== `none`) { + expect([...visible.values()]).toEqual(rows) + expect(publications).toBe(beforeRelease) + const failure = new Error(`peer replay failed`) + if (replay === `failure`) loads[3]!.deferred.reject(failure) + else loads[3]!.deferred.resolve() + await flushPromises() + expect(subscription.lastError).toBe( + replay === `failure` ? failure : undefined, + ) + } + expect([...visible.values()]).toEqual( + evict && replay !== `failure` ? [rows[1]] : rows, + ) + expect(publications - beforeRelease).toBe( + Number(evict && replay !== `failure`), + ) + expect(subscription.status).toBe(`ready`) + } finally { + releaseTarget = false + subscription.unsubscribe() + await collection.cleanup() + } + for (const { options } of loads) { + expect(unloads.filter((unloaded) => unloaded === options)).toHaveLength(1) + } + }, +) diff --git a/packages/db/tests/collection-subscription.test.ts b/packages/db/tests/collection-subscription.test.ts index dc4444b55f..3ea52df1ab 100644 --- a/packages/db/tests/collection-subscription.test.ts +++ b/packages/db/tests/collection-subscription.test.ts @@ -1,12 +1,161 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' +import { Temporal } from 'temporal-polyfill' import { createCollection } from '../src/collection/index.js' +import { CollectionSubscription } from '../src/collection/subscription.js' import { createDeferred } from '../src/deferred.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' import { Func, PropRef, Value } from '../src/query/ir.js' import { DeduplicatedLoadSubset } from '../src/query/subset-dedupe.js' import { flushPromises } from './utils' import type { LoadSubsetOptions } from '../src/types.js' describe(`CollectionSubscription status tracking`, () => { + it.each( + ([`release`, `restart`] as const).flatMap((boundary) => + [false, true].flatMap((rejectOld) => + [false, true].map((oldFirst) => ({ boundary, rejectOld, oldFirst })), + ), + ), + )( + `isolates pending status across retired work: %j`, + async ({ boundary, rejectOld, oldFirst }) => { + const old = createDeferred() + const current = createDeferred() + const last = createDeferred() + const pending = [old, current, last] + let loadCount = 0 + const load = vi.fn(() => pending[loadCount++]!.promise) + const collection = createCollection<{ id: string }>({ + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset: load, unloadSubset: () => {} } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]) + const statuses: Array = [] + subscription.on(`status:change`, ({ status }) => statuses.push(status)) + const settleOld = async () => { + if (rejectOld) old.reject(new Error(`retired work failed`)) + else old.resolve() + await flushPromises() + } + + try { + subscription.requestSnapshot({ where }) + expect(subscription.status).toBe(`loadingSubset`) + if (boundary === `release`) { + subscription.releaseSnapshot(where) + subscription.releaseSnapshot(where) + expect(subscription.status).toBe(`ready`) + subscription.requestSnapshot({ where }) + } else { + await collection.cleanup() + collection.startSyncImmediate() + } + await flushPromises() + expect(load).toHaveBeenCalledTimes(2) + expect(subscription.status).toBe(`loadingSubset`) + const before = [...statuses] + if (oldFirst) { + await settleOld() + expect(subscription.status).toBe(`loadingSubset`) + expect(statuses).toEqual(before) + } + current.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + if (!oldFirst) { + const after = [...statuses] + await settleOld() + expect(statuses).toEqual(after) + } + // A double decrement can hide until the next load starts. + subscription.requestSnapshot({ where }) + expect(load).toHaveBeenCalledTimes(3) + expect(subscription.status).toBe(`loadingSubset`) + last.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + } finally { + for (const result of pending) result.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each([ + { terms: 2, values: [0, 0] }, + { terms: 2, values: [0] }, + { terms: 1, values: [0, 0] }, + ])( + `rejects a $terms-term composite cursor before delivery or acquisition`, + async ({ terms, values }) => { + const load = vi.fn(() => true as const) + const unload = vi.fn() + const delivery = vi.fn() + const observer = vi.fn() + const collection = createCollection<{ id: string; rank: number }>({ + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row`, rank: 1 } }) + commit() + markReady() + return { loadSubset: load, unloadSubset: unload } + }, + }, + }) + const index = collection.createIndex((row) => row.rank, { + indexType: BTreeIndex, + }) + const subscription = collection.subscribeChanges(delivery, { + includeInitialState: false, + }) + subscription.setOrderByIndex(index) + const orderBy = Array.from({ length: terms }, () => ({ + expression: new PropRef([`rank`]), + compareOptions: { direction: `asc` as const, nulls: `first` as const }, + })) + try { + expect(() => + subscription.requestLimitedSnapshot({ + orderBy, + limit: 1, + minValues: values, + onLoadSubsetResult: observer, + }), + ).toThrow(`Only single-column cursors are supported`) + expect(delivery).not.toHaveBeenCalled() + expect(load).not.toHaveBeenCalled() + expect(observer).not.toHaveBeenCalled() + expect(subscription.status).toBe(`ready`) + // A rejected input must not consume local sent keys or an acquisition slot. + subscription.requestLimitedSnapshot({ + orderBy: orderBy.slice(0, 1), + limit: 1, + minValues: [0], + }) + expect(delivery).toHaveBeenCalledTimes(1) + expect(load).toHaveBeenCalledTimes(1) + expect(load.mock.calls[0]).toBeDefined() + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + expect(unload).toHaveBeenCalledTimes(1) + }, + ) + it(`subscription starts with status 'ready'`, () => { const collection = createCollection<{ id: string; value: string }>({ id: `test`, @@ -211,6 +360,112 @@ describe(`CollectionSubscription status tracking`, () => { subscription.unsubscribe() }) + it.each( + ([`generic`, `specific`] as const).flatMap((eventKind) => + ([`clean`, `throw`] as const).map((releaseKind) => ({ + eventKind, + releaseKind, + })), + ), + )( + `stops status delivery when a $eventKind loading listener unsubscribes with $releaseKind cleanup`, + async ({ eventKind, releaseKind }) => { + const pending = createDeferred() + const releaseFailure = new Error(`release failed during status callback`) + const deferredMicrotasks: Array = [] + const queueMicrotaskSpy = vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => deferredMicrotasks.push(callback)) + const collection = createCollection<{ id: string }>({ + id: `unsubscribe-during-${eventKind}-loading-status-${releaseKind}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => pending.promise, + unloadSubset: () => { + if (releaseKind === `throw`) throw releaseFailure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const eventsAfterTeardown: Array = [] + let teardownStarted = false + const unsubscribeOnLoading = () => { + teardownStarted = true + subscription.unsubscribe() + } + const recordAfterTeardown = (event: { status: string }) => { + if (teardownStarted) eventsAfterTeardown.push(event.status) + } + + if (eventKind === `generic`) { + subscription.on(`status:change`, ({ status }) => { + if (status === `loadingSubset`) unsubscribeOnLoading() + }) + subscription.on(`status:change`, recordAfterTeardown) + } else { + subscription.on(`status:loadingSubset`, unsubscribeOnLoading) + subscription.on(`status:loadingSubset`, recordAfterTeardown) + subscription.on(`status:change`, recordAfterTeardown) + } + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + expect(eventsAfterTeardown).toEqual([]) + expect(collection.subscriberCount).toBe(0) + expect(deferredMicrotasks).toHaveLength(releaseKind === `throw` ? 1 : 0) + if (releaseKind === `throw`) { + expect(() => deferredMicrotasks[0]!()).toThrow(releaseFailure) + } + + pending.resolve() + await flushPromises() + expect(eventsAfterTeardown).toEqual([]) + } finally { + queueMicrotaskSpy.mockRestore() + await collection.cleanup() + } + }, + ) + + it(`unsubscribes once when an unsubscribed listener reenters`, async () => { + const deferredMicrotasks: Array = [] + const queueMicrotaskSpy = vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => deferredMicrotasks.push(callback)) + const collection = createCollection<{ id: string }>({ + id: `reentrant-unsubscribed-event`, + getKey: ({ id }) => id, + sync: { sync: ({ markReady }) => markReady() }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + let events = 0 + subscription.on(`unsubscribed`, () => { + events++ + if (events === 1) subscription.unsubscribe() + }) + + try { + subscription.unsubscribe() + + expect(events).toBe(1) + expect(collection.subscriberCount).toBe(0) + expect(deferredMicrotasks).toEqual([]) + } finally { + queueMicrotaskSpy.mockRestore() + await collection.cleanup() + } + }) + it(`promise rejection still cleans up and sets status back to 'ready'`, async () => { let rejectLoadSubset: (error: Error) => void const loadSubsetPromise = new Promise((_, reject) => { @@ -317,59 +572,1317 @@ describe(`CollectionSubscription status tracking`, () => { expect(subscription.lastError).toBe(error) expect(failures).toEqual([error]) - subscription.unsubscribe() - await collection.cleanup() + subscription.unsubscribe() + await collection.cleanup() + }) + + it(`does not unload a subset when loadSubset throws before acquisition`, async () => { + const failure = new Error(`subset failed before acquisition`) + const unloadedOptions: Array = [] + const collection = createCollection<{ id: string }>({ + id: `failed-subset-acquisition`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + throw failure + }, + unloadSubset: (options) => unloadedOptions.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + expect(() => + subscription.requestSnapshot({ optimizedOnly: false }), + ).toThrow(failure) + subscription.unsubscribe() + + expect(unloadedOptions).toEqual([]) + await collection.cleanup() + }) + + it(`releases a subset when its load-result observer throws`, async () => { + const failure = new Error(`load-result observer failed`) + let acquiredOptions: unknown + const unloadedOptions: Array = [] + const collection = createCollection<{ id: string }>({ + id: `subset-observer-failure`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + acquiredOptions = options + return true + }, + unloadSubset: (options) => unloadedOptions.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + expect(() => + subscription.requestSnapshot({ + optimizedOnly: false, + onLoadSubsetResult: () => { + throw failure + }, + }), + ).toThrow(failure) + subscription.unsubscribe() + + expect(unloadedOptions).toEqual([acquiredOptions]) + await collection.cleanup() + }) + + it.each([ + { position: `failed-first`, nestedCleanup: `clean` }, + { position: `failed-first`, nestedCleanup: `throw` }, + { position: `failed-last`, nestedCleanup: `clean` }, + { position: `failed-last`, nestedCleanup: `throw` }, + ] as const)( + `re-finds the $position demand after reentrant $nestedCleanup cleanup`, + async ({ position, nestedCleanup }) => { + const primaryFailure = new Error(`request failed after acquisition`) + const cleanupFailure = new Error(`nested cleanup failed`) + const loaded: Array = [] + const unloaded: Array = [] + const reported: Array = [] + let caughtCleanup: unknown + const collection = createCollection<{ id: string }>({ + id: `reentrant-primary-release-${position}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loaded.push(options) + return true + }, + unloadSubset: (options) => { + unloaded.push(options) + if ( + nestedCleanup === `throw` && + options === loaded[0] && + unloaded.filter((entry) => entry === loaded[0]).length === 1 + ) { + throw cleanupFailure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const firstWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`first`), + ]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`second`), + ]) + let releaseFirst: + | ((primaryFailure?: { error: unknown }) => void) + | undefined + let releaseSecond: + | ((primaryFailure?: { error: unknown }) => void) + | undefined + + subscription.requestSnapshot({ + where: firstWhere, + onLoadSubsetResult: (_result, _options, release) => { + releaseFirst = release + }, + }) + subscription.requestSnapshot({ + where: secondWhere, + onLoadSubsetResult: (_result, _options, release) => { + releaseSecond = release + }, + }) + subscription.on(`loadSubset:error`, ({ error }) => { + reported.push(error) + try { + subscription.releaseSnapshot(firstWhere) + } catch (cleanupError) { + caughtCleanup = cleanupError + } + }) + + if (position === `failed-first`) { + releaseFirst!({ error: primaryFailure }) + expect(unloaded).toEqual([loaded[0]]) + } else { + releaseSecond!({ error: primaryFailure }) + expect(unloaded).toEqual([loaded[0], loaded[1]]) + } + expect(subscription.lastError).toBe(primaryFailure) + expect(reported).toEqual([primaryFailure]) + expect(caughtCleanup).toBe( + nestedCleanup === `throw` ? cleanupFailure : undefined, + ) + + subscription.unsubscribe() + expect(unloaded).toEqual([loaded[0], loaded[1]]) + expect(subscription.lastError).toBe(primaryFailure) + expect(reported).toEqual([primaryFailure]) + await collection.cleanup() + }, + ) + + it.each([`releaseSnapshot`, `unsubscribe`] as const)( + `attempts a failed exact release only once through %s`, + async (releaseMode) => { + const loads: Array = [] + const unloads: Array = [] + const failure = new Error(`release failed`) + const collection = createCollection<{ id: string }>({ + id: `failed-exact-release-${releaseMode}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: false, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return Promise.resolve() + }, + unloadSubset: (options) => { + unloads.push(options) + if (unloads.length === 1) throw failure + }, + } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const where = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`requested`), + ]) + + try { + subscription.requestSnapshot({ + where, + limit: 1, + optimizedOnly: false, + }) + collection._resumeSyncStart() + await flushPromises() + + expect(loads).toHaveLength(1) + const firstRelease = () => + releaseMode === `releaseSnapshot` + ? subscription.releaseSnapshot(where) + : subscription.unsubscribe() + expect(firstRelease).toThrow(failure) + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads).toEqual([loads[0]]) + + subscription.unsubscribe() + expect(unloads).toHaveLength(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it.each( + ([`before`, `after`] as const).flatMap((throwAt) => + [false, true].map((reenter) => ({ throwAt, reenter })), + ), + )( + `bounds throwing adapter cleanup at $throwAt release, reentry=$reenter`, + async ({ throwAt, reenter }) => { + const failure = new Error(`adapter cleanup failed`) + const loads: Array = [] + const unloads: Array = [] + const externalLeases = new Set() + let dispose = () => {} + const collection = createCollection<{ id: string }>({ + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + externalLeases.add(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[0]) { + if (reenter) dispose() + if (throwAt === `before`) throw failure + externalLeases.delete(options) + throw failure + } + externalLeases.delete(options) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + dispose = () => subscription.unsubscribe() + try { + subscription.requestSnapshot({ where: new Value(true) }) + subscription.requestSnapshot({ where: new Value(false) }) + expect(dispose).toThrow(failure) + expect(dispose).not.toThrow() + expect(unloads).toEqual(loads) + expect(loads).toHaveLength(2) + expect(loads.every(({ signal }) => signal?.aborted)).toBe(true) + expect(collection.subscriberCount).toBe(0) + expect(subscription.lastError).toBe(failure) + // Intentional support boundary: core cannot repair an adapter that throws + // before freeing its resource, nor safely repeat a possibly completed release. + expect([...externalLeases]).toEqual( + throwAt === `before` ? [loads[0]] : [], + ) + } finally { + dispose() + await collection.cleanup() + } + }, + ) + + it(`preserves a primary error across reentrant teardown failure`, async () => { + const primaryFailure = new Error(`request failed after acquisition`) + const cleanupFailure = new Error(`teardown failed`) + const loads: Array = [] + const unloads: Array = [] + const reported: Array = [] + let releaseFailedDemand: + | ((primaryFailure?: { error: unknown }) => void) + | undefined + let cleanupAttempts = 0 + let caughtCleanup: unknown + const collection = createCollection<{ id: string }>({ + id: `primary-error-reentrant-teardown`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[0] && cleanupAttempts++ === 0) { + throw cleanupFailure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.requestSnapshot({ where: new Value(true) }) + subscription.requestSnapshot({ + where: new Value(false), + onLoadSubsetResult: (_result, _options, release) => { + releaseFailedDemand = release + }, + }) + subscription.on(`loadSubset:error`, ({ error }) => { + reported.push(error) + if (error !== primaryFailure) return + try { + subscription.unsubscribe() + } catch (cleanupError) { + caughtCleanup = cleanupError + } + }) + + releaseFailedDemand!({ error: primaryFailure }) + + expect(caughtCleanup).toBe(cleanupFailure) + expect(reported).toEqual([primaryFailure]) + expect(subscription.lastError).toBe(primaryFailure) + expect(unloads).toEqual([loads[0], loads[1]]) + + subscription.unsubscribe() + expect(unloads).toEqual([loads[0], loads[1]]) + expect(subscription.lastError).toBe(primaryFailure) + await collection.cleanup() + }) + + it.each([`sync`, `async`, `replay`] as const)( + `preserves a %s adapter error across reentrant teardown failure`, + async (failureMode) => { + const primaryFailure = new Error(`${failureMode} load failed`) + const cleanupFailure = new Error(`teardown failed`) + const victimWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`victim`), + ]) + const failedWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`failed`), + ]) + const loads: Array = [] + const reported: Array = [] + let truncateSource = () => {} + let cleanupAttempts = 0 + let caughtCleanup: unknown + let deliveringPrimary = false + const collection = createCollection<{ id: string }>({ + id: `primary-${failureMode}-reentrant-teardown`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, commit, markReady, truncate }) => { + truncateSource = () => { + begin() + truncate() + commit() + } + markReady() + return { + loadSubset: (options) => { + loads.push(options) + const shouldFail = + failureMode === `replay` + ? loads.length === 4 + : loads.length === 2 + if (!shouldFail) return true + if (failureMode === `sync`) throw primaryFailure + return Promise.reject(primaryFailure) + }, + unloadSubset: () => { + if (deliveringPrimary && cleanupAttempts++ === 0) { + throw cleanupFailure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => { + reported.push(error) + if (error !== primaryFailure) return + deliveringPrimary = true + try { + subscription.releaseSnapshot(victimWhere) + } catch (cleanupError) { + caughtCleanup = cleanupError + } finally { + deliveringPrimary = false + } + }) + + try { + subscription.requestSnapshot({ where: victimWhere }) + if (failureMode === `replay`) { + subscription.requestSnapshot({ where: failedWhere }) + truncateSource() + } else { + const request = () => + subscription.requestSnapshot({ where: failedWhere }) + if (failureMode === `sync`) { + expect(request).toThrow(primaryFailure) + } else { + request() + } + } + await flushPromises() + + expect(caughtCleanup).toBe(cleanupFailure) + expect(reported).toEqual([primaryFailure]) + expect(subscription.lastError).toBe(primaryFailure) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`does not unload a synchronous acquisition that never started`, async () => { + const failure = new Error(`load failed before acquisition`) + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`failed`)]) + const unloads: Array = [] + const collection = createCollection<{ id: string }>({ + id: `reentrant-failed-start-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + throw failure + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + subscription.on(`loadSubset:error`, ({ error }) => { + if (error === failure) subscription.releaseSnapshot(where) + }) + + try { + expect(() => subscription.requestSnapshot({ where })).toThrow(failure) + expect(unloads).toEqual([]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not replay a logically retired demand after its unload fails`, async () => { + const loads: Array = [] + const unloads: Array = [] + const releaseError = new Error(`release failed`) + let allowUnload = false + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const collection = createCollection<{ id: string }>({ + id: `failed-release-is-not-replayed`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return Promise.resolve() + }, + unloadSubset: (options) => { + unloads.push(options) + if (!allowUnload && options === loads[0]) throw releaseError + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`first`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`second`), + ]) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + await flushPromises() + + expect(() => subscription.releaseSnapshot(firstWhere)).toThrow( + releaseError, + ) + + begin() + truncate() + commit() + await flushPromises() + + // The release attempt retired the demand, even though the adapter threw. + // It must not join later replays or be released a second time. + expect(loads).toHaveLength(3) + expect(loads[2]?.where).toBe(secondWhere) + } finally { + allowUnload = true + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`retires pending status per demand even when physical cleanup fails`, async () => { + const firstLoad = createDeferred() + const secondLoad = createDeferred() + const loads: Array = [] + const unloads: Array = [] + const releaseError = new Error(`release failed`) + let firstReleaseAttempts = 0 + const collection = createCollection<{ id: string }>({ + id: `retired-pending-status-and-cleanup-debt`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return loads.length === 1 ? firstLoad.promise : secondLoad.promise + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[0] && ++firstReleaseAttempts < 3) { + throw releaseError + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`first`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`second`), + ]) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + expect(subscription.status).toBe(`loadingSubset`) + + expect(() => subscription.releaseSnapshot(firstWhere)).toThrow( + releaseError, + ) + expect(subscription.status).toBe(`loadingSubset`) + + secondLoad.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + + expect(() => subscription.unsubscribe()).not.toThrow() + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads).toEqual([loads[0], loads[1]]) + } finally { + firstLoad.resolve() + secondLoad.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`attempts both leases once when throwing cleanup reenters teardown`, async () => { + const releaseFailure = new Error(`release failed`) + const duplicateFailure = new Error(`duplicate release`) + const loads: Array = [] + const unloads: Array = [] + const attempts = new Map() + const collection = createCollection<{ id: string }>({ + id: `reentrant-cleanup-debt-retirement`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + const attempt = (attempts.get(options) ?? 0) + 1 + attempts.set(options, attempt) + if (attempt > 1) throw duplicateFailure + if (options === loads[0]) { + subscription.unsubscribe() + } + throw releaseFailure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`first`)]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`second`), + ]) + + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + expect(() => subscription.releaseSnapshot(firstWhere)).toThrow( + releaseFailure, + ) + expect(() => subscription.releaseSnapshot(secondWhere)).not.toThrow() + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads).toEqual([loads[0], loads[1]]) + expect(collection.subscriberCount).toBe(0) + + subscription.unsubscribe() + expect(unloads).toHaveLength(2) + } finally { + try { + subscription.unsubscribe() + } catch { + // Keep cleanup available after a red assertion. + } + await collection.cleanup() + } + }) + + it.each([`sync`, `async`] as const)( + `reopens a failed %s replay only after its last logical demand retires`, + async (failureMode) => { + const failure = new Error(`replay failed`) + let begin!: () => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + let replayStarts = 0 + let replaySuccesses = 0 + const collection = createCollection<{ id: string }>({ + id: `failed-replay-logical-demand-cardinality-${failureMode}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount += 1 + if (loadCount <= 2) return true + if (failureMode === `sync`) throw failure + return Promise.reject(failure) + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + truncateReplayPublication: { + start: () => { + replayStarts += 1 + }, + succeed: () => { + replaySuccesses += 1 + }, + }, + }) + const firstWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`first`), + ]) + const secondWhere = new Func(`eq`, [ + new PropRef([`id`]), + new Value(`second`), + ]) + + try { + subscription.requestSnapshot({ + where: firstWhere, + optimizedOnly: false, + }) + subscription.requestSnapshot({ + where: secondWhere, + optimizedOnly: false, + }) + + begin() + truncate() + commit() + await flushPromises() + expect(loadCount).toBe(4) + expect(replayStarts).toBe(1) + expect(replaySuccesses).toBe(0) + + subscription.releaseSnapshot(firstWhere) + expect(replaySuccesses).toBe(0) + + subscription.releaseSnapshot(secondWhere) + expect(replaySuccesses).toBe(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`attempts the exact in-flight replay release once`, async () => { + const replay = createDeferred() + const loads: Array = [] + const unloads: Array = [] + const failure = new Error(`replay release failed`) + let failed = false + let begin!: () => void + let commit!: () => void + let truncate!: () => void + const collection = createCollection<{ id: string }>({ + id: `failed-replay-release`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + return loads.length === 1 ? Promise.resolve() : replay.promise + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[1] && !failed) { + failed = true + throw failure + } + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + await flushPromises() + begin() + truncate() + commit() + await flushPromises() + + expect(loads).toHaveLength(2) + expect(() => subscription.unsubscribe()).toThrow(failure) + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads.filter((options) => options === loads[0])).toEqual([ + loads[0], + ]) + expect(unloads.filter((options) => options === loads[1])).toEqual([ + loads[1], + ]) + } finally { + replay.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it.each( + ([`direct`, `deferred`] as const).flatMap((start) => + ([`return`, `resolve`] as const).map((result) => ({ + name: `${start} ${result}`, + start, + result, + })), + ), + )( + `publishes ownership before a reentrant unsubscribe: $name`, + async ({ start, result }) => { + const loads: Array = [] + const unloads: Array = [] + let unsubscribeDuringLoad = () => {} + const collection = createCollection<{ id: string }>({ + id: `reentrant-ownership-${start}-${result}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: start === `direct`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + unsubscribeDuringLoad() + return result === `return` ? true : Promise.resolve() + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + if (start === `deferred`) expect(collection._deferSyncStart()).toBe(true) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + unsubscribeDuringLoad = () => subscription.unsubscribe() + + try { + subscription.requestSnapshot({ limit: 1, optimizedOnly: false }) + if (start === `deferred`) collection._resumeSyncStart() + await flushPromises() + + expect(loads).toHaveLength(1) + expect(unloads).toEqual([loads[0]]) + subscription.unsubscribe() + expect(unloads).toHaveLength(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`does not register a subscription closed during its automatic snapshot`, async () => { + const loads: Array = [] + const unloads: Array = [] + const onChange = vi.fn() + let writeAfterUnsubscribe = () => {} + const collection = createCollection<{ id: string }>({ + id: `closed-during-automatic-snapshot`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + writeAfterUnsubscribe = () => { + begin() + write({ type: `insert`, value: { id: `later` } }) + commit() + } + markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (!(options.subscription instanceof CollectionSubscription)) { + throw new Error(`automatic snapshot requires its subscription`) + } + options.subscription.unsubscribe() + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + + const subscription = collection.subscribeChanges(onChange, { + includeInitialState: true, + }) + + expect(loads).toHaveLength(1) + expect(unloads).toEqual(loads) + expect(collection.subscriberCount).toBe(0) + writeAfterUnsubscribe() + expect(onChange).not.toHaveBeenCalled() + + subscription.unsubscribe() + expect(unloads).toHaveLength(1) + await collection.cleanup() + }) + + it(`does not deliver a direct snapshot after adapter work unsubscribes`, async () => { + type Row = { id: string; rank: number } + const loads: Array = [] + const unloads: Array = [] + const callbacks: Array> = [] + let unsubscribeDuringLoad = () => {} + const collection = createCollection({ + id: `direct-snapshot-reentrant-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row`, rank: 1 } }) + commit() + markReady() + return { + loadSubset: (options) => { + loads.push(options) + unsubscribeDuringLoad() + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const subscription = collection.subscribeChanges((changes) => { + callbacks.push(changes.map(({ value }) => value.id)) + }) + unsubscribeDuringLoad = () => subscription.unsubscribe() + + try { + subscription.requestSnapshot({ optimizedOnly: false }) + + expect(callbacks).toEqual([]) + expect(loads).toHaveLength(1) + expect(unloads).toEqual([loads[0]]) + + subscription.requestSnapshot({ optimizedOnly: false }) + expect(callbacks).toEqual([]) + expect(loads).toHaveLength(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not continue a direct snapshot after its result hook unsubscribes`, async () => { + type Row = { id: string } + const callbacks: Array> = [] + const collection = createCollection({ + id: `direct-result-hook-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row` } }) + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }) + const subscription = collection.subscribeChanges((changes) => { + callbacks.push(changes.map(({ value }) => value.id)) + }) + + try { + subscription.requestSnapshot({ + optimizedOnly: false, + onLoadSubsetResult: () => subscription.unsubscribe(), + }) + + expect(callbacks).toEqual([]) + expect(subscription.status).toBe(`ready`) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not continue an unoptimized snapshot after its hook unsubscribes`, async () => { + type Row = { id: string } + const callbacks: Array> = [] + const collection = createCollection({ + id: `direct-unoptimized-hook-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row` } }) + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }) + const subscription = collection.subscribeChanges((changes) => { + callbacks.push(changes.map(({ value }) => value.id)) + }) + + try { + subscription.requestSnapshot({ + where: new Func(`eq`, [new PropRef([`id`]), new Value(`row`)]), + onUnoptimized: () => subscription.unsubscribe(), + }) + + expect(callbacks).toEqual([]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not start limited adapter work after local delivery unsubscribes`, async () => { + type Row = { id: string; rank: number } + const loads: Array = [] + const unloads: Array = [] + const collection = createCollection({ + id: `limited-snapshot-reentrant-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `row`, rank: 1 } }) + commit() + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => unloads.push(options), + } + }, + }, + }) + const index = collection.createIndex((row) => row.rank, { + indexType: BTreeIndex, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => subscription.unsubscribe(), + ) + subscription.setOrderByIndex(index) + + try { + subscription.requestLimitedSnapshot({ + orderBy: [ + { + expression: new PropRef([`rank`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + }) + + expect(loads).toEqual([]) + expect(unloads).toEqual([]) + + subscription.requestLimitedSnapshot({ + orderBy: [ + { + expression: new PropRef([`rank`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + }) + expect(loads).toEqual([]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not observe limited adapter work after it unsubscribes`, async () => { + type Row = { id: string; rank: number } + const pending = createDeferred() + let resultCallbacks = 0 + const collection = createCollection({ + id: `limited-adapter-unsubscribe`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + subscription.unsubscribe() + return pending.promise + }, + } + }, + }, + }) + const index = collection.createIndex((row) => row.rank, { + indexType: BTreeIndex, + }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + ) + subscription.setOrderByIndex(index) + + try { + subscription.requestLimitedSnapshot({ + orderBy: [ + { + expression: new PropRef([`rank`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ], + limit: 1, + onLoadSubsetResult: () => resultCallbacks++, + }) + + expect(resultCallbacks).toBe(0) + expect(subscription.status).toBe(`ready`) + } finally { + pending.resolve() + subscription.unsubscribe() + await collection.cleanup() + } }) - it(`does not unload a subset when loadSubset throws before acquisition`, async () => { - const failure = new Error(`subset failed before acquisition`) - const unloadedOptions: Array = [] + it(`does not release one acquisition twice during nested unsubscribe`, async () => { + const unloads: Array = [] + let reentered = false const collection = createCollection<{ id: string }>({ - id: `failed-subset-acquisition`, - getKey: (item) => item.id, + id: `nested-unsubscribe-release`, + getKey: ({ id }) => id, syncMode: `on-demand`, sync: { sync: ({ markReady }) => { markReady() return { - loadSubset: () => { - throw failure + loadSubset: () => true, + unloadSubset: (options) => { + unloads.push(options) + if (!reentered) { + reentered = true + subscription.unsubscribe() + } }, - unloadSubset: (options) => unloadedOptions.push(options), } }, }, }) - const subscription = collection.subscribeChanges(() => {}, { - includeInitialState: false, - }) + const subscription: CollectionSubscription = collection.subscribeChanges( + () => {}, + { + includeInitialState: false, + }, + ) - expect(() => - subscription.requestSnapshot({ optimizedOnly: false }), - ).toThrow(failure) - subscription.unsubscribe() + try { + subscription.requestSnapshot({ optimizedOnly: false }) + subscription.unsubscribe() - expect(unloadedOptions).toEqual([]) - await collection.cleanup() + expect(unloads).toHaveLength(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } }) - it(`releases a subset when its load-result observer throws`, async () => { - const failure = new Error(`load-result observer failed`) - let acquiredOptions: unknown - const unloadedOptions: Array = [] + it.each( + ([false, true] as const).flatMap((adapterCatches) => + ([`return`, `resolve`] as const).map((result) => ({ + name: `${adapterCatches ? `caught` : `escaped`} ${result}`, + adapterCatches, + result, + })), + ), + )( + `attempts a deferred failed release once after adapter startup: $name`, + async ({ adapterCatches, result }) => { + const failure = new Error(`reentrant release failed`) + const loads: Array = [] + const unloads: Array = [] + let observedReleaseError: unknown + let unsubscribeDuringLoad = () => {} + const collection = createCollection<{ id: string }>({ + id: `reentrant-release-${adapterCatches}-${result}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + loads.push(options) + if (adapterCatches) { + try { + unsubscribeDuringLoad() + } catch (error) { + observedReleaseError = error + } + } else { + unsubscribeDuringLoad() + } + return result === `return` ? true : Promise.resolve() + }, + unloadSubset: (options) => { + unloads.push(options) + if (unloads.length === 1) throw failure + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + unsubscribeDuringLoad = () => subscription.unsubscribe() + + try { + const request = () => + subscription.requestSnapshot({ limit: 1, optimizedOnly: false }) + expect(request).toThrow(failure) + expect(observedReleaseError).toBeUndefined() + await flushPromises() + + expect(unloads).toEqual([loads[0]]) + expect(() => subscription.unsubscribe()).not.toThrow() + expect(unloads).toEqual([loads[0]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + + it(`releases each acquisition once when synchronous replay drops its demand`, async () => { + const loads: Array = [] + const unloads: Array = [] + const where = new Func(`eq`, [new PropRef([`id`]), new Value(`requested`)]) + let replay = () => {} + let releaseDuringReplay = () => {} const collection = createCollection<{ id: string }>({ - id: `subset-observer-failure`, - getKey: (item) => item.id, + id: `synchronous-replay-release`, + getKey: ({ id }) => id, syncMode: `on-demand`, sync: { - sync: ({ markReady }) => { + sync: ({ begin, commit, markReady, truncate }) => { + replay = () => { + begin() + truncate() + commit() + } markReady() return { loadSubset: (options) => { - acquiredOptions = options + loads.push(options) + if (loads.length === 2) releaseDuringReplay() return true }, - unloadSubset: (options) => unloadedOptions.push(options), + unloadSubset: (options) => unloads.push(options), } }, }, @@ -377,19 +1890,21 @@ describe(`CollectionSubscription status tracking`, () => { const subscription = collection.subscribeChanges(() => {}, { includeInitialState: false, }) + releaseDuringReplay = () => subscription.releaseSnapshot(where) - expect(() => - subscription.requestSnapshot({ - optimizedOnly: false, - onLoadSubsetResult: () => { - throw failure - }, - }), - ).toThrow(failure) - subscription.unsubscribe() + try { + subscription.requestSnapshot({ where, optimizedOnly: false }) + replay() + await flushPromises() - expect(unloadedOptions).toEqual([acquiredOptions]) - await collection.cleanup() + expect(loads).toHaveLength(2) + expect(unloads.map((options) => loads.indexOf(options)).sort()).toEqual([ + 0, 1, + ]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } }) it(`reports a rejected subset replay after truncate`, async () => { @@ -502,7 +2017,9 @@ describe(`CollectionSubscription status tracking`, () => { truncate() commit() await flushPromises() - expect(transportCalls).toBe(2) + // Replay creates a fresh abortable acquisition for each logical demand, + // even when the adapter happens to return the same promise for both. + expect(transportCalls).toBe(3) subscription.releaseSnapshot(where) const failure = new Error(`shared replay failed`) @@ -520,7 +2037,7 @@ describe(`CollectionSubscription status tracking`, () => { } }) - it(`keeps the old lease when replacing it fails`, async () => { + it(`retries detached demand after retiring the old lease fails`, async () => { const replay = createDeferred() const loads: Array = [] const unloads: Array = [] @@ -564,19 +2081,28 @@ describe(`CollectionSubscription status tracking`, () => { commit() await flushPromises() - expect(loads).toHaveLength(2) - expect(subscription.status).toBe(`loadingSubset`) - - replay.reject(new DOMException(`replacement abandoned`, `AbortError`)) - await flushPromises() + expect(loads).toHaveLength(1) + expect(unloads).toEqual([loads[0]]) expect(subscription.status).toBe(`ready`) expect(subscription.lastError).toEqual( new Error(`old lease release failed`), ) + begin() + truncate() + commit() + await flushPromises() + expect(loads).toHaveLength(2) + expect(subscription.status).toBe(`loadingSubset`) + expect(loads[1]?.signal?.aborted).toBe(false) + expect(unloads).toEqual([loads[0]]) + replay.resolve() + await flushPromises() + expect(subscription.status).toBe(`ready`) + subscription.unsubscribe() unsubscribed = true - expect(unloads).toEqual([loads[0], loads[1], loads[0]]) + expect(unloads).toEqual([loads[0], loads[1]]) } finally { replay.resolve() if (!unsubscribed) subscription.unsubscribe() @@ -584,6 +2110,142 @@ describe(`CollectionSubscription status tracking`, () => { } }) + it(`does not become ready while replay setup still has a surviving demand`, async () => { + const firstWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`one`)]) + const secondWhere = new Func(`eq`, [new PropRef([`id`]), new Value(`two`)]) + const firstReplay = createDeferred() + const statusEvents: Array<{ status: string; loadCount: number }> = [] + let begin!: () => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const collection = createCollection<{ id: string }>({ + id: `replay-setup-readiness`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + loadCount++ + return loadCount > 2 && options.where === firstWhere + ? firstReplay.promise + : true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + let releaseFirstReplay = false + subscription.on(`status:change`, ({ status }) => { + statusEvents.push({ status, loadCount }) + if (releaseFirstReplay && status === `loadingSubset`) { + releaseFirstReplay = false + subscription.releaseSnapshot(firstWhere) + } + }) + + try { + subscription.requestSnapshot({ where: firstWhere }) + subscription.requestSnapshot({ where: secondWhere }) + releaseFirstReplay = true + begin() + truncate() + commit() + await flushPromises() + + expect(loadCount).toBe(3) + expect(subscription.status).toBe(`ready`) + expect(statusEvents).toEqual([ + { status: `loadingSubset`, loadCount: 2 }, + { status: `ready`, loadCount: 3 }, + ]) + } finally { + firstReplay.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not publish a pending replay after collection cleanup`, async () => { + type Row = { id: string; version: number } + const replay = createDeferred() + const visible = new Map() + const statusEvents: Array = [] + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const collection = createCollection({ + id: `cleanup-pending-replay`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + const version = ++loadCount + begin() + write({ type: `insert`, value: { id: `row`, version } }) + commit() + return version === 1 ? true : replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else visible.set(change.key, change.value) + } + }, + { includeInitialState: false }, + ) + subscription.on(`status:change`, ({ status }) => { + statusEvents.push(status) + }) + + try { + subscription.requestSnapshot() + expect(visible.get(`row`)?.version).toBe(1) + begin() + truncate() + commit() + await flushPromises() + expect(subscription.status).toBe(`loadingSubset`) + + await collection.cleanup() + const eventsAfterCleanup = [...statusEvents] + replay.resolve() + await flushPromises() + + expect(visible.get(`row`)?.version).toBe(1) + expect(statusEvents).toEqual(eventsAfterCleanup) + } finally { + replay.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + it(`retains a subset after a synchronous truncate replay failure`, async () => { const error = new Error(`synchronous truncate replay failed`) let truncateSource: () => void = () => { @@ -700,7 +2362,9 @@ describe(`CollectionSubscription status tracking`, () => { commit() await flushPromises() - expect([...visible.keys()].sort()).toEqual([`one`, `two`]) + // Ordinary source changes do not establish a complete replacement. + // Keep the last coherent generation until a later replay succeeds. + expect([...visible.keys()]).toEqual([`one`]) failReplay = false begin() @@ -792,6 +2456,88 @@ describe(`CollectionSubscription status tracking`, () => { await collection.cleanup() }) + it(`does not become ready between reentrant truncate replacements`, async () => { + type Row = { id: string; version: number } + const replays = [createDeferred(), createDeferred()] + const statusEvents: Array = [] + const visible = new Map() + let begin!: () => void + let write!: (message: { type: `insert`; value: Row }) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + let startedNestedReplay = false + const collection = createCollection({ + id: `reentrant-truncate-ready-barrier`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: () => { + const version = ++loadCount + begin() + write({ type: `insert`, value: { id: `row`, version } }) + commit() + return version === 1 ? true : replays[version - 2]!.promise + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(change.key) + else visible.set(change.key, change.value) + } + if (visible.get(`row`)?.version === 2 && !startedNestedReplay) { + startedNestedReplay = true + begin() + truncate() + commit() + } + }, + { includeInitialState: false }, + ) + subscription.on(`status:change`, ({ status }) => { + statusEvents.push(status) + }) + + try { + subscription.requestSnapshot() + expect(visible.get(`row`)?.version).toBe(1) + + begin() + truncate() + commit() + await flushPromises() + expect(subscription.status).toBe(`loadingSubset`) + + replays[0]!.resolve() + await flushPromises() + expect(loadCount).toBe(3) + expect(visible.get(`row`)?.version).toBe(2) + expect(subscription.status).toBe(`loadingSubset`) + expect(statusEvents).toEqual([`loadingSubset`]) + + replays[1]!.resolve() + await flushPromises() + expect(visible.get(`row`)?.version).toBe(3) + expect(subscription.status).toBe(`ready`) + expect(statusEvents).toEqual([`loadingSubset`, `ready`]) + } finally { + for (const replay of replays) replay.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + it(`scopes a subset failure to the subscription that requested it`, async () => { const error = new Error(`first subscription failed`) let loadCount = 0 @@ -874,6 +2620,51 @@ describe(`CollectionSubscription status tracking`, () => { await collection.cleanup() }) + it.each([ + [`Temporal`, Temporal.PlainDate.from(`2026-08-24`)], + [ + `opaque class`, + new (class Sortable { + valueOf() { + return 24 + } + })(), + ], + ])( + `passes a %s range operand through to the adapter`, + async (_name, operand) => { + let received: LoadSubsetOptions | undefined + const collection = createCollection<{ id: string }>({ + id: `range-operand-subset`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: (options) => { + received = options + return true + }, + } + }, + }, + }) + const subscription = collection.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const where = new Func(`gt`, [new PropRef([`value`]), new Value(operand)]) + + expect(() => + subscription.requestSnapshot({ where, optimizedOnly: false }), + ).not.toThrow() + expect(((received?.where as Func).args[1] as Value).value).toBe(operand) + + subscription.unsubscribe() + await collection.cleanup() + }, + ) + it(`unsubscribe clears event listeners`, () => { const collection = createCollection<{ id: string; value: string }>({ id: `test`, diff --git a/packages/db/tests/collection-sync-reentrancy.test.ts b/packages/db/tests/collection-sync-reentrancy.test.ts new file mode 100644 index 0000000000..48bd077d48 --- /dev/null +++ b/packages/db/tests/collection-sync-reentrancy.test.ts @@ -0,0 +1,1420 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createDeferred } from '../src/deferred.js' +import { oracleRandomParameters, readOracleRunConfig } from './oracle-config.js' +import { flushPromises } from './utils.js' +import type { SyncConfig } from '../src/types.js' + +type Row = { + id: number + value: string +} + +type SyncOps = Parameters[`sync`]>[0] + +type OrderedRow = Row & { rank: number } +type OrderedSync = Parameters[`sync`]>[0] + +type LayoutCallback = { + changes: Array + keys: Array + values: Array + markedReceiptSettled: boolean + revision: number +} + +type ListenerAction = `commit` | `abort` + +type ListenerScenario = { + beforeOpen: ReadonlyArray + leaveOpen: boolean + afterOpen: ReadonlyArray +} + +const listenerActionArbitrary = fc.constantFrom( + `commit`, + `abort`, +) + +const listenerScenarioArbitrary: fc.Arbitrary = fc.record({ + beforeOpen: fc.array(listenerActionArbitrary, { maxLength: 2 }), + leaveOpen: fc.boolean(), + afterOpen: fc.array(listenerActionArbitrary, { maxLength: 2 }), +}) + +function enumerateActions(maxLength: number): Array> { + const histories: Array> = [[]] + for (let length = 1; length <= maxLength; length++) { + const previous = histories.filter( + (history) => history.length === length - 1, + ) + histories.push( + ...previous.flatMap((history) => + ([`commit`, `abort`] as const).map((action) => [...history, action]), + ), + ) + } + return histories +} + +const exhaustiveListenerScenarios: Array = enumerateActions( + 2, +).flatMap((beforeOpen) => + enumerateActions(2).flatMap((afterOpen) => + [false, true].map((leaveOpen) => ({ + beforeOpen, + leaveOpen, + afterOpen, + })), + ), +) + +let generatedHarnessId = 0 + +function createSyncHarness(id: string) { + let sync!: SyncOps + const collection = createCollection({ + id, + getKey: (row) => row.id, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + ops.markReady() + }, + }, + }) + + return { + collection, + get sync() { + return sync + }, + } +} + +function stageInsert( + sync: SyncOps, + row: Row, + options?: { immediate?: boolean }, +): void { + sync.begin(options) + sync.write({ type: `insert`, value: row }) +} + +function installInitialOrderedRows(sync: OrderedSync): void { + sync.begin({ immediate: true }) + sync.write({ + type: `insert`, + value: { id: 1, value: `one`, rank: 0 }, + }) + sync.write({ + type: `insert`, + value: { id: 2, value: `two`, rank: 1 }, + }) + sync.commit() + sync.markReady() +} + +async function runListenerScenario(scenario: ListenerScenario): Promise { + const harness = createSyncHarness( + `generated-listener-sync-${generatedHarnessId++}`, + ) + const { collection } = harness + const appliedKeys: Array = [] + const originalSet = collection._state.syncedData.set.bind( + collection._state.syncedData, + ) + vi.spyOn(collection._state.syncedData, `set`).mockImplementation( + (key, value) => { + appliedKeys.push(key) + return originalSet(key, value) + }, + ) + const batches: Array> = [] + const committedKeys: Array = [] + const committedReceipts: Array> = [] + const abortedReceipts: Array> = [] + let openKey: number | undefined + let nextKey = 2 + let listenerDepth = 0 + let maxListenerDepth = 0 + let ranActions = false + + const runAction = (action: ListenerAction) => { + const key = nextKey++ + stageInsert(harness.sync, { id: key, value: action }) + if (action === `commit`) { + committedKeys.push(key) + const receipt = harness.sync.commit() + if (receipt !== true) committedReceipts.push(receipt) + return + } + + const controller = new AbortController() + controller.abort() + const receipt = harness.sync.commit(controller.signal) + if (receipt !== true) { + void receipt.then( + () => abortedReceipts.push({ status: `fulfilled`, value: undefined }), + (reason) => abortedReceipts.push({ status: `rejected`, reason }), + ) + } + } + + const subscription = collection.subscribeChanges((changes) => { + listenerDepth++ + maxListenerDepth = Math.max(maxListenerDepth, listenerDepth) + batches.push(changes.map((change) => change.key as number)) + + if (!ranActions && changes.some(({ key }) => key === 1)) { + ranActions = true + scenario.beforeOpen.forEach(runAction) + if (scenario.leaveOpen) { + openKey = nextKey++ + stageInsert(harness.sync, { id: openKey, value: `open` }) + } + scenario.afterOpen.forEach(runAction) + } + + listenerDepth-- + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect(appliedKeys).toEqual([1, ...committedKeys]) + expect(batches).toEqual([ + [1], + ...(committedKeys.length > 0 ? [committedKeys] : []), + ]) + expect(maxListenerDepth).toBe(1) + await Promise.all(committedReceipts) + await flushPromises() + expect(committedReceipts).toHaveLength(committedKeys.length) + expect(abortedReceipts).toHaveLength( + scenario.beforeOpen.filter((action) => action === `abort`).length + + scenario.afterOpen.filter((action) => action === `abort`).length, + ) + expect(abortedReceipts.every(({ status }) => status === `rejected`)).toBe( + true, + ) + + if (openKey !== undefined) { + harness.sync.commit() + expect(appliedKeys).toEqual([1, ...committedKeys, openKey]) + expect(batches.at(-1)).toEqual([openKey]) + } + + expect(collection._state.pendingSyncedTransactions).toHaveLength(0) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } +} + +const { multiplier, ...replay } = readOracleRunConfig() +const generatedRuns = 30 * multiplier + +describe(`sync publication reentrancy`, () => { + it(`publishes nested deferrals as one coherent batch`, async () => { + const harness = createSyncHarness(`nested-publication-cycle`) + const { collection } = harness + const callbacks: Array<{ changes: Array; visibleValue: string }> = + [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map((change) => change.value.value), + visibleValue: collection.get(1)!.value, + }) + }, + { includeInitialState: false }, + ) + + try { + const outer = collection._deferPublication() + stageInsert(harness.sync, { id: 1, value: `first` }, { immediate: true }) + harness.sync.commit() + const inner = collection._deferPublication() + harness.sync.begin({ immediate: true }) + harness.sync.write({ + type: `update`, + value: { id: 1, value: `second` }, + }) + harness.sync.commit() + + inner.publish() + expect(callbacks).toEqual([]) + outer.publish() + expect(callbacks).toEqual([ + { changes: [`first`, `second`], visibleValue: `second` }, + ]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`starts a fresh publication after the previous one closes`, async () => { + const harness = createSyncHarness(`successive-publication-cycles`) + const { collection } = harness + const callbacks: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => callbacks.push(changes.map((change) => change.value.value)), + { includeInitialState: false }, + ) + + try { + const first = collection._deferPublication() + stageInsert(harness.sync, { id: 1, value: `first` }, { immediate: true }) + harness.sync.commit() + first.publish() + + const second = collection._deferPublication() + harness.sync.begin({ immediate: true }) + harness.sync.write({ + type: `update`, + value: { id: 1, value: `second` }, + }) + harness.sync.commit() + second.publish() + + expect(callbacks).toEqual([[`first`], [`second`]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`does not let a discarded deferral poison the next publication`, async () => { + const harness = createSyncHarness(`discarded-publication-cycle`) + const { collection } = harness + const callbacks: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => callbacks.push(changes.map((change) => change.value.value)), + { includeInitialState: false }, + ) + + try { + const discarded = collection._deferPublication() + stageInsert( + harness.sync, + { id: 1, value: `discarded` }, + { immediate: true }, + ) + harness.sync.commit() + discarded.discard() + expect(callbacks).toEqual([]) + + const published = collection._deferPublication() + harness.sync.begin({ immediate: true }) + harness.sync.write({ + type: `update`, + value: { id: 1, value: `published` }, + }) + harness.sync.commit() + published.publish() + expect(callbacks).toEqual([[`published`]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`lets a publication callback start the next publication cycle`, async () => { + const harness = createSyncHarness(`publication-cycle-from-callback`) + const { collection } = harness + const callbacks: Array<{ + changes: Array + visibleValue: string + revision: number + }> = [] + const initialRevision = collection._stateRevision + const write = (type: `insert` | `update`, value: string) => { + harness.sync.begin({ immediate: true }) + harness.sync.write({ type, value: { id: 1, value } }) + harness.sync.commit() + } + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map((change) => change.value.value), + visibleValue: collection.get(1)!.value, + revision: collection._stateRevision, + }) + + if (changes[0]?.value.value === `first`) { + const secondPublication = collection._deferPublication() + write(`update`, `second`) + secondPublication.publish() + } + }, + { includeInitialState: false }, + ) + + try { + const firstPublication = collection._deferPublication() + write(`insert`, `first`) + firstPublication.publish() + + expect(callbacks).toEqual([ + { + changes: [`first`], + visibleValue: `first`, + revision: initialRevision + 1, + }, + { + changes: [`second`], + visibleValue: `second`, + revision: initialRevision + 2, + }, + ]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`publishes an internal layout swap with unchanged endpoints`, async () => { + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-middle-swap`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + ops.begin({ immediate: true }) + for (let id = 1; id <= 5; id++) { + ops.write({ + type: `insert`, + value: { id, value: `value-${id}`, rank: id }, + }) + } + ops.commit() + ops.markReady() + }, + }, + }) + const callbacks: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled: false, + revision: collection._layoutRevision, + }) + }, + { includeInitialState: false }, + ) + + try { + const revisionBeforeSwap = collection._layoutRevision + sync.begin({ immediate: true }) + sync.write({ + type: `update`, + value: { id: 3, value: `value-3`, rank: 4 }, + }) + sync.write({ + type: `update`, + value: { id: 4, value: `value-4`, rank: 3 }, + }) + sync.collection._markLayoutChange() + expect(sync.commit()).toBe(true) + + expect([...collection.keys()]).toEqual([1, 2, 4, 3, 5]) + expect(collection._layoutRevision).toBe(revisionBeforeSwap + 1) + expect(callbacks).toEqual([ + { + changes: [3, 4], + keys: [1, 2, 4, 3, 5], + values: [`value-1`, `value-2`, `value-4`, `value-3`, `value-5`], + markedReceiptSettled: false, + revision: revisionBeforeSwap + 1, + }, + ]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`compares layout with the public state before an immediate prefix drain`, async () => { + const updatePersistence = createDeferred() + const insertPersistence = createDeferred() + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-prefix-drain`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + ops.begin({ immediate: true }) + ops.write({ + type: `insert`, + value: { id: 1, value: `one`, rank: 0 }, + }) + ops.write({ + type: `insert`, + value: { id: 2, value: `two`, rank: 1 }, + }) + ops.commit() + ops.markReady() + }, + }, + onUpdate: () => updatePersistence.promise, + onInsert: () => insertPersistence.promise, + }) + const callbacks: Array<{ + changes: Array + keys: Array + values: Array + }> = [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + }) + }, + { includeInitialState: false }, + ) + const update = collection.update(1, (draft) => { + draft.value = `optimistic-one` + }) + let insert: ReturnType | undefined + + try { + sync.begin() + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.collection._markLayoutChange() + const firstReceipt = sync.commit() + expect(firstReceipt).not.toBe(true) + + insert = collection.insert({ + id: 3, + value: `optimistic-three`, + rank: 3, + }) + callbacks.length = 0 + const revisionBeforeDrain = collection._layoutRevision + expect([...collection.keys()]).toEqual([1, 2, 3]) + + sync.begin({ immediate: true }) + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 0 }, + }) + sync.collection._markLayoutChange() + const secondReceipt = sync.commit() + + expect([...collection.keys()]).toEqual([1, 2, 3]) + expect(collection.toArray.map(({ value }) => value)).toEqual([ + `optimistic-one`, + `two`, + `optimistic-three`, + ]) + expect(callbacks).toEqual([]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain) + await Promise.all( + [firstReceipt, secondReceipt] + .filter((receipt) => receipt !== true) + .map((receipt) => receipt), + ) + + updatePersistence.resolve() + insertPersistence.resolve() + await Promise.all([ + update.isPersisted.promise, + insert.isPersisted.promise, + ]) + } finally { + updatePersistence.resolve() + insertPersistence.resolve() + await update.isPersisted.promise.catch(() => undefined) + await insert?.isPersisted.promise.catch(() => undefined) + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`uses the post-removal public layout before an unmarked prefix drain`, async () => { + const updatePersistence = createDeferred() + const deletePersistence = createDeferred() + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-prefix-removal-drain`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + installInitialOrderedRows(ops) + }, + }, + onUpdate: () => updatePersistence.promise, + onDelete: () => deletePersistence.promise, + }) + const callbacks: Array = [] + let parkedReceiptSettled = false + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled: parkedReceiptSettled, + revision: collection._layoutRevision, + }) + }, + { includeInitialState: false }, + ) + const update = collection.update(2, (draft) => { + draft.value = `optimistic-two` + }) + let deletion: ReturnType | undefined + + try { + sync.begin() + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.collection._markLayoutChange() + const parkedReceipt = sync.commit() + expect(parkedReceipt).not.toBe(true) + if (parkedReceipt !== true) { + void parkedReceipt.then(() => { + parkedReceiptSettled = true + }) + } + + deletion = collection.delete(1) + callbacks.length = 0 + const revisionBeforeDrain = collection._layoutRevision + await Promise.resolve() + expect(parkedReceiptSettled).toBe(false) + expect([...collection.keys()]).toEqual([2]) + + sync.begin({ immediate: true }) + sync.write({ + type: `update`, + value: { id: 2, value: `server-two`, rank: 1 }, + }) + const drainReceipt = sync.commit() + + expect(drainReceipt).toBe(true) + expect([...collection.keys()]).toEqual([2]) + expect(collection.toArray.map(({ value }) => value)).toEqual([ + `optimistic-two`, + ]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain) + expect(callbacks).toEqual([]) + if (parkedReceipt !== true) await parkedReceipt + expect(parkedReceiptSettled).toBe(true) + } finally { + subscription.unsubscribe() + updatePersistence.resolve() + deletePersistence.resolve() + await update.isPersisted.promise.catch(() => undefined) + await deletion?.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } + }) + + it(`publishes a parked layout mark when optimistic persistence drains it`, async () => { + const updatePersistence = createDeferred() + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-prefix-normal-drain`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + installInitialOrderedRows(ops) + }, + }, + onUpdate: () => updatePersistence.promise, + }) + let markedReceiptSettled = false + const callbacks: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled, + revision: collection._layoutRevision, + }) + }, + { includeInitialState: false }, + ) + const update = collection.update(2, (draft) => { + draft.value = `optimistic-two` + }) + + try { + sync.begin() + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.collection._markLayoutChange() + const receipt = sync.commit() + expect(receipt).not.toBe(true) + if (receipt !== true) { + void receipt.then(() => { + markedReceiptSettled = true + }) + } + + callbacks.length = 0 + const revisionBeforeDrain = collection._layoutRevision + await Promise.resolve() + expect(markedReceiptSettled).toBe(false) + expect([...collection.keys()]).toEqual([1, 2]) + + updatePersistence.resolve() + await update.isPersisted.promise + if (receipt !== true) await receipt + + expect([...collection.keys()]).toEqual([2, 1]) + expect(collection.toArray.map(({ value }) => value)).toEqual([ + `two`, + `one`, + ]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain + 1) + expect(callbacks).toEqual([ + { + changes: [1, 2], + keys: [2, 1], + values: [`two`, `one`], + markedReceiptSettled: false, + revision: revisionBeforeDrain + 1, + }, + ]) + expect(markedReceiptSettled).toBe(true) + } finally { + updatePersistence.resolve() + await update.isPersisted.promise.catch(() => undefined) + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it.each([`first`, `middle`, `last`, `first-and-middle`] as const)( + `honors %s layout marks in an immediate causal prefix`, + async (markPosition) => { + const updatePersistence = createDeferred() + const insertPersistence = createDeferred() + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-prefix-immediate-${markPosition}`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + installInitialOrderedRows(ops) + }, + }, + onUpdate: () => updatePersistence.promise, + onInsert: () => insertPersistence.promise, + }) + let firstReceiptSettled = false + const callbacks: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled: firstReceiptSettled, + revision: collection._layoutRevision, + }) + }, + { includeInitialState: false }, + ) + const update = collection.update(2, (draft) => { + draft.value = `optimistic-two` + }) + let insert: ReturnType | undefined + + try { + sync.begin() + sync.write({ + type: `update`, + value: + markPosition === `first` || markPosition === `first-and-middle` + ? { id: 1, value: `one`, rank: 2 } + : { id: 2, value: `server-two-a`, rank: 1 }, + }) + if (markPosition === `first` || markPosition === `first-and-middle`) { + sync.collection._markLayoutChange() + } + const firstReceipt = sync.commit() + expect(firstReceipt).not.toBe(true) + if (firstReceipt !== true) { + void firstReceipt.then(() => { + firstReceiptSettled = true + }) + } + await Promise.resolve() + expect(firstReceiptSettled).toBe(false) + + insert = collection.insert({ + id: 3, + value: `optimistic-three`, + rank: 3, + }) + + sync.begin() + sync.write({ + type: `update`, + value: + markPosition === `middle` + ? { id: 1, value: `one`, rank: 2 } + : { id: 2, value: `server-two-b`, rank: 1 }, + }) + if (markPosition === `middle` || markPosition === `first-and-middle`) { + sync.collection._markLayoutChange() + } + const middleReceipt = sync.commit() + expect(middleReceipt).not.toBe(true) + + callbacks.length = 0 + const revisionBeforeDrain = collection._layoutRevision + expect([...collection.keys()]).toEqual([1, 2, 3]) + + sync.begin({ immediate: true }) + sync.write({ + type: `update`, + value: + markPosition === `last` + ? { id: 1, value: `one`, rank: 2 } + : { id: 2, value: `server-two-c`, rank: 1 }, + }) + if (markPosition === `last`) sync.collection._markLayoutChange() + const lastReceipt = sync.commit() + + expect(lastReceipt).toBe(true) + expect([...collection.keys()]).toEqual([2, 1, 3]) + expect(collection.toArray.map(({ value }) => value)).toEqual([ + `optimistic-two`, + `one`, + `optimistic-three`, + ]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain + 1) + expect(callbacks).toEqual([ + { + changes: [1], + keys: [2, 1, 3], + values: [`optimistic-two`, `one`, `optimistic-three`], + markedReceiptSettled: false, + revision: revisionBeforeDrain + 1, + }, + ]) + + await Promise.all( + [firstReceipt, middleReceipt] + .filter((receipt) => receipt !== true) + .map((receipt) => receipt), + ) + expect(firstReceiptSettled).toBe(true) + } finally { + subscription.unsubscribe() + updatePersistence.resolve() + insertPersistence.resolve() + await update.isPersisted.promise.catch(() => undefined) + await insert?.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } + }, + ) + + it(`honors a parked layout mark when truncate drains its causal prefix`, async () => { + const updatePersistence = createDeferred() + const insertPersistence = createDeferred() + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-prefix-truncate-drain`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + installInitialOrderedRows(ops) + }, + }, + onUpdate: () => updatePersistence.promise, + onInsert: () => insertPersistence.promise, + }) + let markedReceiptSettled = false + const callbacks: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled, + revision: collection._layoutRevision, + }) + }, + { includeInitialState: false }, + ) + const update = collection.update(1, (draft) => { + draft.value = `optimistic-one` + }) + let insert: ReturnType | undefined + + try { + sync.begin() + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.collection._markLayoutChange() + const firstReceipt = sync.commit() + expect(firstReceipt).not.toBe(true) + if (firstReceipt !== true) { + void firstReceipt.then(() => { + markedReceiptSettled = true + }) + } + + insert = collection.insert({ + id: 3, + value: `optimistic-three`, + rank: 3, + }) + callbacks.length = 0 + const revisionBeforeDrain = collection._layoutRevision + expect([...collection.keys()]).toEqual([1, 2, 3]) + + sync.begin() + sync.truncate() + sync.write({ + type: `insert`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.write({ + type: `insert`, + value: { id: 2, value: `two`, rank: 1 }, + }) + const truncateReceipt = sync.commit() + + expect(truncateReceipt).toBe(true) + expect([...collection.keys()]).toEqual([2, 1, 3]) + expect(collection.toArray.map(({ value }) => value)).toEqual([ + `two`, + `optimistic-one`, + `optimistic-three`, + ]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain + 1) + expect(callbacks).toEqual([ + { + changes: [2, 1, 3, 1, 3, 1, 2], + keys: [2, 1, 3], + values: [`two`, `optimistic-one`, `optimistic-three`], + markedReceiptSettled: false, + revision: revisionBeforeDrain + 1, + }, + ]) + if (firstReceipt !== true) await firstReceipt + expect(markedReceiptSettled).toBe(true) + } finally { + subscription.unsubscribe() + updatePersistence.resolve() + insertPersistence.resolve() + await update.isPersisted.promise.catch(() => undefined) + await insert?.isPersisted.promise.catch(() => undefined) + await collection.cleanup() + } + }) + + it(`captures a fresh layout boundary for each reentrant causal prefix`, async () => { + let sync!: OrderedSync + const collection = createCollection({ + id: `layout-reentrant-prefixes`, + getKey: (row) => row.id, + compare: (left, right) => left.rank - right.rank, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + installInitialOrderedRows(ops) + }, + }, + }) + let listenerDepth = 0 + let maxListenerDepth = 0 + let queuedRestore = false + let innerReceipt: Promise | undefined + let innerReceiptSettled = false + const callbacks: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + listenerDepth++ + maxListenerDepth = Math.max(maxListenerDepth, listenerDepth) + callbacks.push({ + changes: changes.map(({ key }) => key as number), + keys: [...collection.keys()], + values: collection.toArray.map(({ value }) => value), + markedReceiptSettled: innerReceiptSettled, + revision: collection._layoutRevision, + }) + + if (!queuedRestore) { + queuedRestore = true + sync.begin() + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 0 }, + }) + sync.collection._markLayoutChange() + const receipt = sync.commit() + if (receipt === true) { + throw new Error(`Expected listener-created work to queue`) + } + innerReceipt = receipt + void receipt.then(() => { + innerReceiptSettled = true + }) + } + + listenerDepth-- + }, + { includeInitialState: false }, + ) + + try { + const revisionBeforeDrain = collection._layoutRevision + sync.begin({ immediate: true }) + sync.write({ + type: `update`, + value: { id: 1, value: `one`, rank: 2 }, + }) + sync.collection._markLayoutChange() + expect(sync.commit()).toBe(true) + + expect([...collection.keys()]).toEqual([1, 2]) + expect(collection._layoutRevision).toBe(revisionBeforeDrain + 2) + expect(callbacks).toEqual([ + { + changes: [1], + keys: [2, 1], + values: [`two`, `one`], + markedReceiptSettled: false, + revision: revisionBeforeDrain + 1, + }, + { + changes: [1], + keys: [1, 2], + values: [`one`, `two`], + markedReceiptSettled: false, + revision: revisionBeforeDrain + 2, + }, + ]) + expect(maxListenerDepth).toBe(1) + expect(innerReceipt).toBeDefined() + await innerReceipt + expect(innerReceiptSettled).toBe(true) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`preserves sync work opened by a listener until it is committed`, async () => { + const harness = createSyncHarness(`listener-opened-sync-work`) + const { collection } = harness + let openedInnerTransaction = false + const batches: Array> = [] + + const subscription = collection.subscribeChanges((changes) => { + batches.push(changes.map((change) => change.key as number)) + if (!openedInnerTransaction && changes.some(({ key }) => key === 1)) { + openedInnerTransaction = true + stageInsert(harness.sync, { id: 2, value: `inner` }) + } + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect(openedInnerTransaction).toBe(true) + expect(collection.get(2)).toBeUndefined() + + expect(() => harness.sync.commit()).not.toThrow() + expect(collection.get(2)).toMatchObject({ id: 2, value: `inner` }) + expect(batches).toEqual([[1], [2]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`publishes listener-committed sync work after the outer batch exactly once`, async () => { + const harness = createSyncHarness(`listener-committed-sync-work`) + const { collection } = harness + const appliedKeys: Array = [] + const originalSet = collection._state.syncedData.set.bind( + collection._state.syncedData, + ) + vi.spyOn(collection._state.syncedData, `set`).mockImplementation( + (key, value) => { + appliedKeys.push(key) + return originalSet(key, value) + }, + ) + const batches: Array> = [] + let listenerDepth = 0 + let maxListenerDepth = 0 + let committedInnerTransaction = false + + const subscription = collection.subscribeChanges((changes) => { + listenerDepth++ + maxListenerDepth = Math.max(maxListenerDepth, listenerDepth) + batches.push(changes.map((change) => change.key as number)) + + if (!committedInnerTransaction && changes.some(({ key }) => key === 1)) { + committedInnerTransaction = true + stageInsert(harness.sync, { id: 2, value: `inner` }) + harness.sync.commit() + } + + listenerDepth-- + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect(appliedKeys).toEqual([1, 2]) + expect(batches).toEqual([[1], [2]]) + expect(maxListenerDepth).toBe(1) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`keeps callback work FIFO across committed, aborted, and open transactions`, async () => { + const harness = createSyncHarness(`listener-sync-action-order`) + const { collection } = harness + const batches: Array> = [] + let ranListenerActions = false + + const subscription = collection.subscribeChanges((changes) => { + batches.push(changes.map((change) => change.key as number)) + if (ranListenerActions || !changes.some(({ key }) => key === 1)) return + ranListenerActions = true + + stageInsert(harness.sync, { id: 2, value: `left-open` }) + + stageInsert(harness.sync, { id: 3, value: `committed` }) + harness.sync.metadata!.row.set(3, { source: `listener` }) + harness.sync.metadata!.collection.set(`listener:commit`, 3) + harness.sync.commit() + + stageInsert(harness.sync, { id: 4, value: `aborted` }) + const controller = new AbortController() + controller.abort() + const abortedReceipt = harness.sync.commit(controller.signal) + if (abortedReceipt !== true) { + void abortedReceipt.catch(() => undefined) + } + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect(collection.get(3)).toMatchObject({ id: 3, value: `committed` }) + expect(collection.get(4)).toBeUndefined() + expect(collection._state.syncedMetadata.get(3)).toEqual({ + source: `listener`, + }) + expect( + collection._state.syncedCollectionMetadata.get(`listener:commit`), + ).toBe(3) + + harness.sync.commit() + expect(collection.get(2)).toMatchObject({ id: 2, value: `left-open` }) + expect(batches).toEqual([[1], [3], [2]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`drains listener-committed transactions in staging order`, async () => { + const harness = createSyncHarness(`listener-sync-fifo`) + const { collection } = harness + const batches: Array> = [] + let stagedInnerTransactions = false + + const subscription = collection.subscribeChanges((changes) => { + batches.push(changes.map((change) => change.key as number)) + if (stagedInnerTransactions || !changes.some(({ key }) => key === 1)) { + return + } + stagedInnerTransactions = true + + stageInsert(harness.sync, { id: 2, value: `first` }) + harness.sync.commit() + stageInsert(harness.sync, { id: 3, value: `second` }) + harness.sync.commit() + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect([...collection._state.syncedData.keys()]).toEqual([1, 2, 3]) + expect(batches).toEqual([[1], [2, 3]]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`drains callback work before surfacing a listener error`, async () => { + const harness = createSyncHarness(`throwing-sync-listener`) + const { collection } = harness + const failure = new Error(`listener failed`) + const appliedKeys: Array = [] + const originalSet = collection._state.syncedData.set.bind( + collection._state.syncedData, + ) + vi.spyOn(collection._state.syncedData, `set`).mockImplementation( + (key, value) => { + appliedKeys.push(key) + return originalSet(key, value) + }, + ) + let queuedReceipt: Promise | undefined + const subscription = collection.subscribeChanges((changes) => { + if (!changes.some(({ key }) => key === 1)) return + stageInsert(harness.sync, { id: 2, value: `queued` }) + const receipt = harness.sync.commit() + if (receipt === true) { + throw new Error(`Expected callback-created work to queue`) + } + queuedReceipt = receipt + throw failure + }) + + try { + stageInsert(harness.sync, { id: 1, value: `first` }) + expect(() => harness.sync.commit()).toThrow(failure) + expect(collection.get(1)).toMatchObject({ id: 1, value: `first` }) + expect(collection.get(2)).toMatchObject({ id: 2, value: `queued` }) + expect(queuedReceipt).toBeDefined() + await expect(queuedReceipt).resolves.toBeUndefined() + + stageInsert(harness.sync, { id: 3, value: `second` }) + expect(() => harness.sync.commit()).not.toThrow() + + expect(appliedKeys).toEqual([1, 2, 3]) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`queues a listener truncate until the outer publication finishes`, async () => { + const harness = createSyncHarness(`listener-sync-truncate`) + const { collection } = harness + const appliedKeys: Array = [] + const originalSet = collection._state.syncedData.set.bind( + collection._state.syncedData, + ) + vi.spyOn(collection._state.syncedData, `set`).mockImplementation( + (key, value) => { + appliedKeys.push(key) + return originalSet(key, value) + }, + ) + let stagedTruncate = false + const subscription = collection.subscribeChanges((changes) => { + if (stagedTruncate || !changes.some(({ key }) => key === 1)) return + stagedTruncate = true + harness.sync.begin() + harness.sync.truncate() + harness.sync.write({ + type: `insert`, + value: { id: 2, value: `replacement` }, + }) + harness.sync.commit() + }) + + try { + stageInsert(harness.sync, { id: 1, value: `outer` }) + harness.sync.commit() + + expect(appliedKeys).toEqual([1, 2]) + expect(collection.get(1)).toBeUndefined() + expect(collection.get(2)).toMatchObject({ + id: 2, + value: `replacement`, + }) + } finally { + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`releases subset demand from a publication callback without nested delivery`, async () => { + let sync!: SyncOps + const unloadSubset = vi.fn() + const collection = createCollection({ + id: `listener-subset-release-row-gc`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + ops.markReady() + return { + loadSubset: async () => { + stageInsert(ops, { id: 2, value: `owned` }) + const receipt = ops.commit() + if (receipt !== true) await receipt + return + }, + unloadSubset, + } + }, + }, + }) + let ownerUnsubscribed = false + const owner = collection.subscribeChanges((changes) => { + if (ownerUnsubscribed || !changes.some(({ key }) => key === 1)) return + ownerUnsubscribed = true + owner.unsubscribe() + }) + owner.requestSnapshot({ optimizedOnly: false }) + await flushPromises() + + const batches: Array> = [] + let listenerDepth = 0 + let maxListenerDepth = 0 + const observer = collection.subscribeChanges( + (changes) => { + listenerDepth++ + maxListenerDepth = Math.max(maxListenerDepth, listenerDepth) + batches.push(changes.map((change) => change.key as number)) + listenerDepth-- + }, + { includeInitialState: true }, + ) + batches.length = 0 + + try { + stageInsert(sync, { id: 1, value: `outer` }) + expect(() => sync.commit()).not.toThrow() + + expect(ownerUnsubscribed).toBe(true) + expect(unloadSubset).toHaveBeenCalledOnce() + expect(collection.get(2)).toMatchObject({ id: 2, value: `owned` }) + expect(batches).toEqual([[1]]) + expect(maxListenerDepth).toBe(1) + } finally { + owner.unsubscribe() + observer.unsubscribe() + await collection.cleanup() + } + }) + + it(`keeps normal listener sync work queued behind optimistic persistence`, async () => { + let sync!: SyncOps + const mutation = createDeferred() + const collection = createCollection({ + id: `listener-sync-with-optimistic-work`, + getKey: (row) => row.id, + startSync: true, + sync: { + sync: (ops) => { + sync = ops + ops.markReady() + }, + }, + onInsert: () => mutation.promise, + }) + const optimisticTransaction = collection.insert({ + id: 2, + value: `optimistic`, + }) + let stagedInnerTransaction = false + const subscription = collection.subscribeChanges((changes) => { + if (stagedInnerTransaction || !changes.some(({ key }) => key === 1)) { + return + } + stagedInnerTransaction = true + stageInsert(sync, { id: 3, value: `queued` }) + sync.commit() + }) + + try { + stageInsert(sync, { id: 1, value: `outer` }, { immediate: true }) + sync.commit() + + expect(stagedInnerTransaction).toBe(true) + expect(collection.get(3)).toBeUndefined() + + mutation.resolve() + await optimisticTransaction.isPersisted.promise + + expect(collection.get(3)).toMatchObject({ id: 3, value: `queued` }) + } finally { + mutation.resolve() + subscription.unsubscribe() + await collection.cleanup() + } + }) + + it(`matches every bounded reentrant listener history`, async () => { + for (const scenario of exhaustiveListenerScenarios) { + await runListenerScenario(scenario) + } + }) + + fcTest.prop([listenerScenarioArbitrary], { + numRuns: generatedRuns, + seed: 1774, + })(`matches the reentrant drain laws for a fixed seed`, runListenerScenario) + + fcTest.prop( + [listenerScenarioArbitrary], + oracleRandomParameters( + generatedRuns, + replay, + `collection-sync.reentrant-drain`, + ), + )( + `matches the reentrant drain laws for a random or replayed seed`, + runListenerScenario, + ) +}) diff --git a/packages/db/tests/collection.test.ts b/packages/db/tests/collection.test.ts index 3ff8ede815..568297f214 100644 --- a/packages/db/tests/collection.test.ts +++ b/packages/db/tests/collection.test.ts @@ -7,6 +7,7 @@ import { DuplicateKeySyncError, InvalidKeyError, KeyUpdateNotAllowedError, + LoadSubsetOperationAbortedError, MissingDeleteHandlerError, MissingInsertHandlerError, MissingUpdateHandlerError, @@ -2329,4 +2330,84 @@ describe(`Collection isLoadingSubset property`, () => { expect(result).toBe(true) expect(collection.isLoadingSubset).toBe(false) }) + + it(`rejects an already-aborted subset request before the adapter branch`, async () => { + const loadSubset = vi.fn(() => true as const) + const collection = createCollection<{ id: string; value: string }>({ + id: `already-aborted-subset-request`, + getKey: (item) => item.id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset } + }, + }, + }) + const request = new AbortController() + request.abort() + + await expect( + collection._sync.loadSubset({ signal: request.signal }), + ).rejects.toBeInstanceOf(LoadSubsetOperationAbortedError) + + expect(loadSubset).not.toHaveBeenCalled() + expect(collection.isLoadingSubset).toBe(false) + await collection.cleanup() + }) + + it(`rejects an already-aborted subset request before the eager return`, async () => { + const loadSubset = vi.fn(() => true as const) + const collection = createCollection<{ id: string; value: string }>({ + id: `already-aborted-eager-subset-request`, + getKey: (item) => item.id, + syncMode: `eager`, + startSync: true, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset } + }, + }, + }) + const request = new AbortController() + request.abort() + + await expect( + collection._sync.loadSubset({ signal: request.signal }), + ).rejects.toMatchObject({ name: `AbortError` }) + + expect(loadSubset).not.toHaveBeenCalled() + expect(collection.isLoadingSubset).toBe(false) + await collection.cleanup() + }) + + it(`rejects an already-aborted subset request before deferred start`, async () => { + const loadSubset = vi.fn(() => true as const) + const collection = createCollection<{ id: string; value: string }>({ + id: `already-aborted-deferred-subset-request`, + getKey: (item) => item.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const request = new AbortController() + request.abort() + + await expect( + collection._sync.loadSubset({ signal: request.signal }), + ).rejects.toMatchObject({ name: `AbortError` }) + + expect(loadSubset).not.toHaveBeenCalled() + expect(collection.isLoadingSubset).toBe(false) + collection._resumeSyncStart() + expect(loadSubset).not.toHaveBeenCalled() + await collection.cleanup() + }) }) diff --git a/packages/db/tests/comparison.property.test.ts b/packages/db/tests/comparison.property.test.ts index dd62790011..8b3fea8332 100644 --- a/packages/db/tests/comparison.property.test.ts +++ b/packages/db/tests/comparison.property.test.ts @@ -375,38 +375,53 @@ describe(`normalizeValue property-based tests`, () => { }) fcTest.prop([fc.uint8Array({ minLength: 0, maxLength: 128 })])( - `small Uint8Arrays normalize to string representation`, + `small Uint8Arrays normalize to a stable key`, (arr) => { const normalized = normalizeValue(arr) expect(typeof normalized).toBe(`string`) - expect(normalized).toMatch(/^__u8__/) + expect(normalized).toBe(normalizeValue(new Uint8Array(arr))) }, ) fcTest.prop([fc.uint8Array({ minLength: 129, maxLength: 200 })])( - `large Uint8Arrays are not normalized`, + `large Uint8Arrays normalize to a stable linear-size key`, (arr) => { const normalized = normalizeValue(arr) - expect(normalized).toBe(arr) + expect(typeof normalized).toBe(`string`) + expect(normalized).toBe(normalizeValue(new Uint8Array(arr))) + expect((normalized as string).length - arr.length).toBeLessThan(32) }, ) - fcTest.prop([fc.string()])(`strings pass through unchanged`, (str) => { - expect(normalizeValue(str)).toBe(str) - }) + fcTest.prop([fc.string()])( + `strings preserve equality after normalization`, + (str) => { + expect(normalizeValue(str)).toBe(normalizeValue(`${str}`)) + }, + ) fcTest.prop([fc.integer()])(`integers pass through unchanged`, (n) => { expect(normalizeValue(n)).toBe(n) }) fcTest.prop([fc.uint8Array({ minLength: 0, maxLength: 128 })])( - `normalization is idempotent for Uint8Arrays`, + `binary keys cannot collide with user strings`, (arr) => { - const normalized1 = normalizeValue(arr) - // For strings (which small arrays become), normalizing again should be identity - expect(normalizeValue(normalized1)).toBe(normalized1) + const normalized = normalizeValue(arr) + expect(normalizeValue(normalized)).not.toBe(normalized) }, ) + + fcTest(`reads binary keys from indexed bytes, not custom iteration`, () => { + const bytes = new Uint8Array([2]) + Object.defineProperty(bytes, Symbol.iterator, { + value: function* () { + yield 1 + }, + }) + + expect(normalizeValue(bytes)).toBe(normalizeValue(new Uint8Array([2]))) + }) }) describe(`areValuesEqual property-based tests`, () => { diff --git a/packages/db/tests/comparison.test.ts b/packages/db/tests/comparison.test.ts index ded56421c0..870b23978b 100644 --- a/packages/db/tests/comparison.test.ts +++ b/packages/db/tests/comparison.test.ts @@ -67,6 +67,21 @@ describe(`ascComparator - Temporal values`, () => { }) }) +describe(`ascComparator - symbols`, () => { + const opts = DEFAULT_COMPARE_OPTIONS + + it(`gives symbols a stable total order`, () => { + const first = Symbol(`group`) + const second = Symbol(`group`) + + expect(ascComparator(first, first, opts)).toBe(0) + expect(ascComparator(first, second, opts)).toBeLessThan(0) + expect(ascComparator(second, first, opts)).toBeGreaterThan(0) + expect(ascComparator(first, 1, opts)).toBeGreaterThan(0) + expect(ascComparator(1, first, opts)).toBeLessThan(0) + }) +}) + describe(`compareValues - NaN behavior`, () => { // NaN satisfies neither < nor >, so the fallback returns 0. In practice // gt/gte/lt/lte catch NaN via isUnorderable before reaching compareValues. diff --git a/packages/db/tests/cursor.property.test.ts b/packages/db/tests/cursor.property.test.ts index 04c2fa5368..45c45f8761 100644 --- a/packages/db/tests/cursor.property.test.ts +++ b/packages/db/tests/cursor.property.test.ts @@ -1,370 +1,190 @@ -import { describe, expect, it } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' -import { buildCursor } from '../src/utils/cursor' -import { Func, PropRef, Value } from '../src/query/ir' -import type { OrderBy, OrderByClause } from '../src/query/ir' -import type { CompareOptions } from '../src/query/builder/types' - -/** - * Property-based tests for cursor building - * - * Key properties: - * 1. Empty inputs return undefined - * 2. Single column produces simple gt/lt based on direction - * 3. Direction affects operator choice (asc = gt, desc = lt) - * 4. Determinism - same inputs always produce same output - * 5. Result structure is always valid - */ - -// Arbitraries for generating test data -const arbitraryDirection = fc.constantFrom(`asc`, `desc`) - -const arbitraryNulls = fc.constantFrom(`first`, `last`) - -const arbitraryStringSort = fc.constantFrom(`locale`, `lexical`) - -const arbitraryCompareOptions = fc.record({ - direction: arbitraryDirection, - nulls: arbitraryNulls, - stringSort: arbitraryStringSort, -}) as fc.Arbitrary - -const arbitraryPropRef = fc - .array(fc.string({ minLength: 1, maxLength: 10 }), { - minLength: 1, - maxLength: 3, - }) - .map((path) => new PropRef(path)) - -const arbitraryOrderByClause = fc - .tuple(arbitraryPropRef, arbitraryCompareOptions) - .map( - ([expr, compareOptions]): OrderByClause => ({ - expression: expr, - compareOptions, - }), - ) - -const arbitraryOrderBy = ( - minLength: number, - maxLength: number, -): fc.Arbitrary => - fc.array(arbitraryOrderByClause, { minLength, maxLength }) +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { PropRef } from '../src/query/ir.js' +import { buildCursor } from '../src/utils/cursor.js' +import { evaluateReferenceExpression } from './reference-expression.js' +import type { OrderBy } from '../src/query/ir.js' + +type Term = { + direction: `asc` | `desc` + nulls: `first` | `last` +} -const arbitraryValue = fc.oneof( - fc.string(), - fc.integer(), - fc.double({ noNaN: true }), - fc.boolean(), +const termArbitrary = fc.record({ + direction: fc.constantFrom(`asc`, `desc`), + nulls: fc.constantFrom(`first`, `last`), +}) +const valueArbitrary = fc.oneof( + fc.integer({ min: -2, max: 2 }), fc.constant(null), + fc.constant(undefined), ) -const arbitraryValues = ( - minLength: number, - maxLength: number, -): fc.Arbitrary> => - fc.array(arbitraryValue, { minLength, maxLength }) +function compareValue(left: unknown, right: unknown, term: Term): number { + if (left == null && right == null) return 0 + if (left == null) return term.nulls === `first` ? -1 : 1 + if (right == null) return term.nulls === `first` ? 1 : -1 + const compared = left === right ? 0 : left < right ? -1 : 1 + return term.direction === `asc` ? compared : -compared +} -// Helper to check if result is a Func -function isFunc(expr: unknown): expr is Func { - return expr instanceof Func +function compareTuple( + left: ReadonlyArray, + right: ReadonlyArray, + terms: ReadonlyArray, +): number { + for (let index = 0; index < terms.length; index++) { + const compared = compareValue(left[index], right[index], terms[index]!) + if (compared !== 0) return compared + } + return 0 } -// Helper to get operator name from Func -function getFuncName(expr: Func): string { - return expr.name +function orderBy(terms: ReadonlyArray): OrderBy { + return terms.map((compareOptions, index) => ({ + expression: new PropRef([`column${index}`]), + compareOptions, + })) } -// Helper to recursively count operators in an expression -function countOperators(expr: unknown, name: string): number { - if (!isFunc(expr)) return 0 - const selfCount = expr.name === name ? 1 : 0 - return ( - selfCount + - expr.args.reduce((sum, arg) => sum + countOperators(arg, name), 0) +function row(values: ReadonlyArray): Record { + return Object.fromEntries( + values.map((value, index) => [`column${index}`, value]), ) } -describe(`buildCursor property-based tests`, () => { - describe(`empty input handling`, () => { - fcTest.prop([arbitraryOrderBy(0, 5)])( - `returns undefined for empty values array`, - (orderBy) => { - const result = buildCursor(orderBy, []) - expect(result).toBeUndefined() - }, - ) - - fcTest.prop([arbitraryValues(0, 5)])( - `returns undefined for empty orderBy array`, - (values) => { - const result = buildCursor([], values) - expect(result).toBeUndefined() - }, - ) - - it(`returns undefined for both empty`, () => { - expect(buildCursor([], [])).toBeUndefined() - }) - }) - - describe(`single column cursor`, () => { - fcTest.prop([arbitraryOrderByClause, arbitraryValue])( - `produces a simple comparison for single column`, - (clause, value) => { - const result = buildCursor([clause], [value]) - - expect(result).toBeDefined() - expect(isFunc(result!)).toBe(true) - - // Should be either 'gt' or 'lt' based on direction - const func = result as Func - expect([`gt`, `lt`]).toContain(getFuncName(func)) - }, - ) - - fcTest.prop([ - arbitraryPropRef, - arbitraryNulls, - arbitraryStringSort, - arbitraryValue, - ])( - `ascending direction produces gt operator`, - (expr, nulls, stringSort, value) => { - const clause: OrderByClause = { - expression: expr, - compareOptions: { - direction: `asc`, - nulls: nulls as `first` | `last`, - stringSort: stringSort as `locale` | `lexical`, - }, - } - const result = buildCursor([clause], [value]) - - expect(result).toBeDefined() - expect(getFuncName(result as Func)).toBe(`gt`) - }, - ) - - fcTest.prop([ - arbitraryPropRef, - arbitraryNulls, - arbitraryStringSort, - arbitraryValue, - ])( - `descending direction produces lt operator`, - (expr, nulls, stringSort, value) => { - const clause: OrderByClause = { - expression: expr, - compareOptions: { - direction: `desc`, - nulls: nulls as `first` | `last`, - stringSort: stringSort as `locale` | `lexical`, - }, - } - const result = buildCursor([clause], [value]) - - expect(result).toBeDefined() - expect(getFuncName(result as Func)).toBe(`lt`) - }, - ) - }) - - describe(`multi-column cursor structure`, () => { - fcTest.prop([arbitraryOrderBy(2, 4), arbitraryValues(2, 4)])( - `multi-column produces or at top level when matching lengths`, - (orderBy, values) => { - // Ensure we have matching lengths for a valid multi-column cursor - const minLen = Math.min(orderBy.length, values.length) - if (minLen < 2) return // Skip if not enough for multi-column - - const trimmedOrderBy = orderBy.slice(0, minLen) - const trimmedValues = values.slice(0, minLen) - - const result = buildCursor(trimmedOrderBy, trimmedValues) - - expect(result).toBeDefined() - expect(isFunc(result!)).toBe(true) - - // For 2+ columns, top level should be 'or' - const func = result as Func - expect(getFuncName(func)).toBe(`or`) - }, +function expectCursorDenotation( + terms: ReadonlyArray, + boundary: ReadonlyArray, + candidate: ReadonlyArray, +): void { + if (terms.length !== 1 || boundary.length !== 1) { + expect(() => buildCursor(orderBy(terms), [...boundary])).toThrow( + `Only single-column cursors are supported`, ) + return + } + const length = Math.min(terms.length, boundary.length) + const usedTerms = terms.slice(0, length) + const usedBoundary = boundary.slice(0, length) + const cursor = buildCursor(orderBy(terms), [...boundary]) + expect(cursor).toBeDefined() + expect(Boolean(evaluateReferenceExpression(cursor!, row(candidate)))).toBe( + compareTuple(candidate, usedBoundary, usedTerms) > 0, + ) +} - fcTest.prop([ - fc.tuple(arbitraryOrderByClause, arbitraryOrderByClause), - fc.tuple(arbitraryValue, arbitraryValue), - ])( - `two columns produces correct structure`, - ([clause1, clause2], [val1, val2]) => { - const result = buildCursor([clause1, clause2], [val1, val2]) - - expect(result).toBeDefined() - const func = result as Func - - // Top level should be 'or' - expect(getFuncName(func)).toBe(`or`) - - // Should have structure: or(comparison1, and(eq, comparison2)) - expect(func.args).toHaveLength(2) - - // First arg should be direct gt/lt - expect(isFunc(func.args[0])).toBe(true) - expect([`gt`, `lt`]).toContain(getFuncName(func.args[0] as Func)) - - // Second arg should be 'and' combining eq and comparison - expect(isFunc(func.args[1])).toBe(true) - expect(getFuncName(func.args[1] as Func)).toBe(`and`) - }, - ) - }) - - describe(`determinism`, () => { - fcTest.prop([arbitraryOrderBy(1, 3), arbitraryValues(1, 3)])( - `buildCursor is deterministic`, - (orderBy, values) => { - const result1 = buildCursor(orderBy, values) - const result2 = buildCursor(orderBy, values) - - // Both should be defined or both undefined - expect(result1 === undefined).toBe(result2 === undefined) - - if (result1 !== undefined && result2 !== undefined) { - // Compare structure by JSON representation - expect(JSON.stringify(result1)).toBe(JSON.stringify(result2)) - } +// Keep the nullable mixed-direction ordering law at the retained production +// snapshot boundary even though direct composite cursor construction is removed. +async function expectLocalTupleOrder( + terms: ReadonlyArray, + boundary: ReadonlyArray, + candidate: ReadonlyArray, +): Promise { + const collection = createCollection<{ id: string; [key: string]: unknown }>({ + getKey: (value) => value.id, + autoIndex: `off`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { ...row(candidate), id: `candidate` } }) + write({ type: `insert`, value: { ...row(boundary), id: `boundary` } }) + commit() + markReady() }, - ) + }, }) + try { + await collection.preload() + const expected = + compareTuple(candidate, boundary, terms) >= 0 + ? [`boundary`, `candidate`] + : [`candidate`, `boundary`] + for (const limit of [1, 2]) { + expect( + collection + .currentStateAsChanges({ + orderBy: [ + ...orderBy(terms), + { + expression: new PropRef([`id`]), + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `lexical`, + }, + }, + ], + limit, + }) + ?.map(({ key }) => key), + ).toEqual(expected.slice(0, limit)) + } + } finally { + await collection.cleanup() + } +} - describe(`value preservation`, () => { - fcTest.prop([arbitraryOrderByClause, arbitraryValue])( - `cursor contains the provided value`, - (clause, value) => { - const result = buildCursor([clause], [value]) - - expect(result).toBeDefined() - const func = result as Func - - // Second argument should be a Value containing our value - expect(func.args[1]).toBeInstanceOf(Value) - expect((func.args[1] as Value).value).toBe(value) - }, - ) - - fcTest.prop([arbitraryOrderByClause, arbitraryValue])( - `cursor references the correct property`, - (clause, value) => { - const result = buildCursor([clause], [value]) +const exactCursorArbitrary = fc + .integer({ min: 1, max: 4 }) + .chain((length) => + fc.tuple( + fc.array(termArbitrary, { minLength: length, maxLength: length }), + fc.array(valueArbitrary, { minLength: length, maxLength: length }), + fc.array(valueArbitrary, { minLength: length, maxLength: length }), + ), + ) - expect(result).toBeDefined() - const func = result as Func +const partialCursorArbitrary = fc + .tuple( + fc.array(termArbitrary, { minLength: 1, maxLength: 4 }), + fc.array(valueArbitrary, { minLength: 1, maxLength: 4 }), + fc.array(valueArbitrary, { minLength: 4, maxLength: 4 }), + ) + .filter(([terms, boundary]) => terms.length !== boundary.length) - // First argument should be the same PropRef - expect(func.args[0]).toBeInstanceOf(PropRef) - expect((func.args[0] as PropRef).path).toEqual( - (clause.expression as PropRef).path, - ) - }, +describe(`buildCursor properties`, () => { + it(`returns no cursor without boundary values and rejects a boundary without an order`, () => { + expect(() => buildCursor([], [1])).toThrow( + `Only single-column cursors are supported`, ) + expect(buildCursor([], [])).toBeUndefined() + expect( + buildCursor(orderBy([{ direction: `asc`, nulls: `first` }]), []), + ).toBeUndefined() }) - describe(`length mismatch handling`, () => { - fcTest.prop([arbitraryOrderBy(3, 5), arbitraryValues(1, 2)])( - `handles more orderBy columns than values gracefully`, - (orderBy, values) => { - // Should use the minimum of the two lengths - const result = buildCursor(orderBy, values) - - if (values.length === 0) { - expect(result).toBeUndefined() - } else { - expect(result).toBeDefined() - expect(isFunc(result!)).toBe(true) - } - }, - ) + fcTest.prop([exactCursorArbitrary], { numRuns: 300 })( + `preserves nullable mixed-direction ordering while restricting cursor width`, + async ([terms, boundary, candidate]) => { + expectCursorDenotation(terms, boundary, candidate) + await expectLocalTupleOrder(terms, boundary, candidate) + }, + ) - fcTest.prop([arbitraryOrderBy(1, 2), arbitraryValues(3, 5)])( - `handles more values than orderBy columns gracefully`, - (orderBy, values) => { - // Should use the minimum of the two lengths - const result = buildCursor(orderBy, values) + fcTest.prop([partialCursorArbitrary], { numRuns: 200 })( + `rejects mismatched cursor widths without restricting local tuple ordering`, + async ([terms, boundary, candidate]) => { + expectCursorDenotation(terms, boundary, candidate) + await expectLocalTupleOrder(terms, boundary, candidate) + }, + ) - if (orderBy.length === 0) { - expect(result).toBeUndefined() - } else { - expect(result).toBeDefined() - expect(isFunc(result!)).toBe(true) + fcTest.prop([exactCursorArbitrary], { numRuns: 100 })( + `repeats the same cursor or unsupported-width error`, + ([terms, boundary]) => { + if (terms.length !== 1) { + for (let attempt = 0; attempt < 2; attempt++) { + expect(() => buildCursor(orderBy(terms), [...boundary])).toThrow( + `Only single-column cursors are supported`, + ) } - }, - ) - }) - - describe(`operator consistency`, () => { - fcTest.prop([ - fc.array( - fc.tuple(arbitraryPropRef, arbitraryNulls, arbitraryStringSort), - { minLength: 2, maxLength: 4 }, - ), - arbitraryValues(2, 4), - ])(`all ascending columns use gt operators`, (clauseParts, values) => { - const orderBy: OrderBy = clauseParts.map(([expr, nulls, stringSort]) => ({ - expression: expr, - compareOptions: { - direction: `asc` as const, - nulls: nulls as `first` | `last`, - stringSort: stringSort as `locale` | `lexical`, - }, - })) - - const minLen = Math.min(orderBy.length, values.length) - const result = buildCursor( - orderBy.slice(0, minLen), - values.slice(0, minLen), - ) - - if (result) { - // Count gt operators - should equal number of columns - const gtCount = countOperators(result, `gt`) - expect(gtCount).toBe(minLen) - // Should have no lt operators - const ltCount = countOperators(result, `lt`) - expect(ltCount).toBe(0) + return } - }) - - fcTest.prop([ - fc.array( - fc.tuple(arbitraryPropRef, arbitraryNulls, arbitraryStringSort), - { minLength: 2, maxLength: 4 }, - ), - arbitraryValues(2, 4), - ])(`all descending columns use lt operators`, (clauseParts, values) => { - const orderBy: OrderBy = clauseParts.map(([expr, nulls, stringSort]) => ({ - expression: expr, - compareOptions: { - direction: `desc` as const, - nulls: nulls as `first` | `last`, - stringSort: stringSort as `locale` | `lexical`, - }, - })) - - const minLen = Math.min(orderBy.length, values.length) - const result = buildCursor( - orderBy.slice(0, minLen), - values.slice(0, minLen), + expect(buildCursor(orderBy(terms), [...boundary])).toEqual( + buildCursor(orderBy(terms), [...boundary]), ) - - if (result) { - // Count lt operators - should equal number of columns - const ltCount = countOperators(result, `lt`) - expect(ltCount).toBe(minLen) - // Should have no gt operators - const gtCount = countOperators(result, `gt`) - expect(gtCount).toBe(0) - } - }) - }) + }, + ) }) diff --git a/packages/db/tests/cursor.test.ts b/packages/db/tests/cursor.test.ts index 5d8f4a2f47..1b269bbf54 100644 --- a/packages/db/tests/cursor.test.ts +++ b/packages/db/tests/cursor.test.ts @@ -1,226 +1,99 @@ import { describe, expect, it } from 'vitest' -import { buildCursor } from '../src/utils/cursor.js' -import { Func, PropRef, Value } from '../src/query/ir.js' -import type { OrderBy, OrderByClause } from '../src/query/ir.js' -import type { CompareOptions } from '../src/query/builder/types.js' - -// Helper to create an OrderByClause for testing -function createOrderByClause( - path: string, - direction: `asc` | `desc`, -): OrderByClause { - const compareOptions: CompareOptions = { - direction, - nulls: direction === `asc` ? `first` : `last`, - } - return { - expression: new PropRef([`t`, path]), - compareOptions, - } +import { PropRef } from '../src/query/ir.js' +import { buildCursor, canExpressCursorOrder } from '../src/utils/cursor.js' +import { evaluateReferenceExpression } from './reference-expression.js' +import type { OrderBy } from '../src/query/ir.js' + +function orderBy( + ...terms: ReadonlyArray +): OrderBy { + return terms.map(([path, direction, nulls]) => ({ + expression: new PropRef([path]), + compareOptions: { direction, nulls }, + })) } -// Helper to check if a Func has the expected structure -function isFuncWithName(expr: unknown, name: string): expr is Func { - return expr instanceof Func && expr.name === name +function matches( + order: OrderBy, + boundary: Array, + row: object, +): boolean { + const cursor = buildCursor(order, boundary) + if (!cursor) throw new Error(`expected a cursor`) + return Boolean(evaluateReferenceExpression(cursor, row)) } describe(`buildCursor`, () => { - describe(`edge cases`, () => { - it(`returns undefined for empty values array`, () => { - const orderBy: OrderBy = [createOrderByClause(`col1`, `asc`)] - expect(buildCursor(orderBy, [])).toBeUndefined() - }) - - it(`returns undefined for empty orderBy array`, () => { - expect(buildCursor([], [1, 2, 3])).toBeUndefined() - }) - - it(`returns undefined for both empty`, () => { - expect(buildCursor([], [])).toBeUndefined() - }) + it(`uses direction for one non-null term`, () => { + expect(matches(orderBy([`rank`, `asc`, `first`]), [10], { rank: 11 })).toBe( + true, + ) + expect(matches(orderBy([`rank`, `asc`, `first`]), [10], { rank: 9 })).toBe( + false, + ) + expect(matches(orderBy([`rank`, `desc`, `first`]), [10], { rank: 9 })).toBe( + true, + ) + expect( + matches(orderBy([`rank`, `desc`, `first`]), [10], { rank: 11 }), + ).toBe(false) }) - describe(`single column`, () => { - it(`produces gt() for ASC direction`, () => { - const orderBy: OrderBy = [createOrderByClause(`col1`, `asc`)] - const result = buildCursor(orderBy, [10]) - - expect(result).toBeInstanceOf(Func) - expect(isFuncWithName(result, `gt`)).toBe(true) - - const func = result as Func - expect(func.args).toHaveLength(2) - expect(func.args[0]).toBeInstanceOf(PropRef) - expect((func.args[0] as PropRef).path).toEqual([`t`, `col1`]) - expect(func.args[1]).toBeInstanceOf(Value) - expect((func.args[1] as Value).value).toBe(10) - }) - - it(`produces lt() for DESC direction`, () => { - const orderBy: OrderBy = [createOrderByClause(`col1`, `desc`)] - const result = buildCursor(orderBy, [10]) - - expect(result).toBeInstanceOf(Func) - expect(isFuncWithName(result, `lt`)).toBe(true) - - const func = result as Func - expect(func.args).toHaveLength(2) - expect(func.args[0]).toBeInstanceOf(PropRef) - expect((func.args[0] as PropRef).path).toEqual([`t`, `col1`]) - expect(func.args[1]).toBeInstanceOf(Value) - expect((func.args[1] as Value).value).toBe(10) - }) - - it(`handles string cursor values`, () => { - const orderBy: OrderBy = [createOrderByClause(`name`, `asc`)] - const result = buildCursor(orderBy, [`alice`]) + it(`places nullish values according to the term`, () => { + const nullsFirst = orderBy([`rank`, `asc`, `first`]) + expect(matches(nullsFirst, [null], { rank: 0 })).toBe(true) + expect(matches(nullsFirst, [null], { rank: undefined })).toBe(false) - expect(result).toBeInstanceOf(Func) - const func = result as Func - expect(func.args[1]).toBeInstanceOf(Value) - expect((func.args[1] as Value).value).toBe(`alice`) - }) - - it(`handles null cursor values`, () => { - const orderBy: OrderBy = [createOrderByClause(`col1`, `asc`)] - const result = buildCursor(orderBy, [null]) - - expect(result).toBeInstanceOf(Func) - const func = result as Func - expect((func.args[1] as Value).value).toBeNull() - }) + const nullsLast = orderBy([`rank`, `asc`, `last`]) + expect(matches(nullsLast, [0], { rank: null })).toBe(true) + expect(matches(nullsLast, [null], { rank: 0 })).toBe(false) }) - describe(`multi-column composite cursor`, () => { - it(`produces or(gt(col1), and(eq(col1), gt(col2))) for two ASC columns`, () => { - const orderBy: OrderBy = [ - createOrderByClause(`col1`, `asc`), - createOrderByClause(`col2`, `asc`), - ] - const result = buildCursor(orderBy, [10, 20]) - - // Should be: or(gt(col1, 10), and(eq(col1, 10), gt(col2, 20))) - expect(result).toBeInstanceOf(Func) - expect(isFuncWithName(result, `or`)).toBe(true) - - const orFunc = result as Func - expect(orFunc.args).toHaveLength(2) - - // First arg: gt(col1, 10) - const gtCol1 = orFunc.args[0] - expect(isFuncWithName(gtCol1, `gt`)).toBe(true) - expect((gtCol1 as Func).args[0]).toBeInstanceOf(PropRef) - expect(((gtCol1 as Func).args[0] as PropRef).path).toEqual([`t`, `col1`]) - expect(((gtCol1 as Func).args[1] as Value).value).toBe(10) - - // Second arg: and(eq(col1, 10), gt(col2, 20)) - const andClause = orFunc.args[1] - expect(isFuncWithName(andClause, `and`)).toBe(true) - const andFunc = andClause as Func - expect(andFunc.args).toHaveLength(2) - - // eq(col1, 10) - expect(isFuncWithName(andFunc.args[0], `eq`)).toBe(true) - const eqCol1 = andFunc.args[0] as Func - expect((eqCol1.args[0] as PropRef).path).toEqual([`t`, `col1`]) - expect((eqCol1.args[1] as Value).value).toBe(10) - - // gt(col2, 20) - expect(isFuncWithName(andFunc.args[1], `gt`)).toBe(true) - const gtCol2 = andFunc.args[1] as Func - expect((gtCol2.args[0] as PropRef).path).toEqual([`t`, `col2`]) - expect((gtCol2.args[1] as Value).value).toBe(20) - }) + it(`rejects composite cursors with mixed-direction terms`, () => { + const order = orderBy([`group`, `asc`, `first`], [`rank`, `desc`, `last`]) - it(`handles mixed ASC/DESC directions`, () => { - const orderBy: OrderBy = [ - createOrderByClause(`col1`, `asc`), - createOrderByClause(`col2`, `desc`), - ] - const result = buildCursor(orderBy, [10, 20]) - - // Should be: or(gt(col1, 10), and(eq(col1, 10), lt(col2, 20))) - expect(isFuncWithName(result, `or`)).toBe(true) - - const orFunc = result as Func - const andClause = orFunc.args[1] as Func - - // Second column should use lt() for DESC - expect(isFuncWithName(andClause.args[1], `lt`)).toBe(true) - }) - - it(`handles three columns correctly`, () => { - const orderBy: OrderBy = [ - createOrderByClause(`col1`, `asc`), - createOrderByClause(`col2`, `asc`), - createOrderByClause(`col3`, `desc`), - ] - const result = buildCursor(orderBy, [1, 2, 3]) - - // Should be: or( - // gt(col1, 1), - // and(eq(col1, 1), gt(col2, 2)), - // and(eq(col1, 1), eq(col2, 2), lt(col3, 3)) - // ) - expect(isFuncWithName(result, `or`)).toBe(true) - - const outerOr = result as Func - // The structure is: or(or(gt, and), and) due to reduce - expect(outerOr.args).toHaveLength(2) - - // First arg is or(gt(col1, 1), and(eq(col1, 1), gt(col2, 2))) - const innerOr = outerOr.args[0] - expect(isFuncWithName(innerOr, `or`)).toBe(true) - - // Second arg is and(and(eq(col1, 1), eq(col2, 2)), lt(col3, 3)) - const thirdClause = outerOr.args[1] - expect(isFuncWithName(thirdClause, `and`)).toBe(true) - - // The innermost and should have eq conditions and lt for col3 - const innerAnd = thirdClause as Func - // Due to reduce, the structure is nested: and(and(eq, eq), lt) - expect(isFuncWithName(innerAnd.args[1], `lt`)).toBe(true) - const ltCol3 = innerAnd.args[1] as Func - expect((ltCol3.args[0] as PropRef).path).toEqual([`t`, `col3`]) - expect((ltCol3.args[1] as Value).value).toBe(3) - }) + expect(() => buildCursor(order, [1, 10])).toThrow( + `Only single-column cursors are supported`, + ) + expect(canExpressCursorOrder(order, [1, 10])).toBe(false) }) - describe(`partial values`, () => { - it(`handles fewer values than orderBy columns`, () => { - const orderBy: OrderBy = [ - createOrderByClause(`col1`, `asc`), - createOrderByClause(`col2`, `asc`), - createOrderByClause(`col3`, `asc`), - ] - const result = buildCursor(orderBy, [10, 20]) - - // Should only use first two columns - expect(isFuncWithName(result, `or`)).toBe(true) - - const orFunc = result as Func - expect(orFunc.args).toHaveLength(2) - - // First clause: gt(col1, 10) - expect(isFuncWithName(orFunc.args[0], `gt`)).toBe(true) - - // Second clause: and(eq(col1, 10), gt(col2, 20)) - expect(isFuncWithName(orFunc.args[1], `and`)).toBe(true) - }) - - it(`handles single value for multi-column orderBy`, () => { - const orderBy: OrderBy = [ - createOrderByClause(`col1`, `asc`), - createOrderByClause(`col2`, `asc`), - ] - const result = buildCursor(orderBy, [10]) - - // Should just be gt(col1, 10) since only one value provided - expect(isFuncWithName(result, `gt`)).toBe(true) + it(`rejects partial composite cursors instead of silently dropping terms`, () => { + const order = orderBy([`first`, `asc`, `first`], [`second`, `asc`, `first`]) + expect(() => buildCursor(order, [1])).toThrow( + `Only single-column cursors are supported`, + ) + expect(canExpressCursorOrder(order, [1])).toBe(false) + }) - const gtFunc = result as Func - expect((gtFunc.args[0] as PropRef).path).toEqual([`t`, `col1`]) - expect((gtFunc.args[1] as Value).value).toBe(10) - }) + it(`rejects cursor pushdown when predicates cannot express the order`, () => { + const localeOrder: OrderBy = [ + { + expression: new PropRef([`label`]), + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `locale`, + localeOptions: { numeric: true }, + }, + }, + ] + + expect(canExpressCursorOrder(localeOrder, [`item2`])).toBe(false) + expect( + canExpressCursorOrder( + [ + { + ...localeOrder[0]!, + compareOptions: { + ...localeOrder[0]!.compareOptions, + stringSort: `lexical`, + }, + }, + ], + [`item2`], + ), + ).toBe(true) + expect(canExpressCursorOrder(localeOrder, [{ rank: 1 }])).toBe(false) }) }) diff --git a/packages/db/tests/d2-source-reconciliation-oracle.property.test.ts b/packages/db/tests/d2-source-reconciliation-oracle.property.test.ts new file mode 100644 index 0000000000..3d1b29cc8d --- /dev/null +++ b/packages/db/tests/d2-source-reconciliation-oracle.property.test.ts @@ -0,0 +1,830 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { expect, it } from 'vitest' +import { + createCollection, + createLiveQueryCollection, + eq, +} from '../src/index.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { createEffect } from '../src/query/effect.js' +import { reconcileChangesForD2 } from '../src/query/live/utils.js' +import { oraclePropertyOptions, oracleRuns } from './oracle-config.js' +import { flushPromises } from './utils.js' +import type { ChangeMessage, SyncConfig } from '../src/types.js' + +type SourceRow = { + id: number + revision: number + value: number +} + +type SourceSyncActions = Parameters[`sync`]>[0] + +type SourceKey = string | number + +type SourceOperation = + | { + type: `upsert` + key: SourceKey + row: SourceRow + reportedPreviousValue: SourceRow + } + | { + type: `rawUpdate` + key: SourceKey + row: SourceRow + reportedPreviousValue: SourceRow + } + | { type: `replay`; key: SourceKey } + | { type: `delete`; key: SourceKey; reportedValue: SourceRow } + +type ReconciliationStep = + | { type: `batch`; operations: ReadonlyArray } + | { type: `truncate` } + | { type: `teardown` } + | { type: `restart` } + +type ReconciliationModel = { + sourceRows: Map + sentRows: Map + relation: Map + graphActive: boolean +} + +const sourceRowArbitrary = fc.record({ + id: fc.integer({ min: 0, max: 3 }), + revision: fc.integer({ min: 0, max: 4 }), + value: fc.integer({ min: -2, max: 2 }), +}) + +const sourceKeyArbitrary: fc.Arbitrary = fc.oneof( + fc.integer({ min: 0, max: 2 }), + fc.constantFrom(`0`, `1`, `source`), +) + +const sourceOperationArbitrary: fc.Arbitrary = fc.oneof( + fc + .record({ + key: sourceKeyArbitrary, + row: sourceRowArbitrary, + reportedPreviousValue: sourceRowArbitrary, + }) + .map((operation) => ({ type: `upsert` as const, ...operation })), + fc + .record({ + key: sourceKeyArbitrary, + row: sourceRowArbitrary, + reportedPreviousValue: sourceRowArbitrary, + }) + .map((operation) => ({ type: `rawUpdate` as const, ...operation })), + sourceKeyArbitrary.map((key) => ({ type: `replay` as const, key })), + fc + .record({ + key: sourceKeyArbitrary, + reportedValue: sourceRowArbitrary, + }) + .map((operation) => ({ type: `delete` as const, ...operation })), +) + +const reconciliationStepArbitrary: fc.Arbitrary = fc.oneof( + { + weight: 8, + arbitrary: fc + .array(sourceOperationArbitrary, { minLength: 1, maxLength: 5 }) + .map((operations) => ({ type: `batch` as const, operations })), + }, + { weight: 1, arbitrary: fc.constant({ type: `truncate` as const }) }, + { weight: 1, arbitrary: fc.constant({ type: `teardown` as const }) }, + { weight: 1, arbitrary: fc.constant({ type: `restart` as const }) }, +) + +const reconciliationHistoryArbitrary = fc.array(reconciliationStepArbitrary, { + minLength: 1, + maxLength: 30, +}) + +function sourceOperationForKeyArbitrary( + key: SourceKey, +): fc.Arbitrary { + return fc.oneof( + fc + .tuple(sourceRowArbitrary, sourceRowArbitrary) + .map(([row, reportedPreviousValue]) => ({ + type: `upsert` as const, + key, + row, + reportedPreviousValue, + })), + fc + .tuple(sourceRowArbitrary, sourceRowArbitrary) + .map(([row, reportedPreviousValue]) => ({ + type: `rawUpdate` as const, + key, + row, + reportedPreviousValue, + })), + fc.constant({ type: `replay` as const, key }), + sourceRowArbitrary.map((reportedValue) => ({ + type: `delete` as const, + key, + reportedValue, + })), + ) +} + +const disjointHistoriesArbitrary = fc.tuple( + fc.array(sourceOperationForKeyArbitrary(0), { + minLength: 1, + maxLength: 8, + }), + fc.array(sourceOperationForKeyArbitrary(`other`), { + minLength: 1, + maxLength: 8, + }), +) + +function rowIdentity(row: SourceRow): string { + return `${row.id}:${row.revision}:${row.value}` +} + +function expectedWeightedRowIdentity(key: SourceKey, row: SourceRow): string { + const sourceIdentity = [typeof key, String(key)].join(`:`) + const payloadIdentity = [row.id, row.revision, row.value] + .map(String) + .join(`:`) + return `${sourceIdentity}|${payloadIdentity}` +} + +function addWeight( + relation: Map, + key: SourceKey, + row: SourceRow, + weight: 1 | -1, +): void { + const identity = `${typeof key}:${String(key)}|${rowIdentity(row)}` + const nextWeight = (relation.get(identity) ?? 0) + weight + if (nextWeight === 0) relation.delete(identity) + else relation.set(identity, nextWeight) +} + +function applyToRelation( + relation: Map, + changes: ReadonlyArray>, +): void { + for (const change of changes) { + if (change.type === `insert`) { + addWeight(relation, change.key, change.value, 1) + } else if (change.type === `update`) { + addWeight(relation, change.key, change.previousValue!, -1) + addWeight(relation, change.key, change.value, 1) + } else { + addWeight(relation, change.key, change.value, -1) + } + } +} + +function sourceChangesFor( + operations: ReadonlyArray, + sourceRows: Map, +): Array> { + const changes: Array> = [] + for (const operation of operations) { + if (operation.type === `upsert`) { + const previousValue = sourceRows.get(operation.key) + changes.push( + previousValue === undefined + ? { type: `insert`, key: operation.key, value: operation.row } + : { + type: `update`, + key: operation.key, + value: operation.row, + previousValue: operation.reportedPreviousValue, + }, + ) + sourceRows.set(operation.key, operation.row) + } else if (operation.type === `rawUpdate`) { + changes.push({ + type: `update`, + key: operation.key, + value: operation.row, + previousValue: operation.reportedPreviousValue, + }) + sourceRows.set(operation.key, operation.row) + } else if (operation.type === `replay`) { + const row = sourceRows.get(operation.key) + if (row !== undefined) { + changes.push({ type: `insert`, key: operation.key, value: row }) + } + } else { + changes.push({ + type: `delete`, + key: operation.key, + value: operation.reportedValue, + }) + sourceRows.delete(operation.key) + } + } + return changes +} + +function expectTrackerMatchesSource( + sourceRows: ReadonlyMap, + sentRows: ReadonlyMap, +): void { + const compareEntries = ( + [a]: readonly [SourceKey, SourceRow], + [b]: readonly [SourceKey, SourceRow], + ) => `${typeof a}:${String(a)}`.localeCompare(`${typeof b}:${String(b)}`) + expect([...sentRows.entries()].sort(compareEntries)).toEqual( + [...sourceRows.entries()].sort(compareEntries), + ) +} + +function expectWeightedRelationMatchesSource( + sourceRows: ReadonlyMap, + relation: ReadonlyMap, +): void { + expect( + [...relation.entries()].sort(([a], [b]) => a.localeCompare(b)), + ).toEqual( + [...sourceRows.entries()] + .map(([key, row]) => [expectedWeightedRowIdentity(key, row), 1] as const) + .sort(([a], [b]) => a.localeCompare(b)), + ) +} + +function createReconciliationModel(): ReconciliationModel { + return { + sourceRows: new Map(), + sentRows: new Map(), + relation: new Map(), + graphActive: true, + } +} + +function snapshotModel(model: ReconciliationModel): { + source: Array + sent: Array + relation: Array +} { + const rows = (entries: ReadonlyMap) => + [...entries] + .map( + ([key, row]) => + `${typeof key}:${String(key)}|${row.id}:${row.revision}:${row.value}`, + ) + .sort() + return { + source: rows(model.sourceRows), + sent: rows(model.sentRows), + relation: [...model.relation].sort(([left], [right]) => + left.localeCompare(right), + ), + } +} + +function applyReconciliationStep( + model: ReconciliationModel, + step: ReconciliationStep, +): void { + if (step.type === `truncate`) { + // Truncate is only an early lifecycle signal. Its later source batch + // still needs the retained exact rows to retract the active graph. + } else if (step.type === `teardown`) { + model.sentRows.clear() + model.relation.clear() + model.graphActive = false + } else if (step.type === `restart`) { + if (!model.graphActive) { + const replay = [...model.sourceRows].map(([key, value]) => ({ + type: `insert` as const, + key, + value, + })) + applyToRelation( + model.relation, + reconcileChangesForD2(replay, model.sentRows), + ) + model.graphActive = true + } + } else { + const changes = sourceChangesFor(step.operations, model.sourceRows) + if (model.graphActive) { + const reconciled = reconcileChangesForD2(changes, model.sentRows) + applyToRelation(model.relation, reconciled) + } + } + + if (model.graphActive) { + expectTrackerMatchesSource(model.sourceRows, model.sentRows) + expectWeightedRelationMatchesSource(model.sourceRows, model.relation) + } else { + expect(model.sentRows.size).toBe(0) + expect(model.relation.size).toBe(0) + } +} + +function upsert( + key: SourceKey, + row: SourceRow, + reportedPreviousValue: SourceRow = row, +): ReconciliationStep { + return { + type: `batch`, + operations: [{ type: `upsert`, key, row, reportedPreviousValue }], + } +} + +function createOrderedSourceHarness(id: string) { + let sync!: SourceSyncActions + let loadSubsetCalls = 0 + const replayResolvers: Array<() => void> = [] + const contributed = { id: 1, revision: 1, value: 1 } + const staleDelete = { id: 1, revision: 2, value: 1 } + const replacement = { id: 1, revision: 3, value: 2 } + const source = createCollection({ + id, + getKey: (row) => row.id, + startSync: true, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (actions) => { + sync = actions + actions.markReady() + return { + loadSubset: () => { + loadSubsetCalls++ + // Initial page and its exact tie-boundary refinement are immediate; + // later calls are truncate replays controlled by the test. + if (loadSubsetCalls > 2) { + return new Promise((resolve) => replayResolvers.push(resolve)) + } + return true + }, + } + }, + }, + }) + sync.begin() + sync.write({ type: `insert`, value: contributed }) + expect(sync.commit()).toBe(true) + + let sourceCallback: Parameters[0] | undefined + let suppressSourceChanges = false + const subscribeChanges = source.subscribeChanges.bind(source) + source.subscribeChanges = ((callback, options) => { + sourceCallback = callback + return subscribeChanges((changes) => { + if (!suppressSourceChanges) callback(changes) + }, options) + }) as typeof source.subscribeChanges + + return { + contributed, + replacement, + source, + staleDelete, + suppressSourceChanges: () => { + suppressSourceChanges = true + }, + publish: (changes: Array>) => { + if (sourceCallback === undefined) { + throw new Error(`Query did not subscribe to its source`) + } + const publish = sourceCallback as unknown as ( + messages: Array>, + ) => void + publish(changes) + }, + truncate: () => { + sync.begin() + sync.truncate() + expect(sync.commit()).toBe(true) + }, + resolveReplay: async () => { + if (replayResolvers.length === 0) { + throw new Error(`No truncate replay is pending`) + } + for (let pass = 0; replayResolvers.length > 0; pass++) { + if (pass === 20) { + throw new Error( + `Truncate replay did not reach a fixed point after 20 passes`, + ) + } + for (const resolve of replayResolvers.splice(0)) resolve() + await flushPromises() + } + }, + } +} + +it(`ignores unknown deletes and inserts unknown updates at the D2 boundary`, () => { + const sentRows = new Map() + const stale = { id: 1, revision: 1, value: 1 } + const current = { id: 2, revision: 2, value: 2 } + + expect( + reconcileChangesForD2( + [{ type: `delete`, key: `row`, value: stale }], + sentRows, + ), + ).toEqual([]) + expect( + reconcileChangesForD2( + [ + { + type: `update`, + key: `row`, + previousValue: stale, + value: current, + }, + ], + sentRows, + ), + ).toEqual([{ type: `insert`, key: `row`, value: current }]) + expect(sentRows).toEqual(new Map([[`row`, current]])) +}) + +it(`retracts the exact Effect source row after an ordered truncate`, async () => { + const harness = createOrderedSourceHarness( + `d2-effect-truncate-reconciliation`, + ) + const { contributed, source, staleDelete } = harness + const events: Array<{ + type: string + value: { id: number; revision: number; value: number } + }> = [] + const effect = createEffect({ + query: (query) => + query + .from({ row: source }) + .where(({ row }) => eq(row.revision, contributed.revision)) + .orderBy(({ row }) => row.value) + .limit(1), + onBatch: (batch) => { + events.push(...batch) + }, + }) + try { + await flushPromises() + expect(events).toHaveLength(1) + expect(events[0]).toMatchObject({ + type: `enter`, + key: 1, + value: contributed, + }) + const publishedValue = events[0]!.value + + harness.suppressSourceChanges() + harness.truncate() + await flushPromises() + expect(events).toEqual([{ type: `enter`, key: 1, value: publishedValue }]) + + harness.publish([{ type: `delete`, key: 1, value: staleDelete }]) + await flushPromises() + expect(events).toEqual([ + { type: `enter`, key: 1, value: publishedValue }, + { type: `exit`, key: 1, value: publishedValue }, + ]) + } finally { + await effect.dispose() + await source.cleanup() + } +}) + +it(`retracts the exact live-query source row after ordered replay settles`, async () => { + const harness = createOrderedSourceHarness( + `d2-live-query-truncate-reconciliation`, + ) + const { contributed, source, staleDelete } = harness + const live = createLiveQueryCollection({ + id: `d2-live-query-truncate-result`, + query: (query) => + query + .from({ row: source }) + .where(({ row }) => eq(row.revision, contributed.revision)) + .orderBy(({ row }) => row.value) + .limit(1), + startSync: true, + }) + + try { + await live.preload() + expect(live.get(contributed.id)).toMatchObject(contributed) + + harness.suppressSourceChanges() + harness.truncate() + await flushPromises() + expect(live.get(contributed.id)).toMatchObject(contributed) + + harness.publish([{ type: `delete`, key: 1, value: staleDelete }]) + await flushPromises() + expect(live.get(contributed.id)).toMatchObject(contributed) + + await harness.resolveReplay() + expect(live.get(contributed.id)).toBeUndefined() + } finally { + await live.cleanup() + await source.cleanup() + } +}) + +it(`replaces the retained Effect source row after an ordered truncate`, async () => { + const harness = createOrderedSourceHarness(`d2-effect-truncate-replacement`) + const { contributed, replacement, source, staleDelete } = harness + const batches: Array< + Array<{ + type: string + value: SourceRow + previousValue?: SourceRow + }> + > = [] + const effect = createEffect({ + query: (query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.value) + .limit(1), + onBatch: (batch) => { + batches.push(batch) + }, + }) + + try { + await flushPromises() + expect(batches).toHaveLength(1) + expect(batches[0]).toHaveLength(1) + expect(batches[0]![0]).toMatchObject({ + type: `enter`, + key: 1, + value: contributed, + }) + const publishedValue = batches[0]![0]!.value + + harness.suppressSourceChanges() + harness.truncate() + await flushPromises() + expect(batches).toHaveLength(1) + + harness.publish([ + { + type: `update`, + key: 1, + previousValue: staleDelete, + value: replacement, + }, + ]) + await flushPromises() + expect(batches).toHaveLength(2) + expect(batches[1]).toHaveLength(1) + expect(batches[1]![0]).toMatchObject({ + type: `update`, + key: 1, + value: replacement, + }) + expect(batches[1]![0]!.previousValue).toBe(publishedValue) + } finally { + await effect.dispose() + await source.cleanup() + } +}) + +it(`replaces the retained live-query source row after ordered replay settles`, async () => { + const harness = createOrderedSourceHarness( + `d2-live-query-truncate-replacement`, + ) + const { contributed, replacement, source, staleDelete } = harness + const live = createLiveQueryCollection({ + id: `d2-live-query-truncate-replacement-result`, + query: (query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.value) + .limit(1), + startSync: true, + }) + const batches: Array>> = [] + + try { + await live.preload() + expect(live.get(contributed.id)).toMatchObject(contributed) + const publishedValue = live.get(contributed.id) + const subscription = live.subscribeChanges( + (changes) => batches.push(changes), + { includeInitialState: false }, + ) + + harness.suppressSourceChanges() + harness.truncate() + await flushPromises() + expect(batches).toEqual([]) + expect(live.get(contributed.id)).toBe(publishedValue) + + harness.publish([ + { + type: `update`, + key: 1, + previousValue: staleDelete, + value: replacement, + }, + ]) + await flushPromises() + expect(batches).toEqual([]) + expect(live.get(contributed.id)).toBe(publishedValue) + + await harness.resolveReplay() + expect(batches).toHaveLength(1) + expect(batches[0]).toHaveLength(1) + expect(batches[0]![0]).toMatchObject({ + type: `update`, + key: 1, + value: replacement, + }) + expect(batches[0]![0]!.previousValue).toEqual(publishedValue) + expect(live.get(replacement.id)).toMatchObject(replacement) + subscription.unsubscribe() + } finally { + await live.cleanup() + await source.cleanup() + } +}) + +it(`keeps revision and value in weighted row identity`, () => { + const key = `row` + const base = { id: 1, revision: 1, value: 1 } + const differentRevision = { id: 1, revision: 2, value: 1 } + const differentValue = { id: 1, revision: 1, value: 2 } + const relation = new Map() + + addWeight(relation, key, base, 1) + addWeight(relation, key, differentRevision, 1) + addWeight(relation, key, differentValue, 1) + + expect(relation).toEqual( + new Map([ + [expectedWeightedRowIdentity(key, base), 1], + [expectedWeightedRowIdentity(key, differentRevision), 1], + [expectedWeightedRowIdentity(key, differentValue), 1], + ]), + ) +}) + +it(`keeps numeric and string source keys distinct across restart`, () => { + const model = createReconciliationModel() + const row = { id: 1, revision: 1, value: 1 } + const keys = [0, `0`] as const + + applyReconciliationStep(model, { + type: `batch`, + operations: keys.map((key) => ({ + type: `upsert` as const, + key, + row, + reportedPreviousValue: row, + })), + }) + expect(model.sourceRows).toEqual( + new Map([ + [0, row], + [`0`, row], + ]), + ) + expect(model.sentRows).toEqual(model.sourceRows) + expect(model.relation).toEqual( + new Map([ + [expectedWeightedRowIdentity(0, row), 1], + [expectedWeightedRowIdentity(`0`, row), 1], + ]), + ) + + applyReconciliationStep(model, { type: `teardown` }) + applyReconciliationStep(model, { type: `restart` }) + expect(model.sourceRows).toEqual( + new Map([ + [0, row], + [`0`, row], + ]), + ) + expect(model.sentRows).toEqual(model.sourceRows) + expect(model.relation).toEqual( + new Map([ + [expectedWeightedRowIdentity(0, row), 1], + [expectedWeightedRowIdentity(`0`, row), 1], + ]), + ) +}) + +it(`preserves external source rows across graph teardown and restart`, () => { + const model = createReconciliationModel() + const row = { id: 1, revision: 1, value: 1 } + + applyReconciliationStep(model, upsert(`row`, row)) + applyReconciliationStep(model, { type: `teardown` }) + expect(model.graphActive).toBe(false) + expect(model.sourceRows).toEqual(new Map([[`row`, row]])) + expect(model.sentRows).toEqual(new Map()) + expect(model.relation).toEqual(new Map()) + + applyReconciliationStep(model, { type: `restart` }) + expect(model.graphActive).toBe(true) + expect(model.sourceRows).toEqual(new Map([[`row`, row]])) + expect(model.sentRows).toEqual(new Map([[`row`, row]])) + expect(model.relation).toEqual( + new Map([[expectedWeightedRowIdentity(`row`, row), 1]]), + ) +}) + +it(`replays external source changes made while the graph is down`, () => { + const model = createReconciliationModel() + const first = { id: 1, revision: 1, value: 1 } + const replacement = { id: 1, revision: 2, value: 2 } + + applyReconciliationStep(model, upsert(`row`, first)) + applyReconciliationStep(model, { type: `teardown` }) + applyReconciliationStep(model, upsert(`row`, replacement, first)) + expect(model.sourceRows).toEqual(new Map([[`row`, replacement]])) + expect(model.sentRows).toEqual(new Map()) + expect(model.relation).toEqual(new Map()) + + applyReconciliationStep(model, { type: `restart` }) + expect(model.sourceRows).toEqual(new Map([[`row`, replacement]])) + expect(model.sentRows).toEqual(new Map([[`row`, replacement]])) + expect(model.relation).toEqual( + new Map([[expectedWeightedRowIdentity(`row`, replacement), 1]]), + ) +}) + +it(`generates teardown, down-state source changes, and restart`, () => { + const histories = fc.sample(reconciliationHistoryArbitrary, { + seed: 1780, + numRuns: 500, + }) + + expect( + histories.some((steps) => { + let graphActive = true + let sawTeardown = false + let sawDownStateSourceChange = false + for (const step of steps) { + if (step.type === `teardown`) { + graphActive = false + sawTeardown = true + } else if (step.type === `restart`) { + if (!graphActive && sawTeardown && sawDownStateSourceChange) { + return true + } + graphActive = true + } else if (step.type === `batch` && !graphActive) { + sawDownStateSourceChange = true + } + } + return false + }), + ).toBe(true) +}) + +fcTest.prop( + [reconciliationHistoryArbitrary], + oraclePropertyOptions(200, `d2-source.exact-retractions`), +)( + `keeps one exact D2 contribution per source key across batched histories`, + (steps) => { + const model = createReconciliationModel() + for (const step of steps) { + applyReconciliationStep(model, step) + } + }, +) + +const assertDisjointHistoriesCommute = ([left, right]: [ + Array, + Array, +]) => { + const leftThenRight = createReconciliationModel() + applyReconciliationStep(leftThenRight, { type: `batch`, operations: left }) + applyReconciliationStep(leftThenRight, { type: `batch`, operations: right }) + + const rightThenLeft = createReconciliationModel() + applyReconciliationStep(rightThenLeft, { type: `batch`, operations: right }) + applyReconciliationStep(rightThenLeft, { type: `batch`, operations: left }) + + expect(snapshotModel(rightThenLeft)).toEqual(snapshotModel(leftThenRight)) +} + +fcTest.prop([disjointHistoriesArbitrary], { + numRuns: oracleRuns(100), + seed: 1781, +})( + `commutes independent source histories for a fixed seed`, + assertDisjointHistoriesCommute, +) + +fcTest.prop( + [disjointHistoriesArbitrary], + oraclePropertyOptions(100, `d2-source.disjoint-commutation`), +)( + `commutes independent source histories for a random or replayed seed`, + assertDisjointHistoriesCommute, +) diff --git a/packages/db/tests/db-client.test.ts b/packages/db/tests/db-client.test.ts index 40f275f6bc..a6a61bab81 100644 --- a/packages/db/tests/db-client.test.ts +++ b/packages/db/tests/db-client.test.ts @@ -1014,6 +1014,17 @@ describe(`DbClient`, () => { expect(() => adapterWrite({ id: `1`, name: `adapter` })).not.toThrow() expect(collection.get(`1`)?.name).toBe(`adapter`) + + client.hydrate({ + collections: [ + { + collectionId: `ready-hydration-seed`, + rows: [{ key: `1`, value: { id: `1`, name: `late hydration` } }], + }, + ], + }) + + expect(collection.get(`1`)?.name).toBe(`adapter`) }) it(`does not let a late stream chunk overwrite adapter rows or metadata`, async () => { diff --git a/packages/db/tests/deep-equals-work.test.ts b/packages/db/tests/deep-equals-work.test.ts new file mode 100644 index 0000000000..7ced0d481e --- /dev/null +++ b/packages/db/tests/deep-equals-work.test.ts @@ -0,0 +1,29 @@ +import { describe, expect, it, vi } from 'vitest' +import { deepEquals } from '../src/utils.js' + +describe(`deep equality enumeration work`, () => { + it.each([false, true])( + `avoids intermediate filtered key arrays with symbols=%s`, + (symbols) => { + const key = Symbol(`field`) + const createRow = () => ({ + id: 1, + nested: { value: 2 }, + ...(symbols ? { [key]: 3 } : {}), + }) + const left = createRow() + const right = createRow() + const spy = vi.spyOn(Array.prototype, `filter`) + let calls: number + let equal: boolean + try { + equal = deepEquals(left, right) + calls = spy.mock.calls.length + } finally { + spy.mockRestore() + } + expect(equal).toBe(true) + expect(calls).toBe(0) + }, + ) +}) diff --git a/packages/db/tests/effect-disposal-oracle.test.ts b/packages/db/tests/effect-disposal-oracle.test.ts new file mode 100644 index 0000000000..7428721eb7 --- /dev/null +++ b/packages/db/tests/effect-disposal-oracle.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, it } from 'vitest' +import { createCollection, createEffect } from '../src/index.js' +import { createDeferred } from '../src/deferred.js' +import { flushPromises } from './utils.js' + +// One disposal attempt has one outcome, even when abort/release callbacks +// reenter it. Counting physical releases alone misses divergent caller results. +const scenarios = ([`abort`, `release`] as const).flatMap((reentry) => + [false, true].flatMap((pendingHandler) => + ([`success`, `error`, `undefined`] as const).map((outcome) => ({ + reentry, + pendingHandler, + outcome, + })), + ), +) + +describe(`Effect disposal outcome oracle`, () => { + it.each(scenarios)( + `joins all callers to one attempt: %j`, + async ({ reentry, pendingHandler, outcome }) => { + const failure = new Error(`release failed`) + const handler = createDeferred() + let nested: Promise | undefined + let releases = 0 + const source = createCollection<{ id: number }>({ + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: () => { + begin() + write({ type: `insert`, value: { id: 1 } }) + commit() + return true + }, + unloadSubset: () => { + releases++ + if (reentry === `release`) nested = effect.dispose() + if (outcome === `error`) throw failure + if (outcome === `undefined`) throw undefined + }, + } + }, + }, + }) + const effect: ReturnType = createEffect({ + query: (q) => q.from({ row: source }), + onBatch: (_events, { signal }) => { + if (reentry === `abort`) + signal.addEventListener( + `abort`, + () => { + nested = effect.dispose() + }, + { once: true }, + ) + return pendingHandler ? handler.promise : undefined + }, + }) + try { + await flushPromises() + const outer = effect.dispose() + // Observe every promise before any assertion can throw. + const results = Promise.allSettled([outer, nested!, effect.dispose()]) + let settled = false + void results.then(() => { + settled = true + }) + expect(nested).toBeDefined() + expect(effect.disposed).toBe(true) + expect(source.subscriberCount).toBe(0) + expect(releases).toBe(1) + if (pendingHandler) { + await flushPromises() + expect(settled).toBe(false) + } + handler.resolve() + const observed = await results + for (const result of observed) { + expect(result.status).toBe( + outcome === `success` ? `fulfilled` : `rejected`, + ) + if (result.status === `rejected`) { + if (outcome === `error`) expect(result.reason).toBe(failure) + else expect(result.reason).toMatchObject({ message: `undefined` }) + } + expect(result).toEqual(observed[0]) + } + // A settled failed attempt does not make the source lease retryable. + await effect.dispose() + expect(releases).toBe(1) + } finally { + handler.resolve() + await Promise.allSettled([nested, effect.dispose()]) + await source.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/effect.test.ts b/packages/db/tests/effect.test.ts index 5c95625a4b..302aa5fd25 100644 --- a/packages/db/tests/effect.test.ts +++ b/packages/db/tests/effect.test.ts @@ -716,6 +716,109 @@ describe(`createEffect`, () => { await expect(secondDispose).rejects.toBe(failure) await source.cleanup() }) + + it.each([ + { name: `undefined`, failure: undefined }, + { name: `null`, failure: null }, + { name: `false`, failure: false }, + { name: `zero`, failure: 0 }, + { name: `empty string`, failure: `` }, + { name: `NaN`, failure: Number.NaN }, + ])( + `reports a falsy cleanup failure once: $name`, + async ({ name, failure }) => { + let unloadCount = 0 + const source = createCollection<{ id: number }>({ + id: `effect-falsy-cleanup-${name}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloadCount++ + if (unloadCount === 1) throw failure + }, + } + }, + }, + }) + const effect = createEffect({ + query: (q) => q.from({ source }), + onBatch: () => {}, + }) + + try { + await flushPromises() + let didReject = false + let rejection: unknown + try { + await effect.dispose() + } catch (error) { + didReject = true + rejection = error + } + expect(didReject).toBe(true) + expect(rejection).toBeInstanceOf(Error) + expect((rejection as Error).message).toBe(String(failure)) + expect(unloadCount).toBe(1) + + await effect.dispose() + expect(unloadCount).toBe(1) + } finally { + await effect.dispose() + await source.cleanup() + } + }, + ) + + it(`does not repeat a failed source release across reentrant disposal`, async () => { + const failure = new Error(`outer source release failed`) + let unloadCount = 0 + const source = createCollection<{ id: number }>({ + id: `effect-reentrant-cleanup-error`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloadCount++ + if (unloadCount === 1) { + void effect.dispose() + throw failure + } + }, + } + }, + }, + }) + const effect = createEffect({ + query: (q) => q.from({ source }), + onBatch: () => {}, + }) + + try { + await flushPromises() + await expect(effect.dispose()).rejects.toBe(failure) + // Reentrant disposal cannot repeat an unload still on the stack. + expect(unloadCount).toBe(1) + expect(source.subscriberCount).toBe(0) + + await effect.dispose() + // Finishing the failed attempt does not make the lease retryable. + expect(unloadCount).toBe(1) + await effect.dispose() + expect(unloadCount).toBe(1) + } finally { + await effect.dispose() + await source.cleanup() + } + }) }) describe(`auto-generated IDs`, () => { @@ -1899,12 +2002,85 @@ describe(`createEffect`, () => { } }) + it(`reports failed obsolete-demand release without failing the source commit`, async () => { + const failure = new Error(`obsolete effect demand release failed`) + const users = createCollection( + mockSyncCollectionOptions({ + id: `effect-obsolete-release-users`, + getKey: (user) => user.id, + initialData: [sampleUsers[0]!], + }), + ) + let loadCount = 0 + let unloadCount = 0 + const consoleError = vi + .spyOn(console, `error`) + .mockImplementation(() => {}) + const issues = createCollection({ + id: `effect-obsolete-release-issues`, + getKey: (issue) => issue.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loadCount++ + return true + }, + unloadSubset: () => { + unloadCount++ + if (unloadCount <= 2) throw failure + }, + } + }, + }, + }) + const sourceErrors: Array = [] + const effect = createEffect({ + query: (q) => + q + .from({ user: users }) + .leftJoin({ issue: issues }, ({ user, issue }) => + eq(user.id, issue.userId), + ) + .select(({ user, issue }) => ({ + id: user.id, + issueId: issue.id, + })), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }) + + try { + await flushPromises() + expect(loadCount).toBe(1) + + expect(() => { + users.utils.begin() + users.utils.write({ type: `delete`, value: sampleUsers[0]! }) + users.utils.commit() + }).not.toThrow() + await flushPromises() + + expect(sourceErrors).toEqual([failure]) + expect(effect.disposed).toBe(true) + expect(unloadCount).toBe(1) + + await effect.dispose() + expect(unloadCount).toBe(1) + } finally { + await effect.dispose() + await Promise.all([users.cleanup(), issues.cleanup()]) + consoleError.mockRestore() + } + }) + it(`reports a rejected ordered subset load and disposes the effect`, async () => { const failure = new Error(`ordered subset failed`) let loadCount = 0 - let removeVisibleRow: () => void = () => { - throw new Error(`source has not started`) - } const users = createCollection({ id: `effect-rejected-ordered-users`, getKey: (user) => user.id, @@ -1914,11 +2090,6 @@ describe(`createEffect`, () => { sync: { sync: ({ begin, write, commit, markReady }) => { markReady() - removeVisibleRow = () => { - begin() - write({ type: `delete`, value: sampleUsers[0]! }) - commit() - } return { loadSubset: () => { loadCount++ @@ -1945,11 +2116,6 @@ describe(`createEffect`, () => { try { await flushPromises() - expect(sourceErrors).toEqual([]) - - removeVisibleRow() - await flushPromises() - expect(sourceErrors).toEqual([failure]) expect(effect.disposed).toBe(true) } finally { @@ -2021,7 +2187,7 @@ describe(`createEffect`, () => { cleanupFailure, ) } finally { - await expect(effect.dispose()).rejects.toBe(cleanupFailure) + await expect(effect.dispose()).resolves.toBeUndefined() consoleErrorSpy.mockRestore() await users.cleanup() } @@ -2266,6 +2432,101 @@ describe(`createEffect`, () => { await effect.dispose() }) + it.each( + ([`projection`, `delivery`] as const).flatMap((phase) => + [false, true].map((throwRelease) => ({ phase, throwRelease })), + ), + )( + `isolates in-turn disposal and nested publication: %j`, + async ({ phase, throwRelease }) => { + const users = createUsersCollection([]) + const issues = createIssuesCollection([]) + const peerEvents: Array> = [] + const peer = createEffect({ + query: (q) => q.from({ user: users }), + onEnter: (event) => { + peerEvents.push(event) + }, + }) + const failure = new Error(`unsubscribe failed after releasing`) + let shouldThrow = throwRelease + const subscribe = users.subscribeChanges.bind(users) + vi.spyOn(users, `subscribeChanges`).mockImplementation((...args) => { + const subscription = subscribe(...args) + const unsubscribe = subscription.unsubscribe.bind(subscription) + vi.spyOn(subscription, `unsubscribe`).mockImplementation(() => { + unsubscribe() + if (shouldThrow) { + shouldThrow = false + throw failure + } + }) + return subscription + }) + const events: Array> = [] + let disposeInTurn: (() => void) | undefined + let outcome: Promise | undefined + const effect = createEffect({ + query: (q) => + q + .from({ user: users }) + .leftJoin({ issue: issues }, ({ user, issue }) => + eq(user.id, issue.userId), + ) + .fn.select(({ user }) => { + if (phase === `projection`) disposeInTurn?.() + return user + }), + onEnter: (event) => { + events.push(event) + if (phase === `delivery`) disposeInTurn?.() + }, + }) + try { + await flushPromises() + expect(users.subscriberCount).toBe(2) + expect(issues.subscriberCount).toBe(1) + disposeInTurn = () => { + disposeInTurn = undefined + outcome = effect.dispose().then( + () => ({ status: `fulfilled` }), + (error: unknown) => ({ status: `rejected`, reason: error }), + ) + users.utils.begin() + users.utils.write({ + type: `insert`, + value: { id: 3, name: `Nested`, active: true }, + }) + users.utils.commit() + } + users.utils.begin() + for (const id of [1, 2]) { + users.utils.write({ + type: `insert`, + value: { id, name: `User ${id}`, active: true }, + }) + } + users.utils.commit() + await flushPromises() + expect(outcome).toBeDefined() + expect(await outcome).toEqual( + throwRelease + ? { status: `rejected`, reason: failure } + : { status: `fulfilled` }, + ) + expect(effect.disposed).toBe(true) + expect(events).toHaveLength(phase === `projection` ? 0 : 1) + expect(peerEvents.map(({ key }) => key).sort()).toEqual([1, 2, 3]) + expect(users.subscriberCount).toBe(1) + expect(issues.subscriberCount).toBe(0) + } finally { + await effect.dispose() + await peer.dispose() + await Promise.all([users.cleanup(), issues.cleanup()]) + } + }, + ) + it(`disposing inside handler should not throw and should stop further events`, async () => { const users = createUsersCollection() const events: Array> = [] diff --git a/packages/db/tests/facade-retention.probe.ts b/packages/db/tests/facade-retention.probe.ts new file mode 100644 index 0000000000..fa45d249b0 --- /dev/null +++ b/packages/db/tests/facade-retention.probe.ts @@ -0,0 +1,117 @@ +// Run manually: node --expose-gc --import tsx tests/facade-retention.probe.ts +// This probes reachability, not total application heap size or GC latency. +import assert from 'node:assert/strict' +import { setImmediate } from 'node:timers/promises' +import { D2, MultiSet } from '@tanstack/db-ivm' +import { BucketFacadeAdapter } from '../src/query/live/bucket-facade-adapter.js' +import { BUCKET_FACADE_REF } from '../src/query/live/materialized-pipeline.js' +import type { Collection } from '../src/collection/index.js' +import type { + BucketFacadeRef, + BucketRow, +} from '../src/query/live/materialized-pipeline.js' + +const gc = globalThis.gc +if (!gc) throw new Error(`Run this probe with --expose-gc`) + +function capture( + released: boolean, + holder: `view` | `method`, + pendingUpdate: boolean, +) { + const graph = new D2() + const rows = graph.newInput<[string, BucketRow]>() + const activeBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `retention-probe`, + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: false }], + () => {}, + ) + graph.finalize() + const value = { id: 1, payload: new ArrayBuffer(1024 * 1024) } + const bucketKey = `group` + activeBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + rows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { + publicKey: 1, + value, + order: undefined, + }, + ], + 1, + ], + ]), + ) + graph.run() + const ref: BucketFacadeRef = { + [BUCKET_FACADE_REF]: { edgeId: `children`, bucketKey }, + } + adapter.flush().publish() + const view = adapter.resolve(ref) as unknown as Collection< + typeof value, + number + > + assert.equal(view.get(1)?.id, 1) + const retained = holder === `view` ? view : view.get.bind(view) + if (pendingUpdate) { + rows.sendData( + new MultiSet([ + [[bucketKey, { publicKey: 2, value: { id: 2 }, order: undefined }], 1], + ]), + ) + graph.run() + } + if (released) adapter.cleanup() + return { retained, value: new WeakRef(value), adapter: new WeakRef(adapter) } +} + +const cells = [false, true].flatMap((released) => + ([`view`, `method`] as const).flatMap((holder) => + [false, true].map((pendingUpdate) => ({ released, holder, pendingUpdate })), + ), +) +const results = cells.map((cell) => ({ + ...cell, + samples: Array.from({ length: 10 }, () => + capture(cell.released, cell.holder, cell.pendingUpdate), + ), +})) + +// WeakRef targets stay alive through the creating job. Cross job boundaries +// before forcing collection and avoid dereferencing targets inside this loop. +for (let turn = 0; turn < 5; turn++) { + await setImmediate() + gc() +} +await setImmediate() + +const report = results.map(({ released, holder, pendingUpdate, samples }) => { + const retainedValues = samples.filter( + (sample) => sample.value.deref() !== undefined, + ).length + const retainedAdapters = samples.filter( + (sample) => sample.adapter.deref() !== undefined, + ).length + // Live public facades are the positive control: this probe must detect them. + assert.equal(retainedValues, released ? 0 : samples.length) + assert.equal(retainedAdapters, 0) + assert.equal(samples.length, 10) + // Keep each public handle or captured method observably reachable to the end. + for (const { retained } of samples) { + const row = typeof retained === `function` ? retained(1) : retained.get(1) + assert.equal(row?.id, released ? undefined : 1) + } + return { + released, + holder, + pendingUpdate, + samples: samples.length, + retainedValues, + retainedAdapters, + } +}) +console.log(JSON.stringify({ node: process.version, report }, null, 2)) diff --git a/packages/db/tests/index-domain-recovery-work.test.ts b/packages/db/tests/index-domain-recovery-work.test.ts new file mode 100644 index 0000000000..74d2436364 --- /dev/null +++ b/packages/db/tests/index-domain-recovery-work.test.ts @@ -0,0 +1,91 @@ +import { expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { BasicIndex } from '../src/indexes/basic-index.js' +import { BTreeIndex } from '../src/indexes/btree-index.js' +import { Func, PropRef, Value } from '../src/query/ir.js' +import type { SyncConfig } from '../src/types.js' + +type Row = { id: number; value: number | Array } + +it.each( + [BasicIndex, BTreeIndex].flatMap((IndexType) => + [100, 10000].flatMap((size) => + ([`delete`, `update`] as const).map((retirement) => ({ + name: IndexType.name, + IndexType, + size, + retirement, + })), + ), + ), +)( + `$name restores range lookup after $retirement in $size rows`, + async ({ IndexType, size, retirement }) => { + let sync!: Parameters[`sync`]>[0] + const collection = createCollection({ + getKey: (row) => row.id, + autoIndex: `off`, + sync: { + sync: (context) => { + sync = context + context.begin() + for (let id = 0; id < size; id++) { + context.write({ type: `insert`, value: { id, value: id } }) + } + context.commit() + context.markReady() + }, + }, + }) + await collection.preload() + const index = collection.createIndex((row) => row.value, { + indexType: IndexType, + }) + const entries = collection.entries.bind(collection) + let scanned = 0 + const spy = vi + .spyOn(collection, `entries`) + .mockImplementation(function* () { + for (const entry of entries()) { + scanned++ + yield entry + } + }) + const where = new Func(`gt`, [ + new PropRef([`value`]), + new Value(size - 2), + ]) + const visits: Array = [] + const read = (expected: Array, repeats = 1) => { + scanned = 0 + for (let i = 0; i < repeats; i++) { + expect( + collection + .currentStateAsChanges({ where }) + ?.map(({ key }) => key) + .sort((a, b) => Number(a) - Number(b)), + ).toEqual(expected) + } + visits.push(scanned) + } + try { + read([size - 1]) + sync.begin() + sync.write({ type: `insert`, value: { id: size, value: [size + 5] } }) + sync.commit() + read([size - 1, size]) + sync.begin() + if (retirement === `delete`) sync.write({ type: `delete`, key: size }) + else sync.write({ type: `update`, value: { id: size, value: 0 } }) + sync.commit() + read([size - 1], 3) + index.build(entries()) + read([size - 1]) + // The transient foreign domain must not leave every future snapshot scanning. + expect(visits).toEqual([0, size + 1, 0, 0]) + } finally { + spy.mockRestore() + await collection.cleanup() + } + }, +) diff --git a/packages/db/tests/index-reader.test-d.ts b/packages/db/tests/index-reader.test-d.ts new file mode 100644 index 0000000000..88ea904bc4 --- /dev/null +++ b/packages/db/tests/index-reader.test-d.ts @@ -0,0 +1,13 @@ +import { expectTypeOf, test } from 'vitest' +import type { + IndexReader, + ReverseIndex, + findIndexForField, +} from '../src/index.js' + +test(`resolved indexes expose a named read interface`, () => { + expectTypeOf>().toEqualTypeOf< + IndexReader | undefined + >() + expectTypeOf>().toMatchTypeOf>() +}) diff --git a/packages/db/tests/index-update-short-circuit.test.ts b/packages/db/tests/index-update-short-circuit.test.ts index 94a990b6aa..c23d9ae2c7 100644 --- a/packages/db/tests/index-update-short-circuit.test.ts +++ b/packages/db/tests/index-update-short-circuit.test.ts @@ -3,6 +3,7 @@ import { BasicIndex } from '../src/indexes/basic-index.js' import { BTreeIndex } from '../src/indexes/btree-index.js' import { PropRef } from '../src/query/ir.js' import { normalizeValue } from '../src/utils/comparison.js' +import { valueMapData } from './utils' import type { BaseIndex } from '../src/indexes/base-index.js' type IndexConstructor = new ( @@ -10,9 +11,7 @@ type IndexConstructor = new ( expression: PropRef, name?: string, options?: unknown, -) => BaseIndex & { - valueMapData: Map> -} +) => BaseIndex const indexTypes: Array<[string, IndexConstructor]> = [ [`BasicIndex`, BasicIndex as IndexConstructor], @@ -24,16 +23,31 @@ describe.each(indexTypes)(`%s update`, (_indexName, IndexType) => { return new IndexType(1, new PropRef([`value`]), `test_index`, options) } + it(`looks up rows without collecting timing diagnostics`, () => { + const index = createIndex() + index.add(`a`, { value: 1 }) + const now = vi.spyOn(performance, `now`) + try { + expect(index.lookup(`eq`, 1)).toEqual(new Set([`a`])) + expect(index.lookup(`in`, [1, 2])).toEqual(new Set([`a`])) + expect(now).not.toHaveBeenCalled() + } finally { + now.mockRestore() + } + }) + it(`keeps the existing bucket when the indexed value does not change`, () => { const index = createIndex() index.add(`a`, { value: 1, version: 1 }) - const bucket = index.valueMapData.get(1) - const lastUpdated = index.getStats().lastUpdated + const bucket = valueMapData(index).get(1) + const add = vi.spyOn(index, `add`) + const remove = vi.spyOn(index, `remove`) index.update(`a`, { value: 1, version: 1 }, { value: 1, version: 2 }) - expect(index.valueMapData.get(1)).toBe(bucket) - expect(index.getStats().lastUpdated).toBe(lastUpdated) + expect(valueMapData(index).get(1)).toBe(bucket) + expect(add).not.toHaveBeenCalled() + expect(remove).not.toHaveBeenCalled() expect(index.lookup(`eq`, 1)).toEqual(new Set([`a`])) }) @@ -46,12 +60,12 @@ describe.each(indexTypes)(`%s update`, (_indexName, IndexType) => { ])(`keeps the existing bucket for %s`, (_caseName, oldValue, newValue) => { const index = createIndex() index.add(`a`, { value: oldValue }) - const bucket = index.valueMapData.get(normalizeValue(oldValue)) + const bucket = valueMapData(index).get(normalizeValue(oldValue)) expect(bucket).toBeDefined() index.update(`a`, { value: oldValue }, { value: newValue }) - expect(index.valueMapData.get(normalizeValue(newValue))).toBe(bucket) + expect(valueMapData(index).get(normalizeValue(newValue))).toBe(bucket) }) it(`moves the key when the indexed value changes`, () => { diff --git a/packages/db/tests/index-update.property.test.ts b/packages/db/tests/index-update.property.test.ts index 2099c77906..501c940920 100644 --- a/packages/db/tests/index-update.property.test.ts +++ b/packages/db/tests/index-update.property.test.ts @@ -1,15 +1,24 @@ -import { describe, expect } from 'vitest' +import { describe, expect, expectTypeOf, test } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' +import { compareKeys } from '@tanstack/db-ivm' import { BasicIndex } from '../src/indexes/basic-index.js' import { BTreeIndex } from '../src/indexes/btree-index.js' import { PropRef } from '../src/query/ir.js' -import type { BaseIndex } from '../src/indexes/base-index.js' +import { DEFAULT_COMPARE_OPTIONS } from '../src/utils.js' +import { makeComparator } from '../src/utils/comparison.js' +import { indexedKeysSet, orderedEntriesArray, valueMapData } from './utils' +import type { BaseIndex, IndexInterface } from '../src/indexes/base-index.js' type IndexValue = number type IndexConstructor = new ( id: number, expression: PropRef, + name?: string, + options?: { + compareFn?: (left: unknown, right: unknown) => number + compareOptions?: typeof DEFAULT_COMPARE_OPTIONS + }, ) => BaseIndex type IndexAction = @@ -63,9 +72,9 @@ function expectIndexMatchesModel( const groups = groupKeysByValue(rows) expect(index.keyCount).toBe(rows.size) - expect(index.indexedKeysSet).toEqual(new Set(rows.keys())) - expect(index.valueMapData).toEqual(groups) - expect(index.orderedEntriesArray).toEqual( + expect(indexedKeysSet(index)).toEqual(new Set(rows.keys())) + expect(valueMapData(index)).toEqual(groups) + expect(orderedEntriesArray(index)).toEqual( [...groups].sort(([left], [right]) => left - right), ) @@ -83,7 +92,10 @@ function expectIndexMatchesModel( expect(index.rangeQuery({ from: boundary })).toEqual(keysAtOrAbove) expect(index.rangeQuery({ to: boundary })).toEqual(keysAtOrBelow) + expect(index.rangeQueryReversed({ from: boundary })).toEqual(keysAtOrBelow) + expect(index.rangeQueryReversed({ to: boundary })).toEqual(keysAtOrAbove) } + expect(index.rangeQueryReversed({})).toEqual(new Set(rows.keys())) } describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { @@ -123,4 +135,235 @@ describe.each(indexTypes)(`%s update properties`, (_indexName, IndexType) => { expectIndexMatchesModel(rebuilt, rows) }, ) + + test(`tracks range-domain safety through updates, rebuilds, and clear`, () => { + const index = new IndexType(1, new PropRef([`value`])) + const other = [20] + + index.add(`number`, { value: 50 }) + expect(index.canOptimizeRangeFor(100)).toBe(true) + + index.add(`other`, { value: other }) + expect(index.canOptimizeRangeFor(100)).toBe(false) + + index.update(`other`, { value: other }, { value: 20 }) + expect(index.canOptimizeRangeFor(100)).toBe(true) + + index.update(`number`, { value: 50 }, { value: new Date(50) }) + expect(index.canOptimizeRangeFor(100)).toBe(false) + index.remove(`other`, { value: 20 }) + expect(index.canOptimizeRangeFor(new Date(100))).toBe(true) + + index.clear() + expect(index.canOptimizeRangeFor(100)).toBe(true) + + index.build([ + [`number`, { value: 50 }], + [`other`, { value: [20] }], + ]) + expect(index.canOptimizeRangeFor(100)).toBe(false) + }) + + test(`accepts indexed values rather than row keys through the index interface`, () => { + const index: IndexInterface = new IndexType( + 1, + new PropRef([`value`]), + ) + expectTypeOf< + Parameters[`take`]>[1] + >().toEqualTypeOf() + expectTypeOf< + Parameters[`takeReversed`]>[1] + >().toEqualTypeOf() + expectTypeOf< + Parameters[`take`]>[1] + >().toEqualTypeOf() + expectTypeOf< + Parameters[`takeReversed`]>[1] + >().toEqualTypeOf() + index.add(`undefined`, { value: undefined }) + index.add(`zero`, { value: 0 }) + index.add(`one`, { value: 1 }) + + expect(index.take(3, 0)).toEqual([`one`]) + expect(index.takeReversed(3, 1)).toEqual([`zero`, `undefined`]) + expect(index.take(3, undefined)).toEqual([`zero`, `one`]) + expect(index.takeReversed(3, undefined)).toEqual([]) + }) + + test(`distinguishes explicit undefined range and cursor bounds`, () => { + const index = new IndexType(1, new PropRef([`value`])) + index.add(`undefined`, { value: undefined }) + index.add(`null`, { value: null }) + index.add(`one`, { value: 1 }) + + expect(index.rangeQuery({ to: undefined })).toEqual( + new Set([`undefined`, `null`]), + ) + expect(index.rangeQueryReversed({ from: undefined })).toEqual( + new Set([`undefined`, `null`]), + ) + expect(index.take(3, undefined)).toEqual([`one`]) + expect(index.takeReversed(3, undefined)).toEqual([]) + }) + + test(`executes the ordering advertised by compare options`, () => { + const compareOptions = { + ...DEFAULT_COMPARE_OPTIONS, + nulls: `last` as const, + stringSort: `lexical` as const, + } + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareOptions, + }) + index.add(`undefined`, { value: undefined }) + index.add(`null`, { value: null }) + index.add(`one`, { value: 1 }) + + expect(index.matchesCompareOptions(compareOptions)).toBe(true) + expect(index.takeFromStart(3)).toEqual([`one`, `null`, `undefined`]) + expect(index.rangeQuery({ to: 1 })).toEqual(new Set([`one`])) + }) +}) + +describe.each(indexTypes)(`%s comparator groups`, (_indexName, IndexType) => { + fcTest.prop([ + fc.array(fc.integer({ min: 0, max: 4 }), { + minLength: 2, + maxLength: 20, + }), + ])( + `preserves exact equality while ordered traversal retains every row`, + (groupIds) => { + const symbols = new Map() + const rows = groupIds.map((groupId, position) => { + const symbol = symbols.get(groupId) ?? Symbol(String(groupId)) + symbols.set(groupId, symbol) + return { + key: String(position), + value: [symbol], + groupId, + } + }) + const index = new IndexType(1, new PropRef([`value`])) + + const expectMatchesModel = ( + subject: BaseIndex, + currentRows: typeof rows, + ) => { + const groups = new Map() + for (const row of currentRows) { + const group = groups.get(row.groupId) ?? [] + group.push(row) + groups.set(row.groupId, group) + } + const compare = makeComparator(DEFAULT_COMPARE_OPTIONS) + const orderedGroups = [...groups.values()].sort((left, right) => + compare(left[0]!.value, right[0]!.value), + ) + const forward = orderedGroups.flatMap((group) => + group.map((row) => row.key).sort(compareKeys), + ) + const reversed = [...orderedGroups].reverse().flatMap((group) => + group + .map((row) => row.key) + .sort(compareKeys) + .reverse(), + ) + + expect(subject.takeFromStart(currentRows.length)).toEqual(forward) + expect(subject.takeReversedFromEnd(currentRows.length)).toEqual( + reversed, + ) + for (const [representative, keys] of orderedEntriesArray(subject)) { + expect( + currentRows.some( + (row) => row.value === representative && keys.has(row.key), + ), + ).toBe(true) + } + for (const row of currentRows) { + expect(subject.equalityLookup(row.value)).toEqual(new Set([row.key])) + expect( + subject.rangeQuery({ from: row.value, to: row.value }), + ).toEqual( + new Set( + currentRows + .filter((candidate) => candidate.groupId === row.groupId) + .map((candidate) => candidate.key), + ), + ) + } + } + + for (const row of rows) index.add(row.key, row) + expectMatchesModel(index, rows) + + const removed = rows.shift()! + index.remove(removed.key, removed) + expectMatchesModel(index, rows) + + const changed = rows[0]! + const previous = { ...changed } + changed.groupId = 99 + changed.value = [Symbol(`updated`)] + index.update(changed.key, previous, changed) + expectMatchesModel(index, rows) + + const rebuilt = new IndexType(2, new PropRef([`value`])) + rebuilt.build(rows.map((row) => [row.key, row])) + expectMatchesModel(rebuilt, rows) + }, + ) + + fcTest.prop([ + fc.array(fc.integer({ min: 0, max: 4 }), { + minLength: 2, + maxLength: 20, + }), + ])(`matches an independent custom-comparator model`, (generatedGroups) => { + const groupIds = [...generatedGroups, generatedGroups[0]!] + const rows = groupIds.map((groupId, position) => ({ + key: String(position).padStart(2, `0`), + value: { groupId, position }, + })) + const index = new IndexType(1, new PropRef([`value`]), undefined, { + compareFn: (left, right) => + (left as { groupId: number }).groupId - + (right as { groupId: number }).groupId, + }) + + const expectMatchesModel = (currentRows: typeof rows) => { + const ordered = [...currentRows].sort( + (left, right) => + left.value.groupId - right.value.groupId || + (left.key < right.key ? -1 : left.key > right.key ? 1 : 0), + ) + const forward = ordered.map(({ key }) => key) + expect(index.takeFromStart(currentRows.length)).toEqual(forward) + expect(index.takeReversedFromEnd(currentRows.length)).toEqual( + [...forward].reverse(), + ) + + for (const row of currentRows) { + expect(index.equalityLookup(row.value)).toEqual(new Set([row.key])) + expect(index.rangeQuery({ from: row.value, to: row.value })).toEqual( + new Set( + currentRows + .filter( + (candidate) => candidate.value.groupId === row.value.groupId, + ) + .map(({ key }) => key), + ), + ) + } + } + + for (const row of rows) index.add(row.key, row) + expectMatchesModel(rows) + + const removed = rows[0]! + index.remove(removed.key, removed) + expectMatchesModel(rows.slice(1)) + }) }) diff --git a/packages/db/tests/integration/uint8array-id-comparison.test.ts b/packages/db/tests/integration/uint8array-id-comparison.test.ts index 7b13c04f63..c0329a5937 100644 --- a/packages/db/tests/integration/uint8array-id-comparison.test.ts +++ b/packages/db/tests/integration/uint8array-id-comparison.test.ts @@ -79,8 +79,7 @@ describe(`Uint8Array ID comparison (user reproduction)`, () => { expect(resultByName?.name).toBe(makeItemName(selectedItemIndex)) }) - it(`should use reference equality for large Uint8Arrays (> 128 bytes)`, async () => { - // Create a large Uint8Array (> 128 bytes) that should use reference equality + it(`should use content equality for large Uint8Arrays`, async () => { const largeId = new Uint8Array(200).fill(42) interface LargeItem { @@ -102,7 +101,6 @@ describe(`Uint8Array ID comparison (user reproduction)`, () => { }), ) - // Query with the exact same reference - this should work const queryWithSameRef = createLiveQueryCollection((q) => q .from({ item: collection }) @@ -113,12 +111,9 @@ describe(`Uint8Array ID comparison (user reproduction)`, () => { await queryWithSameRef.preload() const resultWithSameRef = Array.from(queryWithSameRef.entries())[0]?.[1] - // Should find the item because we're using the same reference expect(resultWithSameRef).toBeDefined() expect(resultWithSameRef?.name).toBe(`Large Item`) - // Query with a different instance but same content - this will NOT work - // because large arrays use reference equality const differentInstance = new Uint8Array(200).fill(42) const queryWithDifferentRef = createLiveQueryCollection((q) => q @@ -132,8 +127,7 @@ describe(`Uint8Array ID comparison (user reproduction)`, () => { queryWithDifferentRef.entries(), )[0]?.[1] - // Should NOT find the item because large arrays use reference equality - // This is expected behavior to avoid memory overhead - expect(resultWithDifferentRef).toBeUndefined() + expect(resultWithDifferentRef).toBeDefined() + expect(resultWithDifferentRef?.name).toBe(`Large Item`) }) }) diff --git a/packages/db/tests/live-query-observer.test.ts b/packages/db/tests/live-query-observer.test.ts index 0ff8963811..fe74985f77 100644 --- a/packages/db/tests/live-query-observer.test.ts +++ b/packages/db/tests/live-query-observer.test.ts @@ -128,6 +128,82 @@ function makeControlledTruncateSource() { } describe(`createLiveQueryObserver`, () => { + it.each( + ([`granular`, `wholesale`] as const).flatMap((mode) => + ([`ordinary`, `reentrant`, `dispose`] as const).flatMap((scenario) => + [false, true].map((throwUndefined) => ({ + mode, + scenario, + throwUndefined, + })), + ), + ), + )( + `delivers peer publications before reporting a listener failure: %j`, + async ({ mode, scenario, throwUndefined }) => { + const source = makeSource() + const observer = createLiveQueryObserver(source, { mode }) + const firstError = throwUndefined + ? undefined + : new Error(`First listener failed`) + const secondError = new Error(`Peer listener failed`) + const peerRows = new Map() + const publications: Array> = [] + let armed = false + observer.subscribe(() => { + if (!armed) return + armed = false + if (scenario === `reentrant`) { + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `4`, name: `D` } }) + source.utils.commit() + } + if (scenario === `dispose`) observer.dispose() + throw firstError + }) + observer.subscribe((changes) => { + if (mode === `wholesale`) { + peerRows.clear() + for (const [key, row] of observer.getSnapshot().state ?? []) + peerRows.set(key, row) + } else { + for (const change of changes ?? []) { + if (change.type === `delete`) peerRows.delete(change.key) + else peerRows.set(change.key, change.value) + } + } + publications.push([...peerRows.keys()].sort()) + if (peerRows.has(`3`)) throw secondError + }) + publications.length = 0 + armed = true + try { + source.utils.begin() + source.utils.write({ type: `insert`, value: { id: `3`, name: `C` } }) + let caught: { error: unknown } | undefined + try { + source.utils.commit() + } catch (error) { + caught = { error } + } + expect(caught).toEqual({ error: firstError }) + expect(publications).toEqual( + scenario === `dispose` + ? [] + : scenario === `reentrant` + ? [ + [`1`, `2`, `3`], + [`1`, `2`, `3`, `4`], + ] + : [[`1`, `2`, `3`]], + ) + } finally { + observer.dispose() + await source.cleanup() + } + }, + ) + it(`registers SSR live-query resources for client-owned cleanup`, async () => { const errorSpy = vi.spyOn(console, `error`).mockImplementation(() => {}) const client = new DbClient() diff --git a/packages/db/tests/live-query-window-controller.test.ts b/packages/db/tests/live-query-window-controller.test.ts index 9ecd44d1eb..6526a2cafd 100644 --- a/packages/db/tests/live-query-window-controller.test.ts +++ b/packages/db/tests/live-query-window-controller.test.ts @@ -9,6 +9,7 @@ import { normalizeLiveQueryWindowPageSize, } from '../src/live-query-window-controller.js' import { mockSyncCollectionOptions } from './utils.js' +import { evaluateReferenceExpression } from './reference-expression.js' import type { Collection } from '../src/collection/index.js' interface Row { @@ -49,6 +50,42 @@ const flush = () => new Promise((r) => setTimeout(r, 0)) const ids = (snap: { data: ReadonlyArray }) => snap.data.map((r) => r.id) describe(`createLiveQueryWindowController`, () => { + it.each( + [0, 2, 5].flatMap((rowCount) => + [`fetch`, `reset`, `dispose`].map((action) => ({ rowCount, action })), + ), + )( + `handles $action during initial loading with $rowCount rows`, + async ({ rowCount, action }) => { + const source = makeSource(ROWS.slice(0, rowCount)) + const lq = makeOrderedLiveQuery(source, 2) + const controller = createLiveQueryWindowController(lq, { + pageSize: 2, + }) + const unsubscribe = controller.subscribe(() => {}) + try { + expect(controller.getSnapshot().isLoading).toBe(true) + const fetch = controller.fetchNextPage() + expect(controller.fetchNextPage()).toBe(fetch) + if (action === `reset`) await controller.reset() + if (action === `dispose`) controller.dispose() + await fetch + const visibleCount = action === `fetch` ? 4 : 2 + expect(ids(controller.getSnapshot())).toEqual( + ROWS.slice(0, Math.min(rowCount, visibleCount)).map((row) => row.id), + ) + expect(controller.getSnapshot().pages).toHaveLength( + action === `fetch` && rowCount > 2 ? 2 : 1, + ) + } finally { + unsubscribe() + controller.dispose() + await lq.cleanup() + await source.cleanup() + } + }, + ) + it.each([ { pageSize: undefined, normalized: 20 }, { pageSize: 0, normalized: 20 }, @@ -274,33 +311,45 @@ describe(`createLiveQueryWindowController`, () => { controller.dispose() }) - it(`restores the initial operator window when a graph run throws`, () => { + it(`retains the settled public window after a graph throw until retry`, async () => { const lq = makeOrderedLiveQuery(makeSource(), 2) + await lq.preload() + const settledRows = lq.toArray const builder = lq.utils[LIVE_QUERY_INTERNAL].getBuilder() const originalWindowFn = Reflect.get(builder, `windowFn`) as (options: { offset?: number limit?: number }) => void const windowFn = vi.fn(originalWindowFn) + const originalRunGraph = Reflect.get( + builder, + `maybeRunGraphFn`, + ) as () => void const requestedError = new Error(`requested window failed`) - const maybeRunGraph = vi - .fn() - .mockImplementationOnce(() => { - throw requestedError - }) - .mockImplementationOnce(() => { - throw new Error(`rollback failed`) - }) + const maybeRunGraph = vi.fn(originalRunGraph).mockImplementationOnce(() => { + throw requestedError + }) Reflect.set(builder, `windowFn`, windowFn) Reflect.set(builder, `maybeRunGraphFn`, maybeRunGraph) - expect(() => lq.utils.setWindow({ offset: 0, limit: 5 })).toThrow( - requestedError, - ) + let caught: unknown + try { + lq.utils.setWindow({ offset: 0, limit: 5 }) + } catch (error) { + caught = error + } + expect(caught).toBe(requestedError) expect(windowFn).toHaveBeenNthCalledWith(1, { offset: 0, limit: 5 }) - expect(windowFn).toHaveBeenNthCalledWith(2, { offset: 0, limit: 3 }) - expect(maybeRunGraph).toHaveBeenCalledTimes(2) + // Retain public state, not a rollback of the already advanced private graph. + expect(windowFn).toHaveBeenCalledTimes(1) + expect(maybeRunGraph).toHaveBeenCalledTimes(1) expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + expect(lq.toArray).toEqual(settledRows) + + await lq.utils.setWindow({ offset: 0, limit: 5 }) + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) + expect(lq.toArray.map(({ id, n }) => ({ id, n }))).toEqual(ROWS) + await lq.cleanup() }) it(`keeps the committed page retryable when a window load rejects`, async () => { @@ -384,33 +433,68 @@ describe(`createLiveQueryWindowController`, () => { controller.dispose() }) - it(`does not shrink the physical window when preload overlaps a page fetch`, async () => { - const lq = makeOrderedLiveQuery(makeSource(), 2) - const controller = createLiveQueryWindowController(lq as any, { - pageSize: 2, - }) - controller.subscribe(() => {}) - await lq.preload() + it.each([`resolve`, `reject`] as const)( + `preload joins a pending page fetch that will %s`, + async (outcome) => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + const controller = createLiveQueryWindowController( + lq as any, + { + pageSize: 2, + }, + ) + controller.subscribe(() => {}) + await lq.preload() - const originalSetWindow = lq.utils.setWindow.bind(lq.utils) - let resolveExpansion!: () => void - vi.spyOn(lq.utils, `setWindow`).mockImplementation((options) => { - const result = originalSetWindow(options) - if (options.limit !== 5) return result - return new Promise((resolve) => { - resolveExpansion = resolve - }) - }) + const originalSetWindow = lq.utils.setWindow.bind(lq.utils) + let resolveExpansion!: () => void + let rejectExpansion!: (error: unknown) => void + const failure = new Error(`expansion failed`) + const setWindow = vi + .spyOn(lq.utils, `setWindow`) + .mockImplementationOnce((options) => { + const pending = new Promise((resolve, reject) => { + resolveExpansion = resolve + rejectExpansion = reject + }) + return Promise.resolve(originalSetWindow(options)).then(() => pending) + }) - const expansion = controller.fetchNextPage() - const preload = controller.preload() - resolveExpansion() - await Promise.all([expansion, preload]) + const expansion = controller.fetchNextPage() + const expansionOutcome = expansion.catch((error: unknown) => error) + const preload = controller.preload() + let preloadSettled = false + const preloadOutcome = preload.then( + () => { + preloadSettled = true + }, + (error: unknown) => { + preloadSettled = true + return error + }, + ) + await flush() + expect(setWindow).toHaveBeenCalledTimes(1) + expect(preloadSettled).toBe(false) + if (outcome === `reject`) { + rejectExpansion(failure) + expect(await expansionOutcome).toBe(failure) + expect(await preloadOutcome).toBe(failure) + expect(controller.getSnapshot().pages).toHaveLength(1) + expect(controller.getSnapshot().error).toBe(failure) + await controller.fetchNextPage() + } else { + resolveExpansion() + expect(await expansionOutcome).toBeUndefined() + expect(await preloadOutcome).toBeUndefined() + } - expect(controller.getSnapshot().pages).toHaveLength(2) - expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) - controller.dispose() - }) + expect(controller.getSnapshot().pages).toHaveLength(2) + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`, `3`, `4`]) + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) + controller.dispose() + }, + ) it(`publishes source changes while a page fetch is pending`, async () => { const source = makeSource() @@ -457,8 +541,11 @@ describe(`createLiveQueryWindowController`, () => { sync: ({ begin, write, commit, markReady }) => { markReady() return { - loadSubset: (options) => - new Promise((resolve, reject) => { + loadSubset: (options) => { + // Boundary refinement asks only for the last loaded tie class. + // The source has already supplied that row. + if (options.where) return Promise.resolve() + return new Promise((resolve, reject) => { queueMicrotask(() => { if (rejectLoads) { reject(failure) @@ -471,7 +558,8 @@ describe(`createLiveQueryWindowController`, () => { commit() resolve() }) - }), + }) + }, } }, }, @@ -507,6 +595,8 @@ describe(`createLiveQueryWindowController`, () => { }) it(`reset supersedes an in-flight page expansion`, async () => { + // Isolate controller generations: the mock accepts reset without source work. + // The real source publication barrier is tested separately below. const lq = makeOrderedLiveQuery(makeSource(), 2) const controller = createLiveQueryWindowController(lq as any, { pageSize: 2, @@ -539,94 +629,169 @@ describe(`createLiveQueryWindowController`, () => { controller.dispose() }) - it(`reset does not inherit a superseded expansion failure`, async () => { - const failure = new Error(`superseded expansion failed`) - let loadCount = 0 - const rejectLoads = new Map void>() - const loaded = new Set() - const source = createCollection({ - id: `window-reset-real-source-${seq++}`, - getKey: (row) => row.id, - syncMode: `on-demand`, - startSync: true, - autoIndex: `eager`, - defaultIndexType: BTreeIndex, - sync: { - sync: ({ begin, write, commit, markReady }) => { - markReady() - return { - loadSubset: (options) => { - loadCount++ - if (loadCount === 2) { - return new Promise((_resolve, reject) => { - rejectLoads.set(loadCount, reject) + it.each([`resolve`, `reject`] as const)( + `reset waits for publication-blocking source work that will %s`, + async (outcome) => { + const failure = new Error(`superseded expansion failed`) + let holdNextRequest = false + let settleExpansion: (() => void) | undefined + const loaded = new Set() + const source = createCollection({ + id: `window-reset-real-source-${seq++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + const matching = ROWS.filter( + (row) => + !options.where || + evaluateReferenceExpression(options.where, row) === true, + ) + const cursor = options.cursor + const from = cursor + ? matching.filter( + (row) => + evaluateReferenceExpression(cursor.whereFrom, row) === + true, + ) + : matching.slice(options.offset ?? 0) + const limited = from.slice(0, options.limit) + const requested = cursor + ? matching.filter( + (row) => + limited.includes(row) || + evaluateReferenceExpression( + cursor.whereCurrent, + row, + ) === true, + ) + : limited + const apply = () => { + begin() + requested.forEach((row) => { + if (loaded.has(row.id)) return + loaded.add(row.id) + write({ type: `insert`, value: row }) + }) + return commit() + } + if (!holdNextRequest) + return Promise.resolve(apply()).then(() => {}) + holdNextRequest = false + return new Promise((resolve, reject) => { + settleExpansion = () => { + if (outcome === `reject`) reject(failure) + else + void Promise.resolve(apply()).then( + () => resolve(), + reject, + ) + } }) - } - begin() - ROWS.slice(0, options.limit).forEach((row) => { - if (loaded.has(row.id)) return - loaded.add(row.id) - write({ type: `insert`, value: row }) - }) - commit() - return Promise.resolve() - }, - } + }, + } + }, }, - }, - }) - const lq = makeOrderedLiveQuery(source, 2) - const controller = createLiveQueryWindowController(lq as any, { - pageSize: 2, - }) - controller.subscribe(() => {}) - - try { - await controller.preload() - const expansion = Promise.resolve(controller.fetchNextPage()) - expect(loadCount).toBe(2) - const rejectExpansion = rejectLoads.get(2) - expect(rejectExpansion).toBeDefined() - const reset = Promise.resolve(controller.reset()) - void expansion.catch(() => undefined) - void reset.catch(() => undefined) - - rejectExpansion!(failure) - - await expect(reset).resolves.toBeUndefined() - await expect(expansion).rejects.toBe(failure) - expect(controller.getSnapshot().pages).toHaveLength(1) - } finally { - controller.dispose() - await Promise.all([lq.cleanup(), source.cleanup()]) - } - }) - - it(`cleanup settles the active load operation before another sync session`, async () => { - const lq = makeOrderedLiveQuery(makeSource(), 2) - await lq.preload() - - let resolveLoad!: () => void - const load = new Promise((resolve) => { - resolveLoad = resolve - }) - const operation = lq._sync.beginLoadSubsetOperation() - lq._sync.trackLoadPromise(load) - const waiting = Promise.resolve(operation.wait()) - let settled = false - void waiting.then(() => { - settled = true - }) + }) + const lq = makeOrderedLiveQuery(source, 2) + const controller = createLiveQueryWindowController( + lq as any, + { + pageSize: 2, + }, + ) + controller.subscribe(() => {}) + + try { + await controller.preload() + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`]) + holdNextRequest = true + const expansion = Promise.resolve(controller.fetchNextPage()) + const expansionOutcome = expansion.catch((error: unknown) => error) + expect(holdNextRequest).toBe(false) + expect(settleExpansion).toBeTypeOf(`function`) + const reset = Promise.resolve(controller.reset()) + let resetSettled = false + const resetOutcome = reset.then( + () => { + resetSettled = true + }, + (error: unknown) => { + resetSettled = true + return error + }, + ) + await flush() + expect(resetSettled).toBe(false) + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`]) + settleExpansion!() + if (outcome === `reject`) { + expect(await resetOutcome).toBe(failure) + expect(await expansionOutcome).toBe(failure) + expect(controller.getSnapshot().error).toBe(failure) + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`]) + await controller.reset() + } else { + expect(await resetOutcome).toBeUndefined() + expect(await expansionOutcome).toBeUndefined() + } + expect(controller.getSnapshot().pages).toHaveLength(1) + expect(controller.getSnapshot().isError).toBe(false) + expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + await controller.fetchNextPage() + expect(ids(controller.getSnapshot())).toEqual([`1`, `2`, `3`, `4`]) + } finally { + controller.dispose() + await Promise.all([lq.cleanup(), source.cleanup()]) + } + }, + ) - lq._sync.cleanup() - await Promise.resolve() + it.each( + [false, true].flatMap((waitBeforeCleanup) => + [false, true].map((pendingLoad) => ({ waitBeforeCleanup, pendingLoad })), + ), + )( + `cleanup cancels unfinished operations with waitFirst=$waitBeforeCleanup, pending=$pendingLoad`, + async ({ waitBeforeCleanup, pendingLoad }) => { + const lq = makeOrderedLiveQuery(makeSource(), 2) + await lq.preload() - expect(settled).toBe(true) + let resolveLoad!: () => void + const load = new Promise((resolve) => { + resolveLoad = resolve + }) + const operation = lq._sync.beginLoadSubsetOperation() + if (pendingLoad) lq._sync.trackLoadPromise(load) + const beforeCleanup = waitBeforeCleanup ? operation.wait() : undefined + const observe = (result: true | Promise) => + Promise.resolve(result).then( + () => undefined, + (error: unknown) => error, + ) + const observedBefore = + beforeCleanup === undefined ? undefined : observe(beforeCleanup) + lq._sync.cleanup() + const observed = observedBefore ?? observe(operation.wait()) + const outcome = await observed + if (waitBeforeCleanup && !pendingLoad) { + expect(beforeCleanup).toBe(true) + expect(outcome).toBeUndefined() + } else { + expect(outcome).toMatchObject({ name: `AbortError` }) + } - resolveLoad() - await waiting - await lq.cleanup() - }) + resolveLoad() + expect(await observed).toBe(outcome) + await lq.cleanup() + }, + ) it(`cleanup settles every superseded load operation`, async () => { const lq = makeOrderedLiveQuery(makeSource(), 2) @@ -642,10 +807,10 @@ describe(`createLiveQueryWindowController`, () => { lq._sync.trackLoadPromise(secondLoad) const secondWaiting = Promise.resolve(secondOperation.wait()) const settled = [false, false] - void firstWaiting.then(() => { + void firstWaiting.catch(() => { settled[0] = true }) - void secondWaiting.then(() => { + void secondWaiting.catch(() => { settled[1] = true }) @@ -653,7 +818,8 @@ describe(`createLiveQueryWindowController`, () => { await Promise.resolve() expect(settled).toEqual([true, true]) - await Promise.all([firstWaiting, secondWaiting]) + await expect(firstWaiting).rejects.toMatchObject({ name: `AbortError` }) + await expect(secondWaiting).rejects.toMatchObject({ name: `AbortError` }) await lq.cleanup() }) @@ -765,8 +931,13 @@ describe(`createLiveQueryWindowController`, () => { await smaller.fetchNextPage() expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) + const setWindow = vi.spyOn(lq.utils, `setWindow`) larger.dispose() + expect(setWindow).toHaveBeenLastCalledWith({ offset: 0, limit: 3 }) + expect(setWindow.mock.results.at(-1)!.type).toBe(`return`) + await setWindow.mock.results.at(-1)!.value expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + expect(lq.toArray.map(({ id, n }) => ({ id, n }))).toEqual(ROWS.slice(0, 3)) smaller.dispose() }) @@ -853,10 +1024,14 @@ describe(`createLiveQueryWindowController`, () => { await controller.fetchNextPage() expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 5 }) + const setWindow = vi.spyOn(lq.utils, `setWindow`) unsubscribe() + expect(setWindow).toHaveBeenLastCalledWith({ offset: 0, limit: 3 }) + expect(setWindow.mock.results.at(-1)!.type).toBe(`return`) + await setWindow.mock.results.at(-1)!.value expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) - expect(lq.toArray).toHaveLength(3) + expect(lq.toArray.map(({ id, n }) => ({ id, n }))).toEqual(ROWS.slice(0, 3)) controller.dispose() await lq.cleanup() }) @@ -891,7 +1066,13 @@ describe(`createLiveQueryWindowController`, () => { const unsubscribeSecond = second.subscribe(() => {}) unsubscribeSecond() + expect(setWindow).toHaveBeenLastCalledWith({ offset: 0, limit: 4 }) + expect(setWindow.mock.results.at(-1)!.type).toBe(`return`) + await setWindow.mock.results.at(-1)!.value expect(lq.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) + expect(lq.toArray.map(({ id, n }) => ({ id, n }))).toEqual( + ROWS.slice(0, 4), + ) first.dispose() second.dispose() await lq.cleanup() diff --git a/packages/db/tests/local-storage-persistence-failure.test.ts b/packages/db/tests/local-storage-persistence-failure.test.ts new file mode 100644 index 0000000000..6aff40db6a --- /dev/null +++ b/packages/db/tests/local-storage-persistence-failure.test.ts @@ -0,0 +1,106 @@ +import { expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index' +import { localStorageCollectionOptions } from '../src/local-storage' +import { createTransaction } from '../src/transactions' + +type Row = { id: string; value: number } + +const cases = ([`insert`, `update`, `delete`] as const).flatMap((operation) => + ([`storage`, `serialization`] as const).flatMap((failure) => + [false, true].map((manual) => ({ operation, failure, manual })), + ), +) + +it.each(cases)( + `does not persist a rejected mutation on the next successful write: %j`, + async ({ operation, failure, manual }) => { + const data = new Map() + const error = new Error(`Persistence failed`) + let fail = false + const storage = { + getItem: (key: string) => data.get(key) ?? null, + removeItem: (key: string) => { + data.delete(key) + }, + setItem: (key: string, value: string) => { + if (fail && failure === `storage`) throw error + data.set(key, value) + }, + } + const makeCollection = () => + createCollection( + localStorageCollectionOptions({ + storageKey: `rows`, + storage, + storageEventApi: { addEventListener() {}, removeEventListener() {} }, + getKey: (row) => row.id, + parser: { + parse: JSON.parse, + stringify: (value: unknown) => { + // Per-row validation succeeds; serializing the full stored map fails. + if ( + fail && + failure === `serialization` && + typeof value === `object` && + value !== null && + !(`id` in value) + ) { + throw error + } + return JSON.stringify(value) + }, + }, + }), + ) + const collection = makeCollection() + const log = vi.spyOn(console, `error`).mockImplementation(() => {}) + try { + await collection.preload() + await collection.insert({ id: `seed`, value: 1 }).isPersisted.promise + const mutate = () => { + if (operation === `insert`) + return collection.insert({ id: `bad`, value: 2 }) + if (operation === `delete`) return collection.delete(`seed`) + return collection.update(`seed`, (draft) => { + draft.value = 2 + }) + } + fail = true + const failed = manual + ? createTransaction({ + autoCommit: false, + mutationFn: ({ transaction }) => { + collection.utils.acceptMutations(transaction) + return Promise.resolve() + }, + }) + : mutate() + const rejection = expect(failed.isPersisted.promise).rejects.toBe(error) + if (manual) { + failed.mutate(mutate) + await failed.commit().catch(() => {}) + } + await rejection + fail = false + await collection.insert({ id: `good`, value: 3 }).isPersisted.promise + const restored = makeCollection() + try { + await restored.preload() + expect( + [...restored.values()] + .map(({ id, value }) => ({ id, value })) + .sort((a, b) => a.id.localeCompare(b.id)), + ).toEqual([ + { id: `good`, value: 3 }, + { id: `seed`, value: 1 }, + ]) + } finally { + await restored.cleanup() + } + } finally { + fail = false + await collection.cleanup() + log.mockRestore() + } + }, +) diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 2a0375432a..ec7731bbdb 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -1,8 +1,148 @@ type OracleEnvironment = Record +const staticOracleProperties = [ + `collection-sync.reentrant-drain`, + `collection-state.retention`, + `collection-publication.metadata-cancellation`, + `collection-publication.metadata-only`, + `collection-publication.metadata-rollback`, + `coverage-registry.claim-churn`, + `coverage-registry.state-machine`, + `d2-source.exact-retractions`, + `d2-source.disjoint-commutation`, + `includes-collection.layout-swap`, + `includes-collection.optimistic-child-history`, + `includes-collection.public-key-order`, + `includes-collection.relationship-history`, + `includes-cross-formulation.equivalence`, + `includes-cross-formulation.ordered-window`, + `includes-cross-formulation.reference-context`, + `includes-cross-formulation.reference-key`, + `includes-cross-formulation.symbol-group-route`, + `includes-optimistic.ancestor-rollback`, + `includes-optimistic.confirm-different-route`, + `includes-optimistic.confirm-same-route`, + `includes-optimistic.descendant-rollback`, + `includes-optimistic.rekey-detach`, + `includes-optimistic.rekey-rollback`, + `includes-optimistic.repeated-history`, + `includes-optimistic.sibling-route-rollback`, + `includes-publication.atomic-parent-replacement`, + `includes-publication.child-scalar`, + `includes-publication.optimistic-rollback`, + `includes-publication.parent-route`, + `includes-temporal.release-reentry`, + `includes-temporal.demand-scheduling`, + `includes.alpha-renaming`, + `includes.incremental-history`, + `includes.nested-scalar-materialization`, + `includes.optimistic-convergence`, + `includes.scenario-statistics`, + `load-subset-full-flow.atomic-replacement`, + `load-subset-full-flow.automatic-progress`, + `load-subset-full-flow.boundary-provenance`, + `load-subset-full-flow.consumer-parity`, + `load-subset-full-flow.continuation-evidence`, + `load-subset-full-flow.continuation-statistics`, + `load-subset-full-flow.multi-source-ordered`, + `load-subset-full-flow.multi-source-statistics`, + `load-subset-full-flow.truncate-evidence`, + `load-subset-lifecycle.state-machine`, + `load-subset.async-settlement`, + `load-subset.changing-predicate`, + `load-subset.concurrent-dedupe`, + `load-subset.coverage`, + `load-subset.distinct-window-predicate`, + `load-subset.exact-completion`, + `load-subset.exact-inflight`, + `load-subset.ordered-window`, + `load-subset.rejected-waiter`, + `ordered-work.forward-exhaustion`, + `ordered-work.forward-prefix`, + `ordered-work.custom-comparator-fallback`, + `ordered-work.public-key-suffix`, + `ordered-work.reverse-exhaustion`, + `ordered-work.reverse-prefix`, + `ordered-work.snapshot-reuse`, + `ordered-work.consumer-parity`, + `ordered-work.lifecycle`, + `pagination.async-cursor`, + `pagination.multi-order`, + `pagination.nullable-cursor`, + `pagination.ordered-window`, + `pagination.pending-history`, + `pagination.pending-mutation`, + `pagination.window-transition`, + `predicate-subtraction.duplicate-terms`, + `predicate-subtraction.finite-world`, + `predicate-subtraction.unbounded`, + `subscription-replay.completion`, + `subscription-replay.optimistic`, + `subscription-replay.ownership`, + `subscription-replay.restart`, + `subscription-replay.sequential`, + `subscription-replay.shared`, + `subscription-replay.same-tick`, + `subscription-lifecycle.history-statistics`, + `subscription-lifecycle.publication-history`, + `subscription-lifecycle.sync-history`, + `subscription-lifecycle.async-history`, + `subscription-lifecycle.async-restart`, + `subscription-lifecycle.async-statistics`, +] as const + +const publicationProperties = [ + `parent-scalar`, + `parent-then-child`, + `optimistic-before-confirm`, + `optimistic-after-confirm`, +].flatMap((law) => + [`direct`, `joined`].flatMap((q1Shape) => + [`passThrough`, `where`, `orderBy`, `select`].map( + (q2Shape) => `includes-publication.${law}.${q1Shape}.${q2Shape}`, + ), + ), +) + +const refinementProperties = Array.from( + { length: 11 }, + (_, index) => `load-subset-refinement.${1_779_001 + index}`, +) + +export function validateOraclePropertyRegistry( + properties: ReadonlyArray, +): ReadonlySet { + const registry = new Set() + for (const property of properties) { + if (registry.has(property)) { + throw new Error(`duplicate oracle property: ${property}`) + } + registry.add(property) + } + return registry +} + +const registeredOracleProperties = validateOraclePropertyRegistry([ + ...staticOracleProperties, + ...publicationProperties, + ...refinementProperties, +]) + +function assertRegisteredOracleProperty(property: string): void { + if (!registeredOracleProperties.has(property)) { + throw new Error(`unknown oracle property: ${property}`) + } +} + +export type OracleReplayConfig = { + replaySeed: number | undefined + replayPath: string | undefined + replayProperty: string | undefined +} + export function readOracleRunConfig( environment: OracleEnvironment = process.env, -): { multiplier: number; replaySeed: number | undefined } { +): OracleReplayConfig & { multiplier: number } { const multiplierValue = environment.TANSTACK_DB_ORACLE_RUNS_MULTIPLIER ?? `1` const multiplier = Number(multiplierValue) if ( @@ -16,23 +156,88 @@ export function readOracleRunConfig( } const seedValue = environment.TANSTACK_DB_ORACLE_SEED - if (seedValue === undefined) return { multiplier, replaySeed: undefined } + const replayPath = environment.TANSTACK_DB_ORACLE_PATH + const replayProperty = environment.TANSTACK_DB_ORACLE_PROPERTY + if (seedValue === undefined) { + if (replayPath !== undefined) { + throw new Error( + `TANSTACK_DB_ORACLE_PATH requires TANSTACK_DB_ORACLE_SEED`, + ) + } + if (replayProperty !== undefined) { + throw new Error( + `TANSTACK_DB_ORACLE_PROPERTY requires TANSTACK_DB_ORACLE_PATH`, + ) + } + return { + multiplier, + replaySeed: undefined, + replayPath: undefined, + replayProperty: undefined, + } + } const replaySeed = Number(seedValue) if (seedValue.trim() === `` || !Number.isSafeInteger(replaySeed)) { throw new Error(`TANSTACK_DB_ORACLE_SEED must be an integer`) } - return { multiplier, replaySeed } + if (replayPath === undefined) { + if (replayProperty !== undefined) { + throw new Error( + `TANSTACK_DB_ORACLE_PROPERTY requires TANSTACK_DB_ORACLE_PATH`, + ) + } + return { + multiplier, + replaySeed, + replayPath: undefined, + replayProperty: undefined, + } + } + if (replayPath.trim() === ``) { + throw new Error(`TANSTACK_DB_ORACLE_PATH must be non-empty`) + } + if (!/^\d+(?::\d+)*$/.test(replayPath)) { + throw new Error( + `TANSTACK_DB_ORACLE_PATH must contain colon-separated nonnegative integers`, + ) + } + if (replayProperty === undefined || replayProperty.trim() === ``) { + throw new Error( + `TANSTACK_DB_ORACLE_PATH requires TANSTACK_DB_ORACLE_PROPERTY`, + ) + } + assertRegisteredOracleProperty(replayProperty) + return { multiplier, replaySeed, replayPath, replayProperty } } export function oracleRandomParameters( numRuns: number, - replaySeed: number | undefined, -): { numRuns: number; seed?: number } { - return replaySeed === undefined ? { numRuns } : { numRuns, seed: replaySeed } + replay: OracleReplayConfig | number | undefined, + property?: string, +): { numRuns: number; seed?: number; path?: string } { + if (property !== undefined) assertRegisteredOracleProperty(property) + const { replaySeed, replayPath, replayProperty } = + typeof replay === `object` + ? replay + : { + replaySeed: replay, + replayPath: undefined, + replayProperty: undefined, + } + if (replaySeed === undefined) return { numRuns } + return { + numRuns, + seed: replaySeed, + ...(property !== undefined && + replayPath !== undefined && + replayProperty === property + ? { path: replayPath } + : {}), + } } -const { multiplier, replaySeed: seed } = readOracleRunConfig() +const { multiplier, ...replay } = readOracleRunConfig() /** Keeps ordinary CI bounded while allowing long randomized oracle campaigns. */ export function oracleRuns(baseRuns: number): number { @@ -40,12 +245,13 @@ export function oracleRuns(baseRuns: number): number { } /** Replays broad randomized properties when a campaign seed is supplied. */ -export function oraclePropertyOptions(baseRuns: number): { +export function oraclePropertyOptions( + baseRuns: number, + property?: string, +): { numRuns: number seed?: number + path?: string } { - return { - numRuns: oracleRuns(baseRuns), - ...(seed === undefined ? {} : { seed }), - } + return oracleRandomParameters(oracleRuns(baseRuns), replay, property) } diff --git a/packages/db/tests/proxy-iteration-contract.test.ts b/packages/db/tests/proxy-iteration-contract.test.ts new file mode 100644 index 0000000000..51c2db3e7f --- /dev/null +++ b/packages/db/tests/proxy-iteration-contract.test.ts @@ -0,0 +1,343 @@ +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../src/collection/index.js' +import { createChangeProxy, withChangeTracking } from '../src/proxy.js' + +// Drafts preserve native live membership, even if a snapshot iterator would +// make mutation tracking simpler. Nested field edits have separate laws. +describe.each([`Map`, `Set`] as const)(`%s draft iteration`, (kind) => { + it(`calls a read-only forEach callback once per entry without reporting changes`, () => { + const values = + kind === `Map` ? new Map([[1, { x: 1 }]]) : new Set([{ x: 1 }]) + const context = {} + const callback = + vi.fn<(value: unknown, key: unknown, collection: unknown) => void>() + const { proxy: draft, getChanges } = createChangeProxy({ values }) + const draftValues = draft.values + draftValues.forEach(callback, context) + expect(callback).toHaveBeenCalledTimes(1) + expect(getChanges()).toEqual({}) + expect(callback.mock.contexts).toEqual([context]) + const [value, key, collection] = callback.mock.calls[0]! + expect(collection).toBe(draftValues) + if (kind === `Set`) expect(key).toBe(value) + }) + + it(`rejects an invalid forEach callback even when empty`, () => { + const values = kind === `Map` ? new Map() : new Set() + withChangeTracking({ values }, (draft) => { + expect(() => draft.values.forEach(null!)).toThrow(TypeError) + }) + }) + it(`visits entries added before consuming an existing iterator`, () => { + const values = kind === `Map` ? new Map([[1, 1]]) : new Set([1]) + withChangeTracking({ values }, (draft) => { + const iterator = draft.values.values() + if (draft.values instanceof Map) draft.values.set(2, 2) + else draft.values.add(2) + expect([...iterator]).toEqual([1, 2]) + }) + }) + + it(`skips entries deleted before consuming an existing iterator`, () => { + const values = + kind === `Map` + ? new Map([ + [1, 1], + [2, 2], + ]) + : new Set([1, 2]) + withChangeTracking({ values }, (draft) => { + const iterator = draft.values.values() + draft.values.delete(2) + expect([...iterator]).toEqual([1]) + }) + }) +}) + +type Item = { x: number } +const protocols = [`values`, `entries`, `iterator`, `forEach`] as const +type Protocol = (typeof protocols)[number] + +function visit( + values: Map | Set, + protocol: Protocol, + callback: (value: Item) => void, +) { + switch (protocol) { + case `forEach`: + values.forEach(callback) + break + case `entries`: + for (const [, value] of values.entries()) callback(value) + break + case `values`: + for (const value of values.values()) callback(value) + break + case `iterator`: + if (values instanceof Map) for (const [, value] of values) callback(value) + else for (const value of values) callback(value) + } +} + +describe.each([`Map`, `Set`] as const)(`%s nested iteration laws`, (kind) => { + it.each(protocols)( + `%s tracks each nested edit once and leaves the input untouched`, + (protocol) => { + const input = [{ x: 1 }, { x: 2 }] + const values = + kind === `Map` + ? new Map(input.map((value) => [value.x, value])) + : new Set(input) + let visits = 0 + const changes = withChangeTracking({ values }, (draft) => { + visit(draft.values, protocol, (value) => { + if (++visits > 2) throw new Error(`An edit reinserted an entry`) + value.x += 10 + }) + }) + expect(visits).toBe(2) + expect([...(changes.values as typeof values).values()]).toEqual([ + { x: 11 }, + { x: 12 }, + ]) + expect([...values.values()]).toEqual([{ x: 1 }, { x: 2 }]) + }, + ) + + it.each(protocols)( + `%s can write and revert without revisiting entries or reporting changes`, + (protocol) => { + const values = + kind === `Map` ? new Map([[1, { x: 1 }]]) : new Set([{ x: 1 }]) + let visits = 0 + const changes = withChangeTracking({ values }, (draft) => { + visit(draft.values, protocol, (value) => { + if (++visits > 1) throw new Error(`An edit reinserted an entry`) + value.x = 2 + value.x = 1 + }) + }) + expect(visits).toBe(1) + expect(changes).toEqual({}) + }, + ) + + it.each(protocols)( + `%s preserves sibling changes when another entry reverts`, + (protocol) => { + const values = + kind === `Map` + ? new Map([ + [1, { x: 1 }], + [2, { x: 2 }], + ]) + : new Set([{ x: 1 }, { x: 2 }]) + let visits = 0 + const changes = withChangeTracking({ values }, (draft) => { + visit(draft.values, protocol, (value) => { + if (++visits > 2) throw new Error(`An edit reinserted an entry`) + const original = value.x + value.x += 10 + if (original === 2) value.x = original + }) + }) + expect([...(changes.values as typeof values).values()]).toEqual([ + { x: 11 }, + { x: 2 }, + ]) + }, + ) + + it.each(protocols)( + `%s matches native membership changes during iteration`, + (protocol) => { + const run = (values: Map | Set) => { + const seen: Array = [] + visit(values, protocol, (value) => { + if (seen.length > 5) throw new Error(`Iteration did not terminate`) + seen.push(value.x) + value.x += 10 + if (seen.length === 1) { + if (values instanceof Map) { + values.delete(2) + values.set(3, { x: 3 }) + } else { + values.delete([...values][1]!) + values.add({ x: 3 }) + } + } + }) + return { seen, values: [...values.values()] } + } + const make = () => + kind === `Map` + ? new Map([ + [1, { x: 1 }], + [2, { x: 2 }], + ]) + : new Set([{ x: 1 }, { x: 2 }]) + const expected = run(make()) + const changes = withChangeTracking({ values: make() }, (draft) => { + expect(run(draft.values)).toEqual(expected) + }) + expect([ + ...(changes.values as Map | Set).values(), + ]).toEqual(expected.values) + }, + ) + + it(`keeps newly added values private and supports chained mutators`, () => { + const item = { x: 1 } + const values = kind === `Map` ? new Map() : new Set() + const changes = withChangeTracking({ values }, (draft) => { + if (draft.values instanceof Map) { + expect(draft.values.set(`a`, item).set(`b`, item)).toBe(draft.values) + draft.values.get(`a`)!.x = 2 + expect(draft.values.get(`b`)!.x).toBe(2) + } else { + expect(draft.values.add(item).add(item)).toBe(draft.values) + expect(draft.values.has(item)).toBe(true) + draft.values.values().next().value!.x = 2 + expect(draft.values.size).toBe(1) + } + }) + expect(item.x).toBe(1) + expect( + [...(changes.values as typeof values).values()].every( + (value) => value.x === 2, + ), + ).toBe(true) + }) +}) + +it.each(protocols)( + `Set %s observes clear and re-add after an edit just like a native iterator`, + (protocol) => { + const run = (values: Set) => { + const seen: Array = [] + visit(values, protocol, (value) => { + if (seen.length > 2) throw new Error(`Iteration did not terminate`) + seen.push(value.x) + if (seen.length === 1) { + value.x = 3 + values.clear() + values.add(value) + } + }) + return seen + } + const expected = run(new Set([{ x: 1 }, { x: 2 }])) + const changes = withChangeTracking( + { values: new Set([{ x: 1 }, { x: 2 }]) }, + (draft) => { + expect(run(draft.values)).toEqual(expected) + }, + ) + expect(changes.values).toEqual(new Set([{ x: 3 }])) + }, +) + +it.each(protocols)( + `Set %s keys and values expose the same draft handle`, + (protocol) => { + const changes = withChangeTracking( + { values: new Set([{ x: 1 }]) }, + (draft) => { + const key = draft.values.keys().next().value! + visit(draft.values, protocol, (value) => expect(value).toBe(key)) + key.x = 2 + }, + ) + expect(changes.values).toEqual(new Set([{ x: 2 }])) + }, +) + +it.each([...protocols, `keys`] as const)( + `Set %s handles retain membership after editing`, + (protocol) => { + const changes = withChangeTracking( + { values: new Set([{ x: 1 }]) }, + (draft) => { + let visits = 0 + const edit = (value: Item) => { + if (++visits > 1) throw new Error(`An edit reinserted an entry`) + value.x = 2 + expect(draft.values.has(value)).toBe(true) + expect(draft.values.add(value)).toBe(draft.values) + draft.values.add(value) + expect(draft.values.size).toBe(1) + expect(draft.values.delete(value)).toBe(true) + expect(draft.values.has(value)).toBe(false) + } + if (protocol === `keys`) + for (const value of draft.values.keys()) edit(value) + else visit(draft.values, protocol, edit) + }, + ) + expect(changes.values).toEqual(new Set()) + }, +) + +it(`Map for-of nested writes reach collection.update`, async () => { + const collection = createCollection<{ + id: number + values: Map + }>({ + getKey: (row) => row.id, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { id: 1, values: new Map([[`a`, { x: 1 }]]) }, + }) + commit() + markReady() + }, + }, + onUpdate: async () => {}, + }) + try { + const tx = collection.update(1, (draft) => { + for (const [, value] of draft.values) value.x = 2 + }) + expect(tx.mutations[0]?.changes.values).toEqual(new Map([[`a`, { x: 2 }]])) + expect(collection.get(1)?.values.get(`a`)?.x).toBe(2) + } finally { + await collection.cleanup() + } +}) + +it.each([`entries`, `values`] as const)( + `taking one Map %s value does not scan every entry`, + (protocol) => { + const { proxy } = createChangeProxy({ + values: new Map(Array.from({ length: 1000 }, (_, i) => [i, i])), + }) + const values = proxy.values + let visits = 0 + const original = Map.prototype.entries + const spy = vi.spyOn(Map.prototype, `entries`).mockImplementation(function ( + this: Map, + ) { + const iterator = original.call(this) + const next = iterator.next.bind(iterator) + iterator.next = () => { + const result = next() + if (!result.done) visits++ + return result + } + return iterator + }) + try { + expect(values[protocol]().next()).toEqual({ + done: false, + value: protocol === `entries` ? [0, 0] : 0, + }) + expect(visits).toBe(1) + } finally { + spy.mockRestore() + } + }, +) diff --git a/packages/db/tests/proxy.test.ts b/packages/db/tests/proxy.test.ts index bbac0151af..d074426b76 100644 --- a/packages/db/tests/proxy.test.ts +++ b/packages/db/tests/proxy.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it } from 'vitest' +import { describe, expect, it, vi } from 'vitest' import { Temporal } from 'temporal-polyfill' import { createArrayChangeProxy, @@ -8,6 +8,32 @@ import { } from '../src/proxy' describe(`Proxy Library`, () => { + it.each([null, `true`])( + `tracks reads, writes and reverts without consulting DEBUG=%s`, + (debug) => { + const getItem = vi.fn(() => debug) + const log = vi.spyOn(console, `log`).mockImplementation(() => {}) + vi.stubGlobal(`localStorage`, { getItem }) + try { + const original = { value: 1, nested: { value: 2 } } + const { proxy, getChanges } = createChangeProxy(original) + expect(proxy.value).toBe(1) + proxy.value = 3 + proxy.nested.value = 4 + expect(getChanges()).toEqual({ value: 3, nested: { value: 4 } }) + proxy.value = 1 + proxy.nested.value = 2 + expect(getChanges()).toEqual({}) + expect(original).toEqual({ value: 1, nested: { value: 2 } }) + expect(getItem).not.toHaveBeenCalled() + expect(log).not.toHaveBeenCalled() + } finally { + vi.unstubAllGlobals() + log.mockRestore() + } + }, + ) + describe(`createChangeProxy`, () => { it(`should track changes to an object`, () => { const obj = { name: `John`, age: 30 } diff --git a/packages/db/tests/query/bucket-facade-adapter.test.ts b/packages/db/tests/query/bucket-facade-adapter.test.ts index 4b969e2114..d3999946e8 100644 --- a/packages/db/tests/query/bucket-facade-adapter.test.ts +++ b/packages/db/tests/query/bucket-facade-adapter.test.ts @@ -3,6 +3,7 @@ import { describe, expect, it, vi } from 'vitest' import { createCollection } from '../../src/collection/index.js' import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' import { eq } from '../../src/query/builder/functions.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' import { BucketFacadeAdapter } from '../../src/query/live/bucket-facade-adapter.js' import { CollectionConfigBuilder } from '../../src/query/live/collection-config-builder.js' import { BUCKET_FACADE_REF } from '../../src/query/live/materialized-pipeline.js' @@ -17,30 +18,249 @@ import type { Context } from '../../src/query/builder/types.js' type FacadeSync = Parameters>[`sync`]>[0] +class ThrowingBuildIndex extends BasicIndex { + throwBeforeBuild = false + throwOnBuild = false + + override build(entries: Iterable<[number, unknown]>): void { + if (this.throwBeforeBuild) { + throw new Error(`facade index rebuild failed`) + } + super.build(entries) + if (this.throwOnBuild) { + throw new Error(`facade index rebuild failed`) + } + } +} + describe(`BucketFacadeAdapter`, () => { - it(`restores facade state when a flush fails after writing`, async () => { + it.each( + [false, true].flatMap((present) => + [`insert`, `replace`, `cancel`].map((change) => ({ present, change })), + ), + )( + `publishes consolidated membership: $present / $change`, + async ({ present, change }) => { + const graph = new D2() + const rows = graph.newInput<[string, BucketRow]>() + const activeBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `public-membership`, + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: false }], + () => {}, + ) + graph.finalize() + const ref: BucketFacadeRef = { + [BUCKET_FACADE_REF]: { edgeId: `children`, bucketKey: `group` }, + } + const oldRow: BucketRow = { + publicKey: 1, + value: { id: 1, name: `old` }, + order: undefined, + } + const newRow: BucketRow = { + publicKey: 1, + value: { id: 1, name: `new` }, + order: undefined, + } + const send = (row: BucketRow, weight: number) => + rows.sendData(new MultiSet([[[`group`, row], weight]])) + try { + activeBuckets.sendData(new MultiSet([[[`group`, true], 1]])) + if (present) send(oldRow, 1) + graph.run() + adapter.flush().publish() + if (change === `cancel`) { + send(present ? oldRow : newRow, 1) + send(present ? oldRow : newRow, -1) + } else { + if (change === `replace`) send(oldRow, -1) + send(newRow, 1) + } + graph.run() + const published = adapter.resolve(ref) as unknown as Collection< + { id: number; name: string }, + number + > + const expected = + change !== `insert` && !present + ? [] + : [{ id: 1, name: change === `cancel` ? `old` : `new` }] + expect(published.toArray.map(stripVirtualProps)).toEqual( + present ? [{ id: 1, name: `old` }] : [], + ) + adapter.flush().publish() + expect(adapter.resolve(ref)).toBe(published) + expect(published.toArray.map(stripVirtualProps)).toEqual(expected) + } finally { + await adapter.cleanup() + } + }, + ) + + it.each( + [10, 100].flatMap((size) => + [false, true].map((ordered) => ({ size, ordered })), + ), + )( + `reads a $size-row facade without repeated scans (ordered=$ordered)`, + ({ size, ordered }) => { + const graph = new D2() + const rows = graph.newInput<[string, BucketRow]>() + const activeBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `facade-read-work`, + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: ordered }], + () => {}, + ) + graph.finalize() + const bucketKey = `group` + const values = Array.from({ length: size }, (_, id) => ({ id })) + activeBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + rows.sendData( + new MultiSet( + values.map((value) => [ + [ + bucketKey, + { + publicKey: value.id, + value, + order: ordered + ? String(size - value.id).padStart(3, `0`) + : undefined, + }, + ], + 1, + ]), + ), + ) + graph.run() + adapter.flush().publish() + const ref: BucketFacadeRef = { + [BUCKET_FACADE_REF]: { edgeId: `children`, bucketKey }, + } + const publicView = adapter.resolve(ref) as unknown as Collection< + { id: number }, + number + > + const entries = publicView.entries.bind(publicView) + let visited = 0 + const scan = vi + .spyOn(publicView, `entries`) + .mockImplementation(function* () { + for (const entry of entries()) { + visited++ + yield entry + } + }) + try { + const published = publicView + for (const { id } of values) { + expect(published.get(id)?.id).toBe(id) + expect(published.has(id)).toBe(true) + expect(published.size).toBe(size) + } + expect([...published.keys()]).toEqual( + ordered + ? values.map(({ id }) => id).reverse() + : values.map(({ id }) => id), + ) + // These counters see the real facade scan, not a modeled operation. + expect + .soft(scan.mock.calls.length, `full bucket scans`) + .toBeLessThanOrEqual(1) + expect.soft(visited, `source rows visited`).toBeLessThanOrEqual(size) + const inserted = { id: size } + rows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { publicKey: inserted.id, value: inserted, order: `999` }, + ], + 1, + ], + ]), + ) + graph.run() + adapter.flush().publish() + expect(published.get(size)?.id).toBe(size) + expect(published.size).toBe(size + 1) + } finally { + scan.mockRestore() + adapter.cleanup() + } + }, + ) + + it(`moves a row when the graph reuses its object for a new order`, async () => { + const graph = new D2() + const rows = graph.newInput<[string, BucketRow]>() + const activeBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `facade-order-parent`, + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: true }], + () => {}, + ) + graph.finalize() + + const bucketKey = `group-1` + const moving = { id: 1, value: `moving` } + const fixed = { id: 2, value: `fixed` } + activeBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + rows.sendData( + new MultiSet([ + [[bucketKey, { publicKey: moving.id, value: moving, order: `0` }], 1], + [[bucketKey, { publicKey: fixed.id, value: fixed, order: `1` }], 1], + ]), + ) + graph.run() + adapter.flush().publish() + + const facadeRef: BucketFacadeRef = { + [BUCKET_FACADE_REF]: { edgeId: `children`, bucketKey }, + } + const facade = adapter.resolve(facadeRef) as unknown as Collection< + typeof moving, + number + > + expect(facade.toArray.map(({ id }) => id)).toEqual([1, 2]) + + rows.sendData( + new MultiSet([ + [[bucketKey, { publicKey: moving.id, value: moving, order: `0` }], -1], + [[bucketKey, { publicKey: moving.id, value: moving, order: `2` }], 1], + ]), + ) + graph.run() + adapter.flush().publish() + + expect(facade.toArray.map(({ id }) => id)).toEqual([2, 1]) + await adapter.cleanup() + }) + + it(`restores facade state without public effects when a flush fails`, async () => { const graph = new D2() const rows = graph.newInput<[string, BucketRow]>() const activeBuckets = graph.newInput<[string, true]>() const adapter = new BucketFacadeAdapter( `facade-rollback-parent`, - [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: false }], + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: true }], () => {}, ) graph.finalize() const bucketKey = `group-1` const original = { id: 1, value: `original` } + const fixed = { id: 3, value: `fixed` } activeBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) rows.sendData( new MultiSet([ [ - [ - bucketKey, - { publicKey: original.id, value: original, order: undefined }, - ], + [bucketKey, { publicKey: original.id, value: original, order: `0` }], 1, ], + [[bucketKey, { publicKey: fixed.id, value: fixed, order: `1` }], 1], ]), ) graph.run() @@ -53,11 +273,25 @@ describe(`BucketFacadeAdapter`, () => { typeof original, number > - expect(facade.toArray.map(stripVirtualProps)).toEqual([original]) + expect(facade.toArray.map(stripVirtualProps)).toEqual([original, fixed]) const publications: Array = [] const subscription = facade.subscribeChanges((changes) => { publications.push(changes) }) + let layoutPublications = 0 + const unsubscribeLayout = facade._subscribeLayoutChanges(() => { + layoutPublications++ + }) + let statusChanges = 0 + const unsubscribeStatus = facade.on(`status:change`, () => { + statusChanges++ + }) + let truncates = 0 + const unsubscribeTruncate = facade.on(`truncate`, () => { + truncates++ + }) + const stateRevision = facade._stateRevision + const layoutRevision = facade._layoutRevision const entries = ( adapter as unknown as { @@ -78,6 +312,319 @@ describe(`BucketFacadeAdapter`, () => { } const replacement = { id: 1, value: `replacement` } + const added = { id: 2, value: `added` } + rows.sendData( + new MultiSet([ + [ + [bucketKey, { publicKey: original.id, value: original, order: `0` }], + -1, + ], + [ + [ + bucketKey, + { + publicKey: replacement.id, + value: replacement, + order: `2`, + }, + ], + 1, + ], + [[bucketKey, { publicKey: added.id, value: added, order: `3` }], 1], + ]), + ) + graph.run() + + expect(() => adapter.flush()).toThrow(`facade flush failed`) + expect(facade.toArray.map(stripVirtualProps)).toEqual([original, fixed]) + expect([ + ...( + entries.get(`children`)?.get(bucketKey) as unknown as { + currentOrder: Map + } + ).currentOrder, + ]).toEqual([ + [original.id, `0`], + [fixed.id, `1`], + ]) + expect(publications).toEqual([]) + expect(layoutPublications).toBe(0) + expect(statusChanges).toBe(0) + expect(truncates).toBe(0) + expect(facade._stateRevision).toBe(stateRevision) + expect(facade._layoutRevision).toBe(layoutRevision) + expect(facade.status).toBe(`ready`) + + const restoredOriginal = facade.get(original.id) + const restoredFixed = facade.get(fixed.id) + if (!restoredOriginal || !restoredFixed) { + throw new Error(`Missing restored facade rows`) + } + expect(facade.getKeyFromItem(restoredOriginal)).toBe(original.id) + expect(facade.getKeyFromItem(restoredFixed)).toBe(fixed.id) + + adapter.flush().publish() + + expect(facade.toArray.map(stripVirtualProps)).toEqual([ + fixed, + replacement, + added, + ]) + expect(layoutPublications).toBe(0) + expect(publications).toHaveLength(1) + expect(publications[0]).toHaveLength(2) + expect(statusChanges).toBe(0) + expect(truncates).toBe(0) + expect(facade._stateRevision).toBe(stateRevision + 1) + expect(facade._layoutRevision).toBe(layoutRevision + 1) + expect(facade.toArray.map((row) => facade.getKeyFromItem(row))).toEqual([ + fixed.id, + replacement.id, + added.id, + ]) + expect([ + ...( + entries.get(`children`)?.get(bucketKey) as unknown as { + currentOrder: Map + } + ).currentOrder, + ]).toEqual([ + [original.id, `2`], + [fixed.id, `1`], + [added.id, `3`], + ]) + + rows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { + publicKey: replacement.id, + value: replacement, + order: `2`, + }, + ], + -1, + ], + [ + [ + bucketKey, + { + publicKey: replacement.id, + value: replacement, + order: `0`, + }, + ], + 1, + ], + ]), + ) + graph.run() + adapter.flush().publish() + + expect(facade.toArray.map(stripVirtualProps)).toEqual([ + replacement, + fixed, + added, + ]) + expect(layoutPublications).toBe(1) + expect(publications).toHaveLength(2) + expect(publications[1]).toEqual([]) + expect(facade._stateRevision).toBe(stateRevision + 1) + expect(facade._layoutRevision).toBe(layoutRevision + 2) + + unsubscribeTruncate() + unsubscribeStatus() + unsubscribeLayout() + subscription.unsubscribe() + await adapter.cleanup() + }) + + it(`publishes fresh facade readiness only after every install succeeds`, async () => { + const graph = new D2() + const firstRows = graph.newInput<[string, BucketRow]>() + const firstActiveBuckets = graph.newInput<[string, true]>() + const secondRows = graph.newInput<[string, BucketRow]>() + const secondActiveBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `facade-ready-parent`, + [ + { + edgeId: `first`, + rows: firstRows, + activeBuckets: firstActiveBuckets, + hasOrderBy: false, + }, + { + edgeId: `second`, + rows: secondRows, + activeBuckets: secondActiveBuckets, + hasOrderBy: false, + }, + ], + () => {}, + ) + graph.finalize() + + const bucketKey = `group-1` + const firstFacade = adapter.resolve({ + [BUCKET_FACADE_REF]: { edgeId: `first`, bucketKey }, + } satisfies BucketFacadeRef) as unknown as Collection< + { id: number; value: string }, + number + > + const secondFacade = adapter.resolve({ + [BUCKET_FACADE_REF]: { edgeId: `second`, bucketKey }, + } satisfies BucketFacadeRef) as unknown as Collection< + { id: number; value: string }, + number + > + const firstStatuses: Array = [] + const secondStatuses: Array = [] + const unsubscribeFirst = firstFacade.on(`status:change`, ({ status }) => { + firstStatuses.push(status) + }) + const unsubscribeSecond = secondFacade.on(`status:change`, ({ status }) => { + secondStatuses.push(status) + }) + + const first = { id: 1, value: `first` } + const second = { id: 2, value: `second` } + firstActiveBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + secondActiveBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + firstRows.sendData( + new MultiSet([ + [ + [bucketKey, { publicKey: first.id, value: first, order: undefined }], + 1, + ], + ]), + ) + secondRows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { publicKey: second.id, value: second, order: undefined }, + ], + 1, + ], + ]), + ) + graph.run() + + const entries = ( + adapter as unknown as { + entries: Map> + } + ).entries + const secondSync = entries.get(`second`)?.get(bucketKey)?.sync + if (!secondSync) throw new Error(`Missing second facade sync`) + const commit = secondSync.commit + let shouldThrow = true + secondSync.commit = () => { + const applied = commit() + if (shouldThrow) { + shouldThrow = false + throw new Error(`second facade failed`) + } + return applied + } + + expect(() => adapter.flush()).toThrow(`second facade failed`) + expect(firstFacade.status).toBe(`loading`) + expect(secondFacade.status).toBe(`loading`) + expect(firstStatuses).toEqual([]) + expect(secondStatuses).toEqual([]) + expect(firstFacade.toArray).toEqual([]) + expect(secondFacade.toArray).toEqual([]) + + const retry = adapter.flush() + expect(firstFacade.status).toBe(`loading`) + expect(secondFacade.status).toBe(`loading`) + retry.prepare() + expect(firstFacade.status).toBe(`ready`) + expect(secondFacade.status).toBe(`ready`) + retry.publish() + expect(firstFacade.toArray.map(stripVirtualProps)).toEqual([first]) + expect(secondFacade.toArray.map(stripVirtualProps)).toEqual([second]) + expect(firstStatuses).toEqual([`ready`]) + expect(secondStatuses).toEqual([`ready`]) + + unsubscribeFirst() + unsubscribeSecond() + await adapter.cleanup() + }) + + it(`restores indexed facade state without rebuilding the index`, async () => { + const graph = new D2() + const rows = graph.newInput<[string, BucketRow]>() + const activeBuckets = graph.newInput<[string, true]>() + const adapter = new BucketFacadeAdapter( + `facade-index-rollback-parent`, + [{ edgeId: `children`, rows, activeBuckets, hasOrderBy: false }], + () => {}, + ) + graph.finalize() + + const bucketKey = `group-1` + const original = { id: 1, value: `original` } + activeBuckets.sendData(new MultiSet([[[bucketKey, true], 1]])) + rows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { publicKey: original.id, value: original, order: undefined }, + ], + 1, + ], + ]), + ) + graph.run() + adapter.flush().publish() + + const facade = adapter.resolve({ + [BUCKET_FACADE_REF]: { edgeId: `children`, bucketKey }, + } satisfies BucketFacadeRef) as unknown as Collection< + typeof original, + number + > + const index = facade.createIndex((row) => row.value, { + indexType: ThrowingBuildIndex, + }) as ThrowingBuildIndex + const publications: Array = [] + const subscription = facade.subscribeChanges( + (changes) => { + publications.push(changes) + }, + { includeInitialState: false }, + ) + const revision = facade._stateRevision + + const entry = ( + adapter as unknown as { + entries: Map> + } + ).entries + .get(`children`) + ?.get(bucketKey) + const sync = entry?.sync + if (!sync) throw new Error(`Missing facade sync`) + const commit = sync.commit + let shouldThrow = true + sync.commit = () => { + const applied = commit() + if (shouldThrow) { + shouldThrow = false + throw new Error(`facade flush failed`) + } + return applied + } + + const replacement = { id: 1, value: `replacement` } + index.throwBeforeBuild = true rows.sendData( new MultiSet([ [ @@ -103,14 +650,47 @@ describe(`BucketFacadeAdapter`, () => { graph.run() expect(() => adapter.flush()).toThrow(`facade flush failed`) - expect(facade.toArray.map(stripVirtualProps)).toEqual([original]) + expect(facade.status).toBe(`ready`) + expect(facade._state.syncedData.get(original.id)).toMatchObject(original) expect(publications).toEqual([]) + expect(facade._stateRevision).toBe(revision) + + const final = { id: 1, value: `final` } + rows.sendData( + new MultiSet([ + [ + [ + bucketKey, + { + publicKey: replacement.id, + value: replacement, + order: undefined, + }, + ], + -1, + ], + [ + [bucketKey, { publicKey: final.id, value: final, order: undefined }], + 1, + ], + ]), + ) + graph.run() + adapter.flush().publish() + expect(facade.status).toBe(`ready`) + expect(facade.toArray.map(stripVirtualProps)).toEqual([final]) + expect(publications).toHaveLength(1) + expect(publications[0]).toHaveLength(1) + expect(facade._stateRevision).toBe(revision + 1) + expect(index.lookup(`eq`, `original`)).toEqual(new Set()) + expect(index.lookup(`eq`, `replacement`)).toEqual(new Set()) + expect(index.lookup(`eq`, `final`)).toEqual(new Set([original.id])) subscription.unsubscribe() await adapter.cleanup() }) - it(`drops pending parent changes when facade flushing fails`, async () => { + it(`retries pending parent changes when facade flushing fails`, async () => { type Parent = { id: number; groupId: number } type Child = { id: number; groupId: number } const parents = createCollection( @@ -169,7 +749,7 @@ describe(`BucketFacadeAdapter`, () => { throw new Error(`Missing live query sync state`) } syncState.flushPendingChanges() - expect(live.has(2)).toBe(false) + expect(live.has(2)).toBe(true) } finally { CollectionConfigBuilder.prototype.getConfig = originalGetConfig vi.restoreAllMocks() diff --git a/packages/db/tests/query/compiler/binary-equality-work.test.ts b/packages/db/tests/query/compiler/binary-equality-work.test.ts new file mode 100644 index 0000000000..ddbbeed670 --- /dev/null +++ b/packages/db/tests/query/compiler/binary-equality-work.test.ts @@ -0,0 +1,106 @@ +import { describe, expect, it } from 'vitest' +import { compileSingleRowExpression } from '../../../src/query/compiler/evaluators.js' +import { Func, PropRef, Value } from '../../../src/query/ir.js' + +const cases = [0, 129, 65536].flatMap((size) => + [`eq`, `in`].flatMap((operator) => + [`array`, `buffer`, `mixed`].flatMap((form) => + [`equal`, `offset`, `different`, `length`].map((shape) => ({ + size, + operator, + form, + shape, + })), + ), + ), +) + +// Count bytes encoded, without retaining one mock-call record per byte. +function measureEncoding(run: () => unknown) { + const original = String.fromCharCode + let bytes = 0 + String.fromCharCode = (...codes) => { + bytes += codes.length + return original(...codes) + } + try { + return { result: run(), bytes } + } finally { + String.fromCharCode = original + } +} + +describe(`binary equality work`, () => { + it.each(cases)( + `compares $size bytes with $operator/$form/$shape without encoding strings`, + ({ size, operator, form, shape }) => { + const left = + form === `buffer` + ? Buffer.alloc(size, 65) + : new Uint8Array(size).fill(65) + const backing = new Uint8Array(size + 2).fill(65) + backing[0] = 99 + backing[backing.length - 1] = 99 + let right: Uint8Array = + shape === `offset` + ? backing.subarray(1, size + 1) + : Uint8Array.from(left) + if (shape === `different`) { + if (size === 0) right = new Uint8Array([66]) + else right[size - 1] = 66 + } + if (shape === `length`) right = new Uint8Array(size + 1).fill(65) + if (form !== `array`) + right = Buffer.from(right.buffer, right.byteOffset, right.byteLength) + const expected = shape === `equal` || shape === `offset` + const evaluate = compileSingleRowExpression( + new Func(operator, [ + new PropRef([`blob`]), + new Value(operator === `in` ? [null, right] : right), + ]), + ) + const observed = measureEncoding(() => evaluate({ blob: left })) + expect(observed.result).toBe(expected) + expect(observed.bytes).toBe(0) + }, + ) + + it.each([`eq`, `in`])( + `compares a MiB using %s without caching mutable bytes`, + (operator) => { + const left = new Uint8Array(1024 * 1024).fill(65) + const right = left.slice() + const evaluate = compileSingleRowExpression( + new Func(operator, [ + new PropRef([`blob`]), + new Value(operator === `in` ? [right] : right), + ]), + ) + const equal = measureEncoding(() => evaluate({ blob: left })) + expect(equal).toEqual({ result: true, bytes: 0 }) + right[right.length - 1] = 66 + const different = measureEncoding(() => evaluate({ blob: left })) + expect(different).toEqual({ result: false, bytes: 0 }) + }, + ) + + it.each([`eq`, `in`])( + `keeps %s binary values separate from normalization-like strings`, + (operator) => { + const bytes = new Uint8Array([65]) + const text = '\u0000tanstack-db:binary:A' + for (const [left, right] of [ + [bytes, text], + [text, bytes], + ]) { + const evaluate = compileSingleRowExpression( + new Func(operator, [ + new PropRef([`value`]), + new Value(operator === `in` ? [right] : right), + ]), + ) + expect(evaluate({ value: left })).toBe(false) + } + }, + ) +}) diff --git a/packages/db/tests/query/compiler/group-by-pipeline.test.ts b/packages/db/tests/query/compiler/group-by-pipeline.test.ts new file mode 100644 index 0000000000..8d09872c43 --- /dev/null +++ b/packages/db/tests/query/compiler/group-by-pipeline.test.ts @@ -0,0 +1,197 @@ +import { D2, MultiSet, output } from '@tanstack/db-ivm' +import { describe, expect, test } from 'vitest' +import { NonAggregateExpressionNotInGroupByError } from '../../../src/errors.js' +import { coalesce } from '../../../src/query/builder/functions.js' +import { processGroupBy } from '../../../src/query/compiler/group-by.js' +import { createValueIdentity } from '../../../src/query/equality-value-identity.js' +import { Aggregate, Func, PropRef, Value } from '../../../src/query/ir.js' +import type { Select } from '../../../src/query/ir.js' +import type { KeyedNamespacedRow } from '../../../src/types.js' + +type Row = { id: number; group: number; amount: number; local: boolean } + +const initial: Array = [ + { id: 1, group: 1, amount: 2, local: false }, + { id: 2, group: 1, amount: 5, local: true }, + { id: 3, group: 2, amount: 9, local: false }, +] +const snapshots = [ + [], + initial, + [initial[0]!, { ...initial[1]!, group: 2, amount: 3, local: false }], + [], + initial, +] + +const cases = [false, true].flatMap((grouped) => + ([`none`, `plain`, `wrapped`] as const).flatMap((selection) => + ([`none`, `expression`, `function`, `false`, `null`] as const).map( + (having) => ({ + grouped, + selection, + having, + }), + ), + ), +) + +describe(`group-by production pipeline`, () => { + test.each([false, true])( + `validates ungrouped SELECT references only with grouping keys: %s`, + (grouped) => { + const graph = new D2() + const compile = () => + processGroupBy( + graph.newInput(), + grouped ? [new PropRef([`row`, `group`])] : [], + createValueIdentity(), + undefined, + { amount: new PropRef([`row`, `amount`]) }, + ) + if (grouped) { + expect(compile).toThrow(NonAggregateExpressionNotInGroupByError) + } else { + expect(compile).not.toThrow() + } + }, + ) + + test.each(cases)( + `recomputes rows and metadata: grouped=$grouped, select=$selection, having=$having`, + ({ grouped, selection, having }) => { + const graph = new D2() + const input = graph.newInput() + const groupRef = new PropRef([`row`, `group`]) + const total = new Aggregate(`sum`, [new PropRef([`row`, `amount`])]) + // Exercise generated-field collision avoidance as well as wrapped refs. + const totalAlias = `__tanstack_group_synced` + const select: Select | undefined = + selection === `none` + ? undefined + : { + ...(grouped ? { group: groupRef } : {}), + [totalAlias]: + selection === `plain` + ? total + : new Func(`add`, [ + coalesce(total, 0), + grouped ? groupRef : new Value(0), + ]), + } + type Result = { + key: unknown + selected: unknown + synced: unknown + origin: unknown + } + let actual = new MultiSet() + processGroupBy( + input, + grouped ? [groupRef] : [], + createValueIdentity(), + having === `expression` + ? [ + selection === `none` + ? new Value(true) + : new Func(`gt`, [ + new PropRef([`$selected`, totalAlias]), + new Value(5), + ]), + ] + : having === `false` || having === `null` + ? [ + having === `false` + ? new Value(false) + : new Func(`gt`, [new Value(null), new Value(5)]), + ] + : undefined, + select, + having === `function` + ? [ + (row: { $selected: Record }) => + selection === `none` || row.$selected[totalAlias]! > 5, + ] + : undefined, + `aggregate-result`, + ).pipe( + output((delta) => { + // Observe the public projection, not transient reducer bookkeeping. + actual = actual + .concat( + delta.map(([key, row]) => { + expect(row.$key).toBe(key) + expect(row.$collectionId).toBe(`aggregate-result`) + return { + key, + selected: row.$selected, + synced: row.$synced, + origin: row.$origin, + } + }), + ) + .consolidate() + }), + ) + graph.finalize() + + let previous = new MultiSet() + for (const rows of snapshots) { + const next = new MultiSet( + rows.map((row) => [ + [ + String(row.id), + { + row: { + ...row, + $synced: !row.local, + $origin: row.local ? `local` : `remote`, + }, + }, + ], + 1, + ]), + ) + input.sendData(previous.negate().concat(next)) + graph.run() + previous = next + + // Independent batch model: partition source rows, then sum directly. + const groups = new Map>() + for (const row of rows) { + const key = grouped ? row.group : `single_group` + groups.set(key, [...(groups.get(key) ?? []), row]) + } + const expected = [...groups].flatMap(([key, members]) => { + if (having === `false` || having === `null`) return [] + const amount = + members.reduce((sum, row) => sum + row.amount, 0) + + (selection === `wrapped` && grouped ? Number(key) : 0) + if (having !== `none` && selection !== `none` && amount <= 5) + return [] + return [ + { + key, + selected: + selection === `none` + ? grouped + ? { __key_0: key } + : {} + : { + ...(grouped ? { group: key } : {}), + [totalAlias]: amount, + }, + synced: members.every((row) => !row.local), + origin: members.some((row) => row.local) ? `local` : `remote`, + }, + ] + }) + const observed = actual.getInner().map(([row, weight]) => { + expect(weight).toBe(1) + return row + }) + expect(observed).toHaveLength(expected.length) + expect(observed).toEqual(expect.arrayContaining(expected)) + } + }, + ) +}) diff --git a/packages/db/tests/query/compiler/lazy-demand.test.ts b/packages/db/tests/query/compiler/lazy-demand.test.ts new file mode 100644 index 0000000000..b1c745f5a7 --- /dev/null +++ b/packages/db/tests/query/compiler/lazy-demand.test.ts @@ -0,0 +1,174 @@ +import { D2, output } from '@tanstack/db-ivm' +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../../src/collection/index.js' +import { compileQuery } from '../../../src/query/compiler/index.js' +import { CollectionRef, PropRef } from '../../../src/query/ir.js' +import type { LazyCollectionCallbacks } from '../../../src/query/compiler/joins.js' + +type Row = { id: number; key: unknown } +type Change = [[number, Row], number] + +function createDemandHarness(joinType: `left` | `right` | `full` = `left`) { + const source = (id: string) => + createCollection>({ + id, + getKey: ({ id: key }) => Number(key), + sync: { sync: () => {} }, + }) + const left = source(`demand-left`) + const right = source(`demand-right`) + const graph = new D2() + const leftInput = graph.newInput<[number, Row]>() + const rightInput = graph.newInput<[number, Row]>() + const callbacks: Record = {} + const lazySources = new Set() + const { pipeline } = compileQuery( + { + from: new CollectionRef(left, `left`), + join: [ + { + type: joinType, + from: new CollectionRef(right, `right`), + left: new PropRef([`left`, `key`]), + right: new PropRef([`right`, `key`]), + }, + ], + }, + { left: leftInput, right: rightInput }, + { [left.id]: left, [right.id]: right }, + {}, + callbacks, + lazySources, + {}, + () => {}, + ) + const transitions: Array> = [] + for (const state of Object.values(callbacks)) { + let previous: Array = [] + state.setDemand = (_plan, keys) => { + const next = [...keys] + // Ignore redundant notifications, but not an intervening empty demand. + if ( + next.length === previous.length && + next.every((key) => previous.includes(key)) + ) + return + previous = next + transitions.push(next) + } + } + let resultWeight = 0 + pipeline.pipe( + output((data) => { + for (const [, weight] of data.getInner()) resultWeight += weight + }), + ) + graph.finalize() + const input = joinType === `right` ? rightInput : leftInput + return { + graph, + input, + transitions, + lazySources, + resultWeight: () => resultWeight, + cleanup: async () => { + await left.cleanup() + await right.cleanup() + }, + } +} + +describe(`compiled lazy demand presence`, () => { + // Characterize the current message boundary before changing batching policy. + it.each( + ([`left`, `right`] as const).flatMap((joinType) => + ([`one-message`, `queued-messages`, `separate-turns`] as const).map( + (delivery) => ({ joinType, delivery }), + ), + ), + )( + `preserves demand transitions for $joinType with $delivery`, + async ({ joinType, delivery }) => { + const h = createDemandHarness(joinType) + const row = { id: 1, key: `shared` } + const insert: Change = [[row.id, row], 1] + const retract: Change = [[row.id, row], -1] + try { + h.input.sendData([insert]) + h.graph.run() + expect(h.lazySources.size).toBe(1) + expect(h.transitions).toEqual([[`shared`]]) + h.transitions.length = 0 + if (delivery === `one-message`) h.input.sendData([retract, insert]) + else { + h.input.sendData([retract]) + if (delivery === `separate-turns`) h.graph.run() + h.input.sendData([insert]) + } + h.graph.run() + expect(h.transitions).toEqual( + delivery === `one-message` ? [] : [[], [`shared`]], + ) + expect(h.resultWeight()).toBe(1) + } finally { + await h.cleanup() + } + }, + ) + + it.each([ + { name: `numbers`, first: 3, second: 3 }, + { name: `signed zero`, first: -0, second: 0 }, + { name: `Date values`, first: new Date(3), second: new Date(3) }, + { + name: `binary values`, + first: Buffer.from([3]), + second: new Uint8Array([3]), + }, + ])( + `retains one demand until the last $name contributor leaves`, + async ({ first, second }) => { + const h = createDemandHarness() + const a: Row = { id: 1, key: first } + const b: Row = { id: 2, key: second } + try { + h.input.sendData([ + [[a.id, a], 1], + [[b.id, b], 1], + ]) + h.graph.run() + expect(h.transitions).toHaveLength(1) + expect(h.transitions[0]).toHaveLength(1) + expect(h.resultWeight()).toBe(2) + h.input.sendData([[[a.id, a], -1]]) + h.graph.run() + expect(h.transitions).toHaveLength(1) + expect(h.resultWeight()).toBe(1) + h.input.sendData([[[b.id, b], -1]]) + h.graph.run() + expect(h.transitions).toHaveLength(2) + expect(h.transitions[1]).toEqual([]) + expect(h.resultWeight()).toBe(0) + } finally { + await h.cleanup() + } + }, + ) + + it(`does not demand nullish keys or add lazy demand for a full join`, async () => { + for (const joinType of [`left`, `full`] as const) { + const h = createDemandHarness(joinType) + try { + h.input.sendData([ + [[1, { id: 1, key: null }], 1], + [[2, { id: 2, key: undefined }], 1], + ]) + h.graph.run() + expect(h.transitions).toEqual([]) + expect(h.lazySources.size).toBe(joinType === `full` ? 0 : 1) + } finally { + await h.cleanup() + } + } + }) +}) diff --git a/packages/db/tests/query/compiler/select.test.ts b/packages/db/tests/query/compiler/select.test.ts index 820209b092..c5459944ae 100644 --- a/packages/db/tests/query/compiler/select.test.ts +++ b/packages/db/tests/query/compiler/select.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest' -import { processArgument } from '../../../src/query/compiler/select.js' +import { compileExpression } from '../../../src/query/compiler/evaluators.js' import { Aggregate, Func, PropRef, Value } from '../../../src/query/ir.js' describe(`select compiler`, () => { @@ -7,12 +7,12 @@ describe(`select compiler`, () => { // tests in basic.test.ts and other compiler tests. Here we focus on the standalone // functions that can be tested in isolation. - describe(`processArgument`, () => { + describe(`compileExpression`, () => { it(`processes non-aggregate expressions correctly`, () => { const arg = new PropRef([`users`, `name`]) const namespacedRow = { users: { name: `John` } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(`John`) }) @@ -20,7 +20,7 @@ describe(`select compiler`, () => { const arg = new Value(42) const namespacedRow = {} - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(42) }) @@ -28,7 +28,7 @@ describe(`select compiler`, () => { const arg = new Func(`upper`, [new Value(`hello`)]) const namespacedRow = {} - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(`HELLO`) }) @@ -37,10 +37,9 @@ describe(`select compiler`, () => { const namespacedRow = { users: { id: 1 } } expect(() => { - processArgument(arg, namespacedRow) - }).toThrow( - `Aggregate expressions are not supported in this context. Use GROUP BY clause for aggregates.`, - ) + // @ts-expect-error Aggregate IR is not a single-row expression. + compileExpression(arg)(namespacedRow) + }).toThrow(`Unknown expression type: agg`) }) it(`processes reference expressions from different tables`, () => { @@ -50,7 +49,7 @@ describe(`select compiler`, () => { orders: { amount: 100.5 }, } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(100.5) }) @@ -64,7 +63,7 @@ describe(`select compiler`, () => { }, } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(`New York`) }) @@ -72,7 +71,7 @@ describe(`select compiler`, () => { const arg = new Func(`length`, [new PropRef([`users`, `name`])]) const namespacedRow = { users: { name: `Alice` } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(5) }) @@ -89,7 +88,7 @@ describe(`select compiler`, () => { }, } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(`John Doe`) }) @@ -97,7 +96,7 @@ describe(`select compiler`, () => { const arg = new PropRef([`users`, `middleName`]) const namespacedRow = { users: { name: `John`, middleName: null } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(null) }) @@ -105,7 +104,7 @@ describe(`select compiler`, () => { const arg = new PropRef([`nonexistent`, `field`]) const namespacedRow = { users: { name: `John` } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(undefined) }) @@ -113,7 +112,7 @@ describe(`select compiler`, () => { const arg = new PropRef([`users`, `nonexistent`]) const namespacedRow = { users: { name: `John` } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(undefined) }) @@ -121,7 +120,7 @@ describe(`select compiler`, () => { const arg = new Value({ nested: { value: 42 } }) const namespacedRow = {} - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toEqual({ nested: { value: 42 } }) }) @@ -129,7 +128,7 @@ describe(`select compiler`, () => { const arg = new Func(`and`, [new Value(true), new Value(false)]) const namespacedRow = {} - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(false) }) @@ -137,7 +136,7 @@ describe(`select compiler`, () => { const arg = new Func(`gt`, [new PropRef([`users`, `age`]), new Value(18)]) const namespacedRow = { users: { age: 25 } } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(true) }) @@ -153,18 +152,14 @@ describe(`select compiler`, () => { }, } - const result = processArgument(arg, namespacedRow) + const result = compileExpression(arg)(namespacedRow) expect(result).toBe(108.5) }) }) describe(`helper functions`, () => { // Test the helper function that can be imported and tested directly - it(`correctly identifies aggregate expressions`, () => { - // This test would require accessing the isAggregateExpression function - // which is private. Since we can't test it directly, we test it indirectly - // through the processArgument function's error handling. - + it(`rejects aggregate IR at the single-row compiler boundary`, () => { const aggregateExpressions = [ new Aggregate(`count`, [new PropRef([`users`, `id`])]), new Aggregate(`sum`, [new PropRef([`orders`, `amount`])]), @@ -183,12 +178,13 @@ describe(`select compiler`, () => { // All of these should throw errors since they're aggregates aggregateExpressions.forEach((expr) => { expect(() => { - processArgument(expr, namespacedRow) - }).toThrow(`Aggregate expressions are not supported in this context`) + // @ts-expect-error Aggregate IR is not a single-row expression. + compileExpression(expr)(namespacedRow) + }).toThrow(`Unknown expression type: agg`) }) }) - it(`correctly identifies non-aggregate expressions`, () => { + it(`accepts supported single-row expression forms`, () => { const nonAggregateExpressions = [ new PropRef([`users`, `name`]), new Value(42), @@ -201,7 +197,7 @@ describe(`select compiler`, () => { // None of these should throw errors since they're not aggregates nonAggregateExpressions.forEach((expr) => { expect(() => { - processArgument(expr, namespacedRow) + compileExpression(expr)(namespacedRow) }).not.toThrow() }) }) diff --git a/packages/db/tests/query/group-by-work.test.ts b/packages/db/tests/query/group-by-work.test.ts new file mode 100644 index 0000000000..00153d9c88 --- /dev/null +++ b/packages/db/tests/query/group-by-work.test.ts @@ -0,0 +1,53 @@ +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createLiveQueryCollection } from '../../src/query/index.js' +import { count } from '../../src/query/builder/functions.js' +import { mockSyncCollectionOptions } from '../utils.js' + +describe(`group representative work`, () => { + it.each([16, 1024, 5000])( + `encodes changed contributions, not all %s retained members`, + async (size) => { + const source = createCollection( + mockSyncCollectionOptions<{ id: number; value: number }>({ + id: `group-work-${size}`, + getKey: (row) => row.id, + initialData: Array.from({ length: size }, (_, id) => ({ + id, + value: 1, + })), + }), + ) + const grouped = createLiveQueryCollection((q) => + q + .from({ row: source }) + .groupBy(({ row }) => row.value) + .select(({ row }) => ({ value: row.value, count: count(row.id) })), + ) + try { + await grouped.preload() + for (const type of [`insert`, `delete`] as const) { + const spy = vi.spyOn(JSON, `stringify`) + let calls: number + try { + source.utils.begin() + source.utils.write({ type, value: { id: size, value: 1 } }) + source.utils.commit() + calls = spy.mock.calls.length + } finally { + spy.mockRestore() + } + // Group reduction still scans members. Encoding its stable input + // keys must scale with the delta, not the retained group size. + expect(calls).toBeLessThanOrEqual(4) + expect(grouped.toArray).toMatchObject([ + { value: 1, count: size + (type === `insert` ? 1 : 0) }, + ]) + } + } finally { + await grouped.cleanup() + await source.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/query/group-by.test.ts b/packages/db/tests/query/group-by.test.ts index 39851ed823..d736d8244b 100644 --- a/packages/db/tests/query/group-by.test.ts +++ b/packages/db/tests/query/group-by.test.ts @@ -1,4 +1,5 @@ import { beforeEach, describe, expect, test } from 'vitest' +import { Temporal } from 'temporal-polyfill' import { createLiveQueryCollection } from '../../src/query/index.js' import { createCollection } from '../../src/collection/index.js' import { mockSyncCollectionOptions, stripVirtualProps } from '../utils.js' @@ -222,8 +223,221 @@ function createOrdersCollection(autoIndex: `off` | `eager` = `eager`) { ) } +const equalityEquivalentGroupValues: Array< + [string, () => readonly [unknown, unknown]] +> = [ + [`a Date and its timestamp`, () => [new Date(0), 0]], + [`an invalid Date and NaN`, () => [new Date(Number.NaN), Number.NaN]], + [`signed zero`, () => [-0, 0]], + [ + `binary values with the same bytes`, + () => [Buffer.from([1, 2, 3]), new Uint8Array([1, 2, 3])], + ], + [ + `equivalent Temporal values`, + () => [ + Temporal.PlainDate.from(`2024-04-05`), + Temporal.PlainDate.from(`2024-04-05`), + ], + ], + [ + `the same symbol reference`, + () => { + const value = Symbol(`group`) + return [value, value] + }, + ], + [ + `the same cyclic object`, + () => { + const value: { self?: unknown } = {} + value.self = value + return [value, value] + }, + ], +] + +function representativeSignature(value: unknown): string { + if (value instanceof Date) return `date` + if (Buffer.isBuffer(value)) return `buffer` + if (value instanceof Uint8Array) return `uint8array` + if (typeof value === `number` && Object.is(value, -0)) return `negative-zero` + if (typeof value === `number` && Number.isNaN(value)) return `nan` + if (typeof value === `number`) return `number` + if (typeof value === `symbol`) return `symbol` + if ( + typeof value === `object` && + value !== null && + (value as { self?: unknown }).self === value + ) { + return `cyclic-object` + } + return `${typeof value}:${String(value)}` +} + function createGroupByTests(autoIndex: `off` | `eager`): void { describe(`with autoIndex ${autoIndex}`, () => { + test(`keeps opaque public group keys stable across graph scopes`, async () => { + const symbol = Symbol(`group`) + const otherSymbol = Symbol(`group`) + const valuesCollection = createCollection( + mockSyncCollectionOptions<{ id: number; value: symbol }>({ + id: `scoped-group-symbol-${autoIndex}`, + getKey: (row) => row.id, + initialData: [ + { id: 1, value: symbol }, + { id: 2, value: otherSymbol }, + ], + autoIndex, + }), + ) + const createSummary = () => + createLiveQueryCollection({ + startSync: true, + query: (q) => + q + .from({ value: valuesCollection }) + .groupBy(({ value }) => value.value) + .select(({ value }) => ({ + value: value.value, + count: count(value.id), + })), + }) + + const first = createSummary() + const second = createSummary() + + try { + const firstKeys = [...first.keys()] + const secondKeys = [...second.keys()] + expect(firstKeys).toHaveLength(2) + expect(firstKeys.every((key) => typeof key === `string`)).toBe(true) + expect(new Set(firstKeys).size).toBe(2) + expect(secondKeys).toEqual(firstKeys) + + const symbolKey = first.toArray.find( + (row) => row.value === symbol, + )!.$key + valuesCollection.utils.begin() + valuesCollection.utils.write({ + type: `delete`, + value: { id: 1, value: symbol }, + }) + valuesCollection.utils.commit() + expect(first.get(symbolKey)).toBeUndefined() + + valuesCollection.utils.begin() + valuesCollection.utils.write({ + type: `insert`, + value: { id: 1, value: symbol }, + }) + valuesCollection.utils.commit() + expect(first.get(symbolKey)?.value).toBe(symbol) + } finally { + await Promise.all([ + first.cleanup(), + second.cleanup(), + valuesCollection.cleanup(), + ]) + } + }) + + test.each(equalityEquivalentGroupValues)( + `groups %s by query equality`, + (_name, createValues) => { + const [left, right] = createValues() + const valuesCollection = createCollection( + mockSyncCollectionOptions<{ id: number; value: unknown }>({ + id: `equality-group-values-${autoIndex}`, + getKey: (row) => row.id, + initialData: [ + { id: 1, value: left }, + { id: 2, value: right }, + ], + autoIndex, + }), + ) + + const summary = createLiveQueryCollection({ + startSync: true, + query: (q) => + q + .from({ value: valuesCollection }) + .groupBy(({ value }) => value.value) + .select(({ value }) => ({ + value: value.value, + count: count(value.id), + })), + }) + + const expectSingleGroup = ( + expectedCount: number, + representative: unknown, + ) => { + expect(summary.toArray).toHaveLength(1) + expect(summary.toArray[0]?.count).toBe(expectedCount) + expect(representativeSignature(summary.toArray[0]?.value)).toBe( + representativeSignature(representative), + ) + } + + expectSingleGroup(2, left) + + valuesCollection.utils.begin() + valuesCollection.utils.write({ + type: `delete`, + value: { id: 1, value: left }, + }) + valuesCollection.utils.commit() + expectSingleGroup(1, right) + + valuesCollection.utils.begin() + valuesCollection.utils.write({ + type: `insert`, + value: { id: 1, value: left }, + }) + valuesCollection.utils.commit() + expectSingleGroup(2, left) + }, + ) + + test.each([ + `__group_value_0`, + `__key_0`, + `__tanstack_group_value_0`, + `__tanstack_group_key_0`, + ])( + `keeps the grouped value when an aggregate uses internal-looking alias %s`, + (alias) => { + const valuesCollection = createCollection( + mockSyncCollectionOptions<{ id: number; value: string }>({ + id: `group-alias-collision-${autoIndex}-${alias}`, + getKey: (row) => row.id, + initialData: [ + { id: 1, value: `x` }, + { id: 2, value: `x` }, + ], + autoIndex, + }), + ) + const summary = createLiveQueryCollection({ + startSync: true, + query: (q) => + q + .from({ value: valuesCollection }) + .groupBy(({ value }) => value.value) + .select(({ value }) => ({ + value: value.value, + [alias]: count(value.id), + })), + }) + + expect(summary.toArray.map(stripVirtualProps)).toEqual([ + { value: `x`, [alias]: 2 }, + ]) + }, + ) + describe(`Single Column Grouping`, () => { let ordersCollection: ReturnType diff --git a/packages/db/tests/query/immutable-demand-boundary.test.ts b/packages/db/tests/query/immutable-demand-boundary.test.ts new file mode 100644 index 0000000000..dccd88f5d2 --- /dev/null +++ b/packages/db/tests/query/immutable-demand-boundary.test.ts @@ -0,0 +1,138 @@ +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection' +import { eq } from '../../src/query/builder/functions' +import { Func, PropRef, Value } from '../../src/query/ir' +import { compileSingleRowExpression } from '../../src/query/compiler/evaluators' +import { DeduplicatedLoadSubset } from '../../src/query/subset-dedupe' +import type { LoadSubsetOptions } from '../../src/types' + +describe.each([`direct`, `deferred`] as const)( + `immutable demand through %s sync startup`, + (start) => { + it.each([`return`, `resolve`, `abort`] as const)( + `preserves request data and live cancellation on %s`, + async (outcome) => { + const date = Object.freeze(new Date(7)) + const candidates = Object.freeze([date]) + const reference = new PropRef([`date`]) + Object.freeze(reference.path) + Object.freeze(reference) + const where = new Func(`in`, [ + reference, + Object.freeze(new Value(candidates)), + ]) + Object.freeze(where.args) + Object.freeze(where) + const owner = new AbortController() + const options: LoadSubsetOptions = Object.freeze({ + where, + limit: 2, + signal: owner.signal, + }) + let finish = () => {} + const loads: Array = [] + const unloadSubset = vi.fn() + const deduplicated = new DeduplicatedLoadSubset({ + loadSubset: (request) => { + loads.push(request) + return outcome === `return` + ? true + : new Promise((resolve) => (finish = resolve)) + }, + }) + const collection = createCollection<{ id: number }>({ + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: start === `direct`, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset: deduplicated.loadSubset, unloadSubset } + }, + }, + }) + try { + if (start === `deferred`) { + expect(collection._deferSyncStart()).toBe(true) + } + const result = collection._sync.loadSubset(options) + if (start === `deferred`) { + expect(loads).toEqual([]) + collection._resumeSyncStart() + } + expect(loads).toHaveLength(1) + expect(loads[0]).toBe(options) + const matches = compileSingleRowExpression(loads[0]!.where!) + expect( + [new Date(7), new Date(8)].map((value) => matches({ date: value })), + ).toEqual([true, false]) + + if (outcome === `abort`) owner.abort() + expect(loads[0]!.signal!.aborted).toBe(outcome === `abort`) + if (outcome !== `return`) finish() + await result + collection._sync.unloadSubset(options) + expect(unloadSubset).toHaveBeenCalledExactlyOnceWith(options) + expect(unloadSubset.mock.calls[0]![0]).toBe(loads[0]) + expect(date.getTime()).toBe(7) + expect(candidates).toEqual([new Date(7)]) + + // New immutable data describes a new request. Completed equal data + // shares; an aborted transport establishes no reusable result. + const repeat = deduplicated.loadSubset({ + where: new Func(`in`, [ + new PropRef([`date`]), + new Value([new Date(7)]), + ]), + limit: 2, + }) + expect(loads).toHaveLength(outcome === `abort` ? 2 : 1) + if (outcome === `abort`) finish() + await repeat + } finally { + finish() + await collection.cleanup() + } + }, + ) + }, +) + +it.each([`release`, `cleanup`] as const)( + `retires a frozen queued request before adapter startup by %s`, + async (action) => { + const loadSubset = vi.fn(() => true as const) + const unloadSubset = vi.fn() + const collection = createCollection<{ id: number }>({ + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ markReady }) => { + markReady() + return { loadSubset, unloadSubset } + }, + }, + }) + expect(collection._deferSyncStart()).toBe(true) + const options = Object.freeze({ + where: eq(new PropRef([`id`]), new Value(1)), + }) + try { + const result = collection._sync.loadSubset(options) + const settled = Promise.allSettled([result]) + if (action === `release`) collection._sync.unloadSubset(options) + else await collection.cleanup() + expect(await settled).toEqual([ + { + status: `rejected`, + reason: expect.objectContaining({ name: `AbortError` }), + }, + ]) + collection._resumeSyncStart() + expect(loadSubset).not.toHaveBeenCalled() + expect(unloadSubset).not.toHaveBeenCalled() + } finally { + await collection.cleanup() + } + }, +) diff --git a/packages/db/tests/query/includes-collection-oracle.property.test.ts b/packages/db/tests/query/includes-collection-oracle.property.test.ts index 5d8c2f45a2..82b99103cf 100644 --- a/packages/db/tests/query/includes-collection-oracle.property.test.ts +++ b/packages/db/tests/query/includes-collection-oracle.property.test.ts @@ -301,6 +301,13 @@ const collectionScenarioArbitrary = fc.record({ actions: fc.array(actionArbitrary, { minLength: 1, maxLength: 16 }), }) +const orderSwapArbitrary = fc.integer({ min: 2, max: 8 }).chain((length) => + fc.integer({ min: 0, max: length - 2 }).map((swapIndex) => ({ + length, + swapIndex, + })), +) + function enumerateActionSequences( actions: ReadonlyArray, maxLength: number, @@ -326,7 +333,10 @@ const exhaustiveActions: ReadonlyArray = [ ] describe(`Collection-valued includes oracle`, () => { - fcTest.prop([collectionScenarioArbitrary], oraclePropertyOptions(30))( + fcTest.prop( + [collectionScenarioArbitrary], + oraclePropertyOptions(30, `includes-collection.relationship-history`), + )( `keeps Collection, toArray, and materialize equivalent across generated relationship histories`, ({ parentGroup, childValue, actions }) => runTrace({ @@ -765,6 +775,7 @@ describe(`Collection-valued includes oracle`, () => { fcTest( `outer fn.select recomputes nested values after a union branch include changes`, async () => { + const callbackRows: Array> = [] const messages = createControlledCollection(`fn-select-messages`, [ { id: 1, group: 1 }, ]) @@ -799,11 +810,14 @@ describe(`Collection-valued includes oracle`, () => { id: tool.id, })) - return q.unionAll(messageRows, toolRows).fn.select((row) => ({ - kind: row.kind, - id: row.id, - payload: { children: row.children }, - })) + return q.unionAll(messageRows, toolRows).fn.select((row) => { + callbackRows.push(row) + return { + kind: row.kind, + id: row.id, + payload: { children: row.children }, + } + }) }) try { @@ -819,6 +833,9 @@ describe(`Collection-valued includes oracle`, () => { expect( live.toArray.find((row) => row.kind === `message`)!.payload.children, ).toEqual([{ id: 10, value: 2 }]) + expect( + callbackRows.flatMap((row) => Object.getOwnPropertySymbols(row)), + ).toEqual([]) } finally { await Promise.all([ live.cleanup(), @@ -830,6 +847,66 @@ describe(`Collection-valued includes oracle`, () => { }, ) + fcTest( + `outer fn.select rejects a bare union include before invoking the callback`, + async () => { + class Box { + constructor(readonly child: unknown) {} + } + + const callbackChildren: Array = [] + const messages = createControlledCollection(`fn-select-bare-messages`, [ + { id: 1, group: 1 }, + ]) + const tools = createControlledCollection(`fn-select-bare-tools`, [ + { id: 2, group: 2 }, + ]) + const children = createControlledCollection(`fn-select-bare-children`, [ + { id: 10, parentGroup: 1, value: 1 }, + { id: 20, parentGroup: 2, value: 2 }, + ]) + const buildQuery = () => + createLiveQueryCollection((q) => { + const messageRows = q + .from({ message: messages.collection }) + .select(({ message }) => ({ + kind: `message` as const, + id: message.id, + children: q + .from({ messageChild: children.collection }) + .where(({ messageChild }) => + eq(messageChild.parentGroup, message.group), + ), + })) + const toolRows = q + .from({ tool: tools.collection }) + .select(({ tool }) => ({ + kind: `tool` as const, + id: tool.id, + })) + + return q.unionAll(messageRows, toolRows).fn.select((row) => { + const child = `children` in row ? row.children : undefined + callbackChildren.push(child) + return { kind: row.kind, id: row.id, box: new Box(child) } + }) + }) + + try { + expect(buildQuery).toThrow( + `fn.select() cannot consume Collection-valued includes`, + ) + expect(callbackChildren).toEqual([]) + } finally { + await Promise.all([ + messages.collection.cleanup(), + tools.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + fcTest( `fn.select rejects query values returned during include rematerialization`, async () => { @@ -1040,6 +1117,75 @@ describe(`Collection-valued includes oracle`, () => { } }) + fcTest( + `cleanup during root publication suppresses the prepared facade callback`, + async () => { + type NodeRow = { + id: number + kind: `parent` | `child` + group: number + value: number + } + const nodes = createControlledCollection(`publication-cleanup`, [ + { id: 1, kind: `parent`, group: 1, value: 1 }, + { id: 10, kind: `child`, group: 1, value: 1 }, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: nodes.collection }) + .where(({ parent }) => eq(parent.kind, `parent`)) + .select(({ parent }) => ({ + id: parent.id, + value: parent.value, + children: q + .from({ child: nodes.collection }) + .where(({ child }) => eq(child.kind, `child`)) + .where(({ child }) => eq(child.group, parent.group)), + })), + ) + + await live.preload() + const facade = live.get(1)!.children + const rootSnapshots: Array> = [] + const facadeSnapshots: Array> = [] + let cleanup: Promise | undefined + const rootSubscription = live.subscribeChanges( + () => { + rootSnapshots.push(facade.toArray.map(({ value }) => value)) + cleanup = facade.cleanup() + }, + { includeInitialState: false }, + ) + const facadeSubscription = facade.subscribeChanges( + () => facadeSnapshots.push(facade.toArray.map(({ value }) => value)), + { includeInitialState: false }, + ) + + try { + nodes.writeBatch([ + { + type: `update`, + value: { id: 1, kind: `parent`, group: 1, value: 2 }, + }, + { + type: `update`, + value: { id: 10, kind: `child`, group: 1, value: 2 }, + }, + ]) + await cleanup + + expect(rootSnapshots).toEqual([[2]]) + expect(facadeSnapshots).toEqual([]) + expect(facade.status).toBe(`cleaned-up`) + expect(facade.toArray).toEqual([]) + } finally { + rootSubscription.unsubscribe() + facadeSubscription.unsubscribe() + await Promise.all([live.cleanup(), nodes.collection.cleanup()]) + } + }, + ) + fcTest( `shared facades remain active until their last parent departs`, async () => { @@ -1107,7 +1253,7 @@ describe(`Collection-valued includes oracle`, () => { wideId: fc.integer({ min: 10, max: 19 }), }), ], - oraclePropertyOptions(20), + oraclePropertyOptions(20, `includes-collection.public-key-order`), )( `uses one raw public-key order across Collection and inline materializations`, async ({ smallId, wideId }) => { @@ -1167,19 +1313,29 @@ describe(`Collection-valued includes oracle`, () => { }, ) - fcTest( - `propagates an order-only child move through every materialization`, - async () => { + fcTest.prop( + [orderSwapArbitrary], + oraclePropertyOptions(20, `includes-collection.layout-swap`), + )( + `propagates generated order-only child swaps through every materialization`, + async ({ length, swapIndex }) => { type OrderedChild = ChildRow & { position: number; label: string } const parents = createControlledCollection(`order-move-parents`, [ { id: 1, group: 1 }, ]) + const initialRows: Array = Array.from( + { length }, + (_, index) => ({ + id: index + 1, + parentGroup: 1, + value: index + 1, + position: index, + label: String(index + 1), + }), + ) const children = createControlledCollection( `order-move-children`, - [ - { id: 10, parentGroup: 1, value: 1, position: 0, label: `a` }, - { id: 20, parentGroup: 1, value: 2, position: 1, label: `b` }, - ], + initialRows, ) const live = createLiveQueryCollection((q) => q.from({ parent: parents.collection }).select(({ parent }) => { @@ -1229,27 +1385,30 @@ describe(`Collection-valued includes oracle`, () => { try { await live.preload() const facade = live.get(1)!.facade - const revision = facade._layoutRevision + const initialIds = initialRows.map(({ id }) => id) expect(project()).toEqual({ - ...expectedMaterializations([10, 20]), - first: 10, - joined: `ab`, + ...expectedMaterializations(initialIds), + first: initialIds[0], + joined: initialRows.map(({ label }) => label).join(``), }) - children.write(`update`, { - id: 10, - parentGroup: 1, - value: 1, - position: 2, - label: `a`, - }) + const first = initialRows[swapIndex]! + const second = initialRows[swapIndex + 1]! + children.writeBatch([ + { type: `update`, value: { ...first, position: second.position } }, + { type: `update`, value: { ...second, position: first.position } }, + ]) + const expectedIds = [...initialIds] + ;[expectedIds[swapIndex], expectedIds[swapIndex + 1]] = [ + expectedIds[swapIndex + 1]!, + expectedIds[swapIndex]!, + ] expect(live.get(1)!.facade).toBe(facade) - expect(facade._layoutRevision).toBeGreaterThan(revision) expect(project()).toEqual({ - ...expectedMaterializations([20, 10]), - first: 20, - joined: `ba`, + ...expectedMaterializations(expectedIds), + first: expectedIds[0], + joined: expectedIds.join(``), }) } finally { await Promise.all([ @@ -1770,7 +1929,7 @@ describe(`Collection-valued includes oracle`, () => { value: fc.integer({ min: -10, max: 10 }), }), ], - oraclePropertyOptions(20), + oraclePropertyOptions(20, `includes-collection.optimistic-child-history`), )( `matches recomputation through optimistic child insert and delete confirmation and rollback`, async ({ group, insertedId, confirmedId, value }) => { diff --git a/packages/db/tests/query/includes-context-transport-oracle.test.ts b/packages/db/tests/query/includes-context-transport-oracle.test.ts index 40d12d1ddf..6fbc31fca2 100644 --- a/packages/db/tests/query/includes-context-transport-oracle.test.ts +++ b/packages/db/tests/query/includes-context-transport-oracle.test.ts @@ -1,6 +1,7 @@ import { describe, expect, test } from 'vitest' import { add, + and, coalesce, count, createLiveQueryCollection, @@ -13,6 +14,10 @@ import { sum, toArray, } from '../../src/query/index.js' +import { + attachRouteMetadata, + stripInternalRouteMetadata, +} from '../../src/query/compiler/route-metadata.js' import { createControlledCollection } from './includes-oracle-helpers.js' import type { Collection } from '../../src/collection/index.js' import type { Context, QueryBuilder } from '../../src/query/builder/index.js' @@ -66,6 +71,29 @@ const routeContextGrammar = { selections: [`expression`, `functional`] as const, domains: [`non-null`, `nullable`] as const, }, + namespaceCollision: { + locations: [`parent-alias`, `selected-field`] as const, + boundaries: [`direct`, `query-ref`, `join`, `group`] as const, + names: [ + `__parentContextIdentity`, + `__parentContext`, + `__correlationKey`, + `value`, + `identity`, + ] as const, + }, + publicSurface: { + shapes: [ + `object-query-ref-scalar`, + `nested-functional-spread`, + `opaque-wrapper`, + `functional-having-input`, + `nested-reference`, + `adversarial-key`, + `user-symbol`, + `implicit-join`, + ] as const, + }, } as const const queryRefMetadataGrammar = { @@ -119,6 +147,18 @@ type DerivedResultCell = { domain: (typeof routeContextGrammar.derivedResult.domains)[number] } +type NamespaceCollisionCell = { + family: `namespace-collision` + location: (typeof routeContextGrammar.namespaceCollision.locations)[number] + boundary: (typeof routeContextGrammar.namespaceCollision.boundaries)[number] + name: (typeof routeContextGrammar.namespaceCollision.names)[number] +} + +type PublicSurfaceCell = { + family: `public-surface` + shape: (typeof routeContextGrammar.publicSurface.shapes)[number] +} + type GrammarCell = | ParentProjectionCell | CorrelationDomainCell @@ -128,6 +168,8 @@ type GrammarCell = | JoinCell | UnionIdentityCell | DerivedResultCell + | NamespaceCollisionCell + | PublicSurfaceCell const grammarCells: Array = [ ...routeContextGrammar.parentProjection.shapes.map( @@ -187,6 +229,21 @@ const grammarCells: Array = [ ), ), ), + ...routeContextGrammar.namespaceCollision.locations.flatMap((location) => + routeContextGrammar.namespaceCollision.boundaries.flatMap((boundary) => + routeContextGrammar.namespaceCollision.names.map( + (name): NamespaceCollisionCell => ({ + family: `namespace-collision`, + location, + boundary, + name, + }), + ), + ), + ), + ...routeContextGrammar.publicSurface.shapes.map( + (shape): PublicSurfaceCell => ({ family: `public-surface`, shape }), + ), ] async function cleanup( @@ -257,6 +314,10 @@ function grammarCellName(cell: GrammarCell): string { return `${cell.family} / ${cell.form}` case `derived-result`: return `${cell.family} / ${cell.boundary} / ${cell.selection} / ${cell.domain}` + case `namespace-collision`: + return `${cell.family} / ${cell.location} / ${cell.boundary} / ${cell.name}` + case `public-surface`: + return `${cell.family} / ${cell.shape}` } } @@ -1458,6 +1519,552 @@ async function runJoinCell({ } } +async function runNamespaceCollisionCell({ + location, + boundary, + name, +}: NamespaceCollisionCell): Promise { + const cellName = `collision-${location}-${boundary}-${name}` + const parents = createGrammarCollection(`${cellName}-parents`, [ + { id: 1, group: 1, token: `one` }, + ]) + const children = createGrammarCollection(`${cellName}-children`, [ + { id: 10, parentGroup: 1, token: `one`, label: `one` }, + { id: 20, parentGroup: 2, token: `two`, label: `two` }, + ]) + const tags = createGrammarCollection(`${cellName}-tags`, [ + { id: 10 }, + { id: 20 }, + ]) + const live = + location === `parent-alias` + ? createLiveQueryCollection((q) => + q.from({ [name]: parents.collection }).select((sources) => { + // This computed alias names the sole, non-optional main source. + const parent = sources[name]! + const correlated = q + .from({ child: children.collection }) + .where(({ child }) => + and( + eq(child.parentGroup, parent.group), + eq(child.token, parent.token), + ), + ) + const forms = (() => { + switch (boundary) { + case `direct`: + return includeInEveryForm( + correlated.select(({ child }) => ({ + id: child.id, + value: child.label, + })), + ) + case `query-ref`: { + const projected = correlated.select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + label: child.label, + })) + return includeInEveryForm( + q + .from({ result: projected }) + .where(({ result }) => + eq(result.parentGroup, parent.group), + ) + .select(({ result }) => ({ + id: result.id, + value: result.label, + })), + ) + } + case `join`: + return includeInEveryForm( + correlated + .innerJoin({ tag: tags.collection }, ({ child, tag }) => + eq(child.id, tag.id), + ) + .select(({ child }) => ({ + id: child.id, + value: child.label, + })), + ) + case `group`: + return includeInEveryForm( + correlated + .groupBy(({ child }) => [child.id, child.label]) + .select(({ child }) => ({ + id: child.id, + value: child.label, + })), + ) + } + })() + return { id: parent.id, ...forms } + }), + ) + : createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + const correlated = q + .from({ child: children.collection }) + .where(({ child }) => + and( + eq(child.parentGroup, parent.group), + eq(child.token, parent.token), + ), + ) + const forms = (() => { + switch (boundary) { + case `direct`: + return includeInEveryForm( + correlated.select(({ child }) => ({ + id: child.id, + [name]: child.label, + })), + ) + case `query-ref`: { + const projected = correlated.select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + label: child.label, + })) + return includeInEveryForm( + q + .from({ result: projected }) + .where(({ result }) => + eq(result.parentGroup, parent.group), + ) + .select(({ result }) => ({ + id: result.id, + [name]: result.label, + })), + ) + } + case `join`: + return includeInEveryForm( + correlated + .innerJoin({ tag: tags.collection }, ({ child, tag }) => + eq(child.id, tag.id), + ) + .select(({ child }) => ({ + id: child.id, + [name]: child.label, + })), + ) + case `group`: + return includeInEveryForm( + correlated + .groupBy(({ child }) => [child.id, child.label]) + .select(({ child }) => ({ + id: child.id, + [name]: child.label, + })), + ) + } + })() + return { id: parent.id, ...forms } + }), + ) + + const expected = () => { + const group = parents.collection.get(1)!.group + return children.collection.toArray + .filter( + (child) => + child.parentGroup === group && + child.token === parents.collection.get(1)!.token, + ) + .map((child) => ({ id: child.id, value: child.label })) + } + const project = (rows: Iterable>) => + [...rows].map((row) => ({ + id: row.id, + value: location === `parent-alias` ? row.value : row[name], + })) + const assertCurrent = () => + expectEveryForm( + live.get(1)! as MaterializedForms>, + project, + expected(), + ) + + try { + await live.preload() + assertCurrent() + + parents.write(`update`, { id: 1, group: 2, token: `two` }) + assertCurrent() + + children.write(`update`, { + id: 20, + parentGroup: 2, + token: `two`, + label: `updated`, + }) + assertCurrent() + } finally { + await cleanup(live, [parents, children, tags]) + } +} + +class PublicSurfaceBox { + readonly map: Map + readonly set: Set + + constructor(readonly row: unknown) { + this.map = new Map([[`row`, row]]) + this.set = new Set([row]) + } +} + +function expectNoPrivateSymbolsDeep( + value: unknown, + allowedSymbols: ReadonlySet, + seen = new WeakSet(), +): void { + if (value == null || typeof value !== `object` || seen.has(value)) return + seen.add(value) + + for (const key of Reflect.ownKeys(value)) { + if (typeof key === `symbol`) { + expect( + allowedSymbols.has(key), + `unexpected private symbol in public query output`, + ).toBe(true) + } + expectNoPrivateSymbolsDeep( + (value as Record)[key], + allowedSymbols, + seen, + ) + } + + if (value instanceof Map) { + for (const [key, entry] of value) { + expectNoPrivateSymbolsDeep(key, allowedSymbols, seen) + expectNoPrivateSymbolsDeep(entry, allowedSymbols, seen) + } + } else if (value instanceof Set) { + for (const entry of value) { + expectNoPrivateSymbolsDeep(entry, allowedSymbols, seen) + } + } +} + +async function runPublicSurfaceCell({ + shape, +}: PublicSurfaceCell): Promise { + const callbackRows: Array = [] + const parents = createGrammarCollection(`surface-${shape}-parents`, [ + { id: 1, group: 1 }, + ]) + const first = new Date(`2026-01-01T00:00:00.000Z`) + const second = new Date(`2026-01-02T00:00:00.000Z`) + const third = new Date(`2026-01-03T00:00:00.000Z`) + const firstPayload = { token: `first` } + const secondPayload = { token: `second` } + const userSymbol = Symbol(`user-owned`) + const createAdversarialPayload = (marker: string) => { + const value: { + safe: string + __proto__: { marker: string } + row?: unknown + } = { safe: marker, [`__proto__`]: { marker } } + return value + } + const firstAdversarial = createAdversarialPayload(`first`) + const secondAdversarial = createAdversarialPayload(`second`) + const children = createGrammarCollection(`surface-${shape}-children`, [ + { + id: 10, + parentGroup: 1, + value: first, + payload: firstPayload, + adversarial: firstAdversarial, + symbols: { [userSymbol]: `first` }, + label: `ten`, + }, + { + id: 20, + parentGroup: 2, + value: second, + payload: secondPayload, + adversarial: secondAdversarial, + symbols: { [userSymbol]: `second` }, + label: `twenty`, + }, + ]) + const candidates = createGrammarCollection(`surface-${shape}-candidates`, [ + { id: 10, value: first }, + { id: 20, value: second }, + ]) + const anchors = createGrammarCollection(`surface-${shape}-anchors`, [ + { id: 100, parentGroup: 1, value: first }, + { id: 200, parentGroup: 2, value: second }, + { id: 300, parentGroup: 2, value: third }, + ]) + const tags = createGrammarCollection(`surface-${shape}-tags`, [ + { id: 1000, childId: 10, label: `first` }, + { id: 2000, childId: 20, label: `second` }, + ]) + + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + if (shape === `object-query-ref-scalar`) { + const values = q + .from({ candidate: candidates.collection }) + .fn.select(({ candidate }) => candidate.value) + const rows = q + .from({ anchor: anchors.collection }) + .innerJoin({ value: values }, ({ anchor, value }) => + eq(anchor.value, value), + ) + .where(({ anchor }) => eq(anchor.parentGroup, parent.group)) + .select(({ anchor, value }) => ({ id: anchor.id, value })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + const correlated = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + if (shape === `nested-functional-spread`) { + const rows = correlated.fn.select((row) => ({ + id: row.child.id, + nested: { ...row }, + })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + if (shape === `opaque-wrapper`) { + const rows = correlated.fn + .where((row) => { + callbackRows.push(row) + return true + }) + .fn.select((row) => ({ + id: row.child.id, + box: new PublicSurfaceBox(row), + })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + if (shape === `functional-having-input`) { + const rows = correlated + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ + parentGroup: child.parentGroup, + total: count(child.id), + })) + .fn.having((row) => { + callbackRows.push(row) + return true + }) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + if (shape === `nested-reference`) { + const rows = correlated.select(({ child }) => ({ + id: child.id, + payload: child.payload, + })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + if (shape === `adversarial-key`) { + const rows = correlated.fn.select((row) => { + const payload = createAdversarialPayload(row.child.adversarial.safe) + payload.row = row + return { id: row.child.id, payload } + }) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + if (shape === `user-symbol`) { + const rows = correlated.fn.select((row) => ({ + id: row.child.id, + payload: { ...row.child.symbols, row }, + })) + return { id: parent.id, ...includeInEveryForm(rows) } + } + + const rows = correlated.innerJoin( + { tag: tags.collection }, + ({ child, tag }) => eq(child.id, tag.childId), + ) + return { id: parent.id, ...includeInEveryForm(rows) } + }), + ) + + const assertCurrent = () => { + const forms = live.get(1)! + const rowsByForm = [ + [...forms.collection.values()], + [...forms.array], + [...forms.materialized], + ] + for (const rows of rowsByForm) { + for (const row of rows) { + expectNoPrivateSymbolsDeep(row, new Set([userSymbol])) + } + } + // Retain earlier callback values: later graph work must not contaminate + // objects already handed to user code with private route metadata. + for (const row of callbackRows) { + expectNoPrivateSymbolsDeep(row, new Set([userSymbol])) + } + + if (shape === `object-query-ref-scalar`) { + const expected = + parents.collection.get(1)!.group === 1 + ? [{ id: 100, value: first }] + : candidates.collection.get(20)!.value === second + ? [{ id: 200, value: second }] + : [{ id: 300, value: third }] + for (const rows of rowsByForm) { + expect(rows).toHaveLength(1) + expect((rows[0] as any).id).toBe(expected[0]!.id) + expect((rows[0] as any).value).toBe(expected[0]!.value) + } + return + } + + if (shape === `nested-functional-spread`) { + const child = + parents.collection.get(1)!.group === 1 + ? children.collection.get(10)! + : children.collection.get(20)! + for (const rows of rowsByForm) { + expect( + rows.map((row: any) => ({ + id: row.id, + child: row.nested.child, + })), + ).toEqual([{ id: child.id, child }]) + } + return + } + + const child = + parents.collection.get(1)!.group === 1 + ? children.collection.get(10)! + : children.collection.get(20)! + + if (shape === `opaque-wrapper`) { + for (const rows of rowsByForm) { + expect(rows).toHaveLength(1) + const row = rows[0] as any + expect(row.box).toBeInstanceOf(PublicSurfaceBox) + expect(row.box.row.child.id).toBe(child.id) + expect(row.box.row.child.label).toBe(child.label) + expect(row.box.map.get(`row`)).toBe(row.box.row) + expect(row.box.set.has(row.box.row)).toBe(true) + } + return + } + + if (shape === `nested-reference`) { + for (const rows of rowsByForm) { + expect((rows[0] as any).payload).toBe(child.payload) + } + return + } + + if (shape === `functional-having-input`) { + const total = children.collection.toArray.filter( + ({ parentGroup }) => parentGroup === parents.collection.get(1)!.group, + ).length + for (const rows of rowsByForm) { + expect( + rows.map((row: any) => ({ + parentGroup: row.parentGroup, + total: row.total, + })), + ).toEqual([{ parentGroup: child.parentGroup, total }]) + } + return + } + + if (shape === `adversarial-key`) { + for (const rows of rowsByForm) { + const payload = (rows[0] as any).payload + expect(Object.prototype.hasOwnProperty.call(payload, `__proto__`)).toBe( + true, + ) + expect(Object.getPrototypeOf(payload)).toBe(Object.prototype) + expect(payload.__proto__).toEqual({ + marker: child.adversarial.__proto__.marker, + }) + expect(payload.row.child.id).toBe(child.id) + } + return + } + + if (shape === `user-symbol`) { + for (const [index, rows] of rowsByForm.entries()) { + expect( + (rows[0] as any).payload[userSymbol], + materializationForms[index], + ).toBe(child.symbols[userSymbol]) + expect((rows[0] as any).payload.row.child.id).toBe(child.id) + } + return + } + + const tag = tags.collection.toArray.find( + ({ childId }) => childId === child.id, + )! + for (const rows of rowsByForm) { + expect( + rows.map((row: any) => ({ child: row.child, tag: row.tag })), + ).toEqual([{ child, tag }]) + } + } + + try { + await live.preload() + assertCurrent() + + parents.write(`update`, { id: 1, group: 2 }) + assertCurrent() + + if (shape === `object-query-ref-scalar`) { + candidates.write(`update`, { id: 20, value: third }) + } else if (shape === `functional-having-input`) { + children.write(`insert`, { + id: 30, + parentGroup: 2, + value: third, + payload: { token: `third` }, + adversarial: createAdversarialPayload(`third`), + symbols: { [userSymbol]: `third` }, + label: `thirty`, + }) + } else if (shape === `nested-reference`) { + children.write(`update`, { + ...children.collection.get(20)!, + payload: { token: `updated` }, + }) + } else if (shape === `adversarial-key`) { + children.write(`update`, { + ...children.collection.get(20)!, + adversarial: createAdversarialPayload(`updated`), + }) + } else if (shape === `user-symbol`) { + children.write(`update`, { + ...children.collection.get(20)!, + symbols: { [userSymbol]: `updated` }, + }) + } else { + children.write(`update`, { + ...children.collection.get(20)!, + label: `updated`, + }) + } + assertCurrent() + } finally { + await cleanup(live, [parents, children, candidates, anchors, tags]) + } +} + async function runGrammarCell(cell: GrammarCell): Promise { switch (cell.family) { case `parent-projection`: @@ -1476,6 +2083,10 @@ async function runGrammarCell(cell: GrammarCell): Promise { return runUnionIdentityCell(cell) case `derived-result`: return runDerivedResultCell(cell) + case `namespace-collision`: + return runNamespaceCollisionCell(cell) + case `public-surface`: + return runPublicSurfaceCell(cell) } } @@ -1494,14 +2105,68 @@ describe(`correlated include route-context transport grammar`, () => { routeContextGrammar.unionIdentity.forms.length + routeContextGrammar.derivedResult.boundaries.length * routeContextGrammar.derivedResult.selections.length * - routeContextGrammar.derivedResult.domains.length + routeContextGrammar.derivedResult.domains.length + + routeContextGrammar.namespaceCollision.locations.length * + routeContextGrammar.namespaceCollision.boundaries.length * + routeContextGrammar.namespaceCollision.names.length + + routeContextGrammar.publicSurface.shapes.length const names = grammarCells.map(grammarCellName) expect(grammarCells).toHaveLength(expectedCellCount) expect(new Set(names)).toHaveLength(expectedCellCount) expect( grammarCells.length * materializationForms.length * checkpoints.length, - ).toBe(387) + ).toBe(819) + }) + + test(`preserves cycles while removing nested route metadata`, () => { + const routed = attachRouteMetadata({ id: 1 }, 1, null) + const value: Record = { routed } + value.self = value + + const cleaned = stripInternalRouteMetadata(value) as typeof value + + expect(cleaned).not.toBe(value) + expect(cleaned.self).toBe(cleaned) + expectNoPrivateSymbolsDeep(cleaned, new Set()) + }) + + test(`does not evaluate unused accessors while cleaning routed callback rows`, async () => { + let reads = 0 + const payload = {} + Object.defineProperty(payload, `unused`, { + get() { + reads++ + throw new Error(`unused getter evaluated`) + }, + enumerable: true, + }) + const parents = createGrammarCollection(`getter-parents`, [ + { id: 1, group: 1 }, + ]) + const children = createGrammarCollection(`getter-children`, [ + { id: 10, parentGroup: 1, payload }, + ]) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .fn.where(() => true) + .fn.select(({ child }) => ({ id: child.id })), + ), + })), + ) + + try { + await live.preload() + expect(live.get(1)?.children).toEqual([{ id: 10 }]) + expect(reads).toBe(0) + } finally { + await cleanup(live, [parents, children]) + } }) for (const cell of grammarCells) { diff --git a/packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts b/packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts index 850aeb02c2..8e50137897 100644 --- a/packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts +++ b/packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts @@ -1,6 +1,11 @@ import { fc, test as fcTest } from '@fast-check/vitest' -import { describe, expect } from 'vitest' +import { describe, expect, test } from 'vitest' +import { Temporal } from 'temporal-polyfill' +import { createCollection } from '../../src/collection/index.js' +import { createFilterFunctionFromExpression } from '../../src/collection/change-events.js' import { + and, + count, createLiveQueryCollection, eq, isNull, @@ -10,9 +15,10 @@ import { toArray, } from '../../src/query/index.js' import { oraclePropertyOptions } from '../oracle-config.js' -import { flushPromises } from '../utils.js' +import { flushPromises, stripVirtualProps } from '../utils.js' import { createControlledCollection as createOracleControlledCollection } from './includes-oracle-helpers.js' import type { Collection } from '../../src/collection/index.js' +import type { LoadSubsetOptions } from '../../src/types.js' import type { ControlledCollection } from './includes-oracle-helpers.js' type ParentRow = { @@ -52,6 +58,30 @@ type FlatRow = { child: ChildRow | undefined } +type ReferenceKey = { code: number } + +type ReferenceParent = { + id: number + group: ReferenceKey +} + +type ReferenceChild = { + id: number + parentGroup: ReferenceKey +} + +type ReferenceContextParent = { + id: number + group: number + expected: ReferenceKey +} + +type ReferenceContextChild = { + id: number + group: number + token: ReferenceKey +} + function createControlledCollection( name: string, initialData: ReadonlyArray, @@ -481,6 +511,461 @@ const windowedScenarioArbitrary = fc.record({ }) describe(`includes cross-formulation oracle`, () => { + fcTest.prop( + [fc.integer()], + oraclePropertyOptions(4, `includes-cross-formulation.reference-context`), + )( + `parent-context routing preserves reference-sensitive predicate values across transitions`, + async (code) => { + const firstToken = { code } + const secondToken = { code } + const parentRows: Array = [ + { id: 1, group: 1, expected: firstToken }, + { id: 2, group: 1, expected: secondToken }, + ] + const childRows: Array = [ + { id: 10, group: 1, token: firstToken }, + { id: 20, group: 1, token: secondToken }, + ] + const parents = createControlledCollection( + `reference-context-parents`, + parentRows, + ) + const fullyLoadedChildren = createControlledCollection( + `reference-context-full-children`, + childRows, + ) + const loadedChildIds = new Set() + const lazyChildren = createCollection({ + id: `reference-context-lazy-children`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: (options: LoadSubsetOptions) => { + const matches = options.where + ? createFilterFunctionFromExpression( + options.where, + ) + : () => true + begin() + for (const row of childRows) { + if (!loadedChildIds.has(row.id) && matches(row)) { + loadedChildIds.add(row.id) + write({ type: `insert`, value: row }) + } + } + commit() + markReady() + return Promise.resolve() + }, + }), + }, + }) + const createReferenceContextQuery = ( + children: Collection, + ) => + createLiveQueryCollection({ + getKey: (row) => row.id, + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children }) + .where(({ child }) => + and( + eq(child.group, parent.group), + eq(child.token, parent.expected), + ), + ) + .select(({ child }) => child.id), + ), + })), + }) + const fullyLoaded = createReferenceContextQuery( + fullyLoadedChildren.collection, + ) + const lazy = createReferenceContextQuery(lazyChildren) + + try { + await Promise.all([fullyLoaded.preload(), lazy.preload()]) + expect(lazy.toArray.map(stripVirtualProps)).toEqual([ + { id: 1, children: [10] }, + { id: 2, children: [20] }, + ]) + expect(lazy.toArray.map(stripVirtualProps)).toEqual( + fullyLoaded.toArray.map(stripVirtualProps), + ) + + parents.write(`delete`, parentRows[0]!) + await flushPromises() + expect(lazy.toArray.map(stripVirtualProps)).toEqual([ + { id: 2, children: [20] }, + ]) + expect(lazy.toArray.map(stripVirtualProps)).toEqual( + fullyLoaded.toArray.map(stripVirtualProps), + ) + + parents.write(`insert`, parentRows[0]!) + await flushPromises() + expect(lazy.toArray.map(stripVirtualProps)).toEqual([ + { id: 1, children: [10] }, + { id: 2, children: [20] }, + ]) + } finally { + await Promise.allSettled([ + fullyLoaded.cleanup(), + lazy.cleanup(), + parents.collection.cleanup(), + fullyLoadedChildren.collection.cleanup(), + lazyChildren.cleanup(), + ]) + } + }, + ) + + test.each([ + [`Date and number`, () => [new Date(0), 0] as const], + [ + `Buffer and Uint8Array`, + () => [Buffer.from([1, 2, 3]), new Uint8Array([1, 2, 3])] as const, + ], + [ + `equivalent Temporal values`, + () => + [ + Temporal.PlainDate.from(`2024-04-05`), + Temporal.PlainDate.from(`2024-04-05`), + ] as const, + ], + ])( + `grouped includes use query equality for %s routes`, + async (_name, createValues) => { + const [parentGroup, equivalentChildGroup] = createValues() + const parents = createControlledCollection(`equality-route-parents`, [ + { id: 1, group: parentGroup as unknown }, + ]) + const children = createControlledCollection(`equality-route-children`, [ + { id: 10, parentGroup: parentGroup as unknown }, + { id: 11, parentGroup: equivalentChildGroup as unknown }, + ]) + const nested = createLiveQueryCollection({ + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + summaries: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ count: count(child.id) })), + ), + })), + }) + const standalone = createLiveQueryCollection({ + query: (q) => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parentGroup)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ count: count(child.id) })), + }) + + try { + await Promise.all([nested.preload(), standalone.preload()]) + const nestedCounts = nested + .get(1) + ?.summaries.map(({ count: childCount }) => ({ count: childCount })) + const standaloneCounts = standalone.toArray.map( + ({ count: childCount }) => ({ count: childCount }), + ) + expect(nestedCounts).toEqual(standaloneCounts) + expect(nestedCounts).toEqual([{ count: 2 }]) + } finally { + await Promise.allSettled([ + nested.cleanup(), + standalone.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + test.each([`__correlationKey`, `__tanstack_group_correlation_key`])( + `grouped includes preserve internal-looking aggregate alias %s`, + async (alias) => { + const parents = createControlledCollection(`aggregate-alias-parents`, [ + { id: 1, group: 1 }, + ]) + const children = createControlledCollection(`aggregate-alias-children`, [ + { id: 10, parentGroup: 1 }, + { id: 11, parentGroup: 1 }, + ]) + const nested = createLiveQueryCollection({ + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + summaries: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ [alias]: count(child.id) })), + ), + })), + }) + + try { + await nested.preload() + expect(nested.get(1)?.summaries.map((row) => row[alias])).toEqual([2]) + } finally { + await Promise.allSettled([ + nested.cleanup(), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + + fcTest.prop( + [fc.integer()], + oraclePropertyOptions(4, `includes-cross-formulation.reference-key`), + )( + `lazy materialization matches fully loaded materialization for reference-sensitive correlation keys`, + async (code) => { + const firstKey = { code } + const secondKey = { code } + const parentRows: Array = [ + { id: 1, group: firstKey }, + { id: 2, group: secondKey }, + ] + const childRows: Array = [ + { id: 10, parentGroup: firstKey }, + { id: 20, parentGroup: secondKey }, + ] + const parents = createControlledCollection( + `reference-key-parents`, + parentRows, + ) + const fullyLoadedChildren = createControlledCollection( + `reference-key-full-children`, + childRows, + ) + const loadedChildIds = new Set() + const lazyChildren = createCollection({ + id: `reference-key-lazy-children`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: (options: LoadSubsetOptions) => { + const matches = options.where + ? createFilterFunctionFromExpression( + options.where, + ) + : () => true + begin() + for (const row of childRows) { + if (!loadedChildIds.has(row.id) && matches(row)) { + loadedChildIds.add(row.id) + write({ type: `insert`, value: row }) + } + } + commit() + markReady() + return Promise.resolve() + }, + }), + }, + }) + + const createReferenceQuery = (children: Collection) => + createLiveQueryCollection({ + getKey: (row) => row.id, + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .select(({ child }) => child.id), + ), + })), + }) + + const createGroupedReferenceQuery = ( + children: Collection, + ) => + createLiveQueryCollection({ + getKey: (row) => row.id, + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + summaries: toArray( + q + .from({ child: children }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ count: count(child.id) })), + ), + })), + }) + + const fullyLoaded = createReferenceQuery(fullyLoadedChildren.collection) + const lazy = createReferenceQuery(lazyChildren) + const fullyLoadedGrouped = createGroupedReferenceQuery( + fullyLoadedChildren.collection, + ) + const lazyGrouped = createGroupedReferenceQuery(lazyChildren) + const groupedRows = ( + query: typeof fullyLoadedGrouped, + ): Array<{ id: number; summaries: Array<{ count: number }> }> => + query.toArray.map((row) => ({ + id: row.id, + summaries: row.summaries.map(({ count: childCount }) => ({ + count: childCount, + })), + })) + + try { + await Promise.all([ + fullyLoaded.preload(), + lazy.preload(), + fullyLoadedGrouped.preload(), + lazyGrouped.preload(), + ]) + const fullyLoadedRows = fullyLoaded.toArray.map(stripVirtualProps) + const lazyRows = lazy.toArray.map(stripVirtualProps) + expect(lazyRows).toEqual(fullyLoadedRows) + expect(lazyRows).toEqual([ + { id: 1, children: [10] }, + { id: 2, children: [20] }, + ]) + expect(groupedRows(lazyGrouped)).toEqual( + groupedRows(fullyLoadedGrouped), + ) + expect(groupedRows(lazyGrouped)).toEqual([ + { id: 1, summaries: [{ count: 1 }] }, + { id: 2, summaries: [{ count: 1 }] }, + ]) + + parents.write(`delete`, parentRows[0]!) + await flushPromises() + expect(lazy.toArray.map(stripVirtualProps)).toEqual([ + { id: 2, children: [20] }, + ]) + expect(lazy.toArray.map(stripVirtualProps)).toEqual( + fullyLoaded.toArray.map(stripVirtualProps), + ) + expect(groupedRows(lazyGrouped)).toEqual([ + { id: 2, summaries: [{ count: 1 }] }, + ]) + expect(groupedRows(lazyGrouped)).toEqual( + groupedRows(fullyLoadedGrouped), + ) + + parents.write(`insert`, parentRows[0]!) + await flushPromises() + expect(lazy.toArray.map(stripVirtualProps)).toEqual([ + { id: 1, children: [10] }, + { id: 2, children: [20] }, + ]) + expect(groupedRows(lazyGrouped)).toEqual([ + { id: 1, summaries: [{ count: 1 }] }, + { id: 2, summaries: [{ count: 1 }] }, + ]) + } finally { + await Promise.allSettled([ + fullyLoaded.cleanup(), + lazy.cleanup(), + fullyLoadedGrouped.cleanup(), + lazyGrouped.cleanup(), + parents.collection.cleanup(), + fullyLoadedChildren.collection.cleanup(), + lazyChildren.cleanup(), + ]) + } + }, + ) + + fcTest.prop( + [fc.integer()], + oraclePropertyOptions(4, `includes-cross-formulation.symbol-group-route`), + )( + `grouped includes agree with standalone groups for symbol routes`, + async (code) => { + const firstGroup = Symbol(`first-${code}`) + const secondGroup = Symbol(`second-${code}`) + const parents = createControlledCollection(`symbol-route-parents`, [ + { id: 1, group: firstGroup }, + { id: 2, group: secondGroup }, + ]) + const children = createControlledCollection(`symbol-route-children`, [ + { id: 10, parentGroup: firstGroup }, + { id: 11, parentGroup: firstGroup }, + { id: 20, parentGroup: secondGroup }, + ]) + + const nested = createLiveQueryCollection({ + getKey: (row) => row.id, + query: (q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + summaries: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ count: count(child.id) })), + ), + })), + }) + const standalone = [firstGroup, secondGroup].map((group) => + createLiveQueryCollection({ + query: (q) => + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, group)) + .groupBy(({ child }) => child.parentGroup) + .select(({ child }) => ({ count: count(child.id) })), + }), + ) + + try { + await Promise.all([ + nested.preload(), + ...standalone.map((query) => query.preload()), + ]) + expect( + nested.toArray.map((row) => ({ + id: row.id, + summaries: row.summaries.map(({ count: childCount }) => ({ + count: childCount, + })), + })), + ).toEqual( + standalone.map((query, index) => ({ + id: index + 1, + summaries: query.toArray.map(({ count: childCount }) => ({ + count: childCount, + })), + })), + ) + } finally { + await Promise.allSettled([ + nested.cleanup(), + ...standalone.map((query) => query.cleanup()), + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, + ) + fcTest(`shared-route child deletion agrees across formulations`, () => expectFormulationsEquivalent({ parents: [ @@ -497,12 +982,18 @@ describe(`includes cross-formulation oracle`, () => { }), ) - fcTest.prop([scenarioArbitrary], oraclePropertyOptions(8))( + fcTest.prop( + [scenarioArbitrary], + oraclePropertyOptions(8, `includes-cross-formulation.equivalence`), + )( `agrees across nested includes, flat joins, per-parent queries, and TLP partitions`, expectFormulationsEquivalent, ) - fcTest.prop([windowedScenarioArbitrary], oraclePropertyOptions(12))( + fcTest.prop( + [windowedScenarioArbitrary], + oraclePropertyOptions(12, `includes-cross-formulation.ordered-window`), + )( `matches recomputation for ordered offset and limit child windows`, ({ scenario, offset, limit }) => expectWindowedIncludeMatches(scenario, offset, limit), diff --git a/packages/db/tests/query/includes-functional-input-boundary.test.ts b/packages/db/tests/query/includes-functional-input-boundary.test.ts new file mode 100644 index 0000000000..0e47a21a86 --- /dev/null +++ b/packages/db/tests/query/includes-functional-input-boundary.test.ts @@ -0,0 +1,408 @@ +import { describe, expect, it } from 'vitest' +import { + createLiveQueryCollection, + eq, + materialize, + toArray, +} from '../../src/query/index.js' +import { createControlledCollection } from './includes-oracle-helpers.js' + +describe(`functional include input boundary`, () => { + it.each( + [`array`, `materialized`].flatMap((form) => + [`none`, `first`, `second`].map((failureAt) => ({ form, failureAt })), + ), + )( + `keeps chained $form projections coherent through $failureAt failure`, + async ({ form, failureAt }) => { + const parents = createControlledCollection(`chain-parent`, [ + { id: 1, group: 1 }, + ]) + const children = createControlledCollection(`chain-child`, [ + { id: 10, group: 1 }, + { id: 20, group: 2 }, + ]) + const peers = createControlledCollection(`chain-peer`, [ + { id: 100, group: 1 }, + { id: 200, group: 2 }, + ]) + const failure = new Error(`projection failed`) + let failing = false + const query = createLiveQueryCollection((q) => { + const source = q.from({ parent: parents.collection }) + const included = + form === `array` + ? source.select(({ parent }) => ({ + id: parent.id, + group: parent.group, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.group, parent.group)), + ), + })) + : source.select(({ parent }) => ({ + id: parent.id, + group: parent.group, + children: materialize( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.group, parent.group)), + ), + })) + const first = q.from({ row: included }).fn.select(({ row }) => { + if (failing && failureAt === `first`) throw failure + return { + id: row.id, + group: row.group, + ids: row.children.map((child) => child.id), + } + }) + const projected = q.from({ row: first }) + const combined = + form === `array` + ? projected.select(({ row }) => ({ + id: row.id, + ids: row.ids, + peers: toArray( + q + .from({ peer: peers.collection }) + .where(({ peer }) => eq(peer.group, row.group)), + ), + })) + : projected.select(({ row }) => ({ + id: row.id, + ids: row.ids, + peers: materialize( + q + .from({ peer: peers.collection }) + .where(({ peer }) => eq(peer.group, row.group)), + ), + })) + return q.from({ row: combined }).fn.select(({ row }) => { + if (failing && failureAt === `second`) throw failure + return { + id: row.id, + ids: [...row.ids, ...row.peers.map((peer) => peer.id)], + } + }) + }) + try { + await query.preload() + const old = query.get(1)! + expect(old.ids).toEqual([10, 100]) + failing = failureAt !== `none` + if (failing) { + expect(() => parents.write(`update`, { id: 1, group: 2 })).toThrow( + failure, + ) + expect(query.get(1)).toBe(old) + expect(old.ids).toEqual([10, 100]) + } else { + parents.write(`update`, { id: 1, group: 2 }) + expect(query.get(1)!.ids).toEqual([20, 200]) + expect(old.ids).toEqual([10, 100]) + } + await query.cleanup() + failing = false + await query.preload() + expect(query.get(1)!.ids).toEqual([20, 200]) + peers.write(`insert`, { id: 201, group: 2 }) + expect(query.get(1)!.ids).toEqual([20, 200, 201]) + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + await peers.collection.cleanup() + } + }, + ) + + it(`keeps singleton materialization reactive through absence`, async () => { + const parents = createControlledCollection(`singleton-parent`, [{ id: 1 }]) + const children = createControlledCollection(`singleton-child`, [ + { id: 10, parentId: 2, value: 3 }, + ]) + const query = createLiveQueryCollection((q) => { + const included = q + .from({ parent: parents.collection }) + .select(({ parent }) => ({ + id: parent.id, + child: materialize( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)) + .findOne(), + ), + })) + return q + .from({ row: included }) + .fn.select(({ row }) => ({ id: row.id, value: row.child?.value ?? 0 })) + }) + try { + await query.preload() + expect(query.get(1)!.value).toBe(0) + children.write(`update`, { id: 10, parentId: 1, value: 3 }) + expect(query.get(1)!.value).toBe(3) + children.write(`update`, { id: 10, parentId: 1, value: 7 }) + expect(query.get(1)!.value).toBe(7) + children.write(`delete`, { id: 10, parentId: 1, value: 7 }) + expect(query.get(1)!.value).toBe(0) + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }) + + it.each([ + `read`, + `pass-through`, + `ignore`, + `subscribe`, + `create-index`, + ] as const)( + `rejects a Collection input before the callback can %s it`, + async (use) => { + const parents = createControlledCollection(`boundary-parent`, [{ id: 1 }]) + const children = createControlledCollection(`boundary-child`, [ + { id: 10, parentId: 1 }, + ]) + let calls = 0 + let query: ReturnType | undefined + try { + await expect( + (async () => { + query = createLiveQueryCollection((q) => + q + .from({ + row: q + .from({ parent: parents.collection }) + .select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + })), + }) + .fn.select(({ row }) => { + calls++ + if (use === `subscribe`) + row.children.subscribeChanges(() => {}) + if (use === `create-index`) + row.children.createIndex((child) => child.id) + return { + id: row.id, + value: + use === `read` + ? row.children.size + : use === `pass-through` + ? row.children + : null, + } + }), + ) + await query.preload() + })(), + ).rejects.toThrow( + `fn.select() cannot consume Collection-valued includes`, + ) + expect(calls).toBe(0) + } finally { + await query?.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it.each([`array`, `materialized`] as const)( + `keeps %s calculations reactive after child-only changes`, + async (form) => { + const parents = createControlledCollection(`inline-parent`, [{ id: 1 }]) + const children = createControlledCollection(`inline-child`, [ + { id: 10, parentId: 1, value: 3 }, + ]) + let calls = 0 + const query = createLiveQueryCollection((q) => { + const source = q.from({ parent: parents.collection }) + const included = + form === `array` + ? source.select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + ), + })) + : source.select(({ parent }) => ({ + id: parent.id, + children: materialize( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + ), + })) + return q.from({ row: included }).fn.select(({ row }) => { + calls++ + return { + id: row.id, + count: row.children.length, + sum: row.children.reduce((sum, child) => sum + child.value, 0), + found: row.children.find((child) => child.id === 10)?.value, + } + }) + }) + try { + await query.preload() + expect(query.get(1)).toMatchObject({ count: 1, sum: 3, found: 3 }) + const initialCalls = calls + children.write(`update`, { id: 10, parentId: 1, value: 7 }) + expect(query.get(1)).toMatchObject({ count: 1, sum: 7, found: 7 }) + expect(calls).toBeGreaterThan(initialCalls) + children.write(`insert`, { id: 11, parentId: 1, value: 5 }) + expect(query.get(1)).toMatchObject({ count: 2, sum: 12, found: 7 }) + children.write(`delete`, { id: 10, parentId: 1, value: 7 }) + expect(query.get(1)).toMatchObject({ + count: 1, + sum: 5, + found: undefined, + }) + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it(`keeps a bare Collection live through an expression projection`, async () => { + const parents = createControlledCollection(`expression-parent`, [{ id: 1 }]) + const children = createControlledCollection(`expression-child`, [ + { id: 10, parentId: 1 }, + ]) + const query = createLiveQueryCollection((q) => + q + .from({ + row: q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + })), + }) + .select(({ row }) => ({ id: row.id, children: row.children })), + ) + try { + await query.preload() + const held = query.get(1)!.children + expect(held.get(10)?.id).toBe(10) + children.write(`insert`, { id: 11, parentId: 1 }) + expect(query.get(1)!.children).toBe(held) + expect(held.get(11)?.id).toBe(11) + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }) + + it(`keeps parent-only functional work before adding live children`, async () => { + const parents = createControlledCollection(`parent-first`, [{ id: 1 }]) + const children = createControlledCollection(`parent-first-child`, [ + { id: 10, parentId: 1 }, + ]) + const query = createLiveQueryCollection((q) => { + const projected = q + .from({ parent: parents.collection }) + .fn.select(({ parent }) => ({ + id: parent.id, + label: `Parent ${parent.id}`, + })) + return q.from({ row: projected }).select(({ row }) => ({ + id: row.id, + label: row.label, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, row.id)), + })) + }) + let publications = 0 + const subscription = query.subscribeChanges(() => { + publications++ + }) + try { + await query.preload() + const held = query.get(1)!.children + expect(query.get(1)!.label).toBe(`Parent 1`) + expect(held.get(10)?.id).toBe(10) + publications = 0 + children.write(`insert`, { id: 11, parentId: 1 }) + expect(held.get(11)?.id).toBe(11) + expect(publications).toBe(0) + } finally { + subscription.unsubscribe() + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }) + + it.each([false, true])( + `checks nested Collection inputs (inline=%s)`, + async (inline) => { + const parents = createControlledCollection(`nested-parent`, [{ id: 1 }]) + const children = createControlledCollection(`nested-child`, [ + { id: 10, parentId: 1 }, + ]) + const leaves = createControlledCollection(`nested-leaf`, [ + { id: 100, childId: 10 }, + ]) + let cleanup = async () => {} + try { + const run = async () => { + const query = createLiveQueryCollection((q) => { + const included = q + .from({ parent: parents.collection }) + .select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)) + .select(({ child }) => { + const leafQuery = q + .from({ leaf: leaves.collection }) + .where(({ leaf }) => eq(leaf.childId, child.id)) + return { + id: child.id, + leaves: inline ? toArray(leafQuery) : leafQuery, + } + }), + ), + })) + return q + .from({ row: included }) + .fn.select(({ row }) => ({ id: row.id, children: row.children })) + }) + cleanup = () => query.cleanup() + await query.preload() + expect(query.get(1)).toMatchObject({ + children: [{ id: 10, leaves: [{ id: 100 }] }], + }) + } + if (inline) await run() + else + await expect(run()).rejects.toThrow( + `fn.select() cannot consume Collection-valued includes`, + ) + } finally { + await cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + await leaves.collection.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/query/includes-functional-projection-oracle.test.ts b/packages/db/tests/query/includes-functional-projection-oracle.test.ts new file mode 100644 index 0000000000..1c10fda82d --- /dev/null +++ b/packages/db/tests/query/includes-functional-projection-oracle.test.ts @@ -0,0 +1,1536 @@ +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { BucketFacadeAdapter } from '../../src/query/live/bucket-facade-adapter.js' +import { + createLiveQueryCollection, + eq, + materialize, + toArray, +} from '../../src/query/index.js' +import { flushPromises } from '../utils.js' +import { createControlledCollection } from './includes-oracle-helpers.js' + +const boundaries = [`query-ref`, `recursive-query-ref`, `union`] as const +const forms = [`collection`, `array`, `materialized`] as const +const outputs = [`expression`, `record`, `opaque-root`] as const +const initialStates = [`empty`, `populated`] as const +const cells = boundaries.flatMap((boundary) => + forms.flatMap((form) => + outputs.flatMap((output) => + initialStates.map((initial) => ({ boundary, form, output, initial })), + ), + ), +) +const consumers = [`expression`, `functional`] as const +const valueShapes = [`number`, `null`, `date`, `dropped-record`] as const +const valueCells = forms.flatMap((form) => + consumers.flatMap((consumer) => + valueShapes.flatMap((shape) => + [false, true].map((withInclude) => ({ + form, + consumer, + shape, + withInclude, + })), + ), + ), +) +const renamedCells = forms.flatMap((form) => + consumers.flatMap((consumer) => + consumers.flatMap((projection) => + [false, true].map((withSibling) => ({ + form, + consumer, + projection, + withSibling, + })), + ), + ), +) +const operatorCells = forms.flatMap((form) => + ([`custom-key`, `selected-order`, `distinct`] as const).flatMap((operator) => + [false, true].map((readsInclude) => ({ form, operator, readsInclude })), + ), +) + +type Child = { id: number; parentGroup: number; value: number } +type Input = { id: number; kind: string; children?: unknown } +type Phase = `initial` | `child-update` | `sibling-update` | `route-move` +type ChildView = { + valid: boolean + ready: boolean | undefined + rows: Array +} + +function readChildren(value: unknown, form: (typeof forms)[number]): ChildView { + if (form !== `collection`) { + return { + valid: Array.isArray(value), + ready: undefined, + rows: Array.isArray(value) ? value : [], + } + } + if ( + typeof value !== `object` || + value === null || + !(`toArray` in value) || + !(`isReady` in value) || + typeof value.isReady !== `function` + ) { + return { valid: false, ready: undefined, rows: [] } + } + return { + valid: Array.isArray(value.toArray), + ready: value.isReady(), + rows: Array.isArray(value.toArray) ? value.toArray : [], + } +} + +// Keep only the selected public fields in row comparisons. Callback-time shape +// and facade readiness have their own assertions rather than being normalized away. +function publicRows(rows: ReadonlyArray) { + return rows + .map(({ id, parentGroup, value }) => ({ id, parentGroup, value })) + .sort((left, right) => left.id - right.id) +} + +const collectionInputError = `fn.select() cannot consume Collection-valued includes` +function rejectsCollectionInput( + form: string, + ...functional: Array +): boolean { + return form === `collection` && functional.some(Boolean) +} + +class Projection { + constructor( + readonly id: number, + readonly kind: string, + readonly children: unknown, + readonly total: number, + ) {} +} + +describe(`functional projection output compatibility`, () => { + it.each( + ([`expression`, `functional`] as const).flatMap((projection) => + ([`resolve`, `reject`, `cleanup-resolve`, `cleanup-reject`] as const).map( + (settlement) => ({ projection, settlement }), + ), + ), + )( + `$projection projection fences $settlement of a pending child load`, + async ({ projection, settlement }) => { + const parents = createControlledCollection(`pending-parent`, [ + { id: 1, groupId: 1 }, + ]) + const requests: Array<{ + gate: ReturnType> + signal: AbortSignal | undefined + }> = [] + const children = createCollection<{ id: number; groupId: number }>({ + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: ({ signal }) => { + const gate = createDeferred() + requests.push({ gate, signal }) + return gate.promise.then(async () => { + if (signal?.aborted) return + begin() + write({ type: `insert`, value: { id: 10, groupId: 1 } }) + await commit() + markReady() + }) + }, + }), + }, + }) + const captured: Array> = [] + const buildQuery = () => + createLiveQueryCollection((q) => { + const source = q.from({ + row: q + .from({ parent: parents.collection }) + .select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.groupId, parent.groupId)), + })), + }) + return projection === `expression` + ? source.select(({ row }) => row) + : source.fn.select(({ row }) => { + captured.push(row.children) + return { id: row.id, children: row.children } + }) + }) + if (rejectsCollectionInput(`collection`, projection === `functional`)) { + try { + expect(buildQuery).toThrow(collectionInputError) + expect(captured).toEqual([]) + } finally { + await parents.collection.cleanup() + await children.cleanup() + } + return + } + const query = buildQuery() + const failure = new Error(`pending child failed`) + // Attach both outcomes immediately; no pending-length assertion may + // leave a rejected preload promise unobserved. + const preload = () => { + const result: { settled: boolean; error?: unknown } = { settled: false } + const observed = query.preload().then( + () => { + result.settled = true + }, + (error) => { + result.settled = true + result.error = error + }, + ) + return { result, observed } + } + try { + const initial = preload() + await flushPromises() + expect(requests).toHaveLength(1) + expect(initial.result.settled).toBe(false) + expect(query.isReady()).toBe(false) + const held = query.get(1)!.children + expect(held.toArray).toEqual([]) + if (projection === `functional`) expect(captured).toHaveLength(1) + const obsoleteViews = [...captured, held] + + if (settlement.startsWith(`cleanup`)) { + await query.cleanup() + await children.cleanup() + await initial.observed + expect(initial.result.error).toMatchObject({ name: `AbortError` }) + expect(requests[0]!.signal?.aborted).toBe(true) + const restarted = preload() + await flushPromises() + expect(requests).toHaveLength(2) + const current = query.get(1)!.children + if (settlement === `cleanup-reject`) requests[0]!.gate.reject(failure) + else requests[0]!.gate.resolve() + await flushPromises() + expect( + restarted.result.settled, + `obsolete completion cannot finish preload`, + ).toBe(false) + expect(query.isReady()).toBe(false) + expect(current.toArray).toEqual([]) + requests[1]!.gate.resolve() + await restarted.observed + expect(restarted.result.error).toBeUndefined() + expect(query.isReady()).toBe(true) + expect(current.toArray.map((child) => child.id)).toEqual([10]) + for (const view of obsoleteViews) expect(view.toArray).toEqual([]) + } else { + if (settlement === `reject`) requests[0]!.gate.reject(failure) + else requests[0]!.gate.resolve() + await initial.observed + if (settlement === `reject`) { + expect(initial.result.error).toBe(failure) + expect(query.isReady()).toBe(false) + expect(held.toArray).toEqual([]) + } else { + expect(initial.result.error).toBeUndefined() + expect(query.isReady()).toBe(true) + expect(held.toArray.map((child) => child.id)).toEqual([10]) + for (const view of captured) + expect(view.toArray.map((child) => child.id)).toEqual([10]) + } + } + } finally { + await query.cleanup() + for (const { gate } of requests) gate.resolve() + await children.cleanup() + await parents.collection.cleanup() + } + }, + ) + + it.each( + ([`publication`, `after-preload`] as const).flatMap((subscribeAt) => + ([`none`, `callback`, `flush`] as const).map((failureAt) => ({ + subscribeAt, + failureAt, + })), + ), + )( + `keeps $subscribeAt subscriptions isolated through $failureAt failure and restart`, + async ({ subscribeAt, failureAt }) => { + const parents = createControlledCollection(`subscription-parent`, [ + { id: 1, groupId: 1 }, + ]) + const children = createControlledCollection(`subscription-child`, [ + { id: 10, groupId: 1 }, + { id: 20, groupId: 2 }, + ]) + const observers: Array<{ + rows: Set + batches: Array> + view: Pick + }> = [] + const releases: Array<() => void> = [] + const observe = (view: (typeof observers)[number][`view`]) => { + const rows = new Set() + const batches: Array> = [] + const subscription = view.subscribeChanges( + (changes) => { + batches.push( + changes.map((change) => `${change.type}:${change.value.id}`), + ) + for (const change of changes) { + if (change.type === `delete`) rows.delete(change.value.id) + else rows.add(change.value.id) + } + }, + { includeInitialState: true }, + ) + releases.push(() => subscription.unsubscribe()) + observers.push({ rows, batches, view }) + } + const failure = new Error(`projection subscription ${failureAt} failure`) + let failing = false + let flushReached = false + const originalFlush = BucketFacadeAdapter.prototype.flush + // Fail after actual facade writes, before any deferred public events. + // An event-listener throw is asynchronous and would not test rollback. + const flush = + failureAt === `flush` + ? vi + .spyOn(BucketFacadeAdapter.prototype, `flush`) + .mockImplementation(function (this: BucketFacadeAdapter) { + const publication = originalFlush.call(this) + return { + ...publication, + prepare: () => { + publication.prepare() + if (failing) { + flushReached = true + throw failure + } + }, + } + }) + : undefined + const query = createLiveQueryCollection((q) => { + const projected = q + .from({ parent: parents.collection }) + .fn.select(({ parent }) => { + if (failing && failureAt === `callback`) throw failure + return parent + }) + return q.from({ row: projected }).select(({ row }) => ({ + id: row.id, + groupId: row.groupId, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.groupId, row.groupId)), + })) + }) + if (subscribeAt === `publication`) { + const rootSubscription = query.subscribeChanges( + (changes) => { + for (const change of changes) + if (change.type !== `delete`) observe(change.value.children) + }, + { includeInitialState: true }, + ) + releases.push(() => rootSubscription.unsubscribe()) + } + const ids = (observer: (typeof observers)[number]) => + [...observer.rows].sort((a, b) => a - b) + try { + await query.preload() + if (subscribeAt === `after-preload`) observe(query.get(1)!.children) + const first = observers[0]! + expect(ids(first), `initial subscription snapshot`).toEqual([10]) + children.write(`insert`, { id: 11, groupId: 1 }) + expect(ids(first), `initial live insert`).toEqual([10, 11]) + const originalRow = query.get(1) + const beforeFailure = first.batches.length + failing = failureAt !== `none` + const move = () => parents.write(`update`, { id: 1, groupId: 2 }) + if (failing) { + let thrown: unknown + try { + move() + } catch (error) { + thrown = error + } + expect(thrown).toBe(failure) + expect(query.get(1), `root rollback`).toBe(originalRow) + expect(ids(first), `old subscriber rollback`).toEqual([10, 11]) + expect(first.batches.length, `no partial public events`).toBe( + beforeFailure, + ) + expect(observers).toHaveLength(1) + if (failureAt === `flush`) expect(flushReached).toBe(true) + } else { + move() + if (subscribeAt === `after-preload`) observe(query.get(1)!.children) + expect(ids(first), `retired route`).toEqual([]) + expect(ids(observers[1]!), `destination subscription`).toEqual([20]) + } + + // Keep the external subscriptions alive across cleanup. They belong to + // the old graph, not the next graph created by preload on this query. + await query.cleanup() + const oldObservers = [...observers] + const oldBatches = oldObservers.map( + (observer) => observer.batches.length, + ) + failing = false + await query.preload() + if (subscribeAt === `after-preload`) observe(query.get(1)!.children) + expect(observers).toHaveLength(oldObservers.length + 1) + expect(query.get(1)!.groupId, `restart uses current source`).toBe(2) + const restarted = observers.at(-1)! + expect(ids(restarted), `restart subscription snapshot`).toEqual([20]) + children.write(`insert`, { id: 22, groupId: 2 }) + expect(ids(restarted), `restart live insert`).toEqual([20, 22]) + expect( + oldObservers.map((observer) => observer.batches.length), + `old graph receives no fresh events`, + ).toEqual(oldBatches) + for (const observer of oldObservers) { + expect( + observer.view.toArray, + `old graph exposes no fresh rows`, + ).toEqual([]) + } + } finally { + failing = false + flush?.mockRestore() + for (const release of releases) release() + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + const readSurfaces = [ + `toArray`, + `get`, + `has`, + `size`, + `keys`, + `values`, + `entries`, + `iterator`, + `forEach`, + `map`, + `state`, + `virtual-key`, + `virtual-metadata`, + `index`, + ] as const + it.each( + readSurfaces.flatMap((surface) => + [false, true].map((ordered) => ({ surface, ordered })), + ), + )( + `keeps published $surface reads live (ordered=$ordered)`, + async ({ surface, ordered }) => { + const parents = createControlledCollection(`read-api-parent`, [ + { id: 1, groupId: 1 }, + ]) + const children = createControlledCollection(`read-api-child`, [ + { id: 10, groupId: 1 }, + { id: 11, groupId: 1 }, + { id: 20, groupId: 2 }, + { id: 21, groupId: 2 }, + ]) + const readers = new Map< + number, + (keys: Array) => Array + >() + const query = createLiveQueryCollection((q) => + q + .from({ + row: q.from({ parent: parents.collection }).select(({ parent }) => { + const childQuery = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.groupId, parent.groupId)) + return { + id: parent.id, + groupId: parent.groupId, + children: ordered + ? childQuery.orderBy(({ child }) => child.id, `desc`) + : childQuery, + } + }), + }) + .select(({ row }) => row), + ) + const capture = (row: NonNullable>) => { + const view = row.children + const expectedKeys = [row.groupId * 10, row.groupId * 10 + 1] + const createIndex = view.createIndex.bind(view) + const read = (keys: Array) => { + let ids: Array + switch (surface) { + case `toArray`: + ids = view.toArray.map((child) => child.id) + break + case `get`: + ids = keys.flatMap((key) => view.get(key)?.id ?? []) + break + case `has`: + ids = keys.filter((key) => view.has(key)) + break + case `size`: + ids = [view.size] + break + case `keys`: + ids = [...view.keys()] + break + case `values`: + ids = [...view.values()].map((child) => child.id) + break + case `entries`: + ids = [...view.entries()].map(([key]) => key) + break + case `iterator`: + ids = [...view].map(([key]) => key) + break + case `forEach`: + ids = [] + view.forEach((child) => ids.push(child.id)) + break + case `map`: + ids = view.map((child) => child.id) + break + case `state`: + ids = [...view.state.keys()] + break + case `virtual-key`: + ids = view.toArray.map((child) => child.$key) + break + case `virtual-metadata`: + ids = view.toArray.map((child) => { + expect(child.$collectionId).toBe(children.collection.id) + expect(child.$synced).toBe(true) + expect(child.$origin).toBe(`remote`) + return child.id + }) + break + case `index`: { + const index = createIndex((child) => child.id, { + indexType: BasicIndex, + }) + ids = keys.flatMap((key) => [...index.lookup(`eq`, key)]) + break + } + } + return ids + } + readers.set(row.groupId, read) + const ids = read(expectedKeys) + return { id: row.id, ids, children: view } + } + const expected = (group: number) => { + if (surface === `size`) return [2] + const ids = [group * 10, group * 10 + 1] + return ordered && ![`get`, `has`, `index`].includes(surface) + ? ids.reverse() + : ids + } + const checkPublished = ( + group: number, + ids: Array, + phase: string, + ) => { + const actual = readers.get(group)!([ + group * 10, + group * 10 + 1, + group * 10 + 2, + ]) + const result = + surface === `size` + ? [ids.length] + : ordered && ![`get`, `has`, `index`].includes(surface) + ? [...ids].reverse() + : ids + expect.soft(actual, phase).toEqual(result) + } + try { + await query.preload() + const initialRead = capture(query.get(1)!) + checkPublished(1, [10, 11], `initial published read`) + expect + .soft(initialRead.ids, `initial published input`) + .toEqual(expected(1)) + parents.write(`update`, { id: 1, groupId: 2 }) + const movedRead = capture(query.get(1)!) + expect.soft(movedRead.ids, `moved published input`).toEqual(expected(2)) + checkPublished(1, [], `retired route read`) + checkPublished(2, [20, 21], `moved published read`) + const held = query.get(1)!.children + children.write(`insert`, { id: 22, groupId: 2 }) + expect(held.toArray.map((child) => child.id)).toEqual( + ordered ? [22, 21, 20] : [20, 21, 22], + ) + checkPublished(1, [], `retired route ignores later insert`) + checkPublished(2, [20, 21, 22], `published insertion read`) + children.write(`delete`, { id: 21, groupId: 2 }) + checkPublished(2, [20, 22], `published deletion read`) + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it.each([`expression`, `plain`, `opaque`, `closure`] as const)( + `keeps retained views live across a same-route parent update through a %s holder`, + async (holder) => { + const parents = createControlledCollection(`facade-identity-parent`, [ + { id: 1, groupId: 1, label: `first` }, + { id: 2, groupId: 1, label: `second` }, + ]) + const childSource = createControlledCollection(`facade-identity-child`, [ + { id: 10, groupId: 1 }, + ]) + class Holder { + constructor(readonly children: T) {} + } + const query = createLiveQueryCollection((q) => { + const source = q.from({ + row: q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + label: parent.label, + children: q + .from({ child: childSource.collection }) + .where(({ child }) => eq(child.groupId, parent.groupId)), + })), + }) + return source.select(({ row }) => ({ + id: row.id, + label: row.label, + box: { children: row.children }, + })) + }) + try { + await query.preload() + const childrenAtPublication = query.get(1)!.box.children + const retained = + holder === `opaque` + ? new Holder(childrenAtPublication) + : holder === `closure` + ? { + get children() { + return childrenAtPublication + }, + } + : { children: childrenAtPublication } + const held = retained.children + expect + .soft( + held.toArray.map((child) => child.id), + `initial rows`, + ) + .toEqual([10]) + expect + .soft(query.get(2)!.box.children, `initial shared identity`) + .toBe(held) + parents.write(`update`, { id: 1, groupId: 1, label: `changed` }) + expect + .soft(query.get(1)!.label, `parent update is visible`) + .toBe(`changed`) + expect(query.get(1)!.box.children).toBe(held) + expect + .soft( + query.get(2)!.box.children, + `unchanged parent keeps shared facade`, + ) + .toBe(held) + childSource.write(`insert`, { id: 11, groupId: 1 }) + for (const facade of [ + held, + query.get(1)!.box.children, + query.get(2)!.box.children, + ]) { + expect + .soft( + facade.toArray.map((child) => child.id).sort(), + `retained view stays live`, + ) + .toEqual([10, 11]) + } + } finally { + await query.cleanup() + await parents.collection.cleanup() + await childSource.collection.cleanup() + } + }, + ) + + it.each([`rows`, `index`, `callback-read`, `captured-method`] as const)( + `keeps held facade %s unchanged when a parent projection throws`, + async (surface) => { + const parents = createControlledCollection(`snapshot-parent`, [ + { id: 1, groupId: 1 }, + ]) + const children = createControlledCollection(`snapshot-child`, [ + { id: 10, groupId: 1 }, + { id: 20, groupId: 2 }, + ]) + const failure = new Error(`parent projection failed`) + let fail = false + let readPublished: (() => Array) | undefined + let capturedGet: ((key: number) => { id: number } | undefined) | undefined + let observed: Array | undefined + const query = createLiveQueryCollection((q) => { + const projected = q + .from({ parent: parents.collection }) + .fn.select(({ parent }) => { + if (fail) { + observed = readPublished?.() + throw failure + } + return parent + }) + return q.from({ row: projected }).select(({ row }) => ({ + id: row.id, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.groupId, row.groupId)), + })) + }) + try { + await query.preload() + const original = query.get(1)! + const held = original.children + capturedGet = held.get.bind(held) + readPublished = () => + surface === `captured-method` + ? [capturedGet?.(10)?.id].filter((id) => id !== undefined) + : held.toArray.map((child) => child.id) + const index = held.createIndex((child) => child.id, { + indexType: BasicIndex, + }) + expect(held.toArray.map((child) => child.id)).toEqual([10]) + expect(index.lookup(`eq`, 10)).toEqual(new Set([10])) + fail = true + expect(() => parents.write(`update`, { id: 1, groupId: 2 })).toThrow( + failure, + ) + expect(query.get(1)).toBe(original) + if (surface === `rows`) { + expect(held.toArray.map((child) => child.id)).toEqual([10]) + } else if (surface === `index`) { + expect(index.lookup(`eq`, 10)).toEqual(new Set([10])) + } else { + expect(observed).toEqual([10]) + } + } finally { + await query.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it.each([false, true])( + `preserves projection through public-facade changes and parent restoration (reads=%s)`, + async (readsFacade) => { + const parents = createControlledCollection(`published-parent`, [ + { id: 1 }, + ]) + const children = createControlledCollection(`published-child`, [ + { id: 10, parentId: 1 }, + ]) + const source = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => ({ + id: parent.id, + children: q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + })), + ) + const projected = createLiveQueryCollection((q) => + q + .from({ row: source }) + .fn.select(({ row }) => { + const count = readsFacade + ? readChildren(row.children, `collection`).rows.length + : 1 + return { id: row.id, count } + }) + .distinct(), + ) + const rows = () => + projected.toArray.map(({ id, count }) => ({ id, count })) + try { + await source.preload() + await projected.preload() + expect(rows()).toEqual([{ id: 1, count: 1 }]) + children.write(`insert`, { id: 11, parentId: 1 }) + expect( + readChildren(source.toArray[0]?.children, `collection`).rows, + ).toHaveLength(2) + // A stable facade does not make its scalar reads child dependencies. + expect(rows()).toEqual([{ id: 1, count: 1 }]) + parents.write(`delete`, { id: 1 }) + expect(rows()).toEqual([]) + children.write(`delete`, { id: 11, parentId: 1 }) + parents.write(`insert`, { id: 1 }) + expect(rows()).toEqual([{ id: 1, count: 1 }]) + } finally { + await projected.cleanup() + await source.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it(`covers the declared output and renamed-field products`, () => { + expect(valueCells).toHaveLength(48) + expect(renamedCells).toHaveLength(24) + expect(operatorCells).toHaveLength(18) + expect(new Set(valueCells.map((cell) => JSON.stringify(cell))).size).toBe( + 48, + ) + expect(new Set(renamedCells.map((cell) => JSON.stringify(cell))).size).toBe( + 24, + ) + expect( + new Set(operatorCells.map((cell) => JSON.stringify(cell))).size, + ).toBe(18) + }) + + it.each(operatorCells)( + `$form / $operator / reads-include=$readsInclude consumes the projected value`, + async ({ form, operator, readsInclude }) => { + const parents = createControlledCollection(`operator-parent`, [ + { id: 1, group: 1, base: 3 }, + { id: 2, group: 2, base: 5 }, + ]) + const children = createControlledCollection(`operator-child`, [ + { id: 10, parentGroup: 1, value: 3 }, + { id: 20, parentGroup: 2, value: 5 }, + ]) + // This model stores the scalar computed on a parent projection. A live + // Collection handle does not make its scalar reads child dependencies. + const expectedScores = new Map([ + [1, 3], + [2, 5], + ]) + const observed: Array = [] + const buildQuery = () => + createLiveQueryCollection({ + query: (q) => { + const source = q + .from({ parent: parents.collection }) + .select(({ parent }) => { + const childRows = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + return { + id: parent.id, + base: parent.base, + children: + form === `collection` + ? childRows + : form === `array` + ? toArray(childRows) + : materialize(childRows), + } + }) + const projected = q.from({ row: source }).fn.select(({ row }) => { + const view = readsInclude + ? readChildren(row.children, form) + : undefined + if (view) observed.push({ ...view, rows: publicRows(view.rows) }) + return { + id: operator === `distinct` ? 0 : row.id, + score: view + ? view.rows.reduce((sum, child) => sum + child.value, 0) + : row.base, + } + }) + if (operator === `distinct`) return projected.distinct() + if (operator === `selected-order`) + return projected + .orderBy(({ $selected }) => $selected.score, `desc`) + .orderBy(({ $selected }) => $selected.id) + .limit(1) + return projected + }, + getKey: + operator === `custom-key` ? (row) => `result:${row.id}` : undefined, + }) + if (rejectsCollectionInput(form, true)) { + try { + expect(buildQuery).toThrow(collectionInputError) + expect(observed).toEqual([]) + } finally { + await parents.collection.cleanup() + await children.collection.cleanup() + } + return + } + const live = buildQuery() + const check = () => { + let expected = [...expectedScores].map(([id, score]) => ({ id, score })) + if (operator === `distinct`) + expected = [...new Set(expected.map((row) => row.score))].map( + (score) => ({ id: 0, score }), + ) + if (operator === `selected-order`) + expected = expected + .sort( + (left, right) => right.score - left.score || left.id - right.id, + ) + .slice(0, 1) + const sort = (rows: Array<{ id: number; score: number }>) => + rows.sort( + (left, right) => left.id - right.id || left.score - right.score, + ) + expect + .soft(sort(live.toArray.map(({ id, score }) => ({ id, score })))) + .toEqual(sort(expected)) + if (operator === `custom-key`) + expect + .soft([...live.keys()].sort()) + .toEqual(expected.map((row) => `result:${row.id}`).sort()) + for (const view of observed) { + expect.soft(view.valid, `operator callback input form`).toBe(true) + if (form === `collection`) + expect + .soft(view.ready, `operator callback input readiness`) + .toBe(true) + } + observed.length = 0 + } + try { + await live.preload() + check() + if (readsInclude && form !== `collection`) expectedScores.set(1, 7) + children.write(`update`, { id: 10, parentGroup: 1, value: 7 }) + check() + expectedScores.set(1, 5) + parents.write(`update`, { id: 1, group: 2, base: 5 }) + check() + expectedScores.delete(1) + parents.write(`delete`, { id: 1, group: 2, base: 5 }) + check() + } finally { + await live.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it.each(valueCells)( + `$form / $consumer / $shape / include=$withInclude preserves arbitrary output`, + async ({ form, consumer, shape, withInclude }) => { + const parents = createControlledCollection(`output-parent`, [ + { id: 1, value: 2 }, + ]) + const children = createControlledCollection(`output-child`, [ + { id: 10, parentId: 1 }, + ]) + let expectedValue = 2 + const assertValue = (value: unknown, expected?: number) => { + switch (shape) { + case `number`: + expect.soft(typeof value).toBe(`number`) + if (expected !== undefined) expect.soft(value).toBe(expected) + break + case `null`: + expect.soft(value).toBeNull() + break + case `date`: + expect.soft(value instanceof Date).toBe(true) + if (expected !== undefined && value instanceof Date) + expect.soft(value.getTime()).toBe(expected * 1000) + break + case `dropped-record`: + expect.soft(value !== null && typeof value === `object`).toBe(true) + if (value !== null && typeof value === `object`) { + // Virtual properties are public metadata. Check the selected field + // and forbid input paths without imposing a new metadata contract. + expect.soft(`children` in value || `row` in value).toBe(false) + expect.soft(`code` in value).toBe(true) + if (expected !== undefined && `code` in value) + expect.soft(value.code).toBe(expected) + } + } + } + const buildQuery = () => + createLiveQueryCollection((q) => { + const source = q + .from({ parent: parents.collection }) + .select(({ parent }) => { + const childRows = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)) + return { + id: parent.id, + value: parent.value, + ...(withInclude + ? { + children: + form === `collection` + ? childRows + : form === `array` + ? toArray(childRows) + : materialize(childRows), + } + : {}), + } + }) + const projected = q.from({ row: source }).fn.select(({ row }) => { + switch (shape) { + case `number`: + return row.value + case `null`: + return null + case `date`: + return new Date(row.value * 1000) + case `dropped-record`: + return { code: row.value } + } + }) + const outer = q.from({ result: projected }) + return consumer === `expression` + ? outer.select(({ result }) => ({ value: result })) + : outer.fn.select(({ result }) => { + // Observe the value on entry, including retract callbacks. Those + // may carry an earlier value, but must still have its proper type. + assertValue(result) + return { value: result } + }) + }) + if (rejectsCollectionInput(form, withInclude)) { + try { + expect(buildQuery).toThrow(collectionInputError) + } finally { + await parents.collection.cleanup() + await children.collection.cleanup() + } + return + } + const live = buildQuery() + const check = () => { + expect.soft(live.toArray).toHaveLength(1) + assertValue(live.toArray[0]?.value, expectedValue) + } + try { + await live.preload() + check() + children.write(`insert`, { id: 11, parentId: 1 }) + check() + expectedValue = 4 + parents.write(`update`, { id: 1, value: expectedValue }) + check() + parents.write(`delete`, { id: 1, value: expectedValue }) + expect.soft(live.toArray).toEqual([]) + } finally { + await live.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) + + it.each(renamedCells)( + `$form / $projection / $consumer / sibling=$withSibling materializes inputs before renaming them`, + async ({ form, consumer, projection, withSibling }) => { + const parents = createControlledCollection(`renamed-parent`, [ + { id: 1, group: 1, siblingGroup: 2 }, + ]) + const initial: Array = [ + { id: 10, parentGroup: 1, value: 3 }, + { id: 20, parentGroup: 2, value: 5 }, + ] + const children = createControlledCollection(`renamed-child`, initial) + const truth = new Map(initial.map((row) => [row.id, row])) + let group = 1 + let phase: Phase = `initial` + const calls: Array<{ + phase: Phase + stage: `projection` | `consumer` + primary: ChildView + sibling?: ChildView + }> = [] + const inspect = ( + stage: `projection` | `consumer`, + primary: unknown, + sibling: unknown, + ) => { + const view = readChildren(primary, form) + const second = withSibling ? readChildren(sibling, form) : undefined + calls.push({ + phase, + stage, + primary: { ...view, rows: publicRows(view.rows) }, + sibling: second && { ...second, rows: publicRows(second.rows) }, + }) + return ( + view.rows.reduce((sum, row) => sum + row.value, 0) + + (second?.rows.reduce((sum, row) => sum + row.value, 0) ?? 0) + ) + } + const buildQuery = () => + createLiveQueryCollection((q) => { + const source = q + .from({ parent: parents.collection }) + .select(({ parent }) => { + const primary = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + const sibling = q + .from({ other: children.collection }) + .where(({ other }) => + eq(other.parentGroup, parent.siblingGroup), + ) + return { + id: parent.id, + children: + form === `collection` + ? primary + : form === `array` + ? toArray(primary) + : materialize(primary), + ...(withSibling + ? { + sibling: + form === `collection` + ? sibling + : form === `array` + ? toArray(sibling) + : materialize(sibling), + } + : {}), + } + }) + const input = q.from({ row: source }) + const projected = + projection === `expression` + ? input.select(({ row }) => ({ + id: row.id, + renamed: { primary: row.children, sibling: row.sibling }, + total: 0, + })) + : input.fn.select(({ row }) => ({ + id: row.id, + renamed: { primary: row.children, sibling: row.sibling }, + total: inspect(`projection`, row.children, row.sibling), + })) + const outer = q.from({ result: projected }) + return consumer === `expression` + ? outer.select(({ result }) => ({ value: result })) + : outer.fn.select(({ result }) => ({ + value: { + id: result.id, + renamed: result.renamed, + total: inspect( + `consumer`, + result.renamed.primary, + result.renamed.sibling, + ), + }, + })) + }) + if ( + rejectsCollectionInput( + form, + projection === `functional`, + consumer === `functional`, + ) + ) { + try { + expect(buildQuery).toThrow(collectionInputError) + expect(calls).toEqual([]) + } finally { + await parents.collection.cleanup() + await children.collection.cleanup() + } + return + } + const live = buildQuery() + const check = () => { + const row = live.toArray[0]?.value + expect.soft(live.toArray, `${phase}: row count`).toHaveLength(1) + expect.soft(row?.id, `${phase}: public id`).toBe(1) + const expectedPrimary = publicRows( + [...truth.values()].filter((child) => child.parentGroup === group), + ) + const expectedSibling = withSibling + ? publicRows( + [...truth.values()].filter((child) => child.parentGroup === 2), + ) + : [] + for (const [value, expected] of [ + [row?.renamed.primary, expectedPrimary], + ...(withSibling + ? [[row?.renamed.sibling, expectedSibling] as const] + : []), + ] as const) { + const view = readChildren(value, form) + expect.soft(view.valid, `${phase}: renamed public form`).toBe(true) + expect + .soft(publicRows(view.rows), `${phase}: renamed public rows`) + .toEqual(expected) + if (form === `collection`) + expect + .soft(view.ready, `${phase}: renamed public readiness`) + .toBe(true) + } + if (row) + expect + .soft( + `children` in row || `row` in row, + `${phase}: input paths do not leak`, + ) + .toBe(false) + if ( + form !== `collection` || + phase === `initial` || + phase === `route-move` + ) { + if (projection === `functional` || consumer === `functional`) + expect + .soft(row?.total, `${phase}: derived total`) + .toBe( + [...expectedPrimary, ...expectedSibling].reduce( + (sum, child) => sum + child.value, + 0, + ), + ) + for (const stage of [ + ...(projection === `functional` ? [`projection` as const] : []), + ...(consumer === `functional` ? [`consumer` as const] : []), + ]) { + expect + .soft( + calls.filter( + (call) => call.phase === phase && call.stage === stage, + ).length, + `${phase}: ${stage} callback reach`, + ) + .toBeGreaterThan(0) + } + } + for (const call of calls.filter((item) => item.phase === phase)) { + for (const view of [ + call.primary, + ...(call.sibling ? [call.sibling] : []), + ]) { + expect.soft(view.valid, `${phase}: callback input form`).toBe(true) + if (form === `collection`) + expect + .soft(view.ready, `${phase}: callback input readiness`) + .toBe(true) + } + } + } + try { + await live.preload() + check() + phase = `child-update` + const changed = { id: 10, parentGroup: 1, value: 7 } + truth.set(10, changed) + children.write(`update`, changed) + check() + if (withSibling) { + phase = `sibling-update` + const sibling = { id: 20, parentGroup: 2, value: 11 } + truth.set(20, sibling) + children.write(`update`, sibling) + check() + } + phase = `route-move` + group = 2 + parents.write(`update`, { id: 1, group, siblingGroup: 2 }) + check() + } finally { + await live.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }, + ) +}) + +describe(`functional include projection boundary grammar`, () => { + it(`preserves a scalar result when a functional projection drops its include`, async () => { + const parents = createControlledCollection(`scalar-projection-parent`, [ + { id: 1 }, + ]) + const children = createControlledCollection(`scalar-projection-child`, [ + { id: 10, parentId: 1 }, + ]) + const live = createLiveQueryCollection((q) => { + const included = q + .from({ parent: parents.collection }) + .select(({ parent }) => ({ + id: parent.id, + children: toArray( + q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentId, parent.id)), + ), + })) + const scalar = q.from({ row: included }).fn.select(({ row }) => row.id) + return q + .from({ result: scalar }) + .select(({ result }) => ({ value: result })) + }) + try { + await live.preload() + expect(live.toArray.map((row) => row.value)).toEqual([1]) + } finally { + await live.cleanup() + await parents.collection.cleanup() + await children.collection.cleanup() + } + }) + + it(`preserves opaque-root fields without include materialization`, async () => { + const parents = createControlledCollection(`opaque-root-control`, [ + { id: 1 }, + ]) + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents.collection }) + .fn.select( + ({ parent }) => new Projection(parent.id, `plain`, undefined, 0), + ), + ) + try { + await live.preload() + const row = live.toArray[0] + expect(row?.id).toBe(1) + expect(row?.kind).toBe(`plain`) + expect(row?.total).toBe(0) + // Collection root records already flatten prototypes without includes. + // This matrix checks their fields, not a new prototype-preservation API. + } finally { + await live.cleanup() + await parents.collection.cleanup() + } + }) + + it(`covers every declared boundary product without duplicate cells`, () => { + expect(cells).toHaveLength(54) + expect(new Set(cells.map((cell) => JSON.stringify(cell))).size).toBe(54) + }) + + it.each(cells)( + `$boundary / $form / $output / $initial`, + async ({ boundary, form, output, initial }) => { + const parents = createControlledCollection(`projection-parents`, [ + { id: 1, group: 1 }, + ]) + const absent = createControlledCollection(`projection-absent`, [ + { id: 2 }, + ]) + const initialChildren: Array = [ + ...(initial === `populated` + ? [{ id: 10, parentGroup: 1, value: 3 }] + : []), + { id: 20, parentGroup: 2, value: 5 }, + ] + const children = createControlledCollection( + `projection-children`, + initialChildren, + ) + const truth = new Map(initialChildren.map((row) => [row.id, row])) + let group = 1 + let phase: Phase = `initial` + const calls: Array<{ + phase: Phase + kind: string + child: unknown + view: ChildView + }> = [] + const project = (row: Input) => { + const child = row.children + const view = readChildren(child, form) + // Capture readiness and contents NOW, not through a reference read after preload. + calls.push({ + phase, + kind: row.kind, + child, + view: { ...view, rows: publicRows(view.rows) }, + }) + const total = view.rows.reduce((sum, item) => sum + item.value, 0) + return output === `opaque-root` + ? new Projection(row.id, row.kind, child, total) + : { id: row.id, kind: row.kind, children: child, total } + } + const buildQuery = () => + createLiveQueryCollection((q) => { + const included = q + .from({ parent: parents.collection }) + .select(({ parent }) => { + const childRows = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.parentGroup, parent.group)) + .orderBy(({ child }) => child.id) + .select(({ child }) => ({ + id: child.id, + parentGroup: child.parentGroup, + value: child.value, + })) + return { + id: parent.id, + kind: `included`, + total: 0, + children: + form === `collection` + ? childRows + : form === `array` + ? toArray(childRows) + : materialize(childRows), + } + }) + if (boundary === `union`) { + const withoutInclude = q + .from({ other: absent.collection }) + .select(({ other }) => ({ + id: other.id, + kind: `absent`, + total: 0, + })) + const union = q.unionAll(included, withoutInclude) + return output === `expression` ? union : union.fn.select(project) + } + if (boundary === `recursive-query-ref`) { + const intermediate = q + .from({ inner: included }) + .select(({ inner }) => inner) + const outer = q.from({ row: intermediate }) + return output === `expression` + ? outer.select(({ row }) => row) + : outer.fn.select(({ row }) => project(row)) + } + const outer = q.from({ row: included }) + return output === `expression` + ? outer.select(({ row }) => row) + : outer.fn.select(({ row }) => project(row)) + }) + if (rejectsCollectionInput(form, output !== `expression`)) { + try { + expect(buildQuery).toThrow(collectionInputError) + expect(calls).toEqual([]) + } finally { + await parents.collection.cleanup() + await children.collection.cleanup() + await absent.collection.cleanup() + } + return + } + const live = buildQuery() + let facade: unknown + const check = () => { + const row: (Input & { total: number }) | undefined = live.toArray.find( + (item) => item.kind === `included`, + ) + expect.soft(row, `${phase}: included public row`).toBeDefined() + if (!row) return + const expected = publicRows( + [...truth.values()].filter((item) => item.parentGroup === group), + ) + const view = readChildren(row.children, form) + expect.soft(view.valid, `${phase}: public include form`).toBe(true) + expect + .soft(publicRows(view.rows), `${phase}: public children`) + .toEqual(expected) + if (form === `collection`) { + expect.soft(view.ready, `${phase}: public facade ready`).toBe(true) + if (phase === `initial`) facade = row.children + else if (phase === `child-update`) + expect.soft(row.children, `child-only facade identity`).toBe(facade) + else + expect + .soft(row.children, `route move replaces facade`) + .not.toBe(facade) + } + // A Collection is a live handle, not a dependency-tracked scalar read. + // Assert derived scalars when the parent projection runs, not on child-only + // changes to a retained facade. Inline values do drive parent recomputation. + if ( + output !== `expression` && + (form !== `collection` || phase !== `child-update`) + ) { + expect + .soft(row.total, `${phase}: derived scalar`) + .toBe(expected.reduce((sum, item) => sum + item.value, 0)) + } + if (boundary === `union`) { + const other: Input | undefined = live.toArray.find( + (item) => item.kind === `absent`, + ) + expect.soft(other, `${phase}: absent branch survives`).toBeDefined() + expect + .soft(other?.children, `${phase}: absent branch value`) + .toBeUndefined() + } + const current = calls.filter( + (call) => call.phase === phase && call.kind === `included`, + ) + if ( + output !== `expression` && + (form !== `collection` || phase !== `child-update`) + ) + expect + .soft(current.length, `${phase}: callback reach`) + .toBeGreaterThan(0) + for (const call of current) { + expect + .soft(call.view.valid, `${phase}: callback include form`) + .toBe(true) + if (form === `collection`) + expect + .soft(call.view.ready, `${phase}: callback facade ready`) + .toBe(true) + } + for (const call of calls.filter( + (item) => item.phase === phase && item.kind === `absent`, + )) { + expect + .soft(call.child, `${phase}: valid callback absence`) + .toBeUndefined() + } + } + try { + await live.preload() + check() + phase = `child-update` + const changed = { id: 10, parentGroup: 1, value: 7 } + truth.set(10, changed) + children.write(initial === `empty` ? `insert` : `update`, changed) + check() + phase = `route-move` + group = 2 + parents.write(`update`, { id: 1, group }) + check() + } finally { + await live.cleanup() + await Promise.all([ + parents.collection.cleanup(), + children.collection.cleanup(), + absent.collection.cleanup(), + ]) + } + }, + ) +}) diff --git a/packages/db/tests/query/includes-optimistic-oracle.property.test.ts b/packages/db/tests/query/includes-optimistic-oracle.property.test.ts index 0e125cdfc8..23b4bf762d 100644 --- a/packages/db/tests/query/includes-optimistic-oracle.property.test.ts +++ b/packages/db/tests/query/includes-optimistic-oracle.property.test.ts @@ -499,7 +499,10 @@ describe(`optimistic relationship-transition oracle`, () => { }, ) - fcTest.prop([routeValuesArbitrary], oraclePropertyOptions(12))( + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.rekey-detach`), + )( `an optimistic rekey detaches its old descendants immediately`, async (routes) => { await expectHistoryMatches(routes, [ @@ -514,7 +517,10 @@ describe(`optimistic relationship-transition oracle`, () => { }, ) - fcTest.prop([routeValuesArbitrary], oraclePropertyOptions(12))( + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.rekey-rollback`), + )( `restores the authoritative relationship after an optimistic rekey rolls back`, async (routes) => { await expectHistoryMatches(routes, [ @@ -528,7 +534,10 @@ describe(`optimistic relationship-transition oracle`, () => { }, ) - fcTest.prop([routeValuesArbitrary], oraclePropertyOptions(12))( + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.descendant-rollback`), + )( `rolls back a descendant update made while its ancestor is reparented`, async (routes) => { await expectHistoryMatches(routes, [ @@ -552,7 +561,10 @@ describe(`optimistic relationship-transition oracle`, () => { }, ) - fcTest.prop([routeValuesArbitrary], oraclePropertyOptions(12))( + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.ancestor-rollback`), + )( `rolls back a reparented ancestor while its descendant update remains pending`, async (routes) => { await expectHistoryMatches(routes, [ @@ -576,7 +588,10 @@ describe(`optimistic relationship-transition oracle`, () => { }, ) - fcTest.prop([routeValuesArbitrary], oraclePropertyOptions(12))( + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.confirm-same-route`), + )( `settles a confirmed optimistic reparent on the same authoritative route`, async (routes) => { await expectHistoryMatches(routes, [ @@ -614,7 +629,10 @@ describe(`optimistic relationship-transition oracle`, () => { }, ) - fcTest.prop([routeValuesArbitrary], oraclePropertyOptions(12))( + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.confirm-different-route`), + )( `settles a confirmed optimistic reparent on a different authoritative route`, async (routes) => { await expectHistoryMatches(routes, [ @@ -654,100 +672,100 @@ describe(`optimistic relationship-transition oracle`, () => { }, ) - fcTest.prop([routeValuesArbitrary], oraclePropertyOptions(12))( - `restores a rekey after a sibling enters its old route`, - async (routes) => { - await expectHistoryMatches(routes, [ - { - type: `optimisticRollback`, + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.sibling-route-rollback`), + )(`restores a rekey after a sibling enters its old route`, async (routes) => { + await expectHistoryMatches(routes, [ + { + type: `optimisticRollback`, + level: 1, + id: 11, + patch: { group: routes.optimistic }, + beforeRollback: { level: 1, - id: 11, - patch: { group: routes.optimistic }, - beforeRollback: { - level: 1, - changes: [ - { - type: `insert`, - value: { - id: 12, - parentGroup: routes.rootA, - group: routes.original, - value: 120, - position: 1, - }, - }, - ], - }, - }, - { - type: `sync`, - level: 2, changes: [ { - type: `update`, + type: `insert`, value: { - id: 21, - parentGroup: routes.original, - group: routes.original + 1000, - value: 211, - position: 0, + id: 12, + parentGroup: routes.rootA, + group: routes.original, + value: 120, + position: 1, }, }, ], }, - ]) - }, - ) + }, + { + type: `sync`, + level: 2, + changes: [ + { + type: `update`, + value: { + id: 21, + parentGroup: routes.original, + group: routes.original + 1000, + value: 211, + position: 0, + }, + }, + ], + }, + ]) + }) - fcTest.prop([routeValuesArbitrary], oraclePropertyOptions(12))( - `supports repeated rollback and confirmation histories`, - async (routes) => { - await expectHistoryMatches(routes, [ - { - type: `optimistic`, - handle: `first`, - level: 1, - id: 11, - patch: { parentGroup: routes.rootB }, - }, - { type: `rollback`, handle: `first` }, - { - type: `optimistic`, - handle: `second`, - level: 1, - id: 11, - patch: { parentGroup: routes.rootB }, - }, - { - type: `confirm`, - handle: `second`, - authoritative: firstChild(routes, { - parentGroup: routes.rootB, - }), - }, - { - type: `optimisticRollback`, - level: 1, - id: 11, - patch: { group: routes.optimistic }, - }, - { - type: `sync`, - level: 2, - changes: [ - { - type: `update`, - value: { - id: 21, - parentGroup: routes.original, - group: routes.original + 1000, - value: 212, - position: 0, - }, + fcTest.prop( + [routeValuesArbitrary], + oraclePropertyOptions(12, `includes-optimistic.repeated-history`), + )(`supports repeated rollback and confirmation histories`, async (routes) => { + await expectHistoryMatches(routes, [ + { + type: `optimistic`, + handle: `first`, + level: 1, + id: 11, + patch: { parentGroup: routes.rootB }, + }, + { type: `rollback`, handle: `first` }, + { + type: `optimistic`, + handle: `second`, + level: 1, + id: 11, + patch: { parentGroup: routes.rootB }, + }, + { + type: `confirm`, + handle: `second`, + authoritative: firstChild(routes, { + parentGroup: routes.rootB, + }), + }, + { + type: `optimisticRollback`, + level: 1, + id: 11, + patch: { group: routes.optimistic }, + }, + { + type: `sync`, + level: 2, + changes: [ + { + type: `update`, + value: { + id: 21, + parentGroup: routes.original, + group: routes.original + 1000, + value: 212, + position: 0, }, - ], - }, - ]) - }, - ) + }, + ], + }, + ]) + }) }) diff --git a/packages/db/tests/query/includes-oracle.property.test.ts b/packages/db/tests/query/includes-oracle.property.test.ts index 8fb7625876..a1e4ec2018 100644 --- a/packages/db/tests/query/includes-oracle.property.test.ts +++ b/packages/db/tests/query/includes-oracle.property.test.ts @@ -281,7 +281,7 @@ if (process.env.TANSTACK_DB_ORACLE_STATISTICS === `1`) { fc.statistics( scenarioArbitrary, classifyScenarioCoverage, - oraclePropertyOptions(1_000), + oraclePropertyOptions(1_000, `includes.scenario-statistics`), ) } @@ -4295,7 +4295,10 @@ describe(`includes recompute oracle`, () => { }) }) - fcTest.prop([scenarioArbitrary], oraclePropertyOptions(40))( + fcTest.prop( + [scenarioArbitrary], + oraclePropertyOptions(40, `includes.incremental-history`), + )( `matches naive recomputation after every incremental change`, expectScenarioMatches, ) @@ -4306,7 +4309,7 @@ describe(`includes recompute oracle`, () => { ({ sharedIntermediate }) => !sharedIntermediate, ), ], - oraclePropertyOptions(30), + oraclePropertyOptions(30, `includes.nested-scalar-materialization`), )( `matches recomputation for nested scalar materialization`, expectMaterializeScenarioMatches, @@ -4334,7 +4337,7 @@ describe(`includes recompute oracle`, () => { { selector: (row) => row.id, maxLength: 7 }, ), ], - oraclePropertyOptions(25), + oraclePropertyOptions(25, `includes.alpha-renaming`), )( `is unchanged by alpha-renaming, sibling declaration order, or an unrelated sibling`, async (rootRows, childRows) => { @@ -4446,7 +4449,7 @@ describe(`includes recompute oracle`, () => { fcTest.prop( [fc.integer({ min: -5, max: 5 }).filter((value) => value !== 0)], - oraclePropertyOptions(15), + oraclePropertyOptions(15, `includes.optimistic-convergence`), )( `optimistic updates converge to confirmed-only state`, async (confirmedValue) => { diff --git a/packages/db/tests/query/includes-publication-oracle.test.ts b/packages/db/tests/query/includes-publication-oracle.test.ts index e78656d70e..3450c8e044 100644 --- a/packages/db/tests/query/includes-publication-oracle.test.ts +++ b/packages/db/tests/query/includes-publication-oracle.test.ts @@ -413,8 +413,14 @@ describe(`layered-query publication oracle`, () => { for (const q1Shape of q1Shapes) { for (const q2Shape of q2Shapes) { - fcTest.prop([changedValueArbitrary], oraclePropertyOptions(12))( - `publishes #1713 updates through a ${q1Shape} Q1 and ${q2Shape} Q2`, + fcTest.prop( + [changedValueArbitrary], + oraclePropertyOptions( + 12, + `includes-publication.parent-scalar.${q1Shape}.${q2Shape}`, + ), + )( + `publishes parent scalar updates through a ${q1Shape} Q1 and ${q2Shape} Q2`, async (value) => { await expectPublicationMatches( { type: `parentScalar`, value }, @@ -427,7 +433,10 @@ describe(`layered-query publication oracle`, () => { fcTest.prop( [changedValueArbitrary, changedChildValueArbitrary], - oraclePropertyOptions(12), + oraclePropertyOptions( + 12, + `includes-publication.parent-then-child.${q1Shape}.${q2Shape}`, + ), )( `recovers a ${q1Shape} Q1 and ${q2Shape} Q2 after a child update`, async (parentValue, childValue) => { @@ -440,7 +449,13 @@ describe(`layered-query publication oracle`, () => { }, ) - fcTest.prop([changedValueArbitrary], oraclePropertyOptions(8))( + fcTest.prop( + [changedValueArbitrary], + oraclePropertyOptions( + 8, + `includes-publication.optimistic-before-confirm.${q1Shape}.${q2Shape}`, + ), + )( `publishes optimistic state before confirmation through a ${q1Shape} Q1 and ${q2Shape} Q2`, async (value) => { await expectPublicationMatches( @@ -452,7 +467,13 @@ describe(`layered-query publication oracle`, () => { }, ) - fcTest.prop([changedValueArbitrary], oraclePropertyOptions(8))( + fcTest.prop( + [changedValueArbitrary], + oraclePropertyOptions( + 8, + `includes-publication.optimistic-after-confirm.${q1Shape}.${q2Shape}`, + ), + )( `publishes state after optimistic confirmation through a ${q1Shape} Q1 and ${q2Shape} Q2`, async (value) => { await expectPublicationMatches( @@ -466,19 +487,22 @@ describe(`layered-query publication oracle`, () => { } } - fcTest.prop([changedChildValueArbitrary], oraclePropertyOptions(100))( + fcTest.prop( + [changedChildValueArbitrary], + oraclePropertyOptions(100, `includes-publication.child-scalar`), + )( `publishes child-only scalar updates through both layers`, async (value) => { await expectPublicationMatches({ type: `childScalar`, value }) }, ) - fcTest.prop([fc.constantFrom(20, 30)], oraclePropertyOptions(100))( - `compares route transitions at both query layers`, - async (group) => { - await expectPublicationMatches({ type: `parentRoute`, group }) - }, - ) + fcTest.prop( + [fc.constantFrom(20, 30)], + oraclePropertyOptions(100, `includes-publication.parent-route`), + )(`compares route transitions at both query layers`, async (group) => { + await expectPublicationMatches({ type: `parentRoute`, group }) + }) fcTest.prop( [ @@ -487,18 +511,21 @@ describe(`layered-query publication oracle`, () => { value: changedValueArbitrary, }), ], - oraclePropertyOptions(100), + oraclePropertyOptions( + 100, + `includes-publication.atomic-parent-replacement`, + ), )(`compares atomic parent replacements at both query layers`, async (row) => { await expectPublicationMatches({ type: `atomicReplace`, ...row }) }) - fcTest.prop([changedValueArbitrary], oraclePropertyOptions(100))( - `publishes restored state after optimistic rollback`, - async (value) => { - await expectPublicationMatches({ - type: `optimisticRollback`, - value, - }) - }, - ) + fcTest.prop( + [changedValueArbitrary], + oraclePropertyOptions(100, `includes-publication.optimistic-rollback`), + )(`publishes restored state after optimistic rollback`, async (value) => { + await expectPublicationMatches({ + type: `optimisticRollback`, + value, + }) + }) }) diff --git a/packages/db/tests/query/includes-temporal-oracle.test.ts b/packages/db/tests/query/includes-temporal-oracle.test.ts index 649dccd49a..e54c146196 100644 --- a/packages/db/tests/query/includes-temporal-oracle.test.ts +++ b/packages/db/tests/query/includes-temporal-oracle.test.ts @@ -9,6 +9,7 @@ import { DeduplicatedLoadSubset } from '../../src/query/subset-dedupe.js' import { createLiveQueryCollection, eq, + materialize, toArray, } from '../../src/query/index.js' import { runTrace } from '../trace-runner.js' @@ -164,6 +165,114 @@ function createColdComments(): { return { collection, loads } } +it.each( + ([`array`, `materialized`] as const).flatMap((form) => + ([`expression`, `functional`] as const).map((projection) => ({ + form, + projection, + })), + ), +)( + `$form / $projection preserves child demand and applied settlement across projection`, + async ({ form, projection }) => { + const posts = createColdPosts([{ id: 1, authorId: `one`, title: `post` }]) + const started = createDeferred() + const release = createDeferred() + const loads: Array = [] + const comments = createCollection({ + id: nextCollectionId(`projection-pending-comments`), + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => ({ + loadSubset: (options) => { + loads.push(options) + const keys = correlationKeys([options], `postId`) + started.resolve() + return release.promise.then(async () => { + if (options.signal?.aborted) return + begin() + if (keys.includes(1)) + write({ + type: `insert`, + value: { id: 100, postId: 1, body: `one` }, + }) + await commit() + markReady() + }) + }, + }), + }, + }) + const live = createLiveQueryCollection((q) => { + const included = q.from({ post: posts.collection }).select(({ post }) => { + const childRows = q + .from({ comment: comments }) + .where(({ comment }) => eq(comment.postId, post.id)) + return { + id: post.id, + comments: + form === `array` ? toArray(childRows) : materialize(childRows), + count: 0, + } + }) + const outer = q.from({ row: included }) + return projection === `expression` + ? outer.select(({ row }) => row) + : outer.fn.select(({ row }) => { + expect + .soft( + Array.isArray(row.comments), + `callback receives an inline value`, + ) + .toBe(true) + return { + id: row.id, + comments: row.comments, + count: Array.isArray(row.comments) ? row.comments.length : -1, + } + }) + }) + let settled = false + const preload = live.preload() + const observed = preload.then( + () => { + settled = true + }, + () => { + settled = true + }, + ) + try { + await Promise.race([started.promise, preload]) + expect(loads).toHaveLength(1) + expect(correlationKeys(loads, `postId`)).toEqual([1]) + expect(settled).toBe(false) + release.resolve() + await preload + expect(live.toArray).toHaveLength(1) + // Observe the runtime boundary: a broken projection can omit this value. + const publishedComments = live.toArray[0]?.comments as unknown as + | Array + | undefined + expect( + publishedComments?.map(({ id, postId, body }) => ({ + id, + postId, + body, + })), + ).toEqual([{ id: 100, postId: 1, body: `one` }]) + if (projection === `functional`) expect(live.toArray[0]?.count).toBe(1) + } finally { + release.resolve() + await live.cleanup() + await observed + await posts.collection.cleanup() + await comments.cleanup() + } + }, +) + type ReadinessObservation = { ready: boolean preloadSettled: boolean @@ -839,6 +948,201 @@ async function expectFailedDemandRetriesSameCoverage(): Promise { } } +async function expectDemandReactivationRetriesAfterReleaseFailure( + keys: ReadonlyArray, +): Promise { + let loadCount = 0 + let allowUnload = false + const releaseError = new Error(`child release failed`) + const comments = createCollection({ + id: nextCollectionId(`temporal-release-retry-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: ({ markReady }) => ({ + loadSubset: () => { + loadCount += 1 + markReady() + return true + }, + unloadSubset: () => { + if (!allowUnload) throw releaseError + }, + }), + }, + }) + comments.createIndex((comment) => comment.postId) + const subscription = comments.subscribeChanges(() => {}, { + includeInitialState: false, + }) + const controller = new SubsetDemandController() + const plan: LazyDemandPlan = { + id: `release-failure-retry`, + path: [`postId`], + collectionId: comments.id, + initialKeys: new Set(), + } + + try { + expect( + controller.setDemand(subscription, plan, new Set(keys)), + ).toMatchObject({ changed: true, empty: false }) + expect(loadCount).toBe(1) + + const retired = controller.setDemand(subscription, plan, new Set()) + expect(retired).toMatchObject({ changed: true, empty: true }) + + const reactivated = controller.setDemand(subscription, plan, new Set(keys)) + expect(reactivated).toMatchObject({ changed: true, empty: false }) + expect(loadCount).toBe(2) + } finally { + allowUnload = true + controller.clear() + subscription.unsubscribe() + await comments.cleanup() + } +} + +async function expectRetiredDemandStaysNonfatalAfterReleaseFailure(): Promise { + const post = { id: 1, authorId: `selected`, title: `one` } + const posts = createMutablePosts([post]) + let loadCount = 0 + let allowUnload = false + const releaseError = new Error(`child release failed`) + const comments = createCollection({ + id: nextCollectionId(`temporal-retired-release-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: ({ markReady }) => ({ + loadSubset: () => { + loadCount += 1 + markReady() + return true + }, + unloadSubset: () => { + if (!allowUnload) throw releaseError + }, + }), + }, + }) + const live = createPostsWithCommentsLive(posts.collection, comments) + const consoleError = vi.spyOn(console, `error`).mockImplementation(() => {}) + + try { + await live.preload() + expect(loadCount).toBe(1) + expect(live.status).toBe(`ready`) + + posts.write(`delete`, post) + await flushPromises() + expect(live.size).toBe(0) + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBe(releaseError) + + posts.write(`insert`, post) + await flushPromises() + expect(loadCount).toBe(2) + expect(live.status).toBe(`ready`) + } finally { + allowUnload = true + await live.cleanup() + await Promise.all([posts.collection.cleanup(), comments.cleanup()]) + consoleError.mockRestore() + } +} + +async function expectFailedReplayStopsGatingAfterLastDemandRetires(): Promise { + const post = { id: 1, authorId: `selected`, title: `one` } + const posts = createMutablePosts([post]) + const replay = createDeferred() + let begin!: () => void + let write!: (message: { type: `insert`; value: Comment }) => void + let commit!: () => true | Promise + let truncate!: () => void + let loadCount = 0 + const comments = createCollection({ + id: nextCollectionId(`temporal-retired-replay-comments`), + getKey: (comment) => comment.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + loadCount += 1 + if (loadCount === 1) { + begin() + write({ + type: `insert`, + value: { id: 10, postId: post.id, body: `old` }, + }) + commit() + return true + } + begin() + write({ + type: `insert`, + value: { id: 20, postId: post.id, body: `private replacement` }, + }) + commit() + return replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createPostsWithCommentsLive(posts.collection, comments) + const publications: Array> = [] + const subscription = live.subscribeChanges( + () => publications.push(live.toArray.map(({ id }) => id)), + { includeInitialState: false }, + ) + const consoleError = vi.spyOn(console, `error`).mockImplementation(() => {}) + + try { + await live.preload() + expect(live.get(post.id)?.comments.map(({ id }) => id)).toEqual([10]) + publications.length = 0 + + begin() + truncate() + commit() + await flushPromises() + expect(loadCount).toBe(2) + + replay.reject(new Error(`replacement failed`)) + await flushPromises() + expect(live.get(post.id)?.comments.map(({ id }) => id)).toEqual([10]) + expect(publications).toEqual([]) + + posts.write(`delete`, post) + await flushPromises() + + // Once the parent retires the last child demand, its failed replay can no + // longer gate unrelated parent changes in the shared graph. + expect(live.size).toBe(0) + expect(publications).toEqual([[]]) + } finally { + replay.resolve() + subscription.unsubscribe() + await live.cleanup() + await Promise.all([posts.collection.cleanup(), comments.cleanup()]) + consoleError.mockRestore() + } +} + async function expectSynchronousEmptyDemandIsReady(): Promise { const posts = createMutablePosts([ { id: 1, authorId: `selected`, title: `one` }, @@ -1196,7 +1500,10 @@ describe(`includes temporal oracle`, () => { expectObsoleteDemandCannotPublishAfterReactivation, ) - fcTest.prop([fc.scheduler()], oraclePropertyOptions(20))( + fcTest.prop( + [fc.scheduler()], + oraclePropertyOptions(20, `includes-temporal.demand-scheduling`), + )( `obsolete and current demand completions are generation-safe in either order`, expectScheduledDemandCompletionsStayGenerationSafe, ) @@ -1218,6 +1525,32 @@ describe(`includes temporal oracle`, () => { expectFailedDemandRetriesSameCoverage, ) + it(`reactivated demand retries after its prior release fails`, () => + expectDemandReactivationRetriesAfterReleaseFailure([1])) + + fcTest.prop( + [ + fc.uniqueArray(fc.integer({ min: -3, max: 3 }), { + minLength: 1, + maxLength: 5, + }), + ], + oraclePropertyOptions(20, `includes-temporal.release-reentry`), + )( + `failed release never suppresses a later demand incarnation`, + expectDemandReactivationRetriesAfterReleaseFailure, + ) + + it( + `failed release retires an empty live-query demand without poisoning reentry`, + expectRetiredDemandStaysNonfatalAfterReleaseFailure, + ) + + it( + `failed replay stops gating after its last demand retires`, + expectFailedReplayStopsGatingAfterLastDemandRetires, + ) + it( `a synchronous empty demand can establish ready coverage`, expectSynchronousEmptyDemandIsReady, diff --git a/packages/db/tests/query/includes.test.ts b/packages/db/tests/query/includes.test.ts index b58bcf71a1..9fa07d38a0 100644 --- a/packages/db/tests/query/includes.test.ts +++ b/packages/db/tests/query/includes.test.ts @@ -11,12 +11,12 @@ import { toArray, } from '../../src/query/index.js' import { createCollection } from '../../src/collection/index.js' -import { CleanupQueue } from '../../src/collection/cleanup-queue.js' import { BTreeIndex } from '../../src/indexes/btree-index.js' import { localOnlyCollectionOptions } from '../../src/local-only.js' import { flushPromises, mockSyncCollectionOptions, + resetCleanupQueue, stripVirtualProps, } from '../utils.js' import type { SyncConfig } from '../../src/types.js' @@ -1607,7 +1607,10 @@ describe(`includes subqueries`, () => { defaultIndexType: BTreeIndex, sync: { sync: ({ begin, write, commit, markReady }) => ({ - loadSubset: () => { + loadSubset: (options) => { + // The current tie class is already present. A boundary probe + // must not consume the next page of source rows. + if (options.where) return true loadCount += 1 const row = sourceRows[nextRow++] if (row) { @@ -1662,7 +1665,8 @@ describe(`includes subqueries`, () => { try { await collection.preload() - expect(loadCount).toBe(3) + // Bounded tie probes carry `where` and are not counted as page loads. + expect(loadCount).toBe(sourceRows.length) for (const observation of observations) { for (const parent of observation) { expect(parent.childIds).toEqual([parent.id * 10]) @@ -5040,12 +5044,12 @@ describe(`includes subqueries`, () => { describe(`child collection garbage collection`, () => { beforeEach(() => { vi.useFakeTimers() - CleanupQueue.resetInstance() + resetCleanupQueue() }) afterEach(() => { vi.useRealTimers() - CleanupQueue.resetInstance() + resetCleanupQueue() }) it(`child collections should not be garbage collected when external subscribers unmount`, async () => { @@ -5745,182 +5749,209 @@ describe(`includes subqueries`, () => { expect(data().runs[0].texts[0].text).toBe(`Hello world`) }) - it(`deep buffer change for one parent does not emit spurious update for sibling parent`, async () => { - const TIMELINE_KEY = `tl-spurious` + it.each([0, 1])( + `deep buffer change for run %i leaves its sibling's values and notifications unchanged`, + async (changedIndex) => { + const siblingIndex = 1 - changedIndex + const TIMELINE_KEY = `tl-spurious` - type Seed = { key: string } - type Run = { key: string; _seq: number; status: string } - type Text = { - key: string - run_id: string - _seq: number - status: string - } - type TextDelta = { - key: string - text_id: string - run_id: string - _seq: number - delta: string - } + type Seed = { key: string } + type Run = { key: string; _seq: number; status: string } + type Text = { + key: string + run_id: string + _seq: number + status: string + } + type TextDelta = { + key: string + text_id: string + run_id: string + _seq: number + delta: string + } - const seed = createCollection( - localOnlyCollectionOptions({ - id: `spurious-seed`, - getKey: (s) => s.key, - initialData: [{ key: TIMELINE_KEY }], - }), - ) + const seed = createCollection( + localOnlyCollectionOptions({ + id: `spurious-seed`, + getKey: (s) => s.key, + initialData: [{ key: TIMELINE_KEY }], + }), + ) - const runs = createCollection( - localOnlyCollectionOptions({ - id: `spurious-runs`, - getKey: (r) => r.key, - initialData: [], - }), - ) + const runs = createCollection( + localOnlyCollectionOptions({ + id: `spurious-runs`, + getKey: (r) => r.key, + initialData: [], + }), + ) - const texts = createCollection( - localOnlyCollectionOptions({ - id: `spurious-texts`, - getKey: (t) => t.key, - initialData: [], - }), - ) + const texts = createCollection( + localOnlyCollectionOptions({ + id: `spurious-texts`, + getKey: (t) => t.key, + initialData: [], + }), + ) - const textDeltas = createCollection( - localOnlyCollectionOptions({ - id: `spurious-deltas`, - getKey: (d) => d.key, - initialData: [], - }), - ) + const textDeltas = createCollection( + localOnlyCollectionOptions({ + id: `spurious-deltas`, + getKey: (d) => d.key, + initialData: [], + }), + ) - const runsLive = createLiveQueryCollection({ - id: `spurious-runs-live`, - query: (q) => - q.from({ run: runs }).select(({ run }) => ({ - timelineKey: TIMELINE_KEY, - key: run.key, - order: coalesce(run._seq, -1), - status: run.status, - })), - }) + const runsLive = createLiveQueryCollection({ + id: `spurious-runs-live`, + query: (q) => + q.from({ run: runs }).select(({ run }) => ({ + timelineKey: TIMELINE_KEY, + key: run.key, + order: coalesce(run._seq, -1), + status: run.status, + })), + }) - const textsLive = createLiveQueryCollection({ - id: `spurious-texts-live`, - query: (q) => - q.from({ text: texts }).select(({ text }) => ({ - timelineKey: TIMELINE_KEY, - key: text.key, - run_id: text.run_id, - order: coalesce(text._seq, -1), - status: text.status, - })), - }) + const textsLive = createLiveQueryCollection({ + id: `spurious-texts-live`, + query: (q) => + q.from({ text: texts }).select(({ text }) => ({ + timelineKey: TIMELINE_KEY, + key: text.key, + run_id: text.run_id, + order: coalesce(text._seq, -1), + status: text.status, + })), + }) - const textDeltasLive = createLiveQueryCollection({ - id: `spurious-deltas-live`, - query: (q) => - q.from({ delta: textDeltas }).select(({ delta }) => ({ - timelineKey: TIMELINE_KEY, - key: delta.key, - text_id: delta.text_id, - run_id: delta.run_id, - order: coalesce(delta._seq, -1), - delta: delta.delta, - })), - }) + const textDeltasLive = createLiveQueryCollection({ + id: `spurious-deltas-live`, + query: (q) => + q.from({ delta: textDeltas }).select(({ delta }) => ({ + timelineKey: TIMELINE_KEY, + key: delta.key, + text_id: delta.text_id, + run_id: delta.run_id, + order: coalesce(delta._seq, -1), + delta: delta.delta, + })), + }) - const timeline = createLiveQueryCollection({ - id: `spurious-timeline`, - query: (q) => - q.from({ s: seed }).select(({ s }) => ({ - key: s.key, - runs: toArray( - q - .from({ run: runsLive }) - .where(({ run }) => eq(run.timelineKey, s.key)) - .orderBy(({ run }) => run.order) - .select(({ run }) => ({ - key: run.key, - order: run.order, - status: run.status, - texts: toArray( - q - .from({ text: textsLive }) - .where(({ text }) => eq(text.run_id, run.key)) - .orderBy(({ text }) => text.order) - .select(({ text }) => ({ - key: text.key, - run_id: text.run_id, - order: text.order, - status: text.status, - text: concat( - toArray( - q - .from({ delta: textDeltasLive }) - .where(({ delta }) => eq(delta.text_id, text.key)) - .orderBy(({ delta }) => delta.order) - .select(({ delta }) => delta.delta), + const timeline = createLiveQueryCollection({ + id: `spurious-timeline`, + query: (q) => + q.from({ s: seed }).select(({ s }) => ({ + key: s.key, + runs: toArray( + q + .from({ run: runsLive }) + .where(({ run }) => eq(run.timelineKey, s.key)) + .orderBy(({ run }) => run.order) + .select(({ run }) => ({ + key: run.key, + order: run.order, + status: run.status, + texts: toArray( + q + .from({ text: textsLive }) + .where(({ text }) => eq(text.run_id, run.key)) + .orderBy(({ text }) => text.order) + .select(({ text }) => ({ + key: text.key, + run_id: text.run_id, + order: text.order, + status: text.status, + text: concat( + toArray( + q + .from({ delta: textDeltasLive }) + .where(({ delta }) => + eq(delta.text_id, text.key), + ) + .orderBy(({ delta }) => delta.order) + .select(({ delta }) => delta.delta), + ), ), - ), - })), - ), - })), - ), - })), - }) + })), + ), + })), + ), + })), + }) - await timeline.preload() + await timeline.preload() - const data = () => timeline.get(TIMELINE_KEY) as any + const data = () => timeline.get(TIMELINE_KEY) as any - runs.insert({ key: `run-1`, _seq: 1, status: `started` }) - runs.insert({ key: `run-2`, _seq: 2, status: `started` }) - texts.insert({ - key: `text-1`, - run_id: `run-1`, - _seq: 3, - status: `streaming`, - }) - texts.insert({ - key: `text-2`, - run_id: `run-2`, - _seq: 4, - status: `streaming`, - }) - await new Promise((r) => setTimeout(r, 100)) + runs.insert({ key: `run-1`, _seq: 1, status: `started` }) + runs.insert({ key: `run-2`, _seq: 2, status: `started` }) + texts.insert({ + key: `text-1`, + run_id: `run-1`, + _seq: 3, + status: `streaming`, + }) + texts.insert({ + key: `text-2`, + run_id: `run-2`, + _seq: 4, + status: `streaming`, + }) + await new Promise((r) => setTimeout(r, 100)) + + expect(data().runs).toHaveLength(2) + expect(data().runs[0].texts[0].text).toBe(``) + expect(data().runs[1].texts[0].text).toBe(``) + + const timelineRowBefore = data() + const siblingTextsBefore = timelineRowBefore.runs[siblingIndex].texts + const sibling = createLiveQueryCollection({ + query: (q) => + q.from({ row: timeline }).fn.select(({ row }) => ({ + key: row.key, + texts: row.runs[siblingIndex]!.texts, + })), + getKey: (row) => row.key, + }) + await sibling.preload() + const siblingEvents = vi.fn() + const siblingSubscription = sibling.subscribeChanges(siblingEvents, { + includeInitialState: false, + }) + const updateEvents = vi.fn() + const timelineSubscription = timeline.subscribeChanges(updateEvents, { + includeInitialState: false, + }) - expect(data().runs).toHaveLength(2) - expect(data().runs[0].texts[0].text).toBe(``) - expect(data().runs[1].texts[0].text).toBe(``) - - const timelineRowBefore = data() - const run1TextsBefore = timelineRowBefore.runs[0].texts - const updateEvents: Array = [] - timeline.subscribeChanges((changes) => { - for (const change of changes) { - if (change.type === `update`) { - updateEvents.push(change) - } + try { + textDeltas.insert({ + key: `td-1`, + text_id: `text-${changedIndex + 1}`, + run_id: `run-${changedIndex + 1}`, + _seq: 5, + delta: `Hello`, + }) + await new Promise((r) => setTimeout(r, 100)) + + expect(data().runs[changedIndex].texts[0].text).toBe(`Hello`) + expect(data().runs[siblingIndex].texts[0].text).toBe(``) + + expect(updateEvents).toHaveBeenCalledTimes(1) + expect(updateEvents.mock.calls[0]![0]).toMatchObject([ + { type: `update`, key: TIMELINE_KEY, value: data() }, + ]) + expect(data().runs[siblingIndex].texts).toEqual(siblingTextsBefore) + expect(timelineRowBefore.runs[changedIndex].texts[0].text).toBe(``) + expect(siblingEvents).not.toHaveBeenCalled() + } finally { + timelineSubscription.unsubscribe() + siblingSubscription.unsubscribe() + await sibling.cleanup() } - }) - - textDeltas.insert({ - key: `td-1`, - text_id: `text-2`, - run_id: `run-2`, - _seq: 5, - delta: `Hello`, - }) - await new Promise((r) => setTimeout(r, 100)) - - expect(data().runs[1].texts[0].text).toBe(`Hello`) - expect(data().runs[0].texts[0].text).toBe(``) - - expect(data().runs[0].texts).toBe(run1TextsBefore) - }) + }, + ) // Three collection levels (products -> priceRanges -> region). When two // price ranges in different parent groups point at the same deepest diff --git a/packages/db/tests/query/ir-stable-identity.test.ts b/packages/db/tests/query/ir-stable-identity.test.ts index 6b730cbda3..fed74b9f72 100644 --- a/packages/db/tests/query/ir-stable-identity.test.ts +++ b/packages/db/tests/query/ir-stable-identity.test.ts @@ -32,7 +32,6 @@ import { UnhashableQueryIRError, getLoadSubsetDemandKey, getQueryIdentity, - getStableExpressionHash, getStableQueryIRHash, getStableValueHash, } from '../../src/query/ir-stable-identity.js' @@ -49,8 +48,11 @@ import { compileExpression, toBooleanPredicate, } from '../../src/query/compiler/evaluators.js' -import { isLoadSubsetRequestSubsumedBy } from '../../src/query/predicate-utils.js' -import { createRuntimeReferenceIdentityFactory } from '../../src/query/runtime-reference-identity.js' +import { + createRuntimeReferenceIdentityFactory, + getRuntimeReferenceIdentity, +} from '../../src/query/runtime-reference-identity.js' +import { createValueIdentity } from '../../src/query/equality-value-identity.js' import type { BasicExpression, QueryIR } from '../../src/query/ir.js' import type { LoadSubsetOptions } from '../../src/types.js' @@ -75,6 +77,13 @@ interface User { largeViewCount?: bigint } +function getProjectedExpressionIdentity(expression: BasicExpression): string { + return getQueryIdentity({ + ...getQueryIR(new Query().from({ user: usersCollection })), + select: { value: expression }, + }) +} + const referenceSemanticPairArbitrary = fc.oneof( fc .array(fc.integer()) @@ -232,9 +241,11 @@ describe(`semantic expression identity`, () => { ]) const flat = new Func(`and`, [adult, enabled]) - expect(getStableExpressionHash(nested)).toBe(getStableExpressionHash(flat)) - expect(getStableExpressionHash(new Func(`or`, [adult, adult]))).toBe( - getStableExpressionHash(new Func(`or`, [adult])), + expect(getProjectedExpressionIdentity(nested)).toBe( + getProjectedExpressionIdentity(flat), + ) + expect(getProjectedExpressionIdentity(new Func(`or`, [adult, adult]))).toBe( + getProjectedExpressionIdentity(new Func(`or`, [adult])), ) }) @@ -245,25 +256,25 @@ describe(`semantic expression identity`, () => { expect(toBooleanPredicate(compileExpression(bareAge)(row))).toBe(false) expect(toBooleanPredicate(compileExpression(duplicateAnd)(row))).toBe(true) - expect(getStableExpressionHash(duplicateAnd)).not.toBe( - getStableExpressionHash(bareAge), + expect(getProjectedExpressionIdentity(duplicateAnd)).not.toBe( + getProjectedExpressionIdentity(bareAge), ) }) it(`normalizes equality and reversed inequalities`, () => { - expect(getStableExpressionHash(new Func(`eq`, [age, new Value(18)]))).toBe( - getStableExpressionHash(new Func(`eq`, [new Value(18), age])), - ) - expect(getStableExpressionHash(new Func(`gt`, [age, new Value(18)]))).toBe( - getStableExpressionHash(new Func(`lt`, [new Value(18), age])), - ) + expect( + getProjectedExpressionIdentity(new Func(`eq`, [age, new Value(18)])), + ).toBe(getProjectedExpressionIdentity(new Func(`eq`, [new Value(18), age]))) + expect( + getProjectedExpressionIdentity(new Func(`gt`, [age, new Value(18)])), + ).toBe(getProjectedExpressionIdentity(new Func(`lt`, [new Value(18), age]))) }) it(`preserves order-sensitive function arguments`, () => { expect( - getStableExpressionHash(new Func(`subtract`, [age, new Value(1)])), + getProjectedExpressionIdentity(new Func(`subtract`, [age, new Value(1)])), ).not.toBe( - getStableExpressionHash(new Func(`subtract`, [new Value(1), age])), + getProjectedExpressionIdentity(new Func(`subtract`, [new Value(1), age])), ) }) @@ -276,13 +287,13 @@ describe(`semantic expression identity`, () => { expect(compileExpression(pair.original)(row)).toBe( compileExpression(pair.equivalent)(row), ) - expect(getStableExpressionHash(pair.original)).toBe( - getStableExpressionHash(pair.equivalent), + expect(getProjectedExpressionIdentity(pair.original)).toBe( + getProjectedExpressionIdentity(pair.equivalent), ) }) fcTest.prop([referenceSemanticPairArbitrary])( - `keeps reference-semantic values distinct across identity and coverage`, + `keeps reference-semantic values distinct across expression and demand identity`, ([first, second]) => { const value = new PropRef([`row`, `value`]) const firstPredicate = new Func(`eq`, [value, new Value(first)]) @@ -294,18 +305,12 @@ describe(`semantic expression identity`, () => { expect(compileExpression(firstPredicate)(row)).toBe(true) expect(compileExpression(secondPredicate)(row)).toBe(false) - expect(getStableExpressionHash(firstPredicate)).not.toBe( - getStableExpressionHash(secondPredicate), + expect(getProjectedExpressionIdentity(firstPredicate)).not.toBe( + getProjectedExpressionIdentity(secondPredicate), ) expect( getLoadSubsetDemandKey({ where: firstPredicate, limit: 1 }), ).not.toBe(getLoadSubsetDemandKey({ where: secondPredicate, limit: 1 })) - expect( - isLoadSubsetRequestSubsumedBy( - { where: firstPredicate, limit: 1 }, - { where: secondPredicate, limit: 1 }, - ), - ).toBe(false) }, ) @@ -315,14 +320,13 @@ describe(`semantic expression identity`, () => { vi.resetModules() try { - const { getRuntimeReferenceIdentity } = await import( - `../../src/query/runtime-reference-identity.js` - ) + const { getRuntimeReferenceIdentity: getFreshRuntimeReferenceIdentity } = + await import(`../../src/query/runtime-reference-identity.js`) expect(getRandomValues).not.toHaveBeenCalled() - getRuntimeReferenceIdentity({}) - getRuntimeReferenceIdentity({}) + getFreshRuntimeReferenceIdentity({}) + getFreshRuntimeReferenceIdentity({}) expect(getRandomValues).toHaveBeenCalledOnce() } finally { @@ -337,6 +341,95 @@ describe(`semantic expression identity`, () => { expect(firstRuntime({ a: 1 })).not.toEqual(secondRuntime({ b: 2 })) }) + it(`allocates runtime entropy only when the first identity is requested`, () => { + const getRandomValues = vi.fn((values: Uint32Array) => values) + vi.stubGlobal(`crypto`, { getRandomValues }) + try { + const runtime = createRuntimeReferenceIdentityFactory() + expect(getRandomValues).not.toHaveBeenCalled() + + runtime({}) + runtime({}) + expect(getRandomValues).toHaveBeenCalledOnce() + } finally { + vi.unstubAllGlobals() + } + }) + + it(`keeps symbol identities stable and distinct`, () => { + const first = Symbol(`value`) + const second = Symbol(`value`) + + expect(getRuntimeReferenceIdentity(first)).toEqual( + getRuntimeReferenceIdentity(first), + ) + expect(getRuntimeReferenceIdentity(first)).not.toEqual( + getRuntimeReferenceIdentity(second), + ) + }) + + it(`does not retain symbols in strong identity maps when weak symbol keys are supported`, () => { + const NativeMap = Map + const stronglyStoredSymbols = new Set() + class TrackingMap extends NativeMap { + override set(key: K, value: V): this { + if (typeof key === `symbol`) stronglyStoredSymbols.add(key) + return super.set(key, value) + } + } + const local = Symbol(`local`) + const registered = Symbol.for( + `tanstack-db-runtime-reference-test-${Date.now()}`, + ) + + vi.stubGlobal(`Map`, TrackingMap) + try { + const runtime = createRuntimeReferenceIdentityFactory() + runtime(local) + runtime(registered) + } finally { + vi.unstubAllGlobals() + } + + expect(stronglyStoredSymbols).not.toContain(local) + expect(stronglyStoredSymbols).not.toContain(registered) + }) + + it(`keeps correct symbol identity when weak symbol keys are unavailable`, () => { + const NativeWeakMap = WeakMap + class ObjectOnlyWeakMap extends NativeWeakMap { + override set(key: K, value: V): this { + if (typeof key === `symbol`) { + throw new TypeError(`Symbols cannot be weak keys`) + } + return super.set(key, value) + } + } + const first = Symbol(`value`) + const second = Symbol(`value`) + + vi.stubGlobal(`WeakMap`, ObjectOnlyWeakMap) + try { + const runtime = createRuntimeReferenceIdentityFactory() + + expect(runtime(first)).toEqual(runtime(first)) + expect(runtime(first)).not.toEqual(runtime(second)) + } finally { + vi.unstubAllGlobals() + } + }) + + it(`scopes opaque value identities to their owner`, () => { + const firstScope = createValueIdentity() + const secondScope = createValueIdentity() + const first = Symbol(`value`) + const second = Symbol(`value`) + + expect(firstScope.equality(first)).toEqual(firstScope.equality(first)) + expect(firstScope.equality(first)).not.toEqual(firstScope.equality(second)) + expect(firstScope.equality(first)).not.toEqual(secondScope.equality(first)) + }) + it(`falls back when the runtime crypto object lacks getRandomValues`, () => { vi.stubGlobal(`crypto`, {}) try { @@ -371,8 +464,8 @@ describe(`semantic expression identity`, () => { compileExpression(reordered)(row), ) } - expect(getStableExpressionHash(ordered)).toBe( - getStableExpressionHash(reordered), + expect(getProjectedExpressionIdentity(ordered)).toBe( + getProjectedExpressionIdentity(reordered), ) expect(getLoadSubsetDemandKey({ where: ordered })).toBe( getLoadSubsetDemandKey({ where: reordered }), @@ -460,6 +553,23 @@ describe(`loadSubset demand identity`, () => { ) }) + it.each([ + [`function`, () => `value`, () => `value`], + [`symbol`, Symbol(`value`), Symbol(`value`)], + ])(`uses runtime reference identity for %s demand values`, (_name, a, b) => { + const field = new PropRef([`row`, `value`]) + const demand = (value: unknown): LoadSubsetOptions => ({ + where: new Func(`eq`, [field, new Value(value)]), + }) + + expect(getLoadSubsetDemandKey(demand(a))).toBe( + getLoadSubsetDemandKey(demand(a)), + ) + expect(getLoadSubsetDemandKey(demand(a))).not.toBe( + getLoadSubsetDemandKey(demand(b)), + ) + }) + it.each([ [`signed zero`, -0, 0], [`invalid Date`, new Date(Number.NaN), new Date(Number.NaN)], @@ -494,8 +604,8 @@ describe(`loadSubset demand identity`, () => { expect( compileExpression(firstPredicate)({ row: { value: secondValue } }), ).toBe(true) - expect(getStableExpressionHash(firstPredicate)).toBe( - getStableExpressionHash(secondPredicate), + expect(getProjectedExpressionIdentity(firstPredicate)).toBe( + getProjectedExpressionIdentity(secondPredicate), ) expect(getLoadSubsetDemandKey({ where: firstPredicate })).toBe( getLoadSubsetDemandKey({ where: secondPredicate }), @@ -1459,34 +1569,21 @@ describe(`stable QueryIR identity smoke test`, () => { } }) - it(`rejects function and symbol values inside structured expressions`, () => { - const queries = [ - [ - `function value`, - getQueryIR( - new Query() - .from({ user: usersCollection }) - .where(({ user }) => eq(user.name, (() => `Tanner`) as never)), - ), - /function value/, - ], - [ - `symbol value`, - getQueryIR( - new Query() - .from({ user: usersCollection }) - .where(({ user }) => eq(user.name, Symbol(`name`) as never)), - ), - /symbol value/, - ], - ] as const - - for (const [name, query, message] of queries) { - expect(() => getStableQueryIRHash(query), name).toThrow( - UnhashableQueryIRError, + it.each([ + [`function`, () => `Tanner`, () => `Tanner`], + [`symbol`, Symbol(`name`), Symbol(`name`)], + ])(`keeps %s query values distinct by reference`, (_name, a, b) => { + const query = (value: unknown) => + getQueryIR( + new Query() + .from({ user: usersCollection }) + .where(({ user }) => eq(user.name, value as never)), ) - expect(() => getStableQueryIRHash(query), name).toThrow(message) - } + + expect(getStableQueryIRHash(query(a))).toBe(getStableQueryIRHash(query(a))) + expect(getStableQueryIRHash(query(a))).not.toBe( + getStableQueryIRHash(query(b)), + ) }) it(`accepts opaque object values by reference`, () => { diff --git a/packages/db/tests/query/join-subquery.test.ts b/packages/db/tests/query/join-subquery.test.ts index ed3261cc57..c14e66b113 100644 --- a/packages/db/tests/query/join-subquery.test.ts +++ b/packages/db/tests/query/join-subquery.test.ts @@ -476,7 +476,7 @@ function createJoinSubqueryTests(autoIndex: `off` | `eager`): void { }) }) - test(`should use subquery in LEFT JOIN clause - left join with ordered subquery with limit`, () => { + test(`should use subquery in LEFT JOIN clause - left join with ordered subquery with limit`, async () => { const joinSubquery = createLiveQueryCollection({ query: (q) => { return q @@ -498,6 +498,9 @@ function createJoinSubqueryTests(autoIndex: `off` | `eager`): void { startSync: true, }) + // Initial ordered refinement may hold publication beyond startSync. + await joinSubquery.preload() + expect(joinSubquery.isReady()).toBe(true) const results = joinSubquery.toArray.map((row) => ({ ...stripVirtualProps(row), issue: stripVirtualProps(row.issue), @@ -517,7 +520,7 @@ function createJoinSubqueryTests(autoIndex: `off` | `eager`): void { ]) }) - test(`should use subquery in RIGHT JOIN clause - left join with ordered subquery with limit`, () => { + test(`should use subquery in RIGHT JOIN clause - left join with ordered subquery with limit`, async () => { const joinSubquery = createLiveQueryCollection({ query: (q) => { return q @@ -539,6 +542,8 @@ function createJoinSubqueryTests(autoIndex: `off` | `eager`): void { startSync: true, }) + await joinSubquery.preload() + expect(joinSubquery.isReady()).toBe(true) const results = joinSubquery.toArray.map((row) => ({ ...stripVirtualProps(row), issue: stripVirtualProps(row.issue), diff --git a/packages/db/tests/query/live-query-collection.test.ts b/packages/db/tests/query/live-query-collection.test.ts index c6fc4be397..2ef16a0db1 100644 --- a/packages/db/tests/query/live-query-collection.test.ts +++ b/packages/db/tests/query/live-query-collection.test.ts @@ -18,8 +18,13 @@ import { } from '../utils.js' import { createDeferred } from '../../src/deferred' import { BTreeIndex } from '../../src/indexes/btree-index' +import { createFilterFunctionFromExpression } from '../../src/collection/change-events' import { Func, Value } from '../../src/query/ir.js' -import type { ChangeMessage, LoadSubsetOptions } from '../../src/types.js' +import type { + ChangeMessage, + LoadSubsetOptions, + SyncConfig, +} from '../../src/types.js' // Sample user type for tests type User = { @@ -1591,7 +1596,10 @@ describe(`createLiveQueryCollection`, () => { sync: ({ begin, write, commit, markReady }) => { markReady() return { - loadSubset: () => { + loadSubset: (options) => { + // Boundary refinement asks only for rows tied with rank 1. + // This source has already supplied that whole tie class. + if (options.where) return Promise.resolve() parentLoadCount++ begin() const candidates: Array = [ @@ -1610,72 +1618,1064 @@ describe(`createLiveQueryCollection`, () => { }, }, }) - let childLoadCount = 0 - const children = createCollection({ - id: `window-lazy-demand-children`, - getKey: (child) => child.id, - syncMode: `on-demand`, - autoIndex: `eager`, - defaultIndexType: BTreeIndex, - sync: { - sync: ({ begin, write, commit, markReady }) => { - markReady() - return { - loadSubset: () => { - childLoadCount++ - if (childLoadCount > 1) { - if (delivery === `throw`) throw failure - return Promise.reject(failure) - } - begin() - write({ type: `insert`, value: { id: 10, parentId: 1 } }) - commit() - return Promise.resolve() - }, - } - }, + let childLoadCount = 0 + const children = createCollection({ + id: `window-lazy-demand-children`, + getKey: (child) => child.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: () => { + childLoadCount++ + if (childLoadCount > 1) { + if (delivery === `throw`) throw failure + return Promise.reject(failure) + } + begin() + write({ type: `insert`, value: { id: 10, parentId: 1 } }) + commit() + return Promise.resolve() + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ parent: parents }) + .leftJoin({ child: children }, ({ parent, child }) => + eq(parent.id, child.parentId), + ) + .orderBy(({ parent }) => parent.rank, `asc`) + .limit(1) + .select(({ parent, child }) => ({ + id: parent.id, + childId: child.id, + })), + ) + + try { + await live.preload() + expect(live.status).toBe(`ready`) + + const setWindow = async () => { + const result = live.utils.setWindow({ offset: 0, limit: 2 }) + if (result !== true) await result + } + await expect(setWindow()).rejects.toBe(failure) + expect(live.utils.lastSubsetError).toBe(failure) + } finally { + await Promise.all([ + live.cleanup(), + parents.cleanup(), + children.cleanup(), + ]) + } + }, + ) + + it(`retries the same ordered refill after a transient rejection`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`ordered refill failed`) + let loadCount = 0 + const acquisitions: Array = [] + const source = createCollection({ + id: `ordered-refill-retry-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + acquisitions.push(options) + loadCount++ + if (loadCount === 3) return Promise.reject(failure) + const deliver = (row: Row) => { + begin() + write({ type: `insert`, value: row }) + commit() + } + if (loadCount === 1) { + deliver({ id: 1, rank: 1 }) + return true + } + if (loadCount === 2 || options.where) return true + return Promise.resolve().then(() => deliver({ id: 2, rank: 2 })) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + expect(loadCount).toBe(2) + + const failedWindow = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(failedWindow).toBeInstanceOf(Promise) + await expect(failedWindow).rejects.toBe(failure) + expect(live.utils.lastSubsetError).toBe(failure) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + + const retry = live.utils.setWindow({ offset: 0, limit: 2 }) + if (retry !== true) await retry + // Recovery loads the full source once; it needs no tie-boundary probe. + expect(loadCount).toBe(4) + const recovery = acquisitions[3]! + expect(recovery.where).toBeUndefined() + expect(recovery.orderBy).toBeUndefined() + expect(recovery.limit).toBeUndefined() + expect(recovery.offset).toBeUndefined() + expect(recovery.cursor).toBeUndefined() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`retries a failed full-source window refinement`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`full-source refinement failed`) + let loadCount = 0 + let syncOps!: Parameters[`sync`]>[0] + const acquisitions: Array = [] + const releases: Array = [] + const source = createCollection({ + id: `ordered-full-source-retry-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + const { begin, write, commit, markReady } = operations + markReady() + return { + loadSubset: (options) => { + loadCount++ + acquisitions.push(options) + begin() + write({ + type: `insert`, + value: { id: loadCount, rank: loadCount }, + }) + commit(options.signal) + return loadCount === 1 + ? Promise.reject(failure) + : Promise.resolve() + }, + unloadSubset: (options) => { + releases.push(options) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(0) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + .distinct(), + ) + + try { + await live.preload() + await expect( + live.utils.setWindow({ offset: 0, limit: 2 }), + ).rejects.toBe(failure) + expect(Array.from(live.values())).toEqual([]) + + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(loadCount).toBe(2) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await flushPromises() + await flushPromises() + expect(loadCount).toBe(3) + + await live.cleanup() + expect(releases).toHaveLength(acquisitions.length) + for (const [index, acquisition] of acquisitions.entries()) { + expect(releases[index]).toBe(acquisition) + } + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`publishes a window after its failed full-source demand replays successfully`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`full-source refinement failed`) + let loadCount = 0 + let syncOps!: Parameters[`sync`]>[0] + const publications: Array> = [] + const source = createCollection({ + id: `ordered-full-source-replay-recovery-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.markReady() + return { + loadSubset: (options) => { + loadCount++ + operations.begin() + operations.write({ + type: `insert`, + value: { id: 1, rank: 1 }, + }) + if (loadCount > 1) { + operations.write({ + type: `insert`, + value: { id: 2, rank: 2 }, + }) + } + operations.commit(options.signal) + return loadCount === 1 + ? Promise.reject(failure) + : Promise.resolve() + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(0) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + .distinct(), + ) + const subscription = live.subscribeChanges(() => { + publications.push(Array.from(live.values(), ({ id }) => id)) + }) + + try { + await live.preload() + await expect( + live.utils.setWindow({ offset: 0, limit: 2 }), + ).rejects.toBe(failure) + expect(Array.from(live.values())).toEqual([]) + + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await flushPromises() + await flushPromises() + expect(loadCount).toBe(2) + expect(Array.from(live.values())).toEqual([]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 0 }) + expect(publications).toEqual([]) + + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(publications).toEqual([[1, 2]]) + } finally { + subscription.unsubscribe() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`waits for an active replay before settling a window move`, async () => { + type Row = { id: number; rank: number } + const replayGate = createDeferred() + let recovering = false + let syncOps!: Parameters[`sync`]>[0] + const source = createCollection({ + id: `ordered-window-during-replay-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.begin() + for (let id = 1; id <= 4; id++) { + operations.write({ type: `insert`, value: { id, rank: id } }) + } + operations.commit() + operations.markReady() + return { + loadSubset: (options) => { + if (!recovering) return true + operations.begin() + for (let id = 5; id <= 8; id++) { + operations.write({ type: `insert`, value: { id, rank: id } }) + } + operations.commit(options.signal) + return replayGate.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + + try { + await live.preload() + await live.utils.setWindow({ offset: 0, limit: 4 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2, 3, 4]) + + recovering = true + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await flushPromises() + + const move = live.utils.setWindow({ offset: 0, limit: 3 }) + expect(move).toBeInstanceOf(Promise) + let settled = false + void Promise.resolve(move).then( + () => { + settled = true + }, + () => { + settled = true + }, + ) + await flushPromises() + expect(settled).toBe(false) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2, 3, 4]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) + + replayGate.resolve() + await move + expect(Array.from(live.values(), ({ id }) => id)).toEqual([5, 6, 7]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 3 }) + } finally { + replayGate.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`rejects a replay-blocked window move when cleanup abandons it`, async () => { + type Row = { id: number; rank: number } + const replayGate = createDeferred() + let recovering = false + let syncOps!: Parameters[`sync`]>[0] + const publications: Array> = [] + const source = createCollection({ + id: `ordered-replay-window-cleanup-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, rank: 1 } }) + operations.write({ type: `insert`, value: { id: 2, rank: 2 } }) + operations.commit() + operations.markReady() + return { + loadSubset: () => (recovering ? replayGate.promise : true), + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + const subscription = live.subscribeChanges(() => { + publications.push(Array.from(live.values(), ({ id }) => id)) + }) + + try { + await live.preload() + publications.length = 0 + recovering = true + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await flushPromises() + + const move = live.utils.setWindow({ offset: 0, limit: 3 }) + expect(move).toBeInstanceOf(Promise) + let moveError: unknown + let settled = false + void Promise.resolve(move).then( + () => { + settled = true + }, + (error) => { + moveError = error + settled = true + }, + ) + await flushPromises() + expect(settled).toBe(false) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(publications).toEqual([]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + + await live.cleanup() + await flushPromises() + expect(settled).toBe(true) + expect(moveError).toMatchObject({ name: `AbortError` }) + expect(publications).toEqual([]) + expect(live.status).toBe(`cleaned-up`) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + } finally { + subscription.unsubscribe() + replayGate.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`rejects a window move while source recovery is failed`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`ordered source replay failed`) + let recovering = false + let recoveryLoads = 0 + let syncOps!: Parameters[`sync`]>[0] + const source = createCollection({ + id: `ordered-window-after-failed-replay-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, rank: 1 } }) + operations.write({ type: `insert`, value: { id: 2, rank: 2 } }) + operations.commit() + operations.markReady() + return { + loadSubset: () => { + if (!recovering) return true + recoveryLoads++ + return Promise.reject(failure) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + + try { + await live.preload() + recovering = true + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await vi.waitFor(() => expect(live.utils.lastSubsetError).toBe(failure)) + const loadsAfterFailure = recoveryLoads + + await expect( + live.utils.setWindow({ offset: 0, limit: 3 }), + ).rejects.toBe(failure) + expect(recoveryLoads).toBe(loadsAfterFailure) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it.each( + ( + [ + { label: `Error`, value: new Error(`replay failed`) }, + { label: `undefined`, value: undefined }, + { label: `NaN`, value: Number.NaN }, + { label: `false`, value: false }, + { label: `object`, value: { reason: `replay failed` } }, + ] as const + ).flatMap(({ label, value }) => + ([`throw`, `reject`] as const).flatMap((delivery) => + ([`retained`, `new`] as const).map((demand) => ({ + delivery, + demand, + label, + value, + })), + ), + ), + )( + `scopes a normalized $delivery replay failure with $label to its $demand demand`, + async ({ delivery, demand, value }) => { + type Row = { id: number; rank: number } + const replayGate = createDeferred() + let recovering = false + let failedReplayCalls = 0 + let syncOps!: Parameters[`sync`]>[0] + const source = createCollection({ + id: `ordered-normalized-${delivery}-${String(value)}-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, rank: 1 } }) + operations.commit() + operations.markReady() + return { + loadSubset: (options) => { + if (!recovering) return true + // Choose by request shape, not callback order: a startup + // throw rolls back new demand but retains a replayed owner. + const target = + demand === `retained` + ? options.limit !== undefined + : options.limit === undefined && + options.where === undefined + if (!target || failedReplayCalls > 0) + return replayGate.promise + failedReplayCalls++ + if (delivery === `throw`) throw value + return Promise.reject(value) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + recovering = true + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + await vi.waitFor(() => + expect(live.utils.lastSubsetError).toBeInstanceOf(Error), + ) + const reportedError = live.utils.lastSubsetError + expect(failedReplayCalls).toBe(1) + + const windowMove = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(windowMove).toBeInstanceOf(Promise) + replayGate.resolve() + const settlement = await Promise.resolve(windowMove).then( + () => ({ status: `fulfilled` as const }), + (error: unknown) => ({ status: `rejected` as const, error }), + ) + if (demand === `new` && delivery === `throw`) { + expect(settlement.status).toBe(`fulfilled`) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + } else { + expect(settlement.status).toBe(`rejected`) + if (settlement.status !== `rejected`) + throw new Error(`Expected replay rejection`) + expect(settlement.error).toBe(reportedError) + expect(settlement.error).toBeInstanceOf(Error) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + } + expect(live.utils.lastSubsetError).toBe(reportedError) + } finally { + replayGate.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }, + ) + + it(`ignores queued replay setup after cleanup`, async () => { + type Row = { id: number; rank: number } + let syncOps!: Parameters[`sync`]>[0] + const source = createCollection({ + id: `ordered-replay-success-after-cleanup-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + syncOps = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, rank: 1 } }) + operations.commit() + operations.markReady() + return { loadSubset: () => true } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + const queued: Array<() => void> = [] + + try { + await live.preload() + const queueSpy = vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => queued.push(callback)) + + syncOps.begin() + syncOps.truncate() + const replayReceipt = syncOps.commit() + if (replayReceipt !== true) await replayReceipt + const replaySetup = queued.splice(0) + expect(replaySetup.length).toBeGreaterThan(0) + + await live.cleanup() + for (const callback of replaySetup) expect(callback).not.toThrow() + for (const callback of queued.splice(0)) expect(callback).not.toThrow() + queueSpy.mockRestore() + } finally { + vi.restoreAllMocks() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`resolves omitted window fields from the last requested window`, async () => { + type Row = { id: number; rank: number } + const source = createCollection({ + id: `ordered-partial-window-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (let id = 1; id <= 5; id++) { + write({ type: `insert`, value: { id, rank: id } }) + } + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + + try { + await live.preload() + const requestedWindow = { offset: 2 } + await live.utils.setWindow(requestedWindow) + requestedWindow.offset = 4 + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([3, 4]) + expect(live.utils.getWindow()).toEqual({ offset: 2, limit: 2 }) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`rejects a pending window move when cleanup abandons it`, async () => { + type Row = { id: number; rank: number } + const gate = createDeferred() + let loadCount = 0 + const source = createCollection({ + id: `ordered-cleanup-window-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + commit() + markReady() + return { + loadSubset: () => { + loadCount++ + return loadCount === 3 ? gate.promise : true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + const move = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(move).toBeInstanceOf(Promise) + const rejection = expect(move).rejects.toMatchObject({ + name: `AbortError`, + }) + + await live.cleanup() + await rejection + expect(live.status).toBe(`cleaned-up`) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + } finally { + gate.resolve() + await source.cleanup() + } + }) + + it(`keeps window generations distinct across immediate cleanup and restart`, async () => { + type Row = { id: number; rank: number } + const oldGate = createDeferred() + const newGate = createDeferred() + let limitFourCalls = 0 + const source = createCollection({ + id: `ordered-window-restart-generation-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + operations.begin() + for (let id = 1; id <= 6; id++) { + operations.write({ type: `insert`, value: { id, rank: id } }) + } + operations.commit() + operations.markReady() + return { + loadSubset: (options) => { + if (options.where || options.limit !== 4) return true + limitFourCalls++ + if (limitFourCalls === 1) { + options.signal?.addEventListener( + `abort`, + () => + oldGate.reject(new DOMException(`aborted`, `AbortError`)), + { once: true }, + ) + return oldGate.promise + } + return newGate.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + const abandoned = live.utils.setWindow({ offset: 2, limit: 2 }) + expect(abandoned).toBeInstanceOf(Promise) + const abandonedRejection = expect(abandoned).rejects.toMatchObject({ + name: `AbortError`, + }) + + const cleanup = live.cleanup() + const preload = live.preload() + const replacement = live.utils.setWindow({ offset: 2, limit: 2 }) + expect(replacement).toBeInstanceOf(Promise) + await Promise.all([cleanup, preload, abandonedRejection]) + + newGate.resolve() + await replacement + await live.utils.setWindow({ limit: 1 }) + expect(live.utils.getWindow()).toEqual({ offset: 2, limit: 1 }) + } finally { + oldGate.resolve() + newGate.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`keeps the last complete window when a required tie boundary rejects`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`ordered boundary failed`) + let loadCount = 0 + const source = createCollection({ + id: `ordered-boundary-rollback-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + commit(options.signal) + return true + } + if (loadCount === 2) return true + if (loadCount === 3) { + begin() + write({ type: `insert`, value: { id: 3, rank: 2 } }) + commit(options.signal) + return Promise.resolve() + } + if (loadCount === 4) return Promise.reject(failure) + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + const publications: Array> = [] + const subscription = live.subscribeChanges(() => { + publications.push(Array.from(live.values(), ({ id }) => id)) + }) + + const result = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(result).toBeInstanceOf(Promise) + await expect(result).rejects.toBe(failure) + await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(publications).toEqual([]) + subscription.unsubscribe() + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`settles a superseding window only after that window is visible`, async () => { + type Row = { id: number; rank: number } + const gate = createDeferred() + let loadCount = 0 + const source = createCollection({ + id: `ordered-superseding-window-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + write({ type: `insert`, value: { id: 2, rank: 2 } }) + commit() + markReady() + return { + loadSubset: () => { + loadCount++ + return loadCount <= 2 ? true : gate.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + const first = live.utils.setWindow({ offset: 0, limit: 3 }) + expect(first).toBeInstanceOf(Promise) + const second = live.utils.setWindow({ offset: 1, limit: 1 }) + expect(second).toBeInstanceOf(Promise) + + let secondSettled = false + void Promise.resolve(second).then(() => { + secondSettled = true + }) + await flushPromises() + expect(secondSettled).toBe(false) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + + gate.resolve() + await Promise.all([first, second]) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([2]) + } finally { + gate.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`keeps the restarted session's settled window after a failed move`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`restarted ordered page failed`) + let failPage = false + const source = createCollection({ + id: `ordered-window-restart-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + write({ type: `insert`, value: { id: 2, rank: 2 } }) + commit() + markReady() + return { + loadSubset: (options) => { + if (failPage && !options.where) throw failure + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + + await live.cleanup() + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + + failPage = true + await expect( + Promise.resolve().then(() => + live.utils.setWindow({ offset: 0, limit: 3 }), + ), + ).rejects.toBe(failure) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`does not publish a row that leaves and re-enters during a failed window move`, async () => { + type Row = { id: number; rank: number } + const failure = new Error(`offset page failed`) + let failPage = false + const source = createCollection({ + id: `ordered-window-offset-rollback-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + write({ type: `insert`, value: { id: 2, rank: 2 } }) + commit() + markReady() + return { + loadSubset: (options) => { + if (failPage && !options.where) throw failure + return true + }, + } }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + + try { + await live.preload() + const publications: Array> = [] + const subscription = live.subscribeChanges((changes) => { + publications.push(changes.map(({ type, key }) => ({ type, key }))) }) - const live = createLiveQueryCollection((q) => - q - .from({ parent: parents }) - .leftJoin({ child: children }, ({ parent, child }) => - eq(parent.id, child.parentId), - ) - .orderBy(({ parent }) => parent.rank, `asc`) - .limit(1) - .select(({ parent, child }) => ({ - id: parent.id, - childId: child.id, - })), - ) - try { - await live.preload() - expect(live.status).toBe(`ready`) + failPage = true + await expect( + Promise.resolve().then(() => + live.utils.setWindow({ offset: 1, limit: 2 }), + ), + ).rejects.toBe(failure) + await flushPromises() - const setWindow = async () => { - const result = live.utils.setWindow({ offset: 0, limit: 2 }) - if (result !== true) await result - } - await expect(setWindow()).rejects.toBe(failure) - expect(live.utils.lastSubsetError).toBe(failure) - } finally { - await Promise.all([ - live.cleanup(), - parents.cleanup(), - children.cleanup(), - ]) - } - }, - ) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(publications).toEqual([]) + subscription.unsubscribe() + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) - it(`retries the same ordered refill after a transient rejection`, async () => { + it(`keeps partial ordered source work private when later refinement rejects`, async () => { type Row = { id: number; rank: number } - const failure = new Error(`ordered refill failed`) + const failure = new Error(`ordered boundary failed`) let loadCount = 0 const source = createCollection({ - id: `ordered-refill-retry-source`, + id: `ordered-window-partial-source`, getKey: (row) => row.id, syncMode: `on-demand`, autoIndex: `eager`, @@ -1684,22 +2684,27 @@ describe(`createLiveQueryCollection`, () => { sync: ({ begin, write, commit, markReady }) => { markReady() return { - loadSubset: () => { + loadSubset: (options) => { loadCount++ - if (loadCount === 2) return Promise.reject(failure) - const deliver = () => { - begin() - write({ - type: `insert`, - value: { id: loadCount, rank: loadCount }, - }) - commit() - } if (loadCount === 1) { - deliver() + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + commit(options.signal) return true } - return Promise.resolve().then(deliver) + if (loadCount === 2) return true + if (loadCount === 3) { + begin() + // Fulfill the requested continuation so its new boundary + // needs refinement. Also deliver a live insert before the + // cursor: it would replace the old top-one result if leaked. + write({ type: `insert`, value: { id: 2, rank: 2 } }) + write({ type: `insert`, value: { id: 0, rank: 0 } }) + commit(options.signal) + return Promise.resolve() + } + if (loadCount === 4) return Promise.reject(failure) + return true }, } }, @@ -1708,22 +2713,72 @@ describe(`createLiveQueryCollection`, () => { const live = createLiveQueryCollection((q) => q .from({ row: source }) - .orderBy(({ row }) => row.rank, `asc`) - .limit(2), + .orderBy(({ row }) => row.rank) + .limit(1), ) try { await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + const publications: Array> = [] + const subscription = live.subscribeChanges((changes) => { + publications.push(changes.map(({ type, key }) => ({ type, key }))) + }) + + await expect( + live.utils.setWindow({ offset: 0, limit: 2 }), + ).rejects.toBe(failure) await flushPromises() - expect(loadCount).toBe(2) - expect(live.utils.lastSubsetError).toBe(failure) - const retry = live.utils.setWindow({ offset: 0, limit: 2 }) - if (retry instanceof Promise) await retry + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + expect(publications).toEqual([]) + + await live.utils.setWindow({ offset: 0, limit: 2 }) await flushPromises() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([0, 1]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(publications).toHaveLength(1) + subscription.unsubscribe() + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) - expect(loadCount).toBe(3) - expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 3]) + it(`copies a settled window instead of retaining caller-owned options`, async () => { + type Row = { id: number; rank: number } + const source = createCollection({ + id: `ordered-window-options-copy-source`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + write({ type: `insert`, value: { id: 2, rank: 2 } }) + commit() + markReady() + return { loadSubset: () => true } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + + try { + await live.preload() + const requestedWindow = { offset: 0, limit: 2 } + await live.utils.setWindow(requestedWindow) + requestedWindow.limit = 1 + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) } finally { await Promise.all([live.cleanup(), source.cleanup()]) } @@ -2420,7 +3475,10 @@ describe(`createLiveQueryCollection`, () => { return true } - // Second call (triggered by setWindow) returns a promise + // The second call closes the initial ordered boundary. + if (loadSubsetCallCount === 2) return true + + // The later call triggered by setWindow returns a promise. const loadPromise = new Promise((resolve) => { // Simulate async data loading with a delay setTimeout(() => { @@ -2456,7 +3514,7 @@ describe(`createLiveQueryCollection`, () => { // Initial state: should have 2 items (values 1, 2) expect(liveQuery.size).toBe(2) expect(liveQuery.isLoadingSubset).toBe(false) - expect(loadSubsetCallCount).toBe(1) + expect(loadSubsetCallCount).toBe(2) // Move window to offset 3, which requires loading more data // This should trigger loadSubset and return a Promise @@ -2486,8 +3544,16 @@ describe(`createLiveQueryCollection`, () => { expect(promiseResolved).toBe(false) expect(liveQuery.isLoadingSubset).toBe(true) - // Now advance time to complete the loading (50ms total from loadSubset call) + // Complete the page request. The operation must remain pending while + // the loader closes the ordering boundary so equal sort values cannot + // be omitted from later window moves. await vi.advanceTimersByTimeAsync(40) + expect(loadSubsetCallCount).toBe(4) + expect(promiseResolved).toBe(false) + expect(liveQuery.isLoadingSubset).toBe(true) + + // Complete the boundary request as well. + await vi.advanceTimersByTimeAsync(50) // Wait for the promise to resolve if (result !== true) { @@ -2507,6 +3573,139 @@ describe(`createLiveQueryCollection`, () => { } }) + it(`does not settle a synchronous ordered window before loading its tie boundary`, async () => { + type Row = { id: number; rank: number } + + const remote: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 0 }, + { id: 3, rank: 1 }, + { id: 4, rank: 1 }, + ] + const delivered = new Set() + let calls = 0 + + const source = createCollection({ + id: `sync-ordered-boundary-settlement`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + calls++ + const filter = options.where + ? createFilterFunctionFromExpression(options.where) + : () => true + const candidates = remote + .filter(filter) + .filter(({ id }) => !delivered.has(id)) + .sort( + (left, right) => + left.rank - right.rank || right.id - left.id, + ) + const selected = + options.limit === undefined + ? candidates + : candidates.slice(0, options.limit) + + if (selected.length > 0) { + begin() + for (const row of selected) { + delivered.add(row.id) + write({ type: `insert`, value: row }) + } + commit(options.signal) + } + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + expect(calls).toBe(2) + expect(live.toArray.map(({ id }) => id)).toEqual([1]) + + const settled = live.utils.setWindow({ offset: 2, limit: 1 }) + if (settled !== true) await settled + + expect(calls).toBe(4) + expect(live.toArray.map(({ id }) => id)).toEqual([3]) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it.each([ + { primary: `sync`, boundary: `sync` }, + { primary: `sync`, boundary: `async` }, + { primary: `async`, boundary: `sync` }, + { primary: `async`, boundary: `async` }, + ] as const)( + `rejects initial preload when a required $boundary tie-boundary load fails after a $primary primary load`, + async ({ primary, boundary }) => { + type Row = { id: number; rank: number } + + const failure = new Error(`ordered boundary failed`) + let calls = 0 + const source = createCollection({ + id: `initial-${primary}-${boundary}-ordered-boundary-failure`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + calls++ + if (options.where) { + if (boundary === `async`) return Promise.reject(failure) + throw failure + } + + begin() + write({ type: `insert`, value: { id: 2, rank: 0 } }) + commit(options.signal) + return primary === `async` ? Promise.resolve() : true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await expect(live.preload()).rejects.toBe(failure) + expect(calls).toBe(2) + expect(live.status).toBe(`error`) + expect(live.utils.lastSubsetError).toBe(failure) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }, + ) + it(`advances offset when async loadSubset fills an initially empty window`, async () => { type Item = { id: number; value: number } const remoteData: Array = [ @@ -2529,6 +3728,10 @@ describe(`createLiveQueryCollection`, () => { markReady() return { loadSubset: (options: LoadSubsetOptions) => { + // The last loaded boundary row is already present. Respect + // the exact tie predicate instead of treating it as an + // unbounded offset request. + if (options.where) return Promise.resolve() loadOffsets.push(options.offset) return new Promise((resolve) => { setTimeout(() => { @@ -2571,7 +3774,7 @@ describe(`createLiveQueryCollection`, () => { expect(liveQuery.toArray.map((item) => item.value)).toEqual([3, 4]) }) - it(`requests new offsets when window moves across identical orderBy values`, async () => { + it(`loads an identical orderBy tie class before later window moves`, async () => { type Item = { id: number; rank: number } const remoteData: Array = [ { id: 1, rank: 1 }, @@ -2630,7 +3833,7 @@ describe(`createLiveQueryCollection`, () => { await moveFirst } await flushPromises() - expect(loadOffsets).toEqual([0, 2]) + expect(loadOffsets).toEqual([0, undefined]) expect(liveQuery.toArray.map((item) => item.id)).toEqual([3, 4]) const moveSecond = liveQuery.utils.setWindow({ offset: 4, limit: 2 }) @@ -2638,7 +3841,7 @@ describe(`createLiveQueryCollection`, () => { await moveSecond } await flushPromises() - expect(loadOffsets).toEqual([0, 2, 4]) + expect(loadOffsets).toEqual([0, undefined]) expect(liveQuery.toArray.map((item) => item.id)).toEqual([5, 6]) }) }) diff --git a/packages/db/tests/query/load-subset-oracle.property.test.ts b/packages/db/tests/query/load-subset-oracle.property.test.ts index ff535c18fc..186186819a 100644 --- a/packages/db/tests/query/load-subset-oracle.property.test.ts +++ b/packages/db/tests/query/load-subset-oracle.property.test.ts @@ -4,62 +4,21 @@ import { createCollection } from '../../src/collection/index.js' import { createDeferred } from '../../src/deferred.js' import { createOptimisticAction } from '../../src/optimistic-action.js' import { createLiveQueryCollection, eq } from '../../src/query/index.js' -import { DeduplicatedLoadSubset } from '../../src/query/subset-dedupe.js' import { Func, PropRef, Value } from '../../src/query/ir.js' +import { DeduplicatedLoadSubset } from '../../src/query/subset-dedupe.js' import { createTransaction } from '../../src/transactions.js' import { expectAssertionFailure } from '../expected-failure.js' +import { evaluateReferenceExpression } from '../reference-expression.js' import { oracleRandomParameters, readOracleRunConfig, } from '../oracle-config.js' -import { evaluateReferenceExpression } from '../reference-expression.js' import { TraceAssertionError } from '../trace-runner.js' -import type { BasicExpression } from '../../src/query/ir.js' -import type { LoadSubsetOptions, SyncAppliedReceipt } from '../../src/types.js' - -type PredicateSpec = - | { kind: `all` } - | { kind: `eq`; value: number } - | { kind: `in`; values: ReadonlyArray } - | { - kind: `range` - operator: `gt` | `gte` | `lt` | `lte` - value: number - } - | { kind: `and` | `or`; operands: readonly [PredicateSpec, PredicateSpec] } - | { kind: `not`; operand: PredicateSpec } - -type AsyncScenario = { - first: ReadonlyArray - second: ReadonlyArray - firstOutcome: `resolve` | `reject` - secondOutcome: `resolve` | `reject` - deliveryOrder: `forward` | `reverse` - resetBeforeSettlement: boolean -} - -type ConcurrentAsyncScenario = { - requestedValues: ReadonlyArray> - deliveryOrder: `forward` | `reverse` -} - -type RejectedWaiterScenario = { - covering: ReadonlyArray - covered: ReadonlyArray -} - -type RangeOperator = Extract[`operator`] - -type WindowRequest = { - where?: PredicateSpec - orderField?: `none` | `rank` | `score` - direction: `asc` | `desc` - nulls?: `first` | `last` - stringSort?: `lexical` | `locale` - cursorBoundary?: number - offset: number - limit?: number -} +import type { + LoadSubsetOptions, + LoadSubsetRequestResult, + SyncAppliedReceipt, +} from '../../src/types.js' type PersistedLoadRow = { id: string @@ -71,17 +30,27 @@ type OptimisticDerivedRow = { value: string } -type CoverageSubject = { - loadSubset: (options: LoadSubsetOptions) => true | Promise - reset?: () => void +type ExactDemand = { + values: ReadonlyArray + orderField: `rank` | `score` + direction: `asc` | `desc` + nulls: `first` | `last` + stringSort: `lexical` | `locale` + offset: number + limit: number | undefined + cursorBoundary: number | undefined +} + +type ConcurrentExactScenario = { + trace: ReadonlyArray + settlementOrder: `forward` | `reverse` } -type CoverageSubjectFactory = ( - recordLoad: (options: LoadSubsetOptions) => true | Promise, -) => CoverageSubject +const rankRef = new PropRef([`rank`]) +const scoreRef = new PropRef([`score`]) function requirePendingAppliedReceipt( - receipt: SyncAppliedReceipt, + receipt: LoadSubsetRequestResult, ): Promise { if (receipt === true) { throw new Error(`Expected an asynchronous subset load`) @@ -89,996 +58,204 @@ function requirePendingAppliedReceipt( return receipt } -class CoveredDemandRefetchedError extends Error { - constructor( - readonly checkpoint: number, - readonly requested: ReadonlySet, - readonly loadedRegions: ReadonlyArray>, - readonly requestedFingerprint: string, - readonly loadedRegionFingerprints: ReadonlyArray, - ) { - super(`Covered demand refetched at checkpoint ${checkpoint}`) - } -} - -class UncoveredWindowDeduplicatedError extends Error { - constructor( - readonly checkpoint: number, - readonly requested: WindowRequest, - readonly loadedRegions: ReadonlyArray<{ - request: WindowRequest - positions: ReadonlySet - }>, - ) { - super(`Uncovered window deduplicated at checkpoint ${checkpoint}`) - } -} - -class CoveredWindowRefetchedError extends Error { - constructor( - readonly checkpoint: number, - readonly requested: WindowRequest, - readonly requestedPositions: ReadonlySet, - readonly loadedRegions: ReadonlyArray<{ - request: WindowRequest - positions: ReadonlySet - }>, - ) { - super(`Covered window refetched at checkpoint ${checkpoint}`) - } -} - -// The generated predicates only compare against integers from -3 through 3. -// These points cover every distinct truth partition: both unbounded tails, -// every equality point, and every open interval between adjacent thresholds. -const valueDomain = [ - -4, -3, -2.5, -2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2, 2.5, 3, 4, -] as const -const scoreRef = new PropRef([`score`]) -const rankRef = new PropRef([`rank`]) - -const atomicPredicateSpecArbitrary: fc.Arbitrary = fc.oneof( - { weight: 1, arbitrary: fc.constant({ kind: `all` as const }) }, - { - weight: 3, - arbitrary: fc - .integer({ min: -3, max: 3 }) - .map((value) => ({ kind: `eq` as const, value })), - }, - { - weight: 3, - arbitrary: fc - .uniqueArray(fc.integer({ min: -3, max: 3 }), { - minLength: 0, - maxLength: 7, - }) - .map((values) => ({ kind: `in` as const, values })), - }, - { - weight: 4, - arbitrary: fc.record({ - kind: fc.constant(`range` as const), - operator: fc.constantFrom(`gt`, `gte`, `lt`, `lte`), - value: fc.integer({ min: -3, max: 3 }), - }), - }, -) - -function booleanPredicateSpecArbitrary( - operand: fc.Arbitrary, -): fc.Arbitrary { - return fc.oneof( - fc.record({ - kind: fc.constantFrom(`and` as const, `or` as const), - operands: fc.tuple(operand, operand), - }), - operand.map((nested) => ({ kind: `not` as const, operand: nested })), - ) -} - -const shallowPredicateSpecArbitrary = fc.oneof( - atomicPredicateSpecArbitrary, - booleanPredicateSpecArbitrary(atomicPredicateSpecArbitrary), -) - -const predicateSpecArbitrary: fc.Arbitrary = fc.oneof( - { weight: 8, arbitrary: atomicPredicateSpecArbitrary }, - { weight: 2, arbitrary: shallowPredicateSpecArbitrary }, - { - weight: 1, - arbitrary: booleanPredicateSpecArbitrary(shallowPredicateSpecArbitrary), - }, -) - -const requestTraceArbitrary = fc.array(predicateSpecArbitrary, { - minLength: 1, - maxLength: 20, -}) - -const nonEmptyInValuesArbitrary = fc.uniqueArray( - fc.integer({ min: -3, max: 3 }), - { minLength: 1, maxLength: 7 }, -) - -const asyncScenarioArbitrary: fc.Arbitrary = fc.record({ - first: nonEmptyInValuesArbitrary, - second: nonEmptyInValuesArbitrary, - firstOutcome: fc.constantFrom( - `resolve`, - `reject`, - ), - secondOutcome: fc.constantFrom( - `resolve`, - `reject`, - ), - deliveryOrder: fc.constantFrom( - `forward`, - `reverse`, - ), - resetBeforeSettlement: fc.boolean(), -}) - -const concurrentAsyncScenarioArbitrary: fc.Arbitrary = - fc.record({ - requestedValues: fc.array(nonEmptyInValuesArbitrary, { - minLength: 3, +const exactDemandArbitrary: fc.Arbitrary = fc + .record({ + values: fc.uniqueArray(fc.integer({ min: -3, max: 3 }), { + minLength: 1, maxLength: 5, }), - deliveryOrder: fc.constantFrom(`forward`, `reverse`), - }) - -const rejectedWaiterScenarioArbitrary: fc.Arbitrary = - nonEmptyInValuesArbitrary.chain((covering) => - fc - .subarray(covering, { minLength: 1 }) - .map((covered) => ({ covering, covered })), - ) - -const windowRequestArbitrary: fc.Arbitrary> = - fc.record({ - orderField: fc.constantFrom>( - `none`, - `rank`, - `score`, - ), - direction: fc.constantFrom(`asc`, `desc`), - nulls: fc.constantFrom>( - `first`, - `last`, - ), - stringSort: fc.constantFrom>( - `lexical`, - `locale`, - ), - cursorBoundary: fc.option(fc.integer({ min: -3, max: 3 }), { - nil: undefined, - }), - offset: fc.integer({ min: 0, max: 6 }), - limit: fc.option(fc.integer({ min: 0, max: 6 }), { nil: undefined }), - }) - -const finiteWindowRequestArbitrary: fc.Arbitrary> = - fc.record({ - orderField: fc.constantFrom(`none`, `rank`, `score`), - direction: fc.constantFrom(`asc`, `desc`), - nulls: fc.constantFrom(`first`, `last`), - stringSort: fc.constantFrom(`lexical`, `locale`), + orderField: fc.constantFrom(`rank` as const, `score` as const), + direction: fc.constantFrom(`asc` as const, `desc` as const), + nulls: fc.constantFrom(`first` as const, `last` as const), + stringSort: fc.constantFrom(`lexical` as const, `locale` as const), + offset: fc.integer({ min: 0, max: 4 }), + limit: fc.option(fc.integer({ min: 0, max: 5 }), { nil: undefined }), cursorBoundary: fc.option(fc.integer({ min: -3, max: 3 }), { nil: undefined, }), - offset: fc.integer({ min: 0, max: 6 }), - limit: fc.integer({ min: 0, max: 6 }), - }) - -const windowTraceArbitrary = fc - .record({ - where: fc.option(predicateSpecArbitrary, { nil: undefined }), - requests: fc.array(windowRequestArbitrary, { - minLength: 1, - maxLength: 20, - }), }) - .map(({ where, requests }) => - requests.map((request) => ({ ...request, where })), - ) + .map((demand) => ({ + ...demand, + values: [...demand.values].sort((left, right) => left - right), + })) -const distinctWindowWherePairArbitrary = fc - .tuple(predicateSpecArbitrary, predicateSpecArbitrary) - .filter(isDistinctNonEmptyWindowWherePair) - -function isDistinctNonEmptyWindowWherePair([first, second]: readonly [ - PredicateSpec, - PredicateSpec, -]): boolean { - const firstValues = matchingValues(toWhere(first)) - const secondValues = matchingValues(toWhere(second)) - return ( - firstValues.size > 0 && - secondValues.size > 0 && - (!isSubset(firstValues, secondValues) || - !isSubset(secondValues, firstValues)) - ) +function exactDemandFingerprint(demand: ExactDemand): string { + return JSON.stringify(demand) } -const changingWhereWindowTraceArbitrary = fc - .record({ - wherePair: distinctWindowWherePairArbitrary, - first: finiteWindowRequestArbitrary, - second: finiteWindowRequestArbitrary, - rest: fc.array(fc.tuple(fc.boolean(), finiteWindowRequestArbitrary), { - maxLength: 18, - }), +const exactDemandTraceArbitrary = fc + .uniqueArray(exactDemandArbitrary, { + minLength: 1, + maxLength: 6, + selector: exactDemandFingerprint, }) - .map(({ wherePair, first, second, rest }) => [ - { ...first, where: wherePair[0] }, - { ...second, where: wherePair[1] }, - ...rest.map(([useSecond, request]) => ({ - ...request, - where: wherePair[useSecond ? 1 : 0], - })), - ]) - -function toWhere( - predicate: PredicateSpec, -): BasicExpression | undefined { - switch (predicate.kind) { - case `all`: - return undefined - case `eq`: - return new Func(`eq`, [scoreRef, new Value(predicate.value)]) - case `in`: - return new Func(`in`, [scoreRef, new Value([...predicate.values])]) - case `range`: - return new Func(predicate.operator, [ - scoreRef, - new Value(predicate.value), - ]) - case `and`: - case `or`: - return new Func(predicate.kind, predicate.operands.map(toRequiredWhere)) - case `not`: - return new Func(`not`, [toRequiredWhere(predicate.operand)]) - } -} - -function toRequiredWhere(predicate: PredicateSpec): BasicExpression { - return toWhere(predicate) ?? new Value(true) -} - -function matchingValues( - where: BasicExpression | undefined, -): Set { - return new Set( - valueDomain.filter( - (score) => - where === undefined || - evaluateReferenceExpression(where, { score }) === true, - ), + .chain((pool) => + fc + .array(fc.integer({ min: 0, max: pool.length - 1 }), { + minLength: 1, + maxLength: 20, + }) + .map((indices) => indices.map((index) => pool[index]!)), ) -} - -function difference(left: ReadonlySet, right: ReadonlySet) { - return new Set([...left].filter((value) => !right.has(value))) -} -function isSubset(left: ReadonlySet, right: ReadonlySet) { - return [...left].every((value) => right.has(value)) -} - -function unionSets(sets: ReadonlyArray>): Set { - return new Set(sets.flatMap((set) => [...set])) -} - -function expectSetEqual( - actual: ReadonlySet, - expected: ReadonlySet, -): void { - expect([...actual].sort()).toEqual([...expected].sort()) -} - -const createDeduplicatedCoverageSubject: CoverageSubjectFactory = ( - recordLoad, -) => new DeduplicatedLoadSubset({ loadSubset: recordLoad }) - -const createAlwaysLoadingCoverageSubject: CoverageSubjectFactory = ( - recordLoad, -) => ({ loadSubset: recordLoad }) +const concurrentExactScenarioArbitrary: fc.Arbitrary = + exactDemandTraceArbitrary.map((trace) => ({ + trace, + settlementOrder: trace.length % 2 === 0 ? `forward` : `reverse`, + })) -const createRefetchAfterSettlementSubject: CoverageSubjectFactory = ( - recordLoad, -) => { - let hasSettled = false - const dedupe = new DeduplicatedLoadSubset({ loadSubset: recordLoad }) +function toLoadSubsetOptions(demand: ExactDemand): LoadSubsetOptions { + const orderRef = demand.orderField === `rank` ? rankRef : scoreRef return { - loadSubset: (options) => { - if (hasSettled) return recordLoad(options) - const result = dedupe.loadSubset(options) - if (result instanceof Promise) { - void result.then( - () => { - hasSettled = true - }, - () => { - hasSettled = true - }, - ) - } - return result - }, - reset: () => dedupe.reset(), - } -} - -function runCoverageTrace( - trace: ReadonlyArray, - createSubject = createDeduplicatedCoverageSubject, -): void { - const covered = new Set() - const loadedRegions: Array> = [] - const loadedRegionFingerprints: Array = [] - const loads: Array = [] - const subject = createSubject((options) => { - loads.push(options) - return true - }) - - for (const [checkpoint, predicate] of trace.entries()) { - const where = toWhere(predicate) - const requested = matchingValues(where) - const missing = difference(requested, covered) - const loadCountBefore = loads.length - - const result = subject.loadSubset({ where }) - - expect(result).toBe(true) - expect(loads.length - loadCountBefore).toBeLessThanOrEqual(1) - if (loads.length === loadCountBefore) { - expect(missing.size).toBe(0) - } else { - expect(loads).toHaveLength(loadCountBefore + 1) - const loaded = matchingValues(loads.at(-1)?.where) - expectSetEqual(difference(loaded, requested), new Set()) - if (missing.size === 0) { - throw new CoveredDemandRefetchedError( - checkpoint, - requested, - loadedRegions.map((region) => new Set(region)), - JSON.stringify(predicate), - [...loadedRegionFingerprints], - ) - } - expectSetEqual(difference(missing, loaded), new Set()) - for (const value of loaded) covered.add(value) - loadedRegions.push(loaded) - loadedRegionFingerprints.push(JSON.stringify(predicate)) - } - } -} - -function runCoverageTraceWithKnownFailures( - trace: ReadonlyArray, -): void { - try { - runCoverageTrace(trace) - } catch (error) { - if ( - error instanceof CoveredDemandRefetchedError && - (isKnownUnionCompositionRefetch(error) || - isKnownComposedRegionRefetch(error)) - ) { - return - } - throw error - } -} - -function isKnownComposedRegionRefetch( - error: CoveredDemandRefetchedError, -): boolean { - if (error.requested.size === 0 || error.loadedRegions.length <= 1) { - return false - } - - return error.loadedRegions.some((region) => isSubset(error.requested, region)) -} - -function isKnownUnionCompositionRefetch( - error: CoveredDemandRefetchedError, -): boolean { - if (error.requested.size === 0) return true - // The error can only be built after the independent model proves the demand - // is already covered. This classifier is only for coverage formed by - // composing several regions; a request covered by one region is a different - // defect and must not enter this waiver. - if (error.loadedRegions.length > 1) { - const coveredByOneRegion = error.loadedRegions.some((region) => - isSubset(error.requested, region), - ) - return ( - !coveredByOneRegion && - isSubset(error.requested, unionSets(error.loadedRegions)) - ) - } - - const usesCompoundPredicate = [ - error.requestedFingerprint, - ...error.loadedRegionFingerprints, - ].some((fingerprint) => /"kind":"(?:and|or|not)"/.test(fingerprint)) - return ( - usesCompoundPredicate && - error.loadedRegionFingerprints[0] !== error.requestedFingerprint - ) -} - -function countLoads(trace: ReadonlyArray): number { - let loads = 0 - const dedupe = new DeduplicatedLoadSubset({ - loadSubset: () => { - loads++ - return true - }, - }) - for (const predicate of trace) { - dedupe.loadSubset({ where: toWhere(predicate) }) - } - return loads -} - -function readDedupeTrackingState(dedupe: DeduplicatedLoadSubset): { - unlimitedWhere: BasicExpression | undefined - limitedCalls: ReadonlyArray - inflightCalls: ReadonlyArray -} { - return dedupe as unknown as { - unlimitedWhere: BasicExpression | undefined - limitedCalls: ReadonlyArray - inflightCalls: ReadonlyArray - } -} - -function toWindowOptions(request: WindowRequest): LoadSubsetOptions { - const orderField = request.orderField ?? `rank` - const cursorRef = orderField === `score` ? scoreRef : rankRef - return { - where: request.where ? toWhere(request.where) : undefined, - offset: request.offset, - limit: request.limit, + where: new Func(`in`, [scoreRef, new Value([...demand.values])]), + orderBy: [ + { + expression: orderRef, + compareOptions: { + direction: demand.direction, + nulls: demand.nulls, + stringSort: demand.stringSort, + }, + }, + ], + offset: demand.offset, + limit: demand.limit, cursor: - request.cursorBoundary === undefined + demand.cursorBoundary === undefined ? undefined : { - whereFrom: new Func(request.direction === `asc` ? `gt` : `lt`, [ - cursorRef, - new Value(request.cursorBoundary), + whereFrom: new Func(demand.direction === `asc` ? `gt` : `lt`, [ + orderRef, + new Value(demand.cursorBoundary), ]), whereCurrent: new Func(`eq`, [ - cursorRef, - new Value(request.cursorBoundary), + orderRef, + new Value(demand.cursorBoundary), ]), - lastKey: request.cursorBoundary, + lastKey: demand.cursorBoundary, }, - orderBy: - orderField === `none` - ? undefined - : [ - { - expression: orderField === `rank` ? rankRef : scoreRef, - compareOptions: { - direction: request.direction, - nulls: request.nulls ?? `last`, - stringSort: request.stringSort ?? `lexical`, - }, - }, - ], - } -} - -function hasNoWindowDemand(request: WindowRequest): boolean { - return ( - request.limit === 0 || - matchingValues(toWindowOptions(request).where).size === 0 - ) -} - -function windowPositions(request: WindowRequest): Set { - if (hasNoWindowDemand(request)) return new Set() - // The coverage oracle needs a finite universe. Generated finite windows end - // at position 11, so 16 positions preserve every generated subset relation - // while giving an omitted limit an authoritative "through the end" region. - const length = request.limit ?? 16 - request.offset - return new Set(Array.from({ length }, (_, index) => request.offset + index)) -} - -type WindowCoverageDescriptor = { - request: WindowRequest - whereFingerprint: string - orderFingerprint: string | undefined - cursorFingerprint: string | undefined - matching: Set -} - -function describeWindowCoverage( - request: WindowRequest, -): WindowCoverageDescriptor { - const options = toWindowOptions(request) - return { - request, - whereFingerprint: JSON.stringify(options.where), - orderFingerprint: options.orderBy - ? JSON.stringify(options.orderBy) - : undefined, - cursorFingerprint: options.cursor - ? JSON.stringify(options.cursor) - : undefined, - matching: matchingValues(options.where), - } -} - -function describedWindowCovers( - requested: WindowCoverageDescriptor, - loaded: WindowCoverageDescriptor, -): boolean { - if ( - loaded.request.limit === undefined && - loaded.request.offset === 0 && - loaded.cursorFingerprint === undefined && - isSubset(requested.matching, loaded.matching) - ) { - return true - } - if (requested.cursorFingerprint !== loaded.cursorFingerprint) { - return false - } - if (requested.whereFingerprint !== loaded.whereFingerprint) return false - if (requested.orderFingerprint === undefined) return true - return requested.orderFingerprint === loaded.orderFingerprint -} - -function loadedWindowCovers( - requested: WindowRequest, - loaded: WindowRequest, -): boolean { - return describedWindowCovers( - describeWindowCoverage(requested), - describeWindowCoverage(loaded), - ) -} - -function isKnownCompareOptionsDeduplication( - error: UncoveredWindowDeduplicatedError, -): boolean { - const requestedOptions = toWindowOptions(error.requested) - const requestedOrder = requestedOptions.orderBy?.[0] - if (!requestedOrder) return false - const requestedPositions = windowPositions(error.requested) - - return error.loadedRegions.some(({ request: loaded, positions }) => { - const loadedOptions = toWindowOptions(loaded) - const loadedOrder = loadedOptions.orderBy?.[0] - return ( - loadedOrder !== undefined && - JSON.stringify(requestedOptions.where) === - JSON.stringify(loadedOptions.where) && - JSON.stringify(requestedOrder.expression) === - JSON.stringify(loadedOrder.expression) && - requestedOrder.compareOptions.direction === - loadedOrder.compareOptions.direction && - (requestedOrder.compareOptions.nulls !== - loadedOrder.compareOptions.nulls || - requestedOrder.compareOptions.stringSort !== - loadedOrder.compareOptions.stringSort) && - isSubset(requestedPositions, positions) - ) - }) -} - -function isKnownUnlimitedOffsetDeduplication( - error: UncoveredWindowDeduplicatedError, -): boolean { - const requestedOptions = toWindowOptions(error.requested) - - return error.loadedRegions.some(({ request: loaded }) => { - if (loaded.limit !== undefined || loaded.offset <= error.requested.offset) { - return false - } - const loadedOptions = toWindowOptions(loaded) - return isSubset( - matchingValues(requestedOptions.where), - matchingValues(loadedOptions.where), - ) - }) -} - -function isKnownOffsetTruncatedUnlimitedDeduplication( - error: UncoveredWindowDeduplicatedError, -): boolean { - const unlimitedLoads = error.loadedRegions.filter( - ({ request }) => request.limit === undefined, - ) - const offsetLoads = unlimitedLoads.filter(({ request }) => request.offset > 0) - if (offsetLoads.length === 0) return false - - // The known defect stores unlimited predicate coverage without its offset or - // ordering. Model that loss directly instead of replaying production dedupe. - if (offsetLoads.some(({ request }) => request.where === undefined)) { - return true } - if (error.requested.where === undefined) return false - - const incorrectlyTrackedValues = unionSets( - offsetLoads.map(({ request }) => - matchingValues(toWindowOptions(request).where), - ), - ) - return isSubset( - matchingValues(toWindowOptions(error.requested).where), - incorrectlyTrackedValues, - ) -} - -function isKnownCoveredWindowRefetch( - error: CoveredWindowRefetchedError, -): boolean { - if ( - error.requestedPositions.size === 0 && - hasNoWindowDemand(error.requested) - ) { - return true - } - if (error.loadedRegions.length > 1) { - if ( - !isSubset( - error.requestedPositions, - unionSets(error.loadedRegions.map(({ positions }) => positions)), - ) - ) { - return false - } - const coveredByOneRegion = error.loadedRegions.some(({ positions }) => - isSubset(error.requestedPositions, positions), - ) - return !coveredByOneRegion - } - if (error.requested.where === undefined) return false - - return error.loadedRegions.some( - ({ request: loaded, positions }) => - loadedWindowCovers(error.requested, loaded) && - isSubset(error.requestedPositions, positions), - ) } -function isKnownIndividuallyCoveredWindowRefetch( - error: CoveredWindowRefetchedError, -): boolean { - if (error.loadedRegions.length <= 1) return false - const coveredByOneRegion = error.loadedRegions.some( - ({ request: loaded, positions }) => - loadedWindowCovers(error.requested, loaded) && - isSubset(error.requestedPositions, positions), - ) - if (!coveredByOneRegion) return false - - const replay = [ - ...error.loadedRegions.map(({ request }) => request), - error.requested, - ] - return countWindowLoads(replay) === replay.length -} - -const createWindowKeyBlindSubject: CoverageSubjectFactory = (recordLoad) => { - const coveredWindows = new Set() - return { +function assertCompletedExactDemandTrace( + trace: ReadonlyArray, +): void { + let starts = 0 + const completed = new Set() + let expectedStart: LoadSubsetOptions | undefined + const dedupe = new DeduplicatedLoadSubset({ loadSubset: (options) => { - const key = JSON.stringify({ - offset: options.offset ?? 0, - limit: options.limit, - }) - if (coveredWindows.has(key)) return true - coveredWindows.add(key) - return recordLoad(options) + expect(options).toEqual(expectedStart) + starts++ + return true }, + }) + + for (const demand of trace) { + const startsBefore = starts + expectedStart = toLoadSubsetOptions(demand) + const result = dedupe.loadSubset(expectedStart) + const fingerprint = exactDemandFingerprint(demand) + expect(result).toBe(true) + expect(starts - startsBefore).toBe(completed.has(fingerprint) ? 0 : 1) + completed.add(fingerprint) } } -function runWindowCoverageTrace( - trace: ReadonlyArray, - createSubject = createDeduplicatedCoverageSubject, -): void { - const loadedRegions: Array<{ - request: WindowRequest - positions: Set - coverage: WindowCoverageDescriptor +async function assertConcurrentExactDemandTrace({ + trace, + settlementOrder, +}: ConcurrentExactScenario): Promise { + const transports: Array<{ + deferred: ReturnType> + promise: Promise }> = [] - const loads: Array = [] - const subject = createSubject((options) => { - loads.push(options) - return true + const promisesByDemand = new Map>() + const dedupe = new DeduplicatedLoadSubset({ + loadSubset: () => { + const deferred = createDeferred() + const transport = { deferred, promise: deferred.promise } + transports.push(transport) + return transport.promise + }, }) - for (const [checkpoint, request] of trace.entries()) { - const requested = windowPositions(request) - const requestedCoverage = describeWindowCoverage(request) - const compatibleRegions = loadedRegions.filter(({ coverage }) => - describedWindowCovers(requestedCoverage, coverage), - ) - const covered = new Set( - compatibleRegions.flatMap(({ positions }) => [...positions]), - ) - const missing = difference(requested, covered) - const callsBefore = loads.length - - subject.loadSubset(toWindowOptions(request)) - - expect(loads.length - callsBefore).toBeLessThanOrEqual(1) - if (loads.length === callsBefore) { - if (missing.size > 0) { - throw new UncoveredWindowDeduplicatedError( - checkpoint, - request, - loadedRegions.map(({ request: loaded, positions }) => ({ - request: { ...loaded }, - positions: new Set(positions), - })), - ) - } + const callers = trace.map((demand) => { + const fingerprint = exactDemandFingerprint(demand) + const startsBefore = transports.length + const result = dedupe.loadSubset(toLoadSubsetOptions(demand)) + if (!(result instanceof Promise)) { + throw new Error(`A new in-flight demand must return a promise`) + } + const existing = promisesByDemand.get(fingerprint) + if (existing) { + expect(transports).toHaveLength(startsBefore) + expect(result).toBe(existing) } else { - const loaded = loads.at(-1)! - expect(loaded).toEqual(toWindowOptions(request)) - if (missing.size === 0) { - throw new CoveredWindowRefetchedError( - checkpoint, - { ...request }, - new Set(requested), - compatibleRegions.map(({ request: previous, positions }) => ({ - request: { ...previous }, - positions: new Set(positions), - })), - ) - } - for (const position of requested) covered.add(position) - loadedRegions.push({ - request: { ...request }, - positions: requested, - coverage: requestedCoverage, - }) + expect(transports).toHaveLength(startsBefore + 1) + promisesByDemand.set(fingerprint, result) } - } -} + return result + }) -function runWindowCoverageTraceWithKnownFailures( - trace: ReadonlyArray, -): void { - try { - runWindowCoverageTrace(trace) - } catch (error) { - if ( - error instanceof UncoveredWindowDeduplicatedError && - (isKnownCompareOptionsDeduplication(error) || - isKnownUnlimitedOffsetDeduplication(error) || - isKnownOffsetTruncatedUnlimitedDeduplication(error)) - ) { - return - } - if ( - error instanceof CoveredWindowRefetchedError && - (isKnownCoveredWindowRefetch(error) || - isKnownIndividuallyCoveredWindowRefetch(error)) - ) { - return - } - throw error + const observed = Promise.allSettled(callers) + const settlement = + settlementOrder === `forward` ? transports : [...transports].reverse() + for (const transport of settlement) transport.deferred.resolve() + expect((await observed).every(({ status }) => status === `fulfilled`)).toBe( + true, + ) + + const startsAfterSettlement = transports.length + for (const demand of trace) { + expect(dedupe.loadSubset(toLoadSubsetOptions(demand))).toBe(true) } + expect(transports).toHaveLength(startsAfterSettlement) + + dedupe.reset() + const restarted = dedupe.loadSubset(toLoadSubsetOptions(trace[0]!)) + expect(restarted).toBeInstanceOf(Promise) + expect(transports).toHaveLength(startsAfterSettlement + 1) + transports.at(-1)!.deferred.resolve() + await restarted } -function countWindowLoads(trace: ReadonlyArray): number { - let loads = 0 +async function expectExactWaitersShareRejection(): Promise { + const deferred = createDeferred() + void deferred.promise.catch(() => undefined) const dedupe = new DeduplicatedLoadSubset({ - loadSubset: () => { - loads++ - return true - }, + loadSubset: () => deferred.promise, }) - for (const request of trace) dedupe.loadSubset(toWindowOptions(request)) - return loads -} - -function expectDistinctWhereStartsDistinctLimitedWindowLoads( - predicates: readonly [PredicateSpec, PredicateSpec], -): void { - const createRequest = (where: PredicateSpec): WindowRequest => ({ - where, + const demand: ExactDemand = { + values: [1, 2], orderField: `rank`, direction: `asc`, nulls: `last`, stringSort: `lexical`, offset: 0, limit: 2, - }) - expect(countWindowLoads(predicates.map(createRequest))).toBe(2) -} - -function predicateDepth(predicate: PredicateSpec): number { - if (predicate.kind === `and` || predicate.kind === `or`) { - return 1 + Math.max(...predicate.operands.map(predicateDepth)) + cursorBoundary: undefined, } - if (predicate.kind === `not`) return 1 + predicateDepth(predicate.operand) - return 1 -} -async function runAsyncScenario( - scenario: AsyncScenario, - createSubject: CoverageSubjectFactory = createDeduplicatedCoverageSubject, -): Promise { - const requests: Array<{ - options: LoadSubsetOptions - deferred: ReturnType> - }> = [] - const subject = createSubject((options) => { - const deferred = createDeferred() - // The source promise is intentionally rejectable. Observe it directly as - // well as through the dedupe wrapper so Vitest never mistakes a generated - // transport rejection for an unhandled test error. - void deferred.promise.catch(() => undefined) - requests.push({ options, deferred }) - return deferred.promise - }) + const first = dedupe.loadSubset(toLoadSubsetOptions(demand)) + const second = dedupe.loadSubset(toLoadSubsetOptions(demand)) + expect(first).toBeInstanceOf(Promise) + expect(second).toBe(first) - const firstResult = subject.loadSubset({ - where: toWhere({ kind: `in`, values: scenario.first }), - }) - const secondResult = subject.loadSubset({ - where: toWhere({ kind: `in`, values: scenario.second }), - }) - expect(firstResult).toBeInstanceOf(Promise) - expect(secondResult).toBeInstanceOf(Promise) - if (!(firstResult instanceof Promise) || !(secondResult instanceof Promise)) { - throw new Error(`Initial async requests must return promises`) - } - - const firstSet = new Set(scenario.first) - const secondSet = new Set(scenario.second) - const secondCoveredByFirst = isSubset(secondSet, firstSet) - expect(requests).toHaveLength(secondCoveredByFirst ? 1 : 2) - expect(firstResult === secondResult).toBe(secondCoveredByFirst) - - if (scenario.resetBeforeSettlement) subject.reset?.() - - const outcomes = [scenario.firstOutcome, scenario.secondOutcome] as const - const deliveryIndices = - scenario.deliveryOrder === `forward` - ? requests.map((_, index) => index) - : requests.map((_, index) => index).reverse() - const callerOutcomePromise = Promise.allSettled([firstResult, secondResult]) - for (const index of deliveryIndices) { - const request = requests[index]! - const outcome = outcomes[index]! - if (outcome === `resolve`) request.deferred.resolve() - else request.deferred.reject(new Error(`request ${index} failed`)) - } - - const callerOutcomes = await callerOutcomePromise - const expectedFirstStatus = - scenario.firstOutcome === `resolve` ? `fulfilled` : `rejected` - const expectedSecondStatus = secondCoveredByFirst - ? expectedFirstStatus - : scenario.secondOutcome === `resolve` - ? `fulfilled` - : `rejected` - expect(callerOutcomes.map(({ status }) => status)).toEqual([ - expectedFirstStatus, - expectedSecondStatus, + const outcomes = Promise.allSettled([first, second]) + deferred.reject(new Error(`transport failed`)) + expect((await outcomes).map(({ status }) => status)).toEqual([ + `rejected`, + `rejected`, ]) - const successfullyCovered = new Set() - if (!scenario.resetBeforeSettlement) { - if (scenario.firstOutcome === `resolve`) { - for (const value of firstSet) successfullyCovered.add(value) - } - if (!secondCoveredByFirst && scenario.secondOutcome === `resolve`) { - for (const value of secondSet) successfullyCovered.add(value) - } - } - - const callsBeforeRetry = requests.length - const retry = subject.loadSubset({ - where: toWhere({ kind: `in`, values: scenario.second }), - }) - const retryWasCovered = isSubset(secondSet, successfullyCovered) - if (retry === true) { - expect(retryWasCovered).toBe(true) - expect(retry).toBe(true) - expect(requests).toHaveLength(callsBeforeRetry) - } else { - try { - expect(retryWasCovered).toBe(false) - } catch (error) { - throw new TraceAssertionError(2, error) - } - expect(retry).toBeInstanceOf(Promise) - expect(requests).toHaveLength(callsBeforeRetry + 1) - const retriedValues = matchingValues(requests.at(-1)?.options.where) - const missingRetryValues = difference(secondSet, successfullyCovered) - expectSetEqual(difference(missingRetryValues, retriedValues), new Set()) - expectSetEqual(difference(retriedValues, secondSet), new Set()) - requests.at(-1)?.deferred.resolve() - await retry - } -} - -async function runConcurrentAsyncScenario( - scenario: ConcurrentAsyncScenario, -): Promise { - const transports: Array<{ - values: Set - deferred: ReturnType> - result?: Promise - }> = [] - const subject = createDeduplicatedCoverageSubject((options) => { - const deferred = createDeferred() - transports.push({ values: matchingValues(options.where), deferred }) - return deferred.promise - }) - const callerResults: Array> = [] - - for (const values of scenario.requestedValues) { - const requested = new Set(values) - const coveringIndex = transports.findIndex(({ values: loaded }) => - isSubset(requested, loaded), - ) - const transportCount = transports.length - const result = subject.loadSubset({ - where: toWhere({ kind: `in`, values }), - }) - expect(result).toBeInstanceOf(Promise) - if (!(result instanceof Promise)) { - throw new Error(`Concurrent async requests must remain pending`) - } - callerResults.push(result) - - if (coveringIndex === -1) { - expect(transports).toHaveLength(transportCount + 1) - transports.at(-1)!.result = result - } else { - expect(transports).toHaveLength(transportCount) - expect(result).toBe(transports[coveringIndex]!.result) - } - } - - const delivery = - scenario.deliveryOrder === `forward` - ? transports - : [...transports].reverse() - for (const { deferred } of delivery) deferred.resolve() - await Promise.all(callerResults) -} - -async function runAsyncScenarioWithKnownFailures( - scenario: AsyncScenario, -): Promise { - try { - await runAsyncScenario(scenario) - } catch (error) { - if ( - error instanceof TraceAssertionError && - error.checkpoint === 2 && - !scenario.resetBeforeSettlement && - scenario.firstOutcome === `resolve` && - scenario.secondOutcome === `resolve` && - !isSubset(new Set(scenario.second), new Set(scenario.first)) - ) { - return - } - throw error - } + const retry = dedupe.loadSubset(toLoadSubsetOptions(demand)) + expect(retry).toBeInstanceOf(Promise) + await expect(retry).rejects.toThrow(`transport failed`) } -const { multiplier, replaySeed } = readOracleRunConfig() -const coverageScenarioRuns = 40 * multiplier -const coverageRandomParameters = oracleRandomParameters( - coverageScenarioRuns, - replaySeed, -) +const { multiplier, ...replay } = readOracleRunConfig() +const exactScenarioRuns = 40 * multiplier let collectionSequence = 0 @@ -1334,7 +511,7 @@ async function expectAppliedLoadDoesNotFlushEarlierParkedSync() { } } -async function expectCoverageWaitsForAppliedRows() { +async function expectCompletionWaitsForAppliedRows() { let publishUnrelated!: () => void let transportCalls = 0 const source = createCollection({ @@ -1562,7 +739,7 @@ async function expectLaterImmediateCommitSettlesAppliedSubset() { } } -async function expectAbortedReceiptDoesNotPublishCoverage( +async function expectAbortedReceiptDoesNotSettleDemand( abortPhase: `before-commit` | `while-parked`, ) { let transportCalls = 0 @@ -1753,7 +930,7 @@ async function expectCanceledReceiptReleasesOnlyItsSuppression() { } } -async function expectCleanupRejectsReceiptOnce() { +async function expectCleanupRejectsDemandOnce() { let receipt!: Promise let transportCalls = 0 const deduplicated = new DeduplicatedLoadSubset({ @@ -1891,80 +1068,8 @@ async function expectDerivedSyncDuringOptimisticMutation(): Promise { } } -async function expectDeduplicatedWaiterHandlesRejection( - scenario: RejectedWaiterScenario, -): Promise { - const detachedBranches: Array> = [] - class LocallyTrackedPromise extends Promise { - catch( - onRejected?: ((reason: unknown) => TResult | PromiseLike) | null, - ): Promise { - const branch = super.catch(onRejected) - detachedBranches.push(branch) - return branch - } - } - - let rejectSource!: (reason?: unknown) => void - const sourcePromise = new LocallyTrackedPromise((_resolve, reject) => { - rejectSource = reject - }) - const dedupe = new DeduplicatedLoadSubset({ - loadSubset: () => sourcePromise, - }) - - const first = dedupe.loadSubset({ - where: toWhere({ kind: `in`, values: scenario.covering }), - }) - const second = dedupe.loadSubset({ - where: toWhere({ kind: `in`, values: scenario.covered }), - }) - if (!(first instanceof Promise) || !(second instanceof Promise)) { - throw new Error(`Both callers must wait for the in-flight request`) - } - - const callerOutcomes = Promise.allSettled([first, second]) - const detachedOutcomes = Promise.allSettled(detachedBranches) - rejectSource(new Error(`transport failed`)) - expect((await callerOutcomes).map(({ status }) => status)).toEqual([ - `rejected`, - `rejected`, - ]) - - try { - expect({ - branchCount: detachedBranches.length, - statuses: (await detachedOutcomes).map(({ status }) => status), - }).toEqual({ branchCount: 1, statuses: [`fulfilled`] }) - } catch (error) { - throw new TraceAssertionError(0, error) - } -} - -function expectExactCountFailure( - count: () => number, - actual: number, - expected: number, -): () => Promise { - return expectAssertionFailure( - () => - Promise.resolve().then(() => { - try { - expect(count()).toBe(expected) - } catch (error) { - throw new TraceAssertionError(0, error) - } - }), - { - checkpoint: 0, - classify: ({ actual: received, expected: wanted }) => - received === actual && wanted === expected, - }, - ) -} - -describe(`loadSubset coverage oracle`, () => { - it(`orders a missing reference path with null`, () => { +describe(`exact loadSubset demand oracle`, () => { + it(`uses SQL unknown for nullish comparisons in the independent model`, () => { const missing = new PropRef([`missing`]) expect( @@ -1972,749 +1077,79 @@ describe(`loadSubset coverage oracle`, () => { new Func(`lte`, [missing, new Value(null)]), {}, ), - ).toBe(true) + ).toBeNull() expect( evaluateReferenceExpression(new Func(`lt`, [missing, new Value(0)]), {}), - ).toBe(true) - }) - - it(`rejects one-region coverage from the union-composition classifier`, () => { - expect( - isKnownUnionCompositionRefetch( - new CoveredDemandRefetchedError( - 2, - new Set([1]), - [new Set([1]), new Set([2])], - JSON.stringify({ kind: `eq`, value: 1 }), - [ - JSON.stringify({ kind: `eq`, value: 1 }), - JSON.stringify({ kind: `eq`, value: 2 }), - ], - ), - ), - ).toBe(false) - }) - - it(`classifies a composed state that forgets one loaded region separately`, () => { - const error = new CoveredDemandRefetchedError( - 2, - new Set([2]), - [new Set([0]), new Set([2])], - JSON.stringify({ kind: `eq`, value: 2 }), - [ - JSON.stringify({ kind: `in`, values: [0] }), - JSON.stringify({ kind: `in`, values: [2] }), - ], - ) - - expect(isKnownUnionCompositionRefetch(error)).toBe(false) - expect(isKnownComposedRegionRefetch(error)).toBe(true) - }) - - it(`rejects an uncovered window from the union classifier`, () => { - const request: WindowRequest = { - direction: `asc`, - offset: 2, - limit: 1, - } - expect( - isKnownCoveredWindowRefetch( - new CoveredWindowRefetchedError(2, request, new Set([2]), [ - { - request: { direction: `asc`, offset: 0, limit: 1 }, - positions: new Set([0]), - }, - { - request: { direction: `asc`, offset: 1, limit: 1 }, - positions: new Set([1]), - }, - ]), - ), - ).toBe(false) + ).toBeNull() }) - it(`generates window histories that change predicates`, () => { - const traces = fc.sample(changingWhereWindowTraceArbitrary, { - seed: 1750, - numRuns: 100, + it(`generates repeated, cursor, empty, and unbounded exact demands`, () => { + const traces = fc.sample(exactDemandTraceArbitrary, { + seed: 1656, + numRuns: 200, }) + const demands = traces.flat() expect( traces.some( (trace) => - new Set(trace.map(({ where }) => JSON.stringify(where))).size > 1, + new Set(trace.map(exactDemandFingerprint)).size < trace.length, ), ).toBe(true) - }) - - it(`generates nested boolean predicates`, () => { - const predicates = fc.sample(predicateSpecArbitrary, { - seed: 1751, - numRuns: 500, - }) - - expect(predicates.some((predicate) => predicateDepth(predicate) >= 3)).toBe( - true, - ) - }) - - it(`generates rejected requests shared by a covered waiter`, () => { - const scenarios = fc.sample(asyncScenarioArbitrary, { - seed: 1752, - numRuns: 500, - }) - expect( - scenarios.some( - (scenario) => - scenario.firstOutcome === `reject` && - scenario.second.every((value) => scenario.first.includes(value)), - ), + demands.some(({ cursorBoundary }) => cursorBoundary !== undefined), ).toBe(true) + expect(demands.some(({ limit }) => limit === 0)).toBe(true) + expect(demands.some(({ limit }) => limit === undefined)).toBe(true) + expect(new Set(demands.map(({ offset }) => offset)).size).toBeGreaterThan(1) }) - it(`rejects unrelated offset loss from the truncated-unlimited classifier`, () => { - expect( - isKnownOffsetTruncatedUnlimitedDeduplication( - new UncoveredWindowDeduplicatedError( - 1, - { - where: { kind: `eq`, value: 1 }, - direction: `asc`, - offset: 0, - limit: 1, - }, - [ - { - request: { - where: { kind: `eq`, value: 2 }, - direction: `asc`, - offset: 1, - limit: undefined, - }, - positions: new Set([1, 2]), - }, - ], - ), - ), - ).toBe(false) - }) - - it(`keeps empty predicates out of the distinct-window corpus`, () => { - expect( - isDistinctNonEmptyWindowWherePair([ - { kind: `in`, values: [] }, - { kind: `eq`, value: 0 }, - ]), - ).toBe(false) - }) - - it( - `discovered trace: an empty predicate issues no transport work`, - expectExactCountFailure( - () => countLoads([{ kind: `in`, values: [] }]), - 1, - 0, - ), - ) - - it( - `discovered trace: an empty ordered window issues no transport work`, - expectExactCountFailure( - () => countWindowLoads([{ direction: `asc`, offset: 0, limit: 0 }]), - 1, - 0, - ), - ) - - it( - `discovered trace: an empty filtered window issues no transport work`, - expectExactCountFailure( - () => - countWindowLoads([ - { - where: { kind: `in`, values: [] }, - direction: `asc`, - offset: 0, - limit: 1, - }, - ]), - 1, - 0, - ), - ) - - it( - `discovered trace: a contradictory filtered window issues no transport work`, - expectExactCountFailure( - () => - countWindowLoads([ - { - where: { - kind: `and`, - operands: [ - { kind: `eq`, value: 0 }, - { kind: `eq`, value: 1 }, - ], - }, - direction: `asc`, - offset: 0, - limit: 1, - }, - ]), - 1, - 0, - ), - ) - - it( - `discovered trace: widening an unlimited offset starts another load`, - expectAssertionFailure( - () => - Promise.resolve().then(() => { - expect( - countWindowLoads([ - { direction: `asc`, offset: 1, limit: undefined }, - { direction: `asc`, offset: 0, limit: undefined }, - ]), - ).toBe(2) - }), - { message: /expected 1 to be/ }, - ), - ) - - it( - `discovered trace: an offset-truncated unlimited load does not cover another ordering`, - expectAssertionFailure( - () => - Promise.resolve().then(() => { - expect( - countWindowLoads([ - { - orderField: `rank`, - direction: `asc`, - offset: 1, - limit: undefined, - }, - { - orderField: `score`, - direction: `asc`, - offset: 1, - limit: 1, - }, - ]), - ).toBe(2) - }), - { message: /expected 1 to be 2/ }, - ), - ) - - it( - `discovered trace: an offset-truncated unfiltered load does not cover a filtered request`, - expectExactCountFailure( - () => - countWindowLoads([ - { - direction: `asc`, - offset: 1, - limit: undefined, - }, - { - where: { kind: `not`, operand: { kind: `eq`, value: 0 } }, - direction: `asc`, - offset: 1, - limit: undefined, - }, - ]), - 1, - 2, - ), - ) - - it(`discovered trace: an identical filtered window reuses its load`, () => { - const request: WindowRequest = { - where: { kind: `in`, values: [0] }, - orderField: `none`, - direction: `asc`, - offset: 0, - limit: 1, - } - expect(countWindowLoads([request, request])).toBe(1) - }) - - it(`discovered trace: distinct cursor pages start distinct loads`, () => { - const request: WindowRequest = { - orderField: `rank`, - direction: `asc`, - offset: 0, - limit: 2, - cursorBoundary: 1, - } - - runWindowCoverageTrace([ - request, - { ...request, cursorBoundary: 2 }, - request, - ]) - }) - - it(`discovered trace: a cursor without a limit is not full coverage`, () => { - const request: WindowRequest = { - orderField: `rank`, - direction: `asc`, - offset: 0, - limit: undefined, - cursorBoundary: 1, - } - - runWindowCoverageTrace([ - request, - { ...request, cursorBoundary: 2 }, - { ...request, cursorBoundary: undefined }, - ]) - }) - - it(`rejects repeated transport work for one covered predicate`, () => { - expect(() => - runCoverageTrace( - [ - { kind: `eq`, value: 1 }, - { kind: `eq`, value: 1 }, - ], - createAlwaysLoadingCoverageSubject, - ), - ).toThrow() - }) - - it(`reuses transport work for repeated and strictly covered predicates`, () => { - runCoverageTrace([ - { kind: `range`, operator: `gte`, value: 0 }, - ...Array.from( - { length: 20 }, - (): PredicateSpec => ({ kind: `eq`, value: 1 }), - ), - ]) - }) - - it(`keeps tracking bounded across repeated covered demand`, () => { - const dedupe = new DeduplicatedLoadSubset({ loadSubset: () => true }) - const request: LoadSubsetOptions = { - where: toWhere({ kind: `range`, operator: `gte`, value: 0 }), - offset: 0, - limit: 4, - } - - dedupe.loadSubset(request) - for (let index = 0; index < 20; index++) dedupe.loadSubset(request) - - const state = readDedupeTrackingState(dedupe) - expect(state.limitedCalls).toHaveLength(1) - expect(state.inflightCalls).toHaveLength(0) - }) - - it(`rejects transport work for a strict covered predicate subset`, () => { - expect(() => - runCoverageTrace( - [ - { kind: `range`, operator: `gte`, value: 0 }, - { kind: `eq`, value: 1 }, - ], - createAlwaysLoadingCoverageSubject, - ), - ).toThrow() - }) - - it( - `discovered trace: a covered compound predicate issues no second load`, - expectAssertionFailure( - () => - Promise.resolve().then(() => { - try { - expect( - countLoads([ - { - kind: `and`, - operands: [ - { kind: `range`, operator: `gte`, value: 0 }, - { kind: `not`, operand: { kind: `eq`, value: 2 } }, - ], - }, - { - kind: `or`, - operands: [ - { kind: `eq`, value: 1 }, - { kind: `eq`, value: 3 }, - ], - }, - ]), - ).toBe(1) - } catch (error) { - throw new TraceAssertionError(0, error) - } - }), - { - checkpoint: 0, - classify: ({ actual, expected }) => actual === 2 && expected === 1, - }, - ), - ) - - it( - `discovered trace: a composed predicate state forgets one loaded region`, - expectExactCountFailure( - () => - countLoads([ - { kind: `in`, values: [0] }, - { kind: `in`, values: [2] }, - { kind: `eq`, value: 2 }, - ]), - 3, - 2, - ), - ) - - it(`rejects repeated transport work for one identical compound predicate`, () => { - const predicate: PredicateSpec = { - kind: `and`, - operands: [ - { kind: `range`, operator: `gte`, value: 0 }, - { kind: `not`, operand: { kind: `eq`, value: 2 } }, - ], - } - expect(() => - runCoverageTrace( - [predicate, predicate], - createAlwaysLoadingCoverageSubject, - ), - ).toThrow() - }) - - it(`rejects repeated transport work for a covered compound predicate`, () => { - const covering: PredicateSpec = { - kind: `and`, - operands: [ - { kind: `range`, operator: `gte`, value: 0 }, - { kind: `not`, operand: { kind: `eq`, value: 2 } }, - ], - } - const covered: PredicateSpec = { - kind: `or`, - operands: [ - { kind: `eq`, value: 1 }, - { kind: `eq`, value: 3 }, - ], - } - - expect(() => - runCoverageTrace([covering, covered], createAlwaysLoadingCoverageSubject), - ).toThrow() - }) - - it(`rejects repeated transport work for one covered window`, () => { - expect(() => - runWindowCoverageTrace( - [ - { direction: `asc`, offset: 1, limit: 2 }, - { direction: `asc`, offset: 1, limit: 2 }, - ], - createAlwaysLoadingCoverageSubject, - ), - ).toThrow() - }) - - it(`reuses transport work for repeated and strictly covered windows`, () => { - runWindowCoverageTrace([ - { direction: `asc`, offset: 0, limit: 4 }, - ...Array.from( - { length: 20 }, - (): WindowRequest => ({ direction: `asc`, offset: 1, limit: 2 }), - ), - ]) - }) - - it(`reuses an unlimited load across local orderings`, () => { - runWindowCoverageTrace([ - { - orderField: `none`, - direction: `asc`, - offset: 0, - limit: undefined, - }, - { - where: { kind: `range`, operator: `gt`, value: 0 }, - orderField: `score`, - direction: `desc`, - nulls: `first`, - stringSort: `locale`, - offset: 0, - limit: undefined, - }, - ]) - }) - - it(`does not treat an offset-truncated unlimited load as complete under another ordering`, () => { - expect( - loadedWindowCovers( - { - orderField: `score`, - direction: `asc`, - offset: 1, - limit: 1, - }, - { - orderField: `rank`, - direction: `asc`, - offset: 1, - limit: undefined, - }, - ), - ).toBe(false) - }) - - it(`rejects redundant work for a window covered by one loaded region`, () => { - const first: WindowRequest = { - direction: `asc`, - offset: 0, - limit: 2, - } - expect(() => - runWindowCoverageTrace( - [first, { direction: `asc`, offset: 2, limit: 2 }, first], - createAlwaysLoadingCoverageSubject, - ), - ).toThrow() - }) - - it.each([ - [ - `where`, - { - where: { kind: `eq`, value: 2 }, - orderField: `rank`, - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - offset: 0, - limit: 2, - }, - ], - [ - `order expression`, - { - where: { kind: `eq`, value: 1 }, - orderField: `score`, - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - offset: 0, - limit: 2, - }, - ], - [ - `null placement`, - { - where: { kind: `eq`, value: 1 }, - orderField: `rank`, - direction: `asc`, - nulls: `first`, - stringSort: `lexical`, - offset: 0, - limit: 2, - }, - ], - [ - `string ordering`, - { - where: { kind: `eq`, value: 1 }, - orderField: `rank`, - direction: `asc`, - nulls: `last`, - stringSort: `locale`, - offset: 0, - limit: 2, - }, - ], - ] satisfies ReadonlyArray)( - `does not reuse window coverage across a different %s`, - (_name, changedRequest) => { - const baseRequest: WindowRequest = { - where: { kind: `eq`, value: 1 }, - orderField: `rank`, - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - offset: 0, - limit: 2, - } - expect(() => - runWindowCoverageTrace( - [baseRequest, changedRequest], - createWindowKeyBlindSubject, - ), - ).toThrow() - }, - ) - - it.each([ - [ - `null placement`, - { nulls: `first`, stringSort: `lexical` }, - { nulls: `last`, stringSort: `lexical` }, - ], - [ - `string ordering`, - { nulls: `first`, stringSort: `lexical` }, - { nulls: `first`, stringSort: `locale` }, - ], - ] as const)( - `discovered trace: a different %s starts a distinct window load`, - (_name, firstOptions, secondOptions) => { - const createRequest = ( - compareOptions: typeof firstOptions | typeof secondOptions, - ): WindowRequest => ({ - direction: `asc`, - orderField: `rank`, - offset: 0, - limit: 1, - ...compareOptions, - }) - expect( - countWindowLoads([ - createRequest(firstOptions), - createRequest(secondOptions), - ]), - ).toBe(2) - }, - ) - - it(`rejects async transport work after coverage settles`, async () => { - await expect( - runAsyncScenario( - { - first: [1], - second: [1], - firstOutcome: `resolve`, - secondOutcome: `resolve`, - deliveryOrder: `forward`, - resetBeforeSettlement: false, - }, - createRefetchAfterSettlementSubject, - ), - ).rejects.toThrow() - }) - - it(`discovered trace: settled predicate regions cover their union`, async () => { - await expectAssertionFailure(runAsyncScenario, { - checkpoint: 2, - classify: ({ actual, expected }) => actual === true && expected === false, - })({ - first: [0], - second: [1], - firstOutcome: `resolve`, - secondOutcome: `resolve`, - deliveryOrder: `forward`, - resetBeforeSettlement: false, - }) - }) - - fcTest.prop([requestTraceArbitrary], { - numRuns: coverageScenarioRuns, + fcTest.prop([exactDemandTraceArbitrary], { + numRuns: exactScenarioRuns, seed: 1657, })( - `matches finite-domain coverage for a fixed seed`, - runCoverageTraceWithKnownFailures, - ) - - fcTest.prop([requestTraceArbitrary], coverageRandomParameters)( - `matches finite-domain coverage for a random or replayed seed`, - runCoverageTraceWithKnownFailures, - ) - - fcTest.prop([asyncScenarioArbitrary], { - numRuns: coverageScenarioRuns, - seed: 1658, - })( - `settles, retries, and resets in-flight set requests for a fixed seed`, - runAsyncScenarioWithKnownFailures, + `starts each completed exact demand once for a fixed seed`, + assertCompletedExactDemandTrace, ) - fcTest.prop([asyncScenarioArbitrary], coverageRandomParameters)( - `settles, retries, and resets in-flight set requests for a random or replayed seed`, - runAsyncScenarioWithKnownFailures, + fcTest.prop( + [exactDemandTraceArbitrary], + oracleRandomParameters( + exactScenarioRuns, + replay, + `load-subset.exact-completion`, + ), + )( + `starts each completed exact demand once for a random or replayed seed`, + assertCompletedExactDemandTrace, ) - fcTest.prop([concurrentAsyncScenarioArbitrary], { - numRuns: coverageScenarioRuns, + fcTest.prop([concurrentExactScenarioArbitrary], { + numRuns: exactScenarioRuns, seed: 1661, })( - `deduplicates three or more concurrent requests for a fixed seed`, - runConcurrentAsyncScenario, - ) - - fcTest.prop([concurrentAsyncScenarioArbitrary], coverageRandomParameters)( - `deduplicates three or more concurrent requests for a random or replayed seed`, - runConcurrentAsyncScenario, - ) - - fcTest.prop([rejectedWaiterScenarioArbitrary], { - numRuns: coverageScenarioRuns, - seed: 1665, - })( - `checks rejected requests observed by an in-flight waiter for a fixed seed`, - expectDeduplicatedWaiterHandlesRejection, - ) - - fcTest.prop([rejectedWaiterScenarioArbitrary], coverageRandomParameters)( - `checks rejected requests observed by an in-flight waiter for a random or replayed seed`, - expectDeduplicatedWaiterHandlesRejection, + `shares only identical in-flight demands for a fixed seed`, + assertConcurrentExactDemandTrace, ) - fcTest.prop([windowTraceArbitrary], { - numRuns: coverageScenarioRuns, - seed: 1659, - })( - `never treats uncovered ordered windows as loaded for a fixed seed`, - runWindowCoverageTraceWithKnownFailures, - ) - - fcTest.prop([windowTraceArbitrary], coverageRandomParameters)( - `never treats uncovered ordered windows as loaded for a random or replayed seed`, - runWindowCoverageTraceWithKnownFailures, - ) - - fcTest.prop([changingWhereWindowTraceArbitrary], { - numRuns: coverageScenarioRuns, - seed: 1666, - })( - `keeps changing predicates distinct across window histories for a fixed seed`, - runWindowCoverageTraceWithKnownFailures, - ) - - fcTest.prop([changingWhereWindowTraceArbitrary], coverageRandomParameters)( - `keeps changing predicates distinct across window histories for a random or replayed seed`, - runWindowCoverageTraceWithKnownFailures, - ) - - fcTest.prop([distinctWindowWherePairArbitrary], { - numRuns: coverageScenarioRuns, - seed: 1662, - })( - `keeps distinct limited-window predicates separate for a fixed seed`, - expectDistinctWhereStartsDistinctLimitedWindowLoads, - ) - - fcTest.prop([distinctWindowWherePairArbitrary], coverageRandomParameters)( - `keeps distinct limited-window predicates separate for a random or replayed seed`, - expectDistinctWhereStartsDistinctLimitedWindowLoads, + fcTest.prop( + [concurrentExactScenarioArbitrary], + oracleRandomParameters( + exactScenarioRuns, + replay, + `load-subset.exact-inflight`, + ), + )( + `shares only identical in-flight demands for a random or replayed seed`, + assertConcurrentExactDemandTrace, ) - it(`an in-flight deduplicated waiter rejects without an unhandled branch`, async () => { - await expectDeduplicatedWaiterHandlesRejection({ - covering: [1, 2], - covered: [1], - }) + it(`reports one rejection to every exact waiter and then retries`, async () => { + await expectExactWaitersShareRejection() }) +}) +describe(`loadSubset application and cancellation`, () => { it(`applies loaded rows when no mutation is persisting`, async () => { await expectPersistingLoadIsApplied(false) }) @@ -2745,8 +1180,8 @@ describe(`loadSubset coverage oracle`, () => { await expectAppliedLoadDoesNotFlushEarlierParkedSync() }) - it(`publishes coverage only after its establishing rows apply`, async () => { - await expectCoverageWaitsForAppliedRows() + it(`settles a demand only after its rows apply`, async () => { + await expectCompletionWaitsForAppliedRows() }) it(`keeps an unrelated stream commit parked during a subset acquisition`, async () => { @@ -2758,8 +1193,8 @@ describe(`loadSubset coverage oracle`, () => { }) it.each([`before-commit`, `while-parked`] as const)( - `does not publish coverage when a parked receipt is aborted %s`, - expectAbortedReceiptDoesNotPublishCoverage, + `does not settle a demand when its parked receipt is aborted %s`, + expectAbortedReceiptDoesNotSettleDemand, ) it(`ignores an abort raised after application starts publishing`, async () => { @@ -2770,8 +1205,8 @@ describe(`loadSubset coverage oracle`, () => { await expectCanceledReceiptReleasesOnlyItsSuppression() }) - it(`rejects an abandoned receipt once without publishing coverage`, async () => { - await expectCleanupRejectsReceiptOnce() + it(`rejects an abandoned demand once`, async () => { + await expectCleanupRejectsDemandOnce() }) it(`publishes synced source rows while a derived mutation persists`, async () => { @@ -2784,66 +1219,4 @@ describe(`loadSubset coverage oracle`, () => { expected.join(`,`) === `optimistic,synced`, })() }) - - it( - `discovered trace: adjacent ordered windows do not cover their combined window`, - expectAssertionFailure( - () => - Promise.resolve().then(() => { - expect( - countWindowLoads([ - { direction: `asc`, offset: 0, limit: 2 }, - { direction: `asc`, offset: 2, limit: 2 }, - { direction: `asc`, offset: 0, limit: 4 }, - ]), - ).toBe(2) - }), - { message: /expected 3 to be 2/ }, - ), - ) - - it(`discovered trace: widening a window remembers an earlier covered window`, () => { - const first: WindowRequest = { - orderField: `none`, - direction: `asc`, - offset: 0, - limit: 1, - where: { kind: `in`, values: [0] }, - } - expect(countWindowLoads([first, { ...first, limit: 2 }, first])).toBe(2) - }) - - it( - `discovered trace: complementary ranges redundantly reload an all-data request`, - expectAssertionFailure( - () => - Promise.resolve().then(() => { - expect( - countLoads([ - { kind: `range`, operator: `gt`, value: 0 }, - { kind: `range`, operator: `lte`, value: 0 }, - { kind: `all` }, - ]), - ).toBe(2) - }), - { message: /expected 3 to be 2/ }, - ), - ) - - it( - `discovered trace: a range plus boundary point redundantly reloads a covered set`, - expectAssertionFailure( - () => - Promise.resolve().then(() => { - expect( - countLoads([ - { kind: `range`, operator: `gt`, value: 0 }, - { kind: `eq`, value: 0 }, - { kind: `in`, values: [0, 1] }, - ]), - ).toBe(2) - }), - { message: /expected 3 to be 2/ }, - ), - ) }) diff --git a/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts new file mode 100644 index 0000000000..1637094b3e --- /dev/null +++ b/packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts @@ -0,0 +1,878 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { + createLiveQueryCollection, + eq, + toArray, +} from '../../src/query/index.js' +import { BasicIndex } from '../../src/indexes/basic-index.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { flushPromises } from '../utils.js' +import type { + ChangeMessageOrDeleteKeyMessage, + LoadSubsetOptions, + SyncConfig, +} from '../../src/types.js' + +type Row = { id: string; version: number } +type ObservedRow = { sourceId: string; rowKey: string; version: number } + +describe(`loadSubset replay refinement`, () => { + // A direct subscriber survives source cleanup. A dependent live query enters + // a terminal error instead; restarting only its source must not revive it. + it.each( + ([`direct`, `live`] as const).flatMap((consumer) => + ([`resolve`, `reject`] as const).map((outcome) => ({ + consumer, + outcome, + })), + ), + )( + `separates direct restart from fatal live source cleanup: %j`, + async ({ consumer, outcome }) => { + let operations!: Parameters[`sync`]>[0] + let loads = 0 + const pending = createDeferred() + void pending.promise.catch(() => undefined) + const initial = [{ id: `row`, version: 1 }] + const replacement = [{ id: `row`, version: 2 }] + const source = createCollection({ + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (next) => { + operations = next + next.markReady() + return { + loadSubset: () => { + if (++loads > 1) return pending.promise + next.begin() + next.write({ type: `insert`, value: initial[0]! }) + next.commit() + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = + consumer === `live` + ? createLiveQueryCollection((q) => q.from({ row: source })) + : undefined + const visible = new Map() + const rows = (values: ReadonlyArray) => + values.map(({ id, version }) => ({ id, version })) + const readEvents = () => rows([...visible.values()]) + const read = () => (live ? rows(live.toArray) : readEvents()) + const publications: Array> = [] + const subscription = (live ?? source).subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) visible.delete(String(change.key)) + else visible.set(String(change.key), { ...change.value }) + } + publications.push(readEvents()) + }, + { includeInitialState: consumer === `live` }, + ) + + try { + if (live) await live.preload() + else subscription.requestSnapshot({}) + await flushPromises() + expect(loads).toBe(1) + expect(read()).toEqual(initial) + expect(readEvents()).toEqual(initial) + publications.length = 0 + + await source.cleanup() + if (live) expect(live.status).toBe(`error`) + source.startSyncImmediate() + await flushPromises() + expect(loads).toBe(2) + expect(read()).toEqual(initial) + // Cleanup may change row metadata without changing the public data. + for (const publication of publications) + expect(publication).toEqual(initial) + + operations.begin() + operations.write({ type: `insert`, value: replacement[0]! }) + await operations.commit() + await flushPromises() + expect(rows(source.toArray)).toEqual(replacement) + expect(read()).toEqual(initial) + expect(readEvents()).toEqual(initial) + for (const publication of publications) + expect(publication).toEqual(initial) + publications.length = 0 + + if (outcome === `resolve`) pending.resolve() + else pending.reject(new Error(`restart failed`)) + await flushPromises() + const publishes = consumer === `direct` && outcome === `resolve` + const expected = publishes ? replacement : initial + expect(read()).toEqual(expected) + expect(readEvents()).toEqual(expected) + expect(publications).toEqual(publishes ? [replacement] : []) + if (live) expect(live.status).toBe(`error`) + } finally { + pending.resolve() + subscription.unsubscribe() + await live?.cleanup() + await source.cleanup() + } + }, + ) + + it(`publishes a successful sibling after a settled failed include route retires`, async () => { + type Parent = { id: string; left: number | null; right: number } + type Child = { id: number; version: number } + let parentSync!: Parameters[`sync`]>[0] + let childSync!: Parameters[`sync`]>[0] + const failed = createDeferred() + const successful = createDeferred() + const loads: Array<{ options: LoadSubsetOptions; ids: Array }> = [] + const unloads: Array = [] + const parents = createCollection({ + id: `settled-peer-parent`, + getKey: ({ id }) => id, + sync: { + sync: (operations) => { + parentSync = operations + operations.begin() + operations.write({ + type: `insert`, + value: { id: `parent`, left: 1, right: 2 }, + }) + operations.commit() + operations.markReady() + }, + }, + }) + const children = createCollection({ + id: `settled-peer-children`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: (operations) => { + childSync = operations + operations.markReady() + return { + loadSubset: (options) => { + const rows = [1, 2] + .map((id) => ({ id, version: loads.length < 2 ? 1 : 2 })) + .filter( + (row) => + !options.where || + evaluateReferenceExpression(options.where, row) === true, + ) + loads.push({ options, ids: rows.map(({ id }) => id) }) + operations.begin() + for (const value of rows) + operations.write({ type: `insert`, value }) + operations.commit() + if (loads.length <= 2) return true + return rows.some(({ id }) => id === 1) + ? failed.promise + : successful.promise + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents }).select(({ parent }) => ({ + id: parent.id, + left: toArray( + q + .from({ leftChild: children }) + .where(({ leftChild }) => eq(leftChild.id, parent.left)), + ), + right: toArray( + q + .from({ rightChild: children }) + .where(({ rightChild }) => eq(rightChild.id, parent.right)), + ), + })), + ) + const read = () => + live.toArray.map(({ id, left, right }) => ({ + id, + left: left.map(({ id: key, version }) => ({ id: key, version })), + right: right.map(({ id: key, version }) => ({ id: key, version })), + })) + const publications: Array> = [] + const subscription = live.subscribeChanges( + () => publications.push(read()), + { includeInitialState: false }, + ) + try { + await live.preload() + expect(loads.map(({ ids }) => ids)).toEqual([[1], [2]]) + const initial = [ + { + id: `parent`, + left: [{ id: 1, version: 1 }], + right: [{ id: 2, version: 1 }], + }, + ] + expect(read()).toEqual(initial) + publications.length = 0 + childSync.begin() + childSync.truncate() + childSync.commit() + await flushPromises() + expect(loads.slice(2).map(({ ids }) => ids)).toEqual([[1], [2]]) + failed.reject(new Error(`left replay failed`)) + successful.resolve() + await flushPromises() + expect(read()).toEqual(initial) + expect(publications).toEqual([]) + parentSync.begin() + parentSync.write({ + type: `update`, + value: { id: `parent`, left: null, right: 2 }, + }) + parentSync.commit() + await flushPromises() + expect(read()).toEqual([ + { id: `parent`, left: [], right: [{ id: 2, version: 2 }] }, + ]) + expect(publications).toEqual([ + [{ id: `parent`, left: [], right: [{ id: 2, version: 2 }] }], + ]) + expect(loads).toHaveLength(4) + expect(unloads).toContain(loads[2]!.options) + expect(loads[2]!.options.signal?.aborted).toBe(true) + expect(loads[3]!.options.signal?.aborted).toBe(false) + childSync.begin() + childSync.write({ type: `update`, value: { id: 2, version: 3 } }) + childSync.commit() + await flushPromises() + expect(read()).toEqual([ + { id: `parent`, left: [], right: [{ id: 2, version: 3 }] }, + ]) + expect(publications).toHaveLength(2) + } finally { + failed.resolve() + successful.resolve() + subscription.unsubscribe() + await live.cleanup() + await Promise.all([parents.cleanup(), children.cleanup()]) + } + expect(unloads).toHaveLength(loads.length) + for (const { options } of loads) { + expect(unloads.filter((unloaded) => unloaded === options)).toHaveLength(1) + } + }) + + function createHarness( + sourceId: string, + initialRows: ReadonlyArray = [{ id: `row`, version: 1 }], + ) { + let begin!: () => void + let write!: (message: ChangeMessageOrDeleteKeyMessage) => void + let commit!: () => void + let truncate!: () => void + let loadCount = 0 + const pending: Array<{ + options: LoadSubsetOptions + deferred: ReturnType> + }> = [] + const batches: Array< + Array<{ + type: `insert` | `update` | `delete` + row: { sourceId: string; rowKey: string; version: number } + previousVersion?: number + }> + > = [] + const source = createCollection({ + id: sourceId, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + truncate = params.truncate + params.markReady() + return { + loadSubset: (options) => { + loadCount++ + if (loadCount === 1) { + begin() + for (const value of initialRows) { + write({ type: `insert`, value }) + } + commit() + return true + } + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const downstream = createLiveQueryCollection({ + id: `${sourceId}-downstream`, + query: (q) => + q.from({ row: source }).select(({ row }) => ({ + id: row.id, + version: row.version, + })), + startSync: true, + }) + const callbackReads: Array> = [] + const subscription = downstream.subscribeChanges( + (changes) => { + const batch = changes.map((change) => ({ + type: change.type, + row: { + sourceId, + rowKey: String(change.key), + version: change.value.version, + }, + ...(change.previousValue === undefined + ? {} + : { previousVersion: change.previousValue.version }), + })) + if (batch.length > 0) { + batches.push(batch) + callbackReads.push( + downstream.toArray.map(({ id, version }) => ({ + sourceId, + rowKey: id, + version, + })), + ) + } + }, + { includeInitialState: true }, + ) + + const replaceCore = (version: number) => { + begin() + write({ type: `insert`, value: { id: `row`, version } }) + commit() + } + const updateCore = (previousVersion: number, version: number) => { + begin() + write({ + type: `update`, + value: { id: `row`, version }, + previousValue: { id: `row`, version: previousVersion }, + }) + commit() + } + const applyCore = ( + changes: ReadonlyArray>, + ) => { + begin() + for (const change of changes) write(change) + commit() + } + const startReplay = async () => { + begin() + truncate() + commit() + await flushPromises() + } + const coreRows = () => + source.toArray.map(({ id, version }) => ({ + sourceId, + rowKey: id, + version, + })) + const visibleRows = () => + downstream.toArray.map(({ id, version }) => ({ + sourceId, + rowKey: id, + version, + })) + + return { + source, + downstream, + subscription, + pending, + batches, + callbackReads, + replaceCore, + updateCore, + applyCore, + startReplay, + coreRows, + visibleRows, + } + } + + it(`retains the last complete publication when replay fails after writing`, async () => { + const sourceId = `replay-refinement-failure` + const row = (version: number) => ({ + sourceId, + rowKey: `row`, + version, + }) + const harness = createHarness(sourceId) + + try { + await harness.downstream.preload() + await harness.startReplay() + + harness.replaceCore(2) + harness.pending[0]?.deferred.reject(new Error(`replay failed`)) + await flushPromises() + + expect(harness.coreRows()).toEqual([row(2)]) + expect(harness.visibleRows()).toEqual([row(1)]) + expect(harness.batches).toEqual([[{ type: `insert`, row: row(1) }]]) + expect(harness.callbackReads).toEqual([[row(1)]]) + } finally { + harness.subscription.unsubscribe() + await Promise.all([ + harness.downstream.cleanup(), + harness.source.cleanup(), + ]) + } + }) + + it(`publishes a replay replacement before its source reports ready`, async () => { + const replay = createDeferred() + let loadCount = 0 + let begin!: () => void + let write!: (message: ChangeMessageOrDeleteKeyMessage) => void + let commit!: () => void + let truncate!: () => void + let sourceSubscription: LoadSubsetOptions[`subscription`] + const source = createCollection({ + id: `replay-ready-publication-source`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options) => { + sourceSubscription = options.subscription + loadCount++ + if (loadCount === 1) { + begin() + write({ type: `insert`, value: { id: `row`, version: 1 } }) + commit() + return true + } + return replay.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q.from({ row: source }).select(({ row }) => ({ + id: row.id, + version: row.version, + })), + ) + const readVersions = () => live.toArray.map(({ version }) => version) + const readyReads: Array> = [] + + try { + await live.preload() + expect(readVersions()).toEqual([1]) + sourceSubscription!.on(`status:ready`, () => { + readyReads.push(readVersions()) + }) + + begin() + truncate() + commit() + await flushPromises() + begin() + write({ type: `insert`, value: { id: `row`, version: 2 } }) + commit() + + replay.resolve() + await flushPromises() + + expect(readyReads).toEqual([[2]]) + expect(readVersions()).toEqual([2]) + } finally { + replay.resolve() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`keeps a failed replay private until a later authoritative replay`, async () => { + const sourceId = `replay-refinement-failure-liveness` + const row = (version: number) => ({ sourceId, rowKey: `row`, version }) + const harness = createHarness(sourceId) + + try { + await harness.downstream.preload() + expect(harness.visibleRows().map(({ version }) => version)).toEqual([1]) + + await harness.startReplay() + harness.replaceCore(2) + harness.pending[0]!.deferred.reject(new Error(`replay failed`)) + await flushPromises() + + expect(harness.visibleRows()).toEqual([row(1)]) + expect(harness.downstream.status).toBe(`ready`) + expect(harness.batches).toEqual([[{ type: `insert`, row: row(1) }]]) + + harness.updateCore(2, 3) + await flushPromises() + + expect(harness.visibleRows()).toEqual([row(1)]) + expect(harness.batches).toEqual([[{ type: `insert`, row: row(1) }]]) + expect(harness.callbackReads).toEqual([[row(1)]]) + + await harness.startReplay() + harness.replaceCore(4) + harness.pending[1]!.deferred.resolve() + await flushPromises() + + expect(harness.visibleRows()).toEqual([row(4)]) + expect(harness.batches).toEqual([ + [{ type: `insert`, row: row(1) }], + [{ type: `update`, row: row(4), previousVersion: 1 }], + ]) + expect(harness.callbackReads).toEqual([[row(1)], [row(4)]]) + } finally { + for (const replay of harness.pending) replay.deferred.resolve() + harness.subscription.unsubscribe() + await Promise.all([ + harness.downstream.cleanup(), + harness.source.cleanup(), + ]) + } + }) + + it(`replaces a multi-row failed replay only with later authoritative state`, async () => { + const sourceId = `replay-refinement-multi-row-failure` + const observed = (id: string, version: number) => ({ + sourceId, + rowKey: id, + version, + }) + const harness = createHarness(sourceId, [ + { id: `a`, version: 1 }, + { id: `b`, version: 1 }, + { id: `c`, version: 1 }, + ]) + const sortedVisible = () => + harness + .visibleRows() + .sort((left, right) => left.rowKey.localeCompare(right.rowKey)) + const sortedCore = () => + harness + .coreRows() + .sort((left, right) => left.rowKey.localeCompare(right.rowKey)) + + try { + await harness.downstream.preload() + expect(sortedVisible()).toEqual([ + observed(`a`, 1), + observed(`b`, 1), + observed(`c`, 1), + ]) + const publishedBatches = harness.batches.length + + await harness.startReplay() + harness.applyCore([ + { type: `insert`, value: { id: `a`, version: 2 } }, + { type: `insert`, value: { id: `d`, version: 1 } }, + ]) + harness.pending[0]!.deferred.reject(new Error(`partial replay failed`)) + await flushPromises() + + harness.applyCore([ + { + type: `update`, + value: { id: `a`, version: 3 }, + previousValue: { id: `a`, version: 2 }, + }, + { type: `delete`, key: `d` }, + { type: `insert`, value: { id: `e`, version: 1 } }, + ]) + await flushPromises() + + expect(sortedCore()).toEqual([observed(`a`, 3), observed(`e`, 1)]) + expect(sortedVisible()).toEqual([ + observed(`a`, 1), + observed(`b`, 1), + observed(`c`, 1), + ]) + expect(harness.batches).toHaveLength(publishedBatches) + + await harness.startReplay() + harness.applyCore([ + { type: `insert`, value: { id: `a`, version: 4 } }, + { type: `insert`, value: { id: `b`, version: 1 } }, + { type: `insert`, value: { id: `e`, version: 2 } }, + ]) + harness.pending[1]!.deferred.resolve() + await flushPromises() + + expect(sortedVisible()).toEqual([ + observed(`a`, 4), + observed(`b`, 1), + observed(`e`, 2), + ]) + expect(harness.batches).toHaveLength(publishedBatches + 1) + expect(harness.batches.at(-1)).toEqual([ + { + type: `update`, + row: observed(`a`, 4), + previousVersion: 1, + }, + { type: `delete`, row: observed(`c`, 1) }, + { type: `insert`, row: observed(`e`, 2) }, + ]) + expect( + harness.callbackReads + .at(-1) + ?.sort((left, right) => left.rowKey.localeCompare(right.rowKey)), + ).toEqual([observed(`a`, 4), observed(`b`, 1), observed(`e`, 2)]) + } finally { + for (const replay of harness.pending) replay.deferred.resolve() + harness.subscription.unsubscribe() + await Promise.all([ + harness.downstream.cleanup(), + harness.source.cleanup(), + ]) + } + }) + + it(`waits for every overlapping replay before publishing the newest success`, async () => { + const sourceId = `replay-refinement-overlap` + const row = (version: number) => ({ + sourceId, + rowKey: `row`, + version, + }) + const harness = createHarness(sourceId) + + try { + await harness.downstream.preload() + await harness.startReplay() + await harness.startReplay() + + expect(harness.pending[0]?.options.signal?.aborted).toBe(true) + harness.replaceCore(3) + harness.pending[1]?.deferred.resolve() + await flushPromises() + + expect(harness.visibleRows()).toEqual([row(1)]) + expect(harness.batches).toEqual([[{ type: `insert`, row: row(1) }]]) + expect(harness.callbackReads).toEqual([[row(1)]]) + + harness.pending[0]?.deferred.reject( + new DOMException(`obsolete`, `AbortError`), + ) + await flushPromises() + + expect(harness.coreRows()).toEqual([row(3)]) + expect(harness.visibleRows()).toEqual([row(3)]) + expect(harness.batches).toEqual([ + [{ type: `insert`, row: row(1) }], + [{ type: `update`, row: row(3), previousVersion: 1 }], + ]) + expect(harness.callbackReads).toEqual([[row(1)], [row(3)]]) + } finally { + for (const replay of harness.pending) replay.deferred.resolve() + harness.subscription.unsubscribe() + await Promise.all([ + harness.downstream.cleanup(), + harness.source.cleanup(), + ]) + } + }) + + it(`waits for every recovering source before publishing a joined replacement`, async () => { + type Primary = { id: string; joinKey: string; version: number } + type Secondary = { id: string; joinKey: string; version: number } + + const createSource = (id: string) => { + let begin!: () => void + let write!: (message: { type: `insert`; value: T }) => void + let commit!: () => true | Promise + let truncate!: () => void + const pending: Array>> = [] + const collection = createCollection({ + id, + getKey: ({ id: key }) => key, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: () => { + const request = createDeferred() + pending.push(request) + return request.promise + }, + unloadSubset: () => {}, + } + }, + }, + }) + return { + collection, + pending, + async apply(row: T) { + begin() + write({ type: `insert`, value: row }) + const receipt = commit() + if (receipt !== true) await receipt + }, + replay() { + begin() + truncate() + return commit() + }, + } + } + + const primary = createSource(`joined-replay-primary`) + const secondary = createSource(`joined-replay-secondary`) + const live = createLiveQueryCollection((q) => + q + .from({ primary: primary.collection }) + .innerJoin( + { secondary: secondary.collection }, + ({ primary: left, secondary: right }) => + eq(left.joinKey, right.joinKey), + ) + .orderBy(({ primary: row }) => row.version) + .limit(1) + .select(({ primary: left, secondary: right }) => ({ + id: left.id, + secondaryId: right.id, + primaryVersion: left.version, + secondaryVersion: right.version, + })), + ) + const read = () => + live.toArray.map( + ({ id, secondaryId, primaryVersion, secondaryVersion }) => ({ + id, + secondaryId, + primaryVersion, + secondaryVersion, + }), + ) + const publications: Array> = [] + let subscription: ReturnType | undefined + let primaryReplay: true | Promise = true + let secondaryReplay: true | Promise = true + + try { + const preload = live.preload() + await flushPromises() + expect(primary.pending).toHaveLength(1) + await primary.apply({ id: `p`, joinKey: `shared`, version: 1 }) + primary.pending[0]!.resolve() + await flushPromises() + expect(secondary.pending).toHaveLength(1) + await secondary.apply({ id: `s`, joinKey: `shared`, version: 1 }) + secondary.pending[0]!.resolve() + await flushPromises() + for (const request of primary.pending.slice(1)) request.resolve() + await preload + expect(read()).toEqual([ + { + id: `p`, + secondaryId: `s`, + primaryVersion: 1, + secondaryVersion: 1, + }, + ]) + + subscription = live.subscribeChanges(() => publications.push(read()), { + includeInitialState: false, + }) + const initialPrimaryLoads = primary.pending.length + const initialSecondaryLoads = secondary.pending.length + primaryReplay = primary.replay() + secondaryReplay = secondary.replay() + await flushPromises() + expect(primary.pending.length).toBeGreaterThan(initialPrimaryLoads) + expect(secondary.pending.length).toBeGreaterThan(initialSecondaryLoads) + + await primary.apply({ id: `p`, joinKey: `shared`, version: 2 }) + await secondary.apply({ id: `s`, joinKey: `shared`, version: 2 }) + for (const request of primary.pending.slice(initialPrimaryLoads)) { + request.resolve() + } + await flushPromises() + + expect(read()).toEqual([ + { + id: `p`, + secondaryId: `s`, + primaryVersion: 1, + secondaryVersion: 1, + }, + ]) + expect(publications).toEqual([]) + + for (const request of secondary.pending.slice(initialSecondaryLoads)) { + request.resolve() + } + await Promise.all([primaryReplay, secondaryReplay]) + await flushPromises() + + expect(read()).toEqual([ + { + id: `p`, + secondaryId: `s`, + primaryVersion: 2, + secondaryVersion: 2, + }, + ]) + expect(publications).toEqual([ + [ + { + id: `p`, + secondaryId: `s`, + primaryVersion: 2, + secondaryVersion: 2, + }, + ], + ]) + } finally { + for (const request of [...primary.pending, ...secondary.pending]) { + request.resolve() + } + subscription?.unsubscribe() + await Promise.all([ + Promise.resolve(primaryReplay).catch(() => undefined), + Promise.resolve(secondaryReplay).catch(() => undefined), + live.cleanup(), + primary.collection.cleanup(), + secondary.collection.cleanup(), + ]) + } + }) +}) diff --git a/packages/db/tests/query/load-subset-source-readiness-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-source-readiness-refinement-oracle.test.ts new file mode 100644 index 0000000000..ff67c4ec98 --- /dev/null +++ b/packages/db/tests/query/load-subset-source-readiness-refinement-oracle.test.ts @@ -0,0 +1,346 @@ +import { expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { BTreeIndex } from '../../src/index.js' +import { extractSimpleComparisons } from '../../src/query/expression-helpers.js' +import { + createLiveQueryCollection, + eq, + toArray, +} from '../../src/query/index.js' +import { flushPromises } from '../utils.js' +import type { LoadSubsetOptions } from '../../src/types.js' + +type Row = { id: string; group: string } + +it.each([ + { oldOutcome: `resolve`, settlementOrder: `old-first` }, + { oldOutcome: `reject`, settlementOrder: `old-first` }, + { oldOutcome: `resolve`, settlementOrder: `fresh-first` }, + { oldOutcome: `reject`, settlementOrder: `fresh-first` }, +] as const)( + `fences a retired source-demand attempt across $settlementOrder $oldOutcome settlement`, + async ({ oldOutcome, settlementOrder }) => { + type Parent = { id: string; group: string } + type Child = { id: string; group: string } + type PendingRequest = { + options: LoadSubsetOptions + rows: ReturnType>> + } + const caseId = `${oldOutcome}-${settlementOrder}` + const parentId = `readiness-generation-parent-${caseId}` + const childId = `readiness-generation-child-${caseId}` + let parentBegin!: () => void + let parentWrite!: (message: { + type: `update` + value: Parent + previousValue: Parent + }) => void + let parentCommit!: () => true | Promise + const oldParent: Parent = { id: `parent`, group: `old` } + const freshParent: Parent = { ...oldParent, group: `fresh` } + const parent = createCollection({ + id: parentId, + getKey: (row) => row.id, + sync: { + sync: ({ begin, write, commit, markReady }) => { + parentBegin = begin + parentWrite = write + parentCommit = commit + begin() + write({ type: `insert`, value: oldParent }) + commit() + markReady() + }, + }, + }) + let childBegin!: () => void + let childWrite!: (message: { type: `insert`; value: Child }) => void + let childCommit!: () => true | Promise + const pending: Array = [] + const unloads: Array<{ + options: LoadSubsetOptions + abortedAtUnload: boolean | undefined + }> = [] + const child = createCollection({ + id: childId, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + childBegin = begin + childWrite = write + childCommit = commit + markReady() + return { + loadSubset: (options) => { + const rows = createDeferred>() + pending.push({ options, rows }) + return rows.promise.then(async (acquiredRows) => { + if (acquiredRows.length > 0) { + childBegin() + for (const row of acquiredRows) { + childWrite({ type: `insert`, value: row }) + } + const applied = childCommit() + if (applied !== true) await applied + } + return + }) + }, + unloadSubset: (options) => { + unloads.push({ + options, + abortedAtUnload: options.signal?.aborted, + }) + }, + } + }, + }, + }) + const live = createLiveQueryCollection({ + id: `readiness-generation-live-${caseId}`, + query: (q) => + q.from({ parent }).select(({ parent: parentRow }) => ({ + id: parentRow.id, + children: toArray( + q + .from({ child }) + .where(({ child: childRow }) => + eq(childRow.group, parentRow.group), + ), + ), + })), + startSync: true, + }) + let preloadState: `pending` | `resolved` | `rejected` = `pending` + const preload = live.preload() + void preload.then( + () => { + preloadState = `resolved` + }, + () => { + preloadState = `rejected` + }, + ) + const requestedGroups = (options: LoadSubsetOptions): Array => + extractSimpleComparisons(options.where).flatMap((comparison) => { + if (comparison.field.join(`.`) !== `group`) return [] + if (comparison.operator === `eq`) { + return typeof comparison.value === `string` ? [comparison.value] : [] + } + if (comparison.operator !== `in` || !Array.isArray(comparison.value)) { + return [] + } + return comparison.value.filter( + (value): value is string => typeof value === `string`, + ) + }) + const expectUnloads = ( + ...expectedOptions: ReadonlyArray + ): void => { + expect(unloads).toHaveLength(expectedOptions.length) + for (const [index, options] of expectedOptions.entries()) { + expect(unloads[index]!.options).toBe(options) + expect(unloads[index]!.abortedAtUnload).toBe(true) + } + } + let liveCleaned = false + + try { + await flushPromises() + expect(pending).toHaveLength(1) + expect(requestedGroups(pending[0]!.options)).toEqual([`old`]) + expect(live.status).toBe(`loading`) + expect(preloadState).toBe(`pending`) + + parentBegin() + parentWrite({ + type: `update`, + value: freshParent, + previousValue: oldParent, + }) + const parentApplied = parentCommit() + if (parentApplied !== true) await parentApplied + await flushPromises() + + expect(pending).toHaveLength(2) + expect(requestedGroups(pending[0]!.options)).toEqual([`old`]) + expect(requestedGroups(pending[1]!.options)).toEqual([`fresh`]) + expect(pending[0]!.options.signal?.aborted).toBe(true) + expect(pending[1]!.options.signal?.aborted).toBe(false) + expectUnloads(pending[0]!.options) + expect(live.status).toBe(`loading`) + expect(preloadState).toBe(`pending`) + + const freshChild: Child = { id: `fresh-child`, group: `fresh` } + let freshHasSettled = false + const settleOld = async () => { + if (oldOutcome === `resolve`) { + pending[0]!.rows.resolve([]) + } else { + pending[0]!.rows.reject(new Error(`retired source demand failed`)) + } + await flushPromises() + } + const settleFresh = async () => { + expect(child.get(freshChild.id)).toBeUndefined() + pending[1]!.rows.resolve([freshChild]) + await flushPromises() + freshHasSettled = true + } + const settlements = + settlementOrder === `old-first` + ? [settleOld, settleFresh] + : [settleFresh, settleOld] + for (const settle of settlements) { + await settle() + expect(live.status).toBe(freshHasSettled ? `ready` : `loading`) + expect(preloadState).toBe(freshHasSettled ? `resolved` : `pending`) + expect(live.utils.lastSubsetError).toBeUndefined() + } + + await preload + await flushPromises() + + expect(live.status).toBe(`ready`) + expect(preloadState).toBe(`resolved`) + expect(live.utils.lastSubsetError).toBeUndefined() + expect(child.get(freshChild.id)).toEqual( + expect.objectContaining(freshChild), + ) + expect(live.toArray).toEqual([ + expect.objectContaining({ + id: `parent`, + children: [expect.objectContaining({ id: `fresh-child` })], + }), + ]) + expect(pending[1]!.options.signal?.aborted).toBe(false) + expectUnloads(pending[0]!.options) + + await live.cleanup() + liveCleaned = true + expect(pending[1]!.options.signal?.aborted).toBe(true) + expectUnloads(pending[0]!.options, pending[1]!.options) + } finally { + for (const request of pending) { + request.rows.resolve([]) + } + await Promise.all([ + preload.catch(() => undefined), + liveCleaned ? Promise.resolve() : live.cleanup(), + ]) + await Promise.all([parent.cleanup(), child.cleanup()]) + } + }, +) + +it.each([`resolve`, `reject`, `cleanup`] as const)( + `matches cross-source initial readiness through %s`, + async (secondOutcome) => { + const leftId = `readiness-left-${secondOutcome}` + const rightId = `readiness-right-${secondOutcome}` + const leftDelivery = createDeferred() + const rightDelivery = createDeferred() + const createSource = ( + id: string, + row: Row, + delivery: ReturnType>, + ) => + createCollection({ + id, + getKey: (value) => value.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: () => + delivery.promise.then(async () => { + begin() + write({ type: `insert`, value: row }) + const applied = commit() + if (applied !== true) await applied + return + }), + unloadSubset: () => {}, + } + }, + }, + }) + const left = createSource( + leftId, + { id: `left`, group: `shared` }, + leftDelivery, + ) + const right = createSource( + rightId, + { id: `right`, group: `shared` }, + rightDelivery, + ) + const live = createLiveQueryCollection({ + id: `readiness-live-${secondOutcome}`, + query: (q) => + q + .from({ left }) + .innerJoin({ right }, ({ left: leftRow, right: rightRow }) => + eq(leftRow.group, rightRow.group), + ) + .select(({ left: leftRow, right: rightRow }) => ({ + leftId: leftRow.id, + rightId: rightRow.id, + })), + startSync: true, + }) + const preload = live.preload() + void preload.catch(() => undefined) + + try { + expect(live.status).toBe(`loading`) + + leftDelivery.resolve() + await flushPromises() + + expect(live.status).toBe(`loading`) + expect(live.toArray).toEqual([]) + + if (secondOutcome === `cleanup`) { + await live.cleanup() + expect(live.status).toBe(`cleaned-up`) + + rightDelivery.resolve() + await flushPromises() + + expect(live.status).toBe(`cleaned-up`) + expect(live.toArray).toEqual([]) + return + } else if (secondOutcome === `resolve`) { + rightDelivery.resolve() + } else { + rightDelivery.reject(new Error(`right source failed`)) + } + await flushPromises() + + expect(live.status).toBe(secondOutcome === `resolve` ? `ready` : `error`) + if (secondOutcome === `resolve`) { + await expect(preload).resolves.toBeUndefined() + expect(live.toArray).toEqual([ + expect.objectContaining({ leftId: `left`, rightId: `right` }), + ]) + } else { + await expect(preload).rejects.toThrow(`right source failed`) + } + } finally { + leftDelivery.resolve() + rightDelivery.resolve() + await live.cleanup() + await Promise.all([left.cleanup(), right.cleanup()]) + } + }, +) diff --git a/packages/db/tests/query/load-subset-subquery.test.ts b/packages/db/tests/query/load-subset-subquery.test.ts index 3f6eee13b3..ead6b921d3 100644 --- a/packages/db/tests/query/load-subset-subquery.test.ts +++ b/packages/db/tests/query/load-subset-subquery.test.ts @@ -328,20 +328,23 @@ describe(`loadSubset with subqueries`, () => { // Verify loadSubset was called expect(loadSubsetCalls.length).toBeGreaterThan(0) - // Verify the last call has the orderBy clause and limit - const lastCall = loadSubsetCalls[loadSubsetCalls.length - 1] - expect(lastCall).toBeDefined() - expect(lastCall!.orderBy).toBeDefined() - expect(lastCall!.limit).toBe(2) - const expectedOrderBy: OrderBy = [ { expression: new PropRef([`scheduled_at`]), - compareOptions: { direction: `desc`, nulls: `first` }, + compareOptions: { + direction: `desc`, + nulls: `first`, + stringSort: `locale`, + }, }, ] - expect(lastCall!.orderBy).toEqual(expectedOrderBy) + const orderedCalls = loadSubsetCalls.filter(({ orderBy }) => orderBy) + expect(orderedCalls).not.toHaveLength(0) + for (const { orderBy, limit } of orderedCalls) { + expect(orderBy).toEqual(expectedOrderBy) + expect(limit).toBe(2) + } }) it(`should call loadSubset with orderBy clause for subquery`, async () => { @@ -369,20 +372,23 @@ describe(`loadSubset with subqueries`, () => { // Verify loadSubset was called for the orders collection expect(loadSubsetCalls.length).toBeGreaterThan(0) - // Verify the last call has the orderBy clause and limit - const lastCall = loadSubsetCalls[loadSubsetCalls.length - 1] - expect(lastCall).toBeDefined() - expect(lastCall!.orderBy).toBeDefined() - expect(lastCall!.limit).toBe(2) - const expectedOrderBy: OrderBy = [ { expression: new PropRef([`scheduled_at`]), - compareOptions: { direction: `desc`, nulls: `first` }, + compareOptions: { + direction: `desc`, + nulls: `first`, + stringSort: `locale`, + }, }, ] - expect(lastCall!.orderBy).toEqual(expectedOrderBy) + const orderedCalls = loadSubsetCalls.filter(({ orderBy }) => orderBy) + expect(orderedCalls).not.toHaveLength(0) + for (const { orderBy, limit } of orderedCalls) { + expect(orderBy).toEqual(expectedOrderBy) + expect(limit).toBe(2) + } }) it(`does not forward a computed subquery order to loadSubset`, async () => { diff --git a/packages/db/tests/query/load-subset-transaction-refinement-oracle.test.ts b/packages/db/tests/query/load-subset-transaction-refinement-oracle.test.ts new file mode 100644 index 0000000000..f92a867d1b --- /dev/null +++ b/packages/db/tests/query/load-subset-transaction-refinement-oracle.test.ts @@ -0,0 +1,86 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { createTransaction } from '../../src/transactions.js' + +type Row = { id: string; group: string } + +describe(`loadSubset transaction refinement`, () => { + it.each([`at-commit`, `while-parked`, `after-publication-starts`] as const)( + `matches the independent receipt and publication model when aborting %s`, + async (abortPhase) => { + const sourceId = `transaction-refinement-${abortPhase}` + const remoteRow: Row = { id: `remote`, group: `requested` } + const controller = new AbortController() + const persistence = createDeferred() + const publishedBatches: Array> = [] + const callbackReads: Array> = [] + const source = createCollection({ + id: sourceId, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: ({ signal }) => { + begin() + write({ type: `insert`, value: remoteRow }) + if (abortPhase === `at-commit`) controller.abort() + return commit(signal) + }, + } + }, + }, + }) + source.startSyncImmediate() + const blocker = createTransaction({ + mutationFn: () => persistence.promise, + }) + blocker.mutate(() => + source.insert({ id: `local`, group: `outside-request` }), + ) + const subscription = source.subscribeChanges( + (changes) => { + const remoteKeys = changes + .filter((change) => change.key === remoteRow.id) + .map((change) => String(change.key)) + if (remoteKeys.length === 0) return + publishedBatches.push(remoteKeys) + callbackReads.push(source.has(remoteRow.id) ? [remoteRow.id] : []) + if (abortPhase === `after-publication-starts`) { + controller.abort() + } + }, + { includeInitialState: false }, + ) + const load = source._sync.loadSubset({ signal: controller.signal }) + expect(load).toBeInstanceOf(Promise) + + try { + if (abortPhase === `while-parked`) { + controller.abort() + } + + persistence.resolve() + await blocker.isPersisted.promise + + if (abortPhase !== `after-publication-starts`) { + await expect(load).rejects.toMatchObject({ name: `AbortError` }) + } else { + await expect(load).resolves.toBeUndefined() + } + + const published = abortPhase === `after-publication-starts` + expect(source.has(remoteRow.id)).toBe(published) + expect(publishedBatches).toEqual(published ? [[remoteRow.id]] : []) + expect(callbackReads).toEqual(published ? [[remoteRow.id]] : []) + } finally { + persistence.resolve() + await blocker.isPersisted.promise.catch(() => undefined) + subscription.unsubscribe() + await source.cleanup() + } + }, + ) +}) diff --git a/packages/db/tests/query/order-by.test.ts b/packages/db/tests/query/order-by.test.ts index 9b48805eb9..ba55fbc209 100644 --- a/packages/db/tests/query/order-by.test.ts +++ b/packages/db/tests/query/order-by.test.ts @@ -255,10 +255,6 @@ function createEmployeesWithNullableCollection( function createOrderByTests(autoIndex: `off` | `eager`): void { describe(`with autoIndex ${autoIndex}`, () => { - // Some tests require an index for incremental updates (loadMoreIfNeeded). - // These only work with autoIndex: 'eager' which auto-creates the needed indexes. - const itWhenAutoIndexEager = autoIndex === `eager` ? it : it.skip - let employeesCollection: ReturnType let departmentsCollection: ReturnType @@ -620,59 +616,56 @@ function createOrderByTests(autoIndex: `off` | `eager`): void { ]) }) - itWhenAutoIndexEager( - `applies incremental insert of a new row inside the topK but after max sent value correctly`, - async () => { - const collection = createLiveQueryCollection((q) => - q - .from({ employees: employeesCollection }) - .orderBy(({ employees }) => employees.salary, `asc`) - .offset(1) - .limit(10) - .select(({ employees }) => ({ - id: employees.id, - name: employees.name, - salary: employees.salary, - })), - ) - await collection.preload() + it(`applies incremental insert of a new row inside the topK but after max sent value correctly`, async () => { + const collection = createLiveQueryCollection((q) => + q + .from({ employees: employeesCollection }) + .orderBy(({ employees }) => employees.salary, `asc`) + .offset(1) + .limit(10) + .select(({ employees }) => ({ + id: employees.id, + name: employees.name, + salary: employees.salary, + })), + ) + await collection.preload() - const results = Array.from(collection.values()) + const results = Array.from(collection.values()) - expect(results.map((r) => r.salary)).toEqual([ - 52_000, 55_000, 60_000, 65_000, - ]) + expect(results.map((r) => r.salary)).toEqual([ + 52_000, 55_000, 60_000, 65_000, + ]) - // Now insert a new employee with highest salary - // this should now become part of the topK because - // the topK isn't full yet, so even though it's after the max sent value - // it should still be part of the topK - const newEmployee = { - id: 6, - name: `George`, - department_id: 1, - salary: 72_000, - hire_date: `2023-01-01`, - } - - employeesCollection.utils.begin() - employeesCollection.utils.write({ - type: `insert`, - value: newEmployee, - }) - employeesCollection.utils.commit() - - const newResults = Array.from(collection.values()) - - expect(newResults.map((r) => [r.id, r.salary])).toEqual([ - [5, 52_000], - [3, 55_000], - [2, 60_000], - [4, 65_000], - [6, 72_000], - ]) - }, - ) + // Now insert a new employee with highest salary + // this should now become part of the topK because + // the topK isn't full yet, so even though it's after the max sent value + // it should still be part of the topK + const newEmployee = { + id: 6, + name: `George`, + department_id: 1, + salary: 72_000, + hire_date: `2023-01-01`, + } + + employeesCollection.utils.begin() + employeesCollection.utils.write({ + type: `insert`, + value: newEmployee, + }) + employeesCollection.utils.commit() + + const newResults = Array.from(collection.values()) + + expect(newResults.map((r) => [r.id, r.salary])).toEqual([ + [5, 52_000], + [3, 55_000], + [2, 60_000], + [4, 65_000], + [6, 72_000], + ]) + }) it(`applies incremental insert of a new row after the topK correctly`, async () => { const collection = createLiveQueryCollection((q) => @@ -800,40 +793,37 @@ function createOrderByTests(autoIndex: `off` | `eager`): void { ]) }) - itWhenAutoIndexEager( - `handles deletion from partial page with limit larger than data`, - async () => { - const collection = createLiveQueryCollection((q) => - q - .from({ employees: employeesCollection }) - .orderBy(({ employees }) => employees.salary, `desc`) - .limit(20) // Limit larger than number of employees (5) - .select(({ employees }) => ({ - id: employees.id, - name: employees.name, - salary: employees.salary, - })), - ) - await collection.preload() + it(`handles deletion from partial page with limit larger than data`, async () => { + const collection = createLiveQueryCollection((q) => + q + .from({ employees: employeesCollection }) + .orderBy(({ employees }) => employees.salary, `desc`) + .limit(20) // Limit larger than number of employees (5) + .select(({ employees }) => ({ + id: employees.id, + name: employees.name, + salary: employees.salary, + })), + ) + await collection.preload() - const results = Array.from(collection.values()) - expect(results).toHaveLength(5) - expect(results[0]!.name).toBe(`Diana`) - - // Delete Diana (the highest paid employee, first in DESC order) - const dianaData = employeeData.find((e) => e.id === 4)! - employeesCollection.utils.begin() - employeesCollection.utils.write({ - type: `delete`, - value: dianaData, - }) - employeesCollection.utils.commit() - - const newResults = Array.from(collection.values()) - expect(newResults).toHaveLength(4) - expect(newResults[0]!.name).toBe(`Bob`) - }, - ) + const results = Array.from(collection.values()) + expect(results).toHaveLength(5) + expect(results[0]!.name).toBe(`Diana`) + + // Delete Diana (the highest paid employee, first in DESC order) + const dianaData = employeeData.find((e) => e.id === 4)! + employeesCollection.utils.begin() + employeesCollection.utils.write({ + type: `delete`, + value: dianaData, + }) + employeesCollection.utils.commit() + + const newResults = Array.from(collection.values()) + expect(newResults).toHaveLength(4) + expect(newResults[0]!.name).toBe(`Bob`) + }) }) describe(`OrderBy with Joins`, () => { @@ -1851,186 +1841,172 @@ function createOrderByTests(autoIndex: `off` | `eager`): void { }) describe(`OrderBy Optimization Tests`, () => { - const itWhenAutoIndex = autoIndex === `eager` ? it : it.skip - - itWhenAutoIndex( - `optimizes single-column orderBy when passed as single value`, - async () => { - // Patch getConfig to expose the builder on the returned config for test access - const { CollectionConfigBuilder } = await import( - `../../src/query/live/collection-config-builder.js` - ) - const originalGetConfig = CollectionConfigBuilder.prototype.getConfig - - CollectionConfigBuilder.prototype.getConfig = function (this: any) { - const cfg = originalGetConfig.call(this) - ;(cfg as any).__builder = this - return cfg - } - - try { - const collection = createLiveQueryCollection((q) => - q - .from({ employees: employeesCollection }) - .orderBy(({ employees }) => employees.salary, `desc`) - .limit(3) - .select(({ employees }) => ({ - id: employees.id, - name: employees.name, - salary: employees.salary, - })), - ) + it(`optimizes single-column orderBy when passed as single value`, async () => { + // Patch getConfig to expose the builder on the returned config for test access + const { CollectionConfigBuilder } = await import( + `../../src/query/live/collection-config-builder.js` + ) + const originalGetConfig = CollectionConfigBuilder.prototype.getConfig - await collection.preload() + CollectionConfigBuilder.prototype.getConfig = function (this: any) { + const cfg = originalGetConfig.call(this) + ;(cfg as any).__builder = this + return cfg + } - const builder = (collection as any).config.__builder - expect(builder).toBeTruthy() - const orderByInfo = Object.values( - builder.optimizableOrderByCollections, - )[0] as any - const orderedSource = builder.collectionSources.find( - (source: { alias: string }) => source.alias === `employees`, - ) - expect(orderByInfo.sourceId).toBe(orderedSource.sourceId) - } finally { - CollectionConfigBuilder.prototype.getConfig = originalGetConfig - } - }, - ) - - itWhenAutoIndex( - `optimizes orderBy with alias paths in joins`, - async () => { - // Patch getConfig to expose the builder on the returned config for test access - const { CollectionConfigBuilder } = await import( - `../../src/query/live/collection-config-builder.js` + try { + const collection = createLiveQueryCollection((q) => + q + .from({ employees: employeesCollection }) + .orderBy(({ employees }) => employees.salary, `desc`) + .limit(3) + .select(({ employees }) => ({ + id: employees.id, + name: employees.name, + salary: employees.salary, + })), ) - const originalGetConfig = CollectionConfigBuilder.prototype.getConfig - - CollectionConfigBuilder.prototype.getConfig = function (this: any) { - const cfg = originalGetConfig.call(this) - ;(cfg as any).__builder = this - return cfg - } - - try { - const collection = createLiveQueryCollection((q) => - q - .from({ employees: employeesCollection }) - .join( - { departments: departmentsCollection }, - ({ employees, departments }) => - eq(employees.department_id, departments.id), - ) - .orderBy(({ departments }) => departments.name, `asc`) - .limit(5) - .select(({ employees, departments }) => ({ - employeeId: employees.id, - employeeName: employees.name, - departmentName: departments.name, - })), - ) - await collection.preload() + await collection.preload() - const builder = (collection as any).config.__builder - expect(builder).toBeTruthy() + const builder = (collection as any).config.__builder + expect(builder).toBeTruthy() + const orderByInfo = Object.values( + builder.optimizableOrderByCollections, + )[0] as any + const orderedSource = builder.collectionSources.find( + (source: { alias: string }) => source.alias === `employees`, + ) + expect(orderByInfo.sourceId).toBe(orderedSource.sourceId) + } finally { + CollectionConfigBuilder.prototype.getConfig = originalGetConfig + } + }) - // Verify that the order-by optimization is scoped to the departments alias - const orderByInfo = Object.values( - builder.optimizableOrderByCollections, - )[0] as any - const orderedSource = builder.collectionSources.find( - (source: { alias: string }) => source.alias === `departments`, - ) - expect(orderByInfo).toBeDefined() - expect(orderByInfo.alias).toBe(`departments`) - expect(orderByInfo.sourceId).toBe(orderedSource.sourceId) - expect(orderByInfo.offset).toBe(0) - expect(orderByInfo.limit).toBe(5) - } finally { - CollectionConfigBuilder.prototype.getConfig = originalGetConfig - } - }, - ) + it(`optimizes orderBy with alias paths in joins`, async () => { + // Patch getConfig to expose the builder on the returned config for test access + const { CollectionConfigBuilder } = await import( + `../../src/query/live/collection-config-builder.js` + ) + const originalGetConfig = CollectionConfigBuilder.prototype.getConfig - itWhenAutoIndex( - `loads an ordered self-join through the ordered alias`, - async () => { + CollectionConfigBuilder.prototype.getConfig = function (this: any) { + const cfg = originalGetConfig.call(this) + ;(cfg as any).__builder = this + return cfg + } + + try { const collection = createLiveQueryCollection((q) => q - .from({ employee: employeesCollection }) - .join({ manager: employeesCollection }, ({ employee, manager }) => - eq(employee.id, manager.id), + .from({ employees: employeesCollection }) + .join( + { departments: departmentsCollection }, + ({ employees, departments }) => + eq(employees.department_id, departments.id), ) - .orderBy(({ manager }) => manager.name, `asc`) - .limit(3) - .select(({ employee, manager }) => ({ - id: employee.id, - employeeName: employee.name, - managerName: manager.name, + .orderBy(({ departments }) => departments.name, `asc`) + .limit(5) + .select(({ employees, departments }) => ({ + employeeId: employees.id, + employeeName: employees.name, + departmentName: departments.name, })), ) await collection.preload() - expect( - Array.from(collection.values()).map((row) => [ - row.employeeName, - row.managerName, - ]), - ).toEqual([ - [`Alice`, `Alice`], - [`Bob`, `Bob`], - [`Charlie`, `Charlie`], - ]) - }, - ) - - itWhenAutoIndex( - `optimizes single-column orderBy when passed as array with single element`, - async () => { - // Patch getConfig to expose the builder on the returned config for test access - const { CollectionConfigBuilder } = await import( - `../../src/query/live/collection-config-builder.js` + const builder = (collection as any).config.__builder + expect(builder).toBeTruthy() + + // Verify that the order-by optimization is scoped to the departments alias + const orderByInfo = Object.values( + builder.optimizableOrderByCollections, + )[0] as any + const orderedSource = builder.collectionSources.find( + (source: { alias: string }) => source.alias === `departments`, ) - const originalGetConfig = CollectionConfigBuilder.prototype.getConfig - - CollectionConfigBuilder.prototype.getConfig = function (this: any) { - const cfg = originalGetConfig.call(this) - ;(cfg as any).__builder = this - return cfg - } - - try { - const collection = createLiveQueryCollection((q) => - q - .from({ employees: employeesCollection }) - .orderBy(({ employees }) => [employees.salary], `desc`) - .limit(3) - .select(({ employees }) => ({ - id: employees.id, - name: employees.name, - salary: employees.salary, - })), + expect(orderByInfo).toBeDefined() + expect(orderByInfo.alias).toBe(`departments`) + expect(orderByInfo.sourceId).toBe(orderedSource.sourceId) + expect(orderByInfo.offset).toBe(0) + expect(orderByInfo.limit).toBe(5) + } finally { + CollectionConfigBuilder.prototype.getConfig = originalGetConfig + } + }) + + it(`loads an ordered self-join through the ordered alias`, async () => { + const collection = createLiveQueryCollection((q) => + q + .from({ employee: employeesCollection }) + .join({ manager: employeesCollection }, ({ employee, manager }) => + eq(employee.id, manager.id), ) + .orderBy(({ manager }) => manager.name, `asc`) + .limit(3) + .select(({ employee, manager }) => ({ + id: employee.id, + employeeName: employee.name, + managerName: manager.name, + })), + ) - await collection.preload() + await collection.preload() - const builder = (collection as any).config.__builder - expect(builder).toBeTruthy() - const orderByInfo = Object.values( - builder.optimizableOrderByCollections, - )[0] as any - const orderedSource = builder.collectionSources.find( - (source: { alias: string }) => source.alias === `employees`, - ) - expect(orderByInfo.sourceId).toBe(orderedSource.sourceId) - } finally { - CollectionConfigBuilder.prototype.getConfig = originalGetConfig - } - }, - ) + expect( + Array.from(collection.values()).map((row) => [ + row.employeeName, + row.managerName, + ]), + ).toEqual([ + [`Alice`, `Alice`], + [`Bob`, `Bob`], + [`Charlie`, `Charlie`], + ]) + }) + + it(`optimizes single-column orderBy when passed as array with single element`, async () => { + // Patch getConfig to expose the builder on the returned config for test access + const { CollectionConfigBuilder } = await import( + `../../src/query/live/collection-config-builder.js` + ) + const originalGetConfig = CollectionConfigBuilder.prototype.getConfig + + CollectionConfigBuilder.prototype.getConfig = function (this: any) { + const cfg = originalGetConfig.call(this) + ;(cfg as any).__builder = this + return cfg + } + + try { + const collection = createLiveQueryCollection((q) => + q + .from({ employees: employeesCollection }) + .orderBy(({ employees }) => [employees.salary], `desc`) + .limit(3) + .select(({ employees }) => ({ + id: employees.id, + name: employees.name, + salary: employees.salary, + })), + ) + + await collection.preload() + + const builder = (collection as any).config.__builder + expect(builder).toBeTruthy() + const orderByInfo = Object.values( + builder.optimizableOrderByCollections, + )[0] as any + const orderedSource = builder.collectionSources.find( + (source: { alias: string }) => source.alias === `employees`, + ) + expect(orderByInfo.sourceId).toBe(orderedSource.sourceId) + } finally { + CollectionConfigBuilder.prototype.getConfig = originalGetConfig + } + }) }) describe(`String Comparison Tests`, () => { @@ -2676,7 +2652,7 @@ describe(`OrderBy with duplicate values`, () => { ]) // Now move to next page (offset 5, limit 5) - collection.utils.setWindow({ offset: 5, limit: 5 }) + await collection.utils.setWindow({ offset: 5, limit: 5 }) await collection.stateWhenReady() // Second page should return items 6-10 (all with value 5) @@ -2693,7 +2669,7 @@ describe(`OrderBy with duplicate values`, () => { // Now move to third page (offset 10, limit 5) // It should advance past the duplicate 5s - collection.utils.setWindow({ offset: 10, limit: 5 }) + await collection.utils.setWindow({ offset: 10, limit: 5 }) await collection.stateWhenReady() // Third page should return items 11-13 (the items after the duplicate 5s) @@ -2710,7 +2686,7 @@ describe(`OrderBy with duplicate values`, () => { ]) // Verify we can continue to next page - collection.utils.setWindow({ offset: 15, limit: 5 }) + await collection.utils.setWindow({ offset: 15, limit: 5 }) await collection.stateWhenReady() // Should be empty since we've exhausted all items @@ -2895,16 +2871,17 @@ describe(`OrderBy with duplicate values`, () => { { id: 4, a: 4, keep: true }, { id: 5, a: 5, keep: true }, ]) - expect(loadSubsetCallCount).toBe(1) + expect(loadSubsetCallCount).toBeGreaterThanOrEqual(1) + expect(loadSubsetCallCount).toBeLessThanOrEqual(2) // First loadSubset call (initial page at offset 0) has no cursor expect(loadSubsetCursors[0]).toBeUndefined() + const initialLoadSubsetCallCount = loadSubsetCallCount // Now move to next page (offset 5, limit 5) - this should trigger loadSubset with a cursor const moveToSecondPage = collection.utils.setWindow({ offset: 5, limit: 5, }) - expect(moveToSecondPage).toBeInstanceOf(Promise) await moveToSecondPage // Second page should return items 6-10 (all with value 5, loaded from sync layer) @@ -2918,12 +2895,9 @@ describe(`OrderBy with duplicate values`, () => { { id: 9, a: 5, keep: true }, { id: 10, a: 5, keep: true }, ]) - // we expect 1 new loadSubset call (cursor expressions for whereFrom/whereCurrent are now combined in single call) - expect(loadSubsetCallCount).toBe(2) - // Second loadSubset call (pagination) has a cursor with whereFrom and whereCurrent - expect(loadSubsetCursors[1]).toBeDefined() - expect(loadSubsetCursors[1]).toHaveProperty(`whereFrom`) - expect(loadSubsetCursors[1]).toHaveProperty(`whereCurrent`) + // Initial tie expansion already loaded this page; reuse it without a fetch. + expect(loadSubsetCallCount).toBe(initialLoadSubsetCallCount) + const secondPageLoadSubsetCallCount = loadSubsetCallCount // Now move to third page (offset 10, limit 5) // It should advance past the duplicate 5s @@ -2932,11 +2906,7 @@ describe(`OrderBy with duplicate values`, () => { limit: 5, }) - // Now it is `true` because we already have that page - // because when we loaded the 2nd page we loaded all the duplicate 5s and then we loaded - // values > 5 with limit 5 but since the entire 2nd page is filled with the duplicate 5s - // we in fact already loaded the third page so it is immediately available here - expect(moveToThirdPage).toBe(true) + await moveToThirdPage // Third page should return items 11-13 (the items after the duplicate 5s) // The bug would cause this to stall and return empty or get stuck @@ -2950,10 +2920,12 @@ describe(`OrderBy with duplicate values`, () => { { id: 14, a: 14, keep: true }, { id: 15, a: 15, keep: true }, ]) - // We expect no more loadSubset calls because when we loaded the previous page - // we asked for all data equal to max value and LIMIT values greater than max value - // and the LIMIT values greater than max value already loaded the next page - expect(loadSubsetCallCount).toBe(2) + expect(loadSubsetCallCount).toBeGreaterThan( + secondPageLoadSubsetCallCount, + ) + expect(loadSubsetCallCount).toBeLessThanOrEqual( + secondPageLoadSubsetCallCount + 2, + ) }) it(`should correctly advance window when there are duplicate values loaded from both local collection and sync layer`, async () => { @@ -3131,16 +3103,17 @@ describe(`OrderBy with duplicate values`, () => { { id: 4, a: 4, keep: true }, { id: 5, a: 5, keep: true }, ]) - expect(loadSubsetCallCount).toBe(1) + expect(loadSubsetCallCount).toBeGreaterThanOrEqual(1) + expect(loadSubsetCallCount).toBeLessThanOrEqual(2) // First loadSubset call (initial page at offset 0) has no cursor expect(loadSubsetCursors[0]).toBeUndefined() + const initialLoadSubsetCallCount = loadSubsetCallCount // Now move to next page (offset 5, limit 5) - this should trigger loadSubset with a cursor const moveToSecondPage = collection.utils.setWindow({ offset: 5, limit: 5, }) - expect(moveToSecondPage).toBeInstanceOf(Promise) await moveToSecondPage // Second page should return items 6-10 (all with value 5, loaded from sync layer) @@ -3154,12 +3127,9 @@ describe(`OrderBy with duplicate values`, () => { { id: 9, a: 5, keep: true }, { id: 10, a: 5, keep: true }, ]) - // we expect 1 new loadSubset call (cursor expressions for whereFrom/whereCurrent are now combined in single call) - expect(loadSubsetCallCount).toBe(2) - // Second loadSubset call (pagination) has a cursor with whereFrom and whereCurrent - expect(loadSubsetCursors[1]).toBeDefined() - expect(loadSubsetCursors[1]).toHaveProperty(`whereFrom`) - expect(loadSubsetCursors[1]).toHaveProperty(`whereCurrent`) + // Initial tie expansion already loaded this page; reuse it without a fetch. + expect(loadSubsetCallCount).toBe(initialLoadSubsetCallCount) + const secondPageLoadSubsetCallCount = loadSubsetCallCount // Now move to third page (offset 10, limit 5) // It should advance past the duplicate 5s @@ -3168,11 +3138,7 @@ describe(`OrderBy with duplicate values`, () => { limit: 5, }) - // Now it is `true` because we already have that page - // because when we loaded the 2nd page we loaded all the duplicate 5s and then we loaded - // values > 5 with limit 5 but since the entire 2nd page is filled with the duplicate 5s - // we in fact already loaded the third page so it is immediately available here - expect(moveToThirdPage).toBe(true) + await moveToThirdPage // Third page should return items 11-13 (the items after the duplicate 5s) // The bug would cause this to stall and return empty or get stuck @@ -3186,10 +3152,12 @@ describe(`OrderBy with duplicate values`, () => { { id: 14, a: 14, keep: true }, { id: 15, a: 15, keep: true }, ]) - // We expect no more loadSubset calls because when we loaded the previous page - // we asked for all data equal to max value and LIMIT values greater than max value - // and the LIMIT values greater than max value already loaded the next page - expect(loadSubsetCallCount).toBe(2) + expect(loadSubsetCallCount).toBeGreaterThan( + secondPageLoadSubsetCallCount, + ) + expect(loadSubsetCallCount).toBeLessThanOrEqual( + secondPageLoadSubsetCallCount + 2, + ) }) }) } @@ -3233,9 +3201,10 @@ describe(`OrderBy with Date values and precision differences`, () => { const initialData = testData.slice(0, 5) - // Track the cursor expressions sent to loadSubset - // Note: cursor expressions are now passed separately from where (whereFrom/whereCurrent/lastKey) + // Track both forms used by ordered loading: page cursors and boundary + // predicates. const loadSubsetCursors: Array = [] + const loadSubsetWheres: Array = [] const sourceCollection = createCollection( mockSyncCollectionOptions({ @@ -3257,6 +3226,7 @@ describe(`OrderBy with Date values and precision differences`, () => { loadSubset: (options) => { // Capture the cursor for inspection (now contains whereFrom/whereCurrent/lastKey) loadSubsetCursors.push(options.cursor) + loadSubsetWheres.push(options.where) return new Promise((resolve) => { setTimeout(() => { @@ -3362,21 +3332,28 @@ describe(`OrderBy with Date values and precision differences`, () => { // Find the cursor that contains the "whereCurrent" expression (the minValue query) // With the fix, whereCurrent should be: and(gte(createdAt, baseTime), lt(createdAt, baseTime+1ms)) // Without the fix, this would be: eq(createdAt, baseTime) - const cursorWithDateRange = loadSubsetCursors.find((cursor) => { - if (!cursor?.whereCurrent) return false - const whereCurrent = cursor.whereCurrent - // Check if whereCurrent is an 'and' with 'gte' and 'lt' (the fix) - if (whereCurrent.name === `and` && whereCurrent.args?.length === 2) { - const [first, second] = whereCurrent.args - return first?.name === `gte` && second?.name === `lt` + const findDateRange = (expression: any): any => { + if (!expression) return undefined + if (expression.name === `and` && expression.args?.length === 2) { + const [first, second] = expression.args + if (first?.name === `gte` && second?.name === `lt`) { + return expression + } } - return false - }) + return expression.args + ?.map((argument: any) => findDateRange(argument)) + .find(Boolean) + } + const equalValuesQuery = [ + ...loadSubsetWheres, + ...loadSubsetCursors.map((cursor) => cursor?.whereCurrent), + ] + .map(findDateRange) + .find(Boolean) // The fix should produce a range query (and(gte, lt)) for Date values // instead of an exact equality query (eq) - expect(cursorWithDateRange).toBeDefined() - const equalValuesQuery = cursorWithDateRange.whereCurrent + expect(equalValuesQuery).toBeDefined() expect(equalValuesQuery.name).toBe(`and`) expect(equalValuesQuery.args[0].name).toBe(`gte`) expect(equalValuesQuery.args[1].name).toBe(`lt`) diff --git a/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts b/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts new file mode 100644 index 0000000000..eec402036d --- /dev/null +++ b/packages/db/tests/query/ordered-lifecycle-oracle.property.test.ts @@ -0,0 +1,542 @@ +import { isDeepStrictEqual } from 'node:util' +import { describe, expect, it } from 'vitest' +import { fc, test as fcTest } from '@fast-check/vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createLiveQueryCollection } from '../../src/query/index.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { flushPromises } from '../utils.js' +import { + oracleRandomParameters, + readOracleRunConfig, +} from '../oracle-config.js' +import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' + +type Row = { id: number; rank: number; version: number } +type Route = `page` | `prefix` | `boundary` | `full-source` +type Scenario = { + route: Route + delivery: `before-settlement` | `after-success` + window: `keep` | `widen` + outcome: `resolve` | `reject` | `abort-error` + session: `retain` | `restart` + barrier: `initial` | `replay` + rankOffset?: number + rankStep?: number +} + +const routes: ReadonlyArray = [ + `page`, + `prefix`, + `boundary`, + `full-source`, +] + +async function observeHistory(scenario: Scenario) { + const mismatches: Array<{ law: string; actual: unknown; expected: unknown }> = + [] + const check = (law: string, actual: unknown, expected: unknown) => { + if (!isDeepStrictEqual(actual, expected)) + mismatches.push({ law, actual, expected }) + } + type Sync = Parameters[`sync`]>[0] + const truth: Array = [1, 2, 3, 4, 5].map((id) => ({ + id, + version: 1, + rank: + (scenario.rankOffset ?? 0) + + (scenario.rankStep ?? 1) * + (scenario.route === `boundary` && id === 2 ? 1 : id), + })) + const referenceWindow = (limit: number) => truth.slice(0, limit) + const gate = createDeferred() + const failure = + scenario.outcome === `abort-error` + ? Object.assign(new Error(`target canceled`), { name: `AbortError` }) + : new Error(`target rejected`) + const requests: Array<{ + options: LoadSubsetOptions + session: number + ids: Array + indexed: boolean + applied: boolean + }> = [] + const released: Array = [] + const sourceCleanups: Array = [] + const publications: Array> = [] + const deliveredRows = new Map() + let generation = 0 + let activeSync!: Sync + let activeInstalled!: Set + let targetOutcome: string | undefined + let target: (typeof requests)[number] | undefined + let targetWrites = 0 + let appliedBeforeSettlement = false + let replayStarted = false + let allowTarget = scenario.barrier === `initial` + const source = createCollection({ + id: `ordered-history-source-${JSON.stringify(scenario)}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: scenario.route === `prefix` ? `off` : `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (sync: Sync) => { + activeSync = sync + const session = ++generation + const installed = new Set() + activeInstalled = installed + sync.markReady() + return { + loadSubset: (options) => { + if (requests.length >= 30) + throw new Error(`ordered history exceeded source work bound`) + let rows = truth.filter( + (row) => + !options.where || + evaluateReferenceExpression(options.where, row) === true, + ) + if (options.cursor) + rows = rows.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + const offset = options.cursor ? 0 : (options.offset ?? 0) + rows = rows.slice( + offset, + options.limit === undefined ? undefined : offset + options.limit, + ) + const request = { + options, + session, + ids: rows.map(({ id }) => id), + indexed: source.indexes.size > 0, + applied: false, + } + requests.push(request) + const matchesRoute = + scenario.route === `boundary` + ? options.orderBy === undefined && options.where !== undefined + : scenario.route === `full-source` + ? options.limit === undefined && options.where === undefined + : options.orderBy !== undefined && options.limit !== undefined + const gated = allowTarget && !target && matchesRoute + if (gated) target = request + const apply = async () => { + if (options.signal?.aborted || session !== generation) return + request.applied = true + const fresh = rows.filter(({ id }) => !installed.has(id)) + if (fresh.length === 0) return + sync.begin() + for (const value of fresh) { + installed.add(value.id) + sync.write({ type: `insert`, value }) + if (gated) targetWrites++ + } + const receipt = sync.commit() + if (receipt !== true) await receipt + } + return (async () => { + if (gated && scenario.delivery === `before-settlement`) + await apply() + if (gated) + await gate.promise.then( + () => { + targetOutcome = `resolve` + }, + (error: unknown) => { + targetOutcome = + error === failure ? scenario.outcome : `unexpected` + throw error + }, + ) + if (!gated || scenario.delivery === `after-success`) await apply() + })() + }, + unloadSubset: (options) => { + released.push(options) + }, + cleanup: () => { + sourceCleanups.push(session) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => { + const from = q.from({ row: source }) + return (scenario.route === `full-source` ? from.distinct() : from) + .orderBy(({ row }) => row.rank) + .limit(1) + .select(({ row }) => ({ + id: row.id, + rank: row.rank, + version: row.version, + })) + }) + const read = () => + live.toArray.map(({ id, rank, version }) => ({ id, rank, version })) + const subscription = live.subscribeChanges( + (batch) => { + // subscribeChanges also sends an empty initial-snapshot completion callback. + // Count row publications only when there are row deltas. + if (batch.length === 0) return + for (const change of batch) { + const value = { + id: change.value.id, + rank: change.value.rank, + version: change.value.version, + } + if (change.type === `delete`) { + check(`delete-payload`, value, deliveredRows.get(change.key)) + deliveredRows.delete(change.key) + } else { + if (change.type === `update`) + check( + `update-previous`, + change.previousValue && { + id: change.previousValue.id, + rank: change.previousValue.rank, + version: change.previousValue.version, + }, + deliveredRows.get(change.key), + ) + else check(`insert-new-key`, deliveredRows.has(change.key), false) + deliveredRows.set(change.key, value) + } + } + const byId = (rows: Array) => + rows.slice().sort((a, b) => a.id - b.id) + check(`message-snapshot`, byId([...deliveredRows.values()]), byId(read())) + publications.push(read()) + }, + { includeInitialState: false }, + ) + const observe = (promise: Promise | true) => { + const state: { settled: boolean; error?: unknown } = { settled: false } + const done = Promise.resolve(promise).then( + () => { + state.settled = true + }, + (error: unknown) => { + state.settled = true + state.error = error + }, + ) + return { state, done } + } + const preload = observe(live.preload()) + let move: ReturnType | undefined + let baseline: Array = [] + try { + if (scenario.barrier === `replay`) { + await preload.done + expect(preload.state).toEqual({ settled: true }) + expect(read()).toEqual(referenceWindow(1)) + baseline = read() + publications.length = 0 + allowTarget = true + for (let index = 0; index < truth.length; index++) + truth[index] = { ...truth[index]!, version: 2 } + activeInstalled.clear() + activeSync.begin() + activeSync.truncate() + activeSync.commit() + replayStarted = true + } + for (let turn = 0; turn < 8 && !target; turn++) await flushPromises() + expect( + target, + JSON.stringify({ + scenario, + requests: requests.map(({ ids, options }) => ({ + ids, + limit: options.limit, + ordered: options.orderBy !== undefined, + filtered: options.where !== undefined, + })), + }), + ).toBeDefined() + expect(target!.session).toBe(1) + expect(target!.ids.length).toBeGreaterThan(0) + await flushPromises() + if (scenario.barrier === `initial`) + expect(preload.state.settled).toBe(false) + expect(read()).toEqual(baseline) + check(`pending-window`, live.utils.getWindow(), { offset: 0, limit: 1 }) + expect(publications).toEqual([]) + expect(target!.applied).toBe(scenario.delivery === `before-settlement`) + appliedBeforeSettlement = target!.applied + if (scenario.delivery === `before-settlement`) { + // A replay peer may have installed the same rows already. The provider + // still completed this read; its whole selected subset must be present. + expect(target!.ids.every((id) => source.has(id))).toBe(true) + } else expect(targetWrites).toBe(0) + if (scenario.window === `widen`) { + move = observe(live.utils.setWindow({ offset: 0, limit: 3 })) + await flushPromises() + expect(move.state.settled).toBe(false) + check(`pending-move-window`, live.utils.getWindow(), { + offset: 0, + limit: 1, + }) + expect(publications).toEqual([]) + } + if (scenario.session === `restart`) { + allowTarget = false + await live.cleanup() + await source.cleanup() + expect(target!.options.signal?.aborted).toBe(true) + expect(sourceCleanups).toEqual([1]) + await preload.done + if (scenario.barrier === `initial`) + check( + `cleanup-preload`, + preload.state.error instanceof Error + ? preload.state.error.name + : `resolved`, + `AbortError`, + ) + if (move) { + await move.done + check( + `cleanup-window`, + move.state.error instanceof Error + ? move.state.error.name + : `resolved`, + `AbortError`, + ) + } + await live.preload() + expect(generation).toBe(2) + check(`restarted-window`, read(), referenceWindow(1)) + check(`restarted-window-options`, live.utils.getWindow(), { + offset: 0, + limit: 1, + }) + } + const prior = read() + const priorStatus = live.status + const priorError = live.utils.lastSubsetError + const callbacksBeforeSettlement = publications.length + if (scenario.outcome === `resolve`) gate.resolve() + else gate.reject(failure) + for (let turn = 0; turn < 8; turn++) await flushPromises() + expect(targetOutcome).toBe(scenario.outcome) + // Deferred application happens only after a live attempt succeeds. Failure + // and old-session success must not apply its rows through this provider. + expect(target!.applied).toBe( + scenario.delivery === `before-settlement` || + (scenario.outcome === `resolve` && scenario.session === `retain`), + ) + if (scenario.session === `restart`) { + check(`obsolete-status`, live.status, priorStatus) + check(`obsolete-error`, live.utils.lastSubsetError === priorError, true) + check(`obsolete-rows`, read(), prior) + check( + `obsolete-publication`, + publications.length, + callbacksBeforeSettlement, + ) + truth[0] = { ...truth[0]!, rank: truth[0]!.rank - 1 } + activeSync.begin() + activeSync.write({ type: `update`, value: truth[0] }) + activeSync.commit() + for (let turn = 0; turn < 4; turn++) await flushPromises() + check(`restart-reactivity`, read(), referenceWindow(1)) + check( + `restart-callback`, + publications.length, + callbacksBeforeSettlement + 1, + ) + } else if (scenario.outcome === `resolve`) { + check(`success-preload`, preload.state, { settled: true }) + check(`success-window-options`, live.utils.getWindow(), { + offset: 0, + limit: scenario.window === `widen` ? 3 : 1, + }) + if (move) check(`success-window`, move.state, { settled: true }) + check( + `success-rows`, + read(), + referenceWindow(scenario.window === `widen` ? 3 : 1), + ) + const finalWindow = referenceWindow(scenario.window === `widen` ? 3 : 1) + // A move queued behind replay may follow publication of the complete old + // window, or coalesce with it. Neither path may expose a partial window. + const legalPublications = [[finalWindow]] + if (scenario.barrier === `replay` && scenario.window === `widen`) { + legalPublications.push([referenceWindow(1), finalWindow]) + } + check( + `success-publication`, + legalPublications.some((trace) => + isDeepStrictEqual(publications, trace), + ) + ? `valid` + : publications, + `valid`, + ) + } else { + if (scenario.barrier === `initial`) + check( + `failure-preload`, + { + settled: preload.state.settled, + error: + preload.state.error === failure + ? `target` + : preload.state.error === undefined + ? `none` + : `other`, + }, + { settled: true, error: `target` }, + ) + else check(`replay-error`, live.utils.lastSubsetError === failure, true) + if (move) + check( + `failure-window`, + { + settled: move.state.settled, + error: + move.state.error === failure + ? `target` + : move.state.error === undefined + ? `none` + : `other`, + }, + { settled: true, error: `target` }, + ) + check(`failure-rows`, read(), baseline) + check(`failed-window-options`, live.utils.getWindow(), { + offset: 0, + limit: 1, + }) + check(`failure-publication`, publications, []) + } + } finally { + allowTarget = false + gate.resolve() + subscription.unsubscribe() + await live.cleanup() + await source.cleanup() + await preload.done + if (move) await move.done + } + expect(new Set(released).size).toBe(released.length) + for (const { options } of requests) + expect(released.filter((release) => release === options)).toHaveLength(1) + expect(sourceCleanups).toEqual(scenario.session === `restart` ? [1, 2] : [1]) + expect(requests.every(({ options }) => options.signal?.aborted)).toBe(true) + const route = + target!.options.orderBy !== undefined + ? target!.indexed + ? `page` + : `prefix` + : target!.options.where !== undefined + ? `boundary` + : `full-source` + return { + route, + authority: + target!.options.limit === undefined && target!.options.where === undefined + ? `full` + : `finite`, + generation, + coordinates: [ + route, + appliedBeforeSettlement ? `before-settlement` : `after-success`, + move ? `widen` : `keep`, + targetOutcome, + generation === 2 ? `restart` : `retain`, + replayStarted ? `replay` : `initial`, + ], + mismatches, + } +} + +async function assertHistory(scenario: Scenario) { + const result = await observeHistory(scenario) + expect(result.route).toBe(scenario.route) + expect(result.authority).toBe( + scenario.route === `full-source` ? `full` : `finite`, + ) + expect(result.generation).toBe(scenario.session === `restart` ? 2 : 1) + expect(result.mismatches).toEqual([]) + return result +} + +describe(`ordered lifecycle product`, () => { + const observed = new Set() + const cells: Array = routes.flatMap((route) => + ([`before-settlement`, `after-success`] as const).flatMap((delivery) => + ([`keep`, `widen`] as const).flatMap((window) => + ([`resolve`, `reject`, `abort-error`] as const).flatMap((outcome) => + ([`retain`, `restart`] as const).flatMap((session) => + ([`initial`, `replay`] as const).map((barrier) => ({ + route, + delivery, + window, + outcome, + session, + barrier, + })), + ), + ), + ), + ), + ) + it(`keeps all 192 declared histories distinct`, () => { + expect(cells).toHaveLength(192) + expect(new Set(cells.map((cell) => JSON.stringify(cell))).size).toBe(192) + }) + it.each(cells)( + `$route / $delivery / $window / $outcome / $session / $barrier`, + async (scenario) => { + const result = await assertHistory(scenario) + observed.add(JSON.stringify(result.coordinates)) + }, + ) + it(`reaches all 192 histories through physical work and terminal cleanup`, () => { + expect(observed.size).toBe(192) + }) + const arbitrary = fc.record({ + route: fc.constantFrom(...routes), + delivery: fc.constantFrom( + `before-settlement` as const, + `after-success` as const, + ), + window: fc.constantFrom(`keep` as const, `widen` as const), + outcome: fc.constantFrom( + `resolve` as const, + `reject` as const, + `abort-error` as const, + ), + session: fc.constantFrom(`retain` as const, `restart` as const), + barrier: fc.constantFrom(`initial` as const, `replay` as const), + rankOffset: fc.integer({ min: -1000, max: 1000 }), + rankStep: fc.integer({ min: 1, max: 10 }), + }) + const { multiplier, ...replay } = readOracleRunConfig() + fcTest.prop([arbitrary], { numRuns: 20 * multiplier, seed: 93471 })( + `matches the ordered lifecycle for a fixed seed`, + async (scenario) => { + await assertHistory(scenario) + }, + Math.max(10000, multiplier * 1500), + ) + fcTest.prop( + [arbitrary], + oracleRandomParameters(20 * multiplier, replay, `ordered-work.lifecycle`), + )( + `matches the ordered lifecycle for a random or replayed seed`, + async (scenario) => { + await assertHistory(scenario) + }, + Math.max(10000, multiplier * 1500), + ) +}) diff --git a/packages/db/tests/query/ordered-source-loader-state.test.ts b/packages/db/tests/query/ordered-source-loader-state.test.ts new file mode 100644 index 0000000000..1fce9687e3 --- /dev/null +++ b/packages/db/tests/query/ordered-source-loader-state.test.ts @@ -0,0 +1,621 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createLiveQueryCollection } from '../../src/query/index.js' +import { OrderedSourceLoader } from '../../src/query/live/ordered-source-loader.js' +import { PropRef } from '../../src/query/ir.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { flushPromises } from '../utils.js' +import type { + CollectionSubscription, + ReleaseLoadSubset, +} from '../../src/collection/subscription.js' +import type { OrderByOptimizationInfo } from '../../src/query/compiler/order-by.js' +import type { + LoadSubsetOptions, + LoadSubsetRequestResult, +} from '../../src/types.js' + +type RequestOptions = LoadSubsetOptions & { + minValues?: Array + onLoadSubsetResult?: ( + result: LoadSubsetRequestResult, + acquisition: LoadSubsetOptions, + release: ReleaseLoadSubset, + ) => void +} + +function createDeferred() { + let resolve!: () => void + let reject!: (error: unknown) => void + const promise = new Promise((done, fail) => { + resolve = done + reject = fail + }) + return { promise, resolve, reject } +} + +function createOrderByInfo( + overrides: Partial = {}, +): OrderByOptimizationInfo { + return { + sourceId: `source`, + alias: `row`, + orderBy: [ + { + expression: new PropRef([`row`, `rank`]), + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `lexical`, + }, + }, + ], + offset: 0, + limit: 1, + comparator: (left, right) => + (left?.rank as number) - (right?.rank as number), + valueExtractorForRawRow: (row) => row.rank, + index: {} as NonNullable, + dataNeeded: () => 1, + requiresFullSource: false, + ...overrides, + } +} + +type Observed = { + method: `limited` | `snapshot` + options: RequestOptions + acquisition: LoadSubsetOptions + deferred: ReturnType +} + +function fakeSubscription( + requests: Array, + releases: Array, +) { + const request = (method: Observed[`method`], options: RequestOptions) => { + const acquisition: LoadSubsetOptions = { + orderBy: options.orderBy, + limit: options.limit, + where: options.where, + } + const deferred = createDeferred() + requests.push({ method, options, acquisition, deferred }) + options.onLoadSubsetResult?.(deferred.promise, acquisition, () => + releases.push(acquisition), + ) + } + return { + readOrderedSnapshot: () => [], + setOrderByIndex: () => {}, + requestLimitedSnapshot: (options: RequestOptions) => + request(`limited`, options), + requestSnapshot: (options: RequestOptions) => request(`snapshot`, options), + } as unknown as CollectionSubscription +} + +describe(`Ordered source request ownership`, () => { + it(`keeps a failed public window private when its older tie request finishes`, async () => { + type Row = { id: number; rank: number } + const truth: Array = [1, 2, 3].map((id) => ({ id, rank: id })) + const requests: Array<{ + kind: `page` | `boundary` | `full` + options: LoadSubsetOptions + gate: ReturnType + }> = [] + const releases: Array = [] + let hold = false + let update!: (row: Row) => void + const source = createCollection({ + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (sync) => { + const installed = new Map() + update = (row) => { + installed.set(row.id, row) + sync.begin() + sync.write({ type: `update`, value: row }) + sync.commit() + } + sync.markReady() + return { + loadSubset: async (options) => { + const kind = options.orderBy + ? `page` + : options.where + ? `boundary` + : `full` + const gate = createDeferred() + requests.push({ kind, options, gate }) + if (!hold || kind === `page`) gate.resolve() + await gate.promise + if (options.signal?.aborted) return + const rows = truth + .filter( + (row) => + (!options.where || + evaluateReferenceExpression(options.where, row) === + true) && + (!options.cursor || + evaluateReferenceExpression( + options.cursor.whereFrom, + row, + ) === true), + ) + .sort((a, b) => a.rank - b.rank) + const offset = options.cursor ? 0 : (options.offset ?? 0) + const selected = rows.slice( + offset, + options.limit === undefined + ? undefined + : offset + options.limit, + ) + sync.begin() + for (const row of selected) { + if (installed.get(row.id) === row) continue + sync.write({ + type: installed.has(row.id) ? `update` : `insert`, + value: row, + }) + installed.set(row.id, row) + } + await sync.commit() + }, + unloadSubset: (options) => { + releases.push(options) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1) + .select(({ row }) => ({ id: row.id, rank: row.rank })), + ) + const visibleRows = () => live.toArray.map(({ id, rank }) => ({ id, rank })) + const publications: Array> = [] + live.subscribeChanges( + (batch) => { + if (batch.length) publications.push(visibleRows()) + }, + { includeInitialState: false }, + ) + try { + await live.preload() + expect(visibleRows()).toEqual([{ id: 1, rank: 1 }]) + publications.length = 0 + hold = true + const move = Promise.resolve(live.utils.setWindow({ limit: 2 })).then( + () => ({ status: `fulfilled` as const }), + (error) => ({ status: `rejected` as const, error }), + ) + await flushPromises() + const boundary = requests.at(-1)! + expect(boundary.kind).toBe(`boundary`) + truth[0] = { id: 1, rank: 10 } + update(truth[0]) + await flushPromises() + const full = requests.at(-1)! + expect(full.kind).toBe(`full`) + const count = requests.length + const failure = new Error(`authoritative repair failed`) + full.gate.reject(failure) + // The window still waits for its older publication participant to settle. + await flushPromises() + boundary.gate.resolve() + await flushPromises() + expect(requests).toHaveLength(count) + expect(await move).toEqual({ status: `rejected`, error: failure }) + expect(releases).toEqual([]) + expect(publications).toEqual([]) + expect(visibleRows()).toEqual([{ id: 1, rank: 1 }]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + + hold = false + await live.utils.setWindow({ limit: 2 }) + expect(requests).toHaveLength(count + 1) + expect(releases).toEqual([full.options]) + expect(visibleRows()).toEqual([ + { id: 2, rank: 2 }, + { id: 3, rank: 3 }, + ]) + expect(publications).toEqual([ + [ + { id: 2, rank: 2 }, + { id: 3, rank: 3 }, + ], + ]) + } finally { + await live.cleanup() + for (const request of requests) request.gate.resolve() + await source.cleanup() + } + }) + + // Finite success is not authority to repair a newer full-source failure. + // Cross request kind with settlement order instead of testing each alone. + it.each( + ([`page`, `boundary`] as const).flatMap((olderKind) => + ([`older-first`, `full-first`] as const).flatMap((order) => + ([`success`, `failure`] as const).map((outcome) => ({ + olderKind, + order, + outcome, + })), + ), + ), + )( + `keeps full-source recovery authoritative across overlap: %j`, + async ({ olderKind, order, outcome }) => { + const requests: Array = [] + const releases: Array = [] + const participants: Array> = [] + const subscription = fakeSubscription(requests, releases) + subscription.readOrderedSnapshot = () => [ + { type: `insert`, key: 1, value: { id: 1, rank: 1 } }, + ] + const loader = new OrderedSourceLoader( + createOrderByInfo(), + subscription, + `row`, + (result) => { + if (result instanceof Promise) participants.push(result) + }, + ) + try { + loader.start() + if (olderKind === `boundary`) { + requests[0]!.deferred.resolve() + await participants[0] + expect(requests).toHaveLength(2) + expect(requests[1]!.options.where).toBeDefined() + } + const olderIndex = requests.length - 1 + loader.invalidateSourceOrdering() + loader.loadMore() + const fullIndex = olderIndex + 1 + const count = fullIndex + 1 + expect(requests).toHaveLength(count) + expect(requests[fullIndex]!.options.orderBy).toBeUndefined() + expect(requests[fullIndex]!.options.where).toBeUndefined() + const failure = new Error(`full-source failed`) + const settleOlder = async () => { + requests[olderIndex]!.deferred.resolve() + await participants[olderIndex] + expect(requests).toHaveLength(count) + } + const settleFull = async () => { + if (outcome === `failure`) { + requests[fullIndex]!.deferred.reject(failure) + await expect(participants[fullIndex]).rejects.toBe(failure) + } else { + requests[fullIndex]!.deferred.resolve() + await participants[fullIndex] + } + expect(requests).toHaveLength(count) + } + if (order === `older-first`) { + await settleOlder() + await settleFull() + } else { + await settleFull() + await settleOlder() + } + loader.loadMore() + expect(requests).toHaveLength(count) + expect(releases).toEqual([]) + + loader.loadMore(1) + if (outcome === `failure`) { + expect(requests).toHaveLength(count + 1) + expect(releases).toEqual([requests[fullIndex]!.acquisition]) + loader.loadMore(1) + expect(requests).toHaveLength(count + 1) + requests[count]!.deferred.resolve() + await participants[count] + } else expect(requests).toHaveLength(count) + } finally { + loader.dispose() + for (const request of requests) request.deferred.resolve() + await Promise.allSettled(participants) + } + }, + ) + + it(`a page failure while a full-source demand is held releases only the page`, async () => { + const requests: Array = [] + const releases: Array = [] + const loader = new OrderedSourceLoader( + createOrderByInfo(), + fakeSubscription(requests, releases), + `row`, + ) + loader.start() + const page = ( + loader as unknown as { pending: Promise | undefined } + ).pending! + expect(requests.map(({ method }) => method)).toEqual([`limited`]) + + // A delete during the in-flight page requires authoritative repair. + loader.invalidateSourceOrdering() + loader.loadMore() + expect(requests.map(({ method }) => method)).toEqual([ + `limited`, + `snapshot`, + ]) + expect(requests[1]!.options.limit).toBeUndefined() + const fullSource = ( + loader as unknown as { pending: Promise | undefined } + ).pending! + expect(fullSource).not.toBe(page) + + const failure = new Error(`page rejected`) + requests[0]!.deferred.reject(failure) + await expect(page).rejects.toBe(failure) + + // Blocked automatic retry; explicit retry releases the page only and must + // not issue a duplicate full-source demand while one is already held. + expect(loader.loadMore()).toBe(fullSource) + expect(requests).toHaveLength(2) + expect(releases).toEqual([]) + expect(loader.loadMore(1)).toBe(fullSource) + expect(releases).toEqual([requests[0]!.acquisition]) + expect(requests).toHaveLength(2) + + requests[1]!.deferred.resolve() + await fullSource + expect(loader.loadMore(2)).toBeUndefined() + expect(requests).toHaveLength(2) + expect(releases).toEqual([requests[0]!.acquisition]) + loader.dispose() + }) + + it(`sync full-source failure retains no demand: replay settle is a no-op and retry reissues once`, () => { + const releases: Array = [] + const methods: Array = [] + const failure = new Error(`full-source threw after callback`) + const acquisition: LoadSubsetOptions = {} + let fail = true + const subscription = { + setOrderByIndex: () => {}, + requestSnapshot: (options: RequestOptions) => { + methods.push(`snapshot`) + if (!fail) return + fail = false + options.onLoadSubsetResult?.(true, acquisition, () => + releases.push(acquisition), + ) + throw failure + }, + } as unknown as CollectionSubscription + const loader = new OrderedSourceLoader( + createOrderByInfo({ requiresFullSource: true }), + subscription, + `row`, + ) + expect(() => loader.start()).toThrow(failure) + expect(releases).toEqual([acquisition]) + expect(methods).toEqual([`snapshot`]) + + loader.settleFullSourceReplay() + expect(loader.loadMore()).toBeUndefined() + expect(methods).toEqual([`snapshot`]) + + loader.loadMore(1) + expect(methods).toEqual([`snapshot`, `snapshot`]) + expect(releases).toEqual([acquisition]) + loader.dispose() + }) + + it(`replay repairs a failed full-source demand: a later explicit retry releases nothing`, async () => { + const requests: Array = [] + const releases: Array = [] + const loader = new OrderedSourceLoader( + createOrderByInfo({ requiresFullSource: true }), + fakeSubscription(requests, releases), + `row`, + ) + loader.start() + const pending = ( + loader as unknown as { pending: Promise | undefined } + ).pending! + const failure = new Error(`full-source rejected`) + requests[0]!.deferred.reject(failure) + await expect(pending).rejects.toBe(failure) + expect(requests).toHaveLength(1) + + loader.settleFullSourceReplay() + expect(loader.loadMore(1)).toBeUndefined() + expect(releases).toEqual([]) + expect(requests).toHaveLength(1) + expect(loader.loadMore(2)).toBeUndefined() + expect(requests).toHaveLength(1) + loader.dispose() + }) + + it(`without replay the explicit retry releases and reissues exactly once`, async () => { + const requests: Array = [] + const releases: Array = [] + const loader = new OrderedSourceLoader( + createOrderByInfo({ requiresFullSource: true }), + fakeSubscription(requests, releases), + `row`, + ) + loader.start() + const pending = ( + loader as unknown as { pending: Promise | undefined } + ).pending! + requests[0]!.deferred.reject(new Error(`full-source rejected`)) + await expect(pending).rejects.toThrow(`full-source rejected`) + + loader.loadMore(1) + expect(releases).toEqual([requests[0]!.acquisition]) + expect(requests).toHaveLength(2) + loader.loadMore(1) + expect(requests).toHaveLength(2) + loader.dispose() + }) + + it(`a zero window opening with an offset requests the whole prefix from zero`, () => { + const requests: Array = [] + const releases: Array = [] + const info = createOrderByInfo({ offset: 2, limit: 0 }) + // Production dataNeeded is limit - topK size; it never adds the offset. + info.dataNeeded = () => info.limit + const loader = new OrderedSourceLoader( + info, + fakeSubscription(requests, releases), + `row`, + ) + loader.start() + expect(requests).toHaveLength(0) + + info.limit = 3 + loader.loadMore(1) + expect(requests).toHaveLength(1) + expect(requests[0]!.method).toBe(`limited`) + expect(requests[0]!.options.limit).toBe(5) + expect(requests[0]!.options.offset).toBe(0) + expect(requests[0]!.options.minValues).toBeUndefined() + loader.dispose() + }) + + it(`a failed window move keeps the snapshot; the retry loads the source once`, async () => { + type Row = { id: number; rank: number } + const truth: Array = [1, 2, 3, 4, 5, 6].map((id) => ({ id, rank: id })) + const failure = new Error(`page rejected`) + const requests: Array<{ kind: string; options: LoadSubsetOptions }> = [] + const unloads: Array = [] + let failNextCursor = false + const kindOf = (options: LoadSubsetOptions) => + options.orderBy !== undefined + ? options.cursor + ? `cursor-page` + : `page` + : options.where !== undefined + ? `boundary` + : `full` + const source = createCollection({ + id: `cut-c-probe-source`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (sync) => { + const installed = new Set() + sync.markReady() + return { + loadSubset: (options) => { + requests.push({ kind: kindOf(options), options }) + if (options.cursor && failNextCursor) { + failNextCursor = false + return Promise.reject(failure) + } + let rows = truth.filter( + (row) => + !options.where || + evaluateReferenceExpression(options.where, row) === true, + ) + if (options.cursor) + rows = rows.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + const offset = options.cursor ? 0 : (options.offset ?? 0) + rows = rows.slice( + offset, + options.limit === undefined + ? undefined + : offset + options.limit, + ) + return (async () => { + await Promise.resolve() + const fresh = rows.filter(({ id }) => !installed.has(id)) + if (fresh.length === 0) return + sync.begin() + for (const value of fresh) { + installed.add(value.id) + sync.write({ type: `insert`, value }) + } + const receipt = sync.commit() + if (receipt !== true) await receipt + })() + }, + unloadSubset: (options) => { + unloads.push(options) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2) + .select(({ row }) => ({ id: row.id, rank: row.rank })), + ) + const publications: Array> = [] + live.subscribeChanges( + (batch) => { + if (batch.length > 0) + publications.push(live.toArray.map(({ id }) => id)) + }, + { includeInitialState: false }, + ) + try { + await live.preload() + expect(live.toArray.map(({ id }) => id)).toEqual([1, 2]) + expect(requests.map(({ kind }) => kind)).toEqual([`page`, `boundary`]) + expect(unloads).toEqual([]) + const initialRequests = requests.length + publications.length = 0 + + failNextCursor = true + const move = live.utils.setWindow({ limit: 4 }) + expect(move).not.toBe(true) + await expect(move).rejects.toBe(failure) + await flushPromises() + // Rows: last settled snapshot; events: none; wait: rejected; requests: one. + expect(live.toArray.map(({ id }) => id)).toEqual([1, 2]) + expect(publications).toEqual([]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(requests.slice(initialRequests).map(({ kind }) => kind)).toEqual([ + `cursor-page`, + ]) + expect(unloads).toEqual([]) + expect(live.status).not.toBe(`error`) + + const retry = live.utils.setWindow({ limit: 4 }) + expect(retry).not.toBe(true) + await retry + await flushPromises() + expect(live.toArray.map(({ id }) => id)).toEqual([1, 2, 3, 4]) + expect(publications).toEqual([[1, 2, 3, 4]]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) + expect(requests.slice(initialRequests).map(({ kind }) => kind)).toEqual([ + `cursor-page`, + `full`, + ]) + // The explicit retry released exactly the failed page acquisition. + expect(unloads).toEqual([requests[initialRequests]!.options]) + + await flushPromises() + expect(requests).toHaveLength(initialRequests + 2) + } finally { + await live.cleanup() + await source.cleanup() + } + }) +}) diff --git a/packages/db/tests/query/ordered-source-loader.test.ts b/packages/db/tests/query/ordered-source-loader.test.ts new file mode 100644 index 0000000000..df63b6400c --- /dev/null +++ b/packages/db/tests/query/ordered-source-loader.test.ts @@ -0,0 +1,909 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { OrderedSourceLoader } from '../../src/query/live/ordered-source-loader.js' +import { Func, PropRef, Value } from '../../src/query/ir.js' +import type { + CollectionSubscription, + ReleaseLoadSubset, +} from '../../src/collection/subscription.js' +import type { OrderByOptimizationInfo } from '../../src/query/compiler/order-by.js' +import type { + LoadSubsetOptions, + LoadSubsetRequestResult, +} from '../../src/types.js' + +const pendingPromise = (loader: OrderedSourceLoader) => + (loader as unknown as { pending: Promise | undefined }).pending + +type RequestOptions = LoadSubsetOptions & { + minValues?: Array + onLoadSubsetResult?: ( + result: LoadSubsetRequestResult, + acquisition: LoadSubsetOptions, + release: ReleaseLoadSubset, + ) => void +} + +function createDeferred() { + let resolve!: () => void + let reject!: (error: unknown) => void + const promise = new Promise((done, fail) => { + resolve = done + reject = fail + }) + return { promise, resolve, reject } +} + +function createOrderByInfo( + overrides: Partial = {}, +): OrderByOptimizationInfo { + return { + sourceId: `source`, + alias: `row`, + orderBy: [ + { + expression: new PropRef([`row`, `rank`]), + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `lexical`, + }, + }, + ], + offset: 0, + limit: 1, + comparator: (left, right) => + (left?.rank as number) - (right?.rank as number), + valueExtractorForRawRow: (row) => row.rank, + index: {} as NonNullable, + dataNeeded: () => 1, + requiresFullSource: false, + ...overrides, + } +} + +describe(`OrderedSourceLoader`, () => { + const syncRouteCells = ( + [`page`, `prefix`, `boundary`, `full-source`] as const + ).flatMap((route) => + ([`success`, `throw`, `callback-then-throw`] as const).map((outcome) => ({ + route, + outcome, + })), + ) + + it.each(syncRouteCells)( + `preserves $route request semantics with synchronous $outcome`, + async ({ route, outcome }) => { + const requests: Array<{ method: string; options: RequestOptions }> = [] + const released: Array = [] + const failure = new Error(`target request failed`) + const waiting = createDeferred() + let boundaryReads = 0 + const targetIndex = route === `boundary` ? 1 : 0 + const request = (method: string, options: RequestOptions) => { + const index = requests.length + requests.push({ method, options }) + if (index !== targetIndex) { + // Bootstrap the boundary case; leave later refinement/retry in flight. + options.onLoadSubsetResult?.( + index < targetIndex ? true : waiting.promise, + options, + () => {}, + ) + return + } + if (outcome !== `throw`) { + options.onLoadSubsetResult?.(true, options, () => + released.push(options), + ) + } + if (outcome !== `success`) throw failure + } + const subscription = { + setOrderByIndex: () => {}, + readOrderedSnapshot: () => { + boundaryReads++ + return [{ value: { rank: 1 } }] + }, + requestLimitedSnapshot: (options: RequestOptions) => + request(`limited`, options), + requestSnapshot: (options: RequestOptions) => + request(`snapshot`, options), + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ + dataNeeded: () => 0, + ...(route === `prefix` ? { index: undefined } : {}), + requiresFullSource: route === `full-source`, + }), + subscription as unknown as CollectionSubscription, + `row`, + ) + try { + if (outcome !== `success` && route !== `boundary`) { + expect(() => loader.start()).toThrow(failure) + } else { + loader.start() + if (outcome === `success`) await pendingPromise(loader) + else await expect(pendingPromise(loader)).rejects.toBe(failure) + } + // Drain the synchronous boundary's own settlement as well as its parent. + await Promise.resolve() + const target = requests[targetIndex]! + expect(target.method).toBe(route === `page` ? `limited` : `snapshot`) + expect(target.options.limit).toBe( + route === `page` || route === `prefix` ? 1 : undefined, + ) + expect(Boolean(target.options.where)).toBe(route === `boundary`) + expect(released).toEqual( + outcome === `callback-then-throw` ? [target.options] : [], + ) + if (outcome === `success`) { + // Ordered loads establish a cursor and refine ties; neither a tie + // load nor a full-source load may restart that refinement step. + expect(boundaryReads).toBe(route === `full-source` ? 0 : 1) + expect(requests).toHaveLength(route === `full-source` ? 1 : 2) + if (route === `page` || route === `prefix`) { + expect(requests[1]!.options.where).toBeDefined() + expect(requests[1]!.options.orderBy).toBeUndefined() + } + } else { + const count = requests.length + loader.loadMore() + expect(requests).toHaveLength(count) + loader.loadMore(1) + expect(requests).toHaveLength(count + 1) + const retry = requests.at(-1)! + expect(retry.method).toBe(`snapshot`) + expect(retry.options.orderBy).toBeUndefined() + expect(retry.options.where).toBeUndefined() + expect(retry.options.limit).toBeUndefined() + } + } finally { + loader.dispose() + waiting.resolve() + await Promise.resolve() + } + }, + ) + + it(`recovers authoritatively when reading a settled boundary fails`, async () => { + const failure = new Error(`boundary read failed`) + const requests: Array<{ method: string; options: RequestOptions }> = [] + const released: Array = [] + const request = (method: string, options: RequestOptions) => { + requests.push({ method, options }) + options.onLoadSubsetResult?.(Promise.resolve(), options, () => + released.push(options), + ) + } + const subscription = { + setOrderByIndex: () => {}, + readOrderedSnapshot: () => { + throw failure + }, + requestLimitedSnapshot: (options: RequestOptions) => + request(`limited`, options), + requestSnapshot: (options: RequestOptions) => + request(`snapshot`, options), + } + const loader = new OrderedSourceLoader( + createOrderByInfo(), + subscription as unknown as CollectionSubscription, + `row`, + ) + loader.start() + await expect(pendingPromise(loader)).rejects.toBe(failure) + loader.loadMore() + expect(requests).toHaveLength(1) + await loader.loadMore(1) + expect(released).toEqual([requests[0]!.options]) + expect(requests.map(({ method }) => method)).toEqual([ + `limited`, + `snapshot`, + ]) + expect(requests[1]!.options.limit).toBeUndefined() + loader.dispose() + }) + + const asyncRouteCells = ( + [`page`, `prefix`, `boundary`, `full-source`] as const + ).flatMap((route) => + ( + [ + `resolve`, + `reject`, + `abort`, + `dispose-resolve`, + `dispose-reject`, + ] as const + ).map((outcome) => ({ route, outcome })), + ) + + it.each(asyncRouteCells)( + `keeps the $route acquisition lifecycle exact for $outcome`, + async ({ route, outcome }) => { + type ObservedRequest = { + method: `limited` | `snapshot` + options: RequestOptions + acquisition: LoadSubsetOptions + controller: AbortController + deferred: ReturnType + } + const requests: Array = [] + const releases: Array = [] + const request = ( + method: ObservedRequest[`method`], + options: RequestOptions, + ) => { + const controller = new AbortController() + const acquisition: LoadSubsetOptions = { + signal: controller.signal, + orderBy: options.orderBy, + limit: options.limit, + } + const deferred = createDeferred() + requests.push({ method, options, acquisition, controller, deferred }) + options.onLoadSubsetResult?.(deferred.promise, acquisition, () => + releases.push(acquisition), + ) + } + const subscription = { + readOrderedSnapshot: () => + route === `boundary` ? [{ value: { rank: 1 } }] : [], + setOrderByIndex: () => {}, + requestLimitedSnapshot: (options: RequestOptions) => + request(`limited`, options), + requestSnapshot: (options: RequestOptions) => + request(`snapshot`, options), + } + const info = createOrderByInfo( + route === `prefix` + ? { index: undefined } + : route === `full-source` + ? { requiresFullSource: true } + : {}, + ) + const loader = new OrderedSourceLoader( + info, + subscription as unknown as CollectionSubscription, + `row`, + ) + + loader.start() + if (route === `boundary`) { + expect(requests.map(({ method }) => method)).toEqual([`limited`]) + requests[0]!.deferred.resolve() + await Promise.resolve() + await Promise.resolve() + expect(requests.map(({ method }) => method)).toEqual([ + `limited`, + `snapshot`, + ]) + } + const target = requests.at(-1)! + const targetSettlement = pendingPromise(loader)! + const failure = + outcome === `abort` + ? new DOMException(`${route} canceled`, `AbortError`) + : new Error(`${route} rejected`) + + if (outcome === `dispose-resolve` || outcome === `dispose-reject`) { + loader.dispose() + if (outcome === `dispose-resolve`) target.deferred.resolve() + else target.deferred.reject(failure) + await targetSettlement + expect(requests.at(-1)).toBe(target) + expect(releases).toEqual([]) + return + } + + if (outcome === `resolve`) { + target.deferred.resolve() + await targetSettlement + expect(target.controller.signal.aborted).toBe(false) + expect(releases).toEqual([]) + } else { + if (outcome === `abort`) target.controller.abort() + target.deferred.reject(failure) + await expect(targetSettlement).rejects.toBe(failure) + expect(target.controller.signal.aborted).toBe(outcome === `abort`) + + const requestCount = requests.length + expect(loader.loadMore()).toBeUndefined() + expect(requests).toHaveLength(requestCount) + + loader.loadMore(1) + expect(releases).toEqual([target.acquisition]) + expect(requests).toHaveLength(requestCount + 1) + const retry = requests.at(-1)! + expect(retry.method).toBe(`snapshot`) + retry.deferred.resolve() + await pendingPromise(loader) + expect(releases).toEqual([target.acquisition]) + } + + loader.dispose() + }, + ) + + it.each( + ([`reset`, `dispose`] as const).flatMap((lifecycle) => + ([`resolve`, `reject`, `abort`] as const).map((outcome) => ({ + lifecycle, + outcome, + })), + ), + )( + `preserves replacement ownership after $lifecycle and obsolete $outcome`, + async ({ lifecycle, outcome }) => { + const requests: Array<{ + method: string + options: RequestOptions + deferred: ReturnType + }> = [] + const releases: Array = [] + const request = (method: string, options: RequestOptions) => { + const deferred = createDeferred() + requests.push({ method, options, deferred }) + options.onLoadSubsetResult?.(deferred.promise, options, () => + releases.push(options), + ) + } + const subscription = { + setOrderByIndex: () => {}, + readOrderedSnapshot: () => [], + requestLimitedSnapshot: (options: RequestOptions) => + request(`page`, options), + requestSnapshot: (options: RequestOptions) => + request(`full-source`, options), + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ dataNeeded: () => 0 }), + subscription as unknown as CollectionSubscription, + `row`, + ) + try { + loader.start() + const obsolete = pendingPromise(loader)! + if (lifecycle === `reset`) loader.resetCursor() + else loader.dispose() + const replacement = loader.loadMore(1) + expect(requests.map(({ method }) => method)).toEqual( + lifecycle === `reset` ? [`page`, `page`] : [`page`], + ) + if (lifecycle === `reset`) { + expect(replacement).toBeInstanceOf(Promise) + expect(requests[1]!.options.offset).toBe(0) + expect(requests[1]!.options.minValues).toBeUndefined() + } + + if (outcome === `resolve`) requests[0]!.deferred.resolve() + else { + requests[0]!.deferred.reject( + outcome === `abort` + ? new DOMException(`obsolete request canceled`, `AbortError`) + : new Error(`obsolete request failed`), + ) + } + await obsolete + expect(pendingPromise(loader)).toBe(replacement) + expect(releases).toEqual([]) + + if (lifecycle === `reset`) { + requests[1]!.deferred.resolve() + await replacement + // A successful finite replacement cannot prove that partial writes + // from the obsolete failure were repaired. Success and repair debt + // coexist; only an authoritative full-source request clears it. + loader.loadMore(2) + expect(requests.map(({ method }) => method)).toEqual( + outcome === `resolve` + ? [`page`, `page`] + : [`page`, `page`, `full-source`], + ) + if (outcome !== `resolve`) { + expect(requests[2]!.options.orderBy).toBeUndefined() + expect(requests[2]!.options.limit).toBeUndefined() + requests[2]!.deferred.resolve() + await pendingPromise(loader) + loader.loadMore(3) + expect(requests).toHaveLength(3) + } + } else { + expect(loader.loadMore(2)).toBeUndefined() + expect(requests).toHaveLength(1) + } + } finally { + loader.dispose() + requests.forEach(({ deferred }) => deferred.resolve()) + } + }, + ) + + it.each([ + { label: `undefined`, value: undefined, continuation: `full-source` }, + { label: `null`, value: null, continuation: `full-source` }, + { label: `zero`, value: 0, continuation: `tie` }, + { label: `false`, value: false, continuation: `tie` }, + { label: `empty string`, value: ``, continuation: `tie` }, + ])( + `uses $continuation for a $label boundary`, + async ({ value, continuation }) => { + const methods: Array = [] + let needed = 0 + const request = (method: string, options: RequestOptions) => { + methods.push(method) + options.onLoadSubsetResult?.(true, options, () => {}) + } + const subscription = { + setOrderByIndex: () => {}, + readOrderedSnapshot: () => [{ value: { rank: value } }], + requestLimitedSnapshot: (options: RequestOptions) => + request(`page`, options), + requestSnapshot: (options: RequestOptions) => { + const kind = options.where ? `tie` : `full-source` + expect(kind).toBe(continuation) + request(kind, options) + }, + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ dataNeeded: () => needed, comparator: () => 0 }), + subscription as unknown as CollectionSubscription, + `row`, + ) + + loader.start() + await pendingPromise(loader) + await pendingPromise(loader) + expect(methods).toEqual([`page`, continuation]) + + needed = 2 + loader.loadMore(1) + await pendingPromise(loader) + expect(methods).toEqual( + continuation === `tie` + ? [`page`, `tie`, `page`] + : [`page`, `full-source`], + ) + loader.dispose() + }, + ) + + it(`retains only bounded promise state during a long refinement chain`, async () => { + let biggest: { rank: number } | undefined + const requests: Array> = [] + const tracked: Array<{ settled: boolean }> = [] + const request = (options: RequestOptions) => { + const next = createDeferred() + requests.push(next) + options.onLoadSubsetResult?.( + next.promise, + { + orderBy: options.orderBy, + limit: options.limit, + }, + () => {}, + ) + } + const subscription = { + readOrderedSnapshot: () => (biggest ? [{ value: biggest }] : []), + setOrderByIndex: () => {}, + requestLimitedSnapshot: request, + requestSnapshot: request, + } + const info = createOrderByInfo() + const loader = new OrderedSourceLoader( + info, + subscription as unknown as CollectionSubscription, + `row`, + (promise) => { + if (!(promise instanceof Promise)) return + const participant = { settled: false } + tracked.push(participant) + void promise.then( + () => { + participant.settled = true + }, + () => { + participant.settled = true + }, + ) + }, + ) + + loader.start() + for (let step = 0; step < 20; step++) { + expect(requests[step]).toBeDefined() + if (step % 2 === 0) biggest = { rank: step / 2 } + requests[step]!.resolve() + await Promise.resolve() + await Promise.resolve() + await Promise.resolve() + } + + // One request is active and its predecessor may still be settling during + // the handoff. Earlier ancestors must already be collectible. + expect( + tracked.filter(({ settled }) => !settled).length, + ).toBeLessThanOrEqual(2) + loader.dispose() + }) + + it.each([ + { + name: `page`, + info: createOrderByInfo(), + expectedMethod: `limited`, + }, + { + name: `prefix`, + info: createOrderByInfo({ index: undefined }), + expectedMethod: `snapshot`, + }, + { + name: `full source`, + info: createOrderByInfo({ requiresFullSource: true }), + expectedMethod: `snapshot`, + }, + ])( + `keeps a callback-before-throw $name request failed until a later operation`, + async ({ info, expectedMethod }) => { + const failure = new Error(`${expectedMethod} request failed`) + const methods: Array = [] + let fail = true + const request = ( + method: string, + options: { + onLoadSubsetResult?: ( + result: true, + acquisition: LoadSubsetOptions, + release: ReleaseLoadSubset, + ) => void + }, + ) => { + methods.push(method) + if (!fail) return + fail = false + options.onLoadSubsetResult?.(true, {}, () => + subscription.releaseLoadSubset({}), + ) + loader.loadMore() + throw failure + } + const subscription = { + setOrderByIndex: () => {}, + releaseLoadSubset: (_options: LoadSubsetOptions) => {}, + requestLimitedSnapshot: (options: RequestOptions) => + request(`limited`, options), + requestSnapshot: (options: RequestOptions) => + request(`snapshot`, options), + } + const loader = new OrderedSourceLoader( + info, + subscription as unknown as CollectionSubscription, + `row`, + ) + + expect(() => loader.start()).toThrow(failure) + await Promise.resolve() + await Promise.resolve() + expect(methods).toEqual([expectedMethod]) + expect(loader.loadMore()).toBeUndefined() + expect(methods).toEqual([expectedMethod]) + + loader.loadMore(1) + expect(methods).toEqual([expectedMethod, `snapshot`]) + loader.dispose() + }, + ) + + it(`blocks retry reentered from provisional acquisition cleanup`, () => { + const failure = new Error(`prefix request failed`) + const methods: Array = [] + let fail = true + const subscription = { + setOrderByIndex: () => {}, + releaseLoadSubset: (_options: LoadSubsetOptions) => { + loader.loadMore(1) + }, + requestSnapshot: (options: RequestOptions) => { + methods.push(`snapshot`) + if (!fail) return + fail = false + options.onLoadSubsetResult?.(true, {}, () => + subscription.releaseLoadSubset({}), + ) + throw failure + }, + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ index: undefined }), + subscription as unknown as CollectionSubscription, + `row`, + ) + + expect(() => loader.start()).toThrow(failure) + expect(methods).toEqual([`snapshot`]) + + loader.loadMore(2) + expect(methods).toEqual([`snapshot`, `snapshot`]) + loader.dispose() + }) + + it(`preserves the request failure when provisional cleanup also throws`, async () => { + const requestFailure = new Error(`snapshot publication failed`) + const cleanupFailure = new Error(`provisional cleanup failed`) + const reported: Array = [] + const loads: Array = [] + const unloads: Array = [] + let failedReleaseAttempts = 0 + const source = createCollection<{ id: number; rank: number }>({ + id: `ordered-provisional-cleanup-error`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + commit() + markReady() + return { + loadSubset: (options) => { + loads.push(options) + return true + }, + unloadSubset: (options) => { + unloads.push(options) + if (options === loads[1] && ++failedReleaseAttempts === 1) { + throw cleanupFailure + } + }, + } + }, + }, + }) + const subscription = source.subscribeChanges( + (changes) => { + if (changes.length > 0) throw requestFailure + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => reported.push(error)) + const loader = new OrderedSourceLoader( + createOrderByInfo({ index: undefined }), + subscription as unknown as CollectionSubscription, + `row`, + ) + + try { + subscription.requestSnapshot({ + where: new Func(`eq`, [new PropRef([`id`]), new Value(`unrelated`)]), + optimizedOnly: false, + }) + expect(() => loader.start()).toThrow(requestFailure) + expect(subscription.lastError).toBe(requestFailure) + expect(reported).toEqual([requestFailure]) + expect(unloads).toEqual([loads[1]]) + + loader.dispose() + subscription.unsubscribe() + expect(unloads).toEqual([loads[1], loads[0]]) + expect(subscription.lastError).toBe(requestFailure) + expect(reported).toEqual([requestFailure]) + + subscription.unsubscribe() + expect(unloads).toEqual([loads[1], loads[0]]) + } finally { + loader.dispose() + subscription.unsubscribe() + await source.cleanup() + } + }) + + it.each([ + [`string`, `snapshot publication failed`], + [`undefined`, undefined], + ] as const)( + `normalizes a %s provisional failure once for every observer`, + async (_label, thrownValue) => { + const cleanupFailure = new Error(`provisional cleanup failed`) + const reported: Array = [] + let unloads = 0 + const source = createCollection<{ id: number; rank: number }>({ + id: `ordered-provisional-non-error-${String(thrownValue)}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 1, rank: 1 } }) + commit() + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloads++ + if (unloads === 1) throw cleanupFailure + }, + } + }, + }, + }) + const subscription = source.subscribeChanges( + (changes) => { + if (changes.length > 0) throw thrownValue + }, + { includeInitialState: false }, + ) + subscription.on(`loadSubset:error`, ({ error }) => reported.push(error)) + const loader = new OrderedSourceLoader( + createOrderByInfo({ index: undefined }), + subscription as unknown as CollectionSubscription, + `row`, + ) + const notCaught = Symbol(`not caught`) + let caught: unknown = notCaught + + try { + loader.start() + } catch (error) { + caught = error + } + + expect(caught).not.toBe(notCaught) + expect(caught).toBeInstanceOf(Error) + expect((caught as Error).message).toBe(String(thrownValue)) + expect(subscription.lastError).toBe(caught) + expect(reported).toEqual([caught]) + + loader.dispose() + subscription.unsubscribe() + await source.cleanup() + }, + ) + + it(`retires an acquisition when its internal result observer throws`, async () => { + const observerFailure = new Error(`ordered result observer failed`) + const acquisition: LoadSubsetOptions = {} + const methods: Array = [] + const releases: Array = [] + let failObserver = true + const subscription = { + setOrderByIndex: () => {}, + releaseLoadSubset: (options: LoadSubsetOptions) => { + releases.push(options) + loader.loadMore(1) + }, + requestSnapshot: (options: RequestOptions) => { + methods.push(`snapshot`) + options.onLoadSubsetResult?.(true, acquisition, () => + subscription.releaseLoadSubset(acquisition), + ) + }, + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ index: undefined }), + subscription as unknown as CollectionSubscription, + `row`, + () => { + if (!failObserver) return + failObserver = false + throw observerFailure + }, + ) + + expect(() => loader.start()).toThrow(observerFailure) + await Promise.resolve() + await Promise.resolve() + expect(releases).toEqual([acquisition]) + expect(methods).toEqual([`snapshot`]) + + expect(loader.loadMore()).toBeUndefined() + expect(methods).toEqual([`snapshot`]) + + loader.loadMore(1) + expect(methods).toEqual([`snapshot`, `snapshot`]) + loader.dispose() + }) + + it(`does not replace a failed acquisition while its release is running`, async () => { + const requestFailure = new Error(`ordered acquisition rejected`) + const releaseFailure = new Error(`ordered acquisition release failed`) + const acquisition: LoadSubsetOptions = {} + const methods: Array = [] + let firstRequest = true + const subscription = { + setOrderByIndex: () => {}, + releaseLoadSubset: (_options: LoadSubsetOptions) => { + loader.loadMore(2) + throw releaseFailure + }, + requestSnapshot: (options: RequestOptions) => { + methods.push(`snapshot`) + if (!firstRequest) return + firstRequest = false + options.onLoadSubsetResult?.( + Promise.reject(requestFailure), + acquisition, + () => subscription.releaseLoadSubset(acquisition), + ) + }, + } + const loader = new OrderedSourceLoader( + createOrderByInfo({ index: undefined }), + subscription as unknown as CollectionSubscription, + `row`, + ) + + loader.start() + await expect(pendingPromise(loader)).rejects.toBe(requestFailure) + expect(() => loader.loadMore(1)).toThrow(releaseFailure) + expect(methods).toEqual([`snapshot`]) + + loader.loadMore(3) + expect(methods).toEqual([`snapshot`, `snapshot`]) + loader.dispose() + }) + + it(`blocks a reentrant boundary retry until a later operation`, async () => { + const failure = new Error(`boundary request failed`) + const methods: Array = [] + let failBoundary = true + const subscription = { + readOrderedSnapshot: () => [{ value: { rank: 1 } }], + setOrderByIndex: () => {}, + releaseLoadSubset: (_options: LoadSubsetOptions) => {}, + requestLimitedSnapshot: (options: RequestOptions) => { + methods.push(`limited`) + options.onLoadSubsetResult?.( + true, + { + orderBy: options.orderBy, + limit: options.limit, + }, + () => + subscription.releaseLoadSubset({ + orderBy: options.orderBy, + limit: options.limit, + }), + ) + }, + requestSnapshot: (options: { + onLoadSubsetResult?: ( + result: true, + acquisition: LoadSubsetOptions, + release: ReleaseLoadSubset, + ) => void + }) => { + methods.push(`snapshot`) + if (!failBoundary) return + failBoundary = false + options.onLoadSubsetResult?.(true, {}, () => + subscription.releaseLoadSubset({}), + ) + loader.loadMore() + throw failure + }, + } + const loader = new OrderedSourceLoader( + createOrderByInfo(), + subscription as unknown as CollectionSubscription, + `row`, + ) + + loader.start() + const initial = pendingPromise(loader) + await expect(initial).rejects.toBe(failure) + expect(methods).toEqual([`limited`, `snapshot`]) + expect(loader.loadMore()).toBeUndefined() + expect(methods).toEqual([`limited`, `snapshot`]) + + loader.loadMore(1) + expect(methods).toEqual([`limited`, `snapshot`, `snapshot`]) + loader.dispose() + }) +}) diff --git a/packages/db/tests/query/ordered-work-oracle.property.test.ts b/packages/db/tests/query/ordered-work-oracle.property.test.ts new file mode 100644 index 0000000000..c10075a5bd --- /dev/null +++ b/packages/db/tests/query/ordered-work-oracle.property.test.ts @@ -0,0 +1,1960 @@ +import { fc, test as fcTest } from '@fast-check/vitest' +import { describe, expect, it, vi } from 'vitest' +import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' +import { BTreeIndex } from '../../src/indexes/btree-index.js' +import { createEffect } from '../../src/query/effect.js' +import { getLoadSubsetDemandKey } from '../../src/query/ir-stable-identity.js' +import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' +import { eq, gte } from '../../src/query/builder/functions.js' +import { + oracleRandomParameters, + readOracleRunConfig, +} from '../oracle-config.js' +import { evaluateReferenceExpression } from '../reference-expression.js' +import { flushPromises } from '../utils.js' +import type { InitialQueryBuilder } from '../../src/query/builder/index.js' +import type { LoadSubsetOptions, SyncConfig } from '../../src/types.js' + +type Row = { + id: number + rank: number + eligible: boolean + label: string +} + +type Marker = { id: number; rowId: number } + +type Scenario = { + middleCount: number + middleEligible: boolean + lastEligible: boolean + tied: boolean + direction: `asc` | `desc` +} + +type RequestObservation = { + kind: `page` | `boundary` + key: string | undefined + hasCursor: boolean + limit: number | undefined + offset: number | undefined + lastKey: string | number | undefined +} + +type ConsumerObservation = { + rows: Array + requests: Array + compareRequestTrace: boolean + publications: Array> + errors: Array + live: boolean +} + +const scenarioArbitrary: fc.Arbitrary = fc.record({ + middleCount: fc.constantFrom(0 as const, 1 as const, 2 as const, 3 as const), + middleEligible: fc.boolean(), + lastEligible: fc.boolean(), + tied: fc.boolean(), + direction: fc.constantFrom(`asc` as const, `desc` as const), +}) + +const exhaustiveScenarios: ReadonlyArray = ( + [0, 1, 2, 3] as const +).flatMap((middleCount) => + [false, true].flatMap((middleEligible) => + [false, true].flatMap((lastEligible) => + [false, true].flatMap((tied) => + ([`asc`, `desc`] as const).map((direction) => ({ + middleCount, + middleEligible, + lastEligible, + tied, + direction, + })), + ), + ), + ), +) + +function compareRows(direction: Scenario[`direction`]) { + return (left: Row, right: Row): number => { + const rank = left.rank - right.rank + return (direction === `asc` ? rank : -rank) || left.id - right.id + } +} + +function rowsForScenario(scenario: Scenario): Array { + return [ + { id: 1, rank: 0, eligible: true, label: `first` }, + ...Array.from({ length: scenario.middleCount }, (_, index) => ({ + id: index + 3, + rank: scenario.tied ? 0 : index + 1, + eligible: scenario.middleEligible, + label: `middle-${index}`, + })), + { + id: 2, + rank: scenario.middleCount + 1, + eligible: scenario.lastEligible, + label: `last`, + }, + ] +} + +let harnessId = 0 + +async function observeConsumer( + kind: `collection` | `effect`, + scenario: Scenario, + joinedOnlyPredicate = false, +): Promise { + type Sync = Parameters[`sync`]>[0] + const truth = rowsForScenario(scenario).sort(compareRows(scenario.direction)) + const sourceSize = truth.length + const eligibleTruth = truth.filter(({ eligible }) => eligible) + const rowToDelete = + eligibleTruth.length >= 3 && + eligibleTruth[0]!.rank !== eligibleTruth[1]!.rank + ? eligibleTruth[0] + : undefined + const delivered = new Set() + const requests: Array = [] + const errors: Array = [] + const effectRows = new Map() + let sync!: Sync + + const apply = async (rows: ReadonlyArray) => { + const fresh = rows.filter((row) => !delivered.has(row.id)) + if (fresh.length === 0) return + sync.begin() + for (const row of fresh) { + delivered.add(row.id) + sync.write({ type: `insert`, value: { ...row } }) + } + const receipt = sync.commit() + if (receipt !== true) await receipt + } + + const source = createCollection({ + id: `ordered-consumer-${kind}-${harnessId++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: async (options) => { + const isPage = options.orderBy !== undefined + requests.push({ + kind: isPage ? `page` : `boundary`, + key: getLoadSubsetDemandKey(options), + hasCursor: options.cursor !== undefined, + limit: options.limit, + offset: options.offset, + lastKey: options.cursor?.lastKey, + }) + if (requests.length > truth.length * 3 + 4) { + throw new Error(`ordered loading did not reach a fixed point`) + } + + const matching = options.where + ? truth.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === true, + ) + : truth + + if (!isPage) { + await apply(matching.filter((row) => !delivered.has(row.id))) + return + } + + const start = + options.cursor?.lastKey === undefined + ? (options.offset ?? 0) + : matching.findIndex( + ({ id }) => id === options.cursor?.lastKey, + ) + 1 + const page = matching.slice(start).slice(0, options.limit) + if (page.length > 0) { + await apply(page) + } + }, + unloadSubset: () => {}, + } + }, + }, + }) + const markers = truth + .filter(({ eligible }) => eligible) + .map(({ id }) => ({ id, rowId: id })) + const markerSource = createCollection({ + id: `ordered-marker-${kind}-${harnessId++}`, + getKey: ({ id }) => id, + syncMode: `eager`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + for (const marker of markers) { + write({ type: `insert`, value: marker }) + } + commit() + markReady() + }, + }, + }) + + let live: ReturnType | undefined + let effect: ReturnType | undefined + const publications: Array> = [] + const query = (q: InitialQueryBuilder) => { + const ordered = q + .from({ row: source }) + .leftJoin({ marker: markerSource }, ({ row, marker }) => + eq(row.id, marker.rowId), + ) + .where(({ row, marker }) => + joinedOnlyPredicate ? gte(marker.rowId, 0) : eq(row.id, marker.rowId), + ) + .orderBy(({ row }) => row.rank, scenario.direction) + return (rowToDelete ? ordered.orderBy(({ row }) => row.id, `asc`) : ordered) + .limit(2) + .select(({ row }) => ({ + id: row.id, + rank: row.rank, + eligible: row.eligible, + label: row.label, + })) + } + + const visibleRows = () => + (live ? [...live.values()] : [...effectRows.values()]) + .map(({ id, rank, eligible, label }) => ({ id, rank, eligible, label })) + .sort(compareRows(scenario.direction)) + + try { + if (kind === `collection`) { + live = createLiveQueryCollection(query) + live.subscribeChanges(() => { + publications.push(visibleRows()) + }) + await live.preload() + } else { + effect = createEffect({ + query, + onBatch: (events) => { + for (const event of events) { + if (event.type === `exit`) effectRows.delete(event.key) + else effectRows.set(event.key, { ...event.value }) + } + publications.push(visibleRows()) + }, + onSourceError: (error) => errors.push(error.message), + }) + } + + for (let turn = 0; turn < truth.length * 3 + 6; turn++) { + await flushPromises() + } + + const rows = visibleRows() + const expected = eligibleTruth.slice(0, 2) + expect(rows, JSON.stringify({ kind, scenario, requests })).toEqual(expected) + for (const publication of publications) { + expect(publication).toEqual(expected.slice(0, publication.length)) + } + const semanticPublications = publications.filter( + (publication, index) => + index === 0 || + JSON.stringify(publication) !== JSON.stringify(publications[index - 1]), + ) + for (let index = 1; index < semanticPublications.length; index++) { + expect(semanticPublications[index]!.length).toBeGreaterThan( + semanticPublications[index - 1]!.length, + ) + } + // Single-term bootstrap demand should be identical across entry points. + // Multi-term loading may schedule a different bounded number of prefix + // and tie refinements, so compare that path by rows and work bounds. + let finalRows = rows + const publicationsBeforeMutation = publications.length + if (rowToDelete) { + truth.splice(truth.indexOf(rowToDelete), 1) + delivered.delete(rowToDelete.id) + sync.begin({ immediate: true }) + sync.write({ type: `delete`, value: { ...rowToDelete } }) + const receipt = sync.commit() + if (receipt !== true) await receipt + for (let turn = 0; turn < sourceSize * 3 + 6; turn++) { + await flushPromises() + } + finalRows = visibleRows() + expect(finalRows, JSON.stringify({ kind, scenario, requests })).toEqual( + truth.filter(({ eligible }) => eligible).slice(0, 2), + ) + expect( + publications.length - publicationsBeforeMutation, + ).toBeLessThanOrEqual(1) + } + expect(publications.at(-1) ?? []).toEqual(finalRows) + // The explicit source deletion can publish without another provider call. + expect(publications.length).toBeLessThanOrEqual( + requests.length + 1 + Number(rowToDelete !== undefined), + ) + expect(requests.length).toBeLessThanOrEqual(sourceSize * 3 + 2) + expect( + requests.every( + (request) => request.kind === `boundary` || request.limit !== undefined, + ), + ).toBe(true) + + return { + rows: finalRows, + requests, + compareRequestTrace: rowToDelete === undefined, + publications, + errors, + live: live ? live.status === `ready` : effect?.disposed === false, + } + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await markerSource.cleanup() + await source.cleanup() + } +} + +async function assertConsumerParity(scenario: Scenario): Promise { + const [collection, effect] = await Promise.all([ + observeConsumer(`collection`, scenario), + observeConsumer(`effect`, scenario), + ]) + expect(effect.rows).toEqual(collection.rows) + expect(effect.errors).toEqual(collection.errors) + expect(effect.live).toBe(collection.live) + const semanticRequests = (requests: ReadonlyArray) => + requests.map(({ kind, hasCursor, limit, offset }) => ({ + kind, + hasCursor, + limit, + // Once a cursor is present, the original offset no longer changes the + // provider slice. Live collections retain it in the exact demand while + // Effects omit it, so compare the adapter-visible operation instead. + offset: hasCursor ? 0 : offset, + })) + if (effect.compareRequestTrace && collection.compareRequestTrace) { + expect(semanticRequests(effect.requests)).toEqual( + semanticRequests(collection.requests), + ) + } +} + +async function observeLaterOrderTermMutation( + kind: `collection` | `effect`, +): Promise<{ rows: Array; requests: Array }> { + const truth: Array = [ + { id: 1, rank: 0, eligible: true, label: `a` }, + { id: 2, rank: 0, eligible: true, label: `b` }, + { id: 3, rank: 0, eligible: true, label: `c` }, + ] + const delivered = new Set() + const requests: Array = [] + const effectRows = new Map() + let sync!: Parameters[`sync`]>[0] + + const source = createCollection({ + id: `ordered-later-term-${kind}-${harnessId++}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: async (options) => { + requests.push(getLoadSubsetDemandKey(options)) + let selected = options.where + ? truth.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === true, + ) + : [...truth] + selected.sort( + (left, right) => + left.rank - right.rank || + left.label.localeCompare(right.label) || + left.id - right.id, + ) + if (options.cursor) { + selected = selected.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + } else if (options.offset) { + selected = selected.slice(options.offset) + } + if (options.limit !== undefined) { + selected = selected.slice(0, options.limit) + } + const fresh = selected.filter(({ id }) => !delivered.has(id)) + if (fresh.length === 0) return + operations.begin() + for (const row of fresh) { + delivered.add(row.id) + operations.write({ type: `insert`, value: { ...row } }) + } + const receipt = operations.commit() + if (receipt !== true) await receipt + }, + unloadSubset: () => {}, + } + }, + }, + }) + const query = (q: InitialQueryBuilder) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .orderBy(({ row }) => row.label) + .limit(2) + const live = + kind === `collection` ? createLiveQueryCollection({ query }) : undefined + const effect = + kind === `effect` + ? createEffect({ + query, + onBatch: (events) => { + for (const event of events) { + if (event.type === `exit`) effectRows.delete(event.key) + else effectRows.set(event.key, { ...event.value }) + } + }, + }) + : undefined + + const visibleIds = () => + (live ? live.toArray : [...effectRows.values()]) + .sort( + (left, right) => + left.rank - right.rank || + left.label.localeCompare(right.label) || + left.id - right.id, + ) + .map(({ id }) => id) + + try { + if (live) await live.preload() + else await vi.waitFor(() => expect(visibleIds()).toEqual([1, 2])) + expect(visibleIds()).toEqual([1, 2]) + + const first = { ...source.get(1)!, label: `z` } + sync.begin({ immediate: true }) + sync.write({ type: `update`, value: { ...first } }) + const receipt = sync.commit() + if (receipt !== true) await receipt + await vi.waitFor(() => expect(visibleIds()).toEqual([2, 3])) + + expect(requests.length).toBeLessThanOrEqual(6) + return { rows: visibleIds(), requests } + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await source.cleanup() + } +} + +async function observeFinitePrefixMutation( + kind: `collection` | `effect`, + mutation: `move` | `delete`, +): Promise<{ + rows: Array + requests: number + publications: Array> +}> { + const truth = new Map([ + [1, { id: 1, rank: 0, eligible: true, label: `visible` }], + [2, { id: 2, rank: 1, eligible: true, label: `hidden` }], + ]) + const delivered = new Set() + const effectRows = new Map() + const publications: Array> = [] + let requests = 0 + let sync!: Parameters[`sync`]>[0] + + const source = createCollection({ + id: `ordered-finite-prefix-${kind}-${harnessId++}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: async (options) => { + requests++ + let selected = [...truth.values()].sort( + (left, right) => left.rank - right.rank || left.id - right.id, + ) + if (options.where) { + selected = selected.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === true, + ) + } + if (options.cursor) { + selected = selected.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + } else if (options.offset) { + selected = selected.slice(options.offset) + } + if (options.limit !== undefined) { + selected = selected.slice(0, options.limit) + } + const fresh = selected.filter(({ id }) => !delivered.has(id)) + if (fresh.length === 0) return + operations.begin() + for (const row of fresh) { + delivered.add(row.id) + operations.write({ type: `insert`, value: { ...row } }) + } + const receipt = operations.commit() + if (receipt !== true) await receipt + }, + unloadSubset: () => {}, + } + }, + }, + }) + const query = (q: InitialQueryBuilder) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1) + const live = + kind === `collection` ? createLiveQueryCollection({ query }) : undefined + const effect = + kind === `effect` + ? createEffect({ + query, + onBatch: (events) => { + for (const event of events) { + if (event.type === `exit`) effectRows.delete(event.key) + else effectRows.set(event.key, { ...event.value }) + } + publications.push( + [...effectRows.values()] + .sort((left, right) => left.rank - right.rank) + .map(({ id }) => id), + ) + }, + }) + : undefined + const visibleIds = () => + (live ? live.toArray : [...effectRows.values()]) + .sort((left, right) => left.rank - right.rank) + .map(({ id }) => id) + + try { + if (live) { + live.subscribeChanges(() => publications.push(visibleIds())) + await live.preload() + } else { + await vi.waitFor(() => expect(visibleIds()).toEqual([1])) + } + const requestsBeforeMutation = requests + const moved = { ...truth.get(1)!, rank: 10 } + if (mutation === `delete`) truth.delete(1) + else truth.set(1, moved) + sync.begin({ immediate: true }) + sync.write({ + type: mutation === `delete` ? `delete` : `update`, + value: { ...moved }, + }) + const receipt = sync.commit() + if (receipt !== true) await receipt + await vi.waitFor(() => + expect( + visibleIds(), + JSON.stringify({ kind, requests, source: source.toArray }), + ).toEqual([2]), + ) + + expect(requests).toBeGreaterThan(requestsBeforeMutation) + return { rows: visibleIds(), requests, publications } + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await source.cleanup() + } +} + +describe(`ordered source work oracle`, () => { + it(`keeps later order-term invalidation equal across consumers`, async () => { + const [collection, effect] = await Promise.all([ + observeLaterOrderTermMutation(`collection`), + observeLaterOrderTermMutation(`effect`), + ]) + expect(effect.rows).toEqual(collection.rows) + // The two graph entry points may take a different bounded number of + // refinement passes, but they must exercise the same demand forms. + expect(new Set(effect.requests)).toEqual(new Set(collection.requests)) + }) + + it.each([`move`, `delete`] as const)( + `recovers a finite source prefix equally across consumers after %s`, + async (mutation) => { + const [collection, effect] = await Promise.all([ + observeFinitePrefixMutation(`collection`, mutation), + observeFinitePrefixMutation(`effect`, mutation), + ]) + + expect(effect.rows).toEqual(collection.rows) + expect(effect.publications.at(-1)).toEqual(collection.publications.at(-1)) + }, + ) + + it(`loads each source of a filtered join once`, async () => { + type Order = { + id: number + scheduledAt: string + status: string + addressId: number + } + type Charge = { id: number; addressId: number } + let orderLoads = 0 + let chargeLoads = 0 + const orders = createCollection({ + id: `ordered-filtered-join-orders`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ + type: `insert`, + value: { + id: 1, + scheduledAt: `2024-01-15`, + status: `queued`, + addressId: 1, + }, + }) + write({ + type: `insert`, + value: { + id: 2, + scheduledAt: `2024-01-10`, + status: `queued`, + addressId: 2, + }, + }) + commit() + markReady() + return { + loadSubset: () => { + orderLoads++ + return true + }, + } + }, + }, + }) + const charges = createCollection({ + id: `ordered-filtered-join-charges`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: 10, addressId: 1 } }) + write({ type: `insert`, value: { id: 20, addressId: 2 } }) + commit() + markReady() + return { + loadSubset: () => { + chargeLoads++ + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ order: orders }) + .where(({ order }) => gte(order.scheduledAt, `2024-01-12`)) + .where(({ order }) => eq(order.status, `queued`)) + .innerJoin({ charge: charges }, ({ order, charge }) => + eq(order.addressId, charge.addressId), + ), + ) + + try { + await live.preload() + expect( + [...live.values()].map(({ order, charge }) => [order.id, charge.id]), + ).toEqual([[1, 10]]) + expect(orderLoads).toBe(1) + expect(chargeLoads).toBe(1) + } finally { + await Promise.all([live.cleanup(), orders.cleanup(), charges.cleanup()]) + } + }) + + it.each([`collection`, `effect`] as const)( + `does no source work for a zero-sized %s window`, + async (consumer) => { + let loads = 0 + const source = createCollection({ + id: `ordered-zero-window`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + loads++ + return true + }, + } + }, + }, + }) + const query = (q: InitialQueryBuilder) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(0) + const live = + consumer === `collection` + ? createLiveQueryCollection({ + id: `ordered-zero-window-live`, + query, + startSync: true, + }) + : undefined + const effect = + consumer === `effect` + ? createEffect({ query, onBatch: () => {} }) + : undefined + + try { + if (live) await live.preload() + else await flushPromises() + expect(loads).toBe(0) + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await source.cleanup() + } + }, + ) + + it.each( + ([`collection`, `effect`] as const).flatMap((consumer) => + ([`eager`, `off`] as const).map((autoIndex) => ({ + consumer, + autoIndex, + })), + ), + )( + `does no source work for a joined $consumer with a zero-sized $autoIndex window`, + async ({ consumer, autoIndex }) => { + let rowLoads = 0 + let markerLoads = 0 + const source = createCollection({ + id: `ordered-zero-window-unindexed-${consumer}-source`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex, + defaultIndexType: autoIndex === `eager` ? BTreeIndex : undefined, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + rowLoads++ + return true + }, + } + }, + }, + }) + const markers = createCollection({ + id: `ordered-zero-window-unindexed-${consumer}-marker`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex, + defaultIndexType: autoIndex === `eager` ? BTreeIndex : undefined, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + markerLoads++ + return true + }, + } + }, + }, + }) + const query = (q: InitialQueryBuilder) => + q + .from({ row: source }) + .innerJoin({ marker: markers }, ({ row, marker }) => + eq(row.id, marker.rowId), + ) + .orderBy(({ row }) => row.rank) + .limit(0) + const live = + consumer === `collection` + ? createLiveQueryCollection({ query, startSync: true }) + : undefined + const effect = + consumer === `effect` + ? createEffect({ query, onBatch: () => {} }) + : undefined + + try { + if (live) await live.preload() + else await flushPromises() + expect({ rowLoads, markerLoads }).toEqual({ + rowLoads: 0, + markerLoads: 0, + }) + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await Promise.all([source.cleanup(), markers.cleanup()]) + } + }, + ) + + it.each( + ([`collection`, `effect`] as const).flatMap((consumer) => + [2, 5].flatMap((limit) => + ([`first`, `last`] as const).flatMap((position) => + ([`asc`, `desc`] as const).map((direction) => ({ + consumer, + limit, + position, + direction, + })), + ), + ), + ), + )( + `does not refetch when a visible row changes outside the ordering key: %j`, + async ({ consumer, limit, position, direction }) => { + let sync!: Parameters[`sync`]>[0] + let loads = 0 + const rows = rowsForScenario({ + middleCount: 1, + middleEligible: true, + lastEligible: true, + tied: false, + direction, + }) + const source = createCollection({ + id: `ordered-value-update`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: async () => { + loads++ + if (loads > 1) return + operations.begin() + for (const row of rows) { + operations.write({ type: `insert`, value: { ...row } }) + } + const receipt = operations.commit() + if (receipt !== true) await receipt + }, + } + }, + }, + }) + const query = (q: InitialQueryBuilder) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, direction) + .limit(limit) + const effectRows = new Map() + const live = + consumer === `collection` ? createLiveQueryCollection(query) : undefined + const effect = + consumer === `effect` + ? createEffect({ + query, + onBatch: (events) => { + for (const event of events) { + if (event.type === `exit`) effectRows.delete(event.key) + else effectRows.set(event.key, event.value) + } + }, + }) + : undefined + const readRows = () => + (live ? [...live.values()] : [...effectRows.values()]) + .map(({ id, rank, eligible, label }) => ({ + id, + rank, + eligible, + label, + })) + .sort(compareRows(direction)) + + try { + if (live) await live.preload() + await flushPromises() + const expected = [...rows].sort(compareRows(direction)).slice(0, limit) + expect(readRows()).toEqual(expected) + const loadCount = loads + const row = position === `first` ? expected[0]! : expected.at(-1)! + sync.begin({ immediate: true }) + sync.write({ type: `update`, value: { ...row, label: `changed` } }) + sync.commit() + await flushPromises() + + expect(readRows()).toEqual( + expected.map((value) => + value.id === row.id ? { ...value, label: `changed` } : value, + ), + ) + expect(loads).toBe(loadCount) + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + await source.cleanup() + } + }, + ) + + it.each([ + { + name: `matching`, + secondaryRows: [ + { id: `c-child`, joinKey: `c` }, + { id: `d-child`, joinKey: `d` }, + ], + expected: [`c:c-child`, `d:d-child`], + }, + { + name: `empty`, + secondaryRows: [] as Array<{ id: string; joinKey: string }>, + expected: [] as Array, + }, + ])( + `waits for a $name joined source after exhausting tied ordered rows`, + async ({ name, secondaryRows, expected }) => { + type Primary = { id: string; rank: number; joinKey: string } + type Secondary = { id: string; joinKey: string } + const primaryRows: Array = [`a`, `b`, `c`, `d`].map((id) => ({ + id, + rank: 0, + joinKey: id, + })) + const deliveredPrimary = new Set() + const deliveredSecondary = new Set() + const secondaryLoads: Array<{ + options: LoadSubsetOptions + gate: ReturnType> + }> = [] + let primaryExhausted = false + + const primary = createCollection({ + id: `ordered-late-join-primary-${name}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: async (options) => { + const rows = options.orderBy + ? [ + primaryRows[ + options.cursor?.lastKey + ? primaryRows.findIndex( + ({ id }) => id === options.cursor?.lastKey, + ) + 1 + : 0 + ], + ].filter((row): row is Primary => row !== undefined) + : primaryRows.filter( + (row) => + !options.where || + evaluateReferenceExpression(options.where, row) === + true, + ) + const fresh = rows.filter(({ id }) => !deliveredPrimary.has(id)) + if (fresh.length > 0) { + begin() + for (const row of fresh) { + deliveredPrimary.add(row.id) + write({ type: `insert`, value: row }) + } + const receipt = commit(options.signal) + if (receipt !== true) await receipt + } + primaryExhausted = deliveredPrimary.size === primaryRows.length + }, + } + }, + }, + }) + const secondary = createCollection({ + id: `ordered-late-join-secondary-${name}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: async (options) => { + const gate = createDeferred() + secondaryLoads.push({ options, gate }) + await gate.promise + const fresh = secondaryRows.filter( + (row) => + !deliveredSecondary.has(row.id) && + (!options.where || + evaluateReferenceExpression(options.where, row) === true), + ) + for (const row of fresh) { + deliveredSecondary.add(row.id) + begin() + write({ type: `insert`, value: row }) + const receipt = commit(options.signal) + if (receipt !== true) await receipt + } + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ primaryRow: primary }) + .innerJoin( + { secondaryRow: secondary }, + ({ primaryRow, secondaryRow }) => + eq(primaryRow.joinKey, secondaryRow.joinKey), + ) + .orderBy(({ primaryRow }) => primaryRow.rank) + .limit(2), + ) + + try { + const preload = live.preload() + let settled = false + void preload.finally(() => { + settled = true + }) + await vi.waitFor(() => + expect( + primaryExhausted, + JSON.stringify({ + deliveredPrimary: [...deliveredPrimary], + secondaryLoads: secondaryLoads.length, + }), + ).toBe(true), + ) + expect(secondaryLoads.length).toBeGreaterThan(0) + expect(settled).toBe(false) + + for (const load of [...secondaryLoads].reverse()) { + load.gate.resolve() + await flushPromises() + } + await preload + + expect( + live.toArray.map( + ({ primaryRow, secondaryRow }) => + `${primaryRow.id}:${secondaryRow.id}`, + ), + ).toEqual(expected) + expect(live.isLoadingSubset).toBe(false) + expect(live.utils.lastSubsetError).toBeUndefined() + } finally { + for (const { gate } of secondaryLoads) gate.resolve() + await Promise.all([ + live.cleanup(), + primary.cleanup(), + secondary.cleanup(), + ]) + } + }, + ) + + it(`keeps independent joined loads isolated when they settle in reverse`, async () => { + type Primary = { id: string; rank: number; joinKey: string } + type Secondary = { id: string; joinKey: string } + const pending: Array<{ + options: LoadSubsetOptions + gate: ReturnType> + }> = [] + const completionOrder: Array = [] + const primary = createCollection({ + id: `ordered-independent-primary`, + getKey: ({ id }) => id, + syncMode: `eager`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + begin() + write({ type: `insert`, value: { id: `a`, rank: 1, joinKey: `a` } }) + write({ type: `insert`, value: { id: `b`, rank: 2, joinKey: `b` } }) + commit() + markReady() + }, + }, + }) + const secondaryRows: Array = [ + { id: `a-child`, joinKey: `a` }, + { id: `b-child`, joinKey: `b` }, + ] + const delivered = new Set() + const secondary = createCollection({ + id: `ordered-independent-secondary`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: async (options) => { + const index = pending.length + const gate = createDeferred() + pending.push({ options, gate }) + await gate.promise + const rows = secondaryRows.filter( + (candidate) => + !delivered.has(candidate.id) && + (!options.where || + evaluateReferenceExpression(options.where, candidate) === + true), + ) + for (const row of rows) { + delivered.add(row.id) + begin() + write({ type: `insert`, value: row }) + const receipt = commit(options.signal) + if (receipt !== true) await receipt + } + completionOrder.push(index) + }, + } + }, + }, + }) + const createJoined = (id: `a` | `b`) => + createLiveQueryCollection((q) => + q + .from({ primaryRow: primary }) + .where(({ primaryRow }) => eq(primaryRow.id, id)) + .innerJoin( + { secondaryRow: secondary }, + ({ primaryRow, secondaryRow }) => + eq(primaryRow.joinKey, secondaryRow.joinKey), + ) + .orderBy(({ primaryRow }) => primaryRow.rank) + .limit(1), + ) + const first = createJoined(`a`) + const second = createJoined(`b`) + + try { + const firstPreload = first.preload() + await vi.waitFor(() => expect(pending).toHaveLength(1)) + const secondPreload = second.preload() + await vi.waitFor(() => expect(pending).toHaveLength(2)) + let firstSettled = false + void firstPreload.finally(() => { + firstSettled = true + }) + + pending[1]!.gate.resolve() + await secondPreload + await flushPromises() + expect(firstSettled).toBe(false) + expect( + second.toArray.map( + ({ primaryRow, secondaryRow }) => + `${primaryRow.id}:${secondaryRow.id}`, + ), + ).toEqual([`b:b-child`]) + + pending[0]!.gate.resolve() + await firstPreload + expect(completionOrder).toEqual([1, 0]) + expect( + first.toArray.map( + ({ primaryRow, secondaryRow }) => + `${primaryRow.id}:${secondaryRow.id}`, + ), + ).toEqual([`a:a-child`]) + expect(first.utils.lastSubsetError).toBeUndefined() + expect(second.utils.lastSubsetError).toBeUndefined() + } finally { + for (const { gate } of pending) gate.resolve() + await Promise.all([ + first.cleanup(), + second.cleanup(), + primary.cleanup(), + secondary.cleanup(), + ]) + } + }) + + it(`keeps one source's replay from suppressing another source's recovery`, async () => { + type Primary = { id: number; rank: number } + type Secondary = { id: number; primaryId: number } + const primaryTruth = new Map([ + [1, { id: 1, rank: 0 }], + [2, { id: 2, rank: 1 }], + ]) + const deliveredPrimary = new Set() + const secondaryReplay = createDeferred() + let primaryRequests = 0 + let replayingSecondary = false + let primarySync!: Parameters[`sync`]>[0] + let secondarySync!: Parameters[`sync`]>[0] + let secondaryReplayCalls = 0 + + const primary = createCollection({ + id: `ordered-independent-recovery-primary`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + primarySync = operations + operations.markReady() + return { + loadSubset: async (options) => { + primaryRequests++ + let selected = [...primaryTruth.values()].sort( + (left, right) => left.rank - right.rank || left.id - right.id, + ) + if (options.where) { + selected = selected.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === true, + ) + } + if (options.cursor) { + selected = selected.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + } else if (options.offset) { + selected = selected.slice(options.offset) + } + if (options.limit !== undefined) { + selected = selected.slice(0, options.limit) + } + const fresh = selected.filter( + ({ id }) => !deliveredPrimary.has(id), + ) + if (fresh.length === 0) return + operations.begin() + for (const row of fresh) { + deliveredPrimary.add(row.id) + operations.write({ type: `insert`, value: { ...row } }) + } + const receipt = operations.commit(options.signal) + if (receipt !== true) await receipt + }, + unloadSubset: () => {}, + } + }, + }, + }) + const secondary = createCollection({ + id: `ordered-independent-recovery-secondary`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + secondarySync = operations + operations.markReady() + return { + loadSubset: async (options) => { + if (replayingSecondary) { + secondaryReplayCalls++ + await secondaryReplay.promise + } + operations.begin() + operations.write({ + type: `insert`, + value: { id: 1, primaryId: 1 }, + }) + const receipt = operations.commit(options.signal) + if (receipt !== true) await receipt + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: primary }) + .leftJoin({ child: secondary }, ({ row, child }) => + eq(row.id, child.primaryId), + ) + .orderBy(({ row }) => row.rank) + .limit(1) + .select(({ row, child }) => ({ + id: row.id, + rank: row.rank, + childId: child.id, + })), + ) + + try { + await live.preload() + expect(live.toArray.map(({ id }) => id)).toEqual([1]) + + replayingSecondary = true + secondarySync.begin() + secondarySync.truncate() + const truncateReceipt = secondarySync.commit() + if (truncateReceipt !== true) await truncateReceipt + await vi.waitFor(() => expect(secondaryReplayCalls).toBeGreaterThan(0)) + + const requestsBeforeMutation = primaryRequests + const moved = { id: 1, rank: 10 } + primaryTruth.set(1, moved) + primarySync.begin({ immediate: true }) + primarySync.write({ type: `update`, value: moved }) + const mutationReceipt = primarySync.commit() + if (mutationReceipt !== true) await mutationReceipt + await vi.waitFor(() => + expect(primaryRequests).toBeGreaterThan(requestsBeforeMutation), + ) + expect(live.toArray.map(({ id }) => id)).toEqual([1]) + + secondaryReplay.resolve() + await vi.waitFor(() => + expect(live.toArray.map(({ id }) => id)).toEqual([2]), + ) + expect(primaryRequests).toBeGreaterThan(requestsBeforeMutation) + } finally { + secondaryReplay.resolve() + await Promise.all([ + live.cleanup(), + primary.cleanup(), + secondary.cleanup(), + ]) + } + }) + + it(`publishes one complete batch after an indexed loader fills a window`, async () => { + const remoteRows: ReadonlyArray = [ + { id: 1, rank: 1, eligible: true, label: `one` }, + { id: 2, rank: 2, eligible: true, label: `two` }, + ] + const batches: Array> = [] + const callbackReads: Array> = [] + let loads = 0 + const delivered = new Set() + let sync!: Parameters[`sync`]>[0] + const source = createCollection({ + id: `ordered-atomic-indexed-window`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + loads++ + const row = remoteRows.find( + (candidate) => + !delivered.has(candidate.id) && + (!options.where || + evaluateReferenceExpression(options.where, candidate) === + true), + ) + if (!row) return true + delivered.add(row.id) + sync.begin() + sync.write({ type: `insert`, value: row }) + const receipt = sync.commit() + if (receipt !== true) { + throw new Error(`Expected synchronous source application`) + } + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(0), + ) + const readIds = () => live.toArray.map(({ id }) => id) + const subscription = live.subscribeChanges( + (changes) => { + batches.push(changes.map(({ key }) => Number(key)).sort()) + callbackReads.push(readIds()) + }, + { includeInitialState: false }, + ) + + try { + await live.preload() + await live.utils.setWindow({ offset: 0, limit: 2 }) + await flushPromises() + + // Each page turn is followed by a tie-boundary request. The source + // returns one row at a time while honoring both predicates. + expect(loads).toBe(4) + expect(readIds()).toEqual([1, 2]) + expect(batches).toEqual([[1, 2]]) + expect(callbackReads).toEqual([[1, 2]]) + } finally { + subscription.unsubscribe() + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it(`keeps live collections and Effects equal across the exhaustive small domain`, async () => { + for (const scenario of exhaustiveScenarios) { + await assertConsumerParity(scenario) + } + }) + + it(`fills ordered windows filtered only through a left-joined alias`, async () => { + for (const scenario of exhaustiveScenarios) { + const [collection, effect] = await Promise.all([ + observeConsumer(`collection`, scenario, true), + observeConsumer(`effect`, scenario, true), + ]) + expect(collection.rows).toEqual(effect.rows) + expect(collection.errors).toEqual([]) + expect(effect.errors).toEqual([]) + } + }) + + it.each( + [0, 1, 2, 3, 4].flatMap((middleCount) => + ([`asc`, `desc`] as const).flatMap((direction) => + [false, true].map((tied) => ({ middleCount, direction, tied })), + ), + ), + )( + `settles an underfilled source without repeating a continuation: %j`, + async ({ middleCount, direction, tied }) => { + const scenario: Scenario = { + middleCount, + middleEligible: false, + lastEligible: false, + tied, + direction, + } + const [collection, effect] = await Promise.all([ + observeConsumer(`collection`, scenario), + observeConsumer(`effect`, scenario), + ]) + + expect(collection.rows.map(({ id }) => id)).toEqual([1]) + expect(effect.rows).toEqual(collection.rows) + expect(effect.errors).toEqual(collection.errors) + expect(effect.live).toBe(collection.live) + for (const observation of [collection, effect]) { + expect(observation.errors).toEqual([]) + expect(observation.live).toBe(true) + // At most one page and one tie-boundary load per source row, including + // the final empty page. A fixed cap mistakes longer finite walks for loops. + const sourceSize = rowsForScenario(scenario).length + expect( + observation.requests.length, + JSON.stringify(observation.requests), + ).toBeLessThanOrEqual(2 * sourceSize) + expect( + observation.requests.filter(({ kind }) => kind === `page`).length, + ).toBeLessThanOrEqual(sourceSize) + expect( + observation.requests.filter(({ kind }) => kind === `boundary`).length, + ).toBeLessThanOrEqual(sourceSize) + expect( + new Set(observation.requests.map(({ key }) => key)).size, + JSON.stringify(observation.requests), + ).toBe(observation.requests.length) + } + }, + ) + + it(`replaces an ordered snapshot after truncate without repeating void loads`, async () => { + const initial: ReadonlyArray = [ + { id: 1, rank: 1, eligible: true, label: `old-one` }, + { id: 2, rank: 2, eligible: true, label: `old-two` }, + ] + const replacement: ReadonlyArray = [ + { id: 3, rank: 3, eligible: true, label: `new-three` }, + { id: 4, rank: 4, eligible: true, label: `new-four` }, + ] + let truth = initial + let sync!: Parameters[`sync`]>[0] + let loads = 0 + const installed = new Set() + const source = createCollection({ + id: `ordered-void-truncate`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: async (options) => { + loads++ + if (loads > 12) { + throw new Error( + `ordered void loading did not reach a fixed point`, + ) + } + const matching = options.where + ? truth.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === true, + ) + : truth + const rows = matching + .slice( + options.offset ?? 0, + options.limit === undefined + ? undefined + : (options.offset ?? 0) + options.limit, + ) + .filter(({ id }) => !installed.has(id)) + if (rows.length === 0) return + sync.begin() + for (const row of rows) { + installed.add(row.id) + sync.write({ type: `insert`, value: row }) + } + const receipt = sync.commit() + if (receipt !== true) await receipt + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + + try { + await live.preload() + expect(live.toArray.map(({ id }) => id)).toEqual([1, 2]) + + truth = replacement + installed.clear() + sync.begin() + sync.truncate() + const receipt = sync.commit() + if (receipt !== true) await receipt + await flushPromises() + + expect(live.toArray.map(({ id }) => id)).toEqual([3, 4]) + expect(loads).toBeLessThanOrEqual(8) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) + + it.each([false, true])( + `cancels queued ordered recovery on cleanup (restart=%s)`, + async (restart) => { + const seedRow: Row = { id: 1, rank: 1, eligible: true, label: `retained` } + let sync!: Parameters[`sync`]>[0] + let installed = false + const requests: Array = [] + const source = createCollection({ + id: `queued-recovery-cleanup-${harnessId++}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + requests.push(options) + if (installed) return true + installed = true + operations.begin() + operations.write({ type: `insert`, value: seedRow }) + return operations.commit() + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + try { + await live.preload() + const initialRequests = requests.length + sync.begin() + sync.truncate() + installed = false + expect(sync.commit()).toBe(true) + await live.cleanup() + await flushPromises() + expect(requests).toHaveLength(initialRequests) + if (restart) { + await live.preload() + expect(live.toArray.map(({ id }) => id)).toEqual([1]) + expect(live.utils.lastSubsetError).toBeUndefined() + expect(requests.length).toBeGreaterThan(initialRequests) + } + expect( + requests.filter( + ({ where, limit }) => where === undefined && limit === undefined, + ), + ).toEqual([]) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }, + ) + + it.each([ + { name: `until full-source recovery settles`, failure: undefined }, + { + name: `when full-source recovery throws synchronously`, + failure: { + mode: `sync` as const, + attempts: 1, + error: new Error(`full-source recovery failed`), + }, + }, + { + name: `after two synchronous full-source recovery failures`, + failure: { + mode: `sync` as const, + attempts: 2, + error: new Error(`full-source recovery failed twice`), + }, + }, + { + name: `after asynchronous full-source recovery retries`, + failure: { + mode: `async` as const, + attempts: 1, + error: new Error(`full-source recovery rejected`), + }, + }, + { + name: `after two asynchronous full-source recovery failures`, + failure: { + mode: `async` as const, + attempts: 2, + error: new Error(`full-source recovery rejected twice`), + }, + }, + ])(`keeps an ordered snapshot unchanged $name`, async ({ failure }) => { + const makeRows = (ranks: ReadonlyArray): Array => + ranks.map((rank, index) => ({ + id: index + 1, + rank, + eligible: true, + label: `row-${rank}`, + })) + let truth = makeRows([1, 2, 3, 4, 5]) + let sync!: Parameters[`sync`]>[0] + let recovering = false + let fullSourceRequests = 0 + const fullSource = createDeferred() + const installed = new Set() + const publications: Array> = [] + const escapedErrors: Array = [] + const acquisitions: Array = [] + const releases: Array = [] + const enqueueMicrotask = globalThis.queueMicrotask.bind(globalThis) + const queueMicrotaskSpy = + failure?.mode === `sync` + ? vi + .spyOn(globalThis, `queueMicrotask`) + .mockImplementation((callback) => + enqueueMicrotask(() => { + try { + callback() + } catch (error) { + escapedErrors.push(error) + } + }), + ) + : undefined + + const source = createCollection({ + id: `delayed-full-source-recovery`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + const isFullSource = + options.where === undefined && options.limit === undefined + if (recovering && isFullSource) { + fullSourceRequests++ + if (failure && fullSourceRequests <= failure.attempts) { + if (failure.mode === `sync`) throw failure.error + acquisitions.push(options) + return Promise.reject(failure.error) + } + } + acquisitions.push(options) + + return (async () => { + if (recovering && isFullSource && !failure) { + await fullSource.promise + } + + let selected = options.where + ? truth.filter( + (row) => + evaluateReferenceExpression(options.where!, row) === + true, + ) + : [...truth] + if (options.cursor) { + selected = selected.filter( + (row) => + evaluateReferenceExpression( + options.cursor!.whereFrom, + row, + ) === true, + ) + } + selected.sort((left, right) => left.rank - right.rank) + if (!options.cursor && options.offset) { + selected = selected.slice(options.offset) + } + if (options.limit !== undefined) { + selected = selected.slice(0, options.limit) + } + + const fresh = selected.filter(({ id }) => !installed.has(id)) + if (fresh.length === 0) return + sync.begin() + for (const row of fresh) { + installed.add(row.id) + sync.write({ type: `insert`, value: row }) + } + const receipt = sync.commit() + if (receipt !== true) await receipt + })() + }, + unloadSubset: (options) => { + releases.push(options) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(2), + ) + const subscription = live.subscribeChanges(() => { + publications.push(live.toArray.map(({ rank }) => rank)) + }) + + try { + await live.preload() + await live.utils.setWindow({ offset: 0, limit: 4 }) + await flushPromises() + expect(live.toArray.map(({ rank }) => rank)).toEqual([1, 2, 3, 4]) + const publicationCount = publications.length + + truth = makeRows([0, 0.5, 1, 1.5, 2, 3]) + installed.clear() + recovering = true + sync.begin() + sync.truncate() + const receipt = sync.commit() + if (receipt !== true) await receipt + await vi.waitFor(() => expect(fullSourceRequests).toBe(1)) + for (let index = 0; index < 4; index++) await flushPromises() + + expect(live.toArray.map(({ rank }) => rank)).toEqual([1, 2, 3, 4]) + expect(publications).toHaveLength(publicationCount) + + if (failure) { + expect(live.utils.lastSubsetError).toBe(failure.error) + expect(escapedErrors).toEqual([]) + truth[0] = { ...truth[0]!, label: `updated during failed recovery` } + sync.begin() + sync.write({ type: `update`, value: truth[0] }) + const updateReceipt = sync.commit() + if (updateReceipt !== true) await updateReceipt + await flushPromises() + expect(live.toArray.map(({ rank }) => rank)).toEqual([1, 2, 3, 4]) + expect(live.get(1)?.label).toBe(`row-1`) + expect(publications).toHaveLength(publicationCount) + expect(fullSourceRequests).toBe(1) + const retryCount = failure.attempts + for (let retry = 0; retry < retryCount; retry++) { + installed.clear() + sync.begin() + sync.truncate() + const retryReceipt = sync.commit() + if (retryReceipt !== true) await retryReceipt + await vi.waitFor(() => expect(fullSourceRequests).toBe(retry + 2)) + if (retry + 1 < retryCount) { + for (let index = 0; index < 4; index++) await flushPromises() + expect(live.toArray.map(({ rank }) => rank)).toEqual([1, 2, 3, 4]) + expect(publications).toHaveLength(publicationCount) + expect(live.utils.lastSubsetError).toBe(failure.error) + expect(escapedErrors).toEqual([]) + } + } + await vi.waitFor(() => + expect(source.toArray.map(({ rank }) => rank).sort()).toEqual([ + 0, 0.5, 1, 1.5, 2, 3, + ]), + ) + await vi.waitFor(() => + expect(live.toArray.map(({ rank }) => rank)).toEqual([ + 0, 0.5, 1, 1.5, + ]), + ) + expect(fullSourceRequests).toBe(retryCount + 1) + expect(live.get(1)?.label).toBe(`updated during failed recovery`) + } else { + fullSource.resolve() + await flushPromises() + expect(live.toArray.map(({ rank }) => rank)).toEqual([0, 0.5, 1, 1.5]) + } + expect(publications.slice(publicationCount)).toEqual([[0, 0.5, 1, 1.5]]) + expect(escapedErrors).toEqual([]) + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(live.toArray.map(({ rank }) => rank)).toEqual([0, 0.5]) + const loadsBeforeReplay = fullSourceRequests + installed.clear() + sync.begin() + sync.truncate() + const nextReplay = sync.commit() + if (nextReplay !== true) await nextReplay + await flushPromises() + expect(fullSourceRequests).toBeGreaterThan(loadsBeforeReplay) + expect(live.toArray.map(({ rank }) => rank)).toEqual([0, 0.5]) + } finally { + queueMicrotaskSpy?.mockRestore() + fullSource.resolve() + subscription.unsubscribe() + await Promise.all([live.cleanup(), source.cleanup()]) + } + expect(releases).toHaveLength(acquisitions.length) + for (const acquisition of acquisitions) { + expect( + releases.filter((release) => release === acquisition), + ).toHaveLength(1) + } + }) + + const { multiplier, ...replay } = readOracleRunConfig() + const runs = 20 * multiplier + + fcTest.prop([scenarioArbitrary], { numRuns: runs, seed: 17801 })( + `keeps rows, request traces, batches, errors, and liveness equal for a fixed seed`, + assertConsumerParity, + ) + + fcTest.prop( + [scenarioArbitrary], + oracleRandomParameters(runs, replay, `ordered-work.consumer-parity`), + )( + `keeps rows, request traces, batches, errors, and liveness equal for a random or replayed seed`, + assertConsumerParity, + ) +}) diff --git a/packages/db/tests/query/pagination-oracle.property.test.ts b/packages/db/tests/query/pagination-oracle.property.test.ts index 7b878a5f42..ea9638f13b 100644 --- a/packages/db/tests/query/pagination-oracle.property.test.ts +++ b/packages/db/tests/query/pagination-oracle.property.test.ts @@ -4,8 +4,9 @@ import { createCollection } from '../../src/collection/index.js' import { createDeferred } from '../../src/deferred.js' import { BTreeIndex } from '../../src/index.js' import { createLiveQueryCollection } from '../../src/query/live-query-collection.js' +import { eq } from '../../src/query/builder/functions.js' import { PropRef } from '../../src/query/ir.js' -import { expectAssertionFailure } from '../expected-failure.js' +import { makeComparator } from '../../src/utils/comparison.js' import { oracleRandomParameters, readOracleRunConfig, @@ -13,11 +14,25 @@ import { import { evaluateReferenceExpression } from '../reference-expression.js' import { TraceAssertionError } from '../trace-runner.js' import { flushPromises, mockSyncCollectionOptions } from '../utils.js' -import type { LoadSubsetOptions } from '../../src/types.js' +import type { Deferred } from '../../src/deferred.js' +import type { + ChangeMessage, + LoadSubsetOptions, + SyncConfig, +} from '../../src/types.js' type PageRow = { id: number rank: number + keep?: boolean +} + +type PublicPageRow = Pick +type PublicPageChange = { + type: `insert` | `update` | `delete` + key: number + value: PublicPageRow + previousValue?: PublicPageRow } type MultiOrderRow = { @@ -43,6 +58,17 @@ type NullableCursorRow = { rank: number | null } +type LocaleCursorRow = { + id: number + label: string +} + +type AdversarialOrderedRow = { + id: number + rank: number | null | object + label: string +} + type NullableCursorScenario = { rank: number direction: `asc` | `desc` @@ -55,22 +81,37 @@ type PaginationWindow = { type PaginationScenario = { ranks: ReadonlyArray + keeps?: ReadonlyArray direction: `asc` | `desc` windows: ReadonlyArray + explicitPublicKeyOrder?: boolean + includeFilter?: boolean + reverseInsertion?: boolean + reverseProviderTies?: boolean + localRowsBeforeFirstRequest?: ReadonlyArray } type PaginationAction = | ({ type: `window` } & PaginationWindow) - | { type: `put`; id: number; rank: number } + | { type: `put`; id: number; rank: number; keep?: boolean } | { type: `delete`; id: number } type PaginationStateScenario = { ranks: ReadonlyArray + keeps?: ReadonlyArray direction: `asc` | `desc` initialWindow: PaginationWindow actions: ReadonlyArray + explicitPublicKeyOrder?: boolean + includeFilter?: boolean + reverseInsertion?: boolean } +type PaginationStructure = Pick< + PaginationScenario, + `explicitPublicKeyOrder` | `includeFilter` | `reverseInsertion` +> + type PendingCursorLoad = { options: LoadSubsetOptions deferred: ReturnType> @@ -96,6 +137,9 @@ class DeliveredRowsTraceAssertionError extends TraceAssertionError { readonly deliveredRows: ReadonlyArray, ) { super(0, cause) + if (cause instanceof Error) { + this.message += `: ${cause.message}; delivered=${JSON.stringify(deliveredRows)}` + } } } @@ -113,20 +157,58 @@ type PendingHistoryScenario = { secondRank: number } -const scenarioArbitrary: fc.Arbitrary = fc.record({ - ranks: fc.array(fc.integer({ min: -2, max: 2 }), { +const initialRowsArbitrary = fc.array( + fc.record({ + rank: fc.integer({ min: -2, max: 2 }), + keep: fc.boolean(), + }), + { minLength: 1, maxLength: 12, - }), - direction: fc.constantFrom(`asc`, `desc`), - windows: fc.array( - fc.record({ - offset: fc.integer({ min: 0, max: 12 }), - limit: fc.integer({ min: 0, max: 8 }), - }), - { minLength: 1, maxLength: 12 }, + }, +) + +const scenarioPayloadArbitrary: fc.Arbitrary = fc + .record({ + rows: initialRowsArbitrary, + direction: fc.constantFrom(`asc` as const, `desc` as const), + reverseProviderTies: fc.boolean(), + windows: fc.array( + fc.record({ + offset: fc.integer({ min: 0, max: 12 }), + limit: fc.integer({ min: 0, max: 8 }), + }), + { minLength: 1, maxLength: 12 }, + ), + }) + .map(({ rows, ...scenario }) => ({ + ...scenario, + ranks: rows.map(({ rank }) => rank), + keeps: rows.map(({ keep }) => keep), + })) + +const paginationStructures: ReadonlyArray = [ + ...[false, true].flatMap((explicitPublicKeyOrder) => + [false, true].flatMap((includeFilter) => + [false, true].map((reverseInsertion) => ({ + explicitPublicKeyOrder, + includeFilter, + reverseInsertion, + })), + ), ), -}) +] + +const paginationStructureArbitrary: fc.Arbitrary = + fc.record({ + explicitPublicKeyOrder: fc.boolean(), + includeFilter: fc.boolean(), + reverseInsertion: fc.boolean(), + }) + +const scenarioArbitrary: fc.Arbitrary = fc + .tuple(scenarioPayloadArbitrary, paginationStructureArbitrary) + .map(([scenario, structure]) => ({ ...scenario, ...structure })) const windowArbitrary: fc.Arbitrary = fc.record({ offset: fc.integer({ min: 0, max: 12 }), @@ -147,6 +229,7 @@ const paginationActionArbitrary: fc.Arbitrary = fc.oneof( type: fc.constant(`put` as const), id: fc.integer({ min: 1, max: 16 }), rank: fc.integer({ min: -2, max: 2 }), + keep: fc.boolean(), }), }, { @@ -158,20 +241,25 @@ const paginationActionArbitrary: fc.Arbitrary = fc.oneof( }, ) -const stateScenarioArbitrary: fc.Arbitrary = fc.record( - { - ranks: fc.array(fc.integer({ min: -2, max: 2 }), { - minLength: 1, - maxLength: 12, - }), - direction: fc.constantFrom(`asc`, `desc`), +const stateScenarioPayloadArbitrary: fc.Arbitrary = fc + .record({ + rows: initialRowsArbitrary, + direction: fc.constantFrom(`asc` as const, `desc` as const), initialWindow: windowArbitrary, actions: fc.array(paginationActionArbitrary, { minLength: 1, maxLength: 20, }), - }, -) + }) + .map(({ rows, ...scenario }) => ({ + ...scenario, + ranks: rows.map(({ rank }) => rank), + keeps: rows.map(({ keep }) => keep), + })) + +const stateScenarioArbitrary: fc.Arbitrary = fc + .tuple(stateScenarioPayloadArbitrary, paginationStructureArbitrary) + .map(([scenario, structure]) => ({ ...scenario, ...structure })) const pendingMutationScenarioArbitrary: fc.Arbitrary = fc @@ -310,13 +398,25 @@ const nullableCursorScenarioArbitrary: fc.Arbitrary = direction: fc.constantFrom(`asc` as const, `desc` as const), }) -const { multiplier, replaySeed } = readOracleRunConfig() +type CleanupTarget = { + cleanup: () => unknown +} + +async function cleanupAll( + ...targets: ReadonlyArray +): Promise { + const results = await Promise.allSettled( + targets.map((target) => Promise.resolve().then(() => target.cleanup())), + ) + const rejection = results.find( + (result): result is PromiseRejectedResult => result.status === `rejected`, + ) + if (rejection) throw rejection.reason +} + +const { multiplier, ...replay } = readOracleRunConfig() const orderedScenarioRuns = 12 * multiplier const transitionScenarioRuns = 8 * multiplier -const orderedScenarioRandomParameters = oracleRandomParameters( - orderedScenarioRuns, - replaySeed, -) let collectionSequence = 0 @@ -340,24 +440,88 @@ function referenceWindowRows( left.id - right.id, ) .slice(window.offset, window.offset + window.limit) - .map((row) => ({ ...row })) + .map(({ id, rank }) => ({ id, rank })) +} + +function projectPageRow(row: PageRow): PublicPageRow { + return { id: row.id, rank: row.rank } +} + +function normalizePageChanges( + changes: ReadonlyArray>, +): Array { + return changes + .map((change) => ({ + type: change.type, + key: change.key, + value: projectPageRow(change.value), + ...(change.previousValue !== undefined + ? { previousValue: projectPageRow(change.previousValue) } + : {}), + })) + .sort((left, right) => left.key - right.key) +} + +function expectedPageChanges( + before: ReadonlyArray, + after: ReadonlyArray, +): Array { + const beforeById = new Map(before.map((row) => [row.id, row])) + const afterById = new Map(after.map((row) => [row.id, row])) + const changes: Array = [] + + for (const row of before) { + const next = afterById.get(row.id) + if (!next) { + changes.push({ type: `delete`, key: row.id, value: row }) + } else if (next.rank !== row.rank) { + changes.push({ + type: `update`, + key: row.id, + value: next, + previousValue: row, + }) + } + } + for (const row of after) { + if (!beforeById.has(row.id)) { + changes.push({ type: `insert`, key: row.id, value: row }) + } + } + return changes.sort((left, right) => left.key - right.key) +} + +function isKeptRow(id: number): boolean { + return id % 3 !== 0 +} + +function visibleRows( + rows: ReadonlyArray, + includeFilter: boolean | undefined, +): Array { + return includeFilter ? rows.filter(({ keep }) => keep) : [...rows] } function rowsForLoadSubset( rows: ReadonlyArray, options: LoadSubsetOptions, ): Array { + const matchingRows = options.where + ? rows.filter( + (row) => evaluateReferenceExpression(options.where!, row) === true, + ) + : rows if (!options.cursor) { const start = options.offset ?? 0 const end = - options.limit === undefined ? rows.length : start + options.limit - return rows.slice(start, end) + options.limit === undefined ? matchingRows.length : start + options.limit + return matchingRows.slice(start, end) } - const current = rows.filter((row) => + const current = matchingRows.filter((row) => Boolean(evaluateReferenceExpression(options.cursor!.whereCurrent, row)), ) - const from = rows.filter((row) => + const from = matchingRows.filter((row) => Boolean(evaluateReferenceExpression(options.cursor!.whereFrom, row)), ) const limitedFrom = @@ -367,46 +531,95 @@ function rowsForLoadSubset( return [...requested.values()] } +function createConformingOrderedSource( + id: string, + rows: ReadonlyArray, + autoIndex: `eager` | `off` = `eager`, +) { + const requests: Array = [] + const delivered = new Set() + const source = createCollection({ + id, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + const requested = rowsForLoadSubset(rows, options) + begin() + for (const row of requested) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: row }) + } + const receipt = commit(options.signal) + return receipt === true ? Promise.resolve() : receipt + }, + } + }, + }, + }) + + return { requests, source } +} + async function runPaginationScenario( scenario: PaginationScenario, ): Promise { - const rows = scenario.ranks.map((rank, index) => ({ id: index + 1, rank })) + const rows = scenario.ranks.map((rank, index) => ({ + id: index + 1, + rank, + keep: scenario.keeps?.[index] ?? isKeptRow(index + 1), + })) + const initialRows = scenario.reverseInsertion ? [...rows].reverse() : rows + const expectedRows = visibleRows(rows, scenario.includeFilter) const initialWindow = scenario.windows[0]! const source = createCollection( mockSyncCollectionOptions({ id: `pagination-oracle-source-${collectionSequence++}`, - initialData: rows.map((row) => ({ ...row })), + initialData: initialRows.map((row) => ({ ...row })), getKey: (row: PageRow) => row.id, autoIndex: `eager`, }), ) - const live = createLiveQueryCollection((query) => - query - .from({ row: source }) - .orderBy(({ row }) => row.rank, scenario.direction) - .orderBy(({ row }) => row.id, `asc`) + const live = createLiveQueryCollection((query) => { + const from = query.from({ row: source }) + const filtered = scenario.includeFilter + ? from.where(({ row }) => eq(row.keep, true)) + : from + const ordered = filtered.orderBy(({ row }) => row.rank, scenario.direction) + return ( + scenario.explicitPublicKeyOrder === false + ? ordered + : ordered.orderBy(({ row }) => row.id, `asc`) + ) .offset(initialWindow.offset) .limit(initialWindow.limit) - .select(({ row }) => ({ id: row.id, rank: row.rank })), - ) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + }) try { await live.preload() - expect(Array.from(live.values(), ({ id }) => id)).toEqual( - referenceWindow(rows, scenario.direction, initialWindow), + expect(Array.from(live.values(), ({ id, rank }) => ({ id, rank }))).toEqual( + referenceWindowRows(expectedRows, scenario.direction, initialWindow), ) for (const window of scenario.windows.slice(1)) { const result = live.utils.setWindow(window) if (result instanceof Promise) await result - expect(Array.from(live.values(), ({ id }) => id)).toEqual( - referenceWindow(rows, scenario.direction, window), - ) + expect( + Array.from(live.values(), ({ id, rank }) => ({ id, rank })), + ).toEqual(referenceWindowRows(expectedRows, scenario.direction, window)) } } finally { - live.cleanup() - source.cleanup() + await cleanupAll(live, source) } } @@ -461,39 +674,6 @@ function referenceMultiOrder(scenario: MultiOrderScenario): Array { .map(({ id }) => id) } -function referenceMultiOrderWithoutSecondary( - scenario: MultiOrderScenario, -): Array { - // The current top-K boundary selects rows by the first order term and key, - // then applies the full comparator only to the rows that survived selection. - const selectedIds = new Set( - [...scenario.rows] - .sort( - (left, right) => - compareNullableNumber( - left.primary, - right.primary, - scenario.primary, - ) || left.id - right.id, - ) - .slice(0, scenario.limit) - .map(({ id }) => id), - ) - return scenario.rows - .filter(({ id }) => selectedIds.has(id)) - .sort( - (left, right) => - compareNullableNumber(left.primary, right.primary, scenario.primary) || - compareNullableNumber( - left.secondary, - right.secondary, - scenario.secondary, - ) || - left.id - right.id, - ) - .map(({ id }) => id) -} - async function runMultiOrderScenario( scenario: MultiOrderScenario, ): Promise { @@ -525,45 +705,7 @@ async function runMultiOrderScenario( throw new TraceAssertionError(0, error) } } finally { - live.cleanup() - source.cleanup() - } -} - -function isKnownSecondaryOrderBoundaryFailure( - scenario: MultiOrderScenario, - error: unknown, -): boolean { - if ( - !(error instanceof TraceAssertionError) || - error.checkpoint !== 0 || - typeof error.cause !== `object` || - error.cause === null || - !(`actual` in error.cause) || - !(`expected` in error.cause) || - !isNumberArray(error.cause.actual) || - !isNumberArray(error.cause.expected) - ) { - return false - } - - const expected = referenceMultiOrder(scenario) - const defective = referenceMultiOrderWithoutSecondary(scenario) - return ( - defective.join(`,`) !== expected.join(`,`) && - error.cause.actual.join(`,`) === defective.join(`,`) && - error.cause.expected.join(`,`) === expected.join(`,`) - ) -} - -async function runMultiOrderScenarioWithKnownFailures( - scenario: MultiOrderScenario, -): Promise { - try { - await runMultiOrderScenario(scenario) - } catch (error) { - if (isKnownSecondaryOrderBoundaryFailure(scenario, error)) return - throw error + await cleanupAll(live, source) } } @@ -582,6 +724,7 @@ async function runNullableCursorScenario( }) || left.id - right.id, ) const pending: Array = [] + const delivered = new Set() let begin!: () => void let write!: (message: { type: `insert`; value: NullableCursorRow }) => void let commit!: () => void @@ -621,15 +764,22 @@ async function runNullableCursorScenario( try { const preload = live.preload() - expect(pending).toHaveLength(1) - const request = pending[0]! - begin() - for (const row of rowsForLoadSubset(orderedRows, request.options)) { - write({ type: `insert`, value: { ...row } }) + expect(pending.length).toBeGreaterThan(0) + // Settling one request can append its boundary-refinement request. + // eslint-disable-next-line @typescript-eslint/prefer-for-of + for (let index = 0; index < pending.length; index++) { + const request = pending[index]! + begin() + for (const row of rowsForLoadSubset(orderedRows, request.options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.settled = true + request.deferred.resolve() + await flushPromises() } - commit() - request.settled = true - request.deferred.resolve() await preload try { @@ -639,46 +789,7 @@ async function runNullableCursorScenario( } } finally { for (const request of pending) request.deferred.resolve() - live.cleanup() - source.cleanup() - } -} - -// The current ascending cursor boundary can place the non-null row before the -// nulls-first row. Remove this waiver when that request returns row 1. -function isKnownNullableCursorOrderingFailure( - scenario: NullableCursorScenario, - error: unknown, -): boolean { - if ( - scenario.direction !== `asc` || - !(error instanceof TraceAssertionError) || - error.checkpoint !== 0 || - typeof error.cause !== `object` || - error.cause === null || - !(`actual` in error.cause) || - !(`expected` in error.cause) - ) { - return false - } - return ( - isNumberArray(error.cause.actual) && - error.cause.actual.length === 1 && - error.cause.actual[0] === 2 && - isNumberArray(error.cause.expected) && - error.cause.expected.length === 1 && - error.cause.expected[0] === 1 - ) -} - -async function runNullableCursorScenarioWithKnownFailures( - scenario: NullableCursorScenario, -): Promise { - try { - await runNullableCursorScenario(scenario) - } catch (error) { - if (isKnownNullableCursorOrderingFailure(scenario, error)) return - throw error + await cleanupAll(live, source) } } @@ -686,33 +797,56 @@ async function runPaginationStateScenario( scenario: PaginationStateScenario, ): Promise { const rows = new Map( - scenario.ranks.map((rank, index) => [index + 1, { id: index + 1, rank }]), + scenario.ranks.map((rank, index) => [ + index + 1, + { + id: index + 1, + rank, + keep: scenario.keeps?.[index] ?? isKeptRow(index + 1), + }, + ]), ) + const initialRows = [...rows.values()] + if (scenario.reverseInsertion) initialRows.reverse() let currentWindow = scenario.initialWindow const sourceOptions = mockSyncCollectionOptions({ id: `pagination-state-oracle-source-${collectionSequence++}`, - initialData: [...rows.values()].map((row) => ({ ...row })), + initialData: initialRows.map((row) => ({ ...row })), getKey: (row: PageRow) => row.id, autoIndex: `eager` as const, }) const source = createCollection(sourceOptions) - const live = createLiveQueryCollection((query) => - query - .from({ row: source }) - .orderBy(({ row }) => row.rank, scenario.direction) - .orderBy(({ row }) => row.id, `asc`) + const live = createLiveQueryCollection((query) => { + const from = query.from({ row: source }) + const filtered = scenario.includeFilter + ? from.where(({ row }) => eq(row.keep, true)) + : from + const ordered = filtered.orderBy(({ row }) => row.rank, scenario.direction) + return ( + scenario.explicitPublicKeyOrder === false + ? ordered + : ordered.orderBy(({ row }) => row.id, `asc`) + ) .offset(currentWindow.offset) .limit(currentWindow.limit) - .select(({ row }) => ({ id: row.id, rank: row.rank })), - ) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + }) + const publications: Array<{ + changes: Array + rows: Array + }> = [] + let publicationSubscription: + | ReturnType + | undefined + + const readCurrentWindow = () => + Array.from(live.values(), ({ id, rank }) => ({ id, rank })) const expectCurrentWindow = (checkpoint: number) => { try { - expect( - Array.from(live.values(), ({ id, rank }) => ({ id, rank })), - ).toEqual( + expect(readCurrentWindow()).toEqual( referenceWindowRows( - [...rows.values()], + visibleRows([...rows.values()], scenario.includeFilter), scenario.direction, currentWindow, ), @@ -725,14 +859,32 @@ async function runPaginationStateScenario( try { await live.preload() expectCurrentWindow(0) + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBeUndefined() + publicationSubscription = live.subscribeChanges( + (changes) => + publications.push({ + changes: normalizePageChanges( + changes as Array>, + ), + rows: readCurrentWindow(), + }), + { includeInitialState: false }, + ) for (const [index, action] of scenario.actions.entries()) { + const beforeRows = readCurrentWindow() + const publicationCount = publications.length if (action.type === `window`) { currentWindow = { offset: action.offset, limit: action.limit } const result = live.utils.setWindow(currentWindow) if (result instanceof Promise) await result } else if (action.type === `put`) { - const row = { id: action.id, rank: action.rank } + const row = { + id: action.id, + rank: action.rank, + keep: action.keep ?? isKeptRow(action.id), + } const type = rows.has(action.id) ? `update` : `insert` rows.set(action.id, row) sourceOptions.utils.begin() @@ -748,438 +900,209 @@ async function runPaginationStateScenario( } } expectCurrentWindow(index + 1) + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBeUndefined() + const afterRows = readCurrentWindow() + const expectedChanges = expectedPageChanges(beforeRows, afterRows) + expect(publications.slice(publicationCount)).toEqual( + expectedChanges.length > 0 + ? [{ changes: expectedChanges, rows: afterRows }] + : [], + ) } } finally { - live.cleanup() - source.cleanup() + publicationSubscription?.unsubscribe() + await cleanupAll(live, source) } } -type ReferencePaginationState = { - rows: Map - window: PaginationWindow -} - -function replayReferenceState( - scenario: PaginationStateScenario, - actionCount: number, -): ReferencePaginationState { - const state: ReferencePaginationState = { - rows: new Map( - scenario.ranks.map((rank, index) => [index + 1, { id: index + 1, rank }]), - ), - window: { ...scenario.initialWindow }, - } +async function runOnDemandPaginationScenario( + scenario: PaginationScenario, + assertLoads?: (loads: ReadonlyArray) => void, +): Promise { + const authoritativeRows = scenario.ranks.map((rank, index) => ({ + id: index + 1, + rank, + keep: scenario.keeps?.[index] ?? isKeptRow(index + 1), + })) + const expectedRows = visibleRows(authoritativeRows, scenario.includeFilter) + const directionFactor = scenario.direction === `asc` ? 1 : -1 + const orderedRows = [...authoritativeRows].sort( + (left, right) => + (left.rank - right.rank) * directionFactor || + (left.id - right.id) * + (scenario.explicitPublicKeyOrder === false && + scenario.reverseProviderTies + ? -1 + : 1), + ) + const deliveredIds = new Set() + const loads: Array = [] + const initialWindow = scenario.windows[0]! + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void - for (const action of scenario.actions.slice(0, actionCount)) { - if (action.type === `window`) { - state.window = { offset: action.offset, limit: action.limit } - } else if (action.type === `put`) { - state.rows.set(action.id, { id: action.id, rank: action.rank }) - } else { - state.rows.delete(action.id) - } - } - return state -} + const source = createCollection({ + id: `pagination-on-demand-oracle-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + const { markReady } = operations + markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + loads.push({ ...options }) + const requested = rowsForLoadSubset(orderedRows, options) + const delivered = scenario.reverseInsertion + ? [...requested].reverse() + : requested -function isPageRowArray(value: unknown): value is Array { - return ( - Array.isArray(value) && - value.every( - (row) => - typeof row === `object` && - row !== null && - `id` in row && - typeof row.id === `number` && - `rank` in row && - typeof row.rank === `number`, + const settled = new Promise((resolve) => { + queueMicrotask(() => { + begin() + for (const row of delivered) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + resolve() + }) + }) + return settled + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => { + const from = query.from({ row: source }) + const filtered = scenario.includeFilter + ? from.where(({ row }) => eq(row.keep, true)) + : from + const ordered = filtered.orderBy(({ row }) => row.rank, scenario.direction) + return ( + scenario.explicitPublicKeyOrder === false + ? ordered + : ordered.orderBy(({ row }) => row.id, `asc`) ) - ) -} - -type PageRowDifference = { - checkpoint: number - actual: Array - expected: Array -} - -function readPageRowDifference( - error: unknown, - acceptsCheckpoint: (checkpoint: number) => boolean = (checkpoint) => - checkpoint >= 1, -): PageRowDifference | undefined { - if ( - !(error instanceof TraceAssertionError) || - !acceptsCheckpoint(error.checkpoint) || - typeof error.cause !== `object` || - error.cause === null || - !(`actual` in error.cause) || - !(`expected` in error.cause) || - !isPageRowArray(error.cause.actual) || - !isPageRowArray(error.cause.expected) - ) { - return undefined - } - - return { - checkpoint: error.checkpoint, - actual: error.cause.actual, - expected: error.cause.expected, - } -} - -function readPageRowDifferenceAtCheckpoint( - error: unknown, - checkpoint: number, -): PageRowDifference | undefined { - return readPageRowDifference(error, (value) => value === checkpoint) -} - -function sameRows( - left: ReadonlyArray, - right: ReadonlyArray, -): boolean { - return ( - left.length === right.length && - left.every( - (row, index) => - row.id === right[index]!.id && row.rank === right[index]!.rank, - ) - ) -} - -function comparePageRows( - left: PageRow, - right: PageRow, - direction: `asc` | `desc`, -): number { - const directionFactor = direction === `asc` ? 1 : -1 - return (left.rank - right.rank) * directionFactor || left.id - right.id -} - -function replayOrderedSubscriptionWindow( - scenario: PaginationStateScenario, - actionCount: number, -): Array { - const rows = new Map( - scenario.ranks.map((rank, index) => [index + 1, { id: index + 1, rank }]), - ) - const initialRows = [...rows.values()] - const sentRows = new Map( - referenceWindowRows(initialRows, scenario.direction, { - offset: 0, - limit: scenario.initialWindow.offset + scenario.initialWindow.limit, - }).map((row) => [row.id, row]), - ) - let biggest = referenceWindowRows( - [...sentRows.values()], - scenario.direction, - { offset: 0, limit: sentRows.size }, - ).at(-1) - let window = { ...scenario.initialWindow } - - const currentResult = () => - referenceWindowRows([...sentRows.values()], scenario.direction, window) - - const refill = () => { - const orderedRows = referenceWindowRows( - [...rows.values()], - scenario.direction, - { - offset: 0, - limit: rows.size, - }, - ) - while (biggest !== undefined) { - const currentLength = currentResult().length - if (currentLength >= window.limit) break - const needed = window.limit - currentLength - const atCursor = orderedRows.filter( - (row) => row.rank === biggest!.rank && !sentRows.has(row.id), - ) - const afterCursor = orderedRows - .filter( - (row) => - comparePageRows( - { id: 0, rank: row.rank }, - { id: 0, rank: biggest!.rank }, - scenario.direction, - ) > 0 && !sentRows.has(row.id), - ) - .slice(0, Math.max(0, needed - atCursor.length)) - const loaded = [...atCursor, ...afterCursor] - if (loaded.length === 0) break - - for (const row of loaded) { - sentRows.set(row.id, { ...row }) - if (comparePageRows(biggest, row, scenario.direction) < 0) { - biggest = row - } - } - } - } - - for (const action of scenario.actions.slice(0, actionCount)) { - if (action.type === `window`) { - window = { offset: action.offset, limit: action.limit } - } else if (action.type === `put`) { - const previous = rows.get(action.id) - if (previous?.rank !== action.rank) { - const row = { id: action.id, rank: action.rank } - rows.set(action.id, row) - sentRows.set(row.id, { ...row }) - if ( - biggest === undefined || - comparePageRows(biggest, row, scenario.direction) < 0 - ) { - biggest = row - } - } - } else { - rows.delete(action.id) - sentRows.delete(action.id) - } - refill() - } - - return currentResult() -} - -function isKnownOrderedSubscriptionCoverageFailure( - scenario: PaginationStateScenario, - error: unknown, -): boolean { - const difference = readPageRowDifference(error) - if (!difference) return false - - const fullState = replayReferenceState(scenario, difference.checkpoint) - const expected = referenceWindowRows( - [...fullState.rows.values()], - scenario.direction, - fullState.window, - ) - const defective = replayOrderedSubscriptionWindow( - scenario, - difference.checkpoint, - ) - return ( - !sameRows(defective, expected) && - sameRows(difference.actual, defective) && - sameRows(difference.expected, expected) - ) -} - -function isNumberArray(value: unknown): value is Array { - return Array.isArray(value) && value.every((item) => typeof item === `number`) -} - -function isKnownOnDemandOffsetUnderfetch( - scenario: PaginationScenario, - error: unknown, -): boolean { - if ( - !(error instanceof TraceAssertionError) || - error.checkpoint < 1 || - typeof error.cause !== `object` || - error.cause === null || - !(`actual` in error.cause) || - !(`expected` in error.cause) || - !isNumberArray(error.cause.actual) || - !isNumberArray(error.cause.expected) - ) { - return false - } - - const actual = error.cause.actual - const expected = error.cause.expected - const window = scenario.windows[error.checkpoint] - if (window === undefined) return false - const authoritative = referenceWindow( - scenario.ranks.map((rank, index) => ({ id: index + 1, rank })), - scenario.direction, - window, - ) - const defective = replayOnDemandPaginationWindow(scenario, error.checkpoint) - return ( - expected.length === authoritative.length && - expected.every((id, index) => id === authoritative[index]) && - (defective.length !== authoritative.length || - defective.some((id, index) => id !== authoritative[index])) && - actual.length === defective.length && - actual.every((id, index) => id === defective[index]) - ) -} - -function replayOnDemandPaginationWindow( - scenario: PaginationScenario, - checkpoint: number, -): Array { - const authoritativeRows = referenceWindowRows( - scenario.ranks.map((rank, index) => ({ id: index + 1, rank })), - scenario.direction, - { offset: 0, limit: scenario.ranks.length }, - ) - const initialWindow = scenario.windows[0]! - const delivered = new Map( - authoritativeRows - .slice(0, initialWindow.offset + initialWindow.limit) - .map((row) => [row.id, row]), - ) - let biggest = referenceWindowRows( - [...delivered.values()], - scenario.direction, - { offset: 0, limit: delivered.size }, - ).at(-1) - - if (initialWindow.limit === 0) { - return referenceWindow( - [...delivered.values()], - scenario.direction, - scenario.windows[checkpoint]!, - ) - } - - for (const window of scenario.windows.slice(0, checkpoint + 1)) { - const current = referenceWindowRows( - [...delivered.values()], - scenario.direction, - window, - ) - const needed = window.limit - current.length - if (needed <= 0 || biggest === undefined) continue - - const atCursor = authoritativeRows.filter( - (row) => row.rank === biggest!.rank, - ) - const afterCursor = authoritativeRows - .filter((row) => comparePageRows(biggest!, row, scenario.direction) < 0) - .slice(0, needed) - for (const row of [...atCursor, ...afterCursor]) { - if (!delivered.has(row.id)) delivered.set(row.id, row) - if (comparePageRows(biggest, row, scenario.direction) < 0) biggest = row - } - } - - const window = scenario.windows[checkpoint]! - return referenceWindow([...delivered.values()], scenario.direction, window) -} - -function assertionDifference( - checkpoint: number, - actual: unknown, - expected: unknown, -): TraceAssertionError { - try { - expect(actual).toEqual(expected) - } catch (error) { - return new TraceAssertionError(checkpoint, error) - } - throw new Error(`test difference must not be equal`) -} - -async function runPaginationStateScenarioWithKnownFailures( - scenario: PaginationStateScenario, -): Promise { - try { - await runPaginationStateScenario(scenario) - } catch (error) { - if (isKnownOrderedSubscriptionCoverageFailure(scenario, error)) return - throw error - } -} - -async function runOnDemandPaginationScenarioWithKnownFailures( - scenario: PaginationScenario, -): Promise { - try { - await runOnDemandPaginationScenario(scenario) - } catch (error) { - if (isKnownOnDemandOffsetUnderfetch(scenario, error)) return - throw error - } -} - -async function runOnDemandPaginationScenario( - scenario: PaginationScenario, -): Promise { - const authoritativeRows = scenario.ranks.map((rank, index) => ({ - id: index + 1, - rank, - })) - const directionFactor = scenario.direction === `asc` ? 1 : -1 - const orderedRows = [...authoritativeRows].sort( - (left, right) => - (left.rank - right.rank) * directionFactor || left.id - right.id, - ) - const deliveredIds = new Set() - const loads: Array = [] - const initialWindow = scenario.windows[0]! - - const source = createCollection({ - id: `pagination-on-demand-oracle-source-${collectionSequence++}`, - getKey: (row) => row.id, - syncMode: `on-demand`, - startSync: true, - autoIndex: `eager`, - defaultIndexType: BTreeIndex, - sync: { - sync: ({ begin, write, commit, markReady }) => { - markReady() - return { - loadSubset: (options: LoadSubsetOptions) => { - loads.push({ ...options }) - const requested = rowsForLoadSubset(orderedRows, options) - - return new Promise((resolve) => { - queueMicrotask(() => { - begin() - for (const row of requested) { - if (deliveredIds.has(row.id)) continue - deliveredIds.add(row.id) - write({ type: `insert`, value: { ...row } }) - } - commit() - resolve() - }) - }) - }, - } - }, - }, - }) - const live = createLiveQueryCollection((query) => - query - .from({ row: source }) - .orderBy(({ row }) => row.rank, scenario.direction) - .orderBy(({ row }) => row.id, `asc`) - .offset(initialWindow.offset) - .limit(initialWindow.limit) - .select(({ row }) => ({ id: row.id, rank: row.rank })), + .offset(initialWindow.offset) + .limit(initialWindow.limit) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + }) + const publications: Array<{ + changes: Array + rows: Array + status: string + }> = [] + const publicationSubscription = live.subscribeChanges( + (changes) => { + const rows = Array.from(live.values(), ({ id, rank }) => ({ id, rank })) + publications.push({ + changes: normalizePageChanges( + changes as Array>, + ), + rows, + status: live.status, + }) + }, + { includeInitialState: false }, ) try { - await live.preload() - expect(loads.length).toBeGreaterThan(0) + const preloadPublicationCount = publications.length + const preload = live.preload() + expect(Array.from(live.values())).toHaveLength(0) + await preload + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBeUndefined() + if (initialWindow.limit > 0) { + expect(loads.length).toBeGreaterThan(0) + } try { - expect(Array.from(live.values(), ({ id }) => id)).toEqual( - referenceWindow(authoritativeRows, scenario.direction, initialWindow), + expect( + Array.from(live.values(), ({ id, rank }) => ({ id, rank })), + ).toEqual( + referenceWindowRows(expectedRows, scenario.direction, initialWindow), ) } catch (error) { throw new TraceAssertionError(0, error) } + const initialExpected = referenceWindowRows( + expectedRows, + scenario.direction, + initialWindow, + ) + expect(publications.slice(preloadPublicationCount)).toEqual([ + ...(initialExpected.length > 0 + ? [ + { + changes: expectedPageChanges([], initialExpected), + rows: initialExpected, + status: `loading`, + }, + ] + : []), + // A real source acquisition uses one empty batch to wake subscriptions + // when the initial source set becomes ready, even if it produced no + // visible rows. A zero window needs no acquisition or wake-up. + ...(initialWindow.limit > 0 + ? [{ changes: [], rows: initialExpected, status: `ready` }] + : []), + ]) + + if (scenario.localRowsBeforeFirstRequest) { + expect(loads).toHaveLength(0) + const publicationCount = publications.length + begin() + for (const row of scenario.localRowsBeforeFirstRequest) { + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + expect(publications).toHaveLength(publicationCount) + expect(Array.from(live.values())).toHaveLength(0) + } for (const [index, window] of scenario.windows.slice(1).entries()) { + const before = Array.from(live.values(), ({ id, rank }) => ({ id, rank })) + const publicationCount = publications.length const result = live.utils.setWindow(window) + if (result instanceof Promise) { + expect( + Array.from(live.values(), ({ id, rank }) => ({ id, rank })), + ).toEqual(before) + } if (result instanceof Promise) await result + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBeUndefined() try { - expect(Array.from(live.values(), ({ id }) => id)).toEqual( - referenceWindow(authoritativeRows, scenario.direction, window), - ) + expect( + Array.from(live.values(), ({ id, rank }) => ({ id, rank })), + ).toEqual(referenceWindowRows(expectedRows, scenario.direction, window)) } catch (error) { throw new TraceAssertionError(index + 1, error) } + const after = referenceWindowRows( + expectedRows, + scenario.direction, + window, + ) + const expectedChanges = expectedPageChanges(before, after) + expect(publications.slice(publicationCount)).toEqual( + expectedChanges.length > 0 + ? [{ changes: expectedChanges, rows: after, status: `ready` }] + : [], + ) } const expectedOrderBy = [ @@ -1187,15 +1110,45 @@ async function runOnDemandPaginationScenario( expression: new PropRef([`rank`]), compareOptions: { direction: scenario.direction, nulls: `first` }, }, - { - expression: new PropRef([`id`]), - compareOptions: { direction: `asc`, nulls: `first` }, - }, + ...(scenario.explicitPublicKeyOrder === false + ? [] + : [ + { + expression: new PropRef([`id`]), + compareOptions: { direction: `asc`, nulls: `first` }, + }, + ]), ] - for (const load of loads) expect(load.orderBy).toEqual(expectedOrderBy) + for (const load of loads) { + if (load.orderBy) { + expect(load.orderBy).toMatchObject(expectedOrderBy) + } else if (load.where) { + // Boundary refinement asks for the complete tie class with an exact + // predicate. Prefix and cursor requests still carry the source order. + expect(load.limit).toBeUndefined() + } else { + // If the same finite prefix cannot fill the local window, one + // unbounded request safely establishes the remaining source rows. + expect(load.cursor).toBeUndefined() + expect(load.limit).toBeUndefined() + expect(load.offset).toBeUndefined() + } + } + expect( + loads.length, + JSON.stringify( + loads.map(({ limit, offset, cursor, where }) => ({ + limit, + offset, + cursor, + where, + })), + ), + ).toBeLessThanOrEqual(scenario.windows.length * (expectedRows.length + 2)) + assertLoads?.(loads) } finally { - live.cleanup() - source.cleanup() + publicationSubscription.unsubscribe() + await cleanupAll(live, source) } } @@ -1269,7 +1222,13 @@ async function expectOnDemandWindowsAreCompletionOrderIndependent( const request = pending[index]! apply(request.options) request.deferred.resolve() - await Promise.resolve() + await flushPromises() + } + for (let index = 2; index < pending.length; index++) { + const request = pending[index]! + apply(request.options) + request.deferred.resolve() + await flushPromises() } await first await second @@ -1278,15 +1237,133 @@ async function expectOnDemandWindowsAreCompletionOrderIndependent( expect(Array.from(secondLive.values(), ({ id }) => id)).toEqual([1, 2, 3]) } finally { for (const request of pending) request.deferred.resolve() - firstLive.cleanup() - secondLive.cleanup() - source.cleanup() + await cleanupAll(firstLive, secondLive, source) + } +} + +async function runAdversarialOrderedProviderScenario(options: { + providerRows: ReadonlyArray + initialRows?: ReadonlyArray + order: + | { kind: `rank`; direction: `asc` | `desc`; nulls: `first` | `last` } + | { + kind: `reference` + direction?: `asc` | `desc` + nulls?: `first` | `last` + } + | { kind: `locale` } + limit: number + expectedIds: ReadonlyArray + useOffsetWhenAvailable?: boolean +}): Promise> { + const loads: Array = [] + const delivered = new Set(options.initialRows?.map(({ id }) => id) ?? []) + const source = createCollection({ + id: `pagination-adversarial-order-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + if (options.initialRows?.length) { + begin() + for (const row of options.initialRows) { + write({ type: `insert`, value: { ...row } }) + } + commit() + } + markReady() + return { + loadSubset: (loadOptions: LoadSubsetOptions) => { + loads.push(loadOptions) + if (loads.length > options.providerRows.length * 4 + 4) { + throw new Error( + `Ordered refinement exceeded its finite source work bound: ${JSON.stringify( + loads.map(({ limit, offset, cursor }) => ({ + limit, + offset, + lastKey: cursor?.lastKey, + })), + )}`, + ) + } + const providerRows = loadOptions.where + ? options.providerRows.filter( + (row) => + evaluateReferenceExpression(loadOptions.where!, row) === + true, + ) + : options.providerRows + const providerMatch = options.useOffsetWhenAvailable + ? providerRows.slice( + loadOptions.offset ?? 0, + loadOptions.limit === undefined + ? undefined + : (loadOptions.offset ?? 0) + loadOptions.limit, + ) + : rowsForLoadSubset(options.providerRows, loadOptions) + const requested = providerMatch + begin() + for (const row of requested) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + const receipt = commit() + return receipt === true ? Promise.resolve() : receipt + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => { + const from = query.from({ row: source }) + const ordered = + options.order.kind === `locale` + ? from.orderBy(({ row }) => row.label, { + direction: `asc`, + nulls: `first`, + stringSort: `locale`, + locale: `en-US`, + localeOptions: { numeric: true }, + }) + : from.orderBy( + ({ row }) => row.rank, + options.order.kind === `reference` + ? { + direction: options.order.direction ?? `asc`, + nulls: options.order.nulls ?? `first`, + } + : { + direction: options.order.direction, + nulls: options.order.nulls, + }, + ) + return ordered.limit(options.limit).select(({ row }) => ({ id: row.id })) + }) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual( + options.expectedIds, + ) + // Snapshot observations before cleanup. Teardown must not create fresh + // source demand, and callers must not mistake such work for the scenario's + // final refinement request. + return [...loads] + } finally { + await cleanupAll(live, source) } } async function runPendingMutationScenario( scenario: PendingMutationScenario, timing: `before-response` | `after-response`, + finalLimitAfterMutation?: number, + explicitPublicKeyOrder = true, + transport: `cursor` | `offset` | `key` = `cursor`, ): Promise { const rows = new Map( scenario.ranks.map((rank, index) => [index + 1, { id: index + 1, rank }]), @@ -1300,8 +1377,7 @@ async function runPendingMutationScenario( const deliveredIds = new Set([firstDelivered.id]) // A rejected initial subset load is fatal. Establish a ready baseline first // so reject scenarios exercise subscription-scoped window recovery. - let establishInitialCoverageSynchronously = - scenario.responseOutcome === `reject` + let capturePending = scenario.responseOutcome === `resolve` let begin!: () => void let write!: (message: { type: `insert` | `update` | `delete` @@ -1327,10 +1403,7 @@ async function runPendingMutationScenario( params.markReady() return { loadSubset: (options: LoadSubsetOptions) => { - if (establishInitialCoverageSynchronously) { - establishInitialCoverageSynchronously = false - return true - } + if (!capturePending) return true const deferred = createDeferred() pending.push({ options, deferred }) return deferred.promise @@ -1339,13 +1412,16 @@ async function runPendingMutationScenario( }, }, }) - const live = createLiveQueryCollection((query) => - query + const live = createLiveQueryCollection((query) => { + const ordered = query .from({ row: source }) .orderBy(({ row }) => row.rank, scenario.direction) - .orderBy(({ row }) => row.id, `asc`) - .limit(scenario.responseOutcome === `reject` ? 1 : scenario.limit), - ) + return ( + explicitPublicKeyOrder + ? ordered.orderBy(({ row }) => row.id, `asc`) + : ordered + ).limit(scenario.responseOutcome === `reject` ? 1 : scenario.limit) + }) const outstanding: Array> = [] const applyMutation = () => { @@ -1366,7 +1442,10 @@ async function runPendingMutationScenario( } const settlePending = async () => { - for (const request of pending) { + // Settling one request can append its boundary-refinement request. + // eslint-disable-next-line @typescript-eslint/prefer-for-of + for (let index = 0; index < pending.length; index++) { + const request = pending[index]! if (request.settled) continue request.settled = true const orderedRows = referenceWindowRows( @@ -1377,15 +1456,28 @@ async function runPendingMutationScenario( limit: rows.size, }, ) + const options = { ...request.options } + if (transport !== `cursor`) { + // Model providers whose opaque continuation token is indexed by the + // last fetched row key, rather than by the predicate expression. + if (transport === `key` && options.cursor) { + const boundary = orderedRows.findIndex( + ({ id }) => id === options.cursor!.lastKey, + ) + expect(boundary).toBeGreaterThanOrEqual(0) + options.offset = boundary + 1 + } + options.cursor = undefined + } begin() - for (const row of rowsForLoadSubset(orderedRows, request.options)) { + for (const row of rowsForLoadSubset(orderedRows, options)) { if (deliveredIds.has(row.id)) continue deliveredIds.add(row.id) write({ type: `insert`, value: { ...row } }) } commit() request.deferred.resolve() - await Promise.resolve() + await flushPromises() } } @@ -1400,11 +1492,22 @@ async function runPendingMutationScenario( await preload if (timing === `after-response`) { applyMutation() - await Promise.resolve() - await settlePending() + await flushPromises() + } + if (finalLimitAfterMutation !== undefined) { + finalLimit = finalLimitAfterMutation + const widened = live.utils.setWindow({ + offset: 0, + limit: finalLimit, + }) + if (widened instanceof Promise) outstanding.push(widened) } + await settlePending() + await Promise.all(outstanding) } else { await preload + await flushPromises() + capturePending = true expect(pending).toHaveLength(0) finalLimit += 1 const failedWindow = live.utils.setWindow({ @@ -1426,31 +1529,25 @@ async function runPendingMutationScenario( await flushPromises() await settlePending() expect(await observedFailure).toBe(cursorError) + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBe(cursorError) const retry = live.utils.setWindow({ offset: 0, limit: finalLimit }) - let retrySettled = retry === true const observedRetry = retry instanceof Promise - ? retry.then( - () => { - retrySettled = true - }, - (error: unknown) => { - retrySettled = true - throw error - }, - ) + ? retry.then(undefined, (error: unknown) => { + throw error + }) : undefined - if (pending.length === 2) { - await settlePending() - } else { - await flushPromises() - expect(retrySettled).toBe(true) - } + await flushPromises() + await settlePending() + await flushPromises() if (observedRetry) { outstanding.push(observedRetry) await observedRetry } + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBe(cursorError) } try { @@ -1468,108 +1565,14 @@ async function runPendingMutationScenario( referenceWindowRows( [...rows.values()].filter(({ id }) => deliveredIds.has(id)), scenario.direction, - { offset: 0, limit: finalLimit }, + { offset: 0, limit: deliveredIds.size }, ), ) } } finally { for (const request of pending) request.deferred.resolve() await Promise.allSettled(outstanding) - await live.cleanup() - await source.cleanup() - } -} - -function pendingMutationRows( - scenario: PendingMutationScenario, -): Map { - const rows = new Map( - scenario.ranks.map((rank, index) => [index + 1, { id: index + 1, rank }]), - ) - if (scenario.mutation.type === `delete`) { - rows.delete(scenario.mutation.id) - } else { - rows.set(scenario.mutation.row.id, { ...scenario.mutation.row }) - } - return rows -} - -function isKnownSettledTopKMembershipFailure( - scenario: PendingMutationScenario, - timing: `before-response` | `after-response`, - error: unknown, -): boolean { - if (scenario.responseOutcome !== `resolve` || timing !== `after-response`) { - return false - } - const difference = readPageRowDifferenceAtCheckpoint(error, 0) - if (!difference) return false - - const initialRows = scenario.ranks.map((rank, index) => ({ - id: index + 1, - rank, - })) - const initialVisibleIds = new Set( - referenceWindowRows(initialRows, scenario.direction, { - offset: 0, - limit: scenario.limit, - }).map(({ id }) => id), - ) - const finalRows = pendingMutationRows(scenario) - const expected = referenceWindowRows( - [...finalRows.values()], - scenario.direction, - { offset: 0, limit: scenario.limit }, - ) - const defective = referenceWindowRows( - [...finalRows.values()].filter(({ id }) => initialVisibleIds.has(id)), - scenario.direction, - { offset: 0, limit: scenario.limit }, - ) - - return ( - !sameRows(defective, expected) && - sameRows(difference.actual, defective) && - sameRows(difference.expected, expected) - ) -} - -function isKnownRejectedCursorRetryFailure( - scenario: PendingMutationScenario, - error: unknown, -): boolean { - if (scenario.responseOutcome !== `reject`) return false - if (!(error instanceof PendingMutationTraceAssertionError)) return false - - const difference = readPageRowDifferenceAtCheckpoint(error, 0) - if (!difference) return false - - const finalRows = pendingMutationRows(scenario) - const finalLimit = scenario.limit + 1 - const expected = referenceWindowRows( - [...finalRows.values()], - scenario.direction, - { offset: 0, limit: finalLimit }, - ) - const defective = error.deliveredRows - - return ( - !sameRows(defective, expected) && - sameRows(difference.actual, defective) && - sameRows(difference.expected, expected) - ) -} - -async function runPendingMutationScenarioWithKnownFailures( - scenario: PendingMutationScenario, - timing: `before-response` | `after-response`, -): Promise { - try { - await runPendingMutationScenario(scenario, timing) - } catch (error) { - if (isKnownSettledTopKMembershipFailure(scenario, timing, error)) return - if (isKnownRejectedCursorRetryFailure(scenario, error)) return - throw error + await cleanupAll(live, source) } } @@ -1584,7 +1587,7 @@ async function runRejectedCursorRetryAfterMutation(): Promise { const deliveredIds = new Set([1]) // Keep the rejected cursor in the incremental path rather than failing the // live query's initial preload. - let establishInitialCoverageSynchronously = true + let capturePending = false let begin!: () => void let write!: (message: { type: `insert` | `update`; value: PageRow }) => void let commit!: () => void @@ -1606,10 +1609,7 @@ async function runRejectedCursorRetryAfterMutation(): Promise { params.markReady() return { loadSubset: (options: LoadSubsetOptions) => { - if (establishInitialCoverageSynchronously) { - establishInitialCoverageSynchronously = false - return true - } + if (!capturePending) return true const deferred = createDeferred() pending.push({ options, deferred }) return deferred.promise @@ -1640,11 +1640,13 @@ async function runRejectedCursorRetryAfterMutation(): Promise { } commit() request.deferred.resolve() - await Promise.resolve() + await flushPromises() } try { await live.preload() + await flushPromises() + capturePending = true expect(pending).toHaveLength(0) const failedWindow = live.utils.setWindow({ offset: 0, limit: 2 }) @@ -1682,8 +1684,7 @@ async function runRejectedCursorRetryAfterMutation(): Promise { } } finally { for (const request of pending) request.deferred.resolve() - live.cleanup() - source.cleanup() + await cleanupAll(live, source) } } @@ -1763,7 +1764,7 @@ async function runPendingHistoryScenario( } commit() request.deferred.resolve() - await Promise.resolve() + await flushPromises() } const track = (result: true | Promise): void => { @@ -1777,11 +1778,22 @@ async function runPendingHistoryScenario( updateFirstDelivered(scenario.firstRank) track(live.utils.setWindow({ offset: 0, limit: scenario.narrowLimit })) track(live.utils.setWindow({ offset: 0, limit: scenario.wideLimit })) - expect(pending).toHaveLength(1) + expect(pending.length).toBeGreaterThan(0) updateFirstDelivered(scenario.secondRank) await settle(pending[0]!) for (let index = 1; index < pending.length; index++) { + if (index > rows.size * 4) { + throw new Error( + `Ordered continuation exceeded its finite source work bound: ${JSON.stringify( + pending.map(({ options }) => ({ + limit: options.limit, + offset: options.offset, + lastKey: options.cursor?.lastKey, + })), + )}`, + ) + } await settle(pending[index]!) } await Promise.all(outstanding) @@ -1807,8 +1819,7 @@ async function runPendingHistoryScenario( } finally { for (const request of pending) request.deferred.resolve() await Promise.allSettled(outstanding) - live.cleanup() - source.cleanup() + await cleanupAll(live, source) } } @@ -1820,55 +1831,6 @@ function changedRankValue(previous: number, requested: number): number { : requested } -function pendingHistoryRows( - scenario: PendingHistoryScenario, -): Map { - const rows = new Map( - scenario.ranks.map((rank, index) => [index + 1, { id: index + 1, rank }]), - ) - const first = referenceWindowRows([...rows.values()], scenario.direction, { - offset: 0, - limit: 1, - })[0]! - const afterFirst = changedRankValue(first.rank, scenario.firstRank) - const afterSecond = changedRankValue(afterFirst, scenario.secondRank) - rows.set(first.id, { ...first, rank: afterSecond }) - return rows -} - -function isKnownLatePendingHistoryUnderfill( - scenario: PendingHistoryScenario, - error: unknown, -): boolean { - if (!(error instanceof PendingHistoryTraceAssertionError)) { - return false - } - - const difference = readPageRowDifferenceAtCheckpoint(error, 0) - if (!difference) return false - const authoritative = referenceWindowRows( - [...pendingHistoryRows(scenario).values()], - scenario.direction, - { offset: 0, limit: scenario.wideLimit }, - ) - return ( - sameRows(difference.expected, authoritative) && - !sameRows(error.deliveredRows, authoritative) && - sameRows(difference.actual, error.deliveredRows) - ) -} - -async function runPendingHistoryScenarioWithKnownFailures( - scenario: PendingHistoryScenario, -): Promise { - try { - await runPendingHistoryScenario(scenario) - } catch (error) { - if (isKnownLatePendingHistoryUnderfill(scenario, error)) return - throw error - } -} - async function expectInflightRequestFillsNewWindow(): Promise { const rows: Array = [ { id: 1, rank: 0 }, @@ -1933,12 +1895,17 @@ async function expectInflightRequestFillsNewWindow(): Promise { const setWindow = live.utils.setWindow({ offset: 2, limit: 2 }) expect(setWindow).toBeInstanceOf(Promise) await flushPromises() - expect(pending).toHaveLength(1) + expect(pending.length).toBeGreaterThan(0) - await settle(pending[0]!) + for (let index = 0; index < pending.length; index++) { + if (index > rows.length * 2) { + throw new Error(`Ordered continuation exceeded its work bound`) + } + await settle(pending[index]!) + await flushPromises() + } await preload if (setWindow instanceof Promise) await setWindow - expect(pending).toHaveLength(1) try { expect(Array.from(live.values(), ({ id }) => id)).toEqual([3, 4]) @@ -1947,12 +1914,277 @@ async function expectInflightRequestFillsNewWindow(): Promise { } } finally { for (const request of pending) request.deferred.resolve() - live.cleanup() - source.cleanup() + await cleanupAll(live, source) } } describe(`pagination recomputation oracle`, () => { + it.each([ + { label: `Error`, reason: new Error(`first cleanup failed`) }, + { label: `undefined`, reason: undefined }, + { label: `null`, reason: null }, + { label: `false`, reason: false }, + { label: `zero`, reason: 0 }, + { label: `NaN`, reason: Number.NaN }, + { label: `empty string`, reason: `` }, + ])( + `observes $label cleanup failure after every teardown settles`, + async ({ reason: firstFailure }) => { + const secondFailure = new Error(`second cleanup failed`) + const firstFailureRelease = createDeferred() + const lastCleanupRelease = createDeferred() + const repeatedFirstFailureRelease = createDeferred() + const repeatedLastCleanupRelease = createDeferred() + const events: Array = [] + const unhandled: Array = [] + const recordUnhandled = (reason: unknown) => unhandled.push(reason) + const createTargets = ( + firstRelease: Deferred, + lastRelease: Deferred, + ): ReadonlyArray => [ + { + cleanup: async () => { + events.push(`first`) + await firstRelease.promise + throw firstFailure + }, + }, + { + cleanup: () => { + events.push(`second`) + throw secondFailure + }, + }, + { + cleanup: async () => { + events.push(`third`) + await lastRelease.promise + }, + }, + ] + const observeFirstFailure = (cleanup: Promise) => + cleanup.then( + () => { + throw new Error(`expected cleanup to reject`) + }, + (error: unknown) => expect(error).toBe(firstFailure), + ) + let cleanupFinished = false + process.on(`unhandledRejection`, recordUnhandled) + + try { + const cleanup = cleanupAll( + ...createTargets(firstFailureRelease, lastCleanupRelease), + ).finally(() => { + cleanupFinished = true + }) + const observedFailure = observeFirstFailure(cleanup) + + await flushPromises() + expect(events).toEqual([`first`, `second`, `third`]) + expect(cleanupFinished).toBe(false) + expect(unhandled).toEqual([]) + + firstFailureRelease.resolve() + await flushPromises() + expect(cleanupFinished).toBe(false) + expect(unhandled).toEqual([]) + + lastCleanupRelease.resolve() + await observedFailure + await flushPromises() + expect(cleanupFinished).toBe(true) + expect(unhandled).toEqual([]) + + let repeatedCleanupFinished = false + const repeatedCleanup = cleanupAll( + ...createTargets( + repeatedFirstFailureRelease, + repeatedLastCleanupRelease, + ), + ).finally(() => { + repeatedCleanupFinished = true + }) + const repeatedObservedFailure = observeFirstFailure(repeatedCleanup) + await flushPromises() + expect(events).toEqual([ + `first`, + `second`, + `third`, + `first`, + `second`, + `third`, + ]) + expect(repeatedCleanupFinished).toBe(false) + expect(unhandled).toEqual([]) + + repeatedFirstFailureRelease.resolve() + await flushPromises() + expect(repeatedCleanupFinished).toBe(false) + expect(unhandled).toEqual([]) + + repeatedLastCleanupRelease.resolve() + await repeatedObservedFailure + await flushPromises() + expect(repeatedCleanupFinished).toBe(true) + expect(unhandled).toEqual([]) + } finally { + firstFailureRelease.resolve() + lastCleanupRelease.resolve() + repeatedFirstFailureRelease.resolve() + repeatedLastCleanupRelease.resolve() + process.off(`unhandledRejection`, recordUnhandled) + } + }, + ) + + it(`refills a joined result window through a contract-compliant source`, async () => { + type ParentRow = { id: number; rank: number; groupId: number } + type ChildRow = { id: number; groupId: number } + const parents = [ + { id: 1, rank: 0, groupId: 1 }, + { id: 2, rank: 1, groupId: 2 }, + { id: 3, rank: 2, groupId: 3 }, + { id: 4, rank: 3, groupId: 4 }, + ] satisfies ReadonlyArray + const { requests, source: parentSource } = createConformingOrderedSource( + `pagination-joined-underfill-source-${collectionSequence++}`, + parents, + ) + const childSource = createCollection( + mockSyncCollectionOptions({ + id: `pagination-joined-underfill-child-${collectionSequence++}`, + initialData: [ + { id: 20, groupId: 2 }, + { id: 30, groupId: 3 }, + { id: 40, groupId: 4 }, + ] satisfies ReadonlyArray, + getKey: (row: ChildRow) => row.id, + }), + ) + const live = createLiveQueryCollection((query) => + query + .from({ parent: parentSource }) + .innerJoin({ child: childSource }, ({ parent, child }) => + eq(parent.groupId, child.groupId), + ) + .orderBy(({ parent }) => parent.rank, `asc`) + .orderBy(({ parent }) => parent.id, `asc`) + .limit(2) + .select(({ parent }) => ({ id: parent.id })), + ) + + try { + await live.preload() + await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([2, 3]) + expect(requests).toHaveLength(1) + expect(requests[0]?.limit).toBeUndefined() + } finally { + await cleanupAll(live, childSource, parentSource) + } + }) + + it(`loads the full ordered source when no continuation index exists`, async () => { + type ParentRow = { id: number; rank: number; groupId: number } + type ChildRow = { id: number; groupId: number } + const parents = [ + { id: 1, rank: 0, groupId: 1 }, + { id: 2, rank: 1, groupId: 2 }, + { id: 3, rank: 2, groupId: 3 }, + { id: 4, rank: 3, groupId: 4 }, + ] satisfies ReadonlyArray + const { requests, source: parentSource } = createConformingOrderedSource( + `pagination-no-index-underfill-source-${collectionSequence++}`, + parents, + `off`, + ) + const childSource = createCollection( + mockSyncCollectionOptions({ + id: `pagination-no-index-underfill-child-${collectionSequence++}`, + initialData: [ + { id: 20, groupId: 2 }, + { id: 30, groupId: 3 }, + { id: 40, groupId: 4 }, + ] satisfies ReadonlyArray, + getKey: (row: ChildRow) => row.id, + }), + ) + const live = createLiveQueryCollection((query) => + query + .from({ parent: parentSource }) + .innerJoin({ child: childSource }, ({ parent, child }) => + eq(parent.groupId, child.groupId), + ) + .orderBy(({ parent }) => parent.rank, `asc`) + .orderBy(({ parent }) => parent.id, `asc`) + .limit(2) + .select(({ parent }) => ({ id: parent.id })), + ) + + try { + await live.preload() + await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([2, 3]) + expect(requests).toHaveLength(1) + expect(requests[0]?.limit).toBeUndefined() + } finally { + await cleanupAll(live, childSource, parentSource) + } + }) + + it(`refines a joined foreign order term through the source tie class`, async () => { + type ParentRow = { id: number; sourceRank: number; childId: number } + type ChildRow = { id: number; score: number } + const parents = [ + { id: 1, sourceRank: 0, childId: 1 }, + { id: 2, sourceRank: 0, childId: 2 }, + { id: 3, sourceRank: 0, childId: 3 }, + { id: 4, sourceRank: 0, childId: 4 }, + ] satisfies ReadonlyArray + const { requests, source: parentSource } = createConformingOrderedSource( + `pagination-joined-foreign-order-source-${collectionSequence++}`, + parents, + ) + const childSource = createCollection( + mockSyncCollectionOptions({ + id: `pagination-joined-foreign-order-child-${collectionSequence++}`, + initialData: [ + { id: 1, score: 10 }, + { id: 2, score: 20 }, + { id: 3, score: 0 }, + { id: 4, score: 30 }, + ] satisfies ReadonlyArray, + getKey: (row: ChildRow) => row.id, + }), + ) + const live = createLiveQueryCollection((query) => + query + .from({ parent: parentSource }) + .leftJoin({ child: childSource }, ({ parent, child }) => + eq(parent.childId, child.id), + ) + .orderBy(({ parent }) => parent.sourceRank, `asc`) + .orderBy(({ child }) => child.score, `asc`) + .orderBy(({ parent }) => parent.id, `asc`) + .limit(2) + .select(({ parent }) => ({ id: parent.id })), + ) + + try { + await live.preload() + await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([3, 1]) + expect(requests).toHaveLength(1) + expect(requests[0]?.limit).toBeUndefined() + } finally { + await cleanupAll(live, childSource, parentSource) + } + }) + it(`materializes an empty source window`, async () => { await runPaginationScenario({ ranks: [], @@ -1961,6 +2193,68 @@ describe(`pagination recomputation oracle`, () => { }) }) + it(`does not refetch when live insertion fills a settled empty window`, async () => { + let sync!: Parameters[`sync`]>[0] + const requests: Array = [] + const source = createCollection({ + id: `settled-empty-window`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + requests.push(options) + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + try { + await live.preload() + expect(live.toArray).toEqual([]) + expect(requests).toHaveLength(1) + + sync.begin() + sync.write({ type: `insert`, value: { id: 1, rank: 1 } }) + const receipt = sync.commit() + if (receipt !== true) await receipt + await flushPromises() + + expect(live.toArray.map(({ id, rank }) => ({ id, rank }))).toEqual([ + { id: 1, rank: 1 }, + ]) + expect(live.utils.lastSubsetError).toBeUndefined() + // Correct rows alone would miss a repeated prefix and boundary fetch. + expect( + requests.map(({ limit, offset, orderBy, where, cursor }) => ({ + limit, + offset, + ordered: Boolean(orderBy), + filtered: Boolean(where), + cursor: Boolean(cursor), + })), + ).toEqual([ + { limit: 1, offset: 0, ordered: true, filtered: false, cursor: false }, + ]) + } finally { + await cleanupAll(live, source) + } + }) + it(`materializes an offset past the final row`, async () => { await runPaginationScenario({ ranks: [0, 1], @@ -1989,15 +2283,79 @@ describe(`pagination recomputation oracle`, () => { }) }) + it(`advances past an implicit public-key tie class`, async () => { + await runPaginationScenario({ + ranks: [1, 2, 3, 4, 5, 5, 5, 5, 5, 5, 11, 12, 13, 14, 15, 16], + direction: `asc`, + explicitPublicKeyOrder: false, + includeFilter: true, + reverseInsertion: true, + windows: [ + { offset: 0, limit: 5 }, + { offset: 5, limit: 5 }, + { offset: 10, limit: 5 }, + ], + }) + }) + + it(`keeps implicit ties stable across filtered source mutations`, async () => { + await runPaginationStateScenario({ + ranks: [0, 0, 0, 1, 1, 1], + direction: `asc`, + explicitPublicKeyOrder: false, + includeFilter: true, + reverseInsertion: true, + initialWindow: { offset: 0, limit: 3 }, + actions: [ + { type: `put`, id: 7, rank: 0 }, + { type: `delete`, id: 2 }, + { type: `window`, offset: 1, limit: 3 }, + ], + }) + }) + + it.each([ + { + name: `enters the filter`, + keeps: [false, true], + action: { type: `put` as const, id: 1, rank: 0, keep: true }, + expected: [1, 2], + }, + { + name: `leaves the filter`, + keeps: [true, true], + action: { type: `put` as const, id: 1, rank: 0, keep: false }, + expected: [2], + }, + ])(`updates a row that $name`, async ({ keeps, action, expected }) => { + const scenario: PaginationStateScenario = { + ranks: [0, 1], + keeps, + direction: `asc`, + includeFilter: true, + explicitPublicKeyOrder: true, + reverseInsertion: false, + initialWindow: { offset: 0, limit: 2 }, + actions: [action], + } + await runPaginationStateScenario(scenario) + + const finalRows = scenario.ranks.map((rank, index) => ({ + id: index + 1, + rank, + keep: index === 0 ? action.keep : keeps[index], + })) + expect( + referenceWindow( + visibleRows(finalRows, true), + scenario.direction, + scenario.initialWindow, + ), + ).toEqual(expected) + }) + it(`discovered trace: loads an on-demand window after a zero limit`, async () => { - await expectAssertionFailure(runOnDemandPaginationScenario, { - checkpoint: 1, - classify: ({ actual, expected }) => - isNumberArray(actual) && - actual.join(`,`) === `1` && - isNumberArray(expected) && - expected.join(`,`) === `1,2`, - })({ + await runOnDemandPaginationScenario({ ranks: [0, 0], direction: `asc`, windows: [ @@ -2007,124 +2365,1184 @@ describe(`pagination recomputation oracle`, () => { }) }) - const nullableBoundaryRows: ReadonlyArray = [ - { id: 1, primary: null, secondary: 2 }, - { id: 2, primary: null, secondary: 0 }, - { id: 3, primary: null, secondary: 1 }, - { id: 4, primary: 1, secondary: null }, - { id: 5, primary: 1, secondary: 0 }, - { id: 6, primary: 2, secondary: 0 }, - ] + it(`widens an offset on-demand window after starting at zero limit`, async () => { + await runOnDemandPaginationScenario({ + ranks: [-1, 0, 0, 0, -1, 0], + direction: `asc`, + windows: [ + { offset: 1, limit: 0 }, + { offset: 4, limit: 1 }, + { offset: 4, limit: 2 }, + { offset: 0, limit: 0 }, + ], + }) + }) - it.each([ - [ - `discovered trace: orders an ascending nullable boundary by its second term`, - { - rows: nullableBoundaryRows, - primary: { direction: `asc`, nulls: `first` }, - secondary: { direction: `asc`, nulls: `first` }, - limit: 1, - }, - { actual: [1], expected: [2] }, - ], - [ - `orders a descending nullable boundary by its second term`, - { - rows: nullableBoundaryRows, - primary: { direction: `desc`, nulls: `first` }, - secondary: { direction: `desc`, nulls: `first` }, - limit: 1, - }, - undefined, - ], - [ - `orders an ascending and descending mixed nullable boundary`, - { - rows: nullableBoundaryRows, - primary: { direction: `asc`, nulls: `first` }, - secondary: { direction: `desc`, nulls: `first` }, - limit: 1, - }, - undefined, - ], - [ - `discovered trace: orders a descending and ascending mixed nullable boundary`, - { - rows: nullableBoundaryRows, - primary: { direction: `desc`, nulls: `first` }, - secondary: { direction: `asc`, nulls: `first` }, - limit: 1, - }, - { actual: [1], expected: [2] }, - ], - [ - `uses the public key to break a complete tuple tie`, + it(`starts an on-demand source prefix after opening a zero window with a local row`, async () => { + await runOnDemandPaginationScenario( { - rows: [ - { id: 2, primary: 0, secondary: 0 }, - { id: 1, primary: 0, secondary: 0 }, + ranks: [0, 1], + direction: `asc`, + explicitPublicKeyOrder: false, + windows: [ + { offset: 0, limit: 0 }, + { offset: 0, limit: 1 }, ], - primary: { direction: `asc`, nulls: `last` }, - secondary: { direction: `asc`, nulls: `last` }, - limit: 1, - }, - undefined, - ], - [ - `discovered trace: places nulls last in an ascending nullable boundary`, - { - rows: nullableBoundaryRows, - primary: { direction: `asc`, nulls: `last` }, - secondary: { direction: `asc`, nulls: `last` }, - limit: 1, + localRowsBeforeFirstRequest: [{ id: 2, rank: 1 }], }, - { actual: [4], expected: [5] }, - ], - [ - `places nulls last in a descending nullable boundary`, - { - rows: nullableBoundaryRows, - primary: { direction: `desc`, nulls: `last` }, - secondary: { direction: `desc`, nulls: `last` }, - limit: 1, + (loads) => { + expect(loads[0]?.cursor).toBeUndefined() + expect(loads[0]?.offset).toBe(0) + expect(loads[0]?.limit).toBe(1) }, - undefined, - ], - ] satisfies ReadonlyArray< - readonly [ - string, - MultiOrderScenario, - { actual: ReadonlyArray; expected: ReadonlyArray }?, + ) + }) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + ([`sync`, `async`] as const).map((replayDelivery) => ({ + direction, + replayDelivery, + })), + ), + )( + `keeps a failed $direction window private after $replayDelivery source replay until explicit retry`, + async ({ direction, replayDelivery }) => { + const rows: Array = [ + { id: 1, rank: 1 }, + { id: 2, rank: 2 }, + ] + const failure = new Error(`window acquisition failed`) + const replayGate = createDeferred() + let operations!: Parameters[`sync`]>[0] + let loads = 0 + const source = createCollection({ + id: `pagination-failed-window-replay-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (sync) => { + operations = sync + sync.markReady() + return { + loadSubset: (options) => { + loads++ + sync.begin() + for (const row of loads === 1 ? rows.slice(0, 1) : rows) { + sync.write({ type: `insert`, value: { ...row } }) + } + const receipt = sync.commit(options.signal) + if (loads === 1) return Promise.reject(failure) + if (replayDelivery === `sync`) return receipt + return replayGate.promise.then(async () => { + if (receipt !== true) await receipt + }) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, direction) + .limit(0) + .select(({ row }) => ({ id: row.id, rank: row.rank })) + .distinct(), + ) + const publications: Array> = [] + const subscriber = live.subscribeChanges(() => { + publications.push(Array.from(live.values(), ({ id }) => id)) + }) + const assertHeld = () => { + expect(Array.from(live.values())).toEqual([]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 0 }) + expect(publications).toEqual([]) + } + try { + await live.preload() + await expect( + live.utils.setWindow({ offset: 0, limit: 2 }), + ).rejects.toBe(failure) + assertHeld() + operations.begin() + operations.truncate() + const receipt = operations.commit() + if (receipt !== true) await receipt + await flushPromises() + assertHeld() + replayGate.resolve() + await flushPromises() + await flushPromises() + expect(loads).toBe(2) + assertHeld() + + await live.utils.setWindow({ offset: 0, limit: 2 }) + const expected = referenceWindow(rows, direction, { + offset: 0, + limit: 2, + }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual(expected) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(publications).toEqual([expected]) + } finally { + replayGate.resolve() + subscriber.unsubscribe() + await cleanupAll(live, source) + } + }, + ) + + it.each([ + { offset: 0, limit: 1, failureKind: `error` as const }, + { offset: 2, limit: 1, failureKind: `error` as const }, + { offset: 0, limit: 1, failureKind: `abort` as const }, + { offset: 2, limit: 1, failureKind: `abort` as const }, + ])( + `recovers the first $failureKind-rejected ordered request for window $offset:$limit from the full source`, + async ({ failureKind, ...window }) => { + const authoritativeRows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + { id: 3, rank: 2 }, + { id: 4, rank: 3 }, + ] + const requests: Array = [] + const firstRequest = createDeferred() + const deliveredIds = new Set([4]) + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-rejected-first-prefix-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + if (requests.length === 1) return firstRequest.promise + + begin() + for (const row of rowsForLoadSubset( + authoritativeRows, + options, + )) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(0), + ) + + try { + await live.preload() + begin() + write({ type: `insert`, value: { ...authoritativeRows[3]! } }) + commit() + + const requestedPrefix = window.offset + window.limit + const failed = live.utils.setWindow(window) + expect(failed).toBeInstanceOf(Promise) + expect(requests[0]).toMatchObject({ + offset: 0, + limit: requestedPrefix, + }) + expect(requests[0]?.cursor).toBeUndefined() + const failure = + failureKind === `abort` + ? new DOMException(`first ordered request canceled`, `AbortError`) + : new Error(`first ordered request failed`) + firstRequest.reject(failure) + await expect(failed).rejects.toBe(failure) + + const retry = live.utils.setWindow(window) + if (retry instanceof Promise) await retry + expect(requests[1]?.limit).toBeUndefined() + expect(requests[1]?.offset).toBeUndefined() + expect(requests[1]?.cursor).toBeUndefined() + expect(requests).toHaveLength(2) + expect(Array.from(live.values(), ({ id }) => id)).toEqual( + referenceWindow(authoritativeRows, `asc`, window), + ) + } finally { + firstRequest.resolve() + await cleanupAll(live, source) + } + }, + ) + + it.each([`error`, `AbortError`] as const)( + `does not derive a retry cursor from rows written by a %s request`, + async (failureKind) => { + const authoritativeRows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + { id: 3, rank: 2 }, + { id: 4, rank: 99 }, + ] + const requests: Array = [] + const unloaded: Array = [] + const deliveredIds = new Set() + const rejectedPage = createDeferred() + let rejectNextPage = false + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + let truncate!: () => void + const source = createCollection({ + id: `pagination-rejected-partial-page-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + truncate = operations.truncate + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + if (rejectNextPage) { + rejectNextPage = false + begin() + deliveredIds.add(4) + write({ type: `insert`, value: { ...authoritativeRows[3]! } }) + commit() + return rejectedPage.promise + } + + begin() + for (const row of rowsForLoadSubset( + authoritativeRows, + options, + )) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + unloadSubset: (options) => unloaded.push(options), + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + const initialRequestCount = requests.length + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + + rejectNextPage = true + const failed = live.utils.setWindow({ offset: 0, limit: 4 }) + expect(failed).toBeInstanceOf(Promise) + const failure = + failureKind === `AbortError` + ? new DOMException(`partial ordered request canceled`, `AbortError`) + : new Error(`partial ordered request failed`) + rejectedPage.reject(failure) + await expect(failed).rejects.toBe(failure) + await flushPromises() + expect(requests).toHaveLength(initialRequestCount + 1) + + // Replay can replace the failed request's physical options object + // before the explicit retry retires its logical demand. + const beforeFailedReplay = requests.length + deliveredIds.clear() + begin() + truncate() + commit() + await flushPromises() + const failedReplayRequests = requests.slice(beforeFailedReplay) + // The settled first row permits a three-row continuation. Replay + // must preserve that exact demand even after its first attempt fails. + const replayedFailedRequest = failedReplayRequests.find( + ({ limit, cursor }) => limit === 3 && cursor !== undefined, + ) + expect(replayedFailedRequest).toBeDefined() + expect(replayedFailedRequest).toMatchObject({ offset: 1, limit: 3 }) + expect(replayedFailedRequest?.cursor).toEqual( + requests[initialRequestCount]?.cursor, + ) + + const releasesBeforeRetry = unloaded.length + const requestsBeforeRetry = requests.length + const retry = live.utils.setWindow({ offset: 0, limit: 2 }) + if (retry instanceof Promise) await retry + expect(unloaded.slice(releasesBeforeRetry)).toEqual([ + replayedFailedRequest, + ]) + expect(requests).toHaveLength(requestsBeforeRetry) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + + const beforeWiden = requests.length + const widen = live.utils.setWindow({ offset: 0, limit: 3 }) + if (widen instanceof Promise) await widen + expect(requests).toHaveLength(beforeWiden) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2, 3]) + + const beforeReplay = requests.length + deliveredIds.clear() + begin() + truncate() + commit() + await flushPromises() + expect( + requests.slice(beforeReplay).every(({ cursor }) => !cursor), + ).toBe(true) + } finally { + rejectedPage.resolve() + await cleanupAll(live, source) + } + }, + ) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + ([`throw`, `reject`] as const).map((delivery) => ({ + direction, + delivery, + })), + ), + )( + `holds a $direction page and concurrent live insert when its boundary refinement fails by $delivery`, + async ({ direction, delivery }) => { + const sign = direction === `asc` ? 1 : -1 + const rows: Array = [ + { id: 1, rank: sign }, + { id: 2, rank: 2 * sign }, + ] + const liveInsert: PageRow = { id: 0, rank: 0 } + const delivered = new Set() + const failure = new Error(`later boundary failed`) + let widening = false + let failedBoundary: LoadSubsetOptions | undefined + let suppliedPage: Array | undefined + const source = createCollection({ + id: `pagination-boundary-publication-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + const selected = rowsForLoadSubset(rows, options) + if ( + widening && + options.where && + !options.orderBy && + selected.some(({ id }) => id === 2) && + !failedBoundary + ) { + failedBoundary = options + if (delivery === `throw`) throw failure + return Promise.reject(failure) + } + const newRows = selected.filter(({ id }) => !delivered.has(id)) + begin() + for (const row of newRows) { + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + if (widening && options.orderBy && !suppliedPage) { + suppliedPage = newRows + rows.push(liveInsert) + delivered.add(liveInsert.id) + write({ type: `insert`, value: { ...liveInsert } }) + } + const receipt = commit(options.signal) + // Make the page asynchronous so the failure is in its later + // refinement, not the synchronous setWindow call stack. + return Promise.resolve(receipt).then(() => undefined) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, direction) + .limit(1), + ) + const publications: Array> = [] + const subscription = live.subscribeChanges(() => { + publications.push(Array.from(live.values(), ({ id }) => id)) + }) + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + publications.length = 0 + widening = true + await expect( + live.utils.setWindow({ offset: 0, limit: 2 }), + ).rejects.toBe(failure) + expect(suppliedPage?.map(({ id }) => id)).toEqual([2]) + expect(failedBoundary).toBeDefined() + expect( + rowsForLoadSubset(rows, failedBoundary!).map(({ id }) => id), + ).toEqual([2]) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + expect(publications).toEqual([]) + await live.utils.setWindow({ offset: 0, limit: 2 }) + const expected = referenceWindow(rows, direction, { + offset: 0, + limit: 2, + }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual(expected) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(publications).toEqual([expected]) + } finally { + subscription.unsubscribe() + await cleanupAll(live, source) + } + }, + ) + + it(`recovers a failed tie boundary from the authoritative full source`, async () => { + const authoritativeRows: Array = [ + { id: 1, rank: -1 }, + // A provider may return equal-order rows in any order. The local public + // key tie-breaker must choose id 2 after boundary refinement. + { id: 4, rank: 0 }, + { id: 3, rank: 0 }, + { id: 2, rank: 0 }, + { id: 6, rank: 1 }, + { id: 5, rank: 99 }, ] - >)(`%s`, async (_name, scenario, expectedFailure) => { - if (!expectedFailure) { - await runMultiOrderScenario(scenario) - return + const deliveredIds = new Set() + const requests: Array = [] + const failedPage = createDeferred() + let rejectNextPage = false + let begin!: () => void + let write!: (message: { type: `insert` | `delete`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-recovered-prefix-tie-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + if (rejectNextPage) { + rejectNextPage = false + begin() + deliveredIds.add(5) + write({ type: `insert`, value: { id: 5, rank: 99 } }) + commit() + return failedPage.promise + } + + begin() + for (const row of rowsForLoadSubset(authoritativeRows, options)) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + + rejectNextPage = true + const failed = live.utils.setWindow({ offset: 0, limit: 3 }) + expect(failed).toBeInstanceOf(Promise) + failedPage.reject(new Error(`later page failed`)) + await expect(failed).rejects.toThrow(`later page failed`) + + const retry = live.utils.setWindow({ offset: 0, limit: 2 }) + if (retry instanceof Promise) await retry + const recoveryRequest = requests.at(-1) + expect(recoveryRequest?.limit).toBeUndefined() + expect(recoveryRequest?.offset).toBeUndefined() + expect(recoveryRequest?.cursor).toBeUndefined() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + + const deleted = authoritativeRows.filter(({ id }) => + [1, 2, 3].includes(id), + ) + for (const row of deleted) { + authoritativeRows.splice(authoritativeRows.indexOf(row), 1) + deliveredIds.delete(row.id) + } + begin() + for (const row of deleted) write({ type: `delete`, value: { ...row } }) + commit() + await flushPromises() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([4, 6]) + } finally { + failedPage.resolve() + await cleanupAll(live, source) } - await expectAssertionFailure(runMultiOrderScenario, { - checkpoint: 0, - classify: ({ actual, expected }) => - isNumberArray(actual) && - actual.join(`,`) === expectedFailure.actual.join(`,`) && - isNumberArray(expected) && - expected.join(`,`) === expectedFailure.expected.join(`,`), - })(scenario) }) + it(`rejects a reentrant window move when an ordered request writes and then throws`, async () => { + const authoritativeRows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + { id: 3, rank: 2 }, + ] + const requests: Array = [] + const deliveredIds = new Set() + const failure = new Error(`ordered request threw after writing`) + let reentrantError: unknown + let throwNextPage = false + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-synchronous-partial-page-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + if (throwNextPage) { + throwNextPage = false + begin() + deliveredIds.add(3) + write({ type: `insert`, value: { ...authoritativeRows[2]! } }) + commit() + try { + live.utils.setWindow({ offset: 0, limit: 3 }) + } catch (error) { + reentrantError = error + } + throw failure + } + + begin() + for (const row of rowsForLoadSubset(authoritativeRows, options)) { + if (deliveredIds.has(row.id)) continue + deliveredIds.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + const initialRequestCount = requests.length + throwNextPage = true + + expect(() => live.utils.setWindow({ offset: 0, limit: 2 })).toThrow( + failure, + ) + expect(reentrantError).toMatchObject({ name: `SetWindowReentrancyError` }) + expect(requests).toHaveLength(initialRequestCount + 1) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + + const retry = live.utils.setWindow({ offset: 0, limit: 2 }) + if (retry instanceof Promise) await retry + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + } finally { + await cleanupAll(live, source) + } + }) + + it.each( + ([`sync`, `async`] as const).flatMap((delivery) => + [1, 2].map((requestNumber) => ({ delivery, requestNumber })), + ), + )( + `rejects a window move reentered from startup request $requestNumber after $delivery delivery`, + async ({ delivery, requestNumber }) => { + const authoritativeRows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + ] + const delivered = new Set() + let nestedResult: true | Promise | undefined + let nestedError: unknown + let requests = 0 + let firstRequestSettled = false + function createWindowedQuery() { + return createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + } + const source = createCollection({ + id: `pagination-initial-request-reentrancy-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + requests++ + if (requests === requestNumber) { + if (delivery === `async` && requestNumber === 2) { + expect(firstRequestSettled).toBe(true) + } + try { + nestedResult = live.utils.setWindow({ offset: 0, limit: 2 }) + } catch (error) { + nestedError = error + } + } + const fresh = rowsForLoadSubset( + authoritativeRows, + options, + ).filter(({ id }) => !delivered.has(id)) + if (fresh.length === 0) return true + begin() + for (const row of fresh) { + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return delivery === `async` + ? Promise.resolve().then(() => { + firstRequestSettled = true + }) + : true + }, + } + }, + }, + }) + const live = createWindowedQuery() + + try { + await live.preload() + expect(requests).toBeGreaterThanOrEqual(requestNumber) + expect(nestedResult).toBeUndefined() + expect(nestedError).toMatchObject({ name: `SetWindowReentrancyError` }) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + await live.utils.setWindow({ offset: 0, limit: 2 }) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 2 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + } finally { + await cleanupAll(live, source) + } + }, + ) + + it(`rejects a window move reentered from a public change callback`, async () => { + const authoritativeRows: Array = [ + { id: 1, rank: 0, keep: true }, + { id: 2, rank: 1, keep: true }, + ] + const delivered = new Set() + let begin!: () => void + let write!: (message: { + type: `update` + value: PageRow + previousValue: PageRow + }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-publication-reentrancy-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options) => { + const fresh = rowsForLoadSubset( + authoritativeRows, + options, + ).filter(({ id }) => !delivered.has(id)) + if (fresh.length === 0) return true + begin() + for (const row of fresh) { + delivered.add(row.id) + operations.write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + let nestedResult: true | Promise | undefined + let nestedError: unknown + + try { + await live.preload() + const subscription = live.subscribeChanges(() => { + try { + nestedResult = live.utils.setWindow({ offset: 0, limit: 2 }) + } catch (error) { + nestedError = error + } + }) + const previous = authoritativeRows[0]! + const current = { ...previous, keep: false } + authoritativeRows[0] = current + begin() + write({ type: `update`, value: current, previousValue: previous }) + commit() + subscription.unsubscribe() + + expect(nestedResult).toBeUndefined() + expect(nestedError).toMatchObject({ name: `SetWindowReentrancyError` }) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + } finally { + await cleanupAll(live, source) + } + }) + + it.each([`return-only`, `write-after-cleanup`])( + `does not settle a window move after its sync session is cleaned up: %s`, + async (delivery) => { + const authoritativeRows: Array = [ + { id: 1, rank: 0 }, + { id: 2, rank: 1 }, + ] + const delivered = new Set() + let cleanUpDuringNextRequest = false + let cleanupPromise: Promise | undefined + function createWindowedQuery() { + return createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank) + .limit(1), + ) + } + const source = createCollection({ + id: `pagination-window-cleanup-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + return { + loadSubset: (options) => { + if (cleanUpDuringNextRequest) { + cleanUpDuringNextRequest = false + cleanupPromise = live.cleanup() + if (delivery === `return-only`) return true + } + const fresh = rowsForLoadSubset( + authoritativeRows, + options, + ).filter(({ id }) => !delivered.has(id)) + if (fresh.length === 0) return true + begin() + for (const row of fresh) { + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + return true + }, + } + }, + }, + }) + const live = createWindowedQuery() + + try { + await live.preload() + cleanUpDuringNextRequest = true + const move = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(cleanUpDuringNextRequest).toBe(false) + expect(cleanupPromise).toBeInstanceOf(Promise) + await cleanupPromise + + expect(move).toBeInstanceOf(Promise) + await expect(move).rejects.toMatchObject({ name: `AbortError` }) + expect(live.utils.getWindow()).toEqual({ offset: 0, limit: 1 }) + } finally { + await cleanupAll(live, source) + } + }, + ) + + it(`tracks an asynchronous prefix refresh after synchronous satisfaction`, async () => { + const rows: Array = [ + { id: 1, rank: 1 }, + { id: 2, rank: 2 }, + { id: 3, rank: 3 }, + ] + const requests: Array = [] + const delivered = new Set() + const refinement = createDeferred() + let deferLoads = false + const source = createCollection({ + id: `pagination-async-refinement-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ begin, write, commit, markReady }) => { + markReady() + const publish = (options: LoadSubsetOptions) => { + begin() + for (const row of rowsForLoadSubset(rows, options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + } + + return { + loadSubset: (options: LoadSubsetOptions) => { + requests.push(options) + if (!deferLoads) { + publish(options) + return true + } + + return refinement.promise.then(() => publish(options)) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .orderBy(({ row }) => row.id, `asc`) + .limit(1), + ) + + try { + await live.preload() + await flushPromises() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + const initialRequestCount = requests.length + expect(initialRequestCount).toBeGreaterThan(0) + expect( + requests.every( + ({ limit, where }) => limit !== undefined || where !== undefined, + ), + ).toBe(true) + deferLoads = true + + const widened = live.utils.setWindow({ offset: 0, limit: 2 }) + expect(widened).toBeInstanceOf(Promise) + await flushPromises() + expect(requests.length).toBeGreaterThan(initialRequestCount) + const widenedRequest = requests + .slice(initialRequestCount) + .find(({ limit }) => limit === 2) + expect(widenedRequest).toBeDefined() + expect(widenedRequest?.offset).toBeUndefined() + expect(widenedRequest?.cursor).toBeUndefined() + const settledBeforeRefinement = await Promise.race([ + Promise.resolve(widened).then(() => true), + new Promise((resolve) => setTimeout(() => resolve(false), 10)), + ]) + expect(settledBeforeRefinement).toBe(false) + + refinement.resolve() + if (widened instanceof Promise) await widened + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + } finally { + await cleanupAll(live, source) + } + }) + + it(`refines locale-ordered continuations locally when predicate IR cannot express the collation`, async () => { + const rows: Array = [ + { id: 1, label: `item2` }, + { id: 2, label: `item10` }, + { id: 3, label: `item11` }, + ] + const pending: Array = [] + const delivered = new Set() + let begin!: () => void + let write!: (message: { type: `insert`; value: LocaleCursorRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-locale-cursor-source-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.label, { + direction: `asc`, + nulls: `first`, + stringSort: `locale`, + locale: `en-US`, + localeOptions: { numeric: true }, + }) + .orderBy(({ row }) => row.id, `asc`) + .limit(1), + ) + + try { + const preload = live.preload() + expect(pending).toHaveLength(1) + // Settling one request can append its boundary-refinement request. + // eslint-disable-next-line @typescript-eslint/prefer-for-of + for (let index = 0; index < pending.length; index++) { + const request = pending[index]! + begin() + for (const row of rowsForLoadSubset(rows, request.options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.deferred.resolve() + await flushPromises() + } + await preload + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + expect(pending.length).toBeLessThanOrEqual(rows.length * 2) + expect( + pending.some( + ({ options }) => + options.limit === undefined && options.where === undefined, + ), + ).toBe(true) + + const transportCount = pending.length + const widened = live.utils.setWindow({ offset: 0, limit: 2 }) + for (let index = transportCount; index < pending.length; index++) { + const request = pending[index]! + begin() + for (const row of rowsForLoadSubset(rows, request.options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.deferred.resolve() + await flushPromises() + } + if (widened instanceof Promise) await widened + + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2]) + expect(pending.length).toBeLessThanOrEqual(rows.length * 3) + } finally { + for (const request of pending) request.deferred.resolve() + await cleanupAll(live, source) + } + }) + + const nullableBoundaryRows: ReadonlyArray = [ + { id: 1, primary: null, secondary: 2 }, + { id: 2, primary: null, secondary: 0 }, + { id: 3, primary: null, secondary: 1 }, + { id: 4, primary: 1, secondary: null }, + { id: 5, primary: 1, secondary: 0 }, + { id: 6, primary: 2, secondary: 0 }, + ] + + it.each([ + [ + `discovered trace: orders an ascending nullable boundary by its second term`, + { + rows: nullableBoundaryRows, + primary: { direction: `asc`, nulls: `first` }, + secondary: { direction: `asc`, nulls: `first` }, + limit: 1, + }, + ], + [ + `orders a descending nullable boundary by its second term`, + { + rows: nullableBoundaryRows, + primary: { direction: `desc`, nulls: `first` }, + secondary: { direction: `desc`, nulls: `first` }, + limit: 1, + }, + ], + [ + `orders an ascending and descending mixed nullable boundary`, + { + rows: nullableBoundaryRows, + primary: { direction: `asc`, nulls: `first` }, + secondary: { direction: `desc`, nulls: `first` }, + limit: 1, + }, + ], + [ + `discovered trace: orders a descending and ascending mixed nullable boundary`, + { + rows: nullableBoundaryRows, + primary: { direction: `desc`, nulls: `first` }, + secondary: { direction: `asc`, nulls: `first` }, + limit: 1, + }, + ], + [ + `uses the public key to break a complete tuple tie`, + { + rows: [ + { id: 2, primary: 0, secondary: 0 }, + { id: 1, primary: 0, secondary: 0 }, + ], + primary: { direction: `asc`, nulls: `last` }, + secondary: { direction: `asc`, nulls: `last` }, + limit: 1, + }, + ], + [ + `discovered trace: places nulls last in an ascending nullable boundary`, + { + rows: nullableBoundaryRows, + primary: { direction: `asc`, nulls: `last` }, + secondary: { direction: `asc`, nulls: `last` }, + limit: 1, + }, + ], + [ + `places nulls last in a descending nullable boundary`, + { + rows: nullableBoundaryRows, + primary: { direction: `desc`, nulls: `last` }, + secondary: { direction: `desc`, nulls: `last` }, + limit: 1, + }, + ], + ] satisfies ReadonlyArray)( + `%s`, + async (_name, scenario) => runMultiOrderScenario(scenario), + ) + fcTest.prop([multiOrderScenarioArbitrary], { numRuns: orderedScenarioRuns, seed: 1663, })( `matches multi-column nullable ordering for a fixed seed`, - runMultiOrderScenarioWithKnownFailures, + runMultiOrderScenario, ) fcTest.prop( [multiOrderScenarioArbitrary], - oracleRandomParameters(orderedScenarioRuns, replaySeed), + oracleRandomParameters( + orderedScenarioRuns, + replay, + `pagination.multi-order`, + ), )( `matches multi-column nullable ordering for a random or replayed seed`, - runMultiOrderScenarioWithKnownFailures, + runMultiOrderScenario, ) fcTest.prop([nullableCursorScenarioArbitrary], { @@ -2132,45 +3550,21 @@ describe(`pagination recomputation oracle`, () => { seed: 1665, })( `matches nullable cursor ordering while an async response is pending for a fixed seed`, - runNullableCursorScenarioWithKnownFailures, + runNullableCursorScenario, ) fcTest.prop( [nullableCursorScenarioArbitrary], - oracleRandomParameters(transitionScenarioRuns, replaySeed), + oracleRandomParameters( + transitionScenarioRuns, + replay, + `pagination.nullable-cursor`, + ), )( `matches nullable cursor ordering while an async response is pending for a random or replayed seed`, - runNullableCursorScenarioWithKnownFailures, + runNullableCursorScenario, ) - it(`rejects collateral output from the nullable cursor classifier`, () => { - expect( - isKnownNullableCursorOrderingFailure( - { rank: 0, direction: `asc` }, - assertionDifference(0, [], [1]), - ), - ).toBe(false) - }) - - it(`rejects collateral output from the secondary-order classifier`, () => { - const scenario: MultiOrderScenario = { - rows: [ - { id: 2, primary: -2, secondary: 0 }, - { id: 1, primary: -2, secondary: null }, - ], - primary: { direction: `asc`, nulls: `first` }, - secondary: { direction: `asc`, nulls: `last` }, - limit: 1, - } - - expect( - isKnownSecondaryOrderBoundaryFailure( - scenario, - assertionDifference(0, [], [2]), - ), - ).toBe(false) - }) - it.each([ [`boundary insert`, { type: `insert`, row: { id: 5, rank: 0.5 } }], [`visible delete`, { type: `delete`, id: 1 }], @@ -2193,28 +3587,353 @@ describe(`pagination recomputation oracle`, () => { }, ) - it(`discovered trace: a settled rank update refreshes top-k membership`, async () => { - const scenario: PendingMutationScenario = { - ranks: [0, 0, 1], - direction: `desc`, - limit: 1, - mutation: { type: `update`, row: { id: 3, rank: 0 } }, - responseOutcome: `resolve`, + it.each([ + [`insert`, { type: `insert`, row: { id: 9, rank: 0.5 } }], + [`delete`, { type: `delete`, id: 1 }], + [`rank update`, { type: `update`, row: { id: 2, rank: 10 } }], + ] satisfies ReadonlyArray)( + `revalidates a finite ordered prefix after a settled SSE %s`, + async (_name, mutation) => { + await runPendingMutationScenario( + { + ranks: [0, 1, 2, 3, 4, 5, 6, 7], + direction: `asc`, + limit: 2, + mutation, + responseOutcome: `resolve`, + }, + `after-response`, + ) + }, + ) + + it(`retains a finite inactive prefix across shrink, SSE, and re-expansion`, async () => { + const rows = new Map( + Array.from({ length: 5 }, (_, index) => [ + index + 1, + { id: index + 1, rank: index + 1 }, + ]), + ) + const delivered = new Set() + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-retained-prefix-live-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const ordered = [...rows.values()].sort( + (left, right) => left.rank - right.rank || left.id - right.id, + ) + const requested = rowsForLoadSubset(ordered, options) + begin() + for (const row of requested) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + const receipt = commit() + return Promise.resolve(receipt) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .orderBy(({ row }) => row.id, `asc`) + .limit(3), + ) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2, 3]) + + await live.utils.setWindow({ offset: 0, limit: 1 }) + const inserted = { id: 9, rank: 2.5 } + rows.set(inserted.id, inserted) + delivered.add(inserted.id) + begin() + write({ type: `insert`, value: inserted }) + commit() + await flushPromises() + + await live.utils.setWindow({ offset: 0, limit: 3 }) + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1, 2, 9]) + } finally { + await cleanupAll(live, source) } - await expectAssertionFailure( - () => runPendingMutationScenario(scenario, `after-response`), + }) + + it.each([`asc`, `desc`] as const)( + `refreshes from the start when one SSE batch moves the retained prefix (%s)`, + async (direction) => { + const rows = new Map( + Array.from({ length: 6 }, (_, index) => [ + index + 1, + { id: index + 1, rank: index + 1 }, + ]), + ) + const delivered = new Set() + const loads: Array = [] + let begin!: () => void + let write!: (message: { + type: `insert` | `update` + value: PageRow + }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-batch-prefix-refresh-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + loads.push(options) + const settled = new Promise((resolve) => { + queueMicrotask(() => { + const ordered = referenceWindowRows( + [...rows.values()], + direction, + { offset: 0, limit: rows.size }, + ) + begin() + for (const row of rowsForLoadSubset(ordered, options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + resolve() + }) + }) + return settled + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, direction) + .orderBy(({ row }) => row.id, `asc`) + .limit(2), + ) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual( + direction === `asc` ? [1, 2] : [6, 5], + ) + + begin() + const movedIds = direction === `asc` ? [1, 2, 3, 4] : [3, 4, 5, 6] + for (const id of movedIds) { + const row = { + id, + rank: direction === `asc` ? 100 + id : -100 - id, + } + rows.set(id, row) + write({ type: `update`, value: { ...row } }) + } + commit() + for (let index = 0; index < 5; index++) await flushPromises() + + expect(Array.from(live.values(), ({ id }) => id)).toEqual( + direction === `asc` ? [5, 6] : [2, 1], + ) + expect( + loads.some( + ({ limit, cursor }) => limit === 2 && cursor === undefined, + ), + ).toBe(true) + } finally { + await cleanupAll(live, source) + } + }, + ) + + it.each([ + [`insert`, { type: `insert`, row: { id: 7, rank: 0 } }, [7, 1], [7, 1, 2]], + [`update`, { type: `update`, row: { id: 2, rank: -1 } }, [2, 1], [2, 1, 3]], + [`delete`, { type: `delete`, id: 1 }, [2, 3], [2, 3, 4]], + ] satisfies ReadonlyArray< + readonly [ + string, + PendingMutation, + ReadonlyArray, + ReadonlyArray, + ] + >)( + `keeps an SSE %s that arrives during boundary refinement`, + async (_name, mutation, expectedIds, expectedWideIds) => { + const rows = new Map( + Array.from({ length: 6 }, (_, index) => [ + index + 1, + { id: index + 1, rank: index + 1 }, + ]), + ) + const delivered = new Set() + const pending: Array = [] + let begin!: () => void + let write!: (message: { + type: `insert` | `update` | `delete` + value: PageRow + }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-pending-refinement-sse-${collectionSequence++}`, + getKey: (row) => row.id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (params) => { + begin = params.begin + write = params.write + commit = params.commit + params.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + const deferred = createDeferred() + pending.push({ options, deferred }) + return deferred.promise + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .orderBy(({ row }) => row.id, `asc`) + .limit(2), + ) + + const settle = async (request: PendingCursorLoad) => { + const ordered = referenceWindowRows([...rows.values()], `asc`, { + offset: 0, + limit: rows.size, + }) + begin() + for (const row of rowsForLoadSubset(ordered, request.options)) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + request.deferred.resolve() + await flushPromises() + } + + try { + const preload = live.preload() + expect(pending).toHaveLength(1) + await settle(pending[0]!) + expect(pending).toHaveLength(2) + + begin() + if (mutation.type === `delete`) { + const row = rows.get(mutation.id)! + rows.delete(mutation.id) + delivered.delete(mutation.id) + write({ type: `delete`, value: { ...row } }) + } else { + rows.set(mutation.row.id, { ...mutation.row }) + delivered.add(mutation.row.id) + write({ type: mutation.type, value: { ...mutation.row } }) + } + commit() + + await settle(pending[1]!) + expect( + pending.some( + ({ options }) => + options.limit === 2 && options.cursor === undefined, + ), + ).toBe(true) + for (let index = 2; index < pending.length; index++) { + await settle(pending[index]!) + } + await preload + + expect(Array.from(live.values(), ({ id }) => id)).toEqual(expectedIds) + + const pendingBeforeWiden = pending.length + const widened = live.utils.setWindow({ offset: 0, limit: 3 }) + await flushPromises() + if (mutation.type === `insert`) { + expect(pending.length).toBeGreaterThan(pendingBeforeWiden) + expect( + pending + .slice(pendingBeforeWiden) + .some(({ options }) => options.limit === 3), + ).toBe(true) + } else { + expect( + pending.some( + ({ options }) => + options.limit === undefined && + options.where === undefined && + options.cursor === undefined, + ), + ).toBe(true) + expect(widened).toBe(true) + expect(pending).toHaveLength(pendingBeforeWiden) + } + for (let index = pendingBeforeWiden; index < pending.length; index++) { + await settle(pending[index]!) + } + if (widened instanceof Promise) await widened + expect(Array.from(live.values(), ({ id }) => id)).toEqual( + expectedWideIds, + ) + } finally { + for (const request of pending) request.deferred.resolve() + await cleanupAll(live, source) + } + }, + ) + + it(`does not use a new row beyond finite coverage as a widening boundary`, async () => { + await runPendingMutationScenario( { - checkpoint: 0, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - sameRows(actual, [{ id: 3, rank: 0 }]) && - isPageRowArray(expected) && - sameRows(expected, [{ id: 1, rank: 0 }]), + ranks: [0, 1, 2, 3, 4, 5, 6, 7], + direction: `asc`, + limit: 2, + mutation: { type: `insert`, row: { id: 9, rank: 4.5 } }, + responseOutcome: `resolve`, }, - )() + `after-response`, + 5, + ) }) - it(`rejects collateral output from the settled top-k classifier`, () => { + it(`discovered trace: a settled rank update refreshes top-k membership`, async () => { const scenario: PendingMutationScenario = { ranks: [0, 0, 1], direction: `desc`, @@ -2222,45 +3941,10 @@ describe(`pagination recomputation oracle`, () => { mutation: { type: `update`, row: { id: 3, rank: 0 } }, responseOutcome: `resolve`, } - - expect( - isKnownSettledTopKMembershipFailure( - scenario, - `after-response`, - assertionDifference(0, [{ id: 2, rank: 0 }], [{ id: 1, rank: 0 }]), - ), - ).toBe(false) - }) - - it(`discovered trace: a rejected cursor does not treat a live insert as remote coverage`, async () => { - const scenario: PendingMutationScenario = { - ranks: [0, -1, 0], - direction: `asc`, - limit: 1, - mutation: { type: `insert`, row: { id: 4, rank: 0 } }, - responseOutcome: `reject`, - } - - await expectAssertionFailure( - () => runPendingMutationScenario(scenario, `before-response`), - { - checkpoint: 0, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - sameRows(actual, [ - { id: 2, rank: -1 }, - { id: 4, rank: 0 }, - ]) && - isPageRowArray(expected) && - sameRows(expected, [ - { id: 2, rank: -1 }, - { id: 1, rank: 0 }, - ]), - }, - )() + await runPendingMutationScenario(scenario, `after-response`) }) - it(`rejects collateral output from the rejected-cursor retry classifier`, () => { + it(`a rejected cursor does not treat a live insert as remote coverage`, async () => { const scenario: PendingMutationScenario = { ranks: [0, -1, 0], direction: `asc`, @@ -2269,24 +3953,7 @@ describe(`pagination recomputation oracle`, () => { responseOutcome: `reject`, } - const collateral = assertionDifference( - 0, - [{ id: 4, rank: 0 }], - [ - { id: 2, rank: -1 }, - { id: 1, rank: 0 }, - ], - ) - - expect( - isKnownRejectedCursorRetryFailure( - scenario, - new PendingMutationTraceAssertionError(collateral.cause, [ - { id: 2, rank: -1 }, - { id: 4, rank: 0 }, - ]), - ), - ).toBe(false) + await runPendingMutationScenario(scenario, `before-response`) }) fcTest.prop([pendingMutationScenarioArbitrary, responseTimingArbitrary], { @@ -2294,7 +3961,7 @@ describe(`pagination recomputation oracle`, () => { seed: 1660, })( `matches recomputation when source mutations cross a pending cursor response for a fixed seed`, - runPendingMutationScenarioWithKnownFailures, + runPendingMutationScenario, ) it.each( @@ -2314,7 +3981,7 @@ describe(`pagination recomputation oracle`, () => { : mutationKind === `update` ? { type: `update`, row: { id: 2, rank: -1 } } : { type: `delete`, id: 2 } - await runPendingMutationScenarioWithKnownFailures( + await runPendingMutationScenario( { ranks: [0, 1, 2, 3], direction: `asc`, @@ -2329,162 +3996,45 @@ describe(`pagination recomputation oracle`, () => { fcTest.prop( [pendingMutationScenarioArbitrary, responseTimingArbitrary], - oracleRandomParameters(transitionScenarioRuns, replaySeed), - )( - `matches recomputation when source mutations cross a pending cursor response for a random or replayed seed`, - runPendingMutationScenarioWithKnownFailures, - ) - - it( - `discovered trace: retries a rejected cursor after a source and window transition`, - expectAssertionFailure(runRejectedCursorRetryAfterMutation, { - checkpoint: 0, - classify: ({ actual, expected }) => - isNumberArray(actual) && - actual.join(`,`) === `1,4` && - isNumberArray(expected) && - expected.join(`,`) === `2,3,1`, - }), - ) - - fcTest.prop([pendingHistoryScenarioArbitrary], { - numRuns: transitionScenarioRuns, - seed: 1664, - })( - `matches recomputation across multi-action pending histories for a fixed seed`, - runPendingHistoryScenarioWithKnownFailures, - ) - - fcTest.prop( - [pendingHistoryScenarioArbitrary], - oracleRandomParameters(transitionScenarioRuns, replaySeed), + oracleRandomParameters( + transitionScenarioRuns, + replay, + `pagination.pending-mutation`, + ), )( - `matches recomputation across multi-action pending histories for a random or replayed seed`, - runPendingHistoryScenarioWithKnownFailures, - ) - - it(`rejects collateral output from the late pending-history classifier`, () => { - const scenario: PendingHistoryScenario = { - ranks: [0, 0, 0, 0], - direction: `asc`, - initialLimit: 2, - narrowLimit: 1, - wideLimit: 3, - firstRank: 0, - secondRank: 0, - } - const expectedRows = referenceWindowRows( - [...pendingHistoryRows(scenario).values()], - scenario.direction, - { offset: 0, limit: scenario.wideLimit }, - ) - const cause = assertionDifference( - 0, - { - rows: [{ id: 4, rank: 0 }], - modeledDeliveredRows: expectedRows.slice(0, 2), - }, - { - rows: expectedRows, - modeledDeliveredRows: expectedRows.slice(0, 2), - }, - ) - - expect(isKnownLatePendingHistoryUnderfill(scenario, cause)).toBe(false) - }) - - it( - `discovered trace: an in-flight request does not underfill a new window`, - expectAssertionFailure(expectInflightRequestFillsNewWindow, { - checkpoint: 0, - classify: ({ actual, expected }) => - isNumberArray(actual) && - actual.length === 0 && - isNumberArray(expected) && - expected.join(`,`) === `3,4`, - }), - ) - - it(`rejects collateral loss from the ordered-subscription classifier`, () => { - const scenario: PaginationStateScenario = { - ranks: [0, 1, 2], - direction: `asc`, - initialWindow: { offset: 0, limit: 3 }, - actions: [{ type: `put`, id: 4, rank: 2 }], - } - const expected = [ - { id: 1, rank: 0 }, - { id: 2, rank: 1 }, - { id: 3, rank: 2 }, - ] - - expect( - isKnownOrderedSubscriptionCoverageFailure( - scenario, - assertionDifference(1, [expected[0]!], expected), - ), - ).toBe(false) - }) - - it(`rejects arbitrary leading loss after an offset shift`, () => { - const scenario: PaginationStateScenario = { - ranks: [0, 1, 2, 3], - direction: `asc`, - initialWindow: { offset: 0, limit: 1 }, - actions: [ - { type: `put`, id: 5, rank: -1 }, - { type: `window`, offset: 1, limit: 3 }, - ], - } - const expected = [ - { id: 1, rank: 0 }, - { id: 2, rank: 1 }, - { id: 3, rank: 2 }, - ] - - expect( - isKnownOrderedSubscriptionCoverageFailure( - scenario, - assertionDifference(2, [expected[2]!], expected), - ), - ).toBe(false) - }) - - it(`rejects excessive suffix loss from the on-demand classifier`, () => { - const scenario: PaginationScenario = { - ranks: [0, 1, 2, 3], - direction: `asc`, - windows: [ - { offset: 0, limit: 1 }, - { offset: 1, limit: 3 }, - ], - } + `matches recomputation when source mutations cross a pending cursor response for a random or replayed seed`, + runPendingMutationScenario, + ) - expect( - isKnownOnDemandOffsetUnderfetch( - scenario, - assertionDifference(1, [2], [2, 3, 4]), - ), - ).toBe(false) - }) + it( + `discovered trace: retries a rejected cursor after a source and window transition`, + runRejectedCursorRetryAfterMutation, + ) - it(`rejects a corrupted expectation from the on-demand classifier`, () => { - const scenario: PaginationScenario = { - ranks: [0, 0, 0, 0, 0, 0, 1], - direction: `asc`, - windows: [ - { offset: 0, limit: 1 }, - { offset: 2, limit: 5 }, - ], - } + fcTest.prop([pendingHistoryScenarioArbitrary], { + numRuns: transitionScenarioRuns, + seed: 1664, + })( + `matches recomputation across multi-action pending histories for a fixed seed`, + runPendingHistoryScenario, + ) - expect( - isKnownOnDemandOffsetUnderfetch( - scenario, - assertionDifference(1, [3, 4, 5, 6], [3, 4, 5, 6, 99]), - ), - ).toBe(false) - }) + fcTest.prop( + [pendingHistoryScenarioArbitrary], + oracleRandomParameters( + transitionScenarioRuns, + replay, + `pagination.pending-history`, + ), + )( + `matches recomputation across multi-action pending histories for a random or replayed seed`, + runPendingHistoryScenario, + ) + + it( + `discovered trace: an in-flight request does not underfill a new window`, + expectInflightRequestFillsNewWindow, + ) it(`discovered trace: a row moving across an offset window must refill its boundary`, async () => { const scenario: PaginationStateScenario = { @@ -2493,14 +4043,7 @@ describe(`pagination recomputation oracle`, () => { initialWindow: { offset: 1, limit: 1 }, actions: [{ type: `put`, id: 1, rank: -1 }], } - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 1, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - isPageRowArray(expected) && - sameRows(actual, [{ id: 1, rank: -1 }]) && - sameRows(expected, [{ id: 3, rank: 0 }]), - })(scenario) + await runPaginationStateScenario(scenario) }) it(`retains authoritative rows when a later window admits a prior insert`, async () => { @@ -2513,21 +4056,7 @@ describe(`pagination recomputation oracle`, () => { { type: `window`, offset: 0, limit: 3 }, ], } - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 2, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - isPageRowArray(expected) && - sameRows(actual, [ - { id: 1, rank: 0 }, - { id: 3, rank: 1 }, - ]) && - sameRows(expected, [ - { id: 1, rank: 0 }, - { id: 2, rank: 0 }, - { id: 3, rank: 1 }, - ]), - })(scenario) + await runPaginationStateScenario(scenario) }) it(`restores an out-of-window insert when a later offset selects it`, async () => { @@ -2540,14 +4069,7 @@ describe(`pagination recomputation oracle`, () => { { type: `window`, offset: 2, limit: 1 }, ], } - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 2, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - actual.length === 0 && - isPageRowArray(expected) && - sameRows(expected, [{ id: 3, rank: 1 }]), - })(scenario) + await runPaginationStateScenario(scenario) }) it(`restores an out-of-window rank update when a later offset selects it`, async () => { @@ -2560,14 +4082,7 @@ describe(`pagination recomputation oracle`, () => { { type: `window`, offset: 2, limit: 1 }, ], } - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 2, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - actual.length === 0 && - isPageRowArray(expected) && - sameRows(expected, [{ id: 2, rank: 1 }]), - })(scenario) + await runPaginationStateScenario(scenario) }) it(`discovered trace: inserting at an empty offset boundary refills the window`, async () => { @@ -2581,14 +4096,7 @@ describe(`pagination recomputation oracle`, () => { { type: `put`, id: 4, rank: 1 }, ], } - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 3, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - actual.length === 0 && - isPageRowArray(expected) && - sameRows(expected, [{ id: 4, rank: 1 }]), - })(scenario) + await runPaginationStateScenario(scenario) }) it(`discovered trace: an insert before a later offset does not skip its new boundary`, async () => { @@ -2602,14 +4110,7 @@ describe(`pagination recomputation oracle`, () => { ], } - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 2, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - sameRows(actual, [{ id: 9, rank: 1 }]) && - isPageRowArray(expected) && - sameRows(expected, [{ id: 8, rank: 1 }]), - })(scenario) + await runPaginationStateScenario(scenario) }) it(`discovered trace: an async cursor loads the full offset window`, async () => { @@ -2621,14 +4122,7 @@ describe(`pagination recomputation oracle`, () => { { offset: 2, limit: 5 }, ], } - await expectAssertionFailure(runOnDemandPaginationScenario, { - checkpoint: 1, - classify: ({ actual, expected }) => - isNumberArray(actual) && - isNumberArray(expected) && - actual.join(`,`) === `3,4,5,6` && - expected.join(`,`) === `3,4,5,6,7`, - })(scenario) + await runOnDemandPaginationScenario(scenario) }) it(`discovered trace: an async cursor crosses an offset before filling one row`, async () => { @@ -2640,14 +4134,53 @@ describe(`pagination recomputation oracle`, () => { { offset: 2, limit: 1 }, ], } - await expectAssertionFailure(runOnDemandPaginationScenario, { - checkpoint: 1, - classify: ({ actual, expected }) => - isNumberArray(actual) && - actual.length === 0 && - isNumberArray(expected) && - expected.join(`,`) === `2`, - })(scenario) + await runOnDemandPaginationScenario(scenario) + }) + + it.each( + [`asc`, `desc`].flatMap((direction) => + [false, true].flatMap((explicitPublicKeyOrder) => + [false, true].map((includeFilter) => ({ + direction: direction as `asc` | `desc`, + explicitPublicKeyOrder, + includeFilter, + })), + ), + ), + )( + `bounds requests for an underfilled source: $direction, explicit key=$explicitPublicKeyOrder, filter=$includeFilter`, + async (structure) => { + await runOnDemandPaginationScenario({ + ...structure, + ranks: [0, 0], + keeps: [true, false], + windows: [{ offset: 0, limit: 3 }], + }) + }, + ) + + it.each( + paginationStructures.map((structure, index) => ({ + name: `key=${structure.explicitPublicKeyOrder ? `explicit` : `implicit`}, filter=${structure.includeFilter ? `on` : `off`}, insertion=${structure.reverseInsertion ? `reverse` : `forward`}`, + structure, + index, + })), + )(`covers $name`, async ({ structure, index }) => { + const cellRuns = Math.max(1, Math.ceil(transitionScenarioRuns / 8)) + await fc.assert( + fc.asyncProperty(scenarioPayloadArbitrary, async (scenario) => { + const complete = { ...scenario, ...structure } + await runPaginationScenario(complete) + await runOnDemandPaginationScenario(complete) + }), + { numRuns: cellRuns, seed: 16_570 + index }, + ) + await fc.assert( + fc.asyncProperty(stateScenarioPayloadArbitrary, async (scenario) => { + await runPaginationStateScenario({ ...scenario, ...structure }) + }), + { numRuns: cellRuns, seed: 16_580 + index }, + ) }) fcTest.prop([scenarioArbitrary], { @@ -2658,7 +4191,14 @@ describe(`pagination recomputation oracle`, () => { runPaginationScenario, ) - fcTest.prop([scenarioArbitrary], orderedScenarioRandomParameters)( + fcTest.prop( + [scenarioArbitrary], + oracleRandomParameters( + orderedScenarioRuns, + replay, + `pagination.ordered-window`, + ), + )( `matches full recomputation across ordered windows for a random or replayed seed`, runPaginationScenario, ) @@ -2668,15 +4208,139 @@ describe(`pagination recomputation oracle`, () => { seed: 1658, })( `matches full recomputation across source and window transitions for a fixed seed`, - runPaginationStateScenarioWithKnownFailures, + runPaginationStateScenario, ) fcTest.prop( [stateScenarioArbitrary], - oracleRandomParameters(transitionScenarioRuns, replaySeed), + oracleRandomParameters( + transitionScenarioRuns, + replay, + `pagination.window-transition`, + ), )( `matches full recomputation across source and window transitions for a random or replayed seed`, - runPaginationStateScenarioWithKnownFailures, + runPaginationStateScenario, + ) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + ([`pages`, `widen`] as const).flatMap((mode) => + [3, 10].map((pageSize) => ({ direction, mode, pageSize })), + ), + ), + )( + `fetches linear row volume while traversing settled pages: %j`, + async ({ direction, mode, pageSize }) => { + const pageCount = 10 + const rows = Array.from({ length: pageCount * pageSize }, (_, rank) => ({ + id: rank + 1, + rank, + })) + const ordered = direction === `asc` ? rows : [...rows].reverse() + const { source, requests } = createConformingOrderedSource( + `pagination-transfer-${collectionSequence++}`, + ordered, + ) + const live = createLiveQueryCollection((q) => + q + .from({ row: source }) + .orderBy(({ row }) => row.rank, direction) + .limit(pageSize), + ) + try { + await live.preload() + for (let page = 0; page < pageCount; page++) { + const offset = mode === `pages` ? page * pageSize : 0 + const limit = mode === `pages` ? pageSize : (page + 1) * pageSize + if (page > 0) await live.utils.setWindow({ offset, limit }) + expect([...live.values()].map(projectPageRow)).toEqual( + ordered.slice(offset, offset + limit), + ) + } + // Count every provider-returned row, including duplicates and tie + // probes. Request counts alone cannot detect repeated growing prefixes. + const returnedRows = requests.reduce( + (total, request) => + total + rowsForLoadSubset(ordered, request).length, + 0, + ) + expect(returnedRows).toBeLessThanOrEqual(rows.length + 2 * pageCount) + expect(requests.some((request) => request.cursor !== undefined)).toBe( + true, + ) + } finally { + await live.cleanup() + await source.cleanup() + } + }, + ) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + [false, true].flatMap((explicitPublicKeyOrder) => + [false, true].flatMap((tied) => + [1, 2].map((limit) => ({ + direction, + explicitPublicKeyOrder, + tied, + limit, + })), + ), + ), + ), + )( + `loads the source prefix when moving past an intervening insert: %j`, + async ({ direction, explicitPublicKeyOrder, tied, limit }) => { + const sign = direction === `asc` ? 1 : -1 + await runPaginationStateScenario({ + direction, + initialWindow: { offset: 0, limit: 1 }, + actions: [ + { type: `put`, id: 3, rank: sign * (tied ? 1 : 2), keep: false }, + { type: `window`, offset: 1, limit }, + { type: `put`, id: 1, rank: 0, keep: false }, + ], + ranks: [0, sign], + keeps: [false, false], + explicitPublicKeyOrder, + includeFilter: false, + reverseInsertion: false, + }) + }, + ) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + ([`before-response`, `after-response`] as const).flatMap((timing) => + [0.5, 100].flatMap((rank) => + ([`cursor`, `offset`, `key`] as const).map((transport) => ({ + direction, + timing, + rank, + transport, + })), + ), + ), + ), + )( + `keeps live observations separate from a settled acquisition: %j`, + async ({ direction, timing, rank, transport }) => { + const sign = direction === `asc` ? 1 : -1 + await runPendingMutationScenario( + { + ranks: [0, sign, 2 * sign, 3 * sign], + direction, + limit: 1, + mutation: { type: `insert`, row: { id: 9, rank: sign * rank } }, + responseOutcome: `resolve`, + }, + timing, + 3, + false, + transport, + ) + }, ) it(`discovered trace: a rank update must refill a top-1 window`, async () => { @@ -2686,17 +4350,233 @@ describe(`pagination recomputation oracle`, () => { initialWindow: { offset: 0, limit: 1 }, actions: [{ type: `put`, id: 1, rank: 1 }], } - const staleMembership = [{ id: 1, rank: 1 }] - const expected = [{ id: 2, rank: 0 }] + await runPaginationStateScenario(scenario) + }) + + it(`refills an implicit tie window when a visible row moves below it`, async () => { + await runPaginationStateScenario({ + ranks: [0, 0, 0, -1, 0], + direction: `desc`, + explicitPublicKeyOrder: false, + includeFilter: false, + reverseInsertion: false, + initialWindow: { offset: 0, limit: 4 }, + actions: [{ type: `put`, id: 1, rank: -2, keep: false }], + }) + }) + + it.each([`resolve`, `reject`] as const)( + `keeps the last complete implicit window while background recovery %ss`, + async (settlement) => { + const authoritativeRows = new Map([ + [1, { id: 1, rank: 0, keep: true }], + [2, { id: 2, rank: 1, keep: true }], + ]) + const recovery = createDeferred() + const recoveryError = new Error(`background recovery failed`) + const loads: Array = [] + const delivered = new Set() + let recovering = false + let begin!: () => void + let write!: (message: { + type: `insert` | `update` + value: PageRow + }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-background-prefix-recovery-${collectionSequence++}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + loads.push(options) + const isFullSource = + options.where === undefined && + options.limit === undefined && + options.cursor === undefined + const applyRows = () => { + const rows = rowsForLoadSubset( + [...authoritativeRows.values()], + options, + ) + begin() + for (const row of rows) { + if (delivered.has(row.id)) continue + delivered.add(row.id) + write({ type: `insert`, value: { ...row } }) + } + commit() + } + if (!recovering || !isFullSource) { + applyRows() + return true + } + return recovery.promise.then(() => { + if (settlement === `reject`) throw recoveryError + applyRows() + }) + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1) + .select(({ row }) => ({ + id: row.id, + rank: row.rank, + keep: row.keep, + })), + ) + const publications: Array> = [] + const subscription = live.subscribeChanges( + () => publications.push(live.toArray.map(projectPageRow)), + { includeInitialState: false }, + ) + + try { + await live.preload() + expect(live.toArray.map(projectPageRow)).toEqual([{ id: 1, rank: 0 }]) + publications.length = 0 + const loadsBeforeMutation = loads.length + + recovering = true + const moved = { id: 1, rank: 10, keep: true } + authoritativeRows.set(1, moved) + begin() + write({ type: `update`, value: { ...moved } }) + commit() + await flushPromises() + + const recoveryLoads = loads.slice(loadsBeforeMutation) + expect(recoveryLoads).toHaveLength(1) + expect(recoveryLoads[0]?.where).toBeUndefined() + expect(recoveryLoads[0]?.limit).toBeUndefined() + expect(recoveryLoads[0]?.cursor).toBeUndefined() + expect(live.toArray.map(projectPageRow)).toEqual([{ id: 1, rank: 0 }]) + expect(publications).toEqual([]) + + recovery.resolve() + await flushPromises() + + if (settlement === `resolve`) { + expect(live.toArray.map(projectPageRow)).toEqual([{ id: 2, rank: 1 }]) + expect(publications).toEqual([[{ id: 2, rank: 1 }]]) + expect(live.utils.lastSubsetError).toBeUndefined() + } else { + expect(live.toArray.map(projectPageRow)).toEqual([{ id: 1, rank: 0 }]) + expect(publications).toEqual([]) + expect(live.utils.lastSubsetError).toBe(recoveryError) + } + } finally { + recovery.resolve() + subscription.unsubscribe() + await cleanupAll(live, source) + } + }, + ) + + it(`does not recover the full source when a visible row keeps its order`, async () => { + const loads: Array = [] + let begin!: () => void + let write!: (message: { type: `insert` | `update`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-stable-order-update-${collectionSequence++}`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options: LoadSubsetOptions) => { + loads.push(options) + begin() + write({ + type: `insert`, + value: { id: 1, rank: 0, keep: true }, + }) + commit() + return true + }, + } + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + const loadsBeforeMutation = loads.length + begin() + write({ type: `update`, value: { id: 1, rank: 0, keep: false } }) + commit() + await flushPromises() + + expect( + live.toArray.map(({ id, rank, keep }) => ({ id, rank, keep })), + ).toEqual([{ id: 1, rank: 0, keep: false }]) + expect(loads).toHaveLength(loadsBeforeMutation) + } finally { + await cleanupAll(live, source) + } + }) + + it.each([ + [`top-one`, [0, 0], { offset: 0, limit: 1 }, 1, 1], + [`offset`, [0, 0, 1], { offset: 1, limit: 1 }, 2, 2], + ] as const)( + `refills an implicit %s tie window after a rank update`, + async (_name, ranks, initialWindow, id, rank) => { + await runPaginationStateScenario({ + ranks: [...ranks], + direction: `asc`, + explicitPublicKeyOrder: false, + includeFilter: false, + reverseInsertion: false, + initialWindow, + actions: [{ type: `put`, id, rank, keep: false }], + }) + }, + ) - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 1, - classify: (difference) => - isPageRowArray(difference.actual) && - isPageRowArray(difference.expected) && - sameRows(difference.actual, staleMembership) && - sameRows(difference.expected, expected), - })(scenario) + it(`opens an implicit tie window from zero at the lowest public key`, async () => { + await runPaginationStateScenario({ + ranks: [0], + direction: `asc`, + explicitPublicKeyOrder: false, + includeFilter: false, + reverseInsertion: false, + initialWindow: { offset: 0, limit: 0 }, + actions: [ + { type: `put`, id: 2, rank: 0, keep: false }, + { type: `window`, offset: 0, limit: 1 }, + { type: `delete`, id: 1 }, + ], + }) }) it(`ignores an out-of-window insert when refilling after a delete`, async () => { @@ -2709,25 +4589,7 @@ describe(`pagination recomputation oracle`, () => { { type: `delete`, id: 2 }, ], } - const defective = [ - { id: 1, rank: 100 }, - { id: 3, rank: 80 }, - { id: 5, rank: 10 }, - ] - const expected = [ - { id: 1, rank: 100 }, - { id: 3, rank: 80 }, - { id: 4, rank: 70 }, - ] - - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 2, - classify: (difference) => - isPageRowArray(difference.actual) && - isPageRowArray(difference.expected) && - sameRows(difference.actual, defective) && - sameRows(difference.expected, expected), - })(scenario) + await runPaginationStateScenario(scenario) }) it(`ignores an out-of-window rank update when refilling after a delete`, async () => { @@ -2741,14 +4603,7 @@ describe(`pagination recomputation oracle`, () => { ], } - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 2, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - sameRows(actual, [{ id: 2, rank: 1 }]) && - isPageRowArray(expected) && - sameRows(expected, [{ id: 3, rank: 0 }]), - })(scenario) + await runPaginationStateScenario(scenario) }) it(`ignores an out-of-window rank update when the visible row leaves`, async () => { @@ -2762,14 +4617,7 @@ describe(`pagination recomputation oracle`, () => { ], } - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 2, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - sameRows(actual, [{ id: 2, rank: 1 }]) && - isPageRowArray(expected) && - sameRows(expected, [{ id: 3, rank: 0 }]), - })(scenario) + await runPaginationStateScenario(scenario) }) it(`refills untouched rows when widening after an out-of-window rank update`, async () => { @@ -2783,21 +4631,7 @@ describe(`pagination recomputation oracle`, () => { ], } - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 2, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - sameRows(actual, [ - { id: 1, rank: 0 }, - { id: 2, rank: -1 }, - ]) && - isPageRowArray(expected) && - sameRows(expected, [ - { id: 1, rank: 0 }, - { id: 3, rank: 0 }, - { id: 2, rank: -1 }, - ]), - })(scenario) + await runPaginationStateScenario(scenario) }) it(`rebuilds the full boundary when widening after an out-of-window rank update`, async () => { @@ -2811,24 +4645,7 @@ describe(`pagination recomputation oracle`, () => { ], } - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 2, - classify: ({ actual, expected }) => - isPageRowArray(actual) && - sameRows(actual, [ - { id: 1, rank: 1 }, - { id: 2, rank: 0 }, - { id: 3, rank: 0 }, - { id: 4, rank: 0 }, - ]) && - isPageRowArray(expected) && - sameRows(expected, [ - { id: 1, rank: 1 }, - { id: 5, rank: 1 }, - { id: 2, rank: 0 }, - { id: 3, rank: 0 }, - ]), - })(scenario) + await runPaginationStateScenario(scenario) }) it(`ignores an out-of-window insert when widening a tied window`, async () => { @@ -2841,52 +4658,278 @@ describe(`pagination recomputation oracle`, () => { { type: `window`, offset: 0, limit: 2 }, ], } - const defective = [ - { id: 1, rank: 0 }, - { id: 3, rank: 0 }, - ] - const expected = [ + await runPaginationStateScenario(scenario) + }) + + it(`expands a multi-column boundary before choosing top-K`, async () => { + await expectMultiOrderBoundaryMatches() + }) + + it(`expands a provider tie before applying the public-key tie-breaker`, async () => { + const loads = await runAdversarialOrderedProviderScenario({ + providerRows: [ + { id: 2, rank: 0, label: `second` }, + { id: 1, rank: 0, label: `first` }, + { id: 3, rank: 1, label: `third` }, + ], + order: { kind: `rank`, direction: `asc`, nulls: `first` }, + limit: 1, + expectedIds: [1], + }) + + expect(loads).toHaveLength(2) + expect(loads[1]?.where).toBeDefined() + expect(loads[1]?.cursor).toBeUndefined() + }) + + it(`does not derive an ordered boundary from another demand's local row`, async () => { + const unrelated = { id: 100, rank: 100, label: `unrelated` } + const loads = await runAdversarialOrderedProviderScenario({ + providerRows: [ + { id: 1, rank: 1, label: `first` }, + { id: 2, rank: 2, label: `second` }, + unrelated, + ], + initialRows: [unrelated], + order: { kind: `rank`, direction: `asc`, nulls: `first` }, + limit: 1, + expectedIds: [1], + }) + + expect(loads[0]?.offset).toBe(0) + expect(loads[0]?.cursor).toBeUndefined() + }) + + it(`refines an initial locale window without trusting provider collation`, async () => { + const loads = await runAdversarialOrderedProviderScenario({ + // Lexical provider order disagrees with locale numeric order. + providerRows: [ + { id: 2, rank: 0, label: `item10` }, + { id: 1, rank: 0, label: `item2` }, + ], + order: { kind: `locale` }, + limit: 1, + expectedIds: [1], + useOffsetWhenAvailable: true, + }) + + expect(loads).toHaveLength(2) + expect(loads[1]?.limit).toBeUndefined() + expect(loads[1]?.offset).toBeUndefined() + expect(loads[1]?.cursor).toBeUndefined() + }) + + it(`refines an initial reference-ordered window locally`, async () => { + const first = { value: `first` } + const second = { value: `second` } + // Fix their runtime reference order before the provider returns the + // opposite prefix. + makeComparator({ direction: `asc`, nulls: `first` })(first, second) + + const loads = await runAdversarialOrderedProviderScenario({ + providerRows: [ + { id: 2, rank: second, label: `second` }, + { id: 1, rank: first, label: `first` }, + ], + order: { kind: `reference` }, + limit: 1, + expectedIds: [1], + useOffsetWhenAvailable: true, + }) + + expect(loads).toHaveLength(2) + expect(loads[1]?.limit).toBeUndefined() + expect(loads[1]?.offset).toBeUndefined() + }) + + it.each( + ([`asc`, `desc`] as const).flatMap((direction) => + ([`first`, `last`] as const).map((nulls) => ({ direction, nulls })), + ), + )( + `refines invalid Date ties with an unbounded local-order request ($direction, nulls $nulls)`, + async ({ direction, nulls }) => { + const invalid = new Date(Number.NaN) + const loads = await runAdversarialOrderedProviderScenario({ + providerRows: [ + { id: 2, rank: invalid, label: `second` }, + { id: 1, rank: invalid, label: `first` }, + ], + order: { kind: `reference`, direction, nulls }, + limit: 1, + expectedIds: [1], + useOffsetWhenAvailable: true, + }) + + expect(loads).toHaveLength(2) + expect(loads[1]?.limit).toBeUndefined() + expect(loads[1]?.offset).toBeUndefined() + }, + ) + + it(`uses an ascending index for a bounded descending demand`, async () => { + const rows: Array = [ + { id: 3, rank: 1 }, { id: 1, rank: 0 }, { id: 2, rank: 0 }, ] + const loads: Array = [] + const loaded = new Set() + let begin!: () => void + let write!: (message: { type: `insert`; value: PageRow }) => void + let commit!: () => void + const source = createCollection({ + id: `pagination-reversed-index-ties-${collectionSequence++}`, + getKey: (row: PageRow) => row.id, + syncMode: `on-demand`, + startSync: true, + sync: { + sync: (operations) => { + begin = operations.begin + write = operations.write + commit = operations.commit + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + begin() + for (const row of rowsForLoadSubset(rows, options)) { + if (loaded.has(row.id)) continue + loaded.add(row.id) + write({ type: `insert`, value: row }) + } + commit() + return true + }, + } + }, + }, + }) + source.createIndex((row) => row.rank, { + indexType: BTreeIndex, + options: { + compareOptions: { + direction: `asc`, + nulls: `first`, + stringSort: `locale`, + }, + }, + }) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `desc`) + .limit(2), + ) - await expectAssertionFailure(runPaginationStateScenario, { - checkpoint: 2, - classify: (difference) => - isPageRowArray(difference.actual) && - isPageRowArray(difference.expected) && - sameRows(difference.actual, defective) && - sameRows(difference.expected, expected), - })(scenario) + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([3, 1]) + expect(loads.length).toBeGreaterThan(0) + expect( + loads.every( + ({ limit, where }) => limit !== undefined || where !== undefined, + ), + JSON.stringify( + loads.map(({ limit, offset, cursor, orderBy, where }) => ({ + limit, + offset, + cursor: cursor !== undefined, + orderBy: orderBy !== undefined, + where: where !== undefined, + })), + ), + ).toBe(true) + } finally { + await cleanupAll(live, source) + } }) - it(`expands a multi-column boundary before choosing top-K`, async () => { - await expectAssertionFailure(expectMultiOrderBoundaryMatches, { - checkpoint: 0, - classify: ({ actual, expected }) => - Array.isArray(actual) && - actual.every((value) => typeof value === `number`) && - Array.isArray(expected) && - expected.every((value) => typeof value === `number`) && - actual.join(`,`) === `2,3,1,4` && - expected.join(`,`) === `2,3,1,5`, - })() + it.each([{ ids: [1, Number.NaN] }, { ids: [Number.NaN, 1] }])( + `keeps finite public keys before NaN across insertion order`, + async ({ ids }) => { + const source = createCollection( + mockSyncCollectionOptions({ + id: `pagination-nan-key-order-${collectionSequence++}`, + initialData: ids.map((id) => ({ id, rank: 0 })), + getKey: (row: PageRow) => row.id, + autoIndex: `eager`, + }), + ) + const live = createLiveQueryCollection((query) => + query + .from({ row: source }) + .orderBy(({ row }) => row.rank, `asc`) + .limit(1), + ) + + try { + await live.preload() + expect(Array.from(live.values(), ({ id }) => id)).toEqual([1]) + } finally { + await cleanupAll(live, source) + } + }, + ) + + it(`stabilizes an on-demand window with a NaN public-key tie`, async () => { + const loads = await runAdversarialOrderedProviderScenario({ + providerRows: [ + { id: 1, rank: 0, label: `finite` }, + { id: Number.NaN, rank: 0, label: `nan` }, + ], + order: { kind: `rank`, direction: `asc`, nulls: `first` }, + limit: 1, + expectedIds: [1], + }) + + expect(loads).toHaveLength(2) }) + it.each([ + { direction: `asc`, nulls: `first`, expectedIds: [1, 2] }, + { direction: `asc`, nulls: `last`, expectedIds: [2, 3] }, + { direction: `desc`, nulls: `first`, expectedIds: [1, 3] }, + { direction: `desc`, nulls: `last`, expectedIds: [3, 2] }, + ] as const)( + `keeps null placement and $direction across source refinement ($nulls)`, + async ({ direction, nulls, expectedIds }) => { + const providerRows = [ + { id: 1, rank: null, label: `null` }, + { id: 2, rank: 0, label: `zero` }, + { id: 3, rank: 1, label: `one` }, + ].sort( + (left, right) => + compareNullableNumber(left.rank, right.rank, { direction, nulls }) || + left.id - right.id, + ) + await runAdversarialOrderedProviderScenario({ + providerRows, + order: { kind: `rank`, direction, nulls }, + limit: 2, + expectedIds, + }) + }, + ) + fcTest.prop([scenarioArbitrary], { numRuns: transitionScenarioRuns, seed: 1659, })( `matches full recomputation when exact async cursor loads widen ordered coverage for a fixed seed`, - runOnDemandPaginationScenarioWithKnownFailures, + runOnDemandPaginationScenario, ) fcTest.prop( [scenarioArbitrary], - oracleRandomParameters(transitionScenarioRuns, replaySeed), + oracleRandomParameters( + transitionScenarioRuns, + replay, + `pagination.async-cursor`, + ), )( `matches full recomputation when exact async cursor loads widen ordered coverage for a random or replayed seed`, - runOnDemandPaginationScenarioWithKnownFailures, + runOnDemandPaginationScenario, ) it.each([`forward`, `reverse`] as const)( diff --git a/packages/db/tests/query/predicate-utils.test.ts b/packages/db/tests/query/predicate-utils.test.ts deleted file mode 100644 index 6471950dee..0000000000 --- a/packages/db/tests/query/predicate-utils.test.ts +++ /dev/null @@ -1,1598 +0,0 @@ -import { describe, expect, it } from 'vitest' -import { - isLimitSubset, - isLoadSubsetRequestSubsumedBy, - isOffsetLimitSubset, - isOrderBySubset, - isPredicateSubset, - isWhereSubset, - minusWherePredicates, - unionWherePredicates, -} from '../../src/query/predicate-utils' -import { Func, PropRef, Value } from '../../src/query/ir' -import type { - BasicExpression, - OrderBy, - OrderByClause, -} from '../../src/query/ir' -import type { LoadSubsetOptions } from '../../src/types' - -// Helper functions to build expressions more easily -function ref(path: string | Array): PropRef { - return new PropRef(typeof path === `string` ? [path] : path) -} - -function val(value: any): Value { - return new Value(value) -} - -function func(name: string, ...args: Array): Func { - return new Func(name, args) -} - -function eq(left: BasicExpression, right: BasicExpression): Func { - return func(`eq`, left, right) -} - -function gt(left: BasicExpression, right: BasicExpression): Func { - return func(`gt`, left, right) -} - -function gte(left: BasicExpression, right: BasicExpression): Func { - return func(`gte`, left, right) -} - -function lt(left: BasicExpression, right: BasicExpression): Func { - return func(`lt`, left, right) -} - -function lte(left: BasicExpression, right: BasicExpression): Func { - return func(`lte`, left, right) -} - -function and(...args: Array): Func { - return func(`and`, ...args) -} - -function or(...args: Array): Func { - return func(`or`, ...args) -} - -function inOp(left: BasicExpression, values: Array): Func { - return func(`in`, left, val(values)) -} - -function orderByClause( - expression: BasicExpression, - direction: `asc` | `desc` = `asc`, -): OrderByClause { - return { - expression, - compareOptions: { - direction, - nulls: `last`, - stringSort: `lexical`, - }, - } -} - -describe(`isWhereSubset`, () => { - describe(`basic cases`, () => { - it(`should return true for both undefined (all data is subset of all data)`, () => { - expect(isWhereSubset(undefined, undefined)).toBe(true) - }) - - it(`should return false for undefined subset with constrained superset`, () => { - // Requesting ALL data but only loaded SOME data = NOT subset - expect(isWhereSubset(undefined, gt(ref(`age`), val(10)))).toBe(false) - }) - - it(`should return true for constrained subset with undefined superset`, () => { - // Loaded ALL data, so any constrained subset is covered - expect(isWhereSubset(gt(ref(`age`), val(20)), undefined)).toBe(true) - }) - - it(`should return true for identical expressions`, () => { - const expr = gt(ref(`age`), val(10)) - expect(isWhereSubset(expr, expr)).toBe(true) - }) - - it(`should return true for structurally equal expressions`, () => { - expect( - isWhereSubset(gt(ref(`age`), val(10)), gt(ref(`age`), val(10))), - ).toBe(true) - }) - - it(`should return true when subset is false`, () => { - // When subset is false the result will always be the empty set - // and the empty set is a subset of any set - expect(isWhereSubset(val(false), gt(ref(`age`), val(10)))).toBe(true) - }) - }) - - describe(`comparison operators`, () => { - it(`should handle gt: age > 20 is subset of age > 10`, () => { - expect( - isWhereSubset(gt(ref(`age`), val(20)), gt(ref(`age`), val(10))), - ).toBe(true) - }) - - it(`should handle gt: age > 10 is NOT subset of age > 20`, () => { - expect( - isWhereSubset(gt(ref(`age`), val(10)), gt(ref(`age`), val(20))), - ).toBe(false) - }) - - it(`should handle gte: age >= 20 is subset of age >= 10`, () => { - expect( - isWhereSubset(gte(ref(`age`), val(20)), gte(ref(`age`), val(10))), - ).toBe(true) - }) - - it(`should handle lt: age < 10 is subset of age < 20`, () => { - expect( - isWhereSubset(lt(ref(`age`), val(10)), lt(ref(`age`), val(20))), - ).toBe(true) - }) - - it(`should handle lt: age < 20 is NOT subset of age < 10`, () => { - expect( - isWhereSubset(lt(ref(`age`), val(20)), lt(ref(`age`), val(10))), - ).toBe(false) - }) - - it(`should handle lte: age <= 10 is subset of age <= 20`, () => { - expect( - isWhereSubset(lte(ref(`age`), val(10)), lte(ref(`age`), val(20))), - ).toBe(true) - }) - - it(`should handle eq: age = 15 is subset of age > 10`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(15)), gt(ref(`age`), val(10))), - ).toBe(true) - }) - - it(`should handle eq: age = 5 is NOT subset of age > 10`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(5)), gt(ref(`age`), val(10))), - ).toBe(false) - }) - - it(`should handle eq: age = 15 is subset of age >= 15`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(15)), gte(ref(`age`), val(15))), - ).toBe(true) - }) - - it(`should handle eq: age = 15 is subset of age < 20`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(15)), lt(ref(`age`), val(20))), - ).toBe(true) - }) - - it(`should handle mixed operators: gt vs gte`, () => { - expect( - isWhereSubset(gt(ref(`age`), val(10)), gte(ref(`age`), val(10))), - ).toBe(true) - }) - - it(`should handle mixed operators: gte vs gt`, () => { - expect( - isWhereSubset(gte(ref(`age`), val(11)), gt(ref(`age`), val(10))), - ).toBe(true) - expect( - isWhereSubset(gte(ref(`age`), val(10)), gt(ref(`age`), val(10))), - ).toBe(false) - }) - }) - - describe(`IN operator`, () => { - it(`should handle eq vs in: age = 5 is subset of age IN [5, 10, 15]`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(5)), inOp(ref(`age`), [5, 10, 15])), - ).toBe(true) - }) - - it(`should handle eq vs in: age = 20 is NOT subset of age IN [5, 10, 15]`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(20)), inOp(ref(`age`), [5, 10, 15])), - ).toBe(false) - }) - - it(`should handle in vs in: [5, 10] is subset of [5, 10, 15]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [5, 10]), inOp(ref(`age`), [5, 10, 15])), - ).toBe(true) - }) - - it(`should handle in vs in: [5, 20] is NOT subset of [5, 10, 15]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [5, 20]), inOp(ref(`age`), [5, 10, 15])), - ).toBe(false) - }) - - it(`should handle empty IN array: age IN [] is subset of age IN []`, () => { - expect(isWhereSubset(inOp(ref(`age`), []), inOp(ref(`age`), []))).toBe( - true, - ) - }) - - it(`should handle empty IN array: age IN [] is subset of age IN [5, 10]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), []), inOp(ref(`age`), [5, 10])), - ).toBe(true) - }) - - it(`should handle empty IN array: age IN [5, 10] is NOT subset of age IN []`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [5, 10]), inOp(ref(`age`), [])), - ).toBe(false) - }) - - it(`should handle singleton IN array: age = 5 is subset of age IN [5]`, () => { - expect(isWhereSubset(eq(ref(`age`), val(5)), inOp(ref(`age`), [5]))).toBe( - true, - ) - }) - - it(`should handle singleton IN array: age = 10 is NOT subset of age IN [5]`, () => { - expect( - isWhereSubset(eq(ref(`age`), val(10)), inOp(ref(`age`), [5])), - ).toBe(false) - }) - - it(`should handle singleton IN array: age IN [5] is subset of age IN [5, 10, 15]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [5]), inOp(ref(`age`), [5, 10, 15])), - ).toBe(true) - }) - - it(`should handle singleton IN array: age IN [20] is NOT subset of age IN [5, 10, 15]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [20]), inOp(ref(`age`), [5, 10, 15])), - ).toBe(false) - }) - - it(`should handle singleton IN array: age IN [5, 10, 15] is NOT subset of age IN [5]`, () => { - expect( - isWhereSubset(inOp(ref(`age`), [5, 10, 15]), inOp(ref(`age`), [5])), - ).toBe(false) - }) - }) - - describe(`AND combinations`, () => { - it(`should handle AND in subset: (A AND B) is subset of A`, () => { - expect( - isWhereSubset( - and(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - gt(ref(`age`), val(10)), - ), - ).toBe(true) - }) - - it(`should handle AND in subset: (A AND B) is NOT subset of C (different field)`, () => { - expect( - isWhereSubset( - and(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - eq(ref(`name`), val(`John`)), - ), - ).toBe(false) - }) - - it(`should handle AND in superset: A is subset of (A AND B) is false (superset is more restrictive)`, () => { - expect( - isWhereSubset( - gt(ref(`age`), val(10)), - and(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - ), - ).toBe(false) - }) - - it(`should handle AND in both: (age > 20 AND status = 'active') is subset of (age > 10 AND status = 'active')`, () => { - expect( - isWhereSubset( - and(gt(ref(`age`), val(20)), eq(ref(`status`), val(`active`))), - and(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - ), - ).toBe(true) - }) - }) - - describe(`OR combinations`, () => { - it(`should handle OR in superset: A is subset of (A OR B)`, () => { - expect( - isWhereSubset( - gt(ref(`age`), val(10)), - or(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - ), - ).toBe(true) - }) - - it(`should return false when subset doesn't imply any branch of OR superset`, () => { - expect( - isWhereSubset( - eq(ref(`age`), val(10)), - or(gt(ref(`age`), val(10)), lt(ref(`age`), val(5))), - ), - ).toBe(false) - }) - - it(`should handle OR in subset: (A OR B) is subset of C only if both A and B are subsets of C`, () => { - expect( - isWhereSubset( - or(gt(ref(`age`), val(20)), gt(ref(`age`), val(30))), - gt(ref(`age`), val(10)), - ), - ).toBe(true) - }) - - it(`should handle OR in both: (age > 20 OR status = 'active') is subset of (age > 10 OR status = 'active')`, () => { - expect( - isWhereSubset( - or(gt(ref(`age`), val(20)), eq(ref(`status`), val(`active`))), - or(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - ), - ).toBe(true) - }) - - it(`should handle OR in subset: (A OR B) is NOT subset of C if either is not a subset`, () => { - expect( - isWhereSubset( - or(gt(ref(`age`), val(20)), lt(ref(`age`), val(5))), - gt(ref(`age`), val(10)), - ), - ).toBe(false) - }) - }) - - describe(`AND subset with OR superset`, () => { - it(`should recognize and(eq, isNull) as subset of or(and(eq, isNull), and(eq, isNull))`, () => { - const projectX = `4e164373-31b4-4b42-95c9-9c395cfb4916` - const projectY = `2fd4c147-2547-4b02-9554-9cd067187409` - - const queryX = and( - eq(ref(`project_id`), val(projectX)), - func(`isNull`, ref(`soft_deleted_at`)), - ) - const queryY = and( - eq(ref(`project_id`), val(projectY)), - func(`isNull`, ref(`soft_deleted_at`)), - ) - - const unionPredicate = or(queryX, queryY) - - expect(isWhereSubset(queryX, unionPredicate)).toBe(true) - expect(isWhereSubset(queryY, unionPredicate)).toBe(true) - }) - - it(`should recognize and(A, B) as subset of or(and(A, B), and(C, D))`, () => { - const subsetExpr = and(eq(ref(`id`), val(1)), gt(ref(`age`), val(20))) - const supersetExpr = or( - and(eq(ref(`id`), val(1)), gt(ref(`age`), val(20))), - and(eq(ref(`id`), val(2)), gt(ref(`age`), val(30))), - ) - expect(isWhereSubset(subsetExpr, supersetExpr)).toBe(true) - }) - - it(`should return false when and(A, B) matches no disjunct`, () => { - const subsetExpr = and(eq(ref(`id`), val(3)), gt(ref(`age`), val(20))) - const supersetExpr = or( - and(eq(ref(`id`), val(1)), gt(ref(`age`), val(20))), - and(eq(ref(`id`), val(2)), gt(ref(`age`), val(30))), - ) - expect(isWhereSubset(subsetExpr, supersetExpr)).toBe(false) - }) - }) - - describe(`isNull predicates`, () => { - it(`should return true for identical isNull expressions`, () => { - const a = func(`isNull`, ref(`deleted_at`)) - const b = func(`isNull`, ref(`deleted_at`)) - expect(isWhereSubset(a, b)).toBe(true) - }) - - it(`should return false for isNull on different fields`, () => { - const a = func(`isNull`, ref(`deleted_at`)) - const b = func(`isNull`, ref(`created_at`)) - expect(isWhereSubset(a, b)).toBe(false) - }) - - it(`should return true for and(eq, isNull) subset of identical and(eq, isNull)`, () => { - const subset = and( - eq(ref(`project_id`), val(`abc`)), - func(`isNull`, ref(`soft_deleted_at`)), - ) - const superset = and( - eq(ref(`project_id`), val(`abc`)), - func(`isNull`, ref(`soft_deleted_at`)), - ) - expect(isWhereSubset(subset, superset)).toBe(true) - }) - }) - - describe(`different fields`, () => { - it(`should return false for different fields with no relationship`, () => { - expect( - isWhereSubset(gt(ref(`age`), val(20)), gt(ref(`salary`), val(1000))), - ).toBe(false) - }) - }) - - describe(`Date support`, () => { - const date1 = new Date(`2024-01-01`) - const date2 = new Date(`2024-01-15`) - const date3 = new Date(`2024-02-01`) - - it(`should handle Date equality`, () => { - expect( - isWhereSubset( - eq(ref(`createdAt`), val(date2)), - eq(ref(`createdAt`), val(date2)), - ), - ).toBe(true) - }) - - it(`should handle Date range comparisons: date > 2024-01-15 is subset of date > 2024-01-01`, () => { - expect( - isWhereSubset( - gt(ref(`createdAt`), val(date2)), - gt(ref(`createdAt`), val(date1)), - ), - ).toBe(true) - }) - - it(`should handle Date range comparisons: date < 2024-01-15 is subset of date < 2024-02-01`, () => { - expect( - isWhereSubset( - lt(ref(`createdAt`), val(date2)), - lt(ref(`createdAt`), val(date3)), - ), - ).toBe(true) - }) - - it(`should handle Date equality vs range: date = 2024-01-15 is subset of date > 2024-01-01`, () => { - expect( - isWhereSubset( - eq(ref(`createdAt`), val(date2)), - gt(ref(`createdAt`), val(date1)), - ), - ).toBe(true) - }) - - it(`should handle Date equality vs IN: date = 2024-01-15 is subset of date IN [2024-01-01, 2024-01-15, 2024-02-01]`, () => { - expect( - isWhereSubset( - eq(ref(`createdAt`), val(date2)), - inOp(ref(`createdAt`), [date1, date2, date3]), - ), - ).toBe(true) - }) - - it(`should handle Date IN subset: date IN [2024-01-01, 2024-01-15] is subset of date IN [2024-01-01, 2024-01-15, 2024-02-01]`, () => { - expect( - isWhereSubset( - inOp(ref(`createdAt`), [date1, date2]), - inOp(ref(`createdAt`), [date1, date2, date3]), - ), - ).toBe(true) - }) - - it(`should return false when Date not in IN set`, () => { - expect( - isWhereSubset( - eq(ref(`createdAt`), val(date1)), - inOp(ref(`createdAt`), [date2, date3]), - ), - ).toBe(false) - }) - }) -}) - -describe(`unionWherePredicates`, () => { - describe(`basic cases`, () => { - it(`should return false for empty array`, () => { - const result = unionWherePredicates([]) - expect(result.type).toBe(`val`) - expect((result as Value).value).toBe(false) - }) - - it(`should return the single predicate as-is`, () => { - const pred = gt(ref(`age`), val(10)) - const result = unionWherePredicates([pred]) - expect(result).toBe(pred) - }) - }) - - describe(`same field comparisons`, () => { - it(`should take least restrictive for gt: age > 10 OR age > 20 → age > 10`, () => { - const result = unionWherePredicates([ - gt(ref(`age`), val(10)), - gt(ref(`age`), val(20)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`gt`) - const field = (result as Func).args[1] as Value - expect(field.value).toBe(10) - }) - - it(`should take least restrictive for gte: age >= 10 OR age >= 20 → age >= 10`, () => { - const result = unionWherePredicates([ - gte(ref(`age`), val(10)), - gte(ref(`age`), val(20)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`gte`) - const field = (result as Func).args[1] as Value - expect(field.value).toBe(10) - }) - - it(`should take least restrictive for lt: age < 20 OR age < 10 → age < 20`, () => { - const result = unionWherePredicates([ - lt(ref(`age`), val(20)), - lt(ref(`age`), val(10)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`lt`) - const field = (result as Func).args[1] as Value - expect(field.value).toBe(20) - }) - - it(`should combine eq into IN: age = 5 OR age = 10 → age IN [5, 10]`, () => { - const result = unionWherePredicates([ - eq(ref(`age`), val(5)), - eq(ref(`age`), val(10)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`in`) - const values = ((result as Func).args[1] as Value).value - expect(values).toContain(5) - expect(values).toContain(10) - expect(values.length).toBe(2) - }) - - it(`should fold IN and equality into single IN: age IN [1,2] OR age = 3 → age IN [1,2,3]`, () => { - const result = unionWherePredicates([ - inOp(ref(`age`), [1, 2]), - eq(ref(`age`), val(3)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`in`) - const values = ((result as Func).args[1] as Value).value - expect(values).toContain(1) - expect(values).toContain(2) - expect(values).toContain(3) - expect(values.length).toBe(3) - }) - - it(`should handle gte and gt together: age > 10 OR age >= 15 → age > 10`, () => { - const result = unionWherePredicates([ - gt(ref(`age`), val(10)), - gte(ref(`age`), val(15)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`gt`) - const field = (result as Func).args[1] as Value - expect(field.value).toBe(10) - }) - }) - - describe(`different fields`, () => { - it(`should combine with OR: age > 10 OR status = 'active'`, () => { - const result = unionWherePredicates([ - gt(ref(`age`), val(10)), - eq(ref(`status`), val(`active`)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`or`) - expect((result as Func).args.length).toBe(2) - }) - }) - - describe(`flatten OR`, () => { - it(`should flatten nested ORs`, () => { - const result = unionWherePredicates([ - or(gt(ref(`age`), val(10)), eq(ref(`status`), val(`active`))), - eq(ref(`name`), val(`John`)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`or`) - expect((result as Func).args.length).toBe(3) - }) - }) - - describe(`Date support`, () => { - const date1 = new Date(`2024-01-01`) - const date2 = new Date(`2024-01-15`) - const date3 = new Date(`2024-02-01`) - - it(`should combine Date equalities into IN: date = date1 OR date = date2 → date IN [date1, date2]`, () => { - const result = unionWherePredicates([ - eq(ref(`createdAt`), val(date1)), - eq(ref(`createdAt`), val(date2)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`in`) - const values = ((result as Func).args[1] as Value).value - expect(values.length).toBe(2) - expect(values).toContainEqual(date1) - expect(values).toContainEqual(date2) - }) - - it(`should fold Date IN and equality: date IN [date1,date2] OR date = date3 → date IN [date1,date2,date3]`, () => { - const result = unionWherePredicates([ - inOp(ref(`createdAt`), [date1, date2]), - eq(ref(`createdAt`), val(date3)), - ]) - expect(result.type).toBe(`func`) - expect((result as Func).name).toBe(`in`) - const values = ((result as Func).args[1] as Value).value - expect(values.length).toBe(3) - expect(values).toContainEqual(date1) - expect(values).toContainEqual(date2) - expect(values).toContainEqual(date3) - }) - }) -}) - -describe(`isOrderBySubset`, () => { - it(`should return true for undefined subset`, () => { - const orderBy: OrderBy = [orderByClause(ref(`age`), `asc`)] - expect(isOrderBySubset(undefined, orderBy)).toBe(true) - expect(isOrderBySubset([], orderBy)).toBe(true) - }) - - it(`should return false for undefined superset with non-empty subset`, () => { - const orderBy: OrderBy = [orderByClause(ref(`age`), `asc`)] - expect(isOrderBySubset(orderBy, undefined)).toBe(false) - expect(isOrderBySubset(orderBy, [])).toBe(false) - }) - - it(`should return true for identical orderBy`, () => { - const orderBy: OrderBy = [orderByClause(ref(`age`), `asc`)] - expect(isOrderBySubset(orderBy, orderBy)).toBe(true) - }) - - it(`should return true when subset is prefix of superset`, () => { - const subset: OrderBy = [orderByClause(ref(`age`), `asc`)] - const superset: OrderBy = [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - ] - expect(isOrderBySubset(subset, superset)).toBe(true) - }) - - it(`should return false when subset is not a prefix`, () => { - const subset: OrderBy = [orderByClause(ref(`name`), `desc`)] - const superset: OrderBy = [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - ] - expect(isOrderBySubset(subset, superset)).toBe(false) - }) - - it(`should return false when directions differ`, () => { - const subset: OrderBy = [orderByClause(ref(`age`), `desc`)] - const superset: OrderBy = [orderByClause(ref(`age`), `asc`)] - expect(isOrderBySubset(subset, superset)).toBe(false) - }) - - it.each([ - [ - `null placement`, - { direction: `asc`, nulls: `first`, stringSort: `lexical` } as const, - { direction: `asc`, nulls: `last`, stringSort: `lexical` } as const, - ], - [ - `string sort mode`, - { direction: `asc`, nulls: `last`, stringSort: `lexical` } as const, - { direction: `asc`, nulls: `last`, stringSort: `locale` } as const, - ], - [ - `locale`, - { - direction: `asc`, - nulls: `last`, - stringSort: `locale`, - locale: `en-US`, - } as const, - { - direction: `asc`, - nulls: `last`, - stringSort: `locale`, - locale: `de-DE`, - } as const, - ], - [ - `locale options`, - { - direction: `asc`, - nulls: `last`, - stringSort: `locale`, - locale: `en-US`, - localeOptions: { numeric: true, sensitivity: `base` }, - } as const, - { - direction: `asc`, - nulls: `last`, - stringSort: `locale`, - locale: `en-US`, - localeOptions: { numeric: false, sensitivity: `base` }, - } as const, - ], - ])(`should return false when %s differs`, (_label, first, second) => { - const expression = ref(`name`) - expect( - isOrderBySubset( - [{ expression, compareOptions: first }], - [{ expression, compareOptions: second }], - ), - ).toBe(false) - expect( - isLoadSubsetRequestSubsumedBy( - { - orderBy: [{ expression, compareOptions: first }], - limit: 10, - }, - { - orderBy: [{ expression, compareOptions: second }], - limit: 20, - }, - ), - ).toBe(false) - }) - - it(`should return false when subset is longer than superset`, () => { - const subset: OrderBy = [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - orderByClause(ref(`status`), `asc`), - ] - const superset: OrderBy = [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - ] - expect(isOrderBySubset(subset, superset)).toBe(false) - }) -}) - -describe(`isLimitSubset`, () => { - it(`should return false for undefined subset with limited superset (requesting all data but only have limited)`, () => { - expect(isLimitSubset(undefined, 10)).toBe(false) - }) - - it(`should return true for undefined subset with undefined superset (requesting all data and have all data)`, () => { - expect(isLimitSubset(undefined, undefined)).toBe(true) - }) - - it(`should return true for undefined superset`, () => { - expect(isLimitSubset(10, undefined)).toBe(true) - }) - - it(`should return true when subset <= superset`, () => { - expect(isLimitSubset(10, 20)).toBe(true) - expect(isLimitSubset(10, 10)).toBe(true) - }) - - it(`should return false when subset > superset`, () => { - expect(isLimitSubset(20, 10)).toBe(false) - }) -}) - -describe(`isOffsetLimitSubset`, () => { - it(`should return true when subset range is within superset range (same offset)`, () => { - expect( - isOffsetLimitSubset({ offset: 0, limit: 5 }, { offset: 0, limit: 10 }), - ).toBe(true) - expect( - isOffsetLimitSubset({ offset: 0, limit: 10 }, { offset: 0, limit: 10 }), - ).toBe(true) - }) - - it(`should return true when subset starts later but is still within superset range`, () => { - // superset loads rows [0, 10), subset loads rows [5, 10) - subset is within superset - expect( - isOffsetLimitSubset({ offset: 5, limit: 5 }, { offset: 0, limit: 10 }), - ).toBe(true) - }) - - it(`should return false when subset extends beyond superset range`, () => { - // superset loads rows [0, 10), subset loads rows [5, 15) - subset extends beyond - expect( - isOffsetLimitSubset({ offset: 5, limit: 10 }, { offset: 0, limit: 10 }), - ).toBe(false) - }) - - it(`should return false when subset is completely outside superset range`, () => { - // superset loads rows [0, 10), subset loads rows [20, 30) - no overlap - expect( - isOffsetLimitSubset({ offset: 20, limit: 10 }, { offset: 0, limit: 10 }), - ).toBe(false) - }) - - it(`should return false when superset starts after subset`, () => { - // superset loads rows [10, 20), subset loads rows [0, 10) - superset starts too late - expect( - isOffsetLimitSubset({ offset: 0, limit: 10 }, { offset: 10, limit: 10 }), - ).toBe(false) - }) - - it(`should return true when superset is unlimited`, () => { - expect(isOffsetLimitSubset({ offset: 0, limit: 10 }, { offset: 0 })).toBe( - true, - ) - expect(isOffsetLimitSubset({ offset: 20, limit: 10 }, { offset: 0 })).toBe( - true, - ) - }) - - it(`should return false when superset is unlimited but starts after subset`, () => { - // superset loads rows [10, ∞), subset loads rows [0, 10) - superset starts too late - expect(isOffsetLimitSubset({ offset: 0, limit: 10 }, { offset: 10 })).toBe( - false, - ) - }) - - it(`should return false when subset is unlimited but superset has a limit`, () => { - expect(isOffsetLimitSubset({ offset: 0 }, { offset: 0, limit: 10 })).toBe( - false, - ) - }) - - it(`should return true when both are unlimited and superset starts at or before subset`, () => { - expect(isOffsetLimitSubset({ offset: 10 }, { offset: 0 })).toBe(true) - expect(isOffsetLimitSubset({ offset: 10 }, { offset: 10 })).toBe(true) - }) - - it(`should return false when both are unlimited but superset starts after subset`, () => { - expect(isOffsetLimitSubset({ offset: 0 }, { offset: 10 })).toBe(false) - }) - - it(`should default offset to 0 when undefined`, () => { - expect(isOffsetLimitSubset({ limit: 5 }, { limit: 10 })).toBe(true) - expect(isOffsetLimitSubset({ offset: 0, limit: 5 }, { limit: 10 })).toBe( - true, - ) - expect(isOffsetLimitSubset({ limit: 5 }, { offset: 0, limit: 10 })).toBe( - true, - ) - }) -}) - -describe(`isPredicateSubset`, () => { - it(`should check all components for unlimited superset`, () => { - // For unlimited supersets, where-subset logic applies - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(20)), - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 10, - } - const superset: LoadSubsetOptions = { - where: gt(ref(`age`), val(10)), - orderBy: [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - ], - // No limit - unlimited superset - } - expect(isPredicateSubset(subset, superset)).toBe(true) - }) - - it(`should require equal where clauses for limited supersets`, () => { - // For limited supersets, where clauses must be EQUAL - const sameWhere = gt(ref(`age`), val(10)) - - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 5, - } - const superset: LoadSubsetOptions = { - where: sameWhere, // Same where clause - orderBy: [ - orderByClause(ref(`age`), `asc`), - orderByClause(ref(`name`), `desc`), - ], - limit: 20, - } - expect(isPredicateSubset(subset, superset)).toBe(true) - }) - - it(`treats semantic predicate forms as equal coverage`, () => { - const age = ref(`age`) - const status = ref(`status`) - const ageCheck = gt(age, val(18)) - const statusCheck = eq(status, val(`active`)) - const subset: LoadSubsetOptions = { - where: func(`and`, ageCheck, statusCheck), - limit: 10, - } - const superset: LoadSubsetOptions = { - where: func(`and`, eq(val(`active`), status), func(`lt`, val(18), age)), - limit: 20, - } - - expect(isPredicateSubset(subset, superset)).toBe(true) - }) - - it(`does not normalize distinct comparison operators at a limited boundary`, () => { - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(18)), - limit: 10, - } - const superset: LoadSubsetOptions = { - where: gte(ref(`age`), val(18)), - limit: 20, - } - - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`requires equal predicates for a cursor-relative superset`, () => { - const cursor = { - whereFrom: gt(ref(`id`), val(10)), - whereCurrent: eq(ref(`id`), val(10)), - lastKey: 10, - } - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(18)), - cursor, - } - const superset: LoadSubsetOptions = { - where: gte(ref(`age`), val(18)), - cursor, - } - - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`does not retain expression hashes across comparison operations`, () => { - const subset = gt(ref(`age`), val(18)) - const superset = gt(ref(`age`), val(18)) - - expect(isWhereSubset(subset, superset)).toBe(true) - superset.name = `lt` - expect(isWhereSubset(subset, superset)).toBe(false) - }) - - it(`hashes a repeated expression once per subset comparison`, () => { - let valueReads = 0 - const countedValue = val(1) - Object.defineProperty(countedValue, `value`, { - configurable: true, - get: () => { - valueReads++ - return 1 - }, - }) - const subset = func(`custom-subset`, countedValue) - const superset = func( - `or`, - ...Array.from({ length: 4 }, (_, index) => - func(`custom-superset-${index}`, val(index)), - ), - ) - - expect(isWhereSubset(subset, superset)).toBe(false) - expect(valueReads).toBe(1) - }) - - it(`should return false for limited superset with different where clause`, () => { - // Even if subset's where is more restrictive, it can't be a subset - // of a limited superset with a different where clause. - // The top N items of "age > 20" may not be in the top M items of "age > 10" - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(20)), // More restrictive - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 5, - } - const superset: LoadSubsetOptions = { - where: gt(ref(`age`), val(10)), // Less restrictive but LIMITED - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 20, - } - // This should be FALSE because the top 5 of "age > 20" - // might include items outside the top 20 of "age > 10" - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return false for limited superset with no where vs subset with where`, () => { - // This is the reported bug case: pagination with search filter - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(20)), // Has a filter - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 10, - } - const superset: LoadSubsetOptions = { - where: undefined, // No filter but LIMITED - orderBy: [orderByClause(ref(`age`), `asc`)], - limit: 10, - } - // The filtered results might include items outside the unfiltered top 10 - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return false if where is not subset`, () => { - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(5)), - limit: 10, - } - const superset: LoadSubsetOptions = { - where: gt(ref(`age`), val(10)), - limit: 20, - } - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return false if orderBy is not subset`, () => { - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(20)), - orderBy: [orderByClause(ref(`name`), `desc`)], - } - const superset: LoadSubsetOptions = { - where: gt(ref(`age`), val(10)), - orderBy: [orderByClause(ref(`age`), `asc`)], - } - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return false if limit is not subset`, () => { - const subset: LoadSubsetOptions = { - where: gt(ref(`age`), val(20)), - limit: 30, - } - const superset: LoadSubsetOptions = { - where: gt(ref(`age`), val(10)), - limit: 20, - } - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - describe(`with offset`, () => { - it(`should return true when subset offset+limit is within superset range`, () => { - const sameWhere = gt(ref(`age`), val(10)) - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 5, - limit: 5, - } - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 10, - } - // subset loads rows [5, 10), superset loads rows [0, 10) - subset is within - expect(isPredicateSubset(subset, superset)).toBe(true) - }) - - it(`should return false when subset is at different offset outside superset range`, () => { - const sameWhere = gt(ref(`age`), val(10)) - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 20, - limit: 10, - } - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 10, - } - // subset loads rows [20, 30), superset loads rows [0, 10) - no overlap - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return false when subset extends beyond superset even with same where`, () => { - const sameWhere = gt(ref(`age`), val(10)) - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 5, - limit: 10, - } - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 10, - } - // subset loads rows [5, 15), superset loads rows [0, 10) - subset extends beyond - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should return true for unlimited superset with any subset offset`, () => { - const sameWhere = gt(ref(`age`), val(10)) - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 100, - limit: 10, - } - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - // No limit - unlimited - } - expect(isPredicateSubset(subset, superset)).toBe(true) - }) - - it(`should return false when superset has offset that starts after subset needs`, () => { - const sameWhere = gt(ref(`age`), val(10)) - const subset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 10, - } - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 5, - limit: 10, - } - // subset needs rows [0, 10), superset only has rows [5, 15) - expect(isPredicateSubset(subset, superset)).toBe(false) - }) - - it(`should handle pagination correctly - page 2 not subset of page 1`, () => { - const sameWhere = gt(ref(`age`), val(10)) - // Page 1: offset 0, limit 10 - const page1: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 10, - } - // Page 2: offset 10, limit 10 - const page2: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 10, - limit: 10, - } - // Page 2 is NOT a subset of page 1 (different rows) - expect(isPredicateSubset(page2, page1)).toBe(false) - // Page 1 is NOT a subset of page 2 (different rows) - expect(isPredicateSubset(page1, page2)).toBe(false) - }) - - it(`should return true when superset covers multiple pages`, () => { - const sameWhere = gt(ref(`age`), val(10)) - // Superset: offset 0, limit 30 (covers pages 1-3) - const superset: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 0, - limit: 30, - } - // Page 2: offset 10, limit 10 - const page2: LoadSubsetOptions = { - where: sameWhere, - orderBy: [orderByClause(ref(`age`), `asc`)], - offset: 10, - limit: 10, - } - // Page 2 IS a subset of superset (rows 10-19 within 0-29) - expect(isPredicateSubset(page2, superset)).toBe(true) - }) - }) -}) - -describe(`minusWherePredicates`, () => { - describe(`basic cases`, () => { - it(`should return original predicate when nothing to subtract`, () => { - const pred = gt(ref(`age`), val(10)) - const result = minusWherePredicates(pred, undefined) - - expect(result).toEqual(pred) - }) - - it(`should return null when from is undefined (can't simplify NOT(B))`, () => { - const subtract = gt(ref(`age`), val(10)) - const result = minusWherePredicates(undefined, subtract) - - expect(result).toEqual({ - type: `func`, - name: `not`, - args: [subtract], - }) - }) - - it(`should return empty set when from is subset of subtract`, () => { - const from = gt(ref(`age`), val(20)) // age > 20 - const subtract = gt(ref(`age`), val(10)) // age > 10 - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ type: `val`, value: false }) - }) - - it(`should return null when predicates are on different fields`, () => { - const from = gt(ref(`age`), val(10)) - const subtract = eq(ref(`status`), val(`active`)) - const result = minusWherePredicates(from, subtract) - - expect(result).toBeNull() - }) - }) - - describe(`IN minus IN`, () => { - it(`should compute set difference: IN [A,B,C,D] - IN [B,C] = IN [A,D]`, () => { - const from = inOp(ref(`status`), [`A`, `B`, `C`, `D`]) - const subtract = inOp(ref(`status`), [`B`, `C`]) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `in`, - args: [ref(`status`), val([`A`, `D`])], - }) - }) - - it(`should return empty set when all values are subtracted`, () => { - const from = inOp(ref(`status`), [`A`, `B`]) - const subtract = inOp(ref(`status`), [`A`, `B`]) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ type: `val`, value: false }) - }) - - it(`should return original when no overlap`, () => { - const from = inOp(ref(`status`), [`A`, `B`]) - const subtract = inOp(ref(`status`), [`C`, `D`]) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual(from) - }) - - it(`should collapse to equality when one value remains`, () => { - const from = inOp(ref(`status`), [`A`, `B`]) - const subtract = inOp(ref(`status`), [`B`]) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `eq`, - args: [ref(`status`), val(`A`)], - }) - }) - }) - - describe(`IN minus equality`, () => { - it(`should remove value from IN: IN [A,B,C] - eq(B) = IN [A,C]`, () => { - const from = inOp(ref(`status`), [`A`, `B`, `C`]) - const subtract = eq(ref(`status`), val(`B`)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `in`, - args: [ref(`status`), val([`A`, `C`])], - }) - }) - - it(`should collapse to equality when one value remains`, () => { - const from = inOp(ref(`status`), [`A`, `B`]) - const subtract = eq(ref(`status`), val(`A`)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `eq`, - args: [ref(`status`), val(`B`)], - }) - }) - - it(`should return empty set when removing last value`, () => { - const from = inOp(ref(`status`), [`A`]) - const subtract = eq(ref(`status`), val(`A`)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ type: `val`, value: false }) - }) - }) - - describe(`equality minus equality`, () => { - it(`should return empty set when same value`, () => { - const from = eq(ref(`age`), val(15)) - const subtract = eq(ref(`age`), val(15)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ type: `val`, value: false }) - }) - - it(`should return original when different values`, () => { - const from = eq(ref(`age`), val(15)) - const subtract = eq(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual(from) - }) - }) - - describe(`range minus range - gt/gte`, () => { - it(`should compute difference: age > 10 - age > 20 = (age > 10 AND age <= 20)`, () => { - const from = gt(ref(`age`), val(10)) - const subtract = gt(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gt(ref(`age`), val(10)), lte(ref(`age`), val(20))], - }) - }) - - it(`should return original when no overlap: age > 20 - age > 10`, () => { - const from = gt(ref(`age`), val(20)) - const subtract = gt(ref(`age`), val(10)) - const result = minusWherePredicates(from, subtract) - - // age > 20 is subset of age > 10, so result is empty - expect(result).toEqual({ type: `val`, value: false }) - }) - - it(`should compute difference: age >= 10 - age >= 20 = (age >= 10 AND age < 20)`, () => { - const from = gte(ref(`age`), val(10)) - const subtract = gte(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gte(ref(`age`), val(10)), lt(ref(`age`), val(20))], - }) - }) - - it(`should compute difference: age > 10 - age >= 20 = (age > 10 AND age < 20)`, () => { - const from = gt(ref(`age`), val(10)) - const subtract = gte(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gt(ref(`age`), val(10)), lt(ref(`age`), val(20))], - }) - }) - - it(`should compute difference: age >= 10 - age > 20 = (age >= 10 AND age <= 20)`, () => { - const from = gte(ref(`age`), val(10)) - const subtract = gt(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gte(ref(`age`), val(10)), lte(ref(`age`), val(20))], - }) - }) - }) - - describe(`range minus range - lt/lte`, () => { - it(`should compute difference: age < 30 - age < 20 = (age >= 20 AND age < 30)`, () => { - const from = lt(ref(`age`), val(30)) - const subtract = lt(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gte(ref(`age`), val(20)), lt(ref(`age`), val(30))], - }) - }) - - it(`should return original when no overlap: age < 20 - age < 30`, () => { - const from = lt(ref(`age`), val(20)) - const subtract = lt(ref(`age`), val(30)) - const result = minusWherePredicates(from, subtract) - - // age < 20 is subset of age < 30, so result is empty - expect(result).toEqual({ type: `val`, value: false }) - }) - - it(`should compute difference: age <= 30 - age <= 20 = (age > 20 AND age <= 30)`, () => { - const from = lte(ref(`age`), val(30)) - const subtract = lte(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gt(ref(`age`), val(20)), lte(ref(`age`), val(30))], - }) - }) - - it(`should compute difference: age < 30 - age <= 20 = (age > 20 AND age < 30)`, () => { - const from = lt(ref(`age`), val(30)) - const subtract = lte(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gt(ref(`age`), val(20)), lt(ref(`age`), val(30))], - }) - }) - - it(`should compute difference: age <= 30 - age < 20 = (age >= 20 AND age <= 30)`, () => { - const from = lte(ref(`age`), val(30)) - const subtract = lt(ref(`age`), val(20)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [gte(ref(`age`), val(20)), lte(ref(`age`), val(30))], - }) - }) - }) - - describe(`common conditions`, () => { - it(`should handle common conditions: (age > 10 AND status = 'active') - (age > 20 AND status = 'active') = (age > 10 AND age <= 20 AND status = 'active')`, () => { - const from = and( - gt(ref(`age`), val(10)), - eq(ref(`status`), val(`active`)), - ) - const subtract = and( - gt(ref(`age`), val(20)), - eq(ref(`status`), val(`active`)), - ) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [ - eq(ref(`status`), val(`active`)), // common condition - gt(ref(`age`), val(10)), - lte(ref(`age`), val(20)), - ], - }) - }) - - it(`should handle multiple common conditions`, () => { - const from = and( - gt(ref(`age`), val(10)), - eq(ref(`status`), val(`active`)), - eq(ref(`department`), val(`engineering`)), - ) - const subtract = and( - gt(ref(`age`), val(20)), - eq(ref(`status`), val(`active`)), - eq(ref(`department`), val(`engineering`)), - ) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [ - eq(ref(`status`), val(`active`)), // common condition - eq(ref(`department`), val(`engineering`)), // common condition - gt(ref(`age`), val(10)), - lte(ref(`age`), val(20)), - ], - }) - }) - - it(`should handle IN with common conditions: (age IN [10,20,30] AND status = 'active') - (age IN [20,30] AND status = 'active') = (age IN [10] AND status = 'active')`, () => { - const from = and( - inOp(ref(`age`), [10, 20, 30]), - eq(ref(`status`), val(`active`)), - ) - const subtract = and( - inOp(ref(`age`), [20, 30]), - eq(ref(`status`), val(`active`)), - ) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [ - eq(ref(`status`), val(`active`)), // common condition - { - type: `func`, - name: `eq`, - args: [ref(`age`), val(10)], - }, - ], - }) - }) - - it(`should return null when common conditions exist but remaining difference cannot be simplified`, () => { - const from = and( - gt(ref(`age`), val(10)), - eq(ref(`status`), val(`active`)), - ) - const subtract = and( - gt(ref(`name`), val(`Z`)), - eq(ref(`status`), val(`active`)), - ) - const result = minusWherePredicates(from, subtract) - - // Can't simplify age > 10 - name > 'Z' (different fields), so returns null - expect(result).toBeNull() - }) - }) - - describe(`Date support`, () => { - it(`should handle Date IN minus Date IN`, () => { - const date1 = new Date(`2024-01-01`) - const date2 = new Date(`2024-01-15`) - const date3 = new Date(`2024-02-01`) - - const from = inOp(ref(`createdAt`), [date1, date2, date3]) - const subtract = inOp(ref(`createdAt`), [date2]) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `in`, - args: [ref(`createdAt`), val([date1, date3])], - }) - }) - - it(`should handle Date range difference: date > 2024-01-01 - date > 2024-01-15`, () => { - const date1 = new Date(`2024-01-01`) - const date15 = new Date(`2024-01-15`) - - const from = gt(ref(`createdAt`), val(date1)) - const subtract = gt(ref(`createdAt`), val(date15)) - const result = minusWherePredicates(from, subtract) - - expect(result).toEqual({ - type: `func`, - name: `and`, - args: [ - gt(ref(`createdAt`), val(date1)), - lte(ref(`createdAt`), val(date15)), - ], - }) - }) - }) - - describe(`real-world sync scenarios`, () => { - it(`should compute missing data range: need age > 10, already have age > 20`, () => { - const requested = gt(ref(`age`), val(10)) - const alreadyLoaded = gt(ref(`age`), val(20)) - const needToFetch = minusWherePredicates(requested, alreadyLoaded) - - // Need to fetch: 10 < age <= 20 - expect(needToFetch).toEqual({ - type: `func`, - name: `and`, - args: [gt(ref(`age`), val(10)), lte(ref(`age`), val(20))], - }) - }) - - it(`should compute missing IDs: need IN [1..100], already have IN [50..100]`, () => { - const allIds = Array.from({ length: 100 }, (_, i) => i + 1) - const loadedIds = Array.from({ length: 51 }, (_, i) => i + 50) - - const requested = inOp(ref(`id`), allIds) - const alreadyLoaded = inOp(ref(`id`), loadedIds) - const needToFetch = minusWherePredicates(requested, alreadyLoaded) - - // Need to fetch: ids 1..49 - const expectedIds = Array.from({ length: 49 }, (_, i) => i + 1) - expect(needToFetch).toEqual({ - type: `func`, - name: `in`, - args: [ref(`id`), val(expectedIds)], - }) - }) - - it(`should return empty when all requested data is already loaded`, () => { - const requested = gt(ref(`age`), val(20)) - const alreadyLoaded = gt(ref(`age`), val(10)) - const needToFetch = minusWherePredicates(requested, alreadyLoaded) - - // Requested is subset of already loaded - nothing more to fetch - expect(needToFetch).toEqual({ type: `val`, value: false }) - }) - }) -}) diff --git a/packages/db/tests/query/public-container-copy.test.ts b/packages/db/tests/query/public-container-copy.test.ts new file mode 100644 index 0000000000..85e7daff76 --- /dev/null +++ b/packages/db/tests/query/public-container-copy.test.ts @@ -0,0 +1,162 @@ +import { expect, it } from 'vitest' +import { transformPublicContainers } from '../../src/query/compiler/route-metadata.js' +import { + createLiveQueryCollection, + eq, + materialize, + toArray, +} from '../../src/query/index.js' +import { createControlledCollection } from './includes-oracle-helpers.js' + +it.each( + ([`object`, `array`] as const).flatMap((kind) => + [false, true].map((ordered) => ({ kind, ordered })), + ), +)( + `preserves $kind reference-key matches through an ordered=$ordered projected source`, + async ({ kind, ordered }) => { + const makeKey = (code: number): object => + kind === `object` ? { code } : [code] + const key = makeKey(1) + const other = makeKey(2) + const parents = createControlledCollection(`copy-parents`, [ + { id: 1, group: 1, key }, + ]) + const children = createControlledCollection(`copy-children`, [ + { id: 10, group: 1, key }, + { id: 20, group: 1, key: makeKey(1) }, + { id: 30, group: 1, key: other }, + ]) + const live = createLiveQueryCollection((q) => + q.from({ parent: parents.collection }).select(({ parent }) => { + const filtered = q + .from({ child: children.collection }) + .where(({ child }) => eq(child.group, parent.group)) + const source = ( + ordered ? filtered.orderBy(({ child }) => child.id) : filtered + ).select(({ child }) => ({ id: child.id, key: child.key })) + const matches = q + .from({ inner: source }) + .where(({ inner }) => eq(inner.key, parent.key)) + .select(({ inner }) => ({ id: inner.id })) + return { + id: parent.id, + collection: matches, + array: toArray(matches), + materialized: materialize(matches), + } + }), + ) + const check = (expected: Array) => { + const row = live.get(1)! + for (const values of [ + row.collection.toArray, + row.array, + row.materialized, + ]) { + expect(values.map(({ id }) => id).sort((a, b) => a - b)).toEqual( + expected, + ) + } + } + try { + await live.preload() + check([10]) + parents.write(`update`, { id: 1, group: 1, key: other }) + check([30]) + children.write(`update`, { id: 10, group: 1, key: other }) + check([10, 30]) + children.write(`delete`, { id: 30, group: 1, key: other }) + check([10]) + } finally { + await live.cleanup() + await Promise.all([ + parents.collection.cleanup(), + children.collection.cleanup(), + ]) + } + }, +) + +it.each([false, true])( + `copies public descriptors with null prototype=%s`, + (nullPrototype) => { + const privateKey = Symbol(`private`) + const publicKey = Symbol(`public`) + const opaque = new Date(0) + const replacement = new Map() + const reference = { token: true } + const child = { [privateKey]: true, value: 1 } + const input = Object.create( + nullPrototype ? null : Object.prototype, + ) as Record + let reads = 0 + const getter = () => { + reads++ + return 7 + } + Object.defineProperties(input, { + child: { value: child, enumerable: true, writable: false }, + alias: { value: child, enumerable: true }, + leaf: { value: reference, enumerable: true }, + opaque: { value: opaque, enumerable: true }, + hidden: { value: 4, enumerable: false }, + accessor: { get: getter, enumerable: true }, + [`__proto__`]: { value: `user property`, enumerable: true }, + [publicKey]: { value: child, enumerable: true }, + [privateKey]: { value: true }, + self: { value: input, enumerable: true }, + }) + const result = transformPublicContainers( + input, + (value) => (value === reference ? replacement : value), + new Set([privateKey]), + ) as typeof input + expect(reads).toBe(0) + expect(Object.getPrototypeOf(result)).toBe(Object.getPrototypeOf(input)) + expect(Reflect.ownKeys(result)).toEqual( + Reflect.ownKeys(input).filter((key) => key !== privateKey), + ) + expect(result.child).toEqual({ value: 1 }) + expect(result.alias).toBe(result.child) + expect(result[publicKey]).toBe(result.child) + expect(result.self).toBe(result) + expect(result.leaf).toBe(replacement) + expect(result.opaque).toBe(opaque) + expect(result[`__proto__`]).toBe(`user property`) + expect(Object.getOwnPropertyDescriptor(result, `child`)).toEqual({ + value: result.child, + enumerable: true, + writable: false, + configurable: false, + }) + expect(Object.getOwnPropertyDescriptor(result, `accessor`)?.get).toBe( + getter, + ) + expect(Object.getOwnPropertyDescriptor(result, `hidden`)).toEqual( + Object.getOwnPropertyDescriptor(input, `hidden`), + ) + expect(child[privateKey]).toBe(true) + expect(input.self).toBe(input) + }, +) + +it(`preserves sparse arrays and locked lengths while removing private keys`, () => { + const privateKey = Symbol(`private`) + const input: Array> = new Array(4) + input[2] = { value: 2, [privateKey]: true } + Object.defineProperty(input, `length`, { writable: false }) + const result = transformPublicContainers( + input, + (value) => value, + new Set([privateKey]), + ) as Array + const expected = new Array(4) + expected[2] = { value: 2 } + expect(result).toEqual(expected) + expect(Object.hasOwn(result, 0)).toBe(false) + expect(Object.getOwnPropertyDescriptor(result, `length`)).toEqual( + Object.getOwnPropertyDescriptor(input, `length`), + ) + expect(input[2][privateKey]).toBe(true) +}) diff --git a/packages/db/tests/query/replay-failure-boundary.test.ts b/packages/db/tests/query/replay-failure-boundary.test.ts new file mode 100644 index 0000000000..afbebf8bb8 --- /dev/null +++ b/packages/db/tests/query/replay-failure-boundary.test.ts @@ -0,0 +1,216 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../../src/collection' +import { createDeferred } from '../../src/deferred' +import { BasicIndex } from '../../src/indexes/basic-index' +import { createLiveQueryCollection, eq } from '../../src/query' +import { PropRef } from '../../src/query/ir' +import { evaluateReferenceExpression } from '../reference-expression' +import { flushPromises } from '../utils' +import type { SyncConfig } from '../../src/types' + +type Row = { id: number; version: number } + +describe.each([`direct`, `query`] as const)( + `failed replay publication and recovery for %s`, + (consumer) => { + it.each( + ([`throw`, `reject`] as const).flatMap((failureMode) => + [false, true].map((partialWrite) => ({ failureMode, partialWrite })), + ), + )( + `keeps peers and retained results sound: %j`, + async ({ failureMode, partialWrite }) => { + const failure = new Error(`replacement failed`) + const pending = createDeferred() + let phase: `initial` | `failed` | `recovered` = `initial` + let sync!: Parameters[`sync`]>[0] + let childSync!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: (operations) => { + sync = operations + operations.begin() + for (const id of [1, 2]) + operations.write({ type: `insert`, value: { id, version: 1 } }) + operations.commit() + operations.markReady() + return { + loadSubset: (options) => { + const ids = [1, 2].filter( + (id) => + !options.where || + evaluateReferenceExpression(options.where, { + id, + version: 1, + }), + ) + if (phase === `initial`) return true + for (const id of ids) { + if (phase === `failed` && id === 1 && !partialWrite) + continue + operations.begin() + operations.write({ + type: source.has(id) ? `update` : `insert`, + value: { id, version: phase === `recovered` ? 4 : 2 }, + }) + operations.commit() + } + if (phase === `failed` && ids.includes(1)) { + if (failureMode === `throw`) throw failure + return pending.promise + } + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const children = createCollection({ + getKey: (row) => row.id, + sync: { + sync: (operations) => { + childSync = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, version: 1 } }) + operations.commit() + operations.markReady() + }, + }, + }) + const makeLive = () => + createLiveQueryCollection((q) => + q + .from({ row: source }) + .where(({ row }) => eq(row.id, 1)) + .orderBy(({ row }) => row.id) + .limit(1) + .select(({ row }) => ({ + id: row.id, + version: row.version, + children: q + .from({ child: children }) + .where(({ child }) => eq(child.id, row.id)), + })), + ) + const live = consumer === `query` ? makeLive() : undefined + const peer = createLiveQueryCollection((q) => + q.from({ row: source }).where(({ row }) => eq(row.id, 2)), + ) + const visible = new Map() + const direct = source.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.key !== 1) continue + if (change.type === `delete`) visible.delete(1) + else visible.set(1, { id: 1, version: change.value.version }) + } + }, + { includeInitialState: false }, + ) + const errors: Array = [] + direct.on(`loadSubset:error`, ({ error }) => errors.push(error)) + let replacement: ReturnType | undefined + let replacementDirect: typeof direct | undefined + try { + if (live) await live.preload() + else + direct.requestSnapshot({ + where: eq(sourceExpression(), 1), + optimizedOnly: false, + }) + await peer.preload() + const retainedChild = live?.get(1)?.children + if (live) expect(retainedChild).toBeDefined() + const read = () => + live ? live.get(1)?.version : visible.get(1)?.version + expect(read()).toBe(1) + phase = `failed` + sync.begin() + sync.truncate() + sync.commit() + // Observe the waiter before the queued acquisition can reject. + const waiter = live + ? live.utils.setWindow({ limit: 2 }) + : direct.pendingTruncateReplacement + expect(waiter).toBeInstanceOf(Promise) + const settled = Promise.allSettled([waiter]) + await flushPromises() + if (failureMode === `reject`) pending.reject(failure) + expect(await settled).toEqual([ + { status: `rejected`, reason: failure }, + ]) + await flushPromises() + expect(read()).toBe(1) + expect(peer.get(2)?.version).toBe(2) + if (!live) expect(errors).toEqual([failure]) + if (retainedChild) expect(retainedChild.get(1)?.version).toBe(1) + + sync.begin() + sync.write({ + type: source.has(1) ? `update` : `insert`, + value: { id: 1, version: 3 }, + }) + sync.write({ type: `update`, value: { id: 2, version: 3 } }) + sync.commit() + if (retainedChild) { + childSync.begin() + childSync.write({ type: `update`, value: { id: 1, version: 3 } }) + childSync.commit() + } + await flushPromises() + expect(read()).toBe(1) + expect(peer.get(2)?.version).toBe(3) + expect(source.status).toBe(`ready`) + if (retainedChild) expect(retainedChild.get(1)?.version).toBe(1) + + // Recreating only the failed consumer is a valid recovery action. + // Do not reset the shared source or force its healthy peer to restart. + phase = `recovered` + if (live) { + await live.cleanup() + replacement = makeLive() + await replacement.preload() + expect(replacement.get(1)?.version).toBe(4) + expect(replacement.get(1)?.children.get(1)?.version).toBe(3) + } else { + direct.unsubscribe() + replacementDirect = source.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.key === 1 && change.type !== `delete`) + visible.set(1, change.value) + } + }, + { includeInitialState: false }, + ) + replacementDirect.requestSnapshot({ + where: eq(sourceExpression(), 1), + optimizedOnly: false, + }) + expect(visible.get(1)?.version).toBe(4) + } + expect(peer.get(2)?.version).toBe(3) + } finally { + pending.resolve() + direct.unsubscribe() + replacementDirect?.unsubscribe() + await Promise.all([ + live?.cleanup(), + replacement?.cleanup(), + peer.cleanup(), + ]) + await Promise.all([source.cleanup(), children.cleanup()]) + } + }, + ) + }, +) + +function sourceExpression() { + return new PropRef([`id`]) +} diff --git a/packages/db/tests/query/scheduler.test.ts b/packages/db/tests/query/scheduler.test.ts index 3faf361648..d7c8e546fc 100644 --- a/packages/db/tests/query/scheduler.test.ts +++ b/packages/db/tests/query/scheduler.test.ts @@ -1,19 +1,46 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { createCollection } from '../../src/collection/index.js' +import { createDeferred } from '../../src/deferred.js' import { createLiveQueryCollection, eq, isNull } from '../../src/query/index.js' import { createTransaction } from '../../src/transactions.js' import { createOptimisticAction } from '../../src/optimistic-action.js' import { + Scheduler, getActivePublicationContext, + recordPublicationError, transactionScopedScheduler, withPublicationContext, } from '../../src/scheduler.js' import { CollectionConfigBuilder } from '../../src/query/live/collection-config-builder.js' -import { mockSyncCollectionOptions, stripVirtualProps } from '../utils.js' +import { getCollectionBuilder } from '../../src/query/live/collection-registry.js' +import { CollectionSubscriber } from '../../src/query/live/collection-subscriber.js' +import { Query, createEffect } from '../../src/index.js' +import { + flushPromises, + mockSyncCollectionOptions, + stripVirtualProps, +} from '../utils.js' +import type { SchedulerContextId } from '../../src/scheduler.js' import type { OutputWithVirtual } from '../utils.js' import type { FullSyncState } from '../../src/query/live/types.js' import type { SyncConfig } from '../../src/types.js' +type SchedulerInternals = { + contexts: Map }> +} +const flushAll = (scheduler: Scheduler) => { + const { contexts } = scheduler as unknown as SchedulerInternals + for (const contextId of Array.from(contexts.keys())) + scheduler.flush(contextId) +} +const hasPendingJobs = ( + scheduler: Scheduler, + contextId: SchedulerContextId, +) => { + const { contexts } = scheduler as unknown as SchedulerInternals + return (contexts.get(contextId)?.jobs.size ?? 0) > 0 +} + interface ChangeMessageLike { type: string value: any @@ -24,6 +51,17 @@ interface User { name: string } +const falsyListenerFailureCases = [ + { name: `undefined`, failure: undefined }, + { name: `null`, failure: null }, + { name: `false`, failure: false }, + { name: `zero`, failure: 0 }, + { name: `negative zero`, failure: -0 }, + { name: `bigint zero`, failure: 0n }, + { name: `empty string`, failure: `` }, + { name: `NaN`, failure: Number.NaN }, +] + type UserWithVirtual = OutputWithVirtual interface Task { @@ -88,10 +126,83 @@ function recordBatches(collection: any) { } afterEach(() => { - transactionScopedScheduler.flushAll() + flushAll(transactionScopedScheduler) +}) + +describe(`Scheduler dependency reentry`, () => { + it.each( + [false, true].flatMap((sourceFirst) => + [false, true].flatMap((pendingAware) => + [false, true].map((requeue) => ({ + sourceFirst, + pendingAware, + requeue, + })), + ), + ), + )( + `waits for current source work: sourceFirst=$sourceFirst pendingAware=$pendingAware requeue=$requeue`, + ({ sourceFirst, pendingAware, requeue }) => { + const scheduler = new Scheduler() + const contextId = Symbol(`source-reentry`) + let sourceRuns = 0 + let pending = true + const source = pendingAware + ? { hasPendingGraphRun: () => pending } + : Symbol(`source`) + const observedRuns: Array = [] + const runSource = () => { + sourceRuns++ + pending = false + if (requeue && sourceRuns === 1) { + pending = true + scheduler.schedule({ contextId, jobId: source, run: runSource }) + } + } + const jobs = [ + { contextId, jobId: source, run: runSource }, + { + contextId, + jobId: Symbol(`dependent`), + dependencies: [source], + run: () => observedRuns.push(sourceRuns), + }, + ] + for (const job of sourceFirst ? jobs : [...jobs].reverse()) { + scheduler.schedule(job) + } + scheduler.flush(contextId) + expect(sourceRuns).toBe(requeue ? 2 : 1) + expect(observedRuns).toEqual([sourceRuns]) + expect(hasPendingJobs(scheduler, contextId)).toBe(false) + }, + ) }) describe(`Collection publication scheduler context`, () => { + it(`preserves the first listener error when a later graph job fails`, () => { + const listenerFailure = new Error(`listener failed first`) + const graphFailure = new Error(`graph failed later`) + const graphJob = vi.fn(() => { + throw graphFailure + }) + let contextId: ReturnType + expect(() => + withPublicationContext(() => { + contextId = getActivePublicationContext() + recordPublicationError(listenerFailure) + transactionScopedScheduler.schedule({ + contextId, + jobId: graphJob, + run: graphJob, + }) + }), + ).toThrow(listenerFailure) + expect(graphJob).toHaveBeenCalledOnce() + expect(hasPendingJobs(transactionScopedScheduler, contextId!)).toBe(false) + expect(getActivePublicationContext()).toBeUndefined() + }) + it(`shares one context and flushes after the outer publication`, () => { const calls: Array = [] let contextId: ReturnType @@ -139,11 +250,677 @@ describe(`Collection publication scheduler context`, () => { expect(run).not.toHaveBeenCalled() expect(getActivePublicationContext()).toBeUndefined() - expect(transactionScopedScheduler.hasPendingJobs(contextId!)).toBe(false) + expect(hasPendingJobs(transactionScopedScheduler, contextId!)).toBe(false) + }) + + it(`preserves a falsy graph failure through a publication boundary`, () => { + let didThrow = false + let thrown: unknown + + try { + withPublicationContext(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: `failing`, + run: () => { + throw undefined + }, + }) + }) + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(thrown).toBeUndefined() + }) + + it(`attempts every clear listener and preserves its first failure`, () => { + const scheduler = new Scheduler() + const firstFailure = new Error(`first clear listener failed`) + const laterFailure = new Error(`later clear listener failed`) + const calls: Array = [] + let firstClear = true + let removeAdded: (() => void) | undefined + scheduler.onClear(() => { + calls.push(`first`) + if (!firstClear) return + removeSecond() + removeAdded ??= scheduler.onClear(() => calls.push(`added`)) + throw firstFailure + }) + const removeSecond = scheduler.onClear(() => { + calls.push(`second`) + if (firstClear) throw laterFailure + }) + + let thrown: unknown + try { + scheduler.clear(`context`) + } catch (error) { + thrown = error + } + + expect(thrown).toBe(firstFailure) + expect(calls).toEqual([`first`, `second`]) + + firstClear = false + expect(() => scheduler.clear(`next context`)).not.toThrow() + expect(calls).toEqual([`first`, `second`, `first`, `added`]) + removeAdded?.() }) + + it.each([ + { source: `publication`, failureKind: `Error` }, + { source: `publication`, failureKind: `undefined` }, + { source: `graph`, failureKind: `Error` }, + { source: `graph`, failureKind: `undefined` }, + ] as const)( + `does not replace a $failureKind $source failure with a clear-listener failure`, + ({ source, failureKind }) => { + const primaryFailure = + failureKind === `Error` ? new Error(`${source} failed`) : undefined + const clearFailure = new Error(`clear listener failed`) + const laterClear = vi.fn() + const removeThrowingClear = transactionScopedScheduler.onClear(() => { + throw clearFailure + }) + const removeLaterClear = transactionScopedScheduler.onClear(laterClear) + + try { + let didThrow = false + let thrown: unknown + try { + withPublicationContext(() => { + if (source === `publication`) throw primaryFailure + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: `failing graph`, + run: () => { + throw primaryFailure + }, + }) + }) + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(Object.is(thrown, primaryFailure)).toBe(true) + expect(laterClear).toHaveBeenCalledOnce() + } finally { + removeThrowingClear() + removeLaterClear() + } + }, + ) }) describe(`live query scheduler`, () => { + it(`does not deliver a source batch after a snapshotted listener unsubscribes`, async () => { + let begin!: () => void + let write!: (message: { type: `insert`; value: User }) => void + let commit!: () => void + const calls: Array = [] + const source = createCollection({ + id: `ordinary-listener-membership-source`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: (actions) => { + begin = actions.begin + write = actions.write + commit = () => { + actions.commit() + } + actions.markReady() + }, + }, + }) + let added: { unsubscribe: () => void } | undefined + const first = source.subscribeChanges(() => { + calls.push(`first`) + second.unsubscribe() + added ??= source.subscribeChanges(() => calls.push(`added`), { + includeInitialState: false, + }) + }) + const second = source.subscribeChanges(() => calls.push(`second`)) + + try { + begin() + write({ type: `insert`, value: { id: 1, name: `Ada` } }) + commit() + expect(calls).toEqual([`first`]) + + begin() + write({ type: `insert`, value: { id: 2, name: `Grace` } }) + commit() + expect(calls).toEqual([`first`, `first`, `added`]) + } finally { + first.unsubscribe() + second.unsubscribe() + added?.unsubscribe() + await source.cleanup() + } + }) + + it(`delivers a layout-only batch to its frozen listener snapshot`, async () => { + type RankedUser = User & { rank: number } + const calls: Array = [] + const firstFailure = new Error(`first layout listener failed`) + const laterFailure = new Error(`later public listener failed`) + const graphJob = vi.fn(() => calls.push(`graph`)) + const source = createCollection( + mockSyncCollectionOptions({ + id: `layout-listener-membership-source`, + getKey: (user) => user.id, + initialData: [ + { id: 1, name: `Ada`, rank: 1 }, + { id: 2, name: `Grace`, rank: 2 }, + ], + }), + ) + const ordered = createLiveQueryCollection({ + id: `layout-listener-membership-ordered`, + startSync: true, + query: (q) => + q + .from({ user: source }) + .orderBy(({ user }) => user.rank, `asc`) + .select(({ user }) => ({ id: user.id, name: user.name })), + }) + await ordered.preload() + expect(ordered.toArray.map(({ id }) => id)).toEqual([1, 2]) + let firstPublication = true + let addedLayout: (() => void) | undefined + let addedPublic: { unsubscribe: () => void } | undefined + const unsubscribeFirstLayout = ordered._subscribeLayoutChanges(() => { + calls.push(`layout:first`) + if (!firstPublication) return + unsubscribeSecondLayout() + secondPublic.unsubscribe() + addedLayout ??= ordered._subscribeLayoutChanges(() => + calls.push(`layout:added`), + ) + addedPublic ??= ordered.subscribeChanges( + () => calls.push(`public:added`), + { includeInitialState: false }, + ) + throw firstFailure + }) + const unsubscribeSecondLayout = ordered._subscribeLayoutChanges(() => { + calls.push(`layout:second`) + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: graphJob, + run: graphJob, + }) + }) + const firstPublic = ordered.subscribeChanges( + () => { + calls.push(`public:first`) + if (firstPublication) throw laterFailure + }, + { includeInitialState: false }, + ) + const secondPublic = ordered.subscribeChanges( + () => calls.push(`public:second`), + { + includeInitialState: false, + }, + ) + + try { + let thrown: unknown + try { + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: 1, name: `Ada`, rank: 3 }, + }) + source.utils.commit() + } catch (error) { + thrown = error + } + + expect(thrown).toBe(firstFailure) + expect(calls).toEqual([ + `layout:first`, + `layout:second`, + `public:first`, + `graph`, + ]) + expect(graphJob).toHaveBeenCalledOnce() + expect(ordered.toArray.map(({ id }) => id)).toEqual([2, 1]) + + firstPublication = false + source.utils.begin() + source.utils.write({ + type: `update`, + value: { id: 1, name: `Ada`, rank: 0 }, + }) + expect(() => source.utils.commit()).not.toThrow() + expect(calls).toEqual([ + `layout:first`, + `layout:second`, + `public:first`, + `graph`, + `layout:first`, + `layout:added`, + `public:first`, + `public:added`, + ]) + } finally { + unsubscribeFirstLayout() + unsubscribeSecondLayout() + addedLayout?.() + firstPublic.unsubscribe() + secondPublic.unsubscribe() + addedPublic?.unsubscribe() + await ordered.cleanup() + await source.cleanup() + } + }) + + it(`settles a dependent live query when an earlier source listener throws`, async () => { + let begin!: () => void + let write!: (message: { type: `insert`; value: User }) => void + let commit!: () => void + const listenerFailure = new Error(`source listener failed`) + const source = createCollection({ + id: `throwing-listener-live-source`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: (actions) => { + begin = actions.begin + write = actions.write + commit = () => { + actions.commit() + } + actions.markReady() + }, + }, + }) + const throwingSubscription = source.subscribeChanges( + () => { + throw listenerFailure + }, + { includeInitialState: false }, + ) + const live = createLiveQueryCollection({ + id: `throwing-listener-live-dependent`, + startSync: true, + query: (q) => + q + .from({ user: source }) + .select(({ user }) => ({ id: user.id, name: user.name })), + }) + + try { + await live.preload() + begin() + write({ type: `insert`, value: { id: 1, name: `Ada` } }) + expect(() => commit()).toThrow(listenerFailure) + expect(live.get(1)).toEqual(expect.objectContaining({ name: `Ada` })) + } finally { + throwingSubscription.unsubscribe() + await live.cleanup() + await source.cleanup() + } + }) + + it.each(falsyListenerFailureCases)( + `preserves an exact $name row-listener failure after later delivery`, + async ({ name, failure }) => { + let begin!: () => void + let write!: (message: { type: `insert`; value: User }) => void + let commit!: () => void + type UserObservation = { + changes: Array<{ + type: string + key: string | number + value: UserWithVirtual + previousValue: UserWithVirtual | undefined + }> + rows: Array + } + const sourceObservations: Array = [] + const dependentObservations: Array = [] + const snapshotUser = ({ + id, + name: userName, + $collectionId, + $key, + $origin, + $synced, + }: UserWithVirtual): UserWithVirtual => ({ + id, + name: userName, + $collectionId, + $key, + $origin, + $synced, + }) + const source = createCollection({ + id: `falsy-row-listener-${name.replaceAll(` `, `-`)}`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: (actions) => { + begin = actions.begin + write = actions.write + commit = () => { + actions.commit() + } + actions.markReady() + }, + }, + }) + const throwingSubscription = source.subscribeChanges( + () => { + throw failure + }, + { includeInitialState: false }, + ) + const laterSubscription = source.subscribeChanges( + (changes) => { + sourceObservations.push({ + changes: changes.map(({ type, key, value, previousValue }) => ({ + type, + key, + value: snapshotUser(value), + previousValue: + previousValue === undefined + ? undefined + : snapshotUser(previousValue), + })), + rows: [...source.state.values()].map(snapshotUser), + }) + }, + { includeInitialState: false }, + ) + const live = createLiveQueryCollection({ + id: `falsy-row-listener-dependent-${name.replaceAll(` `, `-`)}`, + startSync: true, + query: (q) => + q + .from({ user: source }) + .select(({ user }) => ({ id: user.id, name: user.name })), + }) + let dependentSubscription: + | ReturnType + | undefined + + try { + await live.preload() + dependentSubscription = live.subscribeChanges( + (changes) => { + dependentObservations.push({ + changes: changes.map(({ type, key, value, previousValue }) => ({ + type, + key, + value: snapshotUser(value), + previousValue: + previousValue === undefined + ? undefined + : snapshotUser(previousValue), + })), + rows: [...live.state.values()].map(snapshotUser), + }) + }, + { includeInitialState: false }, + ) + begin() + write({ type: `insert`, value: { id: 1, name: `Ada` } }) + let didThrow = false + let thrown: unknown + try { + commit() + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(Object.is(thrown, failure)).toBe(true) + const expectedObservation = (collectionId: string): UserObservation => { + const row: UserWithVirtual = { + id: 1, + name: `Ada`, + $collectionId: collectionId, + $key: 1, + $origin: `remote`, + $synced: true, + } + return { + changes: [ + { + type: `insert`, + key: 1, + value: row, + previousValue: undefined, + }, + ], + rows: [row], + } + } + const expectedDependent = expectedObservation(live.id) + expect(sourceObservations).toEqual([expectedObservation(source.id)]) + expect(dependentObservations).toEqual([expectedDependent]) + expect([...live.state.values()].map(snapshotUser)).toEqual( + expectedDependent.rows, + ) + } finally { + throwingSubscription.unsubscribe() + laterSubscription.unsubscribe() + dependentSubscription?.unsubscribe() + await live.cleanup() + await source.cleanup() + } + }, + ) + + it.each([ + { + name: `Error`, + failure: new Error(`filtered source listener failed`), + }, + ...falsyListenerFailureCases, + ])( + `preserves an exact $name filtered row-listener failure`, + async ({ name, failure }) => { + let begin!: () => void + let write!: (message: { type: `insert`; value: User }) => void + let commit!: () => void + const filteredCalls = vi.fn() + const laterListener = vi.fn() + const source = createCollection({ + id: `filtered-throwing-listener-source-${name.replaceAll(` `, `-`)}`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: (actions) => { + begin = actions.begin + write = actions.write + commit = () => { + actions.commit() + } + actions.markReady() + }, + }, + }) + const throwingSubscription = source.subscribeChanges( + (changes) => { + filteredCalls(changes) + throw failure + }, + { + includeInitialState: false, + where: (user) => eq(user.name, `Ada`), + }, + ) + const laterSubscription = source.subscribeChanges(laterListener, { + includeInitialState: false, + }) + const live = createLiveQueryCollection({ + id: `filtered-throwing-listener-dependent-${name.replaceAll(` `, `-`)}`, + startSync: true, + query: (q) => + q + .from({ user: source }) + .select(({ user }) => ({ id: user.id, name: user.name })), + }) + + try { + await live.preload() + begin() + write({ type: `insert`, value: { id: 1, name: `Ada` } }) + let didThrow = false + let thrown: unknown + try { + commit() + } catch (error) { + didThrow = true + thrown = error + } + expect(didThrow).toBe(true) + expect(Object.is(thrown, failure)).toBe(true) + expect(filteredCalls).toHaveBeenCalledOnce() + expect(filteredCalls.mock.calls[0]?.[0]).toEqual([ + expect.objectContaining({ type: `insert`, key: 1 }), + ]) + expect(laterListener).toHaveBeenCalledOnce() + expect(live.get(1)).toEqual(expect.objectContaining({ name: `Ada` })) + + begin() + write({ type: `insert`, value: { id: 2, name: `Grace` } }) + expect(() => commit()).not.toThrow() + expect(filteredCalls).toHaveBeenCalledOnce() + expect(laterListener).toHaveBeenCalledTimes(2) + expect(live.get(2)).toEqual(expect.objectContaining({ name: `Grace` })) + } finally { + throwingSubscription.unsubscribe() + laterSubscription.unsubscribe() + await live.cleanup() + await source.cleanup() + } + }, + ) + + it(`keeps a nested ready failure when a later outer listener throws`, async () => { + let markInnerReady!: () => void + const readyFailure = new Error(`nested ready listener failed`) + const laterFailure = new Error(`later outer listener failed`) + const scheduledJob = vi.fn() + const inner = createCollection({ + id: `nested-ready-collision-inner`, + getKey: (user) => user.id, + sync: { + sync: ({ markReady }) => { + markInnerReady = markReady + }, + }, + }) + const innerFirst = inner.subscribeChanges(() => { + const contextId = getActivePublicationContext() + transactionScopedScheduler.schedule({ + contextId, + jobId: scheduledJob, + run: scheduledJob, + }) + }) + const innerSecond = inner.subscribeChanges(() => { + throw readyFailure + }) + + let beginOuter!: () => void + let writeOuter!: (message: { type: `insert`; value: User }) => void + let commitOuter!: () => void + const outer = createCollection({ + id: `nested-ready-collision-outer`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: (actions) => { + beginOuter = actions.begin + writeOuter = actions.write + commitOuter = () => { + actions.commit() + } + actions.markReady() + }, + }, + }) + const outerFirst = outer.subscribeChanges(() => markInnerReady()) + const outerSecond = outer.subscribeChanges(() => { + throw laterFailure + }) + + try { + beginOuter() + writeOuter({ type: `insert`, value: { id: 1, name: `Ada` } }) + expect(() => commitOuter()).toThrow(readyFailure) + expect(scheduledJob).toHaveBeenCalledOnce() + } finally { + outerFirst.unsubscribe() + outerSecond.unsubscribe() + innerFirst.unsubscribe() + innerSecond.unsubscribe() + await outer.cleanup() + await inner.cleanup() + } + }) + + it(`settles a dependent live query before a nested ready failure escapes`, async () => { + let markSourceReady: (() => void) | undefined + const listenerFailure = new Error(`source ready listener failed`) + const source = createCollection({ + id: `nested-ready-live-source`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: ({ begin, commit, markReady }) => { + begin() + commit() + markSourceReady = markReady + }, + }, + }) + const live = createLiveQueryCollection({ + id: `nested-ready-live-dependent`, + startSync: true, + query: (q) => + q + .from({ user: source }) + .select(({ user }) => ({ id: user.id, name: user.name })), + }) + const preload = live.preload() + const throwingSubscription = source.subscribeChanges(() => { + throw listenerFailure + }) + + try { + expect(live.status).toBe(`loading`) + expect(() => withPublicationContext(() => markSourceReady!())).toThrow( + listenerFailure, + ) + await expect(preload).resolves.toBeUndefined() + expect(source.status).toBe(`ready`) + expect(live.status).toBe(`ready`) + } finally { + throwingSubscription.unsubscribe() + await live.cleanup() + await source.cleanup() + } + }) + it(`runs the live query graph once per transaction that touches multiple collections`, async () => { const { users, tasks, assignments } = setupLiveQueryCollections(`single-batch`) @@ -239,7 +1016,7 @@ describe(`live query scheduler`, () => { const latestBatch = recorder.batches.at(-1)! expect(latestBatch[0]?.type).toBe(`delete`) } - expect(transactionScopedScheduler.hasPendingJobs(tx.id)).toBe(false) + expect(hasPendingJobs(transactionScopedScheduler, tx.id)).toBe(false) // We emit the optimistic insert and, after the explicit rollback, possibly a // compensating delete – but no duplicate inserts. expect(recorder.batches[0]![0]).toMatchObject({ type: `insert` }) @@ -279,6 +1056,127 @@ describe(`live query scheduler`, () => { tx.rollback() }) + it.each( + [`collection`, `effect`].flatMap((consumer) => + [false, true].flatMap((sharedSource) => + [false, true].flatMap((derivedRight) => + [false, true].map((reverseWrites) => ({ + consumer, + sharedSource, + derivedRight, + reverseWrites, + })), + ), + ), + ), + )( + `publishes settled dependencies once: $consumer shared=$sharedSource derivedRight=$derivedRight reverse=$reverseWrites`, + async ({ consumer, sharedSource, derivedRight, reverseWrites }) => { + type Row = { id: number; left: string; right: string } + const makeSource = (id: string) => + createCollection( + mockSyncCollectionOptions({ + id, + getKey: (row) => row.id, + initialData: [{ id: 1, left: `old-left`, right: `old-right` }], + }), + ) + const leftSource = makeSource(`dependency-left`) + const rightSource = sharedSource + ? leftSource + : makeSource(`dependency-right`) + const leftQuery = createLiveQueryCollection({ + query: (q) => + q + .from({ row: leftSource }) + .select(({ row }) => ({ id: row.id, value: row.left })), + }) + const rightQuery = derivedRight + ? createLiveQueryCollection({ + query: (q) => + q + .from({ row: rightSource }) + .select(({ row }) => ({ id: row.id, right: row.right })), + }) + : undefined + await Promise.all([ + leftQuery.preload(), + (rightQuery ?? rightSource).preload(), + ]) + const query = new Query() + .from({ left: leftQuery }) + .join( + { right: rightQuery ?? rightSource }, + ({ left, right }) => eq(left.id, right.id), + `inner`, + ) + .select(({ left, right }) => ({ + id: left.id, + left: left.value, + right: right.right, + })) + const publications: Array> = [] + let cleanupConsumer: () => Promise + if (consumer === `collection`) { + const joined = createLiveQueryCollection({ query }) + await joined.preload() + const subscription = joined.subscribeChanges(() => { + publications.push( + joined.toArray.map(({ left, right }) => ({ left, right })), + ) + }) + cleanupConsumer = async () => { + subscription.unsubscribe() + await joined.cleanup() + } + } else { + const effect = createEffect<{ + id: number + left: string + right: string + }>({ + query, + onBatch: (events) => { + publications.push( + events.map(({ value: { left, right } }) => ({ left, right })), + ) + }, + }) + cleanupConsumer = () => effect.dispose() + } + const tx = createTransaction({ + mutationFn: async () => {}, + autoCommit: false, + }) + try { + publications.length = 0 + const writes = [ + () => + leftSource.update(1, (row) => { + row.left = `next-left` + }), + () => + rightSource.update(1, (row) => { + row.right = `next-right` + }), + ] + tx.mutate(() => { + for (const write of reverseWrites ? [...writes].reverse() : writes) + write() + }) + expect([...publications]).toEqual([ + [{ left: `next-left`, right: `next-right` }], + ]) + } finally { + tx.rollback() + await cleanupConsumer() + await Promise.all([leftQuery.cleanup(), rightQuery?.cleanup()]) + await leftSource.cleanup() + if (!sharedSource) await rightSource.cleanup() + } + }, + ) + it(`runs join live queries once after their parent queries settle`, async () => { const collectionA = createCollection<{ id: number; value: string }>({ id: `diamond-A`, @@ -346,7 +1244,7 @@ describe(`live query scheduler`, () => { liveQueryB.preload(), liveQueryJoin.preload(), ]) - const baseRunCount = liveQueryJoin.utils.getRunCount() + const runs = vi.spyOn(getCollectionBuilder(liveQueryJoin)!, `maybeRunGraph`) const tx = createTransaction({ mutationFn: async () => {}, @@ -361,7 +1259,7 @@ describe(`live query scheduler`, () => { expect(liveQueryJoin.toArray.map((row) => stripVirtualProps(row))).toEqual([ { left: `A1`, right: `B1` }, ]) - expect(liveQueryJoin.utils.getRunCount()).toBe(baseRunCount + 1) + expect(runs).toHaveBeenCalledTimes(1) tx.mutate(() => { collectionA.update(1, (draft) => { @@ -375,8 +1273,9 @@ describe(`live query scheduler`, () => { expect(liveQueryJoin.toArray.map((row) => stripVirtualProps(row))).toEqual([ { left: `A1b`, right: `B1b` }, ]) - expect(liveQueryJoin.utils.getRunCount()).toBe(baseRunCount + 2) + expect(runs).toHaveBeenCalledTimes(2) tx.rollback() + runs.mockRestore() }) it(`runs hybrid joins once when they observe both a live query and a collection`, async () => { @@ -433,7 +1332,7 @@ describe(`live query scheduler`, () => { }) await Promise.all([liveQueryA.preload(), hybridJoin.preload()]) - const baseRunCount = hybridJoin.utils.getRunCount() + const runs = vi.spyOn(getCollectionBuilder(hybridJoin)!, `maybeRunGraph`) const tx = createTransaction({ mutationFn: async () => {}, @@ -448,7 +1347,7 @@ describe(`live query scheduler`, () => { expect(hybridJoin.toArray.map((row) => stripVirtualProps(row))).toEqual([ { left: `A7`, right: `B7` }, ]) - expect(hybridJoin.utils.getRunCount()).toBe(baseRunCount + 1) + expect(runs).toHaveBeenCalledTimes(1) tx.mutate(() => { collectionA.update(7, (draft) => { @@ -462,8 +1361,9 @@ describe(`live query scheduler`, () => { expect(hybridJoin.toArray.map((row) => stripVirtualProps(row))).toEqual([ { left: `A7b`, right: `B7b` }, ]) - expect(hybridJoin.utils.getRunCount()).toBe(baseRunCount + 2) + expect(runs).toHaveBeenCalledTimes(2) tx.rollback() + runs.mockRestore() }) it(`currently single batch when the join sees right-side data before the left`, async () => { @@ -520,7 +1420,7 @@ describe(`live query scheduler`, () => { }) await Promise.all([liveQueryA.preload(), join.preload()]) - const baseRunCount = join.utils.getRunCount() + const runs = vi.spyOn(getCollectionBuilder(join)!, `maybeRunGraph`) const tx = createTransaction({ mutationFn: async () => {}, @@ -535,10 +1435,91 @@ describe(`live query scheduler`, () => { expect(join.toArray.map((row) => stripVirtualProps(row))).toEqual([ { left: `left-later`, right: `right-first` }, ]) - expect(join.utils.getRunCount()).toBe(baseRunCount + 1) + expect(runs).toHaveBeenCalledTimes(1) tx.rollback() + runs.mockRestore() }) + it.each( + [`resolve`, `reject`].flatMap((outcome) => + [false, true].map((replacementSettled) => ({ + outcome, + replacementSettled, + })), + ), + )( + `isolates ordered publication participants across restart: $outcome replacementSettled=$replacementSettled`, + async ({ outcome, replacementSettled }) => { + let sync!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: ({ id }) => id, + sync: { + sync: (operations) => { + sync = operations + operations.begin() + operations.write({ type: `insert`, value: { id: 1, name: `old` } }) + operations.commit() + operations.markReady() + }, + }, + }) + const builder = new CollectionConfigBuilder({ + query: (q) => q.from({ user: source }), + }) + const config = builder.getConfig() + const live = createCollection({ ...config, singleResult: undefined }) + const obsolete = createDeferred() + const replacement = createDeferred() + try { + await live.preload() + // Inject participants at the builder boundary: the ordered loader has + // its own stale-result guards, which must not mask this owner's law. + builder.trackOrderedLoadPromise(obsolete.promise, true) + await live.cleanup() + await live.preload() + builder.trackOrderedLoadPromise(replacement.promise, true) + const publications: Array> = [] + live.subscribeChanges(() => { + publications.push(live.toArray.map(({ name }) => name)) + }) + const update = (name: string) => { + sync.begin() + sync.write({ type: `update`, value: { id: 1, name } }) + sync.commit() + } + update(`replacement`) + expect(live.toArray.map(({ name }) => name)).toEqual([`old`]) + expect(publications).toEqual([]) + if (replacementSettled) { + replacement.resolve() + await flushPromises() + } + const beforeObsolete = [...publications] + if (outcome === `resolve`) obsolete.resolve() + else obsolete.reject(new Error(`discarded session failed`)) + await flushPromises() + expect(publications).toEqual(beforeObsolete) + if (!replacementSettled) { + expect(live.toArray.map(({ name }) => name)).toEqual([`old`]) + replacement.resolve() + await flushPromises() + } + expect(live.toArray.map(({ name }) => name)).toEqual([`replacement`]) + expect(publications).toEqual([[`replacement`]]) + update(`later`) + expect(live.toArray.map(({ name }) => name)).toEqual([`later`]) + expect(publications).toEqual([[`replacement`], [`later`]]) + expect(live.status).toBe(`ready`) + expect(config.utils.lastSubsetError).toBeUndefined() + } finally { + obsolete.resolve() + replacement.resolve() + await live.cleanup() + await source.cleanup() + } + }, + ) + it(`coalesces load-more callbacks scheduled within the same context`, () => { const baseCollection = createCollection({ id: `loader-users`, @@ -596,6 +1577,288 @@ describe(`live query scheduler`, () => { maybeRunGraphSpy.mockRestore() }) + it.each( + [false, true].flatMap((initialWork) => + [false, true].map((loaderResult) => ({ initialWork, loaderResult })), + ), + )( + `drains loader writes before publication: initial=$initialWork return=$loaderResult`, + ({ initialWork, loaderResult }) => { + const source = createCollection({ + getKey: (user) => user.id, + sync: { sync: () => () => {} }, + }) + const builder = new CollectionConfigBuilder({ + query: (q) => q.from({ user: source }), + }) + const events: Array = [] + let pendingWork = initialWork + let wrote = false + builder.currentSyncConfig = { + markReady: vi.fn(), + } as unknown as Parameters[`sync`]>[0] + builder.currentSyncState = { + messagesCount: 1, + subscribedToAllCollections: true, + graph: { + pendingWork: () => pendingWork, + run: () => { + events.push(`graph`) + pendingWork = false + }, + }, + flushPendingChanges: () => events.push(`publish`), + } as unknown as FullSyncState + const contextId = Symbol(`loader-write-context`) + builder.scheduleGraphRun( + () => { + events.push(`first`) + if (!wrote) { + wrote = true + pendingWork = true + } + return loaderResult + }, + { contextId }, + ) + builder.scheduleGraphRun( + () => { + events.push(`second`) + return true + }, + { contextId }, + ) + transactionScopedScheduler.flush(contextId) + expect(events).toEqual([ + ...(initialWork ? [`graph`] : []), + `first`, + `second`, + `graph`, + `first`, + `second`, + `publish`, + ]) + expect(builder.hasPendingGraphRun(contextId)).toBe(false) + }, + ) + + it.each( + [ + { name: `undefined`, failure: undefined }, + { name: `null`, failure: null }, + { name: `false`, failure: false }, + { name: `zero`, failure: 0 }, + { name: `empty string`, failure: `` }, + { name: `NaN`, failure: Number.NaN }, + ].flatMap((entry) => + [false, true].map((laterFails) => ({ ...entry, laterFails })), + ), + )( + `preserves the first falsy graph-loader failure: $name laterFails=$laterFails`, + ({ failure, laterFails }) => { + const baseCollection = createCollection({ + id: `falsy-loader-users-${String(failure)}`, + getKey: (user) => user.id, + sync: { + sync: () => () => {}, + }, + }) + const builder = new CollectionConfigBuilder({ + id: `falsy-loader-builder-${String(failure)}`, + query: (q) => q.from({ user: baseCollection }), + }) + const contextId = Symbol(`falsy-loader-context`) + const laterLoader = vi.fn(() => { + if (laterFails) throw new Error(`later loader failed`) + return false + }) + const config = { + begin: vi.fn(), + write: vi.fn(), + commit: vi.fn(), + markReady: vi.fn(), + truncate: vi.fn(), + } as unknown as Parameters[`sync`]>[0] + const syncState = { + messagesCount: 0, + subscribedToAllCollections: true, + unsubscribeCallbacks: new Set<() => void>(), + graph: { + pendingWork: () => false, + run: vi.fn(), + }, + inputs: {}, + pipeline: {}, + } as unknown as FullSyncState + const maybeRunGraphSpy = vi + .spyOn(builder, `maybeRunGraph`) + .mockImplementation((combinedLoader) => { + combinedLoader?.() + }) + + builder.currentSyncConfig = config + builder.currentSyncState = syncState + builder.scheduleGraphRun( + () => { + throw failure + }, + { contextId }, + ) + builder.scheduleGraphRun(laterLoader, { contextId }) + + let didThrow = false + let thrown: unknown + try { + transactionScopedScheduler.flush(contextId) + } catch (error) { + didThrow = true + thrown = error + } finally { + maybeRunGraphSpy.mockRestore() + } + + expect(didThrow).toBe(true) + expect(Object.is(thrown, failure)).toBe(true) + expect(laterLoader).toHaveBeenCalledOnce() + }, + ) + + it(`attempts every repeated-alias source loader and preserves the first failure`, async () => { + const createSource = (name: string) => + createCollection({ + id: `source-loader-${name}`, + getKey: (user) => user.id, + startSync: true, + sync: { + sync: ({ markReady }) => { + markReady() + return () => {} + }, + }, + }) + const firstSource = createSource(`first`) + const secondSource = createSource(`second`) + const thirdSource = createSource(`third`) + const builder = new CollectionConfigBuilder({ + id: `source-loader-builder`, + query: (q) => + q.from({ root: firstSource }).select(({ root }) => ({ + id: root.id, + second: q + .from({ item: secondSource }) + .where(({ item }) => eq(item.id, root.id)), + third: q + .from({ item: thirdSource }) + .where(({ item }) => eq(item.id, root.id)), + })), + }) + type BuilderSyncConfig = Parameters< + ReturnType[`sync`][`sync`] + >[0] + const config = { + begin: vi.fn(), + write: vi.fn(), + commit: vi.fn(), + markReady: vi.fn(), + truncate: vi.fn(), + } as unknown as BuilderSyncConfig + const builderInternals = builder as unknown as { + graphCache: FullSyncState[`graph`] + inputsCache: FullSyncState[`inputs`] + pipelineCache: FullSyncState[`pipeline`] + collectionSources: Array<{ + sourceId: string + alias: string + collection: object + }> + subscribeToAllCollections: ( + syncConfig: typeof config, + state: FullSyncState, + ) => () => void + } + const syncState = { + messagesCount: 0, + unsubscribeCallbacks: new Set<() => void>(), + subscribedToAllCollections: false, + graph: builderInternals.graphCache, + inputs: builderInternals.inputsCache, + pipeline: builderInternals.pipelineCache, + } as unknown as FullSyncState + const sourceIdFor = (collection: object): string => { + const source = builderInternals.collectionSources.find( + (candidate) => candidate.collection === collection, + ) + if (!source) throw new Error(`Expected a lexical source`) + return source.sourceId + } + const firstSourceId = sourceIdFor(firstSource) + const secondSourceId = sourceIdFor(secondSource) + const thirdSourceId = sourceIdFor(thirdSource) + expect( + builderInternals.collectionSources.map(({ alias }) => alias), + ).toEqual([`root`, `item`, `item`]) + expect(new Set([firstSourceId, secondSourceId, thirdSourceId]).size).toBe(3) + const laterFailure = new Error(`later source failed`) + const loaderCalls: Array = [] + const loaderCallCounts = new Map() + const loadMoreSpy = vi + .spyOn(CollectionSubscriber.prototype, `loadMoreIfNeeded`) + .mockImplementation(function (this: unknown) { + const { sourceId } = this as { sourceId: string } + loaderCalls.push(sourceId) + loaderCallCounts.set( + sourceId, + (loaderCallCounts.get(sourceId) ?? 0) + 1, + ) + if (sourceId === firstSourceId) throw undefined + if (sourceId === secondSourceId) throw laterFailure + if (sourceId === thirdSourceId) return true + throw new Error(`Unexpected source: ${sourceId}`) + }) + + try { + builder.currentSyncConfig = config + builder.currentSyncState = syncState + const loadAllSources = builderInternals.subscribeToAllCollections( + config, + syncState, + ) + + let didThrow = false + let thrown: unknown + try { + loadAllSources() + } catch (error) { + didThrow = true + thrown = error + } + + expect(didThrow).toBe(true) + expect(Object.is(thrown, undefined)).toBe(true) + expect(loaderCalls).toEqual([ + firstSourceId, + secondSourceId, + thirdSourceId, + ]) + expect(loaderCallCounts).toEqual( + new Map([ + [firstSourceId, 1], + [secondSourceId, 1], + [thirdSourceId, 1], + ]), + ) + expect(loadMoreSpy).toHaveBeenCalledTimes(3) + } finally { + for (const unsubscribe of syncState.unsubscribeCallbacks) unsubscribe() + loadMoreSpy.mockRestore() + await Promise.all([ + firstSource.cleanup(), + secondSource.cleanup(), + thirdSource.cleanup(), + ]) + } + }) + it(`should handle optimistic mutations with nested left joins without scheduler errors`, async () => { // This test verifies that optimistic mutations on collections with nested live query // collections using left joins complete successfully without scheduler errors. @@ -730,7 +1993,7 @@ describe(`live query scheduler`, () => { await new Promise((resolve) => setTimeout(resolve, 10)) // The scheduler should flush successfully without detecting unresolved dependencies - transactionScopedScheduler.flushAll() + flushAll(transactionScopedScheduler) } catch (e) { error = e as Error } @@ -817,7 +2080,7 @@ describe(`live query scheduler`, () => { try { action(`1`) await new Promise((resolve) => setTimeout(resolve, 10)) - transactionScopedScheduler.flushAll() + flushAll(transactionScopedScheduler) } catch (e) { error = e as Error } diff --git a/packages/db/tests/query/subset-dedupe.test.ts b/packages/db/tests/query/subset-dedupe.test.ts index b76ba2963c..bff40c2c17 100644 --- a/packages/db/tests/query/subset-dedupe.test.ts +++ b/packages/db/tests/query/subset-dedupe.test.ts @@ -1,1449 +1,483 @@ +import { runInNewContext } from 'node:vm' import { describe, expect, it, vi } from 'vitest' -import { - DeduplicatedLoadSubset, - cloneOptions, -} from '../../src/query/subset-dedupe' +import { DeduplicatedLoadSubset } from '../../src/query/subset-dedupe' +import { eq, gt } from '../../src/query/builder/functions' import { Func, PropRef, Value } from '../../src/query/ir' -import type { BasicExpression, OrderBy } from '../../src/query/ir' -import type { LoadSubsetOptions } from '../../src/types' +import { compileSingleRowExpression } from '../../src/query/compiler/evaluators' +import type { LoadSubsetFn, LoadSubsetOptions } from '../../src/types' -// Helper functions to build expressions more easily -function ref(path: string | Array): PropRef { - return new PropRef(typeof path === `string` ? [path] : path) -} - -function val(value: T): Value { - return new Value(value) -} - -function gt(left: BasicExpression, right: BasicExpression): Func { - return new Func(`gt`, [left, right]) -} - -function lt(left: BasicExpression, right: BasicExpression): Func { - return new Func(`lt`, [left, right]) -} - -function eq(left: BasicExpression, right: BasicExpression): Func { - return new Func(`eq`, [left, right]) -} - -function and(...expressions: Array>): Func { - return new Func(`and`, expressions) -} - -function inOp(left: BasicExpression, values: Array): Func { - return new Func(`in`, [left, new Value(values)]) -} - -function lte(left: BasicExpression, right: BasicExpression): Func { - return new Func(`lte`, [left, right]) -} +const ref = (name: string) => new PropRef([name]) +const val = (value: T) => new Value(value) -function not(expression: BasicExpression): Func { - return new Func(`not`, [expression]) -} - -describe(`createDeduplicatedLoadSubset`, () => { - it(`shares in-flight work while any cancellation owner remains active`, async () => { - let resolveLoad: (() => void) | undefined - let sharedSignal: AbortSignal | undefined - const loadSubset = vi.fn( - (options: LoadSubsetOptions) => - new Promise((resolve) => { - sharedSignal = options.signal - resolveLoad = resolve - }), - ) - const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - const owners = Array.from({ length: 10 }, () => new AbortController()) - const where = gt(ref(`age`), val(10)) - - const loads = owners.map((owner) => - deduplicated.loadSubset({ where, signal: owner.signal }), - ) - - expect(loadSubset).toHaveBeenCalledTimes(1) - for (const load of loads) expect(load).toBe(loads[0]) - for (const owner of owners) expect(sharedSignal).not.toBe(owner.signal) - - for (const owner of owners.slice(0, -1)) owner.abort() - expect(sharedSignal?.aborted).toBe(false) - - resolveLoad?.() - await Promise.all(loads) +describe(`DeduplicatedLoadSubset`, () => { + it(`deduplicates only completed exact demands`, async () => { + const loadSubset = vi.fn().mockResolvedValue(undefined) + const onDeduplicate = vi.fn() + const deduplicated = new DeduplicatedLoadSubset({ + loadSubset, + onDeduplicate, + }) + await deduplicated.loadSubset({ + where: gt(ref(`age`), val(10)), + limit: 2, + }) expect( deduplicated.loadSubset({ - where, - signal: new AbortController().signal, + where: gt(ref(`age`), val(10)), + limit: 2, }), ).toBe(true) - }) - - it(`aborts shared in-flight work after every cancellation owner leaves`, async () => { - const releases: Array<() => void> = [] - let sharedSignal: AbortSignal | undefined - let callCount = 0 - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: (options) => { - callCount += 1 - sharedSignal = options.signal - return new Promise((resolve) => releases.push(resolve)) - }, - }) - const first = new AbortController() - const second = new AbortController() - const where = gt(ref(`age`), val(10)) - - const firstLoad = deduplicated.loadSubset({ where, signal: first.signal }) - const secondLoad = deduplicated.loadSubset({ where, signal: second.signal }) - expect(callCount).toBe(1) - expect(secondLoad).toBe(firstLoad) - - first.abort() - expect(sharedSignal?.aborted).toBe(false) - second.abort() - expect(sharedSignal?.aborted).toBe(true) - releases[0]?.() - await Promise.all([firstLoad, secondLoad]) - - const retry = deduplicated.loadSubset({ - where, - signal: new AbortController().signal, - }) - expect(callCount).toBe(2) - expect(retry).toBeInstanceOf(Promise) - releases[1]?.() - await retry - }) - - it(`keeps shared work active for a signal-less owner`, async () => { - let resolveLoad: (() => void) | undefined - let sharedSignal: AbortSignal | undefined - const loadSubset = vi.fn( - (options: LoadSubsetOptions) => - new Promise((resolve) => { - sharedSignal = options.signal - resolveLoad = resolve - }), - ) - const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - const controller = new AbortController() - const where = gt(ref(`age`), val(10)) - - const abortable = deduplicated.loadSubset({ - where, - signal: controller.signal, - }) - const persistent = deduplicated.loadSubset({ where }) - expect(loadSubset).toHaveBeenCalledTimes(1) - expect(persistent).toBe(abortable) - controller.abort() - expect(sharedSignal?.aborted).toBe(false) - - resolveLoad?.() - await Promise.all([abortable, persistent]) - expect(deduplicated.loadSubset({ where })).toBe(true) - }) - - it(`should call underlying loadSubset on first call`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) + expect(onDeduplicate).toHaveBeenCalledTimes(1) - expect(callCount).toBe(1) - }) - - it(`should return true immediately for subset unlimited calls`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: age > 10 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(1) - - // Second call: age > 20 (subset of age > 10) - const result = await deduplicated.loadSubset({ + await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)), + limit: 2, }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not call underlying function - }) - - it(`should call underlying loadSubset for non-subset unlimited calls`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, + await deduplicated.loadSubset({ + where: gt(ref(`age`), val(10)), + limit: 3, }) - - // First call: age > 20 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) - - // Second call: age > 10 (NOT a subset of age > 20) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(2) // Should call underlying function + expect(loadSubset).toHaveBeenCalledTimes(3) }) - it(`should combine unlimited calls with union`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) + it(`does not infer coverage from a broader predicate or window`, async () => { + const loadSubset = vi.fn().mockResolvedValue(undefined) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - // First call: age > 20 + await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) + await deduplicated.loadSubset({ limit: 10, offset: 0 }) + await deduplicated.loadSubset({ limit: 5, offset: 2 }) - // Second call: age < 10 (different range) - await deduplicated.loadSubset({ where: lt(ref(`age`), val(10)) }) - expect(callCount).toBe(2) - - // Third call: age > 25 (subset of age > 20) - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(25)), - }) - expect(result).toBe(true) - expect(callCount).toBe(2) // Should not call - covered by first call + expect(loadSubset).toHaveBeenCalledTimes(4) }) - it(`should track limited calls separately`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - + it(`shares exact in-flight work when it has no cancellation owner`, async () => { + let resolve!: () => void + const loadSubset = vi.fn( + () => new Promise((done) => (resolve = done)), + ) + const onDeduplicate = vi.fn() const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, + loadSubset, + onDeduplicate, }) - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - const whereClause = gt(ref(`age`), val(10)) + const first = deduplicated.loadSubset({ limit: 2 }) + const second = deduplicated.loadSubset({ limit: 2 }) - // First call: age > 10, orderBy age asc, limit 10 - await deduplicated.loadSubset({ - where: whereClause, - orderBy: orderBy1, - limit: 10, - }) - expect(callCount).toBe(1) + expect(second).toBe(first) + expect(loadSubset).toHaveBeenCalledTimes(1) + expect(onDeduplicate).not.toHaveBeenCalled() - // Second call: SAME where clause, same orderBy, smaller limit (subset) - // For limited queries, where clauses must be EQUAL for subset relationship - const result = await deduplicated.loadSubset({ - where: whereClause, // Same where clause - orderBy: orderBy1, - limit: 5, - }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not call - subset of first + resolve() + await Promise.all([first, second]) + expect(onDeduplicate).toHaveBeenCalledTimes(1) + expect(deduplicated.loadSubset({ limit: 2 })).toBe(true) }) - it(`should NOT dedupe limited calls with different where clauses`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - // First call: age > 10, orderBy age asc, limit 10 - await deduplicated.loadSubset({ - where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - limit: 10, - }) - expect(callCount).toBe(1) - - // Second call: DIFFERENT where clause (age > 20) - should NOT be deduped - // even though age > 20 is "more restrictive" than age > 10, - // the top 5 of age > 20 might not be in the top 10 of age > 10 - await deduplicated.loadSubset({ - where: gt(ref(`age`), val(20)), - orderBy: orderBy1, - limit: 5, - }) - expect(callCount).toBe(2) // Should call - different where clause - }) - - it(`should call underlying for non-subset limited calls`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, + describe.each([`resolve`, `reject`] as const)( + `shared transport %s with deduplication observers`, + (outcome) => { + it.each([ + { waiters: 2, throws: false }, + { waiters: 2, throws: true }, + { waiters: 3, throws: false }, + { waiters: 3, throws: true }, + ])( + `preserves settlement without unhandled rejections ($waiters waiters, throws=$throws)`, + async ({ waiters, throws }) => { + const transportError = new Error(`transport failed`) + const observerError = new Error(`deduplication observer failed`) + let resolve!: () => void + let reject!: (reason: unknown) => void + const loadSubset = vi.fn( + () => + new Promise((done, fail) => { + resolve = done + reject = fail + }), + ) + const onDeduplicate = vi.fn(() => { + if (throws) throw observerError + }) + const deduplicated = new DeduplicatedLoadSubset({ + loadSubset, + onDeduplicate, + }) + const unhandled: Array = [] + const recordUnhandled = (reason: unknown) => unhandled.push(reason) + process.on(`unhandledRejection`, recordUnhandled) + try { + const requests = Array.from({ length: waiters }, () => + deduplicated.loadSubset({ limit: 2 }), + ) + const settled = Promise.allSettled(requests) + expect(requests.every((request) => request === requests[0])).toBe( + true, + ) + expect(loadSubset).toHaveBeenCalledTimes(1) + expect(onDeduplicate).not.toHaveBeenCalled() + + if (outcome === `resolve`) resolve() + else reject(transportError) + + expect(await settled).toEqual( + Array.from({ length: waiters }, () => + outcome === `resolve` + ? { status: `fulfilled`, value: undefined } + : { status: `rejected`, reason: transportError }, + ), + ) + // Let the host report rejected detached observer promises too. + await new Promise((done) => setTimeout(done, 0)) + expect(onDeduplicate).toHaveBeenCalledTimes( + outcome === `resolve` ? waiters - 1 : 0, + ) + expect(unhandled).toEqual([]) + } finally { + process.off(`unhandledRejection`, recordUnhandled) + } }, - }, - ] - - // First call: age > 10, orderBy age asc, limit 10 - await deduplicated.loadSubset({ - where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - limit: 10, - }) - expect(callCount).toBe(1) + ) + }, + ) + + it(`gives independently abortable demands independent transports`, async () => { + const pending: Array<() => void> = [] + const signals: Array = [] + const loadSubset = vi.fn( + (options) => + new Promise((resolve) => { + signals.push(options.signal) + pending.push(resolve) + }), + ) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) + const firstOwner = new AbortController() + const secondOwner = new AbortController() - // Second call: age > 10, orderBy age asc, limit 20 (NOT a subset) - await deduplicated.loadSubset({ - where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - limit: 20, + const first = deduplicated.loadSubset({ + limit: 2, + signal: firstOwner.signal, }) - expect(callCount).toBe(2) // Should call - limit is larger - }) - - it(`should check limited calls against unlimited combined predicate`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, + const second = deduplicated.loadSubset({ + limit: 2, + signal: secondOwner.signal, }) - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] + expect(first).not.toBe(second) + expect(loadSubset).toHaveBeenCalledTimes(2) + expect(signals).toEqual([firstOwner.signal, secondOwner.signal]) - // First call: unlimited age > 10 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(1) - - // Second call: limited age > 20 with orderBy + limit - // Even though it has a limit, it's covered by the unlimited call - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(20)), - orderBy: orderBy1, - limit: 10, - }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not call - covered by unlimited + pending.forEach((resolve) => resolve()) + await Promise.all([first, second]) }) - it(`should ignore orderBy for unlimited calls`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) + it(`does not cache work that settles after its owner aborts`, async () => { + let resolve!: () => void + const loadSubset = vi + .fn() + .mockImplementationOnce( + () => new Promise((done) => (resolve = done)), + ) + .mockResolvedValue(undefined) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) + const owner = new AbortController() - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] + const first = deduplicated.loadSubset({ limit: 2, signal: owner.signal }) + owner.abort() + resolve() + await first + await deduplicated.loadSubset({ limit: 2 }) - // First call: unlimited with orderBy - await deduplicated.loadSubset({ - where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - }) - expect(callCount).toBe(1) - - // Second call: subset where, different orderBy, no limit - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(20)), - }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not call - orderBy ignored for unlimited + expect(loadSubset).toHaveBeenCalledTimes(2) }) - it(`should handle undefined where clauses`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) + it(`retries an exact demand after rejection`, async () => { + const loadSubset = vi + .fn() + .mockRejectedValueOnce(new Error(`offline`)) + .mockResolvedValueOnce(undefined) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - // First call: no where clause (all data) - await deduplicated.loadSubset({}) - expect(callCount).toBe(1) + await expect(deduplicated.loadSubset({ limit: 2 })).rejects.toThrow( + `offline`, + ) + await deduplicated.loadSubset({ limit: 2 }) - // Second call: with where clause (should be covered) - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(10)), - }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not call - all data already loaded + expect(loadSubset).toHaveBeenCalledTimes(2) }) - it(`should handle complex real-world scenario`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(options) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderBy1: OrderBy = [ - { - expression: ref(`createdAt`), - compareOptions: { - direction: `desc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - // Load all active users - await deduplicated.loadSubset({ where: eq(ref(`status`), val(`active`)) }) - expect(callCount).toBe(1) - - // Load top 10 active users by createdAt - const result1 = await deduplicated.loadSubset({ - where: eq(ref(`status`), val(`active`)), - orderBy: orderBy1, - limit: 10, - }) - expect(result1).toBe(true) // Covered by unlimited call - expect(callCount).toBe(1) + it(`erases completed and in-flight evidence on reset`, async () => { + const pending: Array<() => void> = [] + const loadSubset = vi.fn( + () => new Promise((resolve) => pending.push(resolve)), + ) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - // Load all inactive users - await deduplicated.loadSubset({ where: eq(ref(`status`), val(`inactive`)) }) - expect(callCount).toBe(2) + const stale = deduplicated.loadSubset({ limit: 2 }) + deduplicated.reset() + const fresh = deduplicated.loadSubset({ limit: 2 }) + expect(loadSubset).toHaveBeenCalledTimes(2) - // Load top 5 inactive users - const result2 = await deduplicated.loadSubset({ - where: eq(ref(`status`), val(`inactive`)), - orderBy: orderBy1, - limit: 5, - }) - expect(result2).toBe(true) // Covered by unlimited inactive call - expect(callCount).toBe(2) + pending[0]!() + await stale + expect(deduplicated.loadSubset({ limit: 2 })).toBe(fresh) - // Verify only 2 actual calls were made - expect(calls).toHaveLength(2) - expect(calls[0]).toEqual({ where: eq(ref(`status`), val(`active`)) }) - expect(calls[1]).toEqual({ where: eq(ref(`status`), val(`inactive`)) }) + pending[1]!() + await fresh + expect(deduplicated.loadSubset({ limit: 2 })).toBe(true) }) - describe(`subset deduplication with minusWherePredicates`, () => { - it(`should request only the difference for range predicates`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: age > 20 (loads data for age > 20) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) - expect(calls[0]).toEqual({ where: gt(ref(`age`), val(20)) }) - - // Second call: age > 10 (should request only age > 10 AND age <= 20) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: and(gt(ref(`age`), val(10)), lte(ref(`age`), val(20))), - }) - }) - - it(`should request only the difference for set predicates`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: status IN ['B', 'C'] (loads data for B and C) - await deduplicated.loadSubset({ - where: inOp(ref(`status`), [`B`, `C`]), - }) - expect(callCount).toBe(1) - expect(calls[0]).toEqual({ where: inOp(ref(`status`), [`B`, `C`]) }) - - // Second call: status IN ['A', 'B', 'C', 'D'] (should request only A and D) - await deduplicated.loadSubset({ - where: inOp(ref(`status`), [`A`, `B`, `C`, `D`]), - }) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: inOp(ref(`status`), [`A`, `D`]), - }) - }) - - it(`should return true immediately for complete overlap`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, + it(`does not retain synchronous work from before a reentrant reset`, () => { + const loadSubset = vi + .fn() + .mockImplementationOnce(() => { + deduplicated.reset() + return true }) + .mockReturnValue(true) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - // First call: age > 10 (loads data for age > 10) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(1) + expect(deduplicated.loadSubset({ limit: 2 })).toBe(true) + expect(deduplicated.loadSubset({ limit: 2 })).toBe(true) + expect(loadSubset).toHaveBeenCalledTimes(2) + }) - // Second call: age > 20 (completely covered by first call) - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(20)), + it(`does not retain asynchronous work from before a reentrant reset`, async () => { + let resolveStale!: () => void + const loadSubset = vi + .fn() + .mockImplementationOnce(() => { + deduplicated.reset() + return new Promise((resolve) => (resolveStale = resolve)) }) - expect(result).toBe(true) - expect(callCount).toBe(1) // Should not make additional call - }) - - it(`should handle complex predicate differences`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } + .mockResolvedValue(undefined) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) + const stale = deduplicated.loadSubset({ limit: 2 }) + const fresh = deduplicated.loadSubset({ limit: 2 }) + expect(loadSubset).toHaveBeenCalledTimes(2) - // First call: age > 20 AND status = 'active' - const firstPredicate = and( - gt(ref(`age`), val(20)), - eq(ref(`status`), val(`active`)), - ) - await deduplicated.loadSubset({ where: firstPredicate }) - expect(callCount).toBe(1) - expect(calls[0]).toEqual({ where: firstPredicate }) + resolveStale() + await Promise.all([stale, fresh]) + }) - // Second call: age > 10 AND status = 'active' (should request only age > 10 AND age <= 20 AND status = 'active') - const secondPredicate = and( - gt(ref(`age`), val(10)), - eq(ref(`status`), val(`active`)), + it.each([ + { + name: `Date`, + value: new Date(7), + equal: new Date(7), + different: new Date(8), + }, + { + name: `binary`, + value: new Uint8Array([1]), + equal: new Uint8Array([1]), + different: new Uint8Array([2]), + }, + { + name: `Buffer`, + value: Buffer.from([1]), + equal: new Uint8Array([1]), + different: Buffer.from([2]), + }, + ])( + `passes immutable $name values through and deduplicates by equality`, + ({ value, equal, different }) => { + const loadSubset = vi.fn().mockReturnValue(true) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) + const options = { where: eq(ref(`key`), val(value)) } + deduplicated.loadSubset(options) + expect(loadSubset.mock.calls[0]![0]).toBe(options) + const matches = compileSingleRowExpression( + loadSubset.mock.calls[0]![0].where!, ) + expect([value, equal, different].map((key) => matches({ key }))).toEqual([ + true, + true, + false, + ]) + expect( + deduplicated.loadSubset({ where: eq(ref(`key`), val(equal)) }), + ).toBe(true) + expect(loadSubset).toHaveBeenCalledTimes(1) + deduplicated.loadSubset({ where: eq(ref(`key`), val(different)) }) + expect(loadSubset).toHaveBeenCalledTimes(2) + }, + ) - await deduplicated.loadSubset({ where: secondPredicate }) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: and( - eq(ref(`status`), val(`active`)), - gt(ref(`age`), val(10)), - lte(ref(`age`), val(20)), - ), - }) - }) - - it(`should not apply subset logic to limited calls`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderBy1: OrderBy = [ + it(`keeps immutable order and cursor data with its opaque identity`, () => { + const opaque = Object.freeze({ id: 1 }) + const options: LoadSubsetOptions = { + orderBy: [ { - expression: ref(`age`), + expression: ref(`rank`), compareOptions: { direction: `asc`, - nulls: `last`, - stringSort: `lexical`, + nulls: `first`, + stringSort: `locale`, + localeOptions: Object.freeze({ numeric: true }), }, }, - ] - - // First call: unlimited age > 20 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) - - // Second call: limited age > 10 with orderBy + limit - // Should request the full predicate, not the difference, because it's limited - await deduplicated.loadSubset({ - where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - limit: 10, - }) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: gt(ref(`age`), val(10)), - orderBy: orderBy1, - limit: 10, - }) - }) - - it(`should handle undefined where clauses in subset logic`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: age > 20 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) - - // Second call: no where clause (all data) - // Should request all data except what we already loaded - // i.e. should request NOT (age > 20) - await deduplicated.loadSubset({}) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ where: not(gt(ref(`age`), val(20))) }) - - // After loading all data, subsequent calls should be deduplicated - const result = await deduplicated.loadSubset({ - where: gt(ref(`age`), val(5)), - }) - expect(result).toBe(true) - expect(callCount).toBe(2) - }) - - describe(`hasLoadedAllData after loading filtered + unfiltered data`, () => { - it(`should set hasLoadedAllData after a filtered load followed by an unfiltered load`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: inOp(ref(`task_id`), [`id1`, `id2`, `id3`]), - }) - expect(callCount).toBe(1) - - await deduplicated.loadSubset({}) - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: not(inOp(ref(`task_id`), [`id1`, `id2`, `id3`])), - }) - - const result = await deduplicated.loadSubset({}) - expect(result).toBe(true) - expect(callCount).toBe(2) - }) - - it(`should set hasLoadedAllData after a filtered load followed by an unfiltered load (with eq)`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`single-id`)), - }) - expect(callCount).toBe(1) - - await deduplicated.loadSubset({}) - expect(callCount).toBe(2) - - const result1 = await deduplicated.loadSubset({}) - expect(result1).toBe(true) - expect(callCount).toBe(2) - - const result2 = await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`other-id`)), - }) - expect(result2).toBe(true) - expect(callCount).toBe(2) - }) - - it(`should not produce exponentially growing predicates on repeated unfiltered loads`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: inOp(ref(`task_id`), [`id1`, `id2`, `id3`]), - }) - expect(callCount).toBe(1) - - await deduplicated.loadSubset({}) - expect(callCount).toBe(2) - - const rounds: Array<{ round: number; whereSize: number }> = [] - for (let i = 0; i < 10; i++) { - const result = await deduplicated.loadSubset({}) - if (result !== true) { - const whereJson = JSON.stringify(calls[calls.length - 1]?.where) - rounds.push({ round: i + 1, whereSize: whereJson.length }) - } - } - - expect(callCount).toBe(2) - expect(rounds).toEqual([]) - }) - }) - - it(`should mark all data as loaded after a narrowed all-data request`, async () => { - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-1`)), - }) - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-2`)), - }) - - await deduplicated.loadSubset({}) - - expect(calls[2]).toEqual({ - where: not(inOp(ref(`task_id`), [`uuid-1`, `uuid-2`])), - }) - - expect((deduplicated as any).hasLoadedAllData).toBe(true) - expect((deduplicated as any).unlimitedWhere).toBeUndefined() - }) - - it(`should not keep issuing increasingly nested all-data predicates`, async () => { - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-1`)), - }) - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-2`)), - }) - - await deduplicated.loadSubset({}) - await deduplicated.loadSubset({}) - - expect(calls[3]).toBeUndefined() - }) - - it(`should deduplicate identical all-data requests while a narrowed all-data request is in flight`, async () => { - let resolveAllDataLoad: (() => void) | undefined - let callCount = 0 - const calls: Array = [] - const allDataLoadPromise = new Promise((resolve) => { - resolveAllDataLoad = resolve - }) - - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - - if (callCount === 2) { - return allDataLoadPromise - } - - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-1`)), - }) - - const firstAllDataLoad = deduplicated.loadSubset({}) - const secondAllDataLoad = deduplicated.loadSubset({}) - - expect(callCount).toBe(2) - expect(calls[1]).toEqual({ - where: not(eq(ref(`task_id`), val(`uuid-1`))), - }) - expect(secondAllDataLoad).toBe(firstAllDataLoad) - - resolveAllDataLoad?.() - await firstAllDataLoad - await secondAllDataLoad - }) - - it(`should not produce unbounded WHERE expressions when loading all data after eq accumulation`, async () => { - // This test reproduces the production bug where accumulating many eq predicates - // and then loading all data (no WHERE clause) caused unboundedly growing - // expressions instead of correctly setting hasLoadedAllData=true. - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // Simulate visiting multiple tasks, each adding an eq predicate - for (let i = 0; i < 10; i++) { - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-${i}`)), - }) - } - // After 10 eq calls, unlimitedWhere should be IN(task_id, [uuid-0, ..., uuid-9]) - expect(callCount).toBe(10) - - // Now load all data (no WHERE clause) - // This should send NOT(IN(...)) to the backend but track as "all data loaded" - await deduplicated.loadSubset({}) - expect(callCount).toBe(11) - - // The load request should be NOT(IN(task_id, [all accumulated uuids])) - const loadWhere = calls[10]!.where as any - expect(loadWhere.name).toBe(`not`) - expect(loadWhere.args[0].name).toBe(`in`) - expect(loadWhere.args[0].args[0].path).toEqual([`task_id`]) - const loadedUuids = ( - loadWhere.args[0].args[1].value as Array - ).sort() - const expectedUuids = Array.from( - { length: 10 }, - (_, i) => `uuid-${i}`, - ).sort() - expect(loadedUuids).toEqual(expectedUuids) - - // Critical: after loading all data, subsequent requests should be deduplicated - const result1 = await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-999`)), - }) - expect(result1).toBe(true) // Covered by "all data" load - expect(callCount).toBe(11) // No additional call - - // Loading all data again should also be deduplicated - const result2 = await deduplicated.loadSubset({}) - expect(result2).toBe(true) - expect(callCount).toBe(11) // Still no additional call - }) - - it(`should not produce unbounded WHERE expressions with synchronous loadSubset`, () => { - // Same scenario as the async accumulation test, but with a sync mock - // to exercise the sync return path (line 150 of subset-dedupe.ts) - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return true as const - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // Accumulate eq predicates via sync returns - for (let i = 0; i < 10; i++) { - deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-${i}`)), - }) - } - expect(callCount).toBe(10) - - // Load all data (no WHERE clause) — should track as "all data loaded" - deduplicated.loadSubset({}) - expect(callCount).toBe(11) - - // Subsequent requests should be deduplicated - const result1 = deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-999`)), - }) - expect(result1).toBe(true) - expect(callCount).toBe(11) - - const result2 = deduplicated.loadSubset({}) - expect(result2).toBe(true) - expect(callCount).toBe(11) - }) - - it(`should handle multiple all-data loads without expression growth`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First: load some specific data - await deduplicated.loadSubset({ - where: eq(ref(`task_id`), val(`uuid-1`)), - }) - expect(callCount).toBe(1) - - // Load all data (first time) - await deduplicated.loadSubset({}) - expect(callCount).toBe(2) - - // Load all data (second time) - should be deduplicated since we already have everything - const result = await deduplicated.loadSubset({}) - expect(result).toBe(true) - expect(callCount).toBe(2) // No additional call - all data already loaded - }) - - it(`should handle multiple overlapping unlimited calls`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(cloneOptions(options)) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - // First call: age > 20 - await deduplicated.loadSubset({ where: gt(ref(`age`), val(20)) }) - expect(callCount).toBe(1) - - // Second call: age < 10 (different range) - await deduplicated.loadSubset({ where: lt(ref(`age`), val(10)) }) - expect(callCount).toBe(2) - - // Third call: age > 5 (should request only age >= 10 AND age <= 20, since age < 10 is already covered) - await deduplicated.loadSubset({ where: gt(ref(`age`), val(5)) }) - expect(callCount).toBe(3) - - // Ideally it would be smart enough to optimize it to request only age >= 10 AND age <= 20, since age < 10 is already covered - // However, it doesn't do that currently, so it will not optimize and execute the original query - expect(calls[2]).toEqual({ - where: gt(ref(`age`), val(5)), - }) - - /* - expect(calls[2]).toEqual({ - where: and(gte(ref(`age`), val(10)), lte(ref(`age`), val(20))), - }) - */ - }) + ], + cursor: { + whereFrom: gt(ref(`rank`), val(opaque)), + whereCurrent: eq(ref(`rank`), val(opaque)), + }, + } + const request = captureRequest(options) + expect(request).toBe(options) + expect(((request.cursor!.whereFrom as Func).args[1] as Value).value).toBe( + opaque, + ) + expect( + compileSingleRowExpression(request.cursor!.whereCurrent)({ + rank: opaque, + }), + ).toBe(true) + expect( + compileSingleRowExpression(request.cursor!.whereCurrent)({ + rank: { id: 1 }, + }), + ).toBe(false) }) - describe(`onDeduplicate callback`, () => { - it(`should call onDeduplicate when all data already loaded`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const onDeduplicate = vi.fn() - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - onDeduplicate, - }) - - // Load all data - await deduplicated.loadSubset({}) - expect(callCount).toBe(1) - - // Any subsequent request should be deduplicated - const subsetOptions = { where: gt(ref(`age`), val(10)) } - const result = await deduplicated.loadSubset(subsetOptions) - expect(result).toBe(true) - expect(callCount).toBe(1) - expect(onDeduplicate).toHaveBeenCalledTimes(1) - expect(onDeduplicate).toHaveBeenCalledWith(subsetOptions) - }) - - it(`should call onDeduplicate when unlimited superset already loaded`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const onDeduplicate = vi.fn() - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - onDeduplicate: onDeduplicate, - }) - - // First call loads a broader set - await deduplicated.loadSubset({ where: gt(ref(`age`), val(10)) }) - expect(callCount).toBe(1) - - // Second call is a subset of the first; should dedupe and call callback - const subsetOptions = { where: gt(ref(`age`), val(20)) } - const result = await deduplicated.loadSubset(subsetOptions) - expect(result).toBe(true) - expect(callCount).toBe(1) - expect(onDeduplicate).toHaveBeenCalledTimes(1) - expect(onDeduplicate).toHaveBeenCalledWith(subsetOptions) - }) - - it(`should call onDeduplicate for limited subset requests`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const onDeduplicate = vi.fn() - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - onDeduplicate, - }) - - const orderBy1: OrderBy = [ - { - expression: ref(`age`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - const whereClause = gt(ref(`age`), val(10)) - - // First limited call - await deduplicated.loadSubset({ - where: whereClause, - orderBy: orderBy1, - limit: 10, - }) - expect(callCount).toBe(1) - - // Second limited call is a subset (SAME where clause and smaller limit) - // For limited queries, where clauses must be EQUAL for subset relationship - const subsetOptions = { - where: whereClause, // Same where clause - orderBy: orderBy1, - limit: 5, - } - const result = await deduplicated.loadSubset(subsetOptions) - expect(result).toBe(true) - expect(callCount).toBe(1) - expect(onDeduplicate).toHaveBeenCalledTimes(1) - expect(onDeduplicate).toHaveBeenCalledWith(subsetOptions) + it(`keeps completed cursor requests distinct from replacement Date constants`, async () => { + const loadSubset = vi.fn().mockResolvedValue(undefined) + const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) + const request = (year: number): LoadSubsetOptions => ({ + cursor: { + whereFrom: gt(ref(`createdAt`), val(new Date(year, 0))), + whereCurrent: eq(ref(`createdAt`), val(new Date(year, 0))), + }, + limit: 10, }) + await deduplicated.loadSubset(request(2025)) + await deduplicated.loadSubset(request(2026)) + expect(deduplicated.loadSubset(request(2025))).toBe(true) + expect(loadSubset).toHaveBeenCalledTimes(2) + }) - it(`should delay onDeduplicate until covering in-flight request completes`, async () => { - let resolveFirst: (() => void) | undefined - let callCount = 0 - const firstPromise = new Promise((resolve) => { - resolveFirst = () => resolve() - }) + it(`does not substitute comparison payloads with custom instance methods`, () => { + const date = new Date(2) + const bytes = new Uint8Array([1, 2, 3]) + Object.defineProperty(date, `getTime`, { value: () => 1 }) + Object.defineProperty(bytes, `slice`, { value: () => bytes }) + const where = new Func(`and`, [ + eq(ref(`date`), val(date)), + eq(ref(`bytes`), val(bytes)), + ]) + const request = captureRequest({ where }) + const rows = [ + { date, bytes }, + { date: new Date(2), bytes: new Uint8Array([1, 2, 3]) }, + ] + expect(request.where).toBe(where) + expect(rows.map(compileSingleRowExpression(request.where!))).toEqual( + rows.map(compileSingleRowExpression(where)), + ) + }) - // First call will remain in-flight until we resolve it - let first = true - const mockLoadSubset = (_options: LoadSubsetOptions) => { - callCount++ - if (first) { - first = false - return firstPromise + describe.each([`Date`, `Uint8Array`] as const)( + `request transport preserves %s predicate matches`, + (type) => { + it.each([`local`, `foreign`] as const)(`in the %s realm`, (realm) => { + const local = type === `Date` ? new Date(2) : new Uint8Array([1, 2]) + const foreign: unknown = runInNewContext( + type === `Date` ? `new Date(2)` : `new Uint8Array([1, 2])`, + ) + const value = realm === `local` ? local : foreign + for (const where of [ + eq(ref(`value`), val(value)), + new Func(`in`, [ref(`value`), val([value])]), + ]) { + const request = captureRequest({ where }) + const matches = compileSingleRowExpression(request.where!) + expect( + [foreign, local].map((item) => matches({ value: item })), + ).toEqual(realm === `foreign` ? [true, false] : [false, true]) } - return Promise.resolve() - } - - const onDeduplicate = vi.fn() - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - onDeduplicate: onDeduplicate, }) - - // Start a broad in-flight request - const inflightOptions = { where: gt(ref(`age`), val(10)) } - const inflight = deduplicated.loadSubset(inflightOptions) - expect(inflight).toBeInstanceOf(Promise) - expect(callCount).toBe(1) - - // Issue a subset request while first is still in-flight - const subsetOptions = { where: gt(ref(`age`), val(20)) } - const subsetPromise = deduplicated.loadSubset(subsetOptions) - expect(subsetPromise).toBeInstanceOf(Promise) - - // onDeduplicate should NOT have fired yet - expect(onDeduplicate).not.toHaveBeenCalled() - - // Complete the first request - resolveFirst?.() - - // Wait for the subset promise to settle (which chains the first) - await subsetPromise - - // Now the callback should have been called exactly once, with the subset options - expect(onDeduplicate).toHaveBeenCalledTimes(1) - expect(onDeduplicate).toHaveBeenCalledWith(subsetOptions) - }) - - it(`reports a signal-bearing request deduplicated after shared work completes`, async () => { - const pending: Array<() => void> = [] - let sharedSignal: AbortSignal | undefined - const loadSubset = vi.fn( - (options: LoadSubsetOptions) => - new Promise((resolve) => { - sharedSignal = options.signal - pending.push(resolve) - }), - ) - const onDeduplicate = vi.fn() - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset, - onDeduplicate, - }) - const firstController = new AbortController() - const secondController = new AbortController() - - const first = deduplicated.loadSubset({ - where: gt(ref(`age`), val(10)), - signal: firstController.signal, - }) - const secondOptions = { - where: gt(ref(`age`), val(20)), - signal: secondController.signal, - } - const second = deduplicated.loadSubset(secondOptions) - - expect(loadSubset).toHaveBeenCalledTimes(1) - expect(second).toBe(first) - - firstController.abort() - expect(sharedSignal?.aborted).toBe(false) - for (const resolve of pending) resolve() - await Promise.all([first, second]) - expect(onDeduplicate).toHaveBeenCalledTimes(1) - expect(onDeduplicate).toHaveBeenCalledWith(secondOptions) - }) + }, + ) + + it.each([`coalesce`, `caseWhen`] as const)( + `preserves membership results through %s`, + (wrapper) => { + const candidates = Object.freeze([new Uint8Array([1])]) + const expression = + wrapper === `coalesce` + ? new Func(`coalesce`, [val(candidates)]) + : new Func(`caseWhen`, [val(true), val(candidates), val([])]) + const request = captureRequest({ + where: new Func(`in`, [ref(`token`), expression]), + }) + const matches = compileSingleRowExpression(request.where!) + expect( + [1, 2, 3].map((n) => matches({ token: new Uint8Array([n]) })), + ).toEqual([true, false, false]) + expect(candidates).toEqual([new Uint8Array([1])]) + }, + ) + + it(`preserves immutable array ordering operands`, () => { + const boundary = Object.freeze([1, Object.freeze([2])]) + const request = captureRequest({ where: gt(ref(`tuple`), val(boundary)) }) + const matches = compileSingleRowExpression(request.where!) + expect( + [ + [1, [1]], + [1, [2]], + [1, [3]], + ].map((tuple) => matches({ tuple })), + ).toEqual([false, false, true]) }) - describe(`limited queries with different where clauses`, () => { - // When a query has a limit, only the top N rows (by orderBy) are loaded. - // A subsequent query with a different where clause cannot reuse that data, - // even if the new where clause is "more restrictive", because the filtered - // top N might include rows outside the original unfiltered top N. - - it(`should NOT dedupe when where clause differs on limited queries`, async () => { - let callCount = 0 - const calls: Array = [] - const mockLoadSubset = (options: LoadSubsetOptions) => { - callCount++ - calls.push(options) - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderByCreatedAt: OrderBy = [ - { - expression: ref(`created_at`), - compareOptions: { - direction: `desc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - // First query: top 10 items with no filter - await deduplicated.loadSubset({ - where: undefined, - orderBy: orderByCreatedAt, - limit: 10, - }) - expect(callCount).toBe(1) - - // Second query: top 10 items WITH a filter - // This requires a separate request because the filtered top 10 - // might include items outside the unfiltered top 10 - const searchWhere = and(eq(ref(`title`), val(`test`))) - await deduplicated.loadSubset({ - where: searchWhere, - orderBy: orderByCreatedAt, - limit: 10, - }) - - expect(callCount).toBe(2) - expect(calls[1]?.where).toEqual(searchWhere) - }) - - it(`should dedupe when where clause is identical on limited queries`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const orderByCreatedAt: OrderBy = [ - { - expression: ref(`created_at`), - compareOptions: { - direction: `desc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - // First query: top 10 items with no filter - await deduplicated.loadSubset({ - where: undefined, - orderBy: orderByCreatedAt, - limit: 10, - }) - expect(callCount).toBe(1) - - // Second query: same where clause (undefined), smaller limit - // The top 5 are contained within the already-loaded top 10 - const result = await deduplicated.loadSubset({ - where: undefined, - orderBy: orderByCreatedAt, - limit: 5, - }) - expect(result).toBe(true) - expect(callCount).toBe(1) - }) - - it(`should not let caller mutations change stored limited call orderBy`, async () => { - let callCount = 0 - const mockLoadSubset = () => { - callCount++ - return Promise.resolve() - } - - const deduplicated = new DeduplicatedLoadSubset({ - loadSubset: mockLoadSubset, - }) - - const mutableOrderBy: OrderBy = [ - { - expression: ref(`created_at`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - await deduplicated.loadSubset({ - where: eq(ref(`status`), val(`active`)), - orderBy: mutableOrderBy, - limit: 10, - }) - expect(callCount).toBe(1) - - mutableOrderBy[0]!.compareOptions.direction = `desc` - - const originalOrderBy: OrderBy = [ - { - expression: ref(`created_at`), - compareOptions: { - direction: `asc`, - nulls: `last`, - stringSort: `lexical`, - }, - }, - ] - - const result = await deduplicated.loadSubset({ - where: eq(ref(`status`), val(`active`)), - orderBy: originalOrderBy, - limit: 5, - }) - - expect(result).toBe(true) - expect(callCount).toBe(1) - }) - - it(`does not let caller mutations change a stored cursor boundary`, async () => { - const loadSubset = vi.fn().mockResolvedValue(undefined) - const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - const mutableBoundary = val(1) - const firstCursor = { - whereFrom: gt(ref(`id`), mutableBoundary), - whereCurrent: eq(ref(`id`), mutableBoundary), - lastKey: 1, - } - - await deduplicated.loadSubset({ cursor: firstCursor, limit: 10 }) - mutableBoundary.value = 2 - - await deduplicated.loadSubset({ - cursor: { - whereFrom: gt(ref(`id`), val(2)), - whereCurrent: eq(ref(`id`), val(2)), - lastKey: 1, - }, - limit: 10, - }) - - expect(loadSubset).toHaveBeenCalledTimes(2) - }) - - it(`does not let Date mutation change a stored cursor boundary`, async () => { - const loadSubset = vi.fn().mockResolvedValue(undefined) - const deduplicated = new DeduplicatedLoadSubset({ loadSubset }) - const mutableBoundary = new Date(`2025-01-01T00:00:00.000Z`) - - await deduplicated.loadSubset({ - cursor: { - whereFrom: gt(ref(`createdAt`), val(mutableBoundary)), - whereCurrent: eq(ref(`createdAt`), val(mutableBoundary)), - lastKey: 1, - }, - limit: 10, - }) - mutableBoundary.setUTCFullYear(2026) + it.each([`in`, `gt`])(`preserves immutable sparse %s array data`, (name) => { + const values = new Array(3) + values[1] = new Date(7) + Object.freeze(values) + const request = captureRequest({ + where: new Func(name, [ref(`value`), val(values)]), + }) + const payload = ((request.where as Func).args[1] as Value>) + .value + expect(payload).toBe(values) + expect(payload.length).toBe(3) + expect(Object.hasOwn(payload, 0)).toBe(false) + expect(Object.hasOwn(payload, 2)).toBe(false) + expect(payload[1]!.getTime()).toBe(7) + }) - await deduplicated.loadSubset({ - cursor: { - whereFrom: gt( - ref(`createdAt`), - val(new Date(`2026-01-01T00:00:00.000Z`)), - ), - whereCurrent: eq( - ref(`createdAt`), - val(new Date(`2026-01-01T00:00:00.000Z`)), - ), - lastKey: 1, - }, - limit: 10, - }) + it.each([`in`, `gt`])( + `preserves nested-array comparison semantics for %s`, + (name) => { + const nested = [2] + const values = Object.freeze([nested]) + const request = captureRequest({ + where: new Func(name, [ref(`value`), val(values)]), + }) + const matches = compileSingleRowExpression(request.where!) + const rows = name === `in` ? [nested, [2]] : [[[1]], [[2]], [[3]]] + expect(rows.map((value) => matches({ value }))).toEqual( + name === `in` ? [true, false] : [false, false, true], + ) + }, + ) +}) - expect(loadSubset).toHaveBeenCalledTimes(2) - }) +function captureRequest(options: LoadSubsetOptions): LoadSubsetOptions { + let request!: LoadSubsetOptions + const deduplicated = new DeduplicatedLoadSubset({ + loadSubset: (value) => { + request = value + return true + }, }) -}) + deduplicated.loadSubset(options) + return request +} diff --git a/packages/db/tests/query/subset-error-matrix.test.ts b/packages/db/tests/query/subset-error-matrix.test.ts index c161f24657..6ee34cf2fc 100644 --- a/packages/db/tests/query/subset-error-matrix.test.ts +++ b/packages/db/tests/query/subset-error-matrix.test.ts @@ -2,12 +2,14 @@ import { describe, expect, it } from 'vitest' import { createCollection } from '../../src/collection/index.js' import { BTreeIndex } from '../../src/indexes/btree-index.js' import { createEffect, createLiveQueryCollection, eq } from '../../src/index.js' +import { getLoadSubsetDemandKey } from '../../src/query/ir-stable-identity.js' import { mockSyncCollectionOptions } from '../utils.js' type Delivery = `throw` | `reject` type Consumer = `effect` | `live` type StartupPath = `direct` | `ordered` | `lazy` type IncrementalPath = Exclude +type FailureValue = `error` | `nan` | `undefined` type Row = { id: number @@ -22,6 +24,16 @@ type FailureCase = { delivery: Delivery } +type IncrementalFailureCase = FailureCase & { + failureValue: FailureValue +} + +type CleanupFailureCase = { + name: string + consumer: Consumer + failure: unknown +} + const row: Row = { id: 1, rank: 1, parentId: 1 } // Every query form can fail while it acquires initial coverage. @@ -40,25 +52,42 @@ const startupCases: ReadonlyArray> = ( // Direct queries have no automatic later demand. Ordered refills and lazy // relationship routes do, so only those paths have incremental cells. -const incrementalCases: ReadonlyArray> = ( +const incrementalCases: ReadonlyArray = ( [`effect`, `live`] as const ).flatMap((consumer) => ([`ordered`, `lazy`] as const).flatMap((path) => - ([`throw`, `reject`] as const).map((delivery) => ({ - name: `${consumer} ${path} ${delivery}`, - consumer, - path, - delivery, - })), + ([`throw`, `reject`] as const).flatMap((delivery) => + ([`error`, `nan`, `undefined`] as const).map((failureValue) => ({ + name: `${consumer} ${path} ${delivery} ${failureValue}`, + consumer, + path, + delivery, + failureValue, + })), + ), ), ) -function fail(delivery: Delivery, error: Error): Promise { +const cleanupFailureObject = { kind: `cleanup-failure` } +const cleanupFailureCases: ReadonlyArray = ( + [`effect`, `live`] as const +).flatMap((consumer) => [ + { name: `${consumer} undefined`, consumer, failure: undefined }, + { name: `${consumer} NaN`, consumer, failure: Number.NaN }, + { name: `${consumer} object`, consumer, failure: cleanupFailureObject }, +]) + +function fail(delivery: Delivery, error: unknown): Promise { if (delivery === `throw`) throw error return Promise.reject(error) } -function createFailingSource(id: string, delivery: Delivery, error: Error) { +function createFailingSource( + id: string, + delivery: Delivery, + error: unknown, + onLoad = () => {}, +) { return createCollection({ id, getKey: (item) => item.id, @@ -69,7 +98,10 @@ function createFailingSource(id: string, delivery: Delivery, error: Error) { sync: ({ markReady }) => { markReady() return { - loadSubset: () => fail(delivery, error), + loadSubset: () => { + onLoad() + return fail(delivery, error) + }, } }, }, @@ -218,18 +250,26 @@ describe(`loadSubset failure matrix`, () => { it.each(incrementalCases)( `reports an incremental failure without escaping its source commit: $name`, - async ({ consumer, path, delivery }) => { - const error = new Error(`${consumer} ${path} incremental failed`) - const suffix = `${consumer}-${path}-${delivery}` + async ({ consumer, path, delivery, failureValue }) => { + const error: unknown = + failureValue === `nan` + ? Number.NaN + : failureValue === `undefined` + ? undefined + : new Error(`${consumer} ${path} incremental failed`) + const suffix = `${consumer}-${path}-${delivery}-${failureValue}` let triggerFailure: () => void let primary: RowCollection let child: RowCollection + let loadCount = 0 + const orderedLoadKeys: Array = [] + let loadsBeforeFailure = 0 + let failureArmed = false if (path === `ordered`) { let begin!: () => void let write!: (message: { type: `insert` | `delete`; value: Row }) => void let commit!: () => void - let loadCount = 0 primary = createCollection({ id: `failure-matrix-incremental-ordered-${suffix}`, getKey: (item) => item.id, @@ -243,9 +283,13 @@ describe(`loadSubset failure matrix`, () => { commit = params.commit params.markReady() return { - loadSubset: () => { + loadSubset: (options) => { loadCount++ - if (loadCount > 1) return fail(delivery, error) + orderedLoadKeys.push(getLoadSubsetDemandKey(options)) + if (failureArmed) return fail(delivery, error) + // Initial coverage includes tie-boundary refinement, not + // just the first page. Inject failure only after it settles. + if (loadCount > 1) return true begin() write({ type: `insert`, value: row }) commit() @@ -257,6 +301,8 @@ describe(`loadSubset failure matrix`, () => { }) child = primary triggerFailure = () => { + loadsBeforeFailure = orderedLoadKeys.length + failureArmed = true begin() write({ type: `delete`, value: row }) commit() @@ -270,6 +316,7 @@ describe(`loadSubset failure matrix`, () => { `failure-matrix-incremental-child-${suffix}`, delivery, error, + () => loadCount++, ) triggerFailure = () => { primary.utils.begin() @@ -283,10 +330,18 @@ describe(`loadSubset failure matrix`, () => { const sourceErrors: Array = [] const effect = startEffect(path, primary, child, sourceErrors) try { - triggerFailure() + await flushFailures() + expect(sourceErrors).toEqual([]) + expect(effect.disposed).toBe(false) + expect(() => triggerFailure()).not.toThrow() await flushFailures() - expect(sourceErrors).toEqual([error]) + expect(sourceErrors).toHaveLength(1) + if (failureValue === `error`) { + expect(sourceErrors[0]).toBe(error) + } else { + expect(sourceErrors[0]).toBeInstanceOf(Error) + } expect(effect.disposed).toBe(true) } finally { await effect.dispose() @@ -295,16 +350,31 @@ describe(`loadSubset failure matrix`, () => { const live = startLive(path, primary, child) try { await live.preload() - triggerFailure() + expect(live.status).toBe(`ready`) + expect(live.utils.lastSubsetError).toBeUndefined() + expect(() => triggerFailure()).not.toThrow() await flushFailures() expect(live.status).toBe(path === `lazy` ? `error` : `ready`) - expect(live.utils.lastSubsetError).toBe(error) + if (failureValue === `error`) { + expect(live.utils.lastSubsetError).toBe(error) + } else { + expect(live.utils.lastSubsetError).toBeInstanceOf(Error) + } } finally { await live.cleanup() } } + if (path === `ordered`) { + const incrementalKeys = orderedLoadKeys.slice(loadsBeforeFailure) + expect(loadsBeforeFailure).toBeGreaterThan(1) + expect(incrementalKeys.length).toBeGreaterThan(0) + expect(new Set(incrementalKeys).size).toBe(incrementalKeys.length) + } else { + expect(loadCount).toBe(1) + } + expect(primary.subscriberCount).toBe(0) if (path === `lazy`) expect(child.subscriberCount).toBe(0) } finally { @@ -316,4 +386,195 @@ describe(`loadSubset failure matrix`, () => { } }, ) + + it.each(cleanupFailureCases)( + `reports obsolete-demand cleanup failure without failing the source commit: $name`, + async ({ consumer, failure }) => { + const suffix = `${consumer}-${ + failure === undefined + ? `undefined` + : typeof failure === `number` + ? `nan` + : `object` + }` + const parent = createStaticSource(`cleanup-failure-parent-${suffix}`, [ + row, + ]) + let unloadCount = 0 + const child = createCollection({ + id: `cleanup-failure-child-${suffix}`, + getKey: (item) => item.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloadCount++ + if (unloadCount === 1) throw failure + }, + } + }, + }, + }) + const sourceErrors: Array = [] + const effect = + consumer === `effect` + ? createEffect({ + query: (q) => + q + .from({ item: parent }) + .leftJoin({ child }, ({ item, child: childRow }) => + eq(item.id, childRow.parentId), + ), + onBatch: () => {}, + onSourceError: (error) => sourceErrors.push(error), + }) + : undefined + const live = + consumer === `live` + ? createLiveQueryCollection((q) => + q + .from({ item: parent }) + .leftJoin({ child }, ({ item, child: childRow }) => + eq(item.id, childRow.parentId), + ), + ) + : undefined + + try { + if (live) await live.preload() + await flushFailures() + + expect(() => { + parent.utils.begin() + parent.utils.write({ type: `delete`, value: row }) + parent.utils.commit() + }).not.toThrow() + + await flushFailures() + + if (effect) { + expect(sourceErrors).toHaveLength(1) + expect(sourceErrors[0]?.message).toBe(String(failure)) + expect(effect.disposed).toBe(true) + } + if (live) { + expect(live.utils.lastSubsetError).toBeInstanceOf(Error) + expect(live.status).toBe(`ready`) + } + } finally { + if (effect) await effect.dispose() + if (live) await live.cleanup() + expect(unloadCount).toBe(1) + await Promise.all([parent.cleanup(), child.cleanup()]) + } + }, + ) + + it.each([undefined, NaN, new Error(`release failed`)])( + `does not repeat failed release after %s survives demand retirement`, + async (failure) => { + const parent = createStaticSource(`undefined-cleanup-retry-parent`, [row]) + let unloadCount = 0 + const child = createCollection({ + id: `undefined-cleanup-retry-child`, + getKey: (item) => item.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => true, + unloadSubset: () => { + unloadCount++ + if (unloadCount <= 2) throw failure + }, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ item: parent }) + .leftJoin({ child }, ({ item, child: childRow }) => + eq(item.id, childRow.parentId), + ), + ) + const originalQueueMicrotask = globalThis.queueMicrotask + const queuedMicrotasks: Array<() => void> = [] + + try { + await live.preload() + + parent.utils.begin() + parent.utils.write({ type: `delete`, value: row }) + parent.utils.commit() + await flushFailures() + + expect(unloadCount).toBe(1) + expect(live.utils.lastSubsetError).toBeInstanceOf(Error) + + globalThis.queueMicrotask = (callback) => { + queuedMicrotasks.push(callback) + } + await live.cleanup() + expect(unloadCount).toBe(1) + expect(queuedMicrotasks).toHaveLength(0) + await live.cleanup() + expect(unloadCount).toBe(1) + expect(parent.subscriberCount).toBe(0) + expect(child.subscriberCount).toBe(0) + await live.cleanup() + expect(unloadCount).toBe(1) + } finally { + globalThis.queueMicrotask = originalQueueMicrotask + await Promise.all([live.cleanup(), parent.cleanup(), child.cleanup()]) + } + }, + ) + + it(`preserves a synchronous ordered error after reentrant cleanup`, async () => { + const error = new Error(`ordered load failed after cleanup`) + let cleanupLive: () => Promise = () => Promise.resolve() + const source = createCollection({ + id: `ordered-reentrant-cleanup-error`, + getKey: ({ id }) => id, + syncMode: `on-demand`, + startSync: true, + autoIndex: `off`, + defaultIndexType: BTreeIndex, + sync: { + sync: ({ markReady }) => { + markReady() + return { + loadSubset: () => { + void cleanupLive() + throw error + }, + unloadSubset: () => {}, + } + }, + }, + }) + const live = createLiveQueryCollection((q) => + q + .from({ item: source }) + .orderBy(({ item }) => item.rank) + .limit(0), + ) + cleanupLive = () => live.cleanup() + + try { + await live.preload() + expect(() => live.utils.setWindow({ offset: 0, limit: 1 })).toThrow(error) + } finally { + await Promise.all([live.cleanup(), source.cleanup()]) + } + }) }) diff --git a/packages/db/tests/query/union-all.test.ts b/packages/db/tests/query/union-all.test.ts index a94308bb27..ce5f0faf51 100644 --- a/packages/db/tests/query/union-all.test.ts +++ b/packages/db/tests/query/union-all.test.ts @@ -18,6 +18,7 @@ import { } from '../utils.js' import { OnlyOneSourceAllowedError } from '../../src/errors.js' import type { LoadSubsetOptions } from '../../src/types.js' +import type { BasicExpression } from '../../src/query/ir.js' type Message = { id: number @@ -34,6 +35,18 @@ type ToolCall = { userId: number } +function referencesField( + expression: BasicExpression | undefined, + field: string, +): boolean { + if (!expression) return false + if (expression.type === `ref`) return expression.path.includes(field) + if (expression.type !== `func`) return false + return expression.args.some((argument) => + referencesField(argument as BasicExpression, field), + ) +} + type Chunk = { id: number messageId: number @@ -1234,7 +1247,9 @@ describe(`unionAll`, () => { expect(messageLoadSubsetCalls.length).toBeGreaterThan(0) expect(toolLoadSubsetCalls.length).toBeGreaterThan(0) expect( - messageLoadSubsetCalls.every((call) => call.where === undefined), + messageLoadSubsetCalls.every( + (call) => !referencesField(call.where, `userId`), + ), ).toBe(true) expect(toolLoadSubsetCalls.every((call) => call.where === undefined)).toBe( true, @@ -1291,7 +1306,9 @@ describe(`unionAll`, () => { expect(messageLoadSubsetCalls.length).toBeGreaterThan(0) expect(toolLoadSubsetCalls.length).toBeGreaterThan(0) expect( - messageLoadSubsetCalls.every((call) => call.where === undefined), + messageLoadSubsetCalls.every( + (call) => !referencesField(call.where, `userId`), + ), ).toBe(true) expect(toolLoadSubsetCalls.some((call) => call.where)).toBe(true) }) diff --git a/packages/db/tests/reference-expression.ts b/packages/db/tests/reference-expression.ts index 51716b901a..839a7de33b 100644 --- a/packages/db/tests/reference-expression.ts +++ b/packages/db/tests/reference-expression.ts @@ -2,13 +2,6 @@ import type { BasicExpression } from '../src/query/ir.js' function compareReferenceValues(left: unknown, right: unknown): number { if (left === right) return 0 - // Query order cursors use nulls-first ordering. Missing reference paths are - // equivalent to null so adapters can evaluate the same boundary independently. - const leftNullish = left === null || left === undefined - const rightNullish = right === null || right === undefined - if (leftNullish && rightNullish) return 0 - if (leftNullish) return -1 - if (rightNullish) return 1 if (typeof left === `number` && typeof right === `number`) { return left < right ? -1 : 1 } @@ -43,15 +36,24 @@ export function evaluateReferenceExpression( return args.some(Boolean) case `not`: return !args[0] + case `isNull`: + return args[0] === null + case `isUndefined`: + return args[0] === undefined case `eq`: + if (args[0] == null || args[1] == null) return null return args[0] === args[1] case `gt`: + if (args[0] == null || args[1] == null) return null return compareReferenceValues(args[0], args[1]) > 0 case `gte`: + if (args[0] == null || args[1] == null) return null return compareReferenceValues(args[0], args[1]) >= 0 case `lt`: + if (args[0] == null || args[1] == null) return null return compareReferenceValues(args[0], args[1]) < 0 case `lte`: + if (args[0] == null || args[1] == null) return null return compareReferenceValues(args[0], args[1]) <= 0 case `in`: if (!Array.isArray(args[1])) throw new Error(`IN requires an array`) diff --git a/packages/db/tests/replay-adapter-ownership.test.ts b/packages/db/tests/replay-adapter-ownership.test.ts new file mode 100644 index 0000000000..a2913cfd80 --- /dev/null +++ b/packages/db/tests/replay-adapter-ownership.test.ts @@ -0,0 +1,143 @@ +import { expect, it } from 'vitest' +import { createCollection } from '../src/collection' +import { createDeferred } from '../src/deferred' +import { flushPromises } from './utils' +import type { LoadSubsetOptions, SyncConfig } from '../src/types' + +type Row = { id: number; version: number } + +it.each( + [1, 2].flatMap((owners) => + ([`resolve`, `reject`, `throw`] as const).map((outcome) => ({ + owners, + outcome, + })), + ), +)( + `keeps replay adapter ownership balanced: %j`, + async ({ owners, outcome }) => { + const liveLeases = new Set() + const loads: Array = [] + const releases: Array = [] + const pending: Array>> = [] + const failure = new Error(`adapter startup failed`) + let generation = 1 + let starts = 0 + let stops = 0 + let sync!: Parameters[`sync`]>[0] + const source = createCollection({ + getKey: ({ id }) => id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + if (liveLeases.size === 0) starts++ + liveLeases.add(options) + operations.begin() + operations.write({ + type: source.has(1) ? `update` : `insert`, + value: { id: 1, version: generation }, + }) + operations.commit() + if (generation === 2 && outcome === `throw`) { + // The adapter, not unloadSubset, owns rollback of a throw. + liveLeases.delete(options) + if (liveLeases.size === 0) stops++ + throw failure + } + loads.push(options) + if (generation !== 2) return true + const result = createDeferred() + pending.push(result) + return result.promise + }, + unloadSubset: (options) => { + expect(liveLeases.delete(options)).toBe(true) + releases.push(options) + if (liveLeases.size === 0) stops++ + }, + } + }, + }, + }) + const views = Array.from( + { length: owners }, + () => new Map(), + ) + const subscriptions = views.map((view) => + source.subscribeChanges( + (changes) => { + for (const change of changes) { + if (change.type === `delete`) view.delete(change.key) + else view.set(change.key, change.value.version) + } + }, + { includeInitialState: false }, + ), + ) + try { + for (const subscription of subscriptions) subscription.requestSnapshot({}) + expect(liveLeases.size).toBe(owners) + expect(starts).toBe(1) + expect(stops).toBe(0) + generation = 2 + sync.begin() + sync.truncate() + sync.commit() + const waiters = Promise.allSettled( + subscriptions.map( + (subscription) => subscription.pendingTruncateReplacement, + ), + ) + await flushPromises() + for (const view of views) expect([...view.values()]).toEqual([1]) + // Distinct logical owners may share one adapter resource. Retiring one + // must never stop it while another successful owner still holds a lease. + if (owners === 2 && outcome !== `throw`) expect(stops).toBe(0) + if (owners === 1) { + expect(starts).toBe(2) + expect(stops).toBe(outcome === `throw` ? 2 : 1) + } + for (const result of pending) { + if (outcome === `reject`) result.reject(failure) + else result.resolve() + } + const settled = await waiters + expect(settled).toEqual( + Array.from({ length: owners }, () => + outcome === `resolve` + ? { status: `fulfilled`, value: undefined } + : { status: `rejected`, reason: failure }, + ), + ) + await flushPromises() + for (const view of views) + expect([...view.values()]).toEqual([outcome === `resolve` ? 2 : 1]) + + generation = 3 + sync.begin() + sync.truncate() + sync.commit() + await flushPromises() + for (const view of views) expect([...view.values()]).toEqual([3]) + expect(liveLeases.size).toBe(owners) + subscriptions[0]!.unsubscribe() + expect(liveLeases.size).toBe(owners - 1) + for (const subscription of subscriptions) subscription.unsubscribe() + expect(liveLeases.size).toBe(0) + expect(starts).toBe(stops) + expect(releases).toHaveLength(loads.length) + for (const options of loads) + expect( + releases.filter((released) => released === options), + ).toHaveLength(1) + } finally { + for (const result of pending) result.resolve() + for (const subscription of subscriptions) subscription.unsubscribe() + await source.cleanup() + } + }, +) diff --git a/packages/db/tests/replay-publication-storage.test.ts b/packages/db/tests/replay-publication-storage.test.ts new file mode 100644 index 0000000000..3be47ce29a --- /dev/null +++ b/packages/db/tests/replay-publication-storage.test.ts @@ -0,0 +1,335 @@ +import { describe, expect, it } from 'vitest' +import { createCollection } from '../src/collection' +import { createDeferred } from '../src/deferred' +import { BasicIndex } from '../src/indexes/basic-index' +import { createLiveQueryCollection, eq } from '../src/query' +import { PropRef } from '../src/query/ir' +import { evaluateReferenceExpression } from './reference-expression' +import { flushPromises } from './utils' +import type { Deferred } from '../src/deferred' +import type { ChangeMessage, LoadSubsetOptions, SyncConfig } from '../src/types' + +type Row = { id: number; version: number } +type Ops = Parameters[`sync`]>[0] +type Batch = Array<[string, string | number, number]> + +const idRef = () => new PropRef([`id`]) +const shape = (changes: Array>): Batch => + changes.map((c) => [c.type, c.key, c.value.version]) + +/** Retention witness: private replacement rows held per subscription. */ +function replaySessions(collection: unknown) { + const internals = collection as { + _changes: { + changeSubscriptions: Iterable<{ + options: { truncateReplayPublication?: unknown } + truncateReplaySession?: { privateRows?: ReadonlyMap } + }> + } + } + return [...internals._changes.changeSubscriptions].flatMap((s) => + s.truncateReplaySession + ? [ + { + delegated: Boolean(s.options.truncateReplayPublication), + privateRows: s.truncateReplaySession.privateRows?.size ?? null, + }, + ] + : [], + ) +} + +function makeSource(id: string) { + let version = 1 + let hold: Deferred | undefined + let sync!: Ops + const loads: Array = [] + const source = createCollection({ + id, + getKey: (row) => row.id, + syncMode: `on-demand`, + autoIndex: `eager`, + defaultIndexType: BasicIndex, + sync: { + sync: (operations) => { + sync = operations + operations.markReady() + return { + loadSubset: (options) => { + loads.push(options) + const ids = [1, 2, 3].filter( + (rowId) => + !options.where || + evaluateReferenceExpression(options.where, { + id: rowId, + version, + }), + ) + operations.begin() + for (const rowId of ids) { + operations.write({ + type: source.has(rowId) ? `update` : `insert`, + value: { id: rowId, version }, + }) + } + operations.commit() + return hold ? hold.promise : true + }, + unloadSubset: () => {}, + } + }, + }, + }) + return { + source, + loads, + get sync() { + return sync + }, + setVersion: (next: number) => { + version = next + }, + setHold: (next: Deferred | undefined) => { + hold = next + }, + truncate: () => { + sync.begin() + sync.truncate() + sync.commit() + }, + } +} + +describe(`Replay publication storage`, () => { + it(`direct: one replacement batch, healthy peers, late demand joins the barrier`, async () => { + const s = makeSource(`probe-direct`) + const batches: Array = [] + const sub = s.source.subscribeChanges( + (changes) => changes.length && batches.push(shape(changes)), + { includeInitialState: false }, + ) + sub.requestSnapshot({ where: eq(idRef(), 1), optimizedOnly: false }) + sub.requestSnapshot({ where: eq(idRef(), 2), optimizedOnly: false }) + // A demand-free peer sees every source delta immediately. + const peerBatches: Array = [] + const peer = s.source.subscribeChanges( + (changes) => changes.length && peerBatches.push(shape(changes)), + { includeInitialState: false }, + ) + // A query peer over the same source uses the delegated publication path. + const peerLive = createLiveQueryCollection((q) => + q.from({ row: s.source }).where(({ row }) => eq(row.id, 2)), + ) + await peerLive.preload() + expect(batches.flat()).toEqual([ + [`insert`, 1, 1], + [`insert`, 2, 1], + ]) + expect(peerLive.get(2)?.version).toBe(1) + batches.length = 0 + peerBatches.length = 0 + + s.setVersion(2) + const hold = createDeferred() + s.setHold(hold) + s.truncate() + const completion = sub.pendingTruncateReplacement + expect(completion).toBeInstanceOf(Promise) + await flushPromises() + expect(sub.status).toBe(`loadingSubset`) + // No flash of missing content for the direct subscriber. + expect(batches).toEqual([]) + // The query peer keeps its last complete result behind its own barrier. + expect(peerLive.get(2)?.version).toBe(1) + // The demand-free peer saw the truncate deletes and the reloads. + expect(peerBatches.flat().sort()).toEqual( + [ + [`delete`, 1, 1], + [`delete`, 2, 1], + [`insert`, 1, 2], + [`insert`, 2, 2], + ].sort(), + ) + + // Reentrant acquisition while the replay is open joins the barrier. + sub.requestSnapshot({ where: eq(idRef(), 3), optimizedOnly: false }) + expect(batches).toEqual([]) + const retention = replaySessions(s.source) + expect(retention).toContainEqual({ delegated: false, privateRows: 3 }) + expect(retention.filter((r) => r.delegated)).toHaveLength(1) + + hold.resolve() + await flushPromises() + await completion + expect(sub.status).toBe(`ready`) + expect(batches).toHaveLength(1) + expect(batches[0]!.sort()).toEqual([ + [`insert`, 3, 2], + [`update`, 1, 2], + [`update`, 2, 2], + ]) + expect(peerLive.get(2)?.version).toBe(2) + expect(replaySessions(s.source)).toEqual([]) + + // A later plain delta publishes normally. + s.sync.begin() + s.sync.write({ type: `update`, value: { id: 3, version: 5 } }) + s.sync.commit() + expect(batches.at(-1)).toEqual([[`update`, 3, 5]]) + + sub.unsubscribe() + peer.unsubscribe() + await peerLive.cleanup() + await s.source.cleanup() + }) + + it(`direct: releasing the last demand during replay retires it; re-acquisition reconciles`, async () => { + const s = makeSource(`probe-release`) + const visible = new Map() + const batches: Array = [] + const sub = s.source.subscribeChanges( + (changes) => { + changes.length && batches.push(shape(changes)) + for (const c of changes) { + if (c.type === `delete`) visible.delete(c.key) + else visible.set(c.key, c.value.version) + } + }, + { includeInitialState: false }, + ) + const where = eq(idRef(), 1) + sub.requestSnapshot({ where, optimizedOnly: false }) + expect(visible.get(1)).toBe(1) + + s.setVersion(2) + const hold = createDeferred() + s.setHold(hold) + s.truncate() + const settled = Promise.allSettled([sub.pendingTruncateReplacement]) + await flushPromises() + expect(sub.status).toBe(`loadingSubset`) + expect(s.source.get(1)?.version).toBe(2) + expect(visible.get(1)).toBe(1) + + sub.releaseSnapshot(where) + const [outcome] = await settled + expect(outcome.status).toBe(`rejected`) + expect((outcome as PromiseRejectedResult).reason.name).toBe(`AbortError`) + expect(sub.status).toBe(`ready`) + expect(sub.hasPendingTruncateReplacement).toBe(false) + expect(visible.get(1)).toBe(1) + expect(replaySessions(s.source)).toEqual([]) + + // Late settlement of the released transport changes nothing. + hold.resolve() + await flushPromises() + expect(visible.get(1)).toBe(1) + expect(sub.status).toBe(`ready`) + + // Re-acquiring reconciles the retained row against the source. + s.setHold(undefined) + s.setVersion(3) + sub.requestSnapshot({ where, optimizedOnly: false }) + await flushPromises() + expect(visible.get(1)).toBe(3) + expect(batches.at(-1)).toEqual([[`update`, 1, 3]]) + expect(sub.status).toBe(`ready`) + + sub.unsubscribe() + await s.source.cleanup() + }) + + it(`direct: on-demand restart reacquires demand behind one private batch`, async () => { + let loadCount = 0 + let ops!: Ops + const batches: Array = [] + const source = createCollection({ + id: `probe-restart-on-demand`, + getKey: (row) => row.id, + syncMode: `on-demand`, + sync: { + sync: (operations) => { + ops = operations + operations.markReady() + return { + loadSubset: () => { + loadCount++ + ops.begin() + ops.write({ + type: `insert`, + value: { id: 1, version: loadCount }, + }) + ops.commit() + return true + }, + unloadSubset: () => {}, + } + }, + }, + }) + const sub = source.subscribeChanges( + (changes) => changes.length && batches.push(shape(changes)), + { includeInitialState: false }, + ) + sub.requestSnapshot() + expect(batches).toEqual([[[`insert`, 1, 1]]]) + + await source.cleanup() + source.startSyncImmediate() + expect(sub.status).toBe(`loadingSubset`) + await flushPromises() + expect(loadCount).toBe(2) + expect(sub.status).toBe(`ready`) + expect(batches).toEqual([[[`insert`, 1, 1]], [[`update`, 1, 2]]]) + + sub.unsubscribe() + await source.cleanup() + }) + + it(`direct: eager restart reconciles retained rows on the next ready batch`, async () => { + let session = 0 + const batches: Array = [] + const source = createCollection({ + id: `probe-restart-eager`, + getKey: (row) => row.id, + sync: { + sync: (operations) => { + session++ + operations.begin() + const rows = + session === 1 + ? [ + { id: 1, version: 1 }, + { id: 2, version: 1 }, + ] + : [{ id: 1, version: 2 }] + for (const value of rows) operations.write({ type: `insert`, value }) + operations.commit() + operations.markReady() + }, + }, + }) + const sub = source.subscribeChanges( + (changes) => changes.length && batches.push(shape(changes)), + { includeInitialState: true }, + ) + expect(batches.flat().sort()).toEqual([ + [`insert`, 1, 1], + [`insert`, 2, 1], + ]) + batches.length = 0 + + await source.cleanup() + source.startSyncImmediate() + await flushPromises() + expect(batches).toHaveLength(1) + expect(batches[0]!.sort()).toEqual([ + [`delete`, 2, 1], + [`update`, 1, 2], + ]) + expect(sub.status).toBe(`ready`) + + sub.unsubscribe() + await source.cleanup() + }) +}) diff --git a/packages/db/tests/transactions.test.ts b/packages/db/tests/transactions.test.ts index d77e196005..d0f70a890a 100644 --- a/packages/db/tests/transactions.test.ts +++ b/packages/db/tests/transactions.test.ts @@ -10,6 +10,51 @@ import { } from '../src/errors' describe(`Transactions`, () => { + it.each([ + { + name: `Error`, + reason: new Error(`mutation failed`), + message: `mutation failed`, + }, + { name: `string`, reason: `mutation failed`, message: `mutation failed` }, + { + name: `unprintable object`, + reason: { + toString() { + throw new Error(`cannot stringify`) + }, + }, + message: `Unknown error`, + }, + ])( + `rolls back a mutation rejected with an $name`, + async ({ reason, message }) => { + const collection = createCollection<{ id: number }>({ + getKey: (row) => row.id, + sync: { sync: () => {} }, + }) + const transaction = createTransaction({ + autoCommit: false, + mutationFn: () => Promise.reject(reason), + }) + const persisted = transaction.isPersisted.promise.catch( + (error: unknown) => error, + ) + try { + transaction.mutate(() => collection.insert({ id: 1 })) + await expect(transaction.commit()).rejects.toThrow(message) + expect(transaction.state).toBe(`failed`) + expect(collection.has(1)).toBe(false) + expect(await persisted).toBe(transaction.error?.error) + if (reason instanceof Error) + expect(transaction.error?.error).toBe(reason) + } finally { + if (transaction.state !== `failed`) transaction.rollback() + await collection.cleanup() + } + }, + ) + it(`keeps a claimed default transaction ambient for later plain collection mutations`, () => { const client = new DbClient() const clientCollection = client.collection( @@ -216,6 +261,173 @@ describe(`Transactions`, () => { transaction.isPersisted.promise.catch(() => {}) expect(transaction.state).toBe(`failed`) }) + it(`keeps a persisting transaction failed when rollback wins`, async () => { + let releasePersistence!: () => void + const persistence = new Promise((resolve) => { + releasePersistence = resolve + }) + const collection = createCollection<{ id: number }>({ + id: `persisting-rollback-wins`, + getKey: (item) => item.id, + sync: { sync: () => {} }, + }) + const transaction = createTransaction({ + autoCommit: false, + mutationFn: () => persistence, + }) + + try { + transaction.mutate(() => collection.insert({ id: 1 })) + const persisted = transaction.isPersisted.promise.then( + (value) => ({ status: `fulfilled` as const, value }), + (reason: unknown) => ({ status: `rejected` as const, reason }), + ) + const commit = transaction.commit() + expect(transaction.state).toBe(`persisting`) + + transaction.rollback() + expect(transaction.state).toBe(`failed`) + + releasePersistence() + await expect(commit).resolves.toBe(transaction) + expect(await persisted).toEqual({ + status: `rejected`, + reason: undefined, + }) + expect(transaction.state).toBe(`failed`) + expect(transaction.error).toBeUndefined() + } finally { + releasePersistence() + await collection.cleanup() + } + }) + it.each([ + [`Error`, (): unknown => new Error(`late persistence rejection`)], + [`undefined`, (): unknown => undefined], + [`false`, (): unknown => false], + [`zero`, (): unknown => 0], + [`NaN`, (): unknown => Number.NaN], + [`string`, (): unknown => `late persistence rejection`], + [`object`, (): unknown => ({ late: true })], + ] as const)( + `ignores a late %s persistence rejection after rollback wins`, + async (reasonName, createReason) => { + type Row = { id: number; owner: string } + let rejectPersistence!: (reason: unknown) => void + const persistence = new Promise((_resolve, reject) => { + rejectPersistence = reject + }) + const collection = createCollection({ + id: `late-persistence-rejection-${reasonName}`, + getKey: (item) => item.id, + sync: { sync: () => {} }, + }) + const batches: Array> = [] + const subscription = collection.subscribeChanges( + (changes) => { + batches.push( + changes.map(({ type, key }) => ({ + type, + key, + })), + ) + }, + { includeInitialState: false }, + ) + const first = createTransaction({ + autoCommit: false, + mutationFn: () => persistence, + }) + const second = createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + + try { + const persisted = first.isPersisted.promise.then( + (value) => ({ status: `fulfilled` as const, value }), + (reason: unknown) => ({ status: `rejected` as const, reason }), + ) + first.mutate(() => collection.insert({ id: 1, owner: `first` })) + const commit = first.commit().then( + (value) => ({ status: `fulfilled` as const, value }), + (reason: unknown) => ({ status: `rejected` as const, reason }), + ) + + first.rollback() + second.mutate(() => collection.insert({ id: 1, owner: `second` })) + rejectPersistence(createReason()) + + const commitOutcome = await commit + expect(commitOutcome.status).toBe(`fulfilled`) + if (commitOutcome.status === `fulfilled`) { + expect(commitOutcome.value).toBe(first) + } + expect(await persisted).toEqual({ + status: `rejected`, + reason: undefined, + }) + expect(first.state).toBe(`failed`) + expect(first.error).toBeUndefined() + expect(second.state).toBe(`pending`) + expect(collection.get(1)).toEqual({ + id: 1, + owner: `second`, + $collectionId: collection.id, + $key: 1, + $origin: `local`, + $synced: false, + }) + expect(batches).toEqual([ + [{ type: `insert`, key: 1 }], + [{ type: `delete`, key: 1 }], + [{ type: `insert`, key: 1 }], + ]) + } finally { + rejectPersistence(new Error(`test cleanup`)) + if (second.state === `pending`) { + second.rollback({ isSecondaryRollback: true }) + } + subscription.unsubscribe() + await collection.cleanup() + } + }, + ) + it(`keeps repeated rollback from affecting newer transactions`, async () => { + type Row = { id: number; owner: string } + const collection = createCollection({ + id: `repeated-rollback-is-terminal`, + getKey: (item) => item.id, + sync: { sync: () => {} }, + }) + const first = createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + const second = createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + + try { + void first.isPersisted.promise.catch(() => undefined) + first.mutate(() => collection.insert({ id: 1, owner: `first` })) + first.rollback() + + second.mutate(() => collection.insert({ id: 1, owner: `second` })) + expect(second.state).toBe(`pending`) + + expect(first.rollback()).toBe(first) + expect(first.state).toBe(`failed`) + expect(second.state).toBe(`pending`) + expect(collection.get(1)).toMatchObject({ id: 1, owner: `second` }) + } finally { + if (second.state === `pending`) { + second.rollback({ isSecondaryRollback: true }) + } + await collection.cleanup() + } + }) it(`should rollback if the mutationFn throws an error`, async () => { const transaction = createTransaction({ mutationFn: async () => { diff --git a/packages/db/tests/utils.test.ts b/packages/db/tests/utils.test.ts index 0cfc2b27a2..15520a0ea2 100644 --- a/packages/db/tests/utils.test.ts +++ b/packages/db/tests/utils.test.ts @@ -1,23 +1,66 @@ import { describe, expect, it } from 'vitest' import { Temporal } from 'temporal-polyfill' import { deepEquals } from '../src/utils' +import { normalizeError } from '../src/utils/error' import { isPromiseLike } from '../src/utils/type-guards' -import { oracleRandomParameters, readOracleRunConfig } from './oracle-config' +import { + oracleRandomParameters, + readOracleRunConfig, + validateOraclePropertyRegistry, +} from './oracle-config' + +describe(`normalizeError`, () => { + it(`normalizes unstringifiable thrown values`, () => { + const revoked = Proxy.revocable({}, {}) + revoked.revoke() + const thrownValues = [ + Object.create(null), + { + [Symbol.toPrimitive]: () => { + throw new Error(`conversion failed`) + }, + }, + new Proxy( + {}, + { + getPrototypeOf: () => { + throw new Error(`prototype lookup failed`) + }, + }, + ), + revoked.proxy, + ] + + for (const thrownValue of thrownValues) { + expect(() => normalizeError(thrownValue)).not.toThrow() + expect(normalizeError(thrownValue)).toEqual(new Error(`Unknown error`)) + } + }) +}) describe(`oracle run configuration`, () => { - it(`reads the multiplier and replay seed from an explicit environment`, () => { + it(`reads the multiplier and replay coordinates from an explicit environment`, () => { expect( readOracleRunConfig({ TANSTACK_DB_ORACLE_RUNS_MULTIPLIER: `100`, TANSTACK_DB_ORACLE_SEED: `-42`, + TANSTACK_DB_ORACLE_PATH: `1:0:2`, + TANSTACK_DB_ORACLE_PROPERTY: `includes.incremental-history`, }), - ).toEqual({ multiplier: 100, replaySeed: -42 }) + ).toEqual({ + multiplier: 100, + replaySeed: -42, + replayPath: `1:0:2`, + replayProperty: `includes.incremental-history`, + }) }) - it(`uses one run multiplier and no replay seed by default`, () => { + it(`uses one run multiplier and no replay coordinates by default`, () => { expect(readOracleRunConfig({})).toEqual({ multiplier: 1, replaySeed: undefined, + replayPath: undefined, + replayProperty: undefined, }) }) @@ -27,6 +70,49 @@ describe(`oracle run configuration`, () => { [{ TANSTACK_DB_ORACLE_RUNS_MULTIPLIER: ` ` }, `positive integer`], [{ TANSTACK_DB_ORACLE_SEED: `1.5` }, `must be an integer`], [{ TANSTACK_DB_ORACLE_SEED: ` ` }, `must be an integer`], + [{ TANSTACK_DB_ORACLE_PATH: `1:0` }, `requires TANSTACK_DB_ORACLE_SEED`], + [ + { TANSTACK_DB_ORACLE_PROPERTY: `includes.incremental-history` }, + `requires TANSTACK_DB_ORACLE_PATH`, + ], + [ + { + TANSTACK_DB_ORACLE_SEED: `42`, + TANSTACK_DB_ORACLE_PATH: ` `, + TANSTACK_DB_ORACLE_PROPERTY: `includes.incremental-history`, + }, + `must be non-empty`, + ], + [ + { + TANSTACK_DB_ORACLE_SEED: `42`, + TANSTACK_DB_ORACLE_PATH: `1:-1`, + TANSTACK_DB_ORACLE_PROPERTY: `includes.incremental-history`, + }, + `colon-separated nonnegative integers`, + ], + [ + { + TANSTACK_DB_ORACLE_SEED: `42`, + TANSTACK_DB_ORACLE_PATH: `1:0`, + }, + `requires TANSTACK_DB_ORACLE_PROPERTY`, + ], + [ + { + TANSTACK_DB_ORACLE_SEED: `42`, + TANSTACK_DB_ORACLE_PATH: `1:0`, + TANSTACK_DB_ORACLE_PROPERTY: `includes.typo`, + }, + `unknown oracle property`, + ], + [ + { + TANSTACK_DB_ORACLE_SEED: `42`, + TANSTACK_DB_ORACLE_PROPERTY: `includes.incremental-history`, + }, + `requires TANSTACK_DB_ORACLE_PATH`, + ], ] satisfies ReadonlyArray, string]>)( `rejects invalid environment values`, (environment, message) => { @@ -34,16 +120,72 @@ describe(`oracle run configuration`, () => { }, ) - it(`adds a seed only for replay runs`, () => { - expect(oracleRandomParameters(40, undefined)).toEqual({ numRuns: 40 }) - expect(oracleRandomParameters(40, -42)).toEqual({ + it(`rejects duplicate registered property names`, () => { + expect(() => + validateOraclePropertyRegistry([`one.property`, `one.property`]), + ).toThrow(`duplicate oracle property`) + }) + + it(`adds a shrink path only to its named property`, () => { + const ordinaryRun = { + replaySeed: undefined, + replayPath: undefined, + replayProperty: undefined, + } + const replayRun = { + replaySeed: -42, + replayPath: `1:0:2`, + replayProperty: `includes.incremental-history`, + } + + expect( + oracleRandomParameters(40, ordinaryRun, `includes.incremental-history`), + ).toEqual({ numRuns: 40 }) + expect( + oracleRandomParameters(40, replayRun, `includes.alpha-renaming`), + ).toEqual({ numRuns: 40, seed: -42, }) + expect( + oracleRandomParameters(40, replayRun, `includes.incremental-history`), + ).toEqual({ numRuns: 40, seed: -42, path: `1:0:2` }) }) }) describe(`deepEquals`, () => { + it.each([`later`, Symbol(`later`)])( + `checks own-key visibility after a getter runs: %s`, + (key) => { + const right = { first: 1, [key]: undefined } + const left = { + get first() { + Object.defineProperty(right, key, { enumerable: false }) + return 1 + }, + [key]: undefined, + } + expect(deepEquals(left, right)).toBe(false) + }, + ) + + it.each( + [`field`, Symbol(`field`)].flatMap((key) => + [false, true].map((inherited) => ({ key, inherited })), + ), + )( + `requires matching enumerable own keys: $key / inherited=$inherited`, + ({ key, inherited }) => { + const own = { [key]: 1 } + const other = { other: 1 } + if (inherited) Object.setPrototypeOf(other, { [key]: 1 }) + else Object.defineProperty(other, key, { value: 1, enumerable: false }) + expect(deepEquals(own, other)).toBe(false) + expect(deepEquals(other, own)).toBe(false) + expect(deepEquals(own, { [key]: 1 })).toBe(true) + }, + ) + describe(`primitives`, () => { it(`should handle identical primitives`, () => { expect(deepEquals(1, 1)).toBe(true) @@ -104,6 +246,16 @@ describe(`deepEquals`, () => { expect(deepEquals({ a: { b: 1 } }, { a: { b: 2 } })).toBe(false) }) + it(`should compare enumerable symbol properties`, () => { + const key = Symbol(`key`) + + expect(deepEquals({ [key]: 1 }, { [key]: 1 })).toBe(true) + expect(deepEquals({ [key]: 1 }, { [key]: 2 })).toBe(false) + expect(deepEquals({ [Symbol(`key`)]: 1 }, { [Symbol(`key`)]: 1 })).toBe( + false, + ) + }) + it(`should handle circular references in objects`, () => { const a: any = { x: 1 } a.self = a diff --git a/packages/db/tests/utils.ts b/packages/db/tests/utils.ts index b31408d0b4..759173a638 100644 --- a/packages/db/tests/utils.ts +++ b/packages/db/tests/utils.ts @@ -1,6 +1,9 @@ import { expect } from 'vitest' +import { createCollection } from '../src/collection/index.js' import { BTreeIndex } from '../src/indexes/btree-index' import { withCollectionConfigFactory } from '../src/client' +import { denormalizeUndefined } from '../src/utils/comparison.js' +import { CleanupQueue } from '../src/collection/cleanup-queue.js' import type { CollectionConfig, MutationFnParams, @@ -10,41 +13,22 @@ import type { import type { IndexConstructor } from '../src/indexes/base-index' import type { WithVirtualProps } from '../src/virtual-props.js' -type OracleEnvironment = Record - -export function readOracleRunConfig( - environment: OracleEnvironment = process.env, -): { multiplier: number; replaySeed: number | undefined } { - const multiplierValue = environment.TANSTACK_DB_ORACLE_RUNS_MULTIPLIER ?? `1` - const multiplier = Number(multiplierValue) - if (!Number.isSafeInteger(multiplier) || multiplier < 1) { - throw new Error( - `TANSTACK_DB_ORACLE_RUNS_MULTIPLIER must be a positive integer`, - ) - } - - const seedValue = environment.TANSTACK_DB_ORACLE_SEED - if (seedValue === undefined) return { multiplier, replaySeed: undefined } - - const replaySeed = Number(seedValue) - if (!Number.isSafeInteger(replaySeed)) { - throw new Error(`TANSTACK_DB_ORACLE_SEED must be an integer`) - } - return { multiplier, replaySeed } -} - -export function oracleRandomParameters( - numRuns: number, - replaySeed: number | undefined, -): { numRuns: number; seed?: number } { - return replaySeed === undefined ? { numRuns } : { numRuns, seed: replaySeed } -} - export type OutputWithVirtual< T extends object, TKey extends string | number = string | number, > = WithVirtualProps +// Keep sync startup, writes, readiness, and load outcomes in the test itself. +export function createOnDemandCollection( + config: Omit, `getKey` | `syncMode`>, +) { + return createCollection({ + ...config, + getKey: ({ id }) => id, + syncMode: `on-demand`, + }) +} + export const stripVirtualProps = | undefined>( value: T, ) => { @@ -528,3 +512,86 @@ export function withExpectedRejection( }) }) } + +type IndexInternals = { indexedKeys: Set } & ( + | { sortedValues: Array; valueMap: Map> } + | { + valueMap: Map }> + orderedEntries: { + size: number + minKey: () => unknown + maxKey: () => unknown + forRange: ( + low: unknown, + high: unknown, + includeHigh: boolean, + onFound: (key: unknown, bucket: { keys: Set }) => void, + ) => void + } + } +) + +function indexInternals(index: object): [IndexInternals, boolean] { + let reversed = false + let current = index as { originalIndex?: object } + while (current.originalIndex) { + reversed = !reversed + current = current.originalIndex as { originalIndex?: object } + } + return [current as IndexInternals, reversed] +} + +/** Test inspection of an index's tracked keys. */ +export function indexedKeysSet(index: object): Set { + return indexInternals(index)[0].indexedKeys +} + +/** Test inspection of an index's value buckets keyed by indexed value. */ +export function valueMapData(index: object): Map> { + const [internals] = indexInternals(index) + if (`sortedValues` in internals) return internals.valueMap + const result = new Map>() + for (const [key, bucket] of internals.valueMap) { + result.set(denormalizeUndefined(key), bucket.keys) + } + return result +} + +/** Test inspection of an index's ordered [value, keys] entries. */ +export function orderedEntriesArray( + index: object, +): Array<[unknown, Set]> { + const [internals, reversed] = indexInternals(index) + let entries: Array<[unknown, Set]> + if (`sortedValues` in internals) { + entries = internals.sortedValues.map((value) => [ + value, + internals.valueMap.get(value) ?? new Set(), + ]) + } else { + const tree = internals.orderedEntries + entries = [] + if (tree.size > 0) { + tree.forRange(tree.minKey(), tree.maxKey(), true, (key, bucket) => { + entries.push([denormalizeUndefined(key), bucket.keys]) + }) + } + } + return reversed ? entries.reverse() : entries +} + +export function orderedEntriesArrayReversed( + index: object, +): Array<[unknown, Set]> { + return orderedEntriesArray(index).reverse() +} + +/** Reset the CleanupQueue singleton between tests. */ +export function resetCleanupQueue(): void { + const holder = CleanupQueue as unknown as { + instance: { timeoutId: ReturnType | null } | null + } + if (holder.instance?.timeoutId != null) + clearTimeout(holder.instance.timeoutId) + holder.instance = null +} diff --git a/packages/db/tests/utils/collection-helpers.test.ts b/packages/db/tests/utils/collection-helpers.test.ts new file mode 100644 index 0000000000..192b934b8b --- /dev/null +++ b/packages/db/tests/utils/collection-helpers.test.ts @@ -0,0 +1,49 @@ +import { describe, expect, it, vi } from 'vitest' +import { getOrCreate } from '../../src/utils/get-or-create.js' +import { isPlainObject } from '../../src/utils/type-guards.js' + +describe(`getOrCreate`, () => { + it.each([`map`, `weak map`] as const)( + `initializes each %s owner once`, + (kind) => { + const createStore = () => + kind === `map` + ? new Map() + : new WeakMap() + const first = createStore() + const second = createStore() + const key = {} + const create = vi.fn(() => ({})) + const value = getOrCreate(first, key, create) + expect(getOrCreate(first, key, create)).toBe(value) + expect(create).toHaveBeenCalledTimes(1) + expect(getOrCreate(second, key, create)).not.toBe(value) + first.delete(key) + expect(getOrCreate(first, key, create)).not.toBe(value) + expect(create).toHaveBeenCalledTimes(3) + }, + ) + + it.each([false, 0, ``, null])(`retains a defined value %j`, (value) => { + const entries = new Map([[`key`, value]]) + const create = vi.fn(() => value) + expect(getOrCreate(entries, `key`, create)).toBe(value) + expect(create).not.toHaveBeenCalled() + }) +}) + +describe(`isPlainObject`, () => { + it.each([ + { name: `ordinary object`, value: {}, expected: true }, + { name: `null prototype`, value: Object.create(null), expected: true }, + { name: `custom prototype`, value: Object.create({}), expected: false }, + { name: `array`, value: [], expected: false }, + { name: `date`, value: new Date(0), expected: false }, + { name: `null`, value: null, expected: false }, + { name: `undefined`, value: undefined, expected: false }, + { name: `function`, value: () => {}, expected: false }, + { name: `string`, value: `value`, expected: false }, + ])(`classifies $name`, ({ value, expected }) => { + expect(isPlainObject(value)).toBe(expected) + }) +}) diff --git a/packages/electric-db-collection/src/electric.ts b/packages/electric-db-collection/src/electric.ts index 39e6540a6a..eaa1ffca54 100644 --- a/packages/electric-db-collection/src/electric.ts +++ b/packages/electric-db-collection/src/electric.ts @@ -8,6 +8,7 @@ import { Store } from '@tanstack/store' import DebugModule from 'debug' import { DeduplicatedLoadSubset, + LoadSubsetOperationAbortedError, and, withCollectionConfigFactory, } from '@tanstack/db' @@ -564,6 +565,9 @@ function createLoadSubsetDedupe>({ const compileOptions = encodeColumnName ? { encodeColumnName } : undefined const logPrefix = collectionId ? `[${collectionId}] ` : `` + const abortReason = (abortedSignal: AbortSignal): unknown => + abortedSignal.reason ?? new LoadSubsetOperationAbortedError() + /** * Handles errors from snapshot operations. Returns true if the error was * handled (signal aborted during cleanup), false if it should be re-thrown. @@ -579,7 +583,11 @@ function createLoadSubsetDedupe>({ const loadSubset = async (opts: LoadSubsetOptions) => { const commitCursor = getCommitCursor() - if (opts.signal?.aborted) return + const throwIfAborted = () => { + if (signal.aborted) throw abortReason(signal) + if (opts.signal?.aborted) throw abortReason(opts.signal) + } + throwIfAborted() if (isBufferingInitialSync()) { const snapshotParams = compileSQL(opts, compileOptions) @@ -628,6 +636,18 @@ function createLoadSubsetDedupe>({ // still works. if (stream.isUpToDate) { let timeoutId: ReturnType | undefined + const abortSignals = [signal, opts.signal].filter( + (candidate): candidate is AbortSignal => candidate !== undefined, + ) + let rejectAbort: (reason: unknown) => void = () => {} + const aborted = new Promise((_resolve, reject) => { + rejectAbort = reject + }) + const abort = (event: Event) => + rejectAbort(abortReason(event.currentTarget as AbortSignal)) + for (const abortSignal of abortSignals) { + abortSignal.addEventListener(`abort`, abort, { once: true }) + } try { await Promise.race([ stream.forceDisconnectAndRefresh(), @@ -637,8 +657,10 @@ function createLoadSubsetDedupe>({ FORCE_DISCONNECT_AND_REFRESH_TIMEOUT_MS, ) }), + aborted, ]) } catch (error) { + if (signal.aborted || opts.signal?.aborted) throw error if (handleSnapshotError(error, `forceDisconnectAndRefresh`)) { return } @@ -648,10 +670,13 @@ function createLoadSubsetDedupe>({ ) } finally { clearTimeout(timeoutId) + for (const abortSignal of abortSignals) { + abortSignal.removeEventListener(`abort`, abort) + } } } - if (opts.signal?.aborted) return + throwIfAborted() // Upstream limitation: ShapeStream.requestSnapshot() publishes its rows // through the stream callback before its Promise resolves. It accepts no @@ -1573,17 +1598,12 @@ function createElectricSync>( // Abort controller for the stream - wraps the signal if provided const abortController = new AbortController() + const forwardExternalAbort = () => abortController.abort() if (shapeOptions.signal) { - shapeOptions.signal.addEventListener( - `abort`, - () => { - abortController.abort() - }, - { - once: true, - }, - ) + shapeOptions.signal.addEventListener(`abort`, forwardExternalAbort, { + once: true, + }) if (shapeOptions.signal.aborted) { abortController.abort() } @@ -2065,6 +2085,10 @@ function createElectricSync>( return { loadSubset: loadSubsetDedupe?.loadSubset, cleanup: () => { + shapeOptions.signal?.removeEventListener( + `abort`, + forwardExternalAbort, + ) // Unsubscribe from the stream unsubscribeStream() // Abort the abort controller to stop the stream diff --git a/packages/electric-db-collection/tests/electric-live-query.test.ts b/packages/electric-db-collection/tests/electric-live-query.test.ts index 8bd5ac7b8a..e16f8a480e 100644 --- a/packages/electric-db-collection/tests/electric-live-query.test.ts +++ b/packages/electric-db-collection/tests/electric-live-query.test.ts @@ -58,6 +58,13 @@ const sampleUsers: Array = [ const mockSubscribe = vi.fn() const mockRequestSnapshot = vi.fn() const mockFetchSnapshot = vi.fn() + +function expectNoRepeatedSnapshotRequests() { + const requests = mockRequestSnapshot.mock.calls.map(([request]) => request) + const keys = requests.map((request) => JSON.stringify(request)) + expect(keys).toHaveLength(new Set(keys).size) +} + const mockStream = { subscribe: mockSubscribe, fetchSnapshot: mockFetchSnapshot, @@ -553,7 +560,20 @@ describe.each([ expect(limitedLiveQuery.status).toBe(`ready`) expect(limitedLiveQuery.size).toBe(2) // Only first 2 active users - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + expect( + mockRequestSnapshot.mock.calls.map(([request]) => request), + ).toEqual([ + { + params: { '1': `true` }, + where: `"active" = $1`, + orderBy: `"age" NULLS FIRST`, + limit: 2, + }, + { + params: { '1': `true`, '2': `22` }, + where: `"active" = $1 AND "age" = $2`, + }, + ]) const callArgs = (index: number) => mockRequestSnapshot.mock.calls[index]?.[0] @@ -566,32 +586,34 @@ describe.each([ // Next call will return a snapshot containing 2 rows // Calls after that will return the default empty snapshot - mockRequestSnapshot.mockResolvedValueOnce({ - data: [ - { - headers: { operation: `insert` }, - key: 5, - value: { - id: 5, - name: `Eve`, - age: 30, - email: `eve@example.com`, - active: true, - }, - }, - { - headers: { operation: `insert` }, - key: 6, - value: { - id: 6, - name: `Frank`, - age: 35, - email: `frank@example.com`, - active: true, - }, - }, - ], - }) + mockRequestSnapshot.mockImplementation(async ({ where }) => ({ + data: where.includes(` > `) + ? [ + { + headers: { operation: `insert` }, + key: 5, + value: { + id: 5, + name: `Eve`, + age: 30, + email: `eve@example.com`, + active: true, + }, + }, + { + headers: { operation: `insert` }, + key: 6, + value: { + id: 6, + name: `Frank`, + age: 35, + email: `frank@example.com`, + active: true, + }, + }, + ] + : [], + })) // Create second live query with higher limit of 6 const expandedLiveQuery = createLiveQueryCollection({ @@ -613,11 +635,7 @@ describe.each([ // Wait for the live query to process await new Promise((resolve) => setTimeout(resolve, 0)) - // Limited queries are only deduplicated when their where clauses are equal. - // Both queries have the same where clause (active = true), but the second query - // with limit 6 needs more data than the first query with limit 2 provided. - // With cursor-based pagination, initial loads (without cursor) make 1 requestSnapshot call each. - expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + expectNoRepeatedSnapshotRequests() // Check that first it requested a limit of 2 users (from first query) expect(callArgs(0)).toMatchObject({ @@ -627,13 +645,18 @@ describe.each([ limit: 2, }) - // Check that second it requested a limit of 6 users (from second query) - expect(callArgs(1)).toMatchObject({ + expect(mockRequestSnapshot).toHaveBeenCalledWith({ params: { '1': `true` }, where: `"active" = $1`, orderBy: `"age" NULLS FIRST`, limit: 6, }) + expect(mockRequestSnapshot).toHaveBeenCalledWith( + expect.objectContaining({ + where: `"active" = $1 AND "age" > $2`, + orderBy: `"age" NULLS FIRST`, + }), + ) // The expanded live query should have the locally available data expect(expandedLiveQuery.status).toBe(`ready`) @@ -883,9 +906,9 @@ describe(`Electric Collection with Live Query - syncMode integration`, () => { }), ) - // For limited queries, only requests with identical where clauses can be deduplicated. - // With cursor-based pagination, initial loads (without cursor) make 1 requestSnapshot call. - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + // Electric maps one cursor demand to a bounded page plus its exact tie. + expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + expectNoRepeatedSnapshotRequests() }) it(`should pass correct WHERE clause to requestSnapshot when live query has filters`, async () => { @@ -1005,15 +1028,14 @@ describe(`Electric Collection - loadSubset deduplication`, () => { subscriber(messages) } - it(`should deduplicate identical concurrent loadSubset requests`, async () => { + it(`keeps independently abortable live-query requests independent`, async () => { const electricCollection = createElectricCollectionWithSyncMode(`on-demand`) simulateInitialSync([]) expect(electricCollection.status).toBe(`ready`) - // Create three identical live queries concurrently - // Without deduplication, this would trigger 3 requestSnapshot calls - // With deduplication, only 1 should be made + // Each live query owns its own abort signal, so canceling one cannot cancel + // transport work still needed by a peer. createLiveQueryCollection({ startSync: true, query: (q) => @@ -1046,19 +1068,18 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // With deduplication, only 1 requestSnapshot call should be made - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) - expect(mockRequestSnapshot).toHaveBeenCalledWith( - expect.objectContaining({ + expect(mockRequestSnapshot).toHaveBeenCalledTimes(3) + for (const [request] of mockRequestSnapshot.mock.calls) { + expect(request).toMatchObject({ where: `"active" = $1`, params: { '1': `true` }, orderBy: `"age" NULLS FIRST`, limit: 10, - }), - ) + }) + } }) - it(`should deduplicate subset loadSubset requests with same where clause`, async () => { + it(`keeps different exact windows independent despite a shared predicate`, async () => { const electricCollection = createElectricCollectionWithSyncMode(`on-demand`) simulateInitialSync([]) @@ -1079,8 +1100,8 @@ describe(`Electric Collection - loadSubset deduplication`, () => { expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) - // Create a live query with SAME where clause but smaller limit - // This SHOULD be deduped because where clauses are equal and limit is smaller + // A smaller limit is a distinct exact demand. A requested wider window does + // not prove that its rows were applied or that the source was exhausted. createLiveQueryCollection({ startSync: true, query: (q) => @@ -1093,8 +1114,10 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // Still only 1 call - the second was deduped (same where, smaller limit) - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + expect( + mockRequestSnapshot.mock.calls.map(([request]) => request.limit), + ).toEqual([20, 10]) }) it(`should NOT deduplicate limited queries with different where clauses`, async () => { @@ -1195,9 +1218,10 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // For limited queries, only requests with identical where clauses can be deduplicated. - // With cursor-based pagination, initial loads (without cursor) make 1 requestSnapshot call. - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + const requestsBeforeReset = mockRequestSnapshot.mock.calls.map( + ([request]) => JSON.stringify(request), + ) + expect(requestsBeforeReset.length).toBeGreaterThan(0) // Simulate a must-refetch (which triggers truncate and reset) subscriber([{ headers: { control: `must-refetch` } }]) @@ -1206,9 +1230,15 @@ describe(`Electric Collection - loadSubset deduplication`, () => { // Wait for the existing live query to re-request data after truncate await new Promise((resolve) => setTimeout(resolve, 0)) - // The existing live query re-requests its data after truncate - // After must-refetch, the query requests data again (1 initial + 1 after truncate) - expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + const requestsAfterReset = mockRequestSnapshot.mock.calls + .slice(requestsBeforeReset.length) + .map(([request]) => JSON.stringify(request)) + expect(requestsAfterReset.length).toBeGreaterThan(0) + expect( + requestsAfterReset.some((request) => + requestsBeforeReset.includes(request), + ), + ).toBe(true) // Create the same live query again after reset // This should NOT be deduped because the reset cleared the deduplication state, @@ -1226,12 +1256,12 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // Should have more calls - the different query triggered a new request - // 1 initial + 1 after must-refetch + 1 for new query = 3 - expect(mockRequestSnapshot).toHaveBeenCalledTimes(3) + expect(mockRequestSnapshot).toHaveBeenCalledWith( + expect.objectContaining({ params: { '1': `false` } }), + ) }) - it(`should deduplicate unlimited queries regardless of orderBy`, async () => { + it(`keeps different exact unlimited orderings independent`, async () => { const electricCollection = createElectricCollectionWithSyncMode(`on-demand`) simulateInitialSync([]) @@ -1251,8 +1281,7 @@ describe(`Electric Collection - loadSubset deduplication`, () => { expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) - // Create another unlimited query with same where but different orderBy - // This should be deduped - orderBy is ignored for unlimited queries + // Order remains part of exact demand identity even without a limit. createLiveQueryCollection({ startSync: true, query: (q) => @@ -1264,11 +1293,13 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // Still only 1 call - different orderBy doesn't matter for unlimited queries - expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + expect( + mockRequestSnapshot.mock.calls.map(([request]) => request.orderBy), + ).toEqual([`"age" NULLS FIRST`, `"name" DESC NULLS FIRST`]) }) - it(`should combine multiple unlimited queries with union`, async () => { + it(`does not infer union coverage across different predicates`, async () => { const electricCollection = createElectricCollectionWithSyncMode(`on-demand`) simulateInitialSync([]) @@ -1301,8 +1332,8 @@ describe(`Electric Collection - loadSubset deduplication`, () => { expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) - // Create third query (age > 35) - this is a subset of (age > 30) - // This should be deduped + // A broader requested predicate does not prove applied coverage for this + // distinct exact predicate. createLiveQueryCollection({ startSync: true, query: (q) => @@ -1313,7 +1344,55 @@ describe(`Electric Collection - loadSubset deduplication`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) - // Still 2 calls - third was covered by the union of first two - expect(mockRequestSnapshot).toHaveBeenCalledTimes(2) + expect(mockRequestSnapshot).toHaveBeenCalledTimes(3) + expect( + mockRequestSnapshot.mock.calls.map(([request]) => request.params), + ).toEqual([{ '1': `30` }, { '1': `20` }, { '1': `35` }]) + }) + + it(`reuses retained Electric rows after the final live-query owner leaves`, async () => { + const electricCollection = createElectricCollectionWithSyncMode(`on-demand`) + const row = sampleUsers[0]! + simulateInitialSync([]) + mockRequestSnapshot.mockResolvedValue({ + data: [ + { + headers: { operation: `insert` }, + key: row.id, + value: row, + }, + ], + }) + const createLive = (id: string) => + createLiveQueryCollection({ + id, + startSync: true, + query: (q) => + q + .from({ user: electricCollection }) + .where(({ user }) => eq(user.active, true)), + }) + const first = createLive(`electric-remount-first`) + let second: ReturnType | undefined + + try { + await first.preload() + expect(first.toArray.map(({ id }) => id)).toEqual([row.id]) + + await first.cleanup() + expect(electricCollection.size).toBe(1) + + second = createLive(`electric-remount-second`) + await second.preload() + + expect(mockRequestSnapshot).toHaveBeenCalledTimes(1) + expect(second.toArray.map(({ id }) => id)).toEqual([row.id]) + } finally { + await Promise.all([ + first.cleanup(), + second?.cleanup(), + electricCollection.cleanup(), + ]) + } }) }) diff --git a/packages/electric-db-collection/tests/electric.test.ts b/packages/electric-db-collection/tests/electric.test.ts index c913f8d973..e17c792979 100644 --- a/packages/electric-db-collection/tests/electric.test.ts +++ b/packages/electric-db-collection/tests/electric.test.ts @@ -2,6 +2,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { ShapeStream } from '@electric-sql/client' import { CollectionImpl, + IR, createCollection, createTransaction, } from '@tanstack/db' @@ -2659,6 +2660,53 @@ describe(`Electric Integration`, () => { // Tests for syncMode configuration describe(`syncMode configuration`, () => { + const createOnDemandCollection = (id: string) => + createCollection( + electricCollectionOptions({ + id, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + }, + syncMode: `on-demand`, + getKey: (item: Row) => item.id as number, + startSync: true, + }), + ) + + it(`removes the external shape abort listener across cleanup and restart`, async () => { + const externalAbort = new NativeAbortController() + const addSpy = vi.spyOn(externalAbort.signal, `addEventListener`) + const removeSpy = vi.spyOn(externalAbort.signal, `removeEventListener`) + const testCollection = createCollection( + electricCollectionOptions({ + id: `shape-signal-listener-cleanup-test`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + signal: externalAbort.signal, + }, + syncMode: `progressive`, + getKey: (item: Row) => item.id as number, + startSync: true, + }), + ) + + await testCollection.cleanup() + const subscription = testCollection.subscribeChanges(() => {}) + await testCollection.cleanup() + subscription.unsubscribe() + + const addedListeners = addSpy.mock.calls + .filter(([type]) => type === `abort`) + .map(([, listener]) => listener) + const removedListeners = removeSpy.mock.calls + .filter(([type]) => type === `abort`) + .map(([, listener]) => listener) + expect(addedListeners).toHaveLength(2) + expect(removedListeners).toEqual(addedListeners) + }) + it(`should not request snapshots during subscription in eager mode`, () => { vi.clearAllMocks() @@ -2749,6 +2797,229 @@ describe(`Electric Integration`, () => { } }) + it(`waits for an on-demand commit to become public`, async () => { + const request = createDeferred() + mockRequestSnapshot.mockReturnValueOnce(request.promise) + const testCollection = createOnDemandCollection( + `on-demand-successful-parked-commit-test`, + ) + const persistence = createDeferred() + const transaction = createTransaction({ + mutationFn: () => persistence.promise, + }) + + try { + transaction.mutate(() => + testCollection.insert({ id: 3, name: `Local row` }), + ) + const load = Promise.resolve( + testCollection._sync.loadSubset({ limit: 10 }), + ) + await vi.waitFor(() => + expect(mockRequestSnapshot).toHaveBeenCalledOnce(), + ) + subscriber([ + { + key: `2`, + value: { id: 2, name: `Applied parked row` }, + headers: { operation: `insert` }, + }, + { headers: { control: `subset-end` } }, + ]) + request.resolve() + + const nextTurn = new Promise<`next-turn`>((resolve) => + setTimeout(() => resolve(`next-turn`), 0), + ) + await expect( + Promise.race([load.then(() => `load-settled` as const), nextTurn]), + ).resolves.toBe(`next-turn`) + expect(testCollection.has(2)).toBe(false) + + persistence.resolve() + await transaction.isPersisted.promise + await load + expect(stripVirtualProps(testCollection.get(2))).toEqual({ + id: 2, + name: `Applied parked row`, + }) + } finally { + request.resolve() + persistence.resolve() + await transaction.isPersisted.promise.catch(() => undefined) + await testCollection.cleanup() + } + }) + + it(`waits for both physical requests of one cursor demand`, async () => { + const whereCurrent = createDeferred() + const whereFrom = createDeferred() + mockRequestSnapshot + .mockReturnValueOnce(whereCurrent.promise) + .mockReturnValueOnce(whereFrom.promise) + const testCollection = createOnDemandCollection( + `on-demand-cursor-all-requests-test`, + ) + const id = new IR.PropRef([`id`]) + + try { + const load = Promise.resolve( + testCollection._sync.loadSubset({ + limit: 10, + orderBy: [ + { + expression: id, + compareOptions: { + direction: `asc`, + nulls: `last`, + stringSort: `lexical`, + }, + }, + ], + cursor: { + whereCurrent: new IR.Func(`eq`, [id, new IR.Value(1)]), + whereFrom: new IR.Func(`gt`, [id, new IR.Value(1)]), + lastKey: 1, + }, + }), + ) + await vi.waitFor(() => + expect(mockRequestSnapshot).toHaveBeenCalledTimes(2), + ) + + whereCurrent.resolve() + const nextTurn = new Promise<`next-turn`>((resolve) => + setTimeout(() => resolve(`next-turn`), 0), + ) + await expect( + Promise.race([load.then(() => `load-settled` as const), nextTurn]), + ).resolves.toBe(`next-turn`) + + whereFrom.resolve() + await load + } finally { + whereCurrent.resolve() + whereFrom.resolve() + await testCollection.cleanup() + } + }) + + it.each([ + { syncMode: `on-demand`, signalSource: `collection` }, + { syncMode: `on-demand`, signalSource: `request` }, + { syncMode: `progressive`, signalSource: `collection` }, + { syncMode: `progressive`, signalSource: `request` }, + ] as const)( + `starts no $syncMode work for an already-aborted $signalSource signal`, + async ({ syncMode, signalSource }) => { + const abortController = new AbortController() + abortController.abort() + const testCollection = createCollection( + electricCollectionOptions({ + id: `${syncMode}-${signalSource}-already-aborted`, + shapeOptions: { + url: `http://test-url`, + params: { table: `test_table` }, + signal: + signalSource === `collection` + ? abortController.signal + : undefined, + }, + syncMode, + getKey: (item: Row) => item.id as number, + startSync: true, + }), + ) + + await expect( + testCollection._sync.loadSubset({ + limit: 10, + signal: + signalSource === `request` ? abortController.signal : undefined, + }), + ).rejects.toMatchObject({ name: `AbortError` }) + expect(mockForceDisconnectAndRefresh).not.toHaveBeenCalled() + expect(mockRequestSnapshot).not.toHaveBeenCalled() + expect(mockFetchSnapshot).not.toHaveBeenCalled() + await testCollection.cleanup() + }, + ) + + it.each([true, false])( + `settles reasonless cancellation after refresh with DOMException available %s`, + async (hasDOMException) => { + const originalDOMException = globalThis.DOMException + const controller = new NativeAbortController() + const refresh = createDeferred() + mockStream.isUpToDate = true + mockForceDisconnectAndRefresh.mockReturnValueOnce(refresh.promise) + const testCollection = createOnDemandCollection( + `reasonless-refresh-abort`, + ) + try { + const load = testCollection._sync.loadSubset({ + limit: 10, + signal: controller.signal, + }) + const outcome = Promise.resolve(load).then( + () => undefined, + (error: unknown) => error, + ) + await Promise.resolve() + // Model a platform signal without reason; no event is required for + // the post-refresh cancellation check to observe its terminal state. + Object.defineProperty(controller.signal, `aborted`, { value: true }) + Object.defineProperty(controller.signal, `reason`, { + value: undefined, + }) + if (!hasDOMException) vi.stubGlobal(`DOMException`, undefined) + refresh.resolve() + await expect(outcome).resolves.toMatchObject({ name: `AbortError` }) + expect(mockRequestSnapshot).not.toHaveBeenCalled() + } finally { + vi.stubGlobal(`DOMException`, originalDOMException) + refresh.resolve() + await testCollection.cleanup() + } + }, + ) + + it(`cancels a pending refresh wait when the collection is cleaned up`, async () => { + vi.useFakeTimers() + const refresh = createDeferred() + try { + mockStream.isUpToDate = true + mockForceDisconnectAndRefresh.mockReturnValueOnce(refresh.promise) + const testCollection = createOnDemandCollection( + `on-demand-refresh-cleanup-test`, + ) + const load = Promise.resolve( + testCollection._sync.loadSubset({ limit: 10 }), + ) + const loadError = load.then( + () => undefined, + (error: unknown) => error, + ) + + await Promise.resolve() + await testCollection.cleanup() + await vi.advanceTimersByTimeAsync(0) + + await expect(loadError).resolves.toMatchObject({ name: `AbortError` }) + expect(mockRequestSnapshot).not.toHaveBeenCalled() + expect(vi.getTimerCount()).toBe(0) + + refresh.resolve() + await refresh.promise + await load.catch(() => undefined) + expect(mockRequestSnapshot).not.toHaveBeenCalled() + } finally { + refresh.resolve() + await vi.runOnlyPendingTimersAsync() + vi.useRealTimers() + } + }) + it(`should refresh the stream before requesting on-demand snapshots when already up-to-date`, async () => { vi.clearAllMocks() diff --git a/packages/powersync-db-collection/src/PowerSyncTransactor.ts b/packages/powersync-db-collection/src/PowerSyncTransactor.ts index 1e174f034e..b5d14a5287 100644 --- a/packages/powersync-db-collection/src/PowerSyncTransactor.ts +++ b/packages/powersync-db-collection/src/PowerSyncTransactor.ts @@ -1,4 +1,5 @@ import { sanitizeSQL } from '@powersync/common' +import { LoadSubsetOperationAbortedError } from '@tanstack/db' import DebugModule from 'debug' import { PendingOperationStore } from './PendingOperationStore' import { asPowerSyncRecord, mapOperationToPowerSync } from './helpers' @@ -94,7 +95,29 @@ export class PowerSyncTransactor { if (collection.isReady()) { return } - await new Promise((resolve) => collection.onFirstReady(resolve)) + // Observe this session without starting new demand from mutationFn. + // Cleanup and startup failure must settle the wait before taking a lock. + await new Promise((resolve, reject) => { + const check = () => { + if (collection.isReady()) { + unsubscribe() + resolve() + } else if ( + collection.status === `error` || + collection.status === `cleaned-up` + ) { + unsubscribe() + reject( + collection.status === `error` + ? (collection._lifecycle.getSyncError() ?? + new Error(`Collection failed before readiness`)) + : new LoadSubsetOperationAbortedError(), + ) + } + } + const unsubscribe = collection.on(`status:change`, check) + check() + }) }), ) diff --git a/packages/powersync-db-collection/src/powersync.ts b/packages/powersync-db-collection/src/powersync.ts index 76b8dedd3e..a469b661d3 100644 --- a/packages/powersync-db-collection/src/powersync.ts +++ b/packages/powersync-db-collection/src/powersync.ts @@ -323,6 +323,7 @@ function createPowerSyncCollectionConfig< let disposeTracking: | ((options?: { context?: LockContext }) => Promise) | null = null + let trackingSetup: Promise | null = null if (syncMode === `eager`) { return runEagerSync() @@ -337,6 +338,13 @@ function createPowerSyncCollectionConfig< async function safelyDisposeTracking( context?: LockContext, ): Promise { + // Cleanup can race trigger creation. Wait until the disposer has been + // published so an abort cannot strand a freshly-created trigger. + const setup = trackingSetup + if (setup) { + await setup.catch(() => undefined) + } + const dispose = disposeTracking if (!dispose) { return @@ -346,6 +354,25 @@ function createPowerSyncCollectionConfig< await dispose(context ? { context } : undefined) } + async function establishTracking( + options: Parameters[0], + appliedReceipts: Array, + ): Promise { + const setup = (async () => { + const dispose = await createDiffTrigger(options, appliedReceipts) + disposeTracking = dispose + })() + trackingSetup = setup + + try { + await setup + } finally { + if (trackingSetup === setup) { + trackingSetup = null + } + } + } + async function createDiffTrigger( options: { setupContext?: LockContext @@ -398,6 +425,17 @@ function createPowerSyncCollectionConfig< } async function flushDiffRecords(): Promise { + // PowerSync can notify after creating the tracking table but before its + // create call returns. Preserve that notification until the disposer, + // which proves the trigger is usable, has been published. + const setup = trackingSetup + if (setup) { + await setup.catch(() => undefined) + } + if (!disposeTracking) { + return + } + const ignoredReceipts: Array = [] await database .writeTransaction(async (context) => { @@ -512,10 +550,15 @@ function createPowerSyncCollectionConfig< let onUnload: CleanupFn | void | null = null start(async () => { - onUnload = await restConfig.onLoad?.() + const cleanup = await restConfig.onLoad?.() + if (abortController.signal.aborted) { + cleanup?.() + return + } + onUnload = cleanup const appliedReceipts: Array = [] - disposeTracking = await createDiffTrigger( + await establishTracking( { // Initial eager hydration must make the source usable before // PowerSync can persist a mutation queued during startup. @@ -562,110 +605,159 @@ function createPowerSyncCollectionConfig< // On-demand mode. // Registers a diff trigger for the active WHERE expressions. function runOnDemandSync() { - const unloadSubsetCallbacks = new Map() + type DemandRecord = { + options: LoadSubsetOptions + active: boolean + cleanup?: CleanupFn + } + type PendingRelease = { + options: LoadSubsetOptions + failures: number + } + + const demands = new Map() const releasedSubsets = new WeakSet() + const pendingReleases: Array = [] let stopped = false - const hasStopped = () => stopped - - start().catch((error) => + let trackingRevision = 0 + let reconciledTrackingRevision = 0 + let rebuildPromise: Promise | null = null + let drainingReleases = false + let releaseRetryTimer: ReturnType | undefined + const startup = start() + void startup.catch((error) => database.logger.error( `Could not start syncing process for ${viewName} into ${trackedTableName}`, error, ), ) - // Tracks all active WHERE expressions for on-demand sync filtering. - // Each loadSubset call pushes its predicate; unloadSubset removes it. - const activeWhereExpressions: Array = [] + const activeWhereExpressions = () => + Array.from(demands.values()) + .filter((demand) => demand.active) + .map((demand) => demand.options.where) - const loadSubset = async ( - options?: LoadSubsetOptions, - ): Promise => { - if (hasStopped()) return - const appliedReceipts: Array = [] - - if (options) { - activeWhereExpressions.push(options.where) - const cleanup = await restConfig.onLoadSubset?.(options) - if (hasStopped()) { - cleanup?.() - return - } - if (cleanup) { - if (releasedSubsets.has(options) || options.signal?.aborted) { - cleanup() - } else { - unloadSubsetCallbacks.set(options, cleanup) - } - } - } + // One reconciliation owns every queued revision so callers cannot + // settle against a stale trigger configuration. + const reconcileTracking = async (): Promise => { + while (!stopped && reconciledTrackingRevision !== trackingRevision) { + const revision = trackingRevision + const isCurrent = () => !stopped && trackingRevision === revision + const appliedReceipts: Array = [] - // No predicates remain, so stop tracking entirely. Both calls are no-ops - // when no tracking table is currently active. - if (activeWhereExpressions.length === 0) { await database.writeLock(async (ctx) => { + if (!isCurrent()) return await flushDiffRecordsWithContext(ctx, appliedReceipts) + if (!isCurrent()) return await safelyDisposeTracking(ctx) + if (!isCurrent()) return + + const active = activeWhereExpressions() + if (active.length === 0) return + const combinedWhere = + active.length === 1 + ? active[0] + : or(active[0], active[1], ...active.slice(2)) + const compiledNewData = compileSQLite( + { where: combinedWhere }, + { jsonColumn: 'NEW.data' }, + ) + const compiledOldData = compileSQLite( + { where: combinedWhere }, + { jsonColumn: 'OLD.data' }, + ) + const compiledView = compileSQLite({ where: combinedWhere }) + const newDataWhenClause = toInlinedWhereClause(compiledNewData) + const oldDataWhenClause = toInlinedWhereClause(compiledOldData) + const viewWhereClause = toInlinedWhereClause(compiledView) + + await establishTracking( + { + setupContext: ctx, + when: { + [DiffTriggerOperation.INSERT]: newDataWhenClause, + [DiffTriggerOperation.UPDATE]: `(${newDataWhenClause}) OR (${oldDataWhenClause})`, + [DiffTriggerOperation.DELETE]: oldDataWhenClause, + }, + writeType: (rowId: string) => + collection.has(rowId) ? `update` : `insert`, + batchQuery: ( + lockContext: LockContext, + batchSize: number, + cursor: number, + ) => + lockContext.getAll( + `SELECT * FROM ${viewName} WHERE ${viewWhereClause} LIMIT ? OFFSET ?`, + [batchSize, cursor], + ), + }, + appliedReceipts, + ) + if (!isCurrent()) await safelyDisposeTracking(ctx) }) await Promise.all(appliedReceipts) - return + if (isCurrent()) { + reconciledTrackingRevision = revision + } } + } - const combinedWhere = - activeWhereExpressions.length === 1 - ? activeWhereExpressions[0] - : or( - activeWhereExpressions[0], - activeWhereExpressions[1], - ...activeWhereExpressions.slice(2), - ) + const rebuildTracking = (): Promise => { + rebuildPromise ??= reconcileTracking() + .catch((error) => { + // A rebuild may already have removed every active diff trigger. + // Do not leave healthy consumers ready against a stale source. + if (!stopped) markError(error) + throw error + }) + .finally(() => { + rebuildPromise = null + }) + return rebuildPromise + } - const compiledNewData = compileSQLite( - { where: combinedWhere }, - { jsonColumn: 'NEW.data' }, - ) + const loadSubset = async ( + options: LoadSubsetOptions, + ): Promise => { + if (stopped) return + // Never create a trigger that has no observer to drain its diff table. + await startup + if ( + // Cleanup can run while startup is pending. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + stopped || + releasedSubsets.has(options) || + options.signal?.aborted + ) { + return + } - const compiledOldData = compileSQLite( - { where: combinedWhere }, - { jsonColumn: 'OLD.data' }, - ) + const demand: DemandRecord = { options, active: false } + demands.set(options, demand) + try { + const cleanup = await restConfig.onLoadSubset?.(options) + if (cleanup) demand.cleanup = cleanup + } catch (error) { + demands.delete(options) + throw error + } - const compiledView = compileSQLite({ where: combinedWhere }) - - const newDataWhenClause = toInlinedWhereClause(compiledNewData) - const oldDataWhenClause = toInlinedWhereClause(compiledOldData) - const viewWhereClause = toInlinedWhereClause(compiledView) - - await database.writeLock(async (ctx) => { - // Replace any active tracking with one covering the new set of - // predicates. - await flushDiffRecordsWithContext(ctx, appliedReceipts) - await safelyDisposeTracking(ctx) - - disposeTracking = await createDiffTrigger( - { - setupContext: ctx, - when: { - [DiffTriggerOperation.INSERT]: newDataWhenClause, - [DiffTriggerOperation.UPDATE]: `(${newDataWhenClause}) OR (${oldDataWhenClause})`, - [DiffTriggerOperation.DELETE]: oldDataWhenClause, - }, - writeType: (rowId: string) => - collection.has(rowId) ? `update` : `insert`, - batchQuery: ( - lockContext: LockContext, - batchSize: number, - cursor: number, - ) => - lockContext.getAll( - `SELECT * FROM ${viewName} WHERE ${viewWhereClause} LIMIT ? OFFSET ?`, - [batchSize, cursor], - ), - }, - appliedReceipts, - ) - }) - await Promise.all(appliedReceipts) + if ( + // The user hook can reenter cleanup. + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + stopped || + releasedSubsets.has(options) || + options.signal?.aborted || + demands.get(options) !== demand + ) { + demands.delete(options) + demand.cleanup?.() + return + } + + demand.active = true + trackingRevision++ + await rebuildTracking() } const toInlinedWhereClause = (compiled: { @@ -680,56 +772,118 @@ function createPowerSyncCollectionConfig< ) } - const unloadSubset = async (options: LoadSubsetOptions) => { - releasedSubsets.add(options) - unloadSubsetCallbacks.get(options)?.() - unloadSubsetCallbacks.delete(options) - - const idx = activeWhereExpressions.indexOf(options.where) - if (idx !== -1) { - activeWhereExpressions.splice(idx, 1) + const cleanupDemand = (demand: DemandRecord): void => { + demands.delete(demand.options) + try { + demand.cleanup?.() + } catch (error) { + database.logger.error( + `Could not clean up subset hook for ${viewName}`, + error, + ) } + } - // Evict rows that were exclusively loaded by the departing predicate. - // These are rows matching the departing WHERE that are no longer covered - // by any remaining active predicate. + const performPhysicalRelease = async ( + options: LoadSubsetOptions, + ): Promise => { const compiledDeparting = compileSQLite({ where: options.where }) const departingWhereSQL = toInlinedWhereClause(compiledDeparting) + let rowsToEvict: Array<{ id: string }> + for (;;) { + if (stopped) return + const revision = trackingRevision + const active = activeWhereExpressions() + let evictionSQL: string + if (active.length === 0) { + evictionSQL = `SELECT id FROM ${viewName} WHERE ${departingWhereSQL}` + } else { + const combinedRemaining = + active.length === 1 + ? active[0]! + : or(active[0], active[1], ...active.slice(2)) + const compiledRemaining = compileSQLite({ + where: combinedRemaining, + }) + const remainingWhereSQL = toInlinedWhereClause(compiledRemaining) + evictionSQL = `SELECT id FROM ${viewName} WHERE (${departingWhereSQL}) AND NOT (${remainingWhereSQL})` + } - let evictionSQL: string - if (activeWhereExpressions.length === 0) { - evictionSQL = `SELECT id FROM ${viewName} WHERE ${departingWhereSQL}` - } else { - const combinedRemaining = - activeWhereExpressions.length === 1 - ? activeWhereExpressions[0]! - : or( - activeWhereExpressions[0], - activeWhereExpressions[1], - ...activeWhereExpressions.slice(2), - ) - const compiledRemaining = compileSQLite({ - where: combinedRemaining, - }) - const remainingWhereSQL = toInlinedWhereClause(compiledRemaining) - evictionSQL = `SELECT id FROM ${viewName} WHERE (${departingWhereSQL}) AND NOT (${remainingWhereSQL})` + rowsToEvict = await database.getAll<{ id: string }>(evictionSQL) + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition -- cleanup can run during the query + if (stopped) return + if (trackingRevision === revision) break } - - const rowsToEvict = await database.getAll<{ id: string }>(evictionSQL) if (rowsToEvict.length > 0) { begin() for (const { id } of rowsToEvict) { write({ type: `delete`, key: id }) } - // Eviction does not establish new subset coverage. Keep trigger - // replacement in the same unload turn even when this delete waits - // behind a persisting mutation; the later load tracks its own - // establishing receipts. void commit() } + await rebuildTracking() + } + + function scheduleReleaseDrain(delay = 0): void { + if (stopped || drainingReleases || releaseRetryTimer) return + if (delay > 0) { + releaseRetryTimer = setTimeout(() => { + releaseRetryTimer = undefined + void drainReleases() + }, delay) + return + } + void drainReleases() + } + + async function drainReleases(): Promise { + if (stopped || drainingReleases) return + drainingReleases = true + let retryDelay = 0 + try { + const attempts = pendingReleases.length + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition -- each release can reenter cleanup + for (let index = 0; !stopped && index < attempts; index++) { + const pending = pendingReleases.shift()! + try { + await performPhysicalRelease(pending.options) + } catch (error) { + pending.failures++ + pendingReleases.push(pending) + const delay = Math.min( + 1000 * 2 ** (pending.failures - 1), + 30000, + ) + retryDelay = + retryDelay === 0 ? delay : Math.min(retryDelay, delay) + database.logger.error( + `Could not release subset tracking for ${viewName}; retrying`, + error, + ) + } + } + } finally { + drainingReleases = false + } + if (pendingReleases.length > 0) scheduleReleaseDrain(retryDelay) + } - // Recreate the diff trigger for the remaining active WHERE expressions. - await loadSubset() + const unloadSubset = (options: LoadSubsetOptions): void => { + releasedSubsets.add(options) + const demand = demands.get(options) + if (!demand) return + + const wasActive = demand.active + if (wasActive) trackingRevision++ + cleanupDemand(demand) + + if (wasActive) { + pendingReleases.push({ options, failures: 0 }) + // New work must not wait for another release's backoff. + clearTimeout(releaseRetryTimer) + releaseRetryTimer = undefined + scheduleReleaseDrain() + } } markReady() @@ -737,16 +891,19 @@ function createPowerSyncCollectionConfig< return { cleanup: () => { stopped = true + clearTimeout(releaseRetryTimer) + releaseRetryTimer = undefined database.logger.info( `Sync has been stopped for ${viewName} into ${trackedTableName}`, ) abortController.abort() - for (const cleanup of unloadSubsetCallbacks.values()) cleanup() - unloadSubsetCallbacks.clear() - activeWhereExpressions.length = 0 + for (const demand of demands.values()) { + cleanupDemand(demand) + } + pendingReleases.length = 0 }, loadSubset: (options: LoadSubsetOptions) => loadSubset(options), - unloadSubset: (options: LoadSubsetOptions) => unloadSubset(options), + unloadSubset, } } }, diff --git a/packages/powersync-db-collection/tests/load-hooks.test.ts b/packages/powersync-db-collection/tests/load-hooks.test.ts index cc428816e8..b5094f6156 100644 --- a/packages/powersync-db-collection/tests/load-hooks.test.ts +++ b/packages/powersync-db-collection/tests/load-hooks.test.ts @@ -2,6 +2,7 @@ import { randomUUID } from 'node:crypto' import { tmpdir } from 'node:os' import { PowerSyncDatabase, Schema, Table, column } from '@powersync/node' import { createCollection, createLiveQueryCollection, eq } from '@tanstack/db' +import pDefer from 'p-defer' import { describe, expect, it, onTestFinished, vi } from 'vitest' import { powerSyncCollectionOptions } from '../src' @@ -91,6 +92,34 @@ describe(`Sync Streams`, () => { expect(collection.status).toBe(`error`) }) + it(`eager mode: releases a load hook that resolves after cleanup`, async () => { + const db = await createDatabase() + const releaseLoad = pDefer() + const loadStarted = pDefer() + const cleanupLoad = vi.fn() + const createDiffTrigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockResolvedValue(async () => {}) + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + onLoad: async () => { + loadStarted.resolve() + await releaseLoad.promise + return cleanupLoad + }, + }), + ) + + await loadStarted.promise + collection.cleanup() + releaseLoad.resolve() + + await vi.waitFor(() => expect(cleanupLoad).toHaveBeenCalledOnce()) + expect(createDiffTrigger).not.toHaveBeenCalled() + }) + it(`on-demand mode: should call onLoadSubset/onUnloadSubset for each live query`, async () => { const db = await createDatabase() await createTestProducts(db) diff --git a/packages/powersync-db-collection/tests/on-demand-sync.test.ts b/packages/powersync-db-collection/tests/on-demand-sync.test.ts index 8d1dc34122..6888524894 100644 --- a/packages/powersync-db-collection/tests/on-demand-sync.test.ts +++ b/packages/powersync-db-collection/tests/on-demand-sync.test.ts @@ -12,8 +12,10 @@ import { lt, or, } from '@tanstack/db' +import pDefer from 'p-defer' import { describe, expect, it, onTestFinished, vi } from 'vitest' import { powerSyncCollectionOptions } from '../src' +import type { LoadSubsetOptions } from '@tanstack/db' const APP_SCHEMA = new Schema({ products: new Table({ @@ -2131,63 +2133,82 @@ describe(`On-Demand Sync Mode`, () => { ) }) - it(`should resolve isPersisted when all live queries are cleaned up during a pending mutation`, async () => { - const db = await createDatabase() - await createTestProducts(db) - - const collection = createCollection( - powerSyncCollectionOptions({ - database: db, - table: APP_SCHEMA.props.products, - syncMode: `on-demand`, - }), - ) - onTestFinished(() => collection.cleanup()) - await collection.stateWhenReady() - - // Start with 1 live query (electronics) - const electronicsQuery = createLiveQueryCollection({ - query: (q) => - q - .from({ product: collection }) - .where(({ product }) => eq(product.category, `electronics`)) - .select(({ product }) => ({ - id: product.id, - name: product.name, - price: product.price, - category: product.category, - })), - }) + it.each([`insert`, `update`, `delete`] as const)( + `persists a pending %s when its last live query is cleaned up`, + async (operation) => { + const db = await createDatabase() + await createTestProducts(db) + + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + syncMode: `on-demand`, + }), + ) + onTestFinished(() => collection.cleanup()) + await collection.stateWhenReady() - await electronicsQuery.preload() + // Start with 1 live query (electronics) + const electronicsQuery = createLiveQueryCollection({ + query: (q) => + q + .from({ product: collection }) + .where(({ product }) => eq(product.category, `electronics`)) + .select(({ product }) => ({ + id: product.id, + name: product.name, + price: product.price, + category: product.category, + })), + }) - await vi.waitFor( - () => { - expect(electronicsQuery.size).toBe(3) - }, - { timeout: 2000 }, - ) + await electronicsQuery.preload() - // Insert a new electronics product — creates a pending mutation - const insertResult = collection.insert({ - id: randomUUID(), - name: `New Gadget`, - price: 99, - category: `electronics`, - }) + await vi.waitFor( + () => { + expect(electronicsQuery.size).toBe(3) + }, + { timeout: 2000 }, + ) - // Immediately clean up the only live query — triggers unloadSubset → loadSubset - // with 0 predicates (early-return path), which must still call resolveAllPendingFor - electronicsQuery.cleanup() + const existing = Array.from(electronicsQuery.values())[0]! + const id = operation === `insert` ? randomUUID() : existing.id + const mutation = + operation === `insert` + ? collection.insert({ + id, + name: `New Gadget`, + price: 99, + category: `electronics`, + }) + : operation === `update` + ? collection.update(id, (draft) => { + draft.name = `New Gadget` + }) + : collection.delete(id) + let settled = false + const observed = mutation.isPersisted.promise.then( + () => { + settled = true + return { status: `fulfilled` as const } + }, + (error: unknown) => { + settled = true + return { status: `rejected` as const, reason: error } + }, + ) - // isPersisted.promise should resolve — if the bug is present, this hangs forever - await vi.waitFor( - async () => { - await insertResult.isPersisted.promise - }, - { timeout: 5000 }, - ) - }) + // Dropping the last demand must still drain the mutation's diff record + // before removing the trigger that acknowledges its persistence. + electronicsQuery.cleanup() + await vi.waitFor(() => expect(settled).toBe(true), { timeout: 2000 }) + expect(await observed).toEqual({ status: `fulfilled` }) + expect( + await db.getAll(`SELECT id, name FROM products WHERE id = ?`, [id]), + ).toEqual(operation === `delete` ? [] : [{ id, name: `New Gadget` }]) + }, + ) }) describe(`Tracking lifecycle`, () => { @@ -2229,6 +2250,590 @@ describe(`On-Demand Sync Mode`, () => { }) } + function startOnDemandSync( + db: PowerSyncDatabase, + settings: { + onLoadSubset?: ( + options: LoadSubsetOptions, + ) => void | (() => void) | Promise void)> + syncBatchSize?: number + } = {}, + overrides: Partial<{ + begin: ReturnType + write: ReturnType + commit: ReturnType + }> = {}, + ) { + const begin = overrides.begin ?? vi.fn() + const write = overrides.write ?? vi.fn() + const commit = overrides.commit ?? vi.fn(() => true) + const config = powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + syncMode: `on-demand`, + onLoadSubset: settings.onLoadSubset, + syncBatchSize: settings.syncBatchSize, + }) + const sync = config.sync.sync({ + collection: { status: `ready`, has: () => false }, + begin, + write, + commit, + markReady: vi.fn(), + markError: vi.fn(), + truncate: vi.fn(), + } as never) + const loadSubset = + sync && typeof sync !== `function` ? sync.loadSubset : undefined + const unloadSubset = + sync && typeof sync !== `function` ? sync.unloadSubset : undefined + if (!sync || typeof sync === `function` || !loadSubset || !unloadSubset) { + throw new Error(`Expected on-demand sync controls`) + } + return { sync, loadSubset, unloadSubset, begin, write, commit } + } + + it(`does not publish a provisional or rejected subset`, async () => { + const db = await createDatabase() + const firstHook = pDefer() + const hookFailure = new Error(`subset hook failed`) + const onLoadSubset = vi + .fn() + .mockReturnValueOnce(firstHook.promise) + .mockRejectedValueOnce(hookFailure) + .mockResolvedValueOnce(undefined) + const createDiffTrigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockResolvedValue(vi.fn()) + const { sync, loadSubset } = startOnDemandSync(db, { onLoadSubset }) + + try { + const provisional = loadSubset({ + where: eq(`category`, `electronics`), + }) + await vi.waitFor(() => expect(onLoadSubset).toHaveBeenCalledOnce()) + await expect( + loadSubset({ where: eq(`category`, `outdoors`) }), + ).rejects.toBe(hookFailure) + await loadSubset({ where: eq(`category`, `clothing`) }) + + const when = createDiffTrigger.mock.calls.at(-1)?.[0].when + expect(when?.INSERT).toContain(`clothing`) + expect(when?.INSERT).not.toContain(`electronics`) + expect(when?.INSERT).not.toContain(`outdoors`) + + firstHook.resolve() + await provisional + } finally { + firstHook.resolve() + sync.cleanup?.() + } + }) + + it(`does not acquire a subset released during startup`, async () => { + const db = await createDatabase() + const onLoadSubset = vi.fn() + const createDiffTrigger = vi.spyOn(db.triggers, `createDiffTrigger`) + const { sync, loadSubset, unloadSubset } = startOnDemandSync(db, { + onLoadSubset, + }) + const controller = new AbortController() + const request = { + where: eq(`category`, `electronics`), + signal: controller.signal, + } + + const load = loadSubset(request) + controller.abort() + unloadSubset(request) + + try { + await load + expect(onLoadSubset).not.toHaveBeenCalled() + expect(createDiffTrigger).not.toHaveBeenCalled() + } finally { + sync.cleanup?.() + } + }) + + it(`settles concurrent loads only after the latest trigger is live`, async () => { + const db = await createDatabase() + const locks: Array<() => Promise> = [] + vi.spyOn(db, `writeLock`).mockImplementation( + (callback) => + new Promise((resolve, reject) => { + locks.push(async () => { + try { + await callback({} as never) + resolve(undefined as never) + } catch (error) { + reject(error) + } + }) + }) as never, + ) + vi.spyOn(db, `getAll`).mockResolvedValue([]) + const createDiffTrigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockResolvedValue(vi.fn()) + const { sync, loadSubset } = startOnDemandSync(db) + let firstSettled = false + let secondSettled = false + + const first = Promise.resolve( + loadSubset({ where: eq(`category`, `electronics`) }), + ).then(() => { + firstSettled = true + }) + await vi.waitFor(() => expect(locks).toHaveLength(1)) + const second = Promise.resolve( + loadSubset({ where: eq(`category`, `clothing`) }), + ).then(() => { + secondSettled = true + }) + + try { + await locks[0]!() + expect(firstSettled).toBe(false) + expect(secondSettled).toBe(false) + expect(createDiffTrigger).not.toHaveBeenCalled() + + await vi.waitFor(() => expect(locks).toHaveLength(2)) + await locks[1]!() + await Promise.all([first, second]) + + expect(createDiffTrigger).toHaveBeenCalledOnce() + const when = createDiffTrigger.mock.calls[0]?.[0].when + expect(when?.INSERT).toContain(`electronics`) + expect(when?.INSERT).toContain(`clothing`) + } finally { + sync.cleanup?.() + await Promise.all(locks.map((run) => run())) + await Promise.allSettled([first, second]) + } + }) + + it(`disposes a trigger superseded while it is being created`, async () => { + const db = await createDatabase() + const triggerStarted = pDefer() + const finishTrigger = pDefer() + const staleDispose = vi.fn(async () => {}) + const currentDispose = vi.fn(async () => {}) + const createDiffTrigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockImplementationOnce(async () => { + triggerStarted.resolve() + await finishTrigger.promise + return staleDispose + }) + .mockResolvedValueOnce(currentDispose) + const { sync, loadSubset } = startOnDemandSync(db) + const first = Promise.resolve( + loadSubset({ where: eq(`category`, `electronics`) }), + ) + + try { + await triggerStarted.promise + const second = Promise.resolve( + loadSubset({ where: eq(`category`, `clothing`) }), + ) + finishTrigger.resolve() + await Promise.all([first, second]) + + expect(createDiffTrigger).toHaveBeenCalledTimes(2) + expect(staleDispose).toHaveBeenCalledOnce() + expect(currentDispose).not.toHaveBeenCalled() + } finally { + finishTrigger.resolve() + sync.cleanup?.() + await first + } + }) + + it(`waits for every applied batch before settling a subset`, async () => { + const db = await createDatabase() + const receipts: Array>> = [] + const rows = [ + { id: `a`, name: `A`, price: 1, category: `electronics` }, + { id: `b`, name: `B`, price: 2, category: `electronics` }, + ] + vi.spyOn(db.triggers, `createDiffTrigger`).mockImplementation( + async (options) => { + let cursor = 0 + await options.hooks?.beforeCreate?.({ + getAll: async () => rows.slice(cursor, ++cursor), + } as never) + return vi.fn() + }, + ) + const commit = vi.fn(() => { + const receipt = pDefer() + receipts.push(receipt) + return receipt.promise + }) + const { sync, loadSubset } = startOnDemandSync( + db, + { syncBatchSize: 1 }, + { commit }, + ) + let settled = false + const load = Promise.resolve( + loadSubset({ where: eq(`category`, `electronics`) }), + ).then(() => { + settled = true + }) + + try { + await vi.waitFor(() => expect(receipts).toHaveLength(3)) + receipts[0]!.resolve() + receipts[1]!.resolve() + await Promise.resolve() + expect(settled).toBe(false) + + receipts[2]!.resolve() + await load + expect(settled).toBe(true) + } finally { + receipts.forEach((receipt) => receipt.resolve()) + sync.cleanup?.() + await load + } + }) + + it(`does not start queued tracking after cleanup`, async () => { + const db = await createDatabase() + const lockQueued = pDefer() + let runLock!: () => Promise + vi.spyOn(db, `writeLock`).mockImplementation( + (callback) => + new Promise((resolve, reject) => { + runLock = async () => { + try { + await callback({} as never) + resolve(undefined as never) + } catch (error) { + reject(error) + } + } + lockQueued.resolve() + }) as never, + ) + const createDiffTrigger = vi + .spyOn(db.triggers, `createDiffTrigger`) + .mockResolvedValue(vi.fn()) + const { sync, loadSubset } = startOnDemandSync(db) + + const load = loadSubset({ where: eq(`category`, `electronics`) }) + await lockQueued.promise + sync.cleanup?.() + await runLock() + await load + + expect(createDiffTrigger).not.toHaveBeenCalled() + }) + + it(`cleans each acquired subset at most once during reentrant cleanup`, async () => { + const db = await createDatabase() + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const first = { where: eq(`category`, `electronics`) } + const second = { where: eq(`category`, `clothing`) } + const firstCleanup = vi.fn() + const secondCleanup = vi.fn(() => started.unloadSubset(first)) + const onLoadSubset = vi.fn((options: LoadSubsetOptions) => + options === first ? firstCleanup : secondCleanup, + ) + const started = startOnDemandSync(db, { onLoadSubset }) + + await Promise.all([started.loadSubset(first), started.loadSubset(second)]) + started.sync.cleanup?.() + + expect(firstCleanup).toHaveBeenCalledOnce() + expect(secondCleanup).toHaveBeenCalledOnce() + }) + + it(`does not repeat release work started by a reentrant cleanup`, async () => { + const db = await createDatabase() + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const getAll = vi.spyOn(db, `getAll`).mockResolvedValue([]) + const first = { where: eq(`category`, `electronics`) } + const second = { where: eq(`category`, `clothing`) } + const onLoadSubset = vi.fn((options: LoadSubsetOptions) => + options === first ? () => started.unloadSubset(second) : undefined, + ) + const started = startOnDemandSync(db, { onLoadSubset }) + + try { + await Promise.all([ + started.loadSubset(first), + started.loadSubset(second), + ]) + started.unloadSubset(first) + await vi.waitFor(() => + expect( + getAll.mock.calls.some(([sql]) => + String(sql).includes(`electronics`), + ), + ).toBe(true), + ) + + expect( + getAll.mock.calls.filter(([sql]) => String(sql).includes(`clothing`)), + ).toHaveLength(1) + } finally { + started.sync.cleanup?.() + } + }) + + it(`does not create tracking when change observation cannot start`, async () => { + const db = await createDatabase() + const startupError = new Error(`change observation failed`) + vi.spyOn(db.logger, `error`).mockImplementation(() => {}) + vi.spyOn(db, `onChangeWithCallback`).mockImplementation(() => { + throw startupError + }) + const createDiffTrigger = vi.spyOn(db.triggers, `createDiffTrigger`) + const collection = makeCollection(db) + onTestFinished(() => collection.cleanup()) + await collection.stateWhenReady() + const query = categoryQuery(collection, `electronics`) + onTestFinished(() => query.cleanup()) + + await expect(query.preload()).rejects.toBe(startupError) + expect(createDiffTrigger).not.toHaveBeenCalled() + }) + + it(`flushes a change observed while eager tracking starts`, async () => { + const db = await createDatabase() + await createTestProducts(db) + let flush: + | ((event: { changedTables: Array }) => Promise | void) + | undefined + vi.spyOn(db, `onChangeWithCallback`).mockImplementation((handler) => { + flush = handler?.onChange + return () => {} + }) + const triggerCreated = pDefer() + const publishTrigger = pDefer() + const createDiffTrigger = db.triggers.createDiffTrigger.bind(db.triggers) + vi.spyOn(db.triggers, `createDiffTrigger`).mockImplementation( + async (options) => { + const dispose = await createDiffTrigger(options) + triggerCreated.resolve() + await publishTrigger.promise + return dispose + }, + ) + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + }), + ) + onTestFinished(() => collection.cleanup()) + + await triggerCreated.promise + await db.execute(` + INSERT INTO products (id, name, price, category) + VALUES ('during-startup', 'During startup', 300, 'electronics') + `) + const observed = Promise.resolve( + flush?.({ + changedTables: [collection.utils.getMeta().trackedTableName], + }), + ) + publishTrigger.resolve() + await Promise.all([observed, collection.stateWhenReady()]) + + expect(collection.get(`during-startup`)?.name).toBe(`During startup`) + }) + + it(`disposes eager tracking that finishes after cleanup`, async () => { + const db = await createDatabase() + vi.spyOn(db, `onChangeWithCallback`).mockImplementation(() => () => {}) + const triggerStarted = pDefer() + const finishTrigger = pDefer() + const dispose = vi.fn(async () => {}) + vi.spyOn(db.triggers, `createDiffTrigger`).mockImplementation( + async () => { + triggerStarted.resolve() + await finishTrigger.promise + return dispose + }, + ) + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + }), + ) + + await triggerStarted.promise + collection.cleanup() + finishTrigger.resolve() + + await vi.waitFor(() => expect(dispose).toHaveBeenCalledOnce()) + }) + + it(`reports a source error when a rebuild removes tracking and cannot replace it`, async () => { + const db = await createDatabase() + const collection = createCollection( + powerSyncCollectionOptions({ + database: db, + table: APP_SCHEMA.props.products, + syncMode: `on-demand`, + }), + ) + const failure = new Error(`trigger installation failed`) + try { + await collection._sync.loadSubset({ + where: eq(`category`, `electronics`), + }) + expect(collection.status).toBe(`ready`) + vi.spyOn(db.triggers, `createDiffTrigger`).mockRejectedValueOnce( + failure, + ) + await expect( + Promise.resolve( + collection._sync.loadSubset({ where: eq(`category`, `clothing`) }), + ), + ).rejects.toBe(failure) + expect(collection.status).toBe(`error`) + } finally { + await collection.cleanup() + } + }) + + it(`retries a failed physical release`, async () => { + vi.useFakeTimers() + const db = await createDatabase() + vi.spyOn(db.logger, `error`).mockImplementation(() => {}) + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const getAll = vi + .spyOn(db, `getAll`) + .mockRejectedValueOnce(new Error(`transient eviction failure`)) + .mockResolvedValueOnce([]) + const { sync, loadSubset, unloadSubset } = startOnDemandSync(db) + const request = { where: eq(`category`, `electronics`) } + + try { + await loadSubset(request) + expect(unloadSubset(request)).toBeUndefined() + await vi.waitFor(() => expect(getAll).toHaveBeenCalledOnce()) + await vi.advanceTimersByTimeAsync(1_000) + await vi.waitFor(() => expect(getAll).toHaveBeenCalledTimes(2)) + } finally { + sync.cleanup?.() + await vi.runOnlyPendingTimersAsync() + vi.useRealTimers() + } + }) + + it(`does not let one failed release block another`, async () => { + vi.useFakeTimers() + const db = await createDatabase() + vi.spyOn(db.logger, `error`).mockImplementation(() => {}) + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const getAll = vi + .spyOn(db, `getAll`) + .mockImplementation((sql) => + String(sql).includes(`electronics`) + ? Promise.reject(new Error(`persistent eviction failure`)) + : Promise.resolve([]), + ) + const { sync, loadSubset, unloadSubset } = startOnDemandSync(db) + const failing = { where: eq(`category`, `electronics`) } + const succeeding = { where: eq(`category`, `clothing`) } + + try { + await Promise.all([loadSubset(failing), loadSubset(succeeding)]) + unloadSubset(failing) + unloadSubset(succeeding) + await vi.waitFor(() => expect(getAll).toHaveBeenCalled()) + await vi.advanceTimersByTimeAsync(1_000) + + expect( + getAll.mock.calls.some(([sql]) => { + const query = String(sql) + return query.includes(`clothing`) && !query.includes(`electronics`) + }), + ).toBe(true) + } finally { + sync.cleanup?.() + await vi.runOnlyPendingTimersAsync() + vi.useRealTimers() + } + }) + + it(`evicts a newly released demand without waiting for another demand's retry timer`, async () => { + vi.useFakeTimers() + const db = await createDatabase() + vi.spyOn(db.logger, `error`).mockImplementation(() => {}) + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const getAll = vi + .spyOn(db, `getAll`) + .mockImplementation((sql) => + String(sql).includes(`electronics`) + ? Promise.reject(new Error(`eviction failed`)) + : Promise.resolve([]), + ) + const { sync, loadSubset, unloadSubset } = startOnDemandSync(db) + const first = { where: eq(`category`, `electronics`) } + const second = { where: eq(`category`, `clothing`) } + try { + await Promise.all([loadSubset(first), loadSubset(second)]) + unloadSubset(first) + for (let turn = 0; turn < 30; turn++) await Promise.resolve() + const callsAfterFailure = getAll.mock.calls.length + expect(callsAfterFailure).toBe(1) + unloadSubset(second) + for (let turn = 0; turn < 30; turn++) await Promise.resolve() + expect( + getAll.mock.calls + .slice(callsAfterFailure) + .some(([sql]) => String(sql).includes(`clothing`)), + ).toBe(true) + } finally { + sync.cleanup?.() + await vi.runOnlyPendingTimersAsync() + vi.useRealTimers() + } + }) + + it(`rechecks active demand before evicting released rows`, async () => { + const db = await createDatabase() + vi.spyOn(db.triggers, `createDiffTrigger`).mockResolvedValue(vi.fn()) + const firstEviction = pDefer>() + const getAll = vi + .spyOn(db, `getAll`) + .mockReturnValueOnce(firstEviction.promise) + .mockResolvedValueOnce([]) + const write = vi.fn() + const { sync, loadSubset, unloadSubset } = startOnDemandSync( + db, + {}, + { write }, + ) + const departing = { where: eq(`category`, `electronics`) } + + try { + await loadSubset(departing) + unloadSubset(departing) + await vi.waitFor(() => expect(getAll).toHaveBeenCalledOnce()) + + await loadSubset({ where: eq(`category`, `clothing`) }) + firstEviction.resolve([{ id: `now-owned` }]) + + await vi.waitFor(() => expect(getAll).toHaveBeenCalledTimes(2)) + expect(write).not.toHaveBeenCalledWith({ + type: `delete`, + key: `now-owned`, + }) + } finally { + firstEviction.resolve([]) + sync.cleanup?.() + } + }) + it(`should start tracking again when a subset is loaded after every subset was unloaded`, async () => { const db = await createDatabase() await createTestProducts(db) diff --git a/packages/powersync-db-collection/tests/transactor-readiness.test.ts b/packages/powersync-db-collection/tests/transactor-readiness.test.ts new file mode 100644 index 0000000000..dad011bda3 --- /dev/null +++ b/packages/powersync-db-collection/tests/transactor-readiness.test.ts @@ -0,0 +1,69 @@ +import { createCollection, createTransaction } from '@tanstack/db' +import { expect, it, vi } from 'vitest' +import { PowerSyncTransactor } from '../src/PowerSyncTransactor' +import type { AbstractPowerSyncDatabase } from '@powersync/common' + +it.each([`cleanup`, `error`, `ready`] as const)( + `settles a transaction waiting for source readiness on %s`, + async (outcome) => { + const writeTransaction = vi + .fn() + .mockResolvedValue({ whenComplete: Promise.resolve() }) + // This boundary must settle before taking a database lock; no SQL runs. + const transactor = new PowerSyncTransactor({ + database: { writeTransaction } as unknown as AbstractPowerSyncDatabase, + }) + let markSourceReady!: () => void + const collection = createCollection<{ id: string }>({ + getKey: (row) => row.id, + sync: { + sync: ({ markReady }) => { + markSourceReady = markReady + return {} + }, + }, + }) + collection.startSyncImmediate() + const transaction = createTransaction({ + autoCommit: false, + mutationFn: async () => {}, + }) + transaction.mutate(() => collection.insert({ id: `pending` })) + let result: { error: unknown } | { ready: true } | undefined + const waiting = transactor.applyTransaction(transaction).then( + () => { + result = { ready: true } + }, + (error: unknown) => { + result = { error } + }, + ) + const failure = new Error(`source failed before readiness`) + try { + expect(collection.status).toBe(`loading`) + if (outcome === `cleanup`) await collection.cleanup() + else if (outcome === `error`) collection._lifecycle.markError(failure) + else markSourceReady() + // Drain promise reactions without waiting on the possibly orphaned wait. + for (let turn = 0; turn < 10; turn++) await Promise.resolve() + expect(result).toBeDefined() + expect(result).toEqual( + outcome === `ready` + ? { ready: true } + : { + error: + outcome === `error` + ? failure + : expect.objectContaining({ name: `AbortError` }), + }, + ) + await waiting + expect(writeTransaction).toHaveBeenCalledTimes( + outcome === `ready` ? 1 : 0, + ) + } finally { + transaction.rollback() + await collection.cleanup() + } + }, +) diff --git a/packages/query-db-collection/package.json b/packages/query-db-collection/package.json index c067e7fcae..2347266711 100644 --- a/packages/query-db-collection/package.json +++ b/packages/query-db-collection/package.json @@ -21,6 +21,7 @@ "dev": "vite build --watch", "lint": "eslint . --fix", "test": "vitest run", + "test:oracles": "vitest run tests/includes-work-counter-oracle.test.ts tests/load-subset-lifecycle-oracle.test.ts tests/ownership-lifecycle.oracle.test.ts", "test:e2e": "vitest run --config vitest.e2e.config.ts" }, "type": "module", diff --git a/packages/query-db-collection/src/query.ts b/packages/query-db-collection/src/query.ts index e6d95e3f57..620a3c5bb3 100644 --- a/packages/query-db-collection/src/query.ts +++ b/packages/query-db-collection/src/query.ts @@ -1,5 +1,6 @@ import { QueryObserver, hashKey } from '@tanstack/query-core' import { + LoadSubsetOperationAbortedError, deepEquals, getLoadSubsetDemandKey, withCollectionConfigFactory, @@ -821,6 +822,10 @@ export function queryCollectionOptions( // 3. Decrements refcount and GCs rows where count reaches 0 const queryRefCounts = new Map() + // Eager startup holds one reference until cleanup. Cache removal detaches + // observation, not that ownership or its rows. + let ensureEagerSubscription = () => {} + const addRowOwner = (rowKey: string | number, hashedQueryKey: string) => { const owners = rowToQueries.get(rowKey) || new Set() owners.add(hashedQueryKey) @@ -890,6 +895,7 @@ export function queryCollectionOptions( // Track whether sync has been started let syncStarted = false let startupRetentionSettled = false + const pendingStartupLoads = new Set() const retainedQueriesPendingRevalidation = new Set() const pendingResultApplications = new Map>() const failedResultApplications = new Map() @@ -1280,7 +1286,7 @@ export function queryCollectionOptions( ) { unsubscribe() const pending = pendingReadyUnsubscribes.get(hashedQueryKey) - pending?.delete(unsubscribe) + pending?.delete(cancel) if (pending?.size === 0) { pendingReadyUnsubscribes.delete(hashedQueryKey) } @@ -1293,9 +1299,13 @@ export function queryCollectionOptions( } }) }) + const cancel = () => { + unsubscribe() + reject(new LoadSubsetOperationAbortedError()) + } const pending = pendingReadyUnsubscribes.get(hashedQueryKey) ?? new Set() - pending.add(unsubscribe) + pending.add(cancel) pendingReadyUnsubscribes.set(hashedQueryKey, pending) }) @@ -1304,7 +1314,11 @@ export function queryCollectionOptions( queryFunction: typeof queryFn = queryFn, ): true | Promise => { if (!startupRetentionSettled) { + pendingStartupLoads.add(opts) return startupRetentionMaintenancePromise.then(() => { + if (!pendingStartupLoads.delete(opts)) { + throw new LoadSubsetOperationAbortedError() + } const resumed = createQueryFromOpts(opts, queryFunction) return resumed === true ? undefined : resumed }) @@ -1570,14 +1584,20 @@ export function queryCollectionOptions( newItemsMap.forEach((newItem, key) => { const owners = getPersistedOwners(key) - if (!owners.has(hashedQueryKey)) { + const addsOwner = !owners.has(hashedQueryKey) + const insertsRow = !currentSyncedItems.has(key) + if (addsOwner) { owners.add(hashedQueryKey) - setPersistedOwners(key, owners) } addRowOwner(key, hashedQueryKey) - if (!currentSyncedItems.has(key)) { + if (insertsRow) { write({ type: `insert`, value: newItem }) } + if (addsOwner || insertsRow) { + // An insert clears stale metadata for its key. Stage ownership + // afterward so rows and ownership commit as one state change. + setPersistedOwners(key, owners) + } }) const applied = commit(signal) @@ -1780,6 +1800,9 @@ export function queryCollectionOptions( hashedQueryKey: string, ) => { if (!isSubscribed(hashedQueryKey)) { + // Cache removal does not retire eager ownership. Reattach the observer + // to the current cache entry before subscribing to its updates. + if (syncMode === `eager`) observer.setOptions(observer.options) const cachedQueryKey = hashToQueryKey.get(hashedQueryKey)! const handleQueryResult = makeQueryResultHandler(cachedQueryKey) const unsubscribeFn = observer.subscribe(handleQueryResult) @@ -1805,6 +1828,16 @@ export function queryCollectionOptions( unsubscribes.clear() } + ensureEagerSubscription = () => { + if (syncMode !== `eager`) return + state.observers.forEach((observer, key) => { + const query = observer.getCurrentQuery() + if (queryClient.getQueryCache().get(query.queryHash) !== query) { + subscribeToQuery(observer, key) + } + }) + } + // Mark that sync has started syncStarted = true @@ -1822,11 +1855,10 @@ export function queryCollectionOptions( // If syncMode is eager, create the initial query without any predicates if (syncMode === `eager`) { - // Catch any errors to prevent unhandled rejections - const initialResult = createQueryFromOpts({}) - if (initialResult instanceof Promise) { - initialResult.catch(() => { - // Errors are already handled by the query result handler + const result = createQueryFromOpts({}) + if (result instanceof Promise) { + void result.catch(() => { + // Errors are handled by the query result handler. }) } } else { @@ -1884,7 +1916,11 @@ export function queryCollectionOptions( const shouldWriteMetadata = metadata !== undefined && nextOwnersByRow.size > 0 - const needsTransaction = shouldWriteMetadata || rowsToDelete.length > 0 + const retentionKey = `${QUERY_COLLECTION_GC_PREFIX}${hashedQueryKey}` + const hasRetentionMarker = + metadata?.collection.get(retentionKey) !== undefined + const needsTransaction = + shouldWriteMetadata || rowsToDelete.length > 0 || hasRetentionMarker if (needsTransaction) { begin() } @@ -1907,6 +1943,10 @@ export function queryCollectionOptions( }) } + if (hasRetentionMarker) { + metadata.collection.delete(retentionKey) + } + if (needsTransaction) { commit() } @@ -1935,6 +1975,12 @@ export function queryCollectionOptions( unsubscribePendingReadyListeners(hashedQueryKey) } + // Refcounts are explicit ownership tokens. A cache event can remove the + // observer while an active acquisition still owns this query. + if (refcount > 0) { + return + } + const hasListeners = observer?.hasListeners() ?? false if (hasListeners) { @@ -1944,16 +1990,6 @@ export function queryCollectionOptions( return } - // No listeners means the query is truly idle. - // Even if refcount > 0, we treat hasListeners as authoritative to prevent leaks. - // This can happen if subscriptions are GC'd without calling unloadSubset. - if (refcount > 0) { - console.warn( - `[cleanupQueryIfIdle] Invariant violation: refcount=${refcount} but no listeners. Cleaning up to prevent leak.`, - { hashedQueryKey }, - ) - } - if ( effectivePersistedGcTime !== undefined && metadata && @@ -2005,10 +2041,19 @@ export function queryCollectionOptions( const unsubscribeQueryCache = queryClient .getQueryCache() .subscribe((event) => { - const hashedKey = event.query.queryHash + // Ownership uses our stable key, not the Query client's optional + // custom cache hash function. + const hashedKey = hashKey(event.query.queryKey) if (event.type === `removed`) { // Only cleanup if this is OUR query (we track it) if (hashToQueryKey.has(hashedKey)) { + if (syncMode === `eager`) { + unsubscribes.get(hashedKey)?.() + unsubscribes.delete(hashedKey) + unsubscribePendingReadyListeners(hashedKey) + if (collection.subscriberCount > 0) ensureEagerSubscription() + return + } // TanStack Query GC'd this query after gcTime expired. // Use the guarded cleanup path to avoid deleting rows for active queries. cleanupQueryIfIdle(hashedKey) @@ -2016,7 +2061,9 @@ export function queryCollectionOptions( } }) - const cleanup = async () => { + const cleanup = () => { + pendingStartupLoads.clear() + ensureEagerSubscription = () => {} unsubscribeFromCollectionEvents() unsubscribeFromQueries() persistedRetentionTimers.forEach((timer) => { @@ -2024,7 +2071,6 @@ export function queryCollectionOptions( }) persistedRetentionTimers.clear() - const allQueryKeys = [...hashToQueryKey.values()] const allHashedKeys = new Set([ ...state.observers.keys(), ...queryToRows.keys(), @@ -2039,13 +2085,11 @@ export function queryCollectionOptions( // Unsubscribe from cache events (cleanup already happened above) unsubscribeQueryCache() - // Remove queries from TanStack Query cache - await Promise.all( - allQueryKeys.map(async (qKey) => { - await queryClient.cancelQueries({ queryKey: qKey, exact: true }) - queryClient.removeQueries({ queryKey: qKey, exact: true }) - }), - ) + // Removing a Query destroys it and synchronously cancels its retryer. + // Finish this before a later collection sync can create a replacement. + queryClient.removeQueries({ + predicate: (query) => allHashedKeys.has(hashKey(query.queryKey)), + }) } /** @@ -2073,6 +2117,8 @@ export function queryCollectionOptions( * by TanStack Query, allowing quick remounts to restore data without refetching. */ const unloadSubset = (options: LoadSubsetOptions) => { + // No observer lease exists until startup maintenance has finished. + if (pendingStartupLoads.delete(options)) return // 1. Same predicates → 2. Same queryKey const key = generateQueryKeyFromOptions(options) const hashedQueryKey = hashKey(key) @@ -2123,6 +2169,8 @@ export function queryCollectionOptions( * @returns Promise that resolves when the refetch is complete, with QueryObserverResult */ const refetch: RefetchFn = async (opts) => { + // An idle eager observer still owns rows; refetch must deliver its result. + ensureEagerSubscription() const allQueryKeys = [...hashToQueryKey.values()] const refetchPromises = allQueryKeys.map((qKey) => { const queryObserver = state.observers.get(hashKey(qKey))! @@ -2296,15 +2344,6 @@ export function queryCollectionOptions( } } - if (typeof process !== `undefined` && process.env.NODE_ENV === `test`) { - Object.defineProperty(enhancedInternalSync, `__getOwnershipMapsForTests`, { - value: () => ({ - rowToQueries, - queryToRows, - }), - }) - } - // Create write utils using the manual-sync module const writeUtils = createWriteUtils( () => writeContext, diff --git a/packages/query-db-collection/tests/optimistic-writeback.test.ts b/packages/query-db-collection/tests/optimistic-writeback.test.ts new file mode 100644 index 0000000000..f9ace4cfb9 --- /dev/null +++ b/packages/query-db-collection/tests/optimistic-writeback.test.ts @@ -0,0 +1,114 @@ +import { expect, it } from 'vitest' +import { QueryClient } from '@tanstack/query-core' +import { + createCollection, + createLiveQueryCollection, + createOptimisticAction, + createTransaction, +} from '@tanstack/db' +import { queryCollectionOptions } from '../src/query' + +type Row = { id: string; text: string } + +it.each([`insert`, `upsert`] as const)( + `keeps repeated optimistic writes valid after direct %s acknowledgement`, + async (method) => { + const queryClient = new QueryClient({ + defaultOptions: { queries: { retry: false } }, + }) + const source = createCollection( + queryCollectionOptions({ + queryKey: [`optimistic-writeback`, method], + queryClient, + queryFn: async (): Promise> => [], + getKey: (row) => row.id, + }), + ) + const live = createLiveQueryCollection({ + query: (q) => + q.from({ row: source }).select(({ row }) => ({ + id: row.id, + text: row.text, + })), + }) + const batches: Array> = [] + const subscription = source.subscribeChanges((changes) => { + batches.push(changes.map((change) => change.key)) + }) + const insert = createOptimisticAction({ + onMutate: (row) => source.insert(row), + mutationFn: async (row) => { + await Promise.resolve() + if (method === `insert`) source.utils.writeInsert({ ...row }) + else source.utils.writeUpsert({ ...row }) + }, + }) + const rename = createOptimisticAction({ + onMutate: (row) => + source.update(row.id, (draft) => { + draft.text = row.text + }), + mutationFn: async (row) => { + await Promise.resolve() + source.utils.writeUpdate({ ...row }) + }, + }) + try { + await live.preload() + await insert({ id: `one`, text: `created` }).isPersisted.promise + for (const text of [`renamed`, `renamed again`]) { + await rename({ id: `one`, text }).isPersisted.promise + expect([...live.values()].map((row) => row.text)).toEqual([text]) + } + await insert({ id: `two`, text: `second` }).isPersisted.promise + expect([...live.values()].map((row) => row.text).sort()).toEqual([ + `renamed again`, + `second`, + ]) + for (const keys of batches) expect(new Set(keys).size).toBe(keys.length) + } finally { + subscription.unsubscribe() + await live.cleanup() + await source.cleanup() + queryClient.clear() + } + }, +) + +it(`keeps repeated optimistic updates valid after direct upsert acknowledgement`, async () => { + const queryClient = new QueryClient({ + defaultOptions: { queries: { retry: false } }, + }) + let position = 0 + const source = createCollection( + queryCollectionOptions({ + queryKey: [`optimistic-upsert-rounds`], + queryClient, + queryFn: async () => [{ id: `one`, position }], + getKey: (row) => row.id, + }), + ) + const live = createLiveQueryCollection((q) => q.from({ row: source })) + try { + await live.preload() + for (const next of [1, 2, 3]) { + const tx = createTransaction({ + mutationFn: async () => { + position = next + source.utils.writeUpsert({ id: `one`, position }) + }, + }) + tx.mutate(() => + source.update(`one`, (draft) => { + draft.position += 1 + }), + ) + await tx.isPersisted.promise + expect([...live.values()].map((row) => row.position)).toEqual([next]) + } + } finally { + await live.cleanup() + await source.cleanup() + queryClient.clear() + } +}) diff --git a/packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts b/packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts index ef9baa5da9..0b6816a0f0 100644 --- a/packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts +++ b/packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts @@ -1,8 +1,7 @@ +import { QueryClient, hashKey, isCancelledError } from '@tanstack/query-core' +import { createCollection, eq, getLoadSubsetDemandKey } from '@tanstack/db' import { afterEach, describe, expect, it, vi } from 'vitest' -import { QueryClient } from '@tanstack/query-core' -import { createCollection, eq } from '@tanstack/db' -import { expectAssertionFailure } from '../../db/tests/expected-failure.js' -import { TraceAssertionError } from '../../db/tests/trace-runner.js' +import { createDeferred } from '../../db/src/deferred.js' import { queryCollectionOptions } from '../src/query.js' import type { Collection, SyncMetadataApi } from '@tanstack/db' import type { NonSingleResult } from '../../db/src/types.js' @@ -14,23 +13,19 @@ type Item = { name: string } -type OwnershipMaps = { - rowToQueries: Map> - queryToRows: Map> -} - type MetadataRecorder = { - rowWrites: Array<{ - type: `set` | `delete` - key: string | number - }> + rows: Map + writes: Array<{ type: `set` | `delete`; key: string | number }> } type OwnershipFixtureOptions = { id: string - results: Array> + results: Array | Promise>> syncMode?: `eager` | `on-demand` + customHash?: boolean + staleTime?: number metadataRecorder?: MetadataRecorder + setupMetadata?: (metadata: SyncMetadataApi) => void } type OwnershipFixture = { @@ -42,7 +37,6 @@ type OwnershipFixture = { Item > & NonSingleResult - maps: OwnershipMaps queryClient: QueryClient queryFn: ReturnType Promise>>> } @@ -52,191 +46,25 @@ const detailOnly = { id: `detail`, category: `detail`, name: `Detail` } const listOnly = { id: `list`, category: `list`, name: `List` } const cleanups: Array<() => Promise> = [] -function createQueryClient(): QueryClient { +function createQueryClient( + customHash = false, + staleTime = Number.POSITIVE_INFINITY, +): QueryClient { return new QueryClient({ defaultOptions: { queries: { gcTime: Number.POSITIVE_INFINITY, retry: false, - staleTime: Number.POSITIVE_INFINITY, + staleTime, + queryKeyHashFn: customHash + ? (key) => `custom:${hashKey(key)}` + : undefined, }, }, }) } -function inspectOwnershipMaps(options: { - sync: { sync: unknown } -}): OwnershipMaps { - const sync = options.sync.sync as { - __getOwnershipMapsForTests?: () => OwnershipMaps - } - const maps = sync.__getOwnershipMapsForTests?.() - if (!maps) { - throw new Error(`Ownership-map test inspection is unavailable`) - } - return maps -} - -function sorted(values: Iterable): Array { - return Array.from(values).sort() -} - -function ownersOf(maps: OwnershipMaps, rowId: string): Array { - return sorted(maps.rowToQueries.get(rowId) ?? []) -} - -function onlyOwner(maps: OwnershipMaps, rowId: string): string { - const owners = ownersOf(maps, rowId) - if (owners.length !== 1) { - throw new Error(`Expected exactly one owner for ${rowId}`) - } - return owners[0]! -} - -function otherOwner( - maps: OwnershipMaps, - rowId: string, - knownOwner: string, -): string { - const owners = ownersOf(maps, rowId).filter((owner) => owner !== knownOwner) - if (owners.length !== 1) { - throw new Error(`Expected one new owner for ${rowId}`) - } - return owners[0]! -} - -function rowsOwnedBy( - maps: OwnershipMaps, - queryHash: string, -): Array { - return sorted(maps.queryToRows.get(queryHash) ?? []) -} - -function observerCount(queryClient: QueryClient, queryHash: string): number { - return ( - queryClient - .getQueryCache() - .getAll() - .find((query) => query.queryHash === queryHash) - ?.getObserversCount() ?? 0 - ) -} - -function collectionRows(collection: { - keys: () => Iterable -}): Array { - return sorted(collection.keys()).map(String) -} - -function assertCheckpoint( - checkpoint: number, - actual: unknown, - expected: unknown, -): void { - try { - expect(actual).toEqual(expected) - } catch (error) { - throw new TraceAssertionError(checkpoint, error) - } -} - -function asRecords({ - actual, - expected, -}: { - actual: unknown - expected: unknown -}): - | { - observed: Record - wanted: Record - } - | undefined { - if ( - !actual || - typeof actual !== `object` || - !expected || - typeof expected !== `object` - ) { - return undefined - } - - return { - observed: actual as Record, - wanted: expected as Record, - } -} - -function classifyEagerOwnerLoss(difference: { - actual: unknown - expected: unknown -}): boolean { - const records = asRecords(difference) - if (!records) return false - const { observed, wanted } = records - return ( - observed.status === `ready` && - Array.isArray(observed.rows) && - observed.rows.length === 0 && - observed.owners === 0 && - wanted.status === `ready` && - Array.isArray(wanted.rows) && - wanted.rows.length === 1 && - wanted.rows[0] === shared.id && - wanted.owners === 1 - ) -} - -function classifyInsertedOwnerMetadataLoss(difference: { - actual: unknown - expected: unknown -}): boolean { - const records = asRecords(difference) - if (!records) return false - const { observed, wanted } = records - return ( - Array.isArray(observed.persistedOwners) && - observed.persistedOwners.length === 0 && - Array.isArray(observed.metadataSetKeys) && - observed.metadataSetKeys.length === 1 && - observed.metadataSetKeys[0] === shared.id && - Array.isArray(wanted.persistedOwners) && - wanted.persistedOwners.length === 1 && - typeof wanted.persistedOwners[0] === `string` && - Array.isArray(wanted.metadataSetKeys) && - wanted.metadataSetKeys.length === 1 && - wanted.metadataSetKeys[0] === shared.id - ) -} - -function sameArray(actual: unknown, expected: unknown): boolean { - return ( - Array.isArray(actual) && - Array.isArray(expected) && - actual.length === expected.length && - actual.every((value, index) => value === expected[index]) - ) -} - -function classifyPersistedBaselineLoss(difference: { - actual: unknown - expected: unknown -}): boolean { - const records = asRecords(difference) - if (!records) return false - const { observed, wanted } = records - return ( - sameArray(observed.liveOwners, wanted.liveOwners) && - sameArray(observed.persistedOwners, wanted.insertedOwners) && - Array.isArray(observed.insertedOwners) && - observed.insertedOwners.length === 0 && - Array.isArray(wanted.persistedOwners) && - wanted.persistedOwners.length === 2 && - sameArray(observed.metadataSetKeys, wanted.metadataSetKeys) - ) -} - -function recordMetadataWrites( +function recordMetadata( metadata: SyncMetadataApi, recorder: MetadataRecorder, ): SyncMetadataApi { @@ -244,11 +72,13 @@ function recordMetadataWrites( row: { get: (key) => metadata.row.get(key), set: (key, value) => { - recorder.rowWrites.push({ type: `set`, key }) + recorder.writes.push({ type: `set`, key }) + recorder.rows.set(key, value) metadata.row.set(key, value) }, delete: (key) => { - recorder.rowWrites.push({ type: `delete`, key }) + recorder.writes.push({ type: `delete`, key }) + recorder.rows.delete(key) metadata.row.delete(key) }, }, @@ -266,11 +96,16 @@ function createOwnershipFixture({ results, syncMode = `on-demand`, metadataRecorder, + setupMetadata, + customHash, + staleTime, }: OwnershipFixtureOptions): OwnershipFixture { - const queryClient = createQueryClient() + const queryClient = createQueryClient(customHash, staleTime) const queryFn = vi.fn<() => Promise>>() - results.forEach((result) => queryFn.mockResolvedValueOnce(result)) - queryFn.mockRejectedValue(new Error(`Unexpected ownership-oracle refetch`)) + results.forEach((result) => + queryFn.mockImplementationOnce(() => Promise.resolve(result)), + ) + queryFn.mockRejectedValue(new Error(`Unexpected ownership refetch`)) const baseOptions = queryCollectionOptions({ id, queryClient, @@ -280,10 +115,10 @@ function createOwnershipFixture({ syncMode, startSync: true, }) - const maps = inspectOwnershipMaps(baseOptions) const originalSync = baseOptions.sync + let pendingSetup = setupMetadata const collection = createCollection( - metadataRecorder + metadataRecorder || setupMetadata ? { ...baseOptions, sync: { @@ -291,12 +126,18 @@ function createOwnershipFixture({ if (!params.metadata) { throw new Error(`Sync metadata API is unavailable`) } + const observedMetadata = metadataRecorder + ? recordMetadata(params.metadata, metadataRecorder) + : params.metadata + if (pendingSetup) { + params.begin() + pendingSetup(observedMetadata) + params.commit() + pendingSetup = undefined + } return originalSync.sync({ ...params, - metadata: recordMetadataWrites( - params.metadata, - metadataRecorder, - ), + metadata: observedMetadata, }) }, }, @@ -307,400 +148,368 @@ function createOwnershipFixture({ await collection.cleanup() queryClient.clear() }) + return { collection, queryClient, queryFn } +} - return { collection, maps, queryClient, queryFn } +function rows(collection: { + keys: () => Iterable +}): Array { + return Array.from(collection.keys()).map(String).sort() } function persistedOwners( - rowMetadata: ReadonlyMap, + metadata: ReadonlyMap, rowId: string, ): Array { - const metadata = rowMetadata.get(rowId) - if (!metadata || typeof metadata !== `object`) { - return [] - } - - const queryCollection = (metadata as Record).queryCollection - if (!queryCollection || typeof queryCollection !== `object`) { - return [] - } - + const rowMetadata = metadata.get(rowId) + if (!rowMetadata || typeof rowMetadata !== `object`) return [] + const queryCollection = (rowMetadata as Record) + .queryCollection + if (!queryCollection || typeof queryCollection !== `object`) return [] const owners = (queryCollection as Record).owners - if (!owners || typeof owners !== `object`) { - return [] - } - - return sorted(Object.keys(owners)) -} - -function setMetadataKeys(recorder: MetadataRecorder): Array { - return sorted( - new Set( - recorder.rowWrites - .filter((write) => write.type === `set`) - .map((write) => write.key), - ), - ) + return owners && typeof owners === `object` ? Object.keys(owners).sort() : [] } -describe(`query collection ownership lifecycle oracle`, () => { +describe(`query collection ownership lifecycle`, () => { afterEach(async () => { await Promise.all(cleanups.splice(0).map((cleanup) => cleanup())) }) - it(`keeps query ownership while a reused subset still has an acquisition`, async () => { - const { collection, maps, queryFn } = createOwnershipFixture({ - id: `ownership-shared-acquisition`, + it(`keeps cached rows until the final exact acquisition is released`, async () => { + const { collection, queryFn } = createOwnershipFixture({ + id: `shared-acquisition`, results: [[shared, detailOnly]], }) const subset = { where: eq(`category`, `detail`) } await collection._sync.loadSubset(subset) - const queryHash = onlyOwner(maps, shared.id) - assertCheckpoint( - 0, - { - fetches: queryFn.mock.calls.length, - owners: ownersOf(maps, shared.id), - ownedRows: rowsOwnedBy(maps, queryHash), - }, - { - fetches: 1, - owners: [queryHash], - ownedRows: [detailOnly.id, shared.id], - }, - ) - await collection._sync.loadSubset(subset) - assertCheckpoint( - 1, - { - fetches: queryFn.mock.calls.length, - owners: ownersOf(maps, shared.id), - }, - { fetches: 1, owners: [queryHash] }, - ) + expect(queryFn).toHaveBeenCalledOnce() + expect(rows(collection)).toEqual([detailOnly.id, shared.id]) collection._sync.unloadSubset(subset) - assertCheckpoint( - 2, - { - rows: collectionRows(collection), - owners: ownersOf(maps, shared.id), - }, - { - rows: [detailOnly.id, shared.id], - owners: [queryHash], - }, - ) - + expect(rows(collection)).toEqual([detailOnly.id, shared.id]) collection._sync.unloadSubset(subset) - assertCheckpoint( - 3, - { - rows: collectionRows(collection), - ownershipRows: maps.rowToQueries.size, - ownershipQueries: maps.queryToRows.size, - }, - { rows: [], ownershipRows: 0, ownershipQueries: 0 }, - ) + expect(rows(collection)).toEqual([]) await collection._sync.loadSubset(subset) - assertCheckpoint( - 4, - { - fetches: queryFn.mock.calls.length, - rows: collectionRows(collection), - owners: ownersOf(maps, shared.id), - }, - { - fetches: 1, - rows: [detailOnly.id, shared.id], - owners: [queryHash], - }, - ) + expect(queryFn).toHaveBeenCalledOnce() + expect(rows(collection)).toEqual([detailOnly.id, shared.id]) }) - it(`#1488 retires ownership with its observer and reacquires it from cached data`, async () => { - const { collection, maps, queryClient, queryFn } = createOwnershipFixture({ - id: `ownership-observer-reuse-1488`, + it(`removes only rows whose final query owner is released`, async () => { + const { collection, queryFn } = createOwnershipFixture({ + id: `overlapping-acquisitions`, results: [ [shared, detailOnly], [shared, listOnly], ], }) - const detailSubset = { where: eq(`category`, `detail`) } - const listSubset = { where: eq(`category`, `list`) } - - await collection._sync.loadSubset(detailSubset) - const detailHash = onlyOwner(maps, shared.id) - await collection._sync.loadSubset(listSubset) - const listHash = otherOwner(maps, shared.id, detailHash) - assertCheckpoint( - 0, - ownersOf(maps, shared.id), - sorted([detailHash, listHash]), - ) + const detail = { where: eq(`category`, `detail`) } + const list = { where: eq(`category`, `list`) } - collection._sync.unloadSubset(detailSubset) - assertCheckpoint( - 1, - { - rows: collectionRows(collection), - owners: ownersOf(maps, shared.id), - tracksDetail: maps.queryToRows.has(detailHash), - detailObservers: observerCount(queryClient, detailHash), - detailCached: queryClient - .getQueryCache() - .getAll() - .some((query) => query.queryHash === detailHash), - }, - { - rows: [listOnly.id, shared.id], - owners: [listHash], - tracksDetail: false, - detailObservers: 0, - detailCached: true, - }, - ) + await collection._sync.loadSubset(detail) + await collection._sync.loadSubset(list) + expect(rows(collection)).toEqual([detailOnly.id, listOnly.id, shared.id]) - // The ownerless existing-observer state reported by #1488 is not reachable - // here: observer and ownership retire together. Reacquisition creates a new - // observer over cached data, which must register ownership again. - await collection._sync.loadSubset(detailSubset) - assertCheckpoint( - 2, - { - fetches: queryFn.mock.calls.length, - owners: ownersOf(maps, shared.id), - tracksDetail: maps.queryToRows.has(detailHash), - detailObservers: observerCount(queryClient, detailHash), - }, - { - fetches: 2, - owners: sorted([detailHash, listHash]), - tracksDetail: true, - detailObservers: 1, - }, - ) + collection._sync.unloadSubset(detail) + expect(rows(collection)).toEqual([listOnly.id, shared.id]) + await collection._sync.loadSubset(detail) + expect(queryFn).toHaveBeenCalledTimes(2) + expect(rows(collection)).toEqual([detailOnly.id, listOnly.id, shared.id]) - collection._sync.unloadSubset(listSubset) - assertCheckpoint( - 3, - { - rows: collectionRows(collection), - owners: ownersOf(maps, shared.id), - }, - { rows: [detailOnly.id, shared.id], owners: [detailHash] }, - ) + collection._sync.unloadSubset(list) + expect(rows(collection)).toEqual([detailOnly.id, shared.id]) }) - it(`keeps overlapping row ownership while acquisition and owner counts differ`, async () => { - const { collection, maps, queryFn } = createOwnershipFixture({ - id: `ownership-count-boundaries`, - results: [ - [shared, detailOnly], - [shared, listOnly], - ], - }) - const detailSubset = { where: eq(`category`, `detail`) } - const listSubset = { where: eq(`category`, `list`) } - let activeAcquisitions = 0 - const acquire = async (subset: typeof detailSubset) => { - activeAcquisitions += 1 - await collection._sync.loadSubset(subset) - } - const release = (subset: typeof detailSubset) => { - activeAcquisitions -= 1 - collection._sync.unloadSubset(subset) - } + it.each([`remount`, `refetch`] as const)( + `keeps eager rows idle after cache removal and recovers on %s`, + async (action) => { + const id = `eager-lifetime-owner` + const { collection, queryClient, queryFn } = createOwnershipFixture({ + id, + syncMode: `eager`, + results: [[shared], [{ ...shared, name: `Refetched` }]], + }) + await collection.stateWhenReady() + const subscription = collection.subscribeChanges(() => {}) + subscription.unsubscribe() - await acquire(detailSubset) - const detailHash = onlyOwner(maps, shared.id) - await acquire(detailSubset) - await acquire(listSubset) - const listHash = otherOwner(maps, shared.id, detailHash) - assertCheckpoint( - 0, - { - acquisitions: activeAcquisitions, - queryOwners: ownersOf(maps, shared.id), - fetches: queryFn.mock.calls.length, - rows: collectionRows(collection), - }, - { - acquisitions: 3, - queryOwners: sorted([detailHash, listHash]), - fetches: 2, - rows: [detailOnly.id, listOnly.id, shared.id], - }, - ) + queryClient.removeQueries({ queryKey: [id], exact: true }) - release(detailSubset) - release(listSubset) - assertCheckpoint( - 1, - { - rows: collectionRows(collection), - owners: ownersOf(maps, shared.id), - }, - { rows: [detailOnly.id, shared.id], owners: [detailHash] }, - ) + expect(rows(collection)).toEqual([shared.id]) + await Promise.resolve() + expect(queryFn).toHaveBeenCalledOnce() + + const remounted = + action === `remount` ? collection.subscribeChanges(() => {}) : undefined + if (action === `refetch`) await collection.utils.refetch() + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(collection.get(shared.id)?.name).toBe(`Refetched`) + }) + remounted?.unsubscribe() + }, + ) + + it.each([false, true])( + `replaces an active eager cache entry with custom hash %s`, + async (customHash) => { + const id = `active-eager-custom-hash-${customHash}` + const { collection, queryClient, queryFn } = createOwnershipFixture({ + id, + customHash, + syncMode: `eager`, + results: [[shared], [{ ...shared, name: `Replaced` }]], + }) + await collection.stateWhenReady() + const subscription = collection.subscribeChanges(() => {}) + try { + queryClient.removeQueries({ queryKey: [id], exact: true }) + for (let turn = 0; turn < 20; turn++) await Promise.resolve() + expect(queryFn).toHaveBeenCalledTimes(2) + expect(collection.get(shared.id)?.name).toBe(`Replaced`) + } finally { + subscription.unsubscribe() + } + }, + ) - release(detailSubset) - assertCheckpoint( - 2, - { - rows: collectionRows(collection), - ownershipRows: maps.rowToQueries.size, - ownershipQueries: maps.queryToRows.size, + it.each( + [false, true].flatMap((mounted) => + [false, true].flatMap((customHash) => + [false, true].map((rejectOld) => ({ mounted, customHash, rejectOld })), + ), + ), + )( + `replaces a removed pending eager refetch without reviving idle demand: %j`, + async ({ mounted, customHash, rejectOld }) => { + const old = createDeferred>() + const next = createDeferred>() + const id = `pending-eager-removal` + const { collection, queryClient, queryFn } = createOwnershipFixture({ + id, + customHash, + syncMode: `eager`, + results: [[shared], old.promise, next.promise], + }) + await collection.stateWhenReady() + let subscription = collection.subscribeChanges(() => {}) + if (!mounted) subscription.unsubscribe() + let settled = false + const refetch = collection.utils.refetch({ throwOnError: true }).then( + () => { + settled = true + }, + (error: unknown) => { + settled = true + return error + }, + ) + try { + await vi.waitFor(() => expect(queryFn).toHaveBeenCalledTimes(2)) + queryClient.removeQueries({ queryKey: [id], exact: true }) + for (let turn = 0; turn < 30; turn++) await Promise.resolve() + expect(queryFn).toHaveBeenCalledTimes(mounted ? 3 : 2) + expect(collection.get(shared.id)?.name).toBe(`Shared`) + if (!mounted) subscription = collection.subscribeChanges(() => {}) + await vi.waitFor(() => expect(queryFn).toHaveBeenCalledTimes(3)) + next.resolve([{ ...shared, name: `Current` }]) + await vi.waitFor(() => + expect(collection.get(shared.id)?.name).toBe(`Current`), + ) + if (rejectOld) old.reject(new Error(`retired request failed`)) + else old.resolve([{ ...shared, name: `Obsolete` }]) + await vi.waitFor(() => expect(settled).toBe(true)) + expect(isCancelledError(await refetch)).toBe(true) + expect(collection.get(shared.id)?.name).toBe(`Current`) + expect(queryFn).toHaveBeenCalledTimes(3) + expect(collection.status).toBe(`ready`) + } finally { + old.resolve([shared]) + next.resolve([shared]) + subscription.unsubscribe() + } + }, + ) + + it.each( + [0, Number.POSITIVE_INFINITY].flatMap((staleTime) => + [false, true].map((customHash) => ({ staleTime, customHash })), + ), + )( + `starts only the requested fetch for an idle eager observer: %j`, + async ({ staleTime, customHash }) => { + const { collection, queryFn } = createOwnershipFixture({ + id: `idle-explicit-refetch`, + syncMode: `eager`, + staleTime, + customHash, + results: [[shared]], + }) + await collection.stateWhenReady() + queryFn.mockResolvedValue([{ ...shared, name: `Refetched` }]) + const subscription = collection.subscribeChanges(() => {}) + subscription.unsubscribe() + for (let turn = 0; turn < 30; turn++) await Promise.resolve() + const before = queryFn.mock.calls.length + await collection.utils.refetch({ throwOnError: true }) + expect(queryFn).toHaveBeenCalledTimes(before + 1) + expect(collection.subscriberCount).toBe(0) + }, + ) + + it.each([`release`, `cleanup`, `retain`] as const)( + `honors %s during startup retention maintenance`, + async (action) => { + const id = `released-startup-retention` + const subset = { where: eq(`category`, `shared`) } + const key = `queryCollection:gc:${hashKey([id, getLoadSubsetDemandKey(subset)])}` + const { collection, queryFn } = createOwnershipFixture({ + id, + results: [[shared]], + setupMetadata: (metadata) => + metadata.collection.set(key, { + queryHash: hashKey([id, getLoadSubsetDemandKey(subset)]), + mode: `until-revalidated`, + }), + }) + collection.startSyncImmediate() + const result = Promise.resolve(collection._sync.loadSubset(subset)).then( + () => `ready`, + (error: unknown) => error, + ) + if (action === `release`) collection._sync.unloadSubset(subset) + if (action === `cleanup`) await collection.cleanup() + for (let turn = 0; turn < 30; turn++) await Promise.resolve() + if (action === `retain`) { + expect(queryFn).toHaveBeenCalledTimes(1) + await expect(result).resolves.toBe(`ready`) + expect(collection.size).toBe(1) + } else { + expect(queryFn).not.toHaveBeenCalled() + await expect(result).resolves.toMatchObject({ name: `AbortError` }) + expect(collection.size).toBe(0) + } + }, + ) + + it.each([false, true])( + `removes owned cache entries on cleanup with custom hash %s`, + async (customHash) => { + const id = `cleanup-custom-${customHash}` + const { collection, queryClient } = createOwnershipFixture({ + id, + customHash, + syncMode: `eager`, + results: [[shared]], + }) + await collection.stateWhenReady() + expect(queryClient.getQueryCache().getAll()).toHaveLength(1) + await collection.cleanup() + expect(queryClient.getQueryCache().getAll()).toHaveLength(0) + }, + ) + + it(`settles an unfinished load when its final owner leaves`, async () => { + const pending = createDeferred>() + const { collection } = createOwnershipFixture({ + id: `release-before-result`, + results: [pending.promise], + }) + const subset = { where: eq(`category`, `shared`) } + let outcome: unknown = `pending` + const load = Promise.resolve(collection._sync.loadSubset(subset)).then( + () => { + outcome = `ready` + }, + (error: unknown) => { + outcome = error }, - { rows: [], ownershipRows: 0, ownershipQueries: 0 }, ) + try { + expect(collection.isLoadingSubset).toBe(true) + collection._sync.unloadSubset(subset) + for (let turn = 0; turn < 20; turn++) await Promise.resolve() + expect(outcome).toMatchObject({ name: `AbortError` }) + expect(collection.isLoadingSubset).toBe(false) + await load + } finally { + pending.resolve([shared]) + } }) - it(`#1631 keeps the eager owner when its last collection listener departs`, async () => { - const id = `ownership-eager-listener-1631` - const { collection, maps, queryClient } = createOwnershipFixture({ + it(`keeps active on-demand rows when the Query cache entry departs`, async () => { + const id = `active-cache-removal` + const { collection, queryClient } = createOwnershipFixture({ id, - syncMode: `eager`, results: [[shared]], }) + const subset = { where: eq(`category`, `detail`) } + await collection._sync.loadSubset(subset) - await collection.stateWhenReady() - const queryHash = onlyOwner(maps, shared.id) - const subscription = collection.subscribeChanges(() => {}) - assertCheckpoint( - 0, - { - status: collection.status, - listeners: collection.subscriberCount, - rows: collectionRows(collection), - owners: ownersOf(maps, shared.id).length, - }, - { status: `ready`, listeners: 1, rows: [shared.id], owners: 1 }, - ) - - subscription.unsubscribe() - assertCheckpoint(1, collection.subscriberCount, 0) - const warning = vi.spyOn(console, `warn`).mockImplementation(() => {}) - try { - // Removing the cache entry emits the same synchronous signal as gcTime, - // without making the defect boundary depend on a timer. - queryClient.removeQueries({ queryKey: [id], exact: true }) - - const assertOwnerSurvives = expectAssertionFailure( - () => - Promise.resolve().then(() => { - assertCheckpoint( - 2, - { - status: collection.status, - rows: collectionRows(collection), - owners: ownersOf(maps, shared.id).length, - }, - { status: `ready`, rows: [shared.id], owners: 1 }, - ) - }), - { - checkpoint: 2, - classify: classifyEagerOwnerLoss, - }, - ) + queryClient.removeQueries({ queryKey: [id] }) + expect(rows(collection)).toEqual([shared.id]) - await assertOwnerSurvives() - expect(warning).toHaveBeenCalledOnce() - expect(warning).toHaveBeenCalledWith( - expect.stringContaining(`[cleanupQueryIfIdle]`), - { hashedQueryKey: queryHash }, - ) - } finally { - warning.mockRestore() - } + collection._sync.unloadSubset(subset) + expect(rows(collection)).toEqual([]) }) - it(`#1656 keeps the first persisted owner when a second query inserts another row`, async () => { - const metadataRecorder: MetadataRecorder = { rowWrites: [] } - const { collection, maps } = createOwnershipFixture({ - id: `ownership-persisted-baseline-1656`, + it(`persists every owner of rows shared by overlapping queries`, async () => { + const metadata: MetadataRecorder = { rows: new Map(), writes: [] } + const { collection } = createOwnershipFixture({ + id: `persisted-overlap`, results: [[shared], [shared, listOnly]], - metadataRecorder, + metadataRecorder: metadata, }) - const detailSubset = { where: eq(`category`, `detail`) } - const listSubset = { where: eq(`category`, `list`) } - - await collection._sync.loadSubset(detailSubset) - const detailHash = onlyOwner(maps, shared.id) - // The production metadata API records the owner write, but the insert's - // commit currently loses it. Accept only that exact #1656 boundary. - const assertInsertedOwnerPersists = expectAssertionFailure( - () => - Promise.resolve().then(() => { - assertCheckpoint( - 0, - { - persistedOwners: persistedOwners( - collection._state.syncedMetadata, - shared.id, - ), - metadataSetKeys: setMetadataKeys(metadataRecorder), - }, - { persistedOwners: [detailHash], metadataSetKeys: [shared.id] }, - ) - }), - { checkpoint: 0, classify: classifyInsertedOwnerMetadataLoss }, - ) - await assertInsertedOwnerPersists() - - await collection._sync.loadSubset(listSubset) - const listHash = otherOwner(maps, shared.id, detailHash) - // A second insert loses its own owner and rebuilds the persisted baseline - // with only the later query, while the in-memory ownership remains sound. - const assertPersistedBaselineSurvives = expectAssertionFailure( - () => - Promise.resolve().then(() => { - assertCheckpoint( - 1, - { - liveOwners: ownersOf(maps, shared.id), - persistedOwners: persistedOwners( - collection._state.syncedMetadata, - shared.id, - ), - insertedOwners: persistedOwners( - collection._state.syncedMetadata, - listOnly.id, - ), - metadataSetKeys: setMetadataKeys(metadataRecorder), - }, - { - liveOwners: sorted([detailHash, listHash]), - persistedOwners: sorted([detailHash, listHash]), - insertedOwners: [listHash], - metadataSetKeys: [listOnly.id, shared.id], - }, - ) - }), - { checkpoint: 1, classify: classifyPersistedBaselineLoss }, - ) - await assertPersistedBaselineSurvives() - - collection._sync.unloadSubset(listSubset) - assertCheckpoint( - 2, - { - rows: collectionRows(collection), - liveOwners: ownersOf(maps, shared.id), - persistedOwners: persistedOwners( - collection._state.syncedMetadata, - shared.id, - ), - }, - { - rows: [shared.id], - liveOwners: [detailHash], - persistedOwners: [detailHash], + const detail = { where: eq(`category`, `detail`) } + const list = { where: eq(`category`, `list`) } + + await collection._sync.loadSubset(detail) + expect(persistedOwners(metadata.rows, shared.id)).toHaveLength(1) + + await collection._sync.loadSubset(list) + expect(persistedOwners(metadata.rows, shared.id)).toHaveLength(2) + expect(persistedOwners(metadata.rows, listOnly.id)).toHaveLength(1) + + collection._sync.unloadSubset(list) + expect(rows(collection)).toEqual([shared.id]) + expect(persistedOwners(metadata.rows, shared.id)).toHaveLength(1) + }) + + it(`restages a persisted owner when its absent row arrives`, async () => { + const id = `persisted-owner-before-row` + const queryHash = hashKey([id]) + const result = createDeferred>() + const metadata: MetadataRecorder = { rows: new Map(), writes: [] } + let setupCalls = 0 + const { collection, queryFn } = createOwnershipFixture({ + id, + syncMode: `eager`, + results: [result.promise, [{ ...shared, name: `Restarted` }]], + metadataRecorder: metadata, + setupMetadata: (api) => { + setupCalls++ + api.row.set(shared.id, { + queryCollection: { owners: { [queryHash]: true } }, + }) }, - ) + }) + + expect(rows(collection)).toEqual([]) + expect(persistedOwners(metadata.rows, shared.id)).toEqual([queryHash]) + result.resolve([shared]) + await collection.stateWhenReady() + expect(rows(collection)).toEqual([shared.id]) + expect(persistedOwners(metadata.rows, shared.id)).toEqual([queryHash]) + + await collection.cleanup() + await collection.preload() + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(collection.get(shared.id)?.name).toBe(`Restarted`) + }) + expect(setupCalls).toBe(1) + expect(persistedOwners(metadata.rows, shared.id)).toEqual([queryHash]) }) }) diff --git a/packages/query-db-collection/tests/query.test.ts b/packages/query-db-collection/tests/query.test.ts index b6f8266813..51de217b1b 100644 --- a/packages/query-db-collection/tests/query.test.ts +++ b/packages/query-db-collection/tests/query.test.ts @@ -22,6 +22,7 @@ import { mockSyncCollectionOptions, stripVirtualProps, } from '../../db/tests/utils' +import { evaluateReferenceExpression } from '../../db/tests/reference-expression' import { persistedCollectionOptions } from '../../db-sqlite-persistence-core/src' import { queryCollectionOptions } from '../src/query' import type { QueryFunctionContext } from '@tanstack/query-core' @@ -50,28 +51,6 @@ interface CategorisedItem { const getKey = (item: TestItem) => item.id -type OwnershipMaps = { - rowToQueries: Map> - queryToRows: Map> -} - -function inspectOwnershipMaps(options: { - sync: { sync: unknown } -}): OwnershipMaps { - const sync = options.sync.sync as { - __getOwnershipMapsForTests?: () => OwnershipMaps - } - const maps = sync.__getOwnershipMapsForTests?.() - if (!maps) { - throw new Error(`Ownership-map test inspection is unavailable`) - } - return maps -} - -function expectNoEmptyRowOwnershipSets(maps: OwnershipMaps): void { - maps.rowToQueries.forEach((owners) => expect(owners.size).toBeGreaterThan(0)) -} - // Helper to advance timers and allow microtasks to flush const flushPromises = () => new Promise((resolve) => setTimeout(resolve, 0)) @@ -2247,7 +2226,7 @@ describe(`QueryCollection`, () => { // We're mainly verifying the collection cleanup works without errors }) - it(`should call cancelQueries and removeQueries on sync cleanup`, async () => { + it(`should remove its Query cache entry on sync cleanup`, async () => { const queryKey = [`sync-cleanup-test`] const items = [{ id: `1`, name: `Item 1` }] const queryFn = vi.fn().mockResolvedValue(items) @@ -2261,12 +2240,6 @@ describe(`QueryCollection`, () => { startSync: true, } - // Spy on the queryClient methods that should be called during sync cleanup - const cancelQueriesSpy = vi - .spyOn(queryClient, `cancelQueries`) - .mockResolvedValue() - const removeQueriesSpy = vi.spyOn(queryClient, `removeQueries`) - const options = queryCollectionOptions(config) const collection = createCollection(options) @@ -2280,6 +2253,7 @@ describe(`QueryCollection`, () => { // be an active subscription to the query expect(collection.subscriberCount).toBe(0) expect(collection.status).toBe(`ready`) + expect(queryClient.getQueryCache().find({ queryKey })).toBeDefined() // Add explicit subscribers to test cleanup with active subscribers const subscription1 = collection.subscribeChanges(() => {}) @@ -2289,27 +2263,13 @@ describe(`QueryCollection`, () => { // Cleanup the collection which should trigger sync cleanup await collection.cleanup() - // Wait a bit to ensure all async operations complete - await flushPromises() - - // Verify collection status expect(collection.status).toBe(`cleaned-up`) - - // Verify that cleanup methods are called regardless of subscriber state - expect(cancelQueriesSpy).toHaveBeenCalledWith({ - queryKey, - exact: true, - }) - expect(removeQueriesSpy).toHaveBeenCalledWith({ queryKey, exact: true }) + expect(queryClient.getQueryCache().find({ queryKey })).toBeUndefined() // Verify subscribers can be safely cleaned up after collection cleanup subscription1.unsubscribe() subscription2.unsubscribe() expect(collection.subscriberCount).toBe(0) - - // Restore spies - cancelQueriesSpy.mockRestore() - removeQueriesSpy.mockRestore() }) it(`should handle multiple cleanup calls gracefully`, async () => { @@ -2422,12 +2382,6 @@ describe(`QueryCollection`, () => { startSync: true, } - // Spy on queryClient methods - const cancelQueriesSpy = vi - .spyOn(queryClient, `cancelQueries`) - .mockResolvedValue() - const removeQueriesSpy = vi.spyOn(queryClient, `removeQueries`) - const options = queryCollectionOptions(config) const collection = createCollection(options) @@ -2436,43 +2390,24 @@ describe(`QueryCollection`, () => { expect(collection.size).toBe(1) }) - // Cleanup which should call query cleanup methods await collection.cleanup() - await flushPromises() expect(collection.status).toBe(`cleaned-up`) - - // Verify cleanup methods were called - expect(cancelQueriesSpy).toHaveBeenCalledWith({ - queryKey, - exact: true, - }) - expect(removeQueriesSpy).toHaveBeenCalledWith({ queryKey, exact: true }) - - // Clear the spies to track new calls - cancelQueriesSpy.mockClear() - removeQueriesSpy.mockClear() + expect(queryClient.getQueryCache().find({ queryKey })).toBeUndefined() // Restart by accessing collection const subscription = collection.subscribeChanges(() => {}) // Should restart sync expect([`loading`, `ready`]).toContain(collection.status) + await vi.waitFor(() => { + expect(queryFn).toHaveBeenCalledTimes(2) + expect(queryClient.getQueryCache().find({ queryKey })).toBeDefined() + }) // Cleanup again to verify the new sync cleanup works subscription.unsubscribe() await collection.cleanup() - await flushPromises() - - // Verify cleanup methods were called again for the restarted sync - expect(cancelQueriesSpy).toHaveBeenCalledWith({ - queryKey, - exact: true, - }) - expect(removeQueriesSpy).toHaveBeenCalledWith({ queryKey, exact: true }) - - // Restore spies - cancelQueriesSpy.mockRestore() - removeQueriesSpy.mockRestore() + expect(queryClient.getQueryCache().find({ queryKey })).toBeUndefined() }) it(`should handle query invalidation and refetch properly`, async () => { @@ -2563,9 +2498,7 @@ describe(`QueryCollection`, () => { await collection.cleanup() } - expect( - queryClient.getQueryCache().find({ queryKey })?.getObserversCount(), - ).toBe(0) + expect(queryClient.getQueryCache().find({ queryKey })).toBeUndefined() }) it(`rematerializes an active eager query after prefix invalidation`, async () => { @@ -2857,7 +2790,7 @@ describe(`QueryCollection`, () => { // existing unit fixtures do not exercise the full persisted retention path // without introducing broader persistence setup. This PR characterizes active, // inactive, removed, overlapping, and failed-refetch behavior first. - it(`does not rematerialize inactive cached query rows after invalidation`, async () => { + it(`does not refetch a cleaned-up query after invalidation`, async () => { const retainedQueryClient = new QueryClient({ defaultOptions: { queries: { @@ -2891,7 +2824,7 @@ describe(`QueryCollection`, () => { expect(collection.status).toBe(`cleaned-up`) expect( retainedQueryClient.getQueryCache().find({ queryKey }), - ).toBeDefined() + ).toBeUndefined() items = [{ id: `1`, name: `Updated Item 1` }] await retainedQueryClient.invalidateQueries({ queryKey, exact: true }) @@ -3165,7 +3098,7 @@ describe(`QueryCollection`, () => { ).toBe(0) }) - it(`cleans listeners immediately and resolves the unloaded preload before its request settles`, async () => { + it(`cleans listeners immediately and rejects the abandoned preload before its request settles`, async () => { const deferred = createDeferred>() const collection = createCollection( queryCollectionOptions({ @@ -3178,10 +3111,14 @@ describe(`QueryCollection`, () => { }), ) const liveQuery = createSubset(collection) - let preloadResolved = false - void liveQuery.preload().then(() => { - preloadResolved = true - }) + let preloadError: unknown + const preloadOutcome = liveQuery.preload().then( + () => undefined, + (error: unknown) => { + preloadError = error + return error + }, + ) await vi.waitFor(() => expect(queryClient.isFetching()).toBe(1)) await liveQuery.cleanup() @@ -3192,19 +3129,19 @@ describe(`QueryCollection`, () => { // This assertion runs while the request is unresolved and directly guards the // ready-listener bookkeeping bug: unload must synchronously detach its observer. expect(subsetQuery?.getObserversCount() ?? 0).toBe(0) - // Live-query cleanup resolves its preload even though Query Core is still fetching. - expect(preloadResolved).toBe(true) + // Cleanup cancels the caller's wait even while Query Core keeps fetching. + expect(preloadError).toMatchObject({ name: `AbortError` }) deferred.resolve([{ id: `1`, name: `Late item` }]) await vi.waitFor(() => expect(queryClient.isFetching()).toBe(0)) expect(collection.size).toBe(0) - expect(preloadResolved).toBe(true) + expect(await preloadOutcome).toBe(preloadError) expect(subsetQuery?.getObserversCount() ?? 0).toBe(0) await collection.cleanup() }) - it(`keeps an unloaded preload resolved when its pending request later rejects`, async () => { + it(`keeps the cleanup error when an abandoned preload's request later rejects`, async () => { const deferred = createDeferred>() const collection = createCollection( queryCollectionOptions({ @@ -3217,7 +3154,10 @@ describe(`QueryCollection`, () => { }), ) const liveQuery = createSubset(collection) - const preloadPromise = liveQuery.preload() + const preloadOutcome = liveQuery.preload().then( + () => undefined, + (error: unknown) => error, + ) await vi.waitFor(() => expect(queryClient.isFetching()).toBe(1)) await liveQuery.cleanup() @@ -3226,12 +3166,14 @@ describe(`QueryCollection`, () => { queryKey: [`late-subset-rejection-test`], })[0] expect(subsetQuery?.getObserversCount() ?? 0).toBe(0) - await expect(preloadPromise).resolves.toBeUndefined() + const preloadError = await preloadOutcome + expect(preloadError).toMatchObject({ name: `AbortError` }) deferred.reject(new Error(`Late query failure`)) await vi.waitFor(() => expect(queryClient.isFetching()).toBe(0)) expect(collection.size).toBe(0) + expect(await preloadOutcome).toBe(preloadError) expect(subsetQuery?.getObserversCount() ?? 0).toBe(0) await collection.cleanup() }) @@ -4740,8 +4682,6 @@ describe(`QueryCollection`, () => { expect(collection.utils.lastError).toBe(applicationError) expect(collection.utils.errorCount).toBe(1) expect(collection.size).toBe(0) - expect(inspectOwnershipMaps(options).rowToQueries.size).toBe(0) - expect(inspectOwnershipMaps(options).queryToRows.size).toBe(0) await collection.cleanup() consoleErrorSpy.mockRestore() @@ -6032,24 +5972,27 @@ describe(`QueryCollection`, () => { it(`should handle GC correctly when queries are ordered and have a LIMIT`, async () => { const baseQueryKey = [`deduplication-gc-test`] - // Mock queryFn to return different data based on predicates + const items = [ + { id: `1`, name: `Item 1`, category: `A` }, + { id: `2`, name: `Item 2`, category: `A` }, + { id: `3`, name: `Item 3`, category: `A` }, + ] + // Honor the complete pushed predicate so an exact tie request does not + // masquerade as another full category load. const queryFn = vi.fn().mockImplementation((context) => { const { meta } = context const loadSubsetOptions = meta?.loadSubsetOptions ?? {} - const { where, limit } = loadSubsetOptions - - // Query 1: all items with category A (no limit) - if (isCategory(`A`, where)) { - const items = [ - { id: `1`, name: `Item 1`, category: `A` }, - { id: `2`, name: `Item 2`, category: `A` }, - { id: `3`, name: `Item 3`, category: `A` }, - ] - // Slice to limit if provided - return Promise.resolve(limit ? items.slice(0, limit) : items) - } - - return Promise.resolve([]) + const { where, offset = 0, limit } = loadSubsetOptions + + const matching = where + ? items.filter((item) => evaluateReferenceExpression(where, item)) + : items + return Promise.resolve( + matching.slice( + offset, + limit === undefined ? undefined : offset + limit, + ), + ) }) const config: QueryCollectionConfig = { @@ -6119,8 +6062,8 @@ describe(`QueryCollection`, () => { await flushPromises() - // queryFn should have been called twice - // because we do not dedupe the 2nd query + // The initial complete category load already proves that no unseen row + // ties the ordered boundary, so the second demand needs only its prefix. expect(queryFn).toHaveBeenCalledTimes(2) // Collection should still have all 3 items (deduplication doesn't remove data) @@ -6134,13 +6077,15 @@ describe(`QueryCollection`, () => { // Wait for async GC to complete await vi.waitFor(() => { - expect(collection.size).toBe(2) // Should only have items 1 and 2 because they are still referenced by query 2 + // Query 2 shares the already-complete category acquisition so it can + // refill locally. It may retain row 3 even though its visible window + // contains only rows 1 and 2. + expect(collection.size).toBe(3) }) - // Verify that only row 3 is removed (it was only referenced by query 1) - expect(collection.has(`1`)).toBe(true) // Still present (referenced by query 2) - expect(collection.has(`2`)).toBe(true) // Still present (referenced by query 2) - expect(collection.has(`3`)).toBe(false) // Removed (only referenced by query 1) + expect(collection.has(`1`)).toBe(true) + expect(collection.has(`2`)).toBe(true) + expect(collection.has(`3`)).toBe(true) // GC the second query (category A with limit 2) await query2.cleanup() @@ -6171,7 +6116,6 @@ describe(`QueryCollection`, () => { getKey, syncMode: `on-demand`, }) - const ownershipMaps = inspectOwnershipMaps(options) const collection = createCollection(options) const firstSubset = createLiveQueryCollection({ query: (q) => @@ -6196,15 +6140,10 @@ describe(`QueryCollection`, () => { expect(collection.has(`2`)).toBe(true) expect(collection.has(`3`)).toBe(true) }) - expectNoEmptyRowOwnershipSets(ownershipMaps) - await secondSubset.cleanup() await vi.waitFor(() => { expect(collection.size).toBe(0) }) - expectNoEmptyRowOwnershipSets(ownershipMaps) - expect(ownershipMaps.rowToQueries.size).toBe(0) - expect(ownershipMaps.queryToRows.size).toBe(0) }) it(`expires the Query cache entry after unload without restoring deleted rows`, async () => { @@ -6276,7 +6215,6 @@ describe(`QueryCollection`, () => { startSync: true, }) const originalSync = baseOptions.sync - const ownershipMaps = inspectOwnershipMaps(baseOptions) const metadataHarness = createInMemorySyncMetadataApi< string | number, CategorisedItem @@ -6325,9 +6263,6 @@ describe(`QueryCollection`, () => { expect(collection.has(retainedRow.id)).toBe(false) }) expect(metadataHarness.rowMetadata.get(retainedRow.id)).toBeUndefined() - expectNoEmptyRowOwnershipSets(ownershipMaps) - expect(ownershipMaps.rowToQueries.size).toBe(0) - expect(ownershipMaps.queryToRows.get(queryHash)).toEqual(new Set()) expect( metadataHarness.collectionMetadata.has( `queryCollection:gc:${queryHash}`, @@ -7077,7 +7012,6 @@ describe(`QueryCollection`, () => { const baseOptions = queryCollectionOptions(config) const originalSync = baseOptions.sync - const ownershipMaps = inspectOwnershipMaps(baseOptions) const metadataHarness = createInMemorySyncMetadataApi< string | number, CategorisedItem @@ -7110,13 +7044,6 @@ describe(`QueryCollection`, () => { await liveQuery.cleanup() - expect(ownershipMaps.queryToRows.has(retainedQueryHash)).toBe(true) - expect(ownershipMaps.queryToRows.get(retainedQueryHash)).toEqual( - new Set([`1`]), - ) - expect(ownershipMaps.rowToQueries.get(`1`)).toEqual( - new Set([retainedQueryHash]), - ) expect( metadataHarness.collectionMetadata.get( `queryCollection:gc:${retainedQueryHash}`, @@ -7136,8 +7063,6 @@ describe(`QueryCollection`, () => { ), ).toBeUndefined() expect(collection.has(`1`)).toBe(false) - expect(ownershipMaps.queryToRows.has(retainedQueryHash)).toBe(false) - expect(ownershipMaps.rowToQueries.has(`1`)).toBe(false) } finally { vi.useRealTimers() } @@ -7161,7 +7086,6 @@ describe(`QueryCollection`, () => { } const baseOptions = queryCollectionOptions(config) const originalSync = baseOptions.sync - const ownershipMaps = inspectOwnershipMaps(baseOptions) const metadataHarness = createInMemorySyncMetadataApi< string | number, CategorisedItem @@ -7182,12 +7106,20 @@ describe(`QueryCollection`, () => { await liveQuery.preload() await liveQuery.cleanup() - expect(ownershipMaps.queryToRows.has(retainedQueryHash)).toBe(true) + expect( + metadataHarness.collectionMetadata.has( + `queryCollection:gc:${retainedQueryHash}`, + ), + ).toBe(true) await collection.cleanup() - expect(ownershipMaps.queryToRows.size).toBe(0) - expect(ownershipMaps.rowToQueries.size).toBe(0) + expect(collection.size).toBe(0) + expect( + metadataHarness.collectionMetadata.has( + `queryCollection:gc:${retainedQueryHash}`, + ), + ).toBe(false) }) it(`should default persisted retention ttl to query gcTime when persistedGcTime is undefined`, async () => { @@ -7443,13 +7375,17 @@ describe(`QueryCollection`, () => { expect(collection.size).toBe(2) }) - // Force GC by calling removeQueries (simulates gcTime expiry) + // Release the first acquisition before its cache entry is removed. + // Cache events do not revoke active collection ownership. + await query1.cleanup() + await vi.waitFor(() => { + expect(collection.size).toBe(0) + }) + + // Force GC by calling removeQueries (simulates gcTime expiry). queryClient.removeQueries({ queryKey: baseQueryKey }) await flushPromises() - // BUG: queryRefCounts still has stale count, wasn't cleaned up by cleanupQuery - // When we load again, the refcount will be wrong (starts at 1 instead of 0, or accumulates) - // Reload the same query const query2 = createLiveQueryCollection({ query: (q) => @@ -7466,14 +7402,11 @@ describe(`QueryCollection`, () => { expect(collection.size).toBe(2) }) - // Cleanup - this should properly decrement from 1 to 0 and clean up + // Cleanup should decrement the new acquisition from one to zero. await query2.cleanup() await vi.waitFor(() => { expect(collection.size).toBe(0) // Should be cleaned up }) - - // BUG SYMPTOM: If refcount was stale (e.g. was 2, decremented to 1), - // the observer won't be destroyed and data won't be cleaned up }) it(`should handle mount/unmount/remount without breaking cache (destroyed observer bug)`, async () => { diff --git a/packages/react-db/tests/useLiveQuery.test.tsx b/packages/react-db/tests/useLiveQuery.test.tsx index 39396bdbfa..7992efad1c 100644 --- a/packages/react-db/tests/useLiveQuery.test.tsx +++ b/packages/react-db/tests/useLiveQuery.test.tsx @@ -3155,7 +3155,7 @@ describe(`Query Collections`, () => { warnSpy.mockRestore() }) - it(`warns when a structured query captures an opaque runtime value without queryKey`, () => { + it(`uses runtime identity for opaque values in a structured query without queryKey`, () => { const warnSpy = vi.spyOn(console, `warn`).mockImplementation(() => {}) const collection = createCollection( mockSyncCollectionOptions({ @@ -3165,24 +3165,27 @@ describe(`Query Collections`, () => { }), ) - expect(() => - renderHook(() => + const runtimeValue = () => `John Doe` + const { result, rerender } = renderHook( + ({ value }) => useLiveQuery({ query: (q) => q .from({ people: collection }) - .where(({ people }) => - eq(people.name, (() => `John Doe`) as never), - ), + .where(({ people }) => eq(people.name, value as never)), }), - ), - ).not.toThrow() + { initialProps: { value: runtimeValue } }, + ) + const firstCollection = result.current.collection + rerender({ value: runtimeValue }) + expect(result.current.collection).toBe(firstCollection) + rerender({ value: () => `John Doe` }) + expect(result.current.collection).not.toBe(firstCollection) const warnings = warnSpy.mock.calls.filter(([message]) => String(message).includes(`function value`), ) - expect(warnings).toHaveLength(1) - expect(warnings[0]![0]).toContain(`queryKey`) + expect(warnings).toHaveLength(0) warnSpy.mockRestore() }) diff --git a/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts b/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts index c7af26871d..274cc21b6d 100644 --- a/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts +++ b/packages/svelte-db/tests/useLiveInfiniteQuery.svelte.test.ts @@ -67,20 +67,26 @@ describe(`useLiveInfiniteQuery`, () => { await livePosts.preload() const warning = vi.spyOn(console, `warn`).mockImplementation(() => {}) + let query!: ReturnType cleanup = $effect.root(() => { - const query = useLiveInfiniteQuery(() => livePosts, { + query = useLiveInfiniteQuery(() => livePosts, { pageSize: 3, getNextPageParam: (lastPage) => lastPage[0]?.createdAt, }) - flushSync() - - expect(query.collection).toBe(livePosts) - expect(query.data.map((post) => post.id)).toEqual([`1`, `2`, `3`]) - expect(query.state.get(`1`)?.title).toBe(`Post 1`) - expect(query.hasNextPage).toBe(true) - expect(warning).toHaveBeenCalledOnce() - expect(livePosts.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) }) + flushSync() + // flushSync starts the subscription but does not settle its window load. + await vi.waitFor(() => + expect(livePosts.utils.getWindow()).toEqual({ offset: 0, limit: 4 }), + ) + flushSync() + + expect(query.collection).toBe(livePosts) + expect(query.data.map((post) => post.id)).toEqual([`1`, `2`, `3`]) + expect(query.state.get(`1`)?.title).toBe(`Post 1`) + expect(query.hasNextPage).toBe(true) + expect(warning).toHaveBeenCalledOnce() + expect(livePosts.utils.getWindow()).toEqual({ offset: 0, limit: 4 }) }) it(`resets to the first page when a collection getter changes`, async () => { diff --git a/packages/vue-db/tests/useLiveInfiniteQuery.test.ts b/packages/vue-db/tests/useLiveInfiniteQuery.test.ts index 9657d1bccf..f8ef809b4f 100644 --- a/packages/vue-db/tests/useLiveInfiniteQuery.test.ts +++ b/packages/vue-db/tests/useLiveInfiniteQuery.test.ts @@ -66,6 +66,10 @@ describe(`useLiveInfiniteQuery`, () => { ) cleanup = () => scope.stop() if (!query) throw new Error(`Failed to mount infinite query`) + // A framework tick does not settle asynchronous window normalization. + await vi.waitFor(() => + expect(livePosts.utils.getWindow()).toEqual({ offset: 0, limit: 4 }), + ) await flushVue() expect(query.collection.value).toBe(livePosts)