TrueSelph · eldonm · Jul 31, 2026 · Jul 31, 2026 · Jul 31, 2026 · Jul 31, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,6 +9,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **`resolve_sort_value(record, field)`** (`jvspatial/db/database.py`) — the
+  dotted-path resolution `finalize_find_results` uses, exported so adapters and
+  cursor logic resolve a sort field the same way. Added to the module's
+  `__all__`.
+
 - **Deferred-task exception hierarchy** (`jvspatial/exceptions.py`) —
   `DeferredTaskError` → `TaskDispatchError` → `TaskSchedulerNotConfiguredError`,
   replacing the bare `RuntimeError`s raised by strict dispatch. A strict caller
@@ -19,6 +24,66 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+
+- **`SQLiteDB.find` treated `sort=[]` as an untranslatable sort**
+  (`jvspatial/db/sqlite.py`). An empty list failed the `sort is None` guard, so
+  it took the fallback branch: the whole collection was loaded and `limit`
+  applied in memory instead of being pushed into SQL. Results were correct, the
+  work was not. A falsy `sort` is now normalized to `None`.
+- **`ObjectPager` re-sorted each page with a key that disagreed with the
+  database slice** (`jvspatial/core/pager.py`; `paginate_by_field` inherited
+  it). The in-Python safety-net sort used
+  `item.get("context", {}).get(order_by, 0)`, so a record missing `order_by`
+  was ordered as `0` — among the real values — while the DB-side
+  `sort` + `limit` that produced the slice had placed it in the trailing
+  missing-value run. Records could therefore appear on two pages or on none.
+  A blanket `contextlib.suppress(KeyError, TypeError)` also left a page
+  silently unsorted on mixed-type keys. The re-sort now routes through
+  `finalize_find_results`, making it a genuine no-op whenever the backend
+  honored the sort.
+- **`GraphContext.find_page` broke on dotted sort fields and could not reach
+  records missing the sort value** (`jvspatial/core/context.py`). Two defects:
+  the cursor payload was minted with a flat `last.get(primary_field)`, so a
+  `sort=[("context.started_at", -1)]` page always encoded `sort: None` and the
+  next page's keyset filter compared against `None` — raising
+  `TypeError: '>' not supported between instances of 'int' and 'NoneType'` from
+  `QueryEngine` on JsonDB. And with records missing the sort field now sorting
+  last, the keyset filter `{field: {"$lt": value}}` could never match them, so
+  iteration silently stopped at the last record that had a value. The cursor now
+  uses `resolve_sort_value`, the filter carries a `{field: None}` branch to reach
+  the trailing run, and a cursor minted inside that run walks it by `id`.
+- **Postgres applied `LIMIT` in SQL even when the sort could not be pushed
+  down** (`jvspatial/db/postgres.py`, both `PostgresDB.find` and
+  `PostgresTransaction.find`). `translate_sort` returns `None` for a field path
+  it cannot safely interpolate (e.g. `context.my-field`), leaving the ordering
+  to `finalize_find_results` — but the `LIMIT` was still pushed, so the database
+  returned an arbitrary N rows and the in-memory sort ordered that arbitrary
+  subset. `find(sort=..., limit=10)` returned "the top 10 of an arbitrary 10"
+  instead of the true top 10. The `LIMIT` is now withheld whenever the sort
+  falls back to memory, matching `SQLiteDB.find` and `DynamoDB.find`. Vector
+  (`$near`) queries additionally no longer have their distance ordering
+  overwritten by an in-memory re-sort on the user's `sort`.
+
+- **In-memory `find` sort ignored dotted field paths** (`jvspatial/db/database.py`).
+  `_find_sort_key` resolved `sort` fields with a flat `record.get(field)`, so a
+  spec like `sort=[("context.started_at", -1)]` produced `None` for every row and
+  left the result in arbitrary order. The SQLite and Postgres pushdowns
+  (`translate_sort`) and Mongo's native sort already resolved dotted paths, so
+  the same query ordered correctly on those backends and silently did not on
+  JsonDB/DynamoDB — and on SQLite/Postgres whenever the query fell back to the
+  in-memory path. Dotted paths now resolve in memory too; a non-dict segment
+  along the path yields `None` rather than raising.
+- **Descending in-memory sorts placed records missing the sort field first**
+  (`jvspatial/db/database.py`). `finalize_find_results` sorts with
+  `reverse=True`, which flipped `_find_sort_key`'s `None` flag along with the
+  values. Both SQL translators emit `NULLS LAST` for descending and Mongo sorts
+  missing values last, so a "newest N" `sort` + `limit` fetch returned real rows
+  on SQLite/Postgres/Mongo and a window of records missing the field on the
+  in-memory path. Missing values now sort last in both directions everywhere.
+  The comment in `_sqlite_translate.translate_sort` asserting the in-memory path
+  already matched has been corrected.
+
+
 - **Strict deferred scheduling only caught the no-op-scheduler case**
   (`jvspatial/serverless/`). `dispatch_deferred_task(..., strict=True)` raised
   when serverless mode resolved a logging no-op, but every *provider* failure
@@ -47,6 +112,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+
 - **`TaskScheduler.schedule` takes a `strict` argument**
   (`jvspatial/serverless/tasks/base.py`). `TaskScheduler` is a public/stable
   extension point and `config.task_scheduler` is duck-typed, so
@@ -56,6 +122,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   raises `TaskSchedulerNotConfiguredError` (it cannot honor the guarantee)
   rather than `TypeError`.
 
+### Documentation
+
+
+- **`find` sort contract moved to SPEC §4.1** (beside the `Database` method
+  table it governs, rather than under §4.2 capability flags) and extended: the
+  `limit`-must-not-outlive-the-sort-pushdown rule, plus a **Known divergences**
+  table covering MongoDB's ascending sorts (native `cursor.sort()` places
+  missing values first — documented, not normalized), array-index path segments,
+  and heterogeneous value types.
+- **Corrected stale NULL-ordering docstrings** in
+  `jvspatial/db/_sqlite_translate.py` (module docstring and `translate_sort`)
+  and `jvspatial/db/_postgres_translate.py` (`translate_sort`). All three still
+  claimed "NULLs sort last for ascending, first for descending, mirroring
+  `finalize_find_results`" — the opposite of what the code emits and of the
+  contract.
+- **`Database.find`** now documents the ordering contract adapter authors must
+  satisfy; **`Database.find_iter`** no longer claims a composite
+  `(sort_value, id)` cursor — the default implementation tracks `id` only, so a
+  non-`id` sort drops records that sort late but carry a lower `id`.
+
+
 ## [0.0.11] - 2026-07-02
 
 ### Fixed

diff --git a/SPEC.md b/SPEC.md
@@ -207,7 +207,7 @@ When `DeferredSaveMixin` is mixed into an entity *and* `deferred_saves_globally_
 | `save(collection, data)` | Yes | Insert-or-replace by ID; returns saved record |
 | `get(collection, id)` | Yes | Fetch by ID or `None` |
 | `delete(collection, id)` | Yes | Idempotent delete by ID |
-| `find(collection, query, *, limit, sort)` | Yes | Mongo-style query; returns list |
+| `find(collection, query, *, limit, sort)` | Yes | Mongo-style query; returns list. Ordering contract below |
 | `count(collection, query=None)` | Default impl | Default counts the result of `find`; adapters should override for efficiency |
 | `find_one(collection, query)` | Default impl | First match or `None` |
 | `find_many(collection, ids)` | Default impl | Bulk-fetch by ID; default is N sequential `get`s — adapters override for round-trip efficiency |
@@ -216,6 +216,36 @@ When `DeferredSaveMixin` is mixed into an entity *and* `deferred_saves_globally_
 | `bulk_save` | Default impl | Multi-record save; partial-success semantics vary by adapter |
 | `begin_transaction` | Optional | Returns a transaction context manager if `supports_transactions=True` |
 
+#### `find` sort contract
+
+`sort` is a list of `(field, direction)` with `1` ascending and `-1` descending.
+An adapter may push the sort into the backend or fall back to
+`finalize_find_results` (`jvspatial/db/database.py`); either way it must produce
+the same ordering:
+
+- **Dotted paths** (`context.started_at`) resolve into nested documents. A
+  non-dict segment along the path resolves to "missing", not an error. Use
+  `resolve_sort_value` rather than a flat `record.get(field)` anywhere a sort
+  field is read — including cursor payloads.
+- **Missing values sort last in both directions** — matching the `NULLS LAST`
+  emitted by `_sqlite_translate.translate_sort` and
+  `_postgres_translate.translate_sort`. A `sort` + `limit` "newest N" fetch
+  therefore never fills its window with records lacking the sort field.
+- Sorting is **stable**; compound sorts apply from the last key to the first.
+- **`limit` must not be pushed down when the sort is not.** An adapter that
+  cannot express the ordering in the backend has to fetch the full match set
+  and apply `sort` and `limit` together in memory — otherwise it orders an
+  arbitrary N rows instead of returning the true top N.
+
+**Known divergences** — the contract holds for homogeneous scalar leaves; these
+cases are documented rather than normalized:
+
+| Case | Behavior |
+|---|---|
+| MongoDB, ascending | `MongoDB.find` uses native `cursor.sort()`. BSON orders null/missing lowest, so **missing values come first** on ascending sorts. Descending matches the contract. Normalizing would require an aggregation pipeline on every `find`. |
+| Array-index segments (`items.0`) | Resolve to "missing" in memory and on SQLite; rejected by the Postgres pushdown (leading digit fails `_safe_field_path`, so it falls back and agrees); resolved natively by MongoDB. |
+| Heterogeneous values on one key | The in-memory path raises `TypeError` (Python cannot order `str` against `int`); SQL pushdowns order by storage class instead. Object/array leaves likewise raise in memory and sort as JSON text on SQLite. |
+
 ### 4.2 Capability flags
 
 Adapters declare capabilities as class attributes:

diff --git a/jvspatial/core/context.py b/jvspatial/core/context.py
@@ -23,7 +23,7 @@
     cast,
 )
 
-from jvspatial.db.database import Database
+from jvspatial.db.database import Database, resolve_sort_value
 from jvspatial.db.factory import create_database, get_current_database
 from jvspatial.db.manager import get_database_manager
 
@@ -1435,17 +1435,43 @@ async def find_page(
         if cursor_payload and "id" in cursor_payload and "sort" in cursor_payload:
             sort_op = "$lt" if primary_dir < 0 else "$gt"
             id_op = "$lt" if id_dir < 0 else "$gt"
-            keyset_filter = {
-                "$or": [
-                    {primary_field: {sort_op: cursor_payload["sort"]}},
+            cursor_sort = cursor_payload["sort"]
+            keyset_branches: List[Dict[str, Any]]
+            if cursor_sort is None:
+                # The cursor sits in the trailing run of records that have
+                # no value for the sort field. Records missing the sort
+                # field sort last in both directions (see
+                # ``finalize_find_results``), so everything still ahead of
+                # us is also missing it — walk that run by id alone.
+                keyset_branches = [
                     {
                         "$and": [
-                            {primary_field: cursor_payload["sort"]},
+                            {primary_field: None},
+                            {"id": {id_op: cursor_payload["id"]}},
+                        ]
+                    }
+                ]
+            else:
+                keyset_branches = [
+                    {primary_field: {sort_op: cursor_sort}},
+                    # ``{field: None}`` matches both an explicit null and a
+                    # missing key. Without this branch the nulls-last tail
+                    # is unreachable: ``{field: {"$lt": v}}`` never matches
+                    # a record that has no value at all, so iteration would
+                    # stop at the last record that does.
+                    {primary_field: None},
+                    {
+                        "$and": [
+                            {primary_field: cursor_sort},
                             {"id": {id_op: cursor_payload["id"]}},
                         ]
                     },
                 ]
-            }
+            keyset_filter: Dict[str, Any] = (
+                keyset_branches[0]
+                if len(keyset_branches) == 1
+                else {"$or": keyset_branches}
+            )
             final_query = (
                 {"$and": [final_query, keyset_filter]} if final_query else keyset_filter
             )
@@ -1459,7 +1485,13 @@ async def find_page(
         next_cursor: Optional[str] = None
         if has_more and page_rows:
             last = page_rows[-1]
-            payload = {"id": last.get("id"), "sort": last.get(primary_field)}
+            # Dotted sort fields (``context.started_at``) need the same
+            # path walk the adapters use; a flat ``.get`` would mint a
+            # ``None`` sort value for every cursor and stall pagination.
+            payload = {
+                "id": last.get("id"),
+                "sort": resolve_sort_value(last, primary_field),
+            }
             next_cursor = base64.urlsafe_b64encode(
                 json.dumps(payload, separators=(",", ":")).encode()
             ).decode()

diff --git a/jvspatial/core/pager.py b/jvspatial/core/pager.py
@@ -5,10 +5,11 @@
 Designed to integrate seamlessly with UI frameworks requiring paginated data.
 """
 
-import contextlib
 from math import ceil
 from typing import TYPE_CHECKING, Any, Dict, List, Optional, Type, TypeVar
 
+from jvspatial.db.database import finalize_find_results
+
 if TYPE_CHECKING:
     from .entities import Object
 
@@ -195,14 +196,15 @@ async def get_page(
             )
             page_items_raw = all_items_raw[offset : offset + self.page_size]
 
-        # Apply in-Python ordering when a non-id order_by is set.
+        # Safety net for a backend that ignored the requested sort. Route it
+        # through the same helper the adapters use so it is a no-op when the
+        # sort was honored: an ad-hoc key here would disagree with the DB-side
+        # ordering that produced the slice above. In particular, defaulting a
+        # missing value to ``0`` ordered it among the real values while the
+        # slice had already placed it in the trailing missing-value run, which
+        # duplicates or drops rows across page boundaries.
         if self.order_by and page_sort != [("id", 1)]:
-            reverse = self.order_direction.lower() == "desc"
-            with contextlib.suppress(KeyError, TypeError):
-                page_items_raw.sort(
-                    key=lambda item: item.get("context", {}).get(self.order_by, 0),
-                    reverse=reverse,
-                )
+            page_items_raw = finalize_find_results(page_items_raw, sort=page_sort)
 
         page_objects: List[T] = []
         for item_data in page_items_raw:

diff --git a/jvspatial/db/_postgres_translate.py b/jvspatial/db/_postgres_translate.py
@@ -589,10 +589,10 @@ def translate_sort(
     or invalid direction). The fragment does NOT include the leading
     ``ORDER BY`` keyword.
 
-    NULLs sort last for ascending, first for descending — mirrors
-    :func:`jvspatial.db.database.finalize_find_results` semantics. (Postgres'
-    default puts NULLs first for ASC, which is the opposite of what
-    callers expect, so we set NULLS LAST / NULLS FIRST explicitly.)
+    NULLs sort last in both directions — mirrors
+    :func:`jvspatial.db.database.finalize_find_results` semantics (SPEC §4.1,
+    find sort contract). Postgres' own default puts NULLs first for ASC and
+    last for DESC, so ``NULLS LAST`` is set explicitly on every key.
     """
     if not sort:
         return None

diff --git a/jvspatial/db/_sqlite_translate.py b/jvspatial/db/_sqlite_translate.py
@@ -28,9 +28,9 @@
 ORDER BY pushdown
 -----------------
 :func:`translate_sort` handles single-/multi-key sorts on simple
-identifiers (no operators in the key). NULLs sort last for ascending and
-first for descending, mirroring the in-memory ``finalize_find_results``
-behavior.
+identifiers (no operators in the key). NULLs sort last in *both*
+directions, mirroring the in-memory ``finalize_find_results`` behavior
+(SPEC §4.1, find sort contract).
 
 Security
 --------
@@ -271,8 +271,8 @@ def translate_sort(sort: Optional[List[Tuple[str, int]]]) -> Optional[str]:
     invalid direction). The fragment does NOT include the leading
     ``ORDER BY`` keyword.
 
-    NULLs sort last for ascending, first for descending -- this matches
-    ``finalize_find_results`` semantics.
+    NULLs sort last in both directions -- this matches
+    ``finalize_find_results`` semantics (SPEC §4.1, find sort contract).
     """
     if not sort:
         return None
@@ -287,10 +287,10 @@ def translate_sort(sort: Optional[List[Tuple[str, int]]]) -> Optional[str]:
             # ascending: NULLs last
             parts.append(f"({column} IS NULL), {column} ASC")
         else:
-            # descending: NULLs last too (matches in-memory behavior:
-            # the in-memory sort uses (value is None, value), reverse=True,
-            # which puts None last because it sorts (True, ...) after
-            # (False, ...).)
+            # descending: NULLs last too. The leading ``IS NULL`` term is
+            # itself sorted ASC, so non-NULL rows (0) precede NULL rows (1)
+            # regardless of direction. ``_find_sort_key`` inverts its None
+            # flag for descending sorts to reach the same ordering.
             parts.append(f"({column} IS NULL), {column} DESC")
     return ", ".join(parts)