feat(drive): support IN over pinned prefix properties in ranked and having-range queries

book/src/drive/document-ranked-trees.md

@@ -91,7 +91,7 @@ One asymmetry is worth knowing when authoring: **the meta-schema demands the lit
 
 Two structural rules, both enforced at contract-parse time in rs-dpp:
 
-- **No aggregating index on a compound ranked index's full prefix.** Ranked flags are allowed on compound indexes, with **per-prefix** semantics: a ranked `[identityId, class]` puts the indexed tree at each prefix value's terminal `class` property-name level — one ordered secondary per `identityId`, each ranking only that identity's `class` groups. There is deliberately no global cross-prefix ordering; the query surfaces require every leading property to be pinned by an equality `where` clause. The one shape that stays impossible — and is rejected per document type, where all indexes are visible (`validate_no_ranked_prefix_overlap`) — is a countable/summable index terminating at exactly the compound's leading prefix: its aggregating value trees would demand the `NonCounted` / `NotSummed` shell around the ranked terminal tree, and the storage layer structurally rejects any wrapper around an indexed tree, because the wrapper would neutralise the very aggregates the secondaries order by. (Drive's fail-closed guard behind the parse-time check is `INDEXED_INNER_UNWRAPPABLE`.) Only the exact `n-1` prefix conflicts: an aggregating index at a shorter prefix wraps a plain intermediate tree, and one extending past the ranked terminal lives inside its value trees — both supported.
+- **No aggregating index on a compound ranked index's full prefix.** Ranked flags are allowed on compound indexes, with **per-prefix** semantics: a ranked `[identityId, class]` puts the indexed tree at each prefix value's terminal `class` property-name level — one ordered secondary per `identityId`, each ranking only that identity's `class` groups. There is deliberately no global cross-prefix ordering; the query surfaces require every leading property to be pinned by a `where` clause — equalities, plus at most one `IN` that fans out across prefix branches and merges deterministically. The one shape that stays impossible — and is rejected per document type, where all indexes are visible (`validate_no_ranked_prefix_overlap`) — is a countable/summable index terminating at exactly the compound's leading prefix: its aggregating value trees would demand the `NonCounted` / `NotSummed` shell around the ranked terminal tree, and the storage layer structurally rejects any wrapper around an indexed tree, because the wrapper would neutralise the very aggregates the secondaries order by. (Drive's fail-closed guard behind the parse-time check is `INDEXED_INNER_UNWRAPPABLE`.) Only the exact `n-1` prefix conflicts: an aggregating index at a shorter prefix wraps a plain intermediate tree, and one extending past the ranked terminal lives inside its value trees — both supported.
 - **Non-unique indexes only.** `ranked aggregates are not supported on unique indexes: each group of a unique index contains at most one document, so there is nothing meaningful to rank`. Contested indexes are covered transitively — a contested index is unique by construction, so it hits the same check rather than needing its own.
 
 ### Version Gate
@@ -231,7 +231,7 @@ Every ranked read — and, on the prove path, every ranked proof — is issued a
   / <last_index_property_name: utf-8>       // e.g. b"restaurantId"
 ```
 
-The children of that tree are the *groups*: one value tree per distinct value of the last index property, keyed by the raw index-key bytes of that value (for a `string` property, its UTF-8 bytes — e.g. `b"alpha"`). The secondary entries a top-k read returns are keyed by those same group keys. A compound index `[a, b]` inserts `<a> / <encoded pinned value of a>` between the doctype and the terminal `<b>` level — the value segment comes from the request's equality `where` pin on `a`, encoded with the same `serialize_value_for_key` the write path used to key that prefix's value tree, so the walk lands on **that prefix's own** indexed tree and secondary.
+The children of that tree are the *groups*: one value tree per distinct value of the last index property, keyed by the raw index-key bytes of that value (for a `string` property, its UTF-8 bytes — e.g. `b"alpha"`). The secondary entries a top-k read returns are keyed by those same group keys. A compound index `[a, b]` inserts `<a> / <encoded pinned value of a>` between the doctype and the terminal `<b>` level — the value segment comes from the request's `where` pin on `a` — an equality pin, or one element of the single permitted `IN` (one branch per element) — encoded with the same `serialize_value_for_key` the write path used to key that prefix's value tree, so the walk lands on **that prefix's own** indexed tree and secondary.
 
 Prover and verifier build this path through the same function, `DriveDocumentRankedQuery::indexed_property_name_tree_path` (with the pinned prefix values encoded by the shared resolver, `resolve_ranked_query_for_mode`), which is why they agree on the root hash by construction.
 
@@ -341,7 +341,7 @@ Note that the fixture puts each shape on its **own document type**. That's not a
 | Top / bottom K groups by sum of a property | `rankedSummable: true` on an index with `summable: "<prop>"` + `rangeSummable: true` |
 | Top / bottom K groups by average of a property | `rankedAverageable: true` on an index with `averageable: "<prop>"` + `rangeAverageable: true` (or the count+sum longhand) |
 | Two rankings on one index (e.g. by count *and* by average) | Both keywords. The tree is a PCPSIT carrying both axes in its TLV; you pay one secondary Merk per axis on every write. |
-| A ranking filtered by another property (`top 5 restaurants in London`) | A **compound ranked index** with the filter property leading: `[city, restaurantId]` with the ranked flags. Each city gets its own secondary; the query pins the prefix with an equality `where` (`WHERE city == "London" GROUP BY restaurantId ORDER BY <agg> DESC LIMIT 5`). Only equality pins — a range or `IN` on the prefix is rejected, and there is no cross-prefix (global) ordering on a compound ranked index. |
+| A ranking filtered by another property (`top 5 restaurants in London`) | A **compound ranked index** with the filter property leading: `[city, restaurantId]` with the ranked flags. Each city gets its own secondary; the query pins the prefix with an equality `where` (`WHERE city == "London" GROUP BY restaurantId ORDER BY <agg> DESC LIMIT 5`). Equality pins select one prefix; at most one pin may be an `IN` (2..=10 distinct elements; a never-written element — or one whose deeper pinned path was never written — contributes an empty branch), which walks one secondary per element and merges by `(aggregate, encoded prefix, group key)`, proved in one branched `PathQuery` envelope (shared ancestors proved once, per-element authenticated absence) — entries then carry `in_key`. A single-element `IN` normalizes to the equality pin; a `null` pin stays legal on its own but cannot combine with an `IN` (null addresses its prefix through an empty path segment the branched proof cannot express); a non-zero `OFFSET` is rejected together with `IN`; and branched proofs are generated from committed state only. A range operator on the prefix stays rejected, and there is still no global cross-prefix ordering beyond that merge. |
 | A ranking on a unique or contested index | Not available, and not meaningful: every group holds at most one document. |
 | Range aggregates without ranking (the 4.0 surface) | Just the `range*` flags. Ranking is strictly additive — adding it never changes what a range query returns. |
 | Nothing ranking-aware (default) | Don't set any `ranked*` flag. The terminal property-name tree keeps the type its range flags give it. |

packages/dapi-grpc/clients/drive/v0/nodejs/drive_pbjs.js

@@ -26734,6 +26734,7 @@ $root.org = (function() {
                                  * @property {number|Long|null} [count] RankedEntry count
                                  * @property {number|Long|null} [sum] RankedEntry sum
                                  * @property {number|null} [avg] RankedEntry avg
+                                 * @property {Uint8Array|null} [inKey] RankedEntry inKey
                                  */
 
                                 /**
@@ -26783,6 +26784,14 @@ $root.org = (function() {
                                  */
                                 RankedEntry.prototype.avg = 0;
 
+                                /**
+                                 * RankedEntry inKey.
+                                 * @member {Uint8Array} inKey
+                                 * @memberof org.dash.platform.dapi.v0.GetDocumentsResponse.GetDocumentsResponseV1.RankedEntry
+                                 * @instance
+                                 */
+                                RankedEntry.prototype.inKey = $util.newBuffer([]);
+
                                 // OneOf field names bound to virtual getters and setters
                                 var $oneOfFields;
 
@@ -26829,6 +26838,8 @@ $root.org = (function() {
                                         writer.uint32(/* id 3, wireType 0 =*/24).sint64(message.sum);
                                     if (message.avg != null && Object.hasOwnProperty.call(message, "avg"))
                                         writer.uint32(/* id 4, wireType 1 =*/33).double(message.avg);
+                                    if (message.inKey != null && Object.hasOwnProperty.call(message, "inKey"))
+                                        writer.uint32(/* id 5, wireType 2 =*/42).bytes(message.inKey);
                                     return writer;
                                 };
 
@@ -26875,6 +26886,9 @@ $root.org = (function() {
                                         case 4:
                                             message.avg = reader.double();
                                             break;
+                                        case 5:
+                                            message.inKey = reader.bytes();
+                                            break;
                                         default:
                                             reader.skipType(tag & 7);
                                             break;
@@ -26933,6 +26947,9 @@ $root.org = (function() {
                                         if (typeof message.avg !== "number")
                                             return "avg: number expected";
                                     }
+                                    if (message.inKey != null && message.hasOwnProperty("inKey"))
+                                        if (!(message.inKey && typeof message.inKey.length === "number" || $util.isString(message.inKey)))
+                                            return "inKey: buffer expected";
                                     return null;
                                 };
 
@@ -26973,6 +26990,11 @@ $root.org = (function() {
                                             message.sum = new $util.LongBits(object.sum.low >>> 0, object.sum.high >>> 0).toNumber();
                                     if (object.avg != null)
                                         message.avg = Number(object.avg);
+                                    if (object.inKey != null)
+                                        if (typeof object.inKey === "string")
+                                            $util.base64.decode(object.inKey, message.inKey = $util.newBuffer($util.base64.length(object.inKey)), 0);
+                                        else if (object.inKey.length >= 0)
+                                            message.inKey = object.inKey;
                                     return message;
                                 };
 
@@ -26989,14 +27011,22 @@ $root.org = (function() {
                                     if (!options)
                                         options = {};
                                     var object = {};
-                                    if (options.defaults)
+                                    if (options.defaults) {
                                         if (options.bytes === String)
                                             object.key = "";
                                         else {
                                             object.key = [];
                                             if (options.bytes !== Array)
                                                 object.key = $util.newBuffer(object.key);
                                         }
+                                        if (options.bytes === String)
+                                            object.inKey = "";
+                                        else {
+                                            object.inKey = [];
+                                            if (options.bytes !== Array)
+                                                object.inKey = $util.newBuffer(object.inKey);
+                                        }
+                                    }
                                     if (message.key != null && message.hasOwnProperty("key"))
                                         object.key = options.bytes === String ? $util.base64.encode(message.key, 0, message.key.length) : options.bytes === Array ? Array.prototype.slice.call(message.key) : message.key;
                                     if (message.count != null && message.hasOwnProperty("count")) {
@@ -27020,6 +27050,8 @@ $root.org = (function() {
                                         if (options.oneofs)
                                             object.value = "avg";
                                     }
+                                    if (message.inKey != null && message.hasOwnProperty("inKey"))
+                                        object.inKey = options.bytes === String ? $util.base64.encode(message.inKey, 0, message.inKey.length) : options.bytes === Array ? Array.prototype.slice.call(message.inKey) : message.inKey;
                                     return object;
                                 };
 

packages/dapi-grpc/clients/platform/v0/nodejs/platform_pbjs.js

@@ -26226,6 +26226,7 @@ $root.org = (function() {
                                  * @property {number|Long|null} [count] RankedEntry count
                                  * @property {number|Long|null} [sum] RankedEntry sum
                                  * @property {number|null} [avg] RankedEntry avg
+                                 * @property {Uint8Array|null} [inKey] RankedEntry inKey
                                  */
 
                                 /**
@@ -26275,6 +26276,14 @@ $root.org = (function() {
                                  */
                                 RankedEntry.prototype.avg = 0;
 
+                                /**
+                                 * RankedEntry inKey.
+                                 * @member {Uint8Array} inKey
+                                 * @memberof org.dash.platform.dapi.v0.GetDocumentsResponse.GetDocumentsResponseV1.RankedEntry
+                                 * @instance
+                                 */
+                                RankedEntry.prototype.inKey = $util.newBuffer([]);
+
                                 // OneOf field names bound to virtual getters and setters
                                 var $oneOfFields;
 
@@ -26321,6 +26330,8 @@ $root.org = (function() {
                                         writer.uint32(/* id 3, wireType 0 =*/24).sint64(message.sum);
                                     if (message.avg != null && Object.hasOwnProperty.call(message, "avg"))
                                         writer.uint32(/* id 4, wireType 1 =*/33).double(message.avg);
+                                    if (message.inKey != null && Object.hasOwnProperty.call(message, "inKey"))
+                                        writer.uint32(/* id 5, wireType 2 =*/42).bytes(message.inKey);
                                     return writer;
                                 };
 
@@ -26367,6 +26378,9 @@ $root.org = (function() {
                                         case 4:
                                             message.avg = reader.double();
                                             break;
+                                        case 5:
+                                            message.inKey = reader.bytes();
+                                            break;
                                         default:
                                             reader.skipType(tag & 7);
                                             break;
@@ -26425,6 +26439,9 @@ $root.org = (function() {
                                         if (typeof message.avg !== "number")
                                             return "avg: number expected";
                                     }
+                                    if (message.inKey != null && message.hasOwnProperty("inKey"))
+                                        if (!(message.inKey && typeof message.inKey.length === "number" || $util.isString(message.inKey)))
+                                            return "inKey: buffer expected";
                                     return null;
                                 };
 
@@ -26465,6 +26482,11 @@ $root.org = (function() {
                                             message.sum = new $util.LongBits(object.sum.low >>> 0, object.sum.high >>> 0).toNumber();
                                     if (object.avg != null)
                                         message.avg = Number(object.avg);
+                                    if (object.inKey != null)
+                                        if (typeof object.inKey === "string")
+                                            $util.base64.decode(object.inKey, message.inKey = $util.newBuffer($util.base64.length(object.inKey)), 0);
+                                        else if (object.inKey.length >= 0)
+                                            message.inKey = object.inKey;
                                     return message;
                                 };
 
@@ -26481,14 +26503,22 @@ $root.org = (function() {
                                     if (!options)
                                         options = {};
                                     var object = {};
-                                    if (options.defaults)
+                                    if (options.defaults) {
                                         if (options.bytes === String)
                                             object.key = "";
                                         else {
                                             object.key = [];
                                             if (options.bytes !== Array)
                                                 object.key = $util.newBuffer(object.key);
                                         }
+                                        if (options.bytes === String)
+                                            object.inKey = "";
+                                        else {
+                                            object.inKey = [];
+                                            if (options.bytes !== Array)
+                                                object.inKey = $util.newBuffer(object.inKey);
+                                        }
+                                    }
                                     if (message.key != null && message.hasOwnProperty("key"))
                                         object.key = options.bytes === String ? $util.base64.encode(message.key, 0, message.key.length) : options.bytes === Array ? Array.prototype.slice.call(message.key) : message.key;
                                     if (message.count != null && message.hasOwnProperty("count")) {
@@ -26512,6 +26542,8 @@ $root.org = (function() {
                                         if (options.oneofs)
                                             object.value = "avg";
                                     }
+                                    if (message.inKey != null && message.hasOwnProperty("inKey"))
+                                        object.inKey = options.bytes === String ? $util.base64.encode(message.inKey, 0, message.inKey.length) : options.bytes === Array ? Array.prototype.slice.call(message.inKey) : message.inKey;
                                     return object;
                                 };