feat(drive): time-range index TTL — O(1) flat-drop drainage and ephemeral-bytes fees

book/src/SUMMARY.md

@@ -68,6 +68,7 @@
 - [Average Index Examples](drive/average-index-examples.md)
 - [Document Ranked Trees](drive/document-ranked-trees.md)
 - [Ranked Index Examples](drive/ranked-index-examples.md)
+- [Time-Range Index TTL](drive/time-range-ttl.md)
 - [Index-Only Document Types](drive/index-only-document-types.md)
 
 # Testing

book/src/drive/time-range-ttl.md

@@ -0,0 +1,233 @@
+# Time-Range Index TTL
+
+Architecture reference for the `ttl` key of `timeRange` indexes: what it
+means, how expired windows are drained, and the invariants every walker
+shares. The storage primitive underneath is grovedb's flat-subtree drop
+([dashpay/grovedb#848](https://github.com/dashpay/grovedb/issues/848),
+landed in grovedb PR #849); see
+[the storage section](#grovedb-dependency-flat-subtree-drop).
+
+## Motivation
+
+A `timeRange` index stores every document once per containing window, and
+a ranked one additionally rewrites a per-window secondary on every write.
+All of those bytes are billed as **storage** — a price that prepays
+~perpetual retention through the epoch-distribution model — even though
+windowed data is intrinsically ephemeral: a "posts liked this hour"
+bucket is worthless once the trending surface has moved past it. The
+result is that the flagship use case (likes feeding a trending index)
+pays perpetuity prices for state with a useful life measured in days,
+multiplied by the grid's overlap factor.
+
+Nobody cleans this up, either. Deletion costs the deleter processing,
+refunds accrue to owners who have no reason to come back for entries this
+small, and the state lingers forever.
+
+## Semantics
+
+A `timeRange` index may declare a **time to live**:
+
+```json
+"timeRange": { "on": "$createdAt", "range": 3600, "step": 3600, "ttl": 604800 }
+```
+
+Entries under this index are queryable until `ttl` seconds past their
+bucket's start (the exact expiry boundary remains inclusive). Physical
+removal depends on subsequent writes and has no wall-clock deadline. Expired
+buckets are drained **lazily, on write**: every state transition that
+writes into the index continues draining the oldest expired bucket,
+deepest-first, under a per-write operation budget. A fully drained
+window is provably absent, exactly like a window that never held
+documents. An expired window is **not queryable at all** — `byStart`
+rejects starts past the horizon, so the drainage lag is purely internal:
+drainage only ever touches expired buckets, which makes every window a
+query can address complete. Everything written under the index's
+grid-qualified level
+bills as **processing, not storage** — including the transitional bytes
+— at an ephemeral-bytes rate.
+
+### Why the fee reclassification is honest, not a subsidy
+
+Storage fees prepay retention distributed across future epochs. TTL
+indexes instead charge a flat per-byte processing surcharge for their
+transitional storage and write amplification. Version 1 caps the queryable
+lifetime at **one week** (`SystemLimits::max_time_range_ttl_seconds = 604 800`).
+Cleanup capacity exceeds the maximum rate at which continued writes can
+create trees. This is an amortized retention model, not a guarantee that
+physical bytes disappear within a week: bursts need subsequent writes to
+drain, and inactive indexes retain residue as described below.
+
+The load-bearing simplification: **TTL'd subtrees never create
+refundable storage.** No `StorageFlags`, no owner/epoch refund entries.
+That single property pays off three times:
+
+1. the fee reroute needs no refund-ledger reconciliation;
+2. cleanup owes nobody anything;
+3. deletion needs no byte metering for consensus — which is what makes
+   O(1) bucket drops possible at all (see the grovedb dependency).
+
+## Grammar and validation
+
+- `ttl` is an optional key of the `timeRange` map, in seconds, parsed
+  into the transform. It is **not part of the grid identity**:
+  [`TimeRangeTransform::storage_key`] excludes it, so declaring or
+  changing a TTL never forks the storage level, and query-side grid
+  matching ([`TimeRangeGridSpec`]) continues to compare
+  `(range, step, phase)` only.
+- **`ttl ≥ range`.** `$createdAt` is consensus-assigned from block time,
+  so writes only ever target windows containing *now*; this invariant
+  guarantees no bucket that can still receive entries (or serve as the
+  `oldest` selector's window) is ever dropped.
+- **`ttl ≤ SystemLimits::max_time_range_ttl_seconds`** (one week in v1).
+  The cap is what makes the flat ephemeral-byte rate safe.
+- **One TTL per grid per field.** Two indexes bucketing the same field
+  with the same grid share one storage level; a differing `ttl` would
+  give the shared subtree two conflicting lifecycles. Rejected at
+  contract validation.
+- Composes with everything the grid already composes with: `countable`,
+  the range axes, ranked levels below the bucket, `unique`
+  (`range == step`, `$createdAt`), indexOnly document types.
+  `preallocated` stays banned with `timeRange` for the pre-existing
+  structural reason.
+
+## Cleanup
+
+**Trigger** — deterministic and write-amortized: **every write** into a
+TTL'd index continues drainage of the oldest expired bucket (start
+`< block_time − ttl`), deepest-first. Each grid's per-write drop budget is
+`max(SystemLimits::min_time_range_ttl_drop_operations_per_write, 2 × overlap × trees)`.
+The versioned floor is 32; `trees` bounds everything one document can
+create under one bucket: value trees, terminal `[0]` trees, and all
+property-name branches in the grid's merged index structure. Shared grids
+and deep suffixes are counted. Thus cleanup has capacity above the maximum
+tree creation rate, including at the supported overlap of 24. A fixed
+32-drop cap cannot keep up with that overlap.
+
+When nothing is expired, the check is a single bounded range read. Large
+expired buckets drain across writes. In a document batch, **all cleanup
+runs before any document mutations are generated**, including nested
+same-type document groups. Each document earns a budget; conversion then
+uses the prepared state without further direct drops. Estimation performs
+neither cleanup nor its bookkeeping reads. Drops share the caller's
+transaction, so rollback restores both the removed paths and their redo
+records.
+
+**Residue** — an index that stops receiving writes retains its remaining
+buckets, including any expired backlog, indefinitely. That state owes no
+refund, but its size depends on past write volume; the TTL cap alone does
+not bound it. An epoch-transition sweep could provide a backstop, following
+`check_for_ended_vote_polls` / `clean_up_after_vote_polls_end`; it remains
+**out of scope for v1**.
+
+**User deletes and updates of expired documents** — handled at
+**full-path granularity**, because a bucket drains piecewise: an entry
+whose bucket (or whose group's trees inside a standing bucket) the drain
+already took is skipped as cleanly removed; one whose trees still stand
+is removed normally, so a not-yet-drained expired bucket never carries
+dangling references. Every check is deterministic — it reads consensus
+state plus the carried `$createdAt` and block time. Writes never target
+expired windows, so an update of a fully expired document simply leaves
+it without entries under the TTL'd index.
+
+**Per-index semantics** — TTL removes entries from *this index only*.
+An indexOnly like whose windowed entries expire keeps counting in the
+all-time ranked `byPost` and in `byLiker`; permanence lives where the
+contract declares it. Ranked per-window secondaries die with their
+bucket — which also caps live leaderboard state at ~`ttl / step` windows
+per index.
+
+## grovedb dependency: flat-subtree drop
+
+Dropping a bucket must never put user-scaled work on the consensus path.
+The primitive that landed (grovedb PR #849) is the **flat-subtree drop**:
+O(1) consensus removal of a subtree *declared to contain no child
+subtrees* — an ordinary parent-Merk element delete whose cost is
+independent of the subtree's contents — staging a durable redo record
+(atomically, outside the root hash) that names every storage prefix the
+drop orphaned: the subtree's own and, for indexed primaries, its three
+per-axis secondary prefixes. Reclamation is DB-level range tombstones,
+drained by `GroveDb::flush_pending_prefix_drops` — idempotent,
+crash-safe, snapshot-correct, and never part of consensus cost.
+
+A time-range bucket is *not* flat, so the platform drains it
+**deepest-first, one flat unit at a time** (`drain_expired_time_range_buckets`):
+
+1. each group's `[0]` reference tree — flat by construction, and where
+   the mass lives — is flat-dropped;
+2. the emptied group value tree leaves through the flat drop — or, under
+   a ranked (indexed-primary) property-name tree, through grovedb's
+   dedicated indexed-tree delete, which mirrors the group out of the
+   ranking secondary;
+3. the drained property-name tree is flat-dropped (dooming its secondary
+   prefixes when ranked);
+4. the emptied bucket is flat-dropped.
+
+Every step is O(1); the number of steps scales with the window's distinct
+groups and is capped by the structure-derived per-write budget above.
+Between writes a bucket may stand partially drained. Removal walkers skip
+only paths already removed from expired buckets and delete standing
+entries normally. The indexOnly delete validation uses the same rule:
+every surviving entry must match the full row commitment, including entries
+in expired but standing trees. Missing live entries, missing terminal
+members in standing trees, and mismatched commitments fail. An
+indexOnly contract must retain a timestamp-independent proof index, which
+still has to prove the row's membership after all its TTL entries drain.
+
+The flat-drop path-reuse contract (never re-create a dropped path before
+its record drains) holds by construction: bucket paths embed their
+window start, and writes never target expired windows. The host side:
+drive-abci calls `flush_pending_prefix_drops` after committing each
+block's transaction and once at startup, completing reclamation a crash
+may have interrupted.
+
+## Fee mechanics
+
+Write operations targeting a TTL'd index's subtrees are classified
+**ephemeral**: the walkers route them into a separate operation batch
+(`LowLevelDriveOperation::EphemeralGroveOperation`), applied after the
+standing batch, whose captured cost is consumed on its own terms — added
+bytes bill to **processing** at
+`FeeStorageVersion::ttl_ephemeral_disk_usage_credit_per_byte`
+(270 credits/byte, 1% of the storage rate, ~27× a pro-rata week of
+retention) and the storage fee contribution is **zero**. The elements
+carry no storage flags, so deletion — the TTL drain or a user delete of
+a not-yet-expired document — is basic removal with no refund entries:
+there is nothing to refund, which is also where TTL writers collectively
+pre-pay the drainage described below. Cost estimation routes through the
+same split, so estimated and actual fees stay in the same class.
+
+Drainage itself and the walkers' TTL bookkeeping reads are **unbilled**:
+their costs go to scratch accounting, never to the triggering user. That
+is load-bearing for the `estimated >= actual` fee invariant — the
+estimation dry run cannot read state and therefore cannot price
+state-dependent drainage, so billing it only on execution would let a
+transition pass validation and then overdraw on apply. The unbilled work
+is bounded: a capped count of O(1) drop operations plus a handful of
+bounded reads per write.
+
+## Queries
+
+Unchanged in shape, with one hard rule: on a TTL'd index, **expired
+windows are not queryable**. `byStart` resolution rejects any start past
+the expiry horizon (the same strictly-below predicate the drain uses),
+on the server from committed block time and on the verifier from the
+quorum-signed response `time_ms` — so a node cannot serve an expired
+window's remnants past a verifying client. The point of the gate is that
+a mid-drainage window would otherwise serve a truncated answer that
+looks authoritative; rejecting the question is deterministic where
+"whatever the drain has left" is not. Because drainage only ever touches
+expired buckets, every window the resolver admits is **complete**, and a
+window a past drain fully emptied inside its lifetime never existed —
+absence proves normally. The relative selectors (`newest` / `oldest`)
+can never address an expired window at all (`ttl >= range` guarantees
+it).
+
+## Versioning
+
+Everything rides the still-unreleased PV14 grammar: the `ttl` key joins
+the meta-schema v3 `timeRange` map, the two limits join the (also
+unreleased) `SYSTEM_LIMITS_V4` in place, and the ephemeral-bytes rate
+joins the shared storage fee table directly — no fee-version fork,
+because the rate is dead below PV14 (the grammar does not parse, so no
+ephemeral-classified operation can exist to read it). No migration
+story exists or is needed.

packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json

@@ -666,6 +666,11 @@
                 "type": "integer",
                 "minimum": 0,
                 "description": "Grid alignment phase, in seconds. Range starts are `phase + k * step`; must be strictly less than `step` (a larger value would be a redundant spelling of `phase % step`) and strictly less than one year (31536000 — a phase further out could sit past current block time on a huge step, leaving valid timestamps before the grid's first bucket). A pure alignment offset — it moves where window boundaries fall (e.g. daily windows cut at 06:00 UTC instead of midnight) and never excludes any real timestamp. Defaults to 0."
+              },
+              "ttl": {
+                "type": "integer",
+                "minimum": 1,
+                "description": "Time to live, in seconds: entries under this index exist for at most this long past their bucket's start, plus a bounded drainage lag — every write into the index continues draining the oldest expired bucket under a per-write operation budget, and expired windows are not queryable (a byStart selection past the horizon is rejected), so every queryable window is complete. Must be at least `range` (a window still able to receive consensus-timestamped writes can never expire) and at most a protocol-versioned cap (604800 — one week — at protocol version 14). Indexes bucketing one field with the same grid share its storage level and must declare the same ttl. Bytes written under a TTL'd index bill to processing at an ephemeral-bytes rate instead of to storage, carry no storage flags, and refund nothing on removal. Omitted means entries live forever. Available from protocol version 14."
               }
             },
             "required": ["on", "range", "step"],

packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/common/mod.rs

@@ -928,6 +928,40 @@ fn parse_indices(
                                     ));
                                 }
                             }
+                            // The TTL cap is likewise a versioned system
+                            // limit — it is what makes billing TTL'd bytes
+                            // at a flat processing rate honest, so retuning
+                            // it is a protocol-version decision. The lower
+                            // bound (`ttl >= range`) is structural and
+                            // checked in `Index` parsing.
+                            if let Some(ttl_seconds) = transform.ttl_seconds {
+                                if let Some(max_ttl) = ctx
+                                    .platform_version
+                                    .system_limits
+                                    .max_time_range_ttl_seconds
+                                {
+                                    if ttl_seconds > max_ttl {
+                                        return Err(consensus_or_protocol_data_contract_error(
+                                            DataContractError::InvalidContractStructure(format!(
+                                                "timeRange.ttl ({} seconds) exceeds the maximum \
+                                                 of {} seconds: the flat ephemeral-storage \
+                                                 pricing TTL'd entries bill under is only an \
+                                                 honest rate while the lifetime it covers is \
+                                                 bounded",
+                                                ttl_seconds, max_ttl
+                                            )),
+                                        ));
+                                    }
+                                } else {
+                                    return Err(consensus_or_protocol_data_contract_error(
+                                        DataContractError::InvalidContractStructure(
+                                            "timeRange.ttl is not supported by this protocol \
+                                             version"
+                                                .to_string(),
+                                        ),
+                                    ));
+                                }
+                            }
                             let source = transform.source.as_str();
                             let is_system_timestamp = matches!(
                                 source,
@@ -1131,9 +1165,45 @@ fn parse_indices(
     // different grids (or not at all) — each grid forks into its own index
     // level, keyed by the property name qualified with the grid parameters
     // (`TimeRangeTransform::storage_key`), so a bucketed level never shares
-    // a keyspace with a plain level or with another grid's level. No
-    // cross-index agreement rule is needed; identical grids simply share
-    // one level.
+    // a keyspace with a plain level or with another grid's level. The ONE
+    // cross-index agreement rule is the TTL: it is deliberately excluded
+    // from the grid identity (declaring or changing it must not fork the
+    // storage level), so two indexes sharing a grid on one field share one
+    // level's subtrees — and a level cannot have two lifecycles. Identical
+    // grids must declare identical TTLs (including both declaring none).
+    for (name_a, index_a) in indices.iter() {
+        let Some(transform_a) = &index_a.time_range else {
+            continue;
+        };
+        for (name_b, index_b) in indices.iter() {
+            if name_b <= name_a {
+                continue;
+            }
+            let Some(transform_b) = &index_b.time_range else {
+                continue;
+            };
+            if transform_a.source == transform_b.source
+                && transform_a.range_seconds == transform_b.range_seconds
+                && transform_a.step_seconds == transform_b.step_seconds
+                && transform_a.phase_seconds == transform_b.phase_seconds
+                && transform_a.ttl_seconds != transform_b.ttl_seconds
+            {
+                return Err(consensus_or_protocol_data_contract_error(
+                    DataContractError::InvalidContractStructure(format!(
+                        "indexes \"{}\" and \"{}\" bucket \"{}\" with the same grid but \
+                         different TTLs ({:?} vs {:?} seconds): indexes sharing a grid share \
+                         its storage level, and one level cannot have two lifecycles — \
+                         declare the same ttl on both (or on neither)",
+                        name_a,
+                        name_b,
+                        transform_a.source,
+                        transform_a.ttl_seconds,
+                        transform_b.ttl_seconds
+                    )),
+                ));
+            }
+        }
+    }
 
     let index_structure =
         IndexLevel::try_from_indices(indices.values(), ctx.name, ctx.platform_version)?;

packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/ranked_prefix_overlap.rs

@@ -309,6 +309,7 @@ mod tests {
             range_seconds: 3_600,
             step_seconds: 3_600,
             phase_seconds: 0,
+            ttl_seconds: None,
         }
     }