Document how a rule checks a host-owned, mutable list

CHANGELOG.md

@@ -30,6 +30,16 @@ version protects is the API surface `ApiSurfaceTest` calls exported.
   two ways to get it wrong in [`embedding.md`](docs/embedding.md#long-lived-sessions-and-eviction).
   A window in a rule and a window in the session are separate decisions that have to agree, and
   nothing checks that they do.
+- **Documentation for host-owned lists and reference data**, the question this engine had no written
+  answer to: "can a rule check whether a value is in a list my application owns, when a rule's own
+  decision may add to it and every node in a cluster must see the addition". The answer is a fact,
+  looked up before the session with `member: true` *or* `false` so that an outage is an absence and
+  not a false, flipped by `setField` so the same session sees the change, and carried out by `emit`
+  for the host to persist. [`dsl-guide.md`](docs/dsl-guide.md#checking-a-list-your-application-owns)
+  has the compiled recipe, [`embedding.md`](docs/embedding.md#host-owned-lists-and-reference-data)
+  the host half and the cluster note, and the spec records in §1 why a lookup *during* matching is
+  structurally off the table rather than deferred. Nothing in the engine changed to support it,
+  which is the point.
 
 ### Changed
 

docs/choosing-this-engine.md

@@ -134,6 +134,7 @@ The specification's §9.1 has the full accounting; these are the ones people ask
 |---|---|
 | `collect` | answers with a collection, so it has no meaningful `having`, and binding a list needs a way to take one apart that the pattern language does not have |
 | a sliding-window *operator*, "nothing for 24h" | there is no window keyword and no engine-owned clock — nothing here notices time passing with *no fact arriving*, the one input an engine that acts on fact movement never receives. What you assemble instead is a bounded temporal join for what a rule matches plus an `EvictionPolicy.window` for what the session keeps, and a caller-advanced `Clock` fact where the window has to end at "now" — [the recipes](dsl-guide.md#counting-things-in-a-window). That is caller-driven session time, made of parts that already existed rather than a contract of its own |
+| a lookup into host-owned data (a blocklist, a feature store) *during matching* | everything deciding which activation fires assumes a match's answer changes only because a fact moved, so a live lookup would blind refraction, the streaming conflict set and truth maintenance at once. Look it up before the session and insert the answer as a fact, `member: true` or `false`; a rule that adds to the list flips that field and emits the write. [The recipe](dsl-guide.md#checking-a-list-your-application-owns) and [the host half](embedding.md#host-owned-lists-and-reference-data) |
 | `or` inside a `where` | write two rules, use `in`, or reach for a `condition:` expression |
 | backward chaining | the forward-only decision stands; it was made before any code was written |
 | distributed evaluation | the immutability split makes it *feasible* and no more. The partitioning, the wire protocol and cross-node routing are an architecture, not a slice |

docs/dsl-guide.md

@@ -14,6 +14,7 @@ and the engine disagree, this page is wrong.
 - [Matching two facts together](#matching-two-facts-together)
 - [Asking that a fact not exist](#asking-that-a-fact-not-exist)
 - [Doing something](#doing-something)
+- [Checking a list your application owns](#checking-a-list-your-application-owns)
 - [Running it](#running-it)
 - [When it does not fire](#when-it-does-not-fire)
 - [Checking your rules in CI](#checking-your-rules-in-ci)
@@ -607,6 +608,120 @@ event comes back as the return value of the fire call, so a rule is testable wit
 A `callFunction` runs real code at commit time, is not transactional, and if it throws, the changes
 that already landed stay landed.
 
+## Checking a list your application owns
+
+Blocklists, allowlists, watchlists. A rule wants to ask "is this card on the blocklist", and the list
+belongs to the application: it changes on its own cadence, a rule's own decision may add to it, and
+it may live in a store shared by every process running this engine. There is no operator that reaches
+out and asks. There is something better placed: **the answer is a fact.**
+
+Look the membership up *before* the session, once per entity the event names, and insert what you
+found. Then the rules read it the way they read everything else:
+
+```yaml
+apiVersion: rules.v1
+rules:
+  - id: decline-blocklisted-card
+    when:
+      - fact: Payment
+        as: p
+      - fact: ListMembership
+        as: m
+        where:
+          list:     { eq: "card-blocklist" }
+          entityId: { eq: { $ref: p.cardId } }
+          member:   { eq: true }
+    then:
+      - action: setField
+        target: p
+        field: decision
+        value: "DECLINE"
+      - action: emit
+        event: payment.declined
+        payload:
+          paymentId: { $ref: p.id }
+          reason: "card-blocklist"
+
+  - id: blocklist-card-after-third-failure
+    when:
+      - fact: Payment
+        as: p
+        where:
+          failureCount: { gte: 3 }
+      - fact: ListMembership
+        as: m
+        where:
+          list:     { eq: "card-blocklist" }
+          entityId: { eq: { $ref: p.cardId } }
+          member:   { eq: false }
+    then:
+      - action: setField            # the first rule sees this in the next cycle of THIS session
+        target: m
+        field: member
+        value: true
+      - action: emit                # your application writes this to the shared store afterwards
+        event: list.entry.add
+        payload:
+          list:     { $ref: m.list }
+          entityId: { $ref: m.entityId }
+
+  - id: review-when-the-list-could-not-be-checked
+    when:
+      - fact: Payment
+        as: p
+      - fact: ListMembership       # no answer at all: the lookup failed, so fail closed
+        as: m
+        quantifier: notExists
+        where:
+          list:     { eq: "card-blocklist" }
+          entityId: { eq: { $ref: p.cardId } }
+    then:
+      - action: setField
+        target: p
+        field: decision
+        value: "REVIEW"
+```
+
+One `ListMembership` fact per (list, entity) the event names, with `member` true **or false**. That
+second half matters: the fact saying "not on the list" is what the second rule matches, and it is also
+what lets you tell an outage from a known non-membership. If the lookup fails, insert nothing. With no
+fact to bind, neither of the first two rules can fire, so a store that is down declines nobody and
+blocklists nobody; the third rule is the one that sees the gap, because `notExists` over the
+membership is true exactly when nothing answered. Whether "could not check" means review, decline or
+approve is a decision the rule file should state, and that third rule is where it says it.
+
+Three things are going on in that file, and each is a decision.
+
+**Why a fact rather than a callback.** Everything that decides which activation fires assumes that
+during one session a match's answer can only change because a fact moved: refraction is cleared for
+the rules testing a changed path, the streaming matcher drops a rejected match knowing an update will
+bring it back, and truth maintenance re-asks a tuple expecting the same answer. A list consulted live
+would change its answer with nothing moving, and every one of those mechanisms would be blind to it.
+The same reasoning is why the engine owns no clock and time arrives as a
+[`Clock` fact](#as-of-now-and-nothing-has-happened). A fact is constant until you update it, and
+updating it is how the change is announced.
+
+**Reading your own write.** The second rule does two things. `setField` flips `member` on the fact,
+which is a change to a path the first rule tests, so the first rule gets another look in the same
+session and declines the payment in the next cycle. `emit` carries the addition to the outside
+world, where your application writes it to the store after `fireAllRules()` returns. The next
+evaluation, in this process or any other, looks the card up and finds it. The membership fact is a
+snapshot that lives exactly as long as the session, which is why this shape wants one short session
+per event: the engine keeps no longer-lived copy, so across a cluster there is nothing to invalidate
+and the store is the only durable one. Two writes are involved, the decision and the list, and a
+crash between them loses the addition after the decision has been acted on. Record the emitted event
+durably before acting on the decision, or accept that loss knowingly.
+
+**What the engine cannot order for you.** Two evaluations for the same card running at the same
+time each look the card up before either has written. Both see `member: false`, both decide, both
+add. Adding to a set twice is harmless; a list write that is not idempotent is not, and the fix is
+outside the engine: route events for one key to one lane, so they run in sequence.
+
+When the list *is* the stream, because entries arrive and expire continuously and the session runs
+for days, model each entry as its own fact in a long-lived session and ask with `notExists` instead.
+[`embedding.md`](embedding.md#host-owned-lists-and-reference-data) sets the two shapes side by side
+and says when each is the right one.
+
 ## When operator maps aren't enough
 
 There is an escape hatch, and it is deliberately a little inconvenient to reach: it needs an extra

docs/dsl-reference.md

@@ -1054,6 +1054,11 @@ than quietly corrected because of what the stale version cost: a reader who grep
 "temporal" before scrolling to the section that documents them concludes the feature does not exist
 and goes off to compute it at ingestion. That is exactly what happened to somebody.
 
+**Neither is a lookup operator, and one is not coming.** "Is this value in a list my application
+owns" is answered by inserting the membership as a fact before the session, `member: true` or
+`false`, and letting a rule that adds to the list flip that field and `emit` the write. The guide has
+the complete file: [Checking a list your application owns](dsl-guide.md#checking-a-list-your-application-owns).
+
 The quantifiers are *not* on that list any more — [`notExists`](#negation-quantifier-notexists) and
 [`forAll`](#universals-quantifier-forall) are both implemented, with the boundaries those sections
 name: never over an evicted type, and — for `forAll` — vacuously true over an

docs/embedding.md

@@ -17,6 +17,7 @@ For a complete application that does all of this, read
 - [`SessionOptions`](#sessionoptions)
 - [Limits, and the one the engine does not enforce](#limits-and-the-one-the-engine-does-not-enforce)
 - [Host functions](#host-functions)
+- [Host-owned lists and reference data](#host-owned-lists-and-reference-data)
 - [Choosing a matcher](#choosing-a-matcher)
 - [Concurrency](#concurrency)
 - [Long-lived sessions and eviction](#long-lived-sessions-and-eviction)
@@ -244,6 +245,96 @@ object serves many sessions.
 atomic, and cannot be withdrawn. Prefer `emit` and act on `FireResult.emitted()` after the call
 returns.
 
+## Host-owned lists and reference data
+
+The question arrives as "can a rule check whether this value is in a list my application owns", and
+the list is usually mutable, often written by the rules' own decisions, and in a cluster it lives in
+a store every node reads. The engine has no lookup operator, no SPI that consults a host structure
+during matching, and no CEL binding that reaches outside the tuple. That is deliberate, and the
+reason is the one contract everything else here rests on: **during one session, a match's answer
+changes only because a fact moved.** Refraction, the streaming matcher's conflict set, truth
+maintenance and replay all assume it. A structure that changed underneath a running session would
+break all four at once, whatever thread-safety it had.
+
+So the list enters as a fact, and there are two shapes for that.
+
+| Shape | Use it when | How |
+|---|---|---|
+| **Read-through per session** | one session per event; the list is large, shared, and lives in a store | before the session, look up each entity the event names and insert one membership fact per (list, entity), `member: true` or `false`. Writes leave as emitted events and the host applies them after the fire call |
+| **Entries as facts in a long-lived session** | the list *is* the stream: entries arrive and expire continuously, and one session runs for days | insert each entry as its own fact, ask with `notExists`, let a rule's `insertFact` add one. The host retracts an entry when it expires, and that retract is the bound on the session's growth: eviction is not available here, because a `notExists` over an evicted type manufactures matches |
+
+The guide has the rule-file half of the first shape, including how a rule that adds to the list
+makes the addition visible to the rest of the same session:
+[Checking a list your application owns](dsl-guide.md#checking-a-list-your-application-owns). The
+host half is a lookup before `newSession()` and a write after `fireAllRules()`:
+
+```java
+List<ExportedFact> memberships = lookups.membershipsFor(event);   // your store, your client
+try (RuleSession session = rules.newSession(options)) {
+    session.insert("Payment", payment);
+    memberships.forEach(m -> session.insert(m.type(), m.payload()));
+    FireResult result;
+    try {
+        result = session.fireAllRules();
+    } catch (RuleEngineLimitExceeded breach) {
+        result = breach.partialResult();       // completed work is never discarded; decide what it means
+    }
+    for (EmittedEvent e : result.emitted()) {   // AFTER the decision, never during it
+        if (e.eventType().equals("list.entry.add")) {
+            // The one-argument form: path() gives a MissingNode for an absent key, and Jackson 3's
+            // stringValue() THROWS on it where the null-returning read is what a payload check wants.
+            String list = e.payload().path("list").stringValue(null);
+            String entityId = e.payload().path("entityId").stringValue(null);
+            if (list != null && entityId != null) {
+                outbox.record(list, entityId);      // durable before the decision is acted on
+            }
+        }
+    }
+}
+```
+
+That last line is the dual-write problem in one word. The session has decided and the store has not
+yet heard; a crash between the two loses the addition while the decision stands. Write the emitted
+event to something durable before acting on the decision, or accept the loss and say so where the
+next reader will find it.
+
+A membership fact, as a fact document, so a fixture can say what the store would have said:
+
+```yaml
+# memberships.yaml: one fact per (list, entity) the event names; member is true OR false
+- type: ListMembership
+  payload: { list: "card-blocklist", entityId: "4111000000001111", member: false, asOfEpochMs: 1756800000000 }
+```
+
+Four things to hold onto:
+
+- **A failed lookup is an absent fact, never `member: false`.** With `member` true *or* false on
+  every fact the store answered for, a missing fact means one thing only: the store did not answer.
+  Insert `false` on an outage and every `member: { eq: false }` fires against a card nobody checked;
+  insert nothing and neither `eq: true` nor `eq: false` has a fact to bind, while a `notExists` over
+  the membership sees the gap and can fail closed. It is the absence-versus-value distinction §2.6.1
+  draws for a field, applied one level up, to the fact. The entries-as-facts shape has the mirror
+  hazard: a load that fails part-way leaves an *empty list*, indistinguishable from a list with no
+  entries, so every `notExists` over it fires. There, a failed load must fail the session rather
+  than leave it half-filled.
+- **Cluster propagation is the store's job, and the engine keeps no copy.** Every node reads through
+  at session start, so a write from any node is visible to the next session anywhere as soon as the
+  store has it. A per-node cache would reintroduce the problem this design removes.
+- **Determinism holds per session, and ordering across sessions is yours.** Two sessions for the same
+  key running concurrently each read before either writes. Make list writes idempotent, or route
+  events for one key to one lane so they run in sequence. The engine cannot order sessions it did not
+  start.
+- **Which lists a rule set reads is derivable from the compiled rules**, without parsing anything:
+  walk `CompiledRule.source().when()` for patterns on the membership type and read the `list`
+  literal. **Read it and never mutate it**: the constraint records deep-copy a literal on the way in
+  but hand back the live node, so an edit there changes what every session matches (§5.5's
+  invariant 1, `ImmutabilityTest`). An explicit declaration beside the rule file is the better audit
+  record; the walk is what checks that the declaration is complete.
+
+`callFunction` is not the door for either half. It is `void`, so it cannot bring a value back; it
+runs at commit, inside the fire loop; and its own contract asks handlers to be deterministic and
+non-blocking, which a store round trip is not.
+
 ## Choosing a matcher
 
 Three matchers, held to producing **identical firing sequences**. Everything deciding *which*