diff --git a/CHANGELOG.md b/CHANGELOG.md index b9d8905..d3b1959 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,7 +36,9 @@ The spec follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). **Breaking** under [`GOVERNANCE.md`](GOVERNANCE.md) §2 — a new **required** member on `MetaLogDiff`, a rewritten §13.2 clause, a `diff_version` rule (§13.1.1) that a producer stamping an older value now violates, and a definition of `window.start` -and `window.end` (§2.2) that a producer writing its own bounds there now violates. +and `window.end` (§2.2) that a producer writing its own bounds there now violates, +and a definition of `behavior.top_ngrams[].probability` (§4) that a producer writing a +joint probability at `ngram_size` above 2 now violates. It also adds one optional member (`withheld_signals`, §13.2.2), which is additive on its own. MINOR bump: MAJOR stays `0`, so §6's *"the MAJOR field of `metalog_version` must equal the MAJOR of the spec"* is @@ -114,6 +116,23 @@ proposed; any of them that is taken lands as an editor change recorded here. into `start`/`end` moves them to an extension (§7). RFC: pull request [#14](https://github.com/CodeRoasted/metalog-spec/pull/14). +- **§4 — `top_ngrams[].probability` is p(last | first n − 1) at every `ngram_size`, among + sequences of the entry's own length; the *"joint prob otherwise"* text is withdrawn.** + The example's comment read *"p(next | prev) for n=2; joint prob otherwise"*, the only + definition the field had. It was wrong against the one shipped producer, which has + always emitted the conditional at every order, and wrong against the field's one + consumer in this specification: §13's `ngram_delta.rate_changed` compares the value + across two windows as a transition rate, and a joint value restates frequency, which + `count` already carries, so it would move with any change in traffic mix. A new + paragraph under §4 now defines the value: the entry's `count` over the summed `count` + of the window's counted sequences of the same length sharing its first n − 1 ids, + computed before the `top_ngrams_size` cut. The "same length" clause also fixes a + defect found in the reference implementation, which pooled sequences of two lengths + under one denominator when its `top_ngrams` carried both; the fix moves only such + documents. **Breaking by the letter** — a producer that followed "joint" becomes + non-conformant — and editor-merged under [`GOVERNANCE.md`](GOVERNANCE.md) §2's 0.x + rule. No schema changed: the field was, and stays, a number in [0, 1]. + ### Added - **§13.2.1 — `x-metalog-vacuous`, a per-property vacuity declaration.** The witness diff --git a/SPEC.md b/SPEC.md index de9399d..6fa9b69 100644 --- a/SPEC.md +++ b/SPEC.md @@ -801,7 +801,7 @@ Captures *how* templates follow each other, beyond raw frequency. { "sequence": ["h:8a3f...", "h:b104..."], // array of template_ids "count": 8421, - "probability": 0.677 // p(next | prev) for n=2; joint prob otherwise + "probability": 0.677 // p(last | first n−1), among sequences of this length — see below } ], "top_ngrams_size": 64, // integer, required @@ -821,6 +821,19 @@ Captures *how* templates follow each other, beyond raw frequency. } ``` +**`probability` — a conditional, never a joint.** An entry's `probability` is +p(last | prefix): its `count` divided by the summed `count` of every sequence **of the +same length** the producer counted in the window whose first n − 1 template ids equal +the entry's own, n being the length of the entry's `sequence`. At `ngram_size` 2 it is +p(next | prev); at 3 it is p(third | first two). It is computed over every counted +sequence, **before** the `top_ngrams_size` cut, so the retained entries sharing a +prefix need not sum to 1. A sequence of another length never enters the sum: in a +`top_ngrams` holding sequences of more than one length, each length is conditioned +among its own. It is a conditional and not a joint probability because frequency is +already carried by `count`, and because §13's `ngram_delta.rate_changed` compares this +value across two windows as a transition RATE: a joint value would move with any change +in traffic mix and would mean a different quantity at each `ngram_size`. + **`dropped_ngram_observations` — the accounting bound, distinct from top-k truncation.** `top_ngrams` is a RANKING cut: every entry it drops was seen, counted and ranked, and the retained set is the top `top_ngrams_size` of them. A producer MAY additionally bound the