From 6152a3ab7c6a52473c478dc166ffc82ed00ac1c6 Mon Sep 17 00:00:00 2001 From: Emmanuel Prunet Date: Fri, 2 Oct 2026 12:21:26 +0200 Subject: [PATCH] =?UTF-8?q?spec(0.10.0):=20=C2=A74's=20probability=20is=20?= =?UTF-8?q?p(last=20|=20first=20n-1)=20among=20sequences=20of=20the=20entr?= =?UTF-8?q?y's=20own=20length=20=E2=80=94=20"joint=20prob=20otherwise"=20w?= =?UTF-8?q?ithdrawn?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The example comment was the field's only definition, and it was wrong at every ngram_size above 2: the reference producer has always emitted the conditional, and §13's ngram_delta.rate_changed reads the value as a transition rate, which a joint probability (a restatement of count) cannot be. A paragraph under §4 now defines it: count over the summed count of the window's counted sequences of the same length sharing the first n-1 ids, before the top_ngrams_size cut. Breaking by the letter, editor-merged under GOVERNANCE §2's 0.x rule; no schema change. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 21 ++++++++++++++++++++- SPEC.md | 15 ++++++++++++++- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b9d8905..d3b1959 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,7 +36,9 @@ The spec follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). **Breaking** under [`GOVERNANCE.md`](GOVERNANCE.md) §2 — a new **required** member on `MetaLogDiff`, a rewritten §13.2 clause, a `diff_version` rule (§13.1.1) that a producer stamping an older value now violates, and a definition of `window.start` -and `window.end` (§2.2) that a producer writing its own bounds there now violates. +and `window.end` (§2.2) that a producer writing its own bounds there now violates, +and a definition of `behavior.top_ngrams[].probability` (§4) that a producer writing a +joint probability at `ngram_size` above 2 now violates. It also adds one optional member (`withheld_signals`, §13.2.2), which is additive on its own. MINOR bump: MAJOR stays `0`, so §6's *"the MAJOR field of `metalog_version` must equal the MAJOR of the spec"* is @@ -114,6 +116,23 @@ proposed; any of them that is taken lands as an editor change recorded here. into `start`/`end` moves them to an extension (§7). RFC: pull request [#14](https://github.com/CodeRoasted/metalog-spec/pull/14). +- **§4 — `top_ngrams[].probability` is p(last | first n − 1) at every `ngram_size`, among + sequences of the entry's own length; the *"joint prob otherwise"* text is withdrawn.** + The example's comment read *"p(next | prev) for n=2; joint prob otherwise"*, the only + definition the field had. It was wrong against the one shipped producer, which has + always emitted the conditional at every order, and wrong against the field's one + consumer in this specification: §13's `ngram_delta.rate_changed` compares the value + across two windows as a transition rate, and a joint value restates frequency, which + `count` already carries, so it would move with any change in traffic mix. A new + paragraph under §4 now defines the value: the entry's `count` over the summed `count` + of the window's counted sequences of the same length sharing its first n − 1 ids, + computed before the `top_ngrams_size` cut. The "same length" clause also fixes a + defect found in the reference implementation, which pooled sequences of two lengths + under one denominator when its `top_ngrams` carried both; the fix moves only such + documents. **Breaking by the letter** — a producer that followed "joint" becomes + non-conformant — and editor-merged under [`GOVERNANCE.md`](GOVERNANCE.md) §2's 0.x + rule. No schema changed: the field was, and stays, a number in [0, 1]. + ### Added - **§13.2.1 — `x-metalog-vacuous`, a per-property vacuity declaration.** The witness diff --git a/SPEC.md b/SPEC.md index de9399d..6fa9b69 100644 --- a/SPEC.md +++ b/SPEC.md @@ -801,7 +801,7 @@ Captures *how* templates follow each other, beyond raw frequency. { "sequence": ["h:8a3f...", "h:b104..."], // array of template_ids "count": 8421, - "probability": 0.677 // p(next | prev) for n=2; joint prob otherwise + "probability": 0.677 // p(last | first n−1), among sequences of this length — see below } ], "top_ngrams_size": 64, // integer, required @@ -821,6 +821,19 @@ Captures *how* templates follow each other, beyond raw frequency. } ``` +**`probability` — a conditional, never a joint.** An entry's `probability` is +p(last | prefix): its `count` divided by the summed `count` of every sequence **of the +same length** the producer counted in the window whose first n − 1 template ids equal +the entry's own, n being the length of the entry's `sequence`. At `ngram_size` 2 it is +p(next | prev); at 3 it is p(third | first two). It is computed over every counted +sequence, **before** the `top_ngrams_size` cut, so the retained entries sharing a +prefix need not sum to 1. A sequence of another length never enters the sum: in a +`top_ngrams` holding sequences of more than one length, each length is conditioned +among its own. It is a conditional and not a joint probability because frequency is +already carried by `count`, and because §13's `ngram_delta.rate_changed` compares this +value across two windows as a transition RATE: a joint value would move with any change +in traffic mix and would mean a different quantity at each `ngram_size`. + **`dropped_ngram_observations` — the accounting bound, distinct from top-k truncation.** `top_ngrams` is a RANKING cut: every entry it drops was seen, counted and ranked, and the retained set is the top `top_ngrams_size` of them. A producer MAY additionally bound the