Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 29 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ joint probability at `ngram_size` above 2 now violates, a rule on which template
`branching_delta` row (§13.1) that a producer emitting one for a template branching on one
side only now violates, and a binding of each param to a whole-token wildcard (§3.5) that a
producer emitting a param for a wildcard inside a token, or none for a whole-token wildcard,
now violates.
now violates, and two rules on `behavior.ngram_size` (§2.4, §12.1) that a producer keeping one
`retention_profile` across two orders, or a composer merging two orders' blocks, now violates.
It also adds one optional member
(`withheld_signals`, §13.2.2), which is additive on its own. MINOR bump: MAJOR stays `0`, so
§6's *"the MAJOR field of `metalog_version` must equal the MAJOR of the spec"* is
Expand Down Expand Up @@ -71,10 +72,36 @@ in pull request [#14](https://github.com/CodeRoasted/metalog-spec/pull/14) under
0.x line without an `rfc:` issue or a comment window, while the reference
implementation is the only producer and consumer. The RFC comes back at the v1.0
freeze, or earlier the day a second implementation is listed. P1, P3 and P4 stay
proposed; any of them that is taken lands as an editor change recorded here.
proposed; any of them that is taken lands as an editor change recorded here. **P3's item (f)**,
the order-dependent `behavior.ngram_size`, is the first taken: see the §2.4 / §12.1 entry under
*Changed*. The rest of P3 (the caps rule) stays proposed.

### Changed

- **§2.4 — `retention_profile` names `behavior.ngram_size`, and §12.1 omits `behavior` when
two composed inputs differ in it.** `ngram_size` was in neither identifier's list, so two
documents produced at different sequence orders could carry equal identifiers and pass
§2.4's comparability gate. `compose()` then had to pick one input's `ngram_size` — a
required field — so `compose(A, B)` and `compose(B, A)` declared different values, against
§12.2's commutativity MUST; and a `MetaLogDiff` across the pair reported every n-gram of
one side as vanished and every n-gram of the other as new, which §13.2 counts as a witness
of `"changed"`. An order-`m` key and an order-`n` key denote different objects, so no
minimum repairs it: the smaller order either declares an order the array does not hold or
drops one input's n-grams with no field reporting the loss. Two changes. **§2.4** states
what `retention_profile` names as a rule — every parameter that fixes which entries a
bounded block retains, how they are ranked, and what an entry's key denotes — with the
existing items and `behavior.ngram_size` as its examples, so the existing gate refuses a
cross-order `compose()` and `MetaLogDiff` at both sites with no new mechanism. **§12.1**
covers the case §2.4's gate does not reach (an input without the identifier): when both
inputs carry `behavior` at different `ngram_size`, `C.behavior` **MUST** be omitted, never
merged at a minimum and never carried from one side; when they agree, `C.behavior.ngram_size`
is that value. Omitting the block also drops `dropped_ngram_observations`: a count lost,
not falsified. The §1 table and the schema's `retention_profile` description name the
parameter. **Breaking** — a producer keeping one `retention_profile` across two orders, or a
composer merging two orders' blocks, becomes non-conformant — and editor-merged under
[`GOVERNANCE.md`](GOVERNANCE.md) §2's 0.x rule. No document's validity changes: both
identifiers stay opaque strings and `behavior` stays optional. RFC #8, P3 (f).

- **A window with no event time has no envelope (§2.2).** `window.start`, `window.end` and
`window.duration_seconds` are present together, or absent together when the producer
attributes an event time to none of the window's lines — a window with no line
Expand Down
29 changes: 24 additions & 5 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ top-level fields:
| `window` | object | yes | The line count and, when any line has one, the event-time envelope of the lines the window contains. See §2.2. |
| `source` | object | yes | What was observed (service, host, fleet). See §2.3. |
| `canonicalization_version` | string | no | Opaque identifier for the canonicalization rules in effect. Gates `compose()`/diff comparability. See §2.4. |
| `retention_profile` | string | no | Opaque identifier for the retention parameters (top_k, reservoir, salience weights, diversity caps). Gates `compose()`/diff comparability. See §2.4. |
| `retention_profile` | string | no | Opaque identifier for the retention parameters (top_k, reservoir, salience weights, diversity caps, `behavior.ngram_size`). Gates `compose()`/diff comparability. See §2.4. |
| `stats` | object | yes | Per-template counts and frequency metrics. See §3. |
| `templates` | object | no | Optional dedup map `template_id → template_str`. See §3.4. **RESERVED** — see §2. |
| `behavior` | object | no | Sequence/transition fingerprint. See §4. |
Expand Down Expand Up @@ -233,10 +233,14 @@ A MetaLog **MAY** carry two opaque processing-identifier strings that name the
templates and structural metadata). It **MUST** be bumped when those rules'
*output-affecting* semantics change; a binary rebuild with no rule change
**MUST NOT** bump it. It is **not** a binary build id.
- `retention_profile` — names the **retention parameters** in effect: `top_k`
size (§3.1), reservoir admission weights and size and diversity caps (§3.7),
and the salience arithmetic. It **MUST** be bumped when any of those
parameters change.
- `retention_profile` — names the **retention parameters** in effect: every
parameter that fixes which entries a bounded block retains, how they are ranked,
and **what an entry's key denotes**. At this version those are `top_k` size
(§3.1), the reservoir admission weights, size and diversity caps and the salience
arithmetic (§3.7), and `behavior.ngram_size` (§4) — the sequence order, which
fixes the *domain* of `top_ngrams` rather than a bound on it. That list
illustrates the rule and does not close it: a parameter meeting the rule joins it
without an edit here. It **MUST** be bumped when any such parameter changes.

The values are **opaque strings**. This spec defines neither a registry of names
nor a canonical format; producers and consumers within an environment **MUST**
Expand Down Expand Up @@ -1364,6 +1368,21 @@ to a 1-hour MetaLog).
- `C.behavior.dominant_path` is re-derived greedily from the merged
graph; consumers **MUST NOT** assume it equals the path of either
input.
- `C.behavior` **MUST** be omitted entirely when both inputs carry a `behavior` block
and their `ngram_size` values **differ**. An order-`m` key and an order-`n` key with
`m ≠ n` denote different objects, so there is no merged n-gram population to derive:
counts across the two orders are not comparable (every order-`n` occurrence implies
an order-`m` one for `m < n`, so a joint ranking is biased toward the shorter order)
and `probability` is conditioned to a different depth on each side. Where the two
agree, `C.behavior.ngram_size` is that value. A composer **MUST NOT** instead take
the smaller order, nor carry one input's block through unchanged: the first declares
an order the merged array does not hold or silently drops one input's n-grams, and
the second describes one input's lines under a `window` covering both. Omitting the
block also drops `dropped_ngram_observations`; that **loses** a count and does not
falsify one, because §4's omission rule governs the field inside a present block,
not the block. The precondition is block-grained, like the `cube` clause below: the
rest of `C` composes as usual. It is the residual case only — whenever both inputs
carry a `retention_profile`, §2.4's gate refuses the pair before composition begins.
- `C.stability` **MUST** be omitted (it is meaningless across
composed inputs); consumers wanting a current-vs-prior view of a
composed document should use §13 Diff explicitly.
Expand Down
2 changes: 1 addition & 1 deletion schema/metalog.v0.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -467,7 +467,7 @@
},
"retention_profile": {
"type": "string",
"description": "Opaque identifier for the retention parameters — top_k size, reservoir admission weights/size/diversity caps, and the salience arithmetic; gates compose()/diff comparability (SPEC §2.4). Added in v0.5.0."
"description": "Opaque identifier for the retention parameters — top_k size, reservoir admission weights/size/diversity caps, the salience arithmetic, and behavior.ngram_size; gates compose()/diff comparability (SPEC §2.4). Added in v0.5.0."
},
"coordinate": {
"description": "Re-derivation coordinate addressing this window back to its source: raw(window) = replay(source, bounds) (SPEC §15). A raw coordinate XOR a composed coordinate. Added in v0.5.0.",
Expand Down
Loading