Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,30 @@ proposed; any of them that is taken lands as an editor change recorded here.

### Changed

- **A window with no event time has no envelope (§2.2).** `window.start`, `window.end` and
`window.duration_seconds` are present together, or absent together when the producer
attributes an event time to none of the window's lines — a window with no line
(`lines_observed` 0) included. Until now the three were required, so a producer whose
lines carried no readable time had to write an instant it never observed (a fixed value,
an epoch sentinel, the window's opening time), and a reader could not tell it from a real
one; composition (§12.1) then took that value into a composed `window.start`. A producer
**MUST NOT** write such a value, and a consumer reads an absent envelope as "no line of
this window has an event time". The empty-window clause changes with it: an empty window
carried `start` equal to `end`, and now carries no envelope. Consequences, each stated in
its section: §5's `previous_window_end` is absent when the previous window has no
envelope; §12.1 takes the envelope of the input that has one, and none when neither does;
§12.4's `provenance[].window` and §13.1's `current.window` / `previous.window` are `{}`
for a document with no envelope. **Schemas:** the document `window` requires only
`lines_observed` and binds the three envelope members with `dependentRequired`;
`stability` no longer requires `previous_window_end`; the provenance and diff `window`
objects no longer require `start` and `end` and bind them to each other. **Conformance
tool:** the window arm judges instants only where they exist, the empty-extent clause
reports an envelope on an empty window, a `dependentRequired` finding names the missing
companions, and the `window-empty-equal-instants` control is replaced by
`window-no-envelope` and `window-half-envelope`. **Breaking by the letter** — a producer
that writes an envelope on an empty window, or a consumer that requires one, becomes
non-conformant — and editor-merged under [`GOVERNANCE.md`](GOVERNANCE.md) §2's 0.x rule.

- **§13.2 now quantifies over a VERDICT, not over field presence.** The old clause
required *"at least one of"* nine named fields to be **present**. Presence is
satisfied by every producer regardless of what it found — `template_deltas` is a
Expand Down
57 changes: 39 additions & 18 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ top-level fields:
|---|---|---|---|
| `metalog_version` | string | yes | Spec version this document conforms to. SemVer string (e.g. `"0.10.0"`). The `MetaLogDiff` (§13) carries the same axis as `diff_version` — see §13.1.1. |
| `producer` | object | yes | Identifies the producing implementation. See §2.1. |
| `window` | object | yes | The event-time envelope of the lines the window contains. See §2.2. |
| `window` | object | yes | The line count and, when any line has one, the event-time envelope of the lines the window contains. See §2.2. |
| `source` | object | yes | What was observed (service, host, fleet). See §2.3. |
| `canonicalization_version` | string | no | Opaque identifier for the canonicalization rules in effect. Gates `compose()`/diff comparability. See §2.4. |
| `retention_profile` | string | no | Opaque identifier for the retention parameters (top_k, reservoir, salience weights, diversity caps). Gates `compose()`/diff comparability. See §2.4. |
Expand Down Expand Up @@ -164,9 +164,9 @@ that as a coupling.

```jsonc
{
"start": "2026-04-24T10:00:00Z", // RFC 3339, UTC, required
"end": "2026-04-24T10:05:00Z", // RFC 3339, UTC, required
"duration_seconds": 300, // number, required, MUST equal end - start
"start": "2026-04-24T10:00:00Z", // RFC 3339, UTC; present iff some line has an event time
"end": "2026-04-24T10:05:00Z", // RFC 3339, UTC; present iff `start` is
"duration_seconds": 300, // integer; present iff `start` is, MUST equal end - start
"lines_observed": 184273 // integer, required, count of log lines that fed this MetaLog
}
```
Expand All @@ -192,11 +192,20 @@ its own — the interval its rule for deciding the extent used —

`start` **MUST** be less than or equal to `end`; the two are equal
when every line the window contains carries the same event time, a
window of one line included. For a window containing no line
(`lines_observed` 0), `start` **MUST** equal `end`, and both name the
time the producer attributes to the window.
`duration_seconds` **MUST** equal `end - start` rounded to the
nearest second. If a producer cannot count `lines_observed` exactly,
window of one line included. `duration_seconds` **MUST** equal
`end - start` rounded to the nearest second.

**A window with no event time has no envelope.** When the producer
attributes an event time to none of the lines the window contains —
a window containing no line (`lines_observed` 0) included — `start`,
`end` and `duration_seconds` **MUST** all be absent, and
`lines_observed` is the window's only member. The three are present
together or absent together. A producer **MUST NOT** write into
`start` or `end` an instant it did not attribute to a line of the
window: a fixed value, an epoch sentinel, or the time the window was
opened or closed is not an event time, and a reader cannot tell any
of them from one. A consumer **MUST** read an absent envelope as "no
line of this window has an event time", never as an instant. If a producer cannot count `lines_observed` exactly,
it **MUST** emit its best estimate and **SHOULD** emit an
`extensions.org.metalog.lines_observed_estimated: true` flag.

Expand Down Expand Up @@ -920,7 +929,7 @@ order.

```jsonc
{
"previous_window_end": "2026-04-24T10:00:00Z", // RFC 3339, required, the previous window's `window.end`
"previous_window_end": "2026-04-24T10:00:00Z", // RFC 3339, the previous window's `window.end`; absent when it has none
"kl_divergence": 0.043, // number ≥ 0, KL(current || previous) over template freqs
"js_divergence": 0.021, // number in [0, 1], symmetric Jensen-Shannon
"new_templates": 3, // integer, templates seen now but not in previous window
Expand All @@ -929,7 +938,11 @@ order.
}
```

A producer **MAY** include only a subset of these fields. The
A producer **MAY** include only a subset of these fields.
`previous_window_end` **MUST** be present when the previous window
carries a `window.end` and **MUST** be absent when it carries none
(§2.2): the block still compares the two template distributions,
which exist whether or not either window has an event time. The
`stability_score` is producer-defined; consumers **MUST NOT** assume
two producers compute it the same way and **SHOULD** prefer the
explicit divergences (`kl_divergence`, `js_divergence`) for
Expand Down Expand Up @@ -1081,17 +1094,22 @@ the file. Clause 2 is mechanically decidable in exactly one place. `window`
(§2.2) is the only required block whose definition states relations
**between** its members — `start` at or before `end`,
`duration_seconds` equal to their difference rounded to the nearest
second, both instants in UTC, and `start` equal to `end` when
`lines_observed` is 0. None of the four is expressible in JSON
second, both instants in UTC, and no envelope at all when
`lines_observed` is 0. The first three are not expressible in JSON
Schema at any draft (`format: date-time` constrains the grammar,
never the offset, and no keyword relates two siblings), and all four
follow from the document alone, so
never the offset, and no keyword compares two siblings' values); the
fourth could be written as a conditional schema but is decided beside
them, so that every relation of §2.2 is reported by one arm; and all
four follow from the document alone, so
[`conformance/metalog_validate.py`](conformance/metalog_validate.py)
decides them — together with the §7
`org.metalog.lines_observed_estimated` flag, which §2.2 admits only
as `true` and which lives inside an extension container the schema
does not type by design. The **types** of those members are clause
1's business and are not re-decided there. Every **other** required
1's business and are not re-decided there, and so is the one window
relation the schema does state: that `start`, `end` and
`duration_seconds` are present together or absent together, which it
writes with `dependentRequired`. Every **other** required
field is checked only as far as the schema expresses it, and clause
3 is not mechanically decidable at all until a pinned
cross-implementation digest vector exists.
Expand Down Expand Up @@ -1320,6 +1338,9 @@ to a 1-hour MetaLog).
- `C.window.start = min(A.window.start, B.window.start)`
- `C.window.end = max(A.window.end, B.window.end)`
- `C.window.duration_seconds = C.window.end - C.window.start` (the event-time envelope of both inputs' lines, **not** the sum of their durations)
- an input with no envelope (§2.2) contributes no bound: `C` carries
the other input's envelope unchanged, and no envelope when neither
input has one
- `C.window.lines_observed = A.window.lines_observed + B.window.lines_observed`
- `C.source` is `A.source` if equal to `B.source`, otherwise the
most-specific common prefix (e.g. same `fleet`, drop differing
Expand Down Expand Up @@ -1443,7 +1464,7 @@ When emitted, `provenance` is an array of objects:
```jsonc
[
{
"window": { "start": "...", "end": "..." },
"window": { "start": "...", "end": "..." }, // the input's bounds; {} when it has none
"source": { "service": "checkout-api", "host": "checkout-3" },
"lines_observed": 91204,
"document_id": "sha256:..." // optional, content hash of the composed input
Expand Down Expand Up @@ -1477,7 +1498,7 @@ not directly comparable.
{
"diff_version": "0.10.0", // string, REQUIRED — the SPEC version, §13.1.1
"comparison_outcome": "changed", // string, REQUIRED — "changed" | "unchanged", §13.2
"current": { "window": { "start": "...", "end": "..." }, "document_id": "sha256:..." },
"current": { "window": { "start": "...", "end": "..." }, "document_id": "sha256:..." }, // window {} when that document has no envelope (§2.2)
"previous": { "window": { "start": "...", "end": "..." }, "document_id": "sha256:..." },
"kl_divergence": 0.043,
"js_divergence": 0.021,
Expand Down
5 changes: 3 additions & 2 deletions conformance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,8 +173,9 @@ have gone green while blind:
| `withheld-signal-is-a-witness` | §13.2.2 lets a `"changed"` diff name, in `withheld_signals`, a finding it computed and did not serialise. The fixture is `changed_without_witness` with **one array's content** changed — `withheld_signals` from `[]` to `["field_histogram_deltas"]` — and it is CONFORMANT while its twin is not. Judge `withheld_signals` by presence rather than by its declared vacuity and both twins flip, which turns the escape into a hole any producer can climb through. |
| `window-consistency-violation` | §2.2's relations between `window`'s members are unreachable from the schema, so every document here is **schema-valid** and wrong between its members: `duration_seconds` 300 over a 600-second window, `start` after `end`, an `end` with a legal non-UTC offset (also the withholding witness: with `end` unreadable, ordering and duration are not decided), and the §7 estimated-lines flag at `false`. |
| `window-composed-envelope` | A composed document whose two raw children are **not contiguous** — 240 seconds of covered span inside a 600-second window. `duration_seconds` is `end - start`, 600; an arm that computed it from the children, or that treated the gap between shards as a defect, reds this conformant document. It is also the one positive witness that the estimated-lines flag admits `true`. |
| `window-empty-extent` | §2.2: a window with **no line** (`lines_observed` 0) carries `start` equal to `end`. The fixture is one schema-valid document with `lines_observed` 0 whose `start` precedes its `end` by 600 seconds and whose `duration_seconds` is that 600, so ordering and duration both hold and this clause is the only thing that can catch it. Before the clause existed the validator read it CONFORMANT, exit 0. |
| `window-empty-equal-instants` | The same clause's positive witness: two empty windows with `start` equal to `end`, the second spelling `end` as `…T10:00:00.000+00:00` against a `start` of `…T10:00:00Z` — one instant, two legal RFC 3339 spellings. A clause that compared the strings rather than the instants reds this conformant document; measured, that mutation reds exactly this fixture. |
| `window-empty-extent` | §2.2 (v0.10.0): a window with **no line** (`lines_observed` 0) has no event time, so it carries **no envelope**. The fixture is one schema-valid document with `lines_observed` 0 that carries one anyway, whose `start` precedes its `end` by 600 seconds and whose `duration_seconds` is that 600, so ordering and duration both hold and this clause is the only thing that can catch it. Until v0.10.0 the clause read `start` equal to `end`. |
| `window-no-envelope` | The absent envelope's positive witness: two schema-valid documents whose `window` carries `lines_observed` alone — one empty, one of three lines with no event time, the latter also carrying a `stability` block with no `previous_window_end` (§5). Both are CONFORMANT; a schema that still required the three envelope members, or an arm that judged instants on a window without any, reds here. |
| `window-half-envelope` | §2.2's all-or-none is the schema's, written with `dependentRequired`: a `window` carrying `start` alone is a clause 1 finding naming both missing companions, `duration_seconds` and `end`, and the window arm withholds its verdict. Without the keyword the document would reach the arm and be reported as an unreadable `end` — a shape defect sent to the reader as a time defect. |

Measured on the committed tool, 2026-08-19: **seven** independent mutations each
red the self-test — section-as-one-document · offending-member computation blinded ·
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"metalog_version":"0.10.0","producer":{"name":"example-producer","version":"1.0.0"},"window":{"start":"2026-04-24T10:00:00Z","lines_observed":3},"source":{"service":"checkout"},"stats":{"unique_templates":1,"top_k_size":64,"tail_count":0,"tail_unique":0,"top_k":[{"template_id":"h:0123456789abcdef0123456789abcdef","count":3,"frequency":1.0}]}}
34 changes: 29 additions & 5 deletions conformance/fixtures/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@
"path": "invalid/window_empty_extent.metalog.jsonl",
"kind": "metalog",
"control": "window-empty-extent",
"why": "SPEC §2.2: a window containing no line (`lines_observed` 0) carries `start` equal to `end`, and §8 counts that equality among the window relations this validator decides. One SCHEMA-VALID document with `lines_observed` 0 whose `start` precedes its `end` by 600 seconds, and whose `duration_seconds` is that 600, so the ordering and duration clauses both hold and the empty-extent clause is the only thing that can catch it. Before this clause existed the validator read this document CONFORMANT, exit 0.",
"why": "SPEC §2.2 (v0.10.0): a window containing no line (`lines_observed` 0) has no line with an event time, so it carries NO envelope, and §8 counts that among the window relations this validator decides. One SCHEMA-VALID document with `lines_observed` 0 that carries an envelope anyway, whose `start` precedes its `end` by 600 seconds and whose `duration_seconds` is that 600, so the ordering and duration clauses both hold and the empty-extent clause is the only thing that can catch it. Before v0.10.0 the clause read `start` equal to `end`, and this document was caught for its unequal instants; it is still caught, now for carrying instants at all.",
"expect": {
"exit": 1,
"documents": 1,
Expand All @@ -96,18 +96,18 @@
"window_violations": [
[
"empty-extent",
"lines_observed is 0 and start 2026-04-24T10:00:00Z is not end 2026-04-24T10:10:00Z",
"lines_observed is 0 and the window carries an envelope [2026-04-24T10:00:00Z .. 2026-04-24T10:10:00Z]",
1
]
],
"window_unjudged": 0
}
},
{
"path": "valid/window_empty_equal_instants.metalog.jsonl",
"path": "valid/window_no_envelope.metalog.jsonl",
"kind": "metalog",
"control": "window-empty-equal-instants",
"why": "The empty-extent clause's positive witness: two SCHEMA-VALID documents with `lines_observed` 0 and `start` equal to `end`, which §2.2 admits and which MUST read CONFORMANT. The second spells `end` as `2026-04-24T10:00:00.000+00:00`, the same instant as its `start` `2026-04-24T10:00:00Z` in another legal RFC 3339 spelling, so a clause that compared the two strings rather than the two instants reds here. A clause tested only by its violation passes forever as `always red on an empty window`.",
"control": "window-no-envelope",
"why": "The absent envelope's positive witness (SPEC §2.2, v0.10.0): two SCHEMA-VALID documents whose `window` carries `lines_observed` alone — one with no line, one with three lines none of which has an event time — and the second also carries a `stability` block with no `previous_window_end` (§5). Both MUST read CONFORMANT. A schema that still required `start`, `end` and `duration_seconds`, or an arm that decided ordering or duration on a window that has no instants, reds here; a rule tested only by its violation passes forever as `always red on a window without event time`.",
"expect": {
"exit": 0,
"documents": 2,
Expand All @@ -117,6 +117,30 @@
"window_unjudged": 0
}
},
{
"path": "invalid/window_half_envelope.metalog.jsonl",
"kind": "metalog",
"control": "window-half-envelope",
"why": "SPEC §2.2 (v0.10.0): `start`, `end` and `duration_seconds` are present together or absent together, and the schema states it with `dependentRequired`, so this is a clause 1 finding and the window arm withholds its verdict. One document carrying `start` alone: both missing companions are named, one error each. Without the keyword the document would validate and reach the window arm, which would report a missing `end` as an unreadable instant — a shape defect sent to the reader as a time defect.",
"expect": {
"exit": 1,
"documents": 1,
"findings": [
[
"window",
"dependentRequired",
[
"duration_seconds",
"end"
],
2,
1
]
],
"undescribed": [],
"window_unjudged": 1
}
},
{
"path": "undescribed/open_containers.metalog.jsonl",
"kind": "metalog",
Expand Down

This file was deleted.

Loading
Loading