Skip to content

rfc: a window's extent is the producer's to decide; window.start and window.end are the earliest and latest event time of the lines it contains - #14

Merged
coderoast-dev merged 2 commits into
mainfrom
spec/window-extent
Sep 30, 2026
Merged

coderoast-dev merged 2 commits into
mainfrom
spec/window-extent

Conversation

@coderoast-dev

Copy link
Copy Markdown
Collaborator

Change class: BREAKING under GOVERNANCE.md §2. MINOR bump during 0.x; MAJOR stays 0, and both schema files keep their v0 names.

This pull request is the RFC. GOVERNANCE.md §2 names an RFC issue; the editor asked for this one as a pull request, so that section 2 below is the actual diff rather than a transcription of it. There is no separate issue. The four sections §2 requires follow.

The 14-day comment window runs from this pull request's opening: 2026-09-29 to 2026-10-13. The editor merges after it closes. The change is written into the unreleased 0.10.0 entry of CHANGELOG.md; if 0.10.0 is cut before the window closes, the entry moves to the next MINOR at merge.

Still to be added to this pull request before merge: the conformance validator's arm for the one new relation between window members (a window with no line has start equal to end), with a fixture that must fail it. §8 in this diff already counts that relation among the ones the validator decides.


1. The problem, with a concrete example

SPEC.md never says how a producer decides where a window starts and ends, and it should not. Several of its sentences assume one way of deciding anyway: a window is a time interval fixed in advance, closed online as lines arrive, and each document is emitted when its interval ends. Section 4 lists those sentences.

A producer is free to decide a window's extent in other ways, and those sentences rule out some good ones:

  • a producer that reads past a candidate boundary before fixing it;
  • a producer that assigns lines to a window only after seeing what followed them;
  • a producer whose extent follows the structure of the log rather than the clock.

None of these decides where a window ends on the line it has just read. A reader who knows a log well decides the same way: by looking ahead, then scrolling back.

§2.2 also leaves start and end undefined. It requires start <= end and duration_seconds = end - start, rounded to the nearest second, but it does not say what the two times are. Consider a producer that collects lines for 25 seconds, whose last line carries an event time one second before that interval ends. It may emit duration_seconds: 25 (its own interval) or 24 (its lines), and a consumer cannot tell which one it received. §12.1 composes documents by taking min(start) and max(end), so documents from two such producers compose into an interval that means neither.

2. The proposed change, as a diff against SPEC.md

The full diff is this pull request's. The hunks that carry the proposal are these.

Glossary:

-- **Window** — A contiguous time interval over which a single MetaLog
-  is computed.
+- **Window** — The log lines a single MetaLog is computed over. How a
+  producer decides which lines a window contains is
+  implementation-defined (§2.2).
+- **Event time** — The time a producer attributes to a log line: the
+  timestamp the line carries, when it carries one the producer can
+  read. How a producer attributes a time to a line that carries none
+  is implementation-defined.

§2.2, after the window block:

-`start` **MUST** be strictly less than or equal to `end`
-(equality is permitted for empty / heartbeat documents).
+**How a producer decides a window's extent is outside this
+specification.** Which lines a window contains, and how and when the
+producer decides it, are implementation-defined, and this
+specification assumes no producer behaviour for that decision. In
+particular it assumes neither a fixed interval, nor a decision taken
+on any single line, nor an online or causal decision: a producer
+**MAY** fix a window's extent only after reading lines beyond it, and
+**MAY** assign lines to a window retrospectively. A document that
+carries a raw re-derivation coordinate still describes exactly the
+lines whose event time falls within that coordinate's `bounds`
+(§15.3).
+
+`start` and `end` are the earliest and the latest event time among
+the lines the window contains: the window's event-time envelope. They
+say nothing about when or how the window's extent was decided, nor
+about when the document was emitted. A producer that has bounds of
+its own — the interval its rule for deciding the extent used —
+**MAY** publish them in an extension (§7).
+
+`start` **MUST** be less than or equal to `end`; the two are equal
+when every line the window contains carries the same event time, a
+window of one line included. For a window containing no line
+(`lines_observed` 0), `start` **MUST** equal `end`, and both name the
+time the producer attributes to the window.
 `duration_seconds` **MUST** equal `end - start` rounded to the
 nearest second.

Every other sentence that assumed a closure model changes in the same diff; section 4 lists each one with its old and new wording.

3. Alternatives considered

  • Pin the producer's own bounds instead. This assumes every producer has bounds to report, decided in advance. A producer that fixes a window's extent retrospectively has none that mean anything to a consumer.
  • Define them as the event times of the first and the last line read. When lines reach a producer out of event-time order, the first line read is not the earliest, so start <= end could fail. §12.1's min/max also does not compose "first and last line read": it does compose envelopes, because the envelope of a union is the min of the starts and the max of the ends.
  • Declare the reading in an optional member (window.bounds: "observed" | "declared"). This keeps two meanings conforming, moves the ambiguity to every consumer, and leaves the composition of mixed documents undefined.
  • An editorial note only ("consumers MUST NOT assume either reading"). This documents the problem and fixes nothing.
  • Leave the closure-assuming sentences as informative context. §16.6's regime note is normative (it carries a MUST), and the glossary defines a term the whole specification uses.

4. Migration impact

Producers.

  • A producer that writes its own interval into start/end becomes non-conformant. It moves that interval to an extension (§7). This is why the change is classed breaking.
  • A producer that emits a document for an interval in which it saw no line (a heartbeat) must now emit start equal to end for it. The old text permitted equality there but did not require it.
  • previous_window_end (§5) is the previous window's window.end. A producer that wrote an interval boundary there writes that window's end instead.
  • A producer whose reservoir and cube can cover different lines must not emit cube_coord (§16.6).
  • The reference implementation's deterministic batch regime already emits the envelope. Its live regime does not in every path: it can emit, as end, the instant at which its interval rule closed the window. Under GOVERNANCE.md §3 that is a defect in the implementation once this text is merged, and the implementation changes to conform.

Consumers. start and end now mean the same thing in every conformant document, so two producers' windows can be compared and composed. Nothing a consumer did before becomes wrong.

Schemas and examples. Neither schema changes: the types are the same. The examples keep their round-minute times, which remain valid: they read as the event times of lines that fell on those instants.

Conformance tooling. The validator decides §2.2's relations between window members. The empty-window equality is a new one. Its arm and a failing fixture are added to this pull request before merge (see the note at the top).

Every passage changed, old and new wording.

Where Today Proposed
Glossary, Window "A contiguous time interval over which a single MetaLog is computed." Section 2 above. "Contiguous" goes too: it is an assumption about extent.
Glossary, Event time (absent) Section 2 above. start and end are defined through it, and §15 already used the term undefined.
§1, window row "The time interval covered." "The event-time envelope of the lines the window contains."
§2.2 "(equality is permitted for empty / heartbeat documents)" Section 2 above. "Heartbeat" presumes a document emitted on a schedule.
§5, first paragraph "the previous document being the previous closed window" "the previous document being the window before it in the producer's order"
§5, previous_window_end "RFC 3339, required" "RFC 3339, required, the previous window's window.end"
§8, the window relations three relations between members four: the empty-window equality is added
§12.1 "C.window.duration_seconds = C.window.end - C.window.start (real time, not sum of inputs)" "… (the event-time envelope of both inputs' lines, not the sum of their durations)"
§16.6, regime precondition "valid only while the reservoir and the cube close on the same window boundary (the fixed-window regime — true today). Under adaptive window closure the two may close on different boundaries, so the cross would point at a cell of the wrong window; it MUST be re-designed before reuse there" "valid only while the reservoir and the cube are computed over the same lines, one window's content. A producer whose reservoir and cube can cover different lines MUST NOT emit cube_coord." The old text also carried a tag that nothing in the specification defines; it goes.
§16.9, twice "in batch over the closed window"; "the trigger reads only the closed window" "in batch over the window's final set of lines"; "the trigger reads only the window's lines" (editorial)
§16.10, TRIGGER "a pure function of the closed window's content" "a pure function of the window's content" (editorial)
RATIONALE.md §R5 "(typically the immediately preceding window of the same duration) and reports its boundary in previous_window_end" "(typically the immediately preceding window) and names it in previous_window_end by that window's window.end"

Checked and left unchanged, because none assumes how a window's extent is decided:

  • §15.3 requires a document with a raw coordinate to describe exactly the lines whose event time falls within its bounds. That constrains which lines such a window contains, not how or when the producer decides it, and §2.2 now points at it.
  • §4's n-gram bound ("an arriving … key", "arrived late") concerns the order lines are processed in within a window.
  • §4.2's "a streaming producer with no buffering" and §13.2.2's "streaming producer" are examples of a producer that cannot afford a block.
  • §3.5.2's "the §11 streaming envelope" is a memory regime.
  • §12's "1-minute MetaLogs merged to a 1-hour MetaLog" and §13.4's "the 5 minutes before and 5 minutes after a deploy" are informative uses that a producer with fixed intervals may meet.

🤖 Generated with Claude Code

coderoast-dev and others added 2 commits September 29, 2026 22:10
… of the window's lines, and how a producer decides a window's extent is its own

The two instants had no definition. A producer could write the bounds its own
rule used, or the times of the lines it read, and a consumer could not tell
which; §12.1 takes their min and max, so two documents could compose into an
interval that meant neither.

§2.2 now defines them as the earliest and the latest event time among the lines
the window contains, requires start equal to end for a window with no line, and
states that which lines a window contains, and how and when a producer decides
it, is implementation-defined: no fixed interval, no decision on a single line,
no online decision. The glossary gains Event time and redefines Window. The §1
table, §5 (previous_window_end is the previous window's window.end), §8 (four
window relations), §12.1, §16.6 (the regime note becomes a MUST NOT, and loses a
tag nothing defined), §16.9, §16.10 and RATIONALE §R5 drop the wording that
assumed a fixed interval closed as lines arrive.

Breaking under GOVERNANCE.md §2; this pull request is its RFC.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…indow with no line has `start` equal to `end`, judged on the instants

§8 now counts that equality among the window relations this validator
decides, and until this commit nothing decided it. A schema-valid document
with `lines_observed` 0 whose `start` precedes its `end` by 600 seconds, and
whose `duration_seconds` is 600, read CONFORMANT, exit 0: ordering and
duration both hold, so no existing clause could see it.

- metalog_validate.py: a fifth window clause, `empty-extent`, decided apart
  from ordering and on the parsed instants, never on the strings. RFC 3339
  spells one instant several ways, and §2.2 constrains the time, not its
  spelling. The printed scope names the clause, and also names what the
  window arm cannot decide now that §2.2 defines `start` and `end` as the
  event-time envelope of the window's lines: a relation to lines the
  document does not carry.
- invalid/window_empty_extent.metalog.jsonl (control `window-empty-extent`):
  the must-fail document above. Red first, against the validator before this
  commit: the self-test with this manifest entry reads
  `SELFTEST FAILED: 1/31 fixtures — ['invalid/window_empty_extent.metalog.jsonl']`,
  and the fixture validated alone reads `VERDICT: CONFORMANT`, exit 0.
- valid/window_empty_equal_instants.metalog.jsonl (control
  `window-empty-equal-instants`): the clause's positive witness, two empty
  windows with `start` equal to `end`, the second spelling `end` as
  `2026-04-24T10:00:00.000+00:00` against a `start` of
  `2026-04-24T10:00:00Z`. A clause that compares the strings reds it, and
  only it: `SELFTEST FAILED: 1/31 fixtures —
  ['valid/window_empty_equal_instants.metalog.jsonl']`.
- README: the counts (thirty-one fixtures, nineteen tagged, seventeen
  distinct controls), a table row for every control the self-test arms and
  the table did not list (five, three of them already armed before this
  commit), and the clause-2 limit restated with the fifth relation and the
  envelope it cannot reach.

After: `SELFTEST PASSED: 31/31 fixtures`, seventeen controls armed; the
shipped example reads CONFORMANT with `--expect-documents 1`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@coderoast-dev
coderoast-dev merged commit 3f193dd into main Sep 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant