Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,9 +213,6 @@ time the log talks about: chunks are selected on the store's clock, then every
entry is verified against its own logline stamp. Where a producer's two clocks
diverge — Apache logs a request's start time and writes the line when the
request completes — that selection leans on a widening of about a minute.
A store whose two clocks sit further apart says so: `timberfs set <store>
logline_lag=8h` widens the selection by what that store actually needs
instead of by the guess.
See [Two clocks](docs/deployment.md#two-clocks-and-when-they-diverge).

## Beyond the getting-started path
Expand Down Expand Up @@ -660,10 +657,6 @@ my-gc-extractor | timberfs tally --fold --width 60s | timberfs append --into ...
`man timberfs`, **tally**, is the reference; the schema is
`docs/tally-extractor.schema.json`.

⚠ Declare `logline_lag` on the tally store. Its lines are numbers about a minute
that closed some minutes ago, so its two clocks sit far apart, and a
logline-time window over it otherwise selects no chunk and answers nothing.

## Replicating to another timberfs (`frames-send`)

OTLP above ships **entries** to anything that speaks the protocol. When the far
Expand Down
20 changes: 10 additions & 10 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -1277,8 +1277,8 @@ tally store, because outliving the log is the entire point. A `DECLARE` that
has drifted from what is on disk is reported, never rewritten over an
operator's `timberfs set`.

⚠ `logline_lag` is derived, not typed, and `class!=tally` is folded into the
selection so a provisioning cannot end up measuring its own output.
⚠ `class!=tally` is folded into the selection, so a provisioning cannot end up
measuring its own output.

Try one against a real file before deploying it:

Expand All @@ -1300,27 +1300,27 @@ their own, one per source store:
timberfs create /var/log/timberfs/apache-access-tally/apache-access-tally.log \
--index --retain 730d \
--set class=tally --set service=apache-access --set host="$(hostname -s)" \
--set derived_op=tally --set logline_lag=1h
--set derived_op=tally

timberfs query --records apache-access --from '13:00' \
| timberfs tally --extractor /usr/lib/timberfs/tally.extractors.d \
| timberfs append --into /var/log/timberfs/apache-access-tally/apache-access-tally.log
```

Three of those declarations are load-bearing:
⚠ Shown for the mechanics; prefer `--provision` for a store you will keep.
`append` stamps arrival, as it does for any pipe, so this store's chunks carry
the moment the numbers were computed rather than the minutes they are about,
and `--from`/`--to` over it will not find them. A tally store the provisioning
writes carries the buckets, and needs nothing declared to be queried by them.

Two of those declarations are load-bearing:

- **`class=tally`** is how a reader tells the numbers from the log. It matters
the other way round too: a tally store inherits the source's labels, so
`[service=~apache-.*]` now matches *both*, and a follower shipping apache logs
onward would start shipping tally lines. Narrow such a selection with
`class!=tally` — an absent key reads as the empty string, so that already
excludes every store on disk today.
- **`logline_lag`** is how far a line's own stamp may sit from the moment it was
written: the bucket width plus `grace` for a live tally, and however old the
data is for a backfill. Chunk selection is widened by it in place of the one-minute guess,
and without it a logline-time window over the buckets selects no chunk at all
and answers nothing — which reads exactly like a quiet minute. `timberfs info`
reports the declared value, because that failure is otherwise silent.
- **`retain`** is the whole point of materialising: the tally store keeps its
numbers long after the log they came from has been head-dropped. Size it in
years where the log is sized in weeks.
Expand Down
6 changes: 1 addition & 5 deletions docs/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,9 +214,6 @@ object — the label on the timber. Plain enough to read by eye; changed with
"timestamp_regex": "^(...)", // content: exotic line-timestamp format, declared once
"timestamp_format": "%m/%d/%Y %H:%M:%S", // (import flags persist these; inherits)
"timestamp_utc": true, // zoneless line stamps are UTC, not local time
"logline_lag": "8h", // how far a line's stamp may sit from its
// WRITE time — widens chunk selection in
// place of the one-minute guess
"name": "apache-error", // what it is CALLED, where the path is opaque
"derived_from": "41d0…", // lineage: source store's id
"derived_op": "export", // …and how: export (copy), rotate (move) or
Expand Down Expand Up @@ -272,8 +269,7 @@ head-drops, travels on rename, and ships inside `.timber` bundles.

**Chunk selection** is the write-time index's job and is deliberately
coarse: every chunk whose write-time window overlaps the requested range
(widened by the store's `logline_lag`, or about a minute where it declares
none, to catch buffered stragglers) is read in full.
(widened by about a minute to catch buffered stragglers) is read in full.
Chunk windows are bounded by `--flush-age` (default 5 s) for slow writers
and by `--chunk-size` (default 256 KiB) for fast ones, so that is the slop
the index alone would leave at the edges.
Expand Down
72 changes: 25 additions & 47 deletions docs/plans/tally.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,44 +249,26 @@ still resolve a window before answering it. Bounded by series × buckets, not by
entries. A **`--follow` of a tally store delivers unresolved lines and must say
so**, exactly as a live-edge entry carries no chunk number.

## A tally store has two clocks too, and they are FAR apart

The bucket stamp is on the declared axis; the store's own chunks are stamped
when the line was appended. So `axis: "write"` asks *when was this computed*
and `axis: "logline"` asks *which minute is this about* — and a revision
written an hour late is findable on the first.

⚠ **That gap breaks the read path, and building it is what showed how.** Chunk
selection runs on the write clock and is widened by a guess of a minute before
each entry is verified against its own stamp — which assumes the two clocks are
close. A tally store's are close only by accident: its lines are numbers about
a minute that closed `GRACE` ago, a revision's are older still, and a
backfill's are about last month. Measured on the first tally store built here:
a logline-time query for the minute the numbers describe read **0 of 1 chunks**
and answered nothing, which is indistinguishable from an empty minute.

The fix is a declaration rather than a bigger guess: **`logline_lag` in the
`.bark`** says how far a line's own stamp may sit from the moment it was
written, and chunk selection is widened by that instead of by `WIDEN_MS`.
Declaring `logline_lag=8h` on that same store made the same query read the
chunk and answer exactly. It is general, not tally's: the roadmap's
zone-map entry describes the same failure for an arrival-stamped Apache store
whose lines carry request-START times, and today that leans on the same guess.
Widening both ways only ever costs I/O — the per-entry verification keeps the
output exact — and the per-chunk logline range the zone-map sidecar would add
makes the declaration unnecessary rather than wrong.

A tally writer knows its own lag exactly (`grace`, plus the width of the bucket
being closed), so it should declare it on the store it creates.

⚠ **A BACKFILL's lag is unbounded** — its lines are about whenever the source
data is from — so a store that has been backfilled must declare a lag wide
enough to cover it, and a wide lag means chunk selection prunes nothing and
every logline-time query becomes a full scan of the tape. That is the honest
cost of a declaration standing in for an index, and it is the argument for the
zone-map sidecar: a per-chunk logline range answers the same question exactly,
and would make the declaration unnecessary rather than merely generous.
`info` reports the declared lag, because its failure mode is silent.
## A tally store's chunks are stamped with the buckets

A chunk's window is the min/max of the bucket stamps in it, not the moment the
lines were folded. So both of a tally store's clocks are the minutes its lines
are about, and `--from`/`--to` selects its chunks exactly — a backfill of last
month lands in chunks stamped last month, where selection prunes.

⚠ The alternative was measured and is why this is written down: stamping the
moment of computation put the two axes a bucket and a grace apart, and a query
for the minute the numbers describe read **0 of 1 chunks** and answered
nothing — indistinguishable from an empty minute. A backfill made it
unbounded.

The same rule the receive intakes follow: they stamp the sender's event time,
so a received store has no arrival axis of its own either.

⚠ **A tally piped through `append` is stamped on arrival**, as any pipe is, so
a logline-time window over such a store finds nothing. That is `append`'s
meaning and not tally's — a store you will query by the minutes it describes
is written by a provisioning.

⚠ **`AXIS=write` buckets at CHUNK granularity**, because the only arrival stamp
an entry carries is its chunk's write window. On a busy log a chunk is a second
Expand Down Expand Up @@ -488,10 +470,6 @@ Five rules fall out:
permissive than "one tally follower per store" and exactly as safe — and it
is checkable at load, where overlapping input selections are not decidable in
general.
* ⚠ **`logline_lag` is DERIVED, not typed.** It is the `width + grace` of the
applied extractors, and the provisioning knows both. Making an operator write
it invites precisely the failure this note records twice. `DECLARE` may
override it, which is what a backfill needs.
* **Provisioning CONVERGES and does not cascade.** A source store appearing
gets its tally store on the next tick (`--check` to declare and say what
resolved, as `file-intake` does). A source store being DELETED does not take
Expand Down Expand Up @@ -1037,8 +1015,8 @@ loss, recorded exactly — the same rule retention already follows.
* **The follower half**: fanning out per source store, the watermark rule
(`Roller::safe_offset` is already the answer — the oldest byte any held
bucket still depends on, so a restart re-derives identical lines), creating
the tally store with its labels, lineage and `logline_lag`, and writing the
`!gap` marker from the registry's GAP.
the tally store with its labels and lineage, and writing the `!gap` marker
from the registry's GAP.
* **The `!gap` marker** — the registry reports a GAP when retention dropped
chunks a follower had not read, and nothing writes it into the tally store
yet. Until it does, a hole in the numbers and a quiet period look alike.
Expand Down Expand Up @@ -1066,15 +1044,15 @@ loss, recorded exactly — the same rule retention already follows.

## What must change elsewhere when this ships

Done with the first slice: `timberfs.1` gains **tally** and `logline_lag`, the
Done with the first slice: `timberfs.1` gains **tally**, the
completions gain the verb, and `packaging/extractors/` holds the shipped
documents with their fixtures under `tests/extractors/`.

Done with the extractor half: `timberfs.1`'s **tally** section, the README, the
deployment guide, the published schema, and the shipped extractors with their
fixtures. Still owed: a `timberfs-tally-extractor(5)` for the document; `use-cases.md`'s "No aggregation, no
dashboards, no alerting" (two of three survive); `concepts.md` gains **tally**,
**observation**, **bucket**, **revision**, **logline lag**; `design.md` gains
the line format and the `logline_lag` manifest key; a `timberfs-tally(5)` and a
**observation** and **bucket**; `design.md` gains the line format; a
`timberfs-tally(5)` and a
`tally.d` section in `deployment.md`; and `timberfs-query-document(5)` gains
`series`, `step` and the `samples` kind.
51 changes: 20 additions & 31 deletions packaging/timberfs.1
Original file line number Diff line number Diff line change
Expand Up @@ -293,10 +293,9 @@ neither. Known settings are parse-checked
.RI ( "an absolute directory, SUPERSEDED by the follower registry \(em see"
.BR FOLLOWERS ));
.BR timestamp_regex ,
.BR timestamp_format ,
.B timestamp_utc
.B timestamp_format
and
.B logline_lag
.B timestamp_utc
are parse-checked too (see THE TWO CLOCKS); anything else is free-form
provenance
.RB ( host =...),
Expand Down Expand Up @@ -1610,16 +1609,12 @@ somebody ran
.B timberfs set
and meant it.
.PP
.B logline_lag
is DERIVED from the applied extractors (their width plus their grace) and
declared on every tally store, because a logline window narrower than the
distance between its two clocks selects no chunk and answers nothing.
So is
.BR wal=true ,
because losing a minute's numbers silently is worse than the fsync it
costs on a store that takes a handful of lines a minute. Naming either in
.B wal=true
is declared on every tally store, because losing a minute's numbers
silently is worse than the fsync it costs on a store that takes a handful
of lines a minute. Naming it in
.B DECLARE
overrides it, which is what a backfill needs.
overrides that.
.PP
.B FOLLOW_FROM
is
Expand Down Expand Up @@ -1689,12 +1684,13 @@ the
that would have been needed to keep the worst of them \(em which is the
number to act on.
.PP
⚠ A tally store's two clocks sit far apart BY CONSTRUCTION \(em its
lines are numbers about a minute that closed some minutes ago \(em so
declare
.B logline_lag
on it (see THE TWO CLOCKS) or a logline-time window will select no chunk
and answer nothing.
⚠ A tally store's chunks are stamped with the BUCKETS, so both its clocks
are the minutes its lines are about and
.BR \-\-from / \-\-to
selects them exactly. A tally piped through
.B append
instead is stamped on arrival, as any pipe is, and a logline-time window
over that store finds nothing.
.SS list
.TP
.BI \-\-select " EXPR"
Expand Down Expand Up @@ -3692,19 +3688,12 @@ Do not backfill historical data through
.B append
(its index gets today's write times): import the file instead.
.PP
The minute of widening is a GUESS, and a store that knows better says so:
.B logline_lag
declares how far a line's own stamp may sit from the moment it was
written, and chunk selection is widened by that instead. It exists
because the guess has a class of store it cannot cover \(em one whose
producer stamps a line long before writing it. A TALLY store is the
clearest case: its lines are numbers about a minute that closed some
minutes earlier (its width, plus the sealing grace), and a backfilled
one's are about last month. Without the
declaration a logline-time window over such a store selects no chunk at
all and answers nothing, which reads exactly like an empty minute.
Widening both ways only ever costs I/O \(em the per-entry verification
keeps the output exactly inside the window asked for.
The minute of widening is a GUESS, and it only ever costs I/O \(em the
per-entry verification keeps the output exactly inside the window asked
for. It covers a producer that writes a line slightly after the stamp it
carries; a store whose two clocks sit further apart than that is written
by an intake that stamps the line's own time, so its axes coincide and
there is nothing to bridge.
.SH TIME FORMATS
.I TIME
arguments accept RFC 3339
Expand Down
43 changes: 0 additions & 43 deletions src/bark.rs
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,6 @@ pub const NOT_PROVENANCE: &[&str] = &[
"timestamp_regex",
"timestamp_format",
"timestamp_utc",
"logline_lag",
"command",
"pattern",
// Which tape this pair holds — an ADDRESS, not provenance, and the
Expand Down Expand Up @@ -345,22 +344,6 @@ pub fn origin_id(map: &Map<String, Value>) -> Option<String> {
/// exactly as declared — flattening a dotted key like `service.name` is a
/// consumer's concern (Loki requires it, timberfs does not), and doing it
/// here would lose the key the operator actually wrote.
/// How far a line's own stamp may sit from the moment it was written,
/// as the store declares it.
///
/// Chunk selection is on the WRITE clock and a logline window is
/// verified per entry, so the two are bridged by widening the selection
/// — a guess of a minute where nothing says otherwise. A store whose
/// producer stamps a line long before it writes it can say so instead,
/// and the commonest such store is a TALLY: its lines are numbers about
/// a minute that closed some minutes earlier, and a backfill's are about
/// last month. Widening both ways only ever costs I/O; the per-entry
/// verification keeps the answer exact.
pub fn logline_lag_ms(map: Option<&Map<String, Value>>) -> Option<u64> {
let v = map?.get("logline_lag")?.as_str()?;
crate::append::parse_duration_ms(v).ok()
}

pub fn provenance(map: &Map<String, Value>) -> Map<String, Value> {
map.iter()
.filter(|(k, _)| !NOT_PROVENANCE.contains(&k.as_str()))
Expand Down Expand Up @@ -388,7 +371,6 @@ const NON_INHERITED: &[&str] = &[
"retain",
"retain_size",
"retain_unconsumed",
"logline_lag",
];

/// Window bounds are operation facts, recorded as RFC3339 UTC.
Expand Down Expand Up @@ -902,10 +884,6 @@ pub fn declare(
crate::append::parse_size_bytes(&v)?;
Value::String(v)
}
"logline_lag" => {
crate::append::parse_duration_ms(&v)?;
Value::String(v)
}
_ if BOOLEAN_KEYS.contains(&k) => declared_value(k, &v)?,
"timestamp_regex" => {
let re = regex::Regex::new(&v)
Expand Down Expand Up @@ -964,26 +942,6 @@ mod tests {
.collect()
}

#[test]
fn a_declared_logline_lag_is_read_and_a_bad_one_is_not_guessed_at() {
// Absent and unreadable must reach the caller the same way — as
// "nothing declared" — so the guess stands rather than a widening
// of zero, which would silently answer a window with nothing.
assert_eq!(
logline_lag_ms(Some(&map(&[("logline_lag", Value::String("8h".into()))]))),
Some(8 * 3_600_000)
);
assert_eq!(logline_lag_ms(Some(&map(&[]))), None);
assert_eq!(
logline_lag_ms(Some(&map(&[("logline_lag", Value::String("soon".into()))]))),
None
);
assert_eq!(
logline_lag_ms(Some(&map(&[("logline_lag", Value::Bool(true))]))),
None
);
}

#[test]
fn provenance_is_where_the_entries_came_from_not_what_the_store_is() {
let m = map(&[
Expand All @@ -998,7 +956,6 @@ mod tests {
("retain_unconsumed", Value::Bool(true)),
("cursors", Value::String("/var/lib/timberfs".into())),
("timestamp_utc", Value::Bool(true)),
("logline_lag", Value::String("1h".into())),
("wal", Value::Bool(true)),
// One hop's bookkeeping, not provenance: under fan-in these
// name only ONE of the origins.
Expand Down
Loading
Loading