A tally follower ran at 51 entries a second, and its numbers were wrong

[torstei]

Design note for the defect behind "tally is slow catching up". Docs only — no code, and the fix is a protocol amendment that wants settling before it is written.

What was measured

A real pipeline: a 500k-line apache store, tally --provision, feed -- timberfs tally --run, the shipped timberfs-apache-combined document.

the extractor alone (--try, 2 metrics)	310,000 entries/s
the pipeline at the hard-coded BATCH_ENTRIES of 512	51 entries/s

Throughput is batch_size ÷ 10 s and nothing else — 512 → 51/s, 4096 → 341/s, 65536 → 5750/s. Anything busier than ~51 entries/s falls permanently behind.

The numbers already written are wrong, in two ways, and only one leaves a marker. 1536 entries delivered and counted, resolved by timbergraph.read to 512, no !late/!cap/!drop. And once the stall has run long enough, every batch is displaced whole — reproduced with a 10 s/0 s window: 5632 entries spanning 28 s of log time placed across 50+ s of buckets, ten !late … count=512 markers.

Why

Two rules, each right on its own:

• the park — a store with anything unacknowledged is parked, which was measured at 99% of a core without it;
• the watermark — tally reports safe_offset, the oldest byte an open bucket depends on, so a restart re-derives identical lines.

Composed they deadlock: safe_offset is the first entry of the open bucket, so nothing is ever acknowledged, so the park withholds the entries that would advance event time and seal it. The consumer's own report stream shows it:

progress offset=0        <- first entry opens a bucket; the watermark pins to byte 0
                            ...nothing for 10 s...
progress offset=93135    <- one 512-entry batch, acknowledged at once
progress offset=186350   <- 10 s later, the next

Only QUIET_TICKS × IDLE_TICK = 10 s of silence breaks it, via finish() — and that force-seal is what fragments the buckets, while the idle() wall-clock advance beside it is what displaces them.

The fix in the note

The protocol has took it and dropped it; a consumer holding an open bucket is neither. One optional field on the report that already exists:

progress id=<id> offset=<safe> taken=<flow-control>

offset keeps its exact meaning and use (durable, persisted, the retention floor); taken is flow control only, in memory, resetting to offset on restart. Omitted means taken == offset, so no existing consumer changes and a shell consumer is still three printfs. The follower parks on and reads from taken, which preserves the anti-duplicate property the park exists for.

⚠ Deliberately not a raised --batch-size: at 65536 the undercount only shrinks to 5.5%, because the batch-edge buckets are still fragments, and a batch that must span width + grace of the busiest store's traffic is a number the operator would have to compute and get silently wrong.

The note also records three things wrong on their own merits (finish() is not a mid-stream flush; idle()'s advance needs a live-edge gate; the --max note leaks BATCH_ENTRIES at one line per batch), the four hot-loop costs that are next once the ceiling lifts — including a 4.16 s → 61.7 s cliff in Roller::add's series-count scan — and the open questions I did not settle.

For the record, since it was the first suspicion: the regexes are compiled once and were never a suspect.

Test gap

tally_provisioning_end_to_end feeds 3 entries — fewer than one batch — so it cannot fail on any of this. The assertion that catches all three: cross several batch boundaries and require the resolved tape to total the entries fed. An integration test of the feed→tally pair; nothing here needs a VM.

Operational note

Tally is being stopped in production. The tally stores already written are not salvageable — an undercount of unknown factor early, displaced buckets after — so the remedy is to drop them and re-derive from a reset position once this is fixed, which is also the best check on the fix. !late … count=512 on a tally store is this defect's signature and is greppable today.

🤖 Generated with Claude Code