Skip to content

feat(slo): name the burning dimension in the alert, and cut the message down - #13

Merged
chris13524 merged 1 commit into
mainfrom
feat/slo-alert-message
Sep 18, 2026
Merged

chris13524 merged 1 commit into
mainfrom
feat/slo-alert-message

Conversation

@chris13524

Copy link
Copy Markdown
Member

The burn-rate alert's job is to say which of sixty-nine chains is burning. It did not.

Before:

Prod - SLO fast burn — SLO rpc-chain-availability (objective: 99% of RPC requests served, per chain) spent 13.888888888888888% of its 30-day error budget in the last 1h, and is still over-spending as of the last 5m. The limit for this tier is 2% per 1h (14.4x the sustainable pace); at that rate the whole month of budget is gone in ~2.1 days. Any chain_id label on this instance names the chain that is burning. This is a budget alert, not an outage alert: the SLI may look fine on a 5-minute chart and still be on track to miss the objective. Check the SLIs row of the dashboard with the range set to 1h, then that chain's row under Chain RPC Router.

After:

Prod - rpc-chain-availability on chain eip155:5000 burned 13.9% of its 30-day error budget in 1h (tier limit 2%)

rpc-chain-availability on chain eip155:5000 burned 13.9% of its 30-day error budget in the last 1h, and is still burning as of the last 5m. Tier limit is 2% per 1h — 14.4x the sustainable pace, which exhausts the month in ~2.1 days. Objective: 99% of RPC requests served, per chain. Budget alert, not an outage: a 5-minute chart can look healthy while the month is still being missed. Dashboard: SLIs row at 1h, then that chain's row under Chain RPC Router.

What changed

dimension_template — a new opt that names the dimension inline in both fields,
right after the SLI. A Go template, because only the caller knows what its SLIs aggregate
by, and guarded ({{ if $labels.chain_id }}) because the same rule covers SLIs that carry
no such label:

dimension_template: '{{ if $labels.chain_id }} on chain {{ $labels.chain_id }}{{ end }}',

The old prose sentence could never have worked — a notification is read as a one-line
title first, and the answer was in the middle of a paragraph.

dimension_hint is kept but now defaults to empty and moves after the objective. A
consumer with several aggregation labels may still want a sentence; nobody needs it by
default. No consumer breaks — pay-core passes its own and will keep rendering it.

printf "%.1f" on $value, so alerts stop reporting 13.888888888888888%.

Subtraction — the message stated the tier arithmetic three ways and spent three lines
on "this is a budget alert, not an outage alert". That idea is worth one clause. ~600
chars → ~480, with the important part first instead of last.

Testing

render_rules.jsonnet renders and all three promtool suites pass. smoke.jsonnet updated
to the new opt.

Note

The printf is the one thing not verifiable offline — Grafana's $value is documented
as a string in some contexts, and the raw float in the "before" above suggests it is a
float64 here. If it turns out to be a string, that one substitution renders
%!f(string=...) and the rest of the message is unaffected. Worth an eyeball on the
first firing.

…ge down

The burn-rate alert's whole job is to say which of sixty-nine chains is burning, and it
did not say. It said "Any chain_id label on this instance names the chain that is
burning" — a description of where to find the answer instead of the answer, two thirds of
the way down a 90-word paragraph, in a notification that is usually read as a one-line
title on a phone.

`dimension_template` puts it in that first line instead, in both the summary and the
description, immediately after the SLI:

    dimension_template: '{{ if $labels.chain_id }} on chain {{ $labels.chain_id }}{{ end }}'

It is a Go template because only the caller knows which labels its SLIs aggregate by, and
guarded because the same rule covers SLIs that carry no such label at all. The old
`dimension_hint` stays, now empty by default and appended after the objective, for a
consumer with several aggregation labels that genuinely needs a sentence to explain them.

The rest is subtraction. The message repeated the tier arithmetic three ways, explained
what a burn rate is, and spelled out "this is a budget alert, not an outage alert: the
SLI may look fine on a 5-minute chart and still be on track to miss the objective" — true
and worth one clause, not three lines. It also printed the raw float, so an alert
announced that it had spent 13.888888888888888% of its budget; `printf "%.1f"` makes that
13.9%.

Summary now leads with what and where and ends with the limit, so it is legible
truncated:

    Prod - rpc-chain-availability on chain eip155:5000 burned 13.9% of its 30-day error
    budget in 1h (tier limit 2%)
@chris13524
chris13524 force-pushed the feat/slo-alert-message branch from 56a5d73 to 96f4645 Compare September 17, 2026 20:50
@chris13524
chris13524 merged commit e8621e6 into main Sep 18, 2026
1 check passed
@chris13524
chris13524 deleted the feat/slo-alert-message branch September 18, 2026 16:20
chris13524 added a commit that referenced this pull request Oct 1, 2026
`labelled(sli, expr)` now applies an SLI's optional `decorate(expr)` outermost,
after all the burn arithmetic. Optional and guarded, so an SLI that does not
define it takes exactly the previous code path.

The placement is the point, not an implementation detail. PromQL binary ops match
on the full label set, so a label added to `bad` but not to the budget built from
`events` yields no series at all — a silently empty rule rather than an error.
Wrapping at the end also applies it once per rule instead of once per window,
which matters because these expressions are already near the size where Amazon
Managed Grafana's rule-group writes get flaky.

Complements #13 rather than overlapping it. That PR gave the caller
`dimension_template` to PRINT a dimension label; this gives an SLI a way to
CREATE one. blockchain-api uses the pair together: its per-chain SLIs aggregate
`by (chain_id)`, `decorate` maps each id to the human chain name from
chain_config.json, and `dimension_template` renders "Avalanche C-Chain
(eip155:43114)" instead of the CAIP-2 id alone.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants