feat(slo): name the burning dimension in the alert, and cut the message down - #13
Merged
Merged
Conversation
…ge down
The burn-rate alert's whole job is to say which of sixty-nine chains is burning, and it
did not say. It said "Any chain_id label on this instance names the chain that is
burning" — a description of where to find the answer instead of the answer, two thirds of
the way down a 90-word paragraph, in a notification that is usually read as a one-line
title on a phone.
`dimension_template` puts it in that first line instead, in both the summary and the
description, immediately after the SLI:
dimension_template: '{{ if $labels.chain_id }} on chain {{ $labels.chain_id }}{{ end }}'
It is a Go template because only the caller knows which labels its SLIs aggregate by, and
guarded because the same rule covers SLIs that carry no such label at all. The old
`dimension_hint` stays, now empty by default and appended after the objective, for a
consumer with several aggregation labels that genuinely needs a sentence to explain them.
The rest is subtraction. The message repeated the tier arithmetic three ways, explained
what a burn rate is, and spelled out "this is a budget alert, not an outage alert: the
SLI may look fine on a 5-minute chart and still be on track to miss the objective" — true
and worth one clause, not three lines. It also printed the raw float, so an alert
announced that it had spent 13.888888888888888% of its budget; `printf "%.1f"` makes that
13.9%.
Summary now leads with what and where and ends with the limit, so it is legible
truncated:
Prod - rpc-chain-availability on chain eip155:5000 burned 13.9% of its 30-day error
budget in 1h (tier limit 2%)
chris13524
force-pushed
the
feat/slo-alert-message
branch
from
September 17, 2026 20:50
56a5d73 to
96f4645
Compare
geekbrother
approved these changes
Sep 18, 2026
chris13524
added a commit
that referenced
this pull request
Oct 1, 2026
`labelled(sli, expr)` now applies an SLI's optional `decorate(expr)` outermost, after all the burn arithmetic. Optional and guarded, so an SLI that does not define it takes exactly the previous code path. The placement is the point, not an implementation detail. PromQL binary ops match on the full label set, so a label added to `bad` but not to the budget built from `events` yields no series at all — a silently empty rule rather than an error. Wrapping at the end also applies it once per rule instead of once per window, which matters because these expressions are already near the size where Amazon Managed Grafana's rule-group writes get flaky. Complements #13 rather than overlapping it. That PR gave the caller `dimension_template` to PRINT a dimension label; this gives an SLI a way to CREATE one. blockchain-api uses the pair together: its per-chain SLIs aggregate `by (chain_id)`, `decorate` maps each id to the human chain name from chain_config.json, and `dimension_template` renders "Avalanche C-Chain (eip155:43114)" instead of the CAIP-2 id alone. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The burn-rate alert's job is to say which of sixty-nine chains is burning. It did not.
Before:
After:
What changed
dimension_template— a new opt that names the dimension inline in both fields,right after the SLI. A Go template, because only the caller knows what its SLIs aggregate
by, and guarded (
{{ if $labels.chain_id }}) because the same rule covers SLIs that carryno such label:
The old prose sentence could never have worked — a notification is read as a one-line
title first, and the answer was in the middle of a paragraph.
dimension_hintis kept but now defaults to empty and moves after the objective. Aconsumer with several aggregation labels may still want a sentence; nobody needs it by
default. No consumer breaks — pay-core passes its own and will keep rendering it.
printf "%.1f"on$value, so alerts stop reporting13.888888888888888%.Subtraction — the message stated the tier arithmetic three ways and spent three lines
on "this is a budget alert, not an outage alert". That idea is worth one clause. ~600
chars → ~480, with the important part first instead of last.
Testing
render_rules.jsonnetrenders and all three promtool suites pass.smoke.jsonnetupdatedto the new opt.
Note
The
printfis the one thing not verifiable offline — Grafana's$valueis documentedas a string in some contexts, and the raw float in the "before" above suggests it is a
float64 here. If it turns out to be a string, that one substitution renders
%!f(string=...)and the rest of the message is unaffected. Worth an eyeball on thefirst firing.