feat(slo): reusable multiwindow multi-burn-rate SLO alerting

[chris13524]

Extracted from pay-core and blockchain-api, which had each grown a copy of the same
burn-rate implementation. This is the shared home so they stop diverging — I created that
divergence last week by copying the file, so this closes it.

What's here

File
slo/burn_rate.libsonnet	tier table, budget sizing, the PromQL conditions
slo/rule.libsonnet	tiers + SLIs → Grafana Unified Alerting rule groups
slo/panels.libsonnet	burn-rate chart + error-budget bar gauge
slo/tests/	promtool suites over the shipping expressions
instant_query.libsonnet	make an alert rule's query instant rather than range
.github/workflows/ci.yml	this repo's first CI

Why so little is parameterised

I diffed the two copies before extracting. Ignoring comments and whitespace, they
differed in two things: the label carrying the priority (priority vs
og_priority) and two sentences of annotation prose. So those are the only opts; the
rest was already identical. Every knob is a hidden field, so a consumer retunes with
burnRate + { tiers:: [...] } instead of forking.

Two bugs the extraction surfaced

Writing fixtures that didn't share the consumers' incidental metric shapes broke the
code immediately — which is the argument for a library having its own tests.

1. events() depended on a label-set accident. Both copies selected the good and bad
   counters with a single __name__ matcher. increase() drops __name__, so the two
   collapse to the same label set and Prometheus rejects the vector — unless some other
   label distinguishes them. In blockchain-api one metric happens to carry
   provider_kind. Remove it and every SLO rule goes to exec_err_state together, from
   an unrelated metrics refactor. Fixed with the tag-and-union idiom max_of already
   uses; the fixtures deliberately give good/bad identical label sets so it cannot
   come back. (Fixed in blockchain-api separately.)

2. sum(good) + sum(bad) — the form I tried first — yields nothing when either
   side has no series, which is exactly a dimension in total outage.

Testing

CI installs go-jsonnet (matching Terraform's alxrem/jsonnet provider, which embeds it)
and promtool, renders rules and both panels from fixtures, then evaluates 3 suites
(~4s). The suites test mechanics — window arithmetic, budget sizing, the noise floor,
expression splicing, the sparse-denominator hazard, panel/rule agreement. Every case is
mutation-checked.

Not a breaking change: consumers pin a submodule SHA, so nothing moves until they
bump it. Follow-ups will migrate blockchain-api and pay-core onto this and delete
their local copies.

🤖 Generated with Claude Code