Skip to content

feat(slos): SLI, error budget and burn rates - #2295

Merged
Makisuo merged 13 commits into
feat/slo-latency-ladderfrom
feat/slo-compute
Nov 6, 2026
Merged

Makisuo merged 13 commits into
feat/slo-latency-ladderfrom
feat/slo-compute

Conversation

@Makisuo

@Makisuo Makisuo commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Stack: #2293 model, #2294 latency ladder, #2295 compute, #2296 burn alerts, #2297 MCP, #2298 UI

Part 3 of 6: computing the SLI, the error budget left and the burn rates.

Queries

  • sloTotalsQuery reads one SLO window from three tiers: raw spans for the partial minutes at each end, the minutely rollup for the partial hours, and the hourly rollup for everything in between. The boundaries come from rollup-splice.
  • sloBucketsQuery buckets the same reads by hour (for the budget chart) or by minute (for the burn windows).

Service and API

  • SlosService.status returns the SLI and budget left over the window, plus the burn rate of the 5m, 30m, 1h and 6h windows. All four come from one 6-hour minute-grain read.
  • history returns hourly budget points over the window.
  • New endpoints: GET /v2/slos/:id/status and GET /v2/slos/:id/history.

Verification

  • A ClickHouse e2e parity test seeds spans on every seam (partial minute, partial hour, hour boundary, one second outside each end). The seeds include sampled spans and environment and service decoys. Totals and buckets match a flat scan across 6 window alignments, 3 SLIs and 3 environment filters: 152 cases.
  • I broke the splice twice on purpose to confirm the test catches it: one change failed 150 of 152 cases, the other 126.
  • Unit tests cover the math (budget, burn rate, burn windows, history), and route tests cover status and history.

Benchmark. Run with bun run bench:queries, suite apps/api/scripts/bench-suites/slos.ts, on ClickHouse 26.2 with 10 runs and 2 threads. The seed was one service at about 1000 spans/min plus 20 others: 30 days of minutely rows, 60 days of hourly rows, and 2 days of raw spans.

case p50 ms p95 ms read rows read bytes
30d totals, availability 6.4 8.0 19,604 422 KiB
30d totals, latency 500 ms 5.7 7.3 19,604 422 KiB
6h burn windows, minute buckets 4.6 6.0 25,036 426 KiB
60d history, hourly 5.6 11.4 39,145 705 KiB
baseline: flat scan of raw spans 9.8 16.8 2,880,000 41.2 MiB

The baseline row only scans the 2 days of raw spans that were seeded. At the real 30-day raw retention it would read about 43M rows, roughly 2,200 times what the spliced read touches.

🤖 Generated with Claude Code

sloTotalsQuery reads the SLO window from raw partial minutes, the minutely
rollup's partial hours and the hourly interior; sloBucketsQuery buckets by
hour (budget chart) or minute (burn windows). SlosService.status returns the
SLI, the budget left and the 5m/30m/1h/6h burn rates; history returns hourly
budget points over the window. GET /v2/slos/:id/status and /history.
A latency SLI's rollup total is EstimatedLadderSpanCount, so buckets written
before migration 0058 drop out instead of reading as all slow.
…O reads

The parity test seeds spans on every splice seam of six windows (aligned and not,
plus a sub-hour one) and compares sloTotalsQuery and sloBucketsQuery against flat
scans of service_overview_spans for availability and latency SLIs, with and
without environment filters. A pre-ladder case shows legacy rollup rows drop out
of a latency total and still count for availability. Runs in the CI rollup step.

bench-suites/slos.ts covers 30-day totals, the 6h minute burn read, the 60-day
hourly history and a flat 30-day scan as the baseline the splice replaces.
@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 15 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ec679024-a37d-4b9a-9500-986135ebb524
Commits

Reviewing files that changed from the base of the PR and between 79897a7 and 12cbb1d.

Files selected for processing: 18

  • .github/workflows/ci.yml
  • apps/api/scripts/bench-suites/slos.ts
  • apps/api/src/routes/v2/slos.http.test.ts
  • apps/api/src/routes/v2/slos.http.ts
  • apps/api/src/routes/v2/v2-test-support.ts
  • packages/backend/src/services/slos/SlosService.ts
  • packages/backend/src/services/slos/slo-reads.ts
  • packages/backend/src/services/warehouse/slo-parity.clickhouse.e2e.test.ts
  • packages/domain/src/http/v2/openapi.test.ts
  • packages/domain/src/http/v2/slos.ts
  • packages/domain/src/slo.test.ts
  • packages/domain/src/slo.ts
  • packages/query-engine/src/__sql_baseline__/catalog.sql
  • packages/query-engine/src/benchmark/builders.ts
  • packages/query-engine/src/benchmark/catalog.test.ts
  • packages/query-engine/src/ch/index.ts
  • packages/query-engine/src/ch/queries/slos.test.ts
  • packages/query-engine/src/ch/queries/slos.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@maple-review-bot

This comment has been minimized.

@Makisuo

Makisuo commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

F1 fixed: SLO_BURN_LOOKBACK_MINUTES is now Math.max(...SLO_BURN_WINDOWS_MINUTES), so a longer tier window widens the burn read with it. Merged up the stack.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

GET /v2/slos/statuses groups SLOs by SLI and window: one totals read per
(SLI, window) and one burn read per SLI, each broken down by service and
environment and summed per SLO. The SLOs list no longer costs two warehouse
reads per SLO. The parity e2e pins the grouped reads against flat scans.
@maple-review-bot

This comment has been minimized.

@maple-review-bot

maple-review-bot Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge. This pass read only the parity e2e change; its ground truth matches the query semantics and the earlier budgetHistory slot duplication (F2) is unchanged and still open.
1 note · tests covered · risk medium · 1/1 new units observable

Adds the spliced SLO reads (window totals, hour/minute buckets, burn windows, budget history), the status/history service paths and endpoints, and a raw-vs-rollup parity e2e plus benchmark suite. The CodeQL escaping fix in the parity test is correct; nothing new blocks merge.

  • sloTotalsQuery / sloBucketsQuery tile a window from raw spans, minutely and hourly rollups
  • readStatuses reads every SLO's totals and burn windows in batched, org-scoped queries
  • New GET /v2/slos/:id/status and /history endpoints
  • Parity e2e compares the splice against a flat scan on 6 window alignments

Still open from earlier reviews

12cbb1d · What was checked, coverage and telemetry details are on the Maple / review check. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo merged commit bf34463 into main Nov 6, 2026
43 checks passed
@Makisuo
Makisuo deleted the feat/slo-compute branch November 6, 2026 11:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants