Add SLI/SLO definitions and burn-rate alerts for Tier-1 services#9
Draft
bdevierno1 wants to merge 1 commit into
Draft
Add SLI/SLO definitions and burn-rate alerts for Tier-1 services#9bdevierno1 wants to merge 1 commit into
bdevierno1 wants to merge 1 commit into
Conversation
- docs/slo-policy.md: service tier classification, SLI definitions, SLO targets (vecna-api 99.9%, vecna-agent 99.0%, vecna-frontend 99.9%), error-budget table, Green/Yellow/Red band thresholds, operational constraints, and SLO review triggers. - monitoring/alerts/slo-burn-rate.yml: Prometheus multi-window burn-rate rules for vecna-api and vecna-agent covering all three budget states (Red ≥14.4×/1h, Yellow ≥6×/6h, Green-elevated ≥3×/3d). Every alert links to the runbook. - docs/runbooks/slo-burn-rate-runbook.md: response runbook stub with alert reference table, diagnosis commands, and per-severity actions. Co-Authored-By: Vecna <noreply@vecna.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/slo-policy.md— In-repo SLO policy: service tier classification (all three services are Tier 1 per the org policy; no Tier-0 payments/auth exists in this app), SLI definitions sourced from the existing/api/metricsPrometheus endpoint (vecna_operations_total), SLO targets (vecna-api 99.9 %, vecna-agent 99.0 %, vecna-frontend 99.9 %), error-budget table, Green/Yellow/Red band thresholds, operational constraints, and review triggers.monitoring/alerts/slo-burn-rate.yml— Prometheus multi-window burn-rate rules forvecna-apiandvecna-agent. Six recording rules (5 m, 30 m, 1 h, 6 h, 3 d error-rate windows) plus 6 alert rules covering all three budget states: Red (≥ 14.4 × / 1 h + 5 m guard), Yellow (≥ 6 × / 6 h + 30 m guard), Green-elevated (≥ 3 × / 3 d + 6 h guard). Every alert links to the runbook.docs/runbooks/slo-burn-rate-runbook.md— Response runbook: alert reference table, diagnosis commands against SQLite and/api/health/deep, per-severity actions (Red = page + freeze deploys, Yellow = business-hours page + ticket, Green = sprint ticket).What is not in scope
vecna_operation_duration_secondshistogram): the backend does not currently emit duration histograms. Noted inslo-policy.md §2.1; tracked for follow-up.vecna-frontend+vecna-desktopalert rules: both require a Blackbox Exporter probe configured in the Prometheus scrape config, which is out of scope for this PR. SLO targets for both are defined in the policy doc.prometheus.yml: not present in this repo. Themonitoring/alerts/slo-burn-rate.ymlis a drop-inrule_files:entry or KubernetesPrometheusRuleCR.Test plan
grep -c '^+' <(git diff main...)— confirm ≤ 400 added lines (actual: 385).monitoring/alerts/slo-burn-rate.ymlinto Prometheus rule linter or runpromtool check rules monitoring/alerts/slo-burn-rate.yml— should pass with no errors.runbook_urlpointing todocs/runbooks/slo-burn-rate-runbook.md.docs/slo-policy.mdagainst org policy (file_ida86fb303) — targets and budget minutes match pinned memory.Rollback
Delete the three new files and the
monitoring/directory. No existing code changed.🤖 Generated with Vecna
Co-Authored-By: Vecna noreply@vecna.ai