From 6707220e5e953af8897259edae0684c060237580 Mon Sep 17 00:00:00 2001 From: Benjamin De Vierno <54540257+bdevierno1@users.noreply.github.com> Date: Thu, 18 Jun 2026 21:18:32 -0400 Subject: [PATCH] Add SLI/SLO definitions and burn-rate alert rules for Tier-1 services MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - docs/slo-policy.md: service tier classification, SLI definitions, SLO targets (vecna-api 99.9%, vecna-agent 99.0%, vecna-frontend 99.9%), error-budget table, Green/Yellow/Red band thresholds, operational constraints, and SLO review triggers. - monitoring/alerts/slo-burn-rate.yml: Prometheus multi-window burn-rate rules for vecna-api and vecna-agent covering all three budget states (Red ≥14.4×/1h, Yellow ≥6×/6h, Green-elevated ≥3×/3d). Every alert links to the runbook. - docs/runbooks/slo-burn-rate-runbook.md: response runbook stub with alert reference table, diagnosis commands, and per-severity actions. Co-Authored-By: Vecna --- docs/runbooks/slo-burn-rate-runbook.md | 109 ++++++++++++++++ docs/slo-policy.md | 110 ++++++++++++++++ monitoring/alerts/slo-burn-rate.yml | 166 +++++++++++++++++++++++++ 3 files changed, 385 insertions(+) create mode 100644 docs/runbooks/slo-burn-rate-runbook.md create mode 100644 docs/slo-policy.md create mode 100644 monitoring/alerts/slo-burn-rate.yml diff --git a/docs/runbooks/slo-burn-rate-runbook.md b/docs/runbooks/slo-burn-rate-runbook.md new file mode 100644 index 0000000..7dabc91 --- /dev/null +++ b/docs/runbooks/slo-burn-rate-runbook.md @@ -0,0 +1,109 @@ +# SLO Burn-Rate Runbook + + + +Alert source: [`monitoring/alerts/slo-burn-rate.yml`](../../monitoring/alerts/slo-burn-rate.yml) +Policy: [`docs/slo-policy.md`](../slo-policy.md) + +--- + +## Alert Reference + +| Alert | Severity | Budget State | Burn Rate | Exhaustion Horizon | +|---|---|---|---|---| +| `VecnaAPIBurnRateRed` | critical | 🔴 Red | ≥ 14.4× | < 2 days | +| `VecnaAPIBurnRateYellow` | warning | 🟡 Yellow | ≥ 6× | ~5 days | +| `VecnaAPIBurnRateGreenElevated` | info | 🟢 Green (elevated) | ≥ 3× | ~9 days | +| `VecnaAgentBurnRateRed` | critical | 🔴 Red | ≥ 14.4× | < 2 days | +| `VecnaAgentBurnRateYellow` | warning | 🟡 Yellow | ≥ 6× | ~5 days | +| `VecnaAgentBurnRateGreenElevated` | info | 🟢 Green (elevated) | ≥ 3× | ~9 days | + +--- + +## Step 1 — Acknowledge (within 5 min for Red/Yellow) + +``` +# Check current error rate from the metrics endpoint +curl -sf https:///api/metrics | grep vecna_operations_total + +# Check deep health (DB, model key, config) +curl -sf https:///api/health/deep | jq . +``` + +--- + +## Step 2 — Identify Root Cause + +### `vecna-api` alerts (operation success rate) + +```bash +# Count terminal operations in the last hour by status +sqlite3 /opt/vecna-ops/backend/vecna_ops.db \ + "SELECT status, count(*) FROM operations + WHERE created_at > datetime('now', '-1 hour') + GROUP BY status;" + +# Inspect recent failed operations for error patterns +sqlite3 /opt/vecna-ops/backend/vecna_ops.db \ + "SELECT id, error_message, created_at FROM operations + WHERE status IN ('failed','error') + ORDER BY created_at DESC LIMIT 20;" +``` + +Common causes: +- **LLM upstream error** — `error_message` contains rate-limit or auth text → check API key in `.env`, verify `LITELLM_MODEL_ID` prefix matches the key type. +- **Bad deploy** — check `git log -5` on host, compare deploy time to alert start time. +- **DB failure** — `"database"` key in `/api/health/deep` shows `"error"`. + +### `vecna-agent` alerts + +Same queries as above. Agent failures always surface as `status=failed|error` on `operations`. + +--- + +## Step 3 — Respond by Severity + +### 🔴 Red + +1. **Freeze all deploys** immediately. +2. If a deploy landed in the last 60 min → **roll back** ([rollback runbook](production-rollback-runbook.md)). +3. Page on-call: PagerDuty `vecna-oncall`. +4. Confirm SLI recovers — watch `vecna_operations_total` for ~15 min. +5. Open postmortem if Red lasted > 20 min or budget < 10 % remaining. + +### 🟡 Yellow + +1. Freeze non-critical deploys. On-call reviews any pending release. +2. Open a ticket to track root cause; assign to current sprint. +3. Monitor every 30 min until burn rate drops below 6×. + +### 🟢 Green Elevated + +1. Create a sprint ticket: "SLO: slow elevated burn on ``." +2. Investigate error patterns at next business-hours opportunity. +3. No immediate deploy freeze required. + +--- + +## Step 4 — Verify Recovery + +```bash +# SLI should recover within 15 min post-fix +curl -sf https:///api/metrics | grep vecna_operations_total + +# Confirm error rate back to baseline +# (completed / (completed + failed + error) ≥ 0.999 over last 30 min) +``` + +--- + +## Contacts + +| Role | Where | +|---|---| +| On-call engineer | PagerDuty: `vecna-oncall` | +| Incident Commander | Declared at SEV start — see [sev1-response.md](sev1-response.md) | + +--- + +*Last updated: 2026-06-18. Owner: on-call rotation.* diff --git a/docs/slo-policy.md b/docs/slo-policy.md new file mode 100644 index 0000000..a6c66d5 --- /dev/null +++ b/docs/slo-policy.md @@ -0,0 +1,110 @@ +# SLO & Error-Budget Policy — Vecna Operations Console + + + +Follows the org-wide [SLO & Error-Budget Policy](https://vecna.ai/docs/slo-policy). Alert +rules live in [`monitoring/alerts/slo-burn-rate.yml`](../monitoring/alerts/slo-burn-rate.yml). +Runbook: [`docs/runbooks/slo-burn-rate-runbook.md`](runbooks/slo-burn-rate-runbook.md). + +--- + +## 1. Service Tier Classification + +> Tier 0 = payments, auth (not present in this repo). +> The rollback runbook's "Tier-0 Services" section uses local terminology for +> "essential to this app" — the org-wide tier for all three services below is **Tier 1**. + +| Service | Tier | Process | Why Tier 1 | +|---|---|---|---| +| `vecna-api` | **1** | `uvicorn app.main:app` | Core REST + WebSocket API; all client traffic | +| `vecna-agent` | **1** | asyncio tasks inside `vecna-api` | Primary product function; failures = user-visible errors | +| `vecna-frontend` | **1** | Vite static bundle / CDN | UI surface; outage = zero usability | +| `vecna-desktop` | 2 | Electron process | Optional desktop wrapper; degraded ≠ product down | + +--- + +## 2. SLI Definitions + +### 2.1 `vecna-api` — Operation Success Rate + +**What:** fraction of terminal-state operations that completed successfully. + +``` +SLI = rate(vecna_operations_total{status="completed"}[window]) + / rate(vecna_operations_total{status!~"pending|running"}[window]) +``` + +Data source: `GET /api/metrics` (exposed by `backend/app/main.py`). + +**Latency SLI (future):** p99 < 500 ms on `POST /api/operations`. Requires adding +`vecna_operation_duration_seconds` histogram to `app/services/runner.py`. Tracked in +follow-up work. + +### 2.2 `vecna-api` — Availability (HTTP) + +**What:** fraction of seconds `GET /health` returns HTTP 200. + +Data source: Blackbox Exporter probe against `/health`. + +### 2.3 `vecna-agent` — Operation Success Rate + +**What:** fraction of started operations that reach `status=completed`. + +Same metric source as 2.1; SLO target is lower to account for expected upstream LLM failures. + +### 2.4 `vecna-frontend` — Availability + +**What:** fraction of seconds `GET /` returns HTTP 200. + +Data source: Blackbox Exporter probe. + +--- + +## 3. SLO Targets and Error Budgets + +All windows are rolling 28-day. + +| Service | SLI | Target | Error Budget | +|---|---|---|---| +| `vecna-api` | Operation success rate | **99.9 %** | 43.2 min / 28 d | +| `vecna-api` | HTTP availability (`/health`) | **99.9 %** | 43.2 min / 28 d | +| `vecna-agent` | Operation success rate | **99.0 %** | 6.7 h / 28 d | +| `vecna-frontend` | HTTP availability | **99.9 %** | 43.2 min / 28 d | +| `vecna-desktop` | Process availability | **99.5 %** | 3.6 h / 28 d | + +--- + +## 4. Error-Budget States and Burn-Rate Alert Thresholds + +| State | Budget Remaining | `vecna-api` (99.9%) | `vecna-agent` (99.0%) | Response | +|---|---|---|---|---| +| 🟢 Green | > 75 % | Slow elevated burn: ≥ 3× over 3 d | ≥ 3× over 3 d | Ticket; fix in sprint | +| 🟡 Yellow | 25 – 75 % | Medium burn: ≥ 6× over 6 h | ≥ 6× over 6 h | Page during business hours | +| 🔴 Red | < 25 % | Fast burn: ≥ 14.4× over 1 h | ≥ 14.4× over 1 h | Page immediately | + +Burn rate = observed error rate ÷ SLO error rate (1 − target). + +--- + +## 5. Operational Constraints by State + +| State | Constraint | +|---|---| +| 🟢 Green | Normal deployments permitted. | +| 🟡 Yellow | Freeze non-critical deploys. On-call reviews each release before shipping. | +| 🔴 Red | **All deploys frozen** until budget recovers to Yellow. Rollback is default action. | + +See [`docs/runbooks/production-rollback-runbook.md`](runbooks/production-rollback-runbook.md) +for rollback procedures. + +--- + +## 6. SLO Review Triggers + +- Budget hits Red for any Tier-1 service → postmortem required within 48 h. +- Budget hits Red twice in one 28-day window → formal SLO target reassessment. +- New service in critical path → must be tiered and have SLOs before first production deploy. + +--- + +*Last updated: 2026-06-18. Owner: on-call rotation.* diff --git a/monitoring/alerts/slo-burn-rate.yml b/monitoring/alerts/slo-burn-rate.yml new file mode 100644 index 0000000..9261e48 --- /dev/null +++ b/monitoring/alerts/slo-burn-rate.yml @@ -0,0 +1,166 @@ +# Vecna Operations Console — SLO Burn-Rate Alert Rules +# Metric source: GET /api/metrics (backend/app/main.py) +# Policy: docs/slo-policy.md +# Runbook: docs/runbooks/slo-burn-rate-runbook.md +# +# Budget bands (28 d window): Green >75% | Yellow 25-75% | Red <25% +# Multi-window guard: long window (sensitivity) + short window (confirmation). + +groups: + + # vecna-api: SLO 99.9 % | error budget 0.001 | 43.2 min / 28 d + - name: vecna.slo.tier1.api + interval: 30s + rules: + + - record: vecna:api_error_rate:rate5m + expr: > + 1 - ( + rate(vecna_operations_total{status="completed"}[5m]) + / clamp_min(rate(vecna_operations_total{status!~"pending|running"}[5m]), 1e-6) + ) + + - record: vecna:api_error_rate:rate30m + expr: > + 1 - ( + rate(vecna_operations_total{status="completed"}[30m]) + / clamp_min(rate(vecna_operations_total{status!~"pending|running"}[30m]), 1e-6) + ) + + - record: vecna:api_error_rate:rate1h + expr: > + 1 - ( + rate(vecna_operations_total{status="completed"}[1h]) + / clamp_min(rate(vecna_operations_total{status!~"pending|running"}[1h]), 1e-6) + ) + + - record: vecna:api_error_rate:rate6h + expr: > + 1 - ( + rate(vecna_operations_total{status="completed"}[6h]) + / clamp_min(rate(vecna_operations_total{status!~"pending|running"}[6h]), 1e-6) + ) + + - record: vecna:api_error_rate:rate3d + expr: > + 1 - ( + rate(vecna_operations_total{status="completed"}[3d]) + / clamp_min(rate(vecna_operations_total{status!~"pending|running"}[3d]), 1e-6) + ) + + # RED — fast burn ≥ 14.4× — budget exhausts in < 2 days + - alert: VecnaAPIBurnRateRed + expr: > + vecna:api_error_rate:rate1h / 0.001 >= 14.4 + and vecna:api_error_rate:rate5m / 0.001 >= 14.4 + for: 2m + labels: + severity: critical + service: vecna-api + tier: "1" + budget_state: red + annotations: + summary: "vecna-api RED burn — error budget exhausted in < 2 days" + description: > + 1 h error rate is {{ printf "%.1f" (div (index $value 0) 0.001) }}× the SLO budget (0.1 %). + At this rate the 28-day error budget (43 min) exhausts within 2 days. + Freeze all deploys. Roll back the most recent change if unsure. + runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md" + + # YELLOW — medium burn ≥ 6× — exhausts in ~5 days + - alert: VecnaAPIBurnRateYellow + expr: > + vecna:api_error_rate:rate6h / 0.001 >= 6 + and vecna:api_error_rate:rate30m / 0.001 >= 6 + for: 15m + labels: + severity: warning + service: vecna-api + tier: "1" + budget_state: yellow + annotations: + summary: "vecna-api YELLOW burn — error budget draining (~5 days to exhaustion)" + description: > + 6 h error rate is {{ printf "%.1f" (div (index $value 0) 0.001) }}× the SLO budget. + At this rate the 28-day budget exhausts in ~5 days. + Page during business hours; freeze non-critical deploys. + runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md" + + # GREEN elevated — slow burn ≥ 3× — exhausts in ~9 days + - alert: VecnaAPIBurnRateGreenElevated + expr: > + vecna:api_error_rate:rate3d / 0.001 >= 3 + and vecna:api_error_rate:rate6h / 0.001 >= 3 + for: 1h + labels: + severity: info + service: vecna-api + tier: "1" + budget_state: green + annotations: + summary: "vecna-api GREEN elevated burn — sustained 3× SLO budget consumption" + description: > + 3-day error rate is {{ printf "%.1f" (div (index $value 0) 0.001) }}× the SLO budget. + Budget is safe today but will slide into Yellow in ~9 days if unchanged. + Create a ticket and fix within the sprint. + runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md" + + # vecna-agent: SLO 99.0 % | error budget 0.01 | 6.7 h / 28 d + # Recording rules from vecna.slo.tier1.api are reused (same metric/SLI). + - name: vecna.slo.tier1.agent + interval: 30s + rules: + + # RED — fast burn ≥ 14.4× + - alert: VecnaAgentBurnRateRed + expr: > + vecna:api_error_rate:rate1h / 0.01 >= 14.4 + and vecna:api_error_rate:rate5m / 0.01 >= 14.4 + for: 2m + labels: + severity: critical + service: vecna-agent + tier: "1" + budget_state: red + annotations: + summary: "vecna-agent RED burn — operation failure rate critical" + description: > + 1 h agent failure rate is {{ printf "%.1f" (div (index $value 0) 0.01) }}× the 99 % SLO budget. + Check LLM upstream errors (OPENROUTER / OPENAI) and backend/app/services/runner.py. + runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md" + + # YELLOW — medium burn ≥ 6× + - alert: VecnaAgentBurnRateYellow + expr: > + vecna:api_error_rate:rate6h / 0.01 >= 6 + and vecna:api_error_rate:rate30m / 0.01 >= 6 + for: 15m + labels: + severity: warning + service: vecna-agent + tier: "1" + budget_state: yellow + annotations: + summary: "vecna-agent YELLOW burn — elevated operation failure rate" + description: > + 6 h agent failure rate is {{ printf "%.1f" (div (index $value 0) 0.01) }}× the 99 % SLO budget. + Review recent model or tool changes. Check /api/health/deep for LLM key warnings. + runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md" + + # GREEN elevated — slow burn ≥ 3× + - alert: VecnaAgentBurnRateGreenElevated + expr: > + vecna:api_error_rate:rate3d / 0.01 >= 3 + and vecna:api_error_rate:rate6h / 0.01 >= 3 + for: 1h + labels: + severity: info + service: vecna-agent + tier: "1" + budget_state: green + annotations: + summary: "vecna-agent GREEN elevated burn — sustained 3× SLO budget consumption" + description: > + 3-day agent failure rate is {{ printf "%.1f" (div (index $value 0) 0.01) }}× the 99 % SLO budget. + Budget is stable; investigate error patterns at /api/operations for failed runs. + runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md"