bdevierno1 · bdevierno1 · Jun 19, 2026
diff --git a/docs/runbooks/slo-burn-rate-runbook.md b/docs/runbooks/slo-burn-rate-runbook.md
@@ -0,0 +1,109 @@
+# SLO Burn-Rate Runbook
+
+<!-- anchor: runbook-slo-burn-rate -->
+
+Alert source: [`monitoring/alerts/slo-burn-rate.yml`](../../monitoring/alerts/slo-burn-rate.yml)
+Policy: [`docs/slo-policy.md`](../slo-policy.md)
+
+---
+
+## Alert Reference
+
+| Alert | Severity | Budget State | Burn Rate | Exhaustion Horizon |
+|---|---|---|---|---|
+| `VecnaAPIBurnRateRed` | critical | 🔴 Red | ≥ 14.4× | < 2 days |
+| `VecnaAPIBurnRateYellow` | warning | 🟡 Yellow | ≥ 6× | ~5 days |
+| `VecnaAPIBurnRateGreenElevated` | info | 🟢 Green (elevated) | ≥ 3× | ~9 days |
+| `VecnaAgentBurnRateRed` | critical | 🔴 Red | ≥ 14.4× | < 2 days |
+| `VecnaAgentBurnRateYellow` | warning | 🟡 Yellow | ≥ 6× | ~5 days |
+| `VecnaAgentBurnRateGreenElevated` | info | 🟢 Green (elevated) | ≥ 3× | ~9 days |
+
+---
+
+## Step 1 — Acknowledge (within 5 min for Red/Yellow)
+
+```
+# Check current error rate from the metrics endpoint
+curl -sf https://<HOST>/api/metrics | grep vecna_operations_total
+
+# Check deep health (DB, model key, config)
+curl -sf https://<HOST>/api/health/deep | jq .
+```
+
+---
+
+## Step 2 — Identify Root Cause
+
+### `vecna-api` alerts (operation success rate)
+
+```bash
+# Count terminal operations in the last hour by status
+sqlite3 /opt/vecna-ops/backend/vecna_ops.db \
+  "SELECT status, count(*) FROM operations
+   WHERE created_at > datetime('now', '-1 hour')
+   GROUP BY status;"
+
+# Inspect recent failed operations for error patterns
+sqlite3 /opt/vecna-ops/backend/vecna_ops.db \
+  "SELECT id, error_message, created_at FROM operations
+   WHERE status IN ('failed','error')
+   ORDER BY created_at DESC LIMIT 20;"
+```
+
+Common causes:
+- **LLM upstream error** — `error_message` contains rate-limit or auth text → check API key in `.env`, verify `LITELLM_MODEL_ID` prefix matches the key type.
+- **Bad deploy** — check `git log -5` on host, compare deploy time to alert start time.
+- **DB failure** — `"database"` key in `/api/health/deep` shows `"error"`.
+
+### `vecna-agent` alerts
+
+Same queries as above. Agent failures always surface as `status=failed|error` on `operations`.
+
+---
+
+## Step 3 — Respond by Severity
+
+### 🔴 Red
+
+1. **Freeze all deploys** immediately.
+2. If a deploy landed in the last 60 min → **roll back** ([rollback runbook](production-rollback-runbook.md)).
+3. Page on-call: PagerDuty `vecna-oncall`.
+4. Confirm SLI recovers — watch `vecna_operations_total` for ~15 min.
+5. Open postmortem if Red lasted > 20 min or budget < 10 % remaining.
+
+### 🟡 Yellow
+
+1. Freeze non-critical deploys. On-call reviews any pending release.
+2. Open a ticket to track root cause; assign to current sprint.
+3. Monitor every 30 min until burn rate drops below 6×.
+
+### 🟢 Green Elevated
+
+1. Create a sprint ticket: "SLO: slow elevated burn on `<service>`."
+2. Investigate error patterns at next business-hours opportunity.
+3. No immediate deploy freeze required.
+
+---
+
+## Step 4 — Verify Recovery
+
+```bash
+# SLI should recover within 15 min post-fix
+curl -sf https://<HOST>/api/metrics | grep vecna_operations_total
+
+# Confirm error rate back to baseline
+# (completed / (completed + failed + error) ≥ 0.999 over last 30 min)
+```
+
+---
+
+## Contacts
+
+| Role | Where |
+|---|---|
+| On-call engineer | PagerDuty: `vecna-oncall` |
+| Incident Commander | Declared at SEV start — see [sev1-response.md](sev1-response.md) |
+
+---
+
+*Last updated: 2026-06-18. Owner: on-call rotation.*
diff --git a/docs/slo-policy.md b/docs/slo-policy.md
@@ -0,0 +1,110 @@
+# SLO & Error-Budget Policy — Vecna Operations Console
+
+<!-- anchor: slo-policy -->
+
+Follows the org-wide [SLO & Error-Budget Policy](https://vecna.ai/docs/slo-policy). Alert
+rules live in [`monitoring/alerts/slo-burn-rate.yml`](../monitoring/alerts/slo-burn-rate.yml).
+Runbook: [`docs/runbooks/slo-burn-rate-runbook.md`](runbooks/slo-burn-rate-runbook.md).
+
+---
+
+## 1. Service Tier Classification
+
+> Tier 0 = payments, auth (not present in this repo).
+> The rollback runbook's "Tier-0 Services" section uses local terminology for
+> "essential to this app" — the org-wide tier for all three services below is **Tier 1**.
+
+| Service | Tier | Process | Why Tier 1 |
+|---|---|---|---|
+| `vecna-api` | **1** | `uvicorn app.main:app` | Core REST + WebSocket API; all client traffic |
+| `vecna-agent` | **1** | asyncio tasks inside `vecna-api` | Primary product function; failures = user-visible errors |
+| `vecna-frontend` | **1** | Vite static bundle / CDN | UI surface; outage = zero usability |
+| `vecna-desktop` | 2 | Electron process | Optional desktop wrapper; degraded ≠ product down |
+
+---
+
+## 2. SLI Definitions
+
+### 2.1 `vecna-api` — Operation Success Rate
+
+**What:** fraction of terminal-state operations that completed successfully.
+
+```
+SLI = rate(vecna_operations_total{status="completed"}[window])
+      / rate(vecna_operations_total{status!~"pending|running"}[window])
+```
+
+Data source: `GET /api/metrics` (exposed by `backend/app/main.py`).
+
+**Latency SLI (future):** p99 < 500 ms on `POST /api/operations`. Requires adding
+`vecna_operation_duration_seconds` histogram to `app/services/runner.py`. Tracked in
+follow-up work.
+
+### 2.2 `vecna-api` — Availability (HTTP)
+
+**What:** fraction of seconds `GET /health` returns HTTP 200.
+
+Data source: Blackbox Exporter probe against `<HOST>/health`.
+
+### 2.3 `vecna-agent` — Operation Success Rate
+
+**What:** fraction of started operations that reach `status=completed`.
+
+Same metric source as 2.1; SLO target is lower to account for expected upstream LLM failures.
+
+### 2.4 `vecna-frontend` — Availability
+
+**What:** fraction of seconds `GET <FRONTEND_ORIGIN>/` returns HTTP 200.
+
+Data source: Blackbox Exporter probe.
+
+---
+
+## 3. SLO Targets and Error Budgets
+
+All windows are rolling 28-day.
+
+| Service | SLI | Target | Error Budget |
+|---|---|---|---|
+| `vecna-api` | Operation success rate | **99.9 %** | 43.2 min / 28 d |
+| `vecna-api` | HTTP availability (`/health`) | **99.9 %** | 43.2 min / 28 d |
+| `vecna-agent` | Operation success rate | **99.0 %** | 6.7 h / 28 d |
+| `vecna-frontend` | HTTP availability | **99.9 %** | 43.2 min / 28 d |
+| `vecna-desktop` | Process availability | **99.5 %** | 3.6 h / 28 d |
+
+---
+
+## 4. Error-Budget States and Burn-Rate Alert Thresholds
+
+| State | Budget Remaining | `vecna-api` (99.9%) | `vecna-agent` (99.0%) | Response |
+|---|---|---|---|---|
+| 🟢 Green | > 75 % | Slow elevated burn: ≥ 3× over 3 d | ≥ 3× over 3 d | Ticket; fix in sprint |
+| 🟡 Yellow | 25 – 75 % | Medium burn: ≥ 6× over 6 h | ≥ 6× over 6 h | Page during business hours |
+| 🔴 Red | < 25 % | Fast burn: ≥ 14.4× over 1 h | ≥ 14.4× over 1 h | Page immediately |
+
+Burn rate = observed error rate ÷ SLO error rate (1 − target).
+
+---
+
+## 5. Operational Constraints by State
+
+| State | Constraint |
+|---|---|
+| 🟢 Green | Normal deployments permitted. |
+| 🟡 Yellow | Freeze non-critical deploys. On-call reviews each release before shipping. |
+| 🔴 Red | **All deploys frozen** until budget recovers to Yellow. Rollback is default action. |
+
+See [`docs/runbooks/production-rollback-runbook.md`](runbooks/production-rollback-runbook.md)
+for rollback procedures.
+
+---
+
+## 6. SLO Review Triggers
+
+- Budget hits Red for any Tier-1 service → postmortem required within 48 h.
+- Budget hits Red twice in one 28-day window → formal SLO target reassessment.
+- New service in critical path → must be tiered and have SLOs before first production deploy.
+
+---
+
+*Last updated: 2026-06-18. Owner: on-call rotation.*
diff --git a/monitoring/alerts/slo-burn-rate.yml b/monitoring/alerts/slo-burn-rate.yml
@@ -0,0 +1,166 @@
+# Vecna Operations Console — SLO Burn-Rate Alert Rules
+# Metric source: GET /api/metrics  (backend/app/main.py)
+# Policy:  docs/slo-policy.md
+# Runbook: docs/runbooks/slo-burn-rate-runbook.md
+#
+# Budget bands (28 d window):  Green >75% | Yellow 25-75% | Red <25%
+# Multi-window guard: long window (sensitivity) + short window (confirmation).
+
+groups:
+
+  # vecna-api: SLO 99.9 % | error budget 0.001 | 43.2 min / 28 d
+  - name: vecna.slo.tier1.api
+    interval: 30s
+    rules:
+
+      - record: vecna:api_error_rate:rate5m
+        expr: >
+          1 - (
+            rate(vecna_operations_total{status="completed"}[5m])
+            / clamp_min(rate(vecna_operations_total{status!~"pending|running"}[5m]), 1e-6)
+          )
+
+      - record: vecna:api_error_rate:rate30m
+        expr: >
+          1 - (
+            rate(vecna_operations_total{status="completed"}[30m])
+            / clamp_min(rate(vecna_operations_total{status!~"pending|running"}[30m]), 1e-6)
+          )
+
+      - record: vecna:api_error_rate:rate1h
+        expr: >
+          1 - (
+            rate(vecna_operations_total{status="completed"}[1h])
+            / clamp_min(rate(vecna_operations_total{status!~"pending|running"}[1h]), 1e-6)
+          )
+
+      - record: vecna:api_error_rate:rate6h
+        expr: >
+          1 - (
+            rate(vecna_operations_total{status="completed"}[6h])
+            / clamp_min(rate(vecna_operations_total{status!~"pending|running"}[6h]), 1e-6)
+          )
+
+      - record: vecna:api_error_rate:rate3d
+        expr: >
+          1 - (
+            rate(vecna_operations_total{status="completed"}[3d])
+            / clamp_min(rate(vecna_operations_total{status!~"pending|running"}[3d]), 1e-6)
+          )
+
+      # RED — fast burn ≥ 14.4× — budget exhausts in < 2 days
+      - alert: VecnaAPIBurnRateRed
+        expr: >
+          vecna:api_error_rate:rate1h / 0.001 >= 14.4
+          and vecna:api_error_rate:rate5m / 0.001 >= 14.4
+        for: 2m
+        labels:
+          severity: critical
+          service: vecna-api
+          tier: "1"
+          budget_state: red
+        annotations:
+          summary: "vecna-api RED burn — error budget exhausted in < 2 days"
+          description: >
+            1 h error rate is {{ printf "%.1f" (div (index $value 0) 0.001) }}× the SLO budget (0.1 %).
+            At this rate the 28-day error budget (43 min) exhausts within 2 days.
+            Freeze all deploys.  Roll back the most recent change if unsure.
+          runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md"
+
+      # YELLOW — medium burn ≥ 6× — exhausts in ~5 days
+      - alert: VecnaAPIBurnRateYellow
+        expr: >
+          vecna:api_error_rate:rate6h / 0.001 >= 6
+          and vecna:api_error_rate:rate30m / 0.001 >= 6
+        for: 15m
+        labels:
+          severity: warning
+          service: vecna-api
+          tier: "1"
+          budget_state: yellow
+        annotations:
+          summary: "vecna-api YELLOW burn — error budget draining (~5 days to exhaustion)"
+          description: >
+            6 h error rate is {{ printf "%.1f" (div (index $value 0) 0.001) }}× the SLO budget.
+            At this rate the 28-day budget exhausts in ~5 days.
+            Page during business hours; freeze non-critical deploys.
+          runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md"
+
+      # GREEN elevated — slow burn ≥ 3× — exhausts in ~9 days
+      - alert: VecnaAPIBurnRateGreenElevated
+        expr: >
+          vecna:api_error_rate:rate3d / 0.001 >= 3
+          and vecna:api_error_rate:rate6h / 0.001 >= 3
+        for: 1h
+        labels:
+          severity: info
+          service: vecna-api
+          tier: "1"
+          budget_state: green
+        annotations:
+          summary: "vecna-api GREEN elevated burn — sustained 3× SLO budget consumption"
+          description: >
+            3-day error rate is {{ printf "%.1f" (div (index $value 0) 0.001) }}× the SLO budget.
+            Budget is safe today but will slide into Yellow in ~9 days if unchanged.
+            Create a ticket and fix within the sprint.
+          runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md"
+
+  # vecna-agent: SLO 99.0 % | error budget 0.01 | 6.7 h / 28 d
+  # Recording rules from vecna.slo.tier1.api are reused (same metric/SLI).
+  - name: vecna.slo.tier1.agent
+    interval: 30s
+    rules:
+
+      # RED — fast burn ≥ 14.4×
+      - alert: VecnaAgentBurnRateRed
+        expr: >
+          vecna:api_error_rate:rate1h / 0.01 >= 14.4
+          and vecna:api_error_rate:rate5m / 0.01 >= 14.4
+        for: 2m
+        labels:
+          severity: critical
+          service: vecna-agent
+          tier: "1"
+          budget_state: red
+        annotations:
+          summary: "vecna-agent RED burn — operation failure rate critical"
+          description: >
+            1 h agent failure rate is {{ printf "%.1f" (div (index $value 0) 0.01) }}× the 99 % SLO budget.
+            Check LLM upstream errors (OPENROUTER / OPENAI) and backend/app/services/runner.py.
+          runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md"
+
+      # YELLOW — medium burn ≥ 6×
+      - alert: VecnaAgentBurnRateYellow
+        expr: >
+          vecna:api_error_rate:rate6h / 0.01 >= 6
+          and vecna:api_error_rate:rate30m / 0.01 >= 6
+        for: 15m
+        labels:
+          severity: warning
+          service: vecna-agent
+          tier: "1"
+          budget_state: yellow
+        annotations:
+          summary: "vecna-agent YELLOW burn — elevated operation failure rate"
+          description: >
+            6 h agent failure rate is {{ printf "%.1f" (div (index $value 0) 0.01) }}× the 99 % SLO budget.
+            Review recent model or tool changes.  Check /api/health/deep for LLM key warnings.
+          runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md"
+
+      # GREEN elevated — slow burn ≥ 3×
+      - alert: VecnaAgentBurnRateGreenElevated
+        expr: >
+          vecna:api_error_rate:rate3d / 0.01 >= 3
+          and vecna:api_error_rate:rate6h / 0.01 >= 3
+        for: 1h
+        labels:
+          severity: info
+          service: vecna-agent
+          tier: "1"
+          budget_state: green
+        annotations:
+          summary: "vecna-agent GREEN elevated burn — sustained 3× SLO budget consumption"
+          description: >
+            3-day agent failure rate is {{ printf "%.1f" (div (index $value 0) 0.01) }}× the 99 % SLO budget.
+            Budget is stable; investigate error patterns at /api/operations for failed runs.
+          runbook_url: "https://github.com/bdevierno1/Vecna-Operations-Console/blob/main/docs/runbooks/slo-burn-rate-runbook.md"