diff --git a/docs/runbooks/postmortem-template.md b/docs/runbooks/postmortem-template.md new file mode 100644 index 0000000..3c46077 --- /dev/null +++ b/docs/runbooks/postmortem-template.md @@ -0,0 +1,138 @@ +# Blameless Postmortem Template + + + +> **Required for:** every SEV1 and SEV2 incident, within 3 business days of resolution. +> Referenced from [sev1-response.md](sev1-response.md) and [sev2-response.md](sev2-response.md). + +**Blame framing:** This postmortem is blameless. People made reasonable decisions given what they knew at the time. The goal is to surface systemic failures so the system improves — not to assign fault to individuals. + +--- + +## Incident Header + +| Field | Value | +|---|---| +| **Title** | _One sentence describing the incident_ | +| **Severity** | SEV1 / SEV2 | +| **Date** | YYYY-MM-DD | +| **Duration** | HH:MM (detection → all-clear) | +| **Incident Commander** | Name | +| **Ops Lead** | Name | +| **Scribe** | Name | +| **Services impacted** | `vecna-api`, `vecna-runner`, `vecna-db` (list those affected) | +| **Postmortem author** | Name | +| **Postmortem review date** | YYYY-MM-DD | + +--- + +## Impact Scope + +*Quantify. Vague impact statements prevent comparison across incidents.* + +| Dimension | Value | +|---|---| +| User-facing error rate at peak | _e.g., 42%_ | +| Operations failed or incomplete | _count_ | +| Billing sessions affected | _count_ | +| Customer reports received | _count_ | +| Revenue impact (if known) | _$ or "unknown"_ | +| Data loss / corruption | _yes/no; describe if yes_ | + +--- + +## Incident Timeline + +List events in UTC. Include detection, each major action, and recovery. Be specific — "restarted the service at 14:32" beats "service was restarted." + +| Time (UTC) | Event | +|---|---| +| HH:MM | Alert fired / first customer report | +| HH:MM | Alert acknowledged by on-call | +| HH:MM | SEV declared; IC assigned | +| HH:MM | _First investigation step_ | +| HH:MM | _Rollback initiated_ / _root cause identified_ | +| HH:MM | _Rollback confirmed good_ | +| HH:MM | Recovery criteria met — all-clear declared | +| HH:MM | Incident closed in PagerDuty | + +--- + +## Detection & Response Timeline + +Answer these explicitly: + +- **How was the incident detected?** _(alert, customer report, engineer noticed)_ +- **Time from first symptom to detection:** _MM min_ +- **Time from detection to SEV declaration:** _MM min_ +- **Time from detection to first mitigation action:** _MM min_ +- **Time from first mitigation to recovery:** _MM min_ +- **Were runbooks followed?** _(yes / no / partial — note gaps)_ + +--- + +## Root Cause Analysis + +*Describe what broke and why. Use the five-whys method if causation is non-obvious.* + +### What happened + +_Technical narrative. No blame. Describe the system behaviour._ + +### Contributing factors + +List each factor that made the incident worse, slower to detect, or harder to mitigate. These become candidates for action items. + +- [ ] Factor 1 — _e.g., no automated health-check alert existed for the runner_ +- [ ] Factor 2 +- [ ] Factor 3 + +### What went well + +*These reinforce good practices and should not be skipped.* + +- _e.g., Rollback procedure was followed immediately and reduced impact within 8 min_ +- + +--- + +## Action Items + +Every action item must have an **owner** and a **due date**. Unowned items do not get done. + +| # | Action | Owner | Due | Priority | +|---|---|---|---|---| +| 1 | _e.g., Add synthetic health-check alert for agent runner_ | @name | YYYY-MM-DD | High | +| 2 | | | | | +| 3 | | | | | + +**Priority guide:** +- **High** — prevents recurrence of the same incident class +- **Medium** — reduces detection or response time +- **Low** — nice-to-have resilience improvement + +--- + +## Anti-Blame Framing Checklist + +Before publishing, verify: + +- [ ] No individual is named as the cause of the incident +- [ ] All "who did X" statements focus on role, not person, or are removed +- [ ] Contributing factors are systemic (process, tooling, alert gaps) not personal +- [ ] Action items target the system, not individuals' judgment +- [ ] The document would be comfortable to share with the person who made the change that contributed to the incident + +--- + +## Sign-Off + +| Role | Name | Date | +|---|---|---| +| Postmortem author | | | +| Incident Commander | | | +| Engineering Lead | | | + +--- + +*Template version: 2026-06-18. Maintained in `docs/runbooks/`. Questions → #incidents Slack.* diff --git a/docs/runbooks/production-rollback-runbook.md b/docs/runbooks/production-rollback-runbook.md new file mode 100644 index 0000000..fd291c1 --- /dev/null +++ b/docs/runbooks/production-rollback-runbook.md @@ -0,0 +1,181 @@ +# Production Rollback Runbook + + + +**Golden rule: restore the user first; debug second.** No production change ships without a tested rollback path. When in doubt, roll back before you diagnose. + +--- + +## Decision: roll back vs. roll forward + +| Condition | Action | +|---|---| +| Bad deploy landed in the last 60 min | **Roll back** (this doc) | +| Impact growing over time | **Roll back** | +| Trivial, well-understood one-line fix, rollback slower than fix | Roll forward (second pair of eyes required) | +| DB migration is the culprit and is not reversible | **Page DB owner and on-call DBA — do NOT auto-rollback schema** | + +--- + +## Tier-0 Services + +| Service | Process | What breaks when it's down | +|---|---|---| +| `vecna-api` | `uvicorn app.main:app` | All HTTP + WebSocket traffic — 100% customer impact | +| `vecna-runner` | Async tasks inside `vecna-api` (see `services/runner.py`) | New operations hang; existing WS streams stop | +| `vecna-db` | SQLite file (`vecna_ops.db` by default) | All reads/writes fail; API returns 500 on any data path | + +--- + +## Rollback Procedures + +### Service 1 — `vecna-api` (FastAPI / uvicorn) + +**Symptoms:** `/health` returns non-200, new operations return 500, WebSocket connects then immediately closes. + +#### Process-managed deployment (systemd / supervisor) + +```bash +# 1. Identify the bad commit +git -C /opt/vecna-ops log --oneline -10 + +# 2. Pin to the last-known-good SHA (replace ) +GOOD_SHA= +git -C /opt/vecna-ops checkout "$GOOD_SHA" + +# 3. Reinstall dependencies if requirements.txt changed +cd /opt/vecna-ops/backend +pip install -r requirements.txt --quiet + +# 4. Restart the service +sudo systemctl restart vecna-api # or: supervisorctl restart vecna-api + +# 5. Verify (wait up to 30 s) +until curl -sf http://localhost:8000/health | grep -q '"status"'; do sleep 2; done +echo "API healthy" +``` + +#### Kubernetes deployment + +```bash +# Inspect revision history +kubectl rollout history deploy/vecna-api -n vecna + +# Roll back to the previous revision +kubectl rollout undo deploy/vecna-api -n vecna + +# Verify rollout completes +kubectl rollout status deploy/vecna-api -n vecna +# Expected output: "successfully rolled out" + +# Confirm pods are Running + Ready +kubectl get pods -n vecna -l app=vecna-api +``` + +**Post-rollback health check:** +```bash +curl -sf https:///api/health/deep | jq . +# All keys must be green; "database" must be "ok" +``` + +--- + +### Service 2 — `vecna-runner` (agent executor) + +The runner is an `asyncio` task spawned inside the `vecna-api` process. Rolling back `vecna-api` also rolls back the runner. No separate step required. + +**If individual operations are stuck (not the whole API):** + +```bash +# Check for stuck operations in DB +sqlite3 /opt/vecna-ops/backend/vecna_ops.db \ + "SELECT id, status, created_at FROM operations WHERE status='running' ORDER BY created_at DESC LIMIT 10;" + +# Force-fail a stuck operation (replace ) +sqlite3 /opt/vecna-ops/backend/vecna_ops.db \ + "UPDATE operations SET status='error', updated_at=datetime('now') WHERE id='' AND status='running';" + +# Restart API to flush in-flight asyncio tasks +sudo systemctl restart vecna-api +``` + +**Kubernetes:** +```bash +# Restart pods to flush in-flight tasks (rolling restart, zero downtime) +kubectl rollout restart deploy/vecna-api -n vecna +kubectl rollout status deploy/vecna-api -n vecna +``` + +--- + +### Service 3 — `vecna-db` (SQLite) + +#### Schema migration rollback + +Migrations in `backend/app/schema_migrate.py` are **additive only** (ADD COLUMN). SQLite does not support DROP COLUMN in older versions. + +**If a new column caused unexpected behaviour:** + +```bash +# Safe: new columns have DEFAULT values; old code ignores unknown columns. +# Roll back the application code (step above) — the extra column is inert. +# Do NOT attempt to drop the column unless you have confirmed SQLite ≥ 3.35.0 +sqlite3 --version +``` + +**If a column must be removed (non-reversible migration):** + +> ⚠️ **Stop. Do not auto-rollback the schema.** +> Page the DB owner and the on-call DBA immediately. +> The downstream table retains yesterday's state until the DBA clears it. + +```bash +# Page DB owner +# Page on-call DBA +# Document the stuck migration in the incident channel +# Do not run DROP COLUMN without explicit DBA sign-off +``` + +#### Database file recovery + +```bash +# Verify integrity before any recovery action +sqlite3 /opt/vecna-ops/backend/vecna_ops.db "PRAGMA integrity_check;" +# Expected: "ok" + +# Restore from most recent backup (adjust path) +BACKUP_DIR=/var/backups/vecna-db +LATEST=$(ls -t "$BACKUP_DIR"/vecna_ops_*.db | head -1) +echo "Restoring from $LATEST" +cp /opt/vecna-ops/backend/vecna_ops.db \ + /opt/vecna-ops/backend/vecna_ops.db.pre-restore-$(date +%s) +cp "$LATEST" /opt/vecna-ops/backend/vecna_ops.db +sudo systemctl restart vecna-api +``` + +--- + +## Post-Rollback Verification Checklist + +Run these in order; do not close the incident until all five pass: + +- [ ] `GET /api/health/deep` returns HTTP 200 with `"database": "ok"` and no warnings +- [ ] Error rate (4xx/5xx) at or below baseline for ≥ 15 min +- [ ] p99 latency at or below SLO for ≥ 15 min +- [ ] Any feature flag that was added by the bad deploy is disabled or removed +- [ ] Incident channel marked resolved; postmortem opened (SEV1/SEV2) — see [postmortem-template.md](postmortem-template.md) + +--- + +## Contacts + +| Role | Where to page | +|---|---| +| On-call engineer | PagerDuty: `vecna-oncall` | +| DB owner | PagerDuty: `vecna-db-owner` | +| On-call DBA | PagerDuty: `vecna-dba-oncall` | +| Incident Commander | Declared at SEV start; see [sev1-response.md](sev1-response.md) | + +--- + +*Last updated: 2026-06-18. Owner: on-call rotation.* diff --git a/docs/runbooks/sev1-response.md b/docs/runbooks/sev1-response.md new file mode 100644 index 0000000..82ac795 --- /dev/null +++ b/docs/runbooks/sev1-response.md @@ -0,0 +1,122 @@ +# SEV1 Incident Response Runbook + + + +> **Alert annotation URL:** `docs/runbooks/sev1-response.md#runbook-sev1` +> Embed this anchor in PagerDuty / Alertmanager annotations so on-call lands here directly. + +--- + +## Definition + +**SEV1 = customer-wide outage or data loss.** + +Triggering conditions (any one is sufficient): +- `vecna-api` health check fails for > 2 consecutive minutes +- Error rate across all endpoints > 25% sustained for > 5 min +- Database unreachable (all `/api/*` routes return 500) +- Any confirmed data loss or data corruption in `vecna_ops.db` +- p99 latency > 10× SLO for > 5 min with no recovery trend + +--- + +## Immediate Actions (first 5 minutes) + +SLA: acknowledge the alert within **5 minutes** of detection. + +``` +[ ] Acknowledge alert in PagerDuty +[ ] Join #incidents Slack channel (or open one if absent) +[ ] Declare severity: post "SEV1 DECLARED — " in channel +[ ] Assign Incident Commander (IC) — IC owns comms, not hands-on work +[ ] Assign Ops Lead — separate from IC; drives mitigation +``` + +**First technical check — in this order:** + +1. **Recent deploy?** → Roll back immediately; do not debug in-place. + ```bash + # See production-rollback-runbook.md#vecna-api for full steps + git -C /opt/vecna-ops log --oneline -5 + # If the top commit is less than ~1h old and correlates with the incident start → rollback + sudo systemctl restart vecna-api # after git checkout + ``` + +2. **Single-region / AZ failure?** → Failover to standby region or secondary instance. + +3. **Load spike?** → Enable rate limiting, scale horizontally, or shed non-essential traffic. + +> **Rule: stabilize before you diagnose.** Rollback first; root-cause analysis happens after users are restored. + +--- + +## IC Responsibilities During a SEV1 + +The **Incident Commander** owns: +- Opening and maintaining the incident channel +- Assigning roles (Ops Lead, Comms, Scribe) +- Executive communication every **30 minutes** until resolved +- Declaring the incident closed once SLIs recover + +The IC does **not** do hands-on technical work while commanding. + +**Executive update template (every 30 min):** +``` +SEV1 UPDATE [HH:MM UTC] +Status: +Impact: +Last action: +Next action: +ETA: +``` + +--- + +## Escalation + +| Time since declare | Action | +|---|---| +| 0 min | Page on-call engineer | +| 10 min | Page engineering lead if no active mitigation in progress | +| 20 min | Page VP Engineering; notify customer success | +| 30 min | First executive update | +| 60 min | Re-evaluate if rollback is not sufficient; consider DR | + +--- + +## Recovery Criteria + +Do not declare recovery until **all** of the following hold for ≥ 15 min: + +- `GET /api/health/deep` returns 200, `"database": "ok"` +- Error rate ≤ baseline +- p99 latency ≤ SLO +- No new customer reports arriving + +Post-recovery steps: +- [ ] Post all-clear in incident channel and #general +- [ ] Disable any feature flags deployed as part of the bad change +- [ ] Close incident in PagerDuty +- [ ] **Open blameless postmortem within 3 business days** → [postmortem-template.md](postmortem-template.md) + +--- + +## Key Links + +| Resource | Link | +|---|---| +| Rollback runbook | [production-rollback-runbook.md](production-rollback-runbook.md) | +| Postmortem template | [postmortem-template.md](postmortem-template.md) | +| SEV2 runbook | [sev2-response.md](sev2-response.md) | +| Health endpoint | `GET /api/health/deep` | +| Metrics endpoint | `GET /api/metrics` | + +--- + +## Downgrade to SEV2 + +Downgrade to SEV2 when impact narrows to a subset of users (error rate drops below 5%) and the path to resolution is clear. See [sev2-response.md](sev2-response.md). + +--- + +*Last updated: 2026-06-18. Owner: on-call rotation.* diff --git a/docs/runbooks/sev2-response.md b/docs/runbooks/sev2-response.md new file mode 100644 index 0000000..9e34f12 --- /dev/null +++ b/docs/runbooks/sev2-response.md @@ -0,0 +1,126 @@ +# SEV2 Incident Response Runbook + + + +> **Alert annotation URL:** `docs/runbooks/sev2-response.md#runbook-sev2` +> Embed this anchor in PagerDuty / Alertmanager annotations so on-call lands here directly. + +--- + +## Definition + +**SEV2 = major degradation affecting a large subset of users; no total outage.** + +Triggering conditions (any one is sufficient): +- Error rate > 5% sustained for > 5 min (but < 25% — above that → SEV1) +- p99 latency > 3× SLO for > 10 min +- Agent runner failing for ≥ 20% of new operations +- WebSocket streaming broken for a significant portion of clients +- Billing / cost calculations producing wrong results affecting multiple sessions + +If impact expands to a full outage, **upgrade to SEV1** immediately — see [sev1-response.md](sev1-response.md). + +--- + +## Immediate Actions (first 5 minutes) + +SLA: acknowledge the alert within **5 minutes** of detection. + +``` +[ ] Acknowledge alert in PagerDuty +[ ] Post "SEV2 DECLARED — " in #incidents Slack +[ ] Assign Incident Commander +[ ] Assign Ops Lead +``` + +**First technical check — in this order:** + +1. **Recent deploy?** → Default action is rollback. + ```bash + git -C /opt/vecna-ops log --oneline -5 + # Correlate top commit timestamp with incident start + # If match → rollback per production-rollback-runbook.md + ``` + +2. **Single endpoint or service degraded?** + ```bash + # Check health breakdown + curl -sf http://localhost:8000/api/health/deep | jq . + # Identify which subsystem reports unhealthy + ``` + +3. **LLM provider degraded?** (Agent failures only) + ```bash + # Check provider status page + # If provider-side: add a user-visible banner, do not page further + # Rate-limit new operations to reduce cost burn during degradation + ``` + +> **Rule: stabilize before diagnosing.** A rollback that restores 80% of traffic is better than 30 minutes of in-place debugging. + +--- + +## IC Responsibilities During a SEV2 + +The **Incident Commander** owns: +- Incident channel and role assignment +- **Hourly updates** to engineering lead (not exec unless it escalates) +- Declaring recovery + +**Hourly update template:** +``` +SEV2 UPDATE [HH:MM UTC] +Status: +Impact: +Last action: +Next action: +``` + +--- + +## Escalation + +| Time since declare | Action | +|---|---| +| 0 min | Page on-call engineer | +| 15 min | Page engineering lead if no active mitigation | +| 30 min | Reassess severity; upgrade to SEV1 if impact grows | +| 60 min | First hourly update to engineering lead | + +--- + +## Recovery Criteria + +Do not declare recovery until **all** hold for ≥ 15 min: + +- Error rate ≤ baseline +- p99 latency ≤ SLO +- `GET /api/health/deep` fully healthy +- No new reports of the degraded behaviour + +Post-recovery steps: +- [ ] Post all-clear in #incidents +- [ ] Close incident in PagerDuty +- [ ] **Open blameless postmortem within 3 business days** → [postmortem-template.md](postmortem-template.md) + +--- + +## Key Links + +| Resource | Link | +|---|---| +| Rollback runbook | [production-rollback-runbook.md](production-rollback-runbook.md) | +| Postmortem template | [postmortem-template.md](postmortem-template.md) | +| SEV1 runbook | [sev1-response.md](sev1-response.md) | +| Health endpoint | `GET /api/health/deep` | +| Metrics endpoint | `GET /api/metrics` | + +--- + +## Downgrade to SEV3 + +Downgrade to SEV3 when a workaround exists, impact is localised, and the fix can ship in the next business-hours window. SEV3 goes into the ticket tracker, not PagerDuty. + +--- + +*Last updated: 2026-06-18. Owner: on-call rotation.*