From f735277f72d347c193ab72360e6002090698bd17 Mon Sep 17 00:00:00 2001 From: Mark Kraemer Date: Tue, 21 Jul 2026 16:07:48 -0400 Subject: [PATCH 1/5] Heroku production debugging skill --- skills/heroku-triage/SKILL.md | 160 +++++++++++++++ skills/heroku-triage/references/heuristics.md | 183 +++++++++++++++++ .../heroku-triage/references/linear-issue.md | 122 +++++++++++ skills/heroku-triage/references/signals.md | 190 ++++++++++++++++++ .../scripts/papertrail-search.sh | 102 ++++++++++ 5 files changed, 757 insertions(+) create mode 100644 skills/heroku-triage/SKILL.md create mode 100644 skills/heroku-triage/references/heuristics.md create mode 100644 skills/heroku-triage/references/linear-issue.md create mode 100644 skills/heroku-triage/references/signals.md create mode 100755 skills/heroku-triage/scripts/papertrail-search.sh diff --git a/skills/heroku-triage/SKILL.md b/skills/heroku-triage/SKILL.md new file mode 100644 index 0000000..8b7ad1c --- /dev/null +++ b/skills/heroku-triage/SKILL.md @@ -0,0 +1,160 @@ +--- +name: heroku-triage +description: > + Diagnose and fix the Heroku production problems Sentry issues don't cover: memory + errors (R14/R15), slow or timing-out requests (H12, high p95 latency), crash and boot + loops, and cost/performance tuning of the formation (Puma workers/threads, dyno size + and count, Postgres plan). Four modes: diagnose (incident triage), apply (confirm + + apply a change, snapshot a baseline, open a Linear tracking issue), verify (compare + metrics against the baseline after a soak period), review (periodic cost/performance + audit). Use when the user says an app "is slow", "is getting R14s", "has memory + errors", "keeps timing out", "tune puma", "right-size the dynos", "heroku cost + review", "why is production slow", or "verify that tuning change". Sentry error + triage belongs to sentry-top-issue / rm-sentry-issue-fixer; this skill covers what + those don't. +license: MIT +--- + +# Heroku Triage + +Production diagnosis and tuning for RoleModel's standard stack: Heroku + Papertrail + +Sentry, Rails on Puma. Requires the `heroku` CLI (authenticated), `jq`, the Sentry MCP, +and the Linear MCP (for apply/verify). Ask the user questions **one at a time**. + +## Operating principles + +1. **Aggregate outside the context window.** Never page raw production logs into the + conversation. Fetch them to files under `/tmp/heroku-triage//` and reduce with + `jq`/`awk`/`sort` — only counts, percentiles, and compact timelines enter context. +2. **Cheapest signal first.** Check the release timeline before log spelunking; check + error-code counts before fetching log bodies. Most incidents correlate with a deploy + or config change visible in `heroku releases` in ten seconds. +3. **Capture before you heal.** A restart erases the evidence (memory curves, stuck + state). Snapshot logs and metrics before restarting or scaling anything. +4. **Two independent signals per conclusion.** Do not propose a fix from a single data + point. A diagnosis must cite at least two of: log timeline, Heroku state, Sentry + data, Postgres stats, code inspection. +5. **One change at a time.** Every mutation goes through apply mode: baseline → confirm + → apply → Linear issue → verify after a soak. Never batch tuning changes; you can't + attribute the result. + +## Inputs ($ARGUMENTS) + +Space-separated `key=value` tokens plus bare words, all optional except the app: + +- `app=` (or first bare word that isn't a mode) — required +- `mode=diagnose|apply|verify|review` (or bare word) — default `diagnose` +- `symptom=memory|latency|availability|unknown` — default `unknown`, diagnose only +- `window=` — analysis window, default 24 (review uses 168) +- `repo=` — local checkout of the app, for code-level steps +- `team=` — Linear team for tracking issues +- `target-p95=` — latency target, default 500 +- `dry-run` — never mutate anything; print what apply would do + +## Preflight (all modes) + +1. `heroku auth:whoami` and `heroku apps:info -a $APP` — stop with instructions if + either fails. +2. Resolve the app repo: use `repo=`, else check whether the cwd's git remotes or + `heroku git:remote` match the app. Code-level steps degrade gracefully without it + (`heroku run cat -a $APP` reads one file from the slug — uses a one-off dyno). +3. Papertrail: `heroku config:get PAPERTRAIL_API_TOKEN -a $APP`. If empty, history is + limited to `heroku logs` (~1,500 lines) — say so and lower confidence accordingly. +4. `heroku labs -a $APP | grep log-runtime-metrics`. If disabled and the investigation + needs memory/CPU data, this is the first proposed change (see signals.md §4 for the + restart caveat). Memory diagnosis needs a few hours of samples after enabling. +5. Sentry MCP: find the project matching the app/repo name; ask if ambiguous. +6. Linear MCP (apply/verify): confirm it responds. If unavailable, apply mode prints + the tracking-issue content for the user to file manually rather than skipping it. +7. **Secrets hygiene:** never run bare `heroku config -a $APP`. Read only whitelisted + keys via `heroku config --json -a $APP | jq '{WEB_CONCURRENCY, RAILS_MAX_THREADS, + MALLOC_ARENA_MAX, RUBY_YJIT_ENABLE, RAILS_ENV, RACK_ENV, WEB_TIMEOUT}'`. + +## Mode: diagnose + +Read `references/signals.md` (recipes) and `references/heuristics.md` (decision trees) +before starting. + +1. **Census** — formation (`heroku ps`), last 20 releases, whitelisted config, dyno + error-code counts for the window vs. the same window a week earlier (Papertrail + counts only). Identify the dominant symptom code (R14/R15, H12, H10/R10…). +2. **Onset correlation** — find when the symptom started (bucketed counts per hour/day) + and line it up against the release timeline. A matching release → `git log`/diff + that release in the repo; the culprit is usually in it. +3. **Playbook** — follow the matching decision tree in heuristics.md: §1 memory, + §2 latency, §3 availability. Gather only the signals the tree asks for. +4. **Diagnosis block** — end with a structured summary: symptom, root cause, confidence + (high/medium/low), evidence (each item citing its source), proposed change (exact + commands or PR sketch), expected effect, risk, revert plan. If confidence is low, + say what additional data would raise it (often: enable instrumentation, wait, rerun). +5. Stop and present. If the user wants to proceed, continue into apply mode. + +## Mode: apply + +Requires a diagnosis (from this session, or restated by the user). + +1. **Baseline first**: capture the metrics snapshot per `references/linear-issue.md` + §Baseline — before touching anything. +2. Present the exact change: commands, expected effect, risk, revert commands, soak + period and success criteria (from heuristics.md §7). **Wait for explicit + confirmation.** `dry-run` stops here. +3. Apply via heroku CLI. Confirm it took (`heroku releases -n 3`, `heroku ps`), then + smoke-check: `/up` returns 200, and 5 minutes of error-code tail shows no new codes. +4. Create the Linear tracking issue per linear-issue.md (title + `[heroku-triage] : `, baseline JSON, criteria, earliest-verify time). + Ask for the team once (`team=` skips this). +5. Tell the user when to run `verify` (e.g., "after Thursday's peak — 48h from now"). + +## Mode: verify + +1. Find open `[heroku-triage] ` issues via Linear MCP. None → report and stop. +2. Parse the baseline JSON and criteria. If the soak period hasn't elapsed, say how + long remains and stop (unless the user insists). +3. Recompute the same metrics over an equivalent window (match weekday/peak where the + baseline did) and build a before/after table. +4. Verdict per the criteria (heuristics.md §7): + - **KEEP** — criteria met: comment with the table, close the issue. + - **ITERATE** — improved but short: comment, keep open, propose the next rung of + the fix ladder (a fresh apply cycle). + - **REVERT** — regression: confirm with the user, run the recorded revert commands, + comment, restart the soak clock for the revert. + +## Mode: review + +The recurring cost/performance audit (default window 168h). Composable with the +`schedule` skill for automated runs, like sentry-top-issue. + +1. Gather the 7-day dossier (signals.md): formation + monthly cost, steady/peak memory + per process type, latency p50/p95/p99, throughput peaks, load averages, R14/H12 + census, Postgres health (cache hit, connections vs. limit), addon plans. +2. Score against the targets table (heuristics.md §6), adjusted by `target-p95=`. +3. Output ranked recommendations — each with: change, evidence, expected effect, + monthly cost delta, risk, confidence. Include "no change" findings so the user sees + what was checked. Flag anything within 20% of a limit as "watch". +4. Offer to run apply on the top recommendation — one change at a time. + +## Guardrails + +- Never run destructive commands: `apps:destroy`, `addons:destroy`, `pg:reset`, + `pg:kill*`, `maintenance:on`, `config:unset DATABASE_URL`. `releases:rollback` and + `ps:restart` are allowed **only** with explicit user confirmation, after evidence + capture (§principle 3). +- Every mutation is confirm-first, has its revert commands recorded in Linear *before* + it runs, and is one change at a time. +- Don't stack memory allocator experiments (MALLOC_ARENA_MAX and jemalloc) in one step. +- Static tables in heuristics.md (dyno RAM, prices, connection limits) are fallbacks — + prefer live values from `heroku ps`, `pg:info`, and the pricing page; note when a + recommendation depends on a price you couldn't verify. +- Rolling restarts / worker killers are mitigations, not fixes: if proposed, record a + follow-up Linear issue naming the unresolved root cause. + +## Reference map + +- `references/signals.md` — every data-gathering recipe: heroku CLI census, Papertrail + API (via `scripts/papertrail-search.sh`), log-runtime-metrics, Sentry MCP intents, + Postgres, in-repo config. Read at the start of diagnose/review. +- `references/heuristics.md` — decision trees (memory §1, latency §2, availability §3), + Puma sizing + connection math §4, formation economics §5, review targets §6, + verification criteria and soak periods §7. +- `references/linear-issue.md` — tracking-issue template, baseline JSON schema, verify + comment format. Read before apply/verify. diff --git a/skills/heroku-triage/references/heuristics.md b/skills/heroku-triage/references/heuristics.md new file mode 100644 index 0000000..b7406bb --- /dev/null +++ b/skills/heroku-triage/references/heuristics.md @@ -0,0 +1,183 @@ +# Heuristics: decision trees, sizing rules, verification criteria + +Section numbers are referenced from SKILL.md and signals.md. + +## 1. Memory playbook (R14/R15) + +Start from the **shape of the memory curve** (signals §3, 10-min buckets, per dyno, +48h+). The shape is the discriminator; each row names its confirmation signal — +remember the two-signal rule. + +| Curve shape | Likely cause | Confirm with | Fix (see ladder) | +|---|---|---|---| +| Steady climb over hours, resets on restart, R14s late in the cycle | Ruby heap growth: glibc malloc fragmentation (most common on Heroku) or a true leak | Fragmentation plateaus/slows near an asymptote; a true leak climbs unbounded into R15. Cheapest confirm: the ladder's step 1 *is* the experiment | Ladder step 1 → 3 | +| Sharp step(s) at repeatable times or after specific requests | Request-driven bloat: an endpoint loading a huge dataset (missing `find_each`, giant `includes`, file processing in-request, big export) | ±90s context fetch around each step (signals §3); Sentry transaction spans at those timestamps; router `bytes=` outliers | Fix the endpoint: stream/batch/paginate, or move to a background job | +| Already at 60–70%+ of quota at boot, before traffic | Too many Puma workers for the dyno, or boot-time bloat (heavy initializers, preloaded data) | Boot math: master + per-worker RSS × `WEB_CONCURRENCY` vs. quota; compare curve minutes after deploy vs. steady state | Reduce `WEB_CONCURRENCY` (mind capacity, §4) or audit initializers; bigger dyno last | +| Sawtooth of climbs ending in R15 kills every few hours | Runaway individual requests/jobs (unbounded query, huge upload/report) | Last requests before each kill (±90s fetch); H13s and job retries at kill times | Find and bound the runaway (limits, streaming, timeouts) | +| Worker dyno climbs; web flat | Job-driven: per-job allocations retained, or concurrency × per-job RSS exceeds quota | Correlate curve with job schedule; job runtimes in Sentry | Reduce job concurrency, split the heavy queue to its own dyno, batch differently | +| `memory_swap` high while RSS under quota | Same as R14 — quota counts swap; treat by shape as above | — | — | + +When memory *and* latency problems co-occur, fix memory first: a swapping dyno is slow +at everything, and the latency signal is unreadable until swap stops. + +**Fix ladder** (web memory; cheapest and most reversible first — one rung per apply): + +1. `heroku config:set MALLOC_ARENA_MAX=2 -a $APP` — Heroku's own recommendation for + glibc arena fragmentation. Free, instant, low risk; commonly cuts steady RSS + 10–30%. This doubles as the fragmentation diagnosis: if the curve flattens lower, + fragmentation confirmed. +2. Right-size workers: `workers = floor((quota × 0.85 − master_RSS) / per_worker_RSS)`. + If lower than current `WEB_CONCURRENCY`, reduce it — but check the capacity math + (§4) and pair with a dyno-count/threads adjustment if throughput needs it. +3. jemalloc buildpack (`gaffneyc/heroku-buildpack-jemalloc`, needs a deploy) — the + stronger allocator fix. Don't stack with step 1 in a single apply; measure one + change at a time. +4. Hunt the bloat in code: Sentry profiles (hot allocators), fattest responses + (router `bytes=`), heaviest transactions; locally `derailed bundle:mem` (gem cost at + boot) and `derailed exec perf:mem_over_time`. Output is a PR, not a config change. +5. Bigger dyno (e.g. standard-1x → standard-2x). Legitimate when per-worker RSS is + irreducible; state the monthly cost in the proposal. +6. Rolling restarts / `puma_worker_killer` — **mitigation, not fix** (guardrails): + allowed to stop the bleeding, but files a follow-up Linear issue naming the + unresolved root cause. + +## 2. Latency playbook (H12, high p95) + +**Discriminator 1 — queued or slow?** For the same window compute router `service=` +p95 vs. lograge `duration=` p95 (signals §3). Router time ≈ queueing + app time. + +- **Gap ≫ 0 (router p95 − app p95 > ~100ms): requests queue at the dyno → capacity.** + - CPU-bound? `sample#load_avg_1m` sustained ≥ ~1.0 on standard dynos (they're + single-core-ish shared) → more dynos/workers, or YJIT if not enabled + (`RUBY_YJIT_ENABLE=1`, Ruby 3.3+ — small memory cost, real CPU win). + - CPU idle but queueing → threads blocked on I/O: check DB pool waits (Papertrail: + `"could not obtain a connection"`), slow external calls holding threads. Raise + threads modestly (3 → 5) only with this evidence, and re-check connection math §4. + - Utilization math (Little's law): `concurrent demand ≈ peak_rps × avg_service_s`; + `capacity = dynos × workers × threads`. Sustained demand/capacity > ~0.7 → add + capacity (another dyno) or reduce service time. + - Spiky traffic (short queueing bursts, quiet otherwise) → mention Judoscale + (queue-time-based autoscaling) as the structural fix instead of permanent + overprovisioning. +- **Gap ≈ 0: the app itself is slow → route by Sentry span breakdown** (signals §5): + - **db-dominant** → `pg:outliers` + Sentry N+1/slow-query issues + `pg:diagnose` + (missing/unused indexes, bloat). Single endpoint → fix its query (PR). Global + + cache hit < 0.99 → working set outgrew the Postgres plan (§5 economics). + - **http.client-dominant** → an external API: add/lower timeouts, move to a job, + circuit-break. The fix is a PR. + - **app/CPU-dominant** → Sentry profile hotspots; YJIT; memoize/cache the hotspot. +- **H12 pattern shortcuts** (path distribution, signals §3): + - One path dominates → that endpoint (usually a report/export/upload) → background + job + polling or Turbo stream; never a bigger timeout. + - Clustered right after deploys/restarts → boot storm: confirm `preload_app!` in + puma.rb, check boot duration in logs ("Listening on" minus start), H19/R10 nearby. + - Spread across paths at peak → saturation → capacity branch above. +- **Rack::Timeout note**: a service timeout of 15–25s (< the router's 30s) converts + opaque H12s into in-app exceptions with Sentry traces. Recommend when H12 cause is + unclear; flag that it must be an intentional, tested addition (mid-request raises + have sharp edges), and it never substitutes for fixing the slow path. + +## 3. Availability playbook (H10 / R10 / crash loops) + +1. `heroku ps` — crashed/restarting dynos and their ages; `heroku releases -n 5` — did + it start with a release? +2. Boot exception: `heroku logs --dyno web --num 300` → the stack trace is almost + always right there. Release-correlated → propose `heroku releases:rollback` + (confirm-first) while the fix is developed; otherwise fix forward. +3. R10 (60s boot timeout): slow boot — measure "Listening on" delta; usual suspects + are heavy initializers, remote calls at boot, or migrations run in-process instead + of the release phase. +4. One dyno crashing while siblings are healthy → almost always R15 kills (memory + playbook), not code. + +## 4. Puma sizing and connection math + +- **Threads**: 3 (Rails 7.2+ default) to 5. More than 5 rarely helps a Rails app — + GVL contention and DB pool pressure eat the gains. Raise only on I/O-wait evidence + (§2); lower toward 3 when CPU-bound or memory-tight. +- **Workers (`WEB_CONCURRENCY`)**: as many as fit `steady_total ≤ 85% of quota` + (ladder step 2 formula). On shared-CPU dynos (basic/standard) load_avg is the + ceiling check, not core counts. Always `preload_app!` (copy-on-write). +- **DB pool**: Rails defaults `pool = RAILS_MAX_THREADS` — correct; don't hand-tune + pool per-app without cause. +- **Connection ceiling** (the classic Heroku trap): + `total ≈ web_dynos × workers × threads + worker_dynos × job_threads (+ ~2 for + console/one-offs)`. Keep under ~80% of the `pg:info` connection limit. Essential + plans allow only ~20 — e.g. 2 web dynos × 2 workers × 5 threads = 20 = saturated + before the worker dyno connects. Fixes in order: lower threads, upgrade the + Postgres plan, pgbouncer buildpack (transaction pooling; check + prepared-statement compatibility). +- **Worker dynos**: job concurrency counts against both memory (per-thread job RSS) + and connections (GoodJob threads each hold one). + +## 5. Formation economics + +Fallback table (confirm live: `heroku ps` for current, Heroku pricing page for rates — +flag any recommendation whose price you couldn't verify): + +| Dyno | RAM | ~$/mo | Notes | +|---|---|---|---| +| eco | 512MB | 5 (pool) | sleeps; never for client production | +| basic | 512MB | 7 | no autoscaling, 1 dyno per process type honest floor | +| standard-1x | 512MB | 25 | shared CPU | +| standard-2x | 1GB | 50 | shared CPU, 2× share | +| performance-m | 2.5GB | 250 | dedicated CPU | +| performance-l | 14GB | 500 | dedicated, big memory | + +Rules of thumb: +- **2× standard-1x vs 1× standard-2x (same $50)**: prefer two 1x for redundancy and + zero-downtime restarts — *unless* per-worker RSS forces the 1GB quota. +- Performance tier: justified when ~4–5 standard-2x can't hold p95 (CPU-bound) or a + single process legitimately needs >1GB. Below that, scale standard horizontally. +- **Downsize candidates** (review mode): steady memory < 50% of quota for 14 days AND + p95 well under target AND queue gap ≈ 0 → step down one rung (size or count, not + both), verify per §7. +- **Postgres plan**: upgrade triggers are the connection ceiling (§4), cache hit + < 0.99, or data size > ~75% of plan limit (`pg:info`). Essential tiers also lack + diagnostics and followers — growing apps outgrow them operationally, not just on + size. +- **Papertrail plan**: check usage vs. plan in the Papertrail account (API: + `/api/v1/accounts.json`); chronic overage or truncated retention → one plan up is + usually cheap insurance for exactly the debugging this skill does. +- **Scheduler vs. always-on worker**: a worker dyno that's idle outside a few + scheduled jobs can often become Heroku Scheduler one-off runs (pay per second). + +## 6. Review-mode targets (defaults; override via args, e.g. `target-p95=`) + +| Signal | Healthy | Watch | Act | +|---|---|---|---| +| Steady memory / quota | 60–85% | 50–60% or 85–95% | <50% (downsize) / >95% or any R14 (memory playbook) | +| R14 or R15 in window | 0 | — | any | +| p95 latency | < target (default 500ms) | within 20% | above | +| H12 in window | 0 | — | any | +| Queue gap p95 (router − app) | < 50ms | 50–100ms | > 100ms (capacity) | +| load_avg_1m sustained | < 0.7 | 0.7–1.0 | > 1.0 (standard dynos) | +| PG cache hit | ≥ 0.99 | 0.985–0.99 | < 0.985 | +| PG connections peak / limit | < 60% | 60–80% | > 80% | +| Monthly cost | — | — | any Act/downsize finding → quantify $ delta | + +Score every process type and the database; report "healthy" rows too (the value of a +review is knowing what was checked). Weekend-vs-weekday and seasonal peaks: compare +like windows before declaring headroom. + +## 7. Verification criteria and soak periods (verify mode) + +General: match the baseline's window shape (same weekday/peak coverage). "±10%" means +relative to baseline, not target. + +| Change type | Soak | Success criteria | +|---|---|---| +| MALLOC_ARENA_MAX / jemalloc | 48h | R14/day → 0 (or ≥90% drop); steady memory down ≥10%; p95 within ±10% | +| WEB_CONCURRENCY reduced | 48h incl. a weekday peak | zero R14; queue gap p95 < 50ms; p95 ±10%; throughput unchanged | +| Threads changed | 48h | as above, plus zero pool-timeout log lines and PG connections < 80% limit | +| Dyno size/count reduced | 72h incl. peak | p95 ±10%; zero new H12; load_avg within Healthy; memory < 85% | +| Dyno size/count increased (incident) | 24h | symptom gone (R14/H12 = 0); note follow-up: revisit cost in next review | +| Query/index fix (PR) | 24h post-deploy | target endpoint p95 improved by the amount predicted in the diagnosis | +| Job concurrency / queue split | 48h | worker memory < 85%; queue latency (oldest-job age) within target; zero R14 | + +Verdicts: +- **KEEP**: all criteria met → close the Linear issue with the before/after table. +- **ITERATE**: direction right, magnitude short → keep the issue open, propose the + next ladder rung as a fresh apply (new baseline = current state). +- **REVERT**: any criterion regressed beyond its bound → confirm, run the recorded + revert, restart the soak clock to confirm the revert restored baseline. diff --git a/skills/heroku-triage/references/linear-issue.md b/skills/heroku-triage/references/linear-issue.md new file mode 100644 index 0000000..548d0dd --- /dev/null +++ b/skills/heroku-triage/references/linear-issue.md @@ -0,0 +1,122 @@ +# Linear tracking issue: template and baseline schema + +The Linear issue is the durable memory between sessions: apply mode writes it, verify +mode (possibly days later, possibly a different developer's session) finds and settles +it. Everything verify needs must be in the issue — assume the applying session's +context is gone. + +## Conventions + +- **Title**: `[heroku-triage] : ` — e.g. + `[heroku-triage] acme-production: set MALLOC_ARENA_MAX=2`. + The `[heroku-triage]` prefix is how verify mode finds open work; never omit it. +- **Team**: use `team=` if given, else list teams via Linear MCP and ask the user once + per session. Add the label `heroku-triage` if the team has one (create it if the + user agrees; don't block on it). +- **One issue per applied change.** An ITERATE verdict adds a comment and later a new + issue for the next rung — it does not grow the existing issue's scope. + +## Issue body template + +```markdown +## Symptom + + +## Diagnosis + + +**Evidence** +- (source: papertrail R14 hourly buckets, 7d) +- (source: heroku releases / sentry transaction p95 / pg:info …) + +## Change applied +```bash + +``` +Applied at: (release v) + +## Revert +```bash + +``` + +## Verification +- Earliest verify: +- Window to reproduce: +- Success criteria: + - [ ] + - [ ] +- Run: `heroku-triage verify app=` + +## Baseline +```json + +``` +``` + +## Baseline JSON schema (machine-read by verify mode) + +Populate every field you measured for the diagnosis; `null` for unmeasured ones — but +each success criterion must have its baseline field populated. Windows in hours; +memory in MB; latency in ms. + +```json +{ + "app": "acme-production", + "captured_at": "2026-07-15T14:30:00Z", + "window_hours": 48, + "formation": { "web": { "size": "standard-1x", "count": 2 }, + "worker": { "size": "basic", "count": 1 } }, + "config": { "WEB_CONCURRENCY": "2", "RAILS_MAX_THREADS": "5", + "MALLOC_ARENA_MAX": null, "RUBY_YJIT_ENABLE": null }, + "memory": { "web_steady_mb": 480, "web_peak_mb": 545, "quota_mb": 512, + "worker_steady_mb": 300, "r14_per_day": 37, "r15_per_day": 0 }, + "latency_ms": { "router_p50": 120, "router_p95": 890, "router_p99": 2100, + "app_p95": 610, "queue_gap_p95": 280, "h12_per_day": 4 }, + "throughput": { "peak_rpm": 260, "peak_window": "Thu 14:00-15:00 UTC" }, + "load_avg_1m_peak": 1.4, + "pg": { "plan": "essential-0", "cache_hit": 0.993, + "connections_peak": 14, "connection_limit": 20 }, + "cost_usd_month": 57, + "change": { + "summary": "set MALLOC_ARENA_MAX=2", + "commands": ["heroku config:set MALLOC_ARENA_MAX=2 -a acme-production"], + "revert": ["heroku config:unset MALLOC_ARENA_MAX -a acme-production"], + "applied_at": "2026-07-15T14:42:00Z", + "ladder_rung": "memory-1", + "next_rung_if_iterate": "memory-3 (jemalloc buildpack)" + }, + "verify": { + "earliest": "2026-07-17T14:42:00Z", + "criteria": [ + { "metric": "memory.r14_per_day", "op": "==", "value": 0 }, + { "metric": "memory.web_steady_mb", "op": "<=", "value": 432 }, + { "metric": "latency_ms.router_p95", "op": "within_pct", "value": 10 } + ] + } +} +``` + +## Verify comment template + +Post as a comment on the issue (then close it on KEEP): + +```markdown +## Verification — () + +| Metric | Baseline | Now | Criterion | Result | +|---|---|---|---|---| +| R14/day | 37 | 0 | == 0 | ✅ | +| Steady memory (web) | 480MB | 395MB | ≤ 432MB | ✅ | +| Router p95 | 890ms | 855ms | ±10% | ✅ | + +Windows compared: vs (matched: Thu peak). + + + +, new soak deadline for confirming baseline restoration.> +``` + +State machine: open → KEEP (close) | ITERATE (stays open until superseded or resolved) +| REVERT (close after the revert-soak confirms baseline restored, with a final comment). diff --git a/skills/heroku-triage/references/signals.md b/skills/heroku-triage/references/signals.md new file mode 100644 index 0000000..eaaf47d --- /dev/null +++ b/skills/heroku-triage/references/signals.md @@ -0,0 +1,190 @@ +# Signals: data-gathering recipes + +Working convention: raw data goes to `/tmp/heroku-triage//` (create it once per +session: `mkdir -p /tmp/heroku-triage/$APP`). Only aggregates enter the conversation. +All recipes assume `APP=` is set. `date` invocations show the macOS form; +on Linux use `date -d '24 hours ago' +%s` instead of `date -v-24H +%s`. + +## 1. State census (cheap — always run first) + +```bash +heroku ps -a $APP # formation, sizes, dyno uptimes (short uptimes = restart loop) +heroku releases -a $APP -n 20 # deploys AND config changes, timestamped — the onset suspect list +heroku addons -a $APP # postgres plan, papertrail plan +heroku buildpacks -a $APP # jemalloc present? metrics buildpack? +heroku labs -a $APP # log-runtime-metrics enabled? +heroku config --json -a $APP | jq '{WEB_CONCURRENCY, RAILS_MAX_THREADS, MALLOC_ARENA_MAX, RUBY_YJIT_ENABLE, WEB_TIMEOUT}' +``` + +Interpretation notes: +- `heroku ps` uptime resets on restart — a dyno hours younger than its siblings + restarted (R15 kill, crash, or manual). +- `heroku releases` includes `config:set` entries — a tuning change someone made by + hand shows up here. Always line incident onset up against this list. + +## 2. Error-code census (recent window, no Papertrail needed) + +```bash +heroku logs -a $APP --num 1500 > /tmp/heroku-triage/$APP/recent.log +grep -oE 'code=(H|R|L)[0-9]+' /tmp/heroku-triage/$APP/recent.log | sort | uniq -c | sort -rn +``` + +| Code | Meaning | First move | +|------|---------|-----------| +| R14 | Memory quota exceeded (swapping) | Memory playbook (heuristics §1) | +| R15 | Quota vastly exceeded — dyno SIGKILLed | Memory playbook; check for runaway request/job | +| H12 | Request exceeded 30s router timeout | Latency playbook (heuristics §2) | +| H13 | Connection closed without response | Often a worker killed mid-request (pairs with R15) | +| H10 | App crashed | Availability playbook (heuristics §3) | +| R10 | Boot timeout (no bind to $PORT in 60s) | Availability playbook | +| H19 | Backend connection timeout (dyno didn't accept in 5s) | Boot storm / saturation | +| H21 | Backend connection refused | Dyno restarting under load | +| H27/H28 | Client interrupted / idle | Usually benign; ignore unless dominant | + +`heroku logs` holds ~1,500 lines — minutes on a busy app. It answers "what's happening +right now"; every historical question goes to Papertrail. + +## 3. Papertrail (history — the agent's primary time-series source) + +Token: `heroku config:get PAPERTRAIL_API_TOKEN -a $APP` (provisioned by the addon). +Fetch with the bundled script — it paginates the search API, dedupes, and emits +TSV (`received_at`, `source`, `program`, `message`) oldest-first: + +```bash +bash skills/heroku-triage/scripts/papertrail-search.sh \ + --app $APP --query '"Error R14"' \ + --from $(date -v-7d +%s) --out /tmp/heroku-triage/$APP/r14.tsv +``` + +**Sampling strategy — do not fetch the firehose.** Router lines on a busy app run to +millions per week. Instead fetch narrow windows and compare: the incident hour, the +same hour yesterday (control), and the weekly peak hour. `--max-events` defaults to +50,000; if the script reports hitting the cap, narrow the window rather than raising it. + +Query cookbook (Papertrail full-text; quote exact phrases, filter precisely with +grep/awk afterwards): + +```bash +# R14/R15 timeline → hourly buckets: which days, what cadence, which dynos? +scripts/papertrail-search.sh --app $APP --query '"Error R1"' --from $(date -v-7d +%s) --out r1x.tsv +cut -f1 r1x.tsv | cut -c1-13 | sort | uniq -c # events per hour +grep -oE 'web\.[0-9]+|worker\.[0-9]+' r1x.tsv | sort | uniq -c # which process/dyno + +# Memory curve for one dyno (needs log-runtime-metrics, §4) → 10-min buckets, max MB +scripts/papertrail-search.sh --app $APP --query '"sample#memory_total" "source=web.1"' \ + --from $(date -v-48H +%s) --out mem.tsv +awk -F'\t' '{ if (split($0, a, "memory_total=") < 2) next; split(a[2], b, "MB"); + ts = substr($1, 1, 15); if (b[1]+0 > mx[ts]) mx[ts] = b[1]+0 } + END { for (t in mx) printf "%s0 %d MB\n", t, mx[t] }' mem.tsv | sort +# Read the SHAPE of this curve against heuristics §1. Repeat for web.2, worker.1. + +# Router latency percentiles for a one-hour window (status/service from heroku router lines) +scripts/papertrail-search.sh --app $APP --query '"heroku/router" "service="' \ + --from --to --out router.tsv +grep -oE 'service=[0-9]+' router.tsv | cut -d= -f2 | sort -n | \ + awk '{ v[NR] = $1 } END { if (!NR) exit; + printf "n=%d p50=%d p95=%d p99=%d max=%d\n", NR, v[int(NR*.5)], v[int(NR*.95)], v[int(NR*.99)], v[NR] }' + +# App-reported duration for the SAME window (lograge lines) — the queue-time discriminator +scripts/papertrail-search.sh --app $APP --query '"duration="' --from --to --out app.tsv +grep -oE 'duration=[0-9.]+' app.tsv | cut -d= -f2 | sort -n | \ + awk '{ v[NR] = $1 } END { if (!NR) exit; + printf "n=%d p50=%.0f p95=%.0f p99=%.0f\n", NR, v[int(NR*.5)], v[int(NR*.95)], v[int(NR*.99)] }' + +# H12 path distribution — one bad endpoint or everything? +grep 'H12' router-or-r1x.tsv | grep -oE 'path="[^"]*"' | sed 's/?.*/"/' | sort | uniq -c | sort -rn | head + +# What ran just before a memory step / R15? (±90s around a timestamp, all sources) +scripts/papertrail-search.sh --app $APP --query 'web.2' --from --to --out ctx.tsv +``` + +Searchable retention on small Papertrail plans is about a week; older history lives in +archives: `curl -sH "X-Papertrail-Token: $TOKEN" +https://papertrailapp.com/api/v1/archives.json` lists daily `.tsv.gz` files with +download URLs — `zgrep`/`awk` them locally, never into context. + +## 4. log-runtime-metrics (the machine-readable memory/CPU source) + +`heroku labs:enable log-runtime-metrics -a $APP` emits, per dyno every ~20s: +`sample#memory_total`, `memory_rss`, `memory_swap`, `memory_quota`, +`sample#load_avg_1m/5m/15m` — into the log stream, hence into Papertrail history. + +Caveats: +- Takes effect on the **next restart**, and a restart resets the memory curve you may + be investigating. If a leak is live right now: capture `heroku logs` and current + `heroku ps` evidence first, then enable + restart, then diagnose from the fresh curve + after a few hours of accumulation. +- Enabling is a mutation: confirm-first, though it's routine and zero-risk to traffic. +- The Heroku Metrics **dashboard** (and its language metrics via the `heroku/metrics` + buildpack + `barnes`) is human-only — there is no supported API. For agent work, + log-runtime-metrics via Papertrail is the source of truth. If the dashboard is + needed as a cross-check, ask the user to read specific charts aloud. + +## 5. Sentry MCP (app-level truth) + +Resolve the project once (find_projects, match on repo/app name). Useful intents: + +- **Slowest transactions**: p95 duration by transaction for the window, sorted desc; + compare against the prior week for regressions. Also throughput per transaction — + a slow+hot endpoint outranks a slow+cold one. +- **Span breakdown** for the worst transactions: is time in `db`, `http.client`, or + app code? This routes the latency playbook (heuristics §2). +- **Performance issues**: Sentry auto-detects N+1 queries, slow DB queries, + consecutive DB queries, large render blocking — list unresolved ones. +- **Profiles** (apps ship `stackprof`): top functions by self-time for CPU-bound + hypotheses. +- **Release correlation**: error/latency deltas around the release identified in the + census. +- **Memory adjacency**: R15 kills surface as H13s and job retries, not memory events — + check for `SIGKILL`/`Errno` error spikes and job-retry bursts at R14/R15 timestamps. + +## 6. Postgres + +```bash +heroku pg:info -a $APP # plan, connection limit + current, cache hit, data size vs plan +heroku pg:diagnose -a $APP # red/yellow report: hit rate, bloat, blocking, long queries, unused indexes +heroku pg:outliers -a $APP # top queries by total time (pg_stat_statements) +heroku pg:locks -a $APP # only during an active incident +``` + +- Cache hit rate belongs ≥ 0.99; below that = working set exceeds RAM → plan upsize or + bloat/missing-index work (pg:diagnose says which). +- On Essential-tier databases some diagnostics (e.g. pg:outliers) are unavailable — + fall back to Sentry slow-query spans. +- Connection math lives in heuristics §4; the limit comes from `pg:info`, live. + +## 7. In-repo reads (needs the app checkout, or `heroku run cat -a $APP`) + +- `config/puma.rb` — threads default, `workers ENV["WEB_CONCURRENCY"]` handling, + `preload_app!` present? +- `Procfile` — worker process? what runs it (sidekiq/good_job)? +- `config/database.yml` — `pool:` (default `RAILS_MAX_THREADS`) +- Job concurrency: `config/sidekiq.yml` concurrency, or GoodJob + `GOOD_JOB_MAX_THREADS` / `config.good_job` settings +- `Gemfile` — rack-timeout? puma_worker_killer (a mitigation smell — see guardrails)? +- Recent deploy diff when onset matches a release: `git log --oneline vN-date..vM-date`, + diff the suspects. + +## 8. Optional deeper signal: Puma pool stats (deploy required — deep tuning only) + +When thread-pool saturation must be measured directly (heuristics §2 capacity path, +low confidence from the router-vs-app gap alone), offer this snippet as a PR to the +app — logs busy/backlog per worker every 30s, readable via the Papertrail recipes: + +```ruby +# config/initializers/puma_stats.rb +return unless defined?(Puma) && Rails.env.production? +Thread.new do + loop do + sleep 30 + s = Puma.stats_hash rescue next + workers = s[:worker_status]&.map { |w| w[:last_status] } || [s] + busy = workers.sum { |w| w[:busy_threads].to_i } + backlog = workers.sum { |w| w[:backlog].to_i } + Rails.logger.info("puma_stats busy=#{busy} backlog=#{backlog} max=#{workers.sum { |w| w[:max_threads].to_i }}") + end +end +``` + +Any sustained `backlog > 0` means requests queued behind the thread pool — capacity or +a blocking-I/O problem, per heuristics §2. diff --git a/skills/heroku-triage/scripts/papertrail-search.sh b/skills/heroku-triage/scripts/papertrail-search.sh new file mode 100755 index 0000000..3387cac --- /dev/null +++ b/skills/heroku-triage/scripts/papertrail-search.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +# Paginated Papertrail events search for a Heroku app. +# +# Emits TSV (received_at, source_name, program, message), oldest-first, deduped, +# to --out FILE (or stdout). Progress and totals go to stderr. +# +# Requires: curl, jq, heroku CLI (unless PAPERTRAIL_API_TOKEN is already exported). +# +# Usage: +# papertrail-search.sh --app --query '' \ +# --from [--to ] [--max-events N] [--out FILE] +# +# Epoch helpers: macOS `date -v-24H +%s`, Linux `date -d '24 hours ago' +%s`. +set -euo pipefail + +APP="" QUERY="" FROM="" TO="" MAX_EVENTS=50000 OUT="" +while [ $# -gt 0 ]; do + case "$1" in + --app) APP=$2; shift 2 ;; + --query) QUERY=$2; shift 2 ;; + --from) FROM=$2; shift 2 ;; + --to) TO=$2; shift 2 ;; + --max-events) MAX_EVENTS=$2; shift 2 ;; + --out) OUT=$2; shift 2 ;; + *) echo "unknown argument: $1" >&2; exit 2 ;; + esac +done + +if [ -z "$APP" ] || [ -z "$QUERY" ] || [ -z "$FROM" ]; then + sed -n '2,13p' "$0" >&2; exit 2 +fi +case "$FROM" in ''|*[!0-9]*) echo "--from must be epoch seconds" >&2; exit 2 ;; esac +TO=${TO:-$(date +%s)} +case "$TO" in ''|*[!0-9]*) echo "--to must be epoch seconds" >&2; exit 2 ;; esac + +TOKEN=${PAPERTRAIL_API_TOKEN:-} +if [ -z "$TOKEN" ]; then + TOKEN=$(heroku config:get PAPERTRAIL_API_TOKEN -a "$APP" 2>/dev/null || true) +fi +if [ -z "$TOKEN" ]; then + echo "no PAPERTRAIL_API_TOKEN on $APP (is the papertrail addon installed?)" >&2 + exit 3 +fi + +ENC_Q=$(jq -rn --arg q "$QUERY" '$q|@uri') +TMP=$(mktemp) HDRS=$(mktemp) +trap 'rm -f "$TMP" "$HDRS"' EXIT + +BASE="https://papertrailapp.com/api/v1/events/search.json" +MAX_ID="" FETCHED=0 PAGES=0 RETRIES=0 + +while :; do + URL="$BASE?q=$ENC_Q&min_time=$FROM&max_time=$TO&limit=1000&tail=false" + [ -n "$MAX_ID" ] && URL="$URL&max_id=$MAX_ID" + + HTTP_CODE=$(curl -sS -o "$TMP.page" -D "$HDRS" -w '%{http_code}' \ + -H "X-Papertrail-Token: $TOKEN" "$URL") + + if [ "$HTTP_CODE" = "429" ]; then + RETRIES=$((RETRIES + 1)) + [ $RETRIES -gt 5 ] && { echo "rate-limited 5 times in a row; giving up" >&2; exit 4; } + WAIT=$(awk -F': *' 'tolower($1)=="retry-after" {print int($2)}' "$HDRS") + sleep "${WAIT:-5}" + continue + fi + RETRIES=0 + if [ "$HTTP_CODE" != "200" ]; then + echo "papertrail API returned HTTP $HTTP_CODE: $(head -c 300 "$TMP.page")" >&2 + exit 4 + fi + + N=$(jq '.events | length' "$TMP.page" 2>/dev/null) || { + echo "unparseable response: $(head -c 300 "$TMP.page")" >&2; exit 4; } + jq -r '.events[] | [(.id|tostring), .received_at, .source_name, (.program // ""), .message] | @tsv' \ + "$TMP.page" >> "$TMP" + + FETCHED=$((FETCHED + N)) PAGES=$((PAGES + 1)) + printf '\rpage %d, %d events' "$PAGES" "$FETCHED" >&2 + + REACHED=$(jq -r '.reached_beginning // false' "$TMP.page") + MIN_ID=$(jq -r '.min_id // empty' "$TMP.page") + if [ "$REACHED" = "true" ] || [ -z "$MIN_ID" ] || [ "$N" -eq 0 ]; then + break + fi + if [ "$FETCHED" -ge "$MAX_EVENTS" ]; then + printf '\nhit --max-events cap (%d) — window not fully covered; narrow --from/--to\n' \ + "$MAX_EVENTS" >&2 + break + fi + MAX_ID="$MIN_ID" + sleep 0.3 +done +rm -f "$TMP.page" +printf '\nfetched %d events in %d pages\n' "$FETCHED" "$PAGES" >&2 + +# Dedupe on id (page boundaries can overlap), oldest-first, then drop the id column. +if [ -n "$OUT" ]; then + sort -t "$(printf '\t')" -k1,1n -u "$TMP" | cut -f2- > "$OUT" + echo "wrote $(wc -l < "$OUT" | tr -d ' ') lines to $OUT" >&2 +else + sort -t "$(printf '\t')" -k1,1n -u "$TMP" | cut -f2- +fi From c8f006795a6bf536500d1303b0134f2f3fdd6dc0 Mon Sep 17 00:00:00 2001 From: Mark Kraemer Date: Tue, 21 Jul 2026 16:08:55 -0400 Subject: [PATCH 2/5] Sentry standard skill --- skills/sentry-standard/SKILL.md | 164 +++++++++++++++ .../references/alert_templates.md | 73 +++++++ .../references/dashboard_template.json | 194 ++++++++++++++++++ 3 files changed, 431 insertions(+) create mode 100644 skills/sentry-standard/SKILL.md create mode 100644 skills/sentry-standard/references/alert_templates.md create mode 100644 skills/sentry-standard/references/dashboard_template.json diff --git a/skills/sentry-standard/SKILL.md b/skills/sentry-standard/SKILL.md new file mode 100644 index 0000000..af2e042 --- /dev/null +++ b/skills/sentry-standard/SKILL.md @@ -0,0 +1,164 @@ +--- +name: sentry-standard +description: Audit or provision a Rails app's Sentry-side setup against the RoleModel standard — dashboards, alert rules, team assignment, cron monitor validation, uptime monitors. Use when the user asks to "set up sentry" for an app, "audit sentry", "check sentry setup", or bring a project up to the Sentry standard. Repo-side SDK config is NOT this skill's job (that's the rolemodel_rails sentry generator). +--- + +# RoleModel Sentry Standard — audit & provision + +Bring one Sentry project up to the org standard. This skill covers only what the +`rolemodel_rails` Sentry generator **cannot** do (it owns gems, initializers, user +context, and sourcemap upload — if those are missing, tell the user to run +`rails g rolemodel:sentry` instead of fixing them here): + +1. Project + team setup +2. Alert rules (new issue, regression, error spike → team Slack channel) +3. The ` SHM` dashboard +4. Cron monitor validation +5. Uptime monitor validation + +**Org constants:** organization `rolemodel-software`, region `https://us.sentry.io`. + +## How to talk to Sentry + +Use the `sentry` CLI (https://cli.sentry.dev — verify with `sentry auth status`): + +- Reads: `sentry project list|view`, `sentry team list`, `sentry dashboard list|view`, + `sentry alert issues|metrics list|view`, `sentry monitor list` +- Writes: `sentry alert issues|metrics create` (supports `--dry-run`), + `sentry dashboard create` + `sentry dashboard widget add` +- Anything without a dedicated command (uptime monitors, posting a full dashboard + JSON): `sentry api ` — endpoint is relative to `/api/0/`, auth is + automatic. `sentry schema --search ` finds endpoint shapes. +- Append `--json` to any command when you need to parse the output. + +CLI gotchas (learned on the c12 pilot): +- `sentry monitor list /` does NOT filter by project — list org-wide + (response shape `{"data": [...]}`) and filter on `.project.slug` yourself. +- `sentry dashboard view` needs the org as a separate arg: `sentry dashboard view rolemodel-software/ `. +- `sentry alert metrics create` can 403 for member-role users (org restricts + metric-alert creation). If it does, classify 2c as 📋 for an org admin. + +The Sentry MCP tools work as an alternative when the CLI isn't installed. + +## Step 0 — Gather inputs + +Determine, asking the user only for what can't be derived: + +- **Project slug + numeric ID** — match the repo to a project via `find_projects`. + If ambiguous, ask. +- **App display name** — for the dashboard title (` SHM`). +- **Team + Slack channel** — teams map 1:1 to Linear teams (some partners have one + team covering several apps). Find the Sentry team via `find_teams`; ask the user + for the Slack channel name. +- **PII flag** — read `config/initializers/sentry.rb` in the repo: + `send_default_pii = true` means user-based widgets are allowed (see dashboard step). +- **Production URL** — for the uptime check; usually derivable from the repo + (README, `production.rb` hosts) — confirm with the user if guessing. +- **Scheduled jobs** — enumerate from the repo: GoodJob cron config + (`config.good_job.cron` in application.rb/initializers), `config/recurring.yml` + (Solid Queue), `sidekiq-cron`/`sidekiq-scheduler` YAML, `whenever` schedule.rb, + and any job classes including `Sentry::Cron::MonitorCheckIns` + (grep `sentry_monitor_check_ins`). + +## Step 1 — Audit + +Check each item and classify: ✅ compliant · 🔧 auto-fixable · 📋 needs a human. + +| # | Check | How | +|---|-------|-----| +| 1a | Project exists | `sentry project list rolemodel-software/ --json` | +| 1b | Project assigned to the right team | `sentry project view rolemodel-software/ --json`; fix via `sentry api projects/rolemodel-software//teams// -X POST` | +| 2a | Issue alert: new issue → team Slack | `sentry alert issues list rolemodel-software/ --json`; inspect actions target the right channel | +| 2b | Issue alert: regression → team Slack | same | +| 2c | Metric alert: error-volume spike | `sentry alert metrics list rolemodel-software --json`; threshold ≈ 10× the app's average hourly error count over 30d (min 50) — get the baseline with `sentry explore` or `sentry event list` | +| 3a | Dashboard ` SHM` exists | `sentry dashboard list rolemodel-software/ --json` | +| 3b | Dashboard widgets match the template | `sentry dashboard view --json`, diff against `references/dashboard_template.json` (widget titles + queries; layout drift is fine) | +| 4a | Every scheduled job in the repo has a cron monitor | repo job list vs `sentry monitor list rolemodel-software/ --json` | +| 4b | Every monitor is active/green and creates issues on failure | monitor details: status ok, `failure_issue_threshold` set (`sentry api organizations/rolemodel-software/monitors//`) | +| 4c | No orphan monitors (monitor exists, job deleted from repo) | reverse diff — report, don't auto-delete | +| 5a | Uptime monitor exists for the production URL and is active | `sentry api organizations/rolemodel-software/uptime/` (confirm shape with `sentry schema --search uptime` first) | + +Present the results as a table **before fixing anything**, then confirm with the user +which 🔧 items to apply (one confirmation for the batch, not per item). + +## Step 2 — Fix + +- **Alerts**: create missing rules with `sentry alert issues create` / + `sentry alert metrics create` per `references/alert_templates.md` — run with + `--dry-run` first, then for real. Cron/uptime failures surface as issues, so rule + 2a is what routes them to Slack — never add an issue-category filter to it. +- **Dashboard**: substitute `{{APP_NAME}}` and `{{PROJECT_ID}}` into + `references/dashboard_template.json`, then post the whole thing: + `sentry api organizations/rolemodel-software/dashboards/ -X POST --input `. + (One `sentry api` call, not 16 `dashboard widget add` calls — several widgets are + multi-series, which `widget add` can't express.) If the app has PII enabled, + replace the "Errors by Transaction" widget with "Affected Users" (line, + error-events, two queries: `has:user.email` and `!has:user.email`, both + `count_unique(user)`). If a dashboard exists but drifts, PUT the merged widget + list back the same way — **add** missing widgets; ask before removing custom + ones someone added deliberately. +- **Cron monitors**: missing monitors are a repo problem (job lacks + `sentry_monitor_check_ins`) — never create monitors API-side for jobs that never + check in. Instead, provision them in code (see "Cron provisioning" below), with + user confirmation on the diff. Monitors that exist but have no + `failure_issue_threshold` → 🔧 update via API. +- **Uptime**: create the monitor if missing (interval 60s, checking the production + URL) after confirming the URL with the user. + +### Cron provisioning (repo-side, confirmation required) + +No API calls needed: `sentry_monitor_check_ins` **upserts the monitor in Sentry on +the job's first check-in** — write the code, and the monitors appear after deploy. + +Keep the schedule config as the **single source of truth** — don't scatter crontab +strings into job classes. For GoodJob (reference implementation: +`c12_core/config/initializers/good_job.rb` + `app/jobs/application_job.rb`): + +1. `include Sentry::Cron::MonitorCheckIns` once in `ApplicationJob` (inert until + the macro is called). + +2. Below the cron hash in the GoodJob initializer, derive every monitor from it: + + ```ruby + Rails.application.config.to_prepare do + next unless Sentry.initialized? && Sentry.configuration.environment == 'production' + + Rails.application.config.good_job[:cron].each do |key, entry| + entry[:class].constantize.sentry_monitor_check_ins( + slug: key.to_s.dasherize, + monitor_config: Sentry::Cron::MonitorConfig.from_crontab(entry[:cron], checkin_margin: 5) + ) + end + end + ``` + + The entry key becomes the monitor slug; schedule changes in the cron hash flow + to Sentry automatically (upsert on next check-in). + +3. This requires each cron entry to map to a **distinct job class** (the macro is + class-level). For a class scheduled twice, create empty per-schedule subclasses + (e.g. `class HourlyBlazerChecksJob < BlazerChecksJob; end`) and point the cron + entries at them — kwargs keep working since subclasses inherit `perform`. + + For Solid Queue (`config/recurring.yml`) the same pattern applies: iterate the + parsed YAML in a `to_prepare` block. + +4. The production-only guard above is **required**, not optional: each monitor + costs $1/month, and staging apps typically run crons under + `RAILS_ENV=production`, so gate on the **Sentry** environment (set via + `SENTRY_ENVIRONMENT`), never `Rails.env`. + +5. Show the user the full diff and get confirmation **before** writing. After + applying, run the test suite and verify both sides of the gate: + `SENTRY_ENVIRONMENT=production bin/rails runner 'Rails.application.config.good_job[:cron].each { |k, e| puts "#{k}: #{e[:class].constantize.sentry_monitor_config.schedule.value}" }'` + (and confirm `sentry_monitor_config` is nil without the env var). + +6. Tell the user monitors appear in Sentry after the next deploy + first scheduled + run. Schedule the follow-up: re-run this skill's audit afterward to set each new + monitor's `failure_issue_threshold` (step 4b) — `from_crontab` can't set it. + +## Step 3 — Report + +End with: the check table (final state), links to the dashboard and alert pages, +and a short 📋 list of anything requiring a human (Heroku dyno metadata, generator +re-run, Slack channel creation, orphan monitors to consider deleting). diff --git a/skills/sentry-standard/references/alert_templates.md b/skills/sentry-standard/references/alert_templates.md new file mode 100644 index 0000000..9e4cdbe --- /dev/null +++ b/skills/sentry-standard/references/alert_templates.md @@ -0,0 +1,73 @@ +# Alert rule templates (RoleModel Sentry Standard v1) + +All alerts route to the app's team Slack channel (teams map to Linear teams; ask the user +for the channel if unknown). Create rules with the `sentry` CLI — always run with +`--dry-run` first. If a payload is rejected, check the current shape with +`sentry schema --search 'alert'`. + +## Finding the Slack integration ID (needed by every action) + +```sh +sentry api 'organizations/rolemodel-software/integrations/?provider_key=slack' --json +``` + +Use the returned integration `id` as `workspace` (issue alerts) / `integrationId` +(metric alert triggers). + +## 1. New issue → Slack (issue alert) + +```sh +sentry alert issues create rolemodel-software/{PROJECT_SLUG} \ + --name 'New Issue → Slack' \ + --condition '{"id":"sentry.rules.conditions.first_seen_event.FirstSeenEventCondition"}' \ + --action '{"id":"sentry.integrations.slack.notify_action.SlackNotifyServiceAction","workspace":"{SLACK_INTEGRATION_ID}","channel":"#{TEAM_CHANNEL}","tags":""}' \ + --action-match any \ + --frequency 1440 \ + --environment production \ + --owner 'team:{TEAM_ID}' +``` + +Note: do **not** add an issue-category filter — cron monitor failures and uptime +failures also surface as issues, and this rule is what routes them to Slack. + +## 2. Regression → Slack (issue alert) + +Same as rule 1 with: + +```sh + --name 'Regression → Slack' \ + --condition '{"id":"sentry.rules.conditions.regression_event.RegressionEventCondition"}' \ +``` + +## 3. Error-volume spike (metric alert) + +Compute the threshold first: average hourly error count over the last 30 days × 10 +(minimum 50). Get the baseline with `sentry explore` (dataset errors, `count()`, +period 30d) and divide by 720 hours. + +```sh +sentry alert metrics create rolemodel-software \ + --name 'Error volume spike' \ + --dataset errors \ + --query 'event.type:error' \ + --aggregate 'count()' \ + --time-window 60 \ + --environment production \ + --project {PROJECT_SLUG} \ + --trigger '{"label":"critical","alertThreshold":{10X_HOURLY_BASELINE},"actions":[{"type":"slack","targetType":"specific","targetIdentifier":"#{TEAM_CHANNEL}","integrationId":{SLACK_INTEGRATION_ID}}]}' \ + --owner 'team:{TEAM_ID}' +``` + +## 4 & 5. Cron and uptime failures + +Cron monitor failures and uptime downtime create **issues** (category `cron` / +`uptime`), so they route through rule 1 automatically. The skill's job is +validation, not new rules: + +- Every cron monitor has `failure_issue_threshold` set (1 is the standard) so + failures actually create issues. Fix: + `sentry api organizations/rolemodel-software/monitors/{MONITOR_SLUG}/ -X PUT -F 'config[failure_issue_threshold]=1'` + (merge with the existing `config` object — fetch it first, PUT the full config back). +- The uptime monitor exists and is active: + `sentry api organizations/rolemodel-software/uptime/` (verify endpoint shape with + `sentry schema --search uptime` before creating one). diff --git a/skills/sentry-standard/references/dashboard_template.json b/skills/sentry-standard/references/dashboard_template.json new file mode 100644 index 0000000..b8805fb --- /dev/null +++ b/skills/sentry-standard/references/dashboard_template.json @@ -0,0 +1,194 @@ +{ + "_comment": "RoleModel ' SHM' dashboard template. Replace {{APP_NAME}} and {{PROJECT_ID}} (numeric Sentry project ID), remove this _comment key, then: sentry api organizations/rolemodel-software/dashboards/ -X POST --input . If the app has send_default_pii=true, see SKILL.md for the 'Affected Users' widget that replaces 'Errors by Transaction'.", + "title": "{{APP_NAME}} SHM", + "projects": ["{{PROJECT_ID}}"], + "environment": ["production"], + "period": "30d", + "filters": {}, + "widgets": [ + { + "title": "Errors", + "displayType": "big_number", + "widgetType": "error-events", + "interval": "5m", + "layout": { "x": 0, "y": 0, "w": 1, "h": 1, "minH": 1 }, + "queries": [ + { "name": "", "fields": ["count()"], "aggregates": ["count()"], "columns": [], "conditions": "", "orderby": "" } + ] + }, + { + "title": "Unique Issues", + "displayType": "big_number", + "widgetType": "error-events", + "interval": "5m", + "layout": { "x": 1, "y": 0, "w": 1, "h": 1, "minH": 1 }, + "queries": [ + { "name": "", "fields": ["count_unique(issue)"], "aggregates": ["count_unique(issue)"], "columns": [], "conditions": "", "orderby": "" } + ] + }, + { + "title": "p95 Latency", + "displayType": "big_number", + "widgetType": "spans", + "interval": "5m", + "layout": { "x": 2, "y": 0, "w": 1, "h": 1, "minH": 1 }, + "queries": [ + { "name": "", "fields": ["p95(span.duration)"], "aggregates": ["p95(span.duration)"], "columns": [], "conditions": "is_transaction:true", "orderby": "" } + ] + }, + { + "title": "Throughput (epm)", + "displayType": "big_number", + "widgetType": "spans", + "interval": "5m", + "layout": { "x": 3, "y": 0, "w": 1, "h": 1, "minH": 1 }, + "queries": [ + { "name": "", "fields": ["epm()"], "aggregates": ["epm()"], "columns": [], "conditions": "is_transaction:true", "orderby": "" } + ] + }, + { + "title": "Failed Requests", + "displayType": "big_number", + "widgetType": "spans", + "interval": "5m", + "layout": { "x": 4, "y": 0, "w": 1, "h": 1, "minH": 1 }, + "queries": [ + { "name": "", "fields": ["count()"], "aggregates": ["count()"], "columns": [], "conditions": "is_transaction:true !span.status:ok", "orderby": "" } + ] + }, + { + "title": "Failure Rate", + "displayType": "big_number", + "widgetType": "spans", + "interval": "5m", + "layout": { "x": 5, "y": 0, "w": 1, "h": 1, "minH": 1 }, + "queries": [ + { "name": "", "fields": ["failure_rate()"], "aggregates": ["failure_rate()"], "columns": [], "conditions": "", "orderby": "" } + ] + }, + { + "title": "Errors Over Time", + "displayType": "line", + "widgetType": "error-events", + "interval": "5m", + "layout": { "x": 0, "y": 1, "w": 3, "h": 2, "minH": 2 }, + "queries": [ + { "name": "Events", "fields": ["count()"], "aggregates": ["count()"], "columns": [], "conditions": "", "orderby": "count()" } + ] + }, + { + "title": "Latency Over Time", + "displayType": "line", + "widgetType": "spans", + "interval": "5m", + "layout": { "x": 3, "y": 1, "w": 3, "h": 2, "minH": 2 }, + "queries": [ + { "name": "", "fields": ["p50(span.duration)", "p95(span.duration)"], "aggregates": ["p50(span.duration)", "p95(span.duration)"], "columns": [], "conditions": "is_transaction:true", "orderby": "" } + ] + }, + { + "title": "Handled vs. Unhandled", + "displayType": "line", + "widgetType": "error-events", + "interval": "5m", + "layout": { "x": 0, "y": 3, "w": 2, "h": 2, "minH": 2 }, + "queries": [ + { "name": "Handled", "fields": ["count()"], "aggregates": ["count()"], "columns": [], "conditions": "error.handled:true", "orderby": "count()" }, + { "name": "Unhandled", "fields": ["count()"], "aggregates": ["count()"], "columns": [], "conditions": "error.handled:false", "orderby": "count()" } + ] + }, + { + "title": "Errors by Transaction", + "displayType": "table", + "widgetType": "error-events", + "interval": "1h", + "layout": { "x": 2, "y": 3, "w": 2, "h": 2, "minH": 2 }, + "queries": [ + { "name": "", "fields": ["transaction", "count()", "count_unique(issue)"], "aggregates": ["count()", "count_unique(issue)"], "columns": ["transaction"], "conditions": "", "orderby": "-count()" } + ] + }, + { + "title": "Throughput Over Time", + "displayType": "area", + "widgetType": "spans", + "interval": "5m", + "layout": { "x": 4, "y": 3, "w": 2, "h": 2, "minH": 2 }, + "queries": [ + { "name": "", "fields": ["epm()"], "aggregates": ["epm()"], "columns": [], "conditions": "is_transaction:true", "orderby": "" } + ] + }, + { + "title": "Endpoints - Response Time & Throughput", + "displayType": "table", + "widgetType": "spans", + "interval": "5m", + "layout": { "x": 0, "y": 5, "w": 6, "h": 4, "minH": 2 }, + "queries": [ + { + "name": "", + "fields": ["transaction", "count()", "epm()", "p50(span.duration)", "p75(span.duration)", "p95(span.duration)", "p99(span.duration)"], + "aggregates": ["count()", "epm()", "p50(span.duration)", "p75(span.duration)", "p95(span.duration)", "p99(span.duration)"], + "columns": ["transaction"], + "conditions": "is_transaction:true", + "orderby": "-p95(span.duration)" + } + ] + }, + { + "title": "Slowest DB Queries (by total time)", + "displayType": "table", + "widgetType": "spans", + "interval": "5m", + "layout": { "x": 0, "y": 9, "w": 6, "h": 4, "minH": 2 }, + "queries": [ + { + "name": "", + "fields": ["span.description", "count()", "avg(span.duration)", "sum(span.duration)"], + "aggregates": ["count()", "avg(span.duration)", "sum(span.duration)"], + "columns": ["span.description"], + "conditions": "span.op:db*", + "orderby": "-sum(span.duration)" + } + ] + }, + { + "title": "Where Time Is Spent (by operation)", + "displayType": "bar", + "widgetType": "spans", + "interval": "5m", + "limit": 10, + "layout": { "x": 0, "y": 13, "w": 6, "h": 3, "minH": 2 }, + "queries": [ + { "name": "", "fields": ["span.op", "sum(span.duration)"], "aggregates": ["sum(span.duration)"], "columns": ["span.op"], "conditions": "", "orderby": "-sum(span.duration)" } + ] + }, + { + "title": "Transactions by Failure Rate", + "displayType": "table", + "widgetType": "spans", + "interval": "5m", + "layout": { "x": 0, "y": 16, "w": 3, "h": 4, "minH": 2 }, + "queries": [ + { + "name": "", + "fields": ["transaction", "failure_rate()", "count()"], + "aggregates": ["failure_rate()", "count()"], + "columns": ["transaction"], + "conditions": "is_transaction:true", + "orderby": "-failure_rate()" + } + ] + }, + { + "title": "Issues For Review", + "displayType": "table", + "widgetType": "issue", + "interval": "5m", + "layout": { "x": 3, "y": 16, "w": 3, "h": 4, "minH": 2 }, + "description": "Most recently seen unresolved issues for review.", + "queries": [ + { "name": "", "fields": ["issue", "assignee", "events", "title"], "aggregates": [], "columns": ["issue", "assignee", "events", "title"], "conditions": "is:unresolved is:for_review", "orderby": "date" } + ] + } + ] +} From 1a9e99328f3e3d2111db1a86757ca43ed2a6ef14 Mon Sep 17 00:00:00 2001 From: Mark Kraemer Date: Thu, 23 Jul 2026 13:18:39 -0400 Subject: [PATCH 3/5] Dashboard template setup --- skills/sentry-standard/SKILL.md | 39 +++++- .../references/dashboard_template.json | 111 +++++------------- 2 files changed, 66 insertions(+), 84 deletions(-) diff --git a/skills/sentry-standard/SKILL.md b/skills/sentry-standard/SKILL.md index af2e042..deaebd0 100644 --- a/skills/sentry-standard/SKILL.md +++ b/skills/sentry-standard/SKILL.md @@ -37,6 +37,17 @@ CLI gotchas (learned on the c12 pilot): - `sentry dashboard view` needs the org as a separate arg: `sentry dashboard view rolemodel-software/ `. - `sentry alert metrics create` can 403 for member-role users (org restricts metric-alert creation). If it does, classify 2c as 📋 for an org admin. +- `sentry dashboard list` paginates — a dashboard can be absent from the first + page. To find one by title, use + `sentry api 'organizations/rolemodel-software/dashboards/?query='`. +- Uptime and cron failures both have `issue.category:outage` + (types `uptime_domain_failure` / `monitor_check_in_failure`). The legacy + `uptime` and `cron` category values still parse without error but match + nothing — a widget or search using them silently returns empty. +- Dashboards have no uptime dataset (valid widgetTypes as of 2026-07: + error-events, spans, issue, logs, discover, metrics, transaction-like), so + uptime **percentage** can't be a widget — pull it from the API instead (see + Step 3). The Sentry MCP tools work as an alternative when the CLI isn't installed. @@ -90,13 +101,20 @@ which 🔧 items to apply (one confirmation for the batch, not per item). - **Dashboard**: substitute `{{APP_NAME}}` and `{{PROJECT_ID}}` into `references/dashboard_template.json`, then post the whole thing: `sentry api organizations/rolemodel-software/dashboards/ -X POST --input <file>`. - (One `sentry api` call, not 16 `dashboard widget add` calls — several widgets are + (One `sentry api` call, not 12 `dashboard widget add` calls — several widgets are multi-series, which `widget add` can't express.) If the app has PII enabled, - replace the "Errors by Transaction" widget with "Affected Users" (line, - error-events, two queries: `has:user.email` and `!has:user.email`, both - `count_unique(user)`). If a dashboard exists but drifts, PUT the merged widget + **add** an "Affected Users" widget (line, error-events, two queries: + `has:user.email` and `!has:user.email`, both `count_unique(user)`) alongside + "Errors by Transaction". If a dashboard exists but drifts, PUT the merged widget list back the same way — **add** missing widgets; ask before removing custom - ones someone added deliberately. + ones someone added deliberately. Exception: widgets from the pre-July-2026 + template revision — Throughput (epm), Failed Requests, Handled vs. Unhandled, + Throughput Over Time, Where Time Is Spent (by operation), Transactions by + Failure Rate — are deprecated, not custom; remove them as part of the confirmed + fix batch. (The current widget set is data-driven: it keeps only what SHM report + writers actually cited across seven months of Almanac health report summaries, + and adds issue-widgets for uptime/cron failures and performance issues, the two + most-written-about topics that previously required leaving the dashboard.) - **Cron monitors**: missing monitors are a repo problem (job lacks `sentry_monitor_check_ins`) — never create monitors API-side for jobs that never check in. Instead, provision them in code (see "Cron provisioning" below), with @@ -162,3 +180,14 @@ strings into job classes. For GoodJob (reference implementation: End with: the check table (final state), links to the dashboard and alert pages, and a short 📋 list of anything requiring a human (Heroku dyno metadata, generator re-run, Slack channel creation, orphan monitors to consider deleting). + +Include the app's 30-day uptime percentage (it can't live on the dashboard — no +uptime dataset exists for widgets). Get the monitor id from +`sentry api 'organizations/rolemodel-software/uptime/'`, then: + +``` +sentry api 'organizations/rolemodel-software/uptime-summary/?uptimeDetectorId=<id>&statsPeriod=30d' +``` + +uptime % = 1 − downtimeChecks/totalChecks. Link the monitor page next to the +dashboard link — it's the SHM uptime section's source of truth. diff --git a/skills/sentry-standard/references/dashboard_template.json b/skills/sentry-standard/references/dashboard_template.json index b8805fb..0805386 100644 --- a/skills/sentry-standard/references/dashboard_template.json +++ b/skills/sentry-standard/references/dashboard_template.json @@ -1,5 +1,5 @@ { - "_comment": "RoleModel '<App> SHM' dashboard template. Replace {{APP_NAME}} and {{PROJECT_ID}} (numeric Sentry project ID), remove this _comment key, then: sentry api organizations/rolemodel-software/dashboards/ -X POST --input <file>. If the app has send_default_pii=true, see SKILL.md for the 'Affected Users' widget that replaces 'Errors by Transaction'.", + "_comment": "RoleModel '<App> SHM' dashboard template. Replace {{APP_NAME}} and {{PROJECT_ID}} (numeric Sentry project ID), remove this _comment key, then: sentry api organizations/rolemodel-software/dashboards/ -X POST --input <file>. Widget set is derived from what SHM report writers actually cite (Almanac product health report summaries, Jan-Jul 2026). If the app has send_default_pii=true, see SKILL.md for the additional 'Affected Users' widget.", "title": "{{APP_NAME}} SHM", "projects": ["{{PROJECT_ID}}"], "environment": ["production"], @@ -36,32 +36,12 @@ { "name": "", "fields": ["p95(span.duration)"], "aggregates": ["p95(span.duration)"], "columns": [], "conditions": "is_transaction:true", "orderby": "" } ] }, - { - "title": "Throughput (epm)", - "displayType": "big_number", - "widgetType": "spans", - "interval": "5m", - "layout": { "x": 3, "y": 0, "w": 1, "h": 1, "minH": 1 }, - "queries": [ - { "name": "", "fields": ["epm()"], "aggregates": ["epm()"], "columns": [], "conditions": "is_transaction:true", "orderby": "" } - ] - }, - { - "title": "Failed Requests", - "displayType": "big_number", - "widgetType": "spans", - "interval": "5m", - "layout": { "x": 4, "y": 0, "w": 1, "h": 1, "minH": 1 }, - "queries": [ - { "name": "", "fields": ["count()"], "aggregates": ["count()"], "columns": [], "conditions": "is_transaction:true !span.status:ok", "orderby": "" } - ] - }, { "title": "Failure Rate", "displayType": "big_number", "widgetType": "spans", "interval": "5m", - "layout": { "x": 5, "y": 0, "w": 1, "h": 1, "minH": 1 }, + "layout": { "x": 3, "y": 0, "w": 1, "h": 1, "minH": 1 }, "queries": [ { "name": "", "fields": ["failure_rate()"], "aggregates": ["failure_rate()"], "columns": [], "conditions": "", "orderby": "" } ] @@ -87,34 +67,46 @@ ] }, { - "title": "Handled vs. Unhandled", - "displayType": "line", - "widgetType": "error-events", + "title": "Issues For Review", + "displayType": "table", + "widgetType": "issue", "interval": "5m", - "layout": { "x": 0, "y": 3, "w": 2, "h": 2, "minH": 2 }, + "layout": { "x": 0, "y": 3, "w": 3, "h": 4, "minH": 2 }, + "description": "Most recently seen unresolved issues for review.", "queries": [ - { "name": "Handled", "fields": ["count()"], "aggregates": ["count()"], "columns": [], "conditions": "error.handled:true", "orderby": "count()" }, - { "name": "Unhandled", "fields": ["count()"], "aggregates": ["count()"], "columns": [], "conditions": "error.handled:false", "orderby": "count()" } + { "name": "", "fields": ["issue", "assignee", "events", "title"], "aggregates": [], "columns": ["issue", "assignee", "events", "title"], "conditions": "is:unresolved is:for_review", "orderby": "date" } ] }, { - "title": "Errors by Transaction", + "title": "Uptime & Cron Failures", "displayType": "table", - "widgetType": "error-events", - "interval": "1h", - "layout": { "x": 2, "y": 3, "w": 2, "h": 2, "minH": 2 }, + "widgetType": "issue", + "interval": "5m", + "layout": { "x": 3, "y": 3, "w": 3, "h": 4, "minH": 2 }, + "description": "Downtime incidents and failing scheduled jobs this period (answers the uptime and background-job report sections). Resolved incidents intentionally included. Category 'outage' covers both uptime_domain_failure and monitor_check_in_failure; the legacy 'uptime'/'cron' categories parse but match nothing.", "queries": [ - { "name": "", "fields": ["transaction", "count()", "count_unique(issue)"], "aggregates": ["count()", "count_unique(issue)"], "columns": ["transaction"], "conditions": "", "orderby": "-count()" } + { "name": "", "fields": ["issue", "assignee", "events", "title"], "aggregates": [], "columns": ["issue", "assignee", "events", "title"], "conditions": "issue.category:outage", "orderby": "date" } ] }, { - "title": "Throughput Over Time", - "displayType": "area", - "widgetType": "spans", + "title": "Performance Issues", + "displayType": "table", + "widgetType": "issue", "interval": "5m", - "layout": { "x": 4, "y": 3, "w": 2, "h": 2, "minH": 2 }, + "layout": { "x": 0, "y": 7, "w": 3, "h": 3, "minH": 2 }, + "description": "N+1 queries and other performance problems detected by Sentry, most frequent first.", "queries": [ - { "name": "", "fields": ["epm()"], "aggregates": ["epm()"], "columns": [], "conditions": "is_transaction:true", "orderby": "" } + { "name": "", "fields": ["issue", "assignee", "events", "title"], "aggregates": [], "columns": ["issue", "assignee", "events", "title"], "conditions": "is:unresolved issue.category:performance", "orderby": "freq" } + ] + }, + { + "title": "Errors by Transaction", + "displayType": "table", + "widgetType": "error-events", + "interval": "1h", + "layout": { "x": 3, "y": 7, "w": 3, "h": 3, "minH": 2 }, + "queries": [ + { "name": "", "fields": ["transaction", "count()", "count_unique(issue)"], "aggregates": ["count()", "count_unique(issue)"], "columns": ["transaction"], "conditions": "", "orderby": "-count()" } ] }, { @@ -122,7 +114,7 @@ "displayType": "table", "widgetType": "spans", "interval": "5m", - "layout": { "x": 0, "y": 5, "w": 6, "h": 4, "minH": 2 }, + "layout": { "x": 0, "y": 10, "w": 6, "h": 4, "minH": 2 }, "queries": [ { "name": "", @@ -139,7 +131,7 @@ "displayType": "table", "widgetType": "spans", "interval": "5m", - "layout": { "x": 0, "y": 9, "w": 6, "h": 4, "minH": 2 }, + "layout": { "x": 0, "y": 14, "w": 6, "h": 4, "minH": 2 }, "queries": [ { "name": "", @@ -150,45 +142,6 @@ "orderby": "-sum(span.duration)" } ] - }, - { - "title": "Where Time Is Spent (by operation)", - "displayType": "bar", - "widgetType": "spans", - "interval": "5m", - "limit": 10, - "layout": { "x": 0, "y": 13, "w": 6, "h": 3, "minH": 2 }, - "queries": [ - { "name": "", "fields": ["span.op", "sum(span.duration)"], "aggregates": ["sum(span.duration)"], "columns": ["span.op"], "conditions": "", "orderby": "-sum(span.duration)" } - ] - }, - { - "title": "Transactions by Failure Rate", - "displayType": "table", - "widgetType": "spans", - "interval": "5m", - "layout": { "x": 0, "y": 16, "w": 3, "h": 4, "minH": 2 }, - "queries": [ - { - "name": "", - "fields": ["transaction", "failure_rate()", "count()"], - "aggregates": ["failure_rate()", "count()"], - "columns": ["transaction"], - "conditions": "is_transaction:true", - "orderby": "-failure_rate()" - } - ] - }, - { - "title": "Issues For Review", - "displayType": "table", - "widgetType": "issue", - "interval": "5m", - "layout": { "x": 3, "y": 16, "w": 3, "h": 4, "minH": 2 }, - "description": "Most recently seen unresolved issues for review.", - "queries": [ - { "name": "", "fields": ["issue", "assignee", "events", "title"], "aggregates": [], "columns": ["issue", "assignee", "events", "title"], "conditions": "is:unresolved is:for_review", "orderby": "date" } - ] } ] } From 47200fdcf20b9dae37dd0ad19e01d3cfdac95dd6 Mon Sep 17 00:00:00 2001 From: Mark Kraemer <megamarker@gmail.com> Date: Thu, 23 Jul 2026 14:38:37 -0400 Subject: [PATCH 4/5] Only need one dashboard --- skills/sentry-standard/SKILL.md | 47 ++++++++++++------- .../references/dashboard_template.json | 6 +-- 2 files changed, 32 insertions(+), 21 deletions(-) diff --git a/skills/sentry-standard/SKILL.md b/skills/sentry-standard/SKILL.md index deaebd0..f7a889e 100644 --- a/skills/sentry-standard/SKILL.md +++ b/skills/sentry-standard/SKILL.md @@ -12,10 +12,18 @@ context, and sourcemap upload — if those are missing, tell the user to run 1. Project + team setup 2. Alert rules (new issue, regression, error spike → team Slack channel) -3. The `<App> SHM` dashboard +3. The shared `RoleModel SHM` dashboard (one org-wide dashboard, not per-app) 4. Cron monitor validation 5. Uptime monitor validation +**One dashboard for the whole org.** A Sentry dashboard's project filter is +switched at view time, so a single `RoleModel SHM` dashboard serves every app — +open it, pick the project (and environment) from the selector, and the widgets +re-scope. This skill ensures that one dashboard exists and matches the template; +it does **not** create a dashboard per app. (This replaces the earlier per-app +`<App> SHM` model — if you find leftover per-app SHM dashboards, treat them as +deprecated and offer to delete them once the shared one is in place.) + **Org constants:** organization `rolemodel-software`, region `https://us.sentry.io`. ## How to talk to Sentry @@ -56,13 +64,11 @@ The Sentry MCP tools work as an alternative when the CLI isn't installed. Determine, asking the user only for what can't be derived: - **Project slug + numeric ID** — match the repo to a project via `find_projects`. - If ambiguous, ask. -- **App display name** — for the dashboard title (`<App> SHM`). + If ambiguous, ask. (Used for alerts, cron/uptime monitors — the dashboard is + org-wide and not scoped to this project.) - **Team + Slack channel** — teams map 1:1 to Linear teams (some partners have one team covering several apps). Find the Sentry team via `find_teams`; ask the user for the Slack channel name. -- **PII flag** — read `config/initializers/sentry.rb` in the repo: - `send_default_pii = true` means user-based widgets are allowed (see dashboard step). - **Production URL** — for the uptime check; usually derivable from the repo (README, `production.rb` hosts) — confirm with the user if guessing. - **Scheduled jobs** — enumerate from the repo: GoodJob cron config @@ -82,8 +88,8 @@ Check each item and classify: ✅ compliant · 🔧 auto-fixable · 📋 needs a | 2a | Issue alert: new issue → team Slack | `sentry alert issues list rolemodel-software/<slug> --json`; inspect actions target the right channel | | 2b | Issue alert: regression → team Slack | same | | 2c | Metric alert: error-volume spike | `sentry alert metrics list rolemodel-software --json`; threshold ≈ 10× the app's average hourly error count over 30d (min 50) — get the baseline with `sentry explore` or `sentry event list` | -| 3a | Dashboard `<App> SHM` exists | `sentry dashboard list rolemodel-software/ --json` | -| 3b | Dashboard widgets match the template | `sentry dashboard view <id> --json`, diff against `references/dashboard_template.json` (widget titles + queries; layout drift is fine) | +| 3a | Shared `RoleModel SHM` dashboard exists (org-wide, one for all apps) | `sentry api 'organizations/rolemodel-software/dashboards/?query=RoleModel SHM'` (dashboard list paginates — search by title) | +| 3b | Dashboard widgets match the template | `sentry dashboard view rolemodel-software/ <id> --json`, diff against `references/dashboard_template.json` (widget titles + queries; layout drift is fine). Confirm its project filter is **not** pinned to one project (empty `projects`/"All Projects") so the selector can re-scope it | | 4a | Every scheduled job in the repo has a cron monitor | repo job list vs `sentry monitor list rolemodel-software/<slug> --json` | | 4b | Every monitor is active/green and creates issues on failure | monitor details: status ok, `failure_issue_threshold` set (`sentry api organizations/rolemodel-software/monitors/<monitor-slug>/`) | | 4c | No orphan monitors (monitor exists, job deleted from repo) | reverse diff — report, don't auto-delete | @@ -98,16 +104,18 @@ which 🔧 items to apply (one confirmation for the batch, not per item). `sentry alert metrics create` per `references/alert_templates.md` — run with `--dry-run` first, then for real. Cron/uptime failures surface as issues, so rule 2a is what routes them to Slack — never add an issue-category filter to it. -- **Dashboard**: substitute `{{APP_NAME}}` and `{{PROJECT_ID}}` into - `references/dashboard_template.json`, then post the whole thing: +- **Dashboard**: create the shared `RoleModel SHM` dashboard **once for the whole + org** — only if 3a found none. Post `references/dashboard_template.json` as-is + (no substitution — it has a fixed title and an empty project filter so the + selector re-scopes it per app): `sentry api organizations/rolemodel-software/dashboards/ -X POST --input <file>`. (One `sentry api` call, not 12 `dashboard widget add` calls — several widgets are - multi-series, which `widget add` can't express.) If the app has PII enabled, - **add** an "Affected Users" widget (line, error-events, two queries: - `has:user.email` and `!has:user.email`, both `count_unique(user)`) alongside - "Errors by Transaction". If a dashboard exists but drifts, PUT the merged widget - list back the same way — **add** missing widgets; ask before removing custom - ones someone added deliberately. Exception: widgets from the pre-July-2026 + multi-series, which `widget add` can't express.) Because it's org-wide, the PII + and per-project decisions are made at **view** time via the project selector, not + baked in — do not add a project-specific "Affected Users" widget here. If the + dashboard exists but drifts, PUT the merged widget list back the same way — + **add** missing widgets; ask before removing custom ones someone added + deliberately. Exception: widgets from the pre-July-2026 template revision — Throughput (epm), Failed Requests, Handled vs. Unhandled, Throughput Over Time, Where Time Is Spent (by operation), Transactions by Failure Rate — are deprecated, not custom; remove them as part of the confirmed @@ -177,9 +185,12 @@ strings into job classes. For GoodJob (reference implementation: ## Step 3 — Report -End with: the check table (final state), links to the dashboard and alert pages, -and a short 📋 list of anything requiring a human (Heroku dyno metadata, generator -re-run, Slack channel creation, orphan monitors to consider deleting). +End with: the check table (final state), links to the alert pages and to the +shared `RoleModel SHM` dashboard (link it with this project + production +pre-selected so the reader lands on the right view), and a short 📋 list of +anything requiring a human (Heroku dyno metadata, generator re-run, Slack channel +creation, orphan monitors to consider deleting, leftover per-app SHM dashboards +to retire). Include the app's 30-day uptime percentage (it can't live on the dashboard — no uptime dataset exists for widgets). Get the monitor id from diff --git a/skills/sentry-standard/references/dashboard_template.json b/skills/sentry-standard/references/dashboard_template.json index 0805386..7dde073 100644 --- a/skills/sentry-standard/references/dashboard_template.json +++ b/skills/sentry-standard/references/dashboard_template.json @@ -1,7 +1,7 @@ { - "_comment": "RoleModel '<App> SHM' dashboard template. Replace {{APP_NAME}} and {{PROJECT_ID}} (numeric Sentry project ID), remove this _comment key, then: sentry api organizations/rolemodel-software/dashboards/ -X POST --input <file>. Widget set is derived from what SHM report writers actually cite (Almanac product health report summaries, Jan-Jul 2026). If the app has send_default_pii=true, see SKILL.md for the additional 'Affected Users' widget.", - "title": "{{APP_NAME}} SHM", - "projects": ["{{PROJECT_ID}}"], + "_comment": "RoleModel shared 'RoleModel SHM' dashboard template — ONE org-wide dashboard for all apps, not per-app. No substitution needed: post as-is after removing this _comment key. The empty 'projects' array means 'All Projects' so the dashboard's project selector re-scopes every widget per app at view time; do the same for 'environment' (production is the default filter, chosen at view time). Create it only once per org (audit step 3a). Post: sentry api organizations/rolemodel-software/dashboards/ -X POST --input <file>. Widget set is derived from what SHM report writers actually cite (Almanac product health report summaries, Jan-Jul 2026).", + "title": "RoleModel SHM", + "projects": [], "environment": ["production"], "period": "30d", "filters": {}, From 8057cab9a0503ee76f916f1e64dde01a1276d22d Mon Sep 17 00:00:00 2001 From: Mark Kraemer <megamarker@gmail.com> Date: Fri, 24 Jul 2026 11:31:43 -0400 Subject: [PATCH 5/5] Paper trail cleanup notes --- skills/heroku-triage/SKILL.md | 8 +++-- skills/heroku-triage/references/heuristics.md | 3 +- skills/heroku-triage/references/signals.md | 30 +++++++++++++++---- .../scripts/papertrail-search.sh | 14 +++++---- 4 files changed, 42 insertions(+), 13 deletions(-) diff --git a/skills/heroku-triage/SKILL.md b/skills/heroku-triage/SKILL.md index 8b7ad1c..f4bd28c 100644 --- a/skills/heroku-triage/SKILL.md +++ b/skills/heroku-triage/SKILL.md @@ -58,8 +58,12 @@ Space-separated `key=value` tokens plus bare words, all optional except the app: 2. Resolve the app repo: use `repo=`, else check whether the cwd's git remotes or `heroku git:remote` match the app. Code-level steps degrade gracefully without it (`heroku run cat <file> -a $APP` reads one file from the slug — uses a one-off dyno). -3. Papertrail: `heroku config:get PAPERTRAIL_API_TOKEN -a $APP`. If empty, history is - limited to `heroku logs` (~1,500 lines) — say so and lower confidence accordingly. +3. Papertrail history: the legacy `heroku config:get PAPERTRAIL_API_TOKEN` no longer + works. Two ways in (signals.md §3), neither assumed — ask the user: (A) they create + a Papertrail API token and export it so `scripts/papertrail-search.sh` can query the + search API, or (B) they download logs for a date range you specify and hand you the + file to parse locally. Without either, history is limited to `heroku logs` (~1,500 + lines) — say so and lower confidence accordingly. 4. `heroku labs -a $APP | grep log-runtime-metrics`. If disabled and the investigation needs memory/CPU data, this is the first proposed change (see signals.md §4 for the restart caveat). Memory diagnosis needs a few hours of samples after enabling. diff --git a/skills/heroku-triage/references/heuristics.md b/skills/heroku-triage/references/heuristics.md index b7406bb..87ca1b2 100644 --- a/skills/heroku-triage/references/heuristics.md +++ b/skills/heroku-triage/references/heuristics.md @@ -137,7 +137,8 @@ Rules of thumb: diagnostics and followers — growing apps outgrow them operationally, not just on size. - **Papertrail plan**: check usage vs. plan in the Papertrail account (API: - `/api/v1/accounts.json`); chronic overage or truncated retention → one plan up is + `/api/v1/accounts.json`, needs the user-created API token — signals.md §3 Option A); + chronic overage or truncated retention → one plan up is usually cheap insurance for exactly the debugging this skill does. - **Scheduler vs. always-on worker**: a worker dyno that's idle outside a few scheduled jobs can often become Heroku Scheduler one-off runs (pay per second). diff --git a/skills/heroku-triage/references/signals.md b/skills/heroku-triage/references/signals.md index eaaf47d..7ac504c 100644 --- a/skills/heroku-triage/references/signals.md +++ b/skills/heroku-triage/references/signals.md @@ -46,16 +46,34 @@ right now"; every historical question goes to Papertrail. ## 3. Papertrail (history — the agent's primary time-series source) -Token: `heroku config:get PAPERTRAIL_API_TOKEN -a $APP` (provisioned by the addon). -Fetch with the bundled script — it paginates the search API, dedupes, and emits -TSV (`received_at`, `source`, `program`, `message`) oldest-first: +Papertrail holds the log history. **The old `heroku config:get PAPERTRAIL_API_TOKEN` +config var is legacy and no longer grants access** — don't rely on it. There are two +ways to get history into the agent's hands; pick whichever fits the situation, and +neither should be assumed available without asking the user: + +**Option A — user creates a Papertrail API token.** Ask the user to create an API +token for this project in Papertrail (Settings → Profile → "API token"), then export +it so the bundled script can query the search API directly. This unlocks the full +recipe cookbook below (narrow-window fetches, hourly buckets, percentiles): ```bash +export PAPERTRAIL_API_TOKEN=<token the user created> bash skills/heroku-triage/scripts/papertrail-search.sh \ --app $APP --query '"Error R14"' \ --from $(date -v-7d +%s) --out /tmp/heroku-triage/$APP/r14.tsv ``` +**Option B — user downloads logs for a date range.** If the user would rather not +create a token, ask them for the specific time window that matters (e.g. the incident +hour, plus the same hour a day earlier as a control), have them download those logs +from the Papertrail UI, and hand you the file. Parse it locally with the same +grep/awk/`cut` recipes below — they operate on TSV/plaintext regardless of how it was +obtained. Give the user a *narrow, purposeful* window rather than "all of it": a few +targeted hours answers the question and keeps the download small. + +Either way the bundled script emits (or the download should be reduced to) TSV +(`received_at`, `source`, `program`, `message`) oldest-first before the recipes run. + **Sampling strategy — do not fetch the firehose.** Router lines on a busy app run to millions per week. Instead fetch narrow windows and compare: the incident hour, the same hour yesterday (control), and the weekly peak hour. `--max-events` defaults to @@ -99,9 +117,11 @@ scripts/papertrail-search.sh --app $APP --query 'web.2' --from <step_epoch-90> - ``` Searchable retention on small Papertrail plans is about a week; older history lives in -archives: `curl -sH "X-Papertrail-Token: $TOKEN" +archives. With the user-created API token (Option A) exported as `$PAPERTRAIL_API_TOKEN`: +`curl -sH "X-Papertrail-Token: $PAPERTRAIL_API_TOKEN" https://papertrailapp.com/api/v1/archives.json` lists daily `.tsv.gz` files with -download URLs — `zgrep`/`awk` them locally, never into context. +download URLs — `zgrep`/`awk` them locally, never into context. (Under Option B, ask +the user to pull the relevant archive day from the Papertrail UI instead.) ## 4. log-runtime-metrics (the machine-readable memory/CPU source) diff --git a/skills/heroku-triage/scripts/papertrail-search.sh b/skills/heroku-triage/scripts/papertrail-search.sh index 3387cac..0f20932 100755 --- a/skills/heroku-triage/scripts/papertrail-search.sh +++ b/skills/heroku-triage/scripts/papertrail-search.sh @@ -4,9 +4,14 @@ # Emits TSV (received_at, source_name, program, message), oldest-first, deduped, # to --out FILE (or stdout). Progress and totals go to stderr. # -# Requires: curl, jq, heroku CLI (unless PAPERTRAIL_API_TOKEN is already exported). +# Requires: curl, jq, and PAPERTRAIL_API_TOKEN exported. +# +# The token is a Papertrail *API* token, created by the user in Papertrail +# (Settings -> Profile -> "API token"). The old PAPERTRAIL_API_TOKEN Heroku +# config var is legacy and no longer grants API access — do not use it. # # Usage: +# PAPERTRAIL_API_TOKEN=<token> \ # papertrail-search.sh --app <heroku-app> --query '<papertrail query>' \ # --from <epoch-seconds> [--to <epoch-seconds>] [--max-events N] [--out FILE] # @@ -35,10 +40,9 @@ case "$TO" in ''|*[!0-9]*) echo "--to must be epoch seconds" >&2; exit 2 ;; esac TOKEN=${PAPERTRAIL_API_TOKEN:-} if [ -z "$TOKEN" ]; then - TOKEN=$(heroku config:get PAPERTRAIL_API_TOKEN -a "$APP" 2>/dev/null || true) -fi -if [ -z "$TOKEN" ]; then - echo "no PAPERTRAIL_API_TOKEN on $APP (is the papertrail addon installed?)" >&2 + echo "PAPERTRAIL_API_TOKEN not set. Create a Papertrail API token in the Papertrail" >&2 + echo "UI (Settings -> Profile -> API token) and export it before running this script." >&2 + echo "The heroku PAPERTRAIL_API_TOKEN config var is legacy and does not work here." >&2 exit 3 fi