S4X5: budget threshold alerts + SLI/SLO dashboards

README.md

@@ -5,6 +5,10 @@
 > - **clickhouse-mcp-server** manifests are superseded by [`krateo-clickhouse-mcp-server-chart`](https://github.com/braghettos/krateo-clickhouse-mcp-server-chart).
 >
 > What remains here: agents (`agents/`, `kagent-overrides/`) pending the agents-versioning effort, plus blueprint-templates/demo/runbooks/docs.
+>
+> New here (built on the split-out ClickStack pipeline):
+> - **`alerts/budget/`** — budget threshold mechanism on showback records (DDL + evaluator CronJob + HyperDX alert bootstrap, in-portal + email delivery).
+> - **`dashboards/slo/`** — SLI queries (availability/latency/error-rate) + SLO dashboard & breach-alert bootstrap; reference doc in [`docs/SLO.md`](docs/SLO.md).
 
 # Krateo ClickHouse Kubernetes Observability Stack
 

alerts/budget/.env.example

@@ -0,0 +1,15 @@
+# Budget Alert Bootstrap Configuration
+# Copy to .env and fill in your values:
+#   cp .env.example .env
+
+# HyperDX base URL (e.g. http://localhost:3000 when port-forwarding)
+HYPERDX_URL=http://localhost:3000
+
+# Personal API key from HyperDX (Settings → API Keys)
+HYPERDX_API_KEY=
+
+# Webhook ID for in-portal delivery (autopilot-alert-proxy → portal notifications)
+BUDGET_PORTAL_WEBHOOK_ID=
+
+# Webhook ID for email delivery (HyperDX email integration / SMTP relay). Optional.
+BUDGET_EMAIL_WEBHOOK_ID=

alerts/budget/README.md

@@ -0,0 +1,144 @@
+# Budget Alerts — threshold rules on ShowbackRecords
+
+A **budget is a threshold on rated showback data** (per Org / Tenant / Service /
+Tag). No separate budget engine: breach detection **reuses the existing alert
+pipeline** (OTel → ClickHouse → HyperDX), delivery is **in-portal + email**
+(optional webhook / Autopilot).
+
+## How it works
+
+```
+budgets (ClickHouse table)          ← budget definitions (scope + amount + warn ratio)
+        │
+        ▼
+budget_status (ClickHouse view)     ← current-period spend vs. amount, per budget
+        │                              (joins showback_daily / showback_daily_by_tag
+        │                               produced by the showback engine)
+        ▼
+budget-evaluator (CronJob)          ← every 15m, logs one JSON line per budget
+        │                              in warning/breached state (stdout)
+        ▼
+OTel DaemonSet → otel_logs          ← standard log collection, nothing new
+        ▼
+HyperDX saved search + alert        ← same mechanism as every other alert
+        ▼
+in-portal (webhook → alert proxy → portal notifications) + email channel
+```
+
+This is exactly the **heartbeat-canary pattern** already used for pipeline
+self-monitoring: a CronJob emits structured stdout logs, the pipeline picks
+them up, HyperDX alerts on them.
+
+## Components
+
+| File | Purpose |
+|------|---------|
+| `ddl/001_budgets.sql` | `budgets` table — budget definitions (scope, period, amount, warn ratio). |
+| `ddl/002_budget_status.sql` | `budget_status` view — spend-to-date vs. amount, status `ok`/`warning`/`breached`. |
+| `budget-evaluator-cronjob.yaml` | CronJob logging warning/breached budgets as OTel-shaped JSON lines. |
+| `bootstrap-budget-alerts.sh` | Creates the HyperDX saved searches + alerts (warning + breach) via API. Exits non-zero with a FAILED summary if any step does not complete. |
+| `tests/smoke-budget-ddl.sh` | Executes the DDL against clickhouse-local (docker fallback) with sample data and asserts the `budget_status` view materializes the expected statuses. |
+| `.env.example` | Configuration template for the bootstrap script. |
+
+## Budget scoping
+
+A budget row selects a scope with empty-string wildcards:
+
+| Column | `''` means |
+|--------|------------|
+| `tenant` | whole Org |
+| `service` | all services |
+| `tag_key` | not tag-scoped (uses `showback_daily`) |
+| `tag_value` | any value of `tag_key` (uses `showback_daily_by_tag`) |
+
+`period` is `monthly` (spend since start of current month) or `daily`.
+`warn_ratio` (default `0.8`) drives the early-warning alert before the hard
+breach at `1.0`.
+
+> Budget **definitions** (actual amounts, scopes) are configuration and are
+> seeded by the consuming assembly — this repo ships only the mechanism with
+> no budget rows.
+
+## Setup
+
+1. Apply the DDL to the ClickHouse database that hosts the showback tables,
+   rendering `{{database}}` with the target database (same convention as the
+   showback engine DDL; its default database is `showback`):
+
+   ```sh
+   sed 's/{{database}}/showback/g' ddl/001_budgets.sql | clickhouse-client --multiquery
+   sed 's/{{database}}/showback/g' ddl/002_budget_status.sql | clickhouse-client --multiquery
+   ```
+
+2. Deploy the evaluator. `CLICKHOUSE_DATABASE` in the CronJob (default
+   `showback`) **must match the database the DDL was rendered with**,
+   otherwise the `budget_status` view is not found and the job fails:
+
+   ```sh
+   kubectl apply -f budget-evaluator-cronjob.yaml
+   ```
+
+3. Create the HyperDX alerts (webhooks for the in-portal channel and the
+   email channel must exist in HyperDX first). The script exits non-zero
+   with a `FAILED` summary when any saved search or alert cannot be
+   created, so a partial bootstrap never looks like success:
+
+   ```sh
+   cp .env.example .env   # fill in values
+   ./bootstrap-budget-alerts.sh
+   ```
+
+## Verifying the DDL locally
+
+`tests/smoke-budget-ddl.sh` renders `{{database}}`, executes DDL 001+002
+against a throwaway ClickHouse (clickhouse-local, or the
+`clickhouse/clickhouse-server:24.8-alpine` image via docker when no local
+binary exists), inserts sample budgets + spend, and asserts the
+`budget_status` view returns the expected `ok`/`warning`/`breached` rows —
+including ReplacingMergeTree FINAL supersede semantics, tag-scoped budgets,
+daily/monthly period windows and the `enabled` flag:
+
+```sh
+tests/smoke-budget-ddl.sh
+```
+
+## Log-line shape (D19a)
+
+The evaluator emits one JSON line per warning/breached budget following the
+OTel Logs Data Model, so a collector json parser can map it 1:1 onto an
+OTLP LogRecord while HyperDX keeps querying it as JSON in `Body`:
+
+```json
+{
+  "timestamp": "2026-01-01T12:00:00Z",
+  "trace_id": "<32 hex — one per evaluator run>",
+  "span_id": "<16 hex — one per line>",
+  "severity_text": "WARN | ERROR",
+  "severity_number": 13,
+  "body": "budget threshold crossed",
+  "attributes": {
+    "event.name": "krateo.budget.threshold",
+    "service.name": "krateo-budget-evaluator",
+    "krateo.budget.id": "...", "krateo.budget.org": "...",
+    "krateo.budget.status": "warning | breached", "...": "..."
+  }
+}
+```
+
+Severity maps `warning`→`WARN`/13 and `breached`→`ERROR`/17. The evaluator
+is a **scheduled origin**: there is no inbound trace context or baggage to
+propagate (nothing calls it), so each run mints a fresh `trace_id` shared by
+all lines of that run — the alerts of one evaluation are correlatable — and
+a per-line `span_id`. The line is built entirely inside ClickHouse
+(`toJSONString` escapes all user-supplied values); the shell never parses
+row data.
+
+## Delivery channels
+
+- **In-portal**: the alert webhook targets the autopilot-alert-proxy, which
+  forwards to the portal notification endpoint (same route as every other
+  alert shown in the portal).
+- **Email**: a second alert channel pointing at an email-integration webhook
+  (HyperDX email integration or an SMTP relay webhook).
+- **Optional**: point `BUDGET_ALERT_PROXY_WEBHOOK_ID` at the agent-routed
+  webhook to let Autopilot react to breaches.

alerts/budget/bootstrap-budget-alerts.sh

@@ -0,0 +1,174 @@
+#!/usr/bin/env bash
+# ---------------------------------------------------------------------------
+# Budget Alerts – HyperDX API Bootstrap
+#
+# Creates the budget warning + breach alerts over the log lines emitted by
+# the budget-evaluator CronJob. Reuses the standard alert pipeline: the
+# evaluator writes to stdout, OTel ships to ClickHouse `otel_logs`, HyperDX
+# fires the alert.
+#
+# Delivery:
+#   - in-portal: BUDGET_PORTAL_WEBHOOK_ID → autopilot-alert-proxy → portal
+#     notification endpoint (same route as every other in-portal alert)
+#   - email:     BUDGET_EMAIL_WEBHOOK_ID → HyperDX email integration / relay
+#
+# Usage:
+#   export HYPERDX_URL="http://localhost:3000"
+#   export HYPERDX_API_KEY="your-api-key"
+#   export BUDGET_PORTAL_WEBHOOK_ID="webhook-id"
+#   export BUDGET_EMAIL_WEBHOOK_ID="webhook-id"     # optional
+#   ./bootstrap-budget-alerts.sh
+#
+# Or use .env file:
+#   cp .env.example .env && edit .env && ./bootstrap-budget-alerts.sh
+# ---------------------------------------------------------------------------
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+[ -f "$SCRIPT_DIR/.env" ] && set -a && source "$SCRIPT_DIR/.env" && set +a
+
+HYPERDX_URL="${HYPERDX_URL:-http://localhost:3000}"
+HYPERDX_API_KEY="${HYPERDX_API_KEY:-}"
+BUDGET_PORTAL_WEBHOOK_ID="${BUDGET_PORTAL_WEBHOOK_ID:-}"
+BUDGET_EMAIL_WEBHOOK_ID="${BUDGET_EMAIL_WEBHOOK_ID:-}"
+
+API_BASE="${HYPERDX_URL%/}/api"
+
+die() { echo "[ERROR] $*" >&2; exit 1; }
+log() { echo "[bootstrap] $*"; }
+
+# Failures are collected in a file (not a counter) because the helpers run
+# inside $(...) subshells; any recorded failure makes the script exit 1 with
+# a FAILED summary, so a partial bootstrap can never look like success.
+FAIL_LOG="$(mktemp)"
+trap 'rm -f "$FAIL_LOG"' EXIT
+fail() { echo "[FAILED] $*" >&2; echo "$*" >> "$FAIL_LOG"; }
+
+[ -n "$HYPERDX_API_KEY" ]          || die "HYPERDX_API_KEY is required"
+[ -n "$BUDGET_PORTAL_WEBHOOK_ID" ] || die "BUDGET_PORTAL_WEBHOOK_ID is required (create webhook in HyperDX UI first)"
+
+# ---------------------------------------------------------------------------
+# Helper: create a saved search
+# Runs inside $(...): stdout is the returned id ONLY — log goes to stderr,
+# otherwise the log line would be captured into the saved-search id.
+# ---------------------------------------------------------------------------
+create_saved_search() {
+  local name="$1"
+  local query="$2"
+
+  log "Creating saved search: $name" >&2
+  RESP=$(curl -s -w "\n%{http_code}" -X POST "$API_BASE/v1/saved-searches" \
+    -H "Authorization: Bearer $HYPERDX_API_KEY" \
+    -H "Content-Type: application/json" \
+    -d "$(jq -n --arg name "$name" --arg query "$query" \
+      '{ name: $name, query: $query }')")
+
+  HTTP_CODE=$(echo "$RESP" | tail -n 1)
+  HTTP_BODY=$(echo "$RESP" | sed '$d')
+
+  if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ]; then
+    local id
+    id=$(echo "$HTTP_BODY" | jq -r '._id // .id // empty' 2>/dev/null)
+    if [ -n "$id" ]; then
+      echo "$id"
+    else
+      fail "saved search '$name': HTTP $HTTP_CODE but no id in response — $HTTP_BODY"
+      echo ""
+    fi
+  else
+    fail "saved search '$name': HTTP $HTTP_CODE — $HTTP_BODY (if it already exists, delete or update it in the HyperDX UI and re-run)"
+    echo ""
+  fi
+}
+
+# ---------------------------------------------------------------------------
+# Helper: create an alert on a saved search, one channel per call
+# ---------------------------------------------------------------------------
+create_alert() {
+  local name="$1"
+  local saved_search_id="$2"
+  local threshold="$3"
+  local interval="$4"
+  local webhook_id="$5"
+  local message="$6"
+
+  log "Creating alert: $name (interval: $interval)"
+  local payload
+  payload=$(jq -n \
+    --arg name "$name" \
+    --arg savedSearchId "$saved_search_id" \
+    --argjson threshold "$threshold" \
+    --arg interval "$interval" \
+    --arg webhookId "$webhook_id" \
+    --arg message "$message" \
+    --arg groupBy "JSONExtractString(Body, 'attributes', 'krateo.budget.id')" \
+    '{
+      name: $name,
+      savedSearchId: $savedSearchId,
+      threshold: $threshold,
+      threshold_type: "above",
+      interval: $interval,
+      source: "search",
+      channel: { type: "slack_webhook", webhookId: $webhookId },
+      message: $message,
+      groupBy: [$groupBy]
+    }')
+
+  RESP=$(curl -s -w "\n%{http_code}" -X POST "$API_BASE/alerts" \
+    -H "Authorization: Bearer $HYPERDX_API_KEY" \
+    -H "Content-Type: application/json" \
+    -d "$payload")
+
+  HTTP_CODE=$(echo "$RESP" | tail -n 1)
+  if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ]; then
+    log "  Created successfully."
+  else
+    HTTP_BODY=$(echo "$RESP" | sed '$d')
+    fail "alert '$name': HTTP $HTTP_CODE — $(echo "$HTTP_BODY" | jq -c . 2>/dev/null || echo "$HTTP_BODY")"
+  fi
+}
+
+# ---------------------------------------------------------------------------
+# Alert 1: Budget breached (spend >= amount)
+# ---------------------------------------------------------------------------
+log ""
+log "=== Alert 1: Budget Breached ==="
+BREACH_QUERY="ResourceAttributes['k8s.pod.labels.app'] = 'krateo-budget-evaluator' AND JSONExtractString(Body, 'attributes', 'krateo.budget.status') = 'breached'"
+
+BREACH_SS_ID=$(create_saved_search "Budget Breached" "$BREACH_QUERY")
+if [ -n "$BREACH_SS_ID" ]; then
+  MESSAGE="Budget breached: spend reached the configured budget amount for this scope."
+  create_alert "Budget Breached (portal)" "$BREACH_SS_ID" 0 "15m" "$BUDGET_PORTAL_WEBHOOK_ID" "$MESSAGE"
+  if [ -n "$BUDGET_EMAIL_WEBHOOK_ID" ]; then
+    create_alert "Budget Breached (email)" "$BREACH_SS_ID" 0 "15m" "$BUDGET_EMAIL_WEBHOOK_ID" "$MESSAGE"
+  fi
+else
+  fail "breach alerts skipped: no saved-search id. Filter: $BREACH_QUERY"
+fi
+
+# ---------------------------------------------------------------------------
+# Alert 2: Budget warning (spend >= warn_ratio * amount)
+# ---------------------------------------------------------------------------
+log ""
+log "=== Alert 2: Budget Warning ==="
+WARN_QUERY="ResourceAttributes['k8s.pod.labels.app'] = 'krateo-budget-evaluator' AND JSONExtractString(Body, 'attributes', 'krateo.budget.status') = 'warning'"
+
+WARN_SS_ID=$(create_saved_search "Budget Warning" "$WARN_QUERY")
+if [ -n "$WARN_SS_ID" ]; then
+  MESSAGE="Budget warning: spend crossed the early-warning threshold for this scope."
+  create_alert "Budget Warning (portal)" "$WARN_SS_ID" 0 "15m" "$BUDGET_PORTAL_WEBHOOK_ID" "$MESSAGE"
+  if [ -n "$BUDGET_EMAIL_WEBHOOK_ID" ]; then
+    create_alert "Budget Warning (email)" "$WARN_SS_ID" 0 "15m" "$BUDGET_EMAIL_WEBHOOK_ID" "$MESSAGE"
+  fi
+else
+  fail "warning alerts skipped: no saved-search id. Filter: $WARN_QUERY"
+fi
+
+log ""
+if [ -s "$FAIL_LOG" ]; then
+  log "Bootstrap FAILED — $(wc -l < "$FAIL_LOG" | tr -d ' ') step(s) did not complete:"
+  sed 's/^/[bootstrap]   - /' "$FAIL_LOG" >&2
+  log "The bootstrap is PARTIAL: fix the failures above and re-run."
+  exit 1
+fi
+log "Done."