From e7d59863b596d8c3b82c90cb87fdd05aafcac132 Mon Sep 17 00:00:00 2001
From: Diego Braga <diego.braga86@gmail.com>
Date: Mon, 20 Jul 2026 09:24:20 +0200
Subject: [PATCH 1/5] feat: budget threshold alerts + SLI/SLO dashboards (S4X5)

Budget mechanism (alerts/budget/): a budget is a threshold on rated
showback data per Org/Tenant/Service/Tag. Adds the budgets table +
budget_status view DDL, an evaluator CronJob that emits breach/warning
JSON log lines (heartbeat-canary pattern), and a HyperDX bootstrap
script wiring in-portal + email delivery through the existing
OTel -> ClickHouse -> HyperDX alert pipeline. Budget definitions are
seeded by the consuming assembly; only the mechanism ships here.

SLI/SLO exposure (dashboards/slo/ + docs/SLO.md): availability,
latency and error-rate SLI queries over otel_traces/otel_logs,
configurable SLO targets, error-budget burn-rate query, dashboard
layout, and an SLO breach-alert bootstrap script. docs/SLO.md holds
the reference definitions and a perf-results capture template.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 README.md                                   |   4 +
 alerts/budget/.env.example                  |  15 ++
 alerts/budget/README.md                     |  94 ++++++++++++
 alerts/budget/bootstrap-budget-alerts.sh    | 154 +++++++++++++++++++
 alerts/budget/budget-evaluator-cronjob.yaml | 101 ++++++++++++
 alerts/budget/ddl/001_budgets.sql           |  24 +++
 alerts/budget/ddl/002_budget_status.sql     |  65 ++++++++
 dashboards/slo/.env.example                 |  24 +++
 dashboards/slo/README.md                    |  54 +++++++
 dashboards/slo/bootstrap-slo-alerts.sh      | 161 ++++++++++++++++++++
 dashboards/slo/sli-queries.sql              |  89 +++++++++++
 docs/SLO.md                                 |  64 ++++++++
 12 files changed, 849 insertions(+)
 create mode 100644 alerts/budget/.env.example
 create mode 100644 alerts/budget/README.md
 create mode 100755 alerts/budget/bootstrap-budget-alerts.sh
 create mode 100644 alerts/budget/budget-evaluator-cronjob.yaml
 create mode 100644 alerts/budget/ddl/001_budgets.sql
 create mode 100644 alerts/budget/ddl/002_budget_status.sql
 create mode 100644 dashboards/slo/.env.example
 create mode 100644 dashboards/slo/README.md
 create mode 100755 dashboards/slo/bootstrap-slo-alerts.sh
 create mode 100644 dashboards/slo/sli-queries.sql
 create mode 100644 docs/SLO.md

diff --git a/README.md b/README.md
index 520ccbc..19192bb 100644
--- a/README.md
+++ b/README.md
@@ -5,6 +5,10 @@
 > - **clickhouse-mcp-server** manifests are superseded by [`krateo-clickhouse-mcp-server-chart`](https://github.com/braghettos/krateo-clickhouse-mcp-server-chart).
 >
 > What remains here: agents (`agents/`, `kagent-overrides/`) pending the agents-versioning effort, plus blueprint-templates/demo/runbooks/docs.
+>
+> New here (built on the split-out ClickStack pipeline):
+> - **`alerts/budget/`** — budget threshold mechanism on showback records (DDL + evaluator CronJob + HyperDX alert bootstrap, in-portal + email delivery).
+> - **`dashboards/slo/`** — SLI queries (availability/latency/error-rate) + SLO dashboard & breach-alert bootstrap; reference doc in [`docs/SLO.md`](docs/SLO.md).
 
 # Krateo ClickHouse Kubernetes Observability Stack
 
diff --git a/alerts/budget/.env.example b/alerts/budget/.env.example
new file mode 100644
index 0000000..12b4147
--- /dev/null
+++ b/alerts/budget/.env.example
@@ -0,0 +1,15 @@
+# Budget Alert Bootstrap Configuration
+# Copy to .env and fill in your values:
+#   cp .env.example .env
+
+# HyperDX base URL (e.g. http://localhost:3000 when port-forwarding)
+HYPERDX_URL=http://localhost:3000
+
+# Personal API key from HyperDX (Settings → API Keys)
+HYPERDX_API_KEY=
+
+# Webhook ID for in-portal delivery (autopilot-alert-proxy → portal notifications)
+BUDGET_PORTAL_WEBHOOK_ID=
+
+# Webhook ID for email delivery (HyperDX email integration / SMTP relay). Optional.
+BUDGET_EMAIL_WEBHOOK_ID=
diff --git a/alerts/budget/README.md b/alerts/budget/README.md
new file mode 100644
index 0000000..f90d78f
--- /dev/null
+++ b/alerts/budget/README.md
@@ -0,0 +1,94 @@
+# Budget Alerts — threshold rules on ShowbackRecords
+
+A **budget is a threshold on rated showback data** (per Org / Tenant / Service /
+Tag). No separate budget engine: breach detection **reuses the existing alert
+pipeline** (OTel → ClickHouse → HyperDX), delivery is **in-portal + email**
+(optional webhook / Autopilot).
+
+## How it works
+
+```
+budgets (ClickHouse table)          ← budget definitions (scope + amount + warn ratio)
+        │
+        ▼
+budget_status (ClickHouse view)     ← current-period spend vs. amount, per budget
+        │                              (joins showback_daily / showback_daily_by_tag
+        │                               produced by the showback engine)
+        ▼
+budget-evaluator (CronJob)          ← every 15m, logs one JSON line per budget
+        │                              in warning/breached state (stdout)
+        ▼
+OTel DaemonSet → otel_logs          ← standard log collection, nothing new
+        ▼
+HyperDX saved search + alert        ← same mechanism as every other alert
+        ▼
+in-portal (webhook → alert proxy → portal notifications) + email channel
+```
+
+This is exactly the **heartbeat-canary pattern** already used for pipeline
+self-monitoring: a CronJob emits structured stdout logs, the pipeline picks
+them up, HyperDX alerts on them.
+
+## Components
+
+| File | Purpose |
+|------|---------|
+| `ddl/001_budgets.sql` | `budgets` table — budget definitions (scope, period, amount, warn ratio). |
+| `ddl/002_budget_status.sql` | `budget_status` view — spend-to-date vs. amount, status `ok`/`warning`/`breached`. |
+| `budget-evaluator-cronjob.yaml` | CronJob logging warning/breached budgets as JSON lines. |
+| `bootstrap-budget-alerts.sh` | Creates the HyperDX saved searches + alerts (warning + breach) via API. |
+| `.env.example` | Configuration template for the bootstrap script. |
+
+## Budget scoping
+
+A budget row selects a scope with empty-string wildcards:
+
+| Column | `''` means |
+|--------|------------|
+| `tenant` | whole Org |
+| `service` | all services |
+| `tag_key` | not tag-scoped (uses `showback_daily`) |
+| `tag_value` | any value of `tag_key` (uses `showback_daily_by_tag`) |
+
+`period` is `monthly` (spend since start of current month) or `daily`.
+`warn_ratio` (default `0.8`) drives the early-warning alert before the hard
+breach at `1.0`.
+
+> Budget **definitions** (actual amounts, scopes) are configuration and are
+> seeded by the consuming assembly — this repo ships only the mechanism with
+> no budget rows.
+
+## Setup
+
+1. Apply the DDL to the ClickHouse database that hosts the showback tables
+   (replace `{{database}}` with the target database, same convention as the
+   showback engine DDL):
+
+   ```sh
+   clickhouse-client --multiquery < ddl/001_budgets.sql
+   clickhouse-client --multiquery < ddl/002_budget_status.sql
+   ```
+
+2. Deploy the evaluator:
+
+   ```sh
+   kubectl apply -f budget-evaluator-cronjob.yaml
+   ```
+
+3. Create the HyperDX alerts (webhooks for the in-portal channel and the
+   email channel must exist in HyperDX first):
+
+   ```sh
+   cp .env.example .env   # fill in values
+   ./bootstrap-budget-alerts.sh
+   ```
+
+## Delivery channels
+
+- **In-portal**: the alert webhook targets the autopilot-alert-proxy, which
+  forwards to the portal notification endpoint (same route as every other
+  alert shown in the portal).
+- **Email**: a second alert channel pointing at an email-integration webhook
+  (HyperDX email integration or an SMTP relay webhook).
+- **Optional**: point `BUDGET_ALERT_PROXY_WEBHOOK_ID` at the agent-routed
+  webhook to let Autopilot react to breaches.
diff --git a/alerts/budget/bootstrap-budget-alerts.sh b/alerts/budget/bootstrap-budget-alerts.sh
new file mode 100755
index 0000000..4fd8307
--- /dev/null
+++ b/alerts/budget/bootstrap-budget-alerts.sh
@@ -0,0 +1,154 @@
+#!/usr/bin/env bash
+# ---------------------------------------------------------------------------
+# Budget Alerts – HyperDX API Bootstrap
+#
+# Creates the budget warning + breach alerts over the log lines emitted by
+# the budget-evaluator CronJob. Reuses the standard alert pipeline: the
+# evaluator writes to stdout, OTel ships to ClickHouse `otel_logs`, HyperDX
+# fires the alert.
+#
+# Delivery:
+#   - in-portal: BUDGET_PORTAL_WEBHOOK_ID → autopilot-alert-proxy → portal
+#     notification endpoint (same route as every other in-portal alert)
+#   - email:     BUDGET_EMAIL_WEBHOOK_ID → HyperDX email integration / relay
+#
+# Usage:
+#   export HYPERDX_URL="http://localhost:3000"
+#   export HYPERDX_API_KEY="your-api-key"
+#   export BUDGET_PORTAL_WEBHOOK_ID="webhook-id"
+#   export BUDGET_EMAIL_WEBHOOK_ID="webhook-id"     # optional
+#   ./bootstrap-budget-alerts.sh
+#
+# Or use .env file:
+#   cp .env.example .env && edit .env && ./bootstrap-budget-alerts.sh
+# ---------------------------------------------------------------------------
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+[ -f "$SCRIPT_DIR/.env" ] && set -a && source "$SCRIPT_DIR/.env" && set +a
+
+HYPERDX_URL="${HYPERDX_URL:-http://localhost:3000}"
+HYPERDX_API_KEY="${HYPERDX_API_KEY:-}"
+BUDGET_PORTAL_WEBHOOK_ID="${BUDGET_PORTAL_WEBHOOK_ID:-}"
+BUDGET_EMAIL_WEBHOOK_ID="${BUDGET_EMAIL_WEBHOOK_ID:-}"
+
+API_BASE="${HYPERDX_URL%/}/api"
+
+die() { echo "[ERROR] $*" >&2; exit 1; }
+log() { echo "[bootstrap] $*"; }
+
+[ -n "$HYPERDX_API_KEY" ]          || die "HYPERDX_API_KEY is required"
+[ -n "$BUDGET_PORTAL_WEBHOOK_ID" ] || die "BUDGET_PORTAL_WEBHOOK_ID is required (create webhook in HyperDX UI first)"
+
+# ---------------------------------------------------------------------------
+# Helper: create a saved search
+# ---------------------------------------------------------------------------
+create_saved_search() {
+  local name="$1"
+  local query="$2"
+
+  log "Creating saved search: $name"
+  RESP=$(curl -s -w "\n%{http_code}" -X POST "$API_BASE/v1/saved-searches" \
+    -H "Authorization: Bearer $HYPERDX_API_KEY" \
+    -H "Content-Type: application/json" \
+    -d "$(jq -n --arg name "$name" --arg query "$query" \
+      '{ name: $name, query: $query }')")
+
+  HTTP_CODE=$(echo "$RESP" | tail -n 1)
+  HTTP_BODY=$(echo "$RESP" | sed '$d')
+
+  if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ]; then
+    echo "$HTTP_BODY" | jq -r '._id // .id // empty' 2>/dev/null
+  else
+    log "  Warning: saved search creation returned HTTP $HTTP_CODE (may already exist)"
+    echo ""
+  fi
+}
+
+# ---------------------------------------------------------------------------
+# Helper: create an alert on a saved search, one channel per call
+# ---------------------------------------------------------------------------
+create_alert() {
+  local name="$1"
+  local saved_search_id="$2"
+  local threshold="$3"
+  local interval="$4"
+  local webhook_id="$5"
+  local message="$6"
+
+  log "Creating alert: $name (interval: $interval)"
+  local payload
+  payload=$(jq -n \
+    --arg name "$name" \
+    --arg savedSearchId "$saved_search_id" \
+    --argjson threshold "$threshold" \
+    --arg interval "$interval" \
+    --arg webhookId "$webhook_id" \
+    --arg message "$message" \
+    '{
+      name: $name,
+      savedSearchId: $savedSearchId,
+      threshold: $threshold,
+      threshold_type: "above",
+      interval: $interval,
+      source: "search",
+      channel: { type: "slack_webhook", webhookId: $webhookId },
+      message: $message,
+      groupBy: ["budget.budget_id"]
+    }')
+
+  RESP=$(curl -s -w "\n%{http_code}" -X POST "$API_BASE/alerts" \
+    -H "Authorization: Bearer $HYPERDX_API_KEY" \
+    -H "Content-Type: application/json" \
+    -d "$payload")
+
+  HTTP_CODE=$(echo "$RESP" | tail -n 1)
+  if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ]; then
+    log "  Created successfully."
+  else
+    HTTP_BODY=$(echo "$RESP" | sed '$d')
+    log "  Warning: returned HTTP $HTTP_CODE"
+    echo "$HTTP_BODY" | jq . 2>/dev/null || echo "$HTTP_BODY"
+  fi
+}
+
+# ---------------------------------------------------------------------------
+# Alert 1: Budget breached (spend >= amount)
+# ---------------------------------------------------------------------------
+log ""
+log "=== Alert 1: Budget Breached ==="
+BREACH_QUERY="ResourceAttributes['k8s.pod.labels.app'] = 'krateo-budget-evaluator' AND JSONExtractString(Body, 'status') = 'breached'"
+
+BREACH_SS_ID=$(create_saved_search "Budget Breached" "$BREACH_QUERY")
+if [ -n "$BREACH_SS_ID" ]; then
+  MESSAGE="Budget breached: spend reached the configured budget amount for this scope."
+  create_alert "Budget Breached (portal)" "$BREACH_SS_ID" 0 "15m" "$BUDGET_PORTAL_WEBHOOK_ID" "$MESSAGE"
+  if [ -n "$BUDGET_EMAIL_WEBHOOK_ID" ]; then
+    create_alert "Budget Breached (email)" "$BREACH_SS_ID" 0 "15m" "$BUDGET_EMAIL_WEBHOOK_ID" "$MESSAGE"
+  fi
+else
+  log "  Skipping alert creation (no saved search ID). Create the saved search manually in HyperDX UI."
+  log "  Filter: $BREACH_QUERY"
+fi
+
+# ---------------------------------------------------------------------------
+# Alert 2: Budget warning (spend >= warn_ratio * amount)
+# ---------------------------------------------------------------------------
+log ""
+log "=== Alert 2: Budget Warning ==="
+WARN_QUERY="ResourceAttributes['k8s.pod.labels.app'] = 'krateo-budget-evaluator' AND JSONExtractString(Body, 'status') = 'warning'"
+
+WARN_SS_ID=$(create_saved_search "Budget Warning" "$WARN_QUERY")
+if [ -n "$WARN_SS_ID" ]; then
+  MESSAGE="Budget warning: spend crossed the early-warning threshold for this scope."
+  create_alert "Budget Warning (portal)" "$WARN_SS_ID" 0 "15m" "$BUDGET_PORTAL_WEBHOOK_ID" "$MESSAGE"
+  if [ -n "$BUDGET_EMAIL_WEBHOOK_ID" ]; then
+    create_alert "Budget Warning (email)" "$WARN_SS_ID" 0 "15m" "$BUDGET_EMAIL_WEBHOOK_ID" "$MESSAGE"
+  fi
+else
+  log "  Skipping alert creation. Create the saved search manually in HyperDX UI."
+  log "  Filter: $WARN_QUERY"
+fi
+
+log ""
+log "Done."
diff --git a/alerts/budget/budget-evaluator-cronjob.yaml b/alerts/budget/budget-evaluator-cronjob.yaml
new file mode 100644
index 0000000..54c4038
--- /dev/null
+++ b/alerts/budget/budget-evaluator-cronjob.yaml
@@ -0,0 +1,101 @@
+---
+# Budget Evaluator
+#
+# A CronJob that queries the `budget_status` view every 15 minutes and writes
+# one structured JSON log line to stdout for every budget in `warning` or
+# `breached` state. The OTel DaemonSet collects the line into `otel_logs`
+# exactly like any other pod log — no new pipeline component.
+#
+# HyperDX then alerts on these lines (see bootstrap-budget-alerts.sh):
+#   ResourceAttributes['k8s.pod.labels.app'] = 'krateo-budget-evaluator'
+#     AND JSONExtractString(Body, 'status') = 'breached'   (or 'warning')
+#
+# This mirrors the heartbeat-canary self-monitoring pattern.
+#
+# Prerequisites:
+#   - budgets + budget_status DDL applied (see ddl/)
+#   - `otel-clickhouse-credentials` Secret (same one used by the OTel
+#     collectors) — adjust names below if your deployment differs.
+#
+# Apply: kubectl apply -f budget-evaluator-cronjob.yaml
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: krateo-budget-evaluator
+  namespace: krateo-system
+  labels:
+    app.kubernetes.io/name: krateo-budget-evaluator
+    app.kubernetes.io/part-of: krateo-observability-stack
+    app.kubernetes.io/component: budget-alerts
+spec:
+  schedule: "*/15 * * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 1
+  jobTemplate:
+    spec:
+      backoffLimit: 1
+      activeDeadlineSeconds: 120
+      template:
+        metadata:
+          labels:
+            app: krateo-budget-evaluator
+            app.kubernetes.io/name: krateo-budget-evaluator
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: evaluator
+              image: clickhouse/clickhouse-server:24.8-alpine
+              command:
+                - sh
+                - -c
+                - |
+                  set -eu
+                  clickhouse-client \
+                    --host "${CLICKHOUSE_HOST}" \
+                    --port "${CLICKHOUSE_PORT}" \
+                    --user "${CLICKHOUSE_USER}" \
+                    --password "${CLICKHOUSE_PASSWORD}" \
+                    --query "
+                      SELECT budget_id, org, tenant, service, tag_key, tag_value,
+                             period, amount, currency, spend,
+                             round(spend_ratio, 4) AS spend_ratio, status
+                      FROM ${CLICKHOUSE_DATABASE}.budget_status
+                      WHERE status != 'ok'
+                      FORMAT JSONEachRow" \
+                  | while IFS= read -r row; do
+                      echo "{\"level\":\"WARN\",\"msg\":\"budget-threshold\",\"component\":\"krateo-budget-evaluator\",\"budget\":${row},\"status\":$(echo "$row" | grep -o '"status":"[^"]*"' | cut -d: -f2),\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}"
+                    done
+              env:
+                - name: CLICKHOUSE_HOST
+                  value: clickhouse.clickhouse-system.svc.cluster.local
+                - name: CLICKHOUSE_PORT
+                  value: "9000"
+                - name: CLICKHOUSE_DATABASE
+                  value: default
+                - name: CLICKHOUSE_USER
+                  valueFrom:
+                    secretKeyRef:
+                      name: otel-clickhouse-credentials
+                      key: username
+                - name: CLICKHOUSE_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: otel-clickhouse-credentials
+                      key: password
+              resources:
+                limits:
+                  cpu: 100m
+                  memory: 128Mi
+                requests:
+                  cpu: 20m
+                  memory: 64Mi
+              securityContext:
+                readOnlyRootFilesystem: true
+                runAsNonRoot: true
+                runAsUser: 65534
+                allowPrivilegeEscalation: false
+                capabilities:
+                  drop: [ALL]
+          securityContext:
+            runAsNonRoot: true
diff --git a/alerts/budget/ddl/001_budgets.sql b/alerts/budget/ddl/001_budgets.sql
new file mode 100644
index 0000000..8f74ebf
--- /dev/null
+++ b/alerts/budget/ddl/001_budgets.sql
@@ -0,0 +1,24 @@
+-- Budget definitions: a threshold on rated showback data, per scope.
+-- Empty string = wildcard (tenant '' = whole org, service '' = all services,
+-- tag_key '' = not tag-scoped, tag_value '' = any value of tag_key).
+-- ReplacingMergeTree on updated_at so a budget update supersedes older rows.
+--
+-- The `{{database}}` token is replaced at apply time (same convention as the
+-- showback engine DDL); it must be the database hosting the showback tables.
+CREATE TABLE IF NOT EXISTS {{database}}.budgets
+(
+    budget_id  String,
+    org        LowCardinality(String),
+    tenant     LowCardinality(String) DEFAULT '',
+    service    LowCardinality(String) DEFAULT '',
+    tag_key    LowCardinality(String) DEFAULT '',
+    tag_value  String                 DEFAULT '',
+    period     LowCardinality(String) DEFAULT 'monthly',  -- 'monthly' | 'daily'
+    amount     Float64,
+    currency   LowCardinality(String),
+    warn_ratio Float64 DEFAULT 0.8,   -- early-warning threshold as fraction of amount
+    enabled    UInt8   DEFAULT 1,
+    updated_at DateTime64(3, 'UTC')
+)
+ENGINE = ReplacingMergeTree(updated_at)
+ORDER BY (org, tenant, service, tag_key, tag_value, budget_id)
diff --git a/alerts/budget/ddl/002_budget_status.sql b/alerts/budget/ddl/002_budget_status.sql
new file mode 100644
index 0000000..8d9d2e1
--- /dev/null
+++ b/alerts/budget/ddl/002_budget_status.sql
@@ -0,0 +1,65 @@
+-- Current-period spend vs. budget, per budget definition.
+-- status: 'ok' | 'warning' (>= warn_ratio) | 'breached' (>= amount).
+--
+-- Non-tag-scoped budgets aggregate showback_daily; tag-scoped budgets
+-- aggregate showback_daily_by_tag (the rollups maintained by the showback
+-- engine). The budgets table is tiny, so the CROSS JOIN + filter over the
+-- daily rollups stays cheap.
+CREATE OR REPLACE VIEW {{database}}.budget_status AS
+WITH spend AS
+(
+    -- budgets without a tag scope
+    SELECT
+        b.budget_id AS budget_id,
+        sum(d.cost) AS spend
+    FROM {{database}}.budgets AS b FINAL
+    CROSS JOIN {{database}}.showback_daily AS d
+    WHERE b.enabled = 1
+      AND b.tag_key = ''
+      AND d.org = b.org
+      AND (b.tenant  = '' OR d.tenant  = b.tenant)
+      AND (b.service = '' OR d.service = b.service)
+      AND d.currency = b.currency
+      AND d.day >= if(b.period = 'daily', today(), toStartOfMonth(today()))
+    GROUP BY b.budget_id
+
+    UNION ALL
+
+    -- tag-scoped budgets
+    SELECT
+        b.budget_id,
+        sum(d.cost)
+    FROM {{database}}.budgets AS b FINAL
+    CROSS JOIN {{database}}.showback_daily_by_tag AS d
+    WHERE b.enabled = 1
+      AND b.tag_key != ''
+      AND d.org = b.org
+      AND d.tag_key = b.tag_key
+      AND (b.tag_value = '' OR d.tag_value = b.tag_value)
+      AND (b.tenant    = '' OR d.tenant   = b.tenant)
+      AND (b.service   = '' OR d.service  = b.service)
+      AND d.currency = b.currency
+      AND d.day >= if(b.period = 'daily', today(), toStartOfMonth(today()))
+    GROUP BY b.budget_id
+)
+SELECT
+    b.budget_id,
+    b.org,
+    b.tenant,
+    b.service,
+    b.tag_key,
+    b.tag_value,
+    b.period,
+    b.amount,
+    b.currency,
+    b.warn_ratio,
+    coalesce(s.spend, 0)                                    AS spend,
+    if(b.amount > 0, coalesce(s.spend, 0) / b.amount, 0)    AS spend_ratio,
+    multiIf(
+        coalesce(s.spend, 0) >= b.amount,                'breached',
+        coalesce(s.spend, 0) >= b.amount * b.warn_ratio, 'warning',
+                                                         'ok')  AS status,
+    now64(3, 'UTC')                                         AS evaluated_at
+FROM {{database}}.budgets AS b FINAL
+LEFT JOIN spend AS s ON s.budget_id = b.budget_id
+WHERE b.enabled = 1
diff --git a/dashboards/slo/.env.example b/dashboards/slo/.env.example
new file mode 100644
index 0000000..6e89830
--- /dev/null
+++ b/dashboards/slo/.env.example
@@ -0,0 +1,24 @@
+# SLO Alert Bootstrap Configuration
+# Copy to .env and fill in your values:
+#   cp .env.example .env
+
+# HyperDX base URL (e.g. http://localhost:3000 when port-forwarding)
+HYPERDX_URL=http://localhost:3000
+
+# Personal API key from HyperDX (Settings → API Keys)
+HYPERDX_API_KEY=
+
+# Webhook ID for alert delivery (portal / alert-proxy / email integration)
+WEBHOOK_ID=
+
+# --- SLO targets (defaults shown; override per deployment) -----------------
+# API request p95 latency target, milliseconds
+SLO_API_LATENCY_P95_MS=500
+# UI page-interaction p95 latency target, milliseconds
+SLO_UI_LATENCY_P95_MS=2000
+# Availability target, percent (used in dashboards / burn-rate math)
+SLO_AVAILABILITY_TARGET=99.9
+# Max ERROR/FATAL log lines per 5m per namespace before alerting
+SLO_ERROR_RATE_MAX=10
+# Max failing server spans per 5m per service before alerting
+SLO_ERROR_SPAN_MAX=5
diff --git a/dashboards/slo/README.md b/dashboards/slo/README.md
new file mode 100644
index 0000000..81a8f16
--- /dev/null
+++ b/dashboards/slo/README.md
@@ -0,0 +1,54 @@
+# SLI / SLO — dashboards and alerts
+
+Exposes the three golden SLIs — **availability**, **latency**, **error-rate** —
+for the platform services, computed from the telemetry already flowing into
+ClickHouse (`otel_traces`, `otel_logs`), and tracks them against configurable
+SLO targets. No new collection: the OTel gateway already receives traces from
+the instrumented services; this directory only adds the queries, the HyperDX
+dashboard and the SLO burn alerts on top.
+
+## SLI definitions
+
+| SLI | Definition | Source |
+|-----|------------|--------|
+| **Availability** | share of server spans with `StatusCode != 'Error'` over the window | `otel_traces` |
+| **Latency** | p50/p95/p99 of server span `Duration`, per service | `otel_traces` |
+| **Error-rate** | `ERROR`/`FATAL` log lines per service over the window | `otel_logs` |
+
+The full SQL lives in [`sli-queries.sql`](sli-queries.sql) — each query is
+copy-pasteable into a HyperDX chart or the ClickHouse client.
+
+## SLO targets
+
+Targets are **configuration, not code** — set them in `.env` (defaults below):
+
+| Variable | Default | Meaning |
+|----------|---------|---------|
+| `SLO_AVAILABILITY_TARGET` | `99.9` | availability %, per service, 30-day window |
+| `SLO_API_LATENCY_P95_MS` | `500` | API request p95 latency (ms) |
+| `SLO_UI_LATENCY_P95_MS` | `2000` | UI page-interaction p95 latency (ms) |
+| `SLO_ERROR_RATE_MAX` | `10` | max ERROR/FATAL log lines per 5m per namespace |
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| `sli-queries.sql` | The SLI queries (availability, latency percentiles, error-rate, burn rate). |
+| `bootstrap-slo-alerts.sh` | Creates the HyperDX saved searches + SLO breach alerts via API. |
+| `.env.example` | Configuration template (targets + HyperDX access). |
+| [`../../docs/SLO.md`](../../docs/SLO.md) | SLI/SLO reference doc + perf-results capture template. |
+
+## Setup
+
+1. Create the dashboard: in HyperDX add one chart per query from
+   `sli-queries.sql` (Availability by service, p95 latency by service,
+   Error-rate by namespace, SLO burn). Save as **"Platform SLOs"**.
+2. Create the breach alerts:
+
+   ```sh
+   cp .env.example .env   # fill in values
+   ./bootstrap-slo-alerts.sh
+   ```
+
+Alerts route through the same channels as every other alert (portal webhook /
+alert proxy / email), so an SLO breach shows up in-portal like any incident.
diff --git a/dashboards/slo/bootstrap-slo-alerts.sh b/dashboards/slo/bootstrap-slo-alerts.sh
new file mode 100755
index 0000000..f4031d1
--- /dev/null
+++ b/dashboards/slo/bootstrap-slo-alerts.sh
@@ -0,0 +1,161 @@
+#!/usr/bin/env bash
+# ---------------------------------------------------------------------------
+# SLO Alerts – HyperDX API Bootstrap
+#
+# Creates the SLO breach alerts on top of the telemetry already collected:
+#   1. Error-budget burn  — server spans ending in error (fast-burn signal)
+#   2. Latency SLO breach — server spans slower than the p95 target
+#   3. Error-rate SLO     — ERROR/FATAL log lines above budget
+#
+# Targets are configurable via env (see .env.example); defaults:
+#   SLO_API_LATENCY_P95_MS=500  SLO_UI_LATENCY_P95_MS=2000  SLO_ERROR_RATE_MAX=10
+#
+# Usage:
+#   export HYPERDX_URL="http://localhost:3000"
+#   export HYPERDX_API_KEY="your-api-key"
+#   export WEBHOOK_ID="your-webhook-id"
+#   ./bootstrap-slo-alerts.sh
+#
+# Or use .env file:
+#   cp .env.example .env && edit .env && ./bootstrap-slo-alerts.sh
+# ---------------------------------------------------------------------------
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+[ -f "$SCRIPT_DIR/.env" ] && set -a && source "$SCRIPT_DIR/.env" && set +a
+
+HYPERDX_URL="${HYPERDX_URL:-http://localhost:3000}"
+HYPERDX_API_KEY="${HYPERDX_API_KEY:-}"
+WEBHOOK_ID="${WEBHOOK_ID:-}"
+SLO_API_LATENCY_P95_MS="${SLO_API_LATENCY_P95_MS:-500}"
+SLO_UI_LATENCY_P95_MS="${SLO_UI_LATENCY_P95_MS:-2000}"
+SLO_ERROR_RATE_MAX="${SLO_ERROR_RATE_MAX:-10}"
+SLO_ERROR_SPAN_MAX="${SLO_ERROR_SPAN_MAX:-5}"
+
+API_BASE="${HYPERDX_URL%/}/api"
+
+die() { echo "[ERROR] $*" >&2; exit 1; }
+log() { echo "[bootstrap] $*"; }
+
+[ -n "$HYPERDX_API_KEY" ] || die "HYPERDX_API_KEY is required"
+[ -n "$WEBHOOK_ID" ]      || die "WEBHOOK_ID is required"
+
+# ---------------------------------------------------------------------------
+# Helper: create a saved search
+# ---------------------------------------------------------------------------
+create_saved_search() {
+  local name="$1"
+  local query="$2"
+
+  log "Creating saved search: $name"
+  RESP=$(curl -s -w "\n%{http_code}" -X POST "$API_BASE/v1/saved-searches" \
+    -H "Authorization: Bearer $HYPERDX_API_KEY" \
+    -H "Content-Type: application/json" \
+    -d "$(jq -n --arg name "$name" --arg query "$query" \
+      '{ name: $name, query: $query }')")
+
+  HTTP_CODE=$(echo "$RESP" | tail -n 1)
+  HTTP_BODY=$(echo "$RESP" | sed '$d')
+
+  if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ]; then
+    echo "$HTTP_BODY" | jq -r '._id // .id // empty' 2>/dev/null
+  else
+    log "  Warning: saved search creation returned HTTP $HTTP_CODE (may already exist)"
+    echo ""
+  fi
+}
+
+# ---------------------------------------------------------------------------
+# Helper: create an alert
+# ---------------------------------------------------------------------------
+create_alert() {
+  local name="$1"
+  local saved_search_id="$2"
+  local threshold="$3"
+  local interval="$4"
+  local message="$5"
+  local group_by="${6:-}"
+
+  log "Creating alert: $name (threshold: above $threshold, interval: $interval)"
+  local payload
+  payload=$(jq -n \
+    --arg name "$name" \
+    --arg savedSearchId "$saved_search_id" \
+    --argjson threshold "$threshold" \
+    --arg interval "$interval" \
+    --arg webhookId "$WEBHOOK_ID" \
+    --arg message "$message" \
+    --arg groupBy "$group_by" \
+    '{
+      name: $name,
+      savedSearchId: $savedSearchId,
+      threshold: $threshold,
+      threshold_type: "above",
+      interval: $interval,
+      source: "search",
+      channel: { type: "slack_webhook", webhookId: $webhookId },
+      message: $message
+    } + (if $groupBy != "" then { groupBy: ($groupBy | split(",")) } else {} end)')
+
+  RESP=$(curl -s -w "\n%{http_code}" -X POST "$API_BASE/alerts" \
+    -H "Authorization: Bearer $HYPERDX_API_KEY" \
+    -H "Content-Type: application/json" \
+    -d "$payload")
+
+  HTTP_CODE=$(echo "$RESP" | tail -n 1)
+  if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ]; then
+    log "  Created successfully."
+  else
+    HTTP_BODY=$(echo "$RESP" | sed '$d')
+    log "  Warning: returned HTTP $HTTP_CODE"
+    echo "$HTTP_BODY" | jq . 2>/dev/null || echo "$HTTP_BODY"
+  fi
+}
+
+# ---------------------------------------------------------------------------
+# Alert 1: Error-budget burn (server spans in error)
+# ---------------------------------------------------------------------------
+log ""
+log "=== Alert 1: Error-Budget Burn ==="
+BURN_QUERY="SpanKind = 'Server' AND StatusCode = 'Error'"
+
+BURN_SS_ID=$(create_saved_search "SLO Error-Budget Burn" "$BURN_QUERY")
+if [ -n "$BURN_SS_ID" ]; then
+  MESSAGE="Availability SLO at risk: server spans are failing — error budget is burning fast."
+  create_alert "SLO Error-Budget Burn" "$BURN_SS_ID" "$SLO_ERROR_SPAN_MAX" "5m" "$MESSAGE" "service.name"
+else
+  log "  Skipping alert creation. Filter: $BURN_QUERY"
+fi
+
+# ---------------------------------------------------------------------------
+# Alert 2: API latency SLO breach (p95 target)
+# ---------------------------------------------------------------------------
+log ""
+log "=== Alert 2: API Latency SLO ==="
+API_LATENCY_QUERY="SpanKind = 'Server' AND Duration > ${SLO_API_LATENCY_P95_MS}000000"
+
+API_LAT_SS_ID=$(create_saved_search "SLO API Latency Breaches" "$API_LATENCY_QUERY")
+if [ -n "$API_LAT_SS_ID" ]; then
+  MESSAGE="Latency SLO at risk: API requests slower than ${SLO_API_LATENCY_P95_MS}ms are accumulating."
+  create_alert "SLO API Latency" "$API_LAT_SS_ID" 20 "5m" "$MESSAGE" "service.name"
+else
+  log "  Skipping alert creation. Filter: $API_LATENCY_QUERY"
+fi
+
+# ---------------------------------------------------------------------------
+# Alert 3: Error-rate SLO (application logs)
+# ---------------------------------------------------------------------------
+log ""
+log "=== Alert 3: Error-Rate SLO ==="
+ERROR_QUERY="SeverityText IN ('ERROR', 'FATAL') AND ResourceAttributes['telemetry.source'] != 'k8s-events'"
+
+ERROR_SS_ID=$(create_saved_search "SLO Error Rate" "$ERROR_QUERY")
+if [ -n "$ERROR_SS_ID" ]; then
+  MESSAGE="Error-rate SLO at risk: ERROR/FATAL log volume above budget."
+  create_alert "SLO Error Rate" "$ERROR_SS_ID" "$SLO_ERROR_RATE_MAX" "5m" "$MESSAGE" "k8s.namespace.name"
+else
+  log "  Skipping alert creation. Filter: $ERROR_QUERY"
+fi
+
+log ""
+log "Done. Build the 'Platform SLOs' dashboard from sli-queries.sql (see README)."
diff --git a/dashboards/slo/sli-queries.sql b/dashboards/slo/sli-queries.sql
new file mode 100644
index 0000000..e252784
--- /dev/null
+++ b/dashboards/slo/sli-queries.sql
@@ -0,0 +1,89 @@
+-- ---------------------------------------------------------------------------
+-- SLI queries over the ClickStack tables (otel_traces / otel_logs).
+-- Each statement is standalone: paste it into a HyperDX chart (SQL mode) or
+-- run it with clickhouse-client. Time windows use HyperDX's chart range where
+-- available; the standalone versions below default to the last 24h.
+-- ---------------------------------------------------------------------------
+
+-- === SLI 1: Availability (per service, 5m buckets) =========================
+-- Share of server spans that did not end in error.
+SELECT
+    toStartOfFiveMinutes(Timestamp)                       AS bucket,
+    ServiceName                                           AS service,
+    countIf(StatusCode != 'Error') / count()              AS availability,
+    count()                                               AS requests
+FROM otel_traces
+WHERE SpanKind = 'Server'
+  AND Timestamp >= now() - INTERVAL 24 HOUR
+GROUP BY bucket, service
+ORDER BY bucket, service;
+
+-- === SLI 1b: Availability vs. SLO (30-day rolling, per service) ============
+-- Compare against SLO_AVAILABILITY_TARGET (e.g. 99.9).
+SELECT
+    ServiceName                                              AS service,
+    round(100 * countIf(StatusCode != 'Error') / count(), 3) AS availability_pct,
+    count()                                                  AS requests
+FROM otel_traces
+WHERE SpanKind = 'Server'
+  AND Timestamp >= now() - INTERVAL 30 DAY
+GROUP BY service
+ORDER BY availability_pct ASC;
+
+-- === SLI 2: Latency percentiles (per service, 5m buckets) ==================
+-- Duration is nanoseconds; results in milliseconds.
+-- Track p95 against SLO_API_LATENCY_P95_MS / SLO_UI_LATENCY_P95_MS.
+SELECT
+    toStartOfFiveMinutes(Timestamp)              AS bucket,
+    ServiceName                                  AS service,
+    quantile(0.50)(Duration) / 1e6               AS p50_ms,
+    quantile(0.95)(Duration) / 1e6               AS p95_ms,
+    quantile(0.99)(Duration) / 1e6               AS p99_ms
+FROM otel_traces
+WHERE SpanKind = 'Server'
+  AND Timestamp >= now() - INTERVAL 24 HOUR
+GROUP BY bucket, service
+ORDER BY bucket, service;
+
+-- === SLI 2b: Latency SLO compliance (per service, 24h) =====================
+-- Share of requests faster than the latency target (request-based SLO).
+-- Replace 500 with the target (ms) for the service class.
+SELECT
+    ServiceName                                                   AS service,
+    round(100 * countIf(Duration / 1e6 <= 500) / count(), 3)      AS within_target_pct,
+    round(quantile(0.95)(Duration) / 1e6, 1)                      AS p95_ms,
+    count()                                                       AS requests
+FROM otel_traces
+WHERE SpanKind = 'Server'
+  AND Timestamp >= now() - INTERVAL 24 HOUR
+GROUP BY service
+ORDER BY within_target_pct ASC;
+
+-- === SLI 3: Error-rate (per namespace, 5m buckets) =========================
+-- Application ERROR/FATAL log lines (excludes k8s events).
+SELECT
+    toStartOfFiveMinutes(Timestamp)                       AS bucket,
+    ResourceAttributes['k8s.namespace.name']              AS namespace,
+    countIf(SeverityText IN ('ERROR', 'FATAL'))           AS errors,
+    count()                                               AS total_lines
+FROM otel_logs
+WHERE ResourceAttributes['telemetry.source'] != 'k8s-events'
+  AND Timestamp >= now() - INTERVAL 24 HOUR
+GROUP BY bucket, namespace
+ORDER BY bucket, namespace;
+
+-- === SLO burn rate (per service, 1h vs 30d error budget) ===================
+-- burn_rate > 1 means the service is consuming error budget faster than the
+-- SLO allows; > 14.4 on a 99.9% SLO ≈ page-worthy fast burn.
+WITH 0.999 AS slo_target
+SELECT
+    ServiceName                                               AS service,
+    countIf(StatusCode = 'Error') / count()                   AS error_ratio_1h,
+    round((countIf(StatusCode = 'Error') / count()) / (1 - slo_target), 2)
+                                                              AS burn_rate
+FROM otel_traces
+WHERE SpanKind = 'Server'
+  AND Timestamp >= now() - INTERVAL 1 HOUR
+GROUP BY service
+HAVING count() > 0
+ORDER BY burn_rate DESC;
diff --git a/docs/SLO.md b/docs/SLO.md
new file mode 100644
index 0000000..451f4dc
--- /dev/null
+++ b/docs/SLO.md
@@ -0,0 +1,64 @@
+# Platform SLIs / SLOs
+
+Reference for the service-level indicators exposed by the observability stack
+and the objectives tracked against them. The queries, dashboard layout and
+breach alerts live in [`dashboards/slo/`](../dashboards/slo/).
+
+## SLIs
+
+All three SLIs are computed from telemetry already collected by the stack —
+no additional instrumentation is required for services that emit OTLP traces
+and structured logs.
+
+| SLI | Definition | Source table |
+|-----|------------|--------------|
+| **Availability** | ratio of server spans with `StatusCode != 'Error'` | `otel_traces` |
+| **Latency** | p50 / p95 / p99 of server-span `Duration`, per service | `otel_traces` |
+| **Error-rate** | `ERROR`/`FATAL` structured log lines per service/namespace | `otel_logs` |
+
+## SLOs (default targets)
+
+Targets are deployment configuration (`dashboards/slo/.env`), not code.
+Defaults:
+
+| Objective | Target | Window |
+|-----------|--------|--------|
+| Availability | ≥ 99.9 % | 30-day rolling |
+| API request latency | p95 ≤ 500 ms | 5-minute buckets |
+| UI interaction latency | p95 ≤ 2 s | 5-minute buckets |
+| Error-rate | ≤ 10 ERROR/FATAL lines / 5 m / namespace | 5-minute buckets |
+
+**Error budget:** at 99.9 % availability the monthly budget is ~43 minutes of
+failed requests. The burn-rate query in `sli-queries.sql` reports consumption;
+a burn rate above ~14× on the 1-hour window is a fast-burn (page-worthy)
+signal.
+
+## Dashboards
+
+The **"Platform SLOs"** HyperDX dashboard is built from the queries in
+[`dashboards/slo/sli-queries.sql`](../dashboards/slo/sli-queries.sql):
+
+1. Availability by service (5m) + 30-day availability vs. target
+2. Latency p50/p95/p99 by service + share of requests within target
+3. Error-rate by namespace
+4. Error-budget burn rate by service
+
+Breach alerts are bootstrapped by
+[`dashboards/slo/bootstrap-slo-alerts.sh`](../dashboards/slo/bootstrap-slo-alerts.sh)
+and route through the standard alert channels (portal / alert-proxy / email).
+
+## Performance results
+
+Performance runs are captured here so the tracked SLOs stay tied to measured
+baselines. Record one row per run:
+
+| Date | Scenario | Load | Availability | API p95 (ms) | UI p95 (ms) | Notes |
+|------|----------|------|--------------|--------------|-------------|-------|
+| _(pending)_ | e2e full-loop (`demo/tests`) | — | — | — | — | first baseline run to be recorded |
+
+How to capture a run:
+
+1. Execute the load/e2e scenario (see `demo/tests/`).
+2. While the run window is active, execute the queries in
+   `dashboards/slo/sli-queries.sql` scoped to the run's time range.
+3. Append the resulting numbers to the table above and link the raw output.

From 468c1aa33046182ed55ecb3f98f9811ed2fccf0d Mon Sep 17 00:00:00 2001
From: Diego Braga <diego.braga86@gmail.com>
Date: Tue, 21 Jul 2026 15:09:32 +0200
Subject: [PATCH 2/5] test: add executable smoke test for the budget DDL

Renders {{database}} and actually executes ddl/001+002 against a
throwaway ClickHouse (clickhouse-local, docker-image fallback) with stub
showback_daily/showback_daily_by_tag rollups mirroring the showback
engine schemas, then asserts budget_status materializes the expected
rows: ok/warning/breached classification, ReplacingMergeTree FINAL
supersede semantics, tag-scoped budgets (tag_value filtering), daily vs
monthly period windows, enabled=0 exclusion and the no-spend LEFT JOIN
path. Verified green locally via the docker fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 alerts/budget/tests/smoke-budget-ddl.sh | 169 ++++++++++++++++++++++++
 1 file changed, 169 insertions(+)
 create mode 100755 alerts/budget/tests/smoke-budget-ddl.sh

diff --git a/alerts/budget/tests/smoke-budget-ddl.sh b/alerts/budget/tests/smoke-budget-ddl.sh
new file mode 100755
index 0000000..7bfc55f
--- /dev/null
+++ b/alerts/budget/tests/smoke-budget-ddl.sh
@@ -0,0 +1,169 @@
+#!/usr/bin/env bash
+# ---------------------------------------------------------------------------
+# Budget DDL smoke test
+#
+# Actually executes ddl/001_budgets.sql + ddl/002_budget_status.sql against a
+# throwaway ClickHouse (clickhouse-local; docker fallback), together with stub
+# `showback_daily` / `showback_daily_by_tag` rollup tables whose schemas
+# mirror the showback engine DDL (006_showback_daily.sql /
+# 008_showback_daily_by_tag.sql). It then inserts sample budgets + spend and
+# asserts that the `budget_status` view materializes the expected statuses.
+#
+# Covered:
+#   - {{database}} rendering (same convention as the showback engine DDL)
+#   - ReplacingMergeTree FINAL semantics (a newer budget row supersedes an
+#     older one with the same scope + budget_id)
+#   - ok / warning / breached classification (warn_ratio and amount edges)
+#   - tag-scoped budgets (showback_daily_by_tag arm, tag_value filtering)
+#   - period windows (daily budgets ignore yesterday; monthly ignore old rows)
+#   - enabled=0 budgets excluded; budgets with no spend report ok/0
+#
+# Usage:
+#   ./smoke-budget-ddl.sh
+#
+# Environment:
+#   SMOKE_DATABASE    database name to render into {{database}} (default: showback)
+#   CLICKHOUSE_IMAGE  image for the docker fallback (default: clickhouse/clickhouse-server:24.8-alpine)
+#
+# Exit codes: 0 = pass, 1 = assertion/DDL failure, 2 = no ClickHouse runtime.
+# ---------------------------------------------------------------------------
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+DDL_DIR="$SCRIPT_DIR/../ddl"
+DB="${SMOKE_DATABASE:-showback}"
+IMAGE="${CLICKHOUSE_IMAGE:-clickhouse/clickhouse-server:24.8-alpine}"
+
+run_clickhouse() {
+  # One clickhouse-local session, multiquery SQL on stdin.
+  if command -v clickhouse-local >/dev/null 2>&1; then
+    clickhouse-local --multiquery
+  elif command -v clickhouse >/dev/null 2>&1; then
+    clickhouse local --multiquery
+  elif command -v docker >/dev/null 2>&1; then
+    docker run --rm -i --entrypoint clickhouse "$IMAGE" local --multiquery
+  else
+    echo "[smoke] SKIP: no clickhouse-local, clickhouse or docker on PATH" >&2
+    exit 2
+  fi
+}
+
+SQL_FILE="$(mktemp)"
+trap 'rm -f "$SQL_FILE"' EXIT
+
+{
+  echo "CREATE DATABASE IF NOT EXISTS $DB;"
+
+  # Stub rollups — schemas copied from the showback engine DDL
+  # (006_showback_daily.sql / 008_showback_daily_by_tag.sql).
+  cat <<EOF
+CREATE TABLE $DB.showback_daily
+(
+    day      Date,
+    org      LowCardinality(String),
+    tenant   LowCardinality(String),
+    service  LowCardinality(String),
+    metric   LowCardinality(String),
+    currency LowCardinality(String),
+    quantity Float64,
+    cost     Float64
+)
+ENGINE = SummingMergeTree((quantity, cost))
+PARTITION BY toYYYYMM(day)
+ORDER BY (org, tenant, service, metric, currency, day);
+
+CREATE TABLE $DB.showback_daily_by_tag
+(
+    day       Date,
+    org       LowCardinality(String),
+    tenant    LowCardinality(String),
+    service   LowCardinality(String),
+    metric    LowCardinality(String),
+    currency  LowCardinality(String),
+    tag_key   LowCardinality(String),
+    tag_value String,
+    quantity  Float64,
+    cost      Float64
+)
+ENGINE = SummingMergeTree((quantity, cost))
+PARTITION BY toYYYYMM(day)
+ORDER BY (org, tenant, service, metric, currency, tag_key, tag_value, day);
+EOF
+
+  # The DDL under test, rendered exactly like the showback engine renders it.
+  sed "s/{{database}}/$DB/g" "$DDL_DIR/001_budgets.sql"
+  echo ";"
+  sed "s/{{database}}/$DB/g" "$DDL_DIR/002_budget_status.sql"
+  echo ";"
+
+  cat <<EOF
+-- Budgets. b2-warn is inserted twice with the same scope + budget_id:
+-- the second (newer updated_at) row must win via ReplacingMergeTree FINAL.
+INSERT INTO $DB.budgets
+    (budget_id, org, tenant, service, tag_key, tag_value, period, amount, currency, warn_ratio, enabled, updated_at)
+VALUES
+    ('b1-ok',      'acme', '', 'svc-a', '',     '',      'monthly', 1000, 'EUR', 0.8, 1, '2026-01-01 00:00:00.000'),
+    ('b2-warn',    'acme', '', 'svc-b', '',     '',      'monthly',   10, 'EUR', 0.8, 1, '2026-01-01 00:00:00.000');
+
+INSERT INTO $DB.budgets
+    (budget_id, org, tenant, service, tag_key, tag_value, period, amount, currency, warn_ratio, enabled, updated_at)
+VALUES
+    ('b2-warn',    'acme', '', 'svc-b', '',     '',      'monthly',  100, 'EUR', 0.8, 1, '2026-01-02 00:00:00.000'),
+    ('b3-breach',  'acme', '', 'svc-c', '',     '',      'monthly',  100, 'EUR', 0.8, 1, '2026-01-01 00:00:00.000'),
+    ('b4-tag',     'acme', '', '',      'team', 'alpha', 'monthly',   50, 'EUR', 0.8, 1, '2026-01-01 00:00:00.000'),
+    ('b5-daily',   'acme', '', 'svc-d', '',     '',      'daily',    100, 'EUR', 0.8, 1, '2026-01-01 00:00:00.000'),
+    ('b6-monthly', 'acme', '', 'svc-e', '',     '',      'monthly',  100, 'EUR', 0.8, 1, '2026-01-01 00:00:00.000'),
+    ('b7-disabled','acme', '', 'svc-f', '',     '',      'monthly',  100, 'EUR', 0.8, 0, '2026-01-01 00:00:00.000'),
+    ('b8-nospend', 'acme', '', 'svc-g', '',     '',      'monthly',  100, 'EUR', 0.8, 1, '2026-01-01 00:00:00.000');
+
+-- Spend. Rows dated 2020-01-01 are outside every current window; the
+-- yesterday() row must be excluded from the daily budget b5 only.
+INSERT INTO $DB.showback_daily
+    (day, org, tenant, service, metric, currency, quantity, cost)
+VALUES
+    (today(),      'acme', 't1', 'svc-a', 'cpu', 'EUR', 1, 100),
+    (today(),      'acme', 't1', 'svc-b', 'cpu', 'EUR', 1,  85),
+    (today(),      'acme', 't1', 'svc-c', 'cpu', 'EUR', 1, 150),
+    (today(),      'acme', 't1', 'svc-d', 'cpu', 'EUR', 1,  30),
+    (yesterday(),  'acme', 't1', 'svc-d', 'cpu', 'EUR', 1, 500),
+    (today(),      'acme', 't1', 'svc-e', 'cpu', 'EUR', 1,  20),
+    ('2020-01-01', 'acme', 't1', 'svc-e', 'cpu', 'EUR', 1, 999),
+    (today(),      'acme', 't1', 'svc-f', 'cpu', 'EUR', 1, 500);
+
+INSERT INTO $DB.showback_daily_by_tag
+    (day, org, tenant, service, metric, currency, tag_key, tag_value, quantity, cost)
+VALUES
+    (today(), 'acme', 't1', 'svc-x', 'cpu', 'EUR', 'team', 'alpha', 1,  60),
+    (today(), 'acme', 't1', 'svc-x', 'cpu', 'EUR', 'team', 'beta',  1, 999);
+
+SELECT budget_id, amount, spend, status
+FROM $DB.budget_status
+ORDER BY budget_id
+FORMAT TSV;
+EOF
+} > "$SQL_FILE"
+
+echo "[smoke] rendering {{database}} -> $DB and executing DDL 001+002 + sample data"
+ACTUAL="$(run_clickhouse < "$SQL_FILE")"
+
+EXPECTED="$(cat <<'EOF'
+b1-ok	1000	100	ok
+b2-warn	100	85	warning
+b3-breach	100	150	breached
+b4-tag	50	60	breached
+b5-daily	100	30	ok
+b6-monthly	100	20	ok
+b8-nospend	100	0	ok
+EOF
+)"
+
+echo "[smoke] budget_status materialized:"
+printf '%s\n' "$ACTUAL" | sed 's/^/  /'
+
+if [ "$ACTUAL" = "$EXPECTED" ]; then
+  echo "[smoke] PASS: budget_status matches expected statuses (FINAL, tag scope, period windows, enabled flag all verified)"
+else
+  echo "[smoke] FAIL: budget_status differs from expected:" >&2
+  diff <(printf '%s\n' "$EXPECTED") <(printf '%s\n' "$ACTUAL") >&2 || true
+  exit 1
+fi

From 96fe057119d93823359056f4a4e8ad3d03df3100 Mon Sep 17 00:00:00 2001
From: Diego Braga <diego.braga86@gmail.com>
Date: Tue, 21 Jul 2026 15:09:42 +0200
Subject: [PATCH 3/5] fix: align evaluator database with rendered DDL; emit
 OTel-shaped lines (D19a)

Two review findings on the budget evaluator CronJob:

- CLICKHOUSE_DATABASE was hardcoded to 'default' while the DDL renders a
  {{database}} token (showback engine convention, default 'showback'),
  so the CronJob queried a database where budget_status does not exist
  (reproduced: UNKNOWN_TABLE, exit 60). Default is now 'showback' with a
  comment binding it to the rendered DDL database.

- The bespoke stdout line (level/msg/component + fragile grep/cut row
  parsing) is replaced by a line following the OTel Logs Data Model:
  timestamp, severity_text/severity_number (warning->WARN/13,
  breached->ERROR/17), body, trace_id/span_id and semconv-style
  attributes (event.name, service.name, krateo.budget.*), so a
  collector json parser can map it 1:1 onto an OTLP LogRecord while
  HyperDX keeps querying Body as JSON. The evaluator is a scheduled
  origin with no inbound trace context/baggage, so each run mints one
  trace_id shared by all lines of the run (scalar subquery) plus a
  per-line span_id. The whole line is built inside ClickHouse
  (toJSONString escapes all user-supplied values; FORMAT TSVRaw keeps
  lines atomic) - the shell no longer parses row data.

Verified end-to-end: the exact command block extracted from this YAML
was run in the CronJob image against a live clickhouse-server container
seeded with the DDL + sample data; output lines are valid JSON (jq),
32-hex shared trace_id / 16-hex span_ids, correct severity mapping, and
hostile values (quote/backslash/newline) stay escaped and atomic.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 alerts/budget/budget-evaluator-cronjob.yaml | 70 ++++++++++++++++-----
 1 file changed, 56 insertions(+), 14 deletions(-)

diff --git a/alerts/budget/budget-evaluator-cronjob.yaml b/alerts/budget/budget-evaluator-cronjob.yaml
index 54c4038..b867254 100644
--- a/alerts/budget/budget-evaluator-cronjob.yaml
+++ b/alerts/budget/budget-evaluator-cronjob.yaml
@@ -6,14 +6,28 @@
 # `breached` state. The OTel DaemonSet collects the line into `otel_logs`
 # exactly like any other pod log — no new pipeline component.
 #
+# Log-line shape (D19a): the line follows the OTel Logs Data Model —
+# `timestamp`, `severity_text`/`severity_number` (warning→WARN/13,
+# breached→ERROR/17), `body`, `trace_id`/`span_id` and semconv-style
+# `attributes` (`event.name`, `service.name`, `krateo.budget.*`) — so a
+# collector json parser can map it 1:1 onto an OTLP LogRecord, and HyperDX
+# can still query it as JSON in Body. The evaluator is a scheduled origin:
+# there is no inbound trace context or baggage to propagate, so each run
+# mints a fresh `trace_id` (shared by every alert line of that run, making a
+# run's alerts correlatable) and a `span_id` per line.
+#
 # HyperDX then alerts on these lines (see bootstrap-budget-alerts.sh):
 #   ResourceAttributes['k8s.pod.labels.app'] = 'krateo-budget-evaluator'
-#     AND JSONExtractString(Body, 'status') = 'breached'   (or 'warning')
+#     AND JSONExtractString(Body, 'attributes', 'krateo.budget.status') = 'breached'
+#   (or 'warning')
 #
 # This mirrors the heartbeat-canary self-monitoring pattern.
 #
 # Prerequisites:
-#   - budgets + budget_status DDL applied (see ddl/)
+#   - budgets + budget_status DDL applied (see ddl/). CLICKHOUSE_DATABASE
+#     below MUST match the database the DDL's `{{database}}` token was
+#     rendered with — i.e. the database hosting the showback tables
+#     (the showback engine default is `showback`).
 #   - `otel-clickhouse-credentials` Secret (same one used by the OTel
 #     collectors) — adjust names below if your deployment differs.
 #
@@ -46,6 +60,13 @@ spec:
           containers:
             - name: evaluator
               image: clickhouse/clickhouse-server:24.8-alpine
+              # ClickHouse builds the complete OTel-shaped JSON line per row
+              # (concat + toJSONString: all user-supplied values are
+              # JSON-escaped server-side; no shell parsing of row data).
+              # The scalar subquery is evaluated once, so every line of a
+              # run shares one trace_id; rand64() gives a per-line span_id.
+              # FORMAT TSVRaw = one unescaped line per row (toJSONString
+              # never emits raw control characters, lines stay atomic).
               command:
                 - sh
                 - -c
@@ -55,24 +76,45 @@ spec:
                     --host "${CLICKHOUSE_HOST}" \
                     --port "${CLICKHOUSE_PORT}" \
                     --user "${CLICKHOUSE_USER}" \
-                    --password "${CLICKHOUSE_PASSWORD}" \
-                    --query "
-                      SELECT budget_id, org, tenant, service, tag_key, tag_value,
-                             period, amount, currency, spend,
-                             round(spend_ratio, 4) AS spend_ratio, status
-                      FROM ${CLICKHOUSE_DATABASE}.budget_status
-                      WHERE status != 'ok'
-                      FORMAT JSONEachRow" \
-                  | while IFS= read -r row; do
-                      echo "{\"level\":\"WARN\",\"msg\":\"budget-threshold\",\"component\":\"krateo-budget-evaluator\",\"budget\":${row},\"status\":$(echo "$row" | grep -o '"status":"[^"]*"' | cut -d: -f2),\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}"
-                    done
+                    --password "${CLICKHOUSE_PASSWORD}" <<SQL
+                  WITH (SELECT lower(hex(randomString(16)))) AS run_trace_id
+                  SELECT concat(
+                      '{"timestamp":"', formatDateTime(now(), '%Y-%m-%dT%H:%i:%SZ', 'UTC'), '",',
+                      '"trace_id":"', run_trace_id, '",',
+                      '"span_id":"', lower(hex(rand64())), '",',
+                      '"severity_text":"', if(status = 'breached', 'ERROR', 'WARN'), '",',
+                      '"severity_number":', if(status = 'breached', '17', '13'), ',',
+                      '"body":"budget threshold crossed",',
+                      '"attributes":', toJSONString(map(
+                          'event.name',                'krateo.budget.threshold',
+                          'service.name',              'krateo-budget-evaluator',
+                          'krateo.budget.id',          budget_id,
+                          'krateo.budget.org',         org,
+                          'krateo.budget.tenant',      tenant,
+                          'krateo.budget.service',     service,
+                          'krateo.budget.tag_key',     tag_key,
+                          'krateo.budget.tag_value',   tag_value,
+                          'krateo.budget.period',      period,
+                          'krateo.budget.amount',      toString(amount),
+                          'krateo.budget.currency',    currency,
+                          'krateo.budget.spend',       toString(spend),
+                          'krateo.budget.spend_ratio', toString(round(spend_ratio, 4)),
+                          'krateo.budget.status',      status)),
+                      '}')
+                  FROM ${CLICKHOUSE_DATABASE}.budget_status
+                  WHERE status != 'ok'
+                  FORMAT TSVRaw
+                  SQL
               env:
                 - name: CLICKHOUSE_HOST
                   value: clickhouse.clickhouse-system.svc.cluster.local
                 - name: CLICKHOUSE_PORT
                   value: "9000"
+                # Must match the {{database}} the ddl/ files were rendered
+                # with (= the showback engine database; its default is
+                # `showback`), otherwise the budget_status view is not found.
                 - name: CLICKHOUSE_DATABASE
-                  value: default
+                  value: showback
                 - name: CLICKHOUSE_USER
                   valueFrom:
                     secretKeyRef:

From e9505ba9f3c79ebae08c5476d9f557f8c0439ba6 Mon Sep 17 00:00:00 2001
From: Diego Braga <diego.braga86@gmail.com>
Date: Tue, 21 Jul 2026 15:10:01 +0200
Subject: [PATCH 4/5] fix: fail HyperDX bootstraps loudly on non-2xx instead of
 continuing

Both bootstrap scripts logged a soft warning on non-2xx and kept going,
so a partial bootstrap looked like success. Failures (saved search or
alert creation, or a 2xx response with no id) are now recorded and the
script ends with a FAILED summary and exit 1. Failures are collected in
a temp file because the helpers run inside $(...) subshells where a
counter would be lost.

Also fixes a real capture bug the loud-failure test exposed: the
'Creating saved search' log line was written to stdout inside the
command-substituted helper, polluting the captured saved-search id (the
alert payload's savedSearchId contained the log line). It now goes to
stderr; verified against a mock HyperDX that the alert payload carries
the bare id, that an all-500 run attempts no alert POSTs and exits 1,
and that partial/success paths behave (0 on success, 1 on partial).

Budget saved-search filters and groupBy updated to the evaluator's new
OTel line shape: JSONExtractString(Body,'attributes','krateo.budget.status')
/ 'krateo.budget.id' (extraction verified in ClickHouse against a real
emitted line).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 alerts/budget/bootstrap-budget-alerts.sh | 44 +++++++++++++++++-------
 dashboards/slo/bootstrap-slo-alerts.sh   | 37 +++++++++++++++-----
 2 files changed, 61 insertions(+), 20 deletions(-)

diff --git a/alerts/budget/bootstrap-budget-alerts.sh b/alerts/budget/bootstrap-budget-alerts.sh
index 4fd8307..3e0b2e5 100755
--- a/alerts/budget/bootstrap-budget-alerts.sh
+++ b/alerts/budget/bootstrap-budget-alerts.sh
@@ -37,17 +37,26 @@ API_BASE="${HYPERDX_URL%/}/api"
 die() { echo "[ERROR] $*" >&2; exit 1; }
 log() { echo "[bootstrap] $*"; }
 
+# Failures are collected in a file (not a counter) because the helpers run
+# inside $(...) subshells; any recorded failure makes the script exit 1 with
+# a FAILED summary, so a partial bootstrap can never look like success.
+FAIL_LOG="$(mktemp)"
+trap 'rm -f "$FAIL_LOG"' EXIT
+fail() { echo "[FAILED] $*" >&2; echo "$*" >> "$FAIL_LOG"; }
+
 [ -n "$HYPERDX_API_KEY" ]          || die "HYPERDX_API_KEY is required"
 [ -n "$BUDGET_PORTAL_WEBHOOK_ID" ] || die "BUDGET_PORTAL_WEBHOOK_ID is required (create webhook in HyperDX UI first)"
 
 # ---------------------------------------------------------------------------
 # Helper: create a saved search
+# Runs inside $(...): stdout is the returned id ONLY — log goes to stderr,
+# otherwise the log line would be captured into the saved-search id.
 # ---------------------------------------------------------------------------
 create_saved_search() {
   local name="$1"
   local query="$2"
 
-  log "Creating saved search: $name"
+  log "Creating saved search: $name" >&2
   RESP=$(curl -s -w "\n%{http_code}" -X POST "$API_BASE/v1/saved-searches" \
     -H "Authorization: Bearer $HYPERDX_API_KEY" \
     -H "Content-Type: application/json" \
@@ -58,9 +67,16 @@ create_saved_search() {
   HTTP_BODY=$(echo "$RESP" | sed '$d')
 
   if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ]; then
-    echo "$HTTP_BODY" | jq -r '._id // .id // empty' 2>/dev/null
+    local id
+    id=$(echo "$HTTP_BODY" | jq -r '._id // .id // empty' 2>/dev/null)
+    if [ -n "$id" ]; then
+      echo "$id"
+    else
+      fail "saved search '$name': HTTP $HTTP_CODE but no id in response — $HTTP_BODY"
+      echo ""
+    fi
   else
-    log "  Warning: saved search creation returned HTTP $HTTP_CODE (may already exist)"
+    fail "saved search '$name': HTTP $HTTP_CODE — $HTTP_BODY (if it already exists, delete or update it in the HyperDX UI and re-run)"
     echo ""
   fi
 }
@@ -85,6 +101,7 @@ create_alert() {
     --arg interval "$interval" \
     --arg webhookId "$webhook_id" \
     --arg message "$message" \
+    --arg groupBy "JSONExtractString(Body, 'attributes', 'krateo.budget.id')" \
     '{
       name: $name,
       savedSearchId: $savedSearchId,
@@ -94,7 +111,7 @@ create_alert() {
       source: "search",
       channel: { type: "slack_webhook", webhookId: $webhookId },
       message: $message,
-      groupBy: ["budget.budget_id"]
+      groupBy: [$groupBy]
     }')
 
   RESP=$(curl -s -w "\n%{http_code}" -X POST "$API_BASE/alerts" \
@@ -107,8 +124,7 @@ create_alert() {
     log "  Created successfully."
   else
     HTTP_BODY=$(echo "$RESP" | sed '$d')
-    log "  Warning: returned HTTP $HTTP_CODE"
-    echo "$HTTP_BODY" | jq . 2>/dev/null || echo "$HTTP_BODY"
+    fail "alert '$name': HTTP $HTTP_CODE — $(echo "$HTTP_BODY" | jq -c . 2>/dev/null || echo "$HTTP_BODY")"
   fi
 }
 
@@ -117,7 +133,7 @@ create_alert() {
 # ---------------------------------------------------------------------------
 log ""
 log "=== Alert 1: Budget Breached ==="
-BREACH_QUERY="ResourceAttributes['k8s.pod.labels.app'] = 'krateo-budget-evaluator' AND JSONExtractString(Body, 'status') = 'breached'"
+BREACH_QUERY="ResourceAttributes['k8s.pod.labels.app'] = 'krateo-budget-evaluator' AND JSONExtractString(Body, 'attributes', 'krateo.budget.status') = 'breached'"
 
 BREACH_SS_ID=$(create_saved_search "Budget Breached" "$BREACH_QUERY")
 if [ -n "$BREACH_SS_ID" ]; then
@@ -127,8 +143,7 @@ if [ -n "$BREACH_SS_ID" ]; then
     create_alert "Budget Breached (email)" "$BREACH_SS_ID" 0 "15m" "$BUDGET_EMAIL_WEBHOOK_ID" "$MESSAGE"
   fi
 else
-  log "  Skipping alert creation (no saved search ID). Create the saved search manually in HyperDX UI."
-  log "  Filter: $BREACH_QUERY"
+  fail "breach alerts skipped: no saved-search id. Filter: $BREACH_QUERY"
 fi
 
 # ---------------------------------------------------------------------------
@@ -136,7 +151,7 @@ fi
 # ---------------------------------------------------------------------------
 log ""
 log "=== Alert 2: Budget Warning ==="
-WARN_QUERY="ResourceAttributes['k8s.pod.labels.app'] = 'krateo-budget-evaluator' AND JSONExtractString(Body, 'status') = 'warning'"
+WARN_QUERY="ResourceAttributes['k8s.pod.labels.app'] = 'krateo-budget-evaluator' AND JSONExtractString(Body, 'attributes', 'krateo.budget.status') = 'warning'"
 
 WARN_SS_ID=$(create_saved_search "Budget Warning" "$WARN_QUERY")
 if [ -n "$WARN_SS_ID" ]; then
@@ -146,9 +161,14 @@ if [ -n "$WARN_SS_ID" ]; then
     create_alert "Budget Warning (email)" "$WARN_SS_ID" 0 "15m" "$BUDGET_EMAIL_WEBHOOK_ID" "$MESSAGE"
   fi
 else
-  log "  Skipping alert creation. Create the saved search manually in HyperDX UI."
-  log "  Filter: $WARN_QUERY"
+  fail "warning alerts skipped: no saved-search id. Filter: $WARN_QUERY"
 fi
 
 log ""
+if [ -s "$FAIL_LOG" ]; then
+  log "Bootstrap FAILED — $(wc -l < "$FAIL_LOG" | tr -d ' ') step(s) did not complete:"
+  sed 's/^/[bootstrap]   - /' "$FAIL_LOG" >&2
+  log "The bootstrap is PARTIAL: fix the failures above and re-run."
+  exit 1
+fi
 log "Done."
diff --git a/dashboards/slo/bootstrap-slo-alerts.sh b/dashboards/slo/bootstrap-slo-alerts.sh
index f4031d1..506bbcc 100755
--- a/dashboards/slo/bootstrap-slo-alerts.sh
+++ b/dashboards/slo/bootstrap-slo-alerts.sh
@@ -37,17 +37,26 @@ API_BASE="${HYPERDX_URL%/}/api"
 die() { echo "[ERROR] $*" >&2; exit 1; }
 log() { echo "[bootstrap] $*"; }
 
+# Failures are collected in a file (not a counter) because the helpers run
+# inside $(...) subshells; any recorded failure makes the script exit 1 with
+# a FAILED summary, so a partial bootstrap can never look like success.
+FAIL_LOG="$(mktemp)"
+trap 'rm -f "$FAIL_LOG"' EXIT
+fail() { echo "[FAILED] $*" >&2; echo "$*" >> "$FAIL_LOG"; }
+
 [ -n "$HYPERDX_API_KEY" ] || die "HYPERDX_API_KEY is required"
 [ -n "$WEBHOOK_ID" ]      || die "WEBHOOK_ID is required"
 
 # ---------------------------------------------------------------------------
 # Helper: create a saved search
+# Runs inside $(...): stdout is the returned id ONLY — log goes to stderr,
+# otherwise the log line would be captured into the saved-search id.
 # ---------------------------------------------------------------------------
 create_saved_search() {
   local name="$1"
   local query="$2"
 
-  log "Creating saved search: $name"
+  log "Creating saved search: $name" >&2
   RESP=$(curl -s -w "\n%{http_code}" -X POST "$API_BASE/v1/saved-searches" \
     -H "Authorization: Bearer $HYPERDX_API_KEY" \
     -H "Content-Type: application/json" \
@@ -58,9 +67,16 @@ create_saved_search() {
   HTTP_BODY=$(echo "$RESP" | sed '$d')
 
   if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ]; then
-    echo "$HTTP_BODY" | jq -r '._id // .id // empty' 2>/dev/null
+    local id
+    id=$(echo "$HTTP_BODY" | jq -r '._id // .id // empty' 2>/dev/null)
+    if [ -n "$id" ]; then
+      echo "$id"
+    else
+      fail "saved search '$name': HTTP $HTTP_CODE but no id in response — $HTTP_BODY"
+      echo ""
+    fi
   else
-    log "  Warning: saved search creation returned HTTP $HTTP_CODE (may already exist)"
+    fail "saved search '$name': HTTP $HTTP_CODE — $HTTP_BODY (if it already exists, delete or update it in the HyperDX UI and re-run)"
     echo ""
   fi
 }
@@ -107,8 +123,7 @@ create_alert() {
     log "  Created successfully."
   else
     HTTP_BODY=$(echo "$RESP" | sed '$d')
-    log "  Warning: returned HTTP $HTTP_CODE"
-    echo "$HTTP_BODY" | jq . 2>/dev/null || echo "$HTTP_BODY"
+    fail "alert '$name': HTTP $HTTP_CODE — $(echo "$HTTP_BODY" | jq -c . 2>/dev/null || echo "$HTTP_BODY")"
   fi
 }
 
@@ -124,7 +139,7 @@ if [ -n "$BURN_SS_ID" ]; then
   MESSAGE="Availability SLO at risk: server spans are failing — error budget is burning fast."
   create_alert "SLO Error-Budget Burn" "$BURN_SS_ID" "$SLO_ERROR_SPAN_MAX" "5m" "$MESSAGE" "service.name"
 else
-  log "  Skipping alert creation. Filter: $BURN_QUERY"
+  fail "error-budget-burn alert skipped: no saved-search id. Filter: $BURN_QUERY"
 fi
 
 # ---------------------------------------------------------------------------
@@ -139,7 +154,7 @@ if [ -n "$API_LAT_SS_ID" ]; then
   MESSAGE="Latency SLO at risk: API requests slower than ${SLO_API_LATENCY_P95_MS}ms are accumulating."
   create_alert "SLO API Latency" "$API_LAT_SS_ID" 20 "5m" "$MESSAGE" "service.name"
 else
-  log "  Skipping alert creation. Filter: $API_LATENCY_QUERY"
+  fail "api-latency alert skipped: no saved-search id. Filter: $API_LATENCY_QUERY"
 fi
 
 # ---------------------------------------------------------------------------
@@ -154,8 +169,14 @@ if [ -n "$ERROR_SS_ID" ]; then
   MESSAGE="Error-rate SLO at risk: ERROR/FATAL log volume above budget."
   create_alert "SLO Error Rate" "$ERROR_SS_ID" "$SLO_ERROR_RATE_MAX" "5m" "$MESSAGE" "k8s.namespace.name"
 else
-  log "  Skipping alert creation. Filter: $ERROR_QUERY"
+  fail "error-rate alert skipped: no saved-search id. Filter: $ERROR_QUERY"
 fi
 
 log ""
+if [ -s "$FAIL_LOG" ]; then
+  log "Bootstrap FAILED — $(wc -l < "$FAIL_LOG" | tr -d ' ') step(s) did not complete:"
+  sed 's/^/[bootstrap]   - /' "$FAIL_LOG" >&2
+  log "The bootstrap is PARTIAL: fix the failures above and re-run."
+  exit 1
+fi
 log "Done. Build the 'Platform SLOs' dashboard from sli-queries.sql (see README)."

From 38fc8c9b0fe72108f5b0a8f6c29b5fe67f46134f Mon Sep 17 00:00:00 2001
From: Diego Braga <diego.braga86@gmail.com>
Date: Tue, 21 Jul 2026 15:10:01 +0200
Subject: [PATCH 5/5] =?UTF-8?q?docs:=20budget=20README=20=E2=80=94=20DDL?=
 =?UTF-8?q?=20rendering,=20database=20alignment,=20smoke=20test,=20D19a=20?=
 =?UTF-8?q?line=20shape?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- Setup now shows rendering {{database}} (the previous command piped the
  raw file with an unresolved token) and states that the CronJob's
  CLICKHOUSE_DATABASE must match the rendered database.
- Document tests/smoke-budget-ddl.sh and the bootstrap's loud-failure
  behaviour.
- Document the OTel Logs Data Model line shape (severity mapping,
  per-run trace_id / per-line span_id, why a scheduled origin has no
  inbound context/baggage).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---
 alerts/budget/README.md | 68 +++++++++++++++++++++++++++++++++++------
 1 file changed, 59 insertions(+), 9 deletions(-)

diff --git a/alerts/budget/README.md b/alerts/budget/README.md
index f90d78f..78a67e9 100644
--- a/alerts/budget/README.md
+++ b/alerts/budget/README.md
@@ -35,8 +35,9 @@ them up, HyperDX alerts on them.
 |------|---------|
 | `ddl/001_budgets.sql` | `budgets` table — budget definitions (scope, period, amount, warn ratio). |
 | `ddl/002_budget_status.sql` | `budget_status` view — spend-to-date vs. amount, status `ok`/`warning`/`breached`. |
-| `budget-evaluator-cronjob.yaml` | CronJob logging warning/breached budgets as JSON lines. |
-| `bootstrap-budget-alerts.sh` | Creates the HyperDX saved searches + alerts (warning + breach) via API. |
+| `budget-evaluator-cronjob.yaml` | CronJob logging warning/breached budgets as OTel-shaped JSON lines. |
+| `bootstrap-budget-alerts.sh` | Creates the HyperDX saved searches + alerts (warning + breach) via API. Exits non-zero with a FAILED summary if any step does not complete. |
+| `tests/smoke-budget-ddl.sh` | Executes the DDL against clickhouse-local (docker fallback) with sample data and asserts the `budget_status` view materializes the expected statuses. |
 | `.env.example` | Configuration template for the bootstrap script. |
 
 ## Budget scoping
@@ -60,29 +61,78 @@ breach at `1.0`.
 
 ## Setup
 
-1. Apply the DDL to the ClickHouse database that hosts the showback tables
-   (replace `{{database}}` with the target database, same convention as the
-   showback engine DDL):
+1. Apply the DDL to the ClickHouse database that hosts the showback tables,
+   rendering `{{database}}` with the target database (same convention as the
+   showback engine DDL; its default database is `showback`):
 
    ```sh
-   clickhouse-client --multiquery < ddl/001_budgets.sql
-   clickhouse-client --multiquery < ddl/002_budget_status.sql
+   sed 's/{{database}}/showback/g' ddl/001_budgets.sql | clickhouse-client --multiquery
+   sed 's/{{database}}/showback/g' ddl/002_budget_status.sql | clickhouse-client --multiquery
    ```
 
-2. Deploy the evaluator:
+2. Deploy the evaluator. `CLICKHOUSE_DATABASE` in the CronJob (default
+   `showback`) **must match the database the DDL was rendered with**,
+   otherwise the `budget_status` view is not found and the job fails:
 
    ```sh
    kubectl apply -f budget-evaluator-cronjob.yaml
    ```
 
 3. Create the HyperDX alerts (webhooks for the in-portal channel and the
-   email channel must exist in HyperDX first):
+   email channel must exist in HyperDX first). The script exits non-zero
+   with a `FAILED` summary when any saved search or alert cannot be
+   created, so a partial bootstrap never looks like success:
 
    ```sh
    cp .env.example .env   # fill in values
    ./bootstrap-budget-alerts.sh
    ```
 
+## Verifying the DDL locally
+
+`tests/smoke-budget-ddl.sh` renders `{{database}}`, executes DDL 001+002
+against a throwaway ClickHouse (clickhouse-local, or the
+`clickhouse/clickhouse-server:24.8-alpine` image via docker when no local
+binary exists), inserts sample budgets + spend, and asserts the
+`budget_status` view returns the expected `ok`/`warning`/`breached` rows —
+including ReplacingMergeTree FINAL supersede semantics, tag-scoped budgets,
+daily/monthly period windows and the `enabled` flag:
+
+```sh
+tests/smoke-budget-ddl.sh
+```
+
+## Log-line shape (D19a)
+
+The evaluator emits one JSON line per warning/breached budget following the
+OTel Logs Data Model, so a collector json parser can map it 1:1 onto an
+OTLP LogRecord while HyperDX keeps querying it as JSON in `Body`:
+
+```json
+{
+  "timestamp": "2026-01-01T12:00:00Z",
+  "trace_id": "<32 hex — one per evaluator run>",
+  "span_id": "<16 hex — one per line>",
+  "severity_text": "WARN | ERROR",
+  "severity_number": 13,
+  "body": "budget threshold crossed",
+  "attributes": {
+    "event.name": "krateo.budget.threshold",
+    "service.name": "krateo-budget-evaluator",
+    "krateo.budget.id": "...", "krateo.budget.org": "...",
+    "krateo.budget.status": "warning | breached", "...": "..."
+  }
+}
+```
+
+Severity maps `warning`→`WARN`/13 and `breached`→`ERROR`/17. The evaluator
+is a **scheduled origin**: there is no inbound trace context or baggage to
+propagate (nothing calls it), so each run mints a fresh `trace_id` shared by
+all lines of that run — the alerts of one evaluation are correlatable — and
+a per-line `span_id`. The line is built entirely inside ClickHouse
+(`toJSONString` escapes all user-supplied values); the shell never parses
+row data.
+
 ## Delivery channels
 
 - **In-portal**: the alert webhook targets the autopilot-alert-proxy, which