feat: USE-method host and container panels, lighter agent and leaner repo

CHANGELOG.md

@@ -6,6 +6,17 @@ Notable changes to this stack. Format follows
 
 ## [Unreleased]
 
+### Upgrade
+
+- Re-vendor `templates/alloy/config.alloy` and `templates/run_scheduled.sh` on
+  each spoke. Host & Containers needs the first for the new panels and the
+  corrected host network panel.
+- On the hub, `git pull` and restart Grafana once. `retired.yaml` removes the
+  old per-project rules; delete it after that restart.
+- **Breaking:** `just ps`, `logs`, `pull`, `down` and `tail` are gone. Use
+  `docker compose ps|logs -f|pull|down --remove-orphans`, which read the same
+  `COMPOSE_FILE` from `.env`.
+
 ### Added
 
 - **`HighLatencyP99`**: p99 request latency above 2s for 5 minutes, per
@@ -15,6 +26,47 @@ Notable changes to this stack. Format follows
   minutes.
 - Service Health shows a **DB connection pool** panel (used and idle) from the
   OpenTelemetry SQLAlchemy instrumentation.
+- **Host & Containers covers the USE method.** Host rows add CPU by mode,
+  load per core, disk utilisation and throughput, transmit traffic,
+  temperature and scheduler activity, from collectors the agent already ran.
+  Container rows add CPU, memory and I/O pressure (PSI wait time, which works
+  without limits), memory against the limit, and network and disk I/O.
+
+### Changed
+
+- **The spoke agent's cAdvisor skips its per-container filesystem walk**, a
+  third of its CPU on a 12-container host, by enabling only the metric kinds
+  the stack reads.
+- **The spoke agent ships 8 of its own series instead of ~415**: `up`, the
+  export failures and the export queue gauges.
+- The spoke agent batches for 5s instead of 200ms, so a trickle of logs is
+  one request per 5s rather than up to five a second.
+- The hub drops its own Prometheus histogram buckets, like the other
+  stack jobs.
+- GPU panels no longer request exemplars, which the GPU exporter never has.
+- **One `ProjectTelemetrySilent` rule covers every project.** `bootstrap.sh`
+  renders it and the coverage backstop into `projects.yaml`, instead of a file
+  per project/env. Each silent pair is still its own alert instance, and
+  removing a project is now one edit to the `# COVERS:` line.
+- `run_scheduled.sh` pings `<ping_url>/<exit status>`; success pings still send
+  no job output.
+- The GPU dashboard drops the MIG and NVLink fabric panels, which only
+  datacentre cards report.
+- Removed config that restated defaults (Loki `server`, datasource `access`,
+  dashboard provider, Prometheus `evaluation_interval` and its unread
+  `origin` label, Grafana sign-up), the empty Cloudflare provider block, and
+  the pre-rename branches of `infra/generate-imports.sh`.
+- The demo installs only the OTLP HTTP exporter, without gRPC, in a
+  single-stage image.
+
+### Fixed
+
+- **The host network panel showed the agent container's own interface.**
+  `/host/proc/net` resolves to the reading process's namespace; the agent now
+  reads PID 1's, and leaves out bridges and veths. Transmit is shown too.
+- `just check` and `bootstrap.sh` no longer stop early on a checkout without
+  `.env`.
+- ADR 0002 no longer claims three GPU alert rules; none exist.
 
 ## [0.3.1] - 2026-09-07
 

bootstrap.sh

@@ -3,8 +3,9 @@
 #
 #   ./bootstrap.sh <project> <env>
 #
-#   1. renders the ProjectTelemetrySilent rule and reloads Grafana;
-#   2. regenerates the coverage rule that catches projects never bootstrapped;
+#   1. renders the ProjectTelemetrySilent rule for every covered project/env;
+#   2. renders the coverage rule that catches projects never bootstrapped, and
+#      reloads Grafana;
 #   3. creates the project's healthchecks.io checks (if an API key is present);
 #   4. prints the `.env` block for the project host and the curls that vendor the
 #      templates at a pinned tag.
@@ -31,7 +32,7 @@ cd "$root"
 
 # A bare ./bootstrap.sh does not load .env, so read single keys out of it.
 # Sourcing the whole file would drag the stack's secrets into scope.
-env_get() { [[ -f .env ]] && sed -n "s/^$1=//p" .env | tail -1; }
+env_get() { [[ ! -f .env ]] || sed -n "s/^$1=//p" .env | tail -1; }
 
 if [[ -z "${HEALTHCHECKS_API_KEY:-}" ]]; then
     HEALTHCHECKS_API_KEY="$(env_get HEALTHCHECKS_API_KEY)"
@@ -58,37 +59,28 @@ if [[ -z "${BOOTSTRAP_OUT_DIR:-}" ]]; then
     repo_raw="https://raw.githubusercontent.com/CMLPlatform/monitoring/${tag}/templates"
 fi
 
-# --------------------------------------------------------------- 1. the keystone rules
-# Every covered project/env pair, read back from the COVERS marker in each
-# rendered file, plus the pair being bootstrapped now.
-pairs="$({ sed -n 's/^# COVERS: //p' "$out_dir"/project-*.yaml 2>/dev/null || true
-           echo "$project $env_name"; } | sort -u)"
-# The markers come off disk and land in a sed replacement and a PromQL label
-# value. Refuse the run rather than render a rule that silently never matches;
-# the fix is to delete the edited file.
+# ------------------------------------------------------- 1-2. keystone rule and backstop
+# Every covered project/env pair, read back from the COVERS marker in the rendered
+# file, plus the pair being bootstrapped now.
+rendered="${out_dir}/projects.yaml"
+pairs="$({ sed -n 's/^# COVERS: //p' "$rendered" 2>/dev/null | tr ' /' '\n ' || true
+           echo "$project $env_name"; } | sed '/^$/d' | sort -u)"
+# The marker comes off disk and lands in a sed replacement and a PromQL label
+# value. Refuse the run rather than render a rule that silently never matches.
 while read -r p e; do
     valid_pair "$p" "$e" \
-        || { echo "error: bad '# COVERS:' marker in $out_dir: '$p $e'" >&2; exit 2; }
+        || { echo "error: bad '# COVERS:' marker in $rendered: '$p $e'" >&2; exit 2; }
 done <<<"$pairs"
 
-# All pairs are re-rendered, so a template fix reaches every project.
-while read -r p e; do
-    rendered="${out_dir}/project-${p}-${e}.yaml"
-    sed -e "s/__PROJECT__/${p}/g" \
-        -e "s/__ENV__/${e}/g" \
-        templates/alerting/project.yaml.tmpl > "$rendered"
-    echo "rendered  $rendered"
-done <<<"$pairs"
-
-# ------------------------------------------------------------ 2. the coverage backstop
-covered="$(echo "$pairs" | sed 's| |/|' | paste -sd',' - | sed 's/,/, /g')"
-covered_expr="$(echo "$pairs" \
-    | sed 's@^\([^ ]*\) \([^ ]*\)$@{__name__=~"telemetry_.+_total", project="\1",env="\2"}@' \
-    | paste -sd'@' - | sed 's/@/ or /g')"
-sed -e "s@__COVERED__@${covered}@" \
+covers="$(echo "$pairs" | sed 's| |/|' | paste -sd' ' -)"
+selector='s@^\([^ ]*\) \([^ ]*\)$@{__name__=~"telemetry_.+_total", project="\1",env="\2"}@'
+silent_expr="$(echo "$pairs" | sed -e "$selector" -e 's/.*/absent(&)/' | paste -sd'@' - | sed 's/@/ or /g')"
+covered_expr="$(echo "$pairs" | sed "$selector" | paste -sd'@' - | sed 's/@/ or /g')"
+sed -e "s@__COVERS__@${covers}@" \
+    -e "s@__SILENT_EXPR__@${silent_expr}@" \
     -e "s@__COVERED_EXPR__@${covered_expr}@" \
-    templates/alerting/coverage.yaml.tmpl > "${out_dir}/coverage.yaml"
-echo "rendered  ${out_dir}/coverage.yaml  (covering: ${covered})"
+    templates/alerting/projects.yaml.tmpl > "$rendered"
+echo "rendered  $rendered  (covering: ${covers})"
 [[ -z "${BOOTSTRAP_OUT_DIR:-}" ]] || exit 0
 
 # ------------------------------------------------------------------ 3. reload Grafana
@@ -109,26 +101,30 @@ if docker compose ps --status running --services 2>/dev/null | grep -qx grafana;
     # curl's config parser unescapes \ and " inside a quoted value, so escape both.
     gpw="${GRAFANA_ADMIN_PASSWORD//\\/\\\\}"
     gpw="${gpw//\"/\\\"}"
-    uid="proj-silent-${project}-${env_name}"
+    # One rule covers every pair, so read it back and look for this pair's selector.
+    want="project=\\\"${project}\\\",env=\\\"${env_name}\\\""
+    verified=""
     for _ in $(seq 30); do
         sleep 2
         # Password on stdin, never argv. A wrong password would otherwise poll
         # for a minute and then report the rule as missing.
-        code="$(printf 'user = "admin:%s"\n' "$gpw" \
-            | curl -s -o /dev/null -w '%{http_code}' -K - "http://localhost:3000/api/v1/provisioning/alert-rules/${uid}")" || continue
-        case "$code" in
+        resp="$(printf 'user = "admin:%s"\n' "$gpw" \
+            | curl -s -w '\n%{http_code}' -K - "http://localhost:3000/api/v1/provisioning/alert-rules/projects-silent")" || continue
+        case "${resp##*$'\n'}" in
             200)
-                echo "verified  rule ${uid} is provisioned"
-                uid=""
-                break
+                if grep -qF "$want" <<<"$resp"; then
+                    echo "verified  rule projects-silent covers ${project}/${env_name}"
+                    verified=1
+                    break
+                fi
                 ;;
             401 | 403)
-                echo "error: grafana rejected the admin credentials (HTTP ${code}); check GRAFANA_ADMIN_PASSWORD" >&2
+                echo "error: grafana rejected the admin credentials (HTTP ${resp##*$'\n'}); check GRAFANA_ADMIN_PASSWORD" >&2
                 exit 1
                 ;;
         esac
     done
-    [[ -z "$uid" ]] || { echo "error: grafana restarted but rule ${uid} is not provisioned; check 'just logs grafana' for the rejected file" >&2; exit 1; }
+    [[ -n "$verified" ]] || { echo "error: grafana restarted but rule projects-silent does not cover ${project}/${env_name}; check 'docker compose logs grafana' for the rejected file" >&2; exit 1; }
 else
     echo "note      grafana is not running; the rules apply next time it starts"
 fi

compose.yml

@@ -185,7 +185,6 @@ services:
       # Expanded by Grafana into the provisioned contact points.
       ALERT_WEBHOOK_URL: ${ALERT_WEBHOOK_URL:-}
       HEARTBEAT_URL: ${HEARTBEAT_URL:-}
-      GF_USERS_ALLOW_SIGN_UP: "false"
       GF_SERVER_ROOT_URL: ${GRAFANA_ROOT_URL:-http://localhost:3000}
       GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/stack-health.json
       # Secure cookies break plain-http localhost logins, so opt-in. With the