Skip to content

ci(smoke): run the TargetDown wait alongside the data-path checks #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
simonvanlierde merged 1 commit into from
Oct 2, 2026
Merged

ci(smoke): run the TargetDown wait alongside the data-path checks #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
# 600, or the exposure guards refuse it as world-readable.
- run: install -m 600 .env.example .env
- run: just validate
# SMOKE_ALERTS adds the TargetDown round trip (about 3 minutes).
# SMOKE_ALERTS adds the TargetDown round trip (about 2 minutes).
- run: just smoke
env:
SMOKE_ALERTS: "1"
Expand Down
16 changes: 11 additions & 5 deletions scripts/smoke.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,12 @@ n_jobs="$(grep -c '^ *- job_name:' config/prometheus.yaml)"
all_up() { [[ "$(promq 'count(up == 1)' | jq -r '.data.result[0].value[1] // 0')" == "$n_jobs" ]]; }
poll all_up || die "not all $n_jobs scrape targets are up: $(promq up | jq -r '.data.result[] | "\(.metric.job)=\(.value[1])"' | tr '\n' ' ')"

# The alert-pipeline check below waits out TargetDown's `for`. Stopping its target
# here lets that clock run during the data-path checks, none of which scrape it.
if [[ "${SMOKE_ALERTS:-}" == 1 ]]; then
docker compose -p "$project" stop -t 1 node-exporter >/dev/null
fi

# ------------------------------------------------------------- data paths
# One metric and one log through the collector's bearer auth, read back with
# the identity labels the alert rules key on. Posted from inside the stack's
Expand Down Expand Up @@ -93,12 +99,12 @@ counted() { promq "count by (project, env) ({__name__=~\"telemetry_.+_total\", p
poll counted || die "the count connector never produced telemetry_*_total{project=smoke,env=ci}; the alert rules that key on it would never fire"

# ------------------------------------------------------------- alert pipeline
# Opt-in (SMOKE_ALERTS=1): stop one scrape target and wait for TargetDown to
# reach firing. Costs the rule's `for` (2m) plus an evaluation, so it is off
# for the local loop and on in CI. Guards the threshold-node contract in
# rules.yaml: a query whose matching value is 0 never fires without `bool`.
# Opt-in (SMOKE_ALERTS=1): node-exporter was stopped after the scrape-target
# check; wait for TargetDown to reach firing. Costs the rule's `for` (2m) plus an
# evaluation, so it is off for the local loop and on in CI. Guards the
# threshold-node contract in rules.yaml: a query whose matching value is 0 never
# fires without `bool`.
if [[ "${SMOKE_ALERTS:-}" == 1 ]]; then
docker compose -p "$project" stop -t 1 node-exporter >/dev/null
target_down_firing() {
gf "$url/api/prometheus/grafana/api/v1/rules" \
| jq -e '.data.groups[].rules[] | select(.name == "TargetDown") | .state == "firing"' >/dev/null
Expand Down
Loading