Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
53b9670
Add monitoring stack and age-encrypted database backups
claude Oct 29, 2026
28b2f53
Weekly backups (8 weekly + 6 monthly), preflight check and ops checklist
claude Oct 30, 2026
6708d63
preflight: check what .env already sets, reserve Prometheus disk, def…
claude Nov 1, 2026
340d583
preflight --deep: show real mount errors, snap textfile dir fallback,…
claude Nov 1, 2026
95f9042
backup.env.example: make SAFEFAMILY_DIR optional and explain it
claude Nov 1, 2026
06cf1c3
db_backup: optional BACKUP_GROUP so a normal user can copy backups
claude Nov 1, 2026
b4f9b44
db_restore: add --drop-test; fix broken cleanup hint
claude Nov 1, 2026
d45ec7b
checklist: show the BACKUP_GROUP line in step 8
claude Nov 1, 2026
f398eb4
Host node-exporter mode for snap Docker
claude Nov 1, 2026
78a16ff
preflight: cadvisor always needs containerd; diagnose unnamed containers
claude Nov 2, 2026
beffcbd
preflight: explain why CONTAINERD_SOCKET has no effect; stop leaking …
claude Nov 2, 2026
591d3da
Monitoring: drop-in scrape/rule dirs and Samba/NFS share alerts
claude Nov 2, 2026
a01cd41
Watch automounted shares: WATCH_MOUNTS check in dir_sizes.sh
claude Nov 2, 2026
ad663b8
preflight: diagnose WATCH_MOUNTS shares step by step
claude Nov 2, 2026
a0e2c91
preflight: name the folder that blocks node-exporter from a share
claude Nov 2, 2026
5af7632
Detect nginx running a stale config; fix proxy recreate instructions
claude Nov 2, 2026
3ef41a0
Monitor AdGuard Home and whether YouTube is blocked
claude Nov 2, 2026
ab7c16d
adguard_check: blocked-services check for YouTube; slow check_host to…
claude Nov 2, 2026
aed28ed
Export a 1280x800 status picture for the ESP32-P4 display
claude Nov 2, 2026
6947c96
display-export: wait for Grafana at start, retry failures after 60s
claude Nov 2, 2026
ea18638
Serve Grafana at https://<site>/grafana/ for the home network only
claude Nov 2, 2026
8171029
preflight: detect backup setup by backup-specific files only
claude Nov 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,22 @@ MAIL_ACCOUNT="email account"
MAIL_PASSWORD='email app engine password'
MAIL_PERSON_LIST='["receiver email list"]'
ADMIN_IDENTITY="admin_id"

# Monitoring stack (docs/monitoring.md)
GRAFANA_ADMIN_PASSWORD="" # this exact name (not GF_SECURITY_ADMIN_PASSWORD)
# PROMETHEUS_RETENTION_SIZE="5GB" # lower it if preflight warns about disk space
# ALERT_EMAIL_TO="you@example.com" # default: MAIL_ACCOUNT
# GRAFANA_BIND="127.0.0.1"
# DOCKER_ROOT_DIR="/var/snap/docker/common/var-lib-docker" # snap Docker
# CONTAINERD_SOCKET="/run/containerd/containerd.sock" # scripts/monitoring/preflight.sh prints the right value
# COMPOSE_FILE=docker-compose.yml:deploy/monitoring/docker-compose.monitoring.yml
# Snap Docker (preflight --deep says containers see a different /): node-exporter on the host:
# COMPOSE_FILE=docker-compose.yml:deploy/monitoring/docker-compose.monitoring.yml:deploy/monitoring/docker-compose.host-node-exporter.yml
# Status picture for the P4 display (deploy/monitoring/docker-compose.display-export.yml):
# COMPOSE_PROFILES=display-export # exporter as a container (else the host timer)
# DISPLAY_EXPORT_DIR="/home/you/code/esphome-config/grafana-export"
# DISPLAY_TZ="America/Toronto"
# DISPLAY_EXPORT_UID=1000 # uid=/gid= of the share in /etc/fstab
# DISPLAY_EXPORT_GID=1000
# DISPLAY_INTERVAL=300 # seconds; the display marks pictures STALE after 10 min
# DISPLAY_EXPORT_REQUIRE_FSTYPE=cifs # "" to allow a plain local folder
9 changes: 9 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,15 @@ pytest -q tests/test_misc_routes.py::test_notes_media_public_note_for_other_user
ruff check .
```

### Monitoring and backups
Monitoring overlay: `deploy/monitoring/docker-compose.monitoring.yml` (Prometheus, Alertmanager, Grafana, exporters). Encrypted backups: `scripts/db_backup.sh` / `scripts/db_restore.sh` (age, public-key). See `docs/monitoring.md` and `docs/operations-checklist.md`; `scripts/monitoring/preflight.sh` checks host requirements. Validate rule changes with:
```bash
docker run --rm -v "$PWD/deploy/monitoring/prometheus:/etc/prometheus:ro" --entrypoint promtool prom/prometheus:v3.5.0 check config /etc/prometheus/prometheus.yml
docker run --rm -v "$PWD/deploy/monitoring/prometheus:/etc/prometheus:ro" --entrypoint promtool prom/prometheus:v3.5.0 test rules /etc/prometheus/tests/network_mounts_test.yml
docker run --rm -v "$PWD/deploy/monitoring/prometheus:/etc/prometheus:ro" --entrypoint promtool prom/prometheus:v3.5.0 test rules /etc/prometheus/tests/adguard_test.yml
```
The P4 display dashboard (`grafana/dashboards/safefamily-display.json`) is generated from the overview's panels by `deploy/monitoring/display-export/make_display_dashboard.py`; edit the overview panels or the script's LAYOUT, not the JSON. User extensions go in `deploy/monitoring/prometheus/scrape.d/` and `rules.d/` (loaded by glob), not in edits to `prometheus.yml`/`alerts.yml`.

### Run log analysis CLI
```bash
python -m safe_family.cli.analyze --range last_5min
Expand Down
16 changes: 15 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,21 @@ gunzip -c backup/full-2026-09-16.sql.gz \
the analyzer pages will simply have no history until new AdGuard logs arrive.
- `backup/` is gitignored, as is `deploy/init_db/`.

### 7. Host Maintenance Notes
### 7. Monitoring and Encrypted Backups
Prometheus, Grafana and Alertmanager (disk space, database size and health,
per-container CPU/memory, nginx load, website/TLS checks, email alerts) and the
age-encrypted weekly database backup are described in
[docs/monitoring.md](docs/monitoring.md); set it up with
[docs/operations-checklist.md](docs/operations-checklist.md), starting with
`sudo scripts/monitoring/preflight.sh --deep`. Quick start:
```bash
# .env: GRAFANA_ADMIN_PASSWORD=... and
# COMPOSE_FILE=docker-compose.yml:deploy/monitoring/docker-compose.monitoring.yml
docker compose up -d # Grafana on https://<site>/grafana/ (LAN only)
sudo systemctl start safefamily-backup # after the one-time setup in the doc
```

### 8. Host Maintenance Notes
- Snap auto-refreshes Docker several times a day (`snap refresh --time`). A
refresh restarts the daemon and briefly stops containers; `restart: always`
brings them back, but the timing is not under your control. Use
Expand Down
15 changes: 15 additions & 0 deletions deploy/monitoring/adguard.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Copy to /etc/safefamily/adguard.env and make it root-only:
# sudo install -m 600 deploy/monitoring/adguard.env.example /etc/safefamily/adguard.env
# Same values as ADGUARD_HOSTPORT / ADGUARD_USERNAME / ADGUARD_PASSWORD in the app's .env.
ADGUARD_HOSTPORT=10.0.0.1:8080
ADGUARD_USERNAME=admin
ADGUARD_PASSWORD=change-me
# Blocked-service ids to report (the list SafeFamily switches; fast, local).
ADGUARD_WATCH_SERVICES="youtube"
# Domains to check with AdGuard's check_host (also covers filter rules), space
# separated. Slow on some routers; set to "" to turn it off.
ADGUARD_WATCH_DOMAINS="youtube.com"
# Seconds per AdGuard API call.
ADGUARD_TIMEOUT=30
# Keep in step with NODE_EXPORTER_TEXTFILE_DIR in /etc/safefamily/backup.env.
NODE_EXPORTER_TEXTFILE_DIR=/var/lib/node_exporter/textfile
43 changes: 43 additions & 0 deletions deploy/monitoring/alertmanager/alertmanager.yml.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Rendered by entrypoint.sh: ${NAME} placeholders come from the container env
# (see docker-compose.monitoring.yml), so no secret is stored in this file.
global:
smtp_smarthost: '${ALERT_SMTP_SMARTHOST}'
smtp_from: '${MAIL_ACCOUNT}'
smtp_auth_username: '${MAIL_ACCOUNT}'
smtp_auth_password: '${MAIL_PASSWORD}'
smtp_require_tls: true

route:
receiver: email
group_by: [alertname]
group_wait: 1m
group_interval: 10m
repeat_interval: 12h
routes:
- matchers: [severity="critical"]
receiver: email
repeat_interval: 3h

receivers:
- name: email
email_configs:
- to: '${ALERT_EMAIL_TO}'
send_resolved: true
headers:
Subject: '[SafeFamily {{ .Status | toUpper }}] {{ .CommonLabels.alertname }} ({{ .Alerts | len }})'

inhibit_rules:
# When the site is down because a container is down, only mail about the container.
- source_matchers: [alertname="ContainerDown"]
target_matchers: [alertname=~"WebsiteDown|WebsiteSlow|ScrapeTargetDown"]
# One mail per broken share: the explicit watch wins over the general alerts.
- source_matchers: [alertname="WatchedMountDown"]
target_matchers: [alertname=~"NetworkMountUnreachable|NetworkMountMissing"]
equal: [mountpoint]
# An unreachable share can also drop out of the filesystem list.
- source_matchers: [alertname="NetworkMountUnreachable"]
target_matchers: [alertname="NetworkMountMissing"]
equal: [mountpoint]
- source_matchers: [alertname="DiskSpaceCritical"]
target_matchers: [alertname=~"DiskSpaceLow|DiskWillFillIn24h"]
equal: [mountpoint]
26 changes: 26 additions & 0 deletions deploy/monitoring/alertmanager/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/bin/sh
# Render alertmanager.yml.tmpl with values from the environment, then start
# Alertmanager. Values are single-quoted in the template, so a ' inside a
# value is doubled to stay valid YAML.
set -eu

# Unset or empty ALERT_EMAIL_TO: mail the sending account itself.
: "${ALERT_EMAIL_TO:=$MAIL_ACCOUNT}"
export ALERT_EMAIL_TO

out=/tmp/alertmanager.yml
awk '{
while (match($0, /\$\{[A-Z_]+\}/)) {
name = substr($0, RSTART + 2, RLENGTH - 3)
value = ENVIRON[name]
gsub(/'\''/, "'\'''\''", value)
$0 = substr($0, 1, RSTART - 1) value substr($0, RSTART + RLENGTH)
}
print
}' /etc/alertmanager/alertmanager.yml.tmpl > "$out"
chmod 600 "$out"

exec /bin/alertmanager \
--config.file="$out" \
--storage.path=/alertmanager \
"$@"
20 changes: 20 additions & 0 deletions deploy/monitoring/backup.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Copy to /etc/safefamily/backup.env (root-owned, mode 600). Read by
# deploy/systemd/safefamily-backup.service and safefamily-dirsize.service.
BACKUP_DIR=/var/backups/safefamily
BACKUP_AGE_RECIPIENTS=/etc/safefamily/backup-recipients.txt
BACKUP_KEEP_WEEKLY=8
BACKUP_KEEP_MONTHLY=6
# Let this group read the backups (they are encrypted), so you can scp/rsync
# them without sudo. Usually your login group: run `id -gn` to see it.
# BACKUP_GROUP=youruser
NODE_EXPORTER_TEXTFILE_DIR=/var/lib/node_exporter/textfile
# Optional: the repo checkout (the folder with docker-compose.yml), measured
# for Grafana's "Directory sizes" panel. Leave unset: dir_sizes.sh finds it
# from its own location.
# SAFEFAMILY_DIR=/home/you/SafeFamily
# Snap-installed Docker:
# DOCKER_ROOT_DIR=/var/snap/docker/common/var-lib-docker
# Network shares to watch (space separated mount points). Checked every 15
# minutes by dir_sizes.sh; this also keeps a systemd automount mounted, so
# Grafana and the share alerts can see it after a reboot.
# WATCH_MOUNTS=/mnt/nas
15 changes: 15 additions & 0 deletions deploy/monitoring/blackbox/blackbox.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
modules:
# Fetch the site through nginx inside the compose network. The Host header
# and TLS server name match server_name in deploy/nginx/nginx.docker.conf, so
# the real certificate is checked (expiry, hostname, chain).
safefamily_https:
prober: http
timeout: 10s
http:
method: GET
valid_status_codes: [200]
fail_if_not_ssl: true
headers:
Host: zzuse.duckdns.org
tls_config:
server_name: zzuse.duckdns.org
102 changes: 102 additions & 0 deletions deploy/monitoring/display-export/export.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
#!/bin/sh
# Render the "SafeFamily Display" Grafana dashboard to a PNG for the P4
# display, every DISPLAY_INTERVAL seconds (--once: a single run).
#
# Each run: Grafana's /render API (backed by the grafana-renderer service)
# -> a hidden temp file in the export folder -> check it is a PNG of the
# right size -> rename over one fixed file name, so readers never see a
# partial file and old files never pile up. A failed run leaves the last good
# picture in place; the display's STALE banner then shows it is old.
#
# GRAFANA_URL http://grafana:3000/grafana (Grafana's sub path)
# GRAFANA_USER / GRAFANA_PASSWORD a Grafana login (default admin user)
# DISPLAY_DASHBOARD_UID safefamily-display
# DISPLAY_WIDTH / DISPLAY_HEIGHT 1280 / 800
# DISPLAY_TZ time zone for the timeline, e.g. America/Toronto
# DISPLAY_INTERVAL seconds between exports (300); after a failed
# export it retries after DISPLAY_RETRY (60) instead
# DISPLAY_FILE file name in the export folder (grafana-status.png)
# DISPLAY_EXPORT_REQUIRE_FSTYPE only write when the export folder is this
# filesystem type, e.g. cifs; "" = any
# EXPORT_DIR export folder inside the container (/export)
set -u

GRAFANA_URL="${GRAFANA_URL:-http://grafana:3000/grafana}"
UID_="${DISPLAY_DASHBOARD_UID:-safefamily-display}"
WIDTH="${DISPLAY_WIDTH:-1280}"
HEIGHT="${DISPLAY_HEIGHT:-800}"
INTERVAL="${DISPLAY_INTERVAL:-300}"
RETRY="${DISPLAY_RETRY:-60}"
FILE="${DISPLAY_FILE:-grafana-status.png}"
DIR="${EXPORT_DIR:-/export}"
REQUIRE_FSTYPE="${DISPLAY_EXPORT_REQUIRE_FSTYPE-cifs}"
TMP="$DIR/.$FILE.tmp"

log() { echo "[display-export] $(date '+%Y-%m-%d %H:%M:%S') $*" >&2; }

# Filesystem type of the export folder, from this process's mount table.
fstype_of() {
awk -v m="$1" '{ for (i = 1; i <= NF; i++) if ($i == "-") { if ($5 == m) t = $(i + 1); break } } END { print t }' \
/proc/self/mountinfo
}

# PNG signature and IHDR width/height (big-endian, bytes 17-24).
png_size() {
sig="$(head -c 8 "$1" | od -An -tx1 | tr -d ' \n')"
[ "$sig" = "89504e470d0a1a0a" ] || return 1
dims="$(head -c 24 "$1" | tail -c 8 | od -An -tu1)"
# shellcheck disable=SC2086 # split the eight byte values on purpose
set -- $dims
echo "$(( ($1 << 24) + ($2 << 16) + ($3 << 8) + $4 ))x$(( ($5 << 24) + ($6 << 16) + ($7 << 8) + $8 ))"
}

export_once() {
if [ -n "$REQUIRE_FSTYPE" ]; then
actual="$(fstype_of "$DIR")"
case "|$REQUIRE_FSTYPE|" in
*"|$actual|"*) ;;
*)
log "not writing: $DIR is '${actual:-not a mount}', expected $REQUIRE_FSTYPE (share not mounted when this container started? docker compose restart display-export)"
return 1 ;;
esac
fi
url="$GRAFANA_URL/render/d/$UID_?orgId=1&width=$WIDTH&height=$HEIGHT&kiosk=true&theme=dark&timeout=60"
[ -n "${DISPLAY_TZ:-}" ] && url="$url&tz=$DISPLAY_TZ"
if ! curl -fsS --max-time 120 -u "${GRAFANA_USER:-admin}:${GRAFANA_PASSWORD:-}" -o "$TMP" "$url"; then
log "render request failed"
rm -f "$TMP"
return 1
fi
size="$(png_size "$TMP")" || { log "Grafana did not return a PNG"; rm -f "$TMP"; return 1; }
if [ "$size" != "${WIDTH}x${HEIGHT}" ]; then
log "PNG is $size, expected ${WIDTH}x${HEIGHT}"
rm -f "$TMP"
return 1
fi
mv -f "$TMP" "$DIR/$FILE" && log "wrote $DIR/$FILE ($size, $(wc -c < "$DIR/$FILE") bytes)"
}

# Grafana answers /api/health once it is ready (no login needed).
wait_for_grafana() {
tries=0
until curl -fsS --max-time 5 -o /dev/null "$GRAFANA_URL/api/health" 2> /dev/null; do
tries=$((tries + 1))
[ "$tries" -eq 1 ] && log "waiting for Grafana at $GRAFANA_URL"
[ "$tries" -ge 60 ] && { log "Grafana still not ready after 5 minutes, trying anyway"; return 0; }
sleep 5
done
}

if [ "${1:-}" = "--once" ]; then
export_once
exit $?
fi
wait_for_grafana
while :; do
if export_once; then
sleep "$INTERVAL"
else
# Retry soon so one failure does not make the display go STALE.
sleep "$RETRY"
fi
done
77 changes: 77 additions & 0 deletions deploy/monitoring/display-export/make_display_dashboard.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
#!/usr/bin/env python3
"""Build the 1280x800 display dashboard from panels of the overview dashboard.

Copies the Status and AdGuard panels (same queries, mappings and colors) into
deploy/monitoring/grafana/dashboards/safefamily-display.json, laid out to fill
one 1280x800 screenshot in kiosk mode. Rerun after changing those panels:

python3 deploy/monitoring/display-export/make_display_dashboard.py
"""

import copy
import json
from pathlib import Path

DASHBOARDS = Path(__file__).resolve().parents[1] / "grafana" / "dashboards"

# (title in the overview dashboard, x, y, w, h) on Grafana's 24-column grid.
LAYOUT = [
("Website", 0, 0, 6, 5),
("Database", 6, 0, 6, 5),
("App containers running", 12, 0, 6, 5),
("Lowest disk free", 18, 0, 6, 5),
("Page load time", 0, 5, 6, 5),
("TLS cert expires in", 6, 5, 6, 5),
("Last DB backup", 12, 5, 6, 5),
("Database size", 18, 5, 6, 5),
("AdGuard", 0, 10, 4, 5),
("Protection", 4, 10, 4, 5),
("Blocked now", 8, 10, 8, 5),
("Blocked queries (24h)", 16, 10, 4, 5),
("DNS queries (24h)", 20, 10, 4, 5),
("Blocking timeline", 0, 15, 24, 5),
]


def main() -> None:
"""Write the display dashboard JSON."""
overview = json.loads((DASHBOARDS / "safefamily-overview.json").read_text())
by_title = {}
for p in overview["panels"]:
# First panel with a title wins: "Database size" is both the Status
# tile and, further down, a graph.
if p.get("type") != "row":
by_title.setdefault(p["title"], p)
panels = []
for panel_id, (title, x, y, w, h) in enumerate(LAYOUT, start=1):
panel = copy.deepcopy(by_title[title])
panel["id"] = panel_id
panel["gridPos"] = {"x": x, "y": y, "w": w, "h": h}
panel["description"] = "" # no info icons on the display
if panel["type"] == "state-timeline":
# Rows are labelled already; the legend would only show thresholds.
panel["options"]["legend"] = {"showLegend": False}
panels.append(panel)
dashboard = {
"uid": "safefamily-display",
"title": "SafeFamily Display",
"tags": ["safefamily", "display"],
"timezone": "browser",
"schemaVersion": overview["schemaVersion"],
"version": 1,
"editable": False,
"refresh": "",
"time": {"from": "now-24h", "to": "now"},
# No time picker bar: the whole 1280x800 is panels.
"timepicker": {"hidden": True},
"panels": panels,
"templating": {"list": []},
"annotations": {"list": []},
}
out = DASHBOARDS / "safefamily-display.json"
out.write_text(json.dumps(dashboard, indent=2) + "\n")
print(f"wrote {out}")


if __name__ == "__main__":
main()
Loading
Loading