Skip to content

Monitoring stack, encrypted weekly DB backups, AdGuard checks and P4 status picture - #36

Merged
zzuse merged 22 commits into
mainfrom
claude/technical-blog-markdown-yqktk3
Nov 2, 2026
Merged

zzuse merged 22 commits into
mainfrom
claude/technical-blog-markdown-yqktk3

Conversation

@zzuse

@zzuse zzuse commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Summary

Adds server monitoring and encrypted database backups for SafeFamily, tuned on the real (snap-Docker) host. Everything is opt-in through compose overlays and systemd timers; the app's Python code is unchanged.

Monitoring (deploy/monitoring/docker-compose.monitoring.yml)

  • Prometheus, Alertmanager (email through the app's Gmail settings) and Grafana with a provisioned SafeFamily Overview dashboard.
  • Exporters: node-exporter (disk, CPU, RAM), cadvisor (per-container), postgres-exporter, nginx-exporter (internal stub_status), blackbox (HTTPS page load and TLS expiry).
  • 24 alert rules (disk, containers, database, website/TLS, backups, network shares, AdGuard, nginx status), with promtool unit tests in prometheus/tests/.
  • docker-compose.host-node-exporter.yml: runs node-exporter on the host for snap Docker, where containers see the snap's own filesystem as /.
  • Drop-in prometheus/scrape.d/ and rules.d/ folders for user additions.
  • Samba/NFS shares: generic unreachable/missing alerts, and WATCH_MOUNTS keeps automounted shares visible.
  • AdGuard: scripts/monitoring/adguard_check.py reports whether AdGuard is up, whether protection is on, query counts, and whether YouTube is blocked (from AdGuard's blocked-services list). Grafana shows a blocking timeline.
  • docker-compose.display-export.yml: Grafana image renderer plus an exporter that writes a 1280x800 PNG of a generated SafeFamily Display dashboard to a share for the ESP32-P4 display. It writes atomically to a fixed name, refuses to write unless the folder is a mounted share, and retries quickly.
  • nginx serves Grafana at https://<site>/grafana/ for home-network addresses only (403 otherwise). nginx still starts when monitoring is off.

Encrypted backups

  • scripts/db_backup.sh: weekly full pg_dump, verified with pg_restore --list before it's kept.
    • Encrypted with age to public keys only, so the server can't read its own backups.
    • Keeps 8 weekly and 6 monthly. Retention counts backups rather than days, so failed backups never prune the last good ones.
    • Writes metrics for the backup alerts. Optional BACKUP_GROUP lets a normal user copy the files.
  • scripts/db_restore.sh: restore drill into a scratch database (with --drop-test cleanup), or a confirmed live restore. Can take the decrypted dump streamed over ssh, so the private key never touches the server.

Operations

  • scripts/monitoring/preflight.sh (--deep): read-only host check that prints the fix for each problem. It covers:
    • Docker and Compose versions, snap install, containerd socket, .env settings
    • disk headroom (including Prometheus's history), ports, TLS certificate, SMTP
    • backup key and timers, watched mounts, AdGuard
    • a stale nginx config, and the live scrape targets
  • docs/monitoring.md and docs/operations-checklist.md: setup, the alert reference, restore steps, routine tasks, troubleshooting.

Notes for deploying

  • After merging: docker compose up -d, then docker compose up -d --force-recreate proxy. The nginx config is a single-file mount, so a running container keeps the old copy.
  • Grafana now serves from /grafana/. Direct access is http://127.0.0.1:3000/grafana/.

Testing

  • Each piece was run end to end in Docker in a sandbox, with stand-ins for the app and for AdGuard. That includes the dashboard and display screenshots and a backup → restore round trip.
  • Shell scripts pass shellcheck, ruff check . is clean, the existing pytest suite passes, and the promtool rule tests pass.
  • Snap confinement, the real Samba shares and the real AdGuard could not be reproduced in the sandbox; the preflight checks them on the host.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21

claude added 21 commits October 29, 2026 20:18
Monitoring (deploy/monitoring/, a compose overlay): Prometheus with 17
alert rules, Alertmanager mailing through the app's Gmail settings, and
Grafana with a provisioned SafeFamily Overview dashboard. Exporters:
node-exporter (disk free per mount, host CPU/RAM), cadvisor (per-container
CPU/memory/up), postgres-exporter (DB up/size/connections), nginx-exporter
(via an internal-only stub_status server) and blackbox (HTTPS page load
and TLS expiry).

Backups: scripts/db_backup.sh dumps the full database, verifies it with
pg_restore --list and encrypts it with age to public keys only, so the
server cannot decrypt its own backups. Daily/monthly retention, metrics
for backup-age alerts, systemd timers. scripts/db_restore.sh restores into
a scratch DB for drills, or into the live DB with explicit confirmation,
including from a stream so the private key never touches the server.
scripts/monitoring/dir_sizes.sh reports directory sizes for Grafana.

Setup, alert runbook and restore steps are in docs/monitoring.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
Backups now run weekly (Sundays 03:17, Persistent=true) and keep the
newest 8 weekly and 6 monthly. Retention is count-based, so a long run of
failed backups never prunes the last good ones. The missing-backup alert
and Grafana tile thresholds move to 8 days.

scripts/monitoring/preflight.sh checks the host before and after setup:
Docker/Compose versions, snap install, Docker root dir, containerd image
store and socket, ports, disk, RAM, TLS certificate, SMTP reachability,
age keys, backup dirs and timers, and the live scrape targets; --deep
also test-mounts the paths the exporters need. Each failure prints its
fix.

cadvisor's containerd socket is now set with CONTAINERD_SOCKET (its
--containerd flag) instead of a bind mount, which conflicted with the
read-only /var/run mount. ALERT_EMAIL_TO defaulting moves into the
Alertmanager entrypoint, avoiding nested variable defaults that older
Compose versions may not support.

docs/operations-checklist.md: requirements table, step-by-step setup
with a check per step, routine tasks, troubleshooting and how to turn
it off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
…er backup checks

Fixes misleading results seen on a real snap-Docker host:
- DOCKER_ROOT_DIR was always reported missing; now compared against the
  merged compose config (cadvisor's mount), or noted as pending while
  compose does not parse.
- The COMPOSE_FILE warning was a side effect of a compose parse error;
  it is now skipped until the files parse.
- A Grafana password set as GF_SECURITY_ADMIN_PASSWORD (Grafana's
  internal name) gets a rename hint (checks the name only, never values).
- Backup checks collapse to one warning until Phase 2 is started, so
  monitoring can be set up first. The textfile dir moves to the
  monitoring prerequisites.
- Disk check uses free/size like the DiskSpaceLow alert and reserves
  Prometheus's retention size on the Docker root filesystem, suggesting
  the largest PROMETHEUS_RETENTION_SIZE that keeps 15% free.
- Adds --help, rejects unknown options, notes when not run as root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
… verify /

- try_run prints Docker's actual error lines instead of the trailing
  "Run 'docker run --help'" footer, which hid the cause.
- The textfile mount test uses the directory compose really mounts and
  skips with a hint when it does not exist yet. If the mount is refused
  and /var/snap/docker/common is mountable (snap Docker's own data
  area), it prints the steps to move the textfile dir there.
- New check: scripts' NODE_EXPORTER_TEXTFILE_DIR must match the one
  node-exporter mounts, or backup metrics silently never arrive.
- New check: the size of / seen inside a container must match the host's,
  otherwise every disk metric and alert for / would be wrong.
- Docs: snap textfile dir and troubleshooting rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
dir_sizes.sh derives the repo path from its own location; the /srv
placeholder made the repo and logs folders silently vanish from the
Directory sizes panel when the checkout lives elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
Backups were root-only (umask 077), so scp/rsync as a login user failed.
They are age-encrypted, so letting one group read them is safe:
BACKUP_GROUP makes the backup dirs 750 and files 640 for that group,
applied to existing backups too. A missing group only logs a warning.
Checklist restore drill updated (list, then scp; sudo fallback).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
The printed hint 'dropdb -U "$DB_USER"' was expanded by the caller's
shell, where DB_USER is normally unset, so it failed. --drop-test drops
the scratch database as the container's POSTGRES_USER, like the restore
itself, and is safe to repeat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
On a real snap-Docker host, containers see the snap's own ~4 GB
filesystem as /, and the textfile mount is refused (read-only file
system). The node-exporter container therefore reports wrong disk sizes
and never delivers the backup metrics (empty Grafana tiles).

docker-compose.host-node-exporter.yml disables the container (profile)
and maps the name node-exporter to the host for Prometheus, so
prometheus.yml, the dashboard and the alerts stay unchanged. The host
runs Debian's prometheus-node-exporter reading the same textfile dir.

preflight: uses the user's COMPOSE_FILE when it includes monitoring,
detects host mode, checks the host exporter answers on :9100 and reads
the scripts' textfile dir (by process name, not command-line text), and
on a different / recommends host mode instead of the snap textfile dir.
Docs: install steps, ufw rule, explicit removal of the old container
(compose --remove-orphans keeps profile-disabled containers).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
Verified with cadvisor v0.54.1 on a classic overlay2 store (non-default
root, like snap): its docker factory still fails without a reachable
containerd socket, and containers then have no names, so the
"App containers running" tile is empty and ContainerDown fires.

- The containerd socket check now runs for every image store (it was
  skipped for classic stores with a wrong "reads layer metadata
  directly" pass), prefers snap's own socket on snap Docker, and flags
  a CONTAINERD_SOCKET that points elsewhere.
- Running stack: new check that cadvisor names app/db/proxy; if not,
  prints the decisive cadvisor log lines and a fix matched to them.
- Docs: troubleshooting for the unnamed-containers case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
…backup.env

- When cadvisor's containerd socket differs from the real one, say why:
  no line in the repo's .env, a commented-out line, the setting placed in
  /etc/safefamily/backup.env (Compose never reads it), or an empty shell
  variable overriding .env. (Compose accepts spaces, quotes and export in
  .env, so formatting is not a cause.)
- backup.env was sourced with set -a, exporting all of it into the
  preflight's own docker compose calls; a CONTAINERD_SOCKET there made the
  check pass while the real stack still used the default. Now only the
  three settings preflight needs are read, in a subshell, unexported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
- prometheus.yml loads scrape.d/*.yml (scrape_config_files) and
  rules.d/*.yml, so new sources and alerts need no edits to tracked
  files. READMEs with examples in both dirs.
- New network_mounts alerts for any cifs/smb3/nfs mount, no per-share
  config: NetworkMountUnreachable (node-exporter device error for 5m) and
  NetworkMountMissing (mounted within 7 days, gone now); Alertmanager
  inhibits Missing while the same share is Unreachable.
- promtool unit tests for both (tests/network_mounts_test.yml).
- docs/monitoring.md section 4: how to add each kind of source, Samba
  specifics (checking collection, old node-exporter mount filters,
  safe fstab options).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
A share with x-systemd.automount is only an autofs placeholder until
first access, which node-exporter ignores: invisible in Grafana after
every reboot, and NetworkMountMissing would fire although nothing broke.

- dir_sizes.sh: for each WATCH_MOUNTS entry, read the mount point with a
  10s limit (which also wakes the automount) and require a real
  non-autofs filesystem mounted exactly there; writes safefamily_mount_ok.
  Runs before du; unset removes the metrics file.
- WatchedMountDown alert (two failed checks). NetworkMountMissing waits
  20m to cover reboots. Alertmanager sends one mail per broken share.
- dirsize service gets TimeoutStartSec so a hung share cannot pile up.
- promtool tests extended; docs explain automount and WATCH_MOUNTS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
For each watched mount: mounted for real or only an automount
placeholder (or not a mount point), watch check result, a ProtectHome
sandbox blocking the host exporter from /home, and whether node-exporter
reports the share, reports a read error, or filters it out (with its
version). Verified node-exporter 1.9.1 reports a filesystem stacked on
an excluded type, as cifs is on autofs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
A device error with permission denied comes from a parent folder the
exporter's user cannot pass through, typically a 750 home directory on
Ubuntu. Preflight (as root) now finds the first such folder, as the
service's own user, and prints a setfacl pass-through grant for it;
deeper folders are checked on the next run since everything below a
blocked folder looks blocked too. Reproduced with node-exporter 1.9.1
running as prometheus under a 750 home.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
The nginx config is a single-file bind mount. git pull replaces the file
(new inode), so a running proxy keeps the old copy, and neither
docker compose up -d (no compose change) nor nginx -s reload picks it
up; only recreating the container does. Verified both ways. The
checklist wrongly said up -d recreates proxy.

- preflight: checks nginx_up (the exporter is 'up' even without data);
  if the container's config lacks the stub_status server the repo has,
  says so and prints the force-recreate command.
- NginxStatusUnavailable alert (nginx_up == 0 for 5m).
- Checklist step 3 and routine table: force-recreate proxy after nginx
  config changes; troubleshooting row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
scripts/monitoring/adguard_check.py (host, every minute via
safefamily-adguard.timer) logs in to the AdGuard API and writes textfile
metrics: API up, protection on, 24h queries/blocked, and per watched
domain (ADGUARD_WATCH_DOMAINS, default youtube.com) whether AdGuard would
block it now, taken from check_host so blocked services (how SafeFamily
blocks YouTube), filter rules and parental control all count. The login
stays in a root-only /etc/safefamily/adguard.env.

- Alerts: AdGuardDown, AdGuardProtectionDisabled, AdGuardCheckStale,
  with promtool tests. A schedule-aware YouTube alert is documented as a
  rules.d example (tested), since allowed YouTube is often intended.
- Grafana: AdGuard row with status tiles and a blocking timeline.
- preflight: AdGuard section (file permissions, freshness, login, current
  block state).
Tested against a mock AdGuard API (auth, password with ':' and spaces,
blocked/allowed, rewrite not counted, protection off, wrong password,
API down) and end to end through node-exporter, Prometheus and Grafana.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
…lerated

On a real router check_host timed out after 10s (the app also sees
AdGuard ReadTimeouts). check_host walks all filters and, with Safe
Browsing/Parental Control on, waits for AdGuard's online lookups, which
stall when the router's internet path is slow or switched off.

- New ADGUARD_WATCH_SERVICES (default youtube): reads AdGuard's
  blocked-services list, the one SafeFamily switches; fast and local.
  Uses blocked_services/get, falling back to the older /list API.
  New metric safefamily_adguard_service_blocked.
- ADGUARD_TIMEOUT (default 30s); a timed-out check_host logs why and
  what to change; ADGUARD_WATCH_DOMAINS="" turns it off.
- collect() split per check kind (ruff C901).
- Grafana tiles/timeline show services and domains; the YouTube night
  alert example uses the service metric; preflight shows service state
  and warns when check_host fails.
Tested against a mock with a hanging check_host and both API versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
docker-compose.display-export.yml (opt-in overlay):
- grafana-renderer (grafana-image-renderer v4.0.17) behind Grafana's
  /render API.
- display-export (curl image, profile display-export): every 5 minutes
  renders the SafeFamily Display dashboard, checks the result is a PNG of
  exactly 1280x800, writes a hidden temp file and renames it over one fixed
  name (grafana-status.png). Failed renders keep the last good picture.
  Runs as the share's uid/gid and refuses to write unless the folder is a
  cifs mount, so pictures never hide under an unmounted mount point.
- Host fallback for snap Docker: the same script via
  safefamily-display-export.timer.

grafana/dashboards/safefamily-display.json is generated from the overview's
Status and AdGuard panels by make_display_dashboard.py, laid out for one
1280x800 kiosk screenshot (no time picker, no timeline legend).

Rendered end to end with live data; tested unmounted/mounted folder, a
renderer outage, uid ownership, and the host-script path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
After docker compose up recreates Grafana, the exporter's first render
hit a Grafana that was not listening yet (connection refused) and then
waited the full 5-minute interval. It now waits for /api/health before
the first export and retries a failed export after DISPLAY_RETRY (60s),
so a restart cannot push the picture past the display's 10-minute STALE
limit. Tested with the exporter started while Grafana was down.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
nginx: location /grafana/ allowed for 10.0.0.0/8 and 192.168.0.0/16 (403
otherwise; Grafana's login on top), no rate limit (dashboards load many
files), WebSocket upgrade for Grafana Live, and the upstream resolved per
request so nginx still starts without the monitoring stack (502 then).
Grafana: serve_from_sub_path with GF_SERVER_ROOT_URL (GRAFANA_ROOT_URL,
default https://zzuse.duckdns.org/grafana/). Renderer callback, display
exporter and its host fallback use the /grafana sub path.

Tested: nginx without monitoring (app 200, /grafana/ 502); LAN client gets
the login page, assets, API and a 101 WebSocket upgrade; a non-LAN client
gets 403 while the app stays reachable; direct :3000/grafana works; the
display export still renders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-02T20:23:55.333140Z ea18638 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ea18638e54

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/monitoring/preflight.sh Resolved
/etc/safefamily and other safefamily-* units also come from the AdGuard
and display-export setups, so AdGuard-first users got backup failures
instead of the 'not set up yet' warning. Now only the recipients file,
backup.env, the backup dir or safefamily-backup.* units count.
(Codex review on #36.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
@zzuse
zzuse merged commit 9d77ce8 into main Nov 2, 2026
2 checks passed
@zzuse
zzuse deleted the claude/technical-blog-markdown-yqktk3 branch November 2, 2026 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants