Repository navigation
Monitoring stack, encrypted weekly DB backups, AdGuard checks and P4 status picture - #36
Merged
Merged
Conversation
Monitoring (deploy/monitoring/, a compose overlay): Prometheus with 17 alert rules, Alertmanager mailing through the app's Gmail settings, and Grafana with a provisioned SafeFamily Overview dashboard. Exporters: node-exporter (disk free per mount, host CPU/RAM), cadvisor (per-container CPU/memory/up), postgres-exporter (DB up/size/connections), nginx-exporter (via an internal-only stub_status server) and blackbox (HTTPS page load and TLS expiry). Backups: scripts/db_backup.sh dumps the full database, verifies it with pg_restore --list and encrypts it with age to public keys only, so the server cannot decrypt its own backups. Daily/monthly retention, metrics for backup-age alerts, systemd timers. scripts/db_restore.sh restores into a scratch DB for drills, or into the live DB with explicit confirmation, including from a stream so the private key never touches the server. scripts/monitoring/dir_sizes.sh reports directory sizes for Grafana. Setup, alert runbook and restore steps are in docs/monitoring.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
Backups now run weekly (Sundays 03:17, Persistent=true) and keep the newest 8 weekly and 6 monthly. Retention is count-based, so a long run of failed backups never prunes the last good ones. The missing-backup alert and Grafana tile thresholds move to 8 days. scripts/monitoring/preflight.sh checks the host before and after setup: Docker/Compose versions, snap install, Docker root dir, containerd image store and socket, ports, disk, RAM, TLS certificate, SMTP reachability, age keys, backup dirs and timers, and the live scrape targets; --deep also test-mounts the paths the exporters need. Each failure prints its fix. cadvisor's containerd socket is now set with CONTAINERD_SOCKET (its --containerd flag) instead of a bind mount, which conflicted with the read-only /var/run mount. ALERT_EMAIL_TO defaulting moves into the Alertmanager entrypoint, avoiding nested variable defaults that older Compose versions may not support. docs/operations-checklist.md: requirements table, step-by-step setup with a check per step, routine tasks, troubleshooting and how to turn it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
…er backup checks Fixes misleading results seen on a real snap-Docker host: - DOCKER_ROOT_DIR was always reported missing; now compared against the merged compose config (cadvisor's mount), or noted as pending while compose does not parse. - The COMPOSE_FILE warning was a side effect of a compose parse error; it is now skipped until the files parse. - A Grafana password set as GF_SECURITY_ADMIN_PASSWORD (Grafana's internal name) gets a rename hint (checks the name only, never values). - Backup checks collapse to one warning until Phase 2 is started, so monitoring can be set up first. The textfile dir moves to the monitoring prerequisites. - Disk check uses free/size like the DiskSpaceLow alert and reserves Prometheus's retention size on the Docker root filesystem, suggesting the largest PROMETHEUS_RETENTION_SIZE that keeps 15% free. - Adds --help, rejects unknown options, notes when not run as root. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
… verify / - try_run prints Docker's actual error lines instead of the trailing "Run 'docker run --help'" footer, which hid the cause. - The textfile mount test uses the directory compose really mounts and skips with a hint when it does not exist yet. If the mount is refused and /var/snap/docker/common is mountable (snap Docker's own data area), it prints the steps to move the textfile dir there. - New check: scripts' NODE_EXPORTER_TEXTFILE_DIR must match the one node-exporter mounts, or backup metrics silently never arrive. - New check: the size of / seen inside a container must match the host's, otherwise every disk metric and alert for / would be wrong. - Docs: snap textfile dir and troubleshooting rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
dir_sizes.sh derives the repo path from its own location; the /srv placeholder made the repo and logs folders silently vanish from the Directory sizes panel when the checkout lives elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
Backups were root-only (umask 077), so scp/rsync as a login user failed. They are age-encrypted, so letting one group read them is safe: BACKUP_GROUP makes the backup dirs 750 and files 640 for that group, applied to existing backups too. A missing group only logs a warning. Checklist restore drill updated (list, then scp; sudo fallback). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
The printed hint 'dropdb -U "$DB_USER"' was expanded by the caller's shell, where DB_USER is normally unset, so it failed. --drop-test drops the scratch database as the container's POSTGRES_USER, like the restore itself, and is safe to repeat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
On a real snap-Docker host, containers see the snap's own ~4 GB filesystem as /, and the textfile mount is refused (read-only file system). The node-exporter container therefore reports wrong disk sizes and never delivers the backup metrics (empty Grafana tiles). docker-compose.host-node-exporter.yml disables the container (profile) and maps the name node-exporter to the host for Prometheus, so prometheus.yml, the dashboard and the alerts stay unchanged. The host runs Debian's prometheus-node-exporter reading the same textfile dir. preflight: uses the user's COMPOSE_FILE when it includes monitoring, detects host mode, checks the host exporter answers on :9100 and reads the scripts' textfile dir (by process name, not command-line text), and on a different / recommends host mode instead of the snap textfile dir. Docs: install steps, ufw rule, explicit removal of the old container (compose --remove-orphans keeps profile-disabled containers). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
Verified with cadvisor v0.54.1 on a classic overlay2 store (non-default root, like snap): its docker factory still fails without a reachable containerd socket, and containers then have no names, so the "App containers running" tile is empty and ContainerDown fires. - The containerd socket check now runs for every image store (it was skipped for classic stores with a wrong "reads layer metadata directly" pass), prefers snap's own socket on snap Docker, and flags a CONTAINERD_SOCKET that points elsewhere. - Running stack: new check that cadvisor names app/db/proxy; if not, prints the decisive cadvisor log lines and a fix matched to them. - Docs: troubleshooting for the unnamed-containers case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
…backup.env - When cadvisor's containerd socket differs from the real one, say why: no line in the repo's .env, a commented-out line, the setting placed in /etc/safefamily/backup.env (Compose never reads it), or an empty shell variable overriding .env. (Compose accepts spaces, quotes and export in .env, so formatting is not a cause.) - backup.env was sourced with set -a, exporting all of it into the preflight's own docker compose calls; a CONTAINERD_SOCKET there made the check pass while the real stack still used the default. Now only the three settings preflight needs are read, in a subshell, unexported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
- prometheus.yml loads scrape.d/*.yml (scrape_config_files) and rules.d/*.yml, so new sources and alerts need no edits to tracked files. READMEs with examples in both dirs. - New network_mounts alerts for any cifs/smb3/nfs mount, no per-share config: NetworkMountUnreachable (node-exporter device error for 5m) and NetworkMountMissing (mounted within 7 days, gone now); Alertmanager inhibits Missing while the same share is Unreachable. - promtool unit tests for both (tests/network_mounts_test.yml). - docs/monitoring.md section 4: how to add each kind of source, Samba specifics (checking collection, old node-exporter mount filters, safe fstab options). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
A share with x-systemd.automount is only an autofs placeholder until first access, which node-exporter ignores: invisible in Grafana after every reboot, and NetworkMountMissing would fire although nothing broke. - dir_sizes.sh: for each WATCH_MOUNTS entry, read the mount point with a 10s limit (which also wakes the automount) and require a real non-autofs filesystem mounted exactly there; writes safefamily_mount_ok. Runs before du; unset removes the metrics file. - WatchedMountDown alert (two failed checks). NetworkMountMissing waits 20m to cover reboots. Alertmanager sends one mail per broken share. - dirsize service gets TimeoutStartSec so a hung share cannot pile up. - promtool tests extended; docs explain automount and WATCH_MOUNTS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
For each watched mount: mounted for real or only an automount placeholder (or not a mount point), watch check result, a ProtectHome sandbox blocking the host exporter from /home, and whether node-exporter reports the share, reports a read error, or filters it out (with its version). Verified node-exporter 1.9.1 reports a filesystem stacked on an excluded type, as cifs is on autofs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
A device error with permission denied comes from a parent folder the exporter's user cannot pass through, typically a 750 home directory on Ubuntu. Preflight (as root) now finds the first such folder, as the service's own user, and prints a setfacl pass-through grant for it; deeper folders are checked on the next run since everything below a blocked folder looks blocked too. Reproduced with node-exporter 1.9.1 running as prometheus under a 750 home. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
The nginx config is a single-file bind mount. git pull replaces the file (new inode), so a running proxy keeps the old copy, and neither docker compose up -d (no compose change) nor nginx -s reload picks it up; only recreating the container does. Verified both ways. The checklist wrongly said up -d recreates proxy. - preflight: checks nginx_up (the exporter is 'up' even without data); if the container's config lacks the stub_status server the repo has, says so and prints the force-recreate command. - NginxStatusUnavailable alert (nginx_up == 0 for 5m). - Checklist step 3 and routine table: force-recreate proxy after nginx config changes; troubleshooting row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
scripts/monitoring/adguard_check.py (host, every minute via safefamily-adguard.timer) logs in to the AdGuard API and writes textfile metrics: API up, protection on, 24h queries/blocked, and per watched domain (ADGUARD_WATCH_DOMAINS, default youtube.com) whether AdGuard would block it now, taken from check_host so blocked services (how SafeFamily blocks YouTube), filter rules and parental control all count. The login stays in a root-only /etc/safefamily/adguard.env. - Alerts: AdGuardDown, AdGuardProtectionDisabled, AdGuardCheckStale, with promtool tests. A schedule-aware YouTube alert is documented as a rules.d example (tested), since allowed YouTube is often intended. - Grafana: AdGuard row with status tiles and a blocking timeline. - preflight: AdGuard section (file permissions, freshness, login, current block state). Tested against a mock AdGuard API (auth, password with ':' and spaces, blocked/allowed, rewrite not counted, protection off, wrong password, API down) and end to end through node-exporter, Prometheus and Grafana. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
…lerated On a real router check_host timed out after 10s (the app also sees AdGuard ReadTimeouts). check_host walks all filters and, with Safe Browsing/Parental Control on, waits for AdGuard's online lookups, which stall when the router's internet path is slow or switched off. - New ADGUARD_WATCH_SERVICES (default youtube): reads AdGuard's blocked-services list, the one SafeFamily switches; fast and local. Uses blocked_services/get, falling back to the older /list API. New metric safefamily_adguard_service_blocked. - ADGUARD_TIMEOUT (default 30s); a timed-out check_host logs why and what to change; ADGUARD_WATCH_DOMAINS="" turns it off. - collect() split per check kind (ruff C901). - Grafana tiles/timeline show services and domains; the YouTube night alert example uses the service metric; preflight shows service state and warns when check_host fails. Tested against a mock with a hanging check_host and both API versions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
docker-compose.display-export.yml (opt-in overlay): - grafana-renderer (grafana-image-renderer v4.0.17) behind Grafana's /render API. - display-export (curl image, profile display-export): every 5 minutes renders the SafeFamily Display dashboard, checks the result is a PNG of exactly 1280x800, writes a hidden temp file and renames it over one fixed name (grafana-status.png). Failed renders keep the last good picture. Runs as the share's uid/gid and refuses to write unless the folder is a cifs mount, so pictures never hide under an unmounted mount point. - Host fallback for snap Docker: the same script via safefamily-display-export.timer. grafana/dashboards/safefamily-display.json is generated from the overview's Status and AdGuard panels by make_display_dashboard.py, laid out for one 1280x800 kiosk screenshot (no time picker, no timeline legend). Rendered end to end with live data; tested unmounted/mounted folder, a renderer outage, uid ownership, and the host-script path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
After docker compose up recreates Grafana, the exporter's first render hit a Grafana that was not listening yet (connection refused) and then waited the full 5-minute interval. It now waits for /api/health before the first export and retries a failed export after DISPLAY_RETRY (60s), so a restart cannot push the picture past the display's 10-minute STALE limit. Tested with the exporter started while Grafana was down. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
nginx: location /grafana/ allowed for 10.0.0.0/8 and 192.168.0.0/16 (403 otherwise; Grafana's login on top), no rate limit (dashboards load many files), WebSocket upgrade for Grafana Live, and the upstream resolved per request so nginx still starts without the monitoring stack (502 then). Grafana: serve_from_sub_path with GF_SERVER_ROOT_URL (GRAFANA_ROOT_URL, default https://zzuse.duckdns.org/grafana/). Renderer callback, display exporter and its host fallback use the /grafana sub path. Tested: nginx without monitoring (app 200, /grafana/ 502); LAN client gets the login page, assets, API and a 101 WebSocket upgrade; a non-LAN client gets 403 while the app stays reachable; direct :3000/grafana works; the display export still renders. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ea18638e54
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
/etc/safefamily and other safefamily-* units also come from the AdGuard and display-export setups, so AdGuard-first users got backup failures instead of the 'not set up yet' warning. Now only the recipients file, backup.env, the backup dir or safefamily-backup.* units count. (Codex review on #36.) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds server monitoring and encrypted database backups for SafeFamily, tuned on the real (snap-Docker) host. Everything is opt-in through compose overlays and systemd timers; the app's Python code is unchanged.
Monitoring (
deploy/monitoring/docker-compose.monitoring.yml)stub_status), blackbox (HTTPS page load and TLS expiry).promtoolunit tests inprometheus/tests/.docker-compose.host-node-exporter.yml: runs node-exporter on the host for snap Docker, where containers see the snap's own filesystem as/.prometheus/scrape.d/andrules.d/folders for user additions.WATCH_MOUNTSkeeps automounted shares visible.scripts/monitoring/adguard_check.pyreports whether AdGuard is up, whether protection is on, query counts, and whether YouTube is blocked (from AdGuard's blocked-services list). Grafana shows a blocking timeline.docker-compose.display-export.yml: Grafana image renderer plus an exporter that writes a 1280x800 PNG of a generated SafeFamily Display dashboard to a share for the ESP32-P4 display. It writes atomically to a fixed name, refuses to write unless the folder is a mounted share, and retries quickly.https://<site>/grafana/for home-network addresses only (403 otherwise). nginx still starts when monitoring is off.Encrypted backups
scripts/db_backup.sh: weekly fullpg_dump, verified withpg_restore --listbefore it's kept.ageto public keys only, so the server can't read its own backups.BACKUP_GROUPlets a normal user copy the files.scripts/db_restore.sh: restore drill into a scratch database (with--drop-testcleanup), or a confirmed live restore. Can take the decrypted dump streamed over ssh, so the private key never touches the server.Operations
scripts/monitoring/preflight.sh(--deep): read-only host check that prints the fix for each problem. It covers:.envsettingsdocs/monitoring.mdanddocs/operations-checklist.md: setup, the alert reference, restore steps, routine tasks, troubleshooting.Notes for deploying
docker compose up -d, thendocker compose up -d --force-recreate proxy. The nginx config is a single-file mount, so a running container keeps the old copy./grafana/. Direct access ishttp://127.0.0.1:3000/grafana/.Testing
ruff check .is clean, the existing pytest suite passes, and thepromtoolrule tests pass.🤖 Generated with Claude Code
https://claude.ai/code/session_01CKZWtPcMA8pSrVvGGwno21