feat(observe): podman metrics + prune timer (ADR-0009 phase 3) - #313
Merged
Conversation
Advances #283 phase 3 ("observe"). The migration removed docker.sock and put nothing in its place, so the three rootless hosts had ZERO container-level visibility: systemd knew whether a unit was up, but nothing exported restart counts, image ages or per-container resource use. Engine (platform-wide): - Weekly `podman-system-prune.timer` as USER units, replacing the prune sidecar ADR-0009 retired. Deliberately NOT --volumes: an unattached volume may hold a stopped service's database, and losing it to a housekeeping timer would be unrecoverable. Images/containers/networks only, older than 168h. - Opt-in rootless API socket via `config.userSocket`. The ROOT podman.socket stays masked — this is a socket owned by the unprivileged `podman` user inside its own userns, which exists because an exporter has no other way to enumerate containers. Off by default, so a host that exports nothing has no API surface. - Both now run BEFORE app units start. Found the hard way: when podman-exporter failed to start, `set -e` aborted the script and the timer + socket steps never ran at all — platform housekeeping was hostage to an application unit. A unit that Requires=podman.socket also needs the socket up first. Regression-tested. Stacks: podman-exporter quadlet on the Monitoring and Media hosts, plus a `podman` scrape job. Prometheus now reports 7/7 targets UP. The exporter took three attempts, all recorded in the quadlet header: 1. image's own uid 65534 → crun "setgroups: Invalid argument" (outside the subuid window an LXC's 65536-uid map allows); 2. keep-id:uid=65534 fixed that but left no home → died on "stat /home/.config: no such file or directory" seeking containers.conf; 3. User=0 under the DEFAULT mapping — container-root maps to the unprivileged `podman` user, which is exactly who owns the socket. No keep-id, no home hack, and no privilege gain. CT 5114 also gained the DHCP reservation + DNS name it was missing (media.homelab.chrison.internal), so the new scrape target is a NAME — the IP I first wrote there would have re-introduced exactly what #303 removed. NOT done, and deliberately: - The SmartHome host (CT 6004) is unscraped. It sits on IoT VLAN 1040, firewalled off from 1010; verified with a TCP probe from CT 4001. Reaching it needs a narrow UniFi allowance (1010 -> 1040 on :9882), which loosens the IoT VLAN's isolation and is an owner decision. - Cockpit (the other half of phase 3) is untouched: it defaults to :9090, which Prometheus already publishes on CT 4001, and it sits awkwardly against the repo's stated minimal-UI-reliance. Needs a decision, not an assumption. 138 engine tests pass. Refs #283
…0009 phase 3) Completes phase 3 of #283. Cockpit is the management UI ADR-0009 named, now on all three rootless hosts. Prometheus moves to host port 9091 so Cockpit can have its default 9090. Only the PUBLISHED port changes — the container still listens on 9090, so Grafana's datasource (http://prometheus:9090, over the quadlet network) needed no change at all. Verified after: 7/7 scrape targets UP and "Prometheus Server is Healthy" from inside the Grafana container. Cockpit deliberately authenticates as the `podman` USER, not root: rootless containers exist only inside that user's session, so a root Cockpit session shows an empty container list. Both `podman` and `root` ship password-LOCKED on a community-scripts CT, so the provisioner sets the user's password from PODMAN_USER_PASSWORD (generated, in Bitwarden SM). cockpit-podman then reads the same rootless socket the exporter uses — which is why cockpit requires userSocket. Requesting cockpit without the secret FAILS loudly rather than installing a login UI nobody can log into. Ordering bug found live on CT 4001: enabling cockpit.socket before the app units restart fails with `Result: resources`, because Prometheus still holds :9090 on its old published port until it restarts. The package + password now land before app units (so they exist even if a unit later fails) while the SOCKET is enabled last, after units have rebound. Regression-tested in both directions. DNS: all three podman hosts are now resolvable by name — monitoring/media/smarthome.homelab.chrison.internal. The SmartHome host had no name at all; added via UniFi's standalone static-DNS API rather than a DHCP reservation, since that host is only ever reached by name. 141 engine tests pass. Refs #283
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Advances #283 phase 3 ("observe"). The migration removed
docker.sockand put nothing in its place, so the three rootless hosts had zero container-level visibility — systemd knew whether a unit was up, but nothing exported restart counts, image ages or per-container resource use.Result: Prometheus now reports 7/7 targets UP, including both new podman exporters.
Engine (platform-wide)
podman-system-prune.timeras USER units, replacing the prune sidecar ADR-0009 retired. Deliberately not--volumes: an unattached volume may hold a stopped service's database, and losing it to a housekeeping timer would be unrecoverable. Images/containers/networks only, older than 168h.config.userSocket). The ROOTpodman.socketstays masked — this is a socket owned by the unprivilegedpodmanuser inside its own userns, which exists because an exporter has no other way to enumerate containers. Off by default, so a host that exports nothing keeps no API surface.podman-exporterfailed to start,set -eaborted the script and the timer + socket steps never ran at all — platform housekeeping was hostage to an application unit. A unit thatRequires=podman.socketalso needs the socket up first. Regression-tested in both directions.Stacks
podman-exporterquadlet on the Monitoring and Media hosts + apodmanscrape job.The exporter took three attempts, all recorded in the quadlet header so nobody repeats them:
crun: setgroups: Invalid argument(outside the subuid window an LXC's 65536-uid map allows);keep-id:uid=65534fixed that but left the process with no home → died onstat /home/.config: no such file or directorylooking forcontainers.conf;User=0under the default mapping — container-root maps to the unprivilegedpodmanuser, which is exactly who owns the socket. No keep-id, no home hack, and no privilege gain.CT 5114 also gained the DHCP reservation + DNS name it was missing (
media.homelab.chrison.internal), so the new scrape target is a name — the IP I first wrote there would have re-introduced exactly what #303 removed.Deliberately not done
1010 → 1040on:9882), which loosens the IoT VLAN's isolation and is an owner decision.138 engine tests pass.
🤖 Generated with Claude Code