feat(monitoring): migrate the stack to rootless Podman (CT 4001) — ADR-0009 Phase 2b - #308
Merged
Merged
Conversation
…nitoring stack Two engine defects that only a real, large, multi-image stack exposes. Both found on CT 4001 (#303). 1. ASSETS BLEW PAST THE COMMAND-LENGTH LIMIT. #305 embedded the whole asset tree in the single `pct exec` deploy script. The monitoring tree is 286 KiB of base64 (a 100 KB generated snmp.yml + a 97 KB Grafana dashboard) and the SSH connection was reset outright — "Connection reset by peer", with NOTHING executed, not even the first useradd. Not a timeout, so yesterday's keepalives were irrelevant. Measured on this path: 96 KiB of command line works, 128 KiB does not. So my 1 MiB cap in #305 was ~10x too generous, and the guard I wrote specifically to prevent a baffling failure produced one. Fixed architecturally rather than with a smaller number: assets now go out as separate 32 KiB chunks BEFORE the deploy script (first chunk truncates, rest append, decode once), so tree size stops mattering. The cap is demoted to a 16 MiB sanity bound. A test asserts no single command exceeds 64 KiB. 2. ROOTLESS CANNOT UNPACK DISTROLESS IMAGES without ignore_chown_errors. The exportarr image owns files as uid 65532 (/home/nonroot), which is outside the subuid window the LXC's own 65536-uid map allows: "potentially insufficient UIDs or GIDs available in user namespace (requested 65532:65532 for /home/nonroot)" Widening the range can't fix it — mapping container uid 65532 needs ~65533 subuids inside a 65536-wide window, leaving nothing for real accounts. The provisioner now writes ~/.config/containers/storage.conf with ignore_chown_errors=true, before anything pulls an image. Note that only fixes the PULL. RUNNING such an image as its own high uid additionally needs `UserNS=keep-id:uid=<uid>` on the quadlet, or crun refuses: "setgroups: Invalid argument". Verified both halves by hand before encoding. 134 engine tests pass (2 new; 3 obsolete ones replaced — they asserted assets appear inside the deploy script, which is now wrong by design). Refs #303, #305
…tover complete Closes #303. Phase 2b of #283 — the last stack in the "clean" tier. CT 4001 replaces the Docker host CT 4000: 10 containers + a shared network as quadlets, 9 config/dashboard files as converge-rendered assets, 4 podman secrets. CT 4000 is STOPPED but NOT destroyed. 3.9 GB of state migrated (tempo 3.3G, pulse 603M, grafana 50M, loki 64K), streamed node-locally in 78s. Verified after cutover: Grafana's 2 dashboards + 3 name- resolving datasources, Pulse healthy with its api_tokens/alerts intact, tempo's 3.3G of blocks present, and 5/5 Prometheus scrape targets UP — which is the real test, since the by-name wiring (exportarr-*/otel-collector/snmp_exporter) fails silently rather than failing a unit. NO MORE HARD-CODED ADDRESSES. CT 4000 pinned 10.10.0.40 and two things referenced it; both now use monitoring.homelab.chrison.internal (UniFi local DNS, attached to the DHCP reservation so name and lease are defined in one place): - OTEL_EXPORTER_OTLP_ENDPOINT in secrets.env.template - stacks/Core/pangolin.lxc.yaml — the Pulse resource. This one was NOT in the original plan; found by grepping for the old IP. Pulse is publicly exposed via Pangolin, so the migration would have silently broken that route. Verified resolvable from CT 2013 (Pangolin, different node) before switching it to the name rather than the new IP. PowerOrchestrator redeployed via the Fallout target so it picks the endpoint up from secrets.env rather than being hand-edited; all five orchestrator_* series confirmed arriving through the new collector. Two corrections to earlier assumptions, both caught by checking the live containers instead of inferring: - pulse runs as ROOT, not 1000. Its data dir being uid 1000 had me put keep-id:uid=1000 on it, which would have mapped container-root to a subuid and left it unable to write its own data. Removed — the default mapping is right. - prometheus runs as nobody (65534) and needs keep-id, plus an EXPLICIT data volume: podman's anonymous volume isn't owned by the mapped uid, so it panicked on "/prometheus/queries.active: permission denied". Fixing that also fixed a pre-existing flaw — compose declared no volume at all, so the TSDB was destroyed on every container recreate. Metrics now persist; a behaviour improvement, not just a port. Runbook gained a wipe-before-extract step: by cutover time CT 4001 had generated its own state, and a plain tar -x would have merged old over new. Closes #303
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #303. Phase 2b of #283 — the last stack in the "clean" tier. Cutover is done and live.
Verified after cutover, not just at start
exportarr-*,otel-collector,snmp_exporter) fails silently rather than failing a unitapi_tokens.json+ alerts intactorchestrator_*series arriving through the new collector3.9 GB migrated in 78s (tempo 3.3G · pulse 603M · grafana 50M · loki 64K), streamed node-locally.
No more hard-coded addresses
Both consumers now use
monitoring.homelab.chrison.internal(UniFi local DNS, attached to the DHCP reservation so name and lease are defined in one place):OTEL_EXPORTER_OTLP_ENDPOINT— PowerOrchestrator redeployed via the Fallout target, not hand-editedstacks/Core/pangolin.lxc.yaml— the Pulse resource. Not in the original plan; found by grepping for the old IP. Pulse is publicly exposed, so the migration would have silently broken that route.Three findings worth review
The Pangolin resource reconciler is add-only. Editing
target.ipin the shape updates desired state but never touches an existing resource, sopulse.lab.chrison.devstayed pointed at the stopped CT 4000 and returned errors until I PATCHed target 15 via the integration API. Now HTTP 200 →monitoring.homelab.chrison.internal:7655. Any future host migration hits this — worth its own issue.Two of my own assumptions were wrong, both caught by checking live containers instead of inferring:
pulseruns as root, not 1000. Its data dir's 1000 ownership had me putkeep-id:uid=1000on it, which would have mapped container-root to a subuid and left it unable to write its own data.prometheusruns as nobody (65534) and needskeep-idplus an explicit data volume — podman's anonymous volume isn't owned by the mapped uid, so it panicked on/prometheus/queries.active: permission denied. Fixing that also fixed a pre-existing flaw: compose declared no volume, so the TSDB was destroyed on every recreate. Metrics now persist — a behaviour improvement, flagged rather than slipped in.Engine fixes in the first commit
Both found by deploying a real, large, multi-image stack:
config.assets) #305 embedded assets in the singlepct execscript; 286 KiB of base64 reset the SSH connection with nothing executed. Measured limit: 96 KiB works, 128 KiB doesn't — my 1 MiB cap was ~10× too generous. Now 32 KiB chunks sent before the deploy script.storage.confwithignore_chown_errors. Rootless can't unpack distroless images (uid 65532) inside an LXC's 65536-uid window, and widening can't fix it. Running them additionally needskeep-id:uid=65532.134 engine tests pass.
Follow-up (documented in
podman-host/README.md)Once CT 4000 is destroyed: delete
compose.yml,deploy/install.sh,secrets.env.local.template, and the now-duplicatedconfig/+grafana/trees. The duplication is deliberate — it preserves the rollback path while CT 4000 exists.🤖 Generated with Claude Code