From b29c229fe583cdc0235f2cb60eafe3588e0f1765 Mon Sep 17 00:00:00 2001 From: Chrison Simtian Date: Sun, 26 Jul 2026 23:40:28 +1200 Subject: [PATCH 1/2] feat(media): rootless Podman + quadlet host CT 5114, youtarr migrated (Phase 2a) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #302. Phase 2a of #283, on the platform from #284 and the pattern proven in #285. Adds the Media stack's generic container HOST (not a youtarr box) and migrates youtarr + its bundled MariaDB onto it as three quadlets. Replaces Docker host CT 5113, which is left STOPPED but NOT destroyed. First MULTI-container quadlet deployment, so it proves the pattern #303 needs: youtarr.network gives youtarr DNS resolution of youtarr-db via aardvark-dns (verified: youtarr-db.dns.podman -> 10.89.0.2). compose gave that for free; quadlets do not. userns decisions, per stack member rather than per host: - youtarr writes to the NFS share, so NO :U (it chowns recursively — 879 GB over NFS) and NO UserNS= (it runs as container-root, which the default rootless mapping already sends to the podman user). - youtarr-db is the opposite case, a fixed uid 999 on local disk, so keep-id:uid=999,gid=999 with the data dir owned by podman. THE OWNERSHIP PREDICTION WAS WRONG, and better than feared. I expected rootless writes to land under a different host uid than Docker's (101000 vs 100000), leaving two owner uids on the share. They don't: the NAS export already SQUASHES every incoming uid to 1024:100. A probe written by the podman container landed as 1024:100 — byte-identical to files the old Docker setup wrote — and Plex (CT 5008, same share) read it back. So the three userns layers are irrelevant to NFS ownership, and the NAS-side squashing #208 would have added as "the real fix" is already in place. Docs corrected accordingly. Two more findings, both now documented: - depends_on: condition: service_healthy has NO quadlet/systemd equivalent. After= waits for STARTED, not READY, and a host-side ExecStartPre wait can't work because youtarr-db is a podman-network DNS name unresolvable from the CT. Covered the systemd way: Restart=always + RestartSec=10, so youtarr crash-retries until MariaDB accepts connections. - A healthcheck IS expressible here, unlike Mate's: this image ships curl, so the command needs no quotes and survives systemd's parser. Confirms gotcha #10 is about QUOTED commands, not healthchecks generally. MariaDB's own check stays dropped — it needs quoting AND would put the root password in the unit file. YOUTUBE_OUTPUT_DIR is passed as the HOST path (/mnt/youtube) while data mounts at /usr/src/app/data — looks wrong, is exactly how CT 5113 runs, verified against the live container and preserved deliberately. Verified on CT 5114 (hpe-01): NFS mount + hookscript applied, both units active, youtarr healthy, migrated DB intact (channels=8 videos=95 jobs=645, and the app log flipped from "Auto-downloads disabled" on the fresh DB to "enabled" on the migrated one), NFS write→Plex read proven, and boot survival 5s after an LXC restart with podman-auto-update.timer active. Three secrets registered in Bitwarden SM + secrets.env.template, consumed as podman secrets: YOUTARR_DB_ROOT_PASSWORD / _AUTH_USERNAME / _AUTH_PASSWORD. Closes #302 --- secrets.env.template | 6 + stacks/Media/podman-host.lxc.yaml | 71 +++++++++++ stacks/Media/podman-host/README.md | 114 ++++++++++++++++++ .../podman-host/quadlets/youtarr-db.container | 51 ++++++++ .../podman-host/quadlets/youtarr.container | 83 +++++++++++++ .../podman-host/quadlets/youtarr.network | 13 ++ 6 files changed, 338 insertions(+) create mode 100644 stacks/Media/podman-host.lxc.yaml create mode 100644 stacks/Media/podman-host/README.md create mode 100644 stacks/Media/podman-host/quadlets/youtarr-db.container create mode 100644 stacks/Media/podman-host/quadlets/youtarr.container create mode 100644 stacks/Media/podman-host/quadlets/youtarr.network diff --git a/secrets.env.template b/secrets.env.template index 78821c0..887f12b 100644 --- a/secrets.env.template +++ b/secrets.env.template @@ -88,3 +88,9 @@ OTEL_EXPORTER_OTLP_ENDPOINT=http://10.10.0.40:4317 # NOTE: MATE_SECRET_KEY is deliberately NOT registered — Mate auto-generates it into its # data dir when unset, and it must stay paired with that database (see podman-host/README.md). MATE_AUTH_PASSWORD= + +# ── Youtarr on the Media podman host (CT 5114) — #302 ── +# Consumed as podman SECRETS (not an env file) by stacks/Media/podman-host/quadlets/. +YOUTARR_DB_ROOT_PASSWORD= +YOUTARR_AUTH_USERNAME= +YOUTARR_AUTH_PASSWORD= diff --git a/stacks/Media/podman-host.lxc.yaml b/stacks/Media/podman-host.lxc.yaml new file mode 100644 index 0000000..a9db6f5 --- /dev/null +++ b/stacks/Media/podman-host.lxc.yaml @@ -0,0 +1,71 @@ +# Media podman host — the stack's rootless Podman + quadlet host (ADR-0009 Phase 2a, #302). +# +# DELIBERATELY GENERIC, like the SmartHome one (CT 6004): this is *the Media stack's +# container host*, not the youtarr box. Youtarr is simply the first (and today only) +# Docker member of this stack, so it's what lands here first; further containerised +# Media services join as extra quadlets rather than as new one-app Docker CTs. +# +# Replaces the Docker host CT 5113. In-block either way (Media owns 5100–5199). +# +# THE NFS BIT — the reason this migration is interesting (see podman-host/README.md): +# the youtube share is mounted at the PROXMOX HOST level and passed in as mp0, exactly +# as CT 5113 has it. Rootless podman then adds a THIRD userns layer on top of the +# unprivileged LXC's own. Consequences, in short: +# * NEVER use `:U` on /mnt/youtube — it chowns RECURSIVELY, i.e. 879 GB over NFS. +# * NO UserNS= on the youtarr container: it runs as container-root, which the DEFAULT +# rootless mapping already sends to the `podman` user — one stable uid. keep-id would +# push container-root out into the subuid range and make it worse. +# * youtarr-db is the opposite case (fixed uid 999 on LOCAL disk) → keep-id there. +# Verified live: the NAS export squashes every incoming uid to 1024:100, so rootless writes are +# indistinguishable from the old Docker ones and Plex reads them unchanged. +# +# Validate against ../../Infrastructure/schema/shape.schema.json. + +apiVersion: homelab/v1 +kind: LXC + +metadata: + name: podman-host + stack: Media + tags: [podman, quadlet, container-host, adr-0009] + +spec: + app: podman # community-scripts ct/podman.sh + the `podman` provisioner + ctid: 5114 # in-block (5100–5199); highest in use was 5113 (the CT this replaces) + cores: 2 # matches CT 5113 — node app + MariaDB + yt-dlp work + memory: 2048 # MB — podman + youtarr + MariaDB + disk: 15 # GB — image layers + MariaDB store (~130 MB) + jobs/config; videos are on NFS + + features: + nesting: true # rootless podman needs nesting for its own nested userns + keyctl: true # containers keyring — settable only via the pct/SSH create path + fuse: true # fuse-overlayfs fallback storage driver + # NOTE: ct/podman.sh silently DROPS var_fuse; the provisioner + # reconciles features from this shape after create (#284). + + # node / network / os / storage inherited from stack.yaml defaults (hpe-01, VLAN 1010, DHCP). + + mounts: + - type: nfs + storage: ds1813-nfs-volume-4 + source: youtube # volume4 subpath only (self-provisioned by MountReconciler) + target: /mnt/youtube # same path as CT 5113, so YOUTUBE_OUTPUT_DIR is unchanged + acl: true + backup: false + + # Pre-start guard, carried over from CT 5113 unchanged: refuse to start unless volume4 is + # actually mounted. LXC-level and runtime-agnostic — rootless podman doesn't affect it, and + # it's what protects against the NAS-drop rootfs-fill failure mode. + hookscript: local:snippets/ensure-data-mount.sh + + config: + user: podman + quadlets: podman-host/quadlets + + # podman secrets seeded from secrets.env, add-only. Namespaced per service, because this + # host will grow more of them (the SM/template keys are YOUTARR_*, the podman secret names + # are lower-case). + secrets: + youtarr_db_root_password: YOUTARR_DB_ROOT_PASSWORD + youtarr_auth_username: YOUTARR_AUTH_USERNAME + youtarr_auth_password: YOUTARR_AUTH_PASSWORD diff --git a/stacks/Media/podman-host/README.md b/stacks/Media/podman-host/README.md new file mode 100644 index 0000000..862cfd4 --- /dev/null +++ b/stacks/Media/podman-host/README.md @@ -0,0 +1,114 @@ +# podman-host (CT 5114) — the Media stack's rootless Podman + quadlet host + +The stack's **container host**, not a per-app box. Services arrive as extra quadlets in +[`quadlets/`](quadlets/). Established by ADR-0009 and Phase 2a ([#302](https://github.com/Chrison-Homelab/Homelab/issues/302)); replaces the Docker host **CT 5113**. + +- **CT 5114** — in-block (Media owns 5100–5199), VLAN 1010, DHCP, node `hpe-01` +- **Rootless user** `podman`, subuid `10000:50000` +- **NFS** `/mnt/youtube` via host-level mount → `mp0`, plus the `ensure-data-mount.sh` hookscript + carried over unchanged from CT 5113 +- Read the superproject's `docs/plans/284-podman-platform.md` **before editing a quadlet** + +## Current members + +| Quadlet | Service | Notes | +|---|---|---| +| `youtarr.network` | shared network | **Required** — gives `youtarr` DNS resolution of `youtarr-db` | +| `youtarr-db.container` | MariaDB 10.3 | fixed uid 999 on local disk → `keep-id` | +| `youtarr.container` | Youtarr | writes video to the NFS share → **no** `:U`, **no** `UserNS=` | + +## The three things that make this migration different from Phase 1 + +### 1. NFS through a third userns layer — and `:U` is dangerous here + +``` +NAS 192.168.179.11:/volume4/Volume-4/youtube 7.3T, 879G used + → Proxmox host /mnt/pve/ds1813-nfs-volume-4/youtube + → LXC mp0 /mnt/youtube (unprivileged → files appear as 65534/nobody) + → rootless podman userns + → container /usr/src/app/data +``` + +**Never put `:U` on that volume.** It chowns recursively — 879 GB across NFS, rewriting +ownership on the NAS. (Contrast the Monitoring stack, #303, where `:U` is merely wasteful.) + +**And no `UserNS=` on `youtarr`.** It runs as container-root, which the *default* rootless +mapping already sends to the `podman` user — one stable uid. `keep-id` would align the +*user's* uid instead and push container-root into the subuid range: strictly worse. + +`youtarr-db` is the opposite case — a fixed uid (999) on local disk — so it *does* use +`keep-id:uid=999,gid=999`, with its data dir owned by `podman` on the host. + +**Ownership does NOT change — verified, and better than predicted.** The worry was that +rootless (which cannot map container-root to CT root) would write under a different host uid +than Docker did, leaving two owner uids on the share. + +It doesn't happen: **the NAS export already squashes every incoming uid to `1024:100`.** A probe +file written by the podman container landed as `1024:100` — byte-identical ownership to files the +old Docker setup wrote — and Plex (CT 5008, same share) read it back. So the three userns layers +turn out to be irrelevant to NFS *ownership*; what matters is only that the share is writable at +all. The NAS-side squashing that #208 would have introduced as "the real fix" is already in place. + +### 2. No health-gated dependency + +compose had `depends_on: { youtarr-db: { condition: service_healthy } }`. **Quadlet and systemd +have no equivalent** — `After=` waits for a unit to have *started*, not to be *ready*. A +host-side `ExecStartPre` wait is not an option either, because `youtarr-db` is a +podman-network DNS name and is unresolvable from the CT. + +The readiness gap is covered the systemd way: `Restart=always` + `RestartSec=10`. If youtarr +starts before MariaDB accepts connections it exits and systemd retries until it can. + +### 3. `YOUTUBE_OUTPUT_DIR` is a HOST path, not the mount target + +The live container is told `YOUTUBE_OUTPUT_DIR=/mnt/youtube` while the data is mounted at +`/usr/src/app/data`. That looks like a bug and isn't ours to fix — it's how CT 5113 runs today +(verified against the running container before conversion). Preserved exactly. Do not +"correct" it to the container path. + +## Cutover runbook + +Videos need **no migration** — they live on the NFS share, which the new CT mounts at the same +path. Only ~172 MB of local state moves. + +```bash +# 0. from the superproject: provision the host + quadlets +dotnet run --project Infrastructure/engine -- converge stacks/Media --apply + +# 1. stop the OLD containers (CT 5113 host stays UP — the rollback path) +ssh root@hpe-01 'pct exec 5113 -- docker stop youtarr youtarr-db' + +# 2. stop the NEW units before seeding their data +ssh root@hpe-01 'pct exec 5114 -- bash -lc "cd /; U=\$(id -u podman); + runuser -u podman -- env XDG_RUNTIME_DIR=/run/user/\$U \ + DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/\$U/bus \ + systemctl --user stop youtarr.service youtarr-db.service"' + +# 3. copy local state (database ~130M, jobs ~36M, server ~6M, config ~12K) +ssh root@hpe-01 'pct exec 5113 -- tar -C /opt/youtarr -cf - database config jobs server' > /tmp/youtarr.tar +ssh root@hpe-01 'pct exec 5114 -- mkdir -p /home/podman/youtarr' +ssh root@hpe-01 'pct exec 5114 -- tar -C /home/podman/youtarr -xf -' < /tmp/youtarr.tar +# keep-id maps container-999 to the podman user, so the DB dir must be owned by podman +ssh root@hpe-01 'pct exec 5114 -- chown -R podman:podman /home/podman/youtarr' +rm -f /tmp/youtarr.tar + +# 4. start the new units (DB first; systemd's Requires=/After= also enforces this) +# same runuser incantation as step 2, with `start` + +# 5. verify, in this order +# a. youtarr-db up; youtarr reaches it (DB_HOST=youtarr-db resolves via youtarr.network) +# b. youtarr healthy (its healthcheck IS expressible — the image ships curl) +# c. web UI on http://<5114-ip>:3087 shows the EXISTING channels/history +# d. a NEW download lands on /mnt/youtube and Plex can read it ← the real NFS test +# e. unit survives an LXC reboot + +# 6. leave CT 5113 STOPPED but NOT destroyed +ssh root@hpe-01 'pct stop 5113' +``` + +**Rollback:** `pct start 5113 && pct exec 5113 -- docker start youtarr-db youtarr`, then stop the +new units. The copy duplicates state rather than moving it, so CT 5113 keeps its own untouched +copy. The NFS share is shared between both, so nothing there is at risk either way. + +CT 5113 is left stopped rather than destroyed on purpose — deleting it is a manual decision once +the new host has proven itself, and its DHCP reservation goes with it. diff --git a/stacks/Media/podman-host/quadlets/youtarr-db.container b/stacks/Media/podman-host/quadlets/youtarr-db.container new file mode 100644 index 0000000..bef3239 --- /dev/null +++ b/stacks/Media/podman-host/quadlets/youtarr-db.container @@ -0,0 +1,51 @@ +# Youtarr's bundled MariaDB. Converted from stacks/Media/youtarr/compose.yml. +# +# FIXED-UID CASE — the opposite of the youtarr container next door. MariaDB runs as +# mysql (uid 999) inside the image and its data dir on CT 5113 is owned by 999. So: +# UserNS=keep-id:uid=999,gid=999 → the host `podman` user appears as 999 inside the +# container, so the migrated data dir must be owned by `podman` on the host (the +# migration chowns it once). No :U — that would chown recursively on EVERY start. +# Local disk (~130 MB), so unlike /mnt/youtube this is a safe place for either approach; +# keep-id is chosen for consistency and to avoid repeated recursive chowns as the DB grows. + +[Unit] +Description=Youtarr MariaDB +Documentation=https://github.com/Chrison-Homelab/Homelab/issues/302 +After=network-online.target +Wants=network-online.target + +[Container] +Image=docker.io/library/mariadb:10.3 +ContainerName=youtarr-db +Network=youtarr.network + +# Root password as a podman secret rather than an env file. +Secret=youtarr_db_root_password,type=env,target=MYSQL_ROOT_PASSWORD +Environment=MYSQL_DATABASE=youtarr +Environment=MYSQL_TCP_PORT=3321 +Environment=MYSQL_CHARSET=utf8mb4 +Environment=MYSQL_COLLATION=utf8mb4_unicode_ci + +Volume=/home/podman/youtarr/database:/var/lib/mysql +UserNS=keep-id:uid=999,gid=999 + +# compose `command:` → quadlet Exec=. No quoting needed here, so it survives systemd's +# parser intact (contrast HealthCmd below). +Exec=--port=3321 --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --innodb-file-per-table=1 --innodb-large-prefix=ON + +AutoUpdate=registry + +# NO HEALTHCHECK — deliberate, and for TWO reasons (see gotcha #10 in +# docs/plans/284-podman-platform.md). The compose healthcheck is +# mysqladmin ping -h localhost -P 3321 -u root -p${DB_ROOT_PASSWORD} +# which (a) needs quoting that quadlet destroys, and (b) would put the root password in the +# unit file. Readiness is handled by youtarr's crash-retry instead. + +[Service] +# Podman does not create missing bind-mount source dirs (gotcha #9). +ExecStartPre=/usr/bin/mkdir -p /home/podman/youtarr/database +Restart=always +TimeoutStartSec=300 + +[Install] +WantedBy=default.target diff --git a/stacks/Media/podman-host/quadlets/youtarr.container b/stacks/Media/podman-host/quadlets/youtarr.container new file mode 100644 index 0000000..8ee325b --- /dev/null +++ b/stacks/Media/podman-host/quadlets/youtarr.container @@ -0,0 +1,83 @@ +# Youtarr — YouTube→Plex downloader. Converted from stacks/Media/youtarr/compose.yml. +# +# NFS + USERNS: this container writes downloaded video to the NFS share at +# /mnt/youtube (host-level mount, passed in as mp0). Two rules, both load-bearing: +# +# * NO `:U` on that Volume=. It chowns RECURSIVELY — 879 GB across NFS, rewriting +# ownership on the NAS. Cheap-looking flag, hours of damage. +# * NO UserNS= at all. The app runs as container-root (compose defaulted +# YOUTARR_UID/GID to 0:0, and the live container confirms uid=0), and the DEFAULT +# rootless mapping already sends container-root to the `podman` user — a single +# stable uid. keep-id would align the *user's* uid instead, pushing container-root +# out into the subuid range: strictly worse here. +# +# OWNERSHIP — verified empirically, and it is BETTER than predicted. The concern was that +# rootless (which cannot map container-root to CT root) would write files under a different +# host uid than Docker did, leaving two owner uids on the share. It does not: the NAS export +# ALREADY SQUASHES every incoming uid to 1024:100. A probe written by this container landed as +# 1024:100 — byte-identical ownership to files the old Docker setup wrote — and Plex (CT 5008, +# same share) read it back fine. So the userns layers are irrelevant to NFS ownership here, and +# the "proper fix" of NAS-side squashing that #208 would have added is already in place. + +[Unit] +Description=Youtarr — YouTube to Plex downloader +Documentation=https://github.com/Chrison-Homelab/Homelab/issues/302 +After=network-online.target youtarr-db.service +Wants=network-online.target +# Ordering only. compose had `depends_on: condition: service_healthy`, which has NO +# quadlet/systemd equivalent — After= waits for the unit to have STARTED, not to be +# healthy. The readiness gap is covered by Restart=always + RestartSec below: if youtarr +# starts before MariaDB accepts connections it exits and systemd retries until it can. +# A host-side ExecStartPre wait is NOT an option — `youtarr-db` is a podman-network DNS +# name, unresolvable from the CT. +Requires=youtarr-db.service + +[Container] +Image=docker.io/dialmaster/youtarr:latest +ContainerName=youtarr +Network=youtarr.network +PublishPort=3087:3011 + +Environment=IN_DOCKER_CONTAINER=1 +Environment=TZ=Pacific/Auckland +Environment=DB_HOST=youtarr-db +Environment=DB_PORT=3321 +Environment=DB_USER=root +Environment=DB_NAME=youtarr +Environment=AUTH_ENABLED=true +Environment=LOG_LEVEL=info +# NOTE: this is the HOST path, not the container path, exactly as CT 5113 has it — the live +# container is told /mnt/youtube while the data is mounted at /usr/src/app/data. Verified on +# the running container before conversion; do not "correct" it to the mount target. +Environment=YOUTUBE_OUTPUT_DIR=/mnt/youtube + +Secret=youtarr_db_root_password,type=env,target=DB_PASSWORD +Secret=youtarr_auth_username,type=env,target=AUTH_PRESET_USERNAME +Secret=youtarr_auth_password,type=env,target=AUTH_PRESET_PASSWORD + +# The NFS video output. No :U, no UserNS — see the header. +Volume=/mnt/youtube:/usr/src/app/data +# Local state, migrated from CT 5113. +Volume=/home/podman/youtarr/config:/app/config +Volume=/home/podman/youtarr/jobs:/app/jobs +Volume=/home/podman/youtarr/server/images:/app/server/images + +# A healthcheck IS expressible here, unlike Mate's (gotcha #10): this image ships curl, so the +# command needs no quotes and survives systemd's parser intact. Worth having — it catches a +# hung-but-alive app, which Restart= alone cannot. +HealthCmd=curl --fail --silent --show-error --output /dev/null http://localhost:3011/api/health +HealthInterval=30s +HealthTimeout=10s +HealthRetries=3 +HealthStartPeriod=40s + +AutoUpdate=registry + +[Service] +ExecStartPre=/usr/bin/mkdir -p /home/podman/youtarr/config /home/podman/youtarr/jobs /home/podman/youtarr/server/images +Restart=always +RestartSec=10 +TimeoutStartSec=300 + +[Install] +WantedBy=default.target diff --git a/stacks/Media/podman-host/quadlets/youtarr.network b/stacks/Media/podman-host/quadlets/youtarr.network new file mode 100644 index 0000000..4494766 --- /dev/null +++ b/stacks/Media/podman-host/quadlets/youtarr.network @@ -0,0 +1,13 @@ +# Shared network for the youtarr containers, so `youtarr` can resolve `youtarr-db` by name. +# +# Compose gave every service a shared default network with service-name DNS for free; +# quadlets do NOT. Without this, DB_HOST=youtarr-db does not resolve and youtarr can't +# reach its database. Rootless podman runs aardvark-dns on a user-defined network, which +# is what provides the name resolution. +# +# This is the first multi-container quadlet deployment in the homelab (Phase 1 was a single +# container with no peers), so this file is the pattern the Monitoring migration (#303, ten +# containers) will follow. + +[Network] +NetworkName=youtarr From 944aef6ec9f9489b77798e624c80e15458723d0c Mon Sep 17 00:00:00 2001 From: Chrison Simtian Date: Mon, 27 Jul 2026 00:01:12 +1200 Subject: [PATCH 2/2] docs(284): gotcha 11 + sharpen 10 + the NFS-squashing correction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From Phase 2a (#302), the first MULTI-container quadlet deployment. - #11: depends_on: condition: service_healthy has no quadlet OR systemd equivalent — After=/Requires= wait for STARTED, not READY. And a host-side ExecStartPre wait can't substitute, because container names only resolve inside the podman network while ExecStartPre runs on the CT. Do it the systemd way: Restart=always + RestartSec=10 and let it crash-retry. - Multi-container stacks need an explicit .network quadlet: compose gives service-name DNS for free, quadlets don't. Verified aardvark-dns resolution on CT 5114. - #10 sharpened: the constraint is on QUOTING, not healthchecks as a category. youtarr's curl probe works precisely because it needs no quotes, so check for a quote-free probe before giving up. Also: never put a secret in HealthCmd — it lands in the unit file. - userns section: check whether the NFS export SQUASHES before reasoning about ownership. Phase 2a's predicted 100000-vs-101000 split never materialised because the Synology export maps every incoming uid to 1024:100. Refs #302, #284 --- docs/plans/284-podman-platform.md | 79 +++++++++++++++++++++++++++---- 1 file changed, 70 insertions(+), 9 deletions(-) diff --git a/docs/plans/284-podman-platform.md b/docs/plans/284-podman-platform.md index 74d04e5..c7d48d3 100644 --- a/docs/plans/284-podman-platform.md +++ b/docs/plans/284-podman-platform.md @@ -37,12 +37,13 @@ Quadlet files (`*.container`, `*.volume`, `*.network`, `*.pod`) go in the `quadl rendered into `~/.config/containers/systemd/`. No `*.kube` — ADR-0009 rules out any `podman kube` path. -## The ten gotchas +## The eleven gotchas All were found by deploying, not by reading docs. **#1–4, #7 and #8 are fixed in the engine** — -you inherit those. **#5, #6, #9 and #10 are rules you must follow when authoring quadlet files**; -nothing enforces them yet. #9 and #10 came out of the first real workload (Phase 1, Leapmotor -Mate on CT 6004) rather than the throwaway host. +you inherit those. **#5, #6, #9, #10 and #11 are rules you must follow when authoring quadlet +files**; nothing enforces them yet. #9 and #10 came out of the first real workload (Phase 1, +Leapmotor Mate on CT 6004); #11 came out of the first *multi-container* one (Phase 2a, youtarr +on CT 5114). ### 1. Rootless networking needs `/dev/net/tun` — an LXC has none @@ -203,11 +204,64 @@ matter how you write it. Both forms verified broken on CT 6004: The first is the dangerous one: the container sits permanently **`unhealthy`** while the application serves HTTP 200 perfectly well, so any monitoring keyed on health status lies. -A healthcheck here must be **quote-free**. If the check needs quotes (e.g. a python one-liner, -because the image ships no `curl`/`wget`), you cannot express it as a quadlet — either drop it and -rely on `Restart=always` for process death, or render a script onto the host and call it -quote-free (`HealthCmd=python /opt/health.py`). The latter needs the provisioner to render -non-quadlet assets, which it currently does not — a platform change, not a quadlet tweak. +A healthcheck here must be **quote-free**. This is a constraint on *quoting*, not on healthchecks +as a category — youtarr's works fine, because its image ships `curl` and the command needs no +quotes at all: + +```ini +HealthCmd=curl --fail --silent --show-error --output /dev/null http://localhost:3011/api/health +``` + +So: check whether the image gives you a quote-free probe before giving up. If the check genuinely +needs quotes (a python one-liner, because the image ships no `curl`/`wget`), you cannot express it +as a quadlet — either drop it and rely on `Restart=always` for process death, or render a script +onto the host and call it quote-free (`HealthCmd=python /opt/health.py`). The latter needs the +provisioner to render non-quadlet assets, which it currently does not — a platform change, not a +quadlet tweak. + +Also never put a **secret** in `HealthCmd`: it lands in the unit file. youtarr's MariaDB check +(`mysqladmin ping … -p$DB_ROOT_PASSWORD`) was dropped for that reason as much as the quoting. + +### 11. `depends_on: condition: service_healthy` has NO quadlet or systemd equivalent + +compose can gate one service on another being *healthy*. Quadlet cannot, and neither can systemd: +`After=`/`Requires=` wait for a unit to have **started**, not to be **ready**. Converting a +compose file with a health-gated `depends_on` therefore silently loses the gate. + +Nor can you paper over it with a host-side wait: + +```ini +# WRONG — youtarr-db is a podman-network DNS name, unresolvable from the CT +ExecStartPre=/usr/bin/sh -c until nc -z youtarr-db 3321; do sleep 2; done +``` + +The container names only resolve *inside* the podman network (via aardvark-dns), and +`ExecStartPre` runs on the CT. + +Do it the systemd way instead — let it fail and retry: + +```ini +[Service] +Restart=always +RestartSec=10 +``` + +The dependent unit crash-loops harmlessly until its dependency accepts connections. Keep +`After=`/`Requires=` as well, so startup *order* is still right and the dependency is pulled in. + +### Multi-container stacks need an explicit `.network` + +Not a gotcha so much as a missing default: compose gives every service a shared network with +service-name DNS for free. **Quadlets do not.** Two containers that talk to each other need a +`.network` quadlet and `Network=.network` on each, which is what enables aardvark-dns: + +```ini +# youtarr.network +[Network] +NetworkName=youtarr +``` + +Verified on CT 5114: `youtarr-db.dns.podman` → `10.89.0.2`, resolvable from the youtarr container. ## Nested user namespaces (the risk ADR-0009 flagged) @@ -216,6 +270,13 @@ An unprivileged LXC is itself userns-mapped — the host grants a window (conven a rootless user `100000:65536` points outside that window and podman fails at first run (`newuidmap: write to uid_map failed`). +**Before agonising over NFS ownership, check whether the export squashes.** Phase 2a expected +rootless writes to land under a different host uid than Docker's, leaving two owner uids on the +share — and it simply doesn't happen: the Synology export maps *every* incoming uid to `1024:100`, +so podman's writes are byte-identical to Docker's and Plex reads them unchanged. The userns layers +turn out to be irrelevant to ownership on that share. Verify with a probe file rather than +reasoning about the map. + Default is therefore `podman:10000:50000` — below `65534` (nobody), leaving room for real accounts. The provisioner **verifies** rather than assumes, reading field 3 of `/proc/self/uid_map` and failing loudly if `start + count` exceeds it. A container needing a uid