feat(media): rootless Podman + quadlet host CT 5114, youtarr migrated (ADR-0009 Phase 2a)

docs/plans/284-podman-platform.md

@@ -37,12 +37,13 @@ Quadlet files (`*.container`, `*.volume`, `*.network`, `*.pod`) go in the `quadl
 rendered into `~<user>/.config/containers/systemd/`. No `*.kube` — ADR-0009 rules out any
 `podman kube` path.
 
-## The ten gotchas
+## The eleven gotchas
 
 All were found by deploying, not by reading docs. **#1–4, #7 and #8 are fixed in the engine** —
-you inherit those. **#5, #6, #9 and #10 are rules you must follow when authoring quadlet files**;
-nothing enforces them yet. #9 and #10 came out of the first real workload (Phase 1, Leapmotor
-Mate on CT 6004) rather than the throwaway host.
+you inherit those. **#5, #6, #9, #10 and #11 are rules you must follow when authoring quadlet
+files**; nothing enforces them yet. #9 and #10 came out of the first real workload (Phase 1,
+Leapmotor Mate on CT 6004); #11 came out of the first *multi-container* one (Phase 2a, youtarr
+on CT 5114).
 
 ### 1. Rootless networking needs `/dev/net/tun` — an LXC has none
 
@@ -203,11 +204,64 @@ matter how you write it. Both forms verified broken on CT 6004:
 The first is the dangerous one: the container sits permanently **`unhealthy`** while the
 application serves HTTP 200 perfectly well, so any monitoring keyed on health status lies.
 
-A healthcheck here must be **quote-free**. If the check needs quotes (e.g. a python one-liner,
-because the image ships no `curl`/`wget`), you cannot express it as a quadlet — either drop it and
-rely on `Restart=always` for process death, or render a script onto the host and call it
-quote-free (`HealthCmd=python /opt/health.py`). The latter needs the provisioner to render
-non-quadlet assets, which it currently does not — a platform change, not a quadlet tweak.
+A healthcheck here must be **quote-free**. This is a constraint on *quoting*, not on healthchecks
+as a category — youtarr's works fine, because its image ships `curl` and the command needs no
+quotes at all:
+
+```ini
+HealthCmd=curl --fail --silent --show-error --output /dev/null http://localhost:3011/api/health
+```
+
+So: check whether the image gives you a quote-free probe before giving up. If the check genuinely
+needs quotes (a python one-liner, because the image ships no `curl`/`wget`), you cannot express it
+as a quadlet — either drop it and rely on `Restart=always` for process death, or render a script
+onto the host and call it quote-free (`HealthCmd=python /opt/health.py`). The latter needs the
+provisioner to render non-quadlet assets, which it currently does not — a platform change, not a
+quadlet tweak.
+
+Also never put a **secret** in `HealthCmd`: it lands in the unit file. youtarr's MariaDB check
+(`mysqladmin ping … -p$DB_ROOT_PASSWORD`) was dropped for that reason as much as the quoting.
+
+### 11. `depends_on: condition: service_healthy` has NO quadlet or systemd equivalent
+
+compose can gate one service on another being *healthy*. Quadlet cannot, and neither can systemd:
+`After=`/`Requires=` wait for a unit to have **started**, not to be **ready**. Converting a
+compose file with a health-gated `depends_on` therefore silently loses the gate.
+
+Nor can you paper over it with a host-side wait:
+
+```ini
+# WRONG — youtarr-db is a podman-network DNS name, unresolvable from the CT
+ExecStartPre=/usr/bin/sh -c until nc -z youtarr-db 3321; do sleep 2; done
+```
+
+The container names only resolve *inside* the podman network (via aardvark-dns), and
+`ExecStartPre` runs on the CT.
+
+Do it the systemd way instead — let it fail and retry:
+
+```ini
+[Service]
+Restart=always
+RestartSec=10
+```
+
+The dependent unit crash-loops harmlessly until its dependency accepts connections. Keep
+`After=`/`Requires=` as well, so startup *order* is still right and the dependency is pulled in.
+
+### Multi-container stacks need an explicit `.network`
+
+Not a gotcha so much as a missing default: compose gives every service a shared network with
+service-name DNS for free. **Quadlets do not.** Two containers that talk to each other need a
+`.network` quadlet and `Network=<name>.network` on each, which is what enables aardvark-dns:
+
+```ini
+# youtarr.network
+[Network]
+NetworkName=youtarr
+```
+
+Verified on CT 5114: `youtarr-db.dns.podman` → `10.89.0.2`, resolvable from the youtarr container.
 
 ## Nested user namespaces (the risk ADR-0009 flagged)
 
@@ -216,6 +270,13 @@ An unprivileged LXC is itself userns-mapped — the host grants a window (conven
 a rootless user `100000:65536` points outside that window and podman fails at first run
 (`newuidmap: write to uid_map failed`).
 
+**Before agonising over NFS ownership, check whether the export squashes.** Phase 2a expected
+rootless writes to land under a different host uid than Docker's, leaving two owner uids on the
+share — and it simply doesn't happen: the Synology export maps *every* incoming uid to `1024:100`,
+so podman's writes are byte-identical to Docker's and Plex reads them unchanged. The userns layers
+turn out to be irrelevant to ownership on that share. Verify with a probe file rather than
+reasoning about the map.
+
 Default is therefore `podman:10000:50000` — below `65534` (nobody), leaving room for real
 accounts. The provisioner **verifies** rather than assumes, reading field 3 of
 `/proc/self/uid_map` and failing loudly if `start + count` exceeds it. A container needing a uid

secrets.env.template

@@ -88,3 +88,9 @@ OTEL_EXPORTER_OTLP_ENDPOINT=http://10.10.0.40:4317
 # NOTE: MATE_SECRET_KEY is deliberately NOT registered — Mate auto-generates it into its
 # data dir when unset, and it must stay paired with that database (see podman-host/README.md).
 MATE_AUTH_PASSWORD=
+
+# ── Youtarr on the Media podman host (CT 5114) — #302 ──
+# Consumed as podman SECRETS (not an env file) by stacks/Media/podman-host/quadlets/.
+YOUTARR_DB_ROOT_PASSWORD=
+YOUTARR_AUTH_USERNAME=
+YOUTARR_AUTH_PASSWORD=

stacks/Media/podman-host.lxc.yaml

@@ -0,0 +1,71 @@
+# Media podman host — the stack's rootless Podman + quadlet host (ADR-0009 Phase 2a, #302).
+#
+# DELIBERATELY GENERIC, like the SmartHome one (CT 6004): this is *the Media stack's
+# container host*, not the youtarr box. Youtarr is simply the first (and today only)
+# Docker member of this stack, so it's what lands here first; further containerised
+# Media services join as extra quadlets rather than as new one-app Docker CTs.
+#
+# Replaces the Docker host CT 5113. In-block either way (Media owns 5100–5199).
+#
+# THE NFS BIT — the reason this migration is interesting (see podman-host/README.md):
+# the youtube share is mounted at the PROXMOX HOST level and passed in as mp0, exactly
+# as CT 5113 has it. Rootless podman then adds a THIRD userns layer on top of the
+# unprivileged LXC's own. Consequences, in short:
+#   * NEVER use `:U` on /mnt/youtube — it chowns RECURSIVELY, i.e. 879 GB over NFS.
+#   * NO UserNS= on the youtarr container: it runs as container-root, which the DEFAULT
+#     rootless mapping already sends to the `podman` user — one stable uid. keep-id would
+#     push container-root out into the subuid range and make it worse.
+#   * youtarr-db is the opposite case (fixed uid 999 on LOCAL disk) → keep-id there.
+# Verified live: the NAS export squashes every incoming uid to 1024:100, so rootless writes are
+# indistinguishable from the old Docker ones and Plex reads them unchanged.
+#
+# Validate against ../../Infrastructure/schema/shape.schema.json.
+
+apiVersion: homelab/v1
+kind: LXC
+
+metadata:
+  name: podman-host
+  stack: Media
+  tags: [podman, quadlet, container-host, adr-0009]
+
+spec:
+  app: podman            # community-scripts ct/podman.sh + the `podman` provisioner
+  ctid: 5114             # in-block (5100–5199); highest in use was 5113 (the CT this replaces)
+  cores: 2               # matches CT 5113 — node app + MariaDB + yt-dlp work
+  memory: 2048           # MB — podman + youtarr + MariaDB
+  disk: 15               # GB — image layers + MariaDB store (~130 MB) + jobs/config; videos are on NFS
+
+  features:
+    nesting: true        # rootless podman needs nesting for its own nested userns
+    keyctl: true         # containers keyring — settable only via the pct/SSH create path
+    fuse: true           # fuse-overlayfs fallback storage driver
+                         # NOTE: ct/podman.sh silently DROPS var_fuse; the provisioner
+                         # reconciles features from this shape after create (#284).
+
+  # node / network / os / storage inherited from stack.yaml defaults (hpe-01, VLAN 1010, DHCP).
+
+  mounts:
+    - type: nfs
+      storage: ds1813-nfs-volume-4
+      source: youtube                    # volume4 subpath only (self-provisioned by MountReconciler)
+      target: /mnt/youtube               # same path as CT 5113, so YOUTUBE_OUTPUT_DIR is unchanged
+      acl: true
+      backup: false
+
+  # Pre-start guard, carried over from CT 5113 unchanged: refuse to start unless volume4 is
+  # actually mounted. LXC-level and runtime-agnostic — rootless podman doesn't affect it, and
+  # it's what protects against the NAS-drop rootfs-fill failure mode.
+  hookscript: local:snippets/ensure-data-mount.sh
+
+  config:
+    user: podman
+    quadlets: podman-host/quadlets
+
+    # podman secrets seeded from secrets.env, add-only. Namespaced per service, because this
+    # host will grow more of them (the SM/template keys are YOUTARR_*, the podman secret names
+    # are lower-case).
+    secrets:
+      youtarr_db_root_password: YOUTARR_DB_ROOT_PASSWORD
+      youtarr_auth_username: YOUTARR_AUTH_USERNAME
+      youtarr_auth_password: YOUTARR_AUTH_PASSWORD

stacks/Media/podman-host/README.md

@@ -0,0 +1,114 @@
+# podman-host (CT 5114) — the Media stack's rootless Podman + quadlet host
+
+The stack's **container host**, not a per-app box. Services arrive as extra quadlets in
+[`quadlets/`](quadlets/). Established by ADR-0009 and Phase 2a ([#302](https://github.com/Chrison-Homelab/Homelab/issues/302)); replaces the Docker host **CT 5113**.
+
+- **CT 5114** — in-block (Media owns 5100–5199), VLAN 1010, DHCP, node `hpe-01`
+- **Rootless user** `podman`, subuid `10000:50000`
+- **NFS** `/mnt/youtube` via host-level mount → `mp0`, plus the `ensure-data-mount.sh` hookscript
+  carried over unchanged from CT 5113
+- Read the superproject's `docs/plans/284-podman-platform.md` **before editing a quadlet**
+
+## Current members
+
+| Quadlet | Service | Notes |
+|---|---|---|
+| `youtarr.network` | shared network | **Required** — gives `youtarr` DNS resolution of `youtarr-db` |
+| `youtarr-db.container` | MariaDB 10.3 | fixed uid 999 on local disk → `keep-id` |
+| `youtarr.container` | Youtarr | writes video to the NFS share → **no** `:U`, **no** `UserNS=` |
+
+## The three things that make this migration different from Phase 1
+
+### 1. NFS through a third userns layer — and `:U` is dangerous here
+
+```
+NAS 192.168.179.11:/volume4/Volume-4/youtube      7.3T, 879G used
+  → Proxmox host  /mnt/pve/ds1813-nfs-volume-4/youtube
+  → LXC mp0       /mnt/youtube        (unprivileged → files appear as 65534/nobody)
+  → rootless podman userns
+  → container     /usr/src/app/data
+```
+
+**Never put `:U` on that volume.** It chowns recursively — 879 GB across NFS, rewriting
+ownership on the NAS. (Contrast the Monitoring stack, #303, where `:U` is merely wasteful.)
+
+**And no `UserNS=` on `youtarr`.** It runs as container-root, which the *default* rootless
+mapping already sends to the `podman` user — one stable uid. `keep-id` would align the
+*user's* uid instead and push container-root into the subuid range: strictly worse.
+
+`youtarr-db` is the opposite case — a fixed uid (999) on local disk — so it *does* use
+`keep-id:uid=999,gid=999`, with its data dir owned by `podman` on the host.
+
+**Ownership does NOT change — verified, and better than predicted.** The worry was that
+rootless (which cannot map container-root to CT root) would write under a different host uid
+than Docker did, leaving two owner uids on the share.
+
+It doesn't happen: **the NAS export already squashes every incoming uid to `1024:100`.** A probe
+file written by the podman container landed as `1024:100` — byte-identical ownership to files the
+old Docker setup wrote — and Plex (CT 5008, same share) read it back. So the three userns layers
+turn out to be irrelevant to NFS *ownership*; what matters is only that the share is writable at
+all. The NAS-side squashing that #208 would have introduced as "the real fix" is already in place.
+
+### 2. No health-gated dependency
+
+compose had `depends_on: { youtarr-db: { condition: service_healthy } }`. **Quadlet and systemd
+have no equivalent** — `After=` waits for a unit to have *started*, not to be *ready*. A
+host-side `ExecStartPre` wait is not an option either, because `youtarr-db` is a
+podman-network DNS name and is unresolvable from the CT.
+
+The readiness gap is covered the systemd way: `Restart=always` + `RestartSec=10`. If youtarr
+starts before MariaDB accepts connections it exits and systemd retries until it can.
+
+### 3. `YOUTUBE_OUTPUT_DIR` is a HOST path, not the mount target
+
+The live container is told `YOUTUBE_OUTPUT_DIR=/mnt/youtube` while the data is mounted at
+`/usr/src/app/data`. That looks like a bug and isn't ours to fix — it's how CT 5113 runs today
+(verified against the running container before conversion). Preserved exactly. Do not
+"correct" it to the container path.
+
+## Cutover runbook
+
+Videos need **no migration** — they live on the NFS share, which the new CT mounts at the same
+path. Only ~172 MB of local state moves.
+
+```bash
+# 0. from the superproject: provision the host + quadlets
+dotnet run --project Infrastructure/engine -- converge stacks/Media --apply
+
+# 1. stop the OLD containers (CT 5113 host stays UP — the rollback path)
+ssh root@hpe-01 'pct exec 5113 -- docker stop youtarr youtarr-db'
+
+# 2. stop the NEW units before seeding their data
+ssh root@hpe-01 'pct exec 5114 -- bash -lc "cd /; U=\$(id -u podman);
+  runuser -u podman -- env XDG_RUNTIME_DIR=/run/user/\$U \
+    DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/\$U/bus \
+    systemctl --user stop youtarr.service youtarr-db.service"'
+
+# 3. copy local state (database ~130M, jobs ~36M, server ~6M, config ~12K)
+ssh root@hpe-01 'pct exec 5113 -- tar -C /opt/youtarr -cf - database config jobs server' > /tmp/youtarr.tar
+ssh root@hpe-01 'pct exec 5114 -- mkdir -p /home/podman/youtarr'
+ssh root@hpe-01 'pct exec 5114 -- tar -C /home/podman/youtarr -xf -' < /tmp/youtarr.tar
+# keep-id maps container-999 to the podman user, so the DB dir must be owned by podman
+ssh root@hpe-01 'pct exec 5114 -- chown -R podman:podman /home/podman/youtarr'
+rm -f /tmp/youtarr.tar
+
+# 4. start the new units (DB first; systemd's Requires=/After= also enforces this)
+#    same runuser incantation as step 2, with `start`
+
+# 5. verify, in this order
+#    a. youtarr-db up; youtarr reaches it (DB_HOST=youtarr-db resolves via youtarr.network)
+#    b. youtarr healthy (its healthcheck IS expressible — the image ships curl)
+#    c. web UI on http://<5114-ip>:3087 shows the EXISTING channels/history
+#    d. a NEW download lands on /mnt/youtube and Plex can read it   ← the real NFS test
+#    e. unit survives an LXC reboot
+
+# 6. leave CT 5113 STOPPED but NOT destroyed
+ssh root@hpe-01 'pct stop 5113'
+```
+
+**Rollback:** `pct start 5113 && pct exec 5113 -- docker start youtarr-db youtarr`, then stop the
+new units. The copy duplicates state rather than moving it, so CT 5113 keeps its own untouched
+copy. The NFS share is shared between both, so nothing there is at risk either way.
+
+CT 5113 is left stopped rather than destroyed on purpose — deleting it is a manual decision once
+the new host has proven itself, and its DHCP reservation goes with it.

stacks/Media/podman-host/quadlets/youtarr-db.container

@@ -0,0 +1,51 @@
+# Youtarr's bundled MariaDB. Converted from stacks/Media/youtarr/compose.yml.
+#
+# FIXED-UID CASE — the opposite of the youtarr container next door. MariaDB runs as
+# mysql (uid 999) inside the image and its data dir on CT 5113 is owned by 999. So:
+#   UserNS=keep-id:uid=999,gid=999  → the host `podman` user appears as 999 inside the
+#   container, so the migrated data dir must be owned by `podman` on the host (the
+#   migration chowns it once). No :U — that would chown recursively on EVERY start.
+# Local disk (~130 MB), so unlike /mnt/youtube this is a safe place for either approach;
+# keep-id is chosen for consistency and to avoid repeated recursive chowns as the DB grows.
+
+[Unit]
+Description=Youtarr MariaDB
+Documentation=https://github.com/Chrison-Homelab/Homelab/issues/302
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+Image=docker.io/library/mariadb:10.3
+ContainerName=youtarr-db
+Network=youtarr.network
+
+# Root password as a podman secret rather than an env file.
+Secret=youtarr_db_root_password,type=env,target=MYSQL_ROOT_PASSWORD
+Environment=MYSQL_DATABASE=youtarr
+Environment=MYSQL_TCP_PORT=3321
+Environment=MYSQL_CHARSET=utf8mb4
+Environment=MYSQL_COLLATION=utf8mb4_unicode_ci
+
+Volume=/home/podman/youtarr/database:/var/lib/mysql
+UserNS=keep-id:uid=999,gid=999
+
+# compose `command:` → quadlet Exec=. No quoting needed here, so it survives systemd's
+# parser intact (contrast HealthCmd below).
+Exec=--port=3321 --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --innodb-file-per-table=1 --innodb-large-prefix=ON
+
+AutoUpdate=registry
+
+# NO HEALTHCHECK — deliberate, and for TWO reasons (see gotcha #10 in
+# docs/plans/284-podman-platform.md). The compose healthcheck is
+#   mysqladmin ping -h localhost -P 3321 -u root -p${DB_ROOT_PASSWORD}
+# which (a) needs quoting that quadlet destroys, and (b) would put the root password in the
+# unit file. Readiness is handled by youtarr's crash-retry instead.
+
+[Service]
+# Podman does not create missing bind-mount source dirs (gotcha #9).
+ExecStartPre=/usr/bin/mkdir -p /home/podman/youtarr/database
+Restart=always
+TimeoutStartSec=300
+
+[Install]
+WantedBy=default.target