feat(engine): app: podman rootless quadlet platform (ADR-0009 Phase 0)

[ChrisonSimtian]

Phase 0 of #283, implementing ADR-0009. Closes #284.

Builds the reusable platform so a stack can be provisioned as rootless Podman + quadlets, with systemd + git as the control plane. Verified live on a throwaway CT (9900 on desktop-01, destroyed — node back to its original 18 guests).

How it's composed

The create half needed no new code path: community-scripts ships ct/podman.sh, and CommunityScriptsCreator already runs pct create as root over SSH — the only way to set keyctl on an unprivileged LXC — and already renders var_unprivileged/nesting/keyctl/fuse from features:. So creation is just declining podman-install.sh's two Portainer prompts, alongside the existing docker entry.

PodmanProvisioner owns the post-create half, converting the stock rootful Podman CT into the rootless quadlet host: masks the root podman.socket, adds the non-root user with an in-map subuid range + linger, renders the stack's quadlets, drives systemctl --user, seeds podman secrets from secrets.env, and enables podman-auto-update.timer (replacing Watchtower and its docker.sock).

Acceptance (all met)

Criterion	Result
Throwaway app: podman LXC provisions end-to-end unattended	✅ one shot, 6m20s, 3 applied / 0 failed
Hello-world quadlet deploys, starts as a --user unit, survives a reboot	✅ active 3s after boot
Nested subuid/subgid verified inside the LXC's uid_map	✅ podman:10000:50000, guarded against /proc/self/uid_map
systemctl --user reliably driveable over pct exec	✅ runuser + explicit XDG_RUNTIME_DIR/DBUS_SESSION_BUS_ADDRESS

Idempotency confirmed: repeat converges report NOCHANGE, and the marker is deterministic across full rebuilds (e71314bdca83 both times).

Seven defects found by live-provisioning, not by reading docs

Written up in full in docs/plans/284-podman-platform.md.

1. Rootless networking needs /dev/net/tun, which an LXC lacks. pasta/slirp4netns open it to build the tap device, so every container start died. ADR-0009 anticipated no routable per-container IP, but not this. Fixed host-side by bind-mounting the /dev/net directory (binding the file fails with no parent to mount onto). The host node is mode 666, so an unprivileged CT can open it — no privileged container needed.
2. ct/podman.sh silently drops var_fuse — shape said fuse: true, CT came up nesting=1,keyctl=1, twice. Features are now reconciled from the shape, merging so an undeclared mount=nfs isn't stripped.
3. pct status reports running the instant a reboot is requested, before shutdown begins. A status-based wait returned immediately, the next pct exec attached mid-shutdown and blocked forever, which wedged the reboot so the CT never restarted. Two 10-minute hangs. Replaced with stop → wait for stopped → start → poll systemctl is-system-running, every probe timeout-wrapped.
4. runuser keeps cwd /root, which the rootless user can't read → cannot chdir to /root. Insidious because systemctl --user worked fine while podman secret create would have failed.
5. Quadlets started 92s late, every boot. Podman injects Wants=/After=podman-user-wait-network-online.service (Podman fail to autostart containers through quadlet/systemd, works when launched manually, error with pasta podman-container-tools/podman#22197), which spins on systemctl is-active network-online.target — never reached in a community-scripts Debian LXC (ifupdown is in use; only systemd-networkd wait-online units ship). It timed out, failed, and only then did the container start. This is the one I'd have shipped broken: it does survive a reboot, just by waiting out a failure, so a naive check passes. Fixed the cause with a oneshot ordered after networking.service. 92s → 2s.
6. The marker hashed only its inputs, not the deploy recipe — so the fix for (5) reported NOCHANGE against the already-stamped CT and never landed. The marker now folds in a hash of the generated script, so a provisioner fix invalidates it on exactly the hosts that need it.
7. SSH keepalives. A create holds one ssh session open for 30+ minutes on a cold template; without keepalives it was reset mid-create and converge reported CREATE FAILED for a create that had actually succeeded — the worst shape, since the next run must reconcile a CT the plan thought absent. Benefits every provisioner; the self-hosted runner on the node LAN is far less exposed than a laptop across VLANs, which is why it hadn't bitten before.

Two more are documented rather than coded, since they're authoring rules: quadlet Exec= goes through systemd specifier expansion so a literal % must be %%, and boot survival requires [Install] WantedBy=default.target (generated units can't be systemctl enabled).

Testing

123 unit tests pass (24 new), covering idempotency, the userns guard, the rootless posture, secret handling, and regression guards for defects 1–6.

Notes for review

• Shape gains a loader-set SourceDir so a provisioner can read sibling assets (the quadlet files) from the stack dir instead of requiring them inline.
• Podman secrets are add-only (exists || create) — rotation stays an explicit operator action, never a silent converge side effect. Values go via stdin, never argv.
• Nothing outside the engine + schema catalogue changes; no existing stack is migrated here. Pilot (Leapmotor CT 4100) is Pilot: migrate Leapmotor (CT 4100) to rootless Podman + quadlet (Phase 1) #285.

🤖 Generated with Claude Code