comfyui: pin the application version and bump to v0.33.1 for MiniMax Music3

docs/configuration.md

@@ -201,10 +201,39 @@ Named volumes (state; back up via a helper container, see [data.md](data.md)):
 | `open-webui-data` | Open WebUI accounts + uploads |
 | `models-gguf` | llama.cpp GGUF weights (what the backends actually read) |
 | `comfyui-models` | ComfyUI checkpoints, LoRAs, VAEs, encoders |
-| `comfyui-app` | ComfyUI application tree + custom nodes |
+| `comfyui-app` | ComfyUI application tree + custom nodes (version-pinned — see below) |
 
 `/tmp` inside containers is tmpfs; nothing there survives a restart.
 
+### ComfyUI application version
+
+The pinned `COMFYUI_IMAGE` digest pins the *runtime* (torch, CUDA, the boot entrypoint). The
+ComfyUI application itself lives on the `comfyui-app` volume, so the image says nothing about
+which ComfyUI is running. That version is declared separately, as **`COMFYUI_APP_REF` in
+`services/comfyui/plugin.yaml`** — a full upstream commit SHA, never a tag or branch.
+
+`scripts/comfyui/boot.sh` runs as the container command and reconciles the checkout to that ref
+on every start. If the volume has drifted it fetches and checks out the declared ref; if it
+cannot, it refuses to start rather than serve a version nobody declared. The Comfy-Org runtime
+pins (`comfyui-frontend-package`, `comfy-kitchen`, `comfy-aimdo`, …) are read out of that ref's
+own `requirements.txt`, so the app and its runtime deps cannot drift apart.
+
+To move ComfyUI versions:
+
+```bash
+# 1. edit COMFYUI_APP_REF in services/comfyui/plugin.yaml to the new commit SHA
+ordo --source out/ordo.yaml render --out out
+cd out && COMPOSE_PROFILES='*' docker compose -p ordo --env-file .env --env-file secrets.env up -d
+# 2. confirm declared == running
+docker compose -p ordo exec comfyui git -C /root/ComfyUI rev-parse HEAD
+```
+
+Rolling back is the same edit in reverse — the reconciler downgrades the checkout and re-pins the
+runtime deps to the older ref's `requirements.txt`.
+
+Current pin: **v0.33.1** (`72865f4f`), chosen as the first release carrying the MiniMax Music3
+nodes that `services/song-gen` declares.
+
 ## Network Ports
 
 Caddy is still the **only** service with published host ports — every other service (dashboard, open-webui, model-gateway, comfyui, n8n, hermes-dashboard, mcp-gateway, qdrant, ops-api, ops-controller, etc.) has no `ports:` entry and is reachable only from other containers on the internal `ordo-net` network. But since 2026-07-24 Caddy itself is **port-per-service**: it publishes seven SSO-gated host ports on `${CADDY_TAILNET_HOSTNAME}` (all bound to `CADDY_BIND`) instead of routing every UI under subpaths of a single `:443`. Each prebuilt SPA is served at the root it was compiled for, ending the subpath-rewrite workarounds (Open WebUI root-catchall 404s, Hermes header-based base injection, n8n `strip_prefix`, codebase-memory nginx rewrites).

docs/data.md

@@ -28,7 +28,7 @@ Reference for where data lives, how it moves, and what survives a restart / rebu
 | `data/dashboard/` | Throughput samples, benchmarks, job tracking | JSON |
 | `hermes-home` named volume | Hermes agent brain (sessions, config, skills, cron) | JSON / SQLite / YAML |
 | `data/comfyui-output/` | Generated media (renders) | mixed |
-| `comfyui-app` named volume | ComfyUI app tree + custom nodes | mixed |
+| `comfyui-app` named volume | ComfyUI app tree + custom nodes — app version pinned by `COMFYUI_APP_REF`, reconciled on boot ([configuration.md](configuration.md#comfyui-application-version)) | mixed |
 | `n8n-data` named volume | n8n workflows and credentials | n8n native |
 | `couchdb-data` named volume | CouchDB (Obsidian LiveSync) | CouchDB native |
 | `open-webui-data` named volume | Open WebUI accounts + uploads | SQLite / files |

docs/operator-guide.md

@@ -66,6 +66,16 @@ ordo preflight --ref out/.env                 # read-only GO/NO-GO readiness gat
 cd out && COMPOSE_PROFILES=edge,webui,… docker compose -p ordo --env-file .env --env-file secrets.env up -d
 ```
 
+To reconcile the **whole** deployment against a fresh render (every profile you have enabled,
+without having to remember the list), use Compose's wildcard:
+
+```bash
+cd out && COMPOSE_PROFILES='*' docker compose -p ordo --env-file .env --env-file secrets.env up -d
+```
+
+`COMPOSE_PROFILES=all` does **not** do this — there is no profile named `all`, so it silently
+selects only the profile-less services and leaves every optional service unreconciled.
+
 Everything below is the reference for *how* that render engine works and *why* it's built this way.
 
 ## Why this exists (from the architecture interrogation)

scripts/comfyui/boot.sh

@@ -0,0 +1,172 @@
+#!/usr/bin/env bash
+# ComfyUI boot reconciler — bind-mounted read-only at /comfyui-scripts and run as the
+# container command (same pattern as scripts/llamacpp/run-llama-server.sh).
+#
+# WHY THIS EXISTS
+# ComfyUI's application tree lives on the `comfyui-app` named volume, NOT in the pinned
+# runtime image (services/comfyui/plugin.yaml explains why: v0.30 startup I/O wedged the
+# old 9p bind). That means the image digest pins the *runtime* — torch, CUDA, the boot
+# entrypoint — but says nothing about which ComfyUI the stack is actually running. Until
+# this script existed, the answer was "whatever someone last checked out by hand on the
+# volume": an unpinned, undeclared, unreproducible version, i.e. exactly the config drift
+# the render substrate exists to make impossible.
+#
+# So the version is declared (COMFYUI_APP_REF, pinned in services/comfyui/plugin.yaml) and
+# reconciled here on every start. Declared == running is enforced, not hoped for.
+#
+# The pip pins are NOT restated here: they are read out of the checked-out ref's own
+# requirements.txt. One source of truth — bumping COMFYUI_APP_REF automatically carries the
+# runtime deps that version was released against, so the two can never drift apart.
+set -euo pipefail
+
+COMFY_DIR=/root/ComfyUI
+SITE_LIB64=/usr/local/lib64/python3.12/site-packages
+
+log() { echo "[boot] $*"; }
+die() { echo "[boot] FATAL: $*" >&2; exit 1; }
+
+# --- 1. reconcile the application tree to the declared ref ----------------------------
+# Only touches the network when the checkout has actually drifted, so the steady-state
+# boot is offline-safe. When it HAS drifted and cannot be fixed, we fail hard rather than
+# start a version nobody declared — a crash-looping container is a visible problem; a
+# silently-wrong ComfyUI is not.
+reconcile_app() {
+  local ref="${COMFYUI_APP_REF:-}"
+  [ -n "$ref" ] || { log "COMFYUI_APP_REF unset — skipping app reconcile"; return 0; }
+  [ -d "$COMFY_DIR/.git" ] || die "$COMFY_DIR is not a git checkout; cannot reconcile to $ref"
+
+  local head
+  head="$(git -C "$COMFY_DIR" rev-parse HEAD)"
+  if [ "$head" = "$ref" ]; then
+    log "ComfyUI already at declared ref ${ref:0:12} ($(cat "$COMFY_DIR/comfyui_version.py" 2>/dev/null | sed -n 's/^__version__ = "\(.*\)"/\1/p'))"
+    return 0
+  fi
+
+  log "ComfyUI drift: running ${head:0:12}, declared ${ref:0:12} — reconciling"
+  git -C "$COMFY_DIR" fetch --tags --quiet origin \
+    || die "git fetch failed; refusing to run undeclared ComfyUI ${head:0:12}"
+  # Detached checkout: custom_nodes/, user/, scripts/ and the models/output mounts are
+  # untracked or unchanged between refs, so they are carried across untouched.
+  git -C "$COMFY_DIR" checkout --quiet --detach "$ref" \
+    || die "git checkout $ref failed; refusing to run undeclared ComfyUI ${head:0:12}"
+  # Stale bytecode from the previous ref shadows renamed modules (e.g. the new
+  # comfy/memory_management.py split out in v0.33).
+  find "$COMFY_DIR" -name '__pycache__' -type d -prune -exec rm -rf {} + 2>/dev/null || true
+  log "ComfyUI now at $(git -C "$COMFY_DIR" describe --tags --always) (${ref:0:12})"
+}
+
+# --- 2. runtime deps the pinned boot image predates ------------------------------------
+# The Comfy-Org runtime packages are versioned in lockstep with the app: the frontend
+# speaks the app's API, and comfy-kitchen/comfy-aimdo are imported directly by
+# comfy/model_management.py. Install exactly what the checked-out ref declares.
+#
+# Installs go to the USER site (/root/.local, on the comfyui-app volume), which precedes
+# every system site-packages dir on sys.path. That is deliberate: the image ships its own
+# copies in /usr/local/lib AND /usr/local/lib64, and a system-site install can land in the
+# loser of that pair — the shadowing bug that silently broke every fp8/nvfp4 model load on
+# 2026-08-07 ("'NoneType' object has no attribute 'Params'"). Owning the highest-priority
+# path ends that class of failure instead of playing whack-a-mole with rm -rf.
+COMFY_ORG_PKGS=(
+  comfyui-frontend-package
+  comfyui-workflow-templates
+  comfyui-embedded-docs
+  comfy-kitchen
+  comfy-aimdo
+)
+# Not in requirements.txt (optional GLSL shader renderer for the comfyui_underwater /
+# shader nodes) — pinned here to the version this stack is validated against.
+EXTRA_PINS=(comfy-angle==0.1.0)
+
+installed_version() {
+  python3 - "$1" <<'PY' 2>/dev/null || true
+import importlib.metadata as m, sys
+try:
+    print(m.version(sys.argv[1]))
+except Exception:
+    pass
+PY
+}
+
+install_runtime_deps() {
+  local req="$COMFY_DIR/requirements.txt"
+  [ -f "$req" ] || die "missing $req"
+
+  # The image bundles comfy-kitchen in lib64. If the user-site install below ever fails we
+  # must NOT silently fall back to it: the bundled copy is older than the app expects and
+  # fails open (import succeeds, quant ops are None). Remove it so a broken install is loud.
+  rm -rf "$SITE_LIB64"/comfy_kitchen* 2>/dev/null || true
+
+  local specs=() pkg spec want have
+  for pkg in "${COMFY_ORG_PKGS[@]}"; do
+    spec="$(grep -m1 -E "^${pkg}==" "$req" || true)"
+    if [ -z "$spec" ]; then
+      log "WARN ${pkg} not pinned in requirements.txt at this ref — skipping"
+      continue
+    fi
+    want="${spec##*==}"
+    have="$(installed_version "$pkg")"
+    if [ "$have" = "$want" ]; then
+      log "dep ok: ${pkg}==${want}"
+    else
+      log "dep needs install: ${pkg} ${have:-<missing>} -> ${want}"
+      specs+=("$spec")
+    fi
+  done
+  for spec in "${EXTRA_PINS[@]}"; do
+    pkg="${spec%%==*}"; want="${spec##*==}"
+    have="$(installed_version "$pkg")"
+    if [ "$have" = "$want" ]; then log "dep ok: ${spec}"; else specs+=("$spec"); fi
+  done
+
+  if [ ${#specs[@]} -eq 0 ]; then
+    log "runtime deps already reconciled"
+    return 0
+  fi
+  log "installing: ${specs[*]}"
+  PIP_USER=true PIP_ROOT_USER_ACTION=ignore \
+    pip install --no-cache-dir --no-warn-script-location -q "${specs[@]}" \
+    || die "pinned runtime dep install failed: ${specs[*]}"
+
+  # Verify rather than trust: pip can resolve to a different version than asked for when a
+  # transitive constraint intervenes, and this is precisely the check that would have
+  # caught the shadowed-kitchen outage.
+  for spec in "${specs[@]}"; do
+    pkg="${spec%%==*}"; want="${spec##*==}"; have="$(installed_version "$pkg")"
+    [ "$have" = "$want" ] || die "after install, ${pkg} resolves to ${have:-<missing>}, expected ${want}"
+  done
+  log "runtime deps reconciled"
+}
+
+# --- 3. repo-owned workflow templates ---------------------------------------------------
+# Seeded into a repo-owned namespace under the user workflows dir so operator-authored
+# graphs in the sibling dirs are never overwritten. Anything in `ordo/` is generated from
+# the repo and replaced on every boot — edit it in git, not in the UI.
+seed_workflows() {
+  local src=/comfyui-scripts/workflows dst="$COMFY_DIR/user/default/workflows/ordo"
+  [ -d "$src" ] || return 0
+  mkdir -p "$dst"
+  cp -f "$src"/*.json "$dst"/ 2>/dev/null || true
+  log "seeded repo workflows -> user/default/workflows/ordo ($(ls -1 "$dst" | wc -l) file(s))"
+}
+
+# --- 4. custom node deps ----------------------------------------------------------------
+# Deliberately NOT user-site: these land in the image's writable layer and are therefore
+# re-resolved from scratch on every recreate. A custom node that is later removed takes its
+# dependency pins with it, instead of leaving a stale copy shadowing the image forever on
+# the volume. Best-effort by design — one broken node's requirements must not block boot.
+install_custom_node_deps() {
+  local r
+  for r in "$COMFY_DIR"/custom_nodes/*/requirements.txt; do
+    [ -f "$r" ] || continue
+    log "installing $r"
+    pip install --no-cache-dir --no-warn-script-location -q -r "$r" || log "WARN failed $r"
+  done
+}
+
+reconcile_app
+install_runtime_deps
+seed_workflows
+install_custom_node_deps
+
+log "handing off to the image entrypoint"
+exec bash /runner-scripts/entrypoint.sh

scripts/comfyui/models.json

@@ -313,6 +313,37 @@
         }
       ]
     },
+    "minimax-music3": {
+      "capability": "audio",
+      "description": "MiniMax Music3 — text+lyrics to full song with vocals (Apache-2.0). DiT fp16 (4.6 GB) + pruned bf16 text encoder (15.6 GB) + DAV VAE (0.2 GB). Needs ComfyUI >= v0.33.1 for the Music3 nodes.",
+      "_comment": "URLs are pinned to the Comfy-Org repack revision 6444666eb6edfb2c7fcab5f8b81da8b84b4b17b6 — /resolve/main/ is a moving ref and would silently hand a different tensor set to a later re-pull.",
+      "models": [
+        {
+          "repo": "Comfy-Org/MiniMax-Music-3",
+          "file": "diffusion_models/minimax_music3_dit_fp16.safetensors",
+          "dest": "diffusion_models",
+          "type": "diffusion_models",
+          "name": "minimax_music3_dit_fp16.safetensors",
+          "url": "https://huggingface.co/Comfy-Org/MiniMax-Music-3/resolve/6444666eb6edfb2c7fcab5f8b81da8b84b4b17b6/diffusion_models/minimax_music3_dit_fp16.safetensors"
+        },
+        {
+          "repo": "Comfy-Org/MiniMax-Music-3",
+          "file": "text_encoders/minimax_music3_text_encoder_pruned_bf16.safetensors",
+          "dest": "text_encoders",
+          "type": "text_encoders",
+          "name": "minimax_music3_text_encoder_pruned_bf16.safetensors",
+          "url": "https://huggingface.co/Comfy-Org/MiniMax-Music-3/resolve/6444666eb6edfb2c7fcab5f8b81da8b84b4b17b6/text_encoders/minimax_music3_text_encoder_pruned_bf16.safetensors"
+        },
+        {
+          "repo": "Comfy-Org/MiniMax-Music-3",
+          "file": "vae/minimax_music3_dav.safetensors",
+          "dest": "vae",
+          "type": "vae",
+          "name": "minimax_music3_dav.safetensors",
+          "url": "https://huggingface.co/Comfy-Org/MiniMax-Music-3/resolve/6444666eb6edfb2c7fcab5f8b81da8b84b4b17b6/vae/minimax_music3_dav.safetensors"
+        }
+      ]
+    },
     "supir-upscaler": {
       "capability": "upscale",
       "description": "SUPIR upscaler (Kijai pruned) — best quality upscaling, replaces 4x-UltraSharp. 2.7 GB each.",

scripts/comfyui/workflows/minimax-music3.json

@@ -0,0 +1,96 @@
+{
+  "_comment": [
+    "MiniMax Music3 text+lyrics -> song. ComfyUI API format (the shape POST /prompt accepts),",
+    "so this file is both the UI starting point and the thing automation submits.",
+    "Seeded to user/default/workflows/ordo/ by scripts/comfyui/boot.sh — repo-owned, replaced",
+    "on every boot. Edit it here, not in the UI.",
+    "",
+    "Requires ComfyUI >= v0.33.1 (services/comfyui pins COMFYUI_APP_REF) and the",
+    "'minimax-music3' model pack from scripts/comfyui/models.json.",
+    "",
+    "Submit THROUGH the gate (http://comfyui-gate:8188/prompt), never straight to comfyui:8188 —",
+    "the gate acquires GPU residency before the render starts. See services/gpu-gate/README.md."
+  ],
+  "1": {
+    "class_type": "UNETLoader",
+    "_meta": { "title": "Music3 DiT" },
+    "inputs": {
+      "unet_name": "minimax_music3_dit_fp16.safetensors",
+      "weight_dtype": "default"
+    }
+  },
+  "2": {
+    "class_type": "CLIPLoader",
+    "_meta": { "title": "Music3 text encoder" },
+    "inputs": {
+      "clip_name": "minimax_music3_text_encoder_pruned_bf16.safetensors",
+      "type": "minimax",
+      "device": "default"
+    }
+  },
+  "3": {
+    "class_type": "VAELoader",
+    "_meta": { "title": "Music3 DAV VAE" },
+    "inputs": { "vae_name": "minimax_music3_dav.safetensors" }
+  },
+  "4": {
+    "class_type": "MiniMaxMusic3TextEncode",
+    "_meta": { "title": "Caption + lyrics" },
+    "inputs": {
+      "clip": ["2", 0],
+      "caption": "Global Metadata: Warm indie-folk, 96 BPM, G major, bright and unhurried. Sunday-morning mood, clean modern production with a little room ambience.\n\nVocal Details: Single relaxed female lead, close-mic'd and conversational, light double-tracking on the chorus, soft harmonies underneath.\n\nArrangement: Fingerpicked acoustic guitar as the spine, brushed kit entering on the first verse, round upright bass, occasional piano answers. Intro: guitar alone. Verses: guitar, bass, brushes. Chorus: full band, harmonies open up. Outro: back to solo guitar, one last held chord.",
+      "lyrics": "[Intro]\n\n[Verse]\nMorning light across the kitchen floor\nKettle humming like it has before\nNothing urgent waiting at the door\nJust the day and nothing more\n\n[Chorus]\nSo let it be slow\nLet the hours go\nNothing left to prove\nNothing left to move\n\n[Verse]\nCoffee cooling while the record turns\nSame four chords, and still I never learn\nEvery season takes its turn\nAnd the quiet ones return\n\n[Chorus]\nSo let it be slow\nLet the hours go\nNothing left to prove\nNothing left to move\n\n[Outro]\n",
+      "seed": 222,
+      "max_duration": 120.0,
+      "cfg_scale": 1.7,
+      "top_k": 50
+    }
+  },
+  "5": {
+    "class_type": "ConditioningZeroOut",
+    "_meta": { "title": "Negative (zeroed)" },
+    "inputs": { "conditioning": ["4", 0] }
+  },
+  "6": {
+    "class_type": "EmptyMiniMaxMusic3LatentAudio",
+    "_meta": { "title": "Empty audio latent" },
+    "_note": "seconds is driven by the encoder's second output — the model can end the song early, and the latent must match the length it actually chose.",
+    "inputs": {
+      "seconds": ["4", 1],
+      "batch_size": 1
+    }
+  },
+  "7": {
+    "class_type": "KSampler",
+    "_meta": { "title": "Sample" },
+    "inputs": {
+      "model": ["1", 0],
+      "positive": ["4", 0],
+      "negative": ["5", 0],
+      "latent_image": ["6", 0],
+      "seed": 222,
+      "steps": 30,
+      "cfg": 1.7,
+      "sampler_name": "euler",
+      "scheduler": "simple",
+      "denoise": 1.0
+    }
+  },
+  "8": {
+    "class_type": "VAEDecodeAudio",
+    "_meta": { "title": "Decode audio" },
+    "inputs": {
+      "samples": ["7", 0],
+      "vae": ["3", 0]
+    }
+  },
+  "9": {
+    "class_type": "SaveAudioMP3",
+    "_meta": { "title": "Save" },
+    "inputs": {
+      "audio": ["8", 0],
+      "filename_prefix": "audio/minimax-music3",
+      "quality": "V0"
+    }
+  }
+}