From 31a1fd3d726686c95f04ef73165f4e606b3d0011 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Tue, 8 Sep 2026 21:36:59 -0500 Subject: [PATCH] Enhance SKILL.md with detailed explanations for MiniMax H3 usage and add new onboarding proposal for Qwen-Image and Wan2.x models --- docs/proposals/qwen-image-wan-onboarding.md | 122 ++++++++++++ ...-09-08-mounted-mcp-file-transfer-design.md | 173 ++++++++++++++++++ plugins/dw/skills/minimax-h3/SKILL.md | 22 ++- scripts/release.sh | 20 +- 4 files changed, 328 insertions(+), 9 deletions(-) create mode 100644 docs/proposals/qwen-image-wan-onboarding.md create mode 100644 docs/superpowers/specs/2026-09-08-mounted-mcp-file-transfer-design.md diff --git a/docs/proposals/qwen-image-wan-onboarding.md b/docs/proposals/qwen-image-wan-onboarding.md new file mode 100644 index 0000000..18fe8de --- /dev/null +++ b/docs/proposals/qwen-image-wan-onboarding.md @@ -0,0 +1,122 @@ +# Proposal: onboarding Qwen-Image and Wan2.x into the catalog + +Status: draft, 2026-09-08. No implementation. + +## Why these two + +Neither has a template under `workflows/models/`. Both rank among the most +downloaded, most actively-developed open models in their category as of +2026-09-08 (Hugging Face trending listings, pipeline tags +`text-to-image` / `text-to-video`): + +- **Qwen/Qwen-Image** (20B) — 314k downloads, plus `Qwen-Image-2512` (71.8k) + and the widely-used `Qwen-Image-Edit` sibling. Sits above every FLUX + variant except `FLUX.1-dev`/`schnell` themselves. +- **Wan-AI/Wan2.x** — the dominant open video base by LoRA-ecosystem size; + the trending video listing is mostly MiniMax-H3 and Wan2.x + LoRAs/quantizations (`Wan2.2-T2V-A14B-GGUF`, `Wan2.2-TI2V-5B`, + `Wan2.1-T2V-1.3B`), which is itself a signal of how much community tooling + targets it. + +That popularity is the whole argument. Nothing about either model is +currently represented in the catalog, so this is greenfield onboarding, not +repair — the `model-family-onboarding` skill's audit step (sources table, +vendor prompt format, reading-order for templates) stands in for the +"contradicted claims" step, since there are no existing claims to check. + +## What's already established (from a first pass, not a full audit) + +Both fit a 24GB card, via the same quantization/offload primitives the +catalog already uses (`flux2-dev.json`'s BitsAndBytes text-encoder pattern, +`group_offload`, GGUF support in `config_objects.py`) — no engine work +implied: + +- **Qwen-Image**: bf16 doesn't fit; 4-bit BitsAndBytes (`nf4`, + `bnb_4bit_compute_dtype=torch.bfloat16`) brings it to ~17GB with `offload: + model`, mirroring `flux2-dev.json` rather than needing anything new. +- **Wan2.x 14B**: bf16 doesn't fit; the proven 24GB path is GGUF or FP8 + quantization plus block/group offload, or `sequential` offload at a real + speed cost (community reports of tens of minutes per 5s clip on aggressive + CPU offload). Wan2.2 also ships a **5B TI2V** variant that likely fits far + more comfortably in bf16 or light quantization — a cheaper starting point + if the 14B's quality isn't the point of a first template. + +Neither of these is load-bearing yet — they're first-pass web findings, not +audit citations. The audit step confirms them against the diffusers pipeline +source and the vendor's own hardware notes before anything is written as a +template default. + +## Does either need a composition skill? + +Undecided by design — that's what the audit is for — but the shape of the +question differs between them: + +- **Qwen-Image** (text-to-image) looks like FLUX/Z-Image/Krea: a flat + text-to-image shape with no chaining or frame arithmetic. On that evidence + alone it likely needs a template, not a skill. **Qwen-Image-Edit** + (instruction-based image editing, sometimes multi-image) is the piece that + would carry real shape decisions and a vendor prompt convention worth + deferring to — if it's in scope, it's the reason a skill might be + worthwhile, not the base text-to-image model. +- **Wan2.x** (video) is exactly the class LTX-2.5 and MiniMax H3 already got + skills for: t2v vs i2v, first/last-frame conditioning, extending a clip, + frame-count and resolution constraints tied to the vendor's own bucket + rules. A skill is plausible here on the same grounds as the existing two, + pending what the audit finds. + +## Proposed sequence + +Two independent onboardings, run through `model-family-onboarding` +(`.claude/skills/model-family-onboarding/SKILL.md`) one at a time rather than +together, since each produces its own audit file, template(s), and (maybe) a +plugin skill: + +1. **Audit** — one research agent per family against the vendor's model + card(s), GitHub repo, and the diffusers pipeline source + (`QwenImagePipeline` / `QwenImageEditPipeline`; `WanPipeline` / + `WanImageToVideoPipeline`). Confirms the VRAM numbers above, the + resolution/step/guidance defaults the vendor recommends, and — for + Qwen-Image-Edit and Wan2.x — the prompt conventions and any frame/aspect + constraints analogous to H3's `17n+5` or LTX's `8k+1`. Saved as + `docs/proposals/audits/-qwen-image-audit.md` and + `-wan2.x-audit.md`. +2. **Decide scope** — brainstorm with the user per family: which variant + (Qwen-Image alone, or with Edit; Wan2.2 5B TI2V, 14B, or both) and which + quantization path, so the template set doesn't sprawl into every + checkpoint variant the way the pre-cleanup catalog did. +3. **Template(s)** — under `workflows/models/`, configuring + `templates/text-to-image` (Qwen-Image) and whichever video template shape + Wan2.x's t2v/i2v split calls for, following the existing FLUX/H3/LTX + templates as the pattern. +4. **Skill, if the audit supports one** — `plugins/dw/skills//SKILL.md` + per the outline in + `docs/superpowers/specs/2026-09-07-dw-plugin-skills-design.md`, numeric + rules pinned to diffusers by `tests/test_plugin_skills.py`, same as H3 and + LTX-2.5. Skipped for Qwen-Image base text-to-image unless Edit is in + scope. +5. **Cold drill** — for whichever family gets a skill, the same + plugin-installed-vs-not comparison the H3/LTX-2.5 drill used. +6. **Ledger** — a row per family in the Part 4 ledger + (`docs/proposals/agent-catalog-legibility.md`), since this extends that + proposal's catalog rather than opening a new one. + +## What this is not + +- Not an engine change. Both fit `offload`, `group_offload`, BitsAndBytes, + and GGUF as they exist today. +- Not a commitment to Qwen-Image-Edit, a specific Wan variant, or a skill for + either — all three are audit outputs, not premises. +- Not urgent relative to each other — nothing here requires doing both at + once; treat as two backlog items that happen to share a rationale. + +## Open questions + +- Whether Qwen-Image-Edit is in scope for the first pass or a dated + follow-up — it's the part of the Qwen family that would actually justify a + skill. +- Which Wan2.x variant to template first: 5B TI2V (cheap, likely bf16-only) + versus 14B (better quality, needs GGUF/FP8 + offload) — possibly both, as + FLUX has `flux-dev` and `flux2-dev` at different cost tiers. +- Whether Wan2.x's LoRA ecosystem (camera-motion, acceleration LoRAs seen in + the trending listing) is worth a template variant the way H3's turbo LoRA + got one, or is out of scope for a first cut. diff --git a/docs/superpowers/specs/2026-09-08-mounted-mcp-file-transfer-design.md b/docs/superpowers/specs/2026-09-08-mounted-mcp-file-transfer-design.md new file mode 100644 index 0000000..197feca --- /dev/null +++ b/docs/superpowers/specs/2026-09-08-mounted-mcp-file-transfer-design.md @@ -0,0 +1,173 @@ +# Mounted MCP file transfer: design + +The MCP tool surface is built once (`dw_mcp.server.build_server`) and served two +ways: the stdio `dw-mcp` on the user's machine, and mounted at `/mcp` inside +`dw.serve --mcp` on the GPU box. Two tools move files between "here" and the +engine - `upload_asset` and `download_output` - and both assume "here" is the +user's machine. Over the mounted transport it is the GPU box, so `upload_asset` +cannot see the file the agent names and `download_output` writes into the +server's working directory. On 2026-09-08 a cold agent, told to upload a voice +reference, tried to ssh to the GPU box to get around it. + +## Goal + +An agent on either transport can put a small input file into the workspace's +asset library and get an `asset:` reference back, with the tool itself saying +what works on this transport when a form does not. A file the engine's decoders +can read is not refused at the upload gate. + +## Non-goals + +- Large uploads over MCP. Anything past the inline cap is told to use the HTTP + route; the browser UI and `curl` already cover it. +- Downloading media inline over MCP. `get_output_image` / `get_output_text` + and the gallery `url` remain the way content reaches the conversation. +- Changing the stdio behaviour of either tool, except for the shared additions + below. + +## Global constraints + +- One tool name per job. `upload_asset` gains a second form rather than a + sibling tool, so a skill can say "upload_asset" on both transports. +- The MCP server knows which transport it is on: `build_server(client, + mounted=False)`; `dw/server/mcp_mount.build_mcp_app` passes `mounted=True`. + Nothing else consults it. +- Refusals explain. A form that cannot work on this transport returns a + `DwApiError` whose message names the form that does; it never falls through + to a bare "No such file". +- Inline content is bounded: 16 MiB decoded (`MAX_INLINE_UPLOAD_BYTES`). A + voice reference, a portrait or a short clip fits; a video of any length does + not, and the message says so and gives the route. +- The extension gate matches the decoders. What `POST /api/uploads` accepts is + what `load_audio` / the H3 audio reference can read. + +## 1. `upload_asset`, two forms + +### Signature + +``` +upload_asset(file_path: str | None = None, + content_base64: str | None = None, + filename: str | None = None) -> dict +``` + +Exactly one of `file_path` or `content_base64`. `filename` is required with +`content_base64` (it carries the extension the gate checks) and ignored with +`file_path` (the basename is used, as now). + +### Rules + +- `file_path` form: unchanged when `mounted=False`. When `mounted=True` it is + refused before touching the filesystem: "This MCP endpoint runs on the + engine's machine, so `file_path` names a path there, not where you are. Pass + the file as `content_base64` with `filename` (up to 16 MiB), or POST the raw + bytes to `/api/uploads?filename=&workspace=` with the + same bearer token." `` and `` are filled from the client, so + the message is a command the agent can run. +- `content_base64` form, both transports: decode (refuse on padding/alphabet + error with a plain message), check the decoded size against + `MAX_INLINE_UPLOAD_BYTES`, check `filename`'s extension against + `ALLOWED_UPLOAD_EXTENSIONS`, then `client.post_bytes("/api/uploads", body, + params={"filename": filename})` exactly as the path form does. Same return + shape: `reference`, `url`, plus `bytes` (decoded size) so the agent can see + the file arrived whole. +- The docstring is set per transport in `build_server` before registration + (`fn.__doc__ = ...`), so a mounted session reads the caveat in the tool + listing rather than discovering it on failure. Both docstrings name the + extension list and the inline cap. +- Annotation stays `WRITES`. + +### Module + +`dw_mcp/assets.py`: `upload_asset(client, file_path=None, content_base64=None, +filename=None, mounted=False)`; the path form's body moves into a private +`_post_upload(client, body, filename)` shared by both. `MAX_INLINE_UPLOAD_BYTES` +beside `MAX_UPLOAD_BYTES`. `dw_mcp/server.py` passes `mounted` through and +selects the docstring. + +## 2. `download_output` when mounted + +### Rules + +- `mounted=False`: unchanged. +- `mounted=True`: the tool is still registered (a hidden tool cannot explain + itself) but refuses before contacting the server, returning a `DwApiError` + whose message gives the gallery `url` for `name` - the same one `list_gallery` + reports, with the workspace selector - and a one-line `curl -H + "Authorization: Bearer " -o `. The token is not + echoed; the placeholder is literal. +- The docstring's existing transport paragraph is kept and moved to the front + of the mounted variant. + +This closes the 2026-09-08 memory note: a cold agent no longer drops mp4s into +the server's working directory. + +## 3. The extension gate follows the decoders + +### Today + +`dw/security.py` `ALLOWED_AUDIO_EXTENSIONS = {.wav .mp3 .flac .ogg}`; the same +set is copied into `dw_mcp/assets.py`. `dw/tasks/audio_utils.load_audio` reads +audio extensions with `soundfile` and routes video extensions through PyAV. +diffusers' `MiniMaxH3AudioReference.from_file` decodes with PyAV and takes +`.m4a`, `.aac`, `.opus`. A phone recording (`.m4a`) is refused at the gate +though the reference it was meant for would have decoded it. + +### Rules + +- `ALLOWED_AUDIO_EXTENSIONS` gains `.m4a`, `.aac`, `.opus`. +- `load_audio` tries `soundfile` first and, on `soundfile.LibsndfileError` or + an extension libsndfile does not handle, decodes through PyAV the way the + video branch already does, returning the same `(samples, channels)` float32 + and sample rate. The audio tasks therefore accept everything the gate does. +- `dw_mcp` does not import `dw` (it is installable alone), so the copy stays, + and a test asserts the two sets are equal. + +## 4. Tests + +- `tests/test_mcp_assets.py`: the `content_base64` form posts decoded bytes to + `/api/uploads` with the given filename and returns `reference`/`url`/`bytes`; + over the cap is refused naming the route; bad base64 is refused plainly; + both forms given, or neither, is refused; `filename` without an allowed + extension is refused before any request; the `file_path` form is refused + when `mounted=True` and the message contains the base URL, the workspace and + `content_base64`. +- `tests/test_mcp_server.py`: `build_server(client, mounted=True)` registers + `upload_asset` whose description contains the transport caveat and + `build_server(client)` does not; `download_output` mounted refuses with the + gallery url and a `curl` line and makes no HTTP call. +- `tests/test_server_mcp.py`: a round trip through the mounted app - a small + wav as `content_base64` lands in the selected workspace's `assets/` and + `asset:` resolves in a validated workflow. +- `tests/test_assets.py` / audio task tests: `.m4a` passes the gate; + `load_audio` decodes an `.m4a` fixture (generated in the test with PyAV, no + binary checked in) to the same shape and rate as its wav twin; the two + extension sets are equal. + +## 5. Docs + +- docs/MCP.md: a "Where files live" section - the two transports, which + machine each tool's paths mean, the inline form and its cap, the `curl` + forms for both directions. Replaces the scattered sentences in the two tool + docstrings' current prose. +- docs/SERVER.md: the widened audio list where uploads are described. +- `plugins/dw/skills/minimax-h3/SKILL.md` "Run and judge" step 5 gains one + sentence: a recorded voice reference reaches the server with `upload_asset` + (`content_base64` over a `dw.serve --mcp` endpoint); the skill already + explains the export-zip fetch in the other direction. Version-pinned by the + existing size test only; no numbers are stated. +- `.remember` / memory: the 2026-09-08 workaround note becomes "fixed in + ". + +## Sequence for the plan + +1. `mounted` flag through `build_server` and `mcp_mount`; per-transport + docstrings; `file_path` refusal when mounted (section 1, first bullet). + Ships alone if needed - it would have prevented the ssh attempt. +2. Extension gate and `load_audio` fallback (section 3). +3. `content_base64` form (section 1, rest). +4. `download_output` mounted refusal (section 2). +5. Tests alongside each step; docs last. + +Do not restart the GPU box's `dw.serve` to pick this up while a drill job is +running - a restart kills the worker mid-job. diff --git a/plugins/dw/skills/minimax-h3/SKILL.md b/plugins/dw/skills/minimax-h3/SKILL.md index d2e23ab..29cc32e 100644 --- a/plugins/dw/skills/minimax-h3/SKILL.md +++ b/plugins/dw/skills/minimax-h3/SKILL.md @@ -38,6 +38,9 @@ not from here. - **Several boards in one generation, one unbroken score**: `templates/minimax/storyboard` - H3 cuts between the boards inside a single generation, which no concat of separate clips can match for continuous audio. + It is one beat with fixed cut points, not a building block: four of them + concatenated give twelve equal-length shots and a cast redrawn four times. + Past one beat with a recurring cast, use the cuts pattern below. - **Longer than 14.4 seconds**: decide first whether the seam is a cut or a continuation. Chain when the same action or line of speech has to cross the seam; cut when the scene changes, and treat each cut as its own generation. @@ -62,7 +65,11 @@ not from here. author the concat and score steps the same way). A character who speaks in several shots keeps one voice by passing the same clip as an audio reference in each (the `voice-timbre-reference` pattern); a repeated voice - description alone drifts. + description alone drifts. Each shot's `num_frames` is its own, so pace the + cut - a trailer builds by varying shot length. The reference carries + delivery as well as timbre: a flat read gives a flat performance. Bark's + presets are conversational; for a narrator with gravitas, `upload_asset` a + recorded read in that register and reference the same file in every shot. - **Music alone**: `templates/minimax/music` (Music3); the `minimax-music3` skill. If none fits, compose from `list_tasks` before authoring a new workflow, and @@ -94,6 +101,10 @@ read the `workflows` guide's authoring section first. cuts is laid under the concat afterwards. - H3 is guidance-distilled: no `guidance_scale`, no negative prompt. Say what is there, never what is not. +- When deriving a variant, keep `release_pipeline` on the step the template + puts it on: it frees the Z-Image boards before H3 loads. A run killed by + SIGKILL near the end, in a worker warm from a previous job, that succeeds + on a retry in a fresh worker is host memory, not the prompt. - Ref2VA limits: at most 9 images, 3 videos, 3 audio clips, 12 files; audio can never be the only reference. References are labelled in the order passed. - Music3 reads `audio_duration` as a ceiling, not a target: ask for more than @@ -119,7 +130,9 @@ paraphrase it from examples: built-in enhancer writes the format from those guides. Its `idea` is framed as `Task: T2VA. Duration: 5.17 seconds. Idea: ...`. -Whichever route: repeat a speaker's voice description verbatim across shots, +Whichever route: write the whole script before the first shot - the lines in +order, read once, should carry the piece on their own - then place them. +Repeat a speaker's voice description verbatim across shots, and when a reference picture should fix identity but not framing, say so in the prompt itself - in a reference-conditioned request, in the lines that define the subject and state what each reference keeps - or every shot @@ -149,7 +162,10 @@ inherits the portrait's composition. `templates/minimax/music-video`. Ask the user to look for the family's failure modes: a character that changes between shots (reference the same portraits in every shot), a reference portrait imposing its framing on every - shot, a storyboard skipped, drift sharpening into noise late in a chain. + shot, a storyboard skipped, drift sharpening into noise late in a chain, + a voice-over without affect (the reference's delivery came through), every + shot the same length, a look word repeated on every board (shallow depth + of field) softening every shot. 5. After an inline run worth keeping, `get_job_workflow` and `save_workflow` it, so the next run is by name rather than by pasting JSON; `export_job` bundles the run — workflow, manifest, job row and media — for git. The bundle is on diff --git a/scripts/release.sh b/scripts/release.sh index ebf8e40..ede9ca8 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -69,11 +69,19 @@ with open(path, "w", encoding="utf-8") as f: f.write(text) EOF echo "pyproject.toml: $current -> $version" +fi - python3 - "$version" <<'EOF' +# The plugin's version is the engine's (tests/test_plugin_skills.py holds +# them equal), and it is the key the plugin cache is stored under. Written +# unconditionally so a plugin.json that drifted while pyproject.toml already +# sat at the target version is still brought into line +plugin_path="plugins/dw/.claude-plugin/plugin.json" +plugin_current=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["version"])' "$plugin_path") +if [ "$plugin_current" != "$version" ]; then + python3 - "$version" "$plugin_path" <<'EOF' import json, sys -path = "plugins/dw/.claude-plugin/plugin.json" +path = sys.argv[2] with open(path, encoding="utf-8") as f: plugin = json.load(f) plugin["version"] = sys.argv[1] @@ -81,13 +89,13 @@ with open(path, "w", encoding="utf-8") as f: json.dump(plugin, f, indent=4, ensure_ascii=False) f.write("\n") EOF - echo "plugins/dw/.claude-plugin/plugin.json: -> $version" + echo "$plugin_path: $plugin_current -> $version" fi -if ! git diff --quiet -- pyproject.toml plugins/dw/.claude-plugin/plugin.json; then - git commit -m "release $version" -- pyproject.toml plugins/dw/.claude-plugin/plugin.json +if ! git diff --quiet -- pyproject.toml "$plugin_path"; then + git commit -m "release $version" -- pyproject.toml "$plugin_path" else - echo "pyproject.toml already at $version and committed - tagging HEAD" + echo "pyproject.toml and $plugin_path already at $version and committed - tagging HEAD" fi git push origin master