Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 122 additions & 0 deletions docs/proposals/qwen-image-wan-onboarding.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# Proposal: onboarding Qwen-Image and Wan2.x into the catalog

Status: draft, 2026-09-08. No implementation.

## Why these two

Neither has a template under `workflows/models/`. Both rank among the most
downloaded, most actively-developed open models in their category as of
2026-09-08 (Hugging Face trending listings, pipeline tags
`text-to-image` / `text-to-video`):

- **Qwen/Qwen-Image** (20B) — 314k downloads, plus `Qwen-Image-2512` (71.8k)
and the widely-used `Qwen-Image-Edit` sibling. Sits above every FLUX
variant except `FLUX.1-dev`/`schnell` themselves.
- **Wan-AI/Wan2.x** — the dominant open video base by LoRA-ecosystem size;
the trending video listing is mostly MiniMax-H3 and Wan2.x
LoRAs/quantizations (`Wan2.2-T2V-A14B-GGUF`, `Wan2.2-TI2V-5B`,
`Wan2.1-T2V-1.3B`), which is itself a signal of how much community tooling
targets it.

That popularity is the whole argument. Nothing about either model is
currently represented in the catalog, so this is greenfield onboarding, not
repair — the `model-family-onboarding` skill's audit step (sources table,
vendor prompt format, reading-order for templates) stands in for the
"contradicted claims" step, since there are no existing claims to check.

## What's already established (from a first pass, not a full audit)

Both fit a 24GB card, via the same quantization/offload primitives the
catalog already uses (`flux2-dev.json`'s BitsAndBytes text-encoder pattern,
`group_offload`, GGUF support in `config_objects.py`) — no engine work
implied:

- **Qwen-Image**: bf16 doesn't fit; 4-bit BitsAndBytes (`nf4`,
`bnb_4bit_compute_dtype=torch.bfloat16`) brings it to ~17GB with `offload:
model`, mirroring `flux2-dev.json` rather than needing anything new.
- **Wan2.x 14B**: bf16 doesn't fit; the proven 24GB path is GGUF or FP8
quantization plus block/group offload, or `sequential` offload at a real
speed cost (community reports of tens of minutes per 5s clip on aggressive
CPU offload). Wan2.2 also ships a **5B TI2V** variant that likely fits far
more comfortably in bf16 or light quantization — a cheaper starting point
if the 14B's quality isn't the point of a first template.

Neither of these is load-bearing yet — they're first-pass web findings, not
audit citations. The audit step confirms them against the diffusers pipeline
source and the vendor's own hardware notes before anything is written as a
template default.

## Does either need a composition skill?

Undecided by design — that's what the audit is for — but the shape of the
question differs between them:

- **Qwen-Image** (text-to-image) looks like FLUX/Z-Image/Krea: a flat
text-to-image shape with no chaining or frame arithmetic. On that evidence
alone it likely needs a template, not a skill. **Qwen-Image-Edit**
(instruction-based image editing, sometimes multi-image) is the piece that
would carry real shape decisions and a vendor prompt convention worth
deferring to — if it's in scope, it's the reason a skill might be
worthwhile, not the base text-to-image model.
- **Wan2.x** (video) is exactly the class LTX-2.5 and MiniMax H3 already got
skills for: t2v vs i2v, first/last-frame conditioning, extending a clip,
frame-count and resolution constraints tied to the vendor's own bucket
rules. A skill is plausible here on the same grounds as the existing two,
pending what the audit finds.

## Proposed sequence

Two independent onboardings, run through `model-family-onboarding`
(`.claude/skills/model-family-onboarding/SKILL.md`) one at a time rather than
together, since each produces its own audit file, template(s), and (maybe) a
plugin skill:

1. **Audit** — one research agent per family against the vendor's model
card(s), GitHub repo, and the diffusers pipeline source
(`QwenImagePipeline` / `QwenImageEditPipeline`; `WanPipeline` /
`WanImageToVideoPipeline`). Confirms the VRAM numbers above, the
resolution/step/guidance defaults the vendor recommends, and — for
Qwen-Image-Edit and Wan2.x — the prompt conventions and any frame/aspect
constraints analogous to H3's `17n+5` or LTX's `8k+1`. Saved as
`docs/proposals/audits/<date>-qwen-image-audit.md` and
`<date>-wan2.x-audit.md`.
2. **Decide scope** — brainstorm with the user per family: which variant
(Qwen-Image alone, or with Edit; Wan2.2 5B TI2V, 14B, or both) and which
quantization path, so the template set doesn't sprawl into every
checkpoint variant the way the pre-cleanup catalog did.
3. **Template(s)** — under `workflows/models/`, configuring
`templates/text-to-image` (Qwen-Image) and whichever video template shape
Wan2.x's t2v/i2v split calls for, following the existing FLUX/H3/LTX
templates as the pattern.
4. **Skill, if the audit supports one** — `plugins/dw/skills/<family>/SKILL.md`
per the outline in
`docs/superpowers/specs/2026-09-07-dw-plugin-skills-design.md`, numeric
rules pinned to diffusers by `tests/test_plugin_skills.py`, same as H3 and
LTX-2.5. Skipped for Qwen-Image base text-to-image unless Edit is in
scope.
5. **Cold drill** — for whichever family gets a skill, the same
plugin-installed-vs-not comparison the H3/LTX-2.5 drill used.
6. **Ledger** — a row per family in the Part 4 ledger
(`docs/proposals/agent-catalog-legibility.md`), since this extends that
proposal's catalog rather than opening a new one.

## What this is not

- Not an engine change. Both fit `offload`, `group_offload`, BitsAndBytes,
and GGUF as they exist today.
- Not a commitment to Qwen-Image-Edit, a specific Wan variant, or a skill for
either — all three are audit outputs, not premises.
- Not urgent relative to each other — nothing here requires doing both at
once; treat as two backlog items that happen to share a rationale.

## Open questions

- Whether Qwen-Image-Edit is in scope for the first pass or a dated
follow-up — it's the part of the Qwen family that would actually justify a
skill.
- Which Wan2.x variant to template first: 5B TI2V (cheap, likely bf16-only)
versus 14B (better quality, needs GGUF/FP8 + offload) — possibly both, as
FLUX has `flux-dev` and `flux2-dev` at different cost tiers.
- Whether Wan2.x's LoRA ecosystem (camera-motion, acceleration LoRAs seen in
the trending listing) is worth a template variant the way H3's turbo LoRA
got one, or is out of scope for a first cut.
173 changes: 173 additions & 0 deletions docs/superpowers/specs/2026-09-08-mounted-mcp-file-transfer-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
# Mounted MCP file transfer: design

The MCP tool surface is built once (`dw_mcp.server.build_server`) and served two
ways: the stdio `dw-mcp` on the user's machine, and mounted at `/mcp` inside
`dw.serve --mcp` on the GPU box. Two tools move files between "here" and the
engine - `upload_asset` and `download_output` - and both assume "here" is the
user's machine. Over the mounted transport it is the GPU box, so `upload_asset`
cannot see the file the agent names and `download_output` writes into the
server's working directory. On 2026-09-08 a cold agent, told to upload a voice
reference, tried to ssh to the GPU box to get around it.

## Goal

An agent on either transport can put a small input file into the workspace's
asset library and get an `asset:` reference back, with the tool itself saying
what works on this transport when a form does not. A file the engine's decoders
can read is not refused at the upload gate.

## Non-goals

- Large uploads over MCP. Anything past the inline cap is told to use the HTTP
route; the browser UI and `curl` already cover it.
- Downloading media inline over MCP. `get_output_image` / `get_output_text`
and the gallery `url` remain the way content reaches the conversation.
- Changing the stdio behaviour of either tool, except for the shared additions
below.

## Global constraints

- One tool name per job. `upload_asset` gains a second form rather than a
sibling tool, so a skill can say "upload_asset" on both transports.
- The MCP server knows which transport it is on: `build_server(client,
mounted=False)`; `dw/server/mcp_mount.build_mcp_app` passes `mounted=True`.
Nothing else consults it.
- Refusals explain. A form that cannot work on this transport returns a
`DwApiError` whose message names the form that does; it never falls through
to a bare "No such file".
- Inline content is bounded: 16 MiB decoded (`MAX_INLINE_UPLOAD_BYTES`). A
voice reference, a portrait or a short clip fits; a video of any length does
not, and the message says so and gives the route.
- The extension gate matches the decoders. What `POST /api/uploads` accepts is
what `load_audio` / the H3 audio reference can read.

## 1. `upload_asset`, two forms

### Signature

```
upload_asset(file_path: str | None = None,
content_base64: str | None = None,
filename: str | None = None) -> dict
```

Exactly one of `file_path` or `content_base64`. `filename` is required with
`content_base64` (it carries the extension the gate checks) and ignored with
`file_path` (the basename is used, as now).

### Rules

- `file_path` form: unchanged when `mounted=False`. When `mounted=True` it is
refused before touching the filesystem: "This MCP endpoint runs on the
engine's machine, so `file_path` names a path there, not where you are. Pass
the file as `content_base64` with `filename` (up to 16 MiB), or POST the raw
bytes to `<base_url>/api/uploads?filename=<name>&workspace=<ws>` with the
same bearer token." `<base_url>` and `<ws>` are filled from the client, so
the message is a command the agent can run.
- `content_base64` form, both transports: decode (refuse on padding/alphabet
error with a plain message), check the decoded size against
`MAX_INLINE_UPLOAD_BYTES`, check `filename`'s extension against
`ALLOWED_UPLOAD_EXTENSIONS`, then `client.post_bytes("/api/uploads", body,
params={"filename": filename})` exactly as the path form does. Same return
shape: `reference`, `url`, plus `bytes` (decoded size) so the agent can see
the file arrived whole.
- The docstring is set per transport in `build_server` before registration
(`fn.__doc__ = ...`), so a mounted session reads the caveat in the tool
listing rather than discovering it on failure. Both docstrings name the
extension list and the inline cap.
- Annotation stays `WRITES`.

### Module

`dw_mcp/assets.py`: `upload_asset(client, file_path=None, content_base64=None,
filename=None, mounted=False)`; the path form's body moves into a private
`_post_upload(client, body, filename)` shared by both. `MAX_INLINE_UPLOAD_BYTES`
beside `MAX_UPLOAD_BYTES`. `dw_mcp/server.py` passes `mounted` through and
selects the docstring.

## 2. `download_output` when mounted

### Rules

- `mounted=False`: unchanged.
- `mounted=True`: the tool is still registered (a hidden tool cannot explain
itself) but refuses before contacting the server, returning a `DwApiError`
whose message gives the gallery `url` for `name` - the same one `list_gallery`
reports, with the workspace selector - and a one-line `curl -H
"Authorization: Bearer <token>" -o <basename> <url>`. The token is not
echoed; the placeholder is literal.
- The docstring's existing transport paragraph is kept and moved to the front
of the mounted variant.

This closes the 2026-09-08 memory note: a cold agent no longer drops mp4s into
the server's working directory.

## 3. The extension gate follows the decoders

### Today

`dw/security.py` `ALLOWED_AUDIO_EXTENSIONS = {.wav .mp3 .flac .ogg}`; the same
set is copied into `dw_mcp/assets.py`. `dw/tasks/audio_utils.load_audio` reads
audio extensions with `soundfile` and routes video extensions through PyAV.
diffusers' `MiniMaxH3AudioReference.from_file` decodes with PyAV and takes
`.m4a`, `.aac`, `.opus`. A phone recording (`.m4a`) is refused at the gate
though the reference it was meant for would have decoded it.

### Rules

- `ALLOWED_AUDIO_EXTENSIONS` gains `.m4a`, `.aac`, `.opus`.
- `load_audio` tries `soundfile` first and, on `soundfile.LibsndfileError` or
an extension libsndfile does not handle, decodes through PyAV the way the
video branch already does, returning the same `(samples, channels)` float32
and sample rate. The audio tasks therefore accept everything the gate does.
- `dw_mcp` does not import `dw` (it is installable alone), so the copy stays,
and a test asserts the two sets are equal.

## 4. Tests

- `tests/test_mcp_assets.py`: the `content_base64` form posts decoded bytes to
`/api/uploads` with the given filename and returns `reference`/`url`/`bytes`;
over the cap is refused naming the route; bad base64 is refused plainly;
both forms given, or neither, is refused; `filename` without an allowed
extension is refused before any request; the `file_path` form is refused
when `mounted=True` and the message contains the base URL, the workspace and
`content_base64`.
- `tests/test_mcp_server.py`: `build_server(client, mounted=True)` registers
`upload_asset` whose description contains the transport caveat and
`build_server(client)` does not; `download_output` mounted refuses with the
gallery url and a `curl` line and makes no HTTP call.
- `tests/test_server_mcp.py`: a round trip through the mounted app - a small
wav as `content_base64` lands in the selected workspace's `assets/` and
`asset:<name>` resolves in a validated workflow.
- `tests/test_assets.py` / audio task tests: `.m4a` passes the gate;
`load_audio` decodes an `.m4a` fixture (generated in the test with PyAV, no
binary checked in) to the same shape and rate as its wav twin; the two
extension sets are equal.

## 5. Docs

- docs/MCP.md: a "Where files live" section - the two transports, which
machine each tool's paths mean, the inline form and its cap, the `curl`
forms for both directions. Replaces the scattered sentences in the two tool
docstrings' current prose.
- docs/SERVER.md: the widened audio list where uploads are described.
- `plugins/dw/skills/minimax-h3/SKILL.md` "Run and judge" step 5 gains one
sentence: a recorded voice reference reaches the server with `upload_asset`
(`content_base64` over a `dw.serve --mcp` endpoint); the skill already
explains the export-zip fetch in the other direction. Version-pinned by the
existing size test only; no numbers are stated.
- `.remember` / memory: the 2026-09-08 workaround note becomes "fixed in
<commit>".

## Sequence for the plan

1. `mounted` flag through `build_server` and `mcp_mount`; per-transport
docstrings; `file_path` refusal when mounted (section 1, first bullet).
Ships alone if needed - it would have prevented the ssh attempt.
2. Extension gate and `load_audio` fallback (section 3).
3. `content_base64` form (section 1, rest).
4. `download_output` mounted refusal (section 2).
5. Tests alongside each step; docs last.

Do not restart the GPU box's `dw.serve` to pick this up while a drill job is
running - a restart kills the worker mid-job.
22 changes: 19 additions & 3 deletions plugins/dw/skills/minimax-h3/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ not from here.
- **Several boards in one generation, one unbroken score**:
`templates/minimax/storyboard` - H3 cuts between the boards inside a single
generation, which no concat of separate clips can match for continuous audio.
It is one beat with fixed cut points, not a building block: four of them
concatenated give twelve equal-length shots and a cast redrawn four times.
Past one beat with a recurring cast, use the cuts pattern below.
- **Longer than 14.4 seconds**: decide first whether the seam is a cut or a
continuation. Chain when the same action or line of speech has to cross the
seam; cut when the scene changes, and treat each cut as its own generation.
Expand All @@ -62,7 +65,11 @@ not from here.
author the concat and score steps the same way). A character who speaks in
several shots keeps one voice by passing the same clip as an audio
reference in each (the `voice-timbre-reference` pattern); a repeated voice
description alone drifts.
description alone drifts. Each shot's `num_frames` is its own, so pace the
cut - a trailer builds by varying shot length. The reference carries
delivery as well as timbre: a flat read gives a flat performance. Bark's
presets are conversational; for a narrator with gravitas, `upload_asset` a
recorded read in that register and reference the same file in every shot.
- **Music alone**: `templates/minimax/music` (Music3); the `minimax-music3` skill.

If none fits, compose from `list_tasks` before authoring a new workflow, and
Expand Down Expand Up @@ -94,6 +101,10 @@ read the `workflows` guide's authoring section first.
cuts is laid under the concat afterwards.
- H3 is guidance-distilled: no `guidance_scale`, no negative prompt. Say what is
there, never what is not.
- When deriving a variant, keep `release_pipeline` on the step the template
puts it on: it frees the Z-Image boards before H3 loads. A run killed by
SIGKILL near the end, in a worker warm from a previous job, that succeeds
on a retry in a fresh worker is host memory, not the prompt.
- Ref2VA limits: at most 9 images, 3 videos, 3 audio clips, 12 files; audio can
never be the only reference. References are labelled in the order passed.
- Music3 reads `audio_duration` as a ceiling, not a target: ask for more than
Expand All @@ -119,7 +130,9 @@ paraphrase it from examples:
built-in enhancer writes the format from those guides. Its `idea` is
framed as `Task: T2VA. Duration: 5.17 seconds. Idea: ...`.

Whichever route: repeat a speaker's voice description verbatim across shots,
Whichever route: write the whole script before the first shot - the lines in
order, read once, should carry the piece on their own - then place them.
Repeat a speaker's voice description verbatim across shots,
and when a reference picture should fix identity but not framing, say so in
the prompt itself - in a reference-conditioned request, in the lines that
define the subject and state what each reference keeps - or every shot
Expand Down Expand Up @@ -149,7 +162,10 @@ inherits the portrait's composition.
`templates/minimax/music-video`. Ask the user to look for the family's
failure modes: a character that changes between shots (reference the same
portraits in every shot), a reference portrait imposing its framing on every
shot, a storyboard skipped, drift sharpening into noise late in a chain.
shot, a storyboard skipped, drift sharpening into noise late in a chain,
a voice-over without affect (the reference's delivery came through), every
shot the same length, a look word repeated on every board (shallow depth
of field) softening every shot.
5. After an inline run worth keeping, `get_job_workflow` and `save_workflow` it,
so the next run is by name rather than by pasting JSON; `export_job` bundles
the run — workflow, manifest, job row and media — for git. The bundle is on
Expand Down
Loading