[Feature] Ideogram4 Qwen3-VL image conditioning

[Green-Sky]

Feature Summary

Effectively providing editing capabilities.

Detailed Description

The embedder/conditioner for ideogram4 is Qwen3-VL, which is capable of taking in images, and technically videos. While ig4 might not be trained for that specifically, the llm was frozen during training, meaning it will work anyway.

I see --llm_vision exists for other models already, so that would be the thing needing extension.

prompt template: Picture {{ image_count.value }}: {% endif -%} <|vision_start|><|image_pad|><|vision_end|>

[stduhpf]

I've already tried (not really hard) to implement it, looks like vision for Qwen3-VL has already been implemented in #1485, but the tensor naming in the llama.cpp mmproj is not recognised, probably because Hidream comes as an all-in-one checkpoint.

[stduhpf]

@Green-Sky Looks like most of the required work was done with #1688

[Green-Sky]

Comfy-Org/ComfyUI#14567

This looks like something similar.

[stduhpf]

Comfy-Org/ComfyUI#14567

This looks like something similar.

That's completely different, it's not based on vision, but uses RoPE offsets, like the edit models. I'm surprised it can work.

[Green-Sky]

Comfy-Org/ComfyUI#14567
This looks like something similar.

That's completely different, it's not based on vision, but uses RoPE offsets, like the edit models. I'm surprised it can work.

I guess I got confused by the img to ref stuff. Still fun.

[stduhpf]

It does not turn Ideogram into a proper edit model, but it works much better than just the visual conditionning approach.

[Green-Sky]

My guess is that pretaining contained some editing to make the editing finetune straight forward.

[stduhpf]

It behaves a bit like an instruct-pix2pix model with img_cfg too high, adhering way too closely to the reference image.

[Green-Sky]

https://github.com/BitPoet/ComfyUI-bitpoet-IG4Inpaint
https://huggingface.co/BitPoet/Ideogram4-Inpaint-LoRA

Someone did an editing lora. Not sure which one this is at first glance, it talks a lot about conditioning, but it looks more like ref latents too me.