Onboard Qwen2.5-Omni Thinker model - #193
Open
rui-ren wants to merge 11 commits into
Open
Conversation
added 2 commits
April 22, 2026 16:02
Performance Comparison
|
|
The author of this PR, rui-ren, is not an activated member of this organization on Codecov. |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Contributor
Author
|
@copilot can you please use this skill of Currently I am only working on the |
Contributor
Reviewed the thinker work at d5a4ec9 using the
|
Pick up the partial Qwen2.5-Omni onboarding (#193) and bring it to a state where the file imports cleanly, the full pytest suite collects without errors, and the registry-completeness test passes. Concrete fixes: * Syntax: removed stray ")" at end of Qwen25OmniDecoderModel that broke every test that imported the module (65 collection errors on plain main). * Missing imports: RMSNorm, DecoderLayer, initialize_rope were referenced by Qwen25OmniDecoderModel but not imported. Also consolidated the OpBuilder import (was using the internal onnxscript._internal.builder module). * Typos in Qwen25OmniEmbeddingModel: - Used audio.image_token_id (image_token_id lives on VisionConfig). - self._image_token_ids plural typo where _image_token_id was meant. - "return input_embeds" referenced a non-existent local; renamed. - Forward arg "inputs_ids" -> "input_ids". * Composite class implemented: Qwen25OmniThinkerForConditionalGeneration was just `pass`. Now mirrors the gemma4 multi-encoder layout — owns decoder, embedding, vision_encoder, and audio_tower (named to match what SpeechLanguageTask looks up). forward() raises NotImplementedError because this is a multi-output split needing a dedicated task. preprocess_weights maps HF thinker.audio_tower.* / thinker.visual.* / thinker.model.* / thinker.lm_head.* onto the new module structure, drops Talker / codec-head keys, and applies the standard embed/lm_head weight tying. Test infrastructure: * Added qwen2_5_omni to _KNOWN_UNTESTED_MODEL_TYPES with a comment explaining the situation: the audio + vision + text 4-model split needs a dedicated task (like Gemma4Task); the existing speech-language task only drives audio + text. When that task is added in a follow-up, this entry should move into a dedicated graph test (mirroring test_gemma4_any_to_any_graph). Also cleaned up: unused n_window lookup, unused dataclasses import, trailing blank lines, and audio-attention docstring punctuation that ruff flagged. Tests: full tests/build_graph_test.py + src/mobius/ suite passes (2767 passed, 43 skipped). Ruff check + format clean on touched files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
Member
|
@copilot update to the latest and ensure the implementation is clean and correct |
Signed-off-by: GitHub <noreply@github.com>
Add a dedicated four-model task and align audio, vision, decoder, configuration, and weight routing with the current Hugging Face Thinker implementation. Cover the nested config and complete graph package with focused regression tests. Signed-off-by: GitHub <noreply@github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR onboards the Qwen2.5-Omni (Thinker) architecture into mobius, implementing a four-model ONNX split (audio encoder, vision encoder, embedding, decoder) and wiring it into the registry/config extraction paths.
Changes:
- Added
Qwen25OmniThinkerForConditionalGeneration+Qwen25OmniTaskto build a 4-partModelPackage(audio_encoder,vision_encoder,embedding,decoder). - Implemented new Omni audio encoder components and a generic
Conv1dbuilding block. - Extended config extraction to pull Omni’s nested
thinker_config.vision_config(includingvideo_token_id) and registered the new model/task in the public APIs and registry.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/build_graph_test.py | Adds a graph-build test for the 4-model package and marks the model type as “specialized” for registry completeness. |
| src/mobius/tasks/_qwen25_omni.py | New task to build the Omni 4-model split (audio/vision/embedding/decoder). |
| src/mobius/tasks/init.py | Exposes Qwen25OmniTask and registers the task name qwen25-omni. |
| src/mobius/models/qwen25_omni.py | New Thinker model implementation (audio + vision + embedding fusion + decoder) and HF weight routing. |
| src/mobius/models/qwen25_omni_test.py | Unit tests for nested-config extraction and preprocess_weights() routing rules. |
| src/mobius/models/init.py | Exports Qwen25OmniThinkerForConditionalGeneration. |
| src/mobius/components/_qwen25_omni_audio.py | Adds Omni packed audio attention + encoder layer components. |
| src/mobius/components/_conv.py | Adds a reusable Conv1d component. |
| src/mobius/components/init.py | Exposes Omni audio components via the public components API. |
| src/mobius/_registry.py | Registers model type qwen2_5_omni with task qwen25-omni. |
| src/mobius/_configs/per_model/_qwen25_omni_vision.py | Adds a per-model hook to extract vision config from thinker_config. |
| src/mobius/_configs/per_model/init.py | Imports the new Omni vision extractor hook. |
| src/mobius/_configs/_sub_configs.py | Adds video_token_id to VisionConfig. |
| src/mobius/_configs/_extractors.py | Lifts video_token_id as a shared vision field. |
| src/mobius/_configs/_base.py | Adds top-level video_token_id and recognizes qwen2_5_omni_text for config extraction. |
Comment on lines
+85
to
+89
| q = op.Unsqueeze(op.Transpose(q, perm=[1, 0, 2]), [0]) | ||
| k = op.Unsqueeze(op.Transpose(k, perm=[1, 0, 2]), [0]) | ||
| v = op.Unsqueeze(op.Transpose(v, perm=[1, 0, 2]), [0]) | ||
| attn_output = op.Attention( | ||
| q, |
Comment on lines
+14
to
+17
| import onnx_ir as ir | ||
| from onnxscript import nn | ||
| from onnxscript._internal import builder | ||
|
|
Comment on lines
+40
to
+48
| def build(self, module: nn.Module, config: ArchitectureConfig) -> ModelPackage: | ||
| self._validate_components(module) | ||
| models = { | ||
| "audio_encoder": self._build_audio(module.audio_encoder, config), | ||
| "vision_encoder": self._build_vision(module.vision_encoder, config), | ||
| "embedding": self._build_embedding(module.embedding, config), | ||
| "decoder": build_decoder_from_embeds(module.decoder, config, mrope=True), | ||
| } | ||
| return ModelPackage(models, config=config) |
Comment on lines
+58
to
+62
| input_features = builder.input( | ||
| "input_features", | ||
| dtype=ir.DataType.FLOAT, | ||
| shape=[num_chunks, n_mels, chunk_len], | ||
| ) |
Comment on lines
+432
to
+435
| - ``decoder``: Qwen2.5 text decoder taking ``inputs_embeds`` | ||
| - ``vision_encoder``: Qwen2.5-VL ViT (pixel_values + grid_thw → image features) | ||
| - ``audio_tower``: 2x Conv1d + transformer audio tower (mel → audio features) | ||
| - ``embedding``: word embedding + multimodal feature fusion |
Comment on lines
5003
to
+5007
| "fun_asr", | ||
| "mms", | ||
| "qwen3_asr", | ||
| "qwen3_forced_aligner", | ||
| "qwen2_5_omni", |
Contributor
Updated onto latest |
Copilot
AI
changed the title
Onboard qwen25-omni model
Onboard Qwen2.5-Omni Thinker model
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Qwen2.5-Omni Thinker support with: