Skip to content

Add BitNet b1.58 ternary-weight model support #83

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
justinchuby wants to merge 13 commits into
base: main
Choose a base branch
from
Draft

Add BitNet b1.58 ternary-weight model support #83

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
b62bbc0
Add LFM2 hybrid conv+attention model and AudioToAudioTask scaffold
justinchuby Apr 1, 2026
c6c83d5
Add LFM2-Audio model with AudioToAudioTask (4-model split)
justinchuby Apr 1, 2026
090cdc6
Fix depthformer codebook head selection and remove dead embedding input
justinchuby Apr 1, 2026
ad09ae4
Add ShortConv unit tests and audio-to-audio categories to dashboard
justinchuby Apr 1, 2026
ddab9c2
DRY: reuse lfm2 decoder layers, add YAML golden tests and coverage
justinchuby Apr 1, 2026
755c326
Add Moshi/PersonaPlex audio-to-audio model
justinchuby Apr 1, 2026
27b7989
Add Moshi/PersonaPlex real-time streaming example
justinchuby Apr 1, 2026
12124c3
fix: address PR #77 review comments
justinchuby Apr 1, 2026
6cef22c
feat: add BitNet b1.58 ternary-weight model support
justinchuby Apr 2, 2026
f3da0ce
docs: add bitnet-quantized-models skill file
justinchuby Apr 2, 2026
3986955
Revert "docs: add bitnet-quantized-models skill file"
justinchuby Apr 2, 2026
be515c4
fix: address PR #83 review findings
justinchuby Apr 2, 2026
8188ca1
fix: add Q/K norm handling to BitNetAttention.forward
justinchuby Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/_generate_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,10 @@
_CATEGORY_DESCRIPTIONS: dict[str, str] = {
"Text Generation": "Standard autoregressive language models (CausalLM).",
"Mixture of Experts": "Models that route tokens to a subset of expert MLPs.",
"Hybrid Conv+Attention": "Hybrid models with alternating conv and attention layers (LFM2).",
"Multimodal": "Models that process images, audio, or other modalities alongside text.",
"Speech-to-Text": "Encoder-decoder models for speech recognition.",
"Audio-to-Audio": "End-to-end audio language models: audio in, audio + text out (LFM2-Audio, Moshi).",
"Audio": "Audio encoder models for feature extraction (Wav2Vec2, HuBERT, WavLM).",
"encoder-only": "Encoder-only models for embeddings and classification (BERT, RoBERTa).",
"encoder-decoder": "Encoder-decoder sequence-to-sequence models (BART, T5, mBART).",
Expand All @@ -39,8 +41,10 @@
_CATEGORY_ORDER = [
"Text Generation",
"Mixture of Experts",
"Hybrid Conv+Attention",
"Multimodal",
"Speech-to-Text",
"Audio-to-Audio",
"Audio",
"encoder-only",
"encoder",
Expand Down
22 changes: 20 additions & 2 deletions docs/model-catalog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Model Catalog

**mobius** supports 273 registered model types across 10 categories.
**mobius** supports 275 registered model types across 12 categories.
This catalog lists every supported architecture with its module class, task type,
and example HuggingFace model IDs.
Comment on lines +3 to 5

Copilot AI Apr 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs/model-catalog.md is updated to reflect new totals/categories, but it still doesn’t mention newly added model types from this PR (e.g., bitnet, moshi, personaplex) anywhere in the catalog tables. If this page is meant to list every supported model_type, add rows for these entries (or regenerate the page from the registry to keep it consistent).

Copilot uses AI. Check for mistakes.

Expand Down Expand Up @@ -96,6 +96,22 @@ Mamba and Mamba2 architectures using selective state-space layers.
| `falcon_mamba` | `MambaCausalLMModel` | `ssm-text-generation` | `tiiuae/falcon-mamba-7b` |
| `mamba2` | `Mamba2CausalLMModel` | `ssm2-text-generation` | `state-spaces/mamba2-2.7b` |

## Hybrid Conv+Attention

Models with alternating depthwise-conv (ShortConv) and attention layers.

| Model Type | Module Class | Task | Example HuggingFace Model |
|---|---|---|---|
| `lfm2` | `Lfm2CausalLMModel` | `hybrid-text-generation` | `LiquidAI/LFM2-1.2B` |

## Audio-to-Audio

End-to-end audio language models: audio and text in, audio and text out.

| Model Type | Module Class | Task | Example HuggingFace Model |
|---|---|---|---|
| `lfm2_audio` | `Lfm2AudioModel` | `audio-to-audio` | `LiquidAI/LFM2-Audio-1.5B` |

## Hybrid SSM+Attention

Models combining Mamba/SSM layers with transformer attention layers.
Expand Down Expand Up @@ -273,11 +289,13 @@ MatMulNBits ops.
|---|---|---|
| Decoder-only LLMs | ~100 | `CausalLMModel`, `GPT2CausalLMModel` |
| Mixture of Experts | ~25 | `MoECausalLMModel`, `DeepSeekV3CausalLMModel` |
| Hybrid Conv+Attention | 1 | `Lfm2CausalLMModel` |
| SSM / Hybrid | 5 | `MambaCausalLMModel`, `JambaCausalLMModel` |
| Vision-Language | ~40 | `LLaVAModel`, `Qwen25VLCausalLMModel` |
| Encoder-only | ~40 | `BertModel`, `DistilBertModel` |
| Encoder-decoder | ~20 | `BartForConditionalGeneration`, `T5ForConditionalGeneration` |
| Speech & Audio | ~20 | `WhisperForConditionalGeneration`, `Wav2Vec2Model` |
| Audio-to-Audio | 1 | `Lfm2AudioModel` |
| Vision | ~25 | `ViTModel`, `CLIPVisionModel` |
| Diffusion | ~10 | `UNet2DConditionModel`, `FluxTransformer2DModel` |
| **Total** | **~273** | |
| **Total** | **~275** | |
Loading
Loading