Repository navigation
feat(model): support Liquid AI LFM3 conv/attention hybrid architecture - #5
Open
falabellamichael wants to merge 2 commits into
Open
falabellamichael wants to merge 2 commits into
falabellamichael wants to merge 2 commits into
Conversation
Add support for Liquid AI's LFM3 conv/attention hybrid model architecture
(tested on LiquidAI/LFM3-1.7B-MLX-4bit and LiquidAI/LFM3-Omni-2B-MLX-4bit):
- Model architecture (src/models/lfm3.cpp):
* Lfm3Attention: gated attention with per-head q/k RMSNorm, per-head
sigmoid output gate expanded at load, rotary channel permutation,
sliding-window and causal masking, and post-operator norm.
* Lfm3Conv: gated depthwise 1D convolution (kernel width 7) with in_proj,
out_proj, zero-bias matching filter dtype, and state gathering for
padded passes.
* Lfm3Mlp: SwiGLU feed-forward network with post-FFN norm.
* Model detection predicate and registration for 'lfm3'.
* Explicit refusal of unsupported multimodal towers (vision, audio, depth)
for text-only inference clarity.
- Config parsing (include/lse/model/config.hpp, src/model/config.cpp):
* Add sliding_rope_dim, sliding_rope_theta, and layer_types schedule.
* from_lfm3_json: support nested text_config or flat root, per-attention
RoPE parameters, and custom layer_types schedule.
* Config::validate: relax GDN checks when no linear_attention layers are present.
- Execution fixes:
* sdpa: fix Loom sliding-window mask condition j + window > abs_i to prevent
unsigned subtraction underflow.
* hybrid_lm: guard gdn_state allocation so non-GDN hybrid stacks don't
allocate unneeded recurrent state; gather tail rows on padded conv passes.
* tokenizer / router: parse declared bos_token from tokenizer_config.json
and prepend on chat path to prevent generation degeneration.
…g parsing; clear global_attention_layers
Geramy
reviewed
Nov 3, 2026
| const auto heads = static_cast<std::int64_t>(config_.gdn_v_heads); | ||
| const auto tail = static_cast<std::int64_t>(config_.gdn_conv_kernel) - 1; | ||
| if (heads <= 0 || tail <= 0 || t_live < tail) | ||
| if (heads <= 0 && tail <= 0) |
Owner
There was a problem hiding this comment.
I need you to verify the existing Qwen3.8-27B still works with this rule here
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for Liquid AI's LFM3 (conv/attention hybrid) model architecture, verified on
LiquidAI/LFM3-1.7B-MLX-4bitandLiquidAI/LFM3-Omni-2B-MLX-4bit.Architecture & Mixer Details (
src/models/lfm3.cpp)Lfm3Attention(IMixer):head_dimchannels at load time.rope_source) matching the adjacent-pair layout used by the engine.is_sliding_attention), and post-operator RMSNorm.Lfm3Conv(IMixer):in_projemits B x C x x, convolved stream is B x x, gated by C, and projected throughout_proj.conv_w_.dtype()).Lfm3Mlp(IFeedForward):w1,w3,w2) with post-FFN RMSNorm.vision_tower.,audio_tower.,audio_projector.,depth_decoder.,multi_modal_projector.,codebook_embedding.) with clear messages so text decode remains transparent about modalities.Config Parsing (
include/lse/model/config.hpp,src/model/config.cpp)sliding_rope_dim,sliding_rope_theta, andlayer_typesschedule toConfig.is_attention_layer(layer_idx)andis_sliding_attention(layer_idx)driven by thelayer_typesschedule.from_lfm3_json: parses nestedtext_configor flat root, per-attention-kindrope_parameters(full_attentionandsliding_attention), customlayer_types, and conv kernel cache width. Clears defaultglobal_attention_layers{7}so models with <= 7 layers validate cleanly and layers aren't misclassified.from_json_stringwhenmodel_typeorarchitecturesstarts withlfm/Lfm.validate(): relaxes Gated DeltaNet checks when nolinear_attentionlayers are in the schedule (LFM3 conv layers carry no recurrent GDN state).Engine & Kernel Fixes
src/kernels/sdpa.cpp):(abs_i - j) < windowtoj + window > abs_ito avoid unsigned 32-bit underflow/wrap in Loom whenj > abs_i.src/model/hybrid_lm.cpp):HybridLM::hidden, guardgdn_stateallocation to only allocate recurrent state if GDN value heads and dimension are positive (vh_m > 0 && vd > 0).heads <= 0andtail > 0(conv-only tail), and gather tail rows for conv layers even without GDN value heads.include/lse/tokenizer/tokenizer.hpp,src/tokenizer/tokenizer.cpp,src/server/router.cpp):Tokenizer::from_filechecks fortokenizer_config.jsonbesidetokenizer.jsonand parsesbos_tokenif declared.Router::Impl::completion, ifchat == true, prependbos_token_id()if declared and not already present at the start of prompt, matching HF/vLLMadd_special_tokenschat behavior (raw completions remain verbatim).Unit Tests Added (
tests/test_model.cpp)every_architecture_claims_only_its_own_checkpoint: verifieslfm3architecture predicate claims LFM3 weights exclusively.an_unrecognized_checkpoint_names_what_was_tried: verifieslfm3appears in tried architectures diagnostic when detection fails.lfm3_config_is_read_with_per_type_rope: verifies parsing of sliding/full RoPE dims, thetas, layer schedule queries (is_attention_layer,is_sliding_attention,is_global_attention), and JSON round-tripping.Verification & Testing
test_model56/56 passed.test_tokenizer10/10 passed.test_chat_protocol15/15 passed.lse-serveron port 8085 withLiquidAI/LFM3-Omni-2B-MLX-4bit):