Skip to content

feat(model): support Liquid AI LFM3 conv/attention hybrid architecture - #5

Open
falabellamichael wants to merge 2 commits into
Geramy:masterfrom
falabellamichael:feat/lfm3
Open

falabellamichael wants to merge 2 commits into
Geramy:masterfrom
falabellamichael:feat/lfm3

Conversation

@falabellamichael

@falabellamichael falabellamichael commented Nov 3, 2026 •

Copy link
Copy Markdown

Summary

This PR adds support for Liquid AI's LFM3 (conv/attention hybrid) model architecture, verified on LiquidAI/LFM3-1.7B-MLX-4bit and LiquidAI/LFM3-Omni-2B-MLX-4bit.

Architecture & Mixer Details (src/models/lfm3.cpp)

  • Lfm3Attention (IMixer):
    • Gated attention with per-head Q/K RMSNorm and per-head sigmoid output gate. The output gate projection is expanded across head_dim channels at load time.
    • Per-head rotary channel permutation mapping (rope_source) matching the adjacent-pair layout used by the engine.
    • Supports causal and sliding-window masks (is_sliding_attention), and post-operator RMSNorm.
  • Lfm3Conv (IMixer):
    • Gated depthwise 1D convolution (kernel width 7): in_proj emits B x C x x, convolved stream is B x x, gated by C, and projected through out_proj.
    • Zero-bias initialized to match the filter's data type (conv_w_.dtype()).
    • Gathers state tail rows during padded passes for conv-only states.
  • Lfm3Mlp (IFeedForward):
    • SwiGLU MLP (w1, w3, w2) with post-FFN RMSNorm.
  • Refusal List:
    • Explicitly refuses unsupported multimodal towers in Omni checkpoints (vision_tower., audio_tower., audio_projector., depth_decoder., multi_modal_projector., codebook_embedding.) with clear messages so text decode remains transparent about modalities.

Config Parsing (include/lse/model/config.hpp, src/model/config.cpp)

  • Added sliding_rope_dim, sliding_rope_theta, and layer_types schedule to Config.
  • Added helper queries is_attention_layer(layer_idx) and is_sliding_attention(layer_idx) driven by the layer_types schedule.
  • from_lfm3_json: parses nested text_config or flat root, per-attention-kind rope_parameters (full_attention and sliding_attention), custom layer_types, and conv kernel cache width. Clears default global_attention_layers {7} so models with <= 7 layers validate cleanly and layers aren't misclassified.
  • Dispatches in from_json_string when model_type or architectures starts with lfm/Lfm.
  • validate(): relaxes Gated DeltaNet checks when no linear_attention layers are in the schedule (LFM3 conv layers carry no recurrent GDN state).

Engine & Kernel Fixes

  • SDPA sliding-window mask (src/kernels/sdpa.cpp):
    • Rewrote sliding window condition from (abs_i - j) < window to j + window > abs_i to avoid unsigned 32-bit underflow/wrap in Loom when j > abs_i.
  • Hybrid LM state handling (src/model/hybrid_lm.cpp):
    • In HybridLM::hidden, guard gdn_state allocation to only allocate recurrent state if GDN value heads and dimension are positive (vh_m > 0 && vd > 0).
    • In padded passes, allow gather when heads <= 0 and tail > 0 (conv-only tail), and gather tail rows for conv layers even without GDN value heads.
  • Tokenizer / Router BOS handling (include/lse/tokenizer/tokenizer.hpp, src/tokenizer/tokenizer.cpp, src/server/router.cpp):
    • Tokenizer::from_file checks for tokenizer_config.json beside tokenizer.json and parses bos_token if declared.
    • In Router::Impl::completion, if chat == true, prepend bos_token_id() if declared and not already present at the start of prompt, matching HF/vLLM add_special_tokens chat behavior (raw completions remain verbatim).

Unit Tests Added (tests/test_model.cpp)

  • every_architecture_claims_only_its_own_checkpoint: verifies lfm3 architecture predicate claims LFM3 weights exclusively.
  • an_unrecognized_checkpoint_names_what_was_tried: verifies lfm3 appears in tried architectures diagnostic when detection fails.
  • lfm3_config_is_read_with_per_type_rope: verifies parsing of sliding/full RoPE dims, thetas, layer schedule queries (is_attention_layer, is_sliding_attention, is_global_attention), and JSON round-tripping.

Verification & Testing

  • Model unit tests: test_model 56/56 passed.
  • Tokenizer unit tests: test_tokenizer 10/10 passed.
  • Chat protocol unit tests: test_chat_protocol 15/15 passed.
  • Full test suite: 106/111 passed (remaining 5 tests are pre-existing host-environment dependencies like JIT compiler and GCC16 reflection).
  • Live model testing on GPU (lse-server on port 8085 with LiquidAI/LFM3-Omni-2B-MLX-4bit):
    • Single-turn chat: verified ("Paris", 1259 prompt tok/s, 0 fallbacks).
    • Streamed chat: verified SSE tokens chunking correctly.
    • Raw completions: verified (/v1/completions, 85.3 decode tok/s).
    • Multi-turn conversational chat: verified context retention.
    • Long prompt stress test: verified 628-token prompt exceeding the 256 sliding window with 0 fallbacks and no GPU hang.

Add support for Liquid AI's LFM3 conv/attention hybrid model architecture
(tested on LiquidAI/LFM3-1.7B-MLX-4bit and LiquidAI/LFM3-Omni-2B-MLX-4bit):

- Model architecture (src/models/lfm3.cpp):
  * Lfm3Attention: gated attention with per-head q/k RMSNorm, per-head
    sigmoid output gate expanded at load, rotary channel permutation,
    sliding-window and causal masking, and post-operator norm.
  * Lfm3Conv: gated depthwise 1D convolution (kernel width 7) with in_proj,
    out_proj, zero-bias matching filter dtype, and state gathering for
    padded passes.
  * Lfm3Mlp: SwiGLU feed-forward network with post-FFN norm.
  * Model detection predicate and registration for 'lfm3'.
  * Explicit refusal of unsupported multimodal towers (vision, audio, depth)
    for text-only inference clarity.

- Config parsing (include/lse/model/config.hpp, src/model/config.cpp):
  * Add sliding_rope_dim, sliding_rope_theta, and layer_types schedule.
  * from_lfm3_json: support nested text_config or flat root, per-attention
    RoPE parameters, and custom layer_types schedule.
  * Config::validate: relax GDN checks when no linear_attention layers are present.

- Execution fixes:
  * sdpa: fix Loom sliding-window mask condition j + window > abs_i to prevent
    unsigned subtraction underflow.
  * hybrid_lm: guard gdn_state allocation so non-GDN hybrid stacks don't
    allocate unneeded recurrent state; gather tail rows on padded conv passes.
  * tokenizer / router: parse declared bos_token from tokenizer_config.json
    and prepend on chat path to prevent generation degeneration.
Comment thread src/model/hybrid_lm.cpp
const auto heads = static_cast<std::int64_t>(config_.gdn_v_heads);
const auto tail = static_cast<std::int64_t>(config_.gdn_conv_kernel) - 1;
if (heads <= 0 || tail <= 0 || t_live < tail)
if (heads <= 0 && tail <= 0)

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need you to verify the existing Qwen3.8-27B still works with this rule here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants