feat: add PPLX Qwen3 bidirectional embeddings (perplexity-ai/pplx-embed-v1-4b)

[StartupBros]

Adds support for perplexity-ai/pplx-embed-v1-4b (model_type: bidirectional_pplx_qwen3), #9 on the MTEB Code leaderboard.

What it is

A stock Qwen3 decoder run with full bidirectional attention and mean pooling. Upstream's modeling.py is a thin Qwen3Model subclass whose entire job is to set layer.self_attn.is_causal = False on every layer and OR a bidirectional mask over the causal one — so the causal triangle disappears and every token sees every non-padded token. The model does no pooling itself; 1_Pooling/config.json selects pooling_mode_mean_tokens.

Shape details worth reviewing

Three things are load-bearing, each documented in the module:

• head_dim is 128 while hidden_size is 2560, so 32 × 128 = 4096 and q_proj is 2560 → 4096. Deriving head_dim as hidden_size // num_heads gives 80 and loads nothing.
• Qwen3 applies per-head RMSNorm to q and k before RoPE. Qwen2 does not, and carries q/k/v biases instead — attention_bias is false here.
• tie_word_embeddings is true, so there is no separate lm_head tensor.

The bidirectional mask is keyed on columns only, so a padded row still attends to real tokens and every softmax row stays finite. Padded rows are discarded by mean pooling regardless.

Config is validated rather than assumed: the pinned revision ships use_sliding_window: false with all 36 layer_types set to full_attention, and ModelArgs raises if a checkpoint disagrees instead of silently ignoring sliding-window settings.

Verification

End to end against the PyTorch reference on an M3 Ultra, in float32, mean-pooled identically on both sides:

case	result
single text, no padding	cosine 1.0
padded batch (3 texts)	cosine 1.0 on all rows

min_cosine: 1.0 across 4 comparisons. Parameter count is 4,022,468,096, matching the published count exactly.

Also serving-verified: loaded under oMLX 0.6.3 and returned a 2560-dimensional vector over /v1/embeddings.