feat(vlm): add a token-exact Phi4MM audio reference path

[inureyes]

Parent

• Parent epic: epic: OpenXLA multimodal / VLM architecture support (Qwen2-VL, Gemma3n, Phi4MM, Molmo, and more) #566
• Current gap: src/multimodal/phi4mm_prompt.rs explicitly rejects <|audio_N|>, Phi4MMVLModel has no audio tower, and src/loading/vlm_special.rs skips audio weights and .speech. LoRA pairs.

Normative upstream sources

• Official checkpoint/code: https://huggingface.co/microsoft/Phi-4-multimodal-instruct/tree/main
• Audio encoder definition: speech_conformer_encoder.py
• Audio embedding, projection, placeholder merge, and adapter selection: modeling_phi4mm.py::Phi4MMAudioEmbedding and Phi4MMImageAudioEmbedding
• Pin the exact upstream checkpoint revision and source revision in tests/docs; do not track mutable main implicitly.

Problem

There is no local executable reference for Phi4MM audio today. Porting only the existing image-capable MLX wrapper to XLA would provide no trustworthy oracle for the cascaded Conformer encoder, audio projection, placeholder replacement, or modality-specific speech LoRA. The current loader intentionally discards precisely those weights.

Goal

Implement and qualify a complete MLX Phi4MM audio reference path before any XLA port, while preserving the existing image and text modes.

Required implementation

• Port the pinned official audio feature-extraction contract from processor_config.json, including feature dimension, frame mask, per-clip sizes, padding, and the checkpoint's documented sample-rate/duration policy.
• Port the configured cascades Conformer encoder from the official source: input/subsampling layers, masks and length propagation, attention/relative-position behavior, convolution modules, feed-forward blocks, normalization, activation, and output selection. Reuse local audio primitives only after intermediate parity proves their semantics match.
• Implement audio_projection exactly for the configured projection class/downsample rate. Support the distinct speech and vision projection branches when present.
• Normalize compatible audio placeholder IDs to the canonical audio special ID, expand one run per clip, and require sum(audio_embed_sizes) == number_of_audio_positions. Replace those text embedding rows with concatenated valid projected frames; do not add them.
• Load model.embed_tokens_extend.audio_embed.* weights and validate every shape. Stop dropping them in rewrite_phi4mm_vision_key/remapping.
• Load complete .lora_A.speech/.lora_B.speech pairs and implement the official input-mode policy: language uses no adapter, speech uses speech adapter plus speech projection, and mixed vision-speech follows the pinned reference's adapter/projection selection. The selected adapter must remain active for decode when the reference requires it.
• Remove the audio prompt guard and implement exact <|audio_N|> ordering/cardinality validation without regressing <|image_N|> normalization.
• Make per-request modality state sequence-scoped; concurrent speech, vision, mixed, and text requests cannot switch a shared adapter or staged input for another request.

Non-goals

• XLA/StableHLO emission.
• Treating Gemma audio encoders as a substitute for the official Phi4MM Conformer.
• Speech generation/output audio.

Validation

• Pin a small deterministic waveform/task fixture plus the official checkpoint revision. Compare feature tensors/masks, subsampling lengths, selected Conformer block outputs, projected audio frames, placeholder positions, merged embeddings, selected decoder layer outputs under speech LoRA, logits/KV, and greedy tokens against the pinned official implementation.
• Cover one and multiple clips, padding, maximum accepted duration boundary, one-frame/empty rejection, placeholder/count mismatch, audio-only, image-only, mixed image+audio, no-media text, and concurrent different input modes.
• Add negative fixtures for missing speech LoRA, wrong projection branch, additive instead of replacement merge, and adapter state leakage.
• Exercise CLI and OpenAI-compatible streaming server paths and record ASR token-exact output for the pinned fixture.
• Run all existing Phi4MM image and Phi3/Phi4MM text regressions.

Acceptance criteria

• The current audio guard is replaced by exact processor/placeholder support.
• Official audio encoder/projection intermediates match within documented tolerances.
• Audio weights and complete speech LoRA pairs are loaded and shape-validated.
• Speech/vision/language adapter selection is per request and concurrency-safe.
• Audio embeddings replace exactly the declared positions and final deterministic output is token-exact.
• CLI and streaming server paths pass for audio and mixed media.
• Existing image-only and text-only Phi4MM behavior remains token-exact.