Skip to content

refactor(vlm): add owned host prefill preprocessing (#859)#881

Merged
inureyes merged 1 commit into
mainfrom
feature/issue-859-host-vlm-preprocessor
Jul 22, 2026
Merged

refactor(vlm): add owned host prefill preprocessing (#859)#881
inureyes merged 1 commit into
mainfrom
feature/issue-859-host-vlm-preprocessor

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

  • Add a backend-neutral owned prepared-prefill DTO with validated embeddings, positions, additive attention bias, logical token history, and modality metadata.
  • Add a LLaVA host preprocessor and canonical filtered loader that retain only the vision tower, projector, and text embedding weights while bypassing transformations not applied by IREE.
  • Share exact image-placeholder cardinality and overflow-safe expansion semantics with the existing MLX path, and export evaluated contiguous tensors through the fallible FFI boundary.
  • Provide a deterministic checkpoint-free fake plus focused parity, dtype, shape, capacity, and loader-filter tests.

Validation

  • cargo fmt --all -- --check
  • cargo check -p mlxcel-core --lib --tests
  • cargo check -p mlxcel --lib --tests
  • cargo test -q -p mlxcel-core multimodal::tests
  • cargo test -q -p mlxcel multimodal::host_preprocessor::tests --lib
  • cargo test -q -p mlxcel multimodal::vlm_prompt::tests --lib
  • cargo test -q -p mlxcel loading::vlm::llava::tests --lib
  • cargo test -q -p mlxcel vision::merge::tests --lib
  • cargo clippy -p mlxcel-core --lib --tests -- -D warnings
  • cargo clippy -p mlxcel --lib --tests -- -D warnings

Closes #859

Introduce a backend-neutral prepared-prefill DTO with owned tensor storage and structural validation.

Add a filtered LLaVA host preprocessor that reuses shared placeholder semantics while retaining only vision, projector, and embedding weights.

Harden fallible tensor export, cardinality/capacity checks, and deterministic fixtures for future XLA consumption.

Refs #859
@inureyes inureyes added status:review Under review type:refactor Code restructuring without changing functionality priority:high High priority area:models Model architectures, weights, loading, metadata area:architecture Architecture and code structure changes status:done Completed and removed status:review Under review labels Jul 22, 2026
@inureyes
inureyes merged commit 199fa40 into main Jul 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:architecture Architecture and code structure changes area:models Model architectures, weights, loading, metadata priority:high High priority status:done Completed type:refactor Code restructuring without changing functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor(vlm): extract a host multimodal preprocessor for XLA

1 participant