Multi-modal: image/multi-modal embeddings + cross-modal retrieval (text → image)

[dcfocus]

Summary

Support embedding non-text payloads (images / audio) via a pluggable multi-modal embedder, auto-embed them on write, and enable cross-modal retrieval — e.g. a text query returning relevant images via a shared (CLIP-style) embedding space. No models bundled; the embedder is user-supplied.

Motivation / current state

• Embedding providers are text-only: the protocol is embed_texts (python/python/lance_context/embeddings.py:5,21); the registry has only openai + sentence-transformers (embeddings.py:112-115).
• Auto-embed is gated on isinstance(payload, str) (python/python/lance_context/api.py:607-609,680,1173), so images / bytes get no embedding unless the caller supplies a vector (python/tests/test_embeddings.py:192-197).
• Retrieval's text channel scores only text_payload (crates/lance-context-core/src/store.rs:1390,2638-2658), so binary-only records can't match the text channel; string queries are embedded with the text provider only (api.py:1202-1205). There is no shared text/image space and no text→image path.

So today an image record is dead weight for retrieval unless the caller computes and supplies its embedding externally, and even then you cannot query it with text.

Why here

Embeddings + retrieval are already core competencies (vector search, hybrid retrieve, pluggable provider registry from #87). Extending the provider interface to multi-modal and querying a shared space is additive.

Proposed work

• A pluggable multi-modal embedding interface (e.g. embed_images(...), or a CLIP-style shared text+image embedder), registered like existing providers — no bundled models (user supplies the model/hook).
• An auto-embed path for image / bytes payloads on add / upsert / batch.
• Cross-modal retrieval: text query → image results via the shared embedding space; ensure binary-only records are retrievable.
• Keep existing text-only retrieval behavior unchanged.

Acceptance criteria

• Register an image / multi-modal embedder and add image records that get embedded.
• Query by text and retrieve relevant image records via the shared space.
• Binary-only records are retrievable; existing text search / retrieve behavior is unchanged.
• Tests with a deterministic stub multi-modal provider covering add + auto-embed + cross-modal query.

Non-goals

• No bundled CLIP/model weights — the embedder is user-supplied (consistent with the existing provider registry).
• ANN index / brute-force-scan scaling is a separate concern (retrieval is currently a full scan, store.rs:1307-1319).

Relationship

The "make it searchable" half of multi-modal. Builds on external media references (#115) and lazy/projected reads (#116).