feat: add TwelveLabs Marengo multimodal embedders (image/audio/video)

[mohit-twelvelabs]

Description

Follow-up to #3480 (the initial TwelveLabs integration). That PR shipped text embeddings
via Marengo (TwelveLabsTextEmbedder, TwelveLabsDocumentEmbedder) plus the Pegasus
TwelveLabsVideoConverter. But Marengo embeds text, images, audio, and video into one shared
512-dimensional space, and only the text path was exposed. This PR adds the image/audio/video
embedding capabilities so the shared space can actually be used for cross-modal retrieval.

Modeled on the Voyage integration's multimodal embedder, and kept consistent with the existing
Text/Document embedder pairing in this integration.

Components added

TwelveLabsMultimodalEmbedder

Embeds a single image, audio, or video (a local file path or a public URL) into Marengo's
shared space and returns {"embedding": [...], "meta": {...}}. This is the cross-modal query
companion to TwelveLabsTextEmbedder — e.g. embed a text query with the text embedder and search
a store of video/image embeddings, or vice-versa. Modality is inferred from the file
extension / MIME type and can be set explicitly via modality=.

from haystack_integrations.components.embedders.twelvelabs import TwelveLabsMultimodalEmbedder

embedder = TwelveLabsMultimodalEmbedder()  # reads TWELVELABS_API_KEY
result = embedder.run(source="https://example.com/cat.jpg")
print(len(result["embedding"]))

TwelveLabsDocumentMultimodalEmbedder

The indexing counterpart to TwelveLabsDocumentEmbedder (which embeds text content). It reads
each Document's media path from meta["file_path"] (configurable via file_path_meta_field /
root_path), embeds the media, and writes Document.embedding — following the same convention as
the existing CohereDocumentImageEmbedder / JinaDocumentImageEmbedder. Modality is inferred
per-Document (override with meta["modality"]).

from haystack import Document
from haystack_integrations.components.embedders.twelvelabs import TwelveLabsDocumentMultimodalEmbedder

docs = [Document(meta={"file_path": "clip.mp4"})]
docs = TwelveLabsDocumentMultimodalEmbedder().run(documents=docs)["documents"]

Implementation notes

• Images and audio are embedded synchronously via client.embed.create.
• Video uses the asynchronous TwelveLabs video-embedding task: the component submits the task,
  polls embed.tasks.status until it is ready, then retrieves the embedding (whole-video scope).
• Both components provide run and run_async.
• No new dependency — uses the twelvelabs SDK already declared by the integration.

Tests and quality

• 22 new unit tests (56 total pass); ruff check, ruff format --check, and mypy all clean.
• Gated integration tests (@pytest.mark.integration, skipped without TWELVELABS_API_KEY) cover
  a live image embedding and a live video-Document embedding.

Notes for reviewers

• Naming: went with TwelveLabsMultimodalEmbedder (query-side) + TwelveLabsDocumentMultimodalEmbedder
  (document-side) to mirror the existing Text / Document pairing and the repo's image-embedder
  convention. Happy to rename or split further if you'd prefer.
• Opening as a draft — feedback on scope, naming, and the media-input interface welcome before
  I finalize.

[github-actions]

Heads-up for maintainers

This PR is from a fork and touches integrations whose integration tests require API keys.
Those tests are skipped in CI because fork PRs don't have access to repo secrets for security reasons.

Affected integrations:

• twelvelabs

Please run the integration tests locally (hatch run test:integration inside each folder) before approving.

[github-actions]

Coverage report (twelvelabs)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
integrations/twelvelabs/src/haystack_integrations/components/embedders/twelvelabs
_media_embed.py					79, 105-106, 146-147, 150-151, 165-166, 183-185, 200-201, 206-207, 222-223
document_multimodal_embedder.py					88, 157-158
multimodal_embedder.py					59, 122-123
Project Total

This report was generated by python-coverage-comment-action

[julian-risch]

Hi @mohit-twelvelabs and thank you for extending the TwelveLabs integration!

I noticed an inconsistency between run and run_async that I would like you to have another look at and implement in a more consistent way:

• run_async resolves all sources and modalities up front before any embed_media_async call. Saves the user costs.
• run interleaves resolution with embedding inside the loop, so a bad modality/missing path on document only raises after other documents have already been embedded and billed.

My initial thinking is that run should also do sth like:

self._validate(documents)
key = self.api_key.resolve_value() or ""
sources = [self._resolve_source(d) for d in documents]
modalities = [self._resolve_modality(d, s) for d, s in zip(documents, sources, strict=True)]
# then loop embed_media(source, modality, ...) over the pre-resolved lists

However, I realized that TwelveLabsVideoConverter takes a different approach than other components in the integration. It only has run (no run_async at all) and each source is wrapped in try/except that logs a warning and continues, which means no fail-fast one bad source doesn't fail the whole batch.