feat(media): OpenAI media adapter — images, speech, embeddings (M3) - #21
Merged
Merged
Conversation
…ainer) Implement FriendliProvider covering all three FriendliAI inference surfaces from one [providers.friendli] section, selected with endpoint_type: "serverless" (Model APIs, the hosted pay-per-token catalog), "dedicated" (the model field is the endpoint ID, or ID:ADAPTER_ROUTE for Multi-LoRA), and "container" (self-hosted Friendli Engine; base_url required, API key optional). The chat endpoint is OpenAI-compatible, plus the Friendli extensions: reasoning controls (reasoning_effort incl. the Friendli-only "ultracode" tier, reasoning_budget, parse_reasoning, include_reasoning), the chat-template switches enable_thinking / clear_thinking folded into chat_template_kwargs, Friendli Engine sampling (top_k, min_p, min_tokens, repetition_penalty, eos_token, XTC), regex-constrained structured output, cache-aware usage, and the exact /tokenize endpoint. Mutually exclusive body fields (tools vs min_tokens/response_format) are dropped with a warning instead of 422-ing. Transport is selectable via `backend` and auto-resolves openai -> httpx -> sdk. The vendor `friendli` SDK is supported but ranked last on purpose: its generated response models ignore unknown fields, so reasoning_content and reasoning are silently dropped, and it offers no extra_body escape hatch. The provider warns at startup when backend="sdk" meets parse_reasoning, and filters kwargs the SDK cannot type rather than surfacing a TypeError from inside the vendor package. Beyond chat: rich catalog discovery (context, pricing, modalities, reasoning options) cached and primed by warm_up(), tokenize/detokenize/ render_chat, text_completion, transcribe_audio, and — gated to dedicated/container — create_embeddings and generate_image. get_team_cost() and get_team_usage() read the Friendli Suite billing APIs for the configured team, which is also sent as X-Friendli-Team on every request. Token counting stays local (tiktoken) by default: llmcore counts tokens every turn and Model APIs rate limits are tier-based, so exact /tokenize counts are opt-in via native_token_count. Register the provider in ProviderManager with friendliai / friendli_ai aliases, add the llmcore[friendli] extra, the [providers.friendli] config section, and the provider_friendli confy schema section. Validated live against api.friendli.ai: catalog discovery, exact tokenization, chat on all three backends, SSE streaming with reasoning deltas, tool-call round trip, team cost/usage, and 401/429 mapping. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Friendli's GET /serverless/v1/models is a rich catalog rather than the minimal OpenAI /models shape: context length, max completion tokens, per-token pricing (input/output/cache-read/cache-write/audio-minute), a functionality capability block, input/output modalities, reasoning support with the available reasoning_options, the canonical models.dev base_model, the serving mode, and the deprecation date. FriendliAdapter derives nearly every card field from that live data, so the friendli.toml enrichment overlay only carries what the API cannot know: architecture family/type for the open-weight checkpoints Friendli hosts, short display names, and aliases. Pricing is deliberately not pinned in the overlay — the live catalog is authoritative and Friendli adjusts rates. Only the hosted Model APIs catalog is discoverable; Dedicated Endpoints and Container serve a single deployment each and expose no listing endpoint. The adapter also accepts every documented key spelling (FRIENDLI_TOKEN, FRIENDLIAI_API_KEY, FRIENDLI_API_KEY), matching the provider. Registered as "friendli" with a "friendliai" alias. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven cards generated from the live Friendli Model APIs catalog with `python -m tools.cardctl generate friendli` (2026-09-20): zai-org/GLM-5.3 1048576 ctx $1.26 / $3.96 per 1M zai-org/GLM-5.3-Flash 1048576 ctx $0.15 / $0.50 (text+image+video) zai-org/GLM-5.2 1048576 ctx $1.40 / $4.40 zai-org/GLM-5.1 202752 ctx $1.40 / $4.40 google/gemma-4-31B-it 262144 ctx $0.14 / $0.40 (text+image) deepseek-ai/DeepSeek-V3.2 163840 ctx $0.50 / $1.50 MiniMaxAI/MiniMax-M2.5 196608 ctx $0.30 / $1.20 Context, pricing, capabilities, modalities and per-model reasoning options come straight from the API. `cardctl diff friendli` reports no differences and all seven validate. Data only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ContextLengthError takes (model_name, limit, actual, message), but the
OpenAI, DeepSeek and Z.ai providers each constructed it with a keyword set
it has never accepted (provider_name / model / max_tokens /
requested_tokens). The raise statement therefore blew up inside __init__:
TypeError: ContextLengthError.__init__() got an unexpected keyword
argument 'provider_name'
Every context-overflow response produced an opaque TypeError carrying no
model and no limit, and nothing catching ContextLengthError — including
llmcore's own context-management and agent retry paths — ever saw it.
Fixing OpenAIProvider also fixes its subclasses (DeepInfra, vLLM, Poe,
OpenRouter). Anthropic, Mistral, Gemini, Kimi and Friendli already used the
documented signature. actual=0 is the faithful translation of the
requested_tokens=None all three were passing.
The defect survived because no test touched those branches, so add
tests/providers/test_context_length_error_mapping.py with two independent
guards:
- A static AST check over src/llmcore asserting that every
ContextLengthError(...) call site uses keywords the constructor accepts.
It is import-free, so it covers providers with no error-path tests and
any added later — this is the guard that would have caught the bug.
- Behavioural tests driving the real chat_completion() failure path of each
fixed provider, asserting the mapped exception carries the model name and
the model's context limit, plus negative cases (a plain 400, a 401) that
must not become ContextLengthError.
Both guards were verified to fail against the pre-fix code.
The behavioural tests skip with an explicit reason when
tests/providers/test_openai_provider.py has already replaced the openai
package in sys.modules with MagicMock placeholders — in that case the
provider binds a mock exception class no except clause can match. The
static check still runs unconditionally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add docs/Friendli_provider_usage.md covering the three endpoint types, the transport backends (including the measured reason the vendor SDK is not the default — its response models drop reasoning_content), reasoning controls, tool calling and regex structured output, multimodal input, catalog and cardctl workflow, the auxiliary endpoints, token-counting trade-offs, and error/rate-limit mapping. Record the Friendli-only "ultracode" reasoning tier in docs/model_cards.md alongside the canonical vocabulary, add Friendli to the per-provider wire mapping table, and note that it has no "none" tier (reasoning is turned off through chat_template_kwargs.enable_thinking instead). Also note two verified Friendli behaviours callers will hit: /detokenize and /chat/render are documented but currently 404 on Model APIs (they work on Dedicated Endpoints and Container), and tier-0 rate limits are adaptive and in practice allow only a couple of requests per minute — which is why native token counting is opt-in and the example paces its calls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Audit every curated provider against its upstream SDK/API and record the result in two tracking documents. docs/PROVIDER_SUPPORT_MATRIX.md is the ongoing tracker: per provider, the vendor SDK clone in /av/avalon/xrepos with its tag, commit and date, our pyproject pin, the installed version, the transport shape, and a capability matrix (chat/stream/tools/structured/reasoning/vision/audio/image/video/ embeddings/OCR/search/tokenizer) extracted from the provider classes rather than assumed. Section 6 is a runnable refresh procedure so the document can be regenerated per release. docs/PROVIDER_MODERNIZATION_PLAN.md turns the gaps into a phased program, starting from the dual-transport and one-contract principles. Findings that drove the plan: - openai (2.31 pin vs 3.22.1), anthropic (0.94 vs 1.9.0) and google-genai (1.72 vs 2.25.0) are each a MAJOR version behind. - openai 3.x and anthropic 1.x moved to httpx2 and no longer install httpx. Verified we never hand httpx objects to those clients and that no respx test routes traffic through a vendor SDK, so the port is packaging-only — but six providers (mistral, kimi, poe, openrouter, vllm, huggingface) import httpx with no extra of their own and would fail at import. - httpx2 verifies against the OS trust store, not certifi: a deployment risk worth documenting. - Only 4 of 16 providers implement the full extractor contract; ollama and gemini surface reasoning under provider-specific names that callers cannot use polymorphically. - anthropic's thinking_budget_tokens config key is now rejected with a 400 on every current Claude model; adaptive thinking + output_config.effort is the current API. Model defaults are several generations stale (openai gpt-4o, anthropic claude-sonnet-4-6, ollama llama3). - Gemini's media surface (Imagen, Veo, native TTS, Live API, embeddings) is entirely unexposed; xai/groq/together now ship native SDKs we do not use; mistralai v3.0.0 sits unused while the provider is httpx-only. - OpenAI deprecated the Sora video APIs in 3.1 — recorded so we don't add them. Vendor SDK clones under /av/avalon/xrepos were fast-forwarded as part of this audit, and xai-sdk-python, groq-python and together-python were cloned (they were missing). No llmcore code changes yet — the phases are the follow-up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 0 of docs/PROVIDER_MODERNIZATION_PLAN.md: unblock the SDK upgrade so the later capability phases have something current to build on. BREAKING: minimum SDK versions move across a major boundary. openai >=2.31.0 -> >=3.0.0,<4 anthropic >=0.94.0 -> >=1,<2 google-genai >=1.72.0 -> >=2,<3 ollama >=0.6.0 -> >=0.6.3 deepgram-sdk >=7.0.0 -> >=7.11.0 zai-sdk >=0.2.0 -> >=0.2.3 The three majors land together because openai 3.x and anthropic 1.x share one breaking change: their HTTP layer moved from httpx to httpx2 (Pydantic's maintained fork), which is now installed in place of httpx and certifi. Two consequences, both handled here: 1. Six providers (mistral, kimi, poe, openrouter, vllm, huggingface) import httpx but had no extra of their own — they worked only because openai installed httpx transitively. Under openai>=3 they fail at import. Each now has an extra declaring what it actually needs, and all six are in [all]. 2. httpx2 verifies TLS against the OS trust store rather than certifi, which can break minimal containers and TLS-inspecting proxies. Documented in CONFIG_REFERENCE.md with the SSL_CERT_FILE / SSL_CERT_DIR escape hatches. No provider code needed porting: llmcore only ever passes numeric timeouts to the vendor clients (never httpx objects), and no respx test routes traffic through a vendor SDK. Both were verified before bumping rather than assumed. Also fixes a latent test-isolation bug this upgrade exposed. Installing zai-sdk flipped ZaiProvider's backend auto-resolution from "openai" to "sdk", bypassing the AsyncOpenAI mocks in 21 tests — the exact hazard the old CI comment described when it deliberately left zai-sdk uninstalled. The tests now pin `backend` explicitly and patch the availability flags for resolution assertions, so they no longer depend on what happens to be installed. That removes the carve-out, and Z.ai's preferred SDK transport is exercised in CI and validated live for the first time. CI now installs .[dev,all] rather than a hand-maintained extras subset, so a new extra is covered the moment it is added to pyproject. The friendli pin stays at >=0.15.1: the vendor repo's pyproject reads 0.15.2 but that version is not published on PyPI. Verified: all 17 provider modules import; full unit suite green (5164 passed, 29 skipped); live calls through OpenAI 3.22.1, Google Gemini 2.25.0 (47 models discovered), Z.ai on the native SDK backend, and DeepSeek. NOT verified live: anthropic 1.9.0 — import- and test-clean, but no ANTHROPIC_API_KEY is available in this environment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two design+specification documents for the capability programs that add subsystems rather than extending providers. Specification only — no code. docs/MEDIA_SUBSYSTEM_SPEC.md turns the provider survey in /av/data/repos/docs/llmcore/researches into an llmcore-side design: a first-class llmcore.media subsystem with MediaArtifact / MediaUsage / MediaJob, capability Protocols per modality, and — the core distinction — three execution classes rather than one, because image generation is request/response, TTS is a byte stream and video is a long-running job. Capability metadata lives in model cards (extended with media/sourcing/policy blocks) instead of provider-specific tables in code; aggregators get the four-way split between who we call, whose weights, whose licence and whose AUP, and "uncensored" is represented honestly as supports_custom_weights plus provider_policy_applies rather than a boolean no vendor actually offers. It also records what the research could not see: - OpenAI's Sora video APIs were DEPRECATED in openai 3.1.0 (confirmed in the vendor CHANGELOG), so the survey's P0 "add Sora" item is dropped; frontier video comes from Veo and fal-hosted models. - llmcore already returns SpeechResult / ImageGenerationResult / OCRResult from seven providers, so models_multimodal types become views over MediaArtifact rather than being replaced. - Deepgram's surface is 12 public methods including a bidirectional voice agent, so the "refactor behind protocols" step is bigger than it looks — and it is the right first migration precisely because it exercises batch, realtime WebSocket and voice agent. - fal's own env var is FAL_KEY while the configured key is FAL_API_KEY; accept both, as the Friendli provider does for its three spellings. - Artifacts carry expires_at and checksum_sha256 from day one: every aggregator returns short-lived URLs, so storing a URI instead of bytes yields dead links. - Long media jobs get idempotency keys so a retried submit cannot double-bill. docs/COLAB_RUNTIME_SPEC.md designs llmcore.runtimes, a remote-compute abstraction with Colab as the first backend, after studying agent-lens's implemented design (391-line spec + ~3,750 lines across 13 modules) and the official google-colab-cli. The factoring argument: agent-lens already ends its bootstrap by registering the endpoint as an llmcore provider, which means the capability is being built on top of llmcore by a consumer and every other consumer must rebuild it. llmcore should own provisioning/bootstrap/tunnel/ lifecycle; agent-lens keeps its CLI and heuristics and deletes the duplication. Because the endpoint vLLM exposes is OpenAI-compatible, no new provider class is required — only dynamic instance registration in ProviderManager, which is also the one capability the media program needs. The safety model is the part that differs from every other provider: a Colab runtime bills per minute from assignment, not per request. Hence explicit-action-only provisioning (LLMCore.create() must never boot a VM), fail-closed bootstrap that releases the VM on any error, orphan detection so an unmonitored VM is visible, and a new max_lifetime_minutes hard cap on top of the reference idle reaper — an idle reaper does not protect against a runtime that is busy in a loop. Also records this session's live validation in the support matrix, including the two results that are not green: Anthropic 1.9.0 authenticates and maps errors correctly through the new major but every request returns "credit balance is too low", so no completion was validated; and Mistral's refreshed key works (46 models, open-mistral-nemo verified) but the configured default mistral-large-latest returns 403 — not in the account's tier. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase M1 of docs/MEDIA_SUBSYSTEM_SPEC.md: the llmcore.media subsystem, reached through llm.media, plus the one ProviderManager capability that both the media and remote-runtime programs need. No vendor adapters — that is the gate the spec requires before any provider work lands, so that each implementation does not establish its own incompatible conventions. The central design choice is three execution classes rather than one. MediaResult covers request/response (image generation, batch ASR), AsyncIterator[bytes] covers byte streams (TTS, realtime ASR), and MediaJob covers long-running work (video, queue-based vendors). Deepgram's existing 12-method provider-private surface is what happens without that distinction. Core pieces: - models.py — MediaKind, 19 MediaCapability values, MediaExecution, MediaJobStatus, MediaRef, MediaArtifact, MediaProvenance, MediaUsage, MediaResult, MediaJob. MediaRef accepts url/path/bytes/artifact so callers never hand-roll base64. MediaArtifact carries expires_at and checksum_sha256 because every aggregator returns short-lived URLs and a caller who stores the URI gets a dead link hours later. MediaUsage keeps the vendor's own billing units (images, megapixels, seconds, audio minutes, characters, compute seconds) with a stamped estimate, rather than synthesizing a token count for a video. - protocols.py — runtime-checkable Protocols per capability group, plus a CAPABILITY_PROTOCOLS table so adding a capability is one entry rather than an edit to routing code. A capability an adapter declares but does not back is dropped with a warning: better a missing capability than a confident AttributeError at call time. - manager.py — MediaManager with per-modality routers, capability discovery, and the documented resolution order (explicit provider, explicit model, [media.routing], built-in defaults, any capable adapter). Adapters ARE the chat providers: any [providers.*] instance implementing the protocols becomes one, so there is one credential per vendor and no parallel [media.providers.*] tree to keep in sync. - jobs.py — MediaJobManager owns polling, capped backoff with jitter, timeouts and cancellation so no adapter writes its own loop. A timeout raises WITHOUT cancelling the job: the handle stays valid and can be waited on again, because an expensive generation must not be thrown away over a client-side deadline. That is also why the deadline is an explicit parameter rather than asyncio.timeout, which would cancel the task. - artifacts.py — content-addressed store, sharded by SHA-256, atomic publish via a .part rename, with always/on_expiry/never policies. The byte fetcher is injected, so the store carries no hard httpx dependency. - testing.py — FakeMediaProvider implements every protocol and ships inside the package, so downstream projects building adapters can use it too. All 134 new tests run against it: no network, no vendor account. ProviderManager gains register_instance() / unregister_instance() with ephemeral tracking. Providers were previously only constructible during __init__, but a subsystem that creates an endpoint has to add one afterwards — a booted Colab VM's OpenAI-compatible endpoint registers as a vllm instance (docs/COLAB_RUNTIME_SPEC.md §3.2). Guards: a name collision raises unless replace=True so a live provider is never silently swapped out from under its callers; the configured default cannot be unregistered; construction failures surface as ConfigError; a failing close() is logged rather than blocking teardown. Also fixes a real bug the new tests caught: the polling backoff computed 2 ** attempt, which stops converting to float past ~1024 polls and killed the wait loop with OverflowError. A multi-hour video job polled every few seconds reaches that. The exponent is now capped and regression tested at 10,000 polls. Backward compatible throughout: BaseProvider's five media methods and the models_multimodal result types are untouched, so the seven providers using them keep working. Until M2 migrates them, llm.media.adapter_names is empty and says so honestly rather than pretending to route. Verified: 134 new tests; full unit suite 5298 passed, 29 skipped; ruff CI gate clean; llm.media exercised end-to-end through LLMCore.create(). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase M2 of docs/MEDIA_SUBSYSTEM_SPEC.md. Deepgram becomes the first real media adapter, and the legacy multimodal result types gain a bridge to MediaArtifact. Deepgram is the reference migration on purpose: it is the only integration that already exercises batch STT, realtime WebSocket STT and a bidirectional voice agent, so it stress-tests the parts of the abstraction most likely to be wrong before any new vendor commits to them. Its twelve provider-specific methods were the evidence that the chat facade had nowhere to put realtime audio. DeepgramProvider now implements MediaCapableProvider, ASRProvider, TTSProvider, StreamingTTSProvider and StreamingASRProvider. It declares exactly the five capabilities it can serve and reports no async-job execution class, because it has none. The new methods delegate rather than duplicate: transcribe_media, synthesize_speech_media, stream_speech_media and open_transcription_session translate MediaRef in and MediaArtifact out, then call the existing implementations. One code path per operation, so the two surfaces cannot drift. Details worth noting: - A remote MediaRef is handed to Deepgram's own transcribe_url path rather than downloaded locally and re-uploaded. - `timestamps` maps onto Deepgram's `utterances`, which is what actually produces per-segment timings. - `voice` folds into `model` because Deepgram encodes the voice in the model id; an explicit model wins. - Options the caller omits are NOT forwarded, so they cannot override the provider's configured [providers.deepgram.*] defaults. All twelve provider-specific methods are untouched and still return the legacy types; the three existing Deepgram suites pass unchanged. Also adds the models_multimodal <-> MediaArtifact bridge (spec §4.3). SpeechResult, TranscriptionResult, OCRResult, GeneratedImage and ImageGenerationResult gain to_artifact()/to_artifacts(), and the two round-trippable ones gain from_artifact(). These types are public API returned by seven providers, so they are bridged rather than replaced. The conversions preserve what matters — audio format to MIME type, diarization segments and timings, revised_prompt, OCR page structure — and decode base64 image payloads to real bytes, since the media layer deals in bytes. Malformed base64 degrades to the URI path instead of raising. Verified live against the Deepgram API through the media routers: TTS produced 146 KB of WAV with character-based usage, that artifact was fed straight back in as an ASR input via MediaRef.from_artifact() and transcribed correctly, and streaming TTS yielded 39 chunks. Offline: 74 new tests; media + all three Deepgram suites 262 passed; full unit suite 5372 passed, 29 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase M3 of docs/MEDIA_SUBSYSTEM_SPEC.md. OpenAI becomes a media adapter for image_generate, image_edit, tts, tts_stream and asr, and gains provider-level embeddings. Image generation, TTS and ASR delegate to the existing provider methods. Two things are new: image editing (POST /v1/images/edits, including mask support and remote-ref upload) and streaming TTS via the SDK's streaming response, so the first bytes arrive before synthesis completes. create_embeddings() closes a long-standing gap recorded in the support matrix: OpenAI embeddings were reachable only through the separate [embedding.openai] subsystem, so a caller holding a provider could not embed with it. Sora is deliberately absent. openai 3.1 deprecated the video APIs, so video_generate is not declared and a test asserts it stays that way — frontier video comes from Veo (M4) and fal (M5). Parameters with no OpenAI equivalent (seed, negative_prompt, sample_rate_hz) are dropped with a debug log rather than forwarded, where forwarding would 400. Supplying reference_images routes to the edit endpoint, which is how OpenAI expresses reference-conditioned generation. --- The subclassing hazard this phase surfaced --- DeepInfra, vLLM, Poe and OpenRouter all extend OpenAIProvider, so they inherit the media protocol METHODS without inheriting the endpoints behind them. Left alone, the router would confidently call /v1/images on a vLLM server. Each subclass now declares its own _MEDIA_CAPABILITIES (DeepInfra: image/TTS/ASR; the other three: none), and a test walks the subclass tree asserting every one declares explicitly — so a future subclass cannot silently inherit. MediaManager.from_provider_manager() also now skips providers that implement the protocols but declare no capabilities, so adapter_names keeps meaning "can actually do something" rather than listing providers that route nothing. The M1 test that asserted no shipped provider implements the protocols was updated to the new truth rather than the assertion being relaxed. --- A real bug live validation caught --- transcribe_audio() labelled every raw-bytes upload "audio.wav". OpenAI infers the container format from the upload filename, so mp3 bytes were rejected with "This model does not support the format you provided". This surfaced the first time a TTS artifact was fed straight back in as an ASR input — exactly the chaining the media subsystem makes natural, and something no unit test with a mocked client would have caught. transcribe_audio() gained an optional `filename` parameter defaulting to the previous "audio.wav", so existing callers are unaffected, and the media adapter derives the correct name from the MediaRef's mime type, filename or URL. Regression tested both at the helper and through the chaining path. Verified live: TTS produced 55 KB of mp3, the artifact round-tripped through ASR and transcribed correctly, streaming TTS yielded 4 chunks, and embeddings returned 256-dimension vectors. Offline: 42 new tests; media suite 254 passed; full unit suite 5418 passed, 29 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase M3 of
docs/MEDIA_SUBSYSTEM_SPEC.md. OpenAI becomes a media adapter forimage_generate,image_edit,tts,tts_streamandasr, and gains provider-level embeddings.What's new vs. delegated
Image generation, TTS and ASR delegate to the existing provider methods. Two things are genuinely new:
POST /v1/images/edits) with mask support; a remoteMediaRefis fetched first, because the endpoint takes an upload rather than a URL.Plus
create_embeddings()on the provider — a gap recorded in the support matrix. OpenAI embeddings were reachable only through the separate[embedding.openai]subsystem, so a caller holding a provider couldn't embed with it.Sora is deliberately absent.
openai3.1 deprecated the video APIs, sovideo_generateisn't declared and a test asserts it stays that way.Parameters with no OpenAI equivalent (
seed,negative_prompt,sample_rate_hz) are dropped with a debug log rather than forwarded — forwarding them would 400.reference_imagesroutes to the edit endpoint, which is how OpenAI expresses reference-conditioned generation.The subclassing hazard this phase surfaced
DeepInfra,vLLM,PoeandOpenRouterall extendOpenAIProvider, so they inherit the media protocol methods without inheriting the endpoints behind them. Left alone, the router would confidently call/v1/imageson a vLLM server.Each subclass now declares its own
_MEDIA_CAPABILITIES(DeepInfra: image/TTS/ASR; the other three: none), and a test walks the subclass tree asserting every one declares explicitly — so a future subclass can't silently inherit:MediaManager.from_provider_manager()also now skips providers that implement the protocols but declare nothing, soadapter_nameskeeps meaning "can actually do something". The M1 test that asserted no shipped provider implements the protocols was updated to the new truth rather than having its assertion relaxed.A real bug live validation caught
transcribe_audio()labelled every raw-bytes uploadaudio.wav. OpenAI infers the container format from the upload filename, so mp3 bytes were rejected:This surfaced the first time a TTS artifact was fed straight back in as an ASR input — exactly the chaining the media subsystem makes natural, and something no unit test with a mocked client would have caught. It's also a live bug for anyone passing non-wav bytes today.
transcribe_audio()gained an optionalfilenameparameter defaulting to the previous"audio.wav"(existing callers unaffected), and the adapter derives the right name from theMediaRef's mime type, filename or URL. Regression tested at the helper and through the chaining path.Live validation
42 new tests · media suite 254 passed · full unit suite 5418 passed, 29 skipped · ruff CI gate clean.
Next
M4: Google/Gemini media — Imagen images, Veo video (the first true async-job provider, which validates the
MediaJoblifecycle against a real vendor), and native TTS.🤖 Generated with Claude Code