Skip to content

feat: add Media Studio and shared media generation runtime - #16141

Draft
usnavy13 wants to merge 66 commits into
LibreChat-AI:devfrom
usnavy13:feat/media-studio
Draft

usnavy13 wants to merge 66 commits into
LibreChat-AI:devfrom
usnavy13:feat/media-studio

Conversation

@usnavy13

@usnavy13 usnavy13 commented Sep 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds Media Studio for creating and iterating on images and videos with persistent history and recoverable generation jobs. With media.enabled configured, users can generate in Studio, reuse results in chat, and open chat media in Studio. Compatible follow-ups reuse the previous result; history cards show the latest available output.

  • Shared provider connections, model capabilities, credentials, presets, and parameter validation across Studio and media tools.
  • Durable generation, cancellation, retry, provider diagnostics, usage accounting, and administrator recovery for uncertain provider outcomes.
  • Native Gemini image output in chat: images become ordinary Files before streaming, while private signed continuation data remains available for later turns.
  • Theme-aware controls, bounded dialogs, accessible history actions, preserved generation settings, and automatic plain-text titles.

Depends on LibreChat-AI/agents#553. Related proposal: discussion #16140.

How it works

The API host supplies configuration, persistence, storage, and provider transports. Studio records its intent before a paid request; a worker publishes ordered File outputs and reconciles interrupted work. Native chat uses the SDK port and the existing chat accounting path.

Studio / media tools
  validated request
    durable job and admission
      worker -> provider adapter -> provider
        Files + ordered outputs + usage settlement
          history / activity events / chat handoff

Native Google chat
  SDK NativeMediaPort
    authorize invocation -> persist File -> emit content
      private Message continuation metadata -> authorized replay

Database contracts and queries live in packages/data-schemas, runtime behavior in packages/api/src/media, shared configuration and DTOs in packages/data-provider/src/media, and Studio UI/state in client/src/components/Media. Legacy /api code wires the host into the application. Startup uses canonical user IDs, starts chat history alongside conversation metadata, and loads the Media activity snapshot before opening its event stream.

Type of change

  • Feature
  • Bug fix
  • Refactor
  • Performance improvement
  • Tests / tooling / CI

Testing

Current head: d7e616f49d8f3a334d47139c2cbda645758dfb56.

  • All 40 CI checks passed, including frontend/backend unit tests, typechecks, static checks, production image builds and startup, MongoDB/Redis integration, Lighthouse, and every browser suite.
  • Focused local regression suites cover Media persistence and scoped job materialization, SDK contracts, auth response redaction/identity, startup, message-cache races, navigation, active-stream recovery, and restored UI state. The final chat-loading change passed 116 tests in six existing suites; the installed SDK bridge passed 34 contract tests.
  • Changed-workspace typechecks, production builds, scoped lint/format checks, and commit hooks passed.
  • Focused browser tests passed upload/handoff and configured-footer cases, plus all three affected native quote-selection cases without retries.
  • Lighthouse with 250 ms injected per Mongo query passed locally and in Linux CI. CI median LCP is 2.93 s for chat and 3.24 s for restored Studio; CLS and TBT also passed. The 4.5 s LCP, 0.1 CLS, and 500 ms TBT limits are unchanged.
  • Chrome smoke testing against the six-service root stack built from this head passed login, history/reload, existing-thread restore, provider/model controls, advanced settings, and deletion-dialog sizing (448 px), cancellation, and focus restoration. No browser errors, Media writes, or paid generations occurred in the smoke test.
  • Two independent local reviewers found no remaining material findings at the exact pushed head; there are no inline PR review threads.

Manual browser testing also exercised configured Azure, Gemini, Vertex, and OpenRouter models and follow-up requests. CI browser flows use local fixtures without paid inference.

Risk / compatibility

  • Media is opt-in. Provider operations and follow-up semantics vary: supported Vertex Veo continuation extends a clip; OpenRouter video iteration remains limited by provider/reference support.
  • The persistence and accounting paths include owner/tenant fencing, retention, and reconciliation. An uncertain paid submission must not be silently submitted again. Native chat stays outside Studio job settlement.
  • Native continuation metadata is private; public events, exports, and shares strip it. Legacy stored native records retain a compatibility reader.
  • The branch temporarily pins and patches @librechat/agents 4.0.1, the release dev targets; the patch is generated from feat: add native media port for Google image output and host model-invocation tracing agents#553 rebased onto 4.0.1. It must be replaced by published SDK contracts before this PR is ready.

Draft readiness

  • Final cleanup reviewed and committed; meaningful regression coverage retained.
  • SDK dependency opened and linked.
  • SDK PR merged and a containing version published.
  • Consume that release and remove the temporary patch, pins, postinstall/build wiring, and patch-specific cache inputs; rerun the installed-package contract and affected checks.
  • Add screenshots to this PR and operator-facing documentation in the companion librechat.ai repository as part of final review.
  • CI and review pass again after SDK patch removal before marking ready.

usnavy13 and others added 30 commits September 16, 2026 16:39
… temporary creations

Gallery and controls
- Gallery cards are image-first square tiles with model and relative-time captions; status chips only appear for unfinished work; density is a Radio segmented control and phones keep two columns
- Studio header uses the chat header-action icon recipe with tooltips; thread rename/delete move into an ellipsis options menu; result actions collapse to one row with icon-only download, expand and cover; the composer upload trigger is a paperclip like chat
- Settings panel starts with a button that toggles between Generation history and New creation; document title follows the thread title

Generated titles
- media.titles config (enabled, endpoint, model, prompt, timeoutMs) falling back to endpoints.all title settings; packages/api/src/media/title.ts reuses chat provider resolution, sanitizeTitle and the PII title filter and records usage
- Titles are applied with a conditional write that never overwrites a user rename; skipped for temporary creations

Presets
- Server-side Studio presets under /api/media/presets backed by a MediaPreset collection, media.limits.maxPresets, default preset seeding new drafts; dialog to save, apply, star and delete

Model comparison
- A second model picker sends the same prompt and references to two models sharing a comparisonId; the thread renders those turns side by side under one prompt

Temporary creations
- Header toggle flags a new thread as temporary; the server stamps expiresAt from temporaryChatRetention, hides it from the library, and the media worker retires it on expiry

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ventions

- Eyebrow section labels (Label section variant) with quiet right-aligned actions replace stacked outline boxes
- The history/new-creation toggle becomes a header-row action beside the settings heading
- Presets are a labelled field that names the matching preset or reads Custom, with a Manage action for saving, defaults and deletion
- Image/Edit/Video is one segmented Radio control; the compare row is a label with an Add model action and an inline picker
- Advanced settings collapse under an eyebrow disclosure

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… Studio client

Backend
- Acquire deployment, integration and owner permits atomically so a job that fails its
  owner slot no longer parks in the queue holding deployment capacity for other users
- Fold assistant replay content to text unless a part carries signed native media, so
  non-Google providers keep string AIMessages next to tool_calls; cover all call sites
- Stop counting media debt as an account-deletion obligation; holds and pending
  settlements still block, and a debt-only balance can be deleted
- Retry re-prepares the execution so a rotated key or refreshed catalog dispatches
- Report queue-age expiry as queue_expired and leave provider cancellation without an
  error code; log router, worker and transport failures with a redacted cause
- Reject unsupported or oversized uploads before they reach disk and sweep the staging
  directory from the worker tick; stream hosted references through a digest instead of
  buffering them
- Assemble the media runtime once in createMediaRuntimeFromApp for both server entry
  points; resolve /api/config media permissions from the role; drop the app.locals reach
  in favour of the wiring layer; seed MEDIA permissions through the shared helper with
  ADMIN defaulting to use and create
- Document the media configuration section in librechat.example.yaml

Data schemas
- Rewrite $facet lookups, the pipeline retainer update and raw createIndexes calls so the
  DocumentDB compatibility guard passes; add the retirement sweep index
- Store native part expiry as a Date with a TTL index; pass settlement write concern per
  call; allowlist the deployment-wide permit table in the tenant-isolation guard

Client
- Stop receipt polling on a definitive 4xx and offer Dismiss for unresolved commands
- Route thread, job, command and upload calls through useMutation with the existing keys
- Gate Open in Studio once per chat view and mount the chat media host only when enabled
- Register session cleanups through a store registry instead of importing Media from auth
- Wire combobox labels and descriptions for assistive tech, use i18n locale for numbers
  and dates, restore the document title, remove the unused Composer prop and five unused
  translation keys, and add Studio route, handoff, cancel and session-cleanup coverage

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Reuse existing storage strategies, balances, transaction records and credentials. Add authenticated original and rendition delivery, durable Runway and Krea cancellation, and configuration checks before paid work.

Validate with focused API, persistence and UI tests, four workspace typechecks, production builds, browser workflows and the unchanged Lighthouse performance gate.
Unify credentials, admission, billing, retention, observability and Insights with existing LibreChat services. Add operator recovery, resilient client transitions, native replay ownership, SDK contract verification and regression coverage for the integration audit.
usnavy13 and others added 23 commits September 20, 2026 13:17
Start title generation alongside provider submission while retaining the job's
lease and accounting protections through cancellation and shutdown. Use the
shared uncapped title-model options and normalize replies to plain text.

Remove duplicated tests and unused aliases, simplify the media configuration
example, and include all media browser scenarios in the test shortcut.
Reuse the shared deletion dialog, share form selection, and simplify provider
key configuration. Batch thread file caching while preserving metadata and order.

Remove unused persistence exports and forwarding wrappers, and consolidate
duplicated tests while retaining failure, authorization, and replay coverage.
Preserve background receipt handling and both terminal-event projections.
Keep the temporary media bridge on SDK 3.8.8 so conversation-isolated
workspace routing remains intact.
Keeps the temporary @librechat/agents 3.9.0 pin: the native media patch
does not apply to 3.9.1, and nothing on dev consumes a 3.9.1-only export.
ChatRoute combines dev's route reconciliation with the branch's
parallel message load, and its new spec mocks the two hooks that adds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A substring lookup priced gemini-3.1-flash-image at the gemini-3.1 text
rate, and Gemini usage dropped thoughtsTokenCount, so image jobs settled
far below cost. Media pricing now accepts a pattern match only when it
names the whole model, and thinking tokens count as output.

In balance mode a provider-reported failure without a cost now releases
its hold instead of waiting in requires_attention for an operator.
Uncertain outcomes and cancellations stay unresolved.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…able

A Google model listed in a media integration defaulted to TEXT+IMAGE, and
start() threw when the chat surface was off, the user lacked MEDIA.CREATE
or media was disabled, so the whole chat turn failed. A default image
request that cannot be recorded now asks the provider for TEXT only; an
explicit IMAGE request still fails as before.

Native signature publishing also recovers after one failed metadata
write instead of rejecting every later publication.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every account deletion ran media index builds, prerelease migrations and
owner upserts, even on deployments that never enabled media. Deletion now
skips the media fence entirely until media has been activated.

The media completion step ran after the User row was deleted without its
own error handling, so a transient failure skipped the remaining cleanup
and returned an error for an account that no longer existed. It now logs
and leaves the owner to media account reconciliation, which already
completes prepared owners whose User is gone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- deleteMessages reported an already-completed deletion as failed when
  the immediate media consumer reconcile threw; the periodic sweep now
  owns the retry.
- Duplicate-key recovery in saveMessage returned another writer's
  mediaConsumerToken to the client.
- The TS ban check reads IP bans without the proxy port, but the writer
  kept storing them under req.ip, whose assignment is a no-op in Express.
- saveGeneratedImage rejected the raw base64 and generic data URLs tool
  artifacts could send before; native originals stay strict.
- GET /api/messages/:conversationId and /:messageId skipped the public
  media file projection that the other message reads apply.
- Azure and Firebase stream helpers dropped their failure log.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A file cached before the Files query loaded became the whole list,
because the query never refetches on mount. The seeded list is now
provisional, so the next observer loads every file and an in-flight
first load restarts after the file exists.

Saving or revoking a key waited for the media catalog to refetch from
every provider before resolving; the refresh now runs in the background
like the other invalidations.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…able

- An upload switched the draft between image operations (the derived
  edit context dropping, or the first image enabling edit), which
  changed the reference owner and aborted the upload it started. The
  owner now tracks the media kind, not the exact operation.
- A 202 rejected receipt stayed pending, so sending the same draft
  again replayed the rejection. A new send now supersedes it.
- Older turn pages were keyed by the detail cursor, so every new turn
  collapsed loaded history to one page. They now refetch in place.
- Open in Studio and Edit attachments could land on the gallery and
  hide the draft they had just seeded.
- A command error from one thread stayed under the next thread.
- The live event stream never reconnected after a clean server close,
  and event bursts kept cancelling the in-flight thread fetch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- The worker advanced its due-scope cursor before a page that could
  stop at maxActiveTotal, so owners after the break waited a full cycle.
- Unknown provider states read as running and polled forever outside
  balance mode; polling now shares recovery.attentionAfterMs.
- Hosted URL imports had no client abort and only a socket idle
  timeout, so a dripping server could hold memory and capacity slots.
  The whole read is now bounded by timeouts.downloadMs.
- Catalog discovery was cached per owner even for a shared credential.
  It is now keyed by tenant, binding and resolved headers, so users on
  one deployment key share discovery while user keys stay separate.
- MiniMax server errors, or an error that still names a task, no longer
  release a hold as a definite rejection.
- A key sent under a custom Authorization scheme (Sourceful) is now
  redacted from diagnostics.
- OpenRouter bring-your-own-key usage adds the upstream inference cost
  to OpenRouter's fee.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The six media method factories this branch adds sat beside the existing
methods/media directory as multi-word files. They now live inside it as
accounting, consumers, native, preset, recovery and title, with their
specs, and the package exports are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The e2e cleanup helper assumed prepareMediaAccountDeletion always opened
a media session; it now skips media reconciliation on runs that never
activated media and passes the required log. The storage read helper
imports its error metadata relatively, so /api specs that load it from
source resolve it without the packages/api path alias.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
usnavy13 and others added 2 commits September 24, 2026 13:55
Keeps the temporary @librechat/agents 3.9.0 pin over dev's ^3.9.3: the
native media patch targets 3.9.0, and nothing on dev consumes an export
added in 3.9.1-3.9.3. ChatView nests StudioProvider around dev's
QueuedTurnPortalProvider and takes dev's z-10 composer stacking fix;
dev's new stacking spec passes the branch's required messagesReady prop.
ControlCombobox keeps the forwarded trigger ref and option actions and
adds dev's popoverMaxHeight and unsearchedLimit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…t path schemes

Public share links passed Studio files through a blocklist that did not
cover the media* File fields, so anonymous viewers received rendition
storage keys, the owner's thread and conversation ids, and claim and
deletion tokens. The share sanitizer now drops media* fields, matching
what toPublicMessageFiles already strips on logged-in routes.

mediaAssetSchema accepted any filepath, and toAbsoluteFilePath passes
javascript: through to download links. Asset and rendition paths now
accept only same-origin paths and http(s), read with URL so whitespace,
tab and case variants are covered.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@usnavy13

Copy link
Copy Markdown
Collaborator Author

Security review of 5bcab1f32.

Checked: authorization and owner/tenant scoping on every media route and query, admin routes, SSRF on user-submitted and provider-returned URLs, storage paths and file serving, credential handling, balance holds and settlement, agent tool, chat and share paths, frontend sinks, and the SDK patch, lockfile and CI changes.

Found: nothing high-severity. Two low-severity issues:

  1. Public share links returned the media* fields of Studio files to anonymous viewers: rendition storage keys, the owner's thread and conversation ids, and claim and deletion tokens. None of it grants access.
  2. Studio asset paths weren't restricted by scheme, so a javascript: path could have reached a download link. The server builds every asset path, so nothing could reach this today.

Fixed in 168c72c:

  1. The share sanitizer now drops media* fields.
  2. mediaAssetSchema now accepts only same-origin and http(s) paths.

Both fixes have tests that fail on the old code.

usnavy13 and others added 3 commits October 3, 2026 21:52
Moves the native media bridge to @librechat/agents 4.0.1, the release dev
targets: both consumers pin it exactly and patches/@librechat+agents+4.0.1.patch
is regenerated from LibreChat-AI/agents#553 rebased onto 4.0.1, replacing the
3.9.0 patch.

Conflict resolutions carry Studio onto dev's reshaped code:
- "Create media" moves from the removed AttachFileMenu into the Attach and
  tools palette (useAttachItems), with the same chat/create gating.
- Native media stream handling follows dev's move into hooks/SSE/steps.
- Media file claims and native signatures wrap dev's single writeMessage path.
- Cookie authentication for share, image and media files stays one module
  (images/cookies.ts) and gains dev's two-factor enrollment and token
  retirement checks; dev's parallel auth/share.ts is dropped.
- Ban checks use dev's getBanIp without mutating req.ip.

Studio components now satisfy dev's design lint: spacing and color move off
Collapsible primitives onto plain wrappers, section actions use the
section-action Button variant, image frames size through a --media-ratio
custom property, and two shared variants cover needs other screens share
(Input leading-icon for search fields, Button composer for composer-row
controls; pressed header-action buttons fill through aria-pressed).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Keeps the native media factory beside dev's scheduled MCP execution options
in both createRun calls, passes routePending to ChatForm now that dev has
removed the composer tips setting, and adopts dev's split PixelCard frame
loop and DiffRow rendering. Reduced-motion PixelCard stills reset the shared
fill style cache, since they draw outside the animation frame that resets it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolves conflicts with dev's async local file saves, tool approval
grants, UI scale, settings-open atom, running-chat rename and route
re-nesting. Local storage keeps the branch's stream storage and unlink
helpers alongside writeLocalFile; ControlCombobox keeps its forwarded ref
and nested-dialog Escape handling and gains portalElement and rem-scaled
rows; the studio routes move into dev's re-nested Root children.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants