Skip to content

Derive attachment content type from MIME type#150

Open
Adam-D-Lewis wants to merge 3 commits into
mainfrom
fix/derive-attachment-mime-type
Open

Derive attachment content type from MIME type#150
Adam-D-Lewis wants to merge 3 commits into
mainfrom
fix/derive-attachment-mime-type

Conversation

@Adam-D-Lewis

Copy link
Copy Markdown
Member

Derive attachment content type from MIME type

Repo: nebari-chat-pack
Branch: fix/derive-attachment-mime-type
Base: main
Commit: 4efb6d2

Summary

Attachments in the chat input were always tagged with the AG-UI document
content type. This mislabels image/audio/video uploads, so a vision-capable
model never receives an uploaded image as an actual image. This PR derives the
AG-UI content discriminator from each file's MIME type instead.

Problem

In frontend/src/chat/chatinput.tsx, when attached files are converted to
FileInputContent, the discriminator was hardcoded:

return {
  type: "document",
  source: { type: "data", mimeType: file.type, value: ... },
  metadata: { name: file.name },
};

The downstream pipeline (Ravnar file handling → pydantic-ai AG-UI adapter) only
turns an ImageInputContent into an image part for the model. A document-typed
image is never converted to an image, so vision models can't see uploaded images
at all.

Change

Derive the discriminator from file.type:

  • image/*image
  • audio/*audio
  • video/*video
  • otherwise → document (safe fallback)

The valid discriminator literals (image / audio / video / document) were
confirmed against the @ag-ui/core FileInputContent discriminated union
(frontend/src/api/files.ts → the agui.*InputContentSchema definitions).

Files changed

  • frontend/src/chat/chatinput.tsx

Testing

  • tsc --noEmit passes.
  • Not yet exercised in a running browser.

Notes

  • This is a prerequisite for image upload actually working, but on its own it is
    not user-visible: the file picker's accept attribute still restricts which
    files can be selected. The companion change (feat/capability-driven-uploads)
    makes the picker capability-driven and is stacked on top of this branch.

Attachments were hardcoded to the "document" AG-UI content type, so
uploaded images (and audio/video) were mislabeled and never reached the
model as the correct modality. Derive the discriminator from the file's
MIME type instead, falling back to "document".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Adam-D-Lewis Adam-D-Lewis requested a review from jbouder June 18, 2026 15:14

@jbouder jbouder left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just need to resolve the conflict

Adam-D-Lewis and others added 2 commits June 18, 2026 18:58
Resolve chatinput.tsx: keep main's refactor (error handling, file
permission gating) and re-apply the MIME-derived attachment content type.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants