feat(ai): add image input support to GLM

[andrewheard]

GeminiLanguageModel now declares the vision capability. Foundation Models prompts can include image attachments, which are sent to Gemini as inline data:

let response = try await session.respond {
  "What's in this image?"
  Attachment(image)
}

This is Phase 1 of image input support. It covers images created from a CGImage, CIImage, or CVPixelBuffer, and images from local files. Images from remote URLs (https, gs, …) currently throw an error; Phase 2 will forward them as Gemini fileData.

How images are sent

Image attachments in prompts are translated by the new GeminiImageTranslator:

Source	What's sent
File URL: PNG, JPEG, WebP, HEIC, or HEIF that needs no rotation	The original file bytes, with the matching MIME type
File URL: any other format, or a non-.up orientation	The image rotated to its display orientation and re-encoded as JPEG
CGImage, CIImage, or CVPixelBuffer	The image rotated to its display orientation and re-encoded as JPEG
Any non-file URL (https, gs, data, …)	LanguageModelError.unsupportedTranscriptContent (Phase 2)

Details

• Format detection: a file's format is detected from its contents, not its extension, so a PNG named photo.jpg is still sent as image/png.
• Orientation: Transcript.ImageAttachment.cgImage doesn't apply orientation, so the translator rotates or flips the pixels itself for all 8 EXIF orientations. An explicit orientation: overrides the orientation embedded in the file, matching ImageAttachment.
• JPEG encoding: quality 0.9 at full resolution, the same format Transcript's Codable conformance uses when it saves images.
• Transparency: composited onto black, again matching how Transcript saves images.
• Color spaces: standard RGB color spaces (for example, sRGB and Display P3) are preserved. Grayscale, extended-range, and HDR images are converted to sRGB, since they can't be represented in an 8-bit RGB JPEG.
• Files with several images (for example, HEIC): the first image is used, because ImageAttachment reads both its orientation and its pixels from the first image. Such a file is only sent unchanged if its primary image is also its first image; otherwise Gemini would decode a different image from the original bytes.
• Prompt layout: segment order is preserved, adjacent text segments are merged into one text part, and empty text is dropped around images. Text-only prompts are translated exactly as before.
• Errors instead of crashes: reading cgImage crashes for URL attachments that can't be loaded (for example, gs://) and for CIImages with an empty or infinite extent. The translator never reads cgImage for URL attachments, and it rejects those CIImages first. All translation failures surface as LanguageModelError.unsupportedTranscriptContent, with the offending prompt entry and a description. Unsupported URLs are described by their scheme only, so error messages don't include data: payloads or signed-URL tokens.

Current limitations

These are documented on GeminiLanguageModel.capabilities:

• Prompts only: image attachments in instructions, responses, and tool output still throw unsupportedTranscriptContent, and so do Transcript.Attachment.data attachments.
• No remote URLs yet: Phase 2 will forward them as Gemini fileData.
• Labels aren't sent to the model in this PR: tools that take an ImageReference only work if the prompt text mentions the label. The stacked follow-up PR fixes this.
• Request size: inline data counts toward the backend's request size limit (about 20 MB), and earlier images in the session are re-sent with every request.
• Files must stay available: image files are read again for every request, so a file must remain available for as long as the session that references it is used.

Testing

Unit tests (GeminiImageTranslatorTests, GeminiTranscriptTranslatorTests, GeminiLanguageModelTests), using images generated in code with no bundled resources:

• Orientation: all 8 orientations for CGImage sources, checked against a hand-derived table of quadrant colors and against ImageIO's own EXIF transform. Also covered for CVPixelBuffer and CIImage sources, and for JPEG and HEIC files with embedded orientation (including an explicit .up overriding it).
• Pass-through: PNG, JPEG, and HEIC files are sent byte for byte, with or without an explicit Orientation = 1 tag, and the format is detected from the contents. TIFF, GIF, and BMP are re-encoded.
• Color and transparency: Display P3 is preserved; grayscale, extended-range, and HDR images are converted to sRGB; transparent white pixels are composited onto black.
• Files with several images: HEIC files whose primary image isn't first, plus a test that fails if ImageAttachment ever stops using the first image.
• Errors: missing and non-image files, remote and data: URLs (including the scheme-only messages), and unrenderable CIImages.
• Prompt layout: order, text merging, empty text, and text-only prompts.
• End to end: an Attachment sent through LanguageModelSession reaches the request as inline JPEG data (mocked transport).
• Portability: fixture formats the platform can't encode (for example, HEIC on some simulators and VMs) are skipped instead of failing.

Integration tests (ImageUnderstandingIntegrationTests), on all three backends (Developer API, Firebase AI Logic Developer API, and Firebase AI Logic Agent Platform):

• one image, two images compared, and @Generable shape classification
• an orientation check, where an image attached with .down must report the colors in its display orientation
• PNG and HEIC file URLs
• streaming
• an image earlier in a multi-turn conversation

These use the default sampling mode, since many Gemini 3 models don't support greedy sampling. Prompts are tightly constrained so responses stay predictable.

Toolchains: unit tests pass on Xcode 27.2 beta 2 (Swift 6.4) and beta 3 (Swift 6.4.2). swift-format lint --strict is clean.

Usage examples

Describe an image

import FirebaseAILogic
import FoundationModels

func describe(_ image: CGImage) async throws -> String {
  let model = FirebaseAI.firebaseAI().geminiLanguageModel(name: "gemini-model-name")
  let session = LanguageModelSession(model: model)

  let response = try await session.respond {
    "Describe this image in one sentence."
    Attachment(image)
  }
  return response.content
}

Compare two images

Prompt segments keep their order, so text can introduce or follow each image.

func compare(_ before: CGImage, _ after: CGImage) async throws -> String {
  let model = FirebaseAI.firebaseAI().geminiLanguageModel(name: "gemini-model-name")
  let session = LanguageModelSession(model: model)

  let response = try await session.respond {
    "Compare these two photos of the same room using three bullet points."
    "Before:"
    Attachment(before)
    "After:"
    Attachment(after)
  }
  return response.content
}

Analyze a camera frame with its orientation

Camera frames are often stored sideways. Pass the display orientation, and the image is rotated before it's sent.

import CoreVideo
import FirebaseAILogic
import FoundationModels

func describeFrame(_ pixelBuffer: CVPixelBuffer) async throws -> String {
  let model = FirebaseAI.firebaseAI().geminiLanguageModel(name: "gemini-model-name")
  let session = LanguageModelSession(model: model)

  let response = try await session.respond {
    "What is the main object in this camera frame?"
    Attachment(pixelBuffer, orientation: .right)
  }
  return response.content
}

Send a photo from a file

Upright PNG, JPEG, WebP, HEIC, and HEIF files are sent unchanged. Photos with embedded rotation are rotated and re-encoded as JPEG. The file must stay available for as long as the session is used.

func summarizeDocument(at fileURL: URL) async throws -> String {
  let model = FirebaseAI.firebaseAI().geminiLanguageModel(name: "gemini-model-name")
  let session = LanguageModelSession(model: model)

  let response = try await session.respond {
    "Summarize the document shown in this photo."
    Attachment(imageURL: fileURL)
  }
  return response.content
}

Classify an image with guided generation

Wrap the enum in a @Generable struct; top-level enums currently fail to decode (see "Notes for reviewers").

@Generable
enum Animal: String {
  case cat, dog, bird, other
}

@Generable
struct Classification {
  @Guide(description: "The animal shown in the image.")
  var animal: Animal
}

func classify(_ image: CGImage) async throws -> Animal {
  let model = FirebaseAI.firebaseAI().geminiLanguageModel(name: "gemini-model-name")
  let session = LanguageModelSession(model: model)

  let response = try await session.respond(generating: Classification.self) {
    "Classify the animal in this image. Choose cat, dog, bird, or other."
    Attachment(image)
  }
  return response.content.animal
}

Ask follow-up questions about an image

Images stay in the session's history, so later turns can refer to them. Earlier images are re-sent with every request, which counts toward the request size limit.

let session = LanguageModelSession(model: model)

_ = try await session.respond {
  "Here is a photo of my fridge. Remember what's in it."
  Attachment(fridgePhoto)
}
let response = try await session.respond(
  to: "Suggest a dinner I can make with what's in the fridge."
)

[gemini-code-assist]

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

[gemini-code-assist]

Summary of Changes

This pull request introduces image input support for the GeminiLanguageModel by declaring the vision capability. It implements a robust translation layer that processes various image sources, ensuring they are correctly rotated and re-encoded as JPEG for the Gemini API. The changes also update the transcript translation logic to support image attachments within prompts and provide comprehensive test coverage to ensure reliability across different image formats and orientations.

Highlights

• Vision Capability: Added the vision capability to GeminiLanguageModel, enabling the model to process image attachments in prompts.
• Image Translation Layer: Implemented GeminiImageTranslator to handle rotation and JPEG re-encoding for CGImage, CVPixelBuffer, and file URL sources.
• Transcript Integration: Updated GeminiTranscriptTranslator to support image attachments within prompt segments while maintaining segment order.
• Comprehensive Testing: Added extensive unit and integration tests, including a new TestImages utility, to verify image handling, orientation, and format conversion.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Activity

• Pull request created by andrewheard.
• Gemini Code Assist bot provided usage instructions.
• Author requested a summary via /gemini summary.

[gemini-code-assist]

Code Review

This pull request adds image input support (the vision capability) to GeminiLanguageModel. It introduces GeminiImageTranslator to process image attachments from CGImage, CVPixelBuffer, or file URLs, handling orientation adjustments and JPEG re-encoding. The GeminiTranscriptTranslator is updated to extract image parts from prompt segments, and the model's capabilities are expanded to include .vision. Comprehensive unit and integration tests are added to validate these changes. As there are no review comments, I have no feedback to provide.