Skip to content

Create model capability catalog for multimodal model features #1147

Description

@paullizer

Summary

Create an initial maintained JSON catalog that maps model names/families to supported capabilities so SimpleChat can stop relying on regex-only model name checks for multimodal features.

User Value

Admins and developers get a clearer source of truth for which deployed models support vision, image generation, audio input/output, video input, binary/file input, and coding-oriented use cases. This will make new model support easier to add and can reduce the need to deploy separate speech or image services when a selected model already supports those capabilities.

Proposed Behavior

Add a data-only JSON catalog covering:

  • OpenAI models from GPT 5.0 onward.
  • Claude models released in the last two years.
  • Meta models.
  • xAI Grok models.
  • Microsoft Phi and Microsoft AI models.

No backend or frontend behavior should be changed in this first step; the catalog is only the initial source data for future integration.

Acceptance Criteria

  • A JSON catalog exists in the app with model/provider identifiers and capability flags.
  • Catalog entries include at least vision/image input, audio input, audio generation, image generation, video input, binary/file input, and coding capability fields.
  • Initial entries are populated for OpenAI GPT 5+, Claude recent models, Meta, xAI Grok, and Microsoft Phi/MAI model families.
  • JSON structure is valid and documented through clear field names.
  • No backend or frontend runtime behavior is changed in this initial PR.

Notes

  • Current vision capability detection is regex based in functions_model_capabilities.py and mirrored in admin_settings.js.
  • The catalog should be designed so those checks can be replaced in a follow-up change.
  • Priority: P2.
  • Assignee: paullizer.

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

Status
Pending Evaluation

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions