Skip to content

bug(copilot): model catalog includes unavailable and unconfigured models #4697

Description

@hgaol

Title

bug(copilot): model catalog includes unavailable and unconfigured models

Description

What happened

After connecting a GitHub Copilot account and refreshing its model catalog, Maka can show models that the account cannot actually use.

Two paths cause unavailable models to appear:

  1. GitHub historically returns model policy states including:

    enabled | disabled | unconfigured
    

    Maka currently filters only disabled. A model with policy.state = "unconfigured" therefore appears selectable even though Maka has no policy-acceptance flow and cannot use it.

  2. Interactive Copilot connections initially carry the provider fallback model IDs. After account discovery, generic model reconciliation preserves previously enabled IDs even when the account-scoped /models response omits them. Those bootstrap IDs are then added back to the displayed catalog.

The result is that the model picker can contain:

  • policy-unconfigured models;
  • fallback models unavailable to the account;
  • previously available models that the account no longer exposes.

Expected behavior:

  • A policy-free model remains available.
  • A model with policy.state = "enabled" remains available.
  • disabled, unconfigured, unknown, and malformed policy states are excluded.
  • Copilot's filtered /models response is treated as the account's authoritative usable catalog.
  • Initial fallback IDs not present in that catalog are removed.
  • Later refreshes remove withdrawn models without automatically opting the user into newly introduced models.

How to reproduce

  1. Connect a GitHub Copilot account.

  2. Refresh or inspect the connection's model catalog.

  3. Use a /models response containing a mixture such as:

    {
      "data": [
        {
          "id": "available",
          "model_picker_enabled": true,
          "policy": { "state": "enabled" }
        },
        {
          "id": "not-configured",
          "model_picker_enabled": true,
          "policy": { "state": "unconfigured" }
        }
      ]
    }
  4. Observe that not-configured is offered by Maka.

  5. If the connection was bootstrapped with fallback IDs, observe that omitted fallback IDs can also remain selectable after discovery.

A deterministic source-level reproduction exists in the GitHub Copilot provider-contract discovery fixture by adding an unconfigured model. Before the fix, that entry appears in the returned ModelInfo[].

Environment

  • Maka commit: 91624f1f3 or earlier affected build
  • Surface: Desktop / Runtime Host model discovery
  • Provider: GitHub Copilot
  • OS and version: [fill in]
  • Node.js version, if running from source: [fill in]

Logs, screenshots, or additional context

GitHub's Copilot client historically represented model policy as:

type ModelPolicy = {
  state: 'enabled' | 'disabled' | 'unconfigured';
  terms?: string;
};

A policy-free model is also valid because newer Copilot responses may omit the retired inline policy field.

Please redact access tokens and account information from any captured /models response.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions