Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 12 additions & 14 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ src/router.mjs The local proxy: model injection, provider routing, pa
src/catalog.mjs Turns providers into the model entries Copilot expects from GET /models
src/launcher.mjs Starts the router on an ephemeral port and spawns Copilot against it
src/upstream.mjs Resolves the Copilot API tier (individual / business / enterprise)
src/discovery.mjs Asks each provider for its live model list; falls back to the catalog
src/discovery.mjs Asks each provider for its live model list; falls back to the remembered list
src/config.mjs Provider config loading and validation; loads providers.default.json
src/providers.default.json Generated catalog — never edit by hand, run npm run catalog:update
src/keystore.mjs Optional on-disk key store (0600 / owner-only ACL)
Expand All @@ -60,8 +60,8 @@ src/copilot-bin.mjs Resolves the Copilot executable, skipping the stale VS
src/model-ranking.mjs Pure ranking of provider model catalogs (legacy mode)
src/process-env.mjs Strips stale/secret variables from the child environment
src/provider-env.mjs Builds the COPILOT_PROVIDER_* variables (legacy mode)
scripts/update-catalog.mjs Rebuilds the catalog from models.dev
scripts/catalog-sources.json Curated provider/model selection feeding the generator
scripts/update-catalog.mjs Rebuilds the endpoint catalog (offline, no network)
scripts/catalog-sources.json Provider endpoints feeding the generator
extensions/byok/extension.mjs The /byok slash command, copied into ~/.copilot/extensions/
test/*.test.mjs One suite per src module
schemas/providers.schema.json Published JSON Schema for providers.json
Expand All @@ -80,7 +80,7 @@ All commands run from the repository root.
| Test | `npm test` |
| Test with coverage | `npm run test:coverage` |
| Full quality gate | `npm run check` (lint + test) |
| Verify curated models still exist | `npm run catalog:audit` |
| Verify providers serve usable models | `npm run catalog:audit` |
| Package contents check | `npm pack --dry-run` |
| Dependency audit | `npm audit --omit=dev --audit-level=high` |
| Local install for manual testing | `npm link` then `copilot-byok --help` |
Expand Down Expand Up @@ -132,20 +132,18 @@ This project handles third-party provider API keys. Every change must preserve t
The catalog is generated. **Never hand-edit `src/providers.default.json`** — it is overwritten.

1. Add the provider to `scripts/catalog-sources.json`, using only endpoints documented by the
provider, then run `npm run catalog:update`. List model ids under `models`; the generator fills
in context window and capabilities from models.dev and drops anything without tool-calling
support. Leave `models` empty when names are account-specific (Volcengine endpoint ids) or the
catalog is too large to curate (OpenRouter).
2. Run `npm run catalog:audit` after changing any curated list. A model that does
not exist on the provider is worse than a missing one: it is offered to the user
and fails only when picked.
provider, then run `npm run catalog:update`. List no models: the picker is filled
by live discovery from the provider's `GET /models`.
2. Run `npm run catalog:audit` after changing the catalog. A provider that answers
but yields no usable chat model fails the audit; providers without a configured
key are reported as unverifiable.
3. Verify the endpoint yourself before adding it: `POST {baseUrl}/chat/completions` with an invalid
key must answer `400`/`401`/`403`, not `404`. A `404` usually means the path is wrong.
3. For authenticated providers, add every credential environment name to `DEFAULT_SECRET_SOURCE_ENV`
4. For authenticated providers, add every credential environment name to `DEFAULT_SECRET_SOURCE_ENV`
in `src/process-env.mjs`. For intentionally authless providers set `authRequired: false` and do
not invent a credential environment name.
4. Add the row to the provider table in `README.md`.
5. Update `docs/provider-verification.md` honestly: endpoint reachability, authenticated catalog
5. Add the row to the provider table in `README.md`.
6. Update `docs/provider-verification.md` honestly: endpoint reachability, authenticated catalog
access, and end-to-end inference are three distinct evidence levels. Never claim a level that was
not actually verified.

Expand Down
34 changes: 34 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,40 @@ All notable changes to this project are documented in this file.

## [Unreleased]

### Added

- Meta Muse (`muse`) and NVIDIA NIM (`nim`) providers, both verified with
OpenAI-compatible chat and model-list endpoints.
- Image-input support is now read from each provider's live catalog, the same
way reasoning support already was.
- `--no-discovery` is now an offline mode: providers use their pinned models or
the list each key returned last time, without any network request.

### Changed

- The built-in catalog ships endpoints only; model lists always come from live
discovery. There are no shipped model lists to go stale.
- Model ranking no longer prefers any model family; order is recency, version,
context window, then name.
- `npm run catalog:update` is offline and instant: no more models.dev fetch.
- `npm run catalog:audit` now checks that every provider serves at least one
usable chat model instead of checking pinned model ids.
- `--list-providers` marks providers with a configured key as usable and shows
pinned or remembered model counts.

### Removed

- **Breaking:** `stepfun`, `doubao`, `hunyuan` and `qianfan` providers were
removed: the catalog keeps only coding-oriented frontier providers. Configs
referencing them need updating; account-specific endpoint ids still work
through a custom provider entry.

### Fixed

- The `/byok` in-session command lists every provider needing a credential
again instead of an empty list, and no longer reports model counts that only
exist after discovery.

## [1.4.2] - 2026-08-12

### Changed
Expand Down
83 changes: 40 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,18 +100,18 @@ Every provider below is reached through its OpenAI-compatible endpoint, verified
|---|---|---|---|
| OpenAI | `OPENAI_API_KEY` | DeepSeek | `DEEPSEEK_API_KEY` |
| Anthropic | `ANTHROPIC_API_KEY` | Alibaba Qwen | `DASHSCOPE_API_KEY` |
| Google Gemini | `GEMINI_API_KEY` | Z.AI GLM | `ZAI_API_KEY` |
| xAI Grok | `XAI_API_KEY` | Zhipu AI (CN) | `ZHIPU_CN_API_KEY` |
| Mistral | `MISTRAL_API_KEY` | Moonshot Kimi | `MOONSHOT_API_KEY` |
| Groq | `GROQ_API_KEY` | MiniMax | `MINIMAX_API_KEY` |
| Cerebras | `CEREBRAS_API_KEY` | StepFun | `STEPFUN_API_KEY` |
| Together AI | `TOGETHER_API_KEY` | ByteDance Doubao | `ARK_API_KEY` |
| DeepInfra | `DEEPINFRA_API_KEY` | Tencent Hunyuan | `HUNYUAN_API_KEY` |
| Fireworks AI | `FIREWORKS_API_KEY` | Baidu Qianfan | `QIANFAN_API_KEY` |
| OpenRouter | `OPENROUTER_API_KEY` | SiliconFlow | `SILICONFLOW_API_KEY` |
| Chutes | `CHUTES_API_KEY` | ModelScope | `MODELSCOPE_API_KEY` |
| OpenCode Zen | `OPENCODE_ZEN_API_KEY` | Alibaba Token Plan | `ALIBABA_TOKEN_PLAN_API_KEY` |
| OpenCode Go | `OPENCODE_GO_API_KEY` | Ollama / LM Studio | no key needed |
| Google Gemini | `GEMINI_API_KEY` | Alibaba Qwen (CN) | `DASHSCOPE_API_KEY_CN` |
| xAI Grok | `XAI_API_KEY` | Z.AI GLM | `ZAI_API_KEY` |
| Mistral | `MISTRAL_API_KEY` | Zhipu AI (CN) | `ZHIPU_CN_API_KEY` |
| Meta Muse | `MODEL_API_KEY` | Moonshot Kimi | `MOONSHOT_API_KEY` |
| NVIDIA NIM | `NVIDIA_API_KEY` | MiniMax | `MINIMAX_API_KEY` |
| Groq | `GROQ_API_KEY` | SiliconFlow | `SILICONFLOW_API_KEY` |
| Cerebras | `CEREBRAS_API_KEY` | ModelScope | `MODELSCOPE_API_KEY` |
| Together AI | `TOGETHER_API_KEY` | OpenRouter | `OPENROUTER_API_KEY` |
| DeepInfra | `DEEPINFRA_API_KEY` | Chutes | `CHUTES_API_KEY` |
| Fireworks AI | `FIREWORKS_API_KEY` | OpenCode Zen | `OPENCODE_ZEN_API_KEY` |
| Alibaba Token Plan | `ALIBABA_TOKEN_PLAN_API_KEY` | OpenCode Go | `OPENCODE_GO_API_KEY` |
| Ollama | no key needed | LM Studio | no key needed |

Each provider also accepts a `COPILOT_BYOK_<PROVIDER>_API_KEY` variant, useful when you already use the plain name for something else.

Expand All @@ -126,17 +126,17 @@ The store lives at `~/.config/copilot-byok/keys.json` (`%APPDATA%\copilot-byok\k

Either way the keys stay out of the Copilot process: the router uses them in its own, and strips them from the environment it hands to the agent.

Some entries ship without a preset model list, either because their model names are account-specific (Volcengine endpoint ids) or because the catalog is too large to curate (OpenRouter). Add the ones you want in your own config — see below.
No entry ships a model list: the picker is filled by live discovery. To pin exact models instead — for an offline setup or a fixed line-up — add them in your own config — see below.

### Where the model list comes from

**Providers are asked directly.** At startup each configured provider is queried
for the models it serves right now, so the picker shows today's line-up rather
than whatever was curated at release time — a model released this morning is
there this morning. All 30 providers were verified to expose an OpenAI-style
`GET /models`; the ones that also publish capabilities (Chutes, OpenRouter and
others) supply the real context window, output limit and tool-calling support
along with it.
than whatever was pinned at release time — a model released this morning is
there this morning. All 28 providers expose an OpenAI-style `GET /models`
(verified in `docs/provider-verification.md`); the ones that also publish
capabilities (Chutes, OpenRouter, NIM and others) supply the real context
window, output limit and tool-calling support along with it.

While this happens the startup shows what it is waiting on, since the pause comes
before Copilot has drawn anything:
Expand All @@ -153,31 +153,28 @@ Only the providers you have a key for are queried, in parallel, with a hard
timeout. Measured on three configured providers: about a second. Results are
cached for ten minutes, failures for one.

Anything the provider does not report is filled in from the shipped catalog,
which is generated from [models.dev](https://models.dev) and refreshed with:

```sh
npm run catalog:update
```
Anything the provider does not report falls back to conservative defaults (128k
context, 32k output); image input and the reasoning selector appear only when
the provider's own metadata declares them.

If a request fails — a rate limit, a blip, an expired key — the list that key
returned last time is used instead, kept in `~/.cache/copilot-byok/models.json`
(`%LOCALAPPDATA%` on Windows). That matters: the shipped list is generic, while a
provider's catalog belongs to the plan behind your key, so falling back to it can
offer models your plan does not include and hide ones it does. Only when nothing
was ever remembered does the shipped list step in, so a hiccup never empties the
picker. To skip discovery entirely and always use the shipped lists:
(`%LOCALAPPDATA%` on Windows). That matters: a provider's catalog belongs to the
plan behind your key, so only what your key actually returned is ever offered —
never a generic guess. When nothing was ever remembered, the provider stays out
of the picker and the log says why. To skip discovery entirely and work from
remembered lists:

```sh
copilot-byok --no-discovery
```

Models that cannot call tools are excluded, whether that is known from the
provider's own metadata or from the catalog: Copilot's harness trusts what a
model entry declares, and an agent that cannot call tools breaks halfway through
a session instead of failing cleanly. Embedding, image, audio and moderation
models are filtered out for the same reason. Each provider contributes at most
twelve models, so a catalog like OpenRouter's 400+ cannot swamp the picker.
Models that cannot call tools are excluded based on the provider's own metadata:
Copilot's harness trusts what a model entry declares, and an agent that cannot
call tools breaks halfway through a session instead of failing cleanly.
Embedding, image, audio and moderation models are filtered out for the same
reason. Each provider contributes at most twelve models, so a catalog like
OpenRouter's 460+ cannot swamp the picker.

## The /byok command

Expand All @@ -193,9 +190,9 @@ Then `/byok` opens a provider list and asks for the key:

```
Choose a provider to configure:
❯ Anthropic — not configured (4 models)
Google Gemini — not configured (3 models)
DeepSeek — not configured (4 models)
❯ Anthropic — not configured
Google Gemini — not configured
DeepSeek — not configured
OpenAI — set via environment
Chutes — set via environment

Expand Down Expand Up @@ -356,13 +353,13 @@ Everything has a working default; these are for when you need to change one.
### Keeping the catalog honest

```sh
npm run catalog:update # refresh models and limits from models.dev
npm run catalog:audit # check every curated model still exists on its provider
npm run catalog:update # regenerate the endpoint catalog from scripts/catalog-sources.json
npm run catalog:audit # check every provider serves a usable model list
```

The audit exits non-zero when a curated entry is not served by its provider.
Curated lists are only a fallback, but a wrong entry is worse than a missing one:
it is offered in the picker and fails only when selected.
The audit queries each provider's live catalog and exits non-zero when one that
answers yields no usable chat model. Providers that need a key you have not
configured are reported as unverifiable rather than failing.

### Single-provider mode

Expand Down
44 changes: 25 additions & 19 deletions docs/provider-verification.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Provider verification

Last verified: 2026-08-12 (model-list endpoints re-verified the same day)
Last verified: 2026-10-14 for endpoints and model lists; end-to-end runs below are from 2026-08-12.

This document separates three different levels of evidence, because they are not
interchangeable:
Expand All @@ -19,6 +19,8 @@ interchangeable:
- GitHub Copilot CLI 1.0.79, personal (free) plan
- Endpoint probes: `POST /chat/completions` with a deliberately invalid key and a
25-second timeout. No credentials were printed.
- Endpoint and model-list tables were re-probed on 2026-10-14 (Windows,
Node.js 24.21, invalid-key probes, no credentials).

## Endpoint reachability

Expand All @@ -45,15 +47,13 @@ All entries below answered on their OpenAI-compatible route.
| Zhipu AI (CN) | `https://open.bigmodel.cn/api/paas/v4` | 401 |
| Moonshot Kimi | `https://api.moonshot.ai/v1` | 401 |
| MiniMax | `https://api.minimax.io/v1` | 401 |
| StepFun | `https://api.stepfun.com/v1` | 401 |
| Meta Muse | `https://api.meta.ai/v1` | 401 |
| SiliconFlow | `https://api.siliconflow.com/v1` | 401 |
| ModelScope | `https://api-inference.modelscope.cn/v1` | 400 |
| ByteDance Doubao | `https://ark.cn-beijing.volces.com/api/v3` | 401 |
| Tencent Hunyuan | `https://api.hunyuan.cloud.tencent.com/v1` | 401 |
| Baidu Qianfan | `https://qianfan.baidubce.com/v2` | 401 |
| Baichuan | `https://api.baichuan-ai.com/v1` | 401 |
| Chutes | `https://llm.chutes.ai/v1` | 401 |
| OpenCode Zen | `https://opencode.ai/zen/v1` | 401 |
| OpenCode Go | `https://opencode.ai/zen/go/v1` | 400 |
| Baseten | `https://inference.baseten.co/v1` | 403 |

Two endpoints answered `404` to the probe because they validate the model name
Expand All @@ -67,40 +67,46 @@ before the credential, and the probe deliberately sends a non-existent model:
## Model-list endpoints

Discovery reads each provider's `GET /models`, so that route was probed for every
catalog entry with an invalid key. **None returned 404** — every provider exposes
it. Four serve it without authentication, which is how the model counts below
were observed:
catalog entry. Sending no credentials returns the public catalogs below;
sending an invalid key is rejected instead, so a 401 proves the route exists
but says nothing about catalog contents. Observed 2026-10-14 via
`npm run catalog:audit` with no keys configured:

| Provider | Result |
|---|---|
| ModelScope | 200 — 43 models |
| OpenCode Go | 200 — 25 models |
| OpenCode Zen | 200 — 61 models |
| OpenRouter | 200 — 408 models, 340 declaring tool support |
| Gemini, xAI | 400 (route exists, request rejected) |
| Qianfan | 403 |
| Chutes | 200 — 14 models, all usable |
| DeepInfra | 200 — 179 models, 137 usable after ranking |
| ModelScope | 200 — 36 models, 34 usable |
| Nvidia NIM | 200 — 80 models, 69 usable |
| OpenCode Go | 200 — 40 models, all usable |
| OpenCode Zen | 200 — 85 models, all usable |
| OpenRouter | 200 — 462 models, 447 usable |
| Gemini | 404 without a key (400 with an invalid one) |
| xAI | 400 (route exists, request rejected) |
| All others | 401 |
| Ollama, LM Studio | connection refused — local servers, not running here |

Some providers publish the metadata discovery needs alongside the list, which is
what lets the picker show real limits rather than guesses:
what lets the picker show real limits rather than defaults:

| Provider | Fields observed |
|---|---|
| Chutes | `context_length`, `max_output_length`, `supported_features: ["json_mode","tools","structured_outputs","reasoning"]` |
| OpenRouter | `context_length`, `top_provider.max_completion_tokens`, `supported_parameters` including `tools` |
| Nvidia NIM | ids only (`id`, `created`, `owned_by`) |
| ModelScope | ids only (`id`, `created`, `owned_by`) |

Providers that publish only ids still work: their limits come from the shipped
catalog, and a model is excluded only when a provider explicitly says it cannot
call tools.
Providers that publish only ids still work: their limits fall back to
conservative defaults (128k context, 32k output), and a model is excluded only
when a provider explicitly says it cannot call tools.

### Discovery against real keys

| Provider | Result |
|---|---|
| Chutes | 12 models, with context and output read from the API |
| Alibaba Token Plan | 7 models |
| OpenAI | 401 — the key on this machine is not valid for `api.openai.com`; the curated list was used, as designed |
| OpenAI | 401 — the key on this machine is not valid for `api.openai.com`; nothing remembered, so the provider stayed out of the picker, as designed |

Measured wall-clock for the three configured providers, queried in parallel: 1.2s.

Expand Down
16 changes: 7 additions & 9 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,8 @@ The list comes from the provider itself. Three things can hide a model:
chosen by the same ranking used elsewhere. Raise it with `maxDiscoveredModels`
in your config, or pin the exact models you want under `models`.
- **Discovery fell back.** If the provider was unreachable or rejected the key,
the list it returned last time is used; failing that, the shipped one. The log
says which and why. A shipped list is generic, so it may name models your plan
does not include — if you see one that should not be there, that is the sign.
the list it returned last time is used; failing that, the provider stays out
of the picker. The log says which and why.

Results are cached for ten minutes, so a model added upstream may take that long
to appear — restart to pick it up immediately.
Expand All @@ -101,16 +100,15 @@ provider actually offers:
copilot-byok --legacy --provider <id> --list-models
```

Then update `models` in your config, or run `npm run catalog:update` if you use
the built-in catalog.
Then pin the working names under `models` in your own config file.

## The agent starts but fails mid-task

Almost always a capability mismatch: the model does not really support tool
calling, and the harness discovers it once it tries to use a tool. The generated
catalog only publishes models that report tool support, but a hand-written config
entry can claim anything. Verify the model supports function calling and
streaming before adding it.
calling, and the harness discovers it once it tries to use a tool. Discovery
only publishes models that do not rule out tool support, but a hand-written
config entry can claim anything. Verify the model supports function calling and
streaming before pinning it.

## Copilot CLI is not found

Expand Down
Loading
Loading