Skip to content

feat(models.dev): prefer provider-advertised token limits - #200

Merged
yuhp merged 3 commits into
yuhp:devfrom
Realmagnum:feat/provider-advertised-limits
Nov 17, 2026
Merged

yuhp merged 3 commits into
yuhp:devfrom
Realmagnum:feat/provider-advertised-limits

Conversation

@Realmagnum

Copy link
Copy Markdown
Contributor

Problem

When modelInfoFormat is "models.dev" (or "aiproxy", which builds on the same enricher), limits come only from the static models.dev catalog. A gateway that fronts several upstreams often advertises the limits it actually accepts inline on GET /v1/models, but the catalog wins, so the discovered model can get a context window or output budget that the endpoint does not enforce.

A concrete case: CLIProxyAPI exposes context_length and max_completion_tokens for routed models, yet a model entry with no metadata fell back to models.dev, while the same model id behind another provider got different numbers.

Change

  • Read provider-advertised limits from the /v1/models entry: context_length or max_context_length for the context window, max_completion_tokens or max_output_tokens for the output budget.
  • Provider values are authoritative; models.dev fills only the fields the endpoint leaves out (for example, models.dev still supplies limit.input and general metadata).
  • Malformed values (strings, zero, negative, Infinity) are ignored.
  • Output is still clamped to the effective context through createModelLimits.
  • "aiproxy" inherits the behavior automatically: the advertised limits are the base that AIProxy's inline limits block then overlays.

Testing

  • New test/models-dev-model-info.test.ts covers precedence, the max_context_length/max_output_tokens aliases, partial provider metadata, malformed values, models absent from models.dev, and output clamping.
  • Added an aiproxy case where provider limits are the base and inline limits still overlay them.
  • npx vitest run: 43 files pass; test/node-loadable.test.ts was skipped by its environment (it stages the package with npm install, which timed out on the network here) — unrelated to this change.
  • npx tsc --noEmit and npx eslint src pass.

Docs: docs/configuration.md now states the precedence for both formats.

Gateways that front several upstreams advertise the limits they actually
accept on GET /v1/models. Keep those values authoritative for the context
window and output budget, then fill any remaining gap from models.dev,
instead of letting the static catalog override the live endpoint.

Reads context_length/max_context_length and
max_completion_tokens/max_output_tokens, ignores malformed values, and
applies the same precedence to the aiproxy format that builds on this
enricher.
@yuhp

yuhp commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Thanks for the implementation. I performed an additional end-to-end check against a local CLIProxyAPI instance using the project's current OpenAI-compatible provider configuration.

The standard GET /v1/models response returned 31 models, but only the standard OpenAI fields. None of the entries contained context_length, max_context_length, max_completion_tokens, max_output_tokens, or max_input_tokens. I also checked the available extended model-list responses; the local instance did not expose usable limit fields there either.

For the current local setup, this means the new Provider-limit precedence path is not exercised and the result continues to come from models.dev. I matched the returned model IDs against the current models.dev catalog: 21 models matched, and none had limit.input > limit.context.

So this PR does not introduce a behavior change for the current local setup, while the feature remains useful for gateways or deployments that do expose route-specific limits inline. I suggest slightly qualifying the documentation to say “gateways that expose inline limits” rather than implying that the standard CLIProxyAPI /v1/models response always does so.

Before merging, could you also consider adding or confirming regression coverage for these two consistency cases?

  1. A Provider context smaller than the models.dev input limit, so the resulting limit.input does not exceed the effective context (or the intended behavior is explicitly documented).
  2. AIProxy inline limits.max_output_tokens larger than context, since the later AIProxy overlay currently appears able to bypass the createModelLimits output clamp.

Finally, please update the branch from the current dev (which now includes PR #192) and rerun the GitHub CI checks. The isolated local validation of this PR passed: 44 test files and 351 tests, with no TypeScript or ESLint errors.

@yuhp

yuhp commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

I took advantage of the maintainer-edit permission and pushed a follow-up commit (8c8133f) to this PR after updating the work onto the current dev line.

The follow-up keeps your precedence model intact and closes the two consistency edges from the review:

  • createModelLimits now bounds both explicit input and output limits to the effective context.
  • AIProxy's inline limits overlay now goes back through the shared normalizer, so it cannot bypass those bounds.
  • Regression coverage now includes a models.dev input limit larger than provider context and an AIProxy inline input limit larger than provider context.
  • Documentation now qualifies the behavior as applying to gateways that expose inline limits.

Validation passed on the updated branch: lint (0 errors; existing warnings only), typecheck, full suite (47 files / 377 tests), V1 (71 tests), V2 (96 tests), build, bundle guards, and notices guards. The PR is now based on current dev and CI is green.

@yuhp
yuhp merged commit 8136fdb into yuhp:dev Nov 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants