Repository navigation
feat(models.dev): prefer provider-advertised token limits - #200
Conversation
Gateways that front several upstreams advertise the limits they actually accept on GET /v1/models. Keep those values authoritative for the context window and output budget, then fill any remaining gap from models.dev, instead of letting the static catalog override the live endpoint. Reads context_length/max_context_length and max_completion_tokens/max_output_tokens, ignores malformed values, and applies the same precedence to the aiproxy format that builds on this enricher.
|
Thanks for the implementation. I performed an additional end-to-end check against a local CLIProxyAPI instance using the project's current OpenAI-compatible provider configuration. The standard For the current local setup, this means the new Provider-limit precedence path is not exercised and the result continues to come from models.dev. I matched the returned model IDs against the current models.dev catalog: 21 models matched, and none had So this PR does not introduce a behavior change for the current local setup, while the feature remains useful for gateways or deployments that do expose route-specific limits inline. I suggest slightly qualifying the documentation to say “gateways that expose inline limits” rather than implying that the standard CLIProxyAPI Before merging, could you also consider adding or confirming regression coverage for these two consistency cases?
Finally, please update the branch from the current |
|
I took advantage of the maintainer-edit permission and pushed a follow-up commit ( The follow-up keeps your precedence model intact and closes the two consistency edges from the review:
Validation passed on the updated branch: lint (0 errors; existing warnings only), typecheck, full suite (47 files / 377 tests), V1 (71 tests), V2 (96 tests), build, bundle guards, and notices guards. The PR is now based on current |
Problem
When
modelInfoFormatis"models.dev"(or"aiproxy", which builds on the same enricher), limits come only from the static models.dev catalog. A gateway that fronts several upstreams often advertises the limits it actually accepts inline onGET /v1/models, but the catalog wins, so the discovered model can get a context window or output budget that the endpoint does not enforce.A concrete case: CLIProxyAPI exposes
context_lengthandmax_completion_tokensfor routed models, yet a model entry with no metadata fell back to models.dev, while the same model id behind another provider got different numbers.Change
/v1/modelsentry:context_lengthormax_context_lengthfor the context window,max_completion_tokensormax_output_tokensfor the output budget.limit.inputand general metadata).Infinity) are ignored.createModelLimits."aiproxy"inherits the behavior automatically: the advertised limits are the base that AIProxy's inlinelimitsblock then overlays.Testing
test/models-dev-model-info.test.tscovers precedence, themax_context_length/max_output_tokensaliases, partial provider metadata, malformed values, models absent from models.dev, and output clamping.limitsstill overlay them.npx vitest run: 43 files pass;test/node-loadable.test.tswas skipped by its environment (it stages the package withnpm install, which timed out on the network here) — unrelated to this change.npx tsc --noEmitandnpx eslint srcpass.Docs:
docs/configuration.mdnow states the precedence for both formats.