fix: duplicate litellm stream call doubles provider requests - #477
Open
quevedoSteven wants to merge 4 commits into
Open
fix: duplicate litellm stream call doubles provider requests#477quevedoSteven wants to merge 4 commits into
quevedoSteven wants to merge 4 commits into
Conversation
Adds cai.util.nim_rotation which cycles NVIDIA_NIM_API_KEY_1..N per request, keeping under NIM per-key rate limits (~40 req/min). Rotation applies in the OpenAI-compatible LLM path and the direct httpx completion path when the API base is a NIM endpoint (api.nvidia.com). Also defaults cache_write_tokens to 0 in the streamed usage object: litellm CustomResponseUsage now requires the field while NIM usage payloads omit it, which previously aborted streaming at end of response.
…penai The streaming path called litellm.acompletion twice with the same kwargs. The first result was assigned to a variable that is never read, and only the second call's stream was returned. Every streamed request therefore consumed two provider requests (e.g. two NVIDIA NIM requests) with the same API key, halving effective per-key rate limits and leaking the first stream. Keep a single acompletion call for the streamed path; same fix in the tool_call_id truncation retry branch.
The NIM key rotation was incorrectly applying to all models when NVIDIA_NIM_API_KEY_N are configured. This broke non-NIM models like z-ai/glm-5.2 which should route through the Alias gateway, not NVIDIA NIM. Add _is_nim_model() check with known NIM provider prefixes to gate rotation. Fix applies to both LiteLLM path and direct httpx path.
z-ai/glm-5.2 is hosted on NVIDIA NIM (build.nvidia.com/z-ai/glm-5.2). The NIM rotation was incorrectly excluding it. Add "z-ai" to the known NIM provider list in both LiteLLM and direct httpx paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The streaming path in \ etch_response_litellm_openai\ called \litellm.acompletion\ twice with the same kwargs. The first result was assigned to a variable that is never read \u2014 only the second call's stream was returned.
Impact
Every streamed request consumed two provider requests (e.g. two NVIDIA NIM requests) with the same API key:
Change
Keep a single \litellm.acompletion\ call for the streamed path, and the same fix in the \ ool_call_id\ truncation retry branch (which had the identical double-call pattern).
Verification
et\ variable remain.
Dependency
This fix requires the NIM key rotation from #475 (round-robin \NVIDIA_NIM_API_KEY_1..N) to provide \custom_llm_provider\ and \�pi_base\ for NIM models. This PR is based on that branch so the full fix works end-to-end.