Skip to content

feat: add LiteLLM as AI gateway provider#120

Open
RheagalFire wants to merge 1 commit into
kharvd:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway provider#120
RheagalFire wants to merge 1 commit into
kharvd:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire
Copy link
Copy Markdown

Summary

  • Adds LiteLLM as a completion provider, giving gpt-cli users access to 100+ LLM providers (Bedrock, Vertex AI, Groq, Together, Mistral, and more) through the LiteLLM Python SDK.
  • LiteLLM is added as an optional dependency (gpt-command-line[litellm]).

Motivation

gpt-cli currently supports OpenAI, Anthropic, Google, Cohere, Azure, and LLaMA. LiteLLM lets users access any additional provider without a new provider implementation per SDK. Users prefix the model name with litellm: and use LiteLLM's model naming convention (e.g., litellm:bedrock/anthropic.claude-v2, litellm:together_ai/mistralai/Mixtral-8x7B).

Changes

  • gptcli/providers/litellm.py - New LiteLLMCompletionProvider implementing CompletionProvider
  • gptcli/assistant.py - Registered litellm: prefix in get_completion_provider()
  • pyproject.toml - Added litellm>=1.80.0,<1.87.0 as optional dependency

Implementation notes

  • drop_params=True on every call so provider-unsupported kwargs (e.g., temperature on reasoning models) are silently dropped
  • litellm is lazy-imported inside the complete() method so the module loads without litellm installed
  • Follows the same pattern as existing providers: yields MessageDeltaEvent for content, UsageEvent for token counts
  • Streaming supported via litellm.completion(stream=True)
  • LiteLLM reads provider-specific env vars automatically (ANTHROPIC_API_KEY, AWS_ACCESS_KEY_ID, etc.)

Risk / Compatibility

  • Additive only. Existing providers untouched.
  • Optional extra - base install unaffected.

Example usage

# ~/.config/gpt-cli/gpt.yml
assistants:
  litellm-claude:
    model: litellm:anthropic/claude-sonnet-4-20250514
    temperature: 0.7
  litellm-bedrock:
    model: litellm:bedrock/anthropic.claude-v2
# Set provider-specific env vars
export ANTHROPIC_API_KEY=sk-ant-xxx
gpt litellm-claude
import litellm

# Under the hood, gpt-cli calls:
response = litellm.completion(
    model="anthropic/claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
    drop_params=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content, end="")

@RheagalFire
Copy link
Copy Markdown
Author

cc @kharvd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant