Skip to content

feat(model): add generic OpenAI-compatible LLM backend#115

Open
nankingjing wants to merge 5 commits into
microsoft:mainfrom
nankingjing:contrib/add-openai-compatible-backend
Open

feat(model): add generic OpenAI-compatible LLM backend#115
nankingjing wants to merge 5 commits into
microsoft:mainfrom
nankingjing:contrib/add-openai-compatible-backend

Conversation

@nankingjing

Copy link
Copy Markdown

Summary

Adds a generic openai_compatible model backend
(skillopt/model/openai_compatible_backend.py) so SkillOpt can talk to any
service that exposes an OpenAI-compatible /chat/completions endpoint using
nothing more than a base_url and an api_key.

A single backend therefore unlocks a large family of providers, including:

  • DeepSeek (https://api.deepseek.com/v1)
  • Groq (https://api.groq.com/openai/v1)
  • Together AI (https://api.together.xyz/v1)
  • Ollama (http://localhost:11434/v1)
  • vLLM / SGLang / TGI self-hosted servers
  • LiteLLM proxy (http://localhost:4000)
  • OpenRouter, Fireworks, Mistral, xAI Grok, Perplexity, and OpenAI itself

What's included

  • New backend module built on the official openai SDK, mirroring the
    callable surface of the existing chat backends (qwen_backend,
    minimax_backend): chat_optimizer, chat_target,
    chat_optimizer_messages, chat_target_messages, plus token tracking,
    configure_openai_compatible(...), and the standard setter functions.
  • Per-role config (optimizer/target) via configure_openai_compatible(...)
    or environment variables (OPENAI_COMPATIBLE_BASE_URL,
    OPENAI_COMPATIBLE_API_KEY, OPENAI_COMPATIBLE_MODEL, plus
    _TEMPERATURE/_MAX_TOKENS/_TIMEOUT_SECONDS, with OPTIMIZER_*/TARGET_*
    overrides). The API key is optional so keyless local servers (Ollama, vLLM)
    work out of the box.
  • count_tokens() using tiktoken when available, with a character-based
    estimate as a fallback for non-OpenAI models.
  • Tool / function calling support through the *_messages(..., tools=...)
    entry points, returning the shared CompatAssistantMessage objects.
  • Retries with exponential backoff and usage recorded through the shared
    TokenTracker.

Registry / routing wiring

  • common.py — registered openai_compatible in _BACKEND_DEFAULT_MODELS and
    _BACKEND_ALIASES (aliases: openai_compatible, openai_compatible_chat,
    openai-compatible, compat).
  • backend_config.py — accepted as a valid optimizer/target chat backend.
  • model/__init__.py — routed through chat_optimizer/chat_target and their
    _messages variants, added to token-summary aggregation and the
    reset/setter helpers, exposed configure_openai_compatible, and wired into
    the legacy set_backend("openai_compatible") shorthand.

Usage

import skillopt.model as model

model.set_backend("openai_compatible")
model.configure_openai_compatible(
    base_url="https://api.deepseek.com/v1",
    api_key="sk-...",
    model="deepseek-chat",
)

or purely via env vars:

export TARGET_BACKEND=openai_compatible
export OPENAI_COMPATIBLE_BASE_URL="https://api.groq.com/openai/v1"
export OPENAI_COMPATIBLE_API_KEY="gsk_..."
export OPENAI_COMPATIBLE_MODEL="llama-3.3-70b-versatile"

Docs

docs/guide/new-backend.md now opens with a section describing this built-in
backend so contributors reach for it before writing a bespoke one.

Testing

  • python -m py_compile on all touched modules.
  • Local import of skillopt.model and backend selection
    (set_backend/get_backend_name/get_optimizer_backend).
  • Functional test with a mocked OpenAI client: verified routing, that the
    configured model/max_tokens are sent, CompatAssistantMessage return
    values, token-summary aggregation, and count_tokens() on both the tiktoken
    and character-fallback paths.

Note: the existing docs/guide/new-backend.md template describes an
aspirational ModelBackend/async generate() interface that does not match
the current module-based backends. This backend intentionally follows the
actual in-tree pattern so it integrates and runs today; the added docs
section reflects the real interface.

@nankingjing

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant