Gate

A local OpenAI-compatible API proxy that manages vLLM docker deployments on demand.

Features

• OpenAI-compatible API: Exposes /v1/models, /v1/chat/completions, and /v1/completions endpoints
• Multi-model support: Configure multiple models in a single config file
• Automatic lifecycle management: Starts vLLM containers on first request, stops them after idle timeout
• Single active container: Only one model runs at a time to optimize GPU memory usage
• Interactive CLI: Pastel-based CLI with a guided configuration wizard

Install

bun install

Quick Start

1. Configure Gate

Run the configuration wizard:

bun run src/cli.ts config

This will guide you through setting up:

• Server settings (host, port)
• Lifecycle settings (idle timeout, startup timeout)
• vLLM Docker settings (image, ports, runtime)
• Model configurations (add as many as you need)

2. Set Environment Variables

Gate passes through environment variables to vLLM containers. Set your Hugging Face token:

export HUGGING_FACE_HUB_TOKEN=your_token_here

3. Start the Proxy

bun run src/cli.ts serve

4. Make Requests

curl http://localhost:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Configuration

Config is stored at ~/.gate/config.json. Example:

{
  "version": 1,
  "server": {
    "host": "127.0.0.1",
    "port": 3000,
    "cors": { "enabled": true }
  },
  "lifecycle": {
    "idleShutdownMinutes": 10,
    "startTimeoutSeconds": 90,
    "pollIntervalMs": 500
  },
  "vllm": {
    "image": "vllm/vllm-openai:latest",
    "hostPort": 8002,
    "containerPort": 8000,
    "runtime": "nvidia",
    "ipc": "host",
    "env": {
      "HUGGING_FACE_HUB_TOKEN": "${env:HUGGING_FACE_HUB_TOKEN}"
    },
    "volumes": [
      {
        "host": "~/.cache/huggingface",
        "container": "/root/.cache/huggingface"
      }
    ]
  },
  "models": [
    {
      "id": "meta-llama/Llama-3.1-8B-Instruct",
      "args": ["--dtype", "auto", "--gpu-memory-utilization", "0.90"]
    },
    {
      "id": "openai/gpt-oss-20b",
      "args": [
        "--async-scheduling",
        "--tensor-parallel-size",
        "2",
        "--dtype",
        "auto",
        "--gpu-memory-utilization",
        "0.90"
      ]
    }
  ]
}

API Endpoints

GET /v1/models

Returns all configured models.

POST /v1/chat/completions

OpenAI-compatible chat completions endpoint. Requires model field in request body.

POST /v1/completions

OpenAI-compatible text completions endpoint. Requires model field in request body.

GET /healthz

Health check endpoint.

CLI Commands

gate config

Interactive wizard to create or update configuration.

Options:

• -c, --config <path>: Custom config file path
• -y, --yes: Use defaults without prompting

gate serve

Start the proxy server.

Options:

• -c, --config <path>: Custom config file path
• -p, --port <number>: Override server port

Development

Run Tests

bun test

Build

bun run build

Watch Mode

bun run dev

Requirements

• Bun runtime
• Docker with NVIDIA runtime (for GPU support)
• NVIDIA GPU with appropriate drivers

License

MIT