Inference Mode

Inference Mode

Run a computing provider and serve AI inference requests — no wallet, no blockchain, no public IP required.

Inference mode is the default and simplest way to participate in the Swan Chain network. The provider connects outbound via WebSocket, so there's nothing to expose to the internet.

---

How it works

1. You run a local model server (SGLang, vLLM, Ollama, or any OpenAI-compatible endpoint)
2. The computing-provider connects to Swan Inference via WebSocket and registers your models
3. Inference requests arrive over the existing WebSocket connection — no inbound port needed
4. The computing-provider forwards each request to your local server and streams the response back

Swan Inference (cloud)
      │  WebSocket (outbound from your machine)
      ▼
computing-provider
      │  POST /v1/chat/completions
      ▼
your local model server (SGLang / vLLM / Ollama / CLIProxyAPI / …)

---

Reliability & performance

The provider manages request flow and backend health automatically:

• Health checking — each model endpoint is polled every 30s; after 3 consecutive failures a model is reported unhealthy and Swan Inference routes traffic away, recovering automatically when it comes back online.
• Rate limiting & concurrency — GPU-aware token-bucket limits plus global and per-model in-flight slots protect your GPU; over-limit requests are rejected with HTTP 429.
• Automatic retries — transient upstream failures (connection refused/reset, 502/503/504, timeouts) are retried with exponential backoff and jitter.
• Failure logging — failed requests are logged with model, latency, and status code, so problems are visible in cp.log.
• Graceful shutdown — SIGTERM/SIGINT drains cleanly before tearing down inference subsystems.

Rate limits and concurrency slots can be tuned at runtime via the REST API.

---

Prerequisites

• Go 1.21+
• A running OpenAI-compatible model server
• A Swan Inference provider account (sk-prov-* key)

No GPU is strictly required — see the CLIProxyAPI guide for serving GPT-5.x via a ChatGPT account.

---

Quickstart

1. Start a model server

SGLang (GPU):

docker run -d --gpus all -p 30000:30000 --name sglang \
  --shm-size 4g --ipc=host \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.2-3B-Instruct \
    --host 0.0.0.0 --port 30000 \
    --served-model-name meta-llama/Llama-3.2-3B-Instruct

Ollama (CPU or GPU):

ollama serve
ollama pull qwen2.5:7b

CLIProxyAPI (no GPU — uses ChatGPT OAuth):

./CLIProxyAPI serve   # serves gpt-5.x on :8317

→ See the CLIProxyAPI guide for full setup.

2. Get a provider API key

# Sign up
curl -X POST https://inference.swanchain.io/api/v1/user/signup \
  -H "Content-Type: application/json" \
  -d '{"email":"you@example.com","password":"YourPass1","display_name":"My Provider"}'

# Upgrade to provider — copy the sk-prov-* key from the response
TOKEN="<token from signup>"
curl -X POST https://inference.swanchain.io/api/v1/user/upgrade-to-provider \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"My Provider"}'

Or use the interactive setup wizard:

computing-provider setup

3. Configure

computing-provider init --node-name my-provider --port 9085

Edit ~/.swan/computing/config.toml:

[Inference]
  Enable = true
  WebSocketURL = "wss://inference-ws.swanchain.io"
  ApiKey = "sk-prov-<your-key>"
  Models = ["meta-llama/Llama-3.2-3B-Instruct"]

Create ~/.swan/computing/models.json:

{
  "meta-llama/Llama-3.2-3B-Instruct": {
    "endpoint": "http://localhost:30000",
    "gpu_memory": 8000,
    "category": "text-generation"
  }
}

4. Run

computing-provider run

Confirm you see:

Connected to Swan Inference
Registration successful: registered successfully
Model meta-llama/Llama-3.2-3B-Instruct health changed: unknown -> healthy

---

Monitoring

# CLI
computing-provider inference status
computing-provider inference config

# REST API (local)
curl http://localhost:9085/api/v1/computing/inference/status
curl http://localhost:9085/api/v1/computing/inference/models

# Dashboard
open http://localhost:9085   # or http://localhost:3005

---

Hot-reload models

Edit models.json at any time — the file watcher picks up changes and re-registers models with Swan Inference automatically. No restart required.

To force a reload:

curl -X POST http://localhost:9085/api/v1/computing/inference/models/reload

---

Local development

Point the provider at a local swan-inference instance instead of production:

export INFERENCE_WS_URL=ws://localhost:8081
export INFERENCE_API_KEY=sk-prov-your-local-key
computing-provider run

Or set in config.toml:

[Inference]
  WebSocketURL = "ws://localhost:8081"   # no /ws suffix