Unified AI inference deployment SDK across cloud GPU providers

Unified AI inference deployment SDK across cloud GPU providers.
Choose your model or workload (Audio, LLM, Image, Video) — InferWeave provisions the GPU infrastructure, sets up the container environment, manages CUDA/VRAM requirements, and exposes a ready-to-use endpoint.

---

🌟 The Vision

Deploying open-weight AI models into production is notoriously complex:

• Different hardware specs (VRAM sizing, tensor parallelism, compute capability).
• Fragmented container runtimes (vLLM, SGLang, ComfyUI, custom FastAPI/PyTorch wrappers).
• Provider lock-in and disparate provisioning APIs (RunPod, Lambda, AWS, GCP, Modal, etc.).

InferWeave hides all infrastructural complexity behind a unified Python interface:

Configure account and Proxy Token credentials using the TTS tutorial, then:

"""Run with python examples/tts.py after configuring Modal and proxy credentials."""

import asyncio
from pathlib import Path

from inferweave import InferWeave


async def main() -> None:
    weave = InferWeave()
    deployment = None
    try:
        deployment = await weave.deploy(
            model="fish-s2-pro",
            provider="modal",
            custom_args={"cleanup_on_failure": True},
        )
        audio = await deployment.synthesize("Hello from InferWeave", format="wav")
        Path("speech.wav").write_bytes(audio)
    finally:
        try:
            if deployment is not None:
                await deployment.stop()
        finally:
            await weave.close()


if __name__ == "__main__":
    asyncio.run(main())

InferWeave automatically handles model profile detection, VRAM budgeting, container runtime templates, healthcheck polling, and endpoint proxying.

---

Inference, recovery and lifecycle

await weave.attach(id) rebuilds a handle after restart; await weave.find(model=..., provider=...) reuses a unique active deployment from SQLite. Ambiguity raises AmbiguousDeploymentError. Mount the database directory in persistent services.

Modal endpoints are protected by default. SDK credentials provision resources; separate MODAL_PROXY_TOKEN_ID/MODAL_PROXY_TOKEN_SECRET authenticate probes/inference. Scale-to-zero releases idle GPU containers while keeping the app reusable; destroy_after_idle_mins stops the entire app. close() releases local resources.

Multiple accounts per provider

Register several Modal, Lightning AI, RunPod or Vast.ai accounts and InferWeave rotates new deployments across them with CredWeave: round-robin, weighted or priority failover, cooldown on rate limits and quota exhaustion, and bounded failover to the next account. Every deployment stays bound to the account that created it, across restarts, for status, stop, autostop and inference.

weave = InferWeave(accounts="accounts.yaml")  # or export INFERWEAVE_ACCOUNTS_FILE
deployment = await weave.deploy("fish-s2-pro", provider="modal")
print(deployment.account)  # e.g. "modal-team-b"

The accounts file only names environment variables or a protected credentials file; secrets are kept out of deployment records, logs and model containers. Uncertain creation blocks retries until the original operation has settled and owner-bound absence/cleanup is confirmed. Changed control keys cannot silently take over existing pooled deployments. Read use multiple accounts.

Documentation and examples

Lightning AI uses the same deployment, inference and recovery API:

weave = InferWeave()
deployment = await weave.deploy("fish-s2-pro", provider="lightning")
audio = await deployment.synthesize("Hello from Lightning AI")
await deployment.stop()
await weave.close()

Install pip install "inferweave[lightning]" and configure LIGHTNING_USER_ID, LIGHTNING_API_KEY and LIGHTNING_TEAMSPACE=owner/teamspace locally. The user API key also authenticates inference; scoped keys are unsupported. Lightning creates one container Deployment with no Studio; stop deletes it. Read the Lightning guide for first-pull readiness, autoscaling, restart recovery and opt-in paid tests. The verified SDK is lightning-sdk==2026.10.1 (Python 3.11/3.12). Its Click requirement conflicts with SkyPilot; InferWeave runs Lightning and SkyPilot operations in worker processes, so each can live in its own virtual environment (INFERWEAVE_LIGHTNING_PYTHON, INFERWEAVE_SKYPILOT_PYTHON) while one InferWeave process drives all providers. inferweave[all] retains SkyPilot, Modal and workers; it excludes Lightning.

• Use multiple provider accounts
• Use Lightning AI
• First TTS on Modal
• Generate images
• Recover deployments
• Secure endpoints
• Handle cold starts
• Long-lived services
• API, configuration, models
• Architecture, lifecycle and cost, usage metering and estimates
• Runnable examples, roadmap, changelog

🏗️ Architecture

               ┌────────────────────────┐
               │    Your Application    │
               └───────────┬────────────┘
                           │
               ┌───────────▼────────────┐
               │     InferWeave SDK     │
               │ (Registry, Router, API)│
               └───────────┬────────────┘
                           │
       ┌───────────────────┴───────────────────┐
       ▼                                       ▼
┌───────────────┐                       ┌───────────────┐
│ Runtime Layer │                       │ Compute Layer │
├───────────────┤                       ├───────────────┤
│ • Audio (TTS) │                       │ • SkyPilot:   │
│ • LLM (vLLM)  │                       │   RunPod/Vast │
│ • Image (FLUX)│                       │   AWS/GCP/etc.│
│ • Video (WAN) │                       │ • Modal       │
│ • Custom      │                       │ • Cloud GPUs  │
└───────────────┘                       └───────────────┘

---

📦 Installation

InferWeave provides a modular installation model so you only install dependencies for the clouds you actually use.

1. Base Installation (Core SDK & CLI)

Installs the unified client, schema definitions, model registry, and CLI tools:

pip install inferweave

2. Provider-Specific Extras

Install drivers for the clouds you plan to target:

# Individual cloud providers (via SkyPilot engine)
pip install "inferweave[runpod]"
pip install "inferweave[aws]"
pip install "inferweave[gcp]"
pip install "inferweave[azure]"
pip install "inferweave[lambda]"
pip install "inferweave[nebius]"
pip install "inferweave[kubernetes]"
pip install "inferweave[vast]"

# Serverless compute engine (verified with modal>=1.6,<1.7)
pip install "inferweave[modal]"

# Inference worker runtime (FastAPI / Uvicorn stack)
pip install "inferweave[workers]"

3. Multi-Cloud Bundles

# All cloud providers supported by SkyPilot
pip install "inferweave[clouds]"

# SkyPilot cloud engines + Modal + workers runtime (Lightning uses a separate environment)
pip install "inferweave[all]"

---

⚡ CLI Quickstart

InferWeave includes a high-performance command-line interface for deploying and inspecting models across GPU providers:

# 1. Deploy models directly to cloud GPUs
inferweave deploy fish-s2-pro --provider modal
inferweave deploy meta-llama/Meta-Llama-3-8B-Instruct --provider runpod --gpu A100

# 2. Dry-run deployment to preview configuration and hardware budgets
inferweave deploy fish-s2-pro --provider modal --dry-run

# 3. Explore supported models and cloud providers
inferweave models
inferweave models --workload audio
inferweave providers

# 4. Inspect, monitor, and stop deployments across processes
inferweave list
inferweave status <deployment-id>
inferweave stop <deployment-id>

---

💻 Platform & OS Support Guidelines

Operating System	InferWeave Core / Client	Modal Engine	SkyPilot Engine (Multi-Cloud)
Linux (Ubuntu/Debian/RHEL)	✅ Native	✅ Native	✅ Native
macOS (Apple Silicon / Intel)	✅ Native	✅ Native	✅ Native
Windows (WSL2)	✅ Native (WSL)	✅ Native (WSL)	✅ Native (WSL)
Windows (Native PowerShell/CMD)	✅ Native	✅ Native	⚠️ Requires WSL2 / Remote Runner

⚠️ Important Note for Windows Users

SkyPilot’s underlying execution engine relies on POSIX system primitives (termios, resource) that are not present in native Windows.

• If you are on Windows: We recommend running InferWeave inside WSL2 (Windows Subsystem for Linux), where all multi-cloud provisioning features work seamlessly.
• InferWeave Graceful Fallback: InferWeave utilizes lazy loading for SkyPilot modules. If invoked directly on native Windows, InferWeave will safely guide you to WSL2 or allow targeting native-supported backends (such as Modal or remote runners) without crashing during import.

---

---

🛠️ Contributing & Development Setup

InferWeave uses uv for lightning-fast dependency management.

# Clone the repository
git clone https://github.com/AmaLS367/InferWeave.git
cd InferWeave

# Install extras needed for mocked tests and dev tools
uv sync --extra modal --extra workers --group dev

# Run linting and type checking
uv run ruff check .
uv run mypy src
uv run pytest -m "not integration" --strict-markers -ra
uv run python examples/simulate.py

Live integration tests (opt-in, spend real money)

Tests marked integration talk to real clouds. They are excluded from CI and skip cleanly unless explicitly enabled and configured:

Test	Required environment
tests/test_modal_integration.py	MODAL_TOKEN_ID, MODAL_TOKEN_SECRET, MODAL_PROXY_TOKEN_ID, MODAL_PROXY_TOKEN_SECRET
tests/test_runpod_integration.py	INFERWEAVE_RUNPOD_INTEGRATION=1, RUNPOD_API_KEY (or ~/.runpod/config.toml), inferweave[runpod] on Linux/macOS/WSL2

For local testing, copy .env.example to .env and fill in the four Modal credentials. .env is ignored by Git. The library does not load it automatically; uv can load it for the test process:

uv run --env-file .env --extra modal pytest -m integration tests/test_modal_integration.py --strict-markers -ra

This test checks protected access, TTS output, recovery with a fresh SDK and deployment cleanup. It provisions a real GPU and incurs charges.

The RunPod test also reads optional INFERWEAVE_RUNPOD_GPU (default L4), INFERWEAVE_RUNPOD_MODEL (default facebook/opt-125m), INFERWEAVE_RUNPOD_READY_TIMEOUT_SECS, INFERWEAVE_RUNPOD_AUTOSTOP_MINS and INFERWEAVE_RUNPOD_ALLOW_SPOT; see the module docstring.

INFERWEAVE_RUNPOD_INTEGRATION=1 RUNPOD_API_KEY=... \
  uv run --extra runpod pytest -m integration tests/test_runpod_integration.py -s

🚀 Release Process

Publishing to PyPI is automated via GitHub Actions using PyPI Trusted Publishing (OIDC):

1. Ensure the CI suite is green on master.
2. Set the package version in src/inferweave/__init__.py.
3. Create and push a Git tag: vX.Y.Z.
4. Publish a GitHub Release from that tag.
5. GitHub Actions (publish.yml) verifies artifact metadata, validates that the release tag matches inferweave.__version__, and publishes the artifacts to PyPI via Trusted Publishing.

---

📄 License

InferWeave's source code is licensed under the Apache License 2.0. See LICENSE for details.

Third-party licenses

The Apache-2.0 license covers InferWeave's own source code only. It does not cover anything InferWeave deploys or references:

• Model weights (including those named by the built-in model profiles) keep their upstream licenses.
• Container images and third-party runtimes (vLLM, Fish Speech, PyTorch, and the Python packages installed into them) keep their upstream licenses.
• You are responsible for reviewing and complying with the upstream terms of every model, image and runtime you deploy. This is not legal advice.

Fish Audio S2 Pro (fish-s2-pro): the fishaudio/s2-pro weights are published under Fish Audio's own license (fish-audio-research-license on Hugging Face), not Apache-2.0. The built-in S2 runtime is beta: it uses the upstream v2.0.0-beta pre-release of fishaudio/fish-speech. Review the upstream model and runtime licenses before any production or commercial use.