Skip to content

index.md

Latest commit

 

History

History
82 lines (52 loc) · 3.51 KB

index.md

File metadata and controls

82 lines (52 loc) · 3.51 KB
Raw

Getting Started & System Requirements

Local LLM Server Manager manages local artificial intelligence engines through a unified interface. The application coordinates language models, image diffusion, 3D mesh reconstruction, video synthesis, and speech generation.

The software runs as a native desktop application, a system tray application, and a headless background service.

System Requirements

Review the hardware and operating system specifications before you install the software.

Operating System Support

Platform Supported Versions Display Environments
Windows Windows 10 (64-bit, 21H2+) and Windows 11 Desktop Shell, System Tray, Windows Service
Linux Ubuntu 22.04+, Debian 12+, Fedora 38+, Arch Linux X11, Wayland, Headless systemd daemon

Hardware Requirements

Hardware Component Minimum Requirement Recommended Specification
Processor (CPU) 4-core 64-bit x64 processor 8-core modern x64 processor
System Memory (RAM) 16 GB RAM 32 GB RAM or higher
Graphics Card (GPU) NVIDIA GPU with 8 GB VRAM NVIDIA RTX 3060/4070+ with 12 GB to 24 GB VRAM
Disk Storage 20 GB free disk space 200 GB+ free space on NVMe SSD
Network Loopback network interface High-speed internet connection for model downloads

Important

Install the latest NVIDIA GPU drivers and CUDA toolkit for hardware acceleration. Verify your GPU setup by running nvidia-smi in your terminal.

Note

The system can run language models on CPU cores when a dedicated GPU is absent. However, CPU inference operates at reduced generation speeds.

Tip

Place your model storage directories on a fast NVMe solid-state drive. Fast storage significantly reduces model loading times.

Supported AI Engines

Local LLM Server Manager connects to and orchestrates multiple local inference engines:

  • Ollama Engine: Runs Large Language Models (LLMs) locally through port 11434.
  • Stable Diffusion Forge: Generates images and manages CivitAI checkpoints through port 7860.
  • ComfyUI Engine: Generates 3D meshes, videos, and complex diffusion workflows through port 8188.
  • Kokoro TTS Engine: Synthesizes speech with OpenAI-compatible audio endpoints through port 8880.

The manager coordinates these engines through a unified reverse proxy on port 5246.

Getting Started Roadmap

Follow these sequential steps to set up and use Local LLM Server Manager:

  1. Installation Guide
    Review the required versus optional component matrix and install the application on Windows or Linux.

  2. First-Time Configuration
    Auto-detect installed engines, configure engine port numbers, and set your model storage directories.

  3. Quickstart Guide
    Download your first language model from Hugging Face or Ollama, and test your first prompt.

  4. Real Engine Test Flight
    Verify that your local inference engines respond to real network requests before queuing heavy workloads.

  5. Remote Access & Reverse Proxy
    Access your dashboard over LAN, configure SSH tunnels, or set up Caddy reverse proxy authentication.

  6. Troubleshooting Guide
    Resolve common operational issues, handle VRAM out-of-memory errors, and eliminate port conflicts.

Next Steps

Proceed to the Installation Guide to install the software on your system.