A privacy-first desktop assistant that runs entirely offline on an ordinary CPU — no GPU, no cloud required. Voice in → local AI pipeline → real OS actions + spoken response, behind a holographic web UI.
Final Year Project · B.Sc. Software Engineering, COMSATS University Islamabad
Most voice assistants ship your audio to the cloud and die without internet. ULTON targets low-resource, privacy-sensitive environments: everything critical — speech recognition, language understanding, command execution — runs locally on a normal Windows PC. Cloud services are an optional upgrade, never a dependency.
95+ voice commands across 25+ categories (300+ phrasing variations), including:
| Category | Examples |
|---|---|
| Files & folders | create / open / rename / delete, open drives, navigate |
| Windows | maximize, minimize, snap, switch, close |
| Browser | tabs, zoom, refresh, Google/YouTube search |
| System | volume, brightness, wallpaper, screenshots, lock, system info |
| Clipboard & text | copy/paste/select-all, undo/redo, read clipboard aloud, dictation mode |
| read your n-th most recent Gmail (OAuth 2.0) | |
| Productivity | notes, timers, calculator, weather, time/date |
| Messaging | send WhatsApp messages to saved contacts |
| AI | conversational Q&A via local LLM, clipboard summarization |
Plus two extra input modalities:
- 🔐 Face-recognition login — Haar cascade detection + histogram feature matching (cosine similarity); per-user enrollment (199 images), system stays locked until an authorized face is seen
- 🖐️ Hand-gesture virtual mouse — MediaPipe 21-joint tracking: pinch to click/drag/scroll, fist to hand off to voice, pinky gesture to toggle the on-screen keyboard, One-Euro-filtered for smooth pointer motion; plus a voice-directed 9-cell click grid
Voice ──► STT ──► Intent classifier ──► Command router ──► 5 handler modules (files / OS / browser / general / grid)
│ (TF-IDF + SGD, │
│ command vs chat) └──► pyautogui · pywin32 · subprocess
│
└─ chat ──► LLM ──► TTS response
│
local: Qwen2.5-0.5B GGUF (llama-cpp-python)
cloud (optional): Groq · llama-3.3-70B
- STT: OpenVINO Whisper (distil-small) offline · AssemblyAI when online
- LLM: Qwen2.5-0.5B-instruct (q4_k_m GGUF) locally · optional Groq cloud fallback; responses cached, commands always executed fresh
- TTS: persistent Windows System.Speech process (low latency) with pyttsx3 fallback
- VAD: Silero
- Backend: FastAPI + WebSockets (real-time UI updates)
- Frontend: React + TypeScript + Vite + React Three Fiber — 3D holographic interface with microphone orb and HUD
Python llama-cpp-python OpenVINO Whisper scikit-learn MediaPipe OpenCV FastAPI WebSockets React TypeScript Three.js pyautogui pywin32 Gmail API
# 1. Install dependencies
pip install -r requirements.txt
# 2. Launch (starts backend on :8000 and serves the UI)
LAUNCH_APP.batSee FINAL_INSTALL_GUIDE.txt and COMMAND_LIST.md for the full setup walkthrough and every supported command.
- Fully functional without internet — STT, LLM, and all command execution are local
- CPU-only by design (4-bit quantized 0.5B model ≈ runs on any modern laptop)
- Hybrid architecture: seamlessly upgrades to cloud STT/LLM when connectivity exists
- Real OS integration, not a demo: window management, file ops, email, messaging
Author: Muhammad Maowiz Saleem · LinkedIn