Real-Time Cognitive Copilot for Call Center Agents
Aura AI is an enterprise-grade, real-time AI copilot that listens to live phone calls and augments call center agents with:
- Live Transcription — Dual-track (agent + customer) speech-to-text using Groq Whisper or Google STT
- Sentiment Analysis — Real-time mood tracking with visual sentiment arc graphs
- Smart Suggestions — AI-generated response recommendations powered by LLaMA 3.3 70B
- Knowledge Retrieval (RAG) — Instant policy/FAQ lookups from a Pinecone vector store
- Compliance Monitoring — Automatic alerts when calls hit sensitive keywords
- Caller Memory — Persistent context across repeat callers
- Call Finalization — Auto-generated summaries, categorization, and audit trails
Built for telecom, insurance, banking, and any high-volume call center operation.
flowchart TB
TW(["📞 Twilio Voice SDK"])
STT(["🎙️ STT Server · FastAPI :3001\nGroq Whisper · Google STT · Silence Filter"])
TW -->|"WebSocket Media Stream"| STT
TW -->|"TwiML"| APP
STT -->|"Transcripts"| APP
subgraph APP ["⚡ Next.js 16 · TypeScript"]
direction LR
subgraph UI ["🖥️ Portals"]
AGENT["👤 Agent\nDashboard · Live Call\nHistory · Insights · Phonebook"]
ADMIN["🔧 Admin\nAgents · Analytics\nCompliance · KB Setup"]
end
subgraph SVC ["🔌 API Routes"]
API["twilio · transcription\nai · company · kb · auth"]
end
subgraph DB ["🗄️ Data Layer"]
DATA["Firestore · Pinecone\nGroq LLaMA 3.3"]
end
end
| Feature | Description |
|---|---|
| 🎙️ Dual-Track Transcription | Separate agent & customer audio streams via Twilio Media Streams |
| 🧠 Hybrid Intelligence | Routes transcription through a hybrid endpoint for optimal latency |
| 📊 Sentiment Arc | Visual sentiment timeline using LLaMA-powered analysis |
| 💡 Response Suggestions | Context-aware reply recommendations during live calls |
| 📚 RAG Knowledge Base | Pinecone-backed retrieval for instant policy lookups |
| 🔒 Compliance Alerts | Real-time flagging of sensitive keywords & phrases |
| 🧠 Caller Memory | Cross-call context persistence for repeat callers |
| 📞 Outbound Dialer | Browser-based calling with Twilio Voice SDK |
| 🏢 Multi-Tenant | Company-level configuration, agent whitelisting, role-based access |
| 📋 Audit Trail | Full call audit logs with compliance tracking |
| 📈 Analytics Dashboard | Company-wide performance, sentiment, and call volume insights |
| Layer | Technology |
|---|---|
| Frontend | Next.js 16, React 19, TailwindCSS, Framer Motion, Three.js |
| Backend API | Next.js API Routes (TypeScript) |
| STT Server | Python, FastAPI, WebSocket |
| Speech-to-Text | Groq Whisper (primary), Google STT (fallback) |
| LLM | Groq LLaMA 3.3 70B Versatile |
| Database | Firebase Firestore (real-time sync) |
| Auth | Firebase Auth (Google Sign-In) + Session Cookies |
| Telephony | Twilio Programmable Voice + Media Streams |
| Vector Store | Pinecone |
| Voice Preview | Deepgram TTS |
| State Management | Zustand |
| UI Components | Radix UI, Lucide Icons, Chart.js |
git clone https://github.com/Aswin-Kumar7/AuraAI.git
cd AuraAI
# Install Node dependencies
npm install
# Set up Python environment
python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS/Linux
source .venv/bin/activate
pip install fastapi uvicorn firebase-admin python-dotenv numpy requests websocketscp .env.example .env.localFill in all the required API keys in .env.local. Refer to .env.example for documentation on each variable.
You need two terminals running simultaneously:
Terminal 1 — Next.js App (port 3000):
npm run devTerminal 2 — Transcription Server (port 3001):
# Activate your virtual environment first
python server/transcription_server.pyTwilio requires public HTTPS URLs to send webhooks. Use Cloudflare Tunnel (free, no account needed):
# Terminal 3 — Expose Next.js
cloudflared tunnel --url http://localhost:3000
# Terminal 4 — Expose WebSocket Server
cloudflared tunnel --url http://localhost:3001Then update your .env.local:
NGROK_URL=https://<your-nextjs-tunnel>.trycloudflare.com
WEBSOCKET_TUNNEL_URL=https://<your-ws-tunnel>.trycloudflare.comAnd configure your Twilio Phone Number webhook to point to:
https://<your-nextjs-tunnel>.trycloudflare.com/api/twilio/inbound
aura-ai/
├── public/ # Static assets
├── server/
│ └── transcription_server.py # Python FastAPI WebSocket STT server
├── src/
│ ├── app/
│ │ ├── (agent)/ # Agent portal pages
│ │ │ └── agent/
│ │ │ ├── dashboard/ # Live call copilot dashboard
│ │ │ ├── calls/ # Call history
│ │ │ ├── insights/ # Agent performance insights
│ │ │ ├── outbound/ # Outbound dialer
│ │ │ ├── phonebook/ # Contact management
│ │ │ └── profile/ # Agent profile
│ │ ├── (company)/ # Admin/company portal pages
│ │ │ └── company/
│ │ │ ├── dashboard/ # Company overview
│ │ │ ├── agents/ # Agent management & whitelisting
│ │ │ ├── analytics/ # Performance analytics
│ │ │ ├── audit/ # Compliance audit logs
│ │ │ └── setup/ # Company config & knowledge base
│ │ ├── (auth)/ # Authentication pages
│ │ └── api/ # API routes (all server-side)
│ ├── components/ # React components
│ │ ├── agent/ # Agent-specific components
│ │ ├── company/ # Admin-specific components
│ │ └── ui/ # Shared UI primitives (Radix)
│ ├── hooks/ # Custom React hooks
│ ├── lib/ # Utilities, Firebase, Groq, RAG
│ └── store/ # Zustand state management
├── .env.example # Environment variable template
├── next.config.ts # Next.js configuration
├── tailwind.config.ts # Tailwind CSS configuration
├── package.json
└── tsconfig.json
- All API keys are loaded via environment variables — no secrets in source code
- Firebase Admin SDK authenticates server-side operations
- API routes are protected with Firebase Auth token verification
- Role-based access control (Agent vs Admin) enforced in middleware
- Session management via secure httpOnly cookies
- Twilio webhook validation using auth tokens
This project is provided for educational and demonstration purposes.
Built with ❤️ by Aswin Kumar, Naveen D, Dhasarat Surya, and Kharshavarthan