🐾 Nova — AI Companion with Google ADK & Gemini

A talking, animated AI companion that runs in the browser. You type a message; Nova, an AI career mentor built with Google's Agent Development Kit (ADK) and Gemini, thinks it over, searches the web when needed, and answers out loud while her mouth moves in sync with her words.

✨ Features

• 🧠 Agent "brain" with Google ADK: an LlmAgent powered by Gemini Flash handles every reply.
• 🎭 Custom persona: Nova is a sharp, witty and encouraging career mentor who gives concise, practical tech-career advice.
• 🔍 Grounded in real-time information: the built-in google_search tool lets Nova answer questions about recent events instead of relying only on training data.
• 💬 Conversation memory: each browser tab gets its own session ID, so Nova remembers the context of the conversation.
• 🗣️ Voice output: replies are spoken through the browser's Web Speech API, with a selectable Google voice.
• 👄 Lip-sync animation: the avatar switches between mouth-open and mouth-closed frames while speaking, with a typewriter effect for the text.
• 🎨 AI-generated avatar: a custom anime-style character created with generative AI.

🏗️ Architecture

flowchart LR
    U[👤 User types a message] --> F[🌐 Frontend<br/>HTML · CSS · JS]
    F -- "POST /chat<br/>{message, session_id}" --> B[🐍 Flask backend<br/>async route]
    B --> R[ADK InMemoryRunner<br/>per-session memory]
    R --> A[🧠 LlmAgent 'Nova'<br/>Gemini Flash]
    A <-.-> S[🔍 google_search tool]
    A --> R --> B
    B -- "{response}" --> F
    F --> T[🗣️ Speech synthesis]
    F --> L[👄 Lip-sync animation]

Loading

The backend is the brain and the frontend is the face. Because the agent sits behind a simple /chat API, any other interface (a mobile app, a CLI, a 3D avatar) could reuse the same brain without changes.

🧠 The Agent

from google.adk.agents.llm_agent import LlmAgent
from google.adk.tools import google_search

root_agent = LlmAgent(
    model='gemini-flash-latest',
    name='companion_agent',
    tools=[google_search],
    instruction="""You are Nova, a sharp, witty, and encouraging AI career mentor
    who helps people grow in tech and their careers. ...
    Keep answers concise and punchy, no more than 3 sentences.""",
)

Persona design:

• Personality: confident, warm and a little playful, like a senior engineer who has mentored hundreds of people
• Style: practical, real-world advice rather than generic platitudes, plus a memorable one-liner when relevant
• Rules: at most 3 sentences (so spoken replies stay short) and no emoji (they don't read well aloud)

📁 Project Structure

├── app.py                # Flask server: serves the UI and the /chat endpoint
├── character.py          # Nova's ADK agent: model, persona, tools
├── templates/
│   └── index.html        # Chat interface
├── static/
│   ├── app.js            # Chat logic, speech synthesis, lip-sync
│   ├── style.css         # Dark UI theme
│   └── images/           # Avatar frames (mouth open / closed)
├── requirements.txt
└── .env.example          # Template for your Gemini API key

🚀 Run It Locally

1. Get a free Gemini API key from Google AI Studio.

2. Set up the project:

git clone https://github.com/MMujtabaX/Real-Time-Companion.git
cd Real-Time-Companion

python -m venv .venv
.venv\Scripts\activate          # Windows
# source .venv/bin/activate     # macOS / Linux

pip install -r requirements.txt

3. Add your API key. Copy .env.example to .env and paste your key:

GEMINI_API_KEY=your-api-key-here

4. Start the app:

python app.py

Open http://localhost:5000, turn your speakers on, and say hi to Nova. Chrome works best, since it ships the Google voices the app uses.

🛠️ Tech Stack

Layer	Technology
Agent framework	Google Agent Development Kit (ADK)
LLM	Gemini Flash (via Google AI Studio)
Tools	ADK built-in google_search
Backend	Python, Flask (async)
Frontend	HTML, CSS, vanilla JavaScript
Voice	Web Speech API (speechSynthesis)

🗺️ Roadmap: Toward a Real-Time 3D Companion

The next version replaces the 2D image swap with a fully real-time, voice-driven experience:

• 3D avatars rendered with three.js (Ready Player Me / VRM models)
• Face tracking: MediaPipe Face Landmarker drives my avatar from the webcam using ARKit blendshapes
• Speech-to-text with faster-whisper, so I can talk to Nova out loud
• Server-side text-to-speech, with Nova's jaw driven by live audio amplitude (Web Audio AnalyserNode → jawOpen morph target)
• Both avatars in one scene, with smoothing and idle blinks
• Retry handling for API rate limits (HTTP 429)

🙏 Acknowledgements

Built during a Google Developer Groups (GDG) workshop, starting from Google's codelab Build Your First AI Companion with ADK and its starter project (Apache 2.0). I then gave the companion its own persona, avatar and grounding, and moved it from Cloud Shell to local development.

👤 Author

Muhammad Mujtaba Khan Suri — CS @ UBIT, University of Karachi GitHub