MAITRI

A Local Retrieval-Augmented AI Companion for Context-Aware Conversations

MAITRI is a fully local Retrieval-Augmented Generation (RAG) AI companion designed for emotionally intelligent, context-aware conversations while preserving user privacy.

The project combines a local Large Language Model (Qwen), semantic retrieval using ChromaDB, long-term conversational memory, and structured prompt engineering to create an AI assistant capable of maintaining context across conversations without relying on cloud services.

Status: 🚧 Active Development

---

Features

• 🔒 Fully Offline AI Assistant
• 🧠 Retrieval-Augmented Generation (RAG)
• 💬 Long-Term Conversational Memory
• 📚 Semantic Search using ChromaDB
• ⚡ Streaming Token Generation
• 🚀 GPU Accelerated Embeddings (CUDA)
• 📖 Token-Aware Document Chunking
• 📝 Session Logging using JSONL
• 🧩 Modular Pipeline Architecture
• ❤️ Emotion-Aware Prompt Engineering
• 🔍 Semantic Memory Retrieval
• 🛡️ Crisis Detection & Safety Handling

---

Architecture

User
   │
   ▼
Maitri Chat
   │
   ▼
Analyst Prompt
   │
   ▼
Retriever
   │
   ├──────────────► ChromaDB
   │                  │
   │                  ▼
   │           Relevant Memories
   │
   ▼
Context Builder
   │
   ▼
Qwen Local LLM
   │
   ▼
Streaming Response
   │
   ▼
Conversation Logger
   │
   ├────────► JSONL History
   │
   └────────► Session Summary
                      │
                      ▼
              Memory Indexer
                      │
                      ▼
                  ChromaDB

---

Project Structure

.
├── builder.py
├── clean_memory.py
├── config.py
├── data
├── embedding
│   ├── embedder.py
│   └── model.py
├── history
├── LLM
│   └── llm_engine.py
├── loader.py
├── maitri_chat.py
├── models.py
├── pipeline
│   ├── indexing.py
│   ├── querying.py
│   └── rag_pipeline.py
├── retrival
│   ├── context_builder.py
│   ├── router.py
│   └── search_service.py
├── src
│   └── chunker.py
├── system_prompt.py
├── vectordb
│   ├── base.py
│   ├── chroma_store.py
│   └── retriever.py
└── vector_db
    └── chroma.sqlite3

---

Technology Stack

Component	Technology
Language	Python 3.13+
LLM	Qwen 2.5 (Ollama)
Vector Database	ChromaDB
Embedding Model	SentenceTransformers
Chunking	tiktoken
Storage	JSON / JSONL
Hardware	NVIDIA CUDA (optional)

---

How It Works

1. User sends a message.
2. The Analyst prompt classifies the conversation.
3. Relevant memories are retrieved from ChromaDB.
4. Context Builder prepares the final prompt.
5. Qwen generates a streamed response.
6. Conversation is logged.
7. Session summaries are periodically indexed into the vector database for future retrieval.

---

Key Improvements

Streaming Generation

Responses are streamed token-by-token instead of waiting for the entire generation to finish.

---

Long-Term Memory

Past conversations are summarized, embedded, and stored inside ChromaDB, allowing MAITRI to retrieve relevant memories using semantic similarity.

---

Token-Aware Chunking

Documents are split according to tokenizer limits instead of arbitrary word counts, improving retrieval quality.

---

Efficient Logging

Conversation history uses append-only JSONL storage for scalable and efficient logging.

---

GPU Support

Sentence embeddings automatically utilize CUDA when available.

---

Current Limitations

• High inference latency depending on hardware
• Memory summarization can still be improved
• Prompt engineering is continuously evolving
• Multi-user support is not yet implemented
• No graphical user interface (CLI only)

---

Future Roadmap

• Reduce inference latency
• Hybrid retrieval (BM25 + Vector Search)
• Memory ranking and decay
• Function calling
• Voice interaction
• Web interface
• Agentic workflow
• Multi-modal support
• Quantized model benchmarking
• Docker deployment

---

Running the Project

Clone the repository

git clone https://github.com/<your_username>/<repository>.git
cd <repository>

Create a virtual environment

python -m venv venv

Activate it

Linux

source venv/bin/activate

Install dependencies

pip install -r requirements.txt

Start Ollama

ollama serve

Pull Qwen if necessary

ollama pull qwen2.5:7b

Run the chatbot

python maitri_chat.py

---

Performance

Current observations

Metric	Value
Embedding	GPU Accelerated
Memory Search	ChromaDB
Response Streaming	Yes
Average Latency	3–75 seconds
Offline Support	Yes

---

Motivation

This project was created to explore Retrieval-Augmented Generation, prompt engineering, semantic memory, and local Large Language Models while building an AI companion that prioritizes privacy and contextual understanding.

The primary objective is educational: to gain hands-on experience designing scalable AI systems rather than relying solely on existing frameworks.

---

License

This project is licensed under the MIT License.

---

Author

Vicky

Bachelor's Student — Artificial Intelligence & Machine Learning

Interested in:

• Artificial Intelligence
• Retrieval-Augmented Generation
• Robotics
• Computer Vision
• Deep Learning
• Large Language Models

---

⭐ If you found this project interesting, consider giving it a star.