The TechNova Internal Knowledge Assistant is a Retrieval-Augmented Generation (RAG) system designed to deliver accurate, policy-compliant, and source-grounded answers to support queries about TechNova’s products, policies, FAQs, support tickets, and company information. Built for the Certified AI Engineer Professional Program, this project demonstrates advanced RAG techniques, robust evaluation, and strict compliance with academic integrity.
- Strict Grounding: All answers are derived exclusively from the provided knowledge base context. No hallucination or external knowledge is permitted.
- Policy Precision: Policy-related answers quote exact language from source documents, ensuring compliance and traceability.
- Exhaustive Data Extraction: All numerical values, timeframes, and technical details are explicitly included in responses.
- Direct/Binary Answers: Yes/No questions are answered with clear, direct statements as required by the rubric.
- Citation Protocol: Every factual claim is accompanied by a precise source file citation (e.g.,
source: knowledge-base/products/nova_phone_x1.md). - Category Filtering: Users can filter queries by category (Products, Policies, FAQs, Support Tickets, Company) for targeted retrieval.
- Automated Evaluation: The system includes an automated evaluation pipeline to measure answer quality and compliance with rubric requirements.
- User-Friendly Interface: Gradio-based chat interface for interactive multi-turn conversations.
- Document Ingestion:
- All knowledge base files are loaded from the
notebooks/knowledge-base/directory, covering products, policies, FAQs, support tickets, company info, and release notes. - Documents are chunked with overlap and tagged with metadata (
source,category) for precise retrieval and citation.
- All knowledge base files are loaded from the
- Vector Store:
- Uses Chroma for persistent, efficient vector similarity search.
- Embeddings generated with HuggingFace
all-MiniLM-L6-v2for local, cost-effective operation.
- Retrieval Pipeline:
- Supports category filtering and top-k retrieval.
- Ensures only relevant context is passed to the LLM.
- LLM Integration:
- Uses LangChain and Groq for LLM orchestration (e.g., Llama 3, GPT-4.1-nano).
- System prompt enforces grounding, citation, and refusal protocol.
- Evaluation Harness:
- Automated rubric-based evaluation using
questions.jsonand must-mention term validation. - Results stored in
eval_results.jsonand summarized inREPORT.md.
- Automated rubric-based evaluation using
The knowledge base is organized as follows:
company/— About, careers, contact, sustainabilityfaqs/— Account, payments, returns, shipping, software updates, trade-in, troubleshooting, warrantypolicies/— Privacy, return, shipping, terms of service, warrantyproducts/— Nova Earbuds, Laptop Pro, Phone X1, Smartwatch, Speaker, Tabletrelease_notes/— NovaOS and NovaWatch OS updatessupport_tickets/— Example support tickets for retrieval testing
- Final RAG Score: 82.75% (25/29 Passed)
- Evaluation Date: May 30, 2026
- Evaluation Method: Automated rubric-based testing using
eval_results.jsonand must-mention term validation. Each answer was checked for correctness, required keyword presence, and proper source citation.
Summary:
- The RAG system was evaluated using a set of 29 questions covering all major categories.
- Out of 29 questions, 25 were answered correctly according to the rubric and must-mention criteria.
- The system demonstrates strong grounding, accurate citation, and robust category filtering.
- Two questions did not fully meet the must-mention requirements, indicating minor areas for improvement in context extraction or answer formulation.
Recommendations:
- Review failed cases for possible improvements in context retrieval or prompt engineering.
- Consider tuning the retriever parameters or refining the prompt for stricter adherence to must-mention terms.
See REPORT.md for a full evaluation summary and recommendations.
- Python 3.9+
- Recommended: Create a virtual environment
pip install -r requirements.txt# Launch the Gradio chat interface
python main.py
# Or run the evaluation pipeline in the notebookmain.py— Main entry point for the assistantnotebooks/— Development, evaluation, and rubric notebooksnotebooks/knowledge-base/— Source documents for retrievalnotebooks/eval_results.json— Automated evaluation resultsREPORT.md— Detailed evaluation reportrequirements.txt— Python dependencies
- LangChain — Document loading, chunking, and retrieval pipeline
- Chroma — Vector store for similarity search
- HuggingFace Transformers — Local embedding generation
- Gradio — User interface
- Groq / OpenAI — LLM orchestration
This project is part of the Certified AI Engineer Professional Program conducted by Noob Dev.
Strict Confidentiality & Academic Integrity Notice:
All project files — including this README, source code, the knowledge base, the evaluation dataset, and any related materials — are the exclusive intellectual property of Noob Dev and are provided solely for the personal use of enrolled program participants.
You may NOT:
- Share, copy, redistribute, or publish any part of this project in any form
- Upload it to public repositories (GitHub, GitLab, Hugging Face, Kaggle, etc.) or to AI training datasets
- Forward it to non-enrolled individuals, study groups outside the cohort, or online communities
- Use it for commercial purposes outside the program
- Submit it as your own work to any other course, bootcamp, employer, or platform
Violations constitute academic misconduct and a breach of your program enrollment agreement. Consequences include:
- Immediate termination from the Certified AI Engineer Professional Program
- Forfeiture of all tuition fees, course progress, certifications, and program credentials
- Legal action under applicable copyright, trade-secret, and intellectual property laws
By opening, downloading, or using these materials, you acknowledge and accept these terms.
- Kavindu Chamod
This project is for academic purposes only. All rights reserved by Noob Dev.