A Retrieval-Augmented Generation (RAG) system built over Introduction to Machine Learning with Python by Andreas C. Müller and Sarah Guido.
The goal of this project was to understand and implement a complete RAG pipeline from scratch, starting from a textbook PDF and ending with grounded answers generated by an LLM. Instead of asking the language model to answer a question directly, the system first retrieves relevant passages from the textbook and then gives those passages to Gemini as context for generating the answer.
Retrieval-Augmented Generation combines information retrieval with language generation. A normal language model answers a question using only the knowledge encoded in its parameters. A RAG system instead retrieves relevant information from an external source first, then uses that information to generate the answer.
This project follows a simple pipeline:
User Question -> Query Embedding -> Vector Search -> Top-k Relevant Passages
-> Context Construction -> Gemini -> Grounded Answer
The knowledge source here is a machine learning textbook.
The project has two stages.
Indexing — the textbook is processed once to build a searchable vector database:
Textbook PDF -> PyMuPDF -> Text Blocks -> Cleaning -> Heading-aware Chunking
-> Sentence Transformer Embeddings -> ChromaDB
Querying — when a user asks a question:
User Question -> Query Embedding -> ChromaDB Similarity Search -> Top-k Chunks
-> Context Construction -> Gemini -> Grounded Answer
- PDF text extraction using PyMuPDF, with block-based extraction
- Removal of page-number and footer blocks
- Basic text cleanup
- Heading-aware chunking, capped at roughly 200 words per chunk
- Sentence embeddings using
all-MiniLM-L6-v2 - Persistent ChromaDB vector database
- Top-k similarity retrieval with page metadata attached to each chunk
- Context construction from retrieved passages
- Gemini-powered, prompt-grounded answer generation
- Refusal when the retrieved context is insufficient to answer
- Retrieval evaluation against a manually built test set
rag-ml-book/
│
├── Introduction to Machine Learning.pdf
│
├── build_index.py
├── RAG.py
├── evaluation.py
├── requirements.txt
├── README.md
│
└── chroma_db/
build_index.py builds the vector database. It opens the textbook PDF, extracts text blocks per page, removes footer and page-number blocks, cleans the extracted text, detects headings by font size, forms chunks, generates embeddings, and stores everything (chunks, embeddings, page metadata) in ChromaDB.
RAG.py runs the query-time pipeline. It loads the embedding model, connects to the persistent ChromaDB database, embeds the user's question, retrieves the top-k most similar chunks, builds a context from them, constructs a grounding prompt, sends it to Gemini, and returns the generated answer.
evaluation.py holds a small, manually built evaluation set used to check retrieval quality. It currently covers 10 questions spanning topics such as overfitting, k-nearest neighbors, L1/L2 regularization, cross-validation, decision trees, training and test data, feature selection, linear regression, and validation sets.
chroma_db/ is the persistent ChromaDB storage: processed textbook chunks, their embeddings, and associated metadata.
| Component | Technology |
|---|---|
| Programming Language | Python |
| PDF Extraction | PyMuPDF |
| Embedding Model | all-MiniLM-L6-v2 |
| Vector Database | ChromaDB |
| LLM | Gemini |
| Environment Variables | python-dotenv |
Clone the repository:
git clone https://github.com/RustyLegend/rag-from-scratch-v1.0
cd rag-from-scratch-v1.0Install dependencies:
pip install -r requirements.txtRequired packages:
pymupdf
sentence-transformers
chromadb
google-genai
python-dotenv
The project uses Gemini for answer generation. Create a .env file in the project directory:
GEMINI_API_KEY=your_api_key_here
Do not commit your .env file or API key to GitHub. Add this to .gitignore:
.env
The knowledge base used by this project is Introduction to Machine Learning with Python by Andreas C. Müller and Sarah Guido. The PDF is included in this repository to demonstrate the pipeline end to end.
The textbook is copyrighted material. Redistribute the PDF only if permitted by the applicable license or with permission from the rights holder.
The repository includes a pre-built ChromaDB database generated from the textbook, containing text chunks, their embeddings, and each chunk's starting and ending page number. The current index holds roughly 595 chunks, each represented by a 384-dimensional embedding, so you can clone the repository and start asking questions without rebuilding the index. See below if you want to rebuild it yourself.
If the provided ChromaDB database is available, run:
python RAG.pyYou'll be prompted:
Enter a query:
For example:
What is overfitting?
The system retrieves relevant passages from the textbook, passes them to Gemini, and returns an answer generated from that retrieved content.
To recreate the ChromaDB database from the PDF:
python build_index.pyThis runs the full indexing pipeline: PDF -> text extraction -> block cleaning -> chunk creation -> embedding generation -> ChromaDB storage.
Once the index exists, run python RAG.py as usual. You do not need to run build_index.py every time you ask a question — RAG.py simply connects to the persistent database and retrieves from it.
One of the main goals of this project was understanding why chunking matters in RAG. The PDF isn't treated as one large block of text. Text is first extracted as blocks, each carrying its own text, page number, and font size. Font size is used as a simple heuristic for detecting headings, and a heading starts a new chunk when appropriate. Chunks are also capped at roughly 200 words.
Conceptually: a heading is followed by its related paragraphs until the ~200-word limit is reached, at which point the next chunk begins. This keeps related information together while preventing chunks from growing unwieldy.
The project uses all-MiniLM-L6-v2 from Sentence Transformers. Each text chunk is converted into a 384-dimensional vector, and the user's question is embedded with the same model so the two can be compared directly.
ChromaDB performs the vector similarity search. The system retrieves k = 5 passages by default. For each retrieved passage it keeps the text, the similarity distance, and the starting and ending page numbers, then combines these into the context given to Gemini.
After retrieval, the passages are inserted into a prompt that instructs Gemini to use only the retrieved passages, answer clearly and concisely, cite the relevant page or page range, avoid unsupported claims, and state explicitly when the retrieved passages don't contain enough information to answer.
The overall flow: question -> retrieve relevant textbook passages -> build a prompt around them -> Gemini -> grounded answer.
A core part of this project was preventing the LLM from answering from its own knowledge instead of the retrieved text. The prompt explicitly instructs the model not to use information outside the retrieved passages.
For example, when asked "What is the capital of France?", the retrieved textbook context contained nothing relevant, so the system refused to answer. Similarly, when retrieval was deliberately restricted so the needed information was missing, the model reported that the answer couldn't be determined from the retrieved passages.
This illustrates something important about RAG: retrieval quality directly determines what information is available to the generator. A capable generator still can't answer correctly from information it was never given.
A manually constructed set of 10 questions was used to evaluate retrieval. For each question, the top 5 retrieved passages were checked for relevance.
Evaluation questions: 10
k: 5
Recall@5: 100%
For all 10 questions, at least one relevant passage appeared within the top 5 results. This is a small, manually built test set, not proof that the retriever is perfect — it simply shows the current pipeline performed well on this particular set of questions.
Different values of k were tried during development. A smaller k gives less context; a larger k gives more context but risks pulling in redundant or less relevant passages. k = 1 can be very focused but may miss supporting information, while k = 5 generally provided enough relevant context for the questions tested. Pushing k higher tends to add passages that are only loosely related to the question. This is the basic trade-off in choosing retrieval depth.
The system was deliberately tested against situations where RAG tends to break down.
Out-of-corpus question — a question unrelated to the textbook produced no supporting context, and the system correctly refused to answer.
Ambiguous question — a question like "How does it work?" doesn't specify what "it" refers to. The system recognized the context was insufficient to determine an answer.
Insufficient retrieval — the L1/L2 regularization question was tested with only one retrieved passage, which didn't contain enough information to fully answer it. Under the strict grounding instructions, the model refused rather than filling the gap from its own knowledge.
This project intentionally keeps the pipeline simple.
- No reranking — the initial vector search results go straight to the LLM, with no second-stage reranker.
- No hybrid search — retrieval relies on vector similarity alone, without combining it with keyword-based search.
- Static corpus — the knowledge base is a fixed textbook; adding or changing documents means rebuilding or updating the index.
- Simple chunking — chunking is based on PDF blocks, font size, and a word-count cap. More advanced approaches could account for semantic boundaries, overlap, document structure, or token counts.
- Small evaluation set — only 10 manually selected questions have been evaluated so far. A larger, more systematic evaluation set would give a stronger picture of retrieval quality.
- PDF extraction artifacts — extraction isn't perfect; some formatting and spacing issues remain, particularly around complex layouts and code examples.
- No citation verification — page metadata is provided to the LLM, but the citation it produces is generated by the model and not independently verified.
Semantic chunking, chunk overlap, hybrid keyword and vector retrieval, retrieval reranking, better PDF structure detection, automatic evaluation datasets, retrieval metrics beyond Recall@5, citation verification, streaming responses, a web interface, conversation history, support for multiple documents, and document-level metadata filtering.
This project was built to understand RAG from the ground up, rather than to use a library and move on. Along the way it covered why language models need external retrieval, what embeddings represent, cosine and vector similarity, why chunking affects retrieval quality, how pretrained embedding models get used in practice, how a vector database stores and retrieves embeddings, how top-k retrieval works, how retrieved context gets passed to an LLM, why grounding instructions matter, how poor retrieval limits generation regardless of how capable the LLM is, and how to evaluate retrieval separately from generation.
The source code in this project is for educational purposes. Introduction to Machine Learning with Python and its contents are copyrighted by their respective rights holders. Including the textbook or extracted textbook content in this repository does not imply ownership of that material. Verify you have the appropriate rights before using or redistributing the textbook or the generated database.