🧠 Mini LLM From Scratch

Build • Train • Generate • Learn

A complete educational implementation of a GPT-style Transformer Language Model

built completely from scratch using Python and PyTorch.

---

---

---

🎬 Live Project Preview

A short preview of the complete training and inference pipeline.

📺 Complete YouTube walkthrough will be added soon.

---

⭐ Why This Project?

Most tutorials explain what Transformers are.

Very few explain how to build one from scratch.

This repository was created to bridge that gap.

Instead of relying on high-level libraries such as Hugging Face, this project implements the core components manually to provide a clear understanding of how GPT-style language models work internally.

The implementation focuses on learning, clarity, and clean architecture, making it suitable for students, beginners, and developers interested in understanding modern Natural Language Processing systems.

---

🚀 What You'll Learn

By exploring this repository, you'll understand how a Transformer-based language model works internally.

✔ Build a custom tokenizer

✔ Create a vocabulary from raw text

✔ Encode and decode text

✔ Prepare datasets for next-token prediction

✔ Implement embeddings

✔ Add positional encoding

✔ Build self-attention

✔ Implement multi-head attention

✔ Create transformer blocks

✔ Train a GPT-style language model

✔ Save checkpoints

✔ Generate text autoregressively

---

✨ Project Highlights

🧠 GPT-style Architecture Complete Transformer implementation without external model libraries.	⚡ PyTorch Implemented using clean and modular PyTorch code.	🎓 Educational Designed to explain every important building block of modern LLMs.
📚 Beginner Friendly Simple code structure with detailed documentation.	🚀 Resume Ready Production-quality GitHub project suitable for portfolios.	🛠 Easily Extendable Can be expanded with BPE, RoPE, Flash Attention, GPU training, and more.

---

📑 Repository Navigation

Section	Description
🎥 Demo	Project demonstration
✨ Features	Major project capabilities
🏗 Architecture	Internal Transformer design
⚙️ Workflow	End-to-end pipeline
📂 Folder Structure	Project organization
🚀 Installation	Setup instructions
💻 Usage	Training & Inference
📸 Screenshots	Project walkthrough
📊 Configuration	Hyperparameters
🔮 Future Work	Planned improvements
👨‍💻 Author	About the developer

---

🏗 Architecture

The Mini LLM follows the same high-level pipeline used by modern Transformer-based language models.

flowchart LR

A[Raw Text Corpus]
-->B[Tokenizer]

B
-->C[Vocabulary]

C
-->D[Token IDs]

D
-->E[Embedding Layer]

E
-->F[Positional Encoding]

F
-->G[Transformer Blocks]

G
-->H[Linear Layer]

H
-->I[Softmax]

I
-->J[Next Token Prediction]

Loading

---

⚙️ Complete Workflow

The project is divided into small modular stages so each component can be understood independently.

flowchart TD

A[Load Corpus]

-->B[Build Vocabulary]

-->C[Encode Text]

-->D[Create Dataset]

-->E[Initialize Transformer]

-->F[Train Model]

-->G[Save Checkpoint]

-->H[Load Model]

-->I[Generate Text]

Loading

---

🧩 Core Components

Component	Purpose
🔤 Tokenizer	Converts raw text into numerical token IDs
📚 Vocabulary Builder	Creates vocabulary from the corpus
📄 Dataset	Generates input-target pairs
🔢 Embedding Layer	Learns dense word representations
📍 Positional Encoding	Preserves sequence order
🎯 Self Attention	Learns contextual relationships
👀 Multi-Head Attention	Captures multiple semantic patterns simultaneously
⚡ Feed Forward Network	Improves feature representation
🏗 Transformer Block	Combines Attention + FFN + LayerNorm
💾 Checkpoint	Saves trained model weights
💬 Inference	Generates text token-by-token

---

📂 Project Structure

Mini-LLM-From-Scratch/

├── assets/
│
├── checkpoints/
│
├── data/
│
├── tokenizer/
│
├── training/
│
├── model/
│
├── inference/
│
├── utils/
│
├── main.py
│
├── requirements.txt
│
└── README.md

---

📁 Folder Explanation

Folder	Description
assets	Images, GIFs, Banner & Documentation
checkpoints	Saved Model Weights
data	Training Corpus
tokenizer	Vocabulary Builder & Tokenizer
training	Dataset & Training Pipeline
model	Transformer Implementation
inference	Text Generation
utils	Configuration Files
main.py	Demo Programs

---

📸 Project Walkthrough

The following screenshots show each stage of the complete pipeline.

---

📂 Project Structure	🔤 Vocabulary Building
📊 Dataset Preparation	⚙️ Model Configuration
🚀 Training Started	✅ Training Complete
💬 Text Generation

---

📈 Training Pipeline

flowchart LR

Corpus

-->Tokenizer

-->Vocabulary

-->Encoding

-->Dataset

-->DataLoader

-->Embedding

-->Transformer

-->CrossEntropyLoss

-->Backpropagation

-->AdamOptimizer

-->Checkpoint

Loading

---

💬 Inference Pipeline

flowchart LR

Prompt

-->Tokenize

-->LoadCheckpoint

-->Transformer

-->PredictNextToken

-->AppendToken

-->Repeat

-->GeneratedText

Loading

---

🎯 Design Principles

This project was designed with the following goals in mind.

• ✅ Clean and Modular Code
• ✅ Beginner Friendly
• ✅ Easy to Understand
• ✅ Well Documented
• ✅ Resume Ready
• ✅ Open Source
• ✅ Easy to Extend
• ✅ Educational Implementation

---

🧠 Skills Demonstrated

This repository demonstrates practical knowledge of:

• Python Programming
• PyTorch
• Neural Networks
• Transformer Architecture
• Self Attention
• Multi Head Attention
• Deep Learning
• Natural Language Processing
• Language Modeling
• Software Engineering
• Git & GitHub
• Documentation

---

🚀 Getting Started

Follow the steps below to set up and run the project on your local machine.

---

📋 Prerequisites

Before running the project, make sure the following software is installed.

Software	Version
Python	3.10+
Git	Latest
pip	Latest
VS Code (Recommended)	Latest

Verify your installation.

python --version

pip --version

---

📥 Clone Repository

git clone https://github.com/ENAYATULLA/Mini-LLM-From-Scratch.git

cd Mini-LLM-From-Scratch

---

📦 Create Virtual Environment

Windows

python -m venv venv

venv\Scripts\activate

Linux / macOS

python3 -m venv venv

source venv/bin/activate

---

📚 Install Dependencies

pip install -r requirements.txt

---

▶ Running the Project

The complete pipeline can be executed in the following order.

Step	Description
1	Build Vocabulary
2	Prepare Dataset
3	Configure Model
4	Train Model
5	Generate Text

---

🔤 Step 1 — Build Vocabulary

Run

python main.py

Expected Output

The tokenizer scans the corpus, builds a vocabulary, assigns unique IDs to every token, and performs text encoding and decoding.

---

📊 Step 2 — Dataset Preparation

Run

python main.py

Expected Output

The dataset creates Input–Target pairs for next-token prediction using a sliding window approach.

---

⚙ Step 3 — Model Configuration

Run

python main.py

Expected Output

The configuration defines all model hyperparameters including embedding size, transformer layers, attention heads, sequence length, learning rate, and device.

---

🏋 Step 4 — Train the Model

Run

python -m training.train

Training Output

After successful training

The trained weights are automatically stored inside the checkpoints/ directory.

---

💬 Step 5 — Generate Text

Run

python -m inference.generate

Example Prompt

Artificial Intelligence

Example Output

The model predicts one token at a time using autoregressive next-token prediction.

---

📊 Current Model Configuration

Hyperparameter	Value
Architecture	GPT-style Transformer
Embedding Dimension	64
Attention Heads	4
Transformer Layers	2
Tokenizer	Word Level
Framework	PyTorch
Training	Next Token Prediction
Device	CPU

---

📈 Learning Outcomes

This project demonstrates practical implementation of:

• GPT-style Language Models
• Transformer Architecture
• Word Embeddings
• Positional Encoding
• Self Attention
• Multi Head Attention
• Feed Forward Networks
• Layer Normalization
• Residual Connections
• Cross Entropy Loss
• Backpropagation
• Autoregressive Text Generation
• PyTorch Model Development
• Software Engineering Best Practices

---

🛣 Future Roadmap

The following improvements are planned.

• Byte Pair Encoding (BPE)
• SentencePiece Tokenizer
• Rotary Positional Embeddings (RoPE)
• Flash Attention
• GPU Training
• Mixed Precision Training
• Beam Search
• Top-k Sampling
• Top-p Sampling
• Temperature Sampling
• Streamlit Web Interface
• Hugging Face Integration
• Larger Training Dataset

---

🤝 Contributing

Contributions are always welcome.

If you'd like to improve this project:

Fork Repository

↓

Create Feature Branch

↓

Commit Changes

↓

Push Branch

↓

Open Pull Request

---

⭐ Support

If this repository helped you learn something new,

please consider giving it a ⭐ on GitHub.

Your support motivates future improvements.

---

👨‍💻 About the Author

Enayat Ullah B.Tech Computer Science Engineer Passionate about • Artificial Intelligence • Machine Learning • Deep Learning • Natural Language Processing • Large Language Models • Python • PyTorch

---

🌐 Connect With Me

---

⭐ If you found this project useful, consider giving it a star.

Thank you for visiting the repository.

Made with ❤️ using Python & PyTorch