llm-from-scratch

Build an LLM from scratch with transformers, attention, tokenization, dataset engineering, training, alignment, inference, multimodality, agents, and a runnable PyTorch mini language model.

Implementations, assignments, and experiment notes for Stanford CS336: Language Modeling from Scratch.

Building a complete LLM from scratch. No libraries. No shortcuts. Just NumPy, math, and first principles.

LLM Which Was Trained On A Consumer GPU

An industry-grade, production-ready Multi-Modal AGI Vision-Language LLM & Agentic Reasoning Framework built from scratch in PyTorch. Features ViT patch encoding, RoPE, SDPA Flash-Attention, KV-Cache, autonomous Plan-Act-Reflect agentic loop with tool dispatch & Google Colab T4 GPU support.

DeepSeek-style MoE + MLA from scratch in PyTorch, with a router-specialization probe (mutual information) and a dense/+MoE/+MLA ablation. Educational, nano-scale, measured.

From-scratch PyTorch: frontier LLM techniques as of 2026-Q1 — the Muon optimizer and Multi-Token Prediction, plus a base BPE tokenizer. Self-contained, self-checking modules.

A large language model built from scratch using PyTorch. Currently, it is in the initial stage of the development process using PyTorch.🔧

Some attempts, based on MicroGPT.

From-scratch PyTorch GPT: the GPT-2 skeleton plus the components of current open LLMs — RMSNorm, RoPE, SwiGLU, GQA, no-bias + tied embeddings. Each isolated and self-checking.

面向只有 Python 基础学习者的中文 LLM 从零学习伴侣，配合 rasbt/LLMs-from-scratch，含课程、实验、CLI、个性化计划与测试。

GRAFT (Grounded Reasoning-Action File-agent Toolchain): a from-scratch ReAct implementation across Groq's OpenAI-compatible API and the Anthropic Messages API, with persistent memory, sandboxed calculation, and zero-hallucination answering.

Building a Large Language Model from the ground up.

A from-scratch implementation of a Large Language Model in Python, built while following freeCodeCamp's Create a Large Language Model from Scratch with Python – Tutorial.

An ~8M parameter LLM that talks like a tiny robot vacuum.