Attention is All You Need (https://papers.nips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer - T5 (https://arxiv.org/pdf/1910.10683)
GPT-2 paper (https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
GPT-3 paper (Language Models are Few-Shot Learners, https://arxiv.org/abs/2005.14165)
GPT-4 / MoE paper (Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, https://arxiv.org/abs/1701.06538)
RL - PPO & GRPO Blog (https://yugeten.github.io/posts/2025/01/ppogrpo/)
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (https://arxiv.org/pdf/2501.12948)
https://arxiv.org/html/2412.19437v1
https://api-docs.deepseek.com/news/news260424
Rethinking the role of demonstrations: What makes in-context learning work? (https://arxiv.org/pdf/2202.12837)
Finetuned Language Models are Zero-Shot Learners (https://arxiv.org/pdf/2109.01652)
Self-Consistency Improves Chain of Thought Reasoning in Language Models (https://arxiv.org/pdf/2203.11171)
Constitutional AI: Harmlessness from AI Feedback (https://arxiv.org/pdf/2212.08073)
Humanity's Last Exam (https://arxiv.org/pdf/2501.14249)
CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation (https://arxiv.org/pdf/2310.15638)
The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (https://arxiv.org/pdf/2506.06941)
Inverse Scaling in Test-Time Compute (https://arxiv.org/abs/2507.14417)
HALOGEN: Fantastic LLM Hallucinations and Where to Find Them (https://aclanthology.org/2025.acl-long.71.pdf)
Can LLMs Keep A Secret? Testing Privacy Implications via Contextual Integrity Theory (https://arxiv.org/pdf/2310.17884)
Extracting memorized pieces of (copyrighted) books from open-weight language models (https://arxiv.org/pdf/2505.12546)
Alignment Whack-a-Mole: Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models (https://arxiv.org/pdf/2603.20957)
Fine-Tuning LLMs to Be Better Tutors (RL w dynamic reward weights & simulated students+tutors) (https://aclanthology.org/2025.emnlp-main.15.pdf)
AI-Newton: A Concept-Driven Physical Law Discovery System without Prior Physical Knowledge (https://arxiv.org/pdf/2504.01538)
Mechanistic Interpretability
Induction heads (Nanda):
https://transformer-circuits.pub/2021/framework/index.html
https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html
Linear Probes (Hewitt):
https://arxiv.org/abs/1610.01644
https://www.cs.columbia.edu/~johnhew/interpreting-probes.html
Sparse AutoEncoders (Cunningham):
https://arxiv.org/abs/2309.08600
https://adamkarvonen.github.io/machine_learning/2024/06/11/sae-intuitions.html
Attribution Graphs:
https://transformer-circuits.pub/2025/attribution-graphs/biology.html
https://transformer-circuits.pub/2025/attribution-graphs/methods.html
Instant LLM Updates with Doc-to-LoRA and Text-to-LoRA
https://pub.sakana.ai/doc-to-lora/
AI Kernels, custom programming languages, from Triton to CUDA to ThunderKittens + Tiles, DSL, flash-attention, quantization
https://arxiv.org/abs/2410.20399
https://arxiv.org/abs/2603.05451
https://docs.nvidia.com/cutlass/latest/media/docs/pythonDSL/cute_dsl.html
https://arxiv.org/abs/2208.07339
https://arxiv.org/abs/2404.14469
https://arxiv.org/abs/2510.06189
Efficient Attention Mechanisms for Large Language Models: A Survey (https://arxiv.org/abs/2507.19595?utm_source=chatgpt.com)
https://vladfeinberg.com/assets/2025-04-24-princeton-talk.pdf
C/CUDA Tutorial (https://github.com/karpathy/llm.c)
Modern GPU Programming For MLSys https://mlc.ai/modern-gpu-programming-for-mlsys/#
GLM 5 Series from Z.ai
https://arxiv.org/abs/2602.15763
JAX Exercises
https://docs.jax.dev/en/latest/notebooks/thinking_in_jax.html (tutorial)
https://jax-ml.github.io/scaling-book/ (exercises)
From GPT-2 to Kimi 3 (https://x.com/waterloo_intern/status/2081762065392541951?s=51)