Triton Kernel Study Group

Introduction

Triton crash course prepared for beginners.

Outline

0. Introduction to GPU architecture
1. Write a simple CUDA kernel: softmax
2. Introduction to Triton and Triton softmax kernel
3. Tensor Core and Triton matrix multiplication
4. Debugging kernels using NVIDIA NCU
5. Flash-Attention algorithm
6. Triton Flash-Attention kernels (fwd & bwd)
7. Triton kernel examples #1
8. Triton kernel examples #2