fsdp

Train speculative decoding models effortlessly and port them smoothly to SGLang serving.

A scalable, agentic-first, and HuggingFace-native RL framework for research (9k lines).

Best practices & guides on how to write distributed pytorch training code

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

Repo for Qwen Image Finetune

A PyTorch native library for training speculative decoding models

Find why PyTorch training is slow, compare runs, and catch performance regressions in CI.

Write once, run anywhere; ezpz 🍋

A comprehensive hands-on guide to building production-grade distributed applications with Ray - from distributed training and multimodal data processing to inference and reinforcement learning.

Research platform for model training, evaluation, and experimentation across architectures, benchmarks, and recipes.

META LLAMA3 GENAI Real World UseCases End To End Implementation Guide

Fast and easy distributed model training examples.

Interactive exercises for understanding compute, communication, and dependencies in distributed LLM training traces.

Billus 大模型技能库：面向 LLM、VL、多模态与图像生成模型的训练、调参、剪枝、量化与工程化技能库

Forge kernels — fused Triton kernels for faster, leaner LLM fine-tuning. One-call patching into Hugging Face models via forge.patch(model), with FSDP2 multi-GPU support and a growing set of kernels and architectures. Every speedup backed by a committed benchmark. Apache-2.0.

A minimal, hackable pre-training stack for GPT-style language models

A script for training the ConvNextV2 on CIFAR10 dataset using the FSDP technique for a distributed training scheme.

Simple and efficient implementation of 671B DeepSeek V3 that trainable with FSDP+EP and minimal requirement of 256x A100/H100, targeted for HuggingFace ecosystem

A minimal, hackable pre-training stack for GPT-style language models