A runnable, production-grade demonstration of data-parallel language-model training with PyTorch DistributedDataParallel (DDP). Run a small Transformer without downloading a model, or fine-tune Qwen2.5-0.5B-Instruct with LoRA on a CUDA GPU. A standalone Jupyter notebook recreates the training script and launches it with torchrun.
Scope: This is a training demonstration, not an LLM serving application. Multi-GPU Qwen runs are production-grade examples.
- Tiny causal Transformer built in PyTorch: byte-level embeddings, learned positions, causal attention, next-token loss; no Hugging Face download.
- Qwen LoRA training:
Qwen/Qwen2.5-0.5B-Instructwith PEFT adapters targeting query and value projections. - Data parallelism:
torchrunlaunches one process per worker;DistributedSamplerpartitions examples; DDP synchronizes gradients with AllReduce. - CPU demonstration: run 2 processes with Gloo, even on computers with a single GPU or no CUDA GPU.
- Single-and multi-GPU launch: use 1 GPU or a Linux notebook environment with at least 2 CUDA GPUs.
- Custom corpus: read any UTF-8 text file with one sample per line using
--corpus-file. - Model saving: rank 0 writes the checkpoint (tiny mode) or LoRA adapters (Qwen mode).
.
├── README.md
├── requirements.txt
├── .gitignore
├── llm_ddp_parallel_training.py
└── LLM_DDP_Parallel_Training.ipynb
The notebook is self-contained: it writes the full Python training file and requirements file to the notebook's working directory. Training runs in subprocesses rather than directly inside the Jupyter kernel, avoiding notebook multiprocessing / argparse problems.
- Python: 3.10+ (Python 3.12 recommended on Windows).
- CPU mode: any computer with a supported PyTorch installation.
- CUDA mode: a compatible NVIDIA driver and CUDA-enabled PyTorch; check with
torch.cuda.is_available(). - Two GPUs: only if two NVIDIA GPUs are available to your runtime (for example, a suitable Kaggle dual-GPU session or a dual-GPU Linux workstation).
- Google Colab: commonly provides one GPU; use
--nproc_per_node=1unlesstorch.cuda.device_count()reports 2 or more. - Kaggle : commonly provides two GPUs; use
--nproc_per_node=1unlesstorch.cuda.device_count()reports 2 or more.
DDP makes a complete model replica on each GPU. Two 4 GB GPUs do not provide an 8 GB combined model-memory pool. Larger models may require FSDP / tensor parallelism instead.
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip checkFor GPU training: first install an official CUDA-enabled PyTorch wheel compatible with your hardware from https://pytorch.org/get-started/locally/. The torch>=2.6 dependency alone does not guarantee CUDA support.
Check device availability:
python -c "import torch; print('PyTorch:',torch.__version__); print('CUDA build:',torch.version.cuda); print('CUDA available:',torch.cuda.is_available()); print('GPU count:',torch.cuda.device_count())"Open LLM_DDP_Parallel_Training.ipynb. It creates requirements.txt and the training script. When dependencies are missing, run the following as a separate notebook cell, then restart the kernel if prompted:
%pip install -r requirements.txtFor a GPU runtime, do not blindly replace an already-working CUDA-enabled PyTorch installation. If needed, install other packages separately (transformers, peft, accelerate, safetensors).
From the repository directory, test 1 CPU process:
python llm_ddp_parallel_training.py --model tiny --cpu --max-steps 2Test 2 CPU processes with real DDP and Gloo (also works without CUDA):
python -m torch.distributed.run --standalone --nproc_per_node=2 llm_ddp_parallel_training.py --model tiny --cpu --batch-size 2 --max-steps 2Expected log patterns (exact losses and sample IDs may vary):
Mode=tiny, world_size=2, backend=gloo, per_device_batch=2, global_batch=4
Rank 0/2 on cpu: first batch has sample IDs [...]
Rank 1/2 on cpu: first batch has sample IDs [...]
epoch=1 step=1 mean_loss=...
Tiny Transformer saved at: .../tiny_transformer.pt
Requires downloading Qwen/Qwen2.5-0.5B-Instruct from Hugging Face. It uses 0.5B-scale pretrained parameters and only trains a small number of LoRA parameters. It does not perform QLoRA/4-bit quantization.
python -m torch.distributed.run --standalone --nproc_per_node=1 llm_ddp_parallel_training.py --model qwen --batch-size 1 --seq-len 64 --max-steps 2python -m torch.distributed.run --standalone --nproc_per_node=2 llm_ddp_parallel_training.py --model qwen --batch-size 1 --seq-len 64 --max-steps 2Check the actual number of GPUs before launching multiple processes:
import torch
print(torch.cuda.device_count())Only choose --nproc_per_node=2 if your runtime reports at least 2 CUDA GPUs. On Windows, the script uses the Gloo backend; NCCL is not available on Windows in standard PyTorch builds. Linux CUDA installations use NCCL. GPU runs on Windows may be less performant than Linux. PyTorch's distributed Windows support remains more limited than on Linux.
Create corpus.txt with one UTF-8 text sample per line. Then run:
python llm_ddp_parallel_training.py --model tiny --cpu --corpus-file corpus.txt --repeat 1 --max-steps 4This script's included text corpus is pedagogical. The Qwen mode applies next-token loss to the entire input text. It is not a supervised chat instruction dataset with response-only masking or a validation split; use a properly designed dataset and masking strategy for serious fine-tuning.
| Flag | Default | Purpose |
|---|---|---|
--model |
tiny |
tiny (from scratch) or qwen (LoRA) |
--cpu |
off | Force CPU; suitable for DDP demonstrations |
--corpus-file |
built-in | Optional one-example-per-line UTF-8 dataset |
--batch-size |
2 |
Microbatch size per rank |
--seq-len |
64 |
Maximum tokens/bytes per sample |
--epochs |
1 |
Epoch count |
--max-steps |
4 |
Maximum optimizer steps per rank per epoch; 0 = all |
--lr |
mode-specific | Learning-rate override |
--repeat |
4 |
Repeat the demonstration corpus |
--save-dir |
ddp_example_output |
Model/adapters output location |
The global batch size in this example is batch_size × world_size. No gradient accumulation is implemented.
torchrunlaunches N processes with environment variablesWORLD_SIZE,RANK,andLOCAL_RANK.- Each rank initializes a process group (Gloo for CPU/Windows, NCCL for Linux CUDA).
DistributedSamplergives each rank a different partition of the input examples.- Each rank computes a local forward pass, loss, and backward pass.
- DDP synchronizes model gradients across ranks before the optimizer update.
- Rank 0 saves the checkpoint or the adapter; all ranks shut down the group cleanly.
Verified in a local PyTorch 2.10.0+cpu runtime:
- Python syntax compilation: passed.
- Tiny Transformer, 1 CPU process, 2 steps: passed, checkpoint created.
- Tiny Transformer, 2 CPU processes, 2 steps, Gloo backend: passed, checkpoint created.
Not executed here: Qwen model downloads, single-GPU CUDA training or multi-GPU CUDA training and physical multi-GPU training. No production benchmark, quality metric, model-card evaluation or memory guarantee is claimed.
Potential limitations: small corpus, no evaluation split, no robust fault tolerance, no accumulation, possible GPU out-of-memory errors on consumer-grade GPUs, basic LoRA configuration and low-speed Gloo on Windows. For production training, introduce proper validation, data pipelines, observability, mixed-precision/autocast testing, reproducibility controls, and sharded training as needed.
CUDA available: False:nvidia-smican detect a GPU even when your current PyTorch build is CPU-only (e.g.,+cpu). Install the correct CUDA-enabled wheel in the active environment.LOCAL_RANKexceeds GPU count: do not launch more ranks than CUDA GPUs; use CPU mode to demonstrate multiple ranks.- Notebook
argparseerrors: don't run the CLI source with%runfromipykernel; use the provided notebook'ssubprocesslaunch cells. - Out of memory: start with
--batch-size 1 --seq-len 64; fewer steps do not necessarily reduce peak memory usage. - Qwen dependency errors: ensure
transformers,peft,accelerate, andsafetensorsare in the same Python environment; usepython -m pip check. - Repeated model downloads: configure Hugging Face cache paths, ensure internet access and sufficient disk space.
- PyTorch torchrun
- PyTorch Distributed Overview
- PyTorch Distributed Communication Backends
- Qwen2.5-0.5B-Instruct model card
- Hugging Face PEFT
This project is licensed under the Apache License, Version 2.0.© 2026 Sudhanshu Shukla · Licensed under the Apache License, Version 2.0.