LLM Optimization Benchmark Lab

An applied benchmark of seven large-language-model optimization techniques across GPU memory, inference latency, token throughput, scheduling efficiency, and distributed-inference feasibility.

Author: Farhan Prima Chandra
Positioning: Applied AI Engineer / AI Consultant

Project overview

This project evaluates optimization trade-offs using Qwen/Qwen2.5-1.5B-Instruct as the target model and Qwen/Qwen2.5-0.5B-Instruct as the assistant and student model. Direct benchmarks were executed on a Google Colab NVIDIA Tesla T4 GPU.

The objective is not to identify one universally superior method. It is to determine which technique is appropriate under different memory, latency, throughput, and infrastructure constraints.

Techniques covered

1. Quantization: FP16, INT8, and INT4 NF4
2. Sequence-level knowledge distillation with LoRA
3. KV caching
4. Continuous batching simulation
5. Speculative decoding
6. Tensor-parallel sharding analysis
7. Pipeline-parallel scheduling analysis

Single-GPU-compatible techniques are benchmarked directly. Continuous batching is evaluated through a deterministic scheduler simulation, while tensor and pipeline parallelism are evaluated through architecture-aware analysis and numerical demonstrations. No unsupported multi-GPU speedup claims are made.

Experimental design

• Five prompt categories: technical explanation, construction risk, financial risk, JSON extraction, and Python coding
• Deterministic greedy decoding
• Maximum 80 newly generated tokens
• One warm-up run and three measured runs per prompt
• Metrics: model memory, peak GPU memory, latency, output tokens per second, and output consistency
• Fixed random seed: 42

Key benchmark results

Configuration	Model memory (MB)	Mean latency (s)	Throughput (tokens/s)	Interpretation
FP16 baseline with KV cache	2,944.402	2.575	27.691	Best direct latency and throughput in this setup
FP16 without KV cache	2,944.402	3.054	23.410	KV caching reduced latency by 15.684%
INT8	1,694.902	11.130	6.383	Memory fell 42.436%, but latency increased sharply
INT4 NF4	1,070.152	3.722	18.827	Memory fell 63.655%; a more practical memory-first trade-off
Target with assistant model	3,886.695	3.778	19.415	Speculative decoding produced only 0.682x speedup
Distilled 0.5B student	944.355	2.929	24.067	Lowest memory, with limited quality evidence

Additional findings:

• Continuous batching simulation increased normalized token-slot utilization from 65.581% to 88.679% and reduced makespan from 215 to 159 decoding steps.
• A tensor-parallel degree of two satisfies strict attention-head and KV-head divisibility for the target architecture. Degrees four and eight do not satisfy strict KV-head divisibility.
• Four pipeline stages with eight microbatches produced 72.727% theoretical utilization; utilization rises as additional microbatches reduce pipeline bubbles.
• Distillation used 12 teacher-generated examples. Validation loss improved by only 0.434%, so the experiment demonstrates methodology rather than production-quality generalization.

Practical decision framework

Deployment priority	Preferred approach in this experiment
Lowest single-request latency	FP16 with KV caching
Preserve target architecture under a tighter memory budget	INT4 NF4
Smallest model footprint	Distilled 0.5B student, subject to stronger quality evaluation
Higher concurrent-request utilization	Continuous batching, after validation in a production inference engine
Small multi-GPU sharding plan	Tensor-parallel degree two
Layer-wise multi-device scaling	Pipeline parallelism with enough microbatches to limit idle time

INT8 and speculative decoding are not recommended for this specific model, hardware, and workload because their memory benefits did not offset their latency or throughput penalties.

Repository structure

LLM-Optimization-Benchmark-Lab/
├── LLM_Optimization_Benchmark_Lab.ipynb
├── README.md
├── requirements.txt
└── outputs/
    ├── llm_optimization_overview.png
    └── training_and_pipeline_analysis.png

Reproduction

1. Open the notebook in Google Colab.
2. Select a GPU runtime. The recorded benchmark used an NVIDIA Tesla T4.
3. Run the notebook from the beginning in a fresh runtime.
4. Allow model downloads from Hugging Face.

The notebook installs its main Hugging Face dependencies. requirements.txt records the tested core environment and supporting packages. Model weights and LoRA checkpoints are intentionally excluded from this repository.

Limitations

• Results come from one Colab T4 GPU, five prompt categories, and three measured runs per prompt.
• Lexical similarity and validity checks are proxies, not human or task-specific quality evaluation.
• The distillation corpus is intentionally small and does not support generalization claims.
• Continuous batching results are normalized simulation outputs, not measured vLLM throughput.
• Tensor and pipeline parallelism results are analytical and numerical demonstrations, not multi-GPU latency benchmarks.
• Results should be re-benchmarked before applying them to other models, GPUs, serving engines, or production workloads.

References

• Qwen2.5-1.5B-Instruct model card
• Qwen2.5-0.5B-Instruct model card
• Hugging Face bitsandbytes quantization documentation
• Hugging Face KV cache documentation
• Hugging Face PEFT documentation
• PyTorch tensor parallel tutorial
• PyTorch pipeline parallel tutorial

Ownership and scope

This is an independently implemented portfolio project. It is not an IBM or Microsoft course submission and is not affiliated with Qwen, Hugging Face, or PyTorch. The models and libraries retain their respective licenses.