llm-quantization

Get more intelligence from every bit. Better quantization formats and smarter calibration let larger, stronger models run smoothly on the hardware you already own.

Official PyTorch implementation of "GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance" (ICML 2025)

Open quantization tooling for TurboQuant-style low-bit LLM releases, stock GGUF deployment, and Apple Silicon runtime experiments.

[CAAI AIR'24] Minimize Quantization Output Error with Bias Compensation

A curated list of papers, code, and resources for NVFP4 quantization.

ForgeGPT — A lightweight LLM built from scratch in PyTorch, from Transformer architecture to quantized local inference.

A more deep research about TurboQuant algorithms

Ternary Quantization for LLMs: Implement balanced ternary (T3_K) weights for 2.63-bit quantization—the first working solution for modern large language models.

A high-performance, memory-efficient healthcare framework that deploys fine-tuned Large Language Models (LLMs) on edge devices. Multi-agent system to provide personalized diagnostic reasoning, health education, and dietary planning.

Benchmarking LLM quantization (FP16, INT8, INT4) on a real RAG pipeline, measuring RAM usage, latency, throughput, and answer quality with Ollama, Qwen3, FastAPI, PostgreSQL/pgvector, and React.

LLM quantization project built around `llama.cpp` + `Ollama` + `GGUF`

Production-grade LLM quantization, benchmarking, and edge deployment toolkit. Supports bitsandbytes INT8/INT4, GPTQ (Hessian calibration), AWQ (activation-aware), and GGUF (Q2_K–Q8_0). Four-dimensional benchmarking: perplexity, TPS/TTFT, VRAM profiling, and LLM-as-Judge quality scoring. RTX 5090 Blackwell sm_120 ready.

Paired capability-level GGUF quantization fragility benchmark across Qwen2.5-3B and SmolLM2 1.7B.

PentaNet extends BitNet's ternary quantization to pentanary {-2,-1,0,+1,+2}, improving perplexity by 6.4% at 124M params while preserving zero-multiplier arithmetic.

Local & lightweight LLM inference runtime in C++ with support for GGUF & quantization

GWIQ-Atlas: is a brain-atlasing and model-interpretability suite that combines per-layer census, compliance behaviour tracing, SAE features, and quantization analyses for LLMs.

An experiment to evaluate Quantized LLMs of different parameter sizes and quantization intensities on different NLP tasks comparing their latency, throughput, storage requirements, and task-specific metrics.

Let me make GGUF files quickly

Implementing LLM weight quantization from scratch in PyTorch - per-tensor vs per-channel INT8, with quantization error and WikiText-2 perplexity comparisons on Qwen2.5-0.5B.