AI/ML # LLM Quantization Compared: GPTQ, AWQ, GGUF, and SqueezeLLM 2025-10-119 min read Lucio Durán Engineering Manager & AI Solutions Architect Also available in: [Español]() ## The Core Problem A 7B parameter model in FP16 (the standard training precision) takes ~14GB of VRAM. That's a full RTX 4090, or more than an M2 MacBook Air has. Scale to 70B and you need 140GB, multiple A100s. Quantization compresses the weights from 16-bit floats to lower precision (8-bit, 4-bit, sometimes 3-bit or even 2-bit). The question isn't whether to quantize, for inference, you almost certainly should. The question is *how*, because the methods differ in ways that affect quality, speed, and compatibility. ## Method 1: GPTQ (Post-Training Quantization via Optimal Brain Surgeon) GPTQ was the first method that made 4-bit quantization practical for large models. It's based on a clever mathematical insight: instead of quantizing each weight independently (which is what naive round-to-nearest does), GPTQ quantizes one column of the weight matrix at a time and adjusts the remaining columns to compensate for the quantization error. The algorithm, simplified: ```python # Pseudocode for GPTQ quantization of one linear layer def gptq_quantize(W, H, bits=4, group_size=128): """ W: weight matrix [out_features, in_features] H: Hessian approximation (X^T @ X from calibration data) """ Q = torch.zeros_like(W) Err = torch.zeros_like(W) # Process columns left to right for col in range(W.shape[1]): # Quantize this column q = quantize_to_grid(W[:, col], bits, group_size) Q[:, col] = q # Compute error from quantizing this column err = (W[:, col] - q) / H[col, col] # Compensate: adjust remaining unquantized columns W[:, col+1:] -= err.unsqueeze(1) @ H[col, col+1:].unsqueeze(0) return Q ``` The Hessian matrix ` H ` is computed from a small calibration dataset (typically 128 samples from C4 or WikiText). This is the key, GPTQ uses second-order information about how the loss landscape curves around each weight, making smarter quantization decisions than simple rounding. **Pros:** - Excellent quality for GPU inference - ExLlama v2 CUDA kernels provide fastest batched inference - Mature ecosystem (AutoGPTQ, transformers integration) **Cons:** - Requires GPU for quantization (the Hessian computation is expensive) - Not great for CPU inference - Group quantization adds complexity to the weight format ## Method 2: AWQ (Activation-Aware Weight Quantization) AWQ's insight is elegant: not all weights are equally important. Weights that multiply large activations have more impact on output quality. So instead of treating all weights equally, AWQ finds the "salient" weights (those multiplied by large activations) and protects them. The trick is that AWQ doesn't keep salient weights at higher precision. Instead, it applies a per-channel scaling factor before quantization that spreads the range of salient weights across more quantization bins: ```python # AWQ's core insight def awq_scale_weights(W, activations, alpha=0.5): """ Scale weights based on activation magnitudes before quantizing. This protects important weights by giving them more quantization bins. """ # Compute per-channel activation magnitudes act_scales = activations.abs().mean(dim=0) # Scale factor: channels with larger activations get scaled up # before quantization (more bins), then scaled back down at runtime scales = act_scales.pow(alpha) # Apply: W_scaled = W * diag(scales) # After quantization: W_q * diag(1/scales) ≈ W W_scaled = W * scales.unsqueeze(0) return W_scaled, scales ``` The beauty is that the scaling is mathematically equivalent, you scale up before quantization and scale down at runtime, so the fp16 output is the same. But the quantized representation is more accurate because more bits are allocated to the weights that matter most. **Benchmarks, LLaMA-2 7B, WikiText-2 perplexity (lower is better):** | Method | 4-bit PPL | 3-bit PPL | Memory | Quant time | | --- | --- | --- | --- | --- | | FP16 (baseline) | 5.47 | , | 13.5GB | , | | GPTQ 4-bit g128 | 5.63 | 6.29 | 3.9GB | 12min | | AWQ 4-bit g128 | 5.60 | 6.14 | 3.9GB | 8min | | Round-to-nearest | 6.12 | 8.45 | 3.9GB | 2min | AWQ consistently edges out GPTQ on quality, especially at 3-bit. But GPTQ with ExLlama kernels is faster at inference. Pick your priority. ## Method 3: GGUF (llama.cpp's Format) GGUF isn't a quantization algorithm, it's a file format and ecosystem. llama.cpp implements several quantization methods under the GGUF umbrella, and the naming convention is initially confusing: - **Q4\_0**: Simple 4-bit quantization with 32-weight blocks, one scale per block - **Q4\_1**: Q4\_0 + per-block minimum value (effectively an offset) - **Q4\_K\_S**: K-quant small, uses importance metrics to vary precision per tensor - **Q4\_K\_M**: K-quant medium, more tensors get 5-bit treatment - **Q5\_K\_M**: K-quant medium at 5-bit base - **Q8\_0**: 8-bit quantization (good baseline, nearly lossless) The K-quants are the interesting ones. They use a heuristic: attention projection layers (Q, K, V, O matrices) are more sensitive to quantization error than FFN layers. So Q4\_K\_M quantizes attention layers at 5-bit while keeping FFN at 4-bit: ```bash # Quantizing with llama.cpp ./llama-quantize input-model-f16.gguf output-q4km.gguf Q4_K_M # The output tells you exactly what happened: # [ 1/291] layers.0.attention.wq.weight - [4096, 4096] -> Q5_K (5.5 bpw) # [ 2/291] layers.0.attention.wk.weight - [4096, 4096] -> Q5_K (5.5 bpw) # [ 3/291] layers.0.attention.wv.weight - [4096, 4096] -> Q5_K (5.5 bpw) # [ 4/291] layers.0.ffn_gate.weight - [4096, 11008] -> Q4_K (4.5 bpw) # [ 5/291] layers.0.ffn_up.weight - [4096, 11008] -> Q4_K (4.5 bpw) # [ 6/291] layers.0.ffn_down.weight - [11008, 4096] -> Q4_K (4.5 bpw) ``` The mixed precision approach means Q4\_K\_M averages about 4.85 bits per weight, slightly more than pure 4-bit but measurably better in quality. **Why GGUF wins for local inference:** 1. **CPU+GPU hybrid**: llama.cpp can split layers between CPU and GPU. Got a laptop with 8GB VRAM and 32GB RAM? Put 20 layers on GPU and 12 on CPU. 2. **Memory-mapped loading**: The model file is ` mmap `'d, so loading a 4GB model takes milliseconds (the OS pages in data as needed). 3. **Apple Silicon optimized**: Metal shaders for M-series GPUs use unified memory efficiently. 4. **No Python dependency**: The inference server is pure C++. Deployment is copy a binary and a model file. ## Method 4: SqueezeLLM SqueezeLLM is the method that receives comparatively little attention, which is arguably a mistake. It does something fundamentally different from the other three: it separates weights into a "dense" quantized component and a "sparse" correction matrix: ``` W ≈ W_quantized + S (sparse correction) ``` The sparse matrix ` S ` contains corrections for the weights where quantization error is highest (the outliers). These outliers are typically \<0.5% of all weights, so ` S ` is extremely sparse and adds minimal memory overhead. The result is striking: SqueezeLLM at 4-bit achieves quality very close to 8-bit methods, with memory usage close to other 4-bit methods. | Method | 4-bit PPL (LLaMA-2 7B) | Memory | Notes | | --- | --- | --- | --- | | GPTQ | 5.63 | 3.9GB | g128 | | AWQ | 5.60 | 3.9GB | g128 | | GGUF Q4\_K\_M | 5.68 | 4.1GB | Mixed precision | | SqueezeLLM | 5.54 | 4.3GB | Dense + sparse | | FP16 | 5.47 | 13.5GB | Baseline | SqueezeLLM's 5.54 is only 0.07 above FP16, that's essentially lossless for practical purposes. The extra 400MB of memory comes from the sparse correction matrix. **The catch:** SqueezeLLM inference requires custom CUDA kernels for the sparse-dense decomposition, and the ecosystem support is minimal. You'll be writing your own serving code. For research or internal tools, worth it. For production serving at scale, the ecosystem maturity of GPTQ/AWQ/GGUF wins. ## Real-World Inference Speed All four methods were run on the same hardware with identical prompts (512 input tokens, 256 output tokens), using each method's best available inference engine: **NVIDIA RTX 4090 (24GB VRAM):** | Method | Engine | 7B tok/s | 13B tok/s | 70B\* | | --- | --- | --- | --- | --- | | GPTQ 4-bit | ExLlama v2 | 142 | 88 | 18 | | AWQ 4-bit | AutoAWQ | 128 | 79 | 16 | | GGUF Q4\_K\_M | llama.cpp CUDA | 118 | 72 | 15 | | SqueezeLLM 4-bit | Custom CUDA | 95 | 54 | , | | FP16 | vLLM | 89 | OOM | OOM | \*70B requires offloading to CPU RAM for all 4-bit methods on a single 4090. **Apple M3 Pro (18GB unified memory):** | Method | Engine | 7B tok/s | 13B tok/s | | --- | --- | --- | --- | | GGUF Q4\_K\_M | llama.cpp Metal | 52 | 34 | | GGUF Q4\_0 | llama.cpp Metal | 58 | 37 | | MLX 4-bit | mlx-lm | 47 | 29 | GGUF dominates on Apple Silicon. The Metal backend in llama.cpp is well-optimized for the M-series memory architecture. ## The Decision Matrix After extensive evaluation, here is a practical decision framework: **Scenario 1: GPU server, batched inference, maximum throughput** → GPTQ 4-bit with ExLlama v2 kernels via vLLM or text-generation-inference **Scenario 2: Local inference on your laptop/desktop** → GGUF Q4\_K\_M with llama.cpp (CPU+GPU hybrid if needed) **Scenario 3: Edge deployment, limited VRAM, need small model footprint** → AWQ 4-bit (smallest runtime overhead, good library support) **Scenario 4: Maximum quality, willing to trade some speed and ecosystem convenience** → SqueezeLLM 4-bit (closest to FP16 quality at 4-bit memory) **Scenario 5: You're not sure and just want something that works** → GGUF Q4\_K\_M. Seriously. It's the most portable, has the simplest deployment story, works on every platform, and the quality is within spitting distance of the more complex methods. ## Notable Findings **4-bit is better than naive 8-bit for many tasks.** A GPTQ 4-bit model with proper calibration often outperforms a simple round-to-nearest 8-bit model. The calibration data gives the quantizer information about which weights matter, and 4-bit + intelligence beats 8-bit + ignorance. **Calibration data is the most underrated hyperparameter.** Every guide says "use 128 samples from C4." If your model is fine-tuned for code generation and you calibrate on Wikipedia prose, you're leaving quality on the table. Creating domain-specific calibration sets can recover 0.2-0.4 perplexity points on specialized models. **Quantization and KV cache interact.** A Q4 model saves 70% on weight memory, but the KV cache is still FP16 by default. At 32K context length with a 7B model, the KV cache alone is ~2GB. Look into FP8 KV cache quantization (vLLM supports this) to get the full benefit. The quantization landscape moves fast, by the time you read this, there might be a new method that changes the calculus. But the principles don't change: understand what you're trading (precision, speed, compatibility), measure on your actual data, and independent benchmarks on actual production data remain the most reliable measure. ## Frequently asked questions ### Which quantization method should I use for my 7B parameter model? For GPU inference with batching: GPTQ 4-bit with ExLlama v2 kernels. For single-user local inference: GGUF Q4\_K\_M with llama.cpp. For edge deployment on limited VRAM: AWQ 4-bit. For maximum quality with moderate compression: SqueezeLLM at 4-bit with dense-and-sparse decomposition. ### How much quality do I lose with 4-bit quantization? On a LLaMA-2 7B model, FP16 perplexity on WikiText-2 is 5.47. GPTQ 4-bit gets 5.63 (+0.16), AWQ 4-bit gets 5.60 (+0.13), GGUF Q4\_K\_M gets 5.68 (+0.21), and SqueezeLLM 4-bit gets 5.54 (+0.07). For most applications, these differences are imperceptible in output quality. ### Can I quantize and run models on Apple Silicon? Yes. GGUF with llama.cpp has the best Metal support, it uses the GPU's unified memory directly. For a 13B model in Q4\_K\_M, you get ~35 tok/s on an M3 Pro with 18GB unified memory. MLX also supports quantized models natively but with a different format (MLX quantized safetensors). ## Tips 1. **AWQ calibration set matters enormously:** Don't use the default ` pile-uncopyrighted ` for domain-specific models. Create a calibration set from your actual inference data, even 128 samples is enough. Perplexity has been observed to drop by 0.3 on a medical QA model just by switching calibration data to PubMed abstracts. 2. **GGUF Q4\_K\_M is the sweet spot for most use cases:** It uses a mix of 4-bit and 5-bit quantization, with 5-bit for attention layers and 4-bit for FFN layers. The ` llama-quantize ` flag is ` --type q4_K_M `. It's typically within 0.1 perplexity of Q5\_K\_M at 20% less memory. 3. **Profile your VRAM with ` nvidia-smi dmon -s m ` during inference:** Quantized models can still OOM if KV cache grows unbounded. Set ` max_seq_len ` explicitly in your serving config. A 7B model in Q4 uses ~4GB for weights but KV cache at 32K context adds another 2GB. People forget the cache. 4. **For batch inference, GPTQ with ` exllama_v2 ` kernel beats AWQ:** ExLlama v2's custom CUDA kernels for GPTQ dequantization are faster than AutoAWQ's at batch sizes \>4. Set ` disable_exllama=False ` in your quantization config and ensure ` torch.cuda.is_bf16_supported() ` returns true for best results. - llm - quantization - gptq - awq - gguf - inference-optimization [Back to Blog]()