Which quantization method should I use for my 7B parameter model?

For GPU inference with batching: GPTQ 4-bit with ExLlama v2 kernels. For single-user local inference: GGUF Q4_K_M with llama.cpp. For edge deployment on limited VRAM: AWQ 4-bit. For maximum quality with moderate compression: SqueezeLLM at 4-bit with dense-and-sparse decomposition.

How much quality do I lose with 4-bit quantization?

On a LLaMA-2 7B model, FP16 perplexity on WikiText-2 is 5.47. GPTQ 4-bit gets 5.63 (+0.16), AWQ 4-bit gets 5.60 (+0.13), GGUF Q4_K_M gets 5.68 (+0.21), and SqueezeLLM 4-bit gets 5.54 (+0.07). For most applications, these differences are imperceptible in output quality.

Can I quantize and run models on Apple Silicon?

Yes. GGUF with llama.cpp has the best Metal support, it uses the GPU's unified memory directly. For a 13B model in Q4_K_M, you get ~35 tok/s on an M3 Pro with 18GB unified memory. MLX also supports quantized models natively but with a different format (MLX quantized safetensors).

LLM Quantization Compared: GPTQ, AWQ, GGUF, and SqueezeLLM

The Core Problem

A 7B parameter model in FP16 (the standard training precision) takes ~14GB of VRAM. That's a full RTX 4090, or more than an M2 MacBook Air has. Scale to 70B and you need 140GB, multiple A100s.

Quantization compresses the weights from 16-bit floats to lower precision (8-bit, 4-bit, sometimes 3-bit or even 2-bit). The question isn't whether to quantize, for inference, you almost certainly should. The question is how, because the methods differ in ways that affect quality, speed, and compatibility.

Method 1: GPTQ (Post-Training Quantization via Optimal Brain Surgeon)

GPTQ was the first method that made 4-bit quantization practical for large models. It's based on a clever mathematical insight: instead of quantizing each weight independently (which is what naive round-to-nearest does), GPTQ quantizes one column of the weight matrix at a time and adjusts the remaining columns to compensate for the quantization error.

The algorithm, simplified:

# Pseudocode for GPTQ quantization of one linear layer
def gptq_quantize(W, H, bits=4, group_size=128):
 """
 W: weight matrix [out_features, in_features]
 H: Hessian approximation (X^T @ X from calibration data)
 """
 Q = torch.zeros_like(W)
 Err = torch.zeros_like(W)

 # Process columns left to right
 for col in range(W.shape[1]):
 # Quantize this column
 q = quantize_to_grid(W[:, col], bits, group_size)
 Q[:, col] = q

 # Compute error from quantizing this column
 err = (W[:, col] - q) / H[col, col]

 # Compensate: adjust remaining unquantized columns
 W[:, col+1:] -= err.unsqueeze(1) @ H[col, col+1:].unsqueeze(0)

 return Q

The Hessian matrix H is computed from a small calibration dataset (typically 128 samples from C4 or WikiText). This is the key, GPTQ uses second-order information about how the loss landscape curves around each weight, making smarter quantization decisions than simple rounding.

Pros:

Excellent quality for GPU inference
ExLlama v2 CUDA kernels provide fastest batched inference
Mature ecosystem (AutoGPTQ, transformers integration)

Cons:

Requires GPU for quantization (the Hessian computation is expensive)
Not great for CPU inference
Group quantization adds complexity to the weight format

Method 2: AWQ (Activation-Aware Weight Quantization)

AWQ's insight is elegant: not all weights are equally important. Weights that multiply large activations have more impact on output quality. So instead of treating all weights equally, AWQ finds the "salient" weights (those multiplied by large activations) and protects them.

The trick is that AWQ doesn't keep salient weights at higher precision. Instead, it applies a per-channel scaling factor before quantization that spreads the range of salient weights across more quantization bins:

# AWQ's core insight
def awq_scale_weights(W, activations, alpha=0.5):
 """
 Scale weights based on activation magnitudes before quantizing.
 This protects important weights by giving them more quantization bins.
 """
 # Compute per-channel activation magnitudes
 act_scales = activations.abs().mean(dim=0)

 # Scale factor: channels with larger activations get scaled up
 # before quantization (more bins), then scaled back down at runtime
 scales = act_scales.pow(alpha)

 # Apply: W_scaled = W * diag(scales)
 # After quantization: W_q * diag(1/scales) ≈ W
 W_scaled = W * scales.unsqueeze(0)

 return W_scaled, scales

The beauty is that the scaling is mathematically equivalent, you scale up before quantization and scale down at runtime, so the fp16 output is the same. But the quantized representation is more accurate because more bits are allocated to the weights that matter most.

Benchmarks, LLaMA-2 7B, WikiText-2 perplexity (lower is better):

Method	4-bit PPL	3-bit PPL	Memory	Quant time
FP16 (baseline)	5.47	,	13.5GB	,
GPTQ 4-bit g128	5.63	6.29	3.9GB	12min
AWQ 4-bit g128	5.60	6.14	3.9GB	8min
Round-to-nearest	6.12	8.45	3.9GB	2min

AWQ consistently edges out GPTQ on quality, especially at 3-bit. But GPTQ with ExLlama kernels is faster at inference. Pick your priority.

Method 3: GGUF (llama.cpp's Format)

GGUF isn't a quantization algorithm, it's a file format and ecosystem. llama.cpp implements several quantization methods under the GGUF umbrella, and the naming convention is initially confusing:

Q4_0: Simple 4-bit quantization with 32-weight blocks, one scale per block
Q4_1: Q4_0 + per-block minimum value (effectively an offset)
Q4_K_S: K-quant small, uses importance metrics to vary precision per tensor
Q4_K_M: K-quant medium, more tensors get 5-bit treatment
Q5_K_M: K-quant medium at 5-bit base
Q8_0: 8-bit quantization (good baseline, nearly lossless)

The K-quants are the interesting ones. They use a heuristic: attention projection layers (Q, K, V, O matrices) are more sensitive to quantization error than FFN layers. So Q4_K_M quantizes attention layers at 5-bit while keeping FFN at 4-bit:

# Quantizing with llama.cpp
./llama-quantize input-model-f16.gguf output-q4km.gguf Q4_K_M

# The output tells you exactly what happened:
# [ 1/291] layers.0.attention.wq.weight - [4096, 4096] -> Q5_K (5.5 bpw)
# [ 2/291] layers.0.attention.wk.weight - [4096, 4096] -> Q5_K (5.5 bpw)
# [ 3/291] layers.0.attention.wv.weight - [4096, 4096] -> Q5_K (5.5 bpw)
# [ 4/291] layers.0.ffn_gate.weight - [4096, 11008] -> Q4_K (4.5 bpw)
# [ 5/291] layers.0.ffn_up.weight - [4096, 11008] -> Q4_K (4.5 bpw)
# [ 6/291] layers.0.ffn_down.weight - [11008, 4096] -> Q4_K (4.5 bpw)

The mixed precision approach means Q4_K_M averages about 4.85 bits per weight, slightly more than pure 4-bit but measurably better in quality.

Why GGUF wins for local inference:

CPU+GPU hybrid: llama.cpp can split layers between CPU and GPU. Got a laptop with 8GB VRAM and 32GB RAM? Put 20 layers on GPU and 12 on CPU.
Memory-mapped loading: The model file is mmap'd, so loading a 4GB model takes milliseconds (the OS pages in data as needed).
Apple Silicon optimized: Metal shaders for M-series GPUs use unified memory efficiently.
No Python dependency: The inference server is pure C++. Deployment is copy a binary and a model file.

Method 4: SqueezeLLM

SqueezeLLM is the method that receives comparatively little attention, which is arguably a mistake. It does something fundamentally different from the other three: it separates weights into a "dense" quantized component and a "sparse" correction matrix:

W ≈ W_quantized + S (sparse correction)

The sparse matrix S contains corrections for the weights where quantization error is highest (the outliers). These outliers are typically <0.5% of all weights, so S is extremely sparse and adds minimal memory overhead.

The result is striking: SqueezeLLM at 4-bit achieves quality very close to 8-bit methods, with memory usage close to other 4-bit methods.

Method	4-bit PPL (LLaMA-2 7B)	Memory	Notes
GPTQ	5.63	3.9GB	g128
AWQ	5.60	3.9GB	g128
GGUF Q4_K_M	5.68	4.1GB	Mixed precision
SqueezeLLM	5.54	4.3GB	Dense + sparse
FP16	5.47	13.5GB	Baseline

SqueezeLLM's 5.54 is only 0.07 above FP16, that's essentially lossless for practical purposes. The extra 400MB of memory comes from the sparse correction matrix.

The catch: SqueezeLLM inference requires custom CUDA kernels for the sparse-dense decomposition, and the ecosystem support is minimal. You'll be writing your own serving code. For research or internal tools, worth it. For production serving at scale, the ecosystem maturity of GPTQ/AWQ/GGUF wins.

Real-World Inference Speed

All four methods were run on the same hardware with identical prompts (512 input tokens, 256 output tokens), using each method's best available inference engine:

NVIDIA RTX 4090 (24GB VRAM):

Method	Engine	7B tok/s	13B tok/s	70B*
GPTQ 4-bit	ExLlama v2	142	88	18
AWQ 4-bit	AutoAWQ	128	79	16
GGUF Q4_K_M	llama.cpp CUDA	118	72	15
SqueezeLLM 4-bit	Custom CUDA	95	54	,
FP16	vLLM	89	OOM	OOM

*70B requires offloading to CPU RAM for all 4-bit methods on a single 4090.

Apple M3 Pro (18GB unified memory):

Method	Engine	7B tok/s	13B tok/s
GGUF Q4_K_M	llama.cpp Metal	52	34
GGUF Q4_0	llama.cpp Metal	58	37
MLX 4-bit	mlx-lm	47	29

GGUF dominates on Apple Silicon. The Metal backend in llama.cpp is well-optimized for the M-series memory architecture.

The Decision Matrix

After extensive evaluation, here is a practical decision framework:

Scenario 1: GPU server, batched inference, maximum throughput → GPTQ 4-bit with ExLlama v2 kernels via vLLM or text-generation-inference

Scenario 2: Local inference on your laptop/desktop → GGUF Q4_K_M with llama.cpp (CPU+GPU hybrid if needed)

Scenario 3: Edge deployment, limited VRAM, need small model footprint → AWQ 4-bit (smallest runtime overhead, good library support)

Scenario 4: Maximum quality, willing to trade some speed and ecosystem convenience → SqueezeLLM 4-bit (closest to FP16 quality at 4-bit memory)

Scenario 5: You're not sure and just want something that works → GGUF Q4_K_M. Seriously. It's the most portable, has the simplest deployment story, works on every platform, and the quality is within spitting distance of the more complex methods.

Notable Findings

4-bit is better than naive 8-bit for many tasks. A GPTQ 4-bit model with proper calibration often outperforms a simple round-to-nearest 8-bit model. The calibration data gives the quantizer information about which weights matter, and 4-bit + intelligence beats 8-bit + ignorance.

Calibration data is the most underrated hyperparameter. Every guide says "use 128 samples from C4." If your model is fine-tuned for code generation and you calibrate on Wikipedia prose, you're leaving quality on the table. Creating domain-specific calibration sets can recover 0.2-0.4 perplexity points on specialized models.

Quantization and KV cache interact. A Q4 model saves 70% on weight memory, but the KV cache is still FP16 by default. At 32K context length with a 7B model, the KV cache alone is ~2GB. Look into FP8 KV cache quantization (vLLM supports this) to get the full benefit.

The quantization landscape moves fast, by the time you read this, there might be a new method that changes the calculus. But the principles don't change: understand what you're trading (precision, speed, compatibility), measure on your actual data, and independent benchmarks on actual production data remain the most reliable measure.

LLM Quantization Compared: GPTQ, AWQ, GGUF, and SqueezeLLM