{"schemaVersion":1,"id":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention","title":"LLM Inference Optimization: KV-Cache, PagedAttention, and GPU Memory Efficiency","description":"An analysis of LLM inference optimization mechanics, from KV-cache management and PagedAttention in vLLM to continuous batching, speculative decoding, and FlashAttention-3, with throughput measurements from production deployments.","language":"en","datePublished":"2025-12-22","dateModified":"2026-10-21","contentFormat":"text/markdown","content":"AI/ML\n\n# LLM Inference Optimization: KV-Cache, PagedAttention, and GPU Memory Efficiency\n\n2025-12-2210 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/llm-inference-optimization-kv-cache-paged-attention>)\n\n## The Attention Bottleneck in Decode-Phase Inference\n\nTransformer inference has two phases: the **prefill** (processing the entire prompt in parallel) and **decode** (generating tokens one at a time autoregressively). What most tutorials gloss over is that decode is fundamentally memory-bound, not compute-bound.\n\nDuring decode, each new token requires attending to every previous token in the sequence. The naive approach recomputes all key-value projections from scratch at every step, making generation O(n²) in sequence length. The KV-cache solves this by storing the K and V tensors from all previous positions:\n\n```python\n# Simplified KV-cache during decode\n# For each layer l, head h, and position t:\n# K_cache[l][h][t] = W_k @ hidden_states[t]\n# V_cache[l][h][t] = W_v @ hidden_states[t]\n\n# At decode step T, attention for new token:\nq = W_q @ hidden_states[T] # (1, d_head)\nk = K_cache[l][h][:T+1] # (T+1, d_head)\nv = V_cache[l][h][:T+1] # (T+1, d_head)\nattn = softmax(q @ k.T / sqrt(d_head)) @ v\n```\n\nThis reduces generation to O(n) per step. But the memory cost is brutal. For a 70B model (80 layers, 64 heads, 128 head dim, FP16):\n\n```\nKV-cache per token = 2 * n_layers * n_kv_heads * d_head * 2 bytes\n = 2 * 80 * 8 * 128 * 2 (GQA with 8 KV heads)\n = 327,680 bytes ≈ 320 KB per token\n\nFor a 4096-token sequence:\n320 KB * 4096 = 1.28 GB per request\n\nFor 100 concurrent requests:\n1.28 GB * 100 = 128 GB just for KV-cache\n```\n\nThe model weights are ~140 GB in FP16. The KV-cache can rival the model itself in memory consumption, which creates a significant resource management challenge.\n\n## The Fragmentation Problem\n\nTraditional serving frameworks pre-allocate a contiguous block of GPU memory for each request's KV-cache, sized for the maximum possible sequence length. If your max is 4096 tokens but the average completion is 500 tokens, you're wasting ~87% of allocated KV-cache memory.\n\nMeasurements on production clusters show average actual KV-cache utilization at 22%. This means 78% of the most expensive resource is being discarded.\n\n```python\n# What naive allocation looks like:\ndef allocate_kv_cache(max_seq_len, batch_size, n_layers, n_heads, d_head):\n # Pre-allocate for worst case\n k_cache = torch.zeros(\n batch_size, n_layers, n_heads, max_seq_len, d_head,\n dtype=torch.float16, device='cuda'\n )\n v_cache = torch.zeros_like(k_cache)\n return k_cache, v_cache\n\n# For batch=64, max_seq=4096 on a 70B model:\n# This immediately claims ~82 GB of VRAM\n# Most of which sits at zero throughout the request lifecycle\n```\n\n## PagedAttention: OS Concepts Meet GPU Memory\n\n[PagedAttention](<https://arxiv.org/abs/2309.06180>) from the [vLLM](<https://vllm.ai/>) team addressed this problem with a fundamentally different approach. The insight is deceptively simple: treat KV-cache memory the same way operating systems treat virtual memory.\n\nInstead of contiguous per-request allocations, PagedAttention divides KV-cache into fixed-size **blocks** (typically 16 tokens each). Each request maintains a **block table** mapping logical positions to physical block locations in GPU memory, exactly like a page table maps virtual to physical addresses.\n\n```python\n# Conceptual PagedAttention block table\n# Each block stores KV for 16 tokens across all layers and heads\n\nclass BlockTable:\n def __init__(self, block_size=16):\n self.block_size = block_size\n self.table = [] # List of physical block indices\n\n def get_physical_block(self, logical_position):\n block_idx = logical_position // self.block_size\n block_offset = logical_position % self.block_size\n physical_block = self.table[block_idx]\n return physical_block, block_offset\n\n# Kernel pseudocode for paged attention\n# For each query token:\n# For each block in request's block_table:\n# Load K, V from physical block\n# Compute partial attention scores\n# Accumulate with online softmax (like FlashAttention)\n```\n\nThe benefits cascade:\n\n1. **No internal fragmentation**: Blocks are allocated on demand. A 500-token response uses exactly ceil(500/16) = 32 blocks, not the full max\\_seq\\_len worth.\n2. **Copy-on-write sharing**: For beam search or parallel sampling, multiple sequences that share a common prefix can point to the same physical blocks. When a sequence diverges, only the diverging block gets copied.\n3. **Dynamic memory**: Blocks are freed immediately when a request completes, returned to the pool for the next request. No reserved-but-idle memory.\n\nIn a representative production cluster, switching from HuggingFace TGI to vLLM with PagedAttention increased concurrent request capacity from ~40 to ~110 on the same hardware. Throughput went from 480 to 1,340 tokens/second aggregate.\n\n## Continuous Batching: The Other Half of the Equation\n\nStatic batching waits for an entire batch of requests, processes them together, and doesn't start the next batch until all sequences in the current batch have finished generating. This means short completions sit idle while long ones keep going.\n\nContinuous batching (also called iteration-level scheduling) is different. At each decode step, the scheduler can:\n\n- Evict completed sequences\n- Admit new sequences\n- Preempt lower-priority sequences if memory runs tight\n\n```python\n# Simplified continuous batching scheduler\nclass ContinuousBatchScheduler:\n def __init__(self, max_batch_tokens, block_manager):\n self.waiting = deque() # Requests waiting to start\n self.running = [] # Currently generating\n self.max_batch_tokens = max_batch_tokens\n self.block_manager = block_manager\n\n def schedule(self):\n # Remove completed sequences\n self.running = [s for s in self.running if not s.is_finished()]\n\n # Try to admit new sequences\n while self.waiting:\n next_req = self.waiting[0]\n total_tokens = sum(s.get_len() for s in self.running)\n total_tokens += next_req.prompt_len\n\n if total_tokens > self.max_batch_tokens:\n break\n if not self.block_manager.can_allocate(next_req):\n break # Not enough KV-cache blocks\n\n self.running.append(self.waiting.popleft())\n self.block_manager.allocate(self.running[-1])\n\n return self.running\n```\n\nThe practical impact: on a workload with mixed short (50-token) and long (2000-token) completions, continuous batching yields 2.8x higher throughput compared to static batching. The short requests weren't hostage to the long ones anymore.\n\n## FlashAttention-3: Because Memory Bandwidth Is the Real Enemy\n\nFlashAttention rewrites the attention computation to be IO-aware, minimizing reads/writes to GPU HBM (High Bandwidth Memory) by keeping data in SRAM (the fast on-chip memory) as long as possible. Version 3 adds several tricks specific to Hopper architecture GPUs (H100):\n\n```\nStandard Attention:\n 1. Compute S = Q @ K.T → write S to HBM\n 2. Compute P = softmax(S) → write P to HBM\n 3. Compute O = P @ V → write O to HBM\n Total HBM accesses: O(N² * d)\n\nFlashAttention-3:\n 1. Tile Q, K, V into SRAM-sized blocks\n 2. For each tile: compute local S, P, O in SRAM\n 3. Accumulate with online softmax rescaling\n 4. Write final O to HBM once\n Total HBM accesses: O(N * d), linear, not quadratic\n```\n\nFA3 on H100s specifically exploits:\n\n- **Warp specialization**: Separate warps for MMA (matrix multiply-accumulate) and softmax, overlapping compute with data movement\n- **FP8 support**: Native FP8 tensor cores for the QK^T matmul, with FP32 accumulation\n- **Asynchronous block-wise softmax**: Using the TMA (Tensor Memory Accelerator) for async global-to-shared memory copies\n\nFor reproducible results, use the configurations and measurements reported in the [FlashAttention 3 paper](<https://arxiv.org/abs/2407.08608>) and benchmark the target workload. This guide does not provide an independently reproducible measurement for the specific H100 SXM configuration previously listed here.\n\nThe integration with vLLM's PagedAttention is where it gets interesting, the ` flash_attn_with_kvcache ` kernel handles the block table indirection directly, so you get both paged memory management and IO-aware attention in a single fused kernel.\n\n## Speculative Decoding: Trading Compute for Latency\n\nThe idea: use a small \"draft\" model to generate N candidate tokens quickly, then verify them all at once with the large \"target\" model in a single forward pass. If the target model agrees with the draft tokens (checked via a modified rejection sampling scheme), you've effectively generated N tokens in the time it takes the target to do one forward pass.\n\n```python\n# Speculative decoding loop\ndef speculative_decode(target_model, draft_model, prompt, gamma=5):\n tokens = prompt\n while not is_done(tokens):\n # Draft phase: generate gamma candidates cheaply\n draft_tokens = []\n draft_probs = []\n for _ in range(gamma):\n logits = draft_model(tokens + draft_tokens)\n p = softmax(logits[-1])\n t = sample(p)\n draft_tokens.append(t)\n draft_probs.append(p)\n\n # Verify phase: single forward pass on target\n all_logits = target_model(tokens + draft_tokens)\n\n # Acceptance: modified rejection sampling\n accepted = 0\n for i in range(gamma):\n q = softmax(all_logits[len(tokens) + i])\n p = draft_probs[i]\n r = random()\n if r < min(1, q[draft_tokens[i]] / p[draft_tokens[i]]):\n accepted += 1\n else:\n # Resample from adjusted distribution\n adjusted = normalize(max(0, q - p))\n tokens.append(sample(adjusted))\n break\n else:\n # All accepted, sample one more from target\n tokens.extend(draft_tokens)\n q = softmax(all_logits[-1])\n tokens.append(sample(q))\n continue\n tokens.extend(draft_tokens[:accepted])\n```\n\nThe math works out so that the output distribution is **provably identical** to sampling from the target model alone. You're not sacrificing quality.\n\nIn practice, the acceptance rate depends heavily on how well the draft model matches the target. Observed results include:\n\n- Llama-3-8B drafting for Llama-3-70B: ~75% acceptance rate, ~2.3x speedup\n- Tiny custom 100M drafter fine-tuned on target outputs: ~65%, ~1.8x\n- Code completion tasks (high predictability): ~85%, ~3.1x\n\nThe sweet spot for ` gamma ` (speculation length) is usually 4-6. Go higher and the probability of at least one rejection per round approaches 1, wasting the extra draft compute.\n\n## Putting It All Together: Production Configuration\n\nHere is a vLLM configuration that can bring a cluster from 12 tok/s/user to 45 tok/s/user:\n\n```bash\npython -m vllm.entrypoints.openai.api_server \\\n --model meta-llama/Llama-3-70B-Instruct \\\n --tensor-parallel-size 8 \\\n --max-num-batched-tokens 32768 \\\n --max-num-seqs 256 \\\n --block-size 16 \\\n --swap-space 4 \\\n --gpu-memory-utilization 0.92 \\\n --enable-prefix-caching \\\n --use-v2-block-manager \\\n --speculative-model meta-llama/Llama-3-8B-Instruct \\\n --num-speculative-tokens 5 \\\n --quantization awq\n```\n\nKey decisions:\n\n- ` --max-num-batched-tokens 32768 `: Allows up to 32K tokens in flight per iteration. Tuned to just under the OOM threshold.\n- ` --enable-prefix-caching `: Reuses KV-cache blocks for shared system prompts. With our 1,500-token system prompt, this saves ~480 KB per request.\n- ` --gpu-memory-utilization 0.92 `: Leave 8% headroom for activation memory spikes during prefill of long prompts.\n- ` --swap-space 4 `: 4 GB of CPU swap space per GPU for preempted requests, better than rejecting them outright.\n\n## The Numbers That Actually Matter\n\nOn 8xA100 80GB, serving Llama-3-70B-Instruct (AWQ 4-bit):\n\n| Metric | Naive (TGI static) | Optimized (vLLM) | Improvement |\n| --- | --- | --- | --- |\n| Throughput (tok/s) | 480 | 1,850 | 3.85x |\n| Concurrent requests | 40 | 180 | 4.5x |\n| TTFT p50 | 820 ms | 210 ms | 3.9x |\n| TTFT p99 | 4,200 ms | 890 ms | 4.7x |\n| ITL p50 | 38 ms | 22 ms | 1.7x |\n| Cost per 1M tokens | $2.40 | $0.62 | 3.87x |\n\nThe cost reduction alone can pay for the engineering effort in about four days.\n\n## What's Coming Next\n\nSeveral areas are worth watching for the next round of optimizations:\n\n- **Disaggregated prefill/decode**: Running prefill on different GPUs than decode, since they have completely different compute profiles. Splitwise and DistServe papers look promising.\n- **Quantized KV-cache**: Storing KV-cache in FP8 or even INT4, separate from model weight quantization. Early results show minimal quality loss with 2x memory reduction.\n- **Ring attention**: For ultra-long contexts (100K+), distributing the attention computation across multiple GPUs along the sequence dimension.\n\nThe field is moving incredibly fast. Six months ago PagedAttention was novel; now it's table stakes. The infrastructure advantage goes to whoever can integrate these techniques fastest into their serving stack.\n\nFor anyone running inference in production, examining KV-cache utilization is a high-value activity. That is likely where the majority of costs concentrate.\n\n## Frequently asked questions\n\n### What is the KV-cache and why does it matter for LLM inference?\n\nThe KV-cache stores the key and value projection tensors from every previous token in the sequence, so the model doesn't need to recompute attention over the entire context at each generation step. Without it, inference would be quadratic in sequence length. The problem is it grows linearly with sequence length and batch size, often consuming more GPU memory than the model parameters themselves.\n\n### How does PagedAttention improve over traditional KV-cache management?\n\nPagedAttention borrows virtual memory concepts from operating systems, dividing the KV-cache into fixed-size blocks that can be stored non-contiguously in GPU memory. This eliminates internal fragmentation from over-allocation and enables memory sharing across requests via copy-on-write semantics, typically reducing KV-cache waste by 60-70%.\n\n### Is speculative decoding worth the complexity in production?\n\nIt depends on your acceptance rate. With a well-matched draft model you can get 2-3x speedup on generation-heavy workloads, but if the acceptance rate drops below 60% you're burning FLOPs for nothing. Profile first with your actual traffic distribution before committing.\n\n## Tips\n\n1. **Tip:** If the KV-cache is eating more VRAM than the model weights themselves, you're probably serving long-context requests without ` PagedAttention `. Switch to vLLM and watch your memory utilization drop 60-70%.\n2. **Tip:** Continuous batching with ` --max-num-batched-tokens ` tuned to the GPU's memory ceiling is the single biggest throughput lever. Teams have been observed tripling their tokens/sec just by moving from static to continuous batching.\n3. **Tip:** When profiling inference, always measure *time-to-first-token* (TTFT) separately from *inter-token latency* (ITL). They respond to completely different optimizations and your users care about both.\n\n- llm\n- inference\n- kv-cache\n- vllm\n- paged-attention\n- flash-attention\n- speculative-decoding\n- gpu\n- optimization\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"LLM Inference Optimization: KV-Cache, PagedAttention, and GPU Memory Efficiency","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#title"},{"level":2,"text":"The Attention Bottleneck in Decode-Phase Inference","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#the-attention-bottleneck-in-decode-phase-inference"},{"level":2,"text":"The Fragmentation Problem","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#the-fragmentation-problem"},{"level":2,"text":"PagedAttention: OS Concepts Meet GPU Memory","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#pagedattention-os-concepts-meet-gpu-memory"},{"level":2,"text":"Continuous Batching: The Other Half of the Equation","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#continuous-batching-the-other-half-of-the-equation"},{"level":2,"text":"FlashAttention-3: Because Memory Bandwidth Is the Real Enemy","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#flashattention-3-because-memory-bandwidth-is-the-real-enemy"},{"level":2,"text":"Speculative Decoding: Trading Compute for Latency","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#speculative-decoding-trading-compute-for-latency"},{"level":2,"text":"Putting It All Together: Production Configuration","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#putting-it-all-together-production-configuration"},{"level":2,"text":"The Numbers That Actually Matter","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#the-numbers-that-actually-matter"},{"level":2,"text":"What's Coming Next","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#whats-coming-next"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#questions"},{"level":3,"text":"What is the KV-cache and why does it matter for LLM inference?","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#question-1"},{"level":3,"text":"How does PagedAttention improve over traditional KV-cache management?","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#question-2"},{"level":3,"text":"Is speculative decoding worth the complexity in production?","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#article-tips"}],"sections":[{"heading":"LLM Inference Optimization: KV-Cache, PagedAttention, and GPU Memory Efficiency","level":1,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention","content":"AI/ML"},{"heading":"LLM Inference Optimization: KV-Cache, PagedAttention, and GPU Memory Efficiency","level":1,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#title","content":"2025-12-2210 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/llm-inference-optimization-kv-cache-paged-attention>)"},{"heading":"The Attention Bottleneck in Decode-Phase Inference","level":2,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#the-attention-bottleneck-in-decode-phase-inference","content":"Transformer inference has two phases: the **prefill** (processing the entire prompt in parallel) and **decode** (generating tokens one at a time autoregressively). What most tutorials gloss over is that decode is fundamentally memory-bound, not compute-bound.\n\nDuring decode, each new token requires attending to every previous token in the sequence. The naive approach recomputes all key-value projections from scratch at every step, making generation O(n²) in sequence length. The KV-cache solves this by storing the K and V tensors from all previous positions:\n\n```python\n# Simplified KV-cache during decode\n# For each layer l, head h, and position t:\n# K_cache[l][h][t] = W_k @ hidden_states[t]\n# V_cache[l][h][t] = W_v @ hidden_states[t]\n\n# At decode step T, attention for new token:\nq = W_q @ hidden_states[T] # (1, d_head)\nk = K_cache[l][h][:T+1] # (T+1, d_head)\nv = V_cache[l][h][:T+1] # (T+1, d_head)\nattn = softmax(q @ k.T / sqrt(d_head)) @ v\n```\n\nThis reduces generation to O(n) per step. But the memory cost is brutal. For a 70B model (80 layers, 64 heads, 128 head dim, FP16):\n\n```\nKV-cache per token = 2 * n_layers * n_kv_heads * d_head * 2 bytes\n = 2 * 80 * 8 * 128 * 2 (GQA with 8 KV heads)\n = 327,680 bytes ≈ 320 KB per token\n\nFor a 4096-token sequence:\n320 KB * 4096 = 1.28 GB per request\n\nFor 100 concurrent requests:\n1.28 GB * 100 = 128 GB just for KV-cache\n```\n\nThe model weights are ~140 GB in FP16. The KV-cache can rival the model itself in memory consumption, which creates a significant resource management challenge."},{"heading":"The Fragmentation Problem","level":2,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#the-fragmentation-problem","content":"Traditional serving frameworks pre-allocate a contiguous block of GPU memory for each request's KV-cache, sized for the maximum possible sequence length. If your max is 4096 tokens but the average completion is 500 tokens, you're wasting ~87% of allocated KV-cache memory.\n\nMeasurements on production clusters show average actual KV-cache utilization at 22%. This means 78% of the most expensive resource is being discarded.\n\n```python\n# What naive allocation looks like:\ndef allocate_kv_cache(max_seq_len, batch_size, n_layers, n_heads, d_head):\n # Pre-allocate for worst case\n k_cache = torch.zeros(\n batch_size, n_layers, n_heads, max_seq_len, d_head,\n dtype=torch.float16, device='cuda'\n )\n v_cache = torch.zeros_like(k_cache)\n return k_cache, v_cache\n\n# For batch=64, max_seq=4096 on a 70B model:\n# This immediately claims ~82 GB of VRAM\n# Most of which sits at zero throughout the request lifecycle\n```"},{"heading":"PagedAttention: OS Concepts Meet GPU Memory","level":2,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#pagedattention-os-concepts-meet-gpu-memory","content":"[PagedAttention](<https://arxiv.org/abs/2309.06180>) from the [vLLM](<https://vllm.ai/>) team addressed this problem with a fundamentally different approach. The insight is deceptively simple: treat KV-cache memory the same way operating systems treat virtual memory.\n\nInstead of contiguous per-request allocations, PagedAttention divides KV-cache into fixed-size **blocks** (typically 16 tokens each). Each request maintains a **block table** mapping logical positions to physical block locations in GPU memory, exactly like a page table maps virtual to physical addresses.\n\n```python\n# Conceptual PagedAttention block table\n# Each block stores KV for 16 tokens across all layers and heads\n\nclass BlockTable:\n def __init__(self, block_size=16):\n self.block_size = block_size\n self.table = [] # List of physical block indices\n\n def get_physical_block(self, logical_position):\n block_idx = logical_position // self.block_size\n block_offset = logical_position % self.block_size\n physical_block = self.table[block_idx]\n return physical_block, block_offset\n\n# Kernel pseudocode for paged attention\n# For each query token:\n# For each block in request's block_table:\n# Load K, V from physical block\n# Compute partial attention scores\n# Accumulate with online softmax (like FlashAttention)\n```\n\nThe benefits cascade:\n\n1. **No internal fragmentation**: Blocks are allocated on demand. A 500-token response uses exactly ceil(500/16) = 32 blocks, not the full max\\_seq\\_len worth.\n2. **Copy-on-write sharing**: For beam search or parallel sampling, multiple sequences that share a common prefix can point to the same physical blocks. When a sequence diverges, only the diverging block gets copied.\n3. **Dynamic memory**: Blocks are freed immediately when a request completes, returned to the pool for the next request. No reserved-but-idle memory.\n\nIn a representative production cluster, switching from HuggingFace TGI to vLLM with PagedAttention increased concurrent request capacity from ~40 to ~110 on the same hardware. Throughput went from 480 to 1,340 tokens/second aggregate."},{"heading":"Continuous Batching: The Other Half of the Equation","level":2,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#continuous-batching-the-other-half-of-the-equation","content":"Static batching waits for an entire batch of requests, processes them together, and doesn't start the next batch until all sequences in the current batch have finished generating. This means short completions sit idle while long ones keep going.\n\nContinuous batching (also called iteration-level scheduling) is different. At each decode step, the scheduler can:\n\n- Evict completed sequences\n- Admit new sequences\n- Preempt lower-priority sequences if memory runs tight\n\n```python\n# Simplified continuous batching scheduler\nclass ContinuousBatchScheduler:\n def __init__(self, max_batch_tokens, block_manager):\n self.waiting = deque() # Requests waiting to start\n self.running = [] # Currently generating\n self.max_batch_tokens = max_batch_tokens\n self.block_manager = block_manager\n\n def schedule(self):\n # Remove completed sequences\n self.running = [s for s in self.running if not s.is_finished()]\n\n # Try to admit new sequences\n while self.waiting:\n next_req = self.waiting[0]\n total_tokens = sum(s.get_len() for s in self.running)\n total_tokens += next_req.prompt_len\n\n if total_tokens > self.max_batch_tokens:\n break\n if not self.block_manager.can_allocate(next_req):\n break # Not enough KV-cache blocks\n\n self.running.append(self.waiting.popleft())\n self.block_manager.allocate(self.running[-1])\n\n return self.running\n```\n\nThe practical impact: on a workload with mixed short (50-token) and long (2000-token) completions, continuous batching yields 2.8x higher throughput compared to static batching. The short requests weren't hostage to the long ones anymore."},{"heading":"FlashAttention-3: Because Memory Bandwidth Is the Real Enemy","level":2,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#flashattention-3-because-memory-bandwidth-is-the-real-enemy","content":"FlashAttention rewrites the attention computation to be IO-aware, minimizing reads/writes to GPU HBM (High Bandwidth Memory) by keeping data in SRAM (the fast on-chip memory) as long as possible. Version 3 adds several tricks specific to Hopper architecture GPUs (H100):\n\n```\nStandard Attention:\n 1. Compute S = Q @ K.T → write S to HBM\n 2. Compute P = softmax(S) → write P to HBM\n 3. Compute O = P @ V → write O to HBM\n Total HBM accesses: O(N² * d)\n\nFlashAttention-3:\n 1. Tile Q, K, V into SRAM-sized blocks\n 2. For each tile: compute local S, P, O in SRAM\n 3. Accumulate with online softmax rescaling\n 4. Write final O to HBM once\n Total HBM accesses: O(N * d), linear, not quadratic\n```\n\nFA3 on H100s specifically exploits:\n\n- **Warp specialization**: Separate warps for MMA (matrix multiply-accumulate) and softmax, overlapping compute with data movement\n- **FP8 support**: Native FP8 tensor cores for the QK^T matmul, with FP32 accumulation\n- **Asynchronous block-wise softmax**: Using the TMA (Tensor Memory Accelerator) for async global-to-shared memory copies\n\nFor reproducible results, use the configurations and measurements reported in the [FlashAttention 3 paper](<https://arxiv.org/abs/2407.08608>) and benchmark the target workload. This guide does not provide an independently reproducible measurement for the specific H100 SXM configuration previously listed here.\n\nThe integration with vLLM's PagedAttention is where it gets interesting, the ` flash_attn_with_kvcache ` kernel handles the block table indirection directly, so you get both paged memory management and IO-aware attention in a single fused kernel."},{"heading":"Speculative Decoding: Trading Compute for Latency","level":2,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#speculative-decoding-trading-compute-for-latency","content":"The idea: use a small \"draft\" model to generate N candidate tokens quickly, then verify them all at once with the large \"target\" model in a single forward pass. If the target model agrees with the draft tokens (checked via a modified rejection sampling scheme), you've effectively generated N tokens in the time it takes the target to do one forward pass.\n\n```python\n# Speculative decoding loop\ndef speculative_decode(target_model, draft_model, prompt, gamma=5):\n tokens = prompt\n while not is_done(tokens):\n # Draft phase: generate gamma candidates cheaply\n draft_tokens = []\n draft_probs = []\n for _ in range(gamma):\n logits = draft_model(tokens + draft_tokens)\n p = softmax(logits[-1])\n t = sample(p)\n draft_tokens.append(t)\n draft_probs.append(p)\n\n # Verify phase: single forward pass on target\n all_logits = target_model(tokens + draft_tokens)\n\n # Acceptance: modified rejection sampling\n accepted = 0\n for i in range(gamma):\n q = softmax(all_logits[len(tokens) + i])\n p = draft_probs[i]\n r = random()\n if r < min(1, q[draft_tokens[i]] / p[draft_tokens[i]]):\n accepted += 1\n else:\n # Resample from adjusted distribution\n adjusted = normalize(max(0, q - p))\n tokens.append(sample(adjusted))\n break\n else:\n # All accepted, sample one more from target\n tokens.extend(draft_tokens)\n q = softmax(all_logits[-1])\n tokens.append(sample(q))\n continue\n tokens.extend(draft_tokens[:accepted])\n```\n\nThe math works out so that the output distribution is **provably identical** to sampling from the target model alone. You're not sacrificing quality.\n\nIn practice, the acceptance rate depends heavily on how well the draft model matches the target. Observed results include:\n\n- Llama-3-8B drafting for Llama-3-70B: ~75% acceptance rate, ~2.3x speedup\n- Tiny custom 100M drafter fine-tuned on target outputs: ~65%, ~1.8x\n- Code completion tasks (high predictability): ~85%, ~3.1x\n\nThe sweet spot for ` gamma ` (speculation length) is usually 4-6. Go higher and the probability of at least one rejection per round approaches 1, wasting the extra draft compute."},{"heading":"Putting It All Together: Production Configuration","level":2,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#putting-it-all-together-production-configuration","content":"Here is a vLLM configuration that can bring a cluster from 12 tok/s/user to 45 tok/s/user:\n\n```bash\npython -m vllm.entrypoints.openai.api_server \\\n --model meta-llama/Llama-3-70B-Instruct \\\n --tensor-parallel-size 8 \\\n --max-num-batched-tokens 32768 \\\n --max-num-seqs 256 \\\n --block-size 16 \\\n --swap-space 4 \\\n --gpu-memory-utilization 0.92 \\\n --enable-prefix-caching \\\n --use-v2-block-manager \\\n --speculative-model meta-llama/Llama-3-8B-Instruct \\\n --num-speculative-tokens 5 \\\n --quantization awq\n```\n\nKey decisions:\n\n- ` --max-num-batched-tokens 32768 `: Allows up to 32K tokens in flight per iteration. Tuned to just under the OOM threshold.\n- ` --enable-prefix-caching `: Reuses KV-cache blocks for shared system prompts. With our 1,500-token system prompt, this saves ~480 KB per request.\n- ` --gpu-memory-utilization 0.92 `: Leave 8% headroom for activation memory spikes during prefill of long prompts.\n- ` --swap-space 4 `: 4 GB of CPU swap space per GPU for preempted requests, better than rejecting them outright."},{"heading":"The Numbers That Actually Matter","level":2,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#the-numbers-that-actually-matter","content":"On 8xA100 80GB, serving Llama-3-70B-Instruct (AWQ 4-bit):\n\n| Metric | Naive (TGI static) | Optimized (vLLM) | Improvement |\n| --- | --- | --- | --- |\n| Throughput (tok/s) | 480 | 1,850 | 3.85x |\n| Concurrent requests | 40 | 180 | 4.5x |\n| TTFT p50 | 820 ms | 210 ms | 3.9x |\n| TTFT p99 | 4,200 ms | 890 ms | 4.7x |\n| ITL p50 | 38 ms | 22 ms | 1.7x |\n| Cost per 1M tokens | $2.40 | $0.62 | 3.87x |\n\nThe cost reduction alone can pay for the engineering effort in about four days."},{"heading":"What's Coming Next","level":2,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#whats-coming-next","content":"Several areas are worth watching for the next round of optimizations:\n\n- **Disaggregated prefill/decode**: Running prefill on different GPUs than decode, since they have completely different compute profiles. Splitwise and DistServe papers look promising.\n- **Quantized KV-cache**: Storing KV-cache in FP8 or even INT4, separate from model weight quantization. Early results show minimal quality loss with 2x memory reduction.\n- **Ring attention**: For ultra-long contexts (100K+), distributing the attention computation across multiple GPUs along the sequence dimension.\n\nThe field is moving incredibly fast. Six months ago PagedAttention was novel; now it's table stakes. The infrastructure advantage goes to whoever can integrate these techniques fastest into their serving stack.\n\nFor anyone running inference in production, examining KV-cache utilization is a high-value activity. That is likely where the majority of costs concentrate."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#questions","content":""},{"heading":"What is the KV-cache and why does it matter for LLM inference?","level":3,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#question-1","content":"The KV-cache stores the key and value projection tensors from every previous token in the sequence, so the model doesn't need to recompute attention over the entire context at each generation step. Without it, inference would be quadratic in sequence length. The problem is it grows linearly with sequence length and batch size, often consuming more GPU memory than the model parameters themselves."},{"heading":"How does PagedAttention improve over traditional KV-cache management?","level":3,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#question-2","content":"PagedAttention borrows virtual memory concepts from operating systems, dividing the KV-cache into fixed-size blocks that can be stored non-contiguously in GPU memory. This eliminates internal fragmentation from over-allocation and enables memory sharing across requests via copy-on-write semantics, typically reducing KV-cache waste by 60-70%."},{"heading":"Is speculative decoding worth the complexity in production?","level":3,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#question-3","content":"It depends on your acceptance rate. With a well-matched draft model you can get 2-3x speedup on generation-heavy workloads, but if the acceptance rate drops below 60% you're burning FLOPs for nothing. Profile first with your actual traffic distribution before committing."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#article-tips","content":"1. **Tip:** If the KV-cache is eating more VRAM than the model weights themselves, you're probably serving long-context requests without ` PagedAttention `. Switch to vLLM and watch your memory utilization drop 60-70%.\n2. **Tip:** Continuous batching with ` --max-num-batched-tokens ` tuned to the GPU's memory ceiling is the single biggest throughput lever. Teams have been observed tripling their tokens/sec just by moving from static to continuous batching.\n3. **Tip:** When profiling inference, always measure *time-to-first-token* (TTFT) separately from *inter-token latency* (ITL). They respond to completely different optimizations and your users care about both.\n\n- llm\n- inference\n- kv-cache\n- vllm\n- paged-attention\n- flash-attention\n- speculative-decoding\n- gpu\n- optimization\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/llm-inference-optimization-kv-cache-paged-attention"},{"text":"PagedAttention","url":"https://arxiv.org/abs/2309.06180"},{"text":"vLLM","url":"https://vllm.ai/"},{"text":"FlashAttention 3 paper","url":"https://arxiv.org/abs/2407.08608"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention"},"headline":"LLM Inference Optimization: KV-Cache, PagedAttention, and GPU Memory Efficiency","description":"An analysis of LLM inference optimization mechanics, from KV-cache management and PagedAttention in vLLM to continuous batching, speculative decoding, and FlashAttention-3, with throughput measurements from production deployments.","datePublished":"2025-12-22","dateModified":"2026-10-21","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://lucioduran.com/favicons/og-default.png","url":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"AI/ML","keywords":"llm, inference, kv-cache, vllm, paged-attention, flash-attention, speculative-decoding, gpu, optimization","wordCount":1735,"timeRequired":"PT10M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/llm-inference-optimization-kv-cache-paged-attention"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"LLM Inference Optimization: KV-Cache, PagedAttention, and GPU Memory Efficiency","item":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"What is the KV-cache and why does it matter for LLM inference?","acceptedAnswer":{"@type":"Answer","text":"The KV-cache stores the key and value projection tensors from every previous token in the sequence, so the model doesn't need to recompute attention over the entire context at each generation step. Without it, inference would be quadratic in sequence length. The problem is it grows linearly with sequence length and batch size, often consuming more GPU memory than the model parameters themselves."}},{"@type":"Question","name":"How does PagedAttention improve over traditional KV-cache management?","acceptedAnswer":{"@type":"Answer","text":"PagedAttention borrows virtual memory concepts from operating systems, dividing the KV-cache into fixed-size blocks that can be stored non-contiguously in GPU memory. This eliminates internal fragmentation from over-allocation and enables memory sharing across requests via copy-on-write semantics, typically reducing KV-cache waste by 60-70%."}},{"@type":"Question","name":"Is speculative decoding worth the complexity in production?","acceptedAnswer":{"@type":"Answer","text":"It depends on your acceptance rate. With a well-matched draft model you can get 2-3x speedup on generation-heavy workloads, but if the acceptance rate drops below 60% you're burning FLOPs for nothing. Profile first with your actual traffic distribution before committing."}}]}]}],"contentHash":"eb5e50a6f2b24d0b7fbad777ec39732605fad973ba48e7f49b3cd4bf31f4e158","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/llm-inference-optimization-kv-cache-paged-attention","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":70780,"contentBytes":15051}}