<!-- Canonical: https://lucioduran.com/topics/llm-inference
Language: en
Content-SHA256: 6e91ff0ce4b7c08ef20bfbc0f6b988a3a6e3e70f2f8600de0c5dbe844d76b100
-->

# AI inference: latency, memory and quality

Optimizing inference requires observing what changes for the task and for the person receiving the answer. This collection brings together local experiments on latency, memory, quality, caching and streaming delivery, with published configurations and reports to interpret each result.

By [Lucio Durán](<https://lucioduran.com/>) · Uruguay

## Which decision are you making?

### When measuring user wait time

Separate time to first token, time between tokens and total duration. LLM Serving Bench records actual token arrivals; LLM Stream Backpressure examines slow consumers, buffer limits and cancellation.

### When reducing memory or running locally

Compare quality and latency alongside weight size. Quantization Quality Lab studies MiniLM on CPU; WebGPU Local Inference demonstrates browser execution with a WebAssembly fallback.

### When reusing or routing answers

Account for errors and policy cost. LLM Semantic Cache evaluates correct and incorrect hits with isolation and invalidation; AI Model Routing Policy retains negative results and injected failures.

## Projects and evidence

Each project page provides code, environment details, published results and limitations. The experiments describe those configurations; they do not establish results in client systems.

- ### [LLM Serving Bench](<https://lucioduran.com/projects/llm-serving-bench>)

  CPU inference measured over HTTP/SSE: per-token arrivals, TTFT, TPOT, latency and memory. Eight SmolLM2 configurations with raw evidence and offline replay.

  Python · License: MIT

  [Read the project](<https://lucioduran.com/projects/llm-serving-bench>) · [Repository](<https://github.com/lucioduran/llm-serving-bench>) · [Release v0.1.0](<https://github.com/lucioduran/llm-serving-bench/releases/tag/v0.1.0>)
- ### [LLM Stream Backpressure](<https://lucioduran.com/projects/llm-stream-backpressure>)

  Bounded HTTP/SSE streaming with slow-consumer backpressure, interleaved tool events and measured cancellation behavior over real local sockets.

  Python · License: MIT

  [Read the project](<https://lucioduran.com/projects/llm-stream-backpressure>) · [Repository](<https://github.com/lucioduran/llm-stream-backpressure>) · [Release v0.1.0](<https://github.com/lucioduran/llm-stream-backpressure/releases/tag/v0.1.0>)
- ### [Quantization Quality Lab](<https://lucioduran.com/projects/quantization-quality-lab>)

  A real MiniLM CPU quantization study separating weight size, process memory, retrieval quality and latency, with raw regressions and output drift preserved.

  Python · License: MIT

  [Read the project](<https://lucioduran.com/projects/quantization-quality-lab>) · [Repository](<https://github.com/lucioduran/quantization-quality-lab>) · [Release v1.0.0](<https://github.com/lucioduran/quantization-quality-lab/releases/tag/v1.0.0>)
- ### [WebGPU Local Inference](<https://lucioduran.com/projects/webgpu-local-inference>)

  Inspectable browser ticket classification with native WebGPU, real WebAssembly fallback and recorded cached startup during a local asset outage.

  JavaScript · License: MIT

  [Read the project](<https://lucioduran.com/projects/webgpu-local-inference>) · [Repository](<https://github.com/lucioduran/webgpu-local-inference>) · [Release v0.1.0](<https://github.com/lucioduran/webgpu-local-inference/releases/tag/v0.1.0>)
- ### [LLM Semantic Cache](<https://lucioduran.com/projects/llm-semantic-cache>)

  SQLite exact and semantic answer caching with user, permission and version isolation, transactional invalidation and local evaluation of correct and incorrect reuse.

  Python · License: MIT

  [Read the project](<https://lucioduran.com/projects/llm-semantic-cache>) · [Repository](<https://github.com/lucioduran/llm-semantic-cache>) · [Release v0.1.0](<https://github.com/lucioduran/llm-semantic-cache/releases/tag/v0.1.0>)
- ### [AI Model Routing Policy](<https://lucioduran.com/projects/ai-model-routing-policy>)

  Routing between FLAN-T5-small and base with development-only fitting, held-out evaluation, measured router cost and injected local outages. Includes its negative result.

  Python · License: MIT

  [Read the project](<https://lucioduran.com/projects/ai-model-routing-policy>) · [Repository](<https://github.com/lucioduran/ai-model-routing-policy>) · [Release v0.1.0](<https://github.com/lucioduran/ai-model-routing-policy/releases/tag/v0.1.0>)