AI inference: latency, memory and quality
Optimizing inference requires observing what changes for the task and for the person receiving the answer. This collection brings together local experiments on latency, memory, quality, caching and streaming delivery, with published configurations and reports to interpret each result.
Which decision are you making?
When measuring user wait time
Separate time to first token, time between tokens and total duration. LLM Serving Bench records actual token arrivals; LLM Stream Backpressure examines slow consumers, buffer limits and cancellation.
When reducing memory or running locally
Compare quality and latency alongside weight size. Quantization Quality Lab studies MiniLM on CPU; WebGPU Local Inference demonstrates browser execution with a WebAssembly fallback.
When reusing or routing answers
Account for errors and policy cost. LLM Semantic Cache evaluates correct and incorrect hits with isolation and invalidation; AI Model Routing Policy retains negative results and injected failures.
Projects and evidence
Each project page provides code, environment details, published results and limitations. The experiments describe those configurations; they do not establish results in client systems.
LLM Serving Bench
CPU inference measured over HTTP/SSE: per-token arrivals, TTFT, TPOT, latency and memory. Eight SmolLM2 configurations with raw evidence and offline replay.
LLM Stream Backpressure
Bounded HTTP/SSE streaming with slow-consumer backpressure, interleaved tool events and measured cancellation behavior over real local sockets.
Quantization Quality Lab
A real MiniLM CPU quantization study separating weight size, process memory, retrieval quality and latency, with raw regressions and output drift preserved.
WebGPU Local Inference
Inspectable browser ticket classification with native WebGPU, real WebAssembly fallback and recorded cached startup during a local asset outage.
LLM Semantic Cache
SQLite exact and semantic answer caching with user, permission and version isolation, transactional invalidation and local evaluation of correct and incorrect reuse.
AI Model Routing Policy
Routing between FLAN-T5-small and base with development-only fitting, held-out evaluation, measured router cost and injected local outages. Includes its negative result.