LLM Serving Bench

Problem and relevance to agents

A tokens-per-second counter does not explain user waiting time or queue formation. This benchmark records each generated and received token to separate time to first token, decoding and total latency.

It supports agent latency budgets with measurements from a real server. Results identify the model, precision, hardware, concurrency and input length needed to interpret a performance number.

Architecture

1. Measured server. SmolLM2-135M-Instruct in FP32 on CPU, with one generation slot and one SSE event per token ID. Clients connect through loopback HTTP.
2. Controlled matrix. One or two CPU threads, one or two concurrent clients, and exactly 64 or 256 input tokens. Each configuration processes six recorded prompts after an excluded warmup.
3. Replayable evidence. Requests, IDs, arrival times, errors, RSS samples and model hashes are retained. A dependency-free verifier recomputes report metrics.

• Design and methodology
• HTTP/SSE protocol

What this demonstrates

• Token-level instrumentation with explicit metric boundaries.
• Configuration comparisons with recorded requests and controlled concurrency.
• Separation of observed resource use, assumed costs and unmeasured variables.

Published results

Published measurement on Intel Xeon W-3223, macOS, Python 3.11.13 and PyTorch 2.2.2. All 48 requests succeeded and generated 1,152 output tokens. Eight warmups are excluded.

TTFT includes connection setup, tokenization, queueing and prefill. TPOT divides the first-to-last token interval by the number of generation intervals. Percentiles interpolate six requests per configuration; throughput uses the complete configuration duration.

Two threads produced higher throughput in this sample. Doubling clients mostly added waiting because the server retains one generation slot. Six requests per configuration do not establish an SLA or a general server ranking.

Sampled peak RSS ranged from 838.7 to 1,023.5 MiB. Energy is unmeasured. Reported cost uses an illustrative USD 0.10 per machine-hour allocation; it is not a bill or observed expenditure. Original evidence.

Reproduce

Replay requires Python 3.11 and the standard library. Repeating inference requires model dependencies and downloading public weights; subsequent execution uses local files.

Environment commands use a POSIX shell. Choose a new output directory and run only one compute-intensive measurement at a time. The published lock describes Intel macOS; other systems may need different PyTorch wheels.

git clone https://github.com/lucioduran/llm-serving-bench.git
cd llm-serving-bench
git checkout f291bbe0e3fbd377438b56f4cbf4a7a808720d98
python -m serving_bench verify results/reference/report.json

python -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements-models.txt
python -m serving_bench download
python -m serving_bench run --output work/reproduction --hourly-usd 0.10
python -m serving_bench verify work/reproduction/report.json

Scope and limitations

• One model, host and fixed execution order. Client and server share CPU resources.
• Closed-loop load without continuous batching, GPU, arrival-rate saturation or answer-quality evaluation.
• Sampled RSS can miss peaks. Replay checks consistency; it neither measures timings again nor authenticates hardware.

• Inference optimization: KV-cache, PagedAttention and measurement

Sources and revision

Values on this page come from the linked report. They are not live measurements. The report records the executed revision; the reviewed revision includes the subsequent publication of that report.

• Reviewed revision: f291bbe0e3fbd377438b56f4cbf4a7a808720d98
• Revision recorded by the experiment: 1f78bb2b7926840bbec088c289d56285daac807c
• Report timestamp (UTC): 2026-10-21T21:52:58.127518+00:00
• Release published: v0.1.0 · 2026-10-21T21:57:30Z
• Corpus SHA-256: 39e2b0383320350aef0ae9fbde0f4c81ff9c35991c36148808c834fb34466e5a

• README and usage
• Original evidence
• Results and analysis
• Design and methodology
• HTTP/SSE protocol
• Models, revisions and licenses
• Measured environment dependencies
• MIT license