LLM Stream Backpressure
Bounded HTTP/SSE streaming with slow-consumer backpressure, interleaved tool events and measured cancellation behavior over real local sockets.
Problem and relevance to agents
A slow client can hide growing queues and abandoned upstream work. Closing its connection does not establish that an upstream provider stopped computing.
Explicit frame limits, bounded queues and adapter cancellation contracts make stream cleanup observable before integrating a real model provider.
Architecture
- Separate each request. Every connection owns a producer, bounded FIFO, socket writer and disconnect monitor. Token and tool events share an ordered stream.
- Propagate backpressure. Bounded frames and a drain-aware writer propagate waiting after socket buffers fill. A timeout ends writes to a stalled peer.
- Handle cancellation honestly. Cancellable producers stop before adapter cleanup. Unsupported cancellation drains finite remaining work into a discard consumer.
What this demonstrates
- Seven frozen cases executed over real loopback sockets, including failures and three concurrent requests.
- Eighteen tests cover cancellation races, stalled writes, interleaving, malformed requests, bounded buffering and evidence tampering.
- Raw event timestamps distinguish client close, server detection and completed upstream work.
Published results
One run per case was recorded on an Intel Xeon W-3223 with 48 GiB RAM, macOS 26.7.1 and Python 3.12.11.
The producer is deterministic synthetic work, not LLM inference. Instrumented event traces record queue occupancy, enqueue waiting, socket buffering and client delivery. Received counts include terminal error and completion events.
Observed stream cases
| Case | Produced / received | Queue peak | Completed events after disconnect detection |
|---|---|---|---|
| Fast | 300 / 301 | 4 | 0 |
| Slow | 300 / 301 | 4 | 0 |
| Supported cancellation | 5 / 5 | 1 | 0 |
| Unsupported cancellation | 100 / 5 | 2 | 95 |
| Upstream error | 15 / 17 | 2 | 0 |
| Client abort | 12 / 5 | 4 | 0 |
| Three concurrent requests | 80 / 81 each | 2 | 0 |
The slow case took 1,417 ms versus 479 ms for fast consumption; maximum enqueue waiting rose from 2.54 ms to 15.35 ms. These single-run measurements demonstrate transport behavior, not inference speed or statistical latency guarantees.
Maximum observed write buffering was 68,233 bytes. The slow case recorded 18,947,008 bytes of traced Python peak memory, including retained client output and trace records; this is not server-only memory.
The initial run exposed one residual event during sender cleanup. Its raw evidence is retained, and the corrected implementation was rerun with the same matrix. Preserved initial measurement.
CI passed on Ubuntu 24.04. macOS evidence comes from actual local x86_64 execution; hosted macOS and ARM compatibility were not verified.
Reproduce
Use uv 0.8.23 and Python 3.12.11. The setup command creates a repository-local environment; uv must be able to locate the matching interpreter. Install that Python version first if it is unavailable in uv's download catalog. Initial setup and test dependencies may need Internet access. Replay and unittest tests use only the standard library. They require no model downloads, API credentials or external inference service.
After environment setup, the replay command works offline. Replay is offline and recomputes metrics from saved traces; the small test suite opens local loopback sockets. Do not use Python -O because the verifier uses assertions. Follow the README for fresh measurements under the shared benchmark lock on the original host.
Verify saved evidence
git clone https://github.com/lucioduran/llm-stream-backpressure.git
cd llm-stream-backpressure
git checkout b16020bdc459d56d2628b1655fabc3de98f6dd22
UV_CACHE_DIR="$PWD/.tools/uv-cache" UV_PYTHON_INSTALL_DIR="$PWD/.tools/python" uv venv --python 3.12.11 .venv
PYTHONPATH=src .venv/bin/python experiments/replay.pyRun regression tests
PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -vScope and limitations
- The server accepts only GET /events HTTP/1.0 on loopback. TLS, authentication, proxies, keepalive and production deployment are outside scope.
- Operating-system buffers can absorb work before backpressure reaches the producer. No bound on total process or kernel memory is established.
- Unsupported adapters must finish a finite work budget. A contract-violating adapter can hang cleanup, and closing a socket cannot guarantee cancellation of remote compute.
- The fixture client retains finite output in memory. Timings include instrumentation and scheduling effects; provider behavior, billing and energy were not measured.
Sources and revision
The corrected measurement used commit ae5a1361b7df99eeab67e5609ca9130fba7ce45d and ran from 2026-11-09T20:21:26.925418+00:00 to 2026-11-09T20:21:29.396323+00:00. The report binds source, matrix and trace hashes. The final release adds publication and CI changes without altering the measured transport implementation.
- Reviewed revision:
b16020bdc459d56d2628b1655fabc3de98f6dd22 - Revision recorded by the experiment:
ae5a1361b7df99eeab67e5609ca9130fba7ce45d - Report timestamp (UTC):
- Release published: v0.1.0 ·
- Transport measurement report SHA-256:
63fc4cb950369f2d7f8671d29710478af7718557afdf422dc05b28eb8fd6d082