Networking # TCP vs QUIC vs HTTP/3: Kernel Bypass, Zero-Copy, and Latency Optimization 2025-12-078 min read Lucio Durán Engineering Manager & AI Solutions Architect Also available in: [Español]() ## The testing setup Before examining the numbers, it is worth being transparent about the hardware, too many "benchmark" posts are run on a random EC2 instance with noisy neighbors. - **Server**: Bare metal, AMD EPYC 9454 (48C/96T), 256GB DDR5-4800 - **NIC**: Mellanox ConnectX-7 (100GbE), dual-port - **Client**: Same hardware, direct connection via DAC cable (no switch) - **OS**: Ubuntu 24.04 LTS, kernel 6.8.0 - **DPDK**: 23.11 LTS - **QUIC**: quiche 0.21 (Cloudflare's implementation) and msquic 2.3 - **io\_uring**: liburing 2.6 Each test ran for 60 seconds, repeated 10 times, with the median taken. CPU frequency scaling? Disabled. Turbo boost? Disabled. IRQ affinity? Pinned to specific cores. Hyperthreading? Off. No shortcuts. ## TCP: the old guard still has teeth Plain TCP serves as the baseline everyone compares against, and its performance with modern kernels is frequently underestimated. Standard ` send() `/` recv() ` over TCP with default kernel settings: ```c // Classic blocking TCP, nothing fancy int sock = socket(AF_INET, SOCK_STREAM, 0); setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one)); connect(sock, (struct sockaddr*)&addr, sizeof(addr)); // 64KB messages, 1M iterations for (int i = 0; i < 1000000; i++) { send(sock, buf, 65536, 0); } ``` **Results**: 14.2 Gbps throughput, 11.3us mean latency, 18.7us p99. Reasonable, but substantial performance remains untapped. The problem is that every ` send() ` call is a syscall, context switch into kernel mode, copy the buffer, context switch back. ### TCP + io\_uring: zero-copy sends [io\_uring]() with ` IORING_OP_SEND_ZC ` eliminates the buffer copy: ```c struct io_uring ring; io_uring_queue_init(256, &ring, 0); struct io_uring_sqe *sqe = io_uring_get_sqe(&ring); io_uring_prep_send_zc(sqe, sock, buf, 65536, 0, 0); sqe->user_data = 1; io_uring_submit(&ring); // Wait for completion AND notification that buffer is safe to reuse struct io_uring_cqe *cqe; io_uring_wait_cqe(&ring, &cqe); if (cqe->flags & IORING_CQE_F_NOTIF) { // Buffer can be reused now } io_uring_cqe_seen(&ring, cqe); ``` **Results**: 31.7 Gbps throughput, 5.8us mean latency, 9.2us p99. That's a 2.2x throughput improvement and ~50% latency reduction. The key insight: it's not just about avoiding the copy, batching submissions into the ring means fewer syscalls overall. One ` io_uring_submit() ` can push dozens of operations. ### TCP + DPDK: full kernel bypass When you absolutely, positively need every last nanosecond: ```c // DPDK packet construction, you own the entire stack struct rte_mbuf *pkt = rte_pktmbuf_alloc(mbuf_pool); struct rte_ether_hdr *eth = rte_pktmbuf_mtod(pkt, struct rte_ether_hdr*); // ... build ethernet, IP, TCP headers manually ... // Direct NIC access, no kernel involvement uint16_t nb_tx = rte_eth_tx_burst(port_id, queue_id, &pkt, 1); ``` **Results**: 82.4 Gbps throughput, 1.1us mean latency, 1.8us p99. This is the result of removing the kernel entirely. The cost is significant: TCP must be reimplemented in userspace. In practice, this approach is reserved for specific high-frequency trading workloads, and the debugging experience requires considerable expertise. ## QUIC: not just "TCP 2.0" QUIC is frequently characterized as "the future of networking," but the reality is more nuanced. It runs in userspace, which means no kernel bypass tricks, and it mandates TLS 1.3, so every packet carries encryption overhead. The test used [Cloudflare]()'s [quiche]() implementation, which is extensively proven in production: ```rust // quiche QUIC connection setup let mut config = quiche::Config::new(quiche::PROTOCOL_VERSION)?; config.set_application_protos(b"\x02h3")?; config.set_max_idle_timeout(30000); config.set_initial_max_data(10_000_000); config.set_initial_max_stream_data_bidi_local(1_000_000); config.enable_early_data(); let conn = quiche::connect(Some("server"), &scid, local, peer, &mut config)?; // Send data on a stream let written = conn.stream_send(stream_id, &buf[..65536], false)?; ``` **Results**: 8.7 Gbps throughput, 22.4us mean latency, 38.1us p99. Slower than plain TCP? Yes. And that is on a direct connection with zero packet loss. However, QUIC was not designed to win on this benchmark. It was designed for the real internet, where packets get dropped, connections migrate between networks, and head-of-line blocking kills multiplexed HTTP/2 streams. ### Where QUIC actually wins A tc netem profile simulating a typical mobile connection was configured: 50ms RTT, 1% packet loss, 0.5% reordering. | Protocol | Throughput | p99 Latency | Connection Setup | | --- | --- | --- | --- | | TCP + TLS 1.3 | 2.1 Gbps | 187ms | 3 RTT (SYN + TLS) | | QUIC | 3.8 Gbps | 112ms | 1 RTT (0-RTT resume) | | HTTP/3 | 3.6 Gbps | 98ms\* | 0 RTT (resume) | \*HTTP/3's p99 is lower because independent streams avoid HOL blocking. On lossy networks, QUIC's 0-RTT connection resumption and independent stream multiplexing absolutely destroy TCP. The 0-RTT case is particularly brutal for TCP, by the time TCP finishes its three-way handshake and TLS negotiation, QUIC has already sent the first response. ## Congestion control: the algorithm matters more than the protocol This aspect is frequently overlooked. The choice of congestion control algorithm has more impact on real-world performance than TCP vs QUIC in most scenarios. Four algorithms were tested on a 100ms RTT link with varying packet loss: ```bash # Switching congestion control on Linux sysctl -w net.ipv4.tcp_congestion_control=bbr # or cubic, reno, bbr3 (requires patched kernel) ``` ### BBRv3 vs CUBIC vs Reno, 100ms RTT At 0.1% loss: BBRv3 pushes 42 Gbps vs CUBIC's 28 Gbps. BBR's model-based approach doesn't back off as aggressively. At 1% loss: BBRv3 hits 18 Gbps, CUBIC drops to 8.3 Gbps. BBR is clearly superior here. At 5% loss: Here's the surprise, CUBIC recovers to 3.1 Gbps while BBRv3 only manages 2.4 Gbps. BBR's bandwidth probing gets confused by sustained high loss rates and oscillates between probing and draining. Investigation into why BBRv3 underperforms at high loss rates reveals an interesting finding. The answer is in how BBR estimates the bottleneck bandwidth. It uses a windowed max filter over recent delivery rates, but at 5% loss, the ACK stream becomes so noisy that the max filter keeps picking up outlier measurements, causing the sending rate to oscillate wildly. ```python # Simplified BBR bandwidth estimation class BBRBandwidthFilter: def __init__(self, window_size=10): self.window = deque(maxlen=window_size) def update(self, delivery_rate): self.window.append(delivery_rate) # At high loss, these samples are all over the place return max(self.window) # This max is the problem ``` ## HTTP/3: the full picture HTTP/3 is QUIC with HTTP semantics on top. The performance characteristics inherit from QUIC, but there's an additional layer worth discussing: QPACK header compression. QPACK is HTTP/3's answer to HPACK, but it's designed for QUIC's out-of-order delivery. This means it can't use the simple sequential state machine that HPACK uses. Instead, it has encoder and decoder streams that operate independently: ``` // QPACK dynamic table update Encoder Stream: INSERT name=":status" value="200" INSERT name="content-type" value="application/json" // Header block references dynamic table entries Request Stream: INDEXED ref=0 (":status: 200") INDEXED ref=1 ("content-type: application/json") ``` In practice, QPACK achieves 5-15% worse compression than HPACK on typical web traffic (because it can't risk blocking), but the elimination of head-of-line blocking more than compensates. ## The practical decision framework After all these benchmarks, here is a practical recommendation framework: **Use plain TCP + io\_uring** when you control both endpoints, the network is reliable, and you need raw throughput. Think: inter-datacenter replication, database wire protocols, service mesh data planes. **Use QUIC** when clients are on unreliable networks (mobile, WiFi), you need connection migration, or multiplexing multiple independent streams. Think: client-facing APIs, real-time communication, CDN edge. **Use DPDK** only, and I mean only, when you need sub-2us latency and you have a team that can maintain a custom networking stack. Think: HFT, telco packet processing, DPI appliances. **Avoid HTTP/3** unless there is a clear requirement. If the clients are browsers, they will negotiate it automatically via Alt-Svc headers. If the clients are controlled services, the HTTP framing overhead is likely not justified, use QUIC directly or remain with TCP. ## One more thing: GSO and GRO Generic Segmentation Offload and Generic Receive Offload are the unsung heroes of modern networking performance. They let you hand the NIC a 64KB buffer and let hardware split it into MTU-sized segments: ```c // Enable GSO on a UDP socket (for QUIC) int gso_size = 1200; // QUIC packet size setsockopt(fd, SOL_UDP, UDP_SEGMENT, &gso_size, sizeof(gso_size)); // Now you can send 64KB in a single syscall // The kernel/NIC splits it into 1200-byte UDP packets sendmsg(fd, &msg, 0); // msg contains 64KB buffer ``` This single optimization can improve QUIC throughput by 3.2x. Without GSO, each QUIC packet requires a separate ` sendmsg() ` call. With GSO, you batch 50+ packets into one syscall. This is how Cloudflare gets QUIC to perform at scale. The networking landscape is shifting fast. io\_uring is getting new zero-copy features every kernel release, QUIC implementations are maturing rapidly, and kernel bypass is becoming more accessible. But the fundamentals have not changed: measure the actual workload, on the actual network, with the actual hardware. Synthetic benchmarks are a starting point, not an answer. ## Frequently asked questions ### Is QUIC always faster than TCP? No. On low-latency, low-loss networks, TCP with kernel bypass (io\_uring or DPDK) often matches or beats QUIC because QUIC runs in userspace and carries encryption overhead. QUIC shines on lossy, high-latency connections and for connection migration scenarios. ### What is kernel bypass and when should I use it? Kernel bypass lets your application send and receive packets without going through the OS networking stack. Technologies like DPDK and io\_uring reduce context switches and memory copies. Use it when you need sub-millisecond latency or are pushing millions of packets per second. ### How does zero-copy networking work with io\_uring? io\_uring's IORING\_OP\_SEND\_ZC operation passes buffer references to the kernel instead of copying data. The kernel DMAs directly from your application buffer to the NIC. You get a completion event when the buffer is safe to reuse. ## Tips 1. **Tip:** Use ` io_uring ` with ` IORING_OP_SEND_ZC ` for zero-copy sends, it cuts syscall overhead by ~40% on high-throughput TCP workloads. 2. **Tip:** QUIC's 0-RTT resumption significantly reduces latency for mobile clients, but it is vulnerable to replay attacks, always validate ` early_data ` at the application layer. 3. **Tip:** When benchmarking with DPDK, pin your threads to isolated cores using ` isolcpus ` and disable IRQ balancing, otherwise your numbers are meaningless noise. 4. **Tip:** BBRv3 is not a silver bullet. On lossy links with \>2% packet loss, CUBIC still outperforms it. Profile your actual network conditions before switching. - tcp - quic - http3 - io\_uring - dpdk - zero-copy - kernel-bypass - networking - performance - congestion-control [Back to Blog]()