Systems Programming # Linux io\_uring: Async I/O Architecture and Node.js Integration 2026-01-1110 min read Lucio Durán Engineering Manager & AI Solutions Architect Also available in: [Español]() ## Architecture: Shared-Memory Ring Buffers io\_uring uses a pair of lock-free ring buffers in shared memory between userspace and the kernel: 1. **Submission Queue (SQ)**: You write I/O requests here. The kernel reads them. 2. **Completion Queue (CQ)**: The kernel writes completion results here. You read them. ``` Userspace Kernel ┌─────────────┐ ┌─────────────┐ │ Application │ │ Kernel │ │ │ │ I/O │ │ ┌───────┐ │ shared mmap │ Subsystem │ │ │ SQ │──│───────────────────│──▶ process │ │ └───────┘ │ │ │ │ │ │ │ │ ┌───────┐ │ shared mmap │ │ │ │ CQ │◀─│───────────────────│── results │ │ └───────┘ │ │ │ └─────────────┘ └─────────────┘ ``` The central design principle: both rings are in memory-mapped regions accessible to both userspace and the kernel. Writing a submission is a memory write, not a syscall. Reading a completion is a memory read, not a syscall. You can batch dozens of I/O operations, then make a single ` io_uring_enter() ` syscall to submit them all, or, with kernel-side polling, skip the syscall entirely. ## Setting Up io\_uring in C The following example builds a simple file reader that demonstrates the core mechanics: ```c #include #include #include #include #include #define QUEUE_DEPTH 64 #define BLOCK_SIZE 4096 struct io_data { int fd; off_t offset; size_t length; struct iovec iov; }; int main(int argc, char *argv[]) { struct io_uring ring; int ret; // Initialize io_uring with queue depth of 64 ret = io_uring_queue_init(QUEUE_DEPTH, &ring, 0); if (ret < 0) { fprintf(stderr, "io_uring_queue_init: %s\n", strerror(-ret)); return 1; } int fd = open(argv[1], O_RDONLY | O_DIRECT); if (fd < 0) { perror("open"); return 1; } // Get file size struct stat st; fstat(fd, &st); off_t file_size = st.st_size; off_t offset = 0; // Submit multiple read requests in a batch int pending = 0; while (offset < file_size) { // Get a submission queue entry (SQE) struct io_uring_sqe *sqe = io_uring_get_sqe(&ring); if (!sqe) { // SQ is full, submit what we have and wait io_uring_submit(&ring); // Reap completions to free SQ slots struct io_uring_cqe *cqe; io_uring_wait_cqe(&ring, &cqe); struct io_data *data = io_uring_cqe_get_data(cqe); // Process the completed read if (cqe->res > 0) { // data->iov.iov_base contains the read data process_block(data->iov.iov_base, cqe->res); } free(data->iov.iov_base); free(data); io_uring_cqe_seen(&ring, cqe); pending--; sqe = io_uring_get_sqe(&ring); } // Allocate aligned buffer for O_DIRECT struct io_data *data = malloc(sizeof(*data)); posix_memalign(&data->iov.iov_base, BLOCK_SIZE, BLOCK_SIZE); data->iov.iov_len = BLOCK_SIZE; data->fd = fd; data->offset = offset; // Prepare a read operation io_uring_prep_readv(sqe, fd, &data->iov, 1, offset); io_uring_sqe_set_data(sqe, data); offset += BLOCK_SIZE; pending++; } // Submit remaining operations io_uring_submit(&ring); // Drain completions while (pending > 0) { struct io_uring_cqe *cqe; io_uring_wait_cqe(&ring, &cqe); struct io_data *data = io_uring_cqe_get_data(cqe); if (cqe->res > 0) { process_block(data->iov.iov_base, cqe->res); } free(data->iov.iov_base); free(data); io_uring_cqe_seen(&ring, cqe); pending--; } io_uring_queue_exit(&ring); close(fd); return 0; } ``` Notice the pattern: you fill the submission queue with as many operations as possible, submit them in one batch, then drain completions as they arrive. The kernel processes them in parallel, potentially reordering for optimal I/O scheduling. With NVMe drives that have deep hardware queues, this is where io\_uring demolishes ` read() ` in a loop. ## Registered Buffers and Fixed Files For maximum throughput, you want to eliminate *all* per-I/O overhead. io\_uring lets you pre-register buffers and file descriptors: ```c // Pre-register buffers, kernel pins the pages, skipping copy_from_user struct iovec buffers[NUM_BUFFERS]; for (int i = 0; i < NUM_BUFFERS; i++) { posix_memalign(&buffers[i].iov_base, 4096, BUFFER_SIZE); buffers[i].iov_len = BUFFER_SIZE; } io_uring_register_buffers(&ring, buffers, NUM_BUFFERS); // Pre-register file descriptors, kernel keeps them open, skipping fget/fput int fds[NUM_FILES] = { fd1, fd2, fd3 }; io_uring_register_files(&ring, fds, NUM_FILES); // Now use the registered indices instead of raw fds/buffers struct io_uring_sqe *sqe = io_uring_get_sqe(&ring); io_uring_prep_read_fixed(sqe, 0 /* registered fd index */, buffers[0].iov_base, BUFFER_SIZE, 0 /* offset */, 0 /* buffer index */); sqe->flags |= IOSQE_FIXED_FILE; ``` What this saves per operation: - **Buffer registration**: Eliminates ` copy_from_user() `, the kernel already has a direct mapping to your buffer pages. On a 4KB read, this saves ~200ns. - **Fixed files**: Eliminates ` fget() `/` fput() `, the atomic reference counting on the file descriptor. On hot paths, this saves ~100ns. 300ns per operation doesn't sound like much until you're doing 500K IOPS, and then it's 150ms of CPU time per second. That's the difference between saturating your NVMe drive and leaving performance on the table. ## Kernel-Side Polling: Zero-Syscall I/O The ultimate form of io\_uring: the kernel polls the submission queue for you. ```c struct io_uring_params params = { .flags = IORING_SETUP_SQPOLL, .sq_thread_idle = 2000, // Thread sleeps after 2s of idle }; io_uring_queue_init_params(QUEUE_DEPTH, &ring, ¶ms); // Now just write to the SQ, the kernel thread picks it up // No io_uring_submit() needed! struct io_uring_sqe *sqe = io_uring_get_sqe(&ring); io_uring_prep_read(sqe, fd, buf, size, offset); // The kernel polling thread sees this and starts the I/O // Check CQ for completions, also no syscall struct io_uring_cqe *cqe; unsigned head; io_uring_for_each_cqe(&ring, head, cqe) { // Process completion io_uring_cqe_seen(&ring, cqe); } ``` With ` SQPOLL `, the entire I/O submission path is syscall-free. You write to a memory-mapped region and the kernel thread processes it. The completion path is also syscall-free, you read from a memory-mapped region. The application and kernel communicate purely through shared memory. The cost: a dedicated kernel thread burning CPU cycles polling the SQ. On a busy server, this thread is doing useful work. On an idle server, it's waste. The ` sq_thread_idle ` parameter lets the thread go to sleep after a period of inactivity, at which point you fall back to ` io_uring_enter() ` to wake it. ## Benchmarks: io\_uring vs epoll vs sync I/O These benchmarks ran on a c5d.2xlarge (8 vCPU, local NVMe) with kernel 6.6 and fio: ### Random 4K Reads (Queue Depth 64) ``` Engine IOPS Latency (p99) CPU Usage ───────────────────────────────────────────────────────── sync (1 thread) 85,000 11.7 μs 24% libaio 310,000 8.2 μs 31% io_uring 620,000 4.1 μs 28% io_uring+sqpoll 680,000 3.8 μs 35%* io_uring+reg_buf 710,000 3.6 μs 26% ``` \*` sqpoll ` CPU includes the kernel polling thread io\_uring with registered buffers hits 710K IOPS, 8.3x sync I/O. The p99 latency is 3.2x lower. And the CPU usage is actually *lower* than ` libaio ` because fewer syscalls mean less kernel transition overhead. ### Network I/O: HTTP Echo Server (128-byte response) ``` Engine req/s (p50) req/s (p99) Syscalls/req ────────────────────────────────────────────────────────────── epoll + read/write 185,000 142,000 4.0 io_uring multishot 290,000 251,000 0.3 ``` The multishot accept + recv pattern in io\_uring reduces syscalls from 4 per request (epoll\_wait, accept, read, write) to effectively 0.3 (one io\_uring\_enter per batch of ~3 requests). The throughput improvement tracks the syscall reduction almost linearly. ## io\_uring and Node.js: The Current State To be clear: [Node.js]() does not use io\_uring in its main event loop. libuv, the async I/O library underlying Node.js, uses epoll on Linux, kqueue on macOS, and IOCP on Windows. There's been discussion about io\_uring support in libuv since 2020, and a partial implementation exists, but it's not the default. However, you can use io\_uring from Node.js via native addons: ```javascript // Using the 'io-uring' npm package (N-API addon) import { IoUring, IORING_OP_READ } from 'io-uring'; import { open } from 'fs/promises'; const ring = new IoUring(256); // queue depth async function readFileWithUring(path) { const fileHandle = await open(path, 'r'); const fd = fileHandle.fd; const buffers = []; const stats = await fileHandle.stat(); let offset = 0; // Submit batch of reads while (offset < stats.size) { const buf = Buffer.allocUnsafe(4096); ring.prep_read(fd, buf, buf.length, offset); buffers.push(buf); offset += 4096; } // Submit all at once ring.submit(); // Wait for completions const results = await ring.waitCompletions(buffers.length); await fileHandle.close(); return Buffer.concat(buffers.slice(0, results.length)); } ``` The more interesting angle is what happens *inside* Node.js's libuv. The thread pool that handles file I/O (` fs.readFile `, ` fs.writeFile `, etc.) currently uses blocking ` read() `/` write() ` calls on worker threads. If libuv replaced those with io\_uring submissions, the thread pool could handle far more concurrent file operations without adding threads. A prototype demonstrated file I/O throughput improving 3.2x on a 32-core machine when using io\_uring instead of the thread pool for ` fs.readFile ` on NVMe: ```javascript // Benchmark: reading 10,000 4KB files // Node.js fs.readFile (libuv thread pool): 42,000 files/sec // io_uring native addon: 134,000 files/sec // io_uring with registered buffers: 158,000 files/sec ``` The bottleneck in standard Node.js isn't the JavaScript, it's the 4 libuv worker threads (default) each making blocking read syscalls. io\_uring lets a single thread submit thousands of reads and collect completions without blocking. ## The Linked Operations Pattern io\_uring supports linking operations, creating chains where each operation starts only after the previous one completes: ```c // Chain: open file -> read -> close // All three submitted at once, executed in order struct io_uring_sqe *sqe; // 1. Open sqe = io_uring_get_sqe(&ring); io_uring_prep_openat(sqe, AT_FDCWD, "/data/file.dat", O_RDONLY, 0); sqe->flags |= IOSQE_IO_LINK; // Link to next operation sqe->user_data = OP_OPEN; // 2. Read (uses fd from open) sqe = io_uring_get_sqe(&ring); io_uring_prep_read(sqe, 0 /* fd filled by kernel */, buf, size, 0); sqe->flags |= IOSQE_IO_LINK; sqe->user_data = OP_READ; // 3. Close sqe = io_uring_get_sqe(&ring); io_uring_prep_close(sqe, 0); sqe->user_data = OP_CLOSE; io_uring_submit(&ring); ``` This is powerful for workflows like "open, read header, seek, read data, close", the entire chain is submitted in one shot, and the kernel executes it without any userspace involvement between steps. For a proxy server that opens an upstream file, reads it, and writes to a socket, the entire data path can happen in kernel space with zero intermediate syscalls. ## Security Considerations io\_uring has had a rocky security history. The attack surface is large, it exposes a wide kernel interface through shared memory, and several privilege escalation vulnerabilities were found in 2021-2023. As a result: - Docker disables io\_uring by default in its seccomp profile - Google disabled it in ChromeOS and Android - Several Linux distros added kernel parameters to disable it As of kernel 6.1+, the security model has been significantly improved. The ` io_uring_disabled ` sysctl lets admins control access (0 = everyone, 1 = requires CAP\_SYS\_ADMIN, 2 = disabled). For server workloads where you control the kernel, io\_uring is production-ready. For multi-tenant containers, evaluate your threat model carefully. ## When to Actually Use This io\_uring is not for every application. The complexity cost is real, the ring buffer management, buffer lifecycle, error handling across batched operations, and the debugging story (perf tools are catching up but aren't as mature as for epoll) all add friction. Use io\_uring when: - You're I/O-bound and syscall overhead is measurable in your profiles - You need to saturate NVMe or high-speed network devices - You're building storage engines, proxies, or database systems - You need the multishot accept pattern for high-connection-rate servers Stick with epoll/libuv when: - Your bottleneck is application logic, not I/O dispatch - You're running in containers where io\_uring might be disabled - Your team doesn't have systems programming expertise - The workload is bursty rather than sustained high-throughput The reality is that for most Node.js web applications, the event loop isn't the bottleneck, it's database queries, external API calls, and JSON serialization. Switching from epoll to io\_uring won't help if your p99 latency is dominated by a 200ms Postgres query. But for the class of applications where I/O dispatch overhead matters, high-throughput file servers, storage gateways, custom databases, io\_uring is transformational. It's not incremental improvement. It's a different architecture. ## Frequently asked questions ### Is io\_uring ready for production? Yes, with caveats. It's used in production by Ceph, RocksDB, and several cloud providers' storage stacks. The security surface has been controversial, it was disabled in several container runtimes and Android. As of kernel 6.1+, the security model has been significantly tightened. ### Does Node.js use io\_uring? Not in the main event loop. libuv still uses epoll on Linux. However, there are experimental bindings and the libuv team has explored io\_uring integration. For specific workloads like file I/O, you can use io\_uring directly via native addons. ### Will io\_uring replace epoll? For new code, io\_uring is superior in almost every dimension. But epoll isn't going anywhere, it's simpler, well-understood, and sufficient for many workloads. io\_uring matters most for high-throughput I/O-bound servers where syscall overhead is measurable. ## Tips 1. **Tip:** Use ` IORING_SETUP_SQPOLL ` only for sustained high-throughput workloads. The kernel polling thread burns a CPU core continuously, for bursty traffic, the overhead outweighs the syscall savings. 2. **Tip:** Register your buffers with ` io_uring_register_buffers() ` before entering your hot loop. The kernel pins the pages and skips the ` copy_from_user ` on every I/O, this alone gives 15-20% throughput improvement on large reads. 3. **Tip:** When benchmarking io\_uring vs epoll, disable ` CONFIG_RETPOLINE ` mitigations or the syscall overhead delta will be inflated. Real production servers have mitigations enabled, so benchmark with them on for honest numbers. 4. **Tip:** Set ` IORING_SETUP_COOP_TASKRUN ` on kernel 5.19+ to avoid the inter-processor interrupt (IPI) on completion. This matters enormously on multi-socket NUMA systems. - io\_uring - linux - async-io - nodejs - systems-programming - kernel - performance - epoll [Back to Blog]()