Rust # Async Rust Internals: Tokio Runtime Design, Work-Stealing, and Building a Minimal Runtime from Scratch 2025-11-158 min read Lucio Durán Engineering Manager & AI Solutions Architect Also available in: [Español]() ## The Async Mental Model In Rust, an ` async fn ` compiles down to a state machine that implements the ` Future ` trait: ```rust trait Future { type Output; fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll; } ``` Every ` .await ` point becomes a state transition. The compiler generates something roughly like this for a simple async function: ```rust async fn fetch_and_parse(url: &str) -> Result { let response = reqwest::get(url).await?; // state 0 → state 1 let text = response.text().await?; // state 1 → state 2 serde_json::from_str(&text).map_err(Into::into) } // Conceptually compiles to: enum FetchAndParse { State0 { url: String }, State1 { response: Response }, State2 { text: String }, } impl Future for FetchAndParse { type Output = Result; fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { loop { match self.state { State0 { ref url } => { match self.get_future.poll(cx) { Poll::Ready(resp) => self.transition_to(State1 { response: resp? }), Poll::Pending => return Poll::Pending, } } State1 { ref response } => { match self.text_future.poll(cx) { Poll::Ready(text) => self.transition_to(State2 { text: text? }), Poll::Pending => return Poll::Pending, } } State2 { ref text } => { return Poll::Ready(serde_json::from_str(text).map_err(Into::into)); } } } } } ``` The critical insight: when ` poll ` returns ` Poll::Pending `, the future goes to sleep. Nothing is checking on it. The ` Waker ` stored in ` Context ` is the mechanism by which the I/O system says "hey, this socket has data now, poll that future again." ## Tokio's Architecture [Tokio]() has four major subsystems: ``` ┌──────────────────────────────────────────┐ │ Tokio Runtime │ ├──────────┬──────────┬──────────┬─────────┤ │ Scheduler│ I/O │ Timer │ Blocking│ │ (work- │ Driver │ Driver │ Thread │ │ stealing)│ (epoll/ │ (timer │ Pool │ │ │ kqueue/ │ wheel) │ │ │ │ io_uring)│ │ │ └──────────┴──────────┴──────────┴─────────┘ ``` ### The Work-Stealing Scheduler Tokio runs N worker threads (default: number of CPU cores). Each worker has a local run queue (a fixed-size ring buffer of 256 tasks) plus access to a global injection queue. When a worker finishes polling a task: 1. Check its local queue for the next task 2. If empty, check the global injection queue 3. If empty, steal from another worker's local queue (take half their tasks) 4. If everything's empty, park the thread ```rust // Simplified work-stealing loop fn worker_loop(worker: &Worker, shared: &Shared) { loop { // 1. Local queue if let Some(task) = worker.local_queue.pop() { task.poll(); continue; } // 2. Global queue if let Some(task) = shared.global_queue.pop() { // Also grab a batch to amortize lock overhead while let Some(extra) = shared.global_queue.try_pop() { worker.local_queue.push(extra); } task.poll(); continue; } // 3. Steal from a random other worker let victim = random_worker(shared.workers.len()); if let Some(stolen) = shared.workers[victim].local_queue.steal_half() { stolen.first().poll(); for task in stolen.rest() { worker.local_queue.push(task); } continue; } // 4. Nothing to do, park worker.park(); } } ``` The steal-half strategy is important. If worker A has 100 tasks queued and worker B is idle, B steals 50 tasks at once. This amortizes the synchronization cost and quickly balances the load. I profiled this with ` tokio-console ` on a production service handling 50K req/s. The steal rate stabilized at about 3% of task executions, meaning workers were well-balanced and rarely needed to steal. When I introduced a CPU-heavy task on one endpoint, the steal rate jumped to 18%, and latency on *other* endpoints increased. That's when I moved the CPU work to ` spawn_blocking `. ### The I/O Driver Tokio's I/O driver wraps the OS event notification system, ` epoll ` on Linux, ` kqueue ` on macOS, ` io_uring ` on newer Linux kernels (behind a feature flag). The driver runs on a dedicated thread (or shares time with worker threads, depending on configuration). ```rust // Simplified I/O driver loop fn io_driver_loop(driver: &mut IoDriver) { let mut events = Events::with_capacity(1024); loop { // Block until I/O events arrive or timeout driver.poll.poll(&mut events, Some(Duration::from_millis(61)))?; for event in &events { let token = event.token(); if let Some(waker) = driver.registry.get_waker(token) { // Wake the future that was waiting for this I/O waker.wake(); } } } } ``` When you call ` TcpStream::read `, Tokio: 1. Attempts a non-blocking read 2. If ` WouldBlock `, registers interest with epoll via the I/O driver 3. Returns ` Poll::Pending ` with the waker stored in the registry 4. When data arrives, epoll fires, the driver wakes the task 5. The scheduler re-polls the future, and this time the read succeeds ### The Timer Wheel Tokio's timer system uses a hierarchical timer wheel, the same data structure used in Linux kernel timers. It's a multi-level array where each level covers a different time granularity: ``` Level 0: 64 slots × 1ms = 64ms range Level 1: 64 slots × 64ms = ~4s range Level 2: 64 slots × 4s = ~4min range Level 3: 64 slots × 4min = ~4.5hr range Level 4: 64 slots × 4.5hr = ~12 day range Level 5: 64 slots × 12d = ~2 year range ``` Inserting and cancelling a timer is O(1). Advancing the wheel to fire expired timers is amortized O(1) per timer fired. This is dramatically better than a binary heap (O(log n)) when you have thousands of active timers, which is common in connection timeout management. ```rust // How tokio::time::sleep works under the hood pub async fn sleep(duration: Duration) { let deadline = Instant::now() + duration; let entry = timer_wheel.insert(deadline); poll_fn(|cx| { if Instant::now() >= deadline { Poll::Ready(()) } else { entry.set_waker(cx.waker().clone()); Poll::Pending } }).await; } ``` ### The Waker Mechanism The ` Waker ` is the glue that connects I/O readiness to task scheduling. When a future returns ` Poll::Pending `, it stores the waker. When the event happens, the waker is called, which pushes the task back onto the scheduler's queue. ```rust // Tokio's waker implementation (simplified) impl Wake for TaskHarness { fn wake(self: Arc) { // Set the NOTIFIED flag self.state.fetch_or(NOTIFIED, Ordering::AcqRel); // Push task to the scheduler self.scheduler.schedule(self.clone()); } } ``` The ` Arc ` is critical, it means the waker keeps the task alive and can be cloned cheaply. Multiple I/O sources can hold wakers for the same task (think ` select! `), and whichever fires first wakes the task. ## Building a Minimal Runtime Let's build the simplest possible async runtime to nail down the concepts. No work-stealing, no I/O driver, just a single-threaded executor that can run futures to completion. ```rust use std::collections::VecDeque; use std::future::Future; use std::pin::Pin; use std::sync::{Arc, Mutex}; use std::task::{Context, Poll, Wake, Waker}; type BoxFuture = Pin + Send>>; struct MiniRuntime { queue: Arc>>>, } struct Task { future: Mutex>, queue: Arc>>>, } impl Wake for Task { fn wake(self: Arc) { // Re-enqueue the task when woken self.queue.lock().unwrap().push_back(self.clone()); } } impl MiniRuntime { fn new() -> Self { MiniRuntime { queue: Arc::new(Mutex::new(VecDeque::new())), } } fn spawn(&self, future: impl Future

+ Send + 'static) { let task = Arc::new(Task { future: Mutex::new(Some(Box::pin(future))), queue: self.queue.clone(), }); self.queue.lock().unwrap().push_back(task); } fn run(&self) { loop { let task = { let mut queue = self.queue.lock().unwrap(); match queue.pop_front() { Some(task) => task, None => break, // No more tasks } }; let waker = Waker::from(task.clone()); let mut cx = Context::from_waker(&waker); let mut future_slot = task.future.lock().unwrap(); if let Some(mut future) = future_slot.take() { match future.as_mut().poll(&mut cx) { Poll::Ready(()) => {} // Task complete, drop it Poll::Pending => { // Put the future back, it'll be re-enqueued when woken *future_slot = Some(future); } } } } } } ``` This is about 50 lines and it actually works for futures that wake themselves synchronously. To make it useful for real I/O, you'd need to add an epoll loop that calls the appropriate wakers when file descriptors become ready, essentially the I/O driver. ```rust fn main() { let rt = MiniRuntime::new(); rt.spawn(async { println!("Task 1: start"); // yield_now would re-enqueue via waker println!("Task 1: end"); }); rt.spawn(async { println!("Task 2: start"); println!("Task 2: end"); }); rt.run(); } ``` ## Real-World Gotchas A few things I've hit in production that stem from not understanding these internals: **Holding a MutexGuard across .await:** This blocks the worker thread, and since Tokio is cooperative, no other task on that worker can run until the lock is released. Use ` tokio::sync::Mutex ` if you must hold a lock across await points. ```rust // BAD, blocks the worker thread let data = std_mutex.lock().unwrap(); some_async_operation(&data).await; drop(data); // GOOD, async-aware mutex let data = tokio_mutex.lock().await; some_async_operation(&data).await; drop(data); ``` **Too many tasks with short polls:** Each task has overhead, the ` Arc `, the waker allocation, the scheduling. If you spawn 100,000 tasks that each do one tiny operation, the scheduling overhead dominates. Batch them. **Forgetting ` spawn_blocking ` for file I/O:** ` std::fs ` operations are synchronous and will block the worker thread. Tokio's ` tokio::fs ` is just a wrapper around ` spawn_blocking ` + ` std::fs `. Use it, or use ` spawn_blocking ` directly. ```rust // This blocks a worker thread, don't do this in async context let data = std::fs::read_to_string("config.json")?; // This runs on the blocking thread pool let data = tokio::fs::read_to_string("config.json").await?; ``` ## Summary Tokio's design is a masterclass in systems programming trade-offs. The work-stealing scheduler gives you CPU efficiency. The I/O driver gives you scalability. The timer wheel gives you performance with thousands of concurrent timeouts. And the entire thing is built on Rust's ownership model, which means data races are caught at compile time rather than at 3 AM in production. Understanding these internals won't change how you write basic async code. But it will change how you debug it, how you structure your task hierarchy, and how you avoid the performance cliffs that catch everyone eventually. ## Frequently asked questions ### Why does Rust need an external runtime for async instead of building one into the language? Rust's design philosophy separates the language primitives (Future trait, async/await syntax, Pin) from the executor that drives them. This means you can use Tokio for network servers, smol for lightweight tasks, or Embassy for embedded systems, all with the same async/await syntax. Baking a runtime into the language would force compromises that work for nobody. ### How does Tokio's work-stealing scheduler differ from OS thread scheduling? OS schedulers handle preemptive multitasking, they can interrupt threads at any point. Tokio's scheduler is cooperative, tasks must yield at .await points. Work-stealing means idle worker threads can grab tasks from busy workers' queues, improving CPU utilization. The combination gives you M:N threading, many tasks on fewer OS threads, without preemption overhead. ### When should I use tokio::spawn vs tokio::spawn\_blocking? Use tokio::spawn for async work that will .await on I/O or other futures. Use spawn\_blocking for synchronous code that would block the thread, CPU-intensive computation, synchronous file I/O, or calling C libraries. spawn\_blocking runs on a separate thread pool so it won't starve the async workers. ## Tips 1. **LuchoTip:** Never call ` .await ` inside a ` spawn_blocking ` closure, and never call blocking code inside an async task without ` spawn_blocking `. Mixing these two worlds is the #1 cause of mysterious Tokio deadlocks I've debugged. 2. **LuchoTip:** If your Tokio app's latency spikes under load, check ` tokio_unstable ` + ` tokio-console ` first. Nine times out of ten it's a task holding a ` MutexGuard ` across an ` .await ` point, starving the work-stealing scheduler. 3. **LuchoTip:** Set ` worker_threads ` to your CPU count, not higher. More threads than cores means context-switching overhead eats the concurrency gains. I benchmarked a 16-core machine with 16 vs 32 workers, 32 was 11% *slower*. - rust - async - tokio - runtime - work-stealing - futures - waker - io-driver - concurrency - systems-programming [Back to Blog]()