The Async Mental Model
In Rust, an async fn compiles down to a state machine that implements the Future trait:
trait Future {
type Output;
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
}
Every .await point becomes a state transition. The compiler generates something roughly like this for a simple async function:
async fn fetch_and_parse(url: &str) -> Result<Data> {
let response = reqwest::get(url).await?; // state 0 → state 1
let text = response.text().await?; // state 1 → state 2
serde_json::from_str(&text).map_err(Into::into)
}
// Conceptually compiles to:
enum FetchAndParse {
State0 { url: String },
State1 { response: Response },
State2 { text: String },
}
impl Future for FetchAndParse {
type Output = Result<Data>;
fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
loop {
match self.state {
State0 { ref url } => {
match self.get_future.poll(cx) {
Poll::Ready(resp) => self.transition_to(State1 { response: resp? }),
Poll::Pending => return Poll::Pending,
}
}
State1 { ref response } => {
match self.text_future.poll(cx) {
Poll::Ready(text) => self.transition_to(State2 { text: text? }),
Poll::Pending => return Poll::Pending,
}
}
State2 { ref text } => {
return Poll::Ready(serde_json::from_str(text).map_err(Into::into));
}
}
}
}
}
The critical insight: when poll returns Poll::Pending, the future goes to sleep. Nothing is checking on it. The Waker stored in Context is the mechanism by which the I/O system says "hey, this socket has data now, poll that future again."
Tokio's Architecture
Tokio has four major subsystems:
┌──────────────────────────────────────────┐
│ Tokio Runtime │
├──────────┬──────────┬──────────┬─────────┤
│ Scheduler│ I/O │ Timer │ Blocking│
│ (work- │ Driver │ Driver │ Thread │
│ stealing)│ (epoll/ │ (timer │ Pool │
│ │ kqueue/ │ wheel) │ │
│ │ io_uring)│ │ │
└──────────┴──────────┴──────────┴─────────┘
The Work-Stealing Scheduler
Tokio runs N worker threads (default: number of CPU cores). Each worker has a local run queue (a fixed-size ring buffer of 256 tasks) plus access to a global injection queue.
When a worker finishes polling a task:
- Check its local queue for the next task
- If empty, check the global injection queue
- If empty, steal from another worker's local queue (take half their tasks)
- If everything's empty, park the thread
// Simplified work-stealing loop
fn worker_loop(worker: &Worker, shared: &Shared) {
loop {
// 1. Local queue
if let Some(task) = worker.local_queue.pop() {
task.poll();
continue;
}
// 2. Global queue
if let Some(task) = shared.global_queue.pop() {
// Also grab a batch to amortize lock overhead
while let Some(extra) = shared.global_queue.try_pop() {
worker.local_queue.push(extra);
}
task.poll();
continue;
}
// 3. Steal from a random other worker
let victim = random_worker(shared.workers.len());
if let Some(stolen) = shared.workers[victim].local_queue.steal_half() {
stolen.first().poll();
for task in stolen.rest() {
worker.local_queue.push(task);
}
continue;
}
// 4. Nothing to do, park
worker.park();
}
}
The steal-half strategy is important. If worker A has 100 tasks queued and worker B is idle, B steals 50 tasks at once. This amortizes the synchronization cost and quickly balances the load.
I profiled this with tokio-console on a production service handling 50K req/s. The steal rate stabilized at about 3% of task executions, meaning workers were well-balanced and rarely needed to steal. When I introduced a CPU-heavy task on one endpoint, the steal rate jumped to 18%, and latency on other endpoints increased. That's when I moved the CPU work to spawn_blocking.
The I/O Driver
Tokio's I/O driver wraps the OS event notification system, epoll on Linux, kqueue on macOS, io_uring on newer Linux kernels (behind a feature flag). The driver runs on a dedicated thread (or shares time with worker threads, depending on configuration).
// Simplified I/O driver loop
fn io_driver_loop(driver: &mut IoDriver) {
let mut events = Events::with_capacity(1024);
loop {
// Block until I/O events arrive or timeout
driver.poll.poll(&mut events, Some(Duration::from_millis(61)))?;
for event in &events {
let token = event.token();
if let Some(waker) = driver.registry.get_waker(token) {
// Wake the future that was waiting for this I/O
waker.wake();
}
}
}
}
When you call TcpStream::read, Tokio:
- Attempts a non-blocking read
- If
WouldBlock, registers interest with epoll via the I/O driver - Returns
Poll::Pendingwith the waker stored in the registry - When data arrives, epoll fires, the driver wakes the task
- The scheduler re-polls the future, and this time the read succeeds
The Timer Wheel
Tokio's timer system uses a hierarchical timer wheel, the same data structure used in Linux kernel timers. It's a multi-level array where each level covers a different time granularity:
Level 0: 64 slots × 1ms = 64ms range
Level 1: 64 slots × 64ms = ~4s range
Level 2: 64 slots × 4s = ~4min range
Level 3: 64 slots × 4min = ~4.5hr range
Level 4: 64 slots × 4.5hr = ~12 day range
Level 5: 64 slots × 12d = ~2 year range
Inserting and cancelling a timer is O(1). Advancing the wheel to fire expired timers is amortized O(1) per timer fired. This is dramatically better than a binary heap (O(log n)) when you have thousands of active timers, which is common in connection timeout management.
// How tokio::time::sleep works under the hood
pub async fn sleep(duration: Duration) {
let deadline = Instant::now() + duration;
let entry = timer_wheel.insert(deadline);
poll_fn(|cx| {
if Instant::now() >= deadline {
Poll::Ready(())
} else {
entry.set_waker(cx.waker().clone());
Poll::Pending
}
}).await;
}
The Waker Mechanism
The Waker is the glue that connects I/O readiness to task scheduling. When a future returns Poll::Pending, it stores the waker. When the event happens, the waker is called, which pushes the task back onto the scheduler's queue.
// Tokio's waker implementation (simplified)
impl Wake for TaskHarness {
fn wake(self: Arc<Self>) {
// Set the NOTIFIED flag
self.state.fetch_or(NOTIFIED, Ordering::AcqRel);
// Push task to the scheduler
self.scheduler.schedule(self.clone());
}
}
The Arc<TaskHarness> is critical, it means the waker keeps the task alive and can be cloned cheaply. Multiple I/O sources can hold wakers for the same task (think select!), and whichever fires first wakes the task.
Building a Minimal Runtime
Let's build the simplest possible async runtime to nail down the concepts. No work-stealing, no I/O driver, just a single-threaded executor that can run futures to completion.
use std::collections::VecDeque;
use std::future::Future;
use std::pin::Pin;
use std::sync::{Arc, Mutex};
use std::task::{Context, Poll, Wake, Waker};
type BoxFuture = Pin<Box<dyn Future<Output = ()> + Send>>;
struct MiniRuntime {
queue: Arc<Mutex<VecDeque<Arc<Task>>>>,
}
struct Task {
future: Mutex<Option<BoxFuture>>,
queue: Arc<Mutex<VecDeque<Arc<Task>>>>,
}
impl Wake for Task {
fn wake(self: Arc<Self>) {
// Re-enqueue the task when woken
self.queue.lock().unwrap().push_back(self.clone());
}
}
impl MiniRuntime {
fn new() -> Self {
MiniRuntime {
queue: Arc::new(Mutex::new(VecDeque::new())),
}
}
fn spawn(&self, future: impl Future<Output = ()> + Send + 'static) {
let task = Arc::new(Task {
future: Mutex::new(Some(Box::pin(future))),
queue: self.queue.clone(),
});
self.queue.lock().unwrap().push_back(task);
}
fn run(&self) {
loop {
let task = {
let mut queue = self.queue.lock().unwrap();
match queue.pop_front() {
Some(task) => task,
None => break, // No more tasks
}
};
let waker = Waker::from(task.clone());
let mut cx = Context::from_waker(&waker);
let mut future_slot = task.future.lock().unwrap();
if let Some(mut future) = future_slot.take() {
match future.as_mut().poll(&mut cx) {
Poll::Ready(()) => {} // Task complete, drop it
Poll::Pending => {
// Put the future back, it'll be re-enqueued when woken
*future_slot = Some(future);
}
}
}
}
}
}
This is about 50 lines and it actually works for futures that wake themselves synchronously. To make it useful for real I/O, you'd need to add an epoll loop that calls the appropriate wakers when file descriptors become ready, essentially the I/O driver.
fn main() {
let rt = MiniRuntime::new();
rt.spawn(async {
println!("Task 1: start");
// yield_now would re-enqueue via waker
println!("Task 1: end");
});
rt.spawn(async {
println!("Task 2: start");
println!("Task 2: end");
});
rt.run();
}
Real-World Gotchas
A few things I've hit in production that stem from not understanding these internals:
Holding a MutexGuard across .await: This blocks the worker thread, and since Tokio is cooperative, no other task on that worker can run until the lock is released. Use tokio::sync::Mutex if you must hold a lock across await points.
// BAD, blocks the worker thread
let data = std_mutex.lock().unwrap();
some_async_operation(&data).await;
drop(data);
// GOOD, async-aware mutex
let data = tokio_mutex.lock().await;
some_async_operation(&data).await;
drop(data);
Too many tasks with short polls: Each task has overhead, the Arc<TaskHarness>, the waker allocation, the scheduling. If you spawn 100,000 tasks that each do one tiny operation, the scheduling overhead dominates. Batch them.
Forgetting spawn_blocking for file I/O: std::fs operations are synchronous and will block the worker thread. Tokio's tokio::fs is just a wrapper around spawn_blocking + std::fs. Use it, or use spawn_blocking directly.
// This blocks a worker thread, don't do this in async context
let data = std::fs::read_to_string("config.json")?;
// This runs on the blocking thread pool
let data = tokio::fs::read_to_string("config.json").await?;
Summary
Tokio's design is a masterclass in systems programming trade-offs. The work-stealing scheduler gives you CPU efficiency. The I/O driver gives you scalability. The timer wheel gives you performance with thousands of concurrent timeouts. And the entire thing is built on Rust's ownership model, which means data races are caught at compile time rather than at 3 AM in production.
Understanding these internals won't change how you write basic async code. But it will change how you debug it, how you structure your task hierarchy, and how you avoid the performance cliffs that catch everyone eventually.