Architecture # Cell-Based Architecture: Blast Radius Isolation, Shuffle Sharding, and How AWS and Slack Reduce Failure Domains 2025-12-067 min read Lucio Durán Engineering Manager & AI Solutions Architect Also available in: [Español]() ## The Core Idea Cell-based architecture divides your system into independent, self-contained units called cells. Each cell serves a subset of tenants and contains all the infrastructure needed to operate independently, its own databases, caches, queues, and compute. ``` ┌─────────────┐ │ Router │ │ (Cell │ │ Assignment)│ └──────┬──────┘ │ ┌────────────┼────────────┐ │ │ │ ┌─────▼─────┐ ┌───▼─────┐ ┌───▼─────┐ │ Cell A │ │ Cell B │ │ Cell C │ │ ┌───────┐ │ │┌───────┐│ │┌───────┐│ │ │API │ │ ││API ││ ││API ││ │ │DB │ │ ││DB ││ ││DB ││ │ │Cache │ │ ││Cache ││ ││Cache ││ │ │Queue │ │ ││Queue ││ ││Queue ││ │ └───────┘ │ │└───────┘│ │└───────┘│ │ Tenants: │ │Tenants: │ │Tenants: │ │ A,D,G,J...│ │B,E,H,K.│ │C,F,I,L.│ └───────────┘ └─────────┘ └─────────┘ ``` The critical property: cells share nothing. If Cell A's database goes down, cells B and C continue operating. The blast radius of any failure is limited to the tenants assigned to the affected cell. ## Cell Routing The router is the single shared component. It maps tenants to cells and must be extremely reliable. Here's a reference implementation: ```javascript // Cell router, runs at the edge (Cloudflare Worker) const CELL_ENDPOINTS = { 0: 'https://cell-a.internal.example.com', 1: 'https://cell-b.internal.example.com', 2: 'https://cell-c.internal.example.com', 3: 'https://cell-d.internal.example.com', }; const NUM_CELLS = Object.keys(CELL_ENDPOINTS).length; function assignCell(tenantId) { // Deterministic hash-based assignment const hash = fnv1a(tenantId); return hash % NUM_CELLS; } export default { async fetch(request) { const tenantId = extractTenantId(request); const cellIndex = assignCell(tenantId); const cellUrl = CELL_ENDPOINTS[cellIndex]; // Forward to the assigned cell const cellRequest = new Request( `${cellUrl}${new URL(request.url).pathname}`, request, ); try { const response = await fetch(cellRequest, { cf: { timeout: 10000 } }); response.headers.set('X-Cell', `cell-${cellIndex}`); return response; } catch (error) { // Cell is down, return service unavailable for this tenant // With shuffle sharding, we'd try the secondary cell here return new Response('Service temporarily unavailable', { status: 503 }); } }, }; ``` The FNV-1a hash gives uniform distribution. Consistent hashing (like in Maglev) is an alternative, but the simpler modular approach is sufficient when the cell count rarely changes. ## Shuffle Sharding: Reducing Blast Radius Further Basic cell assignment puts all eggs in one basket, if Cell A goes down, all of Cell A's tenants are down. Shuffle sharding improves this by assigning each tenant to multiple cells. ```python import hashlib import random def shuffle_shard(tenant_id, num_cells, shard_size=2): """Assign a tenant to `shard_size` cells deterministically.""" # Use tenant_id as seed for reproducible randomness seed = int(hashlib.sha256(tenant_id.encode()).hexdigest(), 16) rng = random.Random(seed) # Pick `shard_size` unique cells cells = list(range(num_cells)) rng.shuffle(cells) return cells[:shard_size] # Example: 8 cells, each tenant gets 2 tenant_cells = shuffle_shard("tenant-123", num_cells=8, shard_size=2) # Returns e.g. [3, 7], tenant-123's primary is cell 3, secondary is cell 7 ``` The math is where it gets interesting. With 8 cells and 2-cell shards: - Probability that two random tenants share a cell: ` 1 - C(6,2)/C(8,2) ` = ~46% - A single cell failure affects: ` 2/8 ` = 25% of tenants at most - But those tenants fail over to their secondary cell, so actual impact is only the failover latency - Two simultaneous cell failures affect: only tenants whose shard is exactly those two cells = ` 1/C(8,2) ` = ~3.6% [AWS]() Route 53 uses this pattern. Each DNS zone is assigned to 4 of their 2,048 virtual shards. The probability of a "noisy neighbor" (a DDoS on one zone) affecting another specific zone is vanishingly small. ## Cell Infrastructure with Terraform Each cell is a complete infrastructure stack. Terraform workspaces provide a clean way to manage them: ```hcl # modules/cell/main.tf variable "cell_id" { type = string } variable "cell_index" { type = number } resource "aws_vpc" "cell" { cidr_block = "10.${var.cell_index}.0.0/16" tags = { Name = "cell-${var.cell_id}" } } resource "aws_rds_cluster" "cell_db" { cluster_identifier = "cell-${var.cell_id}-db" engine = "aurora-postgresql" engine_version = "16.1" master_username = var.db_username master_password = var.db_password vpc_security_group_ids = [aws_security_group.db.id] db_subnet_group_name = aws_db_subnet_group.cell.name # Each cell gets its own database, no cross-cell sharing tags = { Cell = var.cell_id } } resource "aws_elasticache_replication_group" "cell_cache" { replication_group_id = "cell-${var.cell_id}-redis" description = "Redis cluster for cell ${var.cell_id}" node_type = "cache.r7g.large" num_cache_clusters = 2 tags = { Cell = var.cell_id } } resource "aws_sqs_queue" "cell_queue" { name = "cell-${var.cell_id}-events" tags = { Cell = var.cell_id } } ``` ```bash # Create cell-a terraform workspace select cell-a terraform apply -var='cell_id=a' -var='cell_index=0' # Create cell-b terraform workspace select cell-b terraform apply -var='cell_id=b' -var='cell_index=1' ``` The VPCs are isolated, cells can't accidentally communicate with each other. This is deliberate. Cross-cell communication defeats the purpose of cell isolation. ## Deployment Strategy: Canary Cells The deployment order matters as much as the architecture: ```yaml # deployment-pipeline.yml stages: - name: canary-cell cells: ["cell-a"] validation: - health_check: 5m - error_rate: < 0.1% - latency_p99: < 200ms rollback_on_failure: true - name: second-wave cells: ["cell-b"] depends_on: canary-cell delay: 15m validation: - health_check: 10m - error_rate: < 0.1% - name: remaining cells: ["cell-c", "cell-d"] depends_on: second-wave delay: 30m ``` Cell A is always the canary. It gets deployments first, and the system watches it for 15 minutes before proceeding. If the deployment introduces a bug, only ~25% of tenants are affected, and the rollback can proceed without touching the other cells. This is exactly how a database migration incident plays out with cells: the migration runs on Cell A, it breaks, Cell A gets rolled back, and cells B/C/D never see the bad migration. Instead of 45 minutes of total downtime, it's 5 minutes of partial outage for a quarter of users. ## How Slack Does It Slack's cell architecture (which they've talked about publicly) is one of the clearest implementations. Each "cell" handles a subset of Slack workspaces: - Cell assignment is by workspace ID - Each cell has its own MySQL cluster, cache layer, and job queue - The "admin plane" (workspace creation, billing) is separate from the "data plane" (messages, channels) - Deployments go cell-by-cell with automated canary analysis Their key insight: the admin plane can be a shared service because it's low-traffic and non-critical-path. The data plane, where latency and availability matter, is fully cellularized. ## Monitoring Per Cell You need per-cell dashboards. Aggregate metrics hide cell-specific issues: ```yaml # Prometheus recording rules per cell groups: - name: cell_metrics rules: - record: cell:request_rate:5m expr: sum(rate(http_requests_total{job="api"}[5m])) by (cell) - record: cell:error_rate:5m expr: | sum(rate(http_requests_total{status=~"5.."}[5m])) by (cell) / sum(rate(http_requests_total[5m])) by (cell) - record: cell:latency_p99:5m expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, cell)) - name: cell_alerts rules: - alert: CellHighErrorRate expr: cell:error_rate:5m > 0.01 for: 5m labels: severity: critical annotations: summary: "Cell {{ $labels.cell }} error rate > 1%" - alert: CellCapacityWarning expr: cell:request_rate:5m / cell:capacity_limit > 0.8 for: 10m labels: severity: warning ``` The ` cell ` label on every metric is non-negotiable. Without it, you can't tell if an error rate spike is a global issue or a single-cell issue, and the response is completely different. ## Trade-offs and Operational Costs **Cost**: Running N copies of everything means 4 cells roughly equals 4x the database cost, 4x the cache cost, 4x the monitoring. In practice, cells add about 40% to infrastructure costs (not 4x, because compute scales sub-linearly with fewer tenants per cell). **Operational complexity**: Every operational procedure needs a "which cell?" step. Database migrations, schema changes, config updates, all need to be cell-aware. Expect to spend significant time building tooling to make this manageable. **Data migration**: Moving a tenant between cells is the hardest operation. It requires a coordinated write-pause, data copy, and routing update. A dedicated migration tool is essential, and even then the operation requires careful coordination. **Global features**: Some things inherently span cells, global search, cross-tenant analytics, admin dashboards. These need a separate aggregation layer that reads from all cells. A CDC (Change Data Capture) pipeline that replicates a subset of data from each cell's database into a shared analytics store is a common solution. ## Wrapping Up Cell-based architecture is the nuclear option for blast radius isolation. It works, systems go from "one bad deployment takes down everything" to "one bad deployment affects 25% of users for 5 minutes", but the operational cost is real. Start simple: circuit breakers, bulkheads, multi-AZ deployments. Graduate to cells when a single failure domain affecting all users becomes an existential risk for the business. And when you do build cells, start with 4, keep them truly independent, and invest heavily in per-cell observability. The architecture is the easy part. Living with it is the hard part. ## Frequently asked questions ### How does cell-based architecture differ from microservices? Microservices split by function, one service for auth, another for payments. Cell-based architecture splits by tenant. Each cell runs ALL services for a SUBSET of tenants. The key difference is the failure boundary: a microservice failure affects all users of that function, while a cell failure affects only users assigned to that cell. ### What is shuffle sharding and why does it matter? Shuffle sharding assigns each tenant to a random subset of cells (typically 2 of N). If cell A fails, only tenants whose shard set includes cell A are affected, and those tenants failover to their second cell. With 8 cells and 2-cell shards, any single cell failure affects only 25% of tenants, and most of those recover automatically. ### When is cell-based architecture overkill? If your system has fewer than 1000 tenants, serves under 10K requests/second, or has a team smaller than 5 engineers. The operational overhead of maintaining multiple independent infrastructure stacks is significant. Consider simpler fault isolation first, multi-region active-active, or just thorough circuit breaking. ## Tips 1. **Tip:** Start with 4 cells, not 40. Each cell is a full copy of your infrastructure, networking, databases, caches, queues. More cells means better isolation but exponentially more operational overhead. Starting with 8 and scaling back to 4 is common once the monitoring burden becomes unsustainable for small teams. 2. **Tip:** Cell assignment must be deterministic and consistent, hash the tenant ID, not the request. If the same user hits different cells on different requests, you lose data locality and cache efficiency. Use ` cell = tenantId % numCells ` or consistent hashing. 3. **Tip:** Always keep one cell as a 'canary cell' that receives deployments first. If the deployment breaks the canary, you've contained the blast radius to ~25% of users (with 4 cells) instead of 100%. This approach can prevent a bad migration from corrupting payment records across the entire system. 4. **Tip:** Don't share databases between cells. Shared databases are shared failure domains. If you absolutely must share state (like a global user directory), make it read-only from the cells and write-only from a separate control plane. - cell-based-architecture - blast-radius - shuffle-sharding - fault-isolation - distributed-systems - aws - slack - resilience - high-availability - architecture [Back to Blog]()