{"schemaVersion":1,"id":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation","title":"Cell-Based Architecture: Blast Radius Isolation, Shuffle Sharding, and How AWS and Slack Reduce Failure Domains","description":"A deep technical look at cell-based architecture, how to isolate failures to a fraction of users through independent cells, shuffle sharding, cell routing, and real implementation patterns drawn from AWS, Slack, and production systems.","language":"en","datePublished":"2025-12-06","dateModified":null,"contentFormat":"text/markdown","content":"Architecture\n\n# Cell-Based Architecture: Blast Radius Isolation, Shuffle Sharding, and How AWS and Slack Reduce Failure Domains\n\n2025-12-067 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/cell-based-architecture-blast-radius-isolation>)\n\n## The Core Idea\n\nCell-based architecture divides your system into independent, self-contained units called cells. Each cell serves a subset of tenants and contains all the infrastructure needed to operate independently, its own databases, caches, queues, and compute.\n\n```\n ┌─────────────┐\n │ Router │\n │ (Cell │\n │ Assignment)│\n └──────┬──────┘\n │\n ┌────────────┼────────────┐\n │ │ │\n ┌─────▼─────┐ ┌───▼─────┐ ┌───▼─────┐\n │ Cell A │ │ Cell B │ │ Cell C │\n │ ┌───────┐ │ │┌───────┐│ │┌───────┐│\n │ │API │ │ ││API ││ ││API ││\n │ │DB │ │ ││DB ││ ││DB ││\n │ │Cache │ │ ││Cache ││ ││Cache ││\n │ │Queue │ │ ││Queue ││ ││Queue ││\n │ └───────┘ │ │└───────┘│ │└───────┘│\n │ Tenants: │ │Tenants: │ │Tenants: │\n │ A,D,G,J...│ │B,E,H,K.│ │C,F,I,L.│\n └───────────┘ └─────────┘ └─────────┘\n```\n\nThe critical property: cells share nothing. If Cell A's database goes down, cells B and C continue operating. The blast radius of any failure is limited to the tenants assigned to the affected cell.\n\n## Cell Routing\n\nThe router is the single shared component. It maps tenants to cells and must be extremely reliable. Here's a reference implementation:\n\n```javascript\n// Cell router, runs at the edge (Cloudflare Worker)\nconst CELL_ENDPOINTS = {\n 0: 'https://cell-a.internal.example.com',\n 1: 'https://cell-b.internal.example.com',\n 2: 'https://cell-c.internal.example.com',\n 3: 'https://cell-d.internal.example.com',\n};\n\nconst NUM_CELLS = Object.keys(CELL_ENDPOINTS).length;\n\nfunction assignCell(tenantId) {\n // Deterministic hash-based assignment\n const hash = fnv1a(tenantId);\n return hash % NUM_CELLS;\n}\n\nexport default {\n async fetch(request) {\n const tenantId = extractTenantId(request);\n const cellIndex = assignCell(tenantId);\n const cellUrl = CELL_ENDPOINTS[cellIndex];\n\n // Forward to the assigned cell\n const cellRequest = new Request(\n `${cellUrl}${new URL(request.url).pathname}`,\n request,\n );\n\n try {\n const response = await fetch(cellRequest, { cf: { timeout: 10000 } });\n response.headers.set('X-Cell', `cell-${cellIndex}`);\n return response;\n } catch (error) {\n // Cell is down, return service unavailable for this tenant\n // With shuffle sharding, we'd try the secondary cell here\n return new Response('Service temporarily unavailable', { status: 503 });\n }\n },\n};\n```\n\nThe FNV-1a hash gives uniform distribution. Consistent hashing (like in Maglev) is an alternative, but the simpler modular approach is sufficient when the cell count rarely changes.\n\n## Shuffle Sharding: Reducing Blast Radius Further\n\nBasic cell assignment puts all eggs in one basket, if Cell A goes down, all of Cell A's tenants are down. Shuffle sharding improves this by assigning each tenant to multiple cells.\n\n```python\nimport hashlib\nimport random\n\ndef shuffle_shard(tenant_id, num_cells, shard_size=2):\n \"\"\"Assign a tenant to `shard_size` cells deterministically.\"\"\"\n # Use tenant_id as seed for reproducible randomness\n seed = int(hashlib.sha256(tenant_id.encode()).hexdigest(), 16)\n rng = random.Random(seed)\n\n # Pick `shard_size` unique cells\n cells = list(range(num_cells))\n rng.shuffle(cells)\n return cells[:shard_size]\n\n# Example: 8 cells, each tenant gets 2\ntenant_cells = shuffle_shard(\"tenant-123\", num_cells=8, shard_size=2)\n# Returns e.g. [3, 7], tenant-123's primary is cell 3, secondary is cell 7\n```\n\nThe math is where it gets interesting. With 8 cells and 2-cell shards:\n\n- Probability that two random tenants share a cell: ` 1 - C(6,2)/C(8,2) ` = ~46%\n- A single cell failure affects: ` 2/8 ` = 25% of tenants at most\n- But those tenants fail over to their secondary cell, so actual impact is only the failover latency\n- Two simultaneous cell failures affect: only tenants whose shard is exactly those two cells = ` 1/C(8,2) ` = ~3.6%\n\n[AWS](<https://aws.amazon.com/>) Route 53 uses this pattern. Each DNS zone is assigned to 4 of their 2,048 virtual shards. The probability of a \"noisy neighbor\" (a DDoS on one zone) affecting another specific zone is vanishingly small.\n\n## Cell Infrastructure with Terraform\n\nEach cell is a complete infrastructure stack. Terraform workspaces provide a clean way to manage them:\n\n```hcl\n# modules/cell/main.tf\nvariable \"cell_id\" {\n type = string\n}\n\nvariable \"cell_index\" {\n type = number\n}\n\nresource \"aws_vpc\" \"cell\" {\n cidr_block = \"10.${var.cell_index}.0.0/16\"\n tags = { Name = \"cell-${var.cell_id}\" }\n}\n\nresource \"aws_rds_cluster\" \"cell_db\" {\n cluster_identifier = \"cell-${var.cell_id}-db\"\n engine = \"aurora-postgresql\"\n engine_version = \"16.1\"\n master_username = var.db_username\n master_password = var.db_password\n\n vpc_security_group_ids = [aws_security_group.db.id]\n db_subnet_group_name = aws_db_subnet_group.cell.name\n\n # Each cell gets its own database, no cross-cell sharing\n tags = { Cell = var.cell_id }\n}\n\nresource \"aws_elasticache_replication_group\" \"cell_cache\" {\n replication_group_id = \"cell-${var.cell_id}-redis\"\n description = \"Redis cluster for cell ${var.cell_id}\"\n node_type = \"cache.r7g.large\"\n num_cache_clusters = 2\n\n tags = { Cell = var.cell_id }\n}\n\nresource \"aws_sqs_queue\" \"cell_queue\" {\n name = \"cell-${var.cell_id}-events\"\n tags = { Cell = var.cell_id }\n}\n```\n\n```bash\n# Create cell-a\nterraform workspace select cell-a\nterraform apply -var='cell_id=a' -var='cell_index=0'\n\n# Create cell-b\nterraform workspace select cell-b\nterraform apply -var='cell_id=b' -var='cell_index=1'\n```\n\nThe VPCs are isolated, cells can't accidentally communicate with each other. This is deliberate. Cross-cell communication defeats the purpose of cell isolation.\n\n## Deployment Strategy: Canary Cells\n\nThe deployment order matters as much as the architecture:\n\n```yaml\n# deployment-pipeline.yml\nstages:\n - name: canary-cell\n cells: [\"cell-a\"]\n validation:\n - health_check: 5m\n - error_rate: < 0.1%\n - latency_p99: < 200ms\n rollback_on_failure: true\n\n - name: second-wave\n cells: [\"cell-b\"]\n depends_on: canary-cell\n delay: 15m\n validation:\n - health_check: 10m\n - error_rate: < 0.1%\n\n - name: remaining\n cells: [\"cell-c\", \"cell-d\"]\n depends_on: second-wave\n delay: 30m\n```\n\nCell A is always the canary. It gets deployments first, and the system watches it for 15 minutes before proceeding. If the deployment introduces a bug, only ~25% of tenants are affected, and the rollback can proceed without touching the other cells.\n\nThis is exactly how a database migration incident plays out with cells: the migration runs on Cell A, it breaks, Cell A gets rolled back, and cells B/C/D never see the bad migration. Instead of 45 minutes of total downtime, it's 5 minutes of partial outage for a quarter of users.\n\n## How Slack Does It\n\nSlack's cell architecture (which they've talked about publicly) is one of the clearest implementations. Each \"cell\" handles a subset of Slack workspaces:\n\n- Cell assignment is by workspace ID\n- Each cell has its own MySQL cluster, cache layer, and job queue\n- The \"admin plane\" (workspace creation, billing) is separate from the \"data plane\" (messages, channels)\n- Deployments go cell-by-cell with automated canary analysis\n\nTheir key insight: the admin plane can be a shared service because it's low-traffic and non-critical-path. The data plane, where latency and availability matter, is fully cellularized.\n\n## Monitoring Per Cell\n\nYou need per-cell dashboards. Aggregate metrics hide cell-specific issues:\n\n```yaml\n# Prometheus recording rules per cell\ngroups:\n - name: cell_metrics\n rules:\n - record: cell:request_rate:5m\n expr: sum(rate(http_requests_total{job=\"api\"}[5m])) by (cell)\n\n - record: cell:error_rate:5m\n expr: |\n sum(rate(http_requests_total{status=~\"5..\"}[5m])) by (cell)\n /\n sum(rate(http_requests_total[5m])) by (cell)\n\n - record: cell:latency_p99:5m\n expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, cell))\n\n - name: cell_alerts\n rules:\n - alert: CellHighErrorRate\n expr: cell:error_rate:5m > 0.01\n for: 5m\n labels:\n severity: critical\n annotations:\n summary: \"Cell {{ $labels.cell }} error rate > 1%\"\n\n - alert: CellCapacityWarning\n expr: cell:request_rate:5m / cell:capacity_limit > 0.8\n for: 10m\n labels:\n severity: warning\n```\n\nThe ` cell ` label on every metric is non-negotiable. Without it, you can't tell if an error rate spike is a global issue or a single-cell issue, and the response is completely different.\n\n## Trade-offs and Operational Costs\n\n**Cost**: Running N copies of everything means 4 cells roughly equals 4x the database cost, 4x the cache cost, 4x the monitoring. In practice, cells add about 40% to infrastructure costs (not 4x, because compute scales sub-linearly with fewer tenants per cell).\n\n**Operational complexity**: Every operational procedure needs a \"which cell?\" step. Database migrations, schema changes, config updates, all need to be cell-aware. Expect to spend significant time building tooling to make this manageable.\n\n**Data migration**: Moving a tenant between cells is the hardest operation. It requires a coordinated write-pause, data copy, and routing update. A dedicated migration tool is essential, and even then the operation requires careful coordination.\n\n**Global features**: Some things inherently span cells, global search, cross-tenant analytics, admin dashboards. These need a separate aggregation layer that reads from all cells. A CDC (Change Data Capture) pipeline that replicates a subset of data from each cell's database into a shared analytics store is a common solution.\n\n## Wrapping Up\n\nCell-based architecture is the nuclear option for blast radius isolation. It works, systems go from \"one bad deployment takes down everything\" to \"one bad deployment affects 25% of users for 5 minutes\", but the operational cost is real.\n\nStart simple: circuit breakers, bulkheads, multi-AZ deployments. Graduate to cells when a single failure domain affecting all users becomes an existential risk for the business. And when you do build cells, start with 4, keep them truly independent, and invest heavily in per-cell observability.\n\nThe architecture is the easy part. Living with it is the hard part.\n\n## Frequently asked questions\n\n### How does cell-based architecture differ from microservices?\n\nMicroservices split by function, one service for auth, another for payments. Cell-based architecture splits by tenant. Each cell runs ALL services for a SUBSET of tenants. The key difference is the failure boundary: a microservice failure affects all users of that function, while a cell failure affects only users assigned to that cell.\n\n### What is shuffle sharding and why does it matter?\n\nShuffle sharding assigns each tenant to a random subset of cells (typically 2 of N). If cell A fails, only tenants whose shard set includes cell A are affected, and those tenants failover to their second cell. With 8 cells and 2-cell shards, any single cell failure affects only 25% of tenants, and most of those recover automatically.\n\n### When is cell-based architecture overkill?\n\nIf your system has fewer than 1000 tenants, serves under 10K requests/second, or has a team smaller than 5 engineers. The operational overhead of maintaining multiple independent infrastructure stacks is significant. Consider simpler fault isolation first, multi-region active-active, or just thorough circuit breaking.\n\n## Tips\n\n1. **Tip:** Start with 4 cells, not 40. Each cell is a full copy of your infrastructure, networking, databases, caches, queues. More cells means better isolation but exponentially more operational overhead. Starting with 8 and scaling back to 4 is common once the monitoring burden becomes unsustainable for small teams.\n2. **Tip:** Cell assignment must be deterministic and consistent, hash the tenant ID, not the request. If the same user hits different cells on different requests, you lose data locality and cache efficiency. Use ` cell = tenantId % numCells ` or consistent hashing.\n3. **Tip:** Always keep one cell as a 'canary cell' that receives deployments first. If the deployment breaks the canary, you've contained the blast radius to ~25% of users (with 4 cells) instead of 100%. This approach can prevent a bad migration from corrupting payment records across the entire system.\n4. **Tip:** Don't share databases between cells. Shared databases are shared failure domains. If you absolutely must share state (like a global user directory), make it read-only from the cells and write-only from a separate control plane.\n\n- cell-based-architecture\n- blast-radius\n- shuffle-sharding\n- fault-isolation\n- distributed-systems\n- aws\n- slack\n- resilience\n- high-availability\n- architecture\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"Cell-Based Architecture: Blast Radius Isolation, Shuffle Sharding, and How AWS and Slack Reduce Failure Domains","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#title"},{"level":2,"text":"The Core Idea","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#the-core-idea"},{"level":2,"text":"Cell Routing","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#cell-routing"},{"level":2,"text":"Shuffle Sharding: Reducing Blast Radius Further","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#shuffle-sharding-reducing-blast-radius-further"},{"level":2,"text":"Cell Infrastructure with Terraform","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#cell-infrastructure-with-terraform"},{"level":2,"text":"Deployment Strategy: Canary Cells","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#deployment-strategy-canary-cells"},{"level":2,"text":"How Slack Does It","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#how-slack-does-it"},{"level":2,"text":"Monitoring Per Cell","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#monitoring-per-cell"},{"level":2,"text":"Trade-offs and Operational Costs","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#trade-offs-and-operational-costs"},{"level":2,"text":"Wrapping Up","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#wrapping-up"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#questions"},{"level":3,"text":"How does cell-based architecture differ from microservices?","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#question-1"},{"level":3,"text":"What is shuffle sharding and why does it matter?","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#question-2"},{"level":3,"text":"When is cell-based architecture overkill?","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#article-tips"}],"sections":[{"heading":"Cell-Based Architecture: Blast Radius Isolation, Shuffle Sharding, and How AWS and Slack Reduce Failure Domains","level":1,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation","content":"Architecture"},{"heading":"Cell-Based Architecture: Blast Radius Isolation, Shuffle Sharding, and How AWS and Slack Reduce Failure Domains","level":1,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#title","content":"2025-12-067 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/cell-based-architecture-blast-radius-isolation>)"},{"heading":"The Core Idea","level":2,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#the-core-idea","content":"Cell-based architecture divides your system into independent, self-contained units called cells. Each cell serves a subset of tenants and contains all the infrastructure needed to operate independently, its own databases, caches, queues, and compute.\n\n```\n ┌─────────────┐\n │ Router │\n │ (Cell │\n │ Assignment)│\n └──────┬──────┘\n │\n ┌────────────┼────────────┐\n │ │ │\n ┌─────▼─────┐ ┌───▼─────┐ ┌───▼─────┐\n │ Cell A │ │ Cell B │ │ Cell C │\n │ ┌───────┐ │ │┌───────┐│ │┌───────┐│\n │ │API │ │ ││API ││ ││API ││\n │ │DB │ │ ││DB ││ ││DB ││\n │ │Cache │ │ ││Cache ││ ││Cache ││\n │ │Queue │ │ ││Queue ││ ││Queue ││\n │ └───────┘ │ │└───────┘│ │└───────┘│\n │ Tenants: │ │Tenants: │ │Tenants: │\n │ A,D,G,J...│ │B,E,H,K.│ │C,F,I,L.│\n └───────────┘ └─────────┘ └─────────┘\n```\n\nThe critical property: cells share nothing. If Cell A's database goes down, cells B and C continue operating. The blast radius of any failure is limited to the tenants assigned to the affected cell."},{"heading":"Cell Routing","level":2,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#cell-routing","content":"The router is the single shared component. It maps tenants to cells and must be extremely reliable. Here's a reference implementation:\n\n```javascript\n// Cell router, runs at the edge (Cloudflare Worker)\nconst CELL_ENDPOINTS = {\n 0: 'https://cell-a.internal.example.com',\n 1: 'https://cell-b.internal.example.com',\n 2: 'https://cell-c.internal.example.com',\n 3: 'https://cell-d.internal.example.com',\n};\n\nconst NUM_CELLS = Object.keys(CELL_ENDPOINTS).length;\n\nfunction assignCell(tenantId) {\n // Deterministic hash-based assignment\n const hash = fnv1a(tenantId);\n return hash % NUM_CELLS;\n}\n\nexport default {\n async fetch(request) {\n const tenantId = extractTenantId(request);\n const cellIndex = assignCell(tenantId);\n const cellUrl = CELL_ENDPOINTS[cellIndex];\n\n // Forward to the assigned cell\n const cellRequest = new Request(\n `${cellUrl}${new URL(request.url).pathname}`,\n request,\n );\n\n try {\n const response = await fetch(cellRequest, { cf: { timeout: 10000 } });\n response.headers.set('X-Cell', `cell-${cellIndex}`);\n return response;\n } catch (error) {\n // Cell is down, return service unavailable for this tenant\n // With shuffle sharding, we'd try the secondary cell here\n return new Response('Service temporarily unavailable', { status: 503 });\n }\n },\n};\n```\n\nThe FNV-1a hash gives uniform distribution. Consistent hashing (like in Maglev) is an alternative, but the simpler modular approach is sufficient when the cell count rarely changes."},{"heading":"Shuffle Sharding: Reducing Blast Radius Further","level":2,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#shuffle-sharding-reducing-blast-radius-further","content":"Basic cell assignment puts all eggs in one basket, if Cell A goes down, all of Cell A's tenants are down. Shuffle sharding improves this by assigning each tenant to multiple cells.\n\n```python\nimport hashlib\nimport random\n\ndef shuffle_shard(tenant_id, num_cells, shard_size=2):\n \"\"\"Assign a tenant to `shard_size` cells deterministically.\"\"\"\n # Use tenant_id as seed for reproducible randomness\n seed = int(hashlib.sha256(tenant_id.encode()).hexdigest(), 16)\n rng = random.Random(seed)\n\n # Pick `shard_size` unique cells\n cells = list(range(num_cells))\n rng.shuffle(cells)\n return cells[:shard_size]\n\n# Example: 8 cells, each tenant gets 2\ntenant_cells = shuffle_shard(\"tenant-123\", num_cells=8, shard_size=2)\n# Returns e.g. [3, 7], tenant-123's primary is cell 3, secondary is cell 7\n```\n\nThe math is where it gets interesting. With 8 cells and 2-cell shards:\n\n- Probability that two random tenants share a cell: ` 1 - C(6,2)/C(8,2) ` = ~46%\n- A single cell failure affects: ` 2/8 ` = 25% of tenants at most\n- But those tenants fail over to their secondary cell, so actual impact is only the failover latency\n- Two simultaneous cell failures affect: only tenants whose shard is exactly those two cells = ` 1/C(8,2) ` = ~3.6%\n\n[AWS](<https://aws.amazon.com/>) Route 53 uses this pattern. Each DNS zone is assigned to 4 of their 2,048 virtual shards. The probability of a \"noisy neighbor\" (a DDoS on one zone) affecting another specific zone is vanishingly small."},{"heading":"Cell Infrastructure with Terraform","level":2,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#cell-infrastructure-with-terraform","content":"Each cell is a complete infrastructure stack. Terraform workspaces provide a clean way to manage them:\n\n```hcl\n# modules/cell/main.tf\nvariable \"cell_id\" {\n type = string\n}\n\nvariable \"cell_index\" {\n type = number\n}\n\nresource \"aws_vpc\" \"cell\" {\n cidr_block = \"10.${var.cell_index}.0.0/16\"\n tags = { Name = \"cell-${var.cell_id}\" }\n}\n\nresource \"aws_rds_cluster\" \"cell_db\" {\n cluster_identifier = \"cell-${var.cell_id}-db\"\n engine = \"aurora-postgresql\"\n engine_version = \"16.1\"\n master_username = var.db_username\n master_password = var.db_password\n\n vpc_security_group_ids = [aws_security_group.db.id]\n db_subnet_group_name = aws_db_subnet_group.cell.name\n\n # Each cell gets its own database, no cross-cell sharing\n tags = { Cell = var.cell_id }\n}\n\nresource \"aws_elasticache_replication_group\" \"cell_cache\" {\n replication_group_id = \"cell-${var.cell_id}-redis\"\n description = \"Redis cluster for cell ${var.cell_id}\"\n node_type = \"cache.r7g.large\"\n num_cache_clusters = 2\n\n tags = { Cell = var.cell_id }\n}\n\nresource \"aws_sqs_queue\" \"cell_queue\" {\n name = \"cell-${var.cell_id}-events\"\n tags = { Cell = var.cell_id }\n}\n```\n\n```bash\n# Create cell-a\nterraform workspace select cell-a\nterraform apply -var='cell_id=a' -var='cell_index=0'\n\n# Create cell-b\nterraform workspace select cell-b\nterraform apply -var='cell_id=b' -var='cell_index=1'\n```\n\nThe VPCs are isolated, cells can't accidentally communicate with each other. This is deliberate. Cross-cell communication defeats the purpose of cell isolation."},{"heading":"Deployment Strategy: Canary Cells","level":2,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#deployment-strategy-canary-cells","content":"The deployment order matters as much as the architecture:\n\n```yaml\n# deployment-pipeline.yml\nstages:\n - name: canary-cell\n cells: [\"cell-a\"]\n validation:\n - health_check: 5m\n - error_rate: < 0.1%\n - latency_p99: < 200ms\n rollback_on_failure: true\n\n - name: second-wave\n cells: [\"cell-b\"]\n depends_on: canary-cell\n delay: 15m\n validation:\n - health_check: 10m\n - error_rate: < 0.1%\n\n - name: remaining\n cells: [\"cell-c\", \"cell-d\"]\n depends_on: second-wave\n delay: 30m\n```\n\nCell A is always the canary. It gets deployments first, and the system watches it for 15 minutes before proceeding. If the deployment introduces a bug, only ~25% of tenants are affected, and the rollback can proceed without touching the other cells.\n\nThis is exactly how a database migration incident plays out with cells: the migration runs on Cell A, it breaks, Cell A gets rolled back, and cells B/C/D never see the bad migration. Instead of 45 minutes of total downtime, it's 5 minutes of partial outage for a quarter of users."},{"heading":"How Slack Does It","level":2,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#how-slack-does-it","content":"Slack's cell architecture (which they've talked about publicly) is one of the clearest implementations. Each \"cell\" handles a subset of Slack workspaces:\n\n- Cell assignment is by workspace ID\n- Each cell has its own MySQL cluster, cache layer, and job queue\n- The \"admin plane\" (workspace creation, billing) is separate from the \"data plane\" (messages, channels)\n- Deployments go cell-by-cell with automated canary analysis\n\nTheir key insight: the admin plane can be a shared service because it's low-traffic and non-critical-path. The data plane, where latency and availability matter, is fully cellularized."},{"heading":"Monitoring Per Cell","level":2,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#monitoring-per-cell","content":"You need per-cell dashboards. Aggregate metrics hide cell-specific issues:\n\n```yaml\n# Prometheus recording rules per cell\ngroups:\n - name: cell_metrics\n rules:\n - record: cell:request_rate:5m\n expr: sum(rate(http_requests_total{job=\"api\"}[5m])) by (cell)\n\n - record: cell:error_rate:5m\n expr: |\n sum(rate(http_requests_total{status=~\"5..\"}[5m])) by (cell)\n /\n sum(rate(http_requests_total[5m])) by (cell)\n\n - record: cell:latency_p99:5m\n expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, cell))\n\n - name: cell_alerts\n rules:\n - alert: CellHighErrorRate\n expr: cell:error_rate:5m > 0.01\n for: 5m\n labels:\n severity: critical\n annotations:\n summary: \"Cell {{ $labels.cell }} error rate > 1%\"\n\n - alert: CellCapacityWarning\n expr: cell:request_rate:5m / cell:capacity_limit > 0.8\n for: 10m\n labels:\n severity: warning\n```\n\nThe ` cell ` label on every metric is non-negotiable. Without it, you can't tell if an error rate spike is a global issue or a single-cell issue, and the response is completely different."},{"heading":"Trade-offs and Operational Costs","level":2,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#trade-offs-and-operational-costs","content":"**Cost**: Running N copies of everything means 4 cells roughly equals 4x the database cost, 4x the cache cost, 4x the monitoring. In practice, cells add about 40% to infrastructure costs (not 4x, because compute scales sub-linearly with fewer tenants per cell).\n\n**Operational complexity**: Every operational procedure needs a \"which cell?\" step. Database migrations, schema changes, config updates, all need to be cell-aware. Expect to spend significant time building tooling to make this manageable.\n\n**Data migration**: Moving a tenant between cells is the hardest operation. It requires a coordinated write-pause, data copy, and routing update. A dedicated migration tool is essential, and even then the operation requires careful coordination.\n\n**Global features**: Some things inherently span cells, global search, cross-tenant analytics, admin dashboards. These need a separate aggregation layer that reads from all cells. A CDC (Change Data Capture) pipeline that replicates a subset of data from each cell's database into a shared analytics store is a common solution."},{"heading":"Wrapping Up","level":2,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#wrapping-up","content":"Cell-based architecture is the nuclear option for blast radius isolation. It works, systems go from \"one bad deployment takes down everything\" to \"one bad deployment affects 25% of users for 5 minutes\", but the operational cost is real.\n\nStart simple: circuit breakers, bulkheads, multi-AZ deployments. Graduate to cells when a single failure domain affecting all users becomes an existential risk for the business. And when you do build cells, start with 4, keep them truly independent, and invest heavily in per-cell observability.\n\nThe architecture is the easy part. Living with it is the hard part."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#questions","content":""},{"heading":"How does cell-based architecture differ from microservices?","level":3,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#question-1","content":"Microservices split by function, one service for auth, another for payments. Cell-based architecture splits by tenant. Each cell runs ALL services for a SUBSET of tenants. The key difference is the failure boundary: a microservice failure affects all users of that function, while a cell failure affects only users assigned to that cell."},{"heading":"What is shuffle sharding and why does it matter?","level":3,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#question-2","content":"Shuffle sharding assigns each tenant to a random subset of cells (typically 2 of N). If cell A fails, only tenants whose shard set includes cell A are affected, and those tenants failover to their second cell. With 8 cells and 2-cell shards, any single cell failure affects only 25% of tenants, and most of those recover automatically."},{"heading":"When is cell-based architecture overkill?","level":3,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#question-3","content":"If your system has fewer than 1000 tenants, serves under 10K requests/second, or has a team smaller than 5 engineers. The operational overhead of maintaining multiple independent infrastructure stacks is significant. Consider simpler fault isolation first, multi-region active-active, or just thorough circuit breaking."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#article-tips","content":"1. **Tip:** Start with 4 cells, not 40. Each cell is a full copy of your infrastructure, networking, databases, caches, queues. More cells means better isolation but exponentially more operational overhead. Starting with 8 and scaling back to 4 is common once the monitoring burden becomes unsustainable for small teams.\n2. **Tip:** Cell assignment must be deterministic and consistent, hash the tenant ID, not the request. If the same user hits different cells on different requests, you lose data locality and cache efficiency. Use ` cell = tenantId % numCells ` or consistent hashing.\n3. **Tip:** Always keep one cell as a 'canary cell' that receives deployments first. If the deployment breaks the canary, you've contained the blast radius to ~25% of users (with 4 cells) instead of 100%. This approach can prevent a bad migration from corrupting payment records across the entire system.\n4. **Tip:** Don't share databases between cells. Shared databases are shared failure domains. If you absolutely must share state (like a global user directory), make it read-only from the cells and write-only from a separate control plane.\n\n- cell-based-architecture\n- blast-radius\n- shuffle-sharding\n- fault-isolation\n- distributed-systems\n- aws\n- slack\n- resilience\n- high-availability\n- architecture\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/cell-based-architecture-blast-radius-isolation"},{"text":"AWS","url":"https://aws.amazon.com/"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation"},"headline":"Cell-Based Architecture: Blast Radius Isolation, Shuffle Sharding, and How AWS and Slack Reduce Failure Domains","description":"A deep technical look at cell-based architecture, how to isolate failures to a fraction of users through independent cells, shuffle sharding, cell routing, and real implementation patterns drawn from AWS, Slack, and production systems.","datePublished":"2025-12-06","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://lucioduran.com/favicons/og-default.png","url":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Architecture","keywords":"cell-based-architecture, blast-radius, shuffle-sharding, fault-isolation, distributed-systems, aws, slack, resilience, high-availability, architecture","wordCount":1362,"timeRequired":"PT7M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/cell-based-architecture-blast-radius-isolation"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"Cell-Based Architecture: Blast Radius Isolation, Shuffle Sharding, and How AWS and Slack Reduce Failure Domains","item":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"How does cell-based architecture differ from microservices?","acceptedAnswer":{"@type":"Answer","text":"Microservices split by function, one service for auth, another for payments. Cell-based architecture splits by tenant. Each cell runs ALL services for a SUBSET of tenants. The key difference is the failure boundary: a microservice failure affects all users of that function, while a cell failure affects only users assigned to that cell."}},{"@type":"Question","name":"What is shuffle sharding and why does it matter?","acceptedAnswer":{"@type":"Answer","text":"Shuffle sharding assigns each tenant to a random subset of cells (typically 2 of N). If cell A fails, only tenants whose shard set includes cell A are affected, and those tenants failover to their second cell. With 8 cells and 2-cell shards, any single cell failure affects only 25% of tenants, and most of those recover automatically."}},{"@type":"Question","name":"When is cell-based architecture overkill?","acceptedAnswer":{"@type":"Answer","text":"If your system has fewer than 1000 tenants, serves under 10K requests/second, or has a team smaller than 5 engineers. The operational overhead of maintaining multiple independent infrastructure stacks is significant. Consider simpler fault isolation first, multi-region active-active, or just thorough circuit breaking."}}]}]}],"contentHash":"e87b9774c28d341104440675c20fc1987b766a18893a29d8c02a7a054c6c5300","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/cell-based-architecture-blast-radius-isolation","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":63895,"contentBytes":13398}}