ScyllaDB: Cassandra-Compatible Low-Latency Database Architecture and Migration

The Problem with Cassandra's Architecture

Consider what happens inside a Cassandra node. You've got a JVM process managing tens of gigabytes of heap, running on a machine with 16-32 cores. Every read and write goes through shared data structures protected by locks. The JVM's garbage collector periodically decides to pause everything, and "everything" means all 32 cores stop doing useful work simultaneously.

Here is a typical Cassandra GC log entry that illustrates the problem:

GC pause (G1 Evacuation Pause) 14.2GB->9.8GB 247.483ms

247 milliseconds. That's not a tail latency outlier, that's a quarter second where your database simply doesn't respond. And this was on a tuned cluster with G1GC, 8GB new gen, and carefully sized regions.

The fundamental issue is that Cassandra's threading model was designed in the mid-2000s. It uses a thread-per-core approach with shared mutable state, and then layers a garbage-collected runtime on top. Every architectural choice compounds:

Thread pool executor (WRITE stage) → Shared memtable → Lock contention
Thread pool executor (READ stage) → Shared row cache → Lock contention
JVM GC → Stop-the-world → All threads pause → Latency spike

Seastar: The Shard-Per-Core Framework

ScyllaDB is built on the Seastar framework, which was originally developed for high-performance networking (it powered DPDK-based applications before ScyllaDB existed). Seastar implements a shard-per-core architecture that eliminates shared state entirely.

Here's the mental model. On a 16-core machine:

Core 0: [Shard 0] → own memtable, own cache, own I/O queue, own network connections
Core 1: [Shard 1] → own memtable, own cache, own I/O queue, own network connections
Core 2: [Shard 2] → own memtable, own cache, own I/O queue, own network connections
...
Core 15: [Shard 15] → own memtable, own cache, own I/O queue, own network connections

Each shard is a complete mini-database. There are no locks, because there's nothing shared to lock. When data needs to move between shards (which happens during range scans or when a request arrives at the wrong shard), Seastar uses explicit message passing, not shared memory.

The Seastar reactor loop on each core looks roughly like this:

void reactor::run() {
 while (true) {
 // Poll for completed I/O (no syscall if using io_uring or AIO)
 poll_io();

 // Run ready tasks from this shard's task queue
 run_tasks();

 // Poll network (epoll or DPDK, depending on config)
 poll_network();

 // Check timers
 expire_timers();
 }
}

This is cooperative multitasking. There are no context switches, no thread migrations, no scheduler overhead. A task runs until it voluntarily yields (typically at an I/O boundary), and then the next task on that shard picks up. The CPU cache stays hot because the same core always processes the same data.

Shard-Per-Core in Practice: What It Means for Queries

When a CQL query arrives at a ScyllaDB node, the coordinator determines which shard owns the partition key using a hash:

unsigned shard_of(const partition_key& pk) {
 auto token = partitioner().get_token(pk);
 return token.raw() % smp::count; // simplified
}

If the request lands on the correct shard's core, processing is entirely local, no cross-core communication, no cache line bouncing, no NUMA penalties. The shard-aware drivers that ScyllaDB provides for Java, Go, Python, and Rust implement this same hashing client-side:

# ScyllaDB Python shard-aware driver
from cassandra.cluster import Cluster
from cassandra.policies import TokenAwarePolicy, DCAwareRoundRobinPolicy

cluster = Cluster(
 ['scylla-node-1', 'scylla-node-2', 'scylla-node-3'],
 load_balancing_policy=TokenAwarePolicy(DCAwareRoundRobinPolicy()),
 # The shard-aware extension adds shard routing on top of token awareness
 shard_aware_options=ShardAwareOptions()
)

With shard-aware routing, the driver opens one connection per shard per node. On a 3-node cluster with 16 cores each, that's 48 connections, but each connection goes directly to the right shard, eliminating the coordinator hop overhead.

Real p99 Latency Comparison

These benchmarks ran on identical hardware, 3 nodes, each with 16 vCPUs (c5d.4xlarge on AWS), 32GB RAM, NVMe local storage. Same schema, same dataset (800M partitions, ~2TB), same YCSB workload.

Workload B (95% read, 5% write), 50K ops/sec sustained

 p50 p95 p99 p99.9
Cassandra 4.1 1.2ms 4.8ms 28.3ms 312ms
ScyllaDB 5.4 0.4ms 0.9ms 1.8ms 4.2ms

Look at p99.9. That 312ms on Cassandra? That's GC. There's nothing you can tune to make it go away, it's fundamental to the architecture. ScyllaDB's 4.2ms at p99.9 is I/O scheduling jitter, and it's predictable.

The write path is even more dramatic because Cassandra's memtable is a shared ConcurrentSkipListMap that becomes contended under write-heavy loads:

Workload A (50% read, 50% write), 50K ops/sec sustained

 p50 p95 p99 p99.9
Cassandra 4.1 1.8ms 8.2ms 45.1ms 580ms
ScyllaDB 5.4 0.3ms 0.7ms 1.4ms 3.8ms

The Compaction Story

Cassandra ships with SizeTieredCompactionStrategy (STCS) and LeveledCompactionStrategy (LCS). Both have well-known problems: STCS causes temporary space amplification up to 2x, LCS causes write amplification that kills throughput.

ScyllaDB introduced Incremental Compaction Strategy (ICS), which is their default and has no direct Cassandra equivalent. It operates at a sub-SSTable granularity, instead of reading and rewriting entire SSTables, it works on sorted runs within SSTables:

Traditional Compaction (STCS):
 SSTable_A (4GB) + SSTable_B (4GB) → SSTable_C (8GB)
 Temporary space needed: 16GB (input + output)

Incremental Compaction (ICS):
 SSTable_A[run_1] + SSTable_B[run_1] → merged_run_1
 SSTable_A[run_2] + SSTable_B[run_2] → merged_run_2
 Temporary space needed: ~1GB (one run at a time)

This means you can run ScyllaDB on 50% disk utilization instead of needing 50% free for compaction headroom. On a 2TB NVMe drive, that's an extra terabyte of usable storage.

Migration Guide: Cassandra to ScyllaDB

Here is a proven migration process, validated across multiple production environments. It is less painful than expected.

Step 1: Schema Compatibility Check

Export your Cassandra schema and validate it against ScyllaDB:

cqlsh cassandra-node -e "DESCRIBE KEYSPACE my_keyspace" > schema.cql

# ScyllaDB supports 99% of CQL. Main incompatibilities:
# - Materialized Views: supported but with different consistency guarantees
# - SASI indexes: not supported, use ScyllaDB's secondary index implementation
# - Custom types from Java: need equivalent UDTs

Step 2: SSTable Streaming

For datasets under 2TB, direct SSTable loading is the fastest path:

# On each Cassandra node, take a snapshot
nodetool snapshot -t migration_snap my_keyspace

# Copy SSTables to ScyllaDB nodes (respecting token ranges)
# ScyllaDB's sstableloader handles the token mapping
sstableloader -d scylla-node-1,scylla-node-2,scylla-node-3 \
 /var/lib/cassandra/data/my_keyspace/my_table-<uuid>/snapshots/migration_snap/

Step 3: Validation

# Compare row counts and checksums between clusters
import hashlib
from cassandra.cluster import Cluster

def checksum_partition_range(session, table, token_start, token_end):
 rows = session.execute(f"""
 SELECT * FROM {table}
 WHERE token(partition_key) >= {token_start}
 AND token(partition_key) < {token_end}
 """)
 h = hashlib.sha256()
 count = 0
 for row in rows:
 h.update(str(row).encode())
 count += 1
 return count, h.hexdigest()

Step 4: Traffic Cutover

A shadow-traffic phase is strongly recommended where you send reads to both clusters and compare results:

async def dual_read(cassandra_session, scylla_session, query, params):
 cass_future = cassandra_session.execute_async(query, params)
 scylla_future = scylla_session.execute_async(query, params)

 cass_result = cass_future.result()
 scylla_result = scylla_future.result()

 if cass_result != scylla_result:
 log.error(f"Mismatch for query {query}: cassandra={cass_result}, scylla={scylla_result}")
 metrics.increment("dual_read.mismatch")

 return scylla_result # serve from ScyllaDB when confident

Operational Differences to Know

A few important operational differences:

Memory management: ScyllaDB uses all available RAM by default for its row cache and doesn't need heap tuning. Don't set memory limits unless you're sharing the machine (which is not recommended, see the tip above).

Workload prioritization: ScyllaDB supports service levels that let you prioritize workloads per CQL user. This is incredible for multi-tenant setups:

CREATE SERVICE LEVEL gold WITH timeout = 500ms AND workload_type = 'interactive';
CREATE SERVICE LEVEL batch WITH timeout = 30s AND workload_type = 'batch';

ALTER ROLE analytics_user WITH SERVICE LEVEL batch;
ALTER ROLE api_user WITH SERVICE LEVEL gold;

Repair: ScyllaDB's row-level repair is significantly faster than Cassandra's. What took 8 hours on a comparable Cassandra cluster finishes in 45 minutes on ScyllaDB, because it only streams the actual differing rows instead of entire SSTables.

When NOT to Use ScyllaDB

ScyllaDB is not the right choice for:

• Datasets under 100GB where PostgreSQL is perfectly adequate
• Workloads that are 99% full-table scans (use ClickHouse or DuckDB instead)
• Teams that need the Apache Cassandra ecosystem of tools without modification

But if you're running Cassandra today and fighting with tail latencies, paying for extra nodes to compensate for GC pauses, or spending too much time tuning JVM parameters, ScyllaDB is the answer that addresses these problems directly.

The migration is not free, but teams consistently come out the other side running fewer nodes, with better latencies, and with significantly less operational toil. That is not marketing, that is production reality.