Infrastructure # Kafka vs Redpanda: Event Streaming Benchmarks and Migration Analysis 2026-01-229 min read Lucio Durán Engineering Manager & AI Solutions Architect Also available in: [Español]() ## The Architecture Split That Matters The fundamental architectural differences are the starting point, as all performance characteristics derive from them. **[Apache Kafka]()** is a JVM application. It's written in Scala and Java, runs on the JVM with its associated garbage collector, and until recently required ZooKeeper as a separate coordination service. KRaft mode (Kafka's built-in Raft consensus) has been production-ready since Kafka 3.5, and ZooKeeper is officially deprecated as of 3.7. **[Redpanda]()** is a C++ application built on the Seastar framework, the same async I/O framework that powers ScyllaDB. No JVM, no garbage collector, no ZooKeeper. It uses Raft for consensus natively, with each partition getting its own Raft group. This is not merely a language difference. It is a fundamental difference in how they interact with hardware. ### Thread-Per-Core vs Thread Pool Seastar's thread-per-core model means each CPU core runs a single thread that owns its memory, its network connections, and its disk I/O. No locks, no context switches between threads, no shared memory between cores. ``` Redpanda (Seastar/thread-per-core): ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ Core 0 │ │ Core 1 │ │ Core 2 │ │ Core 3 │ │ Shard 0 │ │ Shard 1 │ │ Shard 2 │ │ Shard 3 │ │ Own mem │ │ Own mem │ │ Own mem │ │ Own mem │ │ Own I/O │ │ Own I/O │ │ Own I/O │ │ Own I/O │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ ↕ message passing (no shared state) Kafka (JVM/thread pool): ┌───────────────────────────────────────────────────┐ │ JVM Heap (shared) │ │ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │ │ │Thread 1│ │Thread 2│ │Thread 3│ │Thread N│ │ │ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ │ │ └──────────┴──────────┴──────────┘ │ │ Synchronized access │ │ GC pauses affect ALL │ └───────────────────────────────────────────────────┘ ``` Kafka's threads share heap memory and coordinate through locks and the JVM's memory model. When the G1 or ZGC collector kicks in, every thread pauses (or at least gets slowed down during concurrent phases). Redpanda simply doesn't have this problem, there's no GC because there's no managed heap. ## The Benchmarks The following tests ran on identical hardware: 3-node clusters, each node with 16 cores (AMD EPYC 7313), 64GB RAM, 2x NVMe SSDs in RAID 0, 25Gbps network. This represents a "serious but not extravagant" setup. ### Throughput Test Producer configuration: 1KB messages, batching enabled (` linger.ms=5 `, ` batch.size=65536 `), ` acks=all `, 6 partitions, replication factor 3. ```bash # Kafka benchmark bin/kafka-producer-perf-test.sh \ --topic bench-topic \ --num-records 10000000 \ --record-size 1024 \ --throughput -1 \ --producer-props \ bootstrap.servers=kafka1:9092,kafka2:9092,kafka3:9092 \ acks=all \ linger.ms=5 \ batch.size=65536 # Redpanda benchmark (same tool works, API compatible) bin/kafka-producer-perf-test.sh \ --topic bench-topic \ --num-records 10000000 \ --record-size 1024 \ --throughput -1 \ --producer-props \ bootstrap.servers=redpanda1:9092,redpanda2:9092,redpanda3:9092 \ acks=all \ linger.ms=5 \ batch.size=65536 ``` Results (10M messages, 1KB each): | Metric | Kafka 3.7 (KRaft) | Redpanda 24.3 | | --- | --- | --- | | Throughput (MB/s) | 850 | 920 | | Avg latency | 3.2ms | 1.8ms | | p50 latency | 2.1ms | 1.4ms | | p99 latency | 28ms | 4.2ms | | p99.9 latency | 145ms | 8.1ms | Note the p99.9 number. **145ms vs 8.1ms**. That's not a rounding error, it's the JVM's G1GC doing a mixed collection. And this was after extensive JVM flag tuning: ```bash # Kafka JVM tuning (KAFKA_HEAP_OPTS) -Xms12g -Xmx12g -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:G1HeapRegionSize=16M -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=25 -XX:+ParallelRefProcEnabled -XX:+ExplicitGCInvokesConcurrent ``` Without that tuning, Kafka's p99.9 was over 400ms. The JVM is powerful, but you pay for it at the tail. ### Consumer Throughput For consumption, a 3-consumer group was tested reading from the same topic: | Metric | Kafka 3.7 | Redpanda 24.3 | | --- | --- | --- | | Consumer throughput (MB/s) | 1,100 | 1,250 | | Rebalance time (3→4 consumers) | 12s | 3s | | End-to-end latency (produce→consume) | 8ms avg | 3ms avg | The rebalance time difference is significant in practice. Every rebalance means a period where partitions are unassigned and no messages are being processed. Kafka's cooperative sticky assignor improved things, but Redpanda's partition assignment is just faster because there's no coordination with a separate metadata service. ## The Migration: A Case Study Consider a 5-node Kafka cluster handling about 2TB of events per day, user analytics, order events, inventory updates, and related data streams. The decision to try Redpanda came from two things: the on-call team was spending 30% of incident time on ZooKeeper issues or JVM tuning, and the p99 latency SLO was getting harder to meet. ### Step 1: Shadow Cluster First, a 3-node Redpanda cluster was set up and used MirrorMaker 2 to replicate all topics: ```properties # mm2.properties clusters = kafka, redpanda kafka.bootstrap.servers = kafka1:9092,kafka2:9092,kafka3:9092 redpanda.bootstrap.servers = rp1:9092,rp2:9092,rp3:9092 kafka->redpanda.enabled = true kafka->redpanda.topics = .* sync.topic.configs.enabled = true sync.topic.acls.enabled = false replication.factor = 3 # Critical: set this to avoid topic name mangling replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy ``` This worked out of the box. MirrorMaker 2 doesn't know or care that the target is Redpanda, it just speaks the Kafka protocol. ### Step 2: Consumer Validation Shadow consumers ran on the Redpanda cluster and compared message counts, ordering, and content against the Kafka consumers. This caught two issues: 1. **Timestamp handling**: Redpanda's ` LogAppendTime ` had microsecond precision differences from Kafka. Deduplication logic that used timestamp comparisons broke. 2. **Transaction markers**: A few producers were using transactions (` enable.idempotence=true ` + ` transactional.id `). The transaction commit markers showed up differently in Redpanda's internal representation. Consumers using ` read_committed ` isolation worked fine, but a custom offset tracking tool that parsed the raw partition data broke. ### Step 3: The Switchover A rolling migration was performed, moving one consumer group at a time: ```python # Simplified migration script import confluent_kafka def migrate_consumer_group(group_id, kafka_brokers, redpanda_brokers, topics): # 1. Get current committed offsets from Kafka admin = confluent_kafka.admin.AdminClient({ 'bootstrap.servers': kafka_brokers }) committed = admin.list_consumer_group_offsets([ confluent_kafka.admin.ConsumerGroupTopicPartitions(group_id) ]) # 2. Map offsets to Redpanda (MirrorMaker preserves offsets with IdentityReplicationPolicy) rp_admin = confluent_kafka.admin.AdminClient({ 'bootstrap.servers': redpanda_brokers }) # 3. Pre-commit offsets to Redpanda before starting consumers offsets = [] for tp, offset_meta in committed[group_id].result().topic_partitions: offsets.append(confluent_kafka.TopicPartition( tp.topic, tp.partition, offset_meta.offset )) rp_admin.alter_consumer_group_offsets([ confluent_kafka.admin.ConsumerGroupTopicPartitions(group_id, offsets) ]) print(f"Migrated {group_id}: {len(offsets)} partition offsets committed to Redpanda") ``` ### Step 4: Producer Switchover The producer bootstrap servers were changed during a low-traffic window: ```yaml # Before kafka: bootstrap-servers: kafka1:9092,kafka2:9092,kafka3:9092 # After kafka: # kept the config key name to minimize code changes bootstrap-servers: rp1:9092,rp2:9092,rp3:9092 ``` It worked without issues. No code changes needed. The Kafka client library connected to Redpanda and started producing. Idempotent producers, transactional producers, partitioned producers, all of them. ## Schema Registry: The Compatibility Story Redpanda bundles its own schema registry that's API-compatible with Confluent's. The migration covered ~200 Avro schemas: ```bash # Export schemas from Confluent Schema Registry for subject in $(curl -s http://kafka-sr:8081/subjects | jq -r '.[]'); do schema=$(curl -s "http://kafka-sr:8081/subjects/${subject}/versions/latest") echo "$schema" > "schemas/${subject}.json" done # Import into Redpanda's schema registry for file in schemas/*.json; do subject=$(basename "$file" .json) schema=$(cat "$file" | jq -c '{schema: .schema, schemaType: .schemaType}') curl -X POST "http://rp1:8081/subjects/${subject}/versions" \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d "$schema" done ``` Compatibility modes (BACKWARD, FORWARD, FULL) all worked. The only gotcha was that Redpanda's schema registry stores data internally in a ` _schemas ` topic, while Confluent uses ` _schemas ` too, but the message format differs. You can't just replicate the topic, you need to re-register. ## Operational Differences That Matter Day-to-Day ### Monitoring Redpanda exposes Prometheus metrics on port 9644 by default. No JMX, no JMX-to-Prometheus exporters, no fiddling with remote JMX authentication: ```yaml # prometheus.yml scrape_configs: - job_name: redpanda static_configs: - targets: - rp1:9644 - rp2:9644 - rp3:9644 metrics_path: /public_metrics ``` With Kafka, obtaining good metrics requires a JMX exporter sidecar, which itself consumes resources and adds complexity. ### Disk Usage Redpanda uses about 20-30% less disk for the same data, mainly because it doesn't have the overhead of index files and timeindex files that Kafka creates per segment. Redpanda keeps its index in memory using its own storage engine. ### Memory Memory management presents a notable contrast. Kafka needs JVM heap tuning, too little and GC pauses kill you, too much and GC pauses still kill you (just differently). With Redpanda, you tell it how much memory to use and it manages it: ```bash rpk redpanda config set --set redpanda.memory=48G rpk redpanda config set --set redpanda.reserve_memory=4G ``` No heap tuning, no GC tuning, no ` -XX: ` flags. The Seastar allocator handles everything. ## Where Kafka Still Wins Kafka remains a strong choice, and there are substantive reasons to continue using it: 1. **Kafka Streams and ksqlDB**: The stream processing ecosystem is unmatched. Redpanda has no equivalent, you'd pair it with Flink or your own consumers. 2. **Kafka Connect**: While Redpanda can run Kafka Connect (it's just the Kafka protocol), the tested-in-production connector ecosystem is deeper with Kafka. 3. **KRaft maturity**: KRaft mode eliminates ZooKeeper, and it's been stable since 3.5. The operational gap is narrowing. 4. **Community and hiring**: Finding Kafka engineers is easier. Redpanda expertise is rarer. 5. **Tiered storage**: Kafka's tiered storage (writing cold segments to S3) is more mature than Redpanda's implementation. ## The Numbers After 8 Months After migration to Redpanda: - **p99 produce latency**: Dropped from 28ms to 4ms (consistent, not just initial) - **On-call incidents related to streaming**: Down 70% - **Infrastructure cost**: Down 40% (3 Redpanda nodes replaced 5 Kafka + 3 ZooKeeper nodes) - **Time spent on JVM tuning**: Zero, because there's no JVM The migration proved worthwhile for this use case. However, it is not universally advisable. If a Kafka cluster is stable, latency is acceptable, and Kafka Streams is heavily used, staying with Kafka may be the better choice. Migration has real costs and real risks. For greenfield projects, however, Redpanda merits strong consideration. The operational simplicity alone justifies the choice. ## Frequently asked questions ### Is Redpanda a drop-in replacement for Kafka? At the API level, mostly yes. Redpanda implements the Kafka wire protocol, so most Kafka clients, connectors, and tools work without changes. However, internal behaviors like replication timing, compaction, and transaction semantics have subtle differences. ### Does Redpanda really need no JVM? Correct. Redpanda is written in C++ using the Seastar framework, which means no JVM, no GC pauses, and no JVM tuning. This is its biggest architectural advantage for latency-sensitive workloads. ### Should I migrate from Kafka to Redpanda? It depends. If your pain points are operational complexity (ZooKeeper, JVM tuning, GC pauses) and tail latency, Redpanda is compelling. If you're deeply invested in the Kafka Streams ecosystem or need specific KRaft features, staying on Kafka may make more sense. ### How does Redpanda handle schema registry? Redpanda bundles a schema registry that's API-compatible with Confluent Schema Registry. It supports Avro, Protobuf, and JSON Schema with the same REST API endpoints. ## Tips 1. **Tip:** When benchmarking Kafka vs Redpanda, always measure *tail* latency (p99/p99.9), not just throughput. Kafka's JVM GC pauses create latency spikes that averages completely hide. Use ` kafka-producer-perf-test.sh --print-metrics ` to get percentile data. 2. **Tip:** Redpanda's ` rpk ` CLI is a drop-in replacement for most ` kafka-topics.sh ` commands. Start with ` rpk topic list ` and ` rpk cluster info `, they're faster and the output is actually readable. 3. **Tip:** If migrating from Kafka to Redpanda, use MirrorMaker 2 with ` sync.topic.configs.enabled=true `. Redpanda's Kafka API compatibility is good enough that MM2 works without modifications, but test your ` exactly_once ` producers, transaction semantics have subtle differences. 4. **Tip:** Redpanda's ` --smp ` flag controls how many CPU cores it uses. On a 16-core machine, start with ` --smp 14 ` and leave 2 cores for the OS and ` rpk ` monitoring. Over-subscribing CPUs tanks p99 latency. - kafka - redpanda - event-streaming - benchmarks - infrastructure - raft - zookeeper - p99-latency [Back to Blog]()