{"schemaVersion":1,"id":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks","title":"Kafka vs Redpanda: Event Streaming Benchmarks and Migration Analysis","description":"A head-to-head comparison between Apache Kafka and Redpanda. Raft vs ZooKeeper, C++ vs JVM overhead, tail latency at p99, schema registry compatibility, and a detailed analysis of migrating 2TB/day of event streams.","language":"en","datePublished":"2026-01-22","dateModified":null,"contentFormat":"text/markdown","content":"Infrastructure\n\n# Kafka vs Redpanda: Event Streaming Benchmarks and Migration Analysis\n\n2026-01-229 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/kafka-vs-redpanda-event-streaming-benchmarks>)\n\n## The Architecture Split That Matters\n\nThe fundamental architectural differences are the starting point, as all performance characteristics derive from them.\n\n**[Apache Kafka](<https://kafka.apache.org/>)** is a JVM application. It's written in Scala and Java, runs on the JVM with its associated garbage collector, and until recently required ZooKeeper as a separate coordination service. KRaft mode (Kafka's built-in Raft consensus) has been production-ready since Kafka 3.5, and ZooKeeper is officially deprecated as of 3.7.\n\n**[Redpanda](<https://redpanda.com/>)** is a C++ application built on the Seastar framework, the same async I/O framework that powers ScyllaDB. No JVM, no garbage collector, no ZooKeeper. It uses Raft for consensus natively, with each partition getting its own Raft group.\n\nThis is not merely a language difference. It is a fundamental difference in how they interact with hardware.\n\n### Thread-Per-Core vs Thread Pool\n\nSeastar's thread-per-core model means each CPU core runs a single thread that owns its memory, its network connections, and its disk I/O. No locks, no context switches between threads, no shared memory between cores.\n\n```\nRedpanda (Seastar/thread-per-core):\n┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐\n│ Core 0 │ │ Core 1 │ │ Core 2 │ │ Core 3 │\n│ Shard 0 │ │ Shard 1 │ │ Shard 2 │ │ Shard 3 │\n│ Own mem │ │ Own mem │ │ Own mem │ │ Own mem │\n│ Own I/O │ │ Own I/O │ │ Own I/O │ │ Own I/O │\n└──────────┘ └──────────┘ └──────────┘ └──────────┘\n ↕ message passing (no shared state)\n\nKafka (JVM/thread pool):\n┌───────────────────────────────────────────────────┐\n│ JVM Heap (shared) │\n│ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │\n│ │Thread 1│ │Thread 2│ │Thread 3│ │Thread N│ │\n│ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ │\n│ └──────────┴──────────┴──────────┘ │\n│ Synchronized access │\n│ GC pauses affect ALL │\n└───────────────────────────────────────────────────┘\n```\n\nKafka's threads share heap memory and coordinate through locks and the JVM's memory model. When the G1 or ZGC collector kicks in, every thread pauses (or at least gets slowed down during concurrent phases). Redpanda simply doesn't have this problem, there's no GC because there's no managed heap.\n\n## The Benchmarks\n\nThe following tests ran on identical hardware: 3-node clusters, each node with 16 cores (AMD EPYC 7313), 64GB RAM, 2x NVMe SSDs in RAID 0, 25Gbps network. This represents a \"serious but not extravagant\" setup.\n\n### Throughput Test\n\nProducer configuration: 1KB messages, batching enabled (` linger.ms=5 `, ` batch.size=65536 `), ` acks=all `, 6 partitions, replication factor 3.\n\n```bash\n# Kafka benchmark\nbin/kafka-producer-perf-test.sh \\\n --topic bench-topic \\\n --num-records 10000000 \\\n --record-size 1024 \\\n --throughput -1 \\\n --producer-props \\\n bootstrap.servers=kafka1:9092,kafka2:9092,kafka3:9092 \\\n acks=all \\\n linger.ms=5 \\\n batch.size=65536\n\n# Redpanda benchmark (same tool works, API compatible)\nbin/kafka-producer-perf-test.sh \\\n --topic bench-topic \\\n --num-records 10000000 \\\n --record-size 1024 \\\n --throughput -1 \\\n --producer-props \\\n bootstrap.servers=redpanda1:9092,redpanda2:9092,redpanda3:9092 \\\n acks=all \\\n linger.ms=5 \\\n batch.size=65536\n```\n\nResults (10M messages, 1KB each):\n\n| Metric | Kafka 3.7 (KRaft) | Redpanda 24.3 |\n| --- | --- | --- |\n| Throughput (MB/s) | 850 | 920 |\n| Avg latency | 3.2ms | 1.8ms |\n| p50 latency | 2.1ms | 1.4ms |\n| p99 latency | 28ms | 4.2ms |\n| p99.9 latency | 145ms | 8.1ms |\n\nNote the p99.9 number. **145ms vs 8.1ms**. That's not a rounding error, it's the JVM's G1GC doing a mixed collection. And this was after extensive JVM flag tuning:\n\n```bash\n# Kafka JVM tuning (KAFKA_HEAP_OPTS)\n-Xms12g -Xmx12g\n-XX:+UseG1GC\n-XX:MaxGCPauseMillis=20\n-XX:G1HeapRegionSize=16M\n-XX:InitiatingHeapOccupancyPercent=35\n-XX:G1ReservePercent=25\n-XX:+ParallelRefProcEnabled\n-XX:+ExplicitGCInvokesConcurrent\n```\n\nWithout that tuning, Kafka's p99.9 was over 400ms. The JVM is powerful, but you pay for it at the tail.\n\n### Consumer Throughput\n\nFor consumption, a 3-consumer group was tested reading from the same topic:\n\n| Metric | Kafka 3.7 | Redpanda 24.3 |\n| --- | --- | --- |\n| Consumer throughput (MB/s) | 1,100 | 1,250 |\n| Rebalance time (3→4 consumers) | 12s | 3s |\n| End-to-end latency (produce→consume) | 8ms avg | 3ms avg |\n\nThe rebalance time difference is significant in practice. Every rebalance means a period where partitions are unassigned and no messages are being processed. Kafka's cooperative sticky assignor improved things, but Redpanda's partition assignment is just faster because there's no coordination with a separate metadata service.\n\n## The Migration: A Case Study\n\nConsider a 5-node Kafka cluster handling about 2TB of events per day, user analytics, order events, inventory updates, and related data streams. The decision to try Redpanda came from two things: the on-call team was spending 30% of incident time on ZooKeeper issues or JVM tuning, and the p99 latency SLO was getting harder to meet.\n\n### Step 1: Shadow Cluster\n\nFirst, a 3-node Redpanda cluster was set up and used MirrorMaker 2 to replicate all topics:\n\n```properties\n# mm2.properties\nclusters = kafka, redpanda\nkafka.bootstrap.servers = kafka1:9092,kafka2:9092,kafka3:9092\nredpanda.bootstrap.servers = rp1:9092,rp2:9092,rp3:9092\n\nkafka->redpanda.enabled = true\nkafka->redpanda.topics = .*\n\nsync.topic.configs.enabled = true\nsync.topic.acls.enabled = false\nreplication.factor = 3\n\n# Critical: set this to avoid topic name mangling\nreplication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy\n```\n\nThis worked out of the box. MirrorMaker 2 doesn't know or care that the target is Redpanda, it just speaks the Kafka protocol.\n\n### Step 2: Consumer Validation\n\nShadow consumers ran on the Redpanda cluster and compared message counts, ordering, and content against the Kafka consumers. This caught two issues:\n\n1. **Timestamp handling**: Redpanda's ` LogAppendTime ` had microsecond precision differences from Kafka. Deduplication logic that used timestamp comparisons broke.\n2. **Transaction markers**: A few producers were using transactions (` enable.idempotence=true ` + ` transactional.id `). The transaction commit markers showed up differently in Redpanda's internal representation. Consumers using ` read_committed ` isolation worked fine, but a custom offset tracking tool that parsed the raw partition data broke.\n\n### Step 3: The Switchover\n\nA rolling migration was performed, moving one consumer group at a time:\n\n```python\n# Simplified migration script\nimport confluent_kafka\n\ndef migrate_consumer_group(group_id, kafka_brokers, redpanda_brokers, topics):\n # 1. Get current committed offsets from Kafka\n admin = confluent_kafka.admin.AdminClient({\n 'bootstrap.servers': kafka_brokers\n })\n\n committed = admin.list_consumer_group_offsets([\n confluent_kafka.admin.ConsumerGroupTopicPartitions(group_id)\n ])\n\n # 2. Map offsets to Redpanda (MirrorMaker preserves offsets with IdentityReplicationPolicy)\n rp_admin = confluent_kafka.admin.AdminClient({\n 'bootstrap.servers': redpanda_brokers\n })\n\n # 3. Pre-commit offsets to Redpanda before starting consumers\n offsets = []\n for tp, offset_meta in committed[group_id].result().topic_partitions:\n offsets.append(confluent_kafka.TopicPartition(\n tp.topic, tp.partition, offset_meta.offset\n ))\n\n rp_admin.alter_consumer_group_offsets([\n confluent_kafka.admin.ConsumerGroupTopicPartitions(group_id, offsets)\n ])\n\n print(f\"Migrated {group_id}: {len(offsets)} partition offsets committed to Redpanda\")\n```\n\n### Step 4: Producer Switchover\n\nThe producer bootstrap servers were changed during a low-traffic window:\n\n```yaml\n# Before\nkafka:\n bootstrap-servers: kafka1:9092,kafka2:9092,kafka3:9092\n\n# After\nkafka: # kept the config key name to minimize code changes\n bootstrap-servers: rp1:9092,rp2:9092,rp3:9092\n```\n\nIt worked without issues. No code changes needed. The Kafka client library connected to Redpanda and started producing. Idempotent producers, transactional producers, partitioned producers, all of them.\n\n## Schema Registry: The Compatibility Story\n\nRedpanda bundles its own schema registry that's API-compatible with Confluent's. The migration covered ~200 Avro schemas:\n\n```bash\n# Export schemas from Confluent Schema Registry\nfor subject in $(curl -s http://kafka-sr:8081/subjects | jq -r '.[]'); do\n schema=$(curl -s \"http://kafka-sr:8081/subjects/${subject}/versions/latest\")\n echo \"$schema\" > \"schemas/${subject}.json\"\ndone\n\n# Import into Redpanda's schema registry\nfor file in schemas/*.json; do\n subject=$(basename \"$file\" .json)\n schema=$(cat \"$file\" | jq -c '{schema: .schema, schemaType: .schemaType}')\n curl -X POST \"http://rp1:8081/subjects/${subject}/versions\" \\\n -H \"Content-Type: application/vnd.schemaregistry.v1+json\" \\\n -d \"$schema\"\ndone\n```\n\nCompatibility modes (BACKWARD, FORWARD, FULL) all worked. The only gotcha was that Redpanda's schema registry stores data internally in a ` _schemas ` topic, while Confluent uses ` _schemas ` too, but the message format differs. You can't just replicate the topic, you need to re-register.\n\n## Operational Differences That Matter Day-to-Day\n\n### Monitoring\n\nRedpanda exposes Prometheus metrics on port 9644 by default. No JMX, no JMX-to-Prometheus exporters, no fiddling with remote JMX authentication:\n\n```yaml\n# prometheus.yml\nscrape_configs:\n - job_name: redpanda\n static_configs:\n - targets:\n - rp1:9644\n - rp2:9644\n - rp3:9644\n metrics_path: /public_metrics\n```\n\nWith Kafka, obtaining good metrics requires a JMX exporter sidecar, which itself consumes resources and adds complexity.\n\n### Disk Usage\n\nRedpanda uses about 20-30% less disk for the same data, mainly because it doesn't have the overhead of index files and timeindex files that Kafka creates per segment. Redpanda keeps its index in memory using its own storage engine.\n\n### Memory\n\nMemory management presents a notable contrast. Kafka needs JVM heap tuning, too little and GC pauses kill you, too much and GC pauses still kill you (just differently). With Redpanda, you tell it how much memory to use and it manages it:\n\n```bash\nrpk redpanda config set --set redpanda.memory=48G\nrpk redpanda config set --set redpanda.reserve_memory=4G\n```\n\nNo heap tuning, no GC tuning, no ` -XX: ` flags. The Seastar allocator handles everything.\n\n## Where Kafka Still Wins\n\nKafka remains a strong choice, and there are substantive reasons to continue using it:\n\n1. **Kafka Streams and ksqlDB**: The stream processing ecosystem is unmatched. Redpanda has no equivalent, you'd pair it with Flink or your own consumers.\n2. **Kafka Connect**: While Redpanda can run Kafka Connect (it's just the Kafka protocol), the tested-in-production connector ecosystem is deeper with Kafka.\n3. **KRaft maturity**: KRaft mode eliminates ZooKeeper, and it's been stable since 3.5. The operational gap is narrowing.\n4. **Community and hiring**: Finding Kafka engineers is easier. Redpanda expertise is rarer.\n5. **Tiered storage**: Kafka's tiered storage (writing cold segments to S3) is more mature than Redpanda's implementation.\n\n## The Numbers After 8 Months\n\nAfter migration to Redpanda:\n\n- **p99 produce latency**: Dropped from 28ms to 4ms (consistent, not just initial)\n- **On-call incidents related to streaming**: Down 70%\n- **Infrastructure cost**: Down 40% (3 Redpanda nodes replaced 5 Kafka + 3 ZooKeeper nodes)\n- **Time spent on JVM tuning**: Zero, because there's no JVM\n\nThe migration proved worthwhile for this use case. However, it is not universally advisable. If a Kafka cluster is stable, latency is acceptable, and Kafka Streams is heavily used, staying with Kafka may be the better choice. Migration has real costs and real risks.\n\nFor greenfield projects, however, Redpanda merits strong consideration. The operational simplicity alone justifies the choice.\n\n## Frequently asked questions\n\n### Is Redpanda a drop-in replacement for Kafka?\n\nAt the API level, mostly yes. Redpanda implements the Kafka wire protocol, so most Kafka clients, connectors, and tools work without changes. However, internal behaviors like replication timing, compaction, and transaction semantics have subtle differences.\n\n### Does Redpanda really need no JVM?\n\nCorrect. Redpanda is written in C++ using the Seastar framework, which means no JVM, no GC pauses, and no JVM tuning. This is its biggest architectural advantage for latency-sensitive workloads.\n\n### Should I migrate from Kafka to Redpanda?\n\nIt depends. If your pain points are operational complexity (ZooKeeper, JVM tuning, GC pauses) and tail latency, Redpanda is compelling. If you're deeply invested in the Kafka Streams ecosystem or need specific KRaft features, staying on Kafka may make more sense.\n\n### How does Redpanda handle schema registry?\n\nRedpanda bundles a schema registry that's API-compatible with Confluent Schema Registry. It supports Avro, Protobuf, and JSON Schema with the same REST API endpoints.\n\n## Tips\n\n1. **Tip:** When benchmarking Kafka vs Redpanda, always measure *tail* latency (p99/p99.9), not just throughput. Kafka's JVM GC pauses create latency spikes that averages completely hide. Use ` kafka-producer-perf-test.sh --print-metrics ` to get percentile data.\n2. **Tip:** Redpanda's ` rpk ` CLI is a drop-in replacement for most ` kafka-topics.sh ` commands. Start with ` rpk topic list ` and ` rpk cluster info `, they're faster and the output is actually readable.\n3. **Tip:** If migrating from Kafka to Redpanda, use MirrorMaker 2 with ` sync.topic.configs.enabled=true `. Redpanda's Kafka API compatibility is good enough that MM2 works without modifications, but test your ` exactly_once ` producers, transaction semantics have subtle differences.\n4. **Tip:** Redpanda's ` --smp ` flag controls how many CPU cores it uses. On a 16-core machine, start with ` --smp 14 ` and leave 2 cores for the OS and ` rpk ` monitoring. Over-subscribing CPUs tanks p99 latency.\n\n- kafka\n- redpanda\n- event-streaming\n- benchmarks\n- infrastructure\n- raft\n- zookeeper\n- p99-latency\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"Kafka vs Redpanda: Event Streaming Benchmarks and Migration Analysis","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#title"},{"level":2,"text":"The Architecture Split That Matters","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#the-architecture-split-that-matters"},{"level":3,"text":"Thread-Per-Core vs Thread Pool","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#thread-per-core-vs-thread-pool"},{"level":2,"text":"The Benchmarks","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#the-benchmarks"},{"level":3,"text":"Throughput Test","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#throughput-test"},{"level":3,"text":"Consumer Throughput","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#consumer-throughput"},{"level":2,"text":"The Migration: A Case Study","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#the-migration-a-case-study"},{"level":3,"text":"Step 1: Shadow Cluster","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#step-1-shadow-cluster"},{"level":3,"text":"Step 2: Consumer Validation","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#step-2-consumer-validation"},{"level":3,"text":"Step 3: The Switchover","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#step-3-the-switchover"},{"level":3,"text":"Step 4: Producer Switchover","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#step-4-producer-switchover"},{"level":2,"text":"Schema Registry: The Compatibility Story","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#schema-registry-the-compatibility-story"},{"level":2,"text":"Operational Differences That Matter Day-to-Day","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#operational-differences-that-matter-day-to-day"},{"level":3,"text":"Monitoring","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#monitoring"},{"level":3,"text":"Disk Usage","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#disk-usage"},{"level":3,"text":"Memory","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#memory"},{"level":2,"text":"Where Kafka Still Wins","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#where-kafka-still-wins"},{"level":2,"text":"The Numbers After 8 Months","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#the-numbers-after-8-months"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#questions"},{"level":3,"text":"Is Redpanda a drop-in replacement for Kafka?","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#question-1"},{"level":3,"text":"Does Redpanda really need no JVM?","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#question-2"},{"level":3,"text":"Should I migrate from Kafka to Redpanda?","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#question-3"},{"level":3,"text":"How does Redpanda handle schema registry?","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#question-4"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#article-tips"}],"sections":[{"heading":"Kafka vs Redpanda: Event Streaming Benchmarks and Migration Analysis","level":1,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks","content":"Infrastructure"},{"heading":"Kafka vs Redpanda: Event Streaming Benchmarks and Migration Analysis","level":1,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#title","content":"2026-01-229 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/kafka-vs-redpanda-event-streaming-benchmarks>)"},{"heading":"The Architecture Split That Matters","level":2,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#the-architecture-split-that-matters","content":"The fundamental architectural differences are the starting point, as all performance characteristics derive from them.\n\n**[Apache Kafka](<https://kafka.apache.org/>)** is a JVM application. It's written in Scala and Java, runs on the JVM with its associated garbage collector, and until recently required ZooKeeper as a separate coordination service. KRaft mode (Kafka's built-in Raft consensus) has been production-ready since Kafka 3.5, and ZooKeeper is officially deprecated as of 3.7.\n\n**[Redpanda](<https://redpanda.com/>)** is a C++ application built on the Seastar framework, the same async I/O framework that powers ScyllaDB. No JVM, no garbage collector, no ZooKeeper. It uses Raft for consensus natively, with each partition getting its own Raft group.\n\nThis is not merely a language difference. It is a fundamental difference in how they interact with hardware."},{"heading":"Thread-Per-Core vs Thread Pool","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#thread-per-core-vs-thread-pool","content":"Seastar's thread-per-core model means each CPU core runs a single thread that owns its memory, its network connections, and its disk I/O. No locks, no context switches between threads, no shared memory between cores.\n\n```\nRedpanda (Seastar/thread-per-core):\n┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐\n│ Core 0 │ │ Core 1 │ │ Core 2 │ │ Core 3 │\n│ Shard 0 │ │ Shard 1 │ │ Shard 2 │ │ Shard 3 │\n│ Own mem │ │ Own mem │ │ Own mem │ │ Own mem │\n│ Own I/O │ │ Own I/O │ │ Own I/O │ │ Own I/O │\n└──────────┘ └──────────┘ └──────────┘ └──────────┘\n ↕ message passing (no shared state)\n\nKafka (JVM/thread pool):\n┌───────────────────────────────────────────────────┐\n│ JVM Heap (shared) │\n│ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │\n│ │Thread 1│ │Thread 2│ │Thread 3│ │Thread N│ │\n│ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ │\n│ └──────────┴──────────┴──────────┘ │\n│ Synchronized access │\n│ GC pauses affect ALL │\n└───────────────────────────────────────────────────┘\n```\n\nKafka's threads share heap memory and coordinate through locks and the JVM's memory model. When the G1 or ZGC collector kicks in, every thread pauses (or at least gets slowed down during concurrent phases). Redpanda simply doesn't have this problem, there's no GC because there's no managed heap."},{"heading":"The Benchmarks","level":2,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#the-benchmarks","content":"The following tests ran on identical hardware: 3-node clusters, each node with 16 cores (AMD EPYC 7313), 64GB RAM, 2x NVMe SSDs in RAID 0, 25Gbps network. This represents a \"serious but not extravagant\" setup."},{"heading":"Throughput Test","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#throughput-test","content":"Producer configuration: 1KB messages, batching enabled (` linger.ms=5 `, ` batch.size=65536 `), ` acks=all `, 6 partitions, replication factor 3.\n\n```bash\n# Kafka benchmark\nbin/kafka-producer-perf-test.sh \\\n --topic bench-topic \\\n --num-records 10000000 \\\n --record-size 1024 \\\n --throughput -1 \\\n --producer-props \\\n bootstrap.servers=kafka1:9092,kafka2:9092,kafka3:9092 \\\n acks=all \\\n linger.ms=5 \\\n batch.size=65536\n\n# Redpanda benchmark (same tool works, API compatible)\nbin/kafka-producer-perf-test.sh \\\n --topic bench-topic \\\n --num-records 10000000 \\\n --record-size 1024 \\\n --throughput -1 \\\n --producer-props \\\n bootstrap.servers=redpanda1:9092,redpanda2:9092,redpanda3:9092 \\\n acks=all \\\n linger.ms=5 \\\n batch.size=65536\n```\n\nResults (10M messages, 1KB each):\n\n| Metric | Kafka 3.7 (KRaft) | Redpanda 24.3 |\n| --- | --- | --- |\n| Throughput (MB/s) | 850 | 920 |\n| Avg latency | 3.2ms | 1.8ms |\n| p50 latency | 2.1ms | 1.4ms |\n| p99 latency | 28ms | 4.2ms |\n| p99.9 latency | 145ms | 8.1ms |\n\nNote the p99.9 number. **145ms vs 8.1ms**. That's not a rounding error, it's the JVM's G1GC doing a mixed collection. And this was after extensive JVM flag tuning:\n\n```bash\n# Kafka JVM tuning (KAFKA_HEAP_OPTS)\n-Xms12g -Xmx12g\n-XX:+UseG1GC\n-XX:MaxGCPauseMillis=20\n-XX:G1HeapRegionSize=16M\n-XX:InitiatingHeapOccupancyPercent=35\n-XX:G1ReservePercent=25\n-XX:+ParallelRefProcEnabled\n-XX:+ExplicitGCInvokesConcurrent\n```\n\nWithout that tuning, Kafka's p99.9 was over 400ms. The JVM is powerful, but you pay for it at the tail."},{"heading":"Consumer Throughput","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#consumer-throughput","content":"For consumption, a 3-consumer group was tested reading from the same topic:\n\n| Metric | Kafka 3.7 | Redpanda 24.3 |\n| --- | --- | --- |\n| Consumer throughput (MB/s) | 1,100 | 1,250 |\n| Rebalance time (3→4 consumers) | 12s | 3s |\n| End-to-end latency (produce→consume) | 8ms avg | 3ms avg |\n\nThe rebalance time difference is significant in practice. Every rebalance means a period where partitions are unassigned and no messages are being processed. Kafka's cooperative sticky assignor improved things, but Redpanda's partition assignment is just faster because there's no coordination with a separate metadata service."},{"heading":"The Migration: A Case Study","level":2,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#the-migration-a-case-study","content":"Consider a 5-node Kafka cluster handling about 2TB of events per day, user analytics, order events, inventory updates, and related data streams. The decision to try Redpanda came from two things: the on-call team was spending 30% of incident time on ZooKeeper issues or JVM tuning, and the p99 latency SLO was getting harder to meet."},{"heading":"Step 1: Shadow Cluster","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#step-1-shadow-cluster","content":"First, a 3-node Redpanda cluster was set up and used MirrorMaker 2 to replicate all topics:\n\n```properties\n# mm2.properties\nclusters = kafka, redpanda\nkafka.bootstrap.servers = kafka1:9092,kafka2:9092,kafka3:9092\nredpanda.bootstrap.servers = rp1:9092,rp2:9092,rp3:9092\n\nkafka->redpanda.enabled = true\nkafka->redpanda.topics = .*\n\nsync.topic.configs.enabled = true\nsync.topic.acls.enabled = false\nreplication.factor = 3\n\n# Critical: set this to avoid topic name mangling\nreplication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy\n```\n\nThis worked out of the box. MirrorMaker 2 doesn't know or care that the target is Redpanda, it just speaks the Kafka protocol."},{"heading":"Step 2: Consumer Validation","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#step-2-consumer-validation","content":"Shadow consumers ran on the Redpanda cluster and compared message counts, ordering, and content against the Kafka consumers. This caught two issues:\n\n1. **Timestamp handling**: Redpanda's ` LogAppendTime ` had microsecond precision differences from Kafka. Deduplication logic that used timestamp comparisons broke.\n2. **Transaction markers**: A few producers were using transactions (` enable.idempotence=true ` + ` transactional.id `). The transaction commit markers showed up differently in Redpanda's internal representation. Consumers using ` read_committed ` isolation worked fine, but a custom offset tracking tool that parsed the raw partition data broke."},{"heading":"Step 3: The Switchover","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#step-3-the-switchover","content":"A rolling migration was performed, moving one consumer group at a time:\n\n```python\n# Simplified migration script\nimport confluent_kafka\n\ndef migrate_consumer_group(group_id, kafka_brokers, redpanda_brokers, topics):\n # 1. Get current committed offsets from Kafka\n admin = confluent_kafka.admin.AdminClient({\n 'bootstrap.servers': kafka_brokers\n })\n\n committed = admin.list_consumer_group_offsets([\n confluent_kafka.admin.ConsumerGroupTopicPartitions(group_id)\n ])\n\n # 2. Map offsets to Redpanda (MirrorMaker preserves offsets with IdentityReplicationPolicy)\n rp_admin = confluent_kafka.admin.AdminClient({\n 'bootstrap.servers': redpanda_brokers\n })\n\n # 3. Pre-commit offsets to Redpanda before starting consumers\n offsets = []\n for tp, offset_meta in committed[group_id].result().topic_partitions:\n offsets.append(confluent_kafka.TopicPartition(\n tp.topic, tp.partition, offset_meta.offset\n ))\n\n rp_admin.alter_consumer_group_offsets([\n confluent_kafka.admin.ConsumerGroupTopicPartitions(group_id, offsets)\n ])\n\n print(f\"Migrated {group_id}: {len(offsets)} partition offsets committed to Redpanda\")\n```"},{"heading":"Step 4: Producer Switchover","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#step-4-producer-switchover","content":"The producer bootstrap servers were changed during a low-traffic window:\n\n```yaml\n# Before\nkafka:\n bootstrap-servers: kafka1:9092,kafka2:9092,kafka3:9092\n\n# After\nkafka: # kept the config key name to minimize code changes\n bootstrap-servers: rp1:9092,rp2:9092,rp3:9092\n```\n\nIt worked without issues. No code changes needed. The Kafka client library connected to Redpanda and started producing. Idempotent producers, transactional producers, partitioned producers, all of them."},{"heading":"Schema Registry: The Compatibility Story","level":2,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#schema-registry-the-compatibility-story","content":"Redpanda bundles its own schema registry that's API-compatible with Confluent's. The migration covered ~200 Avro schemas:\n\n```bash\n# Export schemas from Confluent Schema Registry\nfor subject in $(curl -s http://kafka-sr:8081/subjects | jq -r '.[]'); do\n schema=$(curl -s \"http://kafka-sr:8081/subjects/${subject}/versions/latest\")\n echo \"$schema\" > \"schemas/${subject}.json\"\ndone\n\n# Import into Redpanda's schema registry\nfor file in schemas/*.json; do\n subject=$(basename \"$file\" .json)\n schema=$(cat \"$file\" | jq -c '{schema: .schema, schemaType: .schemaType}')\n curl -X POST \"http://rp1:8081/subjects/${subject}/versions\" \\\n -H \"Content-Type: application/vnd.schemaregistry.v1+json\" \\\n -d \"$schema\"\ndone\n```\n\nCompatibility modes (BACKWARD, FORWARD, FULL) all worked. The only gotcha was that Redpanda's schema registry stores data internally in a ` _schemas ` topic, while Confluent uses ` _schemas ` too, but the message format differs. You can't just replicate the topic, you need to re-register."},{"heading":"Operational Differences That Matter Day-to-Day","level":2,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#operational-differences-that-matter-day-to-day","content":""},{"heading":"Monitoring","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#monitoring","content":"Redpanda exposes Prometheus metrics on port 9644 by default. No JMX, no JMX-to-Prometheus exporters, no fiddling with remote JMX authentication:\n\n```yaml\n# prometheus.yml\nscrape_configs:\n - job_name: redpanda\n static_configs:\n - targets:\n - rp1:9644\n - rp2:9644\n - rp3:9644\n metrics_path: /public_metrics\n```\n\nWith Kafka, obtaining good metrics requires a JMX exporter sidecar, which itself consumes resources and adds complexity."},{"heading":"Disk Usage","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#disk-usage","content":"Redpanda uses about 20-30% less disk for the same data, mainly because it doesn't have the overhead of index files and timeindex files that Kafka creates per segment. Redpanda keeps its index in memory using its own storage engine."},{"heading":"Memory","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#memory","content":"Memory management presents a notable contrast. Kafka needs JVM heap tuning, too little and GC pauses kill you, too much and GC pauses still kill you (just differently). With Redpanda, you tell it how much memory to use and it manages it:\n\n```bash\nrpk redpanda config set --set redpanda.memory=48G\nrpk redpanda config set --set redpanda.reserve_memory=4G\n```\n\nNo heap tuning, no GC tuning, no ` -XX: ` flags. The Seastar allocator handles everything."},{"heading":"Where Kafka Still Wins","level":2,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#where-kafka-still-wins","content":"Kafka remains a strong choice, and there are substantive reasons to continue using it:\n\n1. **Kafka Streams and ksqlDB**: The stream processing ecosystem is unmatched. Redpanda has no equivalent, you'd pair it with Flink or your own consumers.\n2. **Kafka Connect**: While Redpanda can run Kafka Connect (it's just the Kafka protocol), the tested-in-production connector ecosystem is deeper with Kafka.\n3. **KRaft maturity**: KRaft mode eliminates ZooKeeper, and it's been stable since 3.5. The operational gap is narrowing.\n4. **Community and hiring**: Finding Kafka engineers is easier. Redpanda expertise is rarer.\n5. **Tiered storage**: Kafka's tiered storage (writing cold segments to S3) is more mature than Redpanda's implementation."},{"heading":"The Numbers After 8 Months","level":2,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#the-numbers-after-8-months","content":"After migration to Redpanda:\n\n- **p99 produce latency**: Dropped from 28ms to 4ms (consistent, not just initial)\n- **On-call incidents related to streaming**: Down 70%\n- **Infrastructure cost**: Down 40% (3 Redpanda nodes replaced 5 Kafka + 3 ZooKeeper nodes)\n- **Time spent on JVM tuning**: Zero, because there's no JVM\n\nThe migration proved worthwhile for this use case. However, it is not universally advisable. If a Kafka cluster is stable, latency is acceptable, and Kafka Streams is heavily used, staying with Kafka may be the better choice. Migration has real costs and real risks.\n\nFor greenfield projects, however, Redpanda merits strong consideration. The operational simplicity alone justifies the choice."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#questions","content":""},{"heading":"Is Redpanda a drop-in replacement for Kafka?","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#question-1","content":"At the API level, mostly yes. Redpanda implements the Kafka wire protocol, so most Kafka clients, connectors, and tools work without changes. However, internal behaviors like replication timing, compaction, and transaction semantics have subtle differences."},{"heading":"Does Redpanda really need no JVM?","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#question-2","content":"Correct. Redpanda is written in C++ using the Seastar framework, which means no JVM, no GC pauses, and no JVM tuning. This is its biggest architectural advantage for latency-sensitive workloads."},{"heading":"Should I migrate from Kafka to Redpanda?","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#question-3","content":"It depends. If your pain points are operational complexity (ZooKeeper, JVM tuning, GC pauses) and tail latency, Redpanda is compelling. If you're deeply invested in the Kafka Streams ecosystem or need specific KRaft features, staying on Kafka may make more sense."},{"heading":"How does Redpanda handle schema registry?","level":3,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#question-4","content":"Redpanda bundles a schema registry that's API-compatible with Confluent Schema Registry. It supports Avro, Protobuf, and JSON Schema with the same REST API endpoints."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#article-tips","content":"1. **Tip:** When benchmarking Kafka vs Redpanda, always measure *tail* latency (p99/p99.9), not just throughput. Kafka's JVM GC pauses create latency spikes that averages completely hide. Use ` kafka-producer-perf-test.sh --print-metrics ` to get percentile data.\n2. **Tip:** Redpanda's ` rpk ` CLI is a drop-in replacement for most ` kafka-topics.sh ` commands. Start with ` rpk topic list ` and ` rpk cluster info `, they're faster and the output is actually readable.\n3. **Tip:** If migrating from Kafka to Redpanda, use MirrorMaker 2 with ` sync.topic.configs.enabled=true `. Redpanda's Kafka API compatibility is good enough that MM2 works without modifications, but test your ` exactly_once ` producers, transaction semantics have subtle differences.\n4. **Tip:** Redpanda's ` --smp ` flag controls how many CPU cores it uses. On a 16-core machine, start with ` --smp 14 ` and leave 2 cores for the OS and ` rpk ` monitoring. Over-subscribing CPUs tanks p99 latency.\n\n- kafka\n- redpanda\n- event-streaming\n- benchmarks\n- infrastructure\n- raft\n- zookeeper\n- p99-latency\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/kafka-vs-redpanda-event-streaming-benchmarks"},{"text":"Apache Kafka","url":"https://kafka.apache.org/"},{"text":"Redpanda","url":"https://redpanda.com/"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks"},"headline":"Kafka vs Redpanda: Event Streaming Benchmarks and Migration Analysis","description":"A head-to-head comparison between Apache Kafka and Redpanda. Raft vs ZooKeeper, C++ vs JVM overhead, tail latency at p99, schema registry compatibility, and a detailed analysis of migrating 2TB/day of event streams.","datePublished":"2026-01-22","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://statics.lucioduran.com/images/lucioduran-social-card.jpg","url":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Infrastructure","keywords":"kafka, redpanda, event-streaming, benchmarks, infrastructure, raft, zookeeper, p99-latency","wordCount":1522,"timeRequired":"PT9M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/kafka-vs-redpanda-event-streaming-benchmarks"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"Kafka vs Redpanda: Event Streaming Benchmarks and Migration Analysis","item":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"Is Redpanda a drop-in replacement for Kafka?","acceptedAnswer":{"@type":"Answer","text":"At the API level, mostly yes. Redpanda implements the Kafka wire protocol, so most Kafka clients, connectors, and tools work without changes. However, internal behaviors like replication timing, compaction, and transaction semantics have subtle differences."}},{"@type":"Question","name":"Does Redpanda really need no JVM?","acceptedAnswer":{"@type":"Answer","text":"Correct. Redpanda is written in C++ using the Seastar framework, which means no JVM, no GC pauses, and no JVM tuning. This is its biggest architectural advantage for latency-sensitive workloads."}},{"@type":"Question","name":"Should I migrate from Kafka to Redpanda?","acceptedAnswer":{"@type":"Answer","text":"It depends. If your pain points are operational complexity (ZooKeeper, JVM tuning, GC pauses) and tail latency, Redpanda is compelling. If you're deeply invested in the Kafka Streams ecosystem or need specific KRaft features, staying on Kafka may make more sense."}},{"@type":"Question","name":"How does Redpanda handle schema registry?","acceptedAnswer":{"@type":"Answer","text":"Redpanda bundles a schema registry that's API-compatible with Confluent Schema Registry. It supports Avro, Protobuf, and JSON Schema with the same REST API endpoints."}}]}]}],"contentHash":"df344581ca2eaad1d83bcbe3c7edc6747037766a76fe02f045680c976e5e309b","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/kafka-vs-redpanda-event-streaming-benchmarks","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":74903,"contentBytes":15098}}