Valkey vs Redis: Migrating 14TB of Production Data, What the Documentation Doesn't Cover

Why Valkey Exists (The Short Version)

Redis Labs changed their license from BSD to a dual SSPL/RSAL license in March 2024. This made it impossible for cloud providers to offer managed Redis without a commercial agreement. AWS, Google, Oracle, and others backed the Valkey fork under the Linux Foundation with a BSD-3-Clause license.

Politics aside, the relevant question is whether the software is better. The following sections examine the actual engineering.

RESP3: The Protocol Nobody Uses (But Should)

RESP3 (Redis Serialization Protocol v3) was introduced in Redis 6 but barely anyone uses it because RESP2 still works and most client libraries default to it. This is a mistake.

RESP3 introduces typed responses. In RESP2, everything is either a string, integer, array, or error. Your client library has to guess types:

# RESP2 response to HGETALL
*4
$4
name
$5
Lucio
$3
age
$2
38

# Client must know that alternating elements form key-value pairs
# and that "38" should probably be an integer

# RESP3 response to HGETALL
%2
$4
name
$5
Lucio
$3
age
:38

# The % prefix means "map with 2 entries"
# The : prefix means "this is an integer"
# No guessing needed

The practical impact is bigger than it looks. In Node.js services using ioredis, switching to RESP3 eliminates a pile of parseInt() and JSON.parse() calls on the application side. CPU utilization on the app servers can drop by ~8% on the hot path, not because the protocol is faster over the wire, but because we stopped doing redundant type coercion.

To enable RESP3 on an existing connection:

# Python with valkey-py (fork of redis-py)
import valkey

client = valkey.Valkey(
 host='valkey-primary.internal',
 port=6379,
 protocol=3 # This sends HELLO 3 on connect
)

# Now responses are natively typed
result = client.hgetall('user:1001')
# Returns: {'name': 'Lucio', 'age': 38}
# age is already an int, not b'38'

Multi-Threaded I/O: Where Valkey Actually Diverges

Redis has had io-threads since version 6, but it was limited to write operations and felt like an afterthought. Valkey took this further with both read and write threading, and the implementation is genuinely different.

The architecture:

 Main Thread
 ┌──────────┐
 │ Command │
 │ Execution│
 │ (single) │
 └────┬─────┘
 │
 ┌──────────┼──────────┐
 │ │ │
 ┌────┴───┐ ┌───┴────┐ ┌──┴─────┐
 │ I/O │ │ I/O │ │ I/O │
 │Thread 1│ │Thread 2│ │Thread 3│
 │ read/ │ │ read/ │ │ read/ │
 │ write │ │ write │ │ write │
 └────────┘ └────────┘ └────────┘

Command execution is still single-threaded (this is fundamental to Valkey/Redis's consistency model, no locking needed). But I/O parsing and response serialization happen in parallel across threads. Here is a configuration that works well on 8-core instances:

# valkey.conf
io-threads 4
io-threads-do-reads yes

# Don't set io-threads higher than your core count minus 2
# Main thread + background tasks need CPU headroom
# 4 threads on 8-core is typically the sweet spot

# Increase the backlog for high-connection workloads
tcp-backlog 4096

This was benchmarked systematically using valkey-benchmark with different connection counts:

# Single connection (no threading benefit expected)
valkey-benchmark -h 127.0.0.1 -p 6379 -c 1 -n 1000000 \
 -t set,get -d 256 --threads 1

# 200 connections (where threading matters)
valkey-benchmark -h 127.0.0.1 -p 6379 -c 200 -n 5000000 \
 -t set,get -d 256 --threads 4

# 1000 connections (stress test)
valkey-benchmark -h 127.0.0.1 -p 6379 -c 1000 -n 10000000 \
 -t set,get -d 256 --threads 8

Results on an r7g.2xlarge (8 vCPU, 64GB):

Connections	Redis 7.2 (ops/s)	Valkey 8.0 1-thread (ops/s)	Valkey 8.0 4-thread (ops/s)
1	142,000	144,000	143,000
50	398,000	405,000	612,000
200	445,000	451,000	891,000
500	462,000	468,000	1,024,000
1000	471,000	475,000	1,143,000

At 1 connection, no difference. At 1000 connections, Valkey with 4 I/O threads delivers 2.4x the throughput. This isn't magic, it's simply that socket read/write and response serialization are parallelized while command execution remains serialized.

The Migration: 14TB Across 12 Shards

The setup: 12 Redis clusters (each a primary + 2 replicas) handling session data, feature flags, rate limiting, and a real-time analytics pipeline. Total dataset: ~14TB across all shards.

The migration strategy was zero-downtime rolling replacement:

#!/bin/bash
# Simplified version of our migration script
# Real version had error handling, rollback, and monitoring hooks

SHARD=$1
REDIS_HOST="redis-${SHARD}.internal"
VALKEY_HOST="valkey-${SHARD}.internal"

# Step 1: Configure Valkey as replica of Redis
valkey-cli -h $VALKEY_HOST \
 REPLICAOF $REDIS_HOST 6379

# Step 2: Wait for sync to complete
while true; do
 STATUS=$(valkey-cli -h $VALKEY_HOST INFO replication \
 | grep master_sync_in_progress)
 if [[ $STATUS == *"0"* ]]; then
 echo "Sync complete for shard $SHARD"
 break
 fi
 sleep 5
done

# Step 3: Verify data integrity
REDIS_KEYS=$(redis-cli -h $REDIS_HOST DBSIZE | awk '{print $2}')
VALKEY_KEYS=$(valkey-cli -h $VALKEY_HOST DBSIZE | awk '{print $2}')
DIFF=$((VALKEY_KEYS - REDIS_KEYS))

if [ ${DIFF#-} -gt 100 ]; then
 echo "ERROR: Key count mismatch: Redis=$REDIS_KEYS Valkey=$VALKEY_KEYS"
 exit 1
fi

# Step 4: Promote Valkey and switch traffic
valkey-cli -h $VALKEY_HOST REPLICAOF NO ONE

# Step 5: Update DNS/service discovery
# (using Consul, this is a catalog update)
consul kv put "cache/shard-${SHARD}/primary" "$VALKEY_HOST:6379"

The replication protocol between Redis and Valkey is compatible (Valkey can replicate from Redis and vice versa up to the fork point). This was the critical enabler for zero-downtime migration. The approach was one shard per day over two weeks, monitoring each for 24 hours before moving to the next.

Gotchas Encountered

1. maxmemory-policy behavior difference. Valkey's allkeys-lfu eviction has a slightly different frequency decay curve than Redis 7.2. The cache hit rate dropped by ~3% on one shard after migration. The fix was tuning lfu-decay-time from the default 1 to 3:

lfu-decay-time 3
# Slower frequency decay = better at identifying truly hot keys
# vs. keys that had a burst of access

2. Lua script timeout handling. A Lua script occasionally took >5 seconds on large keyspaces. Redis kills the script and returns BUSY. Valkey also kills it, but the error message format is slightly different, which broke the retry logic that parsed the error string. Don't parse error messages. Use error codes.

3. RDB format version. Valkey introduced RDB version 12 for some of its new data structures. If you need to fall back to Redis, you can't load a Valkey-generated RDB file that uses the new format. It is advisable to keep Redis-compatible RDB backups for the first month:

# Force RDB compatibility during migration period
rdb-compat-version 11

Memory Optimization Analysis

Valkey 8.0 introduced changes to how small data structures are stored. The listpack encoding (which replaced ziplist in Redis 7) got further optimized with better memory alignment and reduced per-entry overhead.

Memory usage was measured on identical datasets:

# Test script: populate identical data structures
import valkey
import json
import random
import string

def random_string(n):
 return ''.join(random.choices(string.ascii_letters, k=n))

def populate(client, prefix, count=1_000_000):
 pipe = client.pipeline()
 for i in range(count):
 # Mix of data types
 pipe.set(f'{prefix}:str:{i}', random_string(64))
 pipe.hset(f'{prefix}:hash:{i}', mapping={
 'field1': random_string(32),
 'field2': str(random.randint(0, 100000)),
 'field3': random_string(16),
 })
 pipe.zadd(f'{prefix}:zset:{i % 10000}',
 {random_string(12): random.random()})
 if i % 5000 == 0:
 pipe.execute()
 pipe = client.pipeline()
 pipe.execute()

Memory usage for 1M entries of each type:

Data Type	Redis 7.2	Valkey 8.0	Savings
Strings (64B values)	89.2 MB	87.4 MB	2.0%
Hashes (3 fields)	142.7 MB	131.3 MB	8.0%
Sorted Sets (100 members)	167.4 MB	158.9 MB	5.1%
Streams (1K entries)	203.1 MB	189.6 MB	6.6%

The hash savings are the most impactful because listpack encoding handles the small-hash case better. The hash-max-listpack-entries default is 128 in both, but Valkey's listpack implementation packs entries more tightly.

The active defragmentation is also noticeably better. After 48 hours of sustained writes and deletes on a test instance:

# Check fragmentation ratio
valkey-cli INFO memory | grep mem_fragmentation_ratio
# Redis 7.2 after 48h: mem_fragmentation_ratio:1.43
# Valkey 8.0 after 48h: mem_fragmentation_ratio:1.12

A fragmentation ratio of 1.43 means Redis is using 43% more memory than the actual data requires. Valkey kept it at 12%. On a 14TB dataset, that is the difference between needing 20TB of RAM and needing 15.7TB. At cloud prices, that is real money, roughly $2,400/month.

Client Library State

This matters more than most articles admit. The client library ecosystem determines whether migration is trivial or painful.

Language	Library	Status
Python	valkey-py	Production-ready, fork of redis-py
Node.js	iovalkey	Production-ready, fork of ioredis
Java	Jedis 5.x	Native Valkey support
Go	go-valkey	Production-ready, Valkey-first design
Rust	valkey-rs	Based on redis-rs, works
.NET	StackExchange.Redis	Works with Valkey (protocol-compatible)

Using iovalkey in Node.js, the migration is a package swap and config change:

// Before
import Redis from 'ioredis';
const client = new Redis({
 host: 'redis-primary.internal',
 port: 6379,
 maxRetriesPerRequest: 3,
});

// After
import { Valkey } from 'iovalkey';
const client = new Valkey({
 host: 'valkey-primary.internal',
 port: 6379,
 maxRetriesPerRequest: 3,
 // Enable RESP3 for typed responses
 protocol: 3,
});

Should You Migrate?

Here is an honest assessment after eight months of production use:

Migrate if: You're running Redis OSS, you want multi-threaded I/O for high-connection workloads, the SSPL license concerns you, or you want better memory efficiency on hash-heavy workloads.

Don't migrate if: You depend on Redis Stack modules (Search, JSON, TimeSeries, Graph), you're on Redis Enterprise with SLAs you trust, or your workload is single-connection (you won't see a difference).

Wait if: You're not sure. Valkey isn't going anywhere, the Linux Foundation backing ensures longevity, and the contributor base is growing. You can migrate in a year and the process will only be smoother.

One important point: framing this as a political choice misses the mark. Valkey is technically superior for high-connection workloads, better memory efficiency, better threading, same compatibility. If Redis were still BSD-licensed and had these features, the recommendation would be Redis. But it does not, and Valkey does. That is the end of the analysis.