{"schemaVersion":1,"id":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle","title":"Deterministic Simulation Testing: Fault Injection Methodology in TigerBeetle and FoundationDB","description":"Seed-based fault injection, deterministic scheduling, and the methodology behind finding impossible bugs in distributed systems. Real techniques from TigerBeetle and FoundationDB's testing frameworks.","language":"en","datePublished":"2025-11-29","dateModified":"2026-11-10","contentFormat":"text/markdown","content":"Testing\n\n# Deterministic Simulation Testing: Fault Injection Methodology in TigerBeetle and FoundationDB\n\n2025-11-29Updated: 2026-11-108 min read\n\n[Lucio Durán](<https://lucioduran.com/>)\nC.T.O. at LOi\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/deterministic-simulation-testing-tigerbeetle>)\n\n## What Makes Testing Distributed Systems So Hard\n\nThe fundamental problem is non-determinism. A distributed system has multiple sources of it:\n\n1. **Thread scheduling**: The OS decides which thread runs when\n2. **Network timing**: Packets arrive in unpredictable order with variable latency\n3. **Disk I/O timing**: Reads and writes complete at variable speeds\n4. **Clock skew**: System clocks on different machines drift independently\n5. **Failure timing**: Crashes, network partitions, and disk errors happen at arbitrary moments\n\nA bug that requires a specific combination of these non-deterministic events might take billions of test runs to hit with random testing. And even if you hit it, you can't reproduce it because you don't control the OS scheduler, the network, or the disk.\n\nTraditional integration tests try to handle this with sleeps and retries:\n\n```python\n# This is how most people test distributed systems (don't do this)\ndef test_leader_election():\n cluster.start(3)\n time.sleep(5) # \"wait for election to complete\" 🤞\n leader = cluster.get_leader()\n assert leader is not None\n\n cluster.kill(leader)\n time.sleep(10) # \"wait for re-election\" 🤞🤞\n new_leader = cluster.get_leader()\n assert new_leader is not None\n assert new_leader != leader\n```\n\n## The FoundationDB Methodology\n\n[FoundationDB](<https://www.foundationdb.org/>) pioneered simulation testing in production databases. Their key insight: if you control all sources of non-determinism, you can explore the entire state space systematically.\n\nThe architecture has three pillars:\n\n### 1. Single-Threaded, Event-Driven Core\n\nThe entire database runs in a single thread (or multiple deterministic threads with explicit scheduling). There is no OS-level concurrency. All asynchronous operations are modeled as events in a priority queue:\n\n```\nEvent Queue (ordered by simulated time):\n t=100ms: Node A receives AppendEntries from Node B\n t=102ms: Node C election timeout fires\n t=103ms: Disk write on Node A completes\n t=105ms: Node B heartbeat timer fires\n t=108ms: Network partition begins (injected fault)\n```\n\nThe simulator pops events in order, executes them, and any resulting events go back into the queue. Because the queue is deterministic (same seed → same initial state → same event order), the execution is perfectly reproducible.\n\n### 2. Abstracted I/O Layer\n\nEvery interaction with the outside world goes through an interface:\n\n```cpp\n// FoundationDB's flow runtime abstracts everything\nclass INetwork {\n virtual Future<Void> connect(NetworkAddress addr) = 0;\n virtual Future<Void> send(Connection conn, Bytes data) = 0;\n virtual Future<Bytes> receive(Connection conn) = 0;\n};\n\nclass IDisk {\n virtual Future<Void> write(FileHandle fh, Offset off, Bytes data) = 0;\n virtual Future<Bytes> read(FileHandle fh, Offset off, Size len) = 0;\n virtual Future<Void> sync(FileHandle fh) = 0;\n};\n\nclass IClock {\n virtual double now() = 0;\n virtual Future<Void> delay(double seconds) = 0;\n};\n```\n\nIn production, these call real syscalls. In simulation, they're replaced with fakes that introduce controlled delays, failures, and reorderings.\n\n### 3. Seed-Based Fault Injection\n\nGiven a random seed, the simulator decides:\n\n- When to fire timers (with jitter)\n- When to deliver or drop network packets\n- When to inject disk errors\n- When to simulate process crashes and restarts\n- When to create and heal network partitions\n\n```python\n# Pseudocode for the simulation fault injector\nclass FaultInjector:\n def __init__(self, seed):\n self.rng = Random(seed)\n\n def should_drop_packet(self) -> bool:\n return self.rng.random() < 0.01 # 1% packet loss\n\n def network_delay_ms(self) -> float:\n return self.rng.exponential(scale=5.0) # avg 5ms delay\n\n def should_inject_partition(self, time) -> bool:\n # Partitions happen ~once per simulated hour\n return self.rng.random() < (1.0 / 3600000.0) * time_step_ms\n\n def should_crash_node(self, node_id) -> bool:\n return self.rng.random() < 0.0001 # rare but devastating\n\n def disk_write_should_fail(self) -> bool:\n return self.rng.random() < 0.001 # occasional disk errors\n```\n\n## TigerBeetle's Approach: VOPR\n\n[TigerBeetle](<https://tigerbeetle.com/>) (a financial transactions database written in Zig) took FoundationDB's ideas and built VOPR, the Viewstamped Operation Replication simulator. What makes TigerBeetle's approach distinctive is how deeply simulation is integrated into the codebase.\n\nIn TigerBeetle, the I/O abstraction isn't an afterthought, it's the foundation. Every syscall goes through ` IO `, which has two implementations:\n\n```zig\n// Production I/O, real io_uring syscalls\nconst IO = if (is_simulation)\n @import(\"io_simulation.zig\").IO\nelse\n @import(\"io_uring.zig\").IO;\n```\n\nThe simulation IO replaces io\\_uring with a deterministic event loop:\n\n```zig\n// Simplified simulation I/O\npub const IO = struct {\n prng: std.rand.DefaultPrng,\n event_queue: PriorityQueue(Event),\n tick: u64,\n\n pub fn submit_write(self: *IO, buffer: []const u8, offset: u64) Completion {\n // In simulation, writes complete after a random delay\n const delay = self.prng.random_int(u64) % 10 + 1; // 1-10 ticks\n const completion = Completion{\n .result = if (self.should_inject_fault())\n error.DiskError\n else\n .success,\n };\n self.event_queue.insert(.{\n .tick = self.tick + delay,\n .completion = completion,\n });\n return completion;\n }\n\n fn should_inject_fault(self: *IO) bool {\n return self.prng.random_int(u64) % 1000 == 0; // 0.1% failure rate\n }\n};\n```\n\nThe VOPR runner executes the entire TigerBeetle cluster, multiple replicas, clients, and the network, in a single process:\n\n```zig\n// VOPR runs a full cluster simulation\npub fn run(seed: u64) !void {\n var prng = std.rand.DefaultPrng.init(seed);\n\n // Create simulated cluster\n var cluster = Cluster.init(&prng, .{\n .replica_count = 6,\n .standby_count = 2,\n .client_count = 3,\n });\n\n // Run simulation for N ticks\n var tick: u64 = 0;\n while (tick < 10_000_000) : (tick += 1) {\n // Maybe inject a fault\n if (prng.random_int(u64) % 100 == 0) {\n const fault = random_fault(&prng);\n cluster.inject_fault(fault);\n }\n\n // Advance all components by one tick\n cluster.tick();\n\n // Check invariants after every tick\n cluster.check_invariants();\n }\n}\n```\n\nThe ` check_invariants() ` call is key. After every single tick of the simulation, not just at the end, TigerBeetle verifies that all safety properties hold: no double-spending, no lost transactions, no committed state divergence. This is what catches the subtle interleaving bugs that only manifest mid-execution.\n\n## Building Your Own Simulation Harness\n\nThis methodology can be applied to smaller-scale projects as well. Here is the practical architecture:\n\n### Step 1: Define Your Determinism Boundary\n\nList every source of non-determinism in your system:\n\n```\nSources of non-determinism:\n ✅ System clock → wrap with injectable Clock interface\n ✅ Random numbers → use seeded PRNG everywhere\n ✅ Network I/O → wrap with injectable Network interface\n ✅ Disk I/O → wrap with injectable Storage interface\n ✅ Thread scheduling → use single-threaded event loop in simulation\n ❌ Memory allocation order → usually deterministic, but watch out for hash maps\n ❌ Signal handling → disable in simulation\n```\n\n### Step 2: Build the Abstract I/O Layer\n\n```go\n// Go example: abstract I/O for a Raft implementation\ntype Clock interface {\n Now() time.Time\n After(d time.Duration) <-chan time.Time\n}\n\ntype Network interface {\n Send(to NodeID, msg Message) error\n Receive() <-chan Message\n}\n\ntype Storage interface {\n Append(entries []LogEntry) error\n ReadRange(start, end uint64) ([]LogEntry, error)\n Sync() error\n}\n\n// Production implementations use real I/O\ntype RealClock struct{}\nfunc (c *RealClock) Now() time.Time { return time.Now() }\n\n// Simulation implementations are deterministic\ntype SimClock struct {\n current time.Time\n}\nfunc (c *SimClock) Now() time.Time { return c.current }\nfunc (c *SimClock) Advance(d time.Duration) { c.current = c.current.Add(d) }\n```\n\n### Step 3: The Simulation Runner\n\n```go\ntype Simulator struct {\n seed int64\n rng *rand.Rand\n clock *SimClock\n network *SimNetwork\n nodes []*RaftNode\n events *PriorityQueue\n}\n\nfunc (s *Simulator) Run(ticks int) error {\n for i := 0; i < ticks; i++ {\n // Advance clock\n s.clock.Advance(time.Millisecond)\n\n // Maybe inject faults\n s.maybeInjectFault()\n\n // Deliver pending network messages (with possible reordering)\n s.network.DeliverPending(s.rng)\n\n // Tick all nodes\n for _, node := range s.nodes {\n node.Tick()\n }\n\n // Check invariants\n if err := s.checkInvariants(); err != nil {\n return fmt.Errorf(\"invariant violated at tick %d (seed %d): %w\",\n i, s.seed, err)\n }\n }\n return nil\n}\n\nfunc (s *Simulator) checkInvariants() error {\n // All committed entries must be identical across nodes\n committed := s.nodes[0].CommittedLog()\n for _, node := range s.nodes[1:] {\n nodeCommitted := node.CommittedLog()\n minLen := min(len(committed), len(nodeCommitted))\n for j := 0; j < minLen; j++ {\n if committed[j] != nodeCommitted[j] {\n return fmt.Errorf(\"log divergence at index %d: node0=%v node%d=%v\",\n j, committed[j], node.ID, nodeCommitted[j])\n }\n }\n }\n return nil\n}\n```\n\n### Step 4: CI Integration\n\n```yaml\n# Run 500 seeds on every PR, different range per CI shard\nsimulation-test:\n strategy:\n matrix:\n seed-range: [\"0-100\", \"100-200\", \"200-300\", \"300-400\", \"400-500\"]\n steps:\n - run: |\n START=$(echo ${{ matrix.seed-range }} | cut -d- -f1)\n END=$(echo ${{ matrix.seed-range }} | cut -d- -f2)\n for seed in $(seq $START $END); do\n echo \"Running simulation with seed $seed\"\n ./bin/simulation-test --seed=$seed --ticks=1000000\n done\n```\n\n## Bugs This Methodology Has Found\n\nIn practice, simulation testing consistently finds bugs in these categories:\n\n**Stale leader writes**: A node believes it's the leader but was deposed. It accepts a write that gets lost when the real leader's log overwrites it. Only happens when the deposition message is delayed by exactly the right amount.\n\n**Split-brain during partition healing**: When a network partition heals, two sub-clusters have divergent state. The reconciliation protocol has a window where both think they have quorum. Simulation catches this by healing partitions at adversarial moments.\n\n**Disk write reordering**: On real hardware, ` write(A); write(B); ` doesn't guarantee A is persisted before B. Simulation injects write reordering to find assumptions about persistence order.\n\n**Timer race conditions**: When two timers fire at the same simulated instant, the order matters. Simulation explores both orderings for each seed.\n\nA simulation harness can find complex interleaving bugs in seconds that would take weeks to discover through manual debugging. The contrast between weeks of manual investigation and seconds of simulation execution underscores the methodology's value.\n\nFor systems that need to be correct under failure, databases, consensus protocols, transaction processors, distributed locks, deterministic simulation testing is not optional. It is the only methodology that systematically finds the bugs that matter most: the ones that corrupt data.\n\n> Editorial update (November 10, 2026): unsupported experience claims and numerical outcomes were removed. Examples are instructional.\n\n## Frequently asked questions\n\n### How is deterministic simulation testing different from chaos engineering?\n\nChaos engineering (Chaos Monkey, Litmus, etc.) injects faults in real deployed systems and observes outcomes. It's non-deterministic, you can't reproduce a failure exactly. Simulation testing runs the entire system in a single process with a fake scheduler, fake network, and fake disk, controlled by a seed. Same seed = same execution = same bug, every time.\n\n### Can I apply simulation testing to an existing codebase?\n\nIt's extremely difficult to retrofit. Simulation testing requires that ALL sources of non-determinism (system time, threading, I/O, random numbers) are abstracted behind injectable interfaces. Most existing codebases have non-determinism baked into every layer. It's best adopted as a foundational design constraint for new systems.\n\n## Tips\n\n1. **Tip:** Start by making your I/O layer injectable. Wrap every syscall, ` read() `, ` write() `, ` clock_gettime() `, ` recv() `, behind an interface. In simulation mode, replace them with deterministic fakes. This is the single most important architectural decision for testability.\n2. **Tip:** When you find a failing seed, **never** fix the bug before writing a targeted regression test that reproduces it with that exact seed. Seeds are ephemeral if your simulation code changes, lock down the reproduction first, then fix.\n\n- deterministic-simulation\n- testing\n- distributed-systems\n- tigerbeetle\n- foundationdb\n- fault-injection\n- correctness\n- zig\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"Deterministic Simulation Testing: Fault Injection Methodology in TigerBeetle and FoundationDB","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#title"},{"level":2,"text":"What Makes Testing Distributed Systems So Hard","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#what-makes-testing-distributed-systems-so-hard"},{"level":2,"text":"The FoundationDB Methodology","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#the-foundationdb-methodology"},{"level":3,"text":"1. Single-Threaded, Event-Driven Core","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#1-single-threaded-event-driven-core"},{"level":3,"text":"2. Abstracted I/O Layer","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#2-abstracted-io-layer"},{"level":3,"text":"3. Seed-Based Fault Injection","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#3-seed-based-fault-injection"},{"level":2,"text":"TigerBeetle's Approach: VOPR","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#tigerbeetles-approach-vopr"},{"level":2,"text":"Building Your Own Simulation Harness","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#building-your-own-simulation-harness"},{"level":3,"text":"Step 1: Define Your Determinism Boundary","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#step-1-define-your-determinism-boundary"},{"level":3,"text":"Step 2: Build the Abstract I/O Layer","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#step-2-build-the-abstract-io-layer"},{"level":3,"text":"Step 3: The Simulation Runner","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#step-3-the-simulation-runner"},{"level":3,"text":"Step 4: CI Integration","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#step-4-ci-integration"},{"level":2,"text":"Bugs This Methodology Has Found","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#bugs-this-methodology-has-found"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#questions"},{"level":3,"text":"How is deterministic simulation testing different from chaos engineering?","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#question-1"},{"level":3,"text":"Can I apply simulation testing to an existing codebase?","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#question-2"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#article-tips"}],"sections":[{"heading":"Deterministic Simulation Testing: Fault Injection Methodology in TigerBeetle and FoundationDB","level":1,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle","content":"Testing"},{"heading":"Deterministic Simulation Testing: Fault Injection Methodology in TigerBeetle and FoundationDB","level":1,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#title","content":"2025-11-29Updated: 2026-11-108 min read\n\n[Lucio Durán](<https://lucioduran.com/>)\nC.T.O. at LOi\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/deterministic-simulation-testing-tigerbeetle>)"},{"heading":"What Makes Testing Distributed Systems So Hard","level":2,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#what-makes-testing-distributed-systems-so-hard","content":"The fundamental problem is non-determinism. A distributed system has multiple sources of it:\n\n1. **Thread scheduling**: The OS decides which thread runs when\n2. **Network timing**: Packets arrive in unpredictable order with variable latency\n3. **Disk I/O timing**: Reads and writes complete at variable speeds\n4. **Clock skew**: System clocks on different machines drift independently\n5. **Failure timing**: Crashes, network partitions, and disk errors happen at arbitrary moments\n\nA bug that requires a specific combination of these non-deterministic events might take billions of test runs to hit with random testing. And even if you hit it, you can't reproduce it because you don't control the OS scheduler, the network, or the disk.\n\nTraditional integration tests try to handle this with sleeps and retries:\n\n```python\n# This is how most people test distributed systems (don't do this)\ndef test_leader_election():\n cluster.start(3)\n time.sleep(5) # \"wait for election to complete\" 🤞\n leader = cluster.get_leader()\n assert leader is not None\n\n cluster.kill(leader)\n time.sleep(10) # \"wait for re-election\" 🤞🤞\n new_leader = cluster.get_leader()\n assert new_leader is not None\n assert new_leader != leader\n```"},{"heading":"The FoundationDB Methodology","level":2,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#the-foundationdb-methodology","content":"[FoundationDB](<https://www.foundationdb.org/>) pioneered simulation testing in production databases. Their key insight: if you control all sources of non-determinism, you can explore the entire state space systematically.\n\nThe architecture has three pillars:"},{"heading":"1. Single-Threaded, Event-Driven Core","level":3,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#1-single-threaded-event-driven-core","content":"The entire database runs in a single thread (or multiple deterministic threads with explicit scheduling). There is no OS-level concurrency. All asynchronous operations are modeled as events in a priority queue:\n\n```\nEvent Queue (ordered by simulated time):\n t=100ms: Node A receives AppendEntries from Node B\n t=102ms: Node C election timeout fires\n t=103ms: Disk write on Node A completes\n t=105ms: Node B heartbeat timer fires\n t=108ms: Network partition begins (injected fault)\n```\n\nThe simulator pops events in order, executes them, and any resulting events go back into the queue. Because the queue is deterministic (same seed → same initial state → same event order), the execution is perfectly reproducible."},{"heading":"2. Abstracted I/O Layer","level":3,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#2-abstracted-io-layer","content":"Every interaction with the outside world goes through an interface:\n\n```cpp\n// FoundationDB's flow runtime abstracts everything\nclass INetwork {\n virtual Future<Void> connect(NetworkAddress addr) = 0;\n virtual Future<Void> send(Connection conn, Bytes data) = 0;\n virtual Future<Bytes> receive(Connection conn) = 0;\n};\n\nclass IDisk {\n virtual Future<Void> write(FileHandle fh, Offset off, Bytes data) = 0;\n virtual Future<Bytes> read(FileHandle fh, Offset off, Size len) = 0;\n virtual Future<Void> sync(FileHandle fh) = 0;\n};\n\nclass IClock {\n virtual double now() = 0;\n virtual Future<Void> delay(double seconds) = 0;\n};\n```\n\nIn production, these call real syscalls. In simulation, they're replaced with fakes that introduce controlled delays, failures, and reorderings."},{"heading":"3. Seed-Based Fault Injection","level":3,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#3-seed-based-fault-injection","content":"Given a random seed, the simulator decides:\n\n- When to fire timers (with jitter)\n- When to deliver or drop network packets\n- When to inject disk errors\n- When to simulate process crashes and restarts\n- When to create and heal network partitions\n\n```python\n# Pseudocode for the simulation fault injector\nclass FaultInjector:\n def __init__(self, seed):\n self.rng = Random(seed)\n\n def should_drop_packet(self) -> bool:\n return self.rng.random() < 0.01 # 1% packet loss\n\n def network_delay_ms(self) -> float:\n return self.rng.exponential(scale=5.0) # avg 5ms delay\n\n def should_inject_partition(self, time) -> bool:\n # Partitions happen ~once per simulated hour\n return self.rng.random() < (1.0 / 3600000.0) * time_step_ms\n\n def should_crash_node(self, node_id) -> bool:\n return self.rng.random() < 0.0001 # rare but devastating\n\n def disk_write_should_fail(self) -> bool:\n return self.rng.random() < 0.001 # occasional disk errors\n```"},{"heading":"TigerBeetle's Approach: VOPR","level":2,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#tigerbeetles-approach-vopr","content":"[TigerBeetle](<https://tigerbeetle.com/>) (a financial transactions database written in Zig) took FoundationDB's ideas and built VOPR, the Viewstamped Operation Replication simulator. What makes TigerBeetle's approach distinctive is how deeply simulation is integrated into the codebase.\n\nIn TigerBeetle, the I/O abstraction isn't an afterthought, it's the foundation. Every syscall goes through ` IO `, which has two implementations:\n\n```zig\n// Production I/O, real io_uring syscalls\nconst IO = if (is_simulation)\n @import(\"io_simulation.zig\").IO\nelse\n @import(\"io_uring.zig\").IO;\n```\n\nThe simulation IO replaces io\\_uring with a deterministic event loop:\n\n```zig\n// Simplified simulation I/O\npub const IO = struct {\n prng: std.rand.DefaultPrng,\n event_queue: PriorityQueue(Event),\n tick: u64,\n\n pub fn submit_write(self: *IO, buffer: []const u8, offset: u64) Completion {\n // In simulation, writes complete after a random delay\n const delay = self.prng.random_int(u64) % 10 + 1; // 1-10 ticks\n const completion = Completion{\n .result = if (self.should_inject_fault())\n error.DiskError\n else\n .success,\n };\n self.event_queue.insert(.{\n .tick = self.tick + delay,\n .completion = completion,\n });\n return completion;\n }\n\n fn should_inject_fault(self: *IO) bool {\n return self.prng.random_int(u64) % 1000 == 0; // 0.1% failure rate\n }\n};\n```\n\nThe VOPR runner executes the entire TigerBeetle cluster, multiple replicas, clients, and the network, in a single process:\n\n```zig\n// VOPR runs a full cluster simulation\npub fn run(seed: u64) !void {\n var prng = std.rand.DefaultPrng.init(seed);\n\n // Create simulated cluster\n var cluster = Cluster.init(&prng, .{\n .replica_count = 6,\n .standby_count = 2,\n .client_count = 3,\n });\n\n // Run simulation for N ticks\n var tick: u64 = 0;\n while (tick < 10_000_000) : (tick += 1) {\n // Maybe inject a fault\n if (prng.random_int(u64) % 100 == 0) {\n const fault = random_fault(&prng);\n cluster.inject_fault(fault);\n }\n\n // Advance all components by one tick\n cluster.tick();\n\n // Check invariants after every tick\n cluster.check_invariants();\n }\n}\n```\n\nThe ` check_invariants() ` call is key. After every single tick of the simulation, not just at the end, TigerBeetle verifies that all safety properties hold: no double-spending, no lost transactions, no committed state divergence. This is what catches the subtle interleaving bugs that only manifest mid-execution."},{"heading":"Building Your Own Simulation Harness","level":2,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#building-your-own-simulation-harness","content":"This methodology can be applied to smaller-scale projects as well. Here is the practical architecture:"},{"heading":"Step 1: Define Your Determinism Boundary","level":3,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#step-1-define-your-determinism-boundary","content":"List every source of non-determinism in your system:\n\n```\nSources of non-determinism:\n ✅ System clock → wrap with injectable Clock interface\n ✅ Random numbers → use seeded PRNG everywhere\n ✅ Network I/O → wrap with injectable Network interface\n ✅ Disk I/O → wrap with injectable Storage interface\n ✅ Thread scheduling → use single-threaded event loop in simulation\n ❌ Memory allocation order → usually deterministic, but watch out for hash maps\n ❌ Signal handling → disable in simulation\n```"},{"heading":"Step 2: Build the Abstract I/O Layer","level":3,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#step-2-build-the-abstract-io-layer","content":"```go\n// Go example: abstract I/O for a Raft implementation\ntype Clock interface {\n Now() time.Time\n After(d time.Duration) <-chan time.Time\n}\n\ntype Network interface {\n Send(to NodeID, msg Message) error\n Receive() <-chan Message\n}\n\ntype Storage interface {\n Append(entries []LogEntry) error\n ReadRange(start, end uint64) ([]LogEntry, error)\n Sync() error\n}\n\n// Production implementations use real I/O\ntype RealClock struct{}\nfunc (c *RealClock) Now() time.Time { return time.Now() }\n\n// Simulation implementations are deterministic\ntype SimClock struct {\n current time.Time\n}\nfunc (c *SimClock) Now() time.Time { return c.current }\nfunc (c *SimClock) Advance(d time.Duration) { c.current = c.current.Add(d) }\n```"},{"heading":"Step 3: The Simulation Runner","level":3,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#step-3-the-simulation-runner","content":"```go\ntype Simulator struct {\n seed int64\n rng *rand.Rand\n clock *SimClock\n network *SimNetwork\n nodes []*RaftNode\n events *PriorityQueue\n}\n\nfunc (s *Simulator) Run(ticks int) error {\n for i := 0; i < ticks; i++ {\n // Advance clock\n s.clock.Advance(time.Millisecond)\n\n // Maybe inject faults\n s.maybeInjectFault()\n\n // Deliver pending network messages (with possible reordering)\n s.network.DeliverPending(s.rng)\n\n // Tick all nodes\n for _, node := range s.nodes {\n node.Tick()\n }\n\n // Check invariants\n if err := s.checkInvariants(); err != nil {\n return fmt.Errorf(\"invariant violated at tick %d (seed %d): %w\",\n i, s.seed, err)\n }\n }\n return nil\n}\n\nfunc (s *Simulator) checkInvariants() error {\n // All committed entries must be identical across nodes\n committed := s.nodes[0].CommittedLog()\n for _, node := range s.nodes[1:] {\n nodeCommitted := node.CommittedLog()\n minLen := min(len(committed), len(nodeCommitted))\n for j := 0; j < minLen; j++ {\n if committed[j] != nodeCommitted[j] {\n return fmt.Errorf(\"log divergence at index %d: node0=%v node%d=%v\",\n j, committed[j], node.ID, nodeCommitted[j])\n }\n }\n }\n return nil\n}\n```"},{"heading":"Step 4: CI Integration","level":3,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#step-4-ci-integration","content":"```yaml\n# Run 500 seeds on every PR, different range per CI shard\nsimulation-test:\n strategy:\n matrix:\n seed-range: [\"0-100\", \"100-200\", \"200-300\", \"300-400\", \"400-500\"]\n steps:\n - run: |\n START=$(echo ${{ matrix.seed-range }} | cut -d- -f1)\n END=$(echo ${{ matrix.seed-range }} | cut -d- -f2)\n for seed in $(seq $START $END); do\n echo \"Running simulation with seed $seed\"\n ./bin/simulation-test --seed=$seed --ticks=1000000\n done\n```"},{"heading":"Bugs This Methodology Has Found","level":2,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#bugs-this-methodology-has-found","content":"In practice, simulation testing consistently finds bugs in these categories:\n\n**Stale leader writes**: A node believes it's the leader but was deposed. It accepts a write that gets lost when the real leader's log overwrites it. Only happens when the deposition message is delayed by exactly the right amount.\n\n**Split-brain during partition healing**: When a network partition heals, two sub-clusters have divergent state. The reconciliation protocol has a window where both think they have quorum. Simulation catches this by healing partitions at adversarial moments.\n\n**Disk write reordering**: On real hardware, ` write(A); write(B); ` doesn't guarantee A is persisted before B. Simulation injects write reordering to find assumptions about persistence order.\n\n**Timer race conditions**: When two timers fire at the same simulated instant, the order matters. Simulation explores both orderings for each seed.\n\nA simulation harness can find complex interleaving bugs in seconds that would take weeks to discover through manual debugging. The contrast between weeks of manual investigation and seconds of simulation execution underscores the methodology's value.\n\nFor systems that need to be correct under failure, databases, consensus protocols, transaction processors, distributed locks, deterministic simulation testing is not optional. It is the only methodology that systematically finds the bugs that matter most: the ones that corrupt data.\n\n> Editorial update (November 10, 2026): unsupported experience claims and numerical outcomes were removed. Examples are instructional."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#questions","content":""},{"heading":"How is deterministic simulation testing different from chaos engineering?","level":3,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#question-1","content":"Chaos engineering (Chaos Monkey, Litmus, etc.) injects faults in real deployed systems and observes outcomes. It's non-deterministic, you can't reproduce a failure exactly. Simulation testing runs the entire system in a single process with a fake scheduler, fake network, and fake disk, controlled by a seed. Same seed = same execution = same bug, every time."},{"heading":"Can I apply simulation testing to an existing codebase?","level":3,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#question-2","content":"It's extremely difficult to retrofit. Simulation testing requires that ALL sources of non-determinism (system time, threading, I/O, random numbers) are abstracted behind injectable interfaces. Most existing codebases have non-determinism baked into every layer. It's best adopted as a foundational design constraint for new systems."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#article-tips","content":"1. **Tip:** Start by making your I/O layer injectable. Wrap every syscall, ` read() `, ` write() `, ` clock_gettime() `, ` recv() `, behind an interface. In simulation mode, replace them with deterministic fakes. This is the single most important architectural decision for testability.\n2. **Tip:** When you find a failing seed, **never** fix the bug before writing a targeted regression test that reproduces it with that exact seed. Seeds are ephemeral if your simulation code changes, lock down the reproduction first, then fix.\n\n- deterministic-simulation\n- testing\n- distributed-systems\n- tigerbeetle\n- foundationdb\n- fault-injection\n- correctness\n- zig\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Lucio Durán","url":"https://lucioduran.com/"},{"text":"Español","url":"https://lucioduran.com/es/blog/deterministic-simulation-testing-tigerbeetle"},{"text":"FoundationDB","url":"https://www.foundationdb.org/"},{"text":"TigerBeetle","url":"https://tigerbeetle.com/"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com/"},{"@type":"WebSite","@id":"https://lucioduran.com/#website","url":"https://lucioduran.com/","name":"Lucio Durán","inLanguage":["en","es"],"publisher":{"@id":"https://lucioduran.com/#person"}},{"@type":"WebPage","@id":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#page","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle","name":"Deterministic Simulation Testing: Fault Injection Methodology in TigerBeetle and FoundationDB | Lucio Durán","description":"Seed-based fault injection, deterministic scheduling, and the methodology behind finding impossible bugs in distributed systems. Real techniques from TigerBeetle and FoundationDB's testing frameworks.","inLanguage":"en","author":{"@id":"https://lucioduran.com/#person"},"publisher":{"@id":"https://lucioduran.com/#person"},"isPartOf":{"@id":"https://lucioduran.com/#website"},"isAccessibleForFree":true,"datePublished":"2025-11-29","dateModified":"2026-11-10","mainEntity":{"@id":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#article"},"breadcrumb":{"@id":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#breadcrumb"}},{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#article","mainEntityOfPage":{"@id":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#page"},"headline":"Deterministic Simulation Testing: Fault Injection Methodology in TigerBeetle and FoundationDB","description":"Seed-based fault injection, deterministic scheduling, and the methodology behind finding impossible bugs in distributed systems. Real techniques from TigerBeetle and FoundationDB's testing frameworks.","datePublished":"2025-11-29","dateModified":"2026-11-10","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@id":"https://lucioduran.com/#person"},"url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Testing","keywords":"deterministic-simulation, testing, distributed-systems, tigerbeetle, foundationdb, fault-injection, correctness, zig","wordCount":1532,"timeRequired":"PT8M","workTranslation":[{"@type":"BlogPosting","@id":"https://lucioduran.com/es/blog/deterministic-simulation-testing-tigerbeetle#article","inLanguage":"es","url":"https://lucioduran.com/es/blog/deterministic-simulation-testing-tigerbeetle"}],"isAccessibleForFree":true},{"@type":"Blog","@id":"https://lucioduran.com/blog#blog","url":"https://lucioduran.com/blog","name":"Blog","inLanguage":"en","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com/"},"publisher":{"@id":"https://lucioduran.com/#person"},"isPartOf":{"@id":"https://lucioduran.com/#website"}},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"Deterministic Simulation Testing: Fault Injection Methodology in TigerBeetle and FoundationDB","item":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle"}],"@id":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#breadcrumb"},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"How is deterministic simulation testing different from chaos engineering?","acceptedAnswer":{"@type":"Answer","text":"Chaos engineering (Chaos Monkey, Litmus, etc.) injects faults in real deployed systems and observes outcomes. It's non-deterministic, you can't reproduce a failure exactly. Simulation testing runs the entire system in a single process with a fake scheduler, fake network, and fake disk, controlled by a seed. Same seed = same execution = same bug, every time."}},{"@type":"Question","name":"Can I apply simulation testing to an existing codebase?","acceptedAnswer":{"@type":"Answer","text":"It's extremely difficult to retrofit. Simulation testing requires that ALL sources of non-determinism (system time, threading, I/O, random numbers) are abstracted behind injectable interfaces. Most existing codebases have non-determinism baked into every layer. It's best adopted as a foundational design constraint for new systems."}}],"@id":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#questions","url":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#questions","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle#page"},"author":{"@id":"https://lucioduran.com/#person"}}]}],"contentHash":"50225aca6f260cf532c42210f8ee66ddf37b5f247938107b39b1bda043f5734b","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/deterministic-simulation-testing-tigerbeetle","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":72809,"contentBytes":13146}}