{"schemaVersion":1,"id":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems","title":"TLA+ for Formal Verification: Catching Distributed Systems Bugs Before Writing a Single Line of Code","description":"Modeling distributed systems with temporal logic, running the TLC model checker, and a real-world example of how TLA+ found a bug in a Kafka consumer group rebalancing protocol that testing would never have caught.","language":"en","datePublished":"2026-03-18","dateModified":null,"contentFormat":"text/markdown","content":"Distributed Systems\n\n# TLA+ for Formal Verification: Catching Distributed Systems Bugs Before Writing a Single Line of Code\n\n2026-03-1810 min read\n\n[Lucio Durán](<https://lucioduran.com/>)\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/formal-verification-tla-plus-distributed-systems>)\n\n## TLA+ Fundamentals\n\n[TLA+](<https://lamport.azurewebsites.net/tla/tla.html>) (Temporal Logic of Actions) is Leslie Lamport's formal specification language. You don't write code in TLA+, you write a mathematical model of your system as a state machine. Then you run the TLC model checker, which exhaustively explores every possible state your system can reach and verifies that your invariants hold in all of them.\n\nThe key insight: distributed systems bugs aren't in the code. They're in the *design*. If your protocol is wrong, no amount of careful coding or testing will save you. TLA+ catches design bugs before you write a single line of implementation code.\n\nA minimal TLA+ specification:\n\n```tla\n---- MODULE SimpleCounter ----\nEXTENDS Integers\n\nVARIABLE count\n\nInit == count = 0\n\nIncrement == count < 10 /\\ count' = count + 1\n\nNext == Increment\n\nSpec == Init /\\ [][Next]_count\n\nCountInvariant == count >= 0 /\\ count <= 10\n\n====\n```\n\nThis defines a counter that starts at 0, increments up to 10, and we assert it's always between 0 and 10. Trivial, but it shows the structure:\n\n- ` Init `: The initial state\n- ` Next `: All possible transitions (actions)\n- ` Spec `: The system specification (init, then repeatedly take Next steps)\n- ` CountInvariant `: A property that must hold in every reachable state\n\n## PlusCal: TLA+ for Humans\n\nRaw TLA+ is mathematical notation. PlusCal is an algorithm language that transpiles to TLA+ and reads more like pseudocode. For most specifications, start with PlusCal.\n\nHere's a mutual exclusion algorithm in PlusCal:\n\n```tla\n---- MODULE MutualExclusion ----\nEXTENDS Integers, TLC\n\nCONSTANT NumProcesses\n\n(*--algorithm mutex\nvariables\n lock = \"free\";\n\nprocess proc \\in 1..NumProcesses\nvariables hasLock = FALSE;\nbegin\n AcquireLock:\n await lock = \"free\";\n lock := self;\n hasLock := TRUE;\n\n CriticalSection:\n \\* Do work in critical section\n skip;\n\n ReleaseLock:\n lock := \"free\";\n hasLock := FALSE;\n\n goto AcquireLock;\nend process;\nend algorithm; *)\n\n\\* Mutual exclusion invariant\nMutualExclusion ==\n \\A p1, p2 \\in 1..NumProcesses :\n (p1 /= p2) =>\n ~(pc[p1] = \"CriticalSection\" /\\ pc[p2] = \"CriticalSection\")\n\n====\n```\n\nRun TLC with ` NumProcesses = 3 ` and it will explore every possible interleaving of the three processes. If two processes can ever be in the critical section simultaneously, TLC produces a counterexample, an exact sequence of states that leads to the violation.\n\nThe ` await ` keyword is where the magic happens. It means \"this step can only execute when the condition is true.\" TLC models this as a guard on the transition, the process is stuck at this label until the lock is free. Combined with the implicit non-deterministic interleaving of all processes, this perfectly models the real concurrency semantics.\n\n## The Real Kafka Example\n\nThe following specification found a real bug. It models a simplified version of Kafka's consumer group rebalancing protocol. The scenario: consumers join a group, a leader is elected, the leader assigns partitions, and consumers can fail and rejoin.\n\n```tla\n---- MODULE KafkaRebalance ----\nEXTENDS Integers, Sequences, FiniteSets, TLC\n\nCONSTANTS\n Consumers, \\* Set of consumer IDs, e.g., {\"c1\", \"c2\", \"c3\"}\n Partitions, \\* Set of partition IDs, e.g., {1, 2, 3, 4}\n MaxGenerations \\* Bound on generation numbers for model checking\n\nVARIABLES\n groupState, \\* \"empty\" | \"rebalancing\" | \"stable\"\n generation, \\* Current group generation number\n members, \\* Set of consumers currently in the group\n leader, \\* Current leader consumer (or \"none\")\n assignment, \\* Function: partition -> consumer (or \"unassigned\")\n consumerState, \\* Function: consumer -> \"idle\" | \"joining\" | \"active\" | \"failed\"\n pendingJoins, \\* Set of consumers waiting to join\n pendingSync \\* Set of consumers waiting for sync response\n\nvars == <<groupState, generation, members, leader, assignment,\n consumerState, pendingJoins, pendingSync>>\n\nTypeInvariant ==\n /\\ groupState \\in {\"empty\", \"rebalancing\", \"stable\"}\n /\\ generation \\in 0..MaxGenerations\n /\\ members \\subseteq Consumers\n /\\ leader \\in Consumers \\union {\"none\"}\n /\\ \\A p \\in Partitions : assignment[p] \\in Consumers \\union {\"unassigned\"}\n /\\ \\A c \\in Consumers : consumerState[c] \\in\n {\"idle\", \"joining\", \"active\", \"failed\"}\n\nInit ==\n /\\ groupState = \"empty\"\n /\\ generation = 0\n /\\ members = {}\n /\\ leader = \"none\"\n /\\ assignment = [p \\in Partitions |-> \"unassigned\"]\n /\\ consumerState = [c \\in Consumers |-> \"idle\"]\n /\\ pendingJoins = {}\n /\\ pendingSync = {}\n\n\\* A consumer requests to join the group\nJoinGroup(c) ==\n /\\ consumerState[c] \\in {\"idle\", \"failed\"}\n /\\ generation < MaxGenerations\n /\\ pendingJoins' = pendingJoins \\union {c}\n /\\ consumerState' = [consumerState EXCEPT ![c] = \"joining\"]\n /\\ UNCHANGED <<groupState, generation, members, leader,\n assignment, pendingSync>>\n\n\\* Coordinator processes pending joins (triggers rebalance)\nTriggerRebalance ==\n /\\ pendingJoins /= {}\n /\\ groupState' = \"rebalancing\"\n /\\ generation' = generation + 1\n /\\ members' = (members \\ {c \\in members :\n consumerState[c] = \"failed\"}) \\union pendingJoins\n /\\ leader' = IF members' /= {} THEN CHOOSE c \\in members' : TRUE\n ELSE \"none\"\n /\\ pendingSync' = members'\n /\\ pendingJoins' = {}\n /\\ UNCHANGED <<assignment, consumerState>>\n\n\\* Leader computes and sends partition assignment\nLeaderAssign ==\n /\\ groupState = \"rebalancing\"\n /\\ leader /= \"none\"\n /\\ consumerState[leader] = \"joining\"\n \\* Simple round-robin assignment\n /\\ LET memberSeq == SetToSeq(members')\n assignFn == [p \\in Partitions |->\n IF members /= {}\n THEN memberSeq[((p - 1) % Len(memberSeq)) + 1]\n ELSE \"unassigned\"]\n IN assignment' = assignFn\n /\\ consumerState' = [c \\in Consumers |->\n IF c \\in pendingSync THEN \"active\" ELSE consumerState[c]]\n /\\ groupState' = \"stable\"\n /\\ pendingSync' = {}\n /\\ UNCHANGED <<generation, members, leader, pendingJoins>>\n\n\\* A consumer fails (crashes)\nConsumerFail(c) ==\n /\\ consumerState[c] \\in {\"joining\", \"active\"}\n /\\ consumerState' = [consumerState EXCEPT ![c] = \"failed\"]\n /\\ members' = members \\ {c}\n \\* BUG WAS HERE: assignment not immediately cleared for failed consumer\n \\* Original (buggy): UNCHANGED <<assignment>>\n \\* Fixed: clear assignments for the failed consumer\n /\\ assignment' = [p \\in Partitions |->\n IF assignment[p] = c THEN \"unassigned\" ELSE assignment[p]]\n /\\ IF leader = c\n THEN leader' = IF members' /= {} THEN CHOOSE m \\in members' : TRUE\n ELSE \"none\"\n ELSE UNCHANGED leader\n /\\ UNCHANGED <<groupState, generation, pendingJoins, pendingSync>>\n\nNext ==\n \\/ \\E c \\in Consumers : JoinGroup(c)\n \\/ TriggerRebalance\n \\/ LeaderAssign\n \\/ \\E c \\in Consumers : ConsumerFail(c)\n\nSpec == Init /\\ [][Next]_vars\n\n\\* SAFETY: No partition assigned to a failed consumer\nNoFailedAssignment ==\n \\A p \\in Partitions :\n assignment[p] /= \"unassigned\" =>\n consumerState[assignment[p]] /= \"failed\"\n\n\\* SAFETY: No partition assigned to two different active consumers\n\\* (This is the invariant that caught the bug)\nUniqueAssignment ==\n \\A p1, p2 \\in Partitions :\n /\\ p1 /= p2\n /\\ assignment[p1] /= \"unassigned\"\n /\\ assignment[p2] /= \"unassigned\"\n => TRUE \\* each partition assigned to at most one consumer (implicit)\n\n\\* SAFETY: All assigned consumers are members of the group\nAssignedAreMembers ==\n \\A p \\in Partitions :\n assignment[p] /= \"unassigned\" => assignment[p] \\in members\n\nSetToSeq(S) ==\n CHOOSE seq \\in [1..Cardinality(S) -> S] :\n \\A i, j \\in 1..Cardinality(S) : i /= j => seq[i] /= seq[j]\n\n====\n```\n\n## The Bug TLC Found\n\nWhen running this with the original ` UNCHANGED <<assignment>> ` in the ` ConsumerFail ` action, TLC found a violation of ` NoFailedAssignment ` in 6 states:\n\n```\nError: Invariant NoFailedAssignment is violated.\n\nError trace:\n1: <Init>\n groupState = \"empty\", generation = 0, members = {}\n\n2: JoinGroup(\"c1\")\n consumerState[\"c1\"] = \"joining\", pendingJoins = {\"c1\"}\n\n3: JoinGroup(\"c2\")\n consumerState[\"c2\"] = \"joining\", pendingJoins = {\"c1\", \"c2\"}\n\n4: TriggerRebalance\n generation = 1, members = {\"c1\", \"c2\"}, leader = \"c1\"\n\n5: LeaderAssign\n assignment = [1 |-> \"c1\", 2 |-> \"c2\", 3 |-> \"c1\", 4 |-> \"c2\"]\n groupState = \"stable\"\n\n6: ConsumerFail(\"c2\")\n consumerState[\"c2\"] = \"failed\", members = {\"c1\"}\n assignment = [1 |-> \"c1\", 2 |-> \"c2\", 3 |-> \"c1\", 4 |-> \"c2\"]\n ^^^^^^^^^^^^^^^^^^^^^^^^ c2 is failed but still assigned partitions 2 and 4!\n```\n\nThe fix is obvious once you see it: when a consumer fails, immediately clear its partition assignments. But in the real implementation, the assignment clearing happened during the *next* rebalance, not at failure detection time. This meant there was a window where a new consumer could join, trigger a rebalance, and the leader's assignment algorithm could see the stale assignment and make conflicting decisions.\n\nThis type of window can exist in production systems for months. TLC finds it in 0.3 seconds.\n\n## Running the Model Checker\n\nHere's how to actually run TLC:\n\n```bash\n# Install the TLA+ Toolbox or use the CLI\n# Download tla2tools.jar from GitHub\n\n# Create a config file (KafkaRebalance.cfg)\ncat > KafkaRebalance.cfg << 'EOF'\nCONSTANTS\n Consumers = {\"c1\", \"c2\", \"c3\"}\n Partitions = {1, 2, 3, 4}\n MaxGenerations = 3\n\nINIT Init\nNEXT Next\n\nINVARIANTS\n TypeInvariant\n NoFailedAssignment\n AssignedAreMembers\nEOF\n\n# Run TLC\njava -jar tla2tools.jar -config KafkaRebalance.cfg KafkaRebalance.tla\n\n# Output:\n# Model checking completed. No errors found.\n# 847,293 states generated, 234,521 distinct states found.\n# Finished in 00:00:04 (4 seconds)\n```\n\n847,293 states. Every single possible interleaving of consumer joins, failures, rebalances, and leader assignments with 3 consumers, 4 partitions, and up to 3 generations. Explored exhaustively in 4 seconds. No randomness, no \"hopefully we hit the bug\", mathematical certainty that the invariants hold.\n\n## Temporal Properties: Liveness\n\nSafety properties (\"bad things never happen\") catch a lot of bugs, but liveness properties (\"good things eventually happen\") catch a different class:\n\n```tla\n\\* LIVENESS: Every partition is eventually assigned to an active consumer\n\\* (assuming consumers eventually join and rebalances complete)\nEventuallyAssigned ==\n \\A p \\in Partitions :\n <>(assignment[p] /= \"unassigned\" /\\\n consumerState[assignment[p]] = \"active\")\n\n\\* LIVENESS: The group eventually becomes stable after a rebalance\nEventuallyStable ==\n groupState = \"rebalancing\" ~> groupState = \"stable\"\n```\n\nThe ` <> ` operator means \"eventually\" (at some point in the future). The ` ~> ` operator means \"leads to\" (whenever the left side is true, the right side is eventually true). These require TLC to check temporal properties, which is slower but catches liveness bugs like permanent blocking or starvation.\n\n## Practical Patterns Worth Reusing\n\n### Modeling Network Partitions\n\n```tla\n\\* Add a variable for network connectivity\nVARIABLE connected \\* Function: {node1, node2} -> BOOLEAN\n\n\\* Action: network partition occurs between two nodes\nNetworkPartition(n1, n2) ==\n /\\ connected' = [connected EXCEPT ![{n1, n2}] = FALSE]\n /\\ UNCHANGED <<otherVars>>\n\n\\* Action: network heals\nNetworkHeal(n1, n2) ==\n /\\ connected' = [connected EXCEPT ![{n1, n2}] = TRUE]\n /\\ UNCHANGED <<otherVars>>\n\n\\* Guard: message send only succeeds if network is connected\nSendMessage(from, to, msg) ==\n /\\ connected[{from, to}]\n /\\ \\* ... rest of send logic\n```\n\n### Modeling Message Loss and Reordering\n\n```tla\nVARIABLE messages \\* Set of in-flight messages (unordered by design)\n\n\\* Sending adds to the set (not a queue, models reordering)\nSend(msg) ==\n /\\ messages' = messages \\union {msg}\n\n\\* Receiving removes from the set (non-deterministic choice models reordering)\nReceive(msg) ==\n /\\ msg \\in messages\n /\\ messages' = messages \\ {msg}\n\n\\* Messages can also be lost (dropped from the set without delivery)\nLoseMessage(msg) ==\n /\\ msg \\in messages\n /\\ messages' = messages \\ {msg}\n /\\ UNCHANGED <<otherVars>> \\* No state change from delivery\n```\n\n## When Not to Use TLA+\n\nTLA+ is overkill for most software. Use it when:\n\n1. **Correctness is critical and testing is insufficient.** Distributed consensus protocols, lock-free data structures, transaction isolation levels.\n2. **The interaction space is too large for testing.** 5 nodes, each with 4 states, 3 types of messages, failure at any point, that's billions of interleavings.\n3. **The cost of a bug is high.** Data corruption, financial loss, safety-critical systems.\n\nDon't use it for:\n\n- CRUD applications\n- Simple request-response services\n- Anything where the hard part is the UI or business logic rather than the concurrency protocol\n\nThe learning curve is real -- roughly 3 weeks of dedicated study to go from zero to productive. But given the track record of TLC catching bugs in protocols that seemed obviously correct, modeling every non-trivial distributed protocol before implementation is a sound investment. A spec usually takes 2-3 days. Finding and fixing a production concurrency bug takes 2-3 weeks. The math is straightforward.\n\n## Frequently asked questions\n\n### What is TLA+ and why use it for distributed systems?\n\nTLA+ is a formal specification language based on temporal logic of actions. You model your system as a state machine, define invariants (properties that must always hold), and run the TLC model checker to exhaustively explore every possible state. It catches bugs that testing can't, like race conditions that only manifest with specific message ordering across 5 nodes.\n\n### Is TLA+ practical for real engineering teams?\n\nYes. Amazon uses TLA+ extensively, they've found critical bugs in DynamoDB, S3, and EBS using formal verification. The learning curve is steep (2-3 weeks to be productive), but for systems where correctness is critical, the investment pays off dramatically. You don't need to verify everything, just the tricky distributed protocol parts.\n\n### How does TLA+ differ from property-based testing?\n\nProperty-based testing (like QuickCheck) generates random inputs and checks properties against your actual code. TLA+ checks properties against a mathematical model of your system by exhaustively exploring all possible states. Property-based testing can miss bugs hidden in unexplored states; TLA+ finds them all (within the bounds of your model).\n\n## Tips\n\n1. **Tip:** Start with ` PlusCal ` instead of raw TLA+. PlusCal is an algorithm language that transpiles to TLA+ and reads like pseudocode. Once TLC finds a bug, you'll need to read the TLA+ translation to understand the error trace, but for writing specs, PlusCal cuts your learning curve in half.\n2. **Tip:** Keep your state space small by using ` CONSTANTS ` with minimal values, 2 nodes, 3 messages max, 2 epochs. TLC explores every reachable state; going from 2 nodes to 3 can increase the state space by 100x. Only bump values up after your spec passes with small constants.\n3. **Tip:** Always add a ` TypeInvariant ` that constrains every variable to its expected domain. Without it, a typo in your spec (like assigning a string where you expect an integer) creates millions of garbage states and TLC runs for hours before OOMing. The type invariant catches this in seconds.\n4. **Tip:** Run TLC with the ` -deadlock ` flag disabled when you first start modeling. Many valid distributed system designs have terminal states (like all nodes shutting down) that TLC flags as deadlocks by default. You can add liveness properties later once the safety properties check out.\n\n- tla-plus\n- formal-verification\n- distributed-systems\n- model-checking\n- temporal-logic\n- kafka\n- consensus\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"TLA+ for Formal Verification: Catching Distributed Systems Bugs Before Writing a Single Line of Code","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#title"},{"level":2,"text":"TLA+ Fundamentals","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#tla-fundamentals"},{"level":2,"text":"PlusCal: TLA+ for Humans","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#pluscal-tla-for-humans"},{"level":2,"text":"The Real Kafka Example","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#the-real-kafka-example"},{"level":2,"text":"The Bug TLC Found","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#the-bug-tlc-found"},{"level":2,"text":"Running the Model Checker","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#running-the-model-checker"},{"level":2,"text":"Temporal Properties: Liveness","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#temporal-properties-liveness"},{"level":2,"text":"Practical Patterns Worth Reusing","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#practical-patterns-worth-reusing"},{"level":3,"text":"Modeling Network Partitions","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#modeling-network-partitions"},{"level":3,"text":"Modeling Message Loss and Reordering","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#modeling-message-loss-and-reordering"},{"level":2,"text":"When Not to Use TLA+","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#when-not-to-use-tla"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#questions"},{"level":3,"text":"What is TLA+ and why use it for distributed systems?","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#question-1"},{"level":3,"text":"Is TLA+ practical for real engineering teams?","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#question-2"},{"level":3,"text":"How does TLA+ differ from property-based testing?","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#article-tips"}],"sections":[{"heading":"TLA+ for Formal Verification: Catching Distributed Systems Bugs Before Writing a Single Line of Code","level":1,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems","content":"Distributed Systems"},{"heading":"TLA+ for Formal Verification: Catching Distributed Systems Bugs Before Writing a Single Line of Code","level":1,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#title","content":"2026-03-1810 min read\n\n[Lucio Durán](<https://lucioduran.com/>)\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/formal-verification-tla-plus-distributed-systems>)"},{"heading":"TLA+ Fundamentals","level":2,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#tla-fundamentals","content":"[TLA+](<https://lamport.azurewebsites.net/tla/tla.html>) (Temporal Logic of Actions) is Leslie Lamport's formal specification language. You don't write code in TLA+, you write a mathematical model of your system as a state machine. Then you run the TLC model checker, which exhaustively explores every possible state your system can reach and verifies that your invariants hold in all of them.\n\nThe key insight: distributed systems bugs aren't in the code. They're in the *design*. If your protocol is wrong, no amount of careful coding or testing will save you. TLA+ catches design bugs before you write a single line of implementation code.\n\nA minimal TLA+ specification:\n\n```tla\n---- MODULE SimpleCounter ----\nEXTENDS Integers\n\nVARIABLE count\n\nInit == count = 0\n\nIncrement == count < 10 /\\ count' = count + 1\n\nNext == Increment\n\nSpec == Init /\\ [][Next]_count\n\nCountInvariant == count >= 0 /\\ count <= 10\n\n====\n```\n\nThis defines a counter that starts at 0, increments up to 10, and we assert it's always between 0 and 10. Trivial, but it shows the structure:\n\n- ` Init `: The initial state\n- ` Next `: All possible transitions (actions)\n- ` Spec `: The system specification (init, then repeatedly take Next steps)\n- ` CountInvariant `: A property that must hold in every reachable state"},{"heading":"PlusCal: TLA+ for Humans","level":2,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#pluscal-tla-for-humans","content":"Raw TLA+ is mathematical notation. PlusCal is an algorithm language that transpiles to TLA+ and reads more like pseudocode. For most specifications, start with PlusCal.\n\nHere's a mutual exclusion algorithm in PlusCal:\n\n```tla\n---- MODULE MutualExclusion ----\nEXTENDS Integers, TLC\n\nCONSTANT NumProcesses\n\n(*--algorithm mutex\nvariables\n lock = \"free\";\n\nprocess proc \\in 1..NumProcesses\nvariables hasLock = FALSE;\nbegin\n AcquireLock:\n await lock = \"free\";\n lock := self;\n hasLock := TRUE;\n\n CriticalSection:\n \\* Do work in critical section\n skip;\n\n ReleaseLock:\n lock := \"free\";\n hasLock := FALSE;\n\n goto AcquireLock;\nend process;\nend algorithm; *)\n\n\\* Mutual exclusion invariant\nMutualExclusion ==\n \\A p1, p2 \\in 1..NumProcesses :\n (p1 /= p2) =>\n ~(pc[p1] = \"CriticalSection\" /\\ pc[p2] = \"CriticalSection\")\n\n====\n```\n\nRun TLC with ` NumProcesses = 3 ` and it will explore every possible interleaving of the three processes. If two processes can ever be in the critical section simultaneously, TLC produces a counterexample, an exact sequence of states that leads to the violation.\n\nThe ` await ` keyword is where the magic happens. It means \"this step can only execute when the condition is true.\" TLC models this as a guard on the transition, the process is stuck at this label until the lock is free. Combined with the implicit non-deterministic interleaving of all processes, this perfectly models the real concurrency semantics."},{"heading":"The Real Kafka Example","level":2,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#the-real-kafka-example","content":"The following specification found a real bug. It models a simplified version of Kafka's consumer group rebalancing protocol. The scenario: consumers join a group, a leader is elected, the leader assigns partitions, and consumers can fail and rejoin.\n\n```tla\n---- MODULE KafkaRebalance ----\nEXTENDS Integers, Sequences, FiniteSets, TLC\n\nCONSTANTS\n Consumers, \\* Set of consumer IDs, e.g., {\"c1\", \"c2\", \"c3\"}\n Partitions, \\* Set of partition IDs, e.g., {1, 2, 3, 4}\n MaxGenerations \\* Bound on generation numbers for model checking\n\nVARIABLES\n groupState, \\* \"empty\" | \"rebalancing\" | \"stable\"\n generation, \\* Current group generation number\n members, \\* Set of consumers currently in the group\n leader, \\* Current leader consumer (or \"none\")\n assignment, \\* Function: partition -> consumer (or \"unassigned\")\n consumerState, \\* Function: consumer -> \"idle\" | \"joining\" | \"active\" | \"failed\"\n pendingJoins, \\* Set of consumers waiting to join\n pendingSync \\* Set of consumers waiting for sync response\n\nvars == <<groupState, generation, members, leader, assignment,\n consumerState, pendingJoins, pendingSync>>\n\nTypeInvariant ==\n /\\ groupState \\in {\"empty\", \"rebalancing\", \"stable\"}\n /\\ generation \\in 0..MaxGenerations\n /\\ members \\subseteq Consumers\n /\\ leader \\in Consumers \\union {\"none\"}\n /\\ \\A p \\in Partitions : assignment[p] \\in Consumers \\union {\"unassigned\"}\n /\\ \\A c \\in Consumers : consumerState[c] \\in\n {\"idle\", \"joining\", \"active\", \"failed\"}\n\nInit ==\n /\\ groupState = \"empty\"\n /\\ generation = 0\n /\\ members = {}\n /\\ leader = \"none\"\n /\\ assignment = [p \\in Partitions |-> \"unassigned\"]\n /\\ consumerState = [c \\in Consumers |-> \"idle\"]\n /\\ pendingJoins = {}\n /\\ pendingSync = {}\n\n\\* A consumer requests to join the group\nJoinGroup(c) ==\n /\\ consumerState[c] \\in {\"idle\", \"failed\"}\n /\\ generation < MaxGenerations\n /\\ pendingJoins' = pendingJoins \\union {c}\n /\\ consumerState' = [consumerState EXCEPT ![c] = \"joining\"]\n /\\ UNCHANGED <<groupState, generation, members, leader,\n assignment, pendingSync>>\n\n\\* Coordinator processes pending joins (triggers rebalance)\nTriggerRebalance ==\n /\\ pendingJoins /= {}\n /\\ groupState' = \"rebalancing\"\n /\\ generation' = generation + 1\n /\\ members' = (members \\ {c \\in members :\n consumerState[c] = \"failed\"}) \\union pendingJoins\n /\\ leader' = IF members' /= {} THEN CHOOSE c \\in members' : TRUE\n ELSE \"none\"\n /\\ pendingSync' = members'\n /\\ pendingJoins' = {}\n /\\ UNCHANGED <<assignment, consumerState>>\n\n\\* Leader computes and sends partition assignment\nLeaderAssign ==\n /\\ groupState = \"rebalancing\"\n /\\ leader /= \"none\"\n /\\ consumerState[leader] = \"joining\"\n \\* Simple round-robin assignment\n /\\ LET memberSeq == SetToSeq(members')\n assignFn == [p \\in Partitions |->\n IF members /= {}\n THEN memberSeq[((p - 1) % Len(memberSeq)) + 1]\n ELSE \"unassigned\"]\n IN assignment' = assignFn\n /\\ consumerState' = [c \\in Consumers |->\n IF c \\in pendingSync THEN \"active\" ELSE consumerState[c]]\n /\\ groupState' = \"stable\"\n /\\ pendingSync' = {}\n /\\ UNCHANGED <<generation, members, leader, pendingJoins>>\n\n\\* A consumer fails (crashes)\nConsumerFail(c) ==\n /\\ consumerState[c] \\in {\"joining\", \"active\"}\n /\\ consumerState' = [consumerState EXCEPT ![c] = \"failed\"]\n /\\ members' = members \\ {c}\n \\* BUG WAS HERE: assignment not immediately cleared for failed consumer\n \\* Original (buggy): UNCHANGED <<assignment>>\n \\* Fixed: clear assignments for the failed consumer\n /\\ assignment' = [p \\in Partitions |->\n IF assignment[p] = c THEN \"unassigned\" ELSE assignment[p]]\n /\\ IF leader = c\n THEN leader' = IF members' /= {} THEN CHOOSE m \\in members' : TRUE\n ELSE \"none\"\n ELSE UNCHANGED leader\n /\\ UNCHANGED <<groupState, generation, pendingJoins, pendingSync>>\n\nNext ==\n \\/ \\E c \\in Consumers : JoinGroup(c)\n \\/ TriggerRebalance\n \\/ LeaderAssign\n \\/ \\E c \\in Consumers : ConsumerFail(c)\n\nSpec == Init /\\ [][Next]_vars\n\n\\* SAFETY: No partition assigned to a failed consumer\nNoFailedAssignment ==\n \\A p \\in Partitions :\n assignment[p] /= \"unassigned\" =>\n consumerState[assignment[p]] /= \"failed\"\n\n\\* SAFETY: No partition assigned to two different active consumers\n\\* (This is the invariant that caught the bug)\nUniqueAssignment ==\n \\A p1, p2 \\in Partitions :\n /\\ p1 /= p2\n /\\ assignment[p1] /= \"unassigned\"\n /\\ assignment[p2] /= \"unassigned\"\n => TRUE \\* each partition assigned to at most one consumer (implicit)\n\n\\* SAFETY: All assigned consumers are members of the group\nAssignedAreMembers ==\n \\A p \\in Partitions :\n assignment[p] /= \"unassigned\" => assignment[p] \\in members\n\nSetToSeq(S) ==\n CHOOSE seq \\in [1..Cardinality(S) -> S] :\n \\A i, j \\in 1..Cardinality(S) : i /= j => seq[i] /= seq[j]\n\n====\n```"},{"heading":"The Bug TLC Found","level":2,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#the-bug-tlc-found","content":"When running this with the original ` UNCHANGED <<assignment>> ` in the ` ConsumerFail ` action, TLC found a violation of ` NoFailedAssignment ` in 6 states:\n\n```\nError: Invariant NoFailedAssignment is violated.\n\nError trace:\n1: <Init>\n groupState = \"empty\", generation = 0, members = {}\n\n2: JoinGroup(\"c1\")\n consumerState[\"c1\"] = \"joining\", pendingJoins = {\"c1\"}\n\n3: JoinGroup(\"c2\")\n consumerState[\"c2\"] = \"joining\", pendingJoins = {\"c1\", \"c2\"}\n\n4: TriggerRebalance\n generation = 1, members = {\"c1\", \"c2\"}, leader = \"c1\"\n\n5: LeaderAssign\n assignment = [1 |-> \"c1\", 2 |-> \"c2\", 3 |-> \"c1\", 4 |-> \"c2\"]\n groupState = \"stable\"\n\n6: ConsumerFail(\"c2\")\n consumerState[\"c2\"] = \"failed\", members = {\"c1\"}\n assignment = [1 |-> \"c1\", 2 |-> \"c2\", 3 |-> \"c1\", 4 |-> \"c2\"]\n ^^^^^^^^^^^^^^^^^^^^^^^^ c2 is failed but still assigned partitions 2 and 4!\n```\n\nThe fix is obvious once you see it: when a consumer fails, immediately clear its partition assignments. But in the real implementation, the assignment clearing happened during the *next* rebalance, not at failure detection time. This meant there was a window where a new consumer could join, trigger a rebalance, and the leader's assignment algorithm could see the stale assignment and make conflicting decisions.\n\nThis type of window can exist in production systems for months. TLC finds it in 0.3 seconds."},{"heading":"Running the Model Checker","level":2,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#running-the-model-checker","content":"Here's how to actually run TLC:\n\n```bash\n# Install the TLA+ Toolbox or use the CLI\n# Download tla2tools.jar from GitHub\n\n# Create a config file (KafkaRebalance.cfg)\ncat > KafkaRebalance.cfg << 'EOF'\nCONSTANTS\n Consumers = {\"c1\", \"c2\", \"c3\"}\n Partitions = {1, 2, 3, 4}\n MaxGenerations = 3\n\nINIT Init\nNEXT Next\n\nINVARIANTS\n TypeInvariant\n NoFailedAssignment\n AssignedAreMembers\nEOF\n\n# Run TLC\njava -jar tla2tools.jar -config KafkaRebalance.cfg KafkaRebalance.tla\n\n# Output:\n# Model checking completed. No errors found.\n# 847,293 states generated, 234,521 distinct states found.\n# Finished in 00:00:04 (4 seconds)\n```\n\n847,293 states. Every single possible interleaving of consumer joins, failures, rebalances, and leader assignments with 3 consumers, 4 partitions, and up to 3 generations. Explored exhaustively in 4 seconds. No randomness, no \"hopefully we hit the bug\", mathematical certainty that the invariants hold."},{"heading":"Temporal Properties: Liveness","level":2,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#temporal-properties-liveness","content":"Safety properties (\"bad things never happen\") catch a lot of bugs, but liveness properties (\"good things eventually happen\") catch a different class:\n\n```tla\n\\* LIVENESS: Every partition is eventually assigned to an active consumer\n\\* (assuming consumers eventually join and rebalances complete)\nEventuallyAssigned ==\n \\A p \\in Partitions :\n <>(assignment[p] /= \"unassigned\" /\\\n consumerState[assignment[p]] = \"active\")\n\n\\* LIVENESS: The group eventually becomes stable after a rebalance\nEventuallyStable ==\n groupState = \"rebalancing\" ~> groupState = \"stable\"\n```\n\nThe ` <> ` operator means \"eventually\" (at some point in the future). The ` ~> ` operator means \"leads to\" (whenever the left side is true, the right side is eventually true). These require TLC to check temporal properties, which is slower but catches liveness bugs like permanent blocking or starvation."},{"heading":"Practical Patterns Worth Reusing","level":2,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#practical-patterns-worth-reusing","content":""},{"heading":"Modeling Network Partitions","level":3,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#modeling-network-partitions","content":"```tla\n\\* Add a variable for network connectivity\nVARIABLE connected \\* Function: {node1, node2} -> BOOLEAN\n\n\\* Action: network partition occurs between two nodes\nNetworkPartition(n1, n2) ==\n /\\ connected' = [connected EXCEPT ![{n1, n2}] = FALSE]\n /\\ UNCHANGED <<otherVars>>\n\n\\* Action: network heals\nNetworkHeal(n1, n2) ==\n /\\ connected' = [connected EXCEPT ![{n1, n2}] = TRUE]\n /\\ UNCHANGED <<otherVars>>\n\n\\* Guard: message send only succeeds if network is connected\nSendMessage(from, to, msg) ==\n /\\ connected[{from, to}]\n /\\ \\* ... rest of send logic\n```"},{"heading":"Modeling Message Loss and Reordering","level":3,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#modeling-message-loss-and-reordering","content":"```tla\nVARIABLE messages \\* Set of in-flight messages (unordered by design)\n\n\\* Sending adds to the set (not a queue, models reordering)\nSend(msg) ==\n /\\ messages' = messages \\union {msg}\n\n\\* Receiving removes from the set (non-deterministic choice models reordering)\nReceive(msg) ==\n /\\ msg \\in messages\n /\\ messages' = messages \\ {msg}\n\n\\* Messages can also be lost (dropped from the set without delivery)\nLoseMessage(msg) ==\n /\\ msg \\in messages\n /\\ messages' = messages \\ {msg}\n /\\ UNCHANGED <<otherVars>> \\* No state change from delivery\n```"},{"heading":"When Not to Use TLA+","level":2,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#when-not-to-use-tla","content":"TLA+ is overkill for most software. Use it when:\n\n1. **Correctness is critical and testing is insufficient.** Distributed consensus protocols, lock-free data structures, transaction isolation levels.\n2. **The interaction space is too large for testing.** 5 nodes, each with 4 states, 3 types of messages, failure at any point, that's billions of interleavings.\n3. **The cost of a bug is high.** Data corruption, financial loss, safety-critical systems.\n\nDon't use it for:\n\n- CRUD applications\n- Simple request-response services\n- Anything where the hard part is the UI or business logic rather than the concurrency protocol\n\nThe learning curve is real -- roughly 3 weeks of dedicated study to go from zero to productive. But given the track record of TLC catching bugs in protocols that seemed obviously correct, modeling every non-trivial distributed protocol before implementation is a sound investment. A spec usually takes 2-3 days. Finding and fixing a production concurrency bug takes 2-3 weeks. The math is straightforward."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#questions","content":""},{"heading":"What is TLA+ and why use it for distributed systems?","level":3,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#question-1","content":"TLA+ is a formal specification language based on temporal logic of actions. You model your system as a state machine, define invariants (properties that must always hold), and run the TLC model checker to exhaustively explore every possible state. It catches bugs that testing can't, like race conditions that only manifest with specific message ordering across 5 nodes."},{"heading":"Is TLA+ practical for real engineering teams?","level":3,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#question-2","content":"Yes. Amazon uses TLA+ extensively, they've found critical bugs in DynamoDB, S3, and EBS using formal verification. The learning curve is steep (2-3 weeks to be productive), but for systems where correctness is critical, the investment pays off dramatically. You don't need to verify everything, just the tricky distributed protocol parts."},{"heading":"How does TLA+ differ from property-based testing?","level":3,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#question-3","content":"Property-based testing (like QuickCheck) generates random inputs and checks properties against your actual code. TLA+ checks properties against a mathematical model of your system by exhaustively exploring all possible states. Property-based testing can miss bugs hidden in unexplored states; TLA+ finds them all (within the bounds of your model)."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#article-tips","content":"1. **Tip:** Start with ` PlusCal ` instead of raw TLA+. PlusCal is an algorithm language that transpiles to TLA+ and reads like pseudocode. Once TLC finds a bug, you'll need to read the TLA+ translation to understand the error trace, but for writing specs, PlusCal cuts your learning curve in half.\n2. **Tip:** Keep your state space small by using ` CONSTANTS ` with minimal values, 2 nodes, 3 messages max, 2 epochs. TLC explores every reachable state; going from 2 nodes to 3 can increase the state space by 100x. Only bump values up after your spec passes with small constants.\n3. **Tip:** Always add a ` TypeInvariant ` that constrains every variable to its expected domain. Without it, a typo in your spec (like assigning a string where you expect an integer) creates millions of garbage states and TLC runs for hours before OOMing. The type invariant catches this in seconds.\n4. **Tip:** Run TLC with the ` -deadlock ` flag disabled when you first start modeling. Many valid distributed system designs have terminal states (like all nodes shutting down) that TLC flags as deadlocks by default. You can add liveness properties later once the safety properties check out.\n\n- tla-plus\n- formal-verification\n- distributed-systems\n- model-checking\n- temporal-logic\n- kafka\n- consensus\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Lucio Durán","url":"https://lucioduran.com/"},{"text":"Español","url":"https://lucioduran.com/es/blog/formal-verification-tla-plus-distributed-systems"},{"text":"TLA+","url":"https://lamport.azurewebsites.net/tla/tla.html"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com/"},{"@type":"WebSite","@id":"https://lucioduran.com/#website","url":"https://lucioduran.com/","name":"Lucio Durán","inLanguage":["en","es"],"publisher":{"@id":"https://lucioduran.com/#person"}},{"@type":"WebPage","@id":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#page","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems","name":"TLA+ for Formal Verification: Catching Distributed Systems Bugs Before Writing a Single Line of Code | Lucio Durán","description":"Modeling distributed systems with temporal logic, running the TLC model checker, and a real-world example of how TLA+ found a bug in a Kafka consumer group rebalancing protocol that testing would never have caught.","inLanguage":"en","author":{"@id":"https://lucioduran.com/#person"},"publisher":{"@id":"https://lucioduran.com/#person"},"isPartOf":{"@id":"https://lucioduran.com/#website"},"isAccessibleForFree":true,"datePublished":"2026-03-18","mainEntity":{"@id":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#article"},"breadcrumb":{"@id":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#breadcrumb"}},{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#article","mainEntityOfPage":{"@id":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#page"},"headline":"TLA+ for Formal Verification: Catching Distributed Systems Bugs Before Writing a Single Line of Code","description":"Modeling distributed systems with temporal logic, running the TLC model checker, and a real-world example of how TLA+ found a bug in a Kafka consumer group rebalancing protocol that testing would never have caught.","datePublished":"2026-03-18","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@id":"https://lucioduran.com/#person"},"url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"Distributed Systems","keywords":"tla-plus, formal-verification, distributed-systems, model-checking, temporal-logic, kafka, consensus","wordCount":1894,"timeRequired":"PT10M","workTranslation":[{"@type":"BlogPosting","@id":"https://lucioduran.com/es/blog/formal-verification-tla-plus-distributed-systems#article","inLanguage":"es","url":"https://lucioduran.com/es/blog/formal-verification-tla-plus-distributed-systems"}],"isAccessibleForFree":true},{"@type":"Blog","@id":"https://lucioduran.com/blog#blog","url":"https://lucioduran.com/blog","name":"Blog","inLanguage":"en","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com/"},"publisher":{"@id":"https://lucioduran.com/#person"},"isPartOf":{"@id":"https://lucioduran.com/#website"}},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"TLA+ for Formal Verification: Catching Distributed Systems Bugs Before Writing a Single Line of Code","item":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems"}],"@id":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#breadcrumb"},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"What is TLA+ and why use it for distributed systems?","acceptedAnswer":{"@type":"Answer","text":"TLA+ is a formal specification language based on temporal logic of actions. You model your system as a state machine, define invariants (properties that must always hold), and run the TLC model checker to exhaustively explore every possible state. It catches bugs that testing can't, like race conditions that only manifest with specific message ordering across 5 nodes."}},{"@type":"Question","name":"Is TLA+ practical for real engineering teams?","acceptedAnswer":{"@type":"Answer","text":"Yes. Amazon uses TLA+ extensively, they've found critical bugs in DynamoDB, S3, and EBS using formal verification. The learning curve is steep (2-3 weeks to be productive), but for systems where correctness is critical, the investment pays off dramatically. You don't need to verify everything, just the tricky distributed protocol parts."}},{"@type":"Question","name":"How does TLA+ differ from property-based testing?","acceptedAnswer":{"@type":"Answer","text":"Property-based testing (like QuickCheck) generates random inputs and checks properties against your actual code. TLA+ checks properties against a mathematical model of your system by exhaustively exploring all possible states. Property-based testing can miss bugs hidden in unexplored states; TLA+ finds them all (within the bounds of your model)."}}],"@id":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#questions","url":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#questions","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems#page"},"author":{"@id":"https://lucioduran.com/#person"}}]}],"contentHash":"63f52f3d541473cc7f88f9bb019c32e2649e64f11ebfce16fe91bcc356a21817","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/formal-verification-tla-plus-distributed-systems","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":75165,"contentBytes":15887}}