The Steady-State Hypothesis

Every chaos experiment starts with a hypothesis. An unstructured action such as terminating a random pod without measurable expectations is not an experiment. A proper hypothesis looks like:

"When we terminate 30% of API pods in the checkout namespace, we expect: request success rate remains above 99.5%, p99 latency stays below 500ms, and the Kubernetes HPA scales replacement pods within 60 seconds."

This is your steady-state hypothesis. It defines:

  1. The perturbation, what you're breaking
  2. The steady-state metrics, what "normal" looks like
  3. The tolerance, how much deviation is acceptable

Before running anything, you need a baseline. Capture at least 30 minutes of steady-state metrics under representative load:

# Capture baseline metrics with Prometheus
cat <<'PROMQL'
# Success rate (should be >99.9% at baseline)
sum(rate(http_requests_total{status=~"2.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# p99 latency (capture baseline value)
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# Pod count (baseline for scaling verification)
count(kube_pod_status_phase{namespace="checkout", phase="Running"})
PROMQL

Record these. They're your "before" snapshot. If your steady-state metrics are already unstable before the experiment, fix that first. You can't detect chaos-induced degradation if the baseline is noisy.

LitmusChaos: Kubernetes-native chaos

LitmusChaos is the CNCF chaos engineering project built for Kubernetes. It uses CRDs to define experiments, which means your chaos experiments are version-controlled, reviewable YAML, just like everything else in your cluster.

Install it:

helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm install litmus litmuschaos/litmus \
 --namespace litmus --create-namespace \
 --set portal.server.serviceType=ClusterIP

Here's a representative experiment, pod deletion in the checkout namespace with specific blast radius controls:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
 name: checkout-pod-kill
 namespace: checkout
spec:
 engineState: active
 appinfo:
 appns: checkout
 applabel: app=checkout-api
 appkind: deployment
 chaosServiceAccount: litmus-admin
 experiments:
 - name: pod-delete
 spec:
 components:
 env:
 - name: TOTAL_CHAOS_DURATION
 value: "120" # 2 minutes
 - name: CHAOS_INTERVAL
 value: "30" # Kill a pod every 30s
 - name: PODS_AFFECTED_PERC
 value: "30" # Kill 30% of pods
 - name: FORCE
 value: "false" # Graceful termination first
 - name: SEQUENCE
 value: "parallel"
 probe:
 - name: success-rate-probe
 type: promProbe
 mode: Continuous
 runProperties:
 probeTimeout: 5s
 interval: 10s
 retry: 3
 promProbe/inputs:
 endpoint: http://prometheus:9090
 query: >
 sum(rate(http_requests_total{namespace="checkout",status=~"2.."}[1m]))
 /
 sum(rate(http_requests_total{namespace="checkout"}[1m]))
 comparator:
 type: float
 criteria: ">="
 value: "0.995" # Abort if success rate drops below 99.5%
 - name: latency-probe
 type: promProbe
 mode: Continuous
 runProperties:
 probeTimeout: 5s
 interval: 10s
 retry: 3
 promProbe/inputs:
 endpoint: http://prometheus:9090
 query: >
 histogram_quantile(0.99,
 sum(rate(http_request_duration_seconds_bucket{namespace="checkout"}[1m]))
 by (le))
 comparator:
 type: float
 criteria: "<="
 value: "0.5" # Abort if p99 > 500ms

The probes are the safety net. If success rate drops below 99.5% or p99 exceeds 500ms, LitmusChaos aborts the experiment and the chaos engine transitions to a stopped state. This is non-negotiable, never run an experiment without abort conditions.

Network chaos: the real killer

Pod kills are the "hello world" of chaos engineering. Real outages usually involve network partitions, DNS failures, and latency injection. LitmusChaos handles these too:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
 name: checkout-network-latency
 namespace: checkout
spec:
 engineState: active
 appinfo:
 appns: checkout
 applabel: app=checkout-api
 appkind: deployment
 experiments:
 - name: pod-network-latency
 spec:
 components:
 env:
 - name: NETWORK_INTERFACE
 value: "eth0"
 - name: TARGET_CONTAINER
 value: "checkout-api"
 - name: NETWORK_LATENCY
 value: "200" # Add 200ms latency
 - name: JITTER
 value: "50" # ±50ms jitter
 - name: TOTAL_CHAOS_DURATION
 value: "180"
 - name: DESTINATION_IPS
 value: "10.0.15.0/24" # Only to database subnet

This type of experiment commonly catches bugs in retry logic. Consider exponential backoff with a max of 3 retries, but the timeout per attempt was 5 seconds. Add 200ms of network latency to every DB call, and suddenly each retry takes 5.2 seconds. Three retries = 15.6 seconds total. The HTTP gateway had a 10-second timeout. Every retry attempt after the second one was wasted work, the client had already received a 504.

The fix was simple: per-attempt timeout of 2 seconds, total timeout of 8 seconds, max 4 retries. But this kind of issue rarely surfaces without injecting latency specifically on the database path.

Chaos Monkey and the Netflix lineage

Chaos Monkey, Netflix's original chaos tool, operates differently from LitmusChaos. It randomly terminates instances in your AWS environment. The philosophy is continuous chaos, it runs constantly, killing things at random, and your systems either handle it or they don't.

For AWS-native setups, the Simian Army (Chaos Monkey's evolution) integrates with Auto Scaling Groups:

import boto3
import random
import logging

logger = logging.getLogger('chaos-monkey')

def chaos_monkey_run(asg_name: str, kill_probability: float = 0.1):
 """Kill a random instance in the ASG with given probability"""
 ec2 = boto3.client('ec2')
 asg = boto3.client('autoscaling')

 # Get instances in the ASG
 response = asg.describe_auto_scaling_groups(
 AutoScalingGroupNames=[asg_name]
 )
 instances = response['AutoScalingGroups'][0]['Instances']
 healthy = [i for i in instances if i['HealthStatus'] == 'Healthy']

 if len(healthy) <= 1:
 logger.warning(f"Only {len(healthy)} healthy instance(s) in {asg_name}, skipping")
 return

 if random.random() > kill_probability:
 logger.info(f"Dice roll says no chaos today for {asg_name}")
 return

 victim = random.choice(healthy)
 instance_id = victim['InstanceId']

 logger.info(f"Terminating {instance_id} in {asg_name}")
 ec2.terminate_instances(InstanceIds=[instance_id])

 return instance_id

The key safeguard: never kill if only one healthy instance remains. And notice the probability parameter, in production, running at 10% probability per cycle is common. Most cycles do nothing. But over a week, every service gets hit.

GameDay methodology

A GameDay is a structured chaos session where the entire team participates. It's not just about the technical experiment, it's about observing how humans respond to failure.

Here's a proven template:

## GameDay Plan: Database Failover

**Date**: 2026-01-09 14:00 UTC
**Duration**: 90 minutes (60 min experiment + 30 min retro)
**Facilitator**: Lucio
**Participants**: Backend team, SRE, on-call engineer

### Pre-conditions
- [ ] All participants briefed on the experiment scope
- [ ] Dashboards open: Grafana main, PagerDuty, Slack #incidents
- [ ] Abort procedure documented and understood
- [ ] Customer support notified (not details, just "we're running resilience tests")

### Steady-state hypothesis
When the primary database (RDS Aurora) fails over to the read replica:
1. Write operations pause for <10 seconds
2. Read operations continue uninterrupted
3. Application reconnects automatically within 30 seconds
4. No data loss occurs
5. PagerDuty alert fires within 2 minutes

### Experiment steps
1. [T+0] Capture baseline metrics (5 min observation)
2. [T+5] Initiate RDS failover via AWS CLI
3. [T+5] Start timer, observe dashboards
4. [T+10] Verify writes resume
5. [T+15] Verify reads were uninterrupted
6. [T+20] Check data consistency (run validation query)
7. [T+30] Verify alerting fired correctly
8. [T+35] Run synthetic transaction end-to-end

### Abort conditions
- Error rate exceeds 20% for more than 60 seconds
- Complete service outage (0 successful requests)
- Data corruption detected
- Any participant calls abort

### Rollback
- Failover back to original primary
- If failover-back fails: restore from latest snapshot

The retro is where the real value lives. A typical GameDay might reveal:

  1. The connection pool reconnected fine, but held stale connections for 45 seconds before health-checking them
  2. The PagerDuty alert didn't fire because the check interval was 5 minutes, tightening it to 30 seconds resolved this
  3. The on-call engineer didn't know where the RDS failover runbook was, adding it to the incident response wiki closed the gap

None of this would have surfaced from reading architecture diagrams.

Blast radius control: the non-negotiable

Every experiment needs blast radius limits. These should be enforced at multiple layers:

Infrastructure level: Label nodes and namespaces that are eligible for chaos. Everything else is off-limits.

# Only these nodes can be targeted
kubectl label nodes worker-{1..3} chaos-eligible=true

# Namespace-level opt-in
kubectl annotate namespace checkout litmuschaos.io/chaos=true

Experiment level: Cap the percentage of affected resources. Never go above 50% for a first experiment on any target.

Temporal level: Set maximum duration. If the experiment exceeds its window, it auto-reverts.

Metric level: Probes that auto-abort when SLO budgets are threatened.

Organizational level: Change management approval for production experiments. A lightweight RFC process works well, a Slack thread in #chaos-proposals with the experiment YAML attached and two SRE approvals required.

Building the habit

Chaos engineering isn't a tool, it's a practice. The tool is the easy part. The hard part is making it a habit. Here's what works:

  1. Weekly chaos slot: Every Thursday at 14:00 UTC, one experiment runs. It's on the team calendar. Non-negotiable.
  2. Chaos scoreboard: Track experiments run, issues found, MTTR improvements. Make it visible.
  3. Incident-to-experiment pipeline: After every incident, ask "what chaos experiment would have caught this?" Add it to the backlog.
  4. Graduated complexity: Start with pod kills. Graduate to network chaos. Then multi-AZ failures. Then regional failovers. Each level requires demonstrated success at the previous level.

A database failover experiment like the one described above can lead to a connection pool fix that prevents multiple potential outages over the following year. Each of those would be a p1 incident affecting thousands of users. Forty-seven seconds of disruption during a controlled experiment is an extraordinarily cheap lesson compared to the alternative.