What is immutable infrastructure and how does it differ from traditional server management?

In immutable infrastructure, servers are never modified after deployment. Instead of SSHing in to patch, update, or configure a running server, you build a new image with the changes, deploy fresh instances from that image, and destroy the old ones. This eliminates configuration drift, the state where servers that should be identical have diverged due to ad-hoc changes.

Is immutable infrastructure practical for databases and stateful services?

For the compute layer, absolutely. For stateful services like databases, the approach is modified, you use immutable images for the database engine and configuration, but the data volume is persistent and mounted separately. RDS and managed database services essentially do this already. The key insight is separating the immutable compute from the mutable state.

How do you handle emergency patches with immutable infrastructure?

The same way as regular deployments, just faster. Your AMI pipeline should be fast enough (under 10 minutes) that building a new image for an emergency patch is viable. If that's too slow, you have two options: keep a pre-built 'hotfix base' image that's always ready, or accept that for true emergencies, you might need to temporarily break immutability and patch in place, but then immediately rebuild the image to prevent drift.

Immutable Infrastructure: Packer Golden Images, AMI Pipelines, Blue-Green Deployments, and the Real Migration from Mutable Servers

The Core Principle

Immutable infrastructure means servers are never modified after deployment. The workflow is:

Code Change → Build New Image → Deploy New Instances → Destroy Old Instances

Not:

Code Change → SSH into server → apt-get install → pray

Every server is born from an image, runs untouched, and dies when replaced. There's no drift because there's no mutation.

Building Golden Images with Packer

Packer builds machine images from a declarative template. Here's the production template I use:

# packer/app-server.pkr.hcl

packer {
 required_plugins {
 amazon = {
 version = ">= 1.3.0"
 source = "github.com/hashicorp/amazon"
 }
 }
}

variable "app_version" {
 type = string
 default = "latest"
}

variable "git_sha" {
 type = string
}

source "amazon-ebs" "app_server" {
 ami_name = "app-server-${var.app_version}-${formatdate("YYYYMMDD-hhmm", timestamp())}"
 instance_type = "t3.medium"
 region = "us-east-1"

 source_ami_filter {
 filters = {
 name = "ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-arm64-server-*"
 root-device-type = "ebs"
 virtualization-type = "hvm"
 }
 owners = ["099720109477"] # Canonical
 most_recent = true
 }

 ssh_username = "ubuntu"

 tags = {
 Name = "app-server-${var.app_version}"
 GitSHA = var.git_sha
 BuildTime = timestamp()
 Base = "ubuntu-24.04"
 }
}

build {
 sources = ["source.amazon-ebs.app_server"]

 # System setup
 provisioner "shell" {
 inline = [
 "sudo apt-get update -y",
 "sudo apt-get upgrade -y",
 "sudo apt-get install -y curl unzip jq",

 # Install Node.js 22 LTS
 "curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -",
 "sudo apt-get install -y nodejs",

 # Install CloudWatch agent
 "wget https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb",
 "sudo dpkg -i amazon-cloudwatch-agent.deb",
 "rm amazon-cloudwatch-agent.deb",

 # Security hardening
 "sudo ufw default deny incoming",
 "sudo ufw default allow outgoing",
 "sudo ufw allow 22/tcp",
 "sudo ufw allow 3000/tcp",
 "sudo ufw --force enable",
 ]
 }

 # Application deployment
 provisioner "file" {
 source = "../dist/"
 destination = "/tmp/app/"
 }

 provisioner "shell" {
 inline = [
 "sudo mkdir -p /opt/app",
 "sudo cp -r /tmp/app/* /opt/app/",
 "cd /opt/app && sudo npm ci --production",

 # Create systemd service
 "sudo tee /etc/systemd/system/app.service > /dev/null <<EOF",
 "[Unit]",
 "Description=Application Server",
 "After=network.target",
 "",
 "[Service]",
 "Type=simple",
 "User=www-data",
 "WorkingDirectory=/opt/app",
 "ExecStart=/usr/bin/node server.js",
 "Restart=always",
 "RestartSec=5",
 "Environment=NODE_ENV=production",
 "Environment=PORT=3000",
 "",
 "[Install]",
 "WantedBy=multi-user.target",
 "EOF",

 "sudo systemctl daemon-reload",
 "sudo systemctl enable app.service",
 ]
 }

 # Cleanup
 provisioner "shell" {
 inline = [
 "sudo apt-get clean",
 "sudo rm -rf /tmp/*",
 "sudo rm -rf /var/tmp/*",
 "sudo journalctl --vacuum-time=1d",
 ]
 }
}

Build it in CI:

# GitHub Actions step
packer build \
 -var "app_version=${GITHUB_REF_NAME}" \
 -var "git_sha=${GITHUB_SHA}" \
 packer/app-server.pkr.hcl

The AMI Pipeline

The full pipeline looks like this:

# .github/workflows/ami-pipeline.yml
name: AMI Pipeline

on:
 push:
 branches: [main]

jobs:
 build:
 runs-on: ubuntu-latest
 outputs:
 ami_id: ${{ steps.build.outputs.ami_id }}
 steps:
 - uses: actions/checkout@v4

 - name: Build application
 run: npm ci && npm run build

 - name: Run tests
 run: npm test

 - name: Build AMI
 id: build
 run: |
 AMI_ID=$(packer build -machine-readable \
 -var "app_version=${{ github.ref_name }}" \
 -var "git_sha=${{ github.sha }}" \
 packer/app-server.pkr.hcl | \
 grep 'artifact,0,id' | cut -d: -f2)
 echo "ami_id=$AMI_ID" >> $GITHUB_OUTPUT

 - name: Validate AMI
 run: |
 # Launch a test instance from the new AMI
 INSTANCE_ID=$(aws ec2 run-instances \
 --image-id ${{ steps.build.outputs.ami_id }} \
 --instance-type t3.small \
 --query 'Instances[0].InstanceId' \
 --output text)

 # Wait for it to be running
 aws ec2 wait instance-status-ok --instance-ids $INSTANCE_ID

 # Health check
 IP=$(aws ec2 describe-instances \
 --instance-ids $INSTANCE_ID \
 --query 'Reservations[0].Instances[0].PublicIpAddress' \
 --output text)

 curl -f http://$IP:3000/health || exit 1

 # Cleanup
 aws ec2 terminate-instances --instance-ids $INSTANCE_ID

 deploy:
 needs: build
 runs-on: ubuntu-latest
 steps:
 - name: Update Launch Template
 run: |
 aws ec2 create-launch-template-version \
 --launch-template-name app-server \
 --source-version '$Latest' \
 --launch-template-data '{"ImageId":"${{ needs.build.outputs.ami_id }}"}'

 - name: Blue-Green Deployment
 run: |
 # Start instance refresh (rolling replacement)
 aws autoscaling start-instance-refresh \
 --auto-scaling-group-name app-server-asg \
 --preferences '{
 "MinHealthyPercentage": 90,
 "InstanceWarmup": 120
 }'

Blue-Green Deployments

The ASG instance refresh is the simplest approach, but for zero-downtime deployments, I use a proper blue-green setup:

# terraform/blue-green.tf

resource "aws_lb_target_group" "blue" {
 name = "app-blue"
 port = 3000
 protocol = "HTTP"
 vpc_id = var.vpc_id

 health_check {
 path = "/health"
 healthy_threshold = 2
 unhealthy_threshold = 3
 interval = 10
 }
}

resource "aws_lb_target_group" "green" {
 name = "app-green"
 port = 3000
 protocol = "HTTP"
 vpc_id = var.vpc_id

 health_check {
 path = "/health"
 healthy_threshold = 2
 unhealthy_threshold = 3
 interval = 10
 }
}

resource "aws_lb_listener_rule" "app" {
 listener_arn = var.listener_arn
 priority = 100

 action {
 type = "forward"
 forward {
 target_group {
 arn = aws_lb_target_group.blue.arn
 weight = var.blue_weight # 100 initially
 }
 target_group {
 arn = aws_lb_target_group.green.arn
 weight = var.green_weight # 0 initially
 }
 }
 }

 condition {
 path_pattern { values = ["/*"] }
 }
}

The deployment script shifts traffic gradually:

#!/bin/bash
# deploy-blue-green.sh

NEW_AMI=$1
CURRENT_ENV=$(get_active_environment) # "blue" or "green"
TARGET_ENV=$([[ "$CURRENT_ENV" == "blue" ]] && echo "green" || echo "blue")

echo "Deploying $NEW_AMI to $TARGET_ENV environment"

# 1. Launch new instances in the target environment
update_asg_ami "$TARGET_ENV" "$NEW_AMI"
wait_for_healthy "$TARGET_ENV" 120

# 2. Canary: send 10% of traffic
set_traffic_weights "$CURRENT_ENV" 90 "$TARGET_ENV" 10
sleep 300 # Bake for 5 minutes

# 3. Check error rates
ERROR_RATE=$(get_error_rate "$TARGET_ENV" "5m")
if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
 echo "Error rate too high ($ERROR_RATE), rolling back"
 set_traffic_weights "$CURRENT_ENV" 100 "$TARGET_ENV" 0
 exit 1
fi

# 4. Shift to 50/50
set_traffic_weights "$CURRENT_ENV" 50 "$TARGET_ENV" 50
sleep 300

# 5. Full cutover
set_traffic_weights "$CURRENT_ENV" 0 "$TARGET_ENV" 100
sleep 600 # Bake for 10 minutes

# 6. Scale down old environment
scale_down_asg "$CURRENT_ENV" 0

echo "Deployment complete. Active environment: $TARGET_ENV"

The Migration Story

Migrating from mutable to immutable took us three months. The hardest parts, in order:

1. Discovering all the manual changes. We ran debsums and find / -newer /var/log/installer on every production server to catalog what had changed since the original AMI. We found 23 manually installed packages, 8 modified config files, 3 cron jobs added by hand, and a Python script someone had written directly on a production server that was critical to the billing pipeline.

2. Application configuration. The app read config from local files that had been hand-edited. We moved everything to environment variables injected via the EC2 instance's user data script and AWS Systems Manager Parameter Store:

# User data script, runs on every boot
#!/bin/bash
export APP_DB_HOST=$(aws ssm get-parameter --name /app/db/host --query 'Parameter.Value' --output text)
export APP_DB_PASSWORD=$(aws ssm get-parameter --name /app/db/password --with-decryption --query 'Parameter.Value' --output text)
export APP_REDIS_URL=$(aws ssm get-parameter --name /app/redis/url --query 'Parameter.Value' --output text)

# Write to environment file
cat > /opt/app/.env <<EOF
DB_HOST=$APP_DB_HOST
DB_PASSWORD=$APP_DB_PASSWORD
REDIS_URL=$APP_REDIS_URL
EOF

systemctl start app.service

3. Log management. Servers that get destroyed lose their logs. We set up CloudWatch Logs agent in the golden image to ship logs before the server dies. For post-mortem debugging, we also enabled VPC Flow Logs and ALB access logs to S3.

4. Team habits. The hardest change was cultural. Engineers had to stop thinking "I'll just SSH in and fix it" and start thinking "I'll fix the template and rebuild." We literally disabled SSH access to production instances (no key pair assigned) to force the change.

Drift Detection

Even with immutable infrastructure, drift can creep in through user data scripts, EBS volumes that persist across replacements, or someone re-enabling SSH "just this once." I run drift detection weekly:

# Compare running instance configuration against the golden AMI
aws ec2 describe-instances \
 --filters "Name=tag:Environment,Values=production" \
 --query 'Reservations[].Instances[].[InstanceId,ImageId,LaunchTime]' \
 --output table

# Alert if any instance is running an AMI older than 7 days
CURRENT_AMI=$(get_latest_ami "app-server")
for instance in $(get_production_instances); do
 INSTANCE_AMI=$(get_instance_ami "$instance")
 if [ "$INSTANCE_AMI" != "$CURRENT_AMI" ]; then
 alert "Instance $instance running stale AMI $INSTANCE_AMI (current: $CURRENT_AMI)"
 fi
done

Wrapping Up

Immutable infrastructure isn't about tools, Packer is just the mechanism. It's about a discipline: servers are cattle, not pets. They're born from an image, they serve requests, and they're replaced. No patches, no hotfixes, no "I'll just tweak this one thing."

The migration was painful. Three months of archeology on servers that had been accumulating changes for over a year. But the result is worth it: every deployment is a known-good state, every rollback is instant (just point to the previous AMI), and configuration drift is impossible by design.

The best infrastructure is the kind that doesn't surprise you.

Immutable Infrastructure: Packer Golden Images, AMI Pipelines, Blue-Green Deployments, and the Real Migration from Mutable Servers