{"schemaVersion":1,"id":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images","title":"Immutable Infrastructure: Packer Golden Images, AMI Pipelines, Blue-Green Deployments, and the Real Migration from Mutable Servers","description":"A practical guide to immutable infrastructure, building golden images with Packer, automating AMI pipelines, implementing blue-green deployments, eliminating configuration drift, and the honest story of migrating a production system from mutable servers.","language":"en","datePublished":"2025-12-21","dateModified":null,"contentFormat":"text/markdown","content":"DevOps\n\n# Immutable Infrastructure: Packer Golden Images, AMI Pipelines, Blue-Green Deployments, and the Real Migration from Mutable Servers\n\n2025-12-217 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/immutable-infrastructure-packer-golden-images>)\n\n## The Core Principle\n\nImmutable infrastructure means servers are never modified after deployment. The workflow is:\n\n```\nCode Change → Build New Image → Deploy New Instances → Destroy Old Instances\n```\n\nNot:\n\n```\nCode Change → SSH into server → apt-get install → pray\n```\n\nEvery server is born from an image, runs untouched, and dies when replaced. There's no drift because there's no mutation.\n\n## Building Golden Images with Packer\n\n[Packer](<https://www.packer.io/>) builds machine images from a declarative template. Here's the production template I use:\n\n```hcl\n# packer/app-server.pkr.hcl\n\npacker {\n required_plugins {\n amazon = {\n version = \">= 1.3.0\"\n source = \"github.com/hashicorp/amazon\"\n }\n }\n}\n\nvariable \"app_version\" {\n type = string\n default = \"latest\"\n}\n\nvariable \"git_sha\" {\n type = string\n}\n\nsource \"amazon-ebs\" \"app_server\" {\n ami_name = \"app-server-${var.app_version}-${formatdate(\"YYYYMMDD-hhmm\", timestamp())}\"\n instance_type = \"t3.medium\"\n region = \"us-east-1\"\n\n source_ami_filter {\n filters = {\n name = \"ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-arm64-server-*\"\n root-device-type = \"ebs\"\n virtualization-type = \"hvm\"\n }\n owners = [\"099720109477\"] # Canonical\n most_recent = true\n }\n\n ssh_username = \"ubuntu\"\n\n tags = {\n Name = \"app-server-${var.app_version}\"\n GitSHA = var.git_sha\n BuildTime = timestamp()\n Base = \"ubuntu-24.04\"\n }\n}\n\nbuild {\n sources = [\"source.amazon-ebs.app_server\"]\n\n # System setup\n provisioner \"shell\" {\n inline = [\n \"sudo apt-get update -y\",\n \"sudo apt-get upgrade -y\",\n \"sudo apt-get install -y curl unzip jq\",\n\n # Install Node.js 22 LTS\n \"curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -\",\n \"sudo apt-get install -y nodejs\",\n\n # Install CloudWatch agent\n \"wget https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb\",\n \"sudo dpkg -i amazon-cloudwatch-agent.deb\",\n \"rm amazon-cloudwatch-agent.deb\",\n\n # Security hardening\n \"sudo ufw default deny incoming\",\n \"sudo ufw default allow outgoing\",\n \"sudo ufw allow 22/tcp\",\n \"sudo ufw allow 3000/tcp\",\n \"sudo ufw --force enable\",\n ]\n }\n\n # Application deployment\n provisioner \"file\" {\n source = \"../dist/\"\n destination = \"/tmp/app/\"\n }\n\n provisioner \"shell\" {\n inline = [\n \"sudo mkdir -p /opt/app\",\n \"sudo cp -r /tmp/app/* /opt/app/\",\n \"cd /opt/app && sudo npm ci --production\",\n\n # Create systemd service\n \"sudo tee /etc/systemd/system/app.service > /dev/null <<EOF\",\n \"[Unit]\",\n \"Description=Application Server\",\n \"After=network.target\",\n \"\",\n \"[Service]\",\n \"Type=simple\",\n \"User=www-data\",\n \"WorkingDirectory=/opt/app\",\n \"ExecStart=/usr/bin/node server.js\",\n \"Restart=always\",\n \"RestartSec=5\",\n \"Environment=NODE_ENV=production\",\n \"Environment=PORT=3000\",\n \"\",\n \"[Install]\",\n \"WantedBy=multi-user.target\",\n \"EOF\",\n\n \"sudo systemctl daemon-reload\",\n \"sudo systemctl enable app.service\",\n ]\n }\n\n # Cleanup\n provisioner \"shell\" {\n inline = [\n \"sudo apt-get clean\",\n \"sudo rm -rf /tmp/*\",\n \"sudo rm -rf /var/tmp/*\",\n \"sudo journalctl --vacuum-time=1d\",\n ]\n }\n}\n```\n\nBuild it in CI:\n\n```bash\n# GitHub Actions step\npacker build \\\n -var \"app_version=${GITHUB_REF_NAME}\" \\\n -var \"git_sha=${GITHUB_SHA}\" \\\n packer/app-server.pkr.hcl\n```\n\n## The AMI Pipeline\n\nThe full pipeline looks like this:\n\n```yaml\n# .github/workflows/ami-pipeline.yml\nname: AMI Pipeline\n\non:\n push:\n branches: [main]\n\njobs:\n build:\n runs-on: ubuntu-latest\n outputs:\n ami_id: ${{ steps.build.outputs.ami_id }}\n steps:\n - uses: actions/checkout@v4\n\n - name: Build application\n run: npm ci && npm run build\n\n - name: Run tests\n run: npm test\n\n - name: Build AMI\n id: build\n run: |\n AMI_ID=$(packer build -machine-readable \\\n -var \"app_version=${{ github.ref_name }}\" \\\n -var \"git_sha=${{ github.sha }}\" \\\n packer/app-server.pkr.hcl | \\\n grep 'artifact,0,id' | cut -d: -f2)\n echo \"ami_id=$AMI_ID\" >> $GITHUB_OUTPUT\n\n - name: Validate AMI\n run: |\n # Launch a test instance from the new AMI\n INSTANCE_ID=$(aws ec2 run-instances \\\n --image-id ${{ steps.build.outputs.ami_id }} \\\n --instance-type t3.small \\\n --query 'Instances[0].InstanceId' \\\n --output text)\n\n # Wait for it to be running\n aws ec2 wait instance-status-ok --instance-ids $INSTANCE_ID\n\n # Health check\n IP=$(aws ec2 describe-instances \\\n --instance-ids $INSTANCE_ID \\\n --query 'Reservations[0].Instances[0].PublicIpAddress' \\\n --output text)\n\n curl -f http://$IP:3000/health || exit 1\n\n # Cleanup\n aws ec2 terminate-instances --instance-ids $INSTANCE_ID\n\n deploy:\n needs: build\n runs-on: ubuntu-latest\n steps:\n - name: Update Launch Template\n run: |\n aws ec2 create-launch-template-version \\\n --launch-template-name app-server \\\n --source-version '$Latest' \\\n --launch-template-data '{\"ImageId\":\"${{ needs.build.outputs.ami_id }}\"}'\n\n - name: Blue-Green Deployment\n run: |\n # Start instance refresh (rolling replacement)\n aws autoscaling start-instance-refresh \\\n --auto-scaling-group-name app-server-asg \\\n --preferences '{\n \"MinHealthyPercentage\": 90,\n \"InstanceWarmup\": 120\n }'\n```\n\n## Blue-Green Deployments\n\nThe ASG instance refresh is the simplest approach, but for zero-downtime deployments, I use a proper blue-green setup:\n\n```hcl\n# terraform/blue-green.tf\n\nresource \"aws_lb_target_group\" \"blue\" {\n name = \"app-blue\"\n port = 3000\n protocol = \"HTTP\"\n vpc_id = var.vpc_id\n\n health_check {\n path = \"/health\"\n healthy_threshold = 2\n unhealthy_threshold = 3\n interval = 10\n }\n}\n\nresource \"aws_lb_target_group\" \"green\" {\n name = \"app-green\"\n port = 3000\n protocol = \"HTTP\"\n vpc_id = var.vpc_id\n\n health_check {\n path = \"/health\"\n healthy_threshold = 2\n unhealthy_threshold = 3\n interval = 10\n }\n}\n\nresource \"aws_lb_listener_rule\" \"app\" {\n listener_arn = var.listener_arn\n priority = 100\n\n action {\n type = \"forward\"\n forward {\n target_group {\n arn = aws_lb_target_group.blue.arn\n weight = var.blue_weight # 100 initially\n }\n target_group {\n arn = aws_lb_target_group.green.arn\n weight = var.green_weight # 0 initially\n }\n }\n }\n\n condition {\n path_pattern { values = [\"/*\"] }\n }\n}\n```\n\nThe deployment script shifts traffic gradually:\n\n```bash\n#!/bin/bash\n# deploy-blue-green.sh\n\nNEW_AMI=$1\nCURRENT_ENV=$(get_active_environment) # \"blue\" or \"green\"\nTARGET_ENV=$([[ \"$CURRENT_ENV\" == \"blue\" ]] && echo \"green\" || echo \"blue\")\n\necho \"Deploying $NEW_AMI to $TARGET_ENV environment\"\n\n# 1. Launch new instances in the target environment\nupdate_asg_ami \"$TARGET_ENV\" \"$NEW_AMI\"\nwait_for_healthy \"$TARGET_ENV\" 120\n\n# 2. Canary: send 10% of traffic\nset_traffic_weights \"$CURRENT_ENV\" 90 \"$TARGET_ENV\" 10\nsleep 300 # Bake for 5 minutes\n\n# 3. Check error rates\nERROR_RATE=$(get_error_rate \"$TARGET_ENV\" \"5m\")\nif (( $(echo \"$ERROR_RATE > 0.01\" | bc -l) )); then\n echo \"Error rate too high ($ERROR_RATE), rolling back\"\n set_traffic_weights \"$CURRENT_ENV\" 100 \"$TARGET_ENV\" 0\n exit 1\nfi\n\n# 4. Shift to 50/50\nset_traffic_weights \"$CURRENT_ENV\" 50 \"$TARGET_ENV\" 50\nsleep 300\n\n# 5. Full cutover\nset_traffic_weights \"$CURRENT_ENV\" 0 \"$TARGET_ENV\" 100\nsleep 600 # Bake for 10 minutes\n\n# 6. Scale down old environment\nscale_down_asg \"$CURRENT_ENV\" 0\n\necho \"Deployment complete. Active environment: $TARGET_ENV\"\n```\n\n## The Migration Story\n\nMigrating from mutable to immutable took us three months. The hardest parts, in order:\n\n**1. Discovering all the manual changes.** We ran ` debsums ` and ` find / -newer /var/log/installer ` on every production server to catalog what had changed since the original AMI. We found 23 manually installed packages, 8 modified config files, 3 cron jobs added by hand, and a Python script someone had written directly on a production server that was critical to the billing pipeline.\n\n**2. Application configuration.** The app read config from local files that had been hand-edited. We moved everything to environment variables injected via the EC2 instance's user data script and [AWS](<https://aws.com/>) Systems Manager Parameter Store:\n\n```bash\n# User data script, runs on every boot\n#!/bin/bash\nexport APP_DB_HOST=$(aws ssm get-parameter --name /app/db/host --query 'Parameter.Value' --output text)\nexport APP_DB_PASSWORD=$(aws ssm get-parameter --name /app/db/password --with-decryption --query 'Parameter.Value' --output text)\nexport APP_REDIS_URL=$(aws ssm get-parameter --name /app/redis/url --query 'Parameter.Value' --output text)\n\n# Write to environment file\ncat > /opt/app/.env <<EOF\nDB_HOST=$APP_DB_HOST\nDB_PASSWORD=$APP_DB_PASSWORD\nREDIS_URL=$APP_REDIS_URL\nEOF\n\nsystemctl start app.service\n```\n\n**3. Log management.** Servers that get destroyed lose their logs. We set up CloudWatch Logs agent in the golden image to ship logs before the server dies. For post-mortem debugging, we also enabled VPC Flow Logs and ALB access logs to S3.\n\n**4. Team habits.** The hardest change was cultural. Engineers had to stop thinking \"I'll just SSH in and fix it\" and start thinking \"I'll fix the template and rebuild.\" We literally disabled SSH access to production instances (no key pair assigned) to force the change.\n\n## Drift Detection\n\nEven with immutable infrastructure, drift can creep in through user data scripts, EBS volumes that persist across replacements, or someone re-enabling SSH \"just this once.\" I run drift detection weekly:\n\n```bash\n# Compare running instance configuration against the golden AMI\naws ec2 describe-instances \\\n --filters \"Name=tag:Environment,Values=production\" \\\n --query 'Reservations[].Instances[].[InstanceId,ImageId,LaunchTime]' \\\n --output table\n\n# Alert if any instance is running an AMI older than 7 days\nCURRENT_AMI=$(get_latest_ami \"app-server\")\nfor instance in $(get_production_instances); do\n INSTANCE_AMI=$(get_instance_ami \"$instance\")\n if [ \"$INSTANCE_AMI\" != \"$CURRENT_AMI\" ]; then\n alert \"Instance $instance running stale AMI $INSTANCE_AMI (current: $CURRENT_AMI)\"\n fi\ndone\n```\n\n## Wrapping Up\n\nImmutable infrastructure isn't about tools, Packer is just the mechanism. It's about a discipline: servers are cattle, not pets. They're born from an image, they serve requests, and they're replaced. No patches, no hotfixes, no \"I'll just tweak this one thing.\"\n\nThe migration was painful. Three months of archeology on servers that had been accumulating changes for over a year. But the result is worth it: every deployment is a known-good state, every rollback is instant (just point to the previous AMI), and configuration drift is impossible by design.\n\nThe best infrastructure is the kind that doesn't surprise you.\n\n## Frequently asked questions\n\n### What is immutable infrastructure and how does it differ from traditional server management?\n\nIn immutable infrastructure, servers are never modified after deployment. Instead of SSHing in to patch, update, or configure a running server, you build a new image with the changes, deploy fresh instances from that image, and destroy the old ones. This eliminates configuration drift, the state where servers that should be identical have diverged due to ad-hoc changes.\n\n### Is immutable infrastructure practical for databases and stateful services?\n\nFor the compute layer, absolutely. For stateful services like databases, the approach is modified, you use immutable images for the database engine and configuration, but the data volume is persistent and mounted separately. RDS and managed database services essentially do this already. The key insight is separating the immutable compute from the mutable state.\n\n### How do you handle emergency patches with immutable infrastructure?\n\nThe same way as regular deployments, just faster. Your AMI pipeline should be fast enough (under 10 minutes) that building a new image for an emergency patch is viable. If that's too slow, you have two options: keep a pre-built 'hotfix base' image that's always ready, or accept that for true emergencies, you might need to temporarily break immutability and patch in place, but then immediately rebuild the image to prevent drift.\n\n## Tips\n\n1. **LuchoTip:** Never SSH into a production instance to fix something. If you need to fix it, fix the Packer template and rebuild the image. The moment you patch a running server, you've created a snowflake that will bite you during the next deployment.\n2. **LuchoTip:** Build your AMIs in a CI pipeline, not on someone's laptop. Tag each AMI with the git commit SHA that produced it, ` git_sha = abc123 `. When something breaks, you can trace any running instance back to the exact code that built it.\n3. **LuchoTip:** Keep your golden image build under 8 minutes. Anything longer and your team will start looking for shortcuts. I pre-bake a base image weekly (OS patches, runtime) and layer the application on top, the app-specific build takes 90 seconds.\n4. **LuchoTip:** Blue-green deployments need a bake time. Don't cut over immediately, run both environments for at least 15 minutes while monitoring error rates. I once rolled out a memory leak that didn't surface until 12 minutes after deployment.\n\n- immutable-infrastructure\n- packer\n- golden-images\n- ami\n- blue-green-deployments\n- configuration-drift\n- devops\n- infrastructure-as-code\n- deployment\n- automation\n\n[Back to Blog](<https://lucioduran.com/blog>)","headings":[{"level":1,"text":"Immutable Infrastructure: Packer Golden Images, AMI Pipelines, Blue-Green Deployments, and the Real Migration from Mutable Servers","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#title"},{"level":2,"text":"The Core Principle","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#the-core-principle"},{"level":2,"text":"Building Golden Images with Packer","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#building-golden-images-with-packer"},{"level":2,"text":"The AMI Pipeline","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#the-ami-pipeline"},{"level":2,"text":"Blue-Green Deployments","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#blue-green-deployments"},{"level":2,"text":"The Migration Story","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#the-migration-story"},{"level":2,"text":"Drift Detection","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#drift-detection"},{"level":2,"text":"Wrapping Up","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#wrapping-up"},{"level":2,"text":"Frequently asked questions","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#questions"},{"level":3,"text":"What is immutable infrastructure and how does it differ from traditional server management?","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#question-1"},{"level":3,"text":"Is immutable infrastructure practical for databases and stateful services?","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#question-2"},{"level":3,"text":"How do you handle emergency patches with immutable infrastructure?","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#question-3"},{"level":2,"text":"Tips","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#article-tips"}],"sections":[{"heading":"Immutable Infrastructure: Packer Golden Images, AMI Pipelines, Blue-Green Deployments, and the Real Migration from Mutable Servers","level":1,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images","content":"DevOps"},{"heading":"Immutable Infrastructure: Packer Golden Images, AMI Pipelines, Blue-Green Deployments, and the Real Migration from Mutable Servers","level":1,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#title","content":"2025-12-217 min read\n\nLucio Durán\nEngineering Manager & AI Solutions Architect\n\nAlso available in: [Español](<https://lucioduran.com/es/blog/immutable-infrastructure-packer-golden-images>)"},{"heading":"The Core Principle","level":2,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#the-core-principle","content":"Immutable infrastructure means servers are never modified after deployment. The workflow is:\n\n```\nCode Change → Build New Image → Deploy New Instances → Destroy Old Instances\n```\n\nNot:\n\n```\nCode Change → SSH into server → apt-get install → pray\n```\n\nEvery server is born from an image, runs untouched, and dies when replaced. There's no drift because there's no mutation."},{"heading":"Building Golden Images with Packer","level":2,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#building-golden-images-with-packer","content":"[Packer](<https://www.packer.io/>) builds machine images from a declarative template. Here's the production template I use:\n\n```hcl\n# packer/app-server.pkr.hcl\n\npacker {\n required_plugins {\n amazon = {\n version = \">= 1.3.0\"\n source = \"github.com/hashicorp/amazon\"\n }\n }\n}\n\nvariable \"app_version\" {\n type = string\n default = \"latest\"\n}\n\nvariable \"git_sha\" {\n type = string\n}\n\nsource \"amazon-ebs\" \"app_server\" {\n ami_name = \"app-server-${var.app_version}-${formatdate(\"YYYYMMDD-hhmm\", timestamp())}\"\n instance_type = \"t3.medium\"\n region = \"us-east-1\"\n\n source_ami_filter {\n filters = {\n name = \"ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-arm64-server-*\"\n root-device-type = \"ebs\"\n virtualization-type = \"hvm\"\n }\n owners = [\"099720109477\"] # Canonical\n most_recent = true\n }\n\n ssh_username = \"ubuntu\"\n\n tags = {\n Name = \"app-server-${var.app_version}\"\n GitSHA = var.git_sha\n BuildTime = timestamp()\n Base = \"ubuntu-24.04\"\n }\n}\n\nbuild {\n sources = [\"source.amazon-ebs.app_server\"]\n\n # System setup\n provisioner \"shell\" {\n inline = [\n \"sudo apt-get update -y\",\n \"sudo apt-get upgrade -y\",\n \"sudo apt-get install -y curl unzip jq\",\n\n # Install Node.js 22 LTS\n \"curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -\",\n \"sudo apt-get install -y nodejs\",\n\n # Install CloudWatch agent\n \"wget https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/arm64/latest/amazon-cloudwatch-agent.deb\",\n \"sudo dpkg -i amazon-cloudwatch-agent.deb\",\n \"rm amazon-cloudwatch-agent.deb\",\n\n # Security hardening\n \"sudo ufw default deny incoming\",\n \"sudo ufw default allow outgoing\",\n \"sudo ufw allow 22/tcp\",\n \"sudo ufw allow 3000/tcp\",\n \"sudo ufw --force enable\",\n ]\n }\n\n # Application deployment\n provisioner \"file\" {\n source = \"../dist/\"\n destination = \"/tmp/app/\"\n }\n\n provisioner \"shell\" {\n inline = [\n \"sudo mkdir -p /opt/app\",\n \"sudo cp -r /tmp/app/* /opt/app/\",\n \"cd /opt/app && sudo npm ci --production\",\n\n # Create systemd service\n \"sudo tee /etc/systemd/system/app.service > /dev/null <<EOF\",\n \"[Unit]\",\n \"Description=Application Server\",\n \"After=network.target\",\n \"\",\n \"[Service]\",\n \"Type=simple\",\n \"User=www-data\",\n \"WorkingDirectory=/opt/app\",\n \"ExecStart=/usr/bin/node server.js\",\n \"Restart=always\",\n \"RestartSec=5\",\n \"Environment=NODE_ENV=production\",\n \"Environment=PORT=3000\",\n \"\",\n \"[Install]\",\n \"WantedBy=multi-user.target\",\n \"EOF\",\n\n \"sudo systemctl daemon-reload\",\n \"sudo systemctl enable app.service\",\n ]\n }\n\n # Cleanup\n provisioner \"shell\" {\n inline = [\n \"sudo apt-get clean\",\n \"sudo rm -rf /tmp/*\",\n \"sudo rm -rf /var/tmp/*\",\n \"sudo journalctl --vacuum-time=1d\",\n ]\n }\n}\n```\n\nBuild it in CI:\n\n```bash\n# GitHub Actions step\npacker build \\\n -var \"app_version=${GITHUB_REF_NAME}\" \\\n -var \"git_sha=${GITHUB_SHA}\" \\\n packer/app-server.pkr.hcl\n```"},{"heading":"The AMI Pipeline","level":2,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#the-ami-pipeline","content":"The full pipeline looks like this:\n\n```yaml\n# .github/workflows/ami-pipeline.yml\nname: AMI Pipeline\n\non:\n push:\n branches: [main]\n\njobs:\n build:\n runs-on: ubuntu-latest\n outputs:\n ami_id: ${{ steps.build.outputs.ami_id }}\n steps:\n - uses: actions/checkout@v4\n\n - name: Build application\n run: npm ci && npm run build\n\n - name: Run tests\n run: npm test\n\n - name: Build AMI\n id: build\n run: |\n AMI_ID=$(packer build -machine-readable \\\n -var \"app_version=${{ github.ref_name }}\" \\\n -var \"git_sha=${{ github.sha }}\" \\\n packer/app-server.pkr.hcl | \\\n grep 'artifact,0,id' | cut -d: -f2)\n echo \"ami_id=$AMI_ID\" >> $GITHUB_OUTPUT\n\n - name: Validate AMI\n run: |\n # Launch a test instance from the new AMI\n INSTANCE_ID=$(aws ec2 run-instances \\\n --image-id ${{ steps.build.outputs.ami_id }} \\\n --instance-type t3.small \\\n --query 'Instances[0].InstanceId' \\\n --output text)\n\n # Wait for it to be running\n aws ec2 wait instance-status-ok --instance-ids $INSTANCE_ID\n\n # Health check\n IP=$(aws ec2 describe-instances \\\n --instance-ids $INSTANCE_ID \\\n --query 'Reservations[0].Instances[0].PublicIpAddress' \\\n --output text)\n\n curl -f http://$IP:3000/health || exit 1\n\n # Cleanup\n aws ec2 terminate-instances --instance-ids $INSTANCE_ID\n\n deploy:\n needs: build\n runs-on: ubuntu-latest\n steps:\n - name: Update Launch Template\n run: |\n aws ec2 create-launch-template-version \\\n --launch-template-name app-server \\\n --source-version '$Latest' \\\n --launch-template-data '{\"ImageId\":\"${{ needs.build.outputs.ami_id }}\"}'\n\n - name: Blue-Green Deployment\n run: |\n # Start instance refresh (rolling replacement)\n aws autoscaling start-instance-refresh \\\n --auto-scaling-group-name app-server-asg \\\n --preferences '{\n \"MinHealthyPercentage\": 90,\n \"InstanceWarmup\": 120\n }'\n```"},{"heading":"Blue-Green Deployments","level":2,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#blue-green-deployments","content":"The ASG instance refresh is the simplest approach, but for zero-downtime deployments, I use a proper blue-green setup:\n\n```hcl\n# terraform/blue-green.tf\n\nresource \"aws_lb_target_group\" \"blue\" {\n name = \"app-blue\"\n port = 3000\n protocol = \"HTTP\"\n vpc_id = var.vpc_id\n\n health_check {\n path = \"/health\"\n healthy_threshold = 2\n unhealthy_threshold = 3\n interval = 10\n }\n}\n\nresource \"aws_lb_target_group\" \"green\" {\n name = \"app-green\"\n port = 3000\n protocol = \"HTTP\"\n vpc_id = var.vpc_id\n\n health_check {\n path = \"/health\"\n healthy_threshold = 2\n unhealthy_threshold = 3\n interval = 10\n }\n}\n\nresource \"aws_lb_listener_rule\" \"app\" {\n listener_arn = var.listener_arn\n priority = 100\n\n action {\n type = \"forward\"\n forward {\n target_group {\n arn = aws_lb_target_group.blue.arn\n weight = var.blue_weight # 100 initially\n }\n target_group {\n arn = aws_lb_target_group.green.arn\n weight = var.green_weight # 0 initially\n }\n }\n }\n\n condition {\n path_pattern { values = [\"/*\"] }\n }\n}\n```\n\nThe deployment script shifts traffic gradually:\n\n```bash\n#!/bin/bash\n# deploy-blue-green.sh\n\nNEW_AMI=$1\nCURRENT_ENV=$(get_active_environment) # \"blue\" or \"green\"\nTARGET_ENV=$([[ \"$CURRENT_ENV\" == \"blue\" ]] && echo \"green\" || echo \"blue\")\n\necho \"Deploying $NEW_AMI to $TARGET_ENV environment\"\n\n# 1. Launch new instances in the target environment\nupdate_asg_ami \"$TARGET_ENV\" \"$NEW_AMI\"\nwait_for_healthy \"$TARGET_ENV\" 120\n\n# 2. Canary: send 10% of traffic\nset_traffic_weights \"$CURRENT_ENV\" 90 \"$TARGET_ENV\" 10\nsleep 300 # Bake for 5 minutes\n\n# 3. Check error rates\nERROR_RATE=$(get_error_rate \"$TARGET_ENV\" \"5m\")\nif (( $(echo \"$ERROR_RATE > 0.01\" | bc -l) )); then\n echo \"Error rate too high ($ERROR_RATE), rolling back\"\n set_traffic_weights \"$CURRENT_ENV\" 100 \"$TARGET_ENV\" 0\n exit 1\nfi\n\n# 4. Shift to 50/50\nset_traffic_weights \"$CURRENT_ENV\" 50 \"$TARGET_ENV\" 50\nsleep 300\n\n# 5. Full cutover\nset_traffic_weights \"$CURRENT_ENV\" 0 \"$TARGET_ENV\" 100\nsleep 600 # Bake for 10 minutes\n\n# 6. Scale down old environment\nscale_down_asg \"$CURRENT_ENV\" 0\n\necho \"Deployment complete. Active environment: $TARGET_ENV\"\n```"},{"heading":"The Migration Story","level":2,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#the-migration-story","content":"Migrating from mutable to immutable took us three months. The hardest parts, in order:\n\n**1. Discovering all the manual changes.** We ran ` debsums ` and ` find / -newer /var/log/installer ` on every production server to catalog what had changed since the original AMI. We found 23 manually installed packages, 8 modified config files, 3 cron jobs added by hand, and a Python script someone had written directly on a production server that was critical to the billing pipeline.\n\n**2. Application configuration.** The app read config from local files that had been hand-edited. We moved everything to environment variables injected via the EC2 instance's user data script and [AWS](<https://aws.com/>) Systems Manager Parameter Store:\n\n```bash\n# User data script, runs on every boot\n#!/bin/bash\nexport APP_DB_HOST=$(aws ssm get-parameter --name /app/db/host --query 'Parameter.Value' --output text)\nexport APP_DB_PASSWORD=$(aws ssm get-parameter --name /app/db/password --with-decryption --query 'Parameter.Value' --output text)\nexport APP_REDIS_URL=$(aws ssm get-parameter --name /app/redis/url --query 'Parameter.Value' --output text)\n\n# Write to environment file\ncat > /opt/app/.env <<EOF\nDB_HOST=$APP_DB_HOST\nDB_PASSWORD=$APP_DB_PASSWORD\nREDIS_URL=$APP_REDIS_URL\nEOF\n\nsystemctl start app.service\n```\n\n**3. Log management.** Servers that get destroyed lose their logs. We set up CloudWatch Logs agent in the golden image to ship logs before the server dies. For post-mortem debugging, we also enabled VPC Flow Logs and ALB access logs to S3.\n\n**4. Team habits.** The hardest change was cultural. Engineers had to stop thinking \"I'll just SSH in and fix it\" and start thinking \"I'll fix the template and rebuild.\" We literally disabled SSH access to production instances (no key pair assigned) to force the change."},{"heading":"Drift Detection","level":2,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#drift-detection","content":"Even with immutable infrastructure, drift can creep in through user data scripts, EBS volumes that persist across replacements, or someone re-enabling SSH \"just this once.\" I run drift detection weekly:\n\n```bash\n# Compare running instance configuration against the golden AMI\naws ec2 describe-instances \\\n --filters \"Name=tag:Environment,Values=production\" \\\n --query 'Reservations[].Instances[].[InstanceId,ImageId,LaunchTime]' \\\n --output table\n\n# Alert if any instance is running an AMI older than 7 days\nCURRENT_AMI=$(get_latest_ami \"app-server\")\nfor instance in $(get_production_instances); do\n INSTANCE_AMI=$(get_instance_ami \"$instance\")\n if [ \"$INSTANCE_AMI\" != \"$CURRENT_AMI\" ]; then\n alert \"Instance $instance running stale AMI $INSTANCE_AMI (current: $CURRENT_AMI)\"\n fi\ndone\n```"},{"heading":"Wrapping Up","level":2,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#wrapping-up","content":"Immutable infrastructure isn't about tools, Packer is just the mechanism. It's about a discipline: servers are cattle, not pets. They're born from an image, they serve requests, and they're replaced. No patches, no hotfixes, no \"I'll just tweak this one thing.\"\n\nThe migration was painful. Three months of archeology on servers that had been accumulating changes for over a year. But the result is worth it: every deployment is a known-good state, every rollback is instant (just point to the previous AMI), and configuration drift is impossible by design.\n\nThe best infrastructure is the kind that doesn't surprise you."},{"heading":"Frequently asked questions","level":2,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#questions","content":""},{"heading":"What is immutable infrastructure and how does it differ from traditional server management?","level":3,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#question-1","content":"In immutable infrastructure, servers are never modified after deployment. Instead of SSHing in to patch, update, or configure a running server, you build a new image with the changes, deploy fresh instances from that image, and destroy the old ones. This eliminates configuration drift, the state where servers that should be identical have diverged due to ad-hoc changes."},{"heading":"Is immutable infrastructure practical for databases and stateful services?","level":3,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#question-2","content":"For the compute layer, absolutely. For stateful services like databases, the approach is modified, you use immutable images for the database engine and configuration, but the data volume is persistent and mounted separately. RDS and managed database services essentially do this already. The key insight is separating the immutable compute from the mutable state."},{"heading":"How do you handle emergency patches with immutable infrastructure?","level":3,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#question-3","content":"The same way as regular deployments, just faster. Your AMI pipeline should be fast enough (under 10 minutes) that building a new image for an emergency patch is viable. If that's too slow, you have two options: keep a pre-built 'hotfix base' image that's always ready, or accept that for true emergencies, you might need to temporarily break immutability and patch in place, but then immediately rebuild the image to prevent drift."},{"heading":"Tips","level":2,"url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#article-tips","content":"1. **LuchoTip:** Never SSH into a production instance to fix something. If you need to fix it, fix the Packer template and rebuild the image. The moment you patch a running server, you've created a snowflake that will bite you during the next deployment.\n2. **LuchoTip:** Build your AMIs in a CI pipeline, not on someone's laptop. Tag each AMI with the git commit SHA that produced it, ` git_sha = abc123 `. When something breaks, you can trace any running instance back to the exact code that built it.\n3. **LuchoTip:** Keep your golden image build under 8 minutes. Anything longer and your team will start looking for shortcuts. I pre-bake a base image weekly (OS patches, runtime) and layer the application on top, the app-specific build takes 90 seconds.\n4. **LuchoTip:** Blue-green deployments need a bake time. Don't cut over immediately, run both environments for at least 15 minutes while monitoring error rates. I once rolled out a memory leak that didn't surface until 12 minutes after deployment.\n\n- immutable-infrastructure\n- packer\n- golden-images\n- ami\n- blue-green-deployments\n- configuration-drift\n- devops\n- infrastructure-as-code\n- deployment\n- automation\n\n[Back to Blog](<https://lucioduran.com/blog>)"}],"links":[{"text":"Español","url":"https://lucioduran.com/es/blog/immutable-infrastructure-packer-golden-images"},{"text":"Packer","url":"https://www.packer.io/"},{"text":"AWS","url":"https://aws.com/"},{"text":"Back to Blog","url":"https://lucioduran.com/blog"}],"structuredData":[{"@context":"https://schema.org","@graph":[{"@type":"BlogPosting","@id":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images#article","mainEntityOfPage":{"@type":"WebPage","@id":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images"},"headline":"Immutable Infrastructure: Packer Golden Images, AMI Pipelines, Blue-Green Deployments, and the Real Migration from Mutable Servers","description":"A practical guide to immutable infrastructure, building golden images with Packer, automating AMI pipelines, implementing blue-green deployments, eliminating configuration drift, and the honest story of migrating a production system from mutable servers.","datePublished":"2025-12-21","author":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"publisher":{"@type":"Person","@id":"https://lucioduran.com/#person","name":"Lucio Durán","url":"https://lucioduran.com"},"image":"https://statics.lucioduran.com/images/lucioduran-social-card.jpg","url":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images","inLanguage":"en","isPartOf":{"@id":"https://lucioduran.com/blog#blog"},"articleSection":"DevOps","keywords":"immutable-infrastructure, packer, golden-images, ami, blue-green-deployments, configuration-drift, devops, infrastructure-as-code, deployment, automation","wordCount":1320,"timeRequired":"PT7M","workTranslation":[{"@type":"BlogPosting","inLanguage":"es","url":"https://lucioduran.com/es/blog/immutable-infrastructure-packer-golden-images"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://lucioduran.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://lucioduran.com/blog"},{"@type":"ListItem","position":3,"name":"Immutable Infrastructure: Packer Golden Images, AMI Pipelines, Blue-Green Deployments, and the Real Migration from Mutable Servers","item":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images"}]},{"@type":"FAQPage","mainEntity":[{"@type":"Question","name":"What is immutable infrastructure and how does it differ from traditional server management?","acceptedAnswer":{"@type":"Answer","text":"In immutable infrastructure, servers are never modified after deployment. Instead of SSHing in to patch, update, or configure a running server, you build a new image with the changes, deploy fresh instances from that image, and destroy the old ones. This eliminates configuration drift, the state where servers that should be identical have diverged due to ad-hoc changes."}},{"@type":"Question","name":"Is immutable infrastructure practical for databases and stateful services?","acceptedAnswer":{"@type":"Answer","text":"For the compute layer, absolutely. For stateful services like databases, the approach is modified, you use immutable images for the database engine and configuration, but the data volume is persistent and mounted separately. RDS and managed database services essentially do this already. The key insight is separating the immutable compute from the mutable state."}},{"@type":"Question","name":"How do you handle emergency patches with immutable infrastructure?","acceptedAnswer":{"@type":"Answer","text":"The same way as regular deployments, just faster. Your AMI pipeline should be fast enough (under 10 minutes) that building a new image for an emergency patch is viable. If that's too slow, you have two options: keep a pre-built 'hotfix base' image that's always ready, or accept that for true emergencies, you might need to temporarily break immutability and patch in place, but then immediately rebuild the image to prevent drift."}}]}]}],"contentHash":"fcde0d6d05d97c5d0d007ca4dc5a57128b9470e1abd4ef2e0029b4b274a1424b","provenance":{"method":"Deterministic serialization of the server-rendered main element","source":"https://lucioduran.com/blog/immutable-infrastructure-packer-golden-images","hashAlgorithm":"sha256","hashScope":"JSON document fields before contentHash and provenance","sourceHTMLBytes":66692,"contentBytes":13395}}