Automating AMI Image Builds & Rollouts with Packer and CodeBuild

To illustrate the AMI image build and update process, let's consider a simple web application hosted on AWS. The application follows a scalable and resilient architecture that leverages several AWS services working together:

• Application Layer (Apache Web Server): The application is served by an Apache web server running on Amazon EC2 instances. Each instance uses a custom-built Amazon Machine Image (AMI) that includes the required operating system, web server software, and any application-specific dependencies.
• EC2 Instances and Launch Template (LT): The EC2 instances are launched based on a Launch Template, which defines the instance configuration, including instance type, network settings, security groups, and the custom AMI to use. The Launch Template also includes an initialization script (user data) that runs during instance startup. This script applies the latest OS and application patches, and retrieves the web server configuration and website content from a designated source, such as an Amazon S3 bucket or configuration management repository.
• Auto Scaling Group (ASG): The EC2 instances are managed by an Auto Scaling Group, which automatically adjusts the number of running instances based on traffic load. When demand increases, new instances are launched using the defined Launch Template; when demand drops, unnecessary instances are terminated, ensuring optimal performance and cost efficiency.
• Load Balancer (ALB): An Application Load Balancer (ALB) fronts the Auto Scaling Group and serves as the single entry point for client requests. The ALB distributes incoming HTTP/HTTPS traffic evenly across the available EC2 instances, performs regular health checks to ensure that only healthy instances receive traffic, and provides metrics that can trigger scaling activities in the ASG.

Back to Top

The described configuration provides scalability and automation; however, over time, the number of system and application updates installed during instance startup tends to grow. As more patches accumulate, the initialization phase (executed through startup scripts) takes longer to complete.

This can lead to delays in instance availability, and in some cases, may even cause autoscaling failures if new instances cannot become healthy quickly enough to handle incoming traffic.

To address this issue, it's important to perform regular AMI refreshes, ensuring that the base image always includes the latest OS updates, application patches, and security fixes. This minimizes startup time and ensures new instances are ready to serve traffic as quickly as possible.

Beyond faster deployments, periodic AMI refresh and patching provide several important operational and security benefits:

• Security: Keeping AMIs up to date ensures that all known vulnerabilities (CVEs) are addressed through the latest OS and package updates, reducing exposure to potential exploits.
• Consistency: By using immutable AMIs, every instance launched from the same image starts with an identical configuration, eliminating configuration drift between instances and across environments.
• Operational Simplicity: Instead of running patching or update scripts on live servers, patches and updates are baked directly into the AMI. This approach reduces complexity, minimizes downtime, and simplifies operational management.
• Compliance and Auditability: Many organizations must demonstrate evidence of regular patching and secure build processes during audits. A version-controlled AMI build pipeline provides a clear, repeatable process that satisfies these compliance requirements.

Regular AMI maintenance not only enhances security and reliability but also contributes to faster autoscaling response times, predictable deployments, and simpler operational management across environments.

Back to Top

Packer is a tool developed by HashiCorp that automates the creation of machine images for multiple platforms, including AWS, Azure, Google Cloud, and VMware. It enables teams to build consistent, version-controlled, and pre-configured images, often referred to as golden images, that can be deployed repeatedly across environments.

In an AWS context, Packer integrates directly with Amazon EC2 and the AMI (Amazon Machine Image) service to streamline the image creation process. Instead of manually launching an instance, applying updates, and creating a snapshot, Packer performs all of these steps automatically and in a controlled, reproducible way.

Here's how the process works:

• Builder Phase: Packer uses the AWS builder plugin to launch a temporary EC2 instance based on a defined base AMI (for example, an Amazon Linux or Ubuntu image).
• Provisioning Phase: It then runs one or more provisioners, such as shell scripts or configuration management tools (e.g., Ansible, Chef, or Puppet), to install software packages, apply updates, configure the operating system, and deploy application components or dependencies.
• Image Creation: Once the instance is fully configured, Packer creates a new Amazon Machine Image (AMI) that captures the system's exact state.
• Cleanup Phase: The temporary EC2 builder instance is terminated automatically, leaving behind only the newly created AMI. This ensures a clean and efficient build process without manual intervention.

The result is a repeatable, versioned AMI that can be tested, validated, and promoted through different environments, for example, from development to staging and production, ensuring consistency and traceability across deployments.

Back to Top

The following workflow describes the end-to-end process for maintaining and deploying up-to-date, production-ready Amazon Machine Images (AMIs) using Packer, integrated with existing Launch Template (LT) and Auto Scaling Group (ASG).

1. Build - Create the Updated AMI: Use Packer to build a new Amazon Machine Image that includes the latest operating system patches, security updates, software dependencies, and the web application code.
2. Create - Register a New Launch Template Version: Once the new AMI is built, create a new version of the existing Launch Template that references the new AMI ID. All other instance configuration parameters, such as instance type, networking, and security groups, can remain unchanged.
3. Test - Validate the New Image: Before deployment, perform functional and integration testing by launching a temporary test instance using the new Launch Template version. Run a predefined set of validation checks, such as:
   • HTTP endpoint availability and response verification
   • Application dependency checks (database connectivity, etc., if any)
   • Security and compliance baseline scans
4. Rollout - Refresh Auto Scaling Group Instances: Initiate an ASG instance refresh operation to gradually replace existing instances with ones launched from the new Launch Template version. Use a Desired Configuration that references the new version and enable:
   • Auto Rollback to automatically revert to the previous template if health checks fail.
   • Conservative rollout settings, such as a higher MinHealthyPercentage and an appropriate InstanceWarmup time to ensure stable transitions and minimize user impact.
   This step ensures zero-downtime deployment, allowing traffic to continue flowing through the Application Load Balancer (ALB) while new instances are provisioned, warmed up, and validated.
5. Promote - Finalize and Update Configuration: After a successful instance refresh and post-deployment validation:
   • Set the new Launch Template version as the default.
   • Validate and if necessary update the Auto Scaling Group configuration to permanently reference the new template version.

This workflow combines Packer's image-building automation with AWS-native deployment mechanisms (Launch Templates and Auto Scaling Groups) to deliver a repeatable, controlled, and secure image lifecycle management process.

Back to Top

The following sections provide minimal, copy-paste friendly examples for each component. These are intentionally compact - adapt them to meet your needs.

During the provisioning phase, Packer executes a shell script (or multiple scripts) to install and configure software, apply updates, and prepare the system image for use.

This script defines how your custom AMI is built, including which packages, tools, and configurations are installed, and therefore should be carefully designed to ensure consistency and reliability.

Below is an example structure and some general recommendations for a Packer provisioning shell script.

File: img_build_v24.sh

Recommendations

• Customize the script for your environment: Adapt the provisioning script to include all required software packages, libraries, modules, and tools specific to your base OS, application, or workload. For example, you might need to install Python dependencies, monitoring agents, or custom runtime components.
• Make the script fully non-interactive: The provisioning process must be completely automated and non-interactive, as Packer runs unattended builds. Ensure that no command waits for user input during execution.
• Test before integrating with Packer: It's strongly recommended to test the provisioning script on a standalone EC2 instance first. Confirm that it runs end-to-end without errors and produces a properly configured environment before incorporating it into the Packer workflow. This helps identify missing dependencies or environment-specific issues early, reducing build-time failures.
• Keep it modular and version-controlled: Store your script in a version control system (e.g., Git), and consider splitting it into multiple logically separate parts (e.g., install.sh, configure.sh, cleanup.sh, etc.) to make maintenance easier.
• Include security and cleanup steps: Remove temporary files, credentials, or installation artifacts to minimize the final AMI size and reduce security risks.

The provisioning shell script is a critical component of the Packer AMI build process. By keeping it non-interactive, modular, and thoroughly tested, teams can ensure that each image build is consistent, secure, and production-ready, forming a reliable foundation for automated EC2 instance deployments.

Back to Top

Packer uses a HashiCorp Configuration Language (HCL)-based configuration file to define how an Amazon Machine Image is built, provisioned, and published.

File: ubuntu-ami.pkr.hcl

Main components of this configuration:

• Packer Block: The packer block configures Packer version requirements and specifies which plugins to install upon initialization.
• Variable Blocks: The variable blocks define variables within your Packer configuration.
• Source Block: The source block defines how and where Packer will build the image. For AWS, the amazon-ebs builder plugin launches a temporary EC2 instance, provisions it, creates an EBS-backed AMI, and then terminates the instance.
  Key amazon-ebs parameters include:
  • ami_name / ami_description - the name and description of the resulting AMI.
  • region - the AWS region where the build takes place.
  • instance_type - instance size for the temporary builder instance.
  • source_ami_filter - the base image used as a starting point.
  • iam_instance_profile - the IAM instance profile with permissions required for AMI creation.
  • ssh_username and communicator - specifies how Packer connects to the builder instance.
• Build Block: The build block orchestrates the provisioning and post-processing phases. It references one or more sources and defines what should happen after the base instance is launched.
  Typical build components:
  • provisioner "shell" - runs scripts or inline commands to install software and configure the system..
  • post-processor "manifest" - generates metadata files listing AMI details (e.g., ID, region, creation time).

Defining a Source AMI Filter

The source_ami_filter block specifies which base image Packer should use for building the new AMI. most_recent = true ensures exactly one AMI is returned by the filter.

Use the aws ec2 describe-images command to fine-tune the AMI filter:

Packer VPC Connectivity Requirements

By default, Packer automatically selects a default VPC and subnet in the target region if none are specified. However, it's often best practice to explicitly define vpc_id and subnet_id or subnet_filter { ... } within the amazon-ebs block to ensure builds occur in a controlled network environment.

If your account does not have a default VPC, explicitly define vpc_id and subnet_id:

Security Group Behavior

If no security group is specified:

• Packer automatically creates a temporary security group allowing SSH (port 22) access from the builder host or access for Session Manager.
• The temporary group is automatically deleted after the build completes.

You can override this by providing either security_group_id or security_group_ids:

IAM Requirements

Packer uses two distinct sets of permissions:

• Instance Profile Permissions (for the temporary builder instance): The EC2 instance launched by Packer must have access and permissions to:
  • Download and install software packages and patches, etc.
  • Access any required repositories or S3 buckets.
  • Upload build logs or artifacts, if applicable.
  • Utilize Session Manager, if applicable.
• IAM User/Role Permissions (for Packer itself): The user or role executing Packer must have permission to launch temporary EC2 instances, apply provisioning steps, create AMIs, and clean up resources afterward.

Passing Values to Packer Configuration Variables

Packer supports multiple ways to provide values for input variables defined in the configuration (for example, region, AMI name, or instance profile). This flexibility allows you to adapt the same template to different requirements without modifying the HCL source files.

Below are the most common methods for passing variable values:

• Default values in the configuration file:
  Variables can include a default value directly in the configuration. This approach is convenient for parameters that rarely change, such as a default region.

  variable "region" {
    type = string 
    default = "us-east-1" 
  }

• Command-line arguments:
  You can override any variable at build time using the -var flag:

  packer build -var "region=us-west-2" -var "instance_profile=packer-builder-role" ubuntu-ami.pkr.hcl

  This method is useful for one-time overrides or CI/CD pipelines.
• Variable definition files (.pkrvars.hcl):
  For repeatable builds or environment-specific configurations, you can store variables in an external file:

  # dev.pkrvars.hcl 
  region = "us-east-1" 
  instance_profile = "packer-dev-role"

  Then pass it to Packer using:

  packer build -var-file=dev.pkrvars.hcl ubuntu-ami.pkr.hcl

• Environment variables:
  Packer automatically reads variables defined as PKR_VAR_name. For example:

  export PKR_VAR_region=us-central1 
  packer build ubuntu-ami.pkr.hcl
  

  This approach is particularly useful in automated CI/CD environments where sensitive values (like credentials or role names) should not be stored in plain files.
• Interactive prompts (optional):
  If a variable has no default and is not provided via CLI, file, or environment variable, Packer prompts the user to input a value interactively. Such approach can be useful during development and testing; for automated builds all required variables must be pre-supplied.

Testing Packer Configuration with AWS CloudShell

AWS CloudShell provides a convenient, browser-based shell environment pre-configured with the AWS CLI, credentials, and permissions associated with your AWS account. It's an ideal environment for quickly testing and validating your Packer configuration.

Below are the high-level steps and example commands that can be used to validate and run your Packer build in AWS CloudShell.

• Install Packer:
  Packer is not preinstalled in CloudShell, so you must install it manually. Follow the Packer installation steps for Amazon Linux:

  # Install yum-config-manager
  sudo yum install -y yum-utils
  
  # Add HashiCorp repo
  sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
  
  # Install Packer
  sudo yum -y install packer
  
  # Verify
  packer version 
  

• Upload Your Configuration Files:
  Upload your main configuration file (e.g., ubuntu-ami.pkr.hcl), variable files (if any), and provisioning scripts (e.g., img_build_v24.sh) into CloudShell.
• Initialize Packer and Validate the Configuration:
  Before running a build, validate the syntax and configuration. Make sure to supply values for Packer variables using one of methods described earlier. Run the following commands from the directory containing the HCL file(s) and provisioning script(s):

  packer init .
  packer validate .

• Run a Test Build:
  Execute the Packer build process to test AMI creation. Make sure to supply values for Packer variables using one of methods described earlier. Run the following command from the directory containing the HCL file(s) and provisioning script(s):

  packer build . 

  Upon successful completion, Packer will display the ID of the created AMI.
• Verify the AMI:
  If desired, create a test EC2 instance using the new AMI and validate it configuration.
• Clean Up:
  After testing, consider removing the AMI and associated snapshot to avoid unnecessary storage costs.

Note on Permissions: Since Packer runs within AWS CloudShell and inherits permissions from the logged-in IAM user or role, the user must have sufficient privileges to perform all actions required by Packer, such as launching temporary EC2 instances, creating AMIs, and managing related resources.

Back to Top

Upon build completion, Packer generates a manifest file contains metadata for the newly created AMI. The next steps are to extract the new AMI ID, create a new Launch Template (LT) version referencing it, and then launch a temporary EC2 instance for validation using this new LT version.

• Extract the AMI ID:

  # Extract new AMI ID from Packer manifest file 
  AMI_ID=$(jq -r '.builds[-1].artifact_id | split(":") | .[1]' manifest.json) 
  echo "New AMI ID: $AMI_ID"
  

• Create a new Launch Template version:

  # Assuming LT_ID and REGION are predefined environment variables 
  LT_NEW_AMI=$(echo "{\"ImageId\":\"${AMI_ID}\"}")
  LT_DATA=$(aws ec2 create-launch-template-version \
    --launch-template-id "${LT_ID}" \
    --source-version "\$Latest" \
    --launch-template-data "${LT_NEW_AMI}" \
    --version-description "Updated AMI ID $(date +%Y-%m-%d)" \
    --region "${REGION}" \
    --output json)
  LT_NEW_VERSION=$(echo "${LT_DATA}" | jq -r '.LaunchTemplateVersion.VersionNumber')
  echo "New LT Version: $LT_NEW_VERSION"
  

• Launch a temporary test instance for AMI validation:

  # Launch a temporary test instance 
  INSTANCE_ID=$(aws ec2 run-instances \
    --launch-template LaunchTemplateId="$LT_ID",Version="$LT_NEW_VERSION" \
    --query "Instances[0].InstanceId" \
    --region "${REGION}" \
    --output text)
  echo "Launched test instance: $INSTANCE_ID"
  

After the instance starts, you can retrieve its public IP and use the Site Test Python Script described in the next section to validate the deployment before updating the Auto Scaling Group.

Back to Top

After building a new AMI, it's a good practice to automatically verify that the web server is correctly configured and serving expected content before deploying it to production. The following simple Python script demonstrates how to perform this validation by checking that specific pages return the expected text. The script accepts the base site URL as an input parameter and tests each page defined in the pages dictionary.

This lightweight test ensures that your web server responds correctly before proceeding with rollout. You can easily modify it to check more pages, perform additional health checks, or integrate with your preferred testing framework. Alternatively, you can replace it with your own custom validation script.

File: site_test.py

Back to Top

Once the AMI is verified, the next step is to update your Auto Scaling Group (ASG) to use the new Launch Template version. Instead of modifying the ASG configuration directly, you can perform an instance refresh with the new desired configuration.

This rolling update strategy ensures that:

• New instances are launched using the new LaunchTemplate version with the updated AMI.
• The Auto Scaling Group maintains at least MinHealthyPercentage of healthy instances at all times (no downtime).
• If the refresh completes successfully, the ASG will retain the configuration with the new LaunchTemplate version.
• If the refresh fails (for example, new instances fail health checks), automatic rollback restores the previous configuration.

Once the rollout is complete, you can mark the new Launch Template version as the default:

Back to Top

To automate this entire pipeline - AMI creation, testing, and rollout, you can implement it in an AWS CodeBuild project. CodeBuild provides a managed build environment where you can run Packer, Python site test, and AWS CLI commands within a single repeatable workflow.

Key Steps to Implement

1. Create a CodeBuild Project - define build environment configuration and variables (e.g., REGION, LT_ID, ASG_NAME).
2. Assign IAM Role - the CodeBuild service role must have permissions for EC2, Launch Template, Auto Scaling, S3 (if used for logs/artifacts)
3. Integrate with CodeCommit - store your Packer files, shell scripts, and buildspec.yml in a CodeCommit repository.
4. Enable VPC Access - if your build needs to reach private subnets or EC2 metadata.
5. Create a build spec file - include all image build and validation steps.

Below is an example build spec file which:

• runs Packer
• extracts the AMI ID from the Packer manifest
• creates a new Launch Template version
• launches a test instance, runs tests
• starts an Auto Scaling Group instance refresh with DesiredConfiguration and AutoRollback
• performs cleanup

To adapt this file for your environment, adjust environment variables, Launch Template and ASG configurations, etc.

File: buildspec.yml

Back to Top

List available AMIs:

aws ec2 describe-images \
 --filters "Name=name,Values=ubuntu/images/*/ubuntu-*-24.04-amd64-server-*" \
           "Name=architecture,Values=x86_64" \
           "Name=root-device-type,Values=ebs" \
 --query 'Images[?CreationDate>=`2025-01-01`].[Name, ImageId, ImageOwnerAlias]' \
 --output text | sort

Extract default Launch Template version number:

aws ec2 describe-launch-templates
 --launch-template-ids ${LT_ID} \
 --region ${REGION} \
 --query 'LaunchTemplates[0].DefaultVersionNumber'
 --output text

Backup ASG configuration to S3:

aws autoscaling describe-auto-scaling-groups \
 --auto-scaling-group-names ${ASG_NAME} \
 --region ${REGION} \ 
 > asg-backup-${ASG_NAME}-$(date +%F-%H%M%S).json 
aws s3 cp asg-backup-*.json s3://my-asg-backups/

Back to Top

This article showed a complete, repeatable, and safe process to patch OS-level vulnerabilities by rebuilding AMIs with Packer, validating them, and rolling them into production using Launch Template versions and Auto Scaling Group instance refresh. The approach balances zero-downtime rollout, automatic rollback, and traceability - ideal for secure, auditable operations.

Back to Top

See Also:
Terraform Tutorials