Business challenge

In large companies, traditional processes are often unchallenged for years — “if it isn’t broken, don’t fix it.” This mindset provides fertile ground for potential improvements and cost optimization. Our project for a British company (NDA) exemplifies this scenario. 

The critical task involved creating Unity-based build artifacts converted into Amazon Machine Images (AMIs). These AMIs are essential for generating high-quality virtual reality (VR) and 360-degree rendered images, aiding customers in visualizing their future room designs. 

Adding new visual assets required new builds and AMIs, but the automation didn’t clean up old AMIs and snapshots. This oversight caused a steady rise in monthly AWS costs, unnoticed due to the project’s large scale across five AWS accounts in different regions.

Our solution to reduce AWS Cloud costs

Initial analysis

The DevOps team initiated an in-depth analysis of workloads, collaborating closely with the partner’s 3D development team. This collaboration revealed gaps in the development pipeline and highlighted the importance of eventual cleanup activities.

Addressing Amazon Machine images proliferation

After consulting the development team, it was clear that retaining old AMIs wasn’t usually necessary, as rollback scenarios were uncommon. The client maintained 16 development and 4 production environments, creating over 300 monthly builds across three technologies (VR, HQ, 360). By November 2022, the client’s accounts hosted over 3500 AMIs with snapshots.

The image creation process is fully automated using Jenkins CI alongside Packer and Ansible. Packer sets up an instance, which is then configured by an Ansible playbook, resulting in an AMI and disk snapshot combination. This entire process typically takes around 2-3 hours.

To address this, we implemented a shell script that runs at the end of every build and deletes unnecessary AMIs. 

The team decided to retain only the following:

• the last AMI plus the newly created one for non-production environments,
• the last four AMIs, plus the newly created one for production environments. 

Quick win – this straightforward script reduced the number of AMIs to 52 across all accounts.

Snapshot management

However, the initial solution only partially resolved the problem due to technical limitations. Deregistering AMIs and deleting their snapshots could not be handled in a single script. 

To address this, we created a Lambda function to scan accounts for unattached snapshots and delete them. 

Written in Python and deployed using the Serverless Framework, this function was scheduled to run daily, maintaining a low level of unused snapshots.

Leveraging AMI copy functionality

We knew that we could squeeze even more. Further optimization revealed that builds with identical source code were triggered separately for each deployment environment, leading to redundant AMIs. 

By leveraging Packer’s AMI copy functionality, AMIs could be created in one account and accessed by others. This reduced the time and resources required for build creation and enabled simultaneous deployment of visual features across different regions.

A one-time EBS cleaning allowed maintaining this cost permanently, as the client previously lacked mechanisms to reduce unnecessary snapshots.

Outcome – $32,000 savings! 

The implemented cost optimizations showed significant savings. 

For EBS-backed storage, costs were reduced from approximately $14,000 to $4,000, saving over 70% on storage. 

EC2 compute costs were reduced from around $54,000 to $32,000, saving 50% on total compute costs compared to the previous months. 

We achieved those results by reducing the number of instances used for AMI creation, consolidating build pipelines, and cutting down Jenkins workers required.

The strategic AWS Cloud cost optimizations streamlined the workflow. They delivered substantial cost savings for our partner, demonstrating the power of custom and targeted cloud cost management in complex, big-scale solutions.