How we slashed AWS cloud costs by 70% on storage and 50% on compute in a large company

AWS cloud costs
Przemysław Gacka

Entrusting TSH with a swift one-week sprint was all it took for our client to ensure AWS cloud cost savings. Our team significantly reduced monthly expenses by addressing inefficiencies and implementing targeted cleanup strategies, showcasing the impact of agile and focused cloud cost optimization.

Business challenge

In large companies, traditional processes are often unchallenged for years — “if it isn’t broken, don’t fix it.” This mindset provides fertile ground for potential improvements and cost optimization. Our project for a British company (NDA) exemplifies this scenario. 

The critical task involved creating Unity-based build artifacts converted into Amazon Machine Images (AMIs). These AMIs are essential for generating high-quality virtual reality (VR) and 360-degree rendered images, aiding customers in visualizing their future room designs. 

Adding new visual assets required new builds and AMIs, but the automation didn’t clean up old AMIs and snapshots. This oversight caused a steady rise in monthly AWS costs, unnoticed due to the project’s large scale across five AWS accounts in different regions.

Make your cloud operations effective and predict costs with an official AWS Advanced Partner

Our solution to reduce AWS Cloud costs

Initial analysis

The DevOps team initiated an in-depth analysis of workloads, collaborating closely with the partner’s 3D development team. This collaboration revealed gaps in the development pipeline and highlighted the importance of eventual cleanup activities.

Addressing Amazon Machine images proliferation

After consulting the development team, it was clear that retaining old AMIs wasn’t usually necessary, as rollback scenarios were uncommon. The client maintained 16 development and 4 production environments, creating over 300 monthly builds across three technologies (VR, HQ, 360). By November 2022, the client’s accounts hosted over 3500 AMIs with snapshots.

The image creation process is fully automated using Jenkins CI alongside Packer and Ansible. Packer sets up an instance, which is then configured by an Ansible playbook, resulting in an AMI and disk snapshot combination. This entire process typically takes around 2-3 hours.

To address this, we implemented a shell script that runs at the end of every build and deletes unnecessary AMIs. 

The team decided to retain only the following:

  • the last AMI plus the newly created one for non-production environments,
  • the last four AMIs, plus the newly created one for production environments. 

Quick win – this straightforward script reduced the number of AMIs to 52 across all accounts.

Snapshot management

However, the initial solution only partially resolved the problem due to technical limitations. Deregistering AMIs and deleting their snapshots could not be handled in a single script. 

To address this, we created a Lambda function to scan accounts for unattached snapshots and delete them. 

Written in Python and deployed using the Serverless Framework, this function was scheduled to run daily, maintaining a low level of unused snapshots.

Leveraging AMI copy functionality

We knew that we could squeeze even more. Further optimization revealed that builds with identical source code were triggered separately for each deployment environment, leading to redundant AMIs. 

By leveraging Packer’s AMI copy functionality, AMIs could be created in one account and accessed by others. This reduced the time and resources required for build creation and enabled simultaneous deployment of visual features across different regions.

EBS snapshot AWS cost cut
This chart displays the total monthly cost of storing EBS Snapshots in customer accounts, with the cost represented on the left axis and individual months listed along the bottom axis.

A one-time EBS cleaning allowed maintaining this cost permanently, as the client previously lacked mechanisms to reduce unnecessary snapshots.

Outcome – $32,000 savings! 

The implemented cost optimizations showed significant savings. 

For EBS-backed storage, costs were reduced from approximately $14,000 to $4,000, saving over 70% on storage. 

EC2 compute costs were reduced from around $54,000 to $32,000, saving 50% on total compute costs compared to the previous months. 

We achieved those results by reducing the number of instances used for AMI creation, consolidating build pipelines, and cutting down Jenkins workers required.

The strategic AWS Cloud cost optimizations streamlined the workflow. They delivered substantial cost savings for our partner, demonstrating the power of custom and targeted cloud cost management in complex, big-scale solutions.

Struggling with high AWS costs?

Book our AWS Cost Cut Sprint and leverage the expertise of over 130 AWS-certified professionals. Start optimizing your cloud spending today!

Report
State of Frontend 2024

👨‍💻 Help the Frontend community! Answer the State of Frontend 2024 global survey. Takes less than 10 mins.

I want to help

The Software House is promoting EU projects and driving innovation with the support of EU funds

What would you like to do?

    Your personal data will be processed in order to handle your question, and their administrator will be The Software House sp. z o.o. with its registered office in Gliwice. Other information regarding the processing of personal data, including information on your rights, can be found in our Privacy Policy.

    This site is protected by reCAPTCHA and the Google
    Privacy Policy and Terms of Service apply.

    We regard the TSH team as co-founders in our business. The entire team from The Software House has invested an incredible amount of time to truly understand our business, our users and their needs.

    Eyass Shakrah

    Co-Founder of Pet Media Group

    Thanks

    Thank you for your inquiry!

    We'll be back to you shortly to discuss your needs in more detail.