From bottlenecks to seamless communication. Mastering message handling and optimizing resource utilization with real-time observability

Business challenge

The application’s success depended on prompt user notification delivery, especially of new offers via email. However, performance analysis identified a bottleneck: the consumers tasked with managing events from RabbitMQ.

Like many businesses, our Saudi Arabian client faced a common dilemma: efficiently handling a deluge of messages while ensuring timely delivery to customers.

With notifications piling up in RabbitMQ queues, the pressure was on to streamline operations and keep the flow of communication moving.

But that was just the tip of the iceberg. Our client’s infrastructure, powered by Kubernetes, needed to adapt on the fly to fluctuating demands. The client sought application and infrastructure observability in response to the following:

• increased website traffic (incl. sudden surges),
• need to generate real-time reports or other business events at specific times,
• unpredictable flow of daily operations.

Plus, the client needed a quick heads-up about any anomalies in their system, so we made sure those notifications were sent ASAP to the relevant teams.

The observability process

A data-gathering RabbitMQ-Exporter

To tackle these challenges head-on, we chose RabbitMQ-Exporter software. This nifty tool allowed us to gather real-time data on message volume, feeding it into Prometheus for analysis with lightning speed. 

Armed with this wealth of information, our teams were able to proactively manage workload distribution and ensure that critical notifications reached their destinations.

Kubernetes and Prometheus

Kubernetes and Prometheus formed a dynamic ecosystem where resources could scale up or down quickly. When additional Kubernetes cluster resources were necessary, the system seamlessly included them. No more graveyard shifts for the team. Now, they operate on autonomous 24/7 system management.

Turning Insights into Results

With RabbitMQ-Exporter seamlessly integrated into the active RabbitMQ cluster, we didn’t just accumulate data – we turned it into actionable insights. By relaying important information to development teams via email and Slack, we kept everyone in the loop and ready to act immediately.

Scaling Smarter, Not Harder

Leveraging Prometheus data, Kubernetes autonomously generated custom metrics that informed the application’s scaling behavior. 

When message volume surged, our system seamlessly included additional Kubernetes cluster resources, optimizing cost efficiency with spot instances. As messages slowed, Kubernetes axed extra instances, optimizing resource use.

Project outcome

The impact of our solution was transformative. Gone are the days of frantic firefighting and manual interventions.

1. With improved message handling and autoscaling capabilities, our client’s system operates smoothly, effortlessly handling a surge in concurrent events without disruption.
2. The implemented solution seamlessly manages a significantly increased volume of concurrent events without requiring changes to the application code or manual intervention. 
3. Scalability operations run autonomously 24/7/365. Designated personnel receive timely notifications in cases of anomalies or issues, enabling swift remedial actions through dedicated Grafana dashboards.
4. Anomalies are detected and addressed in real-time through timely notifications via email and Slack. 
5. Custom metrics guide decision-making and resource utilization for cost efficiency.

In the fast-paced landscape of modern business, agility isn’t just an advantage – it’s a necessity. Yet, many organizations find themselves shackled by the constraints of outdated systems, struggling to keep pace with the demands of today’s dynamic marketplace.

The Sensible Observability Score

Our custom test estimates your ability to measure and analyze your system’s data. In a few minutes, you’ll get the first measurable review of your company’s data capabilities.