Identifying Hotspots

There are standard established techniques for scaling any distributed system like horizontal scaling, query optimization, caching etc. A shared distributed cache forms an integral part of any scalable distributed system and can provide huge benefits in both latency and throughput by acting as a very fast middleman between the application and the database.

However, one of the challenges for caching is to identify the hotspots in code where the benefits for caching would be maximized. Also, sometimes, even the fastest shared cache is not enough for getting desired performance primarily due to fact that the code path being cached is so hot that even the fastest cache becomes a bottleneck. That’s where memoization comes in, where you memoize the information in the application itself and trade off some more memory for performance.

This article outlines some of the experiements we did for identifying the hotspots in our code and how we mixed memoization with distributed caching to achieve the desired performance.

Although, this approach tells us that there might be inefficiencies in specific code paths, it does not give us enough granularity to identify specific parts of code that can be cached or memoized for better performance.

Looking at the files sorted by Avg. Hits/Line and diving into that source file for the coverage numbers reveals the hottest code paths in that file. An example is:

Now, we can target these hotspots for further optimizations like reducing complexity, caching and/or memoization.

There is no golden rule for memoization vs caching choice but caching is generally the preferred choice, especially in a distributed system as the same cache can be used across multiple machines and over a larger time duration.

After caching the computations and queries being done in hotspots, we decided to profile our app and rake tasks again, and much to our surprise the reads from redis cache still showed up as top hotspots in multiple occassions.

That’s when we realized that we would have to go with a combination of caching and memoization, sacrificing some more memory for performance. This enables us to eliminate hotspots in cases where same computation or query is being done in the same process more than once, while the shared redis cache still enables the newer runs to perform those queries faster for the first time.

By using this hybrid caching approach, we were able to eliminate hotspots at some crucial points in our code and were able to reduce having to do network calls to database and to shared cache down by upto 99% in worst case. We were able to optimize the real time path of our Rails app and cut down the worst case query times by more than 50%. Our rake tasks are the ones that benefit mostly from this hybrid caching approach where we were able to reduce the runtimes of some of the worst offenders by upto 60%.

Finding hotspots in code requires mixing multiple approaches but it’s worth the effort and can provide huge benefits. Caching is a great resource for improving performance characteristics but there are often cases where it’s just not sufficient and needs to be supplemented by memoization.

Related posts:

This week FoondaMate has teamed up with the South African Depression and Anxiety Group (SADAG) to help increase access to mental health help in South Africa. SADAG is an incredible group working to…

The role of a stepmother is no longer some obscure idea idea of a potentially wicked woman.

Nexty is a platform based on blockchain technology to help the e-commercial startups to raise funds from community. The startups themselves would create the ecosystem for Nexty. Totally possible…