In the previous 2 posts we described several ways in which sub optimal performance engineering practices manifest themselves, as well as identified the lack of goal commonality between developers and performance engineers as one of the key reasons behind these sub optimal practices. In this post I want to look at the problem from a more holistic and organizational perspective.
Loosing site of the goal
What happens when IT departments lose site of the performance engineering goal? (Reminder in short the goal is to improve the end user’s quality of experience and productivity, while maintaining system costs within budget).
Well what happens is that each department gets lost in its own tactical goal:
The Capacity planning team focuses on efficient and accurate hardware provisioning
The load testing team focuses on test coverage and scale requirements
The network engineering team focuses on the speed and capacity of the pipes
The data base team focuses on the performance of the data bases
The server team focuses on the performance of the backend servers
Desktop team is focusing on the performance of the desktop clients
…
What the organization ends up with is a set of local optimums, but in many cases those local optimums don’t amount to an optimal system. What’s missing in the above list is at least one department that is responsible for meeting the goal, it is very rare to find a team that oversees the end to end responsiveness and performance of the application across all its components from the end user’s perspective. It is even harder to find a team that is held accountable to end user performance.
But is it wrong for each team to improve its domain and make sure it is optimal? Well the counter intuitive answer is yes, it is wrong and for the following reasons:
Focusing on the wrong bottlenecks
Let’s consider the following transaction as an example: this transaction generates a time sheet report for global employees. This transaction is served by a client (web browser with java widgets) a few web servers behind a load balancer, a few application servers and a data base server. Now lets see what happens if there are performance issues with this transaction. Naturally each team will spend time in improving its own domain, so the data base team may index the employee’s data base to reduce the data base response time in half, the server team adds more web servers behind the load balancer to increase the application’s scalability and the network team adds more bandwidth to the data center router. All these steps sound like they should help, no? Well the realistic answer is that in some cases none of these steps help, in fact 3 negative things happen here:
a. The teams spent time and money on the wrong bottlenecks
b. The real bottleneck is still out there
c. Increasing the speed of none bottleneck components places more strain on the real bottleneck, slowing things down even further
In future posts I will give specific examples of several problems that can not be addressed in the realm of one IT department. Those problems usually result from interdependencies between the different systems (servers, networks and data bases). It takes a holistic and multi-disciplinary process to find the right bottleneck let along find a solution for the problem. It may sound complicated, but the concept is quite simple, when dealing with performance, it does little good to focus on local optimums, any optimization effort that is not spent on the actual bottleneck is counter productive and a waste of IT resources and money. Remember, the goal is to improve end user response time at the desktop, not optimize a specific component that is part of a bigger system.
Even though the concept is simple, the solution isn’t always as simple. In future posts, I will offer practical ways to find application performance bottlenecks, I have used them in many engagements and they haven’t failed me yet. But before we can talk about the solutions, it is important to understand the problems so in the next several posts we will cover a few basic performance engineering concepts.
Talk to you soon…
Leave a Reply