Business Process Test Plan – Update employee record

However this series of posts is about the day after a data center move. Now that the data center is remote, how does this paradigm shift impact the way we should develop, test, deploy, monitor and troubleshoot applications. I will try to cover as many topics as possible, but the main focus is still going to be around the role application performance management plays in this new paradigm.

I will start by covering the key reasons behind the performance impact that is experienced when applications are hosted in a remote data center? Those reasons are fairly intuitive, but it is important to understand them in depth in order to adequately plan for those new conditions. Two main things impact how applications perform when application servers are hosted in a remote data center vis a vis their application clients :

1. The performance of the network link between the client and the remote data center. This performance is defined by a set of network performance metrics that are application independent (for now we will ignore application aware networks, however the following basic concepts still hold in this scenario as well).

2. The application efficiency, specifically how efficient the application is when transferring data between the client and the remote server (and other tiers if applicable). This is an application attribute (and some time an attribute of a specific business process within the application). These attributes are application specific and are independent of any underlying network.

Let’s start with understanding the network performance metrics. Consider the following scenario:

An application is hosted in a NYC data center, with users in 2 places, some are in a NYC headquarters next to that NYC data center and some are in a remote branch that is located in San Francisco. The question is: “will the application perform the same for both type of users (local users in headquarters and remote users in SF)? In other words will the application be as responsive to the San Francisco user as it is to the NYC user?”

Well the obvious answer is NO, in most cases a NYC user will enjoy a faster more responsive application. What is less obvious is why? What is it about the network that causes remote users to experience a slower application than local users? The rest of this post will cover that question, future posts will address the application specific attributes. Once we cover that we will be ready to examine best practices in building applications for a remote data center.

When I ask this question during my training seminars, I get a variety of answers, many of them are the right ones, but I would like to address one wrong answer that keeps repeating itself for some reason.

Collisions – there is a general conception that collisions are common phenomena on the network which can explain any bad thing that happens to applications. The truth is that collisions are almost a thing of the past (on Enterprise LANs anyhow) and even when they happen they can’t explain why a remote user has a worst experience than a local user as both will experience a similar collision chance since collisions is a phenomenon that happens on local area Ethernet networks. If there are collisions on the Enterprise LAN it usually points to a configuration issue on a network device (like a duplex miss-match) but is still unrelated to the answer to our question.

Now to the right answers to the question, what is it about the Wide Area Network that causes applications to slow down:

There are 5 key conditions that predominately exist on Wide Area Networks and impact application performance, each in their own way:

1. Network Latency – the time it takes a packet to traverse from a source to the destination across the network, measured in milliseconds [msec]. A typical WAN link will introduce latency in the range of 10msec – 500 msec.

2. Bandwidth constraints – how fast can data be processed by the network link, measured in bits per second [bps, Kbps, Mbps, Gbps]

3. Bandwidth utilization (background traffic) – the percentage of bandwidth that is utilized by traffic that already exists on the link (background traffic).

4. Jitter – the deviation of the inter packet gap of sequential packets across a network link, it is a result of the deviation of the network latency and is sometimes used interchangeably with that standard deviation, measured in milliseconds [msec].

5. Packet Loss – the chance to drop a packet across an end to end network link, measured in %. Sometimes presented as the inverse metric called packet delivery rate.

The above are called network impairments, you can click on each one of the links to learn more about them and their causes.

Network impairments are performance conditions that inhibit the flow of data across a network. Each impairment type has an impact on the performance of business applications and network services. Some applications may be very sensitive to network impairments and some may be almost network agnostic. Sorting applications based on their network sensitivity is one of the important steps in performance engineering

In the next post we will discuss how application design can impact performance across the network. But in the mean time I would like to introduce a question for the group:

“We identified network latency as one of the key reasons that impact application performance; we also said that a typical WAN link will introduce 10 – 500 msec of latency. The question is, why does network latency have a big impact on application performance? surely a user doesn’t notice an increase of a few msec in response time, even 500 msec = ½ second goes by in a flinch. So think about it and let me know what you found based on your experience, why does network latency have such a big impact on application performance?”

Until next time,

Amichai Lesser

Which is why I was glad to stumble upon Shamus McGillicuddy article in www.searchenterpriseWAN.com titled “WAN engineers prepare networks as cloud computing adoption rises” that can be found here.

There are some good observations in the article, I especially appreciate the comment on how application architecture best practices such as minimizing application turns become key when developing applications for the Cloud as well as testing applications for network latency becomes critical as the enterprise is migrating more and more applications to the Cloud.

Many questions still remain though and the impact of Cloud computing on performance engineering is still to be determined. Some of the questions that are on my mind are: (and feel free to chime in with more questions or attempts at answers)

1. How do you manage the performance of an application in the Cloud?

2. What changes if any are needed to the service deployment model and version upgrades?

3. How do you conduct a performance test for an application or a service in the Cloud (internal clouds and external clouds)?

4. How do you troubleshoot a performance issue in the Cloud?

Something to think about…

Best,

Amichai Lesser

Which is why I was very interested in learning about 2 new developments from Microsoft. These developments provide  improved performance for branch office users with a Windows 7 client accessing applications hosted on a Windows Server 2008 in a remote data center.

The first development is the improved TCP stack and improved SMB implementation which mitigates one of the biggest performance problems at remote branch offices around file access and web access (internal web access). More on the TCP stack performance improvements can be found here: http://en.wikipedia.org/wiki/Windows_Vista_networking_technologies#Network_performance and on the file transfer performance improvements (namely SMB2 improvements) you can read here: http://en.wikipedia.org/wiki/Server_Message_Block#SMB2. and more evidence to these performance enhancement for remote branch users can be found here: http://blogs.technet.com/josebda/archive/2008/11/11/file-server-performance-improvements-with-the-smb2-protocol-in-windows-server-2008.aspx

A 2nd development that couples the next generation TCP stack is the new BranchCache feature. You can read more about it here: http://www.microsoft.com/downloads/details.aspx?FamilyID=EE07308F-7C53-4C76-9ED9-670BC25A4C9D&displaylang=en

Why is this so interesting? Well, for many of my clients, slow file transfers and slow access to static data are big pain points. It is also one of the drivers behind the WAN acceleration industry. If the new Windows 7 – Windows 2008 R2 lives up to its promise we may see it being an interesting alternative to WAN accelerators, curious to hear what you think, especially if you had any experience with deploying any of these new solutions.

Talk to you soon,

Amichai Lesser

P.S. peer to peer torrent access to cached data inside the branch, really? !! I challenge the first IT engineer to share his experience with implementing this option.

Loosing site of the goal
What happens when IT departments lose site of the performance engineering goal? (Reminder in short the goal is to improve the end user’s quality of experience and productivity, while maintaining system costs within budget).

Well what happens is that each department gets lost in its own tactical goal:

The Capacity planning team focuses on efficient and accurate hardware provisioning
The load testing team focuses on test coverage and scale requirements
The network engineering team focuses on the speed and capacity of the pipes
The data base team focuses on the performance of the data bases
The server team focuses on the performance of the backend servers
Desktop team is focusing on the performance of the desktop clients
…
What the organization ends up with is a set of local optimums, but in many cases those local optimums don’t amount to an optimal system. What’s missing in the above list is at least one department that is responsible for meeting the goal, it is very rare to find a team that oversees the end to end responsiveness and performance of the application across all its components from the end user’s perspective. It is even harder to find a team that is held accountable to end user performance.

But is it wrong for each team to improve its domain and make sure it is optimal? Well the counter intuitive answer is yes, it is wrong and for the following reasons:

Focusing on the wrong bottlenecks
Let’s consider the following transaction as an example: this transaction generates a time sheet report for global employees. This transaction is served by a client (web browser with java widgets) a few web servers behind a load balancer, a few application servers and a data base server. Now lets see what happens if there are performance issues with this transaction. Naturally each team will spend time in improving its own domain, so the data base team may index the employee’s data base to reduce the data base response time in half, the server team adds more web servers behind the load balancer to increase the application’s scalability and the network team adds more bandwidth to the data center router. All these steps sound like they should help, no? Well the realistic answer is that in some cases none of these steps help, in fact 3 negative things happen here:

a. The teams spent time and money on the wrong bottlenecks

b. The real bottleneck is still out there

c. Increasing the speed of none bottleneck components places more strain on the real bottleneck, slowing things down even further

In future posts I will give specific examples of several problems that can not be addressed in the realm of one IT department. Those problems usually result from interdependencies between the different systems (servers, networks and data bases). It takes a holistic and multi-disciplinary process to find the right bottleneck let along find a solution for the problem. It may sound complicated, but the concept is quite simple, when dealing with performance, it does little good to focus on local optimums, any optimization effort that is not spent on the actual bottleneck is counter productive and a waste of IT resources and money. Remember, the goal is to improve end user response time at the desktop, not optimize a specific component that is part of a bigger system.

Even though the concept is simple, the solution isn’t always as simple. In future posts, I will offer practical ways to find application performance bottlenecks, I have used them in many engagements and they haven’t failed me yet. But before we can talk about the solutions, it is important to understand the problems so in the next several posts we will cover a few basic performance engineering concepts.

Talk to you soon…

Part 2 (for part 1 click here)

Anyone who was ever part of a performance engineering process should be able to relate to the following story:

“…Version 3.5 of a critical application is scheduled for release in 6 weeks, the latest stable build (internal version) of the application finally made it to the hands of the performance engineering team 1 week ago. The team ran a few performance tests and found that this new version performs much slower than the previous 3.0 version. The team even identified potential tuning opportunities, but they require changes in the code. Trying to get time from the developers to address these issues was nearly impossible as they some are already working on a patch that adds more functionality to this version and some are already deployed at another project. The application ends up being deployed as is, clients begin complaining about the poor performance and the business unit points fingers at the performance engineering team for not delivering on its goal…”

Why does this happen?
Lack of goal commonality

The main reason behind this kind of story and others like it is the lack of goal commonality. The development team which has the biggest impact on application performance is measured on timely delivery of specified functionality and is rarely measured on the performance and responsiveness of those applications. While application performance is the goal of the performance engineering team, who can usually just verify the compliance of applications with their performance goals, but can rarely impact any changes that will actually improve performance. The performance engineering team can oversee performance improvement projects but the changes have to be driven through the development team, which as already established is not measured on performance, hence will be somewhat reluctant to spend time on these performance improving projects. There are other examples of lack of goal commonalities within IT (Network Ops and Application development is another common example), but none impact application performance (or lack there off) more than the above.
Performance vs.Functionality: Lack of goal commonality makes this a tough choice

But even if the development team isn’t measured on performance, don’t developers care about the performance and responsiveness of their applications?

Well off course they do.

I come from a development background and I can tell you first hand that developers want to take pride in the code they write, but when faced with the choice of spending time on performance improvements or on functionality enhancements, they are usually forced to choose the later.

Lack of tools and expertise

To make things worst, the lack of goal commonality prevents developers from being able to address performance problems even if they set the time for it, since for the most part developers are not equipped with the right tools, best practices and expertise to address potential performance issues. When development managers make their hiring decisions and development tool selections, they tend to make choices that will improve the team’s ability to deliver functionality fast and tend to focus less on tools and expertise that are performance oriented.

Performance later

There is a tendency in the development world to push performance to the last stage in the development life cycle, also known as “performance last” or “performance later”. While this approach may make sense initially, since you can’t test for performance before you have runable code that passed functional testing, it is one of the reasons behind many of the performance problems that exist in applications today. As we will see in future posts, many performance problems are a result of decisions made very early in the development process, such as platform selection, development language selection and even mundane things such as the type of user interface container selected for a specific web page.

In summary we identified the lack of goal commonality between application development teams and performance engineering groups as one of the reasons behind a sub optimal performance engineering practice. In the next post we will look into the lack of a clear goal as a fundamental roadblock in achieving efficient performance engineering.

In the mean time I am interested in your feedback, check within your company’s software specification documents, how many have performance requirements documented in the specifications? How many end up into the test plan? Curious to hear what you find.

Talk to you soon…

Curious to hear your feedback on the following:

Performance Engineering – Why so many companies don’t get it

Many of you may read the headline and wonder, with so much money spent on performance engineering tools and personnel, surely most companies are getting it right. Well, most companies do get some performance engineering tasks right, but anyone familiar with the industry would agree that all or part of the following happens in almost every IT organization:

Applications are slower than initially expected
This story should sound familiar to anyone who ever dealt with deploying an enterprise application. The application goes through rigorous testing, first functional and later load and stress. Baselines are generated and average response times are predicted or estimated (and sometimes guestimated), only to find out that real end users, mostly remote but sometimes even local users experience a much slower application. It is relatively easy to verify the existence of this problem in your organization, simply compare the pre deployment response time baselines with the Application Performance Monitors (APM) statistics about the user experience post deployment. You are sure to find anything from a 2X up to over a 10X difference in response times, especially in applications that are deployed globally. Another way to validate the existence of this situation is by asking the IT department to predict how an application will perform once deployed globally, 99 times out of 100 you will not find a confident answer to that question.

Return on Investment is Hard to Show
In my line of business I get to work with a variety of organizations with different attitudes towards performance. In general there is a growing understanding that improved application performance has a positive impact on the financial bottom line. As a result more and more companies invest in performance engineering processes, tools and personnel in order to get from thinking about performance to actively addressing performance requirements (reaching the goal). However, even the most sophisticated organizations where performance is treated as a product with specific goals and objectives, even those cutting edge IT departments have a hard time showing the return on investment from their performance initiatives.

Performance Engineering becomes Load Testing
Effective performance engineering is hard to achieve. A practice that meets the performance engineering goal must address application performance from end to end. Very few organizations have the tools and expertise required to gain visibility into all the factors that impact application performance. As a result many organizations tend to focus on what can be measured and predominately that means focusing on load testing. Now don’t get me wrong, assessing an application’s ability to perform under load is critical to any successful deployment, but it is certainly not enough. Focusing on load testing is analogous to looking for a lost coin under the street lamp, load testing will highlight a set of potential performance problems, but many other may lurk in the dark and may have a bigger impact on performance than anything that comes up during the load test alone.

There are many other examples of how a suboptimal performance engineering practice manifests itself, in the 2nd part of this post we
will cover the reasons why most companies have a hard time implementing an efficient performance engineering practice.

Talk to you soon…

So here’s my shameless plug for the certificate program.

This 2 and half day certification program covers everything an IT engineer needs to know about Shunra’s technology, including:

• Performance Engineering Best Practices
• Deep Transaction Analysis and Performance Triage
• Network Capacity Planning with Shunra VE Predictor, VE Profiler, VE Analyzer and HP LoadRunner
• Advanced Network Topics
• Network Aware Load Testing with VE Desktop for HP Software

Completing this program provides a SCPE certification (Shunra Certified Performance Engineer) which comes in very handy in today’s global performance world.

This year will be even more interesting as we will, for the first time, cover the new VE Desktop for HP Software. This product was co-developed by Shunra and HP to transform LoadRunner and Performance Center test beds into virtual network test beds for testing application performance from remote sites. The training session will provide expertise on combining load testing and network performance analysis in a single tool.

Another new addition to the curriculum is the Deep Transaction Analysis and Performance Triage which covers tips and tricks for effective triage and troubleshooting of application performance issues.

There are still some seats open which are sold on a first come first serve basis, so make sure to go here if you are interested in participating.

See you on the 14th at the DoubleTree Hotel in Center City, Philadelphia.

Amichai