Good performance at peak loads should be something that we think about from design and development all the way through to the production environment.  It starts with understanding what users are going to do on the site, producing test scenarios that match that behavior, and applying those scenarios to your functionality and load testing efforts in the preproduction environment. Hopefully you are nodding your head as you just read my last sentence, but here is something that you may not have considered. Have you measured the Latency, Packet Loss, and bandwidth for your typical user population and recreated that in your preproduction environment?  “Why does it matter?”, you ask.  It can dramatically affect system performance and stability and always has an impact on end user performance.

Think about it this way, if you are load testing and functionality testing in the preproduction environment with lightning fast latency, near infinite bandwidth, and zero packet loss then you are gaming the system.  In the lab, your packet latency is less than 1 millisecond, but your users are going to be experiencing latencies of 30, 50, or even 100 milliseconds latency, which translates into keeping network resources tied up for much longer timeframes.  That leads to longer Web Session with more simultaneous HTTP connections and longer Application sessions, which can even affect the database. These are all infrastructure resources that are utilized differently using identical load scenarios and load levels, but have a dramatic impact on end user performance and reduce the scalability of the system.

Real Network Conditions Play a Big Role in System Scalability

Applying end user latency, loss, and bandwidth is easy with tools like Shunra’s VE Desktop, which can be used in functional testing to accurately apply the correct network conditions to boost accuracy with the latency that end users would experience in the real world. Shunra can go a step further and help you to determine the latency, loss, and bandwidth that you can plug right into the environment during testing using the Network Catcher tool.

Shunra also has a variety of options for emulating network conditions during load testing ranging from  appliances Like the STA to software. The appliance acts like a router on steroids applying network impairments dynamically to the network traffic during the test treating traffic from different URLs and IPs like they are in different cities.

If you are using HP’s LoadRunner load testing tool, Shunra has an elegant solution called Shunra for HP Software, which works seamlessly with the LoadRunner during the load tests to apply the latency, packet loss, and bandwidth during these critical scalability tests.

Regardless of your testing methods, be sure to include identical network conditions into your testing for the most accurate results.

It gets better!   For the beta program, HP is waiving all of their license fees!  (Amazon EC2 license/use fees still apply.)

And better yet!  Lots of reasons this is a win-win for HP LoadRunner customers:

• This makes LoadRunner significantly more available to testers.
• Using this delivery model, we can expect LoadRunner to be significantly less expensive  AND easier to deploy.
• Good-bye to  days of needing to own load testing software that you should be using regularly, but in reality may only use during specific phases of your SDLC.  Now you can pay for what you need, when you need it.
• Also gone are the days of installing and upgrading LoadRunner.  Have you ever tried to download a 1.2GB installer over a slow link?
• How about realizing that you need to uninstall a previous version before you can install the new one?  Now that HP and Amazon publish AMIs with the software pre-installed, all you need to do is spin-up the images you want and run them.

While my enthusiasm for this development is clear, it should be said that it’s not all, “moonlight and canoes.”

The Controller AMI in the beta program is limited to 250 virtual users.  Since HP hasn’t published their pricing yet, it’s not clear how larger tests will be accommodated.  For now, the beta testing will limit us to mid-sized tests.  Considering the actual technology (LR 9.5) is already stable and known to work well in virtual environments, there is little doubt that this solution will be massively scalable.

You will also need scripts to run in LoadRunner.  For existing or former LoadRunner customers, this is as easy as accessing your scripts from EC2.  For net-new LoadRunner customers, this will require some work.  Thankfully, the script generation environment, Virtual User Generator makes it easy to create simple scripts and there is an extensive community of users and consultants to help you develop the scripts you need.

Be sure to watch Mark Tomlinson’s video where he explains the benefit of testing applications for the cloud in the cloud.  The main benefit Mark demonstrates is the ability to omit your corporate firewall from the test environment.

One BIG benefit of testing in the cloud is eliminating the public network from your test environment:  If you are testing an application in the cloud from your on-premise lab, then you’re subjecting your test data to an unknown world of latency, packet loss and bandwidth limits.  Testing a cloud application from an on-premise lab can produce useless results as they  have no correlation to how your application will actually perform from your end user’s perspective.

Another benefit I would be remiss if I didn’t mention is that Shunra’s integration with HP LoadRunner, Shunra for HP Software already works well in clouds.  Shunra for HP runs inside of LoadRunner and lets you decide what network impairments to test.  You can import your production data, pick from pre-defined settings, or specify ad-hoc values to make each Load Generator emulate a different network location.  By doing this, you can conduct a performance test for all of your end user locations, entirely in a cloud environment, all the while paying as you go.  I may be a little biased, but my argument is that the functionality of Shunra for HP completes the picture for anyone performance testing applications that will be accessed remotely – and who isn’t?

As a techie who started performance testing using a green-screen and writing my own protocol stacks because the applications I was testing insisted on using proprietary protocols, I am excited to see professional quality performance testing moving into the cloud.  Although this innovative solution hasn’t created much new technology, it did make existing technology consumable in the cloud.  This is sure to challenge other vendors  to step up to the plate, making other products and services more cloud-enabled.

What do you think?  Are there any reasons you would not use this capability?  What are some other products or services that you hope will make their way into the cloud this year?

WAN Simulation methodology is a set of theoretical algorithms which are usually applied on a trace file. These algorithms aim to predict applications response times with different conditions and then applied to the captured trace file. A typical use case would be varying the conditions, such as available bandwidth and distance (latency), and then predicting the impact on response time. The main challenge with this approach is that those theoretical algorithms are based on assumptions.  Assumptions, which are in some cases very wrong and skew the results. Typically, with theoretical WAN Simulation you should expect around 70% accuracy.

On the other hand, when we say Emulation (or WAN Emulation) we mean that the application traffic will traverse in real time through a path that is identical to the path we have in mind. An appliance based WAN Emulator or software based WAN Emulation is usually present to create actual network elements in software, bring into account network impairment characteristics, such as distance (latency), bandwidth and loss. The accuracy of WAN Emulation can exceed 99% depending on the level of configuration. We can expect around 95% accuracy from a straight forward automatic configuration.

Another important difference is time. Theoretical WAN simulation may take days to run, but using WAN Emulation the duration is based on the time it takes the application to run.

The main reason why WAN Emulation triumphs is accuracy.

Recently, I visited a company that develops and distributes software for Independent School District’s across USA. Their main product, originally developed for challenged kids, centers around dynamic computerized teaching and testing programs. This product proved to be so successful among this population that the company adapted it for other kids as well. Initially the product was designed to work in the classroom, hence a very simple architecture: 1 PC working as a learning manager.  This 1 PC managed the progress of individual students, as well as received and captured data back to computers across the LAN. However, because of such a high demand, the company decided to host a solution for multiple customers from a central location, specifically – a Cloud hosted server.

Since the original design was created with only a classroom setting in mind, testing and development only occurred over LAN. But now, with accessibility from different regions, including links from APAC countries to US, consideration of the WAN was critical. This required the need to test their solution in a new environment and see if any problems exist.  By analyzing this information, they gain a better understanding of how to fix any problems. With this challenge, the company needed a WAN emulator solution that would allow them to re-create the complex topology of the application served over WAN.

Once tested, they were able to easily reproduce the problems reported to the QA from the field. One issue that was noted – disconnections without the ability to reconnect, was leading to a loss of data and historical progress of the students in the program.  One of the key tools is tracking “task lists.”  One of the main advantages of the system is the ability to produce these new “task lists” for each student, based on historical data retrieved.  The new tasks matched the student’s progress in learning the subjects and fulfilling the tasks.

One other obstacle the company faced dealt with the USD security measures.  Since product was originally operating on a LAN environment, the security of outside access was a concern. As a result, certain ports were blocked and the company needed to understand how those ports will be affected from blocks in different directions of the link. Shunra VE WAN Emulator was able to reproduce such conditions by filtering application traffic based on protocol and ports, as well as IP addresses and blocking them in one direction – the problems were reproduced instantly. Now QA no longer has to rely on the reports generated from the field to try and fix the problems “in the dark”…  They now have a nice “flash light” to expose those reasons for the problem.

The impact of a broken link or just a simple delay in logging in can significantly impact students, especially the challenged ones.  Picture a child struggling to connect to the program, no sounds, no pictures… instead sitting with headphones and seeing on a screen: “Connecting…”  An unnerving experience for all of us, and even less for them (IMHO).  And what if after working for 20-30 minutes on a progressive task – they have to start over because computer lost all memory of what they were doing until now.

However, we may need to apply a sobriety test, where are those servers? And how far out is my virtual storage?  A recent article in Scientific Computing (http://www.scientificcomputing.com/articles-HPC-Cloud-Computing-Pie-in-the-sky-120109.aspx) reminds us that the flexibility of an elastic infrastructure comes with strings attached.

The author presents a case for testing adequate bandwidth, as well as measuring and testing network latency.  Virtual machines can introduce additional latency through the time-sharing nature of the underlying hardware. Since the service provider — and not the scientist — control the hardware, unanticipated sharing and reallocation of machines can significantly affect runtimes.

My conclusion from this article, and others, dealing with the latency issues induced by Cloud Computing, is to include a rigorous testing phase before committing to a migration. An end to end latency measurement and emulation are probably the best ways to ensure smooth adoption of this exciting new technology.

The following posts will answer this question and more, but in order to get answers we need to first address additional questions:

Consider the remote user in Tokyo, he is accessing multiple applications that are all hosted in the remote NYC data center.

Will all these applications perform the same way once hosted in a remote data center?

The obvious answer is NO, some applications will perform well even when users are remote to the data center, while others will provide intermittent poor performance and some will always perform poorly for a remote user.

The answers to the next questions are less obvious:

• Why do some applications perform well in remote data centers while others fail miserably?
• What is it about the way applications are designed and architected in a remote data center that allows some applications to perform better than others?
• What are the key design flaws that cause applications to perform poorly in a remote data center?

Since there are a lot of different applications, there are also a lot of different answers to these questions. In the next couple of posts we will focus on enterprise-data-driven- transactional applications and their performance flaws over the network (ignoring for a moment back-end or desktop related bottlenecks). We will further limit this category to client server based applications, ignoring for a moment the complexity of N-Tier applications and multi web service based applications, these will be dealt with in future posts.

Over the years I was asked to analyze performance problems for many transactional applications and specifically analyze their performance degradation when users are remote to their data center, the following are the key application metrics that I found are related to performance degradation over the network:

1. The number of application turns per transaction (or how chatty the application is)
2. The transaction size (or how much data needs to be downloaded from the server to the client in order to complete each transaction)
3. The transaction efficiency factor (how much data a transaction downloads per application turn)
4. The blocking nature of object retrieval (can the transaction retrieve multiple objects concurrently or is each object download blocking other object requests from being processed)
5. The transaction redundancy metric (or how much of the same data is being retrieved by all the requests made by this transaction). It seems like this metric should always be zero, but you will be surprised how often this is the single reason behind performance problems.
6. The transaction initialization size (how much data does the transaction download initially Vs. sequential navigational steps)
7. The caching ratio (how much data is cached locally as a percentage of the overall data needed by the application)
8. The latency scale factor (How does the backend’s ability to scale change when network latency is added between front end clients and the backend)

Measuring and observing these metrics allows for the deep level analysis that is required to identify performance bottlenecks. This information also helps to point out to application developers and system engineers what needs to be changed in order to remediate application performance problems.

In the next couple of posts I will explain each of the above factors and describe how each of them impacts application performance, so sign up for the RSS feed to get notification on these future posts.

Until then…

Amichai Lesser

Performance Test Lab testing

Pros: Controlled, repeatable, relatively easy to debug observed problems

Cons: Expensive to build and maintain, commonly implemented without real world network considerations

Live Network testing

Pros: Cheap, good for periodic sanity checks of performance

Cons: Puts non-revenue generating load on the production network, very difficult to debug, high volume testing is not an accurate depiction of end user performance

The good news is that developments in cloud-based testing and in HP LoadRunner have been able to improve the accuracy of Performance Test Lab testing while also reducing the complexity and overall cost.  By moving HP LoadRunner load generators into a cloud-based testing environment, your testing can immediately benefit from the flexibility and scalability of the cloud.

One specific use case is for Peak Testing.  Peak testing is simply another name for load testing.  What “peak” implies is that it is at a scale that is not common.  If you are running, or want to run a test that is not commonly run in your environment, then it is fair to assume that you’ll need to plan ahead to gather the necessary resources.  Working in a cloud environment makes gathering and configuring extra resources on demand easy.  Need an extra five load generators?  No problem; just clone your existing one and wait for them to start.  Need more RAM?  No problem; just shut the machine down, configure more RAM, and restart the machine.  Working with the latest version of HP LoadRunner with WAN emulation makes the story even more compelling.  Need load generation from LA, London, and Tokyo?  No problem; just configure the load generators to emulate those locations and they will.  Sound easy?  It is.  Companies like Skytap already have thought this scenario through and have a great subscription-based model built for you to use today.

The real benefits of peak testing in the cloud are that you can quickly scale your lab up and down without the burden of maintaining a lab large enough to support your peak needs –saving time and money.  It is the end of costly lab build-ups, running tests during maintenance windows in the middle of the night, sending load generators to all corners of the world, and putting test traffic on your production network!

Of course, there are a lot of other ways Cloud Computing can be used in a QA environment; validating new versions of test products and proof of concepts are two that immediately come to mind.  In the end, the same ROI formulae that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.

Now you know what I think.  What do you think??

P.S. Stay tuned for a coming blog on location-aware deployment testing in the cloud…

However this series of posts is about the day after a data center move. Now that the data center is remote, how does this paradigm shift impact the way we should develop, test, deploy, monitor and troubleshoot applications. I will try to cover as many topics as possible, but the main focus is still going to be around the role application performance management plays in this new paradigm.

I will start by covering the key reasons behind the performance impact that is experienced when applications are hosted in a remote data center? Those reasons are fairly intuitive, but it is important to understand them in depth in order to adequately plan for those new conditions. Two main things impact how applications perform when application servers are hosted in a remote data center vis a vis their application clients :

1. The performance of the network link between the client and the remote data center. This performance is defined by a set of network performance metrics that are application independent (for now we will ignore application aware networks, however the following basic concepts still hold in this scenario as well).

2. The application efficiency, specifically how efficient the application is when transferring data between the client and the remote server (and other tiers if applicable). This is an application attribute (and some time an attribute of a specific business process within the application). These attributes are application specific and are independent of any underlying network.

Let’s start with understanding the network performance metrics. Consider the following scenario:

An application is hosted in a NYC data center, with users in 2 places, some are in a NYC headquarters next to that NYC data center and some are in a remote branch that is located in San Francisco. The question is: “will the application perform the same for both type of users (local users in headquarters and remote users in SF)? In other words will the application be as responsive to the San Francisco user as it is to the NYC user?”

Well the obvious answer is NO, in most cases a NYC user will enjoy a faster more responsive application. What is less obvious is why? What is it about the network that causes remote users to experience a slower application than local users? The rest of this post will cover that question, future posts will address the application specific attributes. Once we cover that we will be ready to examine best practices in building applications for a remote data center.

When I ask this question during my training seminars, I get a variety of answers, many of them are the right ones, but I would like to address one wrong answer that keeps repeating itself for some reason.

Collisions – there is a general conception that collisions are common phenomena on the network which can explain any bad thing that happens to applications. The truth is that collisions are almost a thing of the past (on Enterprise LANs anyhow) and even when they happen they can’t explain why a remote user has a worst experience than a local user as both will experience a similar collision chance since collisions is a phenomenon that happens on local area Ethernet networks. If there are collisions on the Enterprise LAN it usually points to a configuration issue on a network device (like a duplex miss-match) but is still unrelated to the answer to our question.

Now to the right answers to the question, what is it about the Wide Area Network that causes applications to slow down:

There are 5 key conditions that predominately exist on Wide Area Networks and impact application performance, each in their own way:

1. Network Latency – the time it takes a packet to traverse from a source to the destination across the network, measured in milliseconds [msec]. A typical WAN link will introduce latency in the range of 10msec – 500 msec.

2. Bandwidth constraints – how fast can data be processed by the network link, measured in bits per second [bps, Kbps, Mbps, Gbps]

3. Bandwidth utilization (background traffic) – the percentage of bandwidth that is utilized by traffic that already exists on the link (background traffic).

4. Jitter – the deviation of the inter packet gap of sequential packets across a network link, it is a result of the deviation of the network latency and is sometimes used interchangeably with that standard deviation, measured in milliseconds [msec].

5. Packet Loss – the chance to drop a packet across an end to end network link, measured in %. Sometimes presented as the inverse metric called packet delivery rate.

The above are called network impairments, you can click on each one of the links to learn more about them and their causes.

Network impairments are performance conditions that inhibit the flow of data across a network. Each impairment type has an impact on the performance of business applications and network services. Some applications may be very sensitive to network impairments and some may be almost network agnostic. Sorting applications based on their network sensitivity is one of the important steps in performance engineering

In the next post we will discuss how application design can impact performance across the network. But in the mean time I would like to introduce a question for the group:

“We identified network latency as one of the key reasons that impact application performance; we also said that a typical WAN link will introduce 10 – 500 msec of latency. The question is, why does network latency have a big impact on application performance? surely a user doesn’t notice an increase of a few msec in response time, even 500 msec = ½ second goes by in a flinch. So think about it and let me know what you found based on your experience, why does network latency have such a big impact on application performance?”

Until next time,

Amichai Lesser

In alignment with global trends, more and more clinical trials take place outside of the United States, while the documentation and analysis is done in the US for submission to the FDA. So it is not out of the ordinary to find an Oracle Clinical user in Eastern Europe or in China submitting data to an Oracle Clinical Server hosted in the united states.

Unfortunately, as commonly found with an off the shelf software package that gets customized over time, this type of application can easily morph into a performance challenged application, especially for remote users. The following analysis shows an example of an Oracle Clinical transaction that completes in 8 seconds for local users, while extending to over a minute when users in E. Europe tried to use it.

Transaction Response Time Analysis for an Oracle Clinical Application Accessed by Local Users Vs. Remote Users in E. Europe

Obviously, that big jump in response time was a great cause for concern with my client’s management. Which is how I got the privilege to be asked to analyze the root cause of the poor performance.

The first thing I noticed was that one particular transaction (Connect to DB) had an extremely high jump in TRT from 8 seconds locally to over a minute when accessed remotely, so I focused on that one first.

Looking at the network fingerprint of the transaction you can observe the following things:

The transaction size (amount of data downloaded from the server required to complete the transaction) is extremely high (over 5 MB).

The transaction generates 34 HTTP calls. Some of those calls take disproportionally longer time than others (notice the Get jar file calls that each exceed 19 seconds marked below, click on the image for a larger view)

Oracle Clinical, Transaction Analysis for "Connect to DB"

When inspecting those JAR files, we saw that they were very heavy in size, over 1 MB each which is pretty big for basically a collection of Java classes that are zipped into a JAR file.

You can also observe in the next bounce diagram that most of the JAR files are downloaded in a serial fashion, blocking other objects from downloading in the mean time. Notice the delta time displayed on the right column

Bounce Diagram for an Oracle Clinical Transaction

What can be done to improve the performance of this application?

Oracle Clinical uses JAR files to package Java code that will run on the client. It is not uncommon for customized off the shelf applications to increase in code size over time until the code is very bloated.

There are several best practices that developers can follow to reduce the size of JAR files:

1. Rationalize the code – as applications develop over time, multiple classes and sometimes adjacent projects might reference similar libraries and other assets. If not careful those libraries and common assets end up packaged multiple times inside the JAR file, causing it to inflate in size.
2. Minify the code – a quick Google search for “reduce the size of JAR files” will reveal several free tools that can minimize the size of JAR files, usually through eliminating white spaces, comments, shortening variable names, etc.
3. Defer loading – chances are that not all the code in the JAR file is needed for the “connect to DB” transaction, which means that users that only want to perform a small transaction are penalized by the download time of code that will never get executed by them. Deferred loading is a design pattern that simply says “only download assets or code when it is needed” I wrote more on that design pattern in this previous post “http://www.shunra.com/shunrablog/index.php/2009/01/29/content-loading-when-being-lazy-pays-off/”

In summary, as more and more web 2.0 technologies hit main stream, new forms of performance pitfalls may present themselves, requiring us to consider application performance for remote users even for applications that traditionally were considered “remote user friendly” such as web based applications.

How was this Analysis Performed?

This analysis was used by integrating a HP LoadRunner script modeling an Oracle Clinical business user with Shunra VE Analyzer. So as the scripted transactions executed across a virtual WAN (simulated by the Shunra VE) the beginning and end of each transaction were marked by the VE Transaction Manager. At the end of the test, the VE Analyzer had sufficient data to generate the attached reports, enabling us to pin point the root cause of the performance problem. You can learn more on how to set this up by going through Shunra Certified Performance Engineering training, either on site or at the next Shunra University. Learn more about Shunra training here:  http://www.shunra.com/shunra-university-overview.php and here http://www.shunra.com/training-overview.php?keyword=services

Questions, comments, feel free to write me at amichai.lesser at Shunra dot com. Or comment on any of my posts.

BTW, we just created a new Application Performance Management Group on LinkedIn, feel free to look it up and join here  http://www.linkedin.com/groupRegistration?gid=2200667

Best,

Amichai Lesser

Which is why I was very interested in learning about 2 new developments from Microsoft. These developments provide  improved performance for branch office users with a Windows 7 client accessing applications hosted on a Windows Server 2008 in a remote data center.

The first development is the improved TCP stack and improved SMB implementation which mitigates one of the biggest performance problems at remote branch offices around file access and web access (internal web access). More on the TCP stack performance improvements can be found here: http://en.wikipedia.org/wiki/Windows_Vista_networking_technologies#Network_performance and on the file transfer performance improvements (namely SMB2 improvements) you can read here: http://en.wikipedia.org/wiki/Server_Message_Block#SMB2. and more evidence to these performance enhancement for remote branch users can be found here: http://blogs.technet.com/josebda/archive/2008/11/11/file-server-performance-improvements-with-the-smb2-protocol-in-windows-server-2008.aspx

A 2nd development that couples the next generation TCP stack is the new BranchCache feature. You can read more about it here: http://www.microsoft.com/downloads/details.aspx?FamilyID=EE07308F-7C53-4C76-9ED9-670BC25A4C9D&displaylang=en

Why is this so interesting? Well, for many of my clients, slow file transfers and slow access to static data are big pain points. It is also one of the drivers behind the WAN acceleration industry. If the new Windows 7 – Windows 2008 R2 lives up to its promise we may see it being an interesting alternative to WAN accelerators, curious to hear what you think, especially if you had any experience with deploying any of these new solutions.

Talk to you soon,

Amichai Lesser

P.S. peer to peer torrent access to cached data inside the branch, really? !! I challenge the first IT engineer to share his experience with implementing this option.