There are many challenges organizations face when it comes to administering and maintaining a dedicated pre-production or staging environment for which accurate performance testing can be conducted. The cost to manage and maintain infrastructure, along with personnel and facilities, can be fairly sizeable and are only a subset of the overall costs to be considered. So, in many cases performance testing can be rather expensive and this is exactly why virtualization can provide significant benefits because there is cost reduction across the board. A prime example is in many performance labs there are a variety of application performance tools typically utilized for testing – one such tool is HP LoadRunner or Performance Center. These tools are a primary part of a performance lab as they provide load generation capabilities and can accurately test applications under real world load and stress scenarios. However, these solutions require a significant amount of infrastructure and resources (A Controller to execute tests, LoadRunner Generators to produce user traffic, Virtual User Generator to record scripts, etc.) and this can make it very difficult to manage the environment when it has to scale to meet higher demand. In this case, virtualization saves time, effort and cost because resources can be allocated dynamically within the environment and any number of virtual machines can be leveraged when needed to handle these resource intensive applications. This is also enabling many organizations to architect and customize elegant configurations that more closely align with their testing requirements – which can minimize unnecessary infrastructure and resources. Yet, the prevalent issue many organizations still grapple with is how to execute performance tests that accurately depict the network for which the application will be deployed across.

The most pervasive approach that many organizations would take is to physically deploy hardware (remote load generators) in offices that they wanted to test an application from. This process was not only time-consuming, but also expensive, inaccurate and cumbersome to manage. For this reason, HP decided to form a partnership with Shunra to develop a seamless solution that provides this capability within the HP LoadRunner and Performance Center solutions – Shunra VE Desktop for HP. This solution aligns very well with the virtualization movement because it is simply a plug-in within the HP products that introduces the network into the existing test bed and can be leveraged across most virtual platforms. For LoadRunner 9.5 and later, there is simply a “WAN emulation” tab that can be accessed from within the Controller to introduce the latency, jitter, packet loss and bandwidth constraints directly into the test. With Performance Center 9.5 and later, this capability can easily be configured directly from the browser UI to allocate WAN emulation parameters across any number of desired load generators. A consequence of this is that organizations can leverage on-demand performance testing from a dynamic virtual environment that is agile, flexible and robust. This therefore eliminates the need to manage testing cycles across multiple time zones and remove any need for additional hardware. Additionally, all of the network metrics from each generator utilizing WAN emulation within the test will automatically be imported into the controller, which can save a significant amount of time when collating results and generating analysis reports. These reasons are precisely why numerous organizations have decided to improve their existing performance test environment with the VE Desktop for HP Software  solution. Not only does this solution address a strategic gap within the functionality of the HP solutions, it embraces virtualization as a means to more effectively administer performance testing. Overall, the VE Desktop for HP Software solution was co-developed with HP to considerably enhance the accuracy and value of these application performance test suites.

Since the January 27^th announcement, it has become public knowledge that also Google is about to release a smart tablet, as well as HP using a Windows 7 platform. So are we reaching critical mass? I believe we are reaching the point tablets cannot longer be ignored.  Before the end of the year you will be able to receive TV streams into your tablet, read E-books, use your favorite social network, including the latest addition, Goggle “Buzz”, order pizza for dinner and yes, check your email and place both video or voice conference calls.

On an interview with the Associated Press, Disney’s CEO, Robert Iger, as quoted at the blog TUAW, The Unofficial Apple Blog,  he depict different scenarios using IPAD to deploy games, media and gaming content, all of it oriented toward the non-technical audiences.

Apple is drawing heavily on the experience gained by both IPod Touch and Iphone, that  in only a couple  of  years had managed to become the industry standard for smart web appliances and smart phones. The primary audience is the millions of users who are too afraid or reluctant to use a laptop/netbook since they are too rigidly bound to an unfriendly Windows OS.  A typical tablet user doesn’t want to use a computer per se.

So, why this is relevant to this blog? Very simple, all the context, media, games, TV shows, music and E-books will be delivered over ever increasing congested networks, served from ever more complex infrastructures at all will require substantial testing. What better tool to test a distributed network that Shunra Virtual Enterprise?  Application providers will have to test their application before deploying them to the millions of users. What will be the user experience, how latency will impact an IPAD user watching 2011 Super Bowl or downloading the latest chapter of their favorite TV show?

The combination of testing for 3G readiness (see other posts in this blog) and end user experience assessment will be a key factor on the success of this game changer.

Shunra VE is the best way to prepare for this new wave of context and delivery mechanism.

That sounds like a lot of clouds!  What we are simply trying to understand is where and how to deploy your application in the cloud, so it will perform well for your customers.

If you take the case of deploying an online store into the Amazon EC2 cloud, there are many things to consider that will impact your customer’s experience.  Two of the most important are where to deploy what, and how your store will function over the Internet.  To compound the issue, where you physically deploy machines within EC2 is not clear.  Amazon goes to great lengths to avoid you knowing where their zones are.

Fortunately, you cannot only find solutions to these problems, but with cloud computing the process gets a lot easier.  To break the problems down into a logical flow, you’ll first need to know the demographics of your customers.  Where are they?  How will they access the store?  What needs to happen for them to have a positive experience at your store?  If you can answer these questions, you’ll likely come up with some numbers that place percentages of your users in different regions around your target market (US, EMEA, global, etc.).  Further, you’ll know the top two browsers they’ll use and with what kind of connections they’ll access the internet (e.g. Firefox with a 768/256kbps DSL, or IE8 with dial-up – yes people still use dial-up!).  Finally, you’ll have some way to quantify a “positive user experience”.  That usually involves consistency (does it work?) and speed (how fast?).

Using the information above and HP LoadRunner with WAN emulation, you can quickly build a to-scale test in your cloud-based lab and understand what would happen if you hosted your application all in one place and serviced your entire customer base.  Odds are that the test will not meet all of your performance goals the first time you run it.  Most likely, you will need to adjust some things in your store (e.g. reduce chattiness and image resolution, optimize Web 2.0 use) and then begin the discussion about where to deploy what.  Can you keep all of your database servers in one place?  If you have to split your database servers, how will that affect performance?  Will you need to deploy web servers in every zone, will you be okay with one in each region, or will you need something in-between?

Again, not only can answers to these questions actually be answered, but by using cloud-based testing you can arrive at valid conclusions faster than ever.  To understand the network impact of deploying machines in different zones, network performance measurement tools like VE Network Catcher can run in each of the zones to measure network performance between zones and to customer representative endpoints.  Importing that data directly into HP LoadRunner with WAN emulation and using the flexible cloud environment, you can rapidly form and either dismiss or validate hypotheses about how your application should best be deployed.

The benefit of testing in this way is that you can experiment with many different configurations in rapid succession to find the one or few that are most appropriate for your needs.  Trial and error with a live application is not advisable.  And building a test lab with physical machines to simulate the same thing is not only significantly more expensive, but a lot slower process.  As I pointed out in my previous blog, the same ROI formula that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.

Now you know what I think.  What do you think??

The following posts will answer this question and more, but in order to get answers we need to first address additional questions:

Consider the remote user in Tokyo, he is accessing multiple applications that are all hosted in the remote NYC data center.

Will all these applications perform the same way once hosted in a remote data center?

The obvious answer is NO, some applications will perform well even when users are remote to the data center, while others will provide intermittent poor performance and some will always perform poorly for a remote user.

The answers to the next questions are less obvious:

• Why do some applications perform well in remote data centers while others fail miserably?
• What is it about the way applications are designed and architected in a remote data center that allows some applications to perform better than others?
• What are the key design flaws that cause applications to perform poorly in a remote data center?

Since there are a lot of different applications, there are also a lot of different answers to these questions. In the next couple of posts we will focus on enterprise-data-driven- transactional applications and their performance flaws over the network (ignoring for a moment back-end or desktop related bottlenecks). We will further limit this category to client server based applications, ignoring for a moment the complexity of N-Tier applications and multi web service based applications, these will be dealt with in future posts.

Over the years I was asked to analyze performance problems for many transactional applications and specifically analyze their performance degradation when users are remote to their data center, the following are the key application metrics that I found are related to performance degradation over the network:

1. The number of application turns per transaction (or how chatty the application is)
2. The transaction size (or how much data needs to be downloaded from the server to the client in order to complete each transaction)
3. The transaction efficiency factor (how much data a transaction downloads per application turn)
4. The blocking nature of object retrieval (can the transaction retrieve multiple objects concurrently or is each object download blocking other object requests from being processed)
5. The transaction redundancy metric (or how much of the same data is being retrieved by all the requests made by this transaction). It seems like this metric should always be zero, but you will be surprised how often this is the single reason behind performance problems.
6. The transaction initialization size (how much data does the transaction download initially Vs. sequential navigational steps)
7. The caching ratio (how much data is cached locally as a percentage of the overall data needed by the application)
8. The latency scale factor (How does the backend’s ability to scale change when network latency is added between front end clients and the backend)

Measuring and observing these metrics allows for the deep level analysis that is required to identify performance bottlenecks. This information also helps to point out to application developers and system engineers what needs to be changed in order to remediate application performance problems.

In the next couple of posts I will explain each of the above factors and describe how each of them impacts application performance, so sign up for the RSS feed to get notification on these future posts.

Until then…

Amichai Lesser

Performance Test Lab testing

Pros: Controlled, repeatable, relatively easy to debug observed problems

Cons: Expensive to build and maintain, commonly implemented without real world network considerations

Live Network testing

Pros: Cheap, good for periodic sanity checks of performance

Cons: Puts non-revenue generating load on the production network, very difficult to debug, high volume testing is not an accurate depiction of end user performance

The good news is that developments in cloud-based testing and in HP LoadRunner have been able to improve the accuracy of Performance Test Lab testing while also reducing the complexity and overall cost.  By moving HP LoadRunner load generators into a cloud-based testing environment, your testing can immediately benefit from the flexibility and scalability of the cloud.

One specific use case is for Peak Testing.  Peak testing is simply another name for load testing.  What “peak” implies is that it is at a scale that is not common.  If you are running, or want to run a test that is not commonly run in your environment, then it is fair to assume that you’ll need to plan ahead to gather the necessary resources.  Working in a cloud environment makes gathering and configuring extra resources on demand easy.  Need an extra five load generators?  No problem; just clone your existing one and wait for them to start.  Need more RAM?  No problem; just shut the machine down, configure more RAM, and restart the machine.  Working with the latest version of HP LoadRunner with WAN emulation makes the story even more compelling.  Need load generation from LA, London, and Tokyo?  No problem; just configure the load generators to emulate those locations and they will.  Sound easy?  It is.  Companies like Skytap already have thought this scenario through and have a great subscription-based model built for you to use today.

The real benefits of peak testing in the cloud are that you can quickly scale your lab up and down without the burden of maintaining a lab large enough to support your peak needs –saving time and money.  It is the end of costly lab build-ups, running tests during maintenance windows in the middle of the night, sending load generators to all corners of the world, and putting test traffic on your production network!

Of course, there are a lot of other ways Cloud Computing can be used in a QA environment; validating new versions of test products and proof of concepts are two that immediately come to mind.  In the end, the same ROI formulae that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.

Now you know what I think.  What do you think??

P.S. Stay tuned for a coming blog on location-aware deployment testing in the cloud…

In alignment with global trends, more and more clinical trials take place outside of the United States, while the documentation and analysis is done in the US for submission to the FDA. So it is not out of the ordinary to find an Oracle Clinical user in Eastern Europe or in China submitting data to an Oracle Clinical Server hosted in the united states.

Unfortunately, as commonly found with an off the shelf software package that gets customized over time, this type of application can easily morph into a performance challenged application, especially for remote users. The following analysis shows an example of an Oracle Clinical transaction that completes in 8 seconds for local users, while extending to over a minute when users in E. Europe tried to use it.

Transaction Response Time Analysis for an Oracle Clinical Application Accessed by Local Users Vs. Remote Users in E. Europe

Obviously, that big jump in response time was a great cause for concern with my client’s management. Which is how I got the privilege to be asked to analyze the root cause of the poor performance.

The first thing I noticed was that one particular transaction (Connect to DB) had an extremely high jump in TRT from 8 seconds locally to over a minute when accessed remotely, so I focused on that one first.

Looking at the network fingerprint of the transaction you can observe the following things:

The transaction size (amount of data downloaded from the server required to complete the transaction) is extremely high (over 5 MB).

The transaction generates 34 HTTP calls. Some of those calls take disproportionally longer time than others (notice the Get jar file calls that each exceed 19 seconds marked below, click on the image for a larger view)

Oracle Clinical, Transaction Analysis for "Connect to DB"

When inspecting those JAR files, we saw that they were very heavy in size, over 1 MB each which is pretty big for basically a collection of Java classes that are zipped into a JAR file.

You can also observe in the next bounce diagram that most of the JAR files are downloaded in a serial fashion, blocking other objects from downloading in the mean time. Notice the delta time displayed on the right column

Bounce Diagram for an Oracle Clinical Transaction

What can be done to improve the performance of this application?

Oracle Clinical uses JAR files to package Java code that will run on the client. It is not uncommon for customized off the shelf applications to increase in code size over time until the code is very bloated.

There are several best practices that developers can follow to reduce the size of JAR files:

1. Rationalize the code – as applications develop over time, multiple classes and sometimes adjacent projects might reference similar libraries and other assets. If not careful those libraries and common assets end up packaged multiple times inside the JAR file, causing it to inflate in size.
2. Minify the code – a quick Google search for “reduce the size of JAR files” will reveal several free tools that can minimize the size of JAR files, usually through eliminating white spaces, comments, shortening variable names, etc.
3. Defer loading – chances are that not all the code in the JAR file is needed for the “connect to DB” transaction, which means that users that only want to perform a small transaction are penalized by the download time of code that will never get executed by them. Deferred loading is a design pattern that simply says “only download assets or code when it is needed” I wrote more on that design pattern in this previous post “http://www.shunra.com/shunrablog/index.php/2009/01/29/content-loading-when-being-lazy-pays-off/”

In summary, as more and more web 2.0 technologies hit main stream, new forms of performance pitfalls may present themselves, requiring us to consider application performance for remote users even for applications that traditionally were considered “remote user friendly” such as web based applications.

How was this Analysis Performed?

This analysis was used by integrating a HP LoadRunner script modeling an Oracle Clinical business user with Shunra VE Analyzer. So as the scripted transactions executed across a virtual WAN (simulated by the Shunra VE) the beginning and end of each transaction were marked by the VE Transaction Manager. At the end of the test, the VE Analyzer had sufficient data to generate the attached reports, enabling us to pin point the root cause of the performance problem. You can learn more on how to set this up by going through Shunra Certified Performance Engineering training, either on site or at the next Shunra University. Learn more about Shunra training here:  http://www.shunra.com/shunra-university-overview.php and here http://www.shunra.com/training-overview.php?keyword=services

Questions, comments, feel free to write me at amichai.lesser at Shunra dot com. Or comment on any of my posts.

BTW, we just created a new Application Performance Management Group on LinkedIn, feel free to look it up and join here  http://www.linkedin.com/groupRegistration?gid=2200667

Best,

Amichai Lesser

In the next 2 posts, I will address the first 2 questions, the third one is probably a good topic for a book, so I will attempt to address that one over several future posts.

Should you really test all your applications for latency prior to a data center relocation?

Before jumping into a direct answer with examples of horror stories from past data center relocations that didn’t test all applications and the consequence of that excessive risk taking I want to first provide a conceptual best practice answer.

I recently received my Six Sigma Champion certification and one of the concepts in Six Sigma and in management for quality in general is “Don’t mass inspect, build quality into the process instead”. This concept was one of Deming’s teachings that revolutionized the Japanese manufacturing industry in the 1950s. So one would think that according to that best practice, we shouldn’t test all applications prior to a data center move, since that would be the equivalent of mass inspection. However, that would be a miss interpretation of that statement, an organization can shift from mass inspection to more statistical sampling only if quality is built into the process and when mission critical processes are in control. To complete the analogy, an organization can settle for sample testing a subset of applications only if application development processes have quality built into them, and specifically with a data center move, application development processes have network aware best practices embedded in them. I have yet to find an organization that reaches that level of maturity prior to a data center move. Usually the causation is reverse, first a firm relocates their data center, then IT realizes that they need to adopt a new application development paradigm since from now on applications will access remote servers in a remote data center and thus network aware best practices (like using VE Desktop as part of the development life cycle) are added.

So this brings us to the reasonable assumption that most companies will have a portfolio of applications that weren’t developed with a distributed WAN in mind and now have to adjust to operating in a remote data center. If that is the scenario your company is facing then mass inspection is your only way to mitigate the enormous risk associated with such a move.

Now if you don’t buy into the conceptual answer, consider the following:

1. In an average data center relocation project, we usually find that 20% – 40% of applications experience performance degradation ranging from visible to severe. (visible means that a user notices the slowdown in response time, severe means that an application stops being usable under latency conditions).
2. Many of the problems we uncover end up being show stoppers for the DCR project and need to go through remediation before the project can complete.
3. The mitigation path varies between applications and is very dependent on the type of problem that causes the performance degradation. Hence, a typical remediation solution may include WAN Acceleration for some applications (mainly server-to-server copy utilities and file services), terminal services (Citrix, remote desktop, VDI, etc.) for other applications, code fixes for some applications and architecture changes for the rest. Therefore engineering a solution based on a sample of applications will results in a sub optimal solution that doesn’t address all the performance problems that will manifest post the move.
4. When a large NY based insurance company relocated their servers from NY to Atlanta, GA without testing, they ended up rolling back the first phase of the relocation when several mission critical applications became unusable due to poor performance.

In summary, unless IT has built network aware best practices into the software development process, an organization planning a data center relocation must test all of its applications to analyze the performance impact that new added latency will have on application performance. More best practices on this topic can be found in my recently updated white paper that can be found at http://www.shunra.com/predicting-the-impact-of-data-center-moves-on-application-performance-whitepaper.php

The next post will cover the question “Which applications are the most vulnerable to the added network latency?” small example, any applications that are running executables from a shared network drive, will experience severe performance problems, that and more will be covered in the next post.

In the meantime, I am curious to hear your experience with data center relocations, what approach did your IT group take? Test all, sample test or cross your fingers and just move? Let me know.

P.S. as part of my Six Sigma Certification, I developed a dashboard and a flow chart for a data center relocation performance analysis, it needs some work before it can be shared, but if enough readers ask for it, I will try to clean it up and post it.

Till next time,

Amichai Lesser

Of the topics covered, WAN emulation is clearly closest to my heart, but please be sure to read up on Microsoft Vista support, the new RDP agent, and what is happening with LoadRunner’s Analysis API.

To read the full story, click here.

Do you plan to move to LoadRunner 9.5 or Performance Center 9.5 in the near future?  Will the integration with Shunra’s WAN emulation motivate you to upgrade or accelerate your upgrade schedule?

In this post I will show you how to generate analysis in less than three minutes that addresses the question that naturally follows. “Why does this transaction perform well locally, but slows down under network conditions?”

This analysis can be generated by installing VE Analyzer on the “remote” load generator and configuring it with the packet analysis settings (reminder: the “remote” load generator is any load generator that has the VE Desktop Professional WAN Emulation client installed on it. This WAN Emulation client enables us to simulate remote network conditions from a local load generator in the lab. Hence the double quotes in “remote” since the generator isn’t physically remote but is simulated to behave that way)

Follow these steps to setup the analysis on the “remote” load generator:

In the LoadRunner Controller Design screen click on the Load Generators button

Select the “remote” load generator

Click on details and select the WAN Emulation Tab:

Check the “Enable WAN Emulation” check box (if not already selected) and click on “WAN Emulation Settings”

See the previous post on how to setup the WAN Emulation parameters in the following screen.

Click on the additional settings button

First check the “Capture Packets” check box, this enables the packet capturing agent on the VE Desktop Client on the LoadGenerator. I recommend assigning at least 100 MB of buffer space for the capture so at least one iteration of the script will be captured.

For web based applications, select HTTP in the “Automatically run post-test analysis” option, for other applications select either TCP or UDP analysis.

Finally, select a folder on the load generator for automatically saving the test results. It is important to re-iterate that this folder is on the “remote” load generator, not on the LoadRunner Controller.

It is recommended to setup only 1 virtual user to run from this “remote” generator, so the analysis will focus on a single instance of the user. If load is needed as part of the scenario, it should be generated from a second load generator, in this case the analysis will show results of the marginal user.

Adding the analysis capabilities in this example doesn’t require any changes to the script and can be applied to the same scenario we presented in the previous post. Running this test results in an analysis report that is generated on the LoadGenerator. This report can be accessed from the VE Reperter that is installed on the “remote” load generator.

VE Reporter stores the reports based on the test names and the test run dates, so we are looking for the last test run for our test.

Once the report opens we are mainly interested in 4 reports:

The Application Efficiency Report shows us how chatty the application is (application chattiness is directly related to response time slow downs due to network latency) as well as how long it took to download various resources from the server. In this example, we see a resource that took 9 seconds to download, explaining why the entire action took 11 seconds for a remote user Vs, 0.6 seconds for a local user.

The Application Efficiency Report highlights a resource that took 9.2 seconds to download

The second report is the Client Network Server Breakdown that shows how much time the transaction spent on the client, network and server.

A Client Network Server Breakdown Report

The 3rd report is the Bandwidth Analysis which shows how much bandwidth was used by this single application user.

Finally, for a real deep dive we can use the Bounce Diagram that displays the entire communication between the client and the server as well as the delta times between consecutive packets, high delta times can point out to blocking events within the transaction.

The Bounce Diagram shows deep packet analysis data on the transaction

In summary, in less than 3 minutes we can generate the above reports on any transaction that seems to be performing poorly over the Wide Area Network. This information can help the developers and network engineers to fine-tune the application and the network to improve the end user experience.

Feel free to suggest additional topics you would like to see covered in this series of posts.

Best,

Amichai

First off, VE Desktop for HP Software 5.0 is a new product.  It is in no way related to LoadRunner MWE.  HP/Mercury customers last saw MWE in LoadRunner 8.1 (July 2005) and the feature was removed from the LoadRunner 9.x releases.

While MWE was indeed “Powered by Shunra,” it was done so through an OEM relationship.  MWE was sold and supported by HP/Mercury.  VE Desktop for HP Software 5.0 development was a collaboration between Shunra and HP.  The product is the result of both companies learning from our experiences, designing, and co-developing a new solution that would exactly meet the needs of our customers.  VE Desktop for HP Software 5.0 can do everything LoadRunner MWE could, and much more.  VE Desktop for HP Software 5.0 is sold and supported by Shunra.

The top three issues we chose to address were:

1) Accuracy: Since the MWE did not support bandwidth emulation or jitter, the results of its emulations often did not fully reflect the real-world performance of applications being testing.  To address this, VE Desktop for HP Software supports bandwidth emulation (symmetric and asymmetric), jitter, imported network characteristics from Shunra’s VE Network Catcher, and a host of other network impairments aimed at precisely emulating network links.

2) Reporting: With MWE, there was no network specific reporting.  If a test was run with multiple locations emulated, the LoadRunner results would still show transaction response time and other metrics averaged across the whole test.  In order to understand how applications fared at individual locations, scripts would have to be modified so the script and transaction names reflected the location they were being run at.  This was very challenging to implement and maintain.  In VE Desktop for HP Software 5.0, there are no script changes required for such reporting.  For users of VE Desktop for HP Software, HP Analysis added the ability to sort results by “Emulated Location.”  HP also added the automatic import of network statistics on a per Load Generator basis.  Now users can understand their application performance by location and merge that data with network statistics like bandwidth utilization, latency and packet loss.  Finally, users can understand if the network is impacting their application performance and if so, how.

3) Performance Center: LoadRunner MWE had no compliment in Performance Center.  VE Desktop for HP Software 5.0 supports both LoadRunner 9.5 and Performance Center 9.5.  Within Performance Center, dynamic host allocation is fully supported.

VE Desktop for HP Software 5.0 provides a lot of other benefits beyond the direct improvements over MWE.  Some of those include a robust API and protocol analysis capabilities, but those are topics for other blog posts.

I look forward to your comments on the explanations above and any feedback you have about our latest addition to the Shunra family of products.