If you’re not familiar with LoadRunner in the Cloud, it’s one of HP’s latest offerings where they partner with Amazon AWS to offer HP LoadRunner in Amazon’s EC2.  The product is currently in beta and is doing very well.  To read my post about it and to get links to join the HP beta, go here.

As a long-time partner of HP, Shunra was invited to augment the HP/Amazon partnership by including Shunra for HP Software.  In our recent white paper with SAP and HP, we already showed how testing with WAN emulation is valuable for anyone testing distributed applications AND how it all works swimmingly in virtual environments, so the only hurdle was to package everything together.

That packaging is now done.  We worked with HP to build an Amazon Machine Image (AMI) that is based off of HP’s 32 bit Load Generator AMI and includes Shunra’s WAN emulation component by default, and we have defined a very fast and easy process to install our component on the Controller instance.  The Controller component installation for Shunra requires a 15MB download and approximately 90s to install.

Once the new Load Generator is approved internally by HP (in the next few days), it will be added to the list of AMIs available for the HP beta program (again, to request inclusion in the beta program, go here).  In the mean time, you can still take advantage of this offer by joining the HP beta program and contacting me directly.  After confirming your participation in the HP program, I’ll send you download and installation instructions for the Load Generator as well as the Controller components from Shunra for HP Software – both are required to use WAN emulation in your LoadRunner in the Cloud tests.

As I mentioned in my previous post, HP is waiving their license fees for this beta program.  In that light, Shunra has agreed to do the same.  The only fees you will be responsible for are from Amazon AWS.  From personal experience, I can say that the AWS usage fees are quite reasonable.

If you’re not planning to participate in this beta program, but would still like to learn more about application performance testing in the cloud, I encourage you to comment below or send me a message directly.  My contact information is listed at the bottom of this post.

Performance Test Lab testing

Pros: Controlled, repeatable, relatively easy to debug observed problems

Cons: Expensive to build and maintain, commonly implemented without real world network considerations

Live Network testing

Pros: Cheap, good for periodic sanity checks of performance

Cons: Puts non-revenue generating load on the production network, very difficult to debug, high volume testing is not an accurate depiction of end user performance

The good news is that developments in cloud-based testing and in HP LoadRunner have been able to improve the accuracy of Performance Test Lab testing while also reducing the complexity and overall cost.  By moving HP LoadRunner load generators into a cloud-based testing environment, your testing can immediately benefit from the flexibility and scalability of the cloud.

One specific use case is for Peak Testing.  Peak testing is simply another name for load testing.  What “peak” implies is that it is at a scale that is not common.  If you are running, or want to run a test that is not commonly run in your environment, then it is fair to assume that you’ll need to plan ahead to gather the necessary resources.  Working in a cloud environment makes gathering and configuring extra resources on demand easy.  Need an extra five load generators?  No problem; just clone your existing one and wait for them to start.  Need more RAM?  No problem; just shut the machine down, configure more RAM, and restart the machine.  Working with the latest version of HP LoadRunner with WAN emulation makes the story even more compelling.  Need load generation from LA, London, and Tokyo?  No problem; just configure the load generators to emulate those locations and they will.  Sound easy?  It is.  Companies like Skytap already have thought this scenario through and have a great subscription-based model built for you to use today.

The real benefits of peak testing in the cloud are that you can quickly scale your lab up and down without the burden of maintaining a lab large enough to support your peak needs –saving time and money.  It is the end of costly lab build-ups, running tests during maintenance windows in the middle of the night, sending load generators to all corners of the world, and putting test traffic on your production network!

Of course, there are a lot of other ways Cloud Computing can be used in a QA environment; validating new versions of test products and proof of concepts are two that immediately come to mind.  In the end, the same ROI formulae that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.

Now you know what I think.  What do you think??

P.S. Stay tuned for a coming blog on location-aware deployment testing in the cloud…

In this post I will show you how to generate analysis in less than three minutes that addresses the question that naturally follows. “Why does this transaction perform well locally, but slows down under network conditions?”

This analysis can be generated by installing VE Analyzer on the “remote” load generator and configuring it with the packet analysis settings (reminder: the “remote” load generator is any load generator that has the VE Desktop Professional WAN Emulation client installed on it. This WAN Emulation client enables us to simulate remote network conditions from a local load generator in the lab. Hence the double quotes in “remote” since the generator isn’t physically remote but is simulated to behave that way)

Follow these steps to setup the analysis on the “remote” load generator:

In the LoadRunner Controller Design screen click on the Load Generators button

Select the “remote” load generator

Click on details and select the WAN Emulation Tab:

Check the “Enable WAN Emulation” check box (if not already selected) and click on “WAN Emulation Settings”

See the previous post on how to setup the WAN Emulation parameters in the following screen.

Click on the additional settings button

First check the “Capture Packets” check box, this enables the packet capturing agent on the VE Desktop Client on the LoadGenerator. I recommend assigning at least 100 MB of buffer space for the capture so at least one iteration of the script will be captured.

For web based applications, select HTTP in the “Automatically run post-test analysis” option, for other applications select either TCP or UDP analysis.

Finally, select a folder on the load generator for automatically saving the test results. It is important to re-iterate that this folder is on the “remote” load generator, not on the LoadRunner Controller.

It is recommended to setup only 1 virtual user to run from this “remote” generator, so the analysis will focus on a single instance of the user. If load is needed as part of the scenario, it should be generated from a second load generator, in this case the analysis will show results of the marginal user.

Adding the analysis capabilities in this example doesn’t require any changes to the script and can be applied to the same scenario we presented in the previous post. Running this test results in an analysis report that is generated on the LoadGenerator. This report can be accessed from the VE Reperter that is installed on the “remote” load generator.

VE Reporter stores the reports based on the test names and the test run dates, so we are looking for the last test run for our test.

Once the report opens we are mainly interested in 4 reports:

The Application Efficiency Report shows us how chatty the application is (application chattiness is directly related to response time slow downs due to network latency) as well as how long it took to download various resources from the server. In this example, we see a resource that took 9 seconds to download, explaining why the entire action took 11 seconds for a remote user Vs, 0.6 seconds for a local user.

The Application Efficiency Report highlights a resource that took 9.2 seconds to download

The second report is the Client Network Server Breakdown that shows how much time the transaction spent on the client, network and server.

A Client Network Server Breakdown Report

The 3rd report is the Bandwidth Analysis which shows how much bandwidth was used by this single application user.

Finally, for a real deep dive we can use the Bounce Diagram that displays the entire communication between the client and the server as well as the delta times between consecutive packets, high delta times can point out to blocking events within the transaction.

The Bounce Diagram shows deep packet analysis data on the transaction

In summary, in less than 3 minutes we can generate the above reports on any transaction that seems to be performing poorly over the Wide Area Network. This information can help the developers and network engineers to fine-tune the application and the network to improve the end user experience.

Feel free to suggest additional topics you would like to see covered in this series of posts.

Best,

Amichai

This will be the first in a series of posts that walks through the steps needed to setup a test scenario that includes end user load and network conditions. By the end of this post you will know how to setup a load test that can answer the question: “How will this application perform for a remote user in [Chicago] compared to a local user?” [replace with any remote site that is relevant for your enterprise network]

Our setup includes 2 machines:

1. A LoadRunner controller version 9.5 with a license for the VE Desktop for HP Software. This machine will also serve as a local load generator and is assigned with IP address 172.31.0.143

2. A LoadRunner load generator with the VED for HP injector and VE Reporter for analysis. This machine is assigned with IP address 172.31.0.142 and will simulate the “remote” load generator.

Both machines are physically in our center of excellence lab in Philadelphia, and so is the server under test.

We begin with a simple test scenario that will run the same script from the 2 load generators – 143 will be the local user and 142 will emulate the remote user in Chicago, so we name the 2 groups accordingly:

Next we need to setup the “remote” load generator with the desired network conditions. The WAN emulation conditions are defined on the load generator details dialog box. These details can be accessed by clicking on the Load Generators Icon (marked with the border),

selecting the “remote” load generator in the load generators setup page, and clicking the details button:

You will notice that the Load Generator Details has a WAN Emulation tab. Selecting the tab and checking the Enable WAN Emulation checkbox will allow you to setup the WAN emulation conditions. It is recommended to change the name of the profile to reflect the network conditions, e.g. Chicago, remote site with 512Kbps, Satellite link, etc.

Clicking on the WAN Emulation settings button will open up the following dialog box:

You will notice the 3 setup modes: Basic, Advanced and Professional.

The Basic mode (shown above) allows you to quickly define a network profile based on predefined settings, the example above sets up a network link from a Chicago user to an Atlanta data center with a T1 link during normal business hours.

The advanced mode lets you setup specific network conditions such as latency, packet loss and bandwidth. The Professional mode will be covered in future posts.

Once the desired settings are configured, the test is ready to go. Our test scenario has 1 user from each generator, but the scenario isn’t limited by the number of users (up to the licensed number of virtual users).

The scenario can be saved and run like any other LoadRunner scenario.

Once the test completes, we can invoke the analysis and generate the reports we are looking for (reminder, we want to see the response time between the local user and the user in Chicago). The analysis is triggered by clicking on the LoadRunner analysis button:

By default the analysis results are averaged across all load generators, so we need to group the results by WAN Emulation location which is done in the next 3 steps:

Step 1 – select the transaction response time chart:

Step 2 – Select the Set Filter\Group By option by clicking on (Ctrl T):

And group by emulated location:

Step 3 – We want to present the data as a bar chart, so we select the display options (Ctrl I):

And setup a Bar chart with the following options:

Finally, we get the report we were looking for (below), answering the question:

“How will this application perform for a remote user compared to a local user?”

The local user response time is identified by the transactions with the “No WAN Emulation” suffix, the Chicago user is identified by the transactions with the Chicago suffix (click on the thumbnail to view the large image).

Hope this helps, stay tuned for more use cases and tutorials.

Best,

Amichai

]]> http://www.shunra.com/shunrablog/index.php/2009/02/26/loadrunner-conditions/feed/ 6 http://www.shunra.com/shunrablog/index.php/2008/11/19/replacing-remote-load-generators/ http://www.shunra.com/shunrablog/index.php/2008/11/19/replacing-remote-load-generators/#comments Wed, 19 Nov 2008 22:25:55 +0000 Dave Berg http://applicationtesting.wordpress.com/?p=140 Have you ever tried to run a test, only to find that your Load Generators half-way around the world are not responding?

If any of that sounds familiar, then you might want to consider using network emulation instead of deploying Load Generators around the globe. By using an emulation solution, you can bring your testing back into a controlled lab environment. The benefits of doing that include:

• Repeatability: by controlling your whole environment, you can quickly run and reproduce exactly the tests that you want.
• Reduced cost: no need to install and administer remote machines, or the people to maintain them
• Reduced time: stable testing environment means less test runs to get reliable results, less setup time – ever try to download a 1GB installer over a slow network?

And many more…

The first question that should come to mind is, “will an emulated solution be as accurate as using the actual production network?” The short answer is yes. The long answer needs to point out that testing over the production network is not “accurate” in the first place. Example: if you run your load tests during off hours, what relevance does that have when the actual users try to work during peak network times? Very little.

If you’re still convinced that you need to model you’re production environment to a high level of accuracy, then you’re still in luck. With network emulation, it is possible to closely model production networks. When measuring on average transaction response time, we typically see correlations around 95%. It should be pointed out that, to get these numbers several runs over the production network are needed so a true average can be understood. When running in an emulated network environment, a single test run is usually sufficient.

Practically speaking, the 80/20 rule applies here as well. If you can get your emulated network to perform at 80% or closer to the actual conditions that your users will operate in, then you’ll be able to far exceed any testing value that you’re getting out of deploying Load Generators around the network.

Remember that network emulation empowers you to return your testing to what it was originally designed to be – a scientific experiment. Once you use network emulation to get all of the variables under your control, you can quickly configure and run tests, troubleshoot, and present results with confidence.