Good performance at peak loads should be something that we think about from design and development all the way through to the production environment.  It starts with understanding what users are going to do on the site, producing test scenarios that match that behavior, and applying those scenarios to your functionality and load testing efforts in the preproduction environment. Hopefully you are nodding your head as you just read my last sentence, but here is something that you may not have considered. Have you measured the Latency, Packet Loss, and bandwidth for your typical user population and recreated that in your preproduction environment?  “Why does it matter?”, you ask.  It can dramatically affect system performance and stability and always has an impact on end user performance.

Think about it this way, if you are load testing and functionality testing in the preproduction environment with lightning fast latency, near infinite bandwidth, and zero packet loss then you are gaming the system.  In the lab, your packet latency is less than 1 millisecond, but your users are going to be experiencing latencies of 30, 50, or even 100 milliseconds latency, which translates into keeping network resources tied up for much longer timeframes.  That leads to longer Web Session with more simultaneous HTTP connections and longer Application sessions, which can even affect the database. These are all infrastructure resources that are utilized differently using identical load scenarios and load levels, but have a dramatic impact on end user performance and reduce the scalability of the system.

Real Network Conditions Play a Big Role in System Scalability

Applying end user latency, loss, and bandwidth is easy with tools like Shunra’s VE Desktop, which can be used in functional testing to accurately apply the correct network conditions to boost accuracy with the latency that end users would experience in the real world. Shunra can go a step further and help you to determine the latency, loss, and bandwidth that you can plug right into the environment during testing using the Network Catcher tool.

Shunra also has a variety of options for emulating network conditions during load testing ranging from  appliances Like the STA to software. The appliance acts like a router on steroids applying network impairments dynamically to the network traffic during the test treating traffic from different URLs and IPs like they are in different cities.

If you are using HP’s LoadRunner load testing tool, Shunra has an elegant solution called Shunra for HP Software, which works seamlessly with the LoadRunner during the load tests to apply the latency, packet loss, and bandwidth during these critical scalability tests.

Regardless of your testing methods, be sure to include identical network conditions into your testing for the most accurate results.

If you’re not familiar with LoadRunner in the Cloud, it’s one of HP’s latest offerings where they partner with Amazon AWS to offer HP LoadRunner in Amazon’s EC2.  The product is currently in beta and is doing very well.  To read my post about it and to get links to join the HP beta, go here.

As a long-time partner of HP, Shunra was invited to augment the HP/Amazon partnership by including Shunra for HP Software.  In our recent white paper with SAP and HP, we already showed how testing with WAN emulation is valuable for anyone testing distributed applications AND how it all works swimmingly in virtual environments, so the only hurdle was to package everything together.

That packaging is now done.  We worked with HP to build an Amazon Machine Image (AMI) that is based off of HP’s 32 bit Load Generator AMI and includes Shunra’s WAN emulation component by default, and we have defined a very fast and easy process to install our component on the Controller instance.  The Controller component installation for Shunra requires a 15MB download and approximately 90s to install.

Once the new Load Generator is approved internally by HP (in the next few days), it will be added to the list of AMIs available for the HP beta program (again, to request inclusion in the beta program, go here).  In the mean time, you can still take advantage of this offer by joining the HP beta program and contacting me directly.  After confirming your participation in the HP program, I’ll send you download and installation instructions for the Load Generator as well as the Controller components from Shunra for HP Software – both are required to use WAN emulation in your LoadRunner in the Cloud tests.

As I mentioned in my previous post, HP is waiving their license fees for this beta program.  In that light, Shunra has agreed to do the same.  The only fees you will be responsible for are from Amazon AWS.  From personal experience, I can say that the AWS usage fees are quite reasonable.

If you’re not planning to participate in this beta program, but would still like to learn more about application performance testing in the cloud, I encourage you to comment below or send me a message directly.  My contact information is listed at the bottom of this post.

My previous post about LoadRunner in the Cloud spoke about how Shunra™ for HP Software not only works well in clouds, but actually “completes the picture” for anyone conducting performance testing of applications that will be accessed remotely.    I am not alone with these statements.  SAP Labs, HP and Shunra are pleased to announce the availability of a new whitepaper, released today, June 7,  titled, “Application Performance Testing in a Virtual Environment.”

In this paper, we explain a best practice for performance testing SAP applications.  This best practice was conceived by SAP LABs and goes through four stages that take an SAP application from the initial steps of verifying functionality and accessibility, to how to tune the solution with WAN acceleration technology.  While the paper specifically focused on SAP applications, it’s clear that this best practice applies to almost any application that will be accessed remotely.

One of the key themes in the paper is virtualization.  As I have mentioned before, Shunra for HP Software already works well in clouds.  This paper took a scientific approach and compared tests using dedicated, physical machines with a completely virtual setup.  Specifically, HP LoadRunner was first run on a standalone machine, through a Shunra Appliance and then to the SAP application servers.  For comparison, HP LoadRunner with Shunra for HP was installed on virtual machines (controller plus load generators) and the SAP application servers were also installed on virtual machines.

When the six-hour load tests were run and the results were compared, the average transaction response time varied by only a few one thousandths of a second – well below any reasonable requirement for accuracy!

Whether you’re interested in using HP’s new LoadRunner in the Cloud offering, or simply trying to reap some of the many benefits of virtualization by moving your testing environment into a virtual server, this paper is a must read.

There are many challenges organizations face when it comes to administering and maintaining a dedicated pre-production or staging environment for which accurate performance testing can be conducted. The cost to manage and maintain infrastructure, along with personnel and facilities, can be fairly sizeable and are only a subset of the overall costs to be considered. So, in many cases performance testing can be rather expensive and this is exactly why virtualization can provide significant benefits because there is cost reduction across the board. A prime example is in many performance labs there are a variety of application performance tools typically utilized for testing – one such tool is HP LoadRunner or Performance Center. These tools are a primary part of a performance lab as they provide load generation capabilities and can accurately test applications under real world load and stress scenarios. However, these solutions require a significant amount of infrastructure and resources (A Controller to execute tests, LoadRunner Generators to produce user traffic, Virtual User Generator to record scripts, etc.) and this can make it very difficult to manage the environment when it has to scale to meet higher demand. In this case, virtualization saves time, effort and cost because resources can be allocated dynamically within the environment and any number of virtual machines can be leveraged when needed to handle these resource intensive applications. This is also enabling many organizations to architect and customize elegant configurations that more closely align with their testing requirements – which can minimize unnecessary infrastructure and resources. Yet, the prevalent issue many organizations still grapple with is how to execute performance tests that accurately depict the network for which the application will be deployed across.

The most pervasive approach that many organizations would take is to physically deploy hardware (remote load generators) in offices that they wanted to test an application from. This process was not only time-consuming, but also expensive, inaccurate and cumbersome to manage. For this reason, HP decided to form a partnership with Shunra to develop a seamless solution that provides this capability within the HP LoadRunner and Performance Center solutions – Shunra VE Desktop for HP. This solution aligns very well with the virtualization movement because it is simply a plug-in within the HP products that introduces the network into the existing test bed and can be leveraged across most virtual platforms. For LoadRunner 9.5 and later, there is simply a “WAN emulation” tab that can be accessed from within the Controller to introduce the latency, jitter, packet loss and bandwidth constraints directly into the test. With Performance Center 9.5 and later, this capability can easily be configured directly from the browser UI to allocate WAN emulation parameters across any number of desired load generators. A consequence of this is that organizations can leverage on-demand performance testing from a dynamic virtual environment that is agile, flexible and robust. This therefore eliminates the need to manage testing cycles across multiple time zones and remove any need for additional hardware. Additionally, all of the network metrics from each generator utilizing WAN emulation within the test will automatically be imported into the controller, which can save a significant amount of time when collating results and generating analysis reports. These reasons are precisely why numerous organizations have decided to improve their existing performance test environment with the VE Desktop for HP Software  solution. Not only does this solution address a strategic gap within the functionality of the HP solutions, it embraces virtualization as a means to more effectively administer performance testing. Overall, the VE Desktop for HP Software solution was co-developed with HP to considerably enhance the accuracy and value of these application performance test suites.

Are you migrating over to the Web 2.0 world or considering adding modern architecture and components? If so, take a look at this interesting video I came across, in which Mark Tomlinson and Steve Feloney from HP, our partner, tackle the challenges of load testing Web 2.0 applications. It will be well worth your time to check out Mark and Steve as they address the top issues that make LoadRunner web testing much harder and more complex. Very helpful and it’s only 4 min long!

In this video, Mark and Steve talk about real challenges their customers faced when migrating to the Web 2.0 without considering the impact of WAN latency, and most importantly how to solve this! Anyone using VE Desktop for HP Software should also definitely check this out and take a light-hearted look at web load testing.

This video is the first in a series put together by Steve and Mark covering some top issues that make LoadRunner web testing much harder and complex. Stay tuned for the next part of “Real Stories of Load Testing Web 2.0: Impacts of WAN latency of Web 2.0 apps”.

…will Steve end up going to Paris? and what will Mark find on google? the plot thickens…

That sounds like a lot of clouds!  What we are simply trying to understand is where and how to deploy your application in the cloud, so it will perform well for your customers.

If you take the case of deploying an online store into the Amazon EC2 cloud, there are many things to consider that will impact your customer’s experience.  Two of the most important are where to deploy what, and how your store will function over the Internet.  To compound the issue, where you physically deploy machines within EC2 is not clear.  Amazon goes to great lengths to avoid you knowing where their zones are.

Fortunately, you cannot only find solutions to these problems, but with cloud computing the process gets a lot easier.  To break the problems down into a logical flow, you’ll first need to know the demographics of your customers.  Where are they?  How will they access the store?  What needs to happen for them to have a positive experience at your store?  If you can answer these questions, you’ll likely come up with some numbers that place percentages of your users in different regions around your target market (US, EMEA, global, etc.).  Further, you’ll know the top two browsers they’ll use and with what kind of connections they’ll access the internet (e.g. Firefox with a 768/256kbps DSL, or IE8 with dial-up – yes people still use dial-up!).  Finally, you’ll have some way to quantify a “positive user experience”.  That usually involves consistency (does it work?) and speed (how fast?).

Using the information above and HP LoadRunner with WAN emulation, you can quickly build a to-scale test in your cloud-based lab and understand what would happen if you hosted your application all in one place and serviced your entire customer base.  Odds are that the test will not meet all of your performance goals the first time you run it.  Most likely, you will need to adjust some things in your store (e.g. reduce chattiness and image resolution, optimize Web 2.0 use) and then begin the discussion about where to deploy what.  Can you keep all of your database servers in one place?  If you have to split your database servers, how will that affect performance?  Will you need to deploy web servers in every zone, will you be okay with one in each region, or will you need something in-between?

Again, not only can answers to these questions actually be answered, but by using cloud-based testing you can arrive at valid conclusions faster than ever.  To understand the network impact of deploying machines in different zones, network performance measurement tools like VE Network Catcher can run in each of the zones to measure network performance between zones and to customer representative endpoints.  Importing that data directly into HP LoadRunner with WAN emulation and using the flexible cloud environment, you can rapidly form and either dismiss or validate hypotheses about how your application should best be deployed.

The benefit of testing in this way is that you can experiment with many different configurations in rapid succession to find the one or few that are most appropriate for your needs.  Trial and error with a live application is not advisable.  And building a test lab with physical machines to simulate the same thing is not only significantly more expensive, but a lot slower process.  As I pointed out in my previous blog, the same ROI formula that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.

Now you know what I think.  What do you think??

Performance Test Lab testing

Pros: Controlled, repeatable, relatively easy to debug observed problems

Cons: Expensive to build and maintain, commonly implemented without real world network considerations

Live Network testing

Pros: Cheap, good for periodic sanity checks of performance

Cons: Puts non-revenue generating load on the production network, very difficult to debug, high volume testing is not an accurate depiction of end user performance

The good news is that developments in cloud-based testing and in HP LoadRunner have been able to improve the accuracy of Performance Test Lab testing while also reducing the complexity and overall cost.  By moving HP LoadRunner load generators into a cloud-based testing environment, your testing can immediately benefit from the flexibility and scalability of the cloud.

One specific use case is for Peak Testing.  Peak testing is simply another name for load testing.  What “peak” implies is that it is at a scale that is not common.  If you are running, or want to run a test that is not commonly run in your environment, then it is fair to assume that you’ll need to plan ahead to gather the necessary resources.  Working in a cloud environment makes gathering and configuring extra resources on demand easy.  Need an extra five load generators?  No problem; just clone your existing one and wait for them to start.  Need more RAM?  No problem; just shut the machine down, configure more RAM, and restart the machine.  Working with the latest version of HP LoadRunner with WAN emulation makes the story even more compelling.  Need load generation from LA, London, and Tokyo?  No problem; just configure the load generators to emulate those locations and they will.  Sound easy?  It is.  Companies like Skytap already have thought this scenario through and have a great subscription-based model built for you to use today.

The real benefits of peak testing in the cloud are that you can quickly scale your lab up and down without the burden of maintaining a lab large enough to support your peak needs –saving time and money.  It is the end of costly lab build-ups, running tests during maintenance windows in the middle of the night, sending load generators to all corners of the world, and putting test traffic on your production network!

Of course, there are a lot of other ways Cloud Computing can be used in a QA environment; validating new versions of test products and proof of concepts are two that immediately come to mind.  In the end, the same ROI formulae that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.

Now you know what I think.  What do you think??

P.S. Stay tuned for a coming blog on location-aware deployment testing in the cloud…

In alignment with global trends, more and more clinical trials take place outside of the United States, while the documentation and analysis is done in the US for submission to the FDA. So it is not out of the ordinary to find an Oracle Clinical user in Eastern Europe or in China submitting data to an Oracle Clinical Server hosted in the united states.

Unfortunately, as commonly found with an off the shelf software package that gets customized over time, this type of application can easily morph into a performance challenged application, especially for remote users. The following analysis shows an example of an Oracle Clinical transaction that completes in 8 seconds for local users, while extending to over a minute when users in E. Europe tried to use it.

Transaction Response Time Analysis for an Oracle Clinical Application Accessed by Local Users Vs. Remote Users in E. Europe

Obviously, that big jump in response time was a great cause for concern with my client’s management. Which is how I got the privilege to be asked to analyze the root cause of the poor performance.

The first thing I noticed was that one particular transaction (Connect to DB) had an extremely high jump in TRT from 8 seconds locally to over a minute when accessed remotely, so I focused on that one first.

Looking at the network fingerprint of the transaction you can observe the following things:

The transaction size (amount of data downloaded from the server required to complete the transaction) is extremely high (over 5 MB).

The transaction generates 34 HTTP calls. Some of those calls take disproportionally longer time than others (notice the Get jar file calls that each exceed 19 seconds marked below, click on the image for a larger view)

Oracle Clinical, Transaction Analysis for "Connect to DB"

When inspecting those JAR files, we saw that they were very heavy in size, over 1 MB each which is pretty big for basically a collection of Java classes that are zipped into a JAR file.

You can also observe in the next bounce diagram that most of the JAR files are downloaded in a serial fashion, blocking other objects from downloading in the mean time. Notice the delta time displayed on the right column

Bounce Diagram for an Oracle Clinical Transaction

What can be done to improve the performance of this application?

Oracle Clinical uses JAR files to package Java code that will run on the client. It is not uncommon for customized off the shelf applications to increase in code size over time until the code is very bloated.

There are several best practices that developers can follow to reduce the size of JAR files:

1. Rationalize the code – as applications develop over time, multiple classes and sometimes adjacent projects might reference similar libraries and other assets. If not careful those libraries and common assets end up packaged multiple times inside the JAR file, causing it to inflate in size.
2. Minify the code – a quick Google search for “reduce the size of JAR files” will reveal several free tools that can minimize the size of JAR files, usually through eliminating white spaces, comments, shortening variable names, etc.
3. Defer loading – chances are that not all the code in the JAR file is needed for the “connect to DB” transaction, which means that users that only want to perform a small transaction are penalized by the download time of code that will never get executed by them. Deferred loading is a design pattern that simply says “only download assets or code when it is needed” I wrote more on that design pattern in this previous post “http://www.shunra.com/shunrablog/index.php/2009/01/29/content-loading-when-being-lazy-pays-off/”

In summary, as more and more web 2.0 technologies hit main stream, new forms of performance pitfalls may present themselves, requiring us to consider application performance for remote users even for applications that traditionally were considered “remote user friendly” such as web based applications.

How was this Analysis Performed?

This analysis was used by integrating a HP LoadRunner script modeling an Oracle Clinical business user with Shunra VE Analyzer. So as the scripted transactions executed across a virtual WAN (simulated by the Shunra VE) the beginning and end of each transaction were marked by the VE Transaction Manager. At the end of the test, the VE Analyzer had sufficient data to generate the attached reports, enabling us to pin point the root cause of the performance problem. You can learn more on how to set this up by going through Shunra Certified Performance Engineering training, either on site or at the next Shunra University. Learn more about Shunra training here:  http://www.shunra.com/shunra-university-overview.php and here http://www.shunra.com/training-overview.php?keyword=services

Questions, comments, feel free to write me at amichai.lesser at Shunra dot com. Or comment on any of my posts.

BTW, we just created a new Application Performance Management Group on LinkedIn, feel free to look it up and join here  http://www.linkedin.com/groupRegistration?gid=2200667

Best,

Amichai Lesser

In this solution I’m taking advantage of the “User Data Point” mechanism in LoadRunner. The “User Data Point” mechanism allows users to include their own custom data points in the LoadRunner test database. These data points are viewable in the LoadRunner Controller UI when the test runs and in the analysis.

The main idea is that for each transaction reported to the controller, the script will also report a “User Data Point”. The data point will include the name of the emulated location or host machine. Adding the host name will guaranty uniqueness of the data point name and will allow pinpointing the location it comes from. If you are worried about performance costs associated with using many data points with multiple virtual users, you can implement a caching mechanism for each location that collects these data points and reports their averages as data points every second or so.

In my script I define StartTransaction and EndTransaction functions and replace all instances of lr_start_transaction and lr_end_transaction with them. These functions call the appropriate lr_ function internally and EndTransaction() also reports the data point. Although StartTransaction just calls lr_start_transaction, it’s a good idea to use it as it keeps the code clean and allows for easy modifications.

One note: I’m using “Host name” and not “Emulated Location” in the script because there is currently no API to get the emulated location name using a script.

Here is how the two functions are defined:

// Encapsulate lr_start_transaction() – for consistency with EndTransaction()

intStartTransaction( char* pName )

{

return lr_start_transaction( pName );

}

// Encapsulate lr_end_transaction(). Measures TRT and status and inserts

// a user data point that includes the host name – to distinguish the same transaction

// coming from other hosts.

int EndTransaction( char* pName, int status )

{

double dTransTime = 0;

int iTransStatus = 0;

char pDataPointName[1024] = {0};

int iRetVal = 0;

// Get the duration of the transaction.

dTransTime = lr_get_transaction_duration(pName );

// Get the status of the transaction.

iTransStatus = lr_get_transaction_status( pName );

// Signal the end of the transaction to LoadRunner.

iRetVal = lr_end_transaction( pName, status );

// If the transaction completed correctly, add it’s instance as a data point

if ( dTransTime > 0 && iTransStatus == LR_PASS && ( status == LR_PASS || status == LR_AUTO ) )

{

// Add the host name.

sprintf( pDataPointName, “%s::%s”, lr_get_host_name(), pName );

// Add the user data point.

lr_user_data_point( pDataPointName, dTransTime );

}

return iRetVal;

}

Here is an example of Action transaction that uses the new functionality:

Action()

{

/* … Removed beginning of script for clarity … */

//Signal a transaction start

StartTransaction( “Second Download” );

web_url(“testpage.asp_2″,

“URL=http://10.0.0.2/testanalyzer/testpage.asp”,

“Resource=0″,

“RecContentType=text/html”,

“Referer=http://10.0.0.2/”,

“Snapshot=t7.inf”,

“Mode=HTML”,

LAST);

web_link(“download 1000 bytes with 1 second delay”,

“Text=download 1000 bytes with 1 second delay”,

“Snapshot=t8.inf”,

LAST);

// Signal a transaction end.

EndTransaction( “Second Download”, LR_AUTO );

return 0;

}

When the test runs each transaction appears in the “Transaction Response Time” graph of LoadRunner but it’s value is aggregated for all locations. Using the StartTransaction / EndTransaction functions above you get the TRTs from each location in the data point graph aggregated only for virtual users running on that location. This allows you to see how each transaction performs from different emulated location.

Stay tuned as In a few weeks I’m planning to post another article detailing how to add bandwidth and latency to the mix and get a better view of how the network affects your users.

Enjoy:

Amichai