If you’re not familiar with LoadRunner in the Cloud, it’s one of HP’s latest offerings where they partner with Amazon AWS to offer HP LoadRunner in Amazon’s EC2.  The product is currently in beta and is doing very well.  To read my post about it and to get links to join the HP beta, go here.

As a long-time partner of HP, Shunra was invited to augment the HP/Amazon partnership by including Shunra for HP Software.  In our recent white paper with SAP and HP, we already showed how testing with WAN emulation is valuable for anyone testing distributed applications AND how it all works swimmingly in virtual environments, so the only hurdle was to package everything together.

That packaging is now done.  We worked with HP to build an Amazon Machine Image (AMI) that is based off of HP’s 32 bit Load Generator AMI and includes Shunra’s WAN emulation component by default, and we have defined a very fast and easy process to install our component on the Controller instance.  The Controller component installation for Shunra requires a 15MB download and approximately 90s to install.

Once the new Load Generator is approved internally by HP (in the next few days), it will be added to the list of AMIs available for the HP beta program (again, to request inclusion in the beta program, go here).  In the mean time, you can still take advantage of this offer by joining the HP beta program and contacting me directly.  After confirming your participation in the HP program, I’ll send you download and installation instructions for the Load Generator as well as the Controller components from Shunra for HP Software – both are required to use WAN emulation in your LoadRunner in the Cloud tests.

As I mentioned in my previous post, HP is waiving their license fees for this beta program.  In that light, Shunra has agreed to do the same.  The only fees you will be responsible for are from Amazon AWS.  From personal experience, I can say that the AWS usage fees are quite reasonable.

If you’re not planning to participate in this beta program, but would still like to learn more about application performance testing in the cloud, I encourage you to comment below or send me a message directly.  My contact information is listed at the bottom of this post.

My previous post about LoadRunner in the Cloud spoke about how Shunra™ for HP Software not only works well in clouds, but actually “completes the picture” for anyone conducting performance testing of applications that will be accessed remotely.    I am not alone with these statements.  SAP Labs, HP and Shunra are pleased to announce the availability of a new whitepaper, released today, June 7,  titled, “Application Performance Testing in a Virtual Environment.”

In this paper, we explain a best practice for performance testing SAP applications.  This best practice was conceived by SAP LABs and goes through four stages that take an SAP application from the initial steps of verifying functionality and accessibility, to how to tune the solution with WAN acceleration technology.  While the paper specifically focused on SAP applications, it’s clear that this best practice applies to almost any application that will be accessed remotely.

One of the key themes in the paper is virtualization.  As I have mentioned before, Shunra for HP Software already works well in clouds.  This paper took a scientific approach and compared tests using dedicated, physical machines with a completely virtual setup.  Specifically, HP LoadRunner was first run on a standalone machine, through a Shunra Appliance and then to the SAP application servers.  For comparison, HP LoadRunner with Shunra for HP was installed on virtual machines (controller plus load generators) and the SAP application servers were also installed on virtual machines.

When the six-hour load tests were run and the results were compared, the average transaction response time varied by only a few one thousandths of a second – well below any reasonable requirement for accuracy!

Whether you’re interested in using HP’s new LoadRunner in the Cloud offering, or simply trying to reap some of the many benefits of virtualization by moving your testing environment into a virtual server, this paper is a must read.

It gets better!   For the beta program, HP is waiving all of their license fees!  (Amazon EC2 license/use fees still apply.)

And better yet!  Lots of reasons this is a win-win for HP LoadRunner customers:

• This makes LoadRunner significantly more available to testers.
• Using this delivery model, we can expect LoadRunner to be significantly less expensive  AND easier to deploy.
• Good-bye to  days of needing to own load testing software that you should be using regularly, but in reality may only use during specific phases of your SDLC.  Now you can pay for what you need, when you need it.
• Also gone are the days of installing and upgrading LoadRunner.  Have you ever tried to download a 1.2GB installer over a slow link?
• How about realizing that you need to uninstall a previous version before you can install the new one?  Now that HP and Amazon publish AMIs with the software pre-installed, all you need to do is spin-up the images you want and run them.

While my enthusiasm for this development is clear, it should be said that it’s not all, “moonlight and canoes.”

The Controller AMI in the beta program is limited to 250 virtual users.  Since HP hasn’t published their pricing yet, it’s not clear how larger tests will be accommodated.  For now, the beta testing will limit us to mid-sized tests.  Considering the actual technology (LR 9.5) is already stable and known to work well in virtual environments, there is little doubt that this solution will be massively scalable.

You will also need scripts to run in LoadRunner.  For existing or former LoadRunner customers, this is as easy as accessing your scripts from EC2.  For net-new LoadRunner customers, this will require some work.  Thankfully, the script generation environment, Virtual User Generator makes it easy to create simple scripts and there is an extensive community of users and consultants to help you develop the scripts you need.

Be sure to watch Mark Tomlinson’s video where he explains the benefit of testing applications for the cloud in the cloud.  The main benefit Mark demonstrates is the ability to omit your corporate firewall from the test environment.

One BIG benefit of testing in the cloud is eliminating the public network from your test environment:  If you are testing an application in the cloud from your on-premise lab, then you’re subjecting your test data to an unknown world of latency, packet loss and bandwidth limits.  Testing a cloud application from an on-premise lab can produce useless results as they  have no correlation to how your application will actually perform from your end user’s perspective.

Another benefit I would be remiss if I didn’t mention is that Shunra’s integration with HP LoadRunner, Shunra for HP Software already works well in clouds.  Shunra for HP runs inside of LoadRunner and lets you decide what network impairments to test.  You can import your production data, pick from pre-defined settings, or specify ad-hoc values to make each Load Generator emulate a different network location.  By doing this, you can conduct a performance test for all of your end user locations, entirely in a cloud environment, all the while paying as you go.  I may be a little biased, but my argument is that the functionality of Shunra for HP completes the picture for anyone performance testing applications that will be accessed remotely – and who isn’t?

As a techie who started performance testing using a green-screen and writing my own protocol stacks because the applications I was testing insisted on using proprietary protocols, I am excited to see professional quality performance testing moving into the cloud.  Although this innovative solution hasn’t created much new technology, it did make existing technology consumable in the cloud.  This is sure to challenge other vendors  to step up to the plate, making other products and services more cloud-enabled.

What do you think?  Are there any reasons you would not use this capability?  What are some other products or services that you hope will make their way into the cloud this year?

There are many challenges organizations face when it comes to administering and maintaining a dedicated pre-production or staging environment for which accurate performance testing can be conducted. The cost to manage and maintain infrastructure, along with personnel and facilities, can be fairly sizeable and are only a subset of the overall costs to be considered. So, in many cases performance testing can be rather expensive and this is exactly why virtualization can provide significant benefits because there is cost reduction across the board. A prime example is in many performance labs there are a variety of application performance tools typically utilized for testing – one such tool is HP LoadRunner or Performance Center. These tools are a primary part of a performance lab as they provide load generation capabilities and can accurately test applications under real world load and stress scenarios. However, these solutions require a significant amount of infrastructure and resources (A Controller to execute tests, LoadRunner Generators to produce user traffic, Virtual User Generator to record scripts, etc.) and this can make it very difficult to manage the environment when it has to scale to meet higher demand. In this case, virtualization saves time, effort and cost because resources can be allocated dynamically within the environment and any number of virtual machines can be leveraged when needed to handle these resource intensive applications. This is also enabling many organizations to architect and customize elegant configurations that more closely align with their testing requirements – which can minimize unnecessary infrastructure and resources. Yet, the prevalent issue many organizations still grapple with is how to execute performance tests that accurately depict the network for which the application will be deployed across.

The most pervasive approach that many organizations would take is to physically deploy hardware (remote load generators) in offices that they wanted to test an application from. This process was not only time-consuming, but also expensive, inaccurate and cumbersome to manage. For this reason, HP decided to form a partnership with Shunra to develop a seamless solution that provides this capability within the HP LoadRunner and Performance Center solutions – Shunra VE Desktop for HP. This solution aligns very well with the virtualization movement because it is simply a plug-in within the HP products that introduces the network into the existing test bed and can be leveraged across most virtual platforms. For LoadRunner 9.5 and later, there is simply a “WAN emulation” tab that can be accessed from within the Controller to introduce the latency, jitter, packet loss and bandwidth constraints directly into the test. With Performance Center 9.5 and later, this capability can easily be configured directly from the browser UI to allocate WAN emulation parameters across any number of desired load generators. A consequence of this is that organizations can leverage on-demand performance testing from a dynamic virtual environment that is agile, flexible and robust. This therefore eliminates the need to manage testing cycles across multiple time zones and remove any need for additional hardware. Additionally, all of the network metrics from each generator utilizing WAN emulation within the test will automatically be imported into the controller, which can save a significant amount of time when collating results and generating analysis reports. These reasons are precisely why numerous organizations have decided to improve their existing performance test environment with the VE Desktop for HP Software  solution. Not only does this solution address a strategic gap within the functionality of the HP solutions, it embraces virtualization as a means to more effectively administer performance testing. Overall, the VE Desktop for HP Software solution was co-developed with HP to considerably enhance the accuracy and value of these application performance test suites.

Are you migrating over to the Web 2.0 world or considering adding modern architecture and components? If so, take a look at this interesting video I came across, in which Mark Tomlinson and Steve Feloney from HP, our partner, tackle the challenges of load testing Web 2.0 applications. It will be well worth your time to check out Mark and Steve as they address the top issues that make LoadRunner web testing much harder and more complex. Very helpful and it’s only 4 min long!

In this video, Mark and Steve talk about real challenges their customers faced when migrating to the Web 2.0 without considering the impact of WAN latency, and most importantly how to solve this! Anyone using VE Desktop for HP Software should also definitely check this out and take a light-hearted look at web load testing.

This video is the first in a series put together by Steve and Mark covering some top issues that make LoadRunner web testing much harder and complex. Stay tuned for the next part of “Real Stories of Load Testing Web 2.0: Impacts of WAN latency of Web 2.0 apps”.

…will Steve end up going to Paris? and what will Mark find on google? the plot thickens…

Be sure to look for the little girl playing hide-and-seek with the camera and the German police patrolling the train!

That sounds like a lot of clouds!  What we are simply trying to understand is where and how to deploy your application in the cloud, so it will perform well for your customers.

If you take the case of deploying an online store into the Amazon EC2 cloud, there are many things to consider that will impact your customer’s experience.  Two of the most important are where to deploy what, and how your store will function over the Internet.  To compound the issue, where you physically deploy machines within EC2 is not clear.  Amazon goes to great lengths to avoid you knowing where their zones are.

Fortunately, you cannot only find solutions to these problems, but with cloud computing the process gets a lot easier.  To break the problems down into a logical flow, you’ll first need to know the demographics of your customers.  Where are they?  How will they access the store?  What needs to happen for them to have a positive experience at your store?  If you can answer these questions, you’ll likely come up with some numbers that place percentages of your users in different regions around your target market (US, EMEA, global, etc.).  Further, you’ll know the top two browsers they’ll use and with what kind of connections they’ll access the internet (e.g. Firefox with a 768/256kbps DSL, or IE8 with dial-up – yes people still use dial-up!).  Finally, you’ll have some way to quantify a “positive user experience”.  That usually involves consistency (does it work?) and speed (how fast?).

Using the information above and HP LoadRunner with WAN emulation, you can quickly build a to-scale test in your cloud-based lab and understand what would happen if you hosted your application all in one place and serviced your entire customer base.  Odds are that the test will not meet all of your performance goals the first time you run it.  Most likely, you will need to adjust some things in your store (e.g. reduce chattiness and image resolution, optimize Web 2.0 use) and then begin the discussion about where to deploy what.  Can you keep all of your database servers in one place?  If you have to split your database servers, how will that affect performance?  Will you need to deploy web servers in every zone, will you be okay with one in each region, or will you need something in-between?

Again, not only can answers to these questions actually be answered, but by using cloud-based testing you can arrive at valid conclusions faster than ever.  To understand the network impact of deploying machines in different zones, network performance measurement tools like VE Network Catcher can run in each of the zones to measure network performance between zones and to customer representative endpoints.  Importing that data directly into HP LoadRunner with WAN emulation and using the flexible cloud environment, you can rapidly form and either dismiss or validate hypotheses about how your application should best be deployed.

The benefit of testing in this way is that you can experiment with many different configurations in rapid succession to find the one or few that are most appropriate for your needs.  Trial and error with a live application is not advisable.  And building a test lab with physical machines to simulate the same thing is not only significantly more expensive, but a lot slower process.  As I pointed out in my previous blog, the same ROI formula that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.

Now you know what I think.  What do you think??

The following posts will answer this question and more, but in order to get answers we need to first address additional questions:

Consider the remote user in Tokyo, he is accessing multiple applications that are all hosted in the remote NYC data center.

Will all these applications perform the same way once hosted in a remote data center?

The obvious answer is NO, some applications will perform well even when users are remote to the data center, while others will provide intermittent poor performance and some will always perform poorly for a remote user.

The answers to the next questions are less obvious:

• Why do some applications perform well in remote data centers while others fail miserably?
• What is it about the way applications are designed and architected in a remote data center that allows some applications to perform better than others?
• What are the key design flaws that cause applications to perform poorly in a remote data center?

Since there are a lot of different applications, there are also a lot of different answers to these questions. In the next couple of posts we will focus on enterprise-data-driven- transactional applications and their performance flaws over the network (ignoring for a moment back-end or desktop related bottlenecks). We will further limit this category to client server based applications, ignoring for a moment the complexity of N-Tier applications and multi web service based applications, these will be dealt with in future posts.

Over the years I was asked to analyze performance problems for many transactional applications and specifically analyze their performance degradation when users are remote to their data center, the following are the key application metrics that I found are related to performance degradation over the network:

1. The number of application turns per transaction (or how chatty the application is)
2. The transaction size (or how much data needs to be downloaded from the server to the client in order to complete each transaction)
3. The transaction efficiency factor (how much data a transaction downloads per application turn)
4. The blocking nature of object retrieval (can the transaction retrieve multiple objects concurrently or is each object download blocking other object requests from being processed)
5. The transaction redundancy metric (or how much of the same data is being retrieved by all the requests made by this transaction). It seems like this metric should always be zero, but you will be surprised how often this is the single reason behind performance problems.
6. The transaction initialization size (how much data does the transaction download initially Vs. sequential navigational steps)
7. The caching ratio (how much data is cached locally as a percentage of the overall data needed by the application)
8. The latency scale factor (How does the backend’s ability to scale change when network latency is added between front end clients and the backend)

Measuring and observing these metrics allows for the deep level analysis that is required to identify performance bottlenecks. This information also helps to point out to application developers and system engineers what needs to be changed in order to remediate application performance problems.

In the next couple of posts I will explain each of the above factors and describe how each of them impacts application performance, so sign up for the RSS feed to get notification on these future posts.

Until then…

Amichai Lesser

Performance Test Lab testing

Pros: Controlled, repeatable, relatively easy to debug observed problems

Cons: Expensive to build and maintain, commonly implemented without real world network considerations

Live Network testing

Pros: Cheap, good for periodic sanity checks of performance

Cons: Puts non-revenue generating load on the production network, very difficult to debug, high volume testing is not an accurate depiction of end user performance

The good news is that developments in cloud-based testing and in HP LoadRunner have been able to improve the accuracy of Performance Test Lab testing while also reducing the complexity and overall cost.  By moving HP LoadRunner load generators into a cloud-based testing environment, your testing can immediately benefit from the flexibility and scalability of the cloud.

One specific use case is for Peak Testing.  Peak testing is simply another name for load testing.  What “peak” implies is that it is at a scale that is not common.  If you are running, or want to run a test that is not commonly run in your environment, then it is fair to assume that you’ll need to plan ahead to gather the necessary resources.  Working in a cloud environment makes gathering and configuring extra resources on demand easy.  Need an extra five load generators?  No problem; just clone your existing one and wait for them to start.  Need more RAM?  No problem; just shut the machine down, configure more RAM, and restart the machine.  Working with the latest version of HP LoadRunner with WAN emulation makes the story even more compelling.  Need load generation from LA, London, and Tokyo?  No problem; just configure the load generators to emulate those locations and they will.  Sound easy?  It is.  Companies like Skytap already have thought this scenario through and have a great subscription-based model built for you to use today.

The real benefits of peak testing in the cloud are that you can quickly scale your lab up and down without the burden of maintaining a lab large enough to support your peak needs –saving time and money.  It is the end of costly lab build-ups, running tests during maintenance windows in the middle of the night, sending load generators to all corners of the world, and putting test traffic on your production network!

Of course, there are a lot of other ways Cloud Computing can be used in a QA environment; validating new versions of test products and proof of concepts are two that immediately come to mind.  In the end, the same ROI formulae that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.

Now you know what I think.  What do you think??

P.S. Stay tuned for a coming blog on location-aware deployment testing in the cloud…

In alignment with global trends, more and more clinical trials take place outside of the United States, while the documentation and analysis is done in the US for submission to the FDA. So it is not out of the ordinary to find an Oracle Clinical user in Eastern Europe or in China submitting data to an Oracle Clinical Server hosted in the united states.

Unfortunately, as commonly found with an off the shelf software package that gets customized over time, this type of application can easily morph into a performance challenged application, especially for remote users. The following analysis shows an example of an Oracle Clinical transaction that completes in 8 seconds for local users, while extending to over a minute when users in E. Europe tried to use it.

Transaction Response Time Analysis for an Oracle Clinical Application Accessed by Local Users Vs. Remote Users in E. Europe

Obviously, that big jump in response time was a great cause for concern with my client’s management. Which is how I got the privilege to be asked to analyze the root cause of the poor performance.

The first thing I noticed was that one particular transaction (Connect to DB) had an extremely high jump in TRT from 8 seconds locally to over a minute when accessed remotely, so I focused on that one first.

Looking at the network fingerprint of the transaction you can observe the following things:

The transaction size (amount of data downloaded from the server required to complete the transaction) is extremely high (over 5 MB).

The transaction generates 34 HTTP calls. Some of those calls take disproportionally longer time than others (notice the Get jar file calls that each exceed 19 seconds marked below, click on the image for a larger view)

Oracle Clinical, Transaction Analysis for "Connect to DB"

When inspecting those JAR files, we saw that they were very heavy in size, over 1 MB each which is pretty big for basically a collection of Java classes that are zipped into a JAR file.

You can also observe in the next bounce diagram that most of the JAR files are downloaded in a serial fashion, blocking other objects from downloading in the mean time. Notice the delta time displayed on the right column

Bounce Diagram for an Oracle Clinical Transaction

What can be done to improve the performance of this application?

Oracle Clinical uses JAR files to package Java code that will run on the client. It is not uncommon for customized off the shelf applications to increase in code size over time until the code is very bloated.

There are several best practices that developers can follow to reduce the size of JAR files:

1. Rationalize the code – as applications develop over time, multiple classes and sometimes adjacent projects might reference similar libraries and other assets. If not careful those libraries and common assets end up packaged multiple times inside the JAR file, causing it to inflate in size.
2. Minify the code – a quick Google search for “reduce the size of JAR files” will reveal several free tools that can minimize the size of JAR files, usually through eliminating white spaces, comments, shortening variable names, etc.
3. Defer loading – chances are that not all the code in the JAR file is needed for the “connect to DB” transaction, which means that users that only want to perform a small transaction are penalized by the download time of code that will never get executed by them. Deferred loading is a design pattern that simply says “only download assets or code when it is needed” I wrote more on that design pattern in this previous post “http://www.shunra.com/shunrablog/index.php/2009/01/29/content-loading-when-being-lazy-pays-off/”

In summary, as more and more web 2.0 technologies hit main stream, new forms of performance pitfalls may present themselves, requiring us to consider application performance for remote users even for applications that traditionally were considered “remote user friendly” such as web based applications.

How was this Analysis Performed?

This analysis was used by integrating a HP LoadRunner script modeling an Oracle Clinical business user with Shunra VE Analyzer. So as the scripted transactions executed across a virtual WAN (simulated by the Shunra VE) the beginning and end of each transaction were marked by the VE Transaction Manager. At the end of the test, the VE Analyzer had sufficient data to generate the attached reports, enabling us to pin point the root cause of the performance problem. You can learn more on how to set this up by going through Shunra Certified Performance Engineering training, either on site or at the next Shunra University. Learn more about Shunra training here:  http://www.shunra.com/shunra-university-overview.php and here http://www.shunra.com/training-overview.php?keyword=services

Questions, comments, feel free to write me at amichai.lesser at Shunra dot com. Or comment on any of my posts.

BTW, we just created a new Application Performance Management Group on LinkedIn, feel free to look it up and join here  http://www.linkedin.com/groupRegistration?gid=2200667

Best,

Amichai Lesser