My friend Mark Tomlinson from HP recently wrote an informative blog about “Understanding the language of hosted load testing.” The blog touched on two competing approaches to application performance testing that were referenced as “behind the firewall” and “outside the firewall”. Behind the firewall testing usually means testing in a lab environment such as a Performance Test Lab. Outside the firewall testing means testing over some part of a live network. As you can imagine, and as Mark pointed out, both approaches have their pros and cons. Selectively borrowing from Mark and adding some of my own gives us the following:
Performance Test Lab testing
Pros: Controlled, repeatable, relatively easy to debug observed problems
Cons: Expensive to build and maintain, commonly implemented without real world network considerations
Live Network testing
Pros: Cheap, good for periodic sanity checks of performance
Cons: Puts non-revenue generating load on the production network, very difficult to debug, high volume testing is not an accurate depiction of end user performance
The good news is that developments in cloud-based testing and in HP LoadRunner have been able to improve the accuracy of Performance Test Lab testing while also reducing the complexity and overall cost. By moving HP LoadRunner load generators into a cloud-based testing environment, your testing can immediately benefit from the flexibility and scalability of the cloud.
One specific use case is for Peak Testing. Peak testing is simply another name for load testing. What “peak” implies is that it is at a scale that is not common. If you are running, or want to run a test that is not commonly run in your environment, then it is fair to assume that you’ll need to plan ahead to gather the necessary resources. Working in a cloud environment makes gathering and configuring extra resources on demand easy. Need an extra five load generators? No problem; just clone your existing one and wait for them to start. Need more RAM? No problem; just shut the machine down, configure more RAM, and restart the machine. Working with the latest version of HP LoadRunner with WAN emulation makes the story even more compelling. Need load generation from LA, London, and Tokyo? No problem; just configure the load generators to emulate those locations and they will. Sound easy? It is. Companies like Skytap already have thought this scenario through and have a great subscription-based model built for you to use today.
The real benefits of peak testing in the cloud are that you can quickly scale your lab up and down without the burden of maintaining a lab large enough to support your peak needs –saving time and money. It is the end of costly lab build-ups, running tests during maintenance windows in the middle of the night, sending load generators to all corners of the world, and putting test traffic on your production network!
Of course, there are a lot of other ways Cloud Computing can be used in a QA environment; validating new versions of test products and proof of concepts are two that immediately come to mind. In the end, the same ROI formulae that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.
Now you know what I think. What do you think??
P.S. Stay tuned for a coming blog on location-aware deployment testing in the cloud…
October 30th, 2009 at 7:33 pm
Nice article David. Thanks for sharing your thoughts. LoadRunner is the high-end legacy load testing tool, and your points are well taken about how cloud makes life easier and cheaper. Have you tried any of the new cloud testing services that were built from the ground up to dynamically allocate/manage all the load generation resources necessary? They abstract all of this layer so you can focus on the performance testing and tuning that is unique to your application. The big benefit is time and money. Some of these services can let you run performance tests for 5,000 concurrent users for $300 per month. And the kicker? You can run 100 test iterations for that same $300!
That’s the real game changer in my opinion. Please take a look at http://loadstorm.com It may be too simple for your needs because you are already a LR guru, but for the 90% of web developers in the world that can’t afford LR, this native cloud load testing tool can be a tremendous addition to the bag of tricks.
Feel free to contact me if you want to collaborate on any performance testing blogging.
Thanks again and good luck.
November 19th, 2009 at 5:05 pm
,..] http://www.shunra.com is another useful source on this subject,..]