Have you ever tried to run a test, only to find that your Load Generators half-way around the world are not responding?
If any of that sounds familiar, then you might want to consider using network emulation instead of deploying Load Generators around the globe. By using an emulation solution, you can bring your testing back into a controlled lab environment. The benefits of doing that include:
- Repeatability: by controlling your whole environment, you can quickly run and reproduce exactly the tests that you want.
- Reduced cost: no need to install and administer remote machines, or the people to maintain them
- Reduced time: stable testing environment means less test runs to get reliable results, less setup time – ever try to download a 1GB installer over a slow network?
And many more…
The first question that should come to mind is, “will an emulated solution be as accurate as using the actual production network?” The short answer is yes. The long answer needs to point out that testing over the production network is not “accurate” in the first place. Example: if you run your load tests during off hours, what relevance does that have when the actual users try to work during peak network times? Very little.
If you’re still convinced that you need to model you’re production environment to a high level of accuracy, then you’re still in luck. With network emulation, it is possible to closely model production networks. When measuring on average transaction response time, we typically see correlations around 95%. It should be pointed out that, to get these numbers several runs over the production network are needed so a true average can be understood. When running in an emulated network environment, a single test run is usually sufficient.
Practically speaking, the 80/20 rule applies here as well. If you can get your emulated network to perform at 80% or closer to the actual conditions that your users will operate in, then you’ll be able to far exceed any testing value that you’re getting out of deploying Load Generators around the network.
Remember that network emulation empowers you to return your testing to what it was originally designed to be – a scientific experiment. Once you use network emulation to get all of the variables under your control, you can quickly configure and run tests, troubleshoot, and present results with confidence.
December 2nd, 2008 at 14:19
I’ve worked in performance testing my entire career, and this seems like a no-brainer. I’m amazed at how many companies still rely on remote generator deployment for their testing efforts when those generators are logistically challenging to deploy, cumbersone to administer, and produce questonable results (because testing is typically ocurring in off-, non-peak hours, when the network is lightly utilized and “available” for testing). Particularly these days, when teams are being asked to do more with less, Shunra’s approach makes a lot of sense to me.
December 3rd, 2008 at 13:46
Dave, I have one other comment. Load tests should be repeatable, so that if any variable is changed the comparison between “A” and “B” will be valid. When using the actual production network for testing, any number of changes can invalidate the ability to compare against a baseline.
December 17th, 2008 at 11:12
Has anyone ever done a comparison? Is there any studies out there that show what the accuracy between WAN emulation and sending remote load generators out? I think WAN emulation sounds great and make sense but I need to prove it to my boss and counterparts.
Any link would be helpful..
January 1st, 2009 at 12:48
Jack,
I have been involved in several comparisons of a remote load generator and one that is running in a WAN Emulation lab. The response time accuracy varies between 92% – 97%, meaning for example that a transaction that took 7 seconds in a remote generator took 6.8 seconds in an emulated lab.
Most of these studies were done with customer scripts so they are proprietary, but one study was done with SAP as part of their evaluation of Shunra for the Co-Innovation lab. This one involved a load generator in Germany compared to a load generator in the lab. I don’t have the data from the comparison but judging by the fact that Shunra was selected for the Co-Innovation lab speaks for itself.
You can read more about the SAP relationship here: http://www.shunra.com/news.aspx?newsType=1&newsId=397
Hope this helps,
Best,
Amichai
May 20th, 2009 at 07:01
Great article / hope to definitely come back again:)