You have probably encountered a situation where pre-launch load tests did not correlate with the way it performed in production.  There are a lot of reasons why this can happen, but for now, I want to focus on 3^rd party content providers and Content Delivery Networks (CDN).  Almost everyone running a business today has some part of their application or Web presence farmed out to a 3^rd party.  Think about what happens when you go online to buy gifts for the upcoming Holidays.

As you browse for that perfect tie for your dad, the site is likely to be serving static content from Akamai or Limelight, adds from 3^rd party providers, and a shopping cart that may not be hosted by the company you are dealing with.  When you actually purchase that Tickle-me Elmo for your niece, the site processes your credit card by interacting with a financial institution and provides shipping tracking numbers by interacting with a major shipping company behind the scenes.  All that typically happens as you wait for the page to return in a few seconds if all goes as it should, but any of those 3^rd party providers have a strong likelihood to disrupt the page load and make the site look like it is not responding.

Increased delays from these and other 3^rd party providers do happen and you should definitely test for those scenarios, but how do you do that when you really do not have control over those sites. You definitely should perform load tests on the site as a whole to ensure that each 3^rd party provider will scale to your targeted peak traffic and perform within their SLA as expected. In addition, you should think about how slow your 3^rd party providers can get without affecting the performance of your site.

“How Slow Can They Go?” Yes, a corny statement, but to the point! You definitely need to identify how slow your 3^rd party providers can go before it starts to affect the performance of your sites.  The real dilemma is replicating the traffic from the 3^rd party site, which in the case of financial transactions can be very difficult plus your team can waste a great deal of time trying to do this. I’m suggesting that you don’t try to replicate that traffic, instead use the actual 3^rd party traffic from the 3^rd party site during the test. Here’s the catch, you impair the traffic coming from the 3^rd party site real time. Simply single out that traffic in your usability studies using a tool like Shunra’s VE Desktop to measure performance of the site with the 3^rd party adds impaired or by slowing down the response from your credit card processing partner using a Shunra VE Network Appliance on your backend systems during a maintenance window.

Now think about scalability load testing for a minute. I recommend that you do not implicitly trust 3^rd party and Content Delivery Network (CDN) provider’s scalability estimates and guarantees. You are probably scratching your head thinking that your CDN’s and other 3^rd party content providers have massive server and network infrastructures that are impossible to overload. While it is true that the major 3^rd party providers have massive infrastructures, the real question to ask yourself is this, “How much of that infrastructure is your content riding on?” Let’s just say that you actually do know how much of their infrastructure is dedicated to your cause, you still have to ask yourself the question, “Do I know for sure that it will handle the peak traffic?” Adding in impairments during a load test is easy using Shunra’s VE Suite or Shunra for HP Software which works seamlessly with Load Runner.

During my career I have load tested literally hundreds of different sites, and many of those sites had CDN and 3^rd party content provided by very large companies that did not hold up to the load during a stress test. Let’s face it, regardless of the size, the CDN or 3^rd party provider’s service can only scale to the level of infrastructure that is available to your site.  CDN content can fail to scale due to complex caching configurations that can get screwed up with one incorrect setting.

In more than one situation, I had a customer tell us that they were going to inform their CDN or 3^rd party providers to “be ready” for the peak traffic that our load test was going to generate. They were usually shocked when I asked, “Why?” Their answer was typically, “We need to be responsible partners and let them know anytime we are doing testing.”  Clearly, I would normally agree with that statement for many different kinds of testing, but not in the case of load testing against an established Service Level Agreement (SLA)!  Here is the bottom line, if the CDN or 3^rd Party has agreed to an SLA then that system should be ready to handle that load any time (barring timeframes specified in the SLA for maintenance windows, etc.).

In several load testing engagements, where the client let the CDN or 3^rd party know about the load test, there were “additional resources” stood up to support the load test. Surprisingly, the system scaled during those times, but failed to scale when no notice was given ahead of time. Does that sound suspicious to you? If the answer is yes, then we are on the same mental wavelength. If you encounter problems with your 3^rd party provider not scaling during your load test, and they will probably insist on “supporting” you during the next load test. Be sure that you ask your 3^rd party providers the following questions:

What did you do to support this load test Vs. the last one?

What did you change in your infrastructure that allowed it to scale successfully this time?

Did you add infrastructure or change a setting?

In one situation, I heard a 3^rd party provider tell my client that they stood up an extra bank of machines to support the testing that was happening that evening. My question to the customer was, “Does that sound strange to you?”

At the end of the day, we all need to admit to ourselves that even our monstrous 3^rd party CDN partners can slow your site down, so you really should test how your site performs to the end user and how the infrastructure scales when they do slow down. Try to find out how slow each 3^rd party provider can get before it impacts your site and ultimately the end user’s response time and build that into your Service Level Objectives and ultimately let that drive your Service Level Agreements with 3^rd party content providers.

     Bruce Milne of CA provides a great overview of how Hyperformix helps IT meet this challenge:

 http://www.hyperformix.com/resources/videos/hyperformix-101 

     A key takeaway is that CIOs are leaving $millions on the table by utilizing < 20% of their server resources in production. A key reason is they lack confidence in their ability to accurately predict production resource requirements given normal business growth, peak period load, seasonal cycles, etc. So, they pay for headroom – way too much.

    In the Hyperformix Capacity Planning best practice, the “Test to Production” phase takes data from your test lab, applies these results to a model of your production network architecture, and then predicts your performance in production. From Hyperformix Web Site:

“Turn Load Testing into Insight – Load testing is a critical component of mission critical product releases. HP Performance Center and LoadRunner users can leverage the data generated by load testing into models that accurately portray how the application will behave in production — no need to be concerned that the test lab is different than production. Optimizer will identify bottlenecks and illustrate parts of the infrastructure that are overbuilt and can be consolidated.”

    A critical flaw of the “Test to Production” phase can occur if you use load test results that are not based on tests that include your production network behavior. The response times and server resource consumption patterns of 1000 “virtual users” in a lab vary significantly compared to 1000 users accessing apps from various network locations while competing with bandwidth limits, latency and occasional packet loss.  We’ve seen system scalability drop from 1000s of users to 100s of users, and response times rise from several seconds to several minutes, when apps move from a test lab into production. Given the limited accuracy of the current best practice it’s no wonder CIOs pay for lots of headroom.

    The good news – HP has solved this problem with the Shunra for HP Software product. The “Test to Production” phase can now include load testing that easily incorporates the production network behavior of your remote locations – and the solution is built-in to HP’s Controller and Load Generator.  Shunra is offering a free 30-day trial license that turns on this feature inside LoadRunner (Version 9.5 & above). You’ll be surprised when you see the significant difference in your test results accuracy. You’ll also gain more confidence in reclaiming some of the 80% of unused server capacity sitting in your data center.

We will be demonstrating this solution at the IXIA booth at Interop Vegas between May 19th. and May 21st.  at the Mandalay Bay Convention Center.

See you there,

Amichai