Improve Test Result Accuracy and Reliability by Discovering and Emulating Real-World Network Conditions

Application performance is the number one factor impacting application success and end user adoption. It also remains the number one cost when it comes to developing, managing and maintaining applications. The rapid adoption and migration to mobile and Cloud applications, and the increasingly distributed nature of application architectures and end user populations, complicates previous methods for managing and ensuring application performance. In fact, traditional methods of managing application performance are no longer reliable and often result in false assurances that end up costing businesses millions of dollars when in-production performance issues cause lost revenue, lost productivity, unnecessary IT remediation costs, and brand damage.

Traditional strategies for ensuring application performance are incomplete. They do not account for the dynamic nature of today’s global networks, and are therefore not reflective of real end-user experience. For example, load testing in a pristine test environment will provide some insight into system scalability under load, but without an understanding of the network conditions experienced by different (virtual) user groups, assumptions made regarding capacity requirements will be misleading. In this example, if one of the virtual user groups is a mobile audience, you need to understand how the longer user sessions that result from the inherent latency of mobile networks will affect system resources.

In the same manner, functional testing in a pristine test environment is not indicative of how a user will experience an application or website. Functional test results may confirm that button A leads to transaction B, but if that transaction takes 30 seconds or longer in the real world due to network constraints, end users may not wait long enough to see the results.

Right or wrong, end users expect faster and faster response times, even when accessing data across slower mobile networks. End users do not account for latency, bandwidth, jitter and packet loss. For an end user, it is simply a matter of how long it takes to receive requested content. And, end user patience is waning:

• 50% of mobile users abandon a page if it doesn’t load in 10 seconds.
• 40% of mobile shoppers will abandon an e-commerce site that doesn’t load in 3 seconds.
• 60% of mobile users who abandon a page due to poor performance will not return to that site.
• 500ms of latency increases bounce rate 4.7% and decreases pageviews 5.7%.
• A single second of web page delay could cost a company like Amazon.com to lose up to $1.6 billion per year

The need for a new and more reliable testing strategy is further highlighted by advancements and enterprise initiatives to leverage technologies like mobile, Cloud and virtualization.

Network Virtualization

Network conditions such as latency, limited bandwidth, packet loss, and jitter, are all critical factors that must be taken into account when testing or validating application performance, especially when considering the multiple network connections required to support 3^rd party services and external resources.

Just as it is challenging and costly to bring real-world services into the test environment, the real-world networks affecting these services and the end user represent, at best, a daunting task to incorporate into a test environment isolated behind a firewall. The global mobile network, for example, is a communication channel that is in a continual state of flux, with dynamic conditions dictated on a moment-by-moment basis according to carrier, connection type, connection strength, time of day, location and myriad other factors.

In the absence of real-world conditions, testing will yield unreliable results as the effect of the network on the end user experience, as well as communication with external services, remains unaccounted for. As with services, virtualizing network conditions in the test lab is essential to creating a realistic environment for testing that yields accurate and reliable insight into application behavior.

With the massive global uptake in mobile devices and the rise of cloud computing, the effect of the network has become even more pronounced since conditions across the “last mile” affect performance more than other factors, and mobile conditions in particular are intrinsically dynamic. Therefore, incorporating network virtualization is particularly crucial when any mobile or cloud component is part of the scenario.

Shunra's Network Virtualization for Software Testing Solutions

The first step required to implement network virtualization is to “discover” the real-world network conditions affecting end users and application services. Shunra NetworkCatcher unobtrusively captures production network constraints like bandwidth, latency, jitter and packet loss. Recorded interval statistics can then be re-played, or virtualized, in the test environment, using the Shunra NV suite of products, including Shunra NV Appliances, Shunra NV Desktop, Shunra NV for Mobile and Shunra NV for HP Software.

With Shunra NV, virtualized network impairments, such as high jitter rate and limited available bandwidth, can be emulated in the test lab so that the effect of the network on transaction response time can be reliably and accurately measured.

In addition to capturing real-world network conditions, Shunra’s network virtualization solutions for software testing include a Global Library of mobile and broadband network conditions. This library, built on millions of data points, provides typical, best and worst case conditions recorded from major cities around the world.

With Shunra’s network virtualization technologies, custom impairments can be applied to determine the specific effect on application and network behavior in order to emulate worst case scenarios or to troubleshoot specific in-production problems. Shunra’s technology also provides the ability to execute tests, analyze results and remediate or optimize performance.

For more information on Network Virtualization, and a real-world example of an integrated Network and Service Virtualization approach, please read this solution brief from Shunra and HP.