Given the application traffic requirements, the challenge is not going to be the amount of traffic (Throughput) , but rather the network conditions, or impairments, that are affecting how the application traffic is going to be transmitted over the network, eventually arriving at the corporate data center south of the nation’s capital.

Using VE Catcher 4.6, a model of the latency and packet loss characteristics was created, for a period of 10 days. Two separate devices where utilized, on one AT&T 3G network, the other, on the Sprint network, to ensure carrier availability (the project call for a 50% split among the networks).  A pilot site, south of the data center, was selected, and VE Catcher created two parallel recordings, one for device “A” and another for device in the alternative carrier.

Once the behavioral model has been created, Shunra’s Virtual Enterprise appliance will allow creating a “test-bed” where the application will be sending real traffic to the data center over an emulated link. The network conditions will be imported from VE Catcher and reflect the exact conditions observed at each of the 3G carriers.

The performance of the application, running over the emulated links will allow the testing team to predict the behavior of the application well before field deployment is completed, and make adjustments required by the application and also predict behavior under extreme conditions, like heavy network traffic, inclement weather or other events.

You may not always use WAN emulation, but when you do, use Shunra.

Shunra Virtual Enterprise is your guaranty to a successful deployment.

When do these issues become as important in LANs as they are in WANs? Probably when our LAN becomes huge – such as Cloud computing Data Centers.

These Data Centers are interconnected with each other and one of the main differences with our traditional Data Centers is that they are managed automatically by very complex and smart management systems. These systems are capable of automatically allocate hardware and software resources for our applications, maintain these resources and expand/decrease capacity of the infrastructure based on the application demands almost in real time.

So imagine that the system is scheduled to perform an OS upgrade on the managed infrastructure: it will download OS images on thousands of machines thus creating huge traffic congestion within the Data Center. What will happen to the system that knows only how to work on LAN and was never tested on coping with delays? Packet Loss? – it might go down partially or completely. Since it was not tested – how can we know? (Sounds a little like Skynet scenario in Terminator, doesn’t it?)

The importance of the management system performance becomes utmost in this case and it makes sense to test how the system will react to the issues that now might happen within it, doesn’t it?

This is where WAN emulation solution can help – it can create those conditions within the Data Center or between the Cloud Data Centers and the “unexpected” can become “expected” and fixed.

While actual deployment may be month away, it is important to remember that the transport mechanism, open Internet over an IP Core will require substantial testing of the end user experience. IPTV is a protocol sensitive to network conditions, like latency and packet loss.  Any successful implementation will have to include significant bandwidth allocation testing, as well as load under latency, jitter and packet loss.

However this series of posts is about the day after a data center move. Now that the data center is remote, how does this paradigm shift impact the way we should develop, test, deploy, monitor and troubleshoot applications. I will try to cover as many topics as possible, but the main focus is still going to be around the role application performance management plays in this new paradigm.

I will start by covering the key reasons behind the performance impact that is experienced when applications are hosted in a remote data center? Those reasons are fairly intuitive, but it is important to understand them in depth in order to adequately plan for those new conditions. Two main things impact how applications perform when application servers are hosted in a remote data center vis a vis their application clients :

1. The performance of the network link between the client and the remote data center. This performance is defined by a set of network performance metrics that are application independent (for now we will ignore application aware networks, however the following basic concepts still hold in this scenario as well).

2. The application efficiency, specifically how efficient the application is when transferring data between the client and the remote server (and other tiers if applicable). This is an application attribute (and some time an attribute of a specific business process within the application). These attributes are application specific and are independent of any underlying network.

Let’s start with understanding the network performance metrics. Consider the following scenario:

An application is hosted in a NYC data center, with users in 2 places, some are in a NYC headquarters next to that NYC data center and some are in a remote branch that is located in San Francisco. The question is: “will the application perform the same for both type of users (local users in headquarters and remote users in SF)? In other words will the application be as responsive to the San Francisco user as it is to the NYC user?”

Well the obvious answer is NO, in most cases a NYC user will enjoy a faster more responsive application. What is less obvious is why? What is it about the network that causes remote users to experience a slower application than local users? The rest of this post will cover that question, future posts will address the application specific attributes. Once we cover that we will be ready to examine best practices in building applications for a remote data center.

When I ask this question during my training seminars, I get a variety of answers, many of them are the right ones, but I would like to address one wrong answer that keeps repeating itself for some reason.

Collisions – there is a general conception that collisions are common phenomena on the network which can explain any bad thing that happens to applications. The truth is that collisions are almost a thing of the past (on Enterprise LANs anyhow) and even when they happen they can’t explain why a remote user has a worst experience than a local user as both will experience a similar collision chance since collisions is a phenomenon that happens on local area Ethernet networks. If there are collisions on the Enterprise LAN it usually points to a configuration issue on a network device (like a duplex miss-match) but is still unrelated to the answer to our question.

Now to the right answers to the question, what is it about the Wide Area Network that causes applications to slow down:

There are 5 key conditions that predominately exist on Wide Area Networks and impact application performance, each in their own way:

1. Network Latency – the time it takes a packet to traverse from a source to the destination across the network, measured in milliseconds [msec]. A typical WAN link will introduce latency in the range of 10msec – 500 msec.

2. Bandwidth constraints – how fast can data be processed by the network link, measured in bits per second [bps, Kbps, Mbps, Gbps]

3. Bandwidth utilization (background traffic) – the percentage of bandwidth that is utilized by traffic that already exists on the link (background traffic).

4. Jitter – the deviation of the inter packet gap of sequential packets across a network link, it is a result of the deviation of the network latency and is sometimes used interchangeably with that standard deviation, measured in milliseconds [msec].

5. Packet Loss – the chance to drop a packet across an end to end network link, measured in %. Sometimes presented as the inverse metric called packet delivery rate.

The above are called network impairments, you can click on each one of the links to learn more about them and their causes.

Network impairments are performance conditions that inhibit the flow of data across a network. Each impairment type has an impact on the performance of business applications and network services. Some applications may be very sensitive to network impairments and some may be almost network agnostic. Sorting applications based on their network sensitivity is one of the important steps in performance engineering

In the next post we will discuss how application design can impact performance across the network. But in the mean time I would like to introduce a question for the group:

“We identified network latency as one of the key reasons that impact application performance; we also said that a typical WAN link will introduce 10 – 500 msec of latency. The question is, why does network latency have a big impact on application performance? surely a user doesn’t notice an increase of a few msec in response time, even 500 msec = ½ second goes by in a flinch. So think about it and let me know what you found based on your experience, why does network latency have such a big impact on application performance?”

Until next time,

Amichai Lesser

Which is why I was glad to stumble upon Shamus McGillicuddy article in www.searchenterpriseWAN.com titled “WAN engineers prepare networks as cloud computing adoption rises” that can be found here.

There are some good observations in the article, I especially appreciate the comment on how application architecture best practices such as minimizing application turns become key when developing applications for the Cloud as well as testing applications for network latency becomes critical as the enterprise is migrating more and more applications to the Cloud.

Many questions still remain though and the impact of Cloud computing on performance engineering is still to be determined. Some of the questions that are on my mind are: (and feel free to chime in with more questions or attempts at answers)

1. How do you manage the performance of an application in the Cloud?

2. What changes if any are needed to the service deployment model and version upgrades?

3. How do you conduct a performance test for an application or a service in the Cloud (internal clouds and external clouds)?

4. How do you troubleshoot a performance issue in the Cloud?

Something to think about…

Best,

Amichai Lesser

• When networks get congested over a long period of time the router buffers get saturated which at some point leads to a situation where there is no room to store more pakets in the buffers, so new arriving packets get dropped
• Some routers implement a mechanism called random early detection, since IP doesn’t provide any explicit way to indicate that the network is congested, routers use loss for that purpose. Such loss is a way of communicating back to the users the need to scale back the offered load on the network. RED is becoming less common as new QoS techniques are used to implement rate control.
• Communications across wireless and cellular networks may encounter packet loss due to interference or a weak signal
• Hardware and cable errors are a common cause of packet loss

From the network perspective, loss can be categorized as random loss or loss due to congestion. The average packet loss rate for a network connection gives an overall sense of the quality of the connection. A connection with less than 1 percent average packet loss is considered a decent connection. But average loss doesn’t tell the whole story. There is importance to the type, or pattern, of packet loss. There are at least two kinds of packet loss that should be considered: ‘Random’ loss and ‘Burst’ loss. To explain the difference between them, let’s suppose we are trying to run 2 Voice over IP conversation over 2 links that have an average of 1 percent packet loss. Call A loses one packet in every 100 packets over the entire call (random loss) while Call B loses 100 packets in two incidents – at the beginning and the end of the call (burst loss). Which call would you rather have? That’s why it is important to consider not just the average packet loss but also the type of loss and information on any bursts of packet loss over time.

Of the topics covered, WAN emulation is clearly closest to my heart, but please be sure to read up on Microsoft Vista support, the new RDP agent, and what is happening with LoadRunner’s Analysis API.

To read the full story, click here.

Do you plan to move to LoadRunner 9.5 or Performance Center 9.5 in the near future?  Will the integration with Shunra’s WAN emulation motivate you to upgrade or accelerate your upgrade schedule?

In this post I will show you how to generate analysis in less than three minutes that addresses the question that naturally follows. “Why does this transaction perform well locally, but slows down under network conditions?”

This analysis can be generated by installing VE Analyzer on the “remote” load generator and configuring it with the packet analysis settings (reminder: the “remote” load generator is any load generator that has the VE Desktop Professional WAN Emulation client installed on it. This WAN Emulation client enables us to simulate remote network conditions from a local load generator in the lab. Hence the double quotes in “remote” since the generator isn’t physically remote but is simulated to behave that way)

Follow these steps to setup the analysis on the “remote” load generator:

In the LoadRunner Controller Design screen click on the Load Generators button

Select the “remote” load generator

Click on details and select the WAN Emulation Tab:

Check the “Enable WAN Emulation” check box (if not already selected) and click on “WAN Emulation Settings”

See the previous post on how to setup the WAN Emulation parameters in the following screen.

Click on the additional settings button

First check the “Capture Packets” check box, this enables the packet capturing agent on the VE Desktop Client on the LoadGenerator. I recommend assigning at least 100 MB of buffer space for the capture so at least one iteration of the script will be captured.

For web based applications, select HTTP in the “Automatically run post-test analysis” option, for other applications select either TCP or UDP analysis.

Finally, select a folder on the load generator for automatically saving the test results. It is important to re-iterate that this folder is on the “remote” load generator, not on the LoadRunner Controller.

It is recommended to setup only 1 virtual user to run from this “remote” generator, so the analysis will focus on a single instance of the user. If load is needed as part of the scenario, it should be generated from a second load generator, in this case the analysis will show results of the marginal user.

Adding the analysis capabilities in this example doesn’t require any changes to the script and can be applied to the same scenario we presented in the previous post. Running this test results in an analysis report that is generated on the LoadGenerator. This report can be accessed from the VE Reperter that is installed on the “remote” load generator.

VE Reporter stores the reports based on the test names and the test run dates, so we are looking for the last test run for our test.

Once the report opens we are mainly interested in 4 reports:

The Application Efficiency Report shows us how chatty the application is (application chattiness is directly related to response time slow downs due to network latency) as well as how long it took to download various resources from the server. In this example, we see a resource that took 9 seconds to download, explaining why the entire action took 11 seconds for a remote user Vs, 0.6 seconds for a local user.

The Application Efficiency Report highlights a resource that took 9.2 seconds to download

The second report is the Client Network Server Breakdown that shows how much time the transaction spent on the client, network and server.

A Client Network Server Breakdown Report

The 3rd report is the Bandwidth Analysis which shows how much bandwidth was used by this single application user.

Finally, for a real deep dive we can use the Bounce Diagram that displays the entire communication between the client and the server as well as the delta times between consecutive packets, high delta times can point out to blocking events within the transaction.

The Bounce Diagram shows deep packet analysis data on the transaction

In summary, in less than 3 minutes we can generate the above reports on any transaction that seems to be performing poorly over the Wide Area Network. This information can help the developers and network engineers to fine-tune the application and the network to improve the end user experience.

Feel free to suggest additional topics you would like to see covered in this series of posts.

Best,

Amichai

First off, VE Desktop for HP Software 5.0 is a new product.  It is in no way related to LoadRunner MWE.  HP/Mercury customers last saw MWE in LoadRunner 8.1 (July 2005) and the feature was removed from the LoadRunner 9.x releases.

While MWE was indeed “Powered by Shunra,” it was done so through an OEM relationship.  MWE was sold and supported by HP/Mercury.  VE Desktop for HP Software 5.0 development was a collaboration between Shunra and HP.  The product is the result of both companies learning from our experiences, designing, and co-developing a new solution that would exactly meet the needs of our customers.  VE Desktop for HP Software 5.0 can do everything LoadRunner MWE could, and much more.  VE Desktop for HP Software 5.0 is sold and supported by Shunra.

The top three issues we chose to address were:

1) Accuracy: Since the MWE did not support bandwidth emulation or jitter, the results of its emulations often did not fully reflect the real-world performance of applications being testing.  To address this, VE Desktop for HP Software supports bandwidth emulation (symmetric and asymmetric), jitter, imported network characteristics from Shunra’s VE Network Catcher, and a host of other network impairments aimed at precisely emulating network links.

2) Reporting: With MWE, there was no network specific reporting.  If a test was run with multiple locations emulated, the LoadRunner results would still show transaction response time and other metrics averaged across the whole test.  In order to understand how applications fared at individual locations, scripts would have to be modified so the script and transaction names reflected the location they were being run at.  This was very challenging to implement and maintain.  In VE Desktop for HP Software 5.0, there are no script changes required for such reporting.  For users of VE Desktop for HP Software, HP Analysis added the ability to sort results by “Emulated Location.”  HP also added the automatic import of network statistics on a per Load Generator basis.  Now users can understand their application performance by location and merge that data with network statistics like bandwidth utilization, latency and packet loss.  Finally, users can understand if the network is impacting their application performance and if so, how.

3) Performance Center: LoadRunner MWE had no compliment in Performance Center.  VE Desktop for HP Software 5.0 supports both LoadRunner 9.5 and Performance Center 9.5.  Within Performance Center, dynamic host allocation is fully supported.

VE Desktop for HP Software 5.0 provides a lot of other benefits beyond the direct improvements over MWE.  Some of those include a robust API and protocol analysis capabilities, but those are topics for other blog posts.

I look forward to your comments on the explanations above and any feedback you have about our latest addition to the Shunra family of products.

In the following series of posts I will walk you through the process of extending QTP scripts to:

1. Test an application under network conditions such as latency, packet loss and limited bandwidth (this post)
2. Analyze the performance of application transactions under network conditions (part 2 in this series)
3. Automatically execute a test for application performance from different simulated network sites (part 3 of this series)

This series will be making use of 3 tools which should be installed on the same machine:

Quick Test Pro from HP Software, Shunra VE Desktop Professional and Shunra VE Analyzer (optional)

We will begin with a very simple QTP script recorded against the HP demo server hosted at http://newtours.demoaut.com.

In the beginning of the process the script looks like this:

Services.StartTransaction "Sign In"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("userName").Set "amichai"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("password").SetSecure "49875eac11258595e0335617ec51bad1590a"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").Image("Sign-In").Click 21,4
Services.EndTransaction "Sign In"
Services.StartTransaction "SearchFlights"
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").Click 42,13
Services.EndTransaction "SearchFlights"
Services.StartTransaction "SignOff"
Browser("Welcome: Mercury Tours").Page("Select a Flight: Mercury").Link("SIGN-OFF").Click
Services.EndTransaction "SignOff"

The first step in integrating QTP with VE Desktop Professional is to load the VE Desktop Professional COM object, this will be done in a code block that is located before the QTP action steps as follows:

'Load the COM interface for VE Desktop API
Set ctrl = CreateObject("ShunraVEAPI.ShunraVEDController")

Next we will activate the network conditions. This function takes, as input, a value for latency [msec] packet loss [%] and bandwidth [kbps where 0 = unlimited]

(NOTE: For some reason WordPress doesn’t allow the larger than and less than signs, so if you copy paste the code below, replace the Is Not statement with the appropriate logical operator)

'Activate the following network conditions 50 msec latency, 0% packet loss and unlimited bandwidth (0)
returnStatus = ctrl.StartAdvanced(50,0,0)
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "Start Advanced", text
End If

By adding the 2 function calls above, the same QTP script will now run under the defined network conditions. We now simply have to remove those network conditions at the end of the script and release the COM object.

'Stop the network impairments
returnStatus = ctrl.StopTest()
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

'Release the COM object
Set ctrl = Nothing

The complete script now looks as follows:

'Load the COM interface for VE Desktop API
Set ctrl = CreateObject("ShunraVEAPI.ShunraVEDController")

' Activate the network conditions 50 msec latency, 0% packet loss and unlimited bandwidth (0)
returnStatus = ctrl.StartAdvanced(50,0,0)
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "Start Advanced", text
End If

'BEGINNING OF THE QTP ACTION
Services.StartTransaction "Sign In"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("userName").Set "amichai"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("password").SetSecure "49875eac11258595e0335617ec51bad1590a"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").Image("Sign-In").Click 21,4
Services.EndTransaction "Sign In"
Services.StartTransaction "SearchFlights"
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").Click 42,13
Services.EndTransaction "SearchFlights"
Services.StartTransaction "SignOff"
Browser("Welcome: Mercury Tours").Page("Select a Flight: Mercury").Link("SIGN-OFF").Click
Services.EndTransaction "SignOff"
'END OF THE QTP ACTION

'Stop the network impairments
returnStatus = ctrl.StopTest()
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

'Release the COM object
Set ctrl = Nothing

Here is a version of the script with a few safety checks at the beginning.

We first remove any previous network conditions by stopping the VE Desktop test. Then we checkout a license, in case the floating license for the VE Desktop client has expired. At the end of the script we return the floating license to the license server by calling the license checking function. The complete script is presented below:

'Load the COM interface for VE Desktop API
Set ctrl = CreateObject("ShunraVEAPI.ShunraVEDController")

'Just in case network conditions are already applied , stop the VED test
returnStatus = ctrl.StopTest()
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

' check out a license from the VED server
returnStatus = ctrl.LicenseCheckout(5)
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "License Checkout", text
End If

' Activate the network conditions 50 msec latency, 0% packet loss and unlimited bandwidth (0)
returnStatus = ctrl.StartAdvanced(50,0,0)
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "Start Advanced", text
End If

'BEGINNING OF THE QTP ACTION
Services.StartTransaction "Sign In"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("userName").Set "amichai"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("password").SetSecure "49875eac11258595e0335617ec51bad1590a"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").Image("Sign-In").Click 21,4
Services.EndTransaction "Sign In"
Services.StartTransaction "SearchFlights"
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").Click 42,13
Services.EndTransaction "SearchFlights"
Services.StartTransaction "SignOff"
Browser("Welcome: Mercury Tours").Page("Select a Flight: Mercury").Link("SIGN-OFF").Click
Services.EndTransaction "SignOff"
'END OF THE QTP ACTION

'Stop the network impairments
returnStatus = ctrl.StopTest()
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

returnStatus = ctrl.LicenseCheckin()
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "License Check In", text
End If

'Release the COM object
Set ctrl = Nothing