In this post I will expand the sample to automatically analyze the performance of transactions under network conditions. This leverages the built in packet capture and analysis capability within VE Desktop and leverages the VE Analyzer product for the reporting and analysis oprtion.

The process for adding the analysis is quite simple, it involves adding synchronization points at the end of each transaction as well as adding VE Desktop markers at the beginning and end of each transaction. The analysis itself can be triggered automatically at the end of the test.

First we need to add synchronization points at the end of each transaction so the playback will wait for the page to finish loading before ending the transaction. The easiest way to add synchronization points is during the recording of the QTP script, in this example the script waits for the “findFlights” button image to be visible before completing the “SignIn” transaction.

Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").WaitProperty "visible", true, 10000

More synchronization points need to be added, one for each transaction (for network analysis it is recommended to mark each none human intervention step in the script, i.e. from the “click the submit button” until the page loads).

Next we need to activate the packet capture before the test begins and we will also configure the analysis to run automatically at the end of the test, reminder in the previous post we defined the ctrl object which provides access to the VED client functionality:

'Activate the packet list and configure the auto analysis
ctrl.ActivatePacketList
ctrl.SetAutoAnalysisParams True, "HTTP", True

Finally we need to mark the beginning and end of each transaction so the VE Analyzer will be able to provide its analysis on a transaction be transaction basis. By default, if you don’t mark the transactions the analysis will be provided on the entire script, but as you will quickly find out, the ability to correlate the business logic to the analysis is very useful.

The marking of the transactions is done with the following functions:

ctrl.StartTransaction "SignIn"
' Transaction runs here
ctrl.StopTransaction 1

The final script looks like this:

'Load the COM interface for VE Desktop API
Set ctrl = CreateObject("ShunraVEAPI.ShunraVEDController")

'Just in case network conditions are already applied , stop the VED test
returnStatus = ctrl.StopTest()
If returnStatus  0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

' check out a license from the VED server
returnStatus = ctrl.LicenseCheckout(5)
If returnStatus  0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "License Checkout", text
End If

' Activate the packet list and configure the auto analysis and auto save
ctrl.ActivatePacketList
ctrl.SetAutoAnalysisParams True, "HTTP", True

' Activate the network conditions 50 msec latency, 0% packet loss and unlimited bandwidth (0)
returnStatus = ctrl.StartAdvanced (100,0,0)
If returnStatus  0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "Start Advanced", text
End If

'BEGINNING OF THE QTP ACTION NOW WITH VED MARKERS
ctrl.StartTransaction "SignIn"
Services.StartTransaction "SignIn"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("userName").Set "amichai"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("password").SetSecure "49c2a6682829d59b84f9eabf8c8d76a8557e"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").Image("Sign-In").Click 24,4
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").WaitProperty "visible", true, 10000
Services.EndTransaction "SignIn"
ctrl.StopTransaction 1

ctrl.StartTransaction "SearchFlight"
Services.StartTransaction "SearchFlight"
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").WebList("fromDay").Select "12"
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").Click 34,6
Browser("Welcome: Mercury Tours").Page("Select a Flight: Mercury").Image("reserveFlights").WaitProperty "visible", true, 10000
Services.EndTransaction "SearchFlight"
ctrl.StopTransaction 1

ctrl.StartTransaction "SignOff"
Services.StartTransaction "SignOff"
Browser("Welcome: Mercury Tours").Page("Select a Flight: Mercury").Link("SIGN-OFF").Click
Browser("Welcome: Mercury Tours").Page("Sign-on: Mercury Tours").Image("Login").WaitProperty "visible", true, 10000
Services.EndTransaction "SignOff"
ctrl.StopTransaction 1

'END OF THE QTP ACTION

'Stop the network impairments
returnStatus = ctrl.StopTest()
If returnStatus  0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

returnStatus = ctrl.LicenseCheckin()
If returnStatus  0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "License Check In", text
End If

'Release the COM object
Set ctrl = Nothing

So here’s my shameless plug for the certificate program.

This 2 and half day certification program covers everything an IT engineer needs to know about Shunra’s technology, including:

• Performance Engineering Best Practices
• Deep Transaction Analysis and Performance Triage
• Network Capacity Planning with Shunra VE Predictor, VE Profiler, VE Analyzer and HP LoadRunner
• Advanced Network Topics
• Network Aware Load Testing with VE Desktop for HP Software

Completing this program provides a SCPE certification (Shunra Certified Performance Engineer) which comes in very handy in today’s global performance world.

This year will be even more interesting as we will, for the first time, cover the new VE Desktop for HP Software. This product was co-developed by Shunra and HP to transform LoadRunner and Performance Center test beds into virtual network test beds for testing application performance from remote sites. The training session will provide expertise on combining load testing and network performance analysis in a single tool.

Another new addition to the curriculum is the Deep Transaction Analysis and Performance Triage which covers tips and tricks for effective triage and troubleshooting of application performance issues.

There are still some seats open which are sold on a first come first serve basis, so make sure to go here if you are interested in participating.

See you on the 14th at the DoubleTree Hotel in Center City, Philadelphia.

Amichai

In this post I will show you how to generate analysis in less than three minutes that addresses the question that naturally follows. “Why does this transaction perform well locally, but slows down under network conditions?”

This analysis can be generated by installing VE Analyzer on the “remote” load generator and configuring it with the packet analysis settings (reminder: the “remote” load generator is any load generator that has the VE Desktop Professional WAN Emulation client installed on it. This WAN Emulation client enables us to simulate remote network conditions from a local load generator in the lab. Hence the double quotes in “remote” since the generator isn’t physically remote but is simulated to behave that way)

Follow these steps to setup the analysis on the “remote” load generator:

In the LoadRunner Controller Design screen click on the Load Generators button

Select the “remote” load generator

Click on details and select the WAN Emulation Tab:

Check the “Enable WAN Emulation” check box (if not already selected) and click on “WAN Emulation Settings”

See the previous post on how to setup the WAN Emulation parameters in the following screen.

Click on the additional settings button

First check the “Capture Packets” check box, this enables the packet capturing agent on the VE Desktop Client on the LoadGenerator. I recommend assigning at least 100 MB of buffer space for the capture so at least one iteration of the script will be captured.

For web based applications, select HTTP in the “Automatically run post-test analysis” option, for other applications select either TCP or UDP analysis.

Finally, select a folder on the load generator for automatically saving the test results. It is important to re-iterate that this folder is on the “remote” load generator, not on the LoadRunner Controller.

It is recommended to setup only 1 virtual user to run from this “remote” generator, so the analysis will focus on a single instance of the user. If load is needed as part of the scenario, it should be generated from a second load generator, in this case the analysis will show results of the marginal user.

Adding the analysis capabilities in this example doesn’t require any changes to the script and can be applied to the same scenario we presented in the previous post. Running this test results in an analysis report that is generated on the LoadGenerator. This report can be accessed from the VE Reperter that is installed on the “remote” load generator.

VE Reporter stores the reports based on the test names and the test run dates, so we are looking for the last test run for our test.

Once the report opens we are mainly interested in 4 reports:

The Application Efficiency Report shows us how chatty the application is (application chattiness is directly related to response time slow downs due to network latency) as well as how long it took to download various resources from the server. In this example, we see a resource that took 9 seconds to download, explaining why the entire action took 11 seconds for a remote user Vs, 0.6 seconds for a local user.

The Application Efficiency Report highlights a resource that took 9.2 seconds to download

The second report is the Client Network Server Breakdown that shows how much time the transaction spent on the client, network and server.

A Client Network Server Breakdown Report

The 3rd report is the Bandwidth Analysis which shows how much bandwidth was used by this single application user.

Finally, for a real deep dive we can use the Bounce Diagram that displays the entire communication between the client and the server as well as the delta times between consecutive packets, high delta times can point out to blocking events within the transaction.

The Bounce Diagram shows deep packet analysis data on the transaction

In summary, in less than 3 minutes we can generate the above reports on any transaction that seems to be performing poorly over the Wide Area Network. This information can help the developers and network engineers to fine-tune the application and the network to improve the end user experience.

Feel free to suggest additional topics you would like to see covered in this series of posts.

Best,

Amichai

In the article, Patric Ancipink says that application response time is the most important metric for network performance management. He further explains that the application response time is mostly affected by the latency of your connection and the number of connections the application makes (application turns). He is right. There are things that you can do to improve, or reduce your latency between two locations, but there will be a point where you simply can not improve further. Likewise with the number of connections; there are many techniques you can try to reduce the number of connections and the number of interdependencies on the connections you use, but you will eventually hit a point of diminishing returns.

The logical next step is to start trying to test for these effects and see how your application performs already. Shunra has a great tool called VE Analyzer that helps you do this. It works with our appliances and desktop WAN emulation solutions, so it’ll fit into your existing testing or offer an easy entry into application delivery testing. I don’t want this blog to sound too “salesy” so I’ll leave the deployment information at that. What VE Analyzer will do is gather the packet data from your test and then tell you things like where your packets are spending their time, how many application turns each transaction is taking and how long each turn takes, where any network bottlenecks exist, etc…

Quickly generating these metrics enables you to understand the most important things: how sensitive your application is to latency, and how many connections each transaction is using. You can then either make changes to your application and re-test or try what-if scenarios by changing the emulated network. Either way you choose to address the problem will give you valuable insights with ease and efficiency of time that you can not find anywhere else.

How are you currently testing application delivery? Do you consider latency, number of connections, and application response times? I’d love to hear about your projects or thoughts on this post.

The problem is that the OPNET products can be time consuming, the data can be overwhelming, and the skill level required to generate and understand the data often requires dedicated people for. One solution is to use Shunra’s VE Desktop Professional to speed-up the OPNET process.

Here is how:

Typical problem: Two data centers that housed the online store servers were just consolidated into one data center. Now customers are complaining that the site is loading “slowly” and their shopping carts keep emptying without reason.

Process:

1) Find out where the problems are being reported and pick the highest priority (or worst network connection).

2) Understand what the network conditions are from that location to the data center.

a. A lot of products exist that can help you understand what your latency, packet loss, and available bandwidth are. For your first pass, rough numbers will often be good enough to identify the problem (e.g. Ping for latency and packet loss, FTP for available bandwidth).

3) Use VE Desktop to emulate the network between the problem location and the data center in your lab.

a. Use the rough network numbers to start with

b. Turn on packet capture

c. Time some of the transactions that users have been complaining about

4) If you can recreate the symptoms already, and it’s likely you can, then you can move on from here. If you can’t then you can deploy the VE Desktop to one of the problem areas, run it with packet capture on and no emulation, and have the end user execute some of the problem transactions while timing them with VE Desktop.

5) The results of either #3 or #4 that are of interest are the packet capture and the transaction timing results. With this data Shunra’s VE Analyzer can correlate the transaction timing with the captured packets. VE Analyzer will then provide a summary analysis of the transactions. In many cases, this will identify the high level problem and give you a focused area to debug further. If it doesn’t, then VE Analyzer will also tell you which frames within the packet capture correlate with the beginning and end of each transaction.

6) Armed with a packet capture and your frame numbers for each transaction, you can quickly use any packet sniffer on the market to isolate the packets for only the transactions you wish to analyze further in OPNET. The ability to use VE Desktop and VE Analyzer to get this information will save you hours…