Application Performance Engineering Blog – Shunra Software

Adding Network Conditions to a QTP Script – Part 1

Quick Test Pro (QTP) is a very useful functional testing tool and now as a result of the Shunra-HP partnership, QTP can integrate with Shunra VE Desktop Professional to automate the testing and analysis of applications under network conditions.

In the following series of posts I will walk you through the process of extending QTP scripts to:

1. Test an application under network conditions such as latency, packet loss and limited bandwidth (this post)
2. Analyze the performance of application transactions under network conditions (part 2 in this series)
3. Automatically execute a test for application performance from different simulated network sites (part 3 of this series)

This series will be making use of 3 tools which should be installed on the same machine:

Quick Test Pro from HP Software, Shunra VE Desktop Professional and Shunra VE Analyzer (optional)

We will begin with a very simple QTP script recorded against the HP demo server hosted at http://newtours.demoaut.com.

In the beginning of the process the script looks like this:

Services.StartTransaction "Sign In"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("userName").Set "amichai"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("password").SetSecure "49875eac11258595e0335617ec51bad1590a"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").Image("Sign-In").Click 21,4
Services.EndTransaction "Sign In"
Services.StartTransaction "SearchFlights"
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").Click 42,13
Services.EndTransaction "SearchFlights"
Services.StartTransaction "SignOff"
Browser("Welcome: Mercury Tours").Page("Select a Flight: Mercury").Link("SIGN-OFF").Click
Services.EndTransaction "SignOff"

The first step in integrating QTP with VE Desktop Professional is to load the VE Desktop Professional COM object, this will be done in a code block that is located before the QTP action steps as follows:

'Load the COM interface for VE Desktop API
Set ctrl = CreateObject("ShunraVEAPI.ShunraVEDController")

Next we will activate the network conditions. This function takes, as input, a value for latency [msec] packet loss [%] and bandwidth [kbps where 0 = unlimited]

(NOTE: For some reason WordPress doesn’t allow the larger than and less than signs, so if you copy paste the code below, replace the Is Not statement with the appropriate logical operator)

'Activate the following network conditions 50 msec latency, 0% packet loss and unlimited bandwidth (0)
returnStatus = ctrl.StartAdvanced(50,0,0)
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "Start Advanced", text
End If

By adding the 2 function calls above, the same QTP script will now run under the defined network conditions. We now simply have to remove those network conditions at the end of the script and release the COM object.

'Stop the network impairments
returnStatus = ctrl.StopTest()
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

'Release the COM object
Set ctrl = Nothing

The complete script now looks as follows:

'Load the COM interface for VE Desktop API
Set ctrl = CreateObject("ShunraVEAPI.ShunraVEDController")

' Activate the network conditions 50 msec latency, 0% packet loss and unlimited bandwidth (0)
returnStatus = ctrl.StartAdvanced(50,0,0)
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "Start Advanced", text
End If

'BEGINNING OF THE QTP ACTION
Services.StartTransaction "Sign In"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("userName").Set "amichai"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("password").SetSecure "49875eac11258595e0335617ec51bad1590a"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").Image("Sign-In").Click 21,4
Services.EndTransaction "Sign In"
Services.StartTransaction "SearchFlights"
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").Click 42,13
Services.EndTransaction "SearchFlights"
Services.StartTransaction "SignOff"
Browser("Welcome: Mercury Tours").Page("Select a Flight: Mercury").Link("SIGN-OFF").Click
Services.EndTransaction "SignOff"
'END OF THE QTP ACTION

'Stop the network impairments
returnStatus = ctrl.StopTest()
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

'Release the COM object
Set ctrl = Nothing

Here is a version of the script with a few safety checks at the beginning.

We first remove any previous network conditions by stopping the VE Desktop test. Then we checkout a license, in case the floating license for the VE Desktop client has expired. At the end of the script we return the floating license to the license server by calling the license checking function. The complete script is presented below:

'Load the COM interface for VE Desktop API
Set ctrl = CreateObject("ShunraVEAPI.ShunraVEDController")

'Just in case network conditions are already applied , stop the VED test
returnStatus = ctrl.StopTest()
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

' check out a license from the VED server
returnStatus = ctrl.LicenseCheckout(5)
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "License Checkout", text
End If

' Activate the network conditions 50 msec latency, 0% packet loss and unlimited bandwidth (0)
returnStatus = ctrl.StartAdvanced(50,0,0)
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "Start Advanced", text
End If

'BEGINNING OF THE QTP ACTION
Services.StartTransaction "Sign In"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("userName").Set "amichai"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("password").SetSecure "49875eac11258595e0335617ec51bad1590a"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").Image("Sign-In").Click 21,4
Services.EndTransaction "Sign In"
Services.StartTransaction "SearchFlights"
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").Click 42,13
Services.EndTransaction "SearchFlights"
Services.StartTransaction "SignOff"
Browser("Welcome: Mercury Tours").Page("Select a Flight: Mercury").Link("SIGN-OFF").Click
Services.EndTransaction "SignOff"
'END OF THE QTP ACTION

'Stop the network impairments
returnStatus = ctrl.StopTest()
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

returnStatus = ctrl.LicenseCheckin()
If returnStatus Is Not 0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "License Check In", text
End If

'Release the COM object
Set ctrl = Nothing

Continue reading...

Adding Network Conditions to a LoadRunner 9.5 Scenario

In the previous post I presented a new solution that increases the accuracy of load tests by easily adding network conditions to the load generators.

This will be the first in a series of posts that walks through the steps needed to setup a test scenario that includes end user load and network conditions. By the end of this post you will know how to setup a load test that can answer the question: “How will this application perform for a remote user in [Chicago] compared to a local user?” [replace with any remote site that is relevant for your enterprise network]

Our setup includes 2 machines:

1. A LoadRunner controller version 9.5 with a license for the VE Desktop for HP Software. This machine will also serve as a local load generator and is assigned with IP address 172.31.0.143

2. A LoadRunner load generator with the VED for HP injector and VE Reporter for analysis. This machine is assigned with IP address 172.31.0.142 and will simulate the “remote” load generator.

Both machines are physically in our center of excellence lab in Philadelphia, and so is the server under test.

We begin with a simple test scenario that will run the same script from the 2 load generators – 143 will be the local user and 142 will emulate the remote user in Chicago, so we name the 2 groups accordingly:

Next we need to setup the “remote” load generator with the desired network conditions. The WAN emulation conditions are defined on the load generator details dialog box. These details can be accessed by clicking on the Load Generators Icon (marked with the border),

selecting the “remote” load generator in the load generators setup page, and clicking the details button:

You will notice that the Load Generator Details has a WAN Emulation tab. Selecting the tab and checking the Enable WAN Emulation checkbox will allow you to setup the WAN emulation conditions. It is recommended to change the name of the profile to reflect the network conditions, e.g. Chicago, remote site with 512Kbps, Satellite link, etc.

Clicking on the WAN Emulation settings button will open up the following dialog box:

You will notice the 3 setup modes: Basic, Advanced and Professional.

The Basic mode (shown above) allows you to quickly define a network profile based on predefined settings, the example above sets up a network link from a Chicago user to an Atlanta data center with a T1 link during normal business hours.

The advanced mode lets you setup specific network conditions such as latency, packet loss and bandwidth. The Professional mode will be covered in future posts.

Once the desired settings are configured, the test is ready to go. Our test scenario has 1 user from each generator, but the scenario isn’t limited by the number of users (up to the licensed number of virtual users).

The scenario can be saved and run like any other LoadRunner scenario.

Once the test completes, we can invoke the analysis and generate the reports we are looking for (reminder, we want to see the response time between the local user and the user in Chicago). The analysis is triggered by clicking on the LoadRunner analysis button:

By default the analysis results are averaged across all load generators, so we need to group the results by WAN Emulation location which is done in the next 3 steps:

Step 1 – select the transaction response time chart:

Step 2 – Select the Set Filter\Group By option by clicking on (Ctrl T):

And group by emulated location:

Step 3 – We want to present the data as a bar chart, so we select the display options (Ctrl I):

And setup a Bar chart with the following options:

Finally, we get the report we were looking for (below), answering the question:

“How will this application perform for a remote user compared to a local user?”

The local user response time is identified by the transactions with the “No WAN Emulation” suffix, the Chicago user is identified by the transactions with the Chicago suffix (click on the thumbnail to view the large image).

Hope this helps, stay tuned for more use cases and tutorials.

Best,

Amichai

Continue reading...

Content Loading – When Being Lazy Pays Off

Building applications on a local area network masks many performance problems that only surface when applications interact with remote users on a wide area network.

One common problem is when remote users experience slow web pages that load content (table records, images, videos, PDF files, widgets). These pages tend to perform much slower for remote users than they do for local users. This is usually because these web pages load more content than needed for rendering. For example, pages that contain grid components may hold a substantial number of records or pages that layout container components (lists, tab navigators, accordions, etc.) may need to load many thumbnails media objects. As a result end users painfully wait for many objects to load when only 5-20 can fit on the screen.

Developers can avoid this problem by implementing a best practice called lazy loading (or deferred loading). This design pattern implements the following basic concept “only load objects that are needed to be rendered on the screen and defer any object loading until the object display is requested.”

For example, a grid component that can potentially hold 10,000 employee records would only load the first 20 and consequently load additional records if and only if the user scrolls down to view other records. Another example could be a PDF file reader that immediately loads the first page and lazy loads the rest of the pages over time or as needed.

Lazy loading is an abstract design pattern but it has many implementation examples in AJAX, .NET and Flex to name a few.

The following are some implementation examples and more information on this useful technique.

Introduction to the lazy loading design pattern

http://en.wikipedia.org/wiki/Lazy_load

AJAX implementation examples of lazy loading

http://www.digital-web.com/articles/improve_page_performance_with_lazy_loading/

Java examples of a lazy loading implementation

http://blogs.msdn.com/jkowalski/archive/2008/05/12/transparent-lazy-loading-for-entity-framework-part-2.aspx

Adobe Flex examples

I found the following example very insightful because it shows some out-of-the-box thinking

http://flexdiary.blogspot.com/2009/01/lazy-loading-tree-example-file-posted.html

But in general when I develop in Flex I follow the performance guidelines in this whitepaper

White Paper: Flex Application Performance: Tips and Techniques for Improving Client Application and Server Performance

So the next time you have to troubleshoot an application page that loads slowly, ask your development team: “Is this page only loading objects that are needed for rendering?” You will be surprised what you find.

Best,

Amichai

Continue reading...

Performance Best Practices from Citrix

I get to work with many clients that are deploying applications over Citrix. In general Citrix is a good way to overcome network latency related performance problems and, although I am a big fan of coding applications right the first time, if the hand you were dealt is to manage an application that can’t perform well once introduced with network latency then Citrix is one good way to go. However, deploying applications over Citrix isn’t free of challenges (or free at all for that matter).

When asked about best practices for improving Citrix performance over the network I offer 2 pieces of advice that I follow whenever I work with a client with Citrix performance tuning: read Brian Madden’s excellent whitepaper on the topic and read his blog posts. The man is a wealth of knowledge when it comes to Citrix as well as any virtualization related topic.

Some of the more useful posts when it comes to Citrix performance tuning are:

Understanding Citrix Printing, a popular issue with Citrix deployments. A guest post by Kevin Buchaniec

Understanding the Citrix Log On Process, which I found instrumental when troubleshooting slow Citrix Log On transactions. Log on is quite a popular transaction on the list of Citrix transactions that slow down under impaired network conditions

Hope this helps. I would love to hear from you, what have been your challenges with performance tuning of Citrix environments and any performance tips you would like to share.

Best,

Amichai

Continue reading...

Slow network users take down an IIS based ASP application server

We often mention the fact that slow network users can impact the server’s ability to scale and consequently degrade user performance (for example here ). This is fundamentally attributed to the fact that slow network users (due to high network latency or limited bandwidth) occupy server resources for longer periods of time per transaction. This negatively impacts the user experience for all users, fast or slow and a scenario that must be part of every pre-deployment performance testing plan.

This week we had a first hand experience of such a scenario. We were testing an IIS based application that under ideal network conditions (LAN) scaled to 200 users. However, the same server when tested with 200 users under WAN conditions (high latency and low bandwidth) started to bog down at around the 80 user count.

When our consulting experts analyzed the server they found a specific resource “w3wp.exe” that was pegged and had high memory usage. Additional research uncovered that this specific resource tends to degrade in performance under low bandwidth conditions. There is a fix to this problem and more information available from Microsoft at http://support.microsoft.com/kb/916984

This is just one example to the type of problems that surface when applications and networks interact.  It was great to find this problem early in the testing phase rather than have his users experience post deployment issues.

I am curious to hear from you: What type of unique problems have you uncovered when applications and networks interact? (I mean beyond the obvious slowness in application response time due to network latency and limited bandwidth)

Looking forward to getting your input

Best,

Amichai

Continue reading...

The Remote Site Performance Range

In the previous post I presented an approach for making sure that the network models used in a test are up to date. I also mentioned that in cases where the enterprise network is very dynamic, a more rigorous approach is needed. It is important in these cases to test applications for a range of network conditions per link, specifically for best case, average case and worst case profiles for each link. In this post I want to describe this best practice and offer some tips in implementing it.

On the surface, answering the question “How will this application perform from a remote site?” seems pretty straightforward. One needs to emulate the network conditions of the link from the remote site to the data center and then test the application under those emulated network conditions. However, in practice answering this question is often more challenging. Specifically, enterprise networks are very dynamic with conditions that change based on the time of day and day of the month. This increases the challenge of selecting which network conditions to use in each test. A test that doesn’t reflect the entire gamut of network conditions may miss potential problems, on one extreme, or lead to teams over reacting to bad results on the other extreme.

This means that there isn’t a single response time metric that can tell us how an application will perform in a remote site. This is where the remote site performance range comes into play. The performance range predicts application response times under best, average and worst case network scenarios for each remote site. By testing for a range of network conditions the tester can confidently make a statement such as “when the network is idle (best case network condition), a user from this remote site will experience a 7 seconds response time, on average the response time will be 10 seconds and during worst case network scenarios the response time will increase to 17 seconds.” (These response times are used for illustration purposes only and aren’t offered as guidelines).

A report that summarizes the test results of a performance range may look like the following, where each transaction from each site is presented with its measured performance range:

Site	Transaction	Best Case [Sec]	Average Case [Sec]	Worst Case [Sec]
NY	Sign in	5.30	7.20	10.30
	View Spending report	6.20	7.30	8.40
	View Credit Report	10.10	15.20	23.50
	Submit changes	3.30	4.50	5.50
	Sign out	1.10	1.20	1.50
Boston	Sign in	5.83	7.92	11.33
	View Spending report	6.82	8.03	9.24
	View Credit Report	11.11	16.72	25.85
	Submit changes	3.63	4.95	6.05
	Sign out	1.21	1.32	1.65
Chicago	Sign in	7.00	9.50	13.60
	View Spending report	8.18	9.64	11.09
	View Credit Report	13.33	20.06	31.02
	Submit changes	4.36	5.94	7.26
	Sign out	1.45	1.58	1.98
San Francisco	Sign in	8.40	11.40	16.32
	View Spending report	9.82	11.56	13.31
	View Credit Report	16.00	24.08	37.22
	Submit changes	5.23	7.13	8.71
	Sign out	1.74	1.90	2.38
Seattle	Sign in	10.07	13.69	19.58
	View Spending report	11.78	13.88	15.97
	View Credit Report	19.20	28.89	44.67
	Submit changes	6.27	8.55	10.45
	Sign out	2.09	2.28	2.85
London	Sign in	12.09	16.42	23.49
	View Spending report	14.14	16.65	19.16
	View Credit Report	23.04	34.67	53.60
	Submit changes	7.53	10.26	12.55
	Sign out	2.51	2.74	3.42
Tokyo	Sign in	14.51	19.71	28.19
	View Spending report	16.97	19.98	22.99
	View Credit Report	27.65	41.60	64.32
	Submit changes	9.03	12.32	15.05
	Sign out	3.01	3.28	4.11

In a future post, I will go deeper into how to model a network link to represent a best case, worst case and average case scenario.

Continue reading...

Network Emulation Gotcha – does your Network Model match your network?

We’re in discussions with an enterprise customer preparing for a major Data Center Consolidation. They are planning to lab test several misison critical applications to understand if application availability and transaction response times will meet performance goals after their West Coast Data Center (USA) is consolidated into an existing facility in the Southeast (SE). They’re concerned about the impact of increased WAN latency to some offices, especially on applications using Microsoft File Sharing and SQL Server. They’re also concerned about the application impact due to the increased traffic into the SE Data Center.

Their existing lab environment includes HP LoadRunner to generate user traffic. They are now considering the purchase of a WAN Emulation appliance to simulate the new transmission behavior of offices accessing applications in the relocated Data Center.  This diagram provides a quick look at the star topology being emulated.

One major requirement they are struggling with is the ability to lab test the impact of increased traffic on the single (redundant) pipe into the SE Data Center. They’re looking at a lower cost network delay box that has an “East to West” hardware design and can support independent Network Profiles for each remote office, e.g. 110ms latency on a T3 from Seattle to SE Data Center, 70 ms on a T1 from Phoenix to the SE Data Center, etc. The problem is the 100 Mbps pipe into the SE Data Center that all remote offices share – the cheaper Network Emulator can’t model the shared pipe into the Data Center in an integrated fashion.

This customer wants a network model in their lab that provides unique latency characteristics for each office + emulates traffic contention on the shared 100Mbps pipe into the Data Center + also provides an option to emulate additional background traffic on the shared 100Mbps pipe so they can do Network Capacity Planning.  The cheaper box can only support the first of these requirements – is that good enough?

Has anyone done WAN emulation testing for a star topology similar to this? Did you setup the remote office links with independent profiles without emulating the shared pipe into the Data Center? How did your your results compare once you went live on your Production Network? Any thoughts or recommendations?

Continue reading...

Keeping Your Network Models Up to Date

“How do we know that the Shunra VE models that we use in our tests are up to date?” I am often asked this question by application testing teams who find themselves using Shunra VE in their testing lab every few weeks. These people are not responsible for maintaining the models and often don’t have visibility into network changes that may require updates to the model.

The concern is that since the enterprise network is so dynamic, a test that uses old data in the model will not accurately reflect the expected performance over current network conditions.

I want to share with you two ways to be confident that your network models are up to date.

Both options use VE Network Catcher. This is a tool that supports the capture of network conditions from various links which can then be played back during a test. VE Network Catcher supports a 24/7 recording of up to 100 links.

The first and easier approach that I suggest is to record your WAN behavior (methods will be described in future how-to posts) prior to running a test. This is simple enough if all we want to verify is that recent network changes are reflected in the model. However, if the concern is that the dynamic nature of the enterprise network may introduce conditions that vary on a daily or monthly basis, then a more detailed approach is needed which includes reviewing the network data since the last update and creating a range of link profiles reflecting the best, average and worst case scenarios. This approach allows the tester to generate a performance range, which is a best practice that I will elaborate on in a future post.

The process of building link profiles from historical data, although not too time consuming, may still be more than what a testing team will want to be bothered with prior to a test. A substantial number of links in the model can also make this very time consuming. This brings me to the second approach for being confident that the network models are up to date – use the new Shunra consulting service to keep these network models up to date. This is a new service that was built based on customer requests which provides the following:

A Shunra VE consulting professional will build the network models for each site to reflect a range of network conditions, including the best case scenario, average case scenario and worst case scenario. The process of setting up these models utilizes data from Shunra VE Network Catcher as well as from additional network monitoring tools that the client may have in place such as SolarWinds, CiscoWorks, HP OpenView, etc.

Once a month, a Shunra VE consulting professional will update the VE models with data from the past 30 days, including best case, average case and worst case conditions, so all the network models for all the sites are reflecting recent conditions on an ongoing basis. This relieves application testing and network performance teams from managing those models and allowing them to focus on testing, confident that they are always using the most up to date models.

In summary, VE Network Catcher allows testers to keep Shunra VE models up to date with the latest network conditions. Utilizing the new service by Shunra consulting services these models can be refreshed and updated on an ongoing basis allowing testing teams to focus on executing tests.

A demo version, VE Network Catcher Lite, is available online so give it a try.

Continue reading...

Shunra VE Modeler – Avoid Common Roadblocks

Shunra VE Modeler, the well known Microsoft Visio-based Shunra console, makes network modeling quick, easy and straightforward – I think most Shunra users will agree with this statement. Still, some users run into issues with VE Modeler which are easy to avoid.

I have been working with Shunra users for several years now and I wanted to share my experience to help others avoid common network modeling roadblocks and fast forward the learning curve for our new users.

(more…)

Continue reading...