Application Performance Engineering Blog – Shunra Software

The Value of Leveraging Virtualization for Application Performance Testing

Virtualization has emerged as one of the leading technologies in today’s market; enabling businesses to more effectively scale operations to meet demand while significantly reducing costs at the same time. Everyone seems to understand what virtualization is, but it’s actually rather difficult to define because the term is used interchangeably to describe a plethora of different things. When I first tried to define what virtualization is in my own terms, I thought of it more as a technology for achieving some end – primarily server consolidation. However, after further investigation I realized that virtualization is really more of a concept than anything else. This certainly became evident in an article I read from the Virtualization Journal where the CTO of Citrix & Founder of XenSource, Simon Crosby, was being interviewed on the topic of virtualization. He stated that “virtualization is already widely used, but primarily for the first-order benefit, namely server consolidation. The second-order benefits of agility, availability and manageability of the IT stack are now becoming better understood,” Crosby continues, “and as a consequence virtualization has moved from a tactical tool for gaining immediate savings, to become a key strategic theme for every IT department.” Essentially, virtualization has become a business enabler for many – and that’s certainly apparent considering the number of organizations gearing up for cloud computing. This is also the case when it comes to how application performance testing can be leveraged within organizations today.

There are many challenges organizations face when it comes to administering and maintaining a dedicated pre-production or staging environment for which accurate performance testing can be conducted. The cost to manage and maintain infrastructure, along with personnel and facilities, can be fairly sizeable and are only a subset of the overall costs to be considered. So, in many cases performance testing can be rather expensive and this is exactly why virtualization can provide significant benefits because there is cost reduction across the board. A prime example is in many performance labs there are a variety of application performance tools typically utilized for testing – one such tool is HP LoadRunner or Performance Center. These tools are a primary part of a performance lab as they provide load generation capabilities and can accurately test applications under real world load and stress scenarios. However, these solutions require a significant amount of infrastructure and resources (A Controller to execute tests, LoadRunner Generators to produce user traffic, Virtual User Generator to record scripts, etc.) and this can make it very difficult to manage the environment when it has to scale to meet higher demand. In this case, virtualization saves time, effort and cost because resources can be allocated dynamically within the environment and any number of virtual machines can be leveraged when needed to handle these resource intensive applications. This is also enabling many organizations to architect and customize elegant configurations that more closely align with their testing requirements – which can minimize unnecessary infrastructure and resources. Yet, the prevalent issue many organizations still grapple with is how to execute performance tests that accurately depict the network for which the application will be deployed across.

The most pervasive approach that many organizations would take is to physically deploy hardware (remote load generators) in offices that they wanted to test an application from. This process was not only time-consuming, but also expensive, inaccurate and cumbersome to manage. For this reason, HP decided to form a partnership with Shunra to develop a seamless solution that provides this capability within the HP LoadRunner and Performance Center solutions – Shunra VE Desktop for HP. This solution aligns very well with the virtualization movement because it is simply a plug-in within the HP products that introduces the network into the existing test bed and can be leveraged across most virtual platforms. For LoadRunner 9.5 and later, there is simply a “WAN emulation” tab that can be accessed from within the Controller to introduce the latency, jitter, packet loss and bandwidth constraints directly into the test. With Performance Center 9.5 and later, this capability can easily be configured directly from the browser UI to allocate WAN emulation parameters across any number of desired load generators. A consequence of this is that organizations can leverage on-demand performance testing from a dynamic virtual environment that is agile, flexible and robust. This therefore eliminates the need to manage testing cycles across multiple time zones and remove any need for additional hardware. Additionally, all of the network metrics from each generator utilizing WAN emulation within the test will automatically be imported into the controller, which can save a significant amount of time when collating results and generating analysis reports. These reasons are precisely why numerous organizations have decided to improve their existing performance test environment with the VE Desktop for HP Software  solution. Not only does this solution address a strategic gap within the functionality of the HP solutions, it embraces virtualization as a means to more effectively administer performance testing. Overall, the VE Desktop for HP Software solution was co-developed with HP to considerably enhance the accuracy and value of these application performance test suites.

Continue reading...

A fix to the awesome WPF Snoop utility.

Anyone developing on the WPF platform surely knows the Snoop utility by Pete Blois. It is absolutely awesome and I cannot imagine developing WPF applications without it.

However, there is a problem with it. One cannot snoop WPF framework elements nested too deep. By too deep I mean somewhere around 60 layers. Trying to snoop at that nesting level crashes the Snoop and since it is injected into the snooped application process space, the application being snooped is crashed as well.  Which is very annoying.

First of all, there is nothing wrong with Snoop. The problem is inherent in WPF – one just cannot nest too many WPF framework elements. The limit is not 60, of course. It is 255 and is very easy to check. Try to write a simple WPF application with 256 nested panels – it is impossible. At some point the application will crash. Debugging it reveals an exception with the following message:

Layout recursion reached allowed limit to avoid stack overflow: ’255′. Either the tree contains a loop or is too deep.

This msdn forum post is precisely about this issue. Note, that a reply from a Microsoft guy Rob Relyea promises to increase the limit from 255 to 4096 in the upcoming .NET 3.5 “servicing release”. But, until then the problem exists.

So, why is Snoop affected at all? Simple, Snoop uses the WPF TreeView control to display the hierarchy of snooped WPF framework elements. The depth of a node in that tree is directly proportional to the nesting level of the respective snooped element. The following image is a snapshot of that tree created using the Snoop itself:

(Actually, this is a slightly different version of the original Snoop. I have added the nesting level in the square brackets to the display of a node)

Inspecting the tree gives us the following formula for the nesting depth of a TreeViewItem object – 8 + 4*n, where n is the nesting depth of the respective snooped element. Hence snooping at the first framework element in a WPF application produces a node nested 12 levels deep, whereas snooping at an element at level 60 corresponds to a TreeViewItem at level 248 – dangerously close to the hard coded WPF nesting limit.

In reality, deep snooping of a complex WPF application almost always hits this limit, bringing down the Snoop code along with the snooped application.

So, how can one deal with this unfortunate situation? I see these ways:

• Be careful not to snoop too deeply – never works.
• Use the filter textbox to change the root node of the element TreeView. This is a good workaround, but from my own experience it is easy to forget oneself and snoop carelessly too deep – bang, you are toast.
• Forget about snooping, until the .NET 3.5 “servicing release” is actually released. The reply promising this release dates back to April 17th 2008. I have checked the state of the matters today (July 21st 2009) – the limit is still 255.
• Fix somehow the Snoop TreeView.

Since I love Snoop very much and use it extensively, I have chosen to fix this issue once and for all. The following items depict the fix:

• When a new node is added to the TreeView calculate its expected nesting level using this formula – rootLayoutDepth + levelLayoutDepth * n, where
  • rootLayoutDepth is the depth of the first TreeViewItem, calculated automatically somewhere in the beginning.
  • levelLayoutDepth is the depth difference between a node and its child, calculated automatically somewhere in the beginning and is assumed to be constant thereafter.
  • n is the zero based nesting level of the particular snooped element.
• If the expected nesting level is above 240, then a new TreeView root node is picked. I use the filtering feature of Snoop to setup a special reduce depth filter behind the scenes.

That’s it. There are, of course, many details to it, but the overall picture is captured pretty accurately by these two items.

A few final notes:

• The Mole visualizer does not have this problem, although it also displays the tree view of inspected elements. The reason is that Mole uses Windows Forms TreeView, which is a single control regardless of how many nodes it displays and how deep they are nested. The nodes are simply drawings on the control’s canvas.
• The described element nesting limit is not a regression from the Win32 world. Plain old Win32 windows have this limit as well, as described in this post.
• I have mentioned this issue some time ago in this msdn blog post. Cory Plotts responded to my post and he has added some code to the original Snoop source to better handle interop scenarios. Refer to his blog for details here.
• In addition to fixing the nesting issue, I have made the following changes to the original Snoop source code:
  • Fixed the property grid view – the last Snoop release has broken it.
  • Added the nesting level in the square brackets to the display of a node in the element tree.
  • Added a test application to the Snoop solution to make sure snooping at deeply nested elements works correctly.
  • Added a few more exception checks.

Last, but not least – the source code. Since I am not the owner of the Snoop application I am not putting the complete source code here, but rather the delta and the Release binaries. In order to compile the Snoop application with my fixes first the complete source code must be downloaded from here. Then, the delta containing my fixes should be copied over the Snoop source tree. Now build it – that’s it.

The release binaries are here.

Continue reading...

Using Linq expressions in a client-server application.

Imagine the following situation:

• A client-server application, where the client side has no knowledge of the DAL. Even the DAL assemblies are not found on the client side.
• The client should have an API to fetch entities using some flexible criteria.

One of the likable ways to satisfy the second bullet is let the client specify Linq expressions, pass them to the server side and let the DAL handle it. The benefits are clear:

• No need to invent criteria API. Granted, some DAL implementations may come with versatile in-house criteria API. However, one of the requirements is that the client side be unaware of the actual DAL, so it may not use any DAL specific criteria API.

• The multitude of DAL implementations consuming Linq expressions – Linq to SQL, Linq to Entities, Linq to NHibernate to name a few.

So, this is the way we have chosen. Ideally we would like the following client side code to be valid:

public static IList<Entity> GetByNamePattern(string namePattern)
{
Expression<Func<Entity, bool>> expr = e => e.Name.Contains(namePattern);
return EntityCollection<Entity>.FetchByExpression(expr);
}

The process seems to be pretty straightforward:

1. Serialize the Linq expression
2. Send it over the wire
3. Deserialize the Linq expression
4. Pass it to the DAL to receive a list of entities
5. Return the entities to the client

There is, however, a problem – linq expressions are not easily serializable! See this msdn forum post, for instance.

There are couple of ways to deal with it. I will describe here ours.

First, the linq expression has to be partially evaluated to remove any anonymous delegates and resolve any member/property access references, which can be evaluated immediately. There is an excellent post by Matt Warren here, which explains exactly this. But I have found that the DAL implementation we use, namely Linq to NHibernate, already has a partial linq expression evaluator, which looks very similar to the  Matt Warren’s code. I suppose they have read his post very thoroughly. So, I just took their partial linq evaluator code – two totally self contained C# source code files, one for the Evaluator and the other for the Expression Visitor, also explained by  Matt Warren in the Appendix to his post here.

Next, the partially evaluated Linq expression must be converted to something easily serializable. Again, there are several solutions out there, but I have choosen the one shared with the community by Luca Bolognese and LukeH here. It allows one to (de)serialize a linq expression to/from XML.

Having these two pieces makes it possible to transmit a linq expression from the client side to the server side.

There is one last thing. How to use a linq expression with Linq to NHibernate? After all, their samples only give this kind of code – from c in db.Customers where linq-expression select c.

Actually, it turns out to be pretty simple – db.Customers.Where(linq-expression), where the client side is responsible for supplying the linq-expression and the server side has access to db.Customers.

That’s it.

Continue reading...

Monitoring Transactions by emulated location at runtime using VED and Vugen script – Part 1

The Analysis application of LoadRunner 9.5 allows grouping transactions by emulated location. This enables users to see how each transaction performs in different locations. This is a very valuable feature but it is only available at analysis time, after the test stopped. In this post I’ll show how simple changes to the vugen scripts allow monitoring transactions from different locations, while the test is running. This is by no means a complete solution but it’ll show you the general idea behind it.

In this solution I’m taking advantage of the “User Data Point” mechanism in LoadRunner. The “User Data Point” mechanism allows users to include their own custom data points in the LoadRunner test database. These data points are viewable in the LoadRunner Controller UI when the test runs and in the analysis.

The main idea is that for each transaction reported to the controller, the script will also report a “User Data Point”. The data point will include the name of the emulated location or host machine. Adding the host name will guaranty uniqueness of the data point name and will allow pinpointing the location it comes from. If you are worried about performance costs associated with using many data points with multiple virtual users, you can implement a caching mechanism for each location that collects these data points and reports their averages as data points every second or so.

In my script I define StartTransaction and EndTransaction functions and replace all instances of lr_start_transaction and lr_end_transaction with them. These functions call the appropriate lr_ function internally and EndTransaction() also reports the data point. Although StartTransaction just calls lr_start_transaction, it’s a good idea to use it as it keeps the code clean and allows for easy modifications.

One note: I’m using “Host name” and not “Emulated Location” in the script because there is currently no API to get the emulated location name using a script.

Here is how the two functions are defined:

// Encapsulate lr_start_transaction() – for consistency with EndTransaction()

intStartTransaction( char* pName )

{

return lr_start_transaction( pName );

}

// Encapsulate lr_end_transaction(). Measures TRT and status and inserts

// a user data point that includes the host name – to distinguish the same transaction

// coming from other hosts.

int EndTransaction( char* pName, int status )

{

double dTransTime = 0;

int iTransStatus = 0;

char pDataPointName[1024] = {0};

int iRetVal = 0;

// Get the duration of the transaction.

dTransTime = lr_get_transaction_duration(pName );

// Get the status of the transaction.

iTransStatus = lr_get_transaction_status( pName );

// Signal the end of the transaction to LoadRunner.

iRetVal = lr_end_transaction( pName, status );

// If the transaction completed correctly, add it’s instance as a data point

if ( dTransTime > 0 && iTransStatus == LR_PASS && ( status == LR_PASS || status == LR_AUTO ) )

{

// Add the host name.

sprintf( pDataPointName, “%s::%s”, lr_get_host_name(), pName );

// Add the user data point.

lr_user_data_point( pDataPointName, dTransTime );

}

return iRetVal;

}

Here is an example of Action transaction that uses the new functionality:

Action()

{

/* … Removed beginning of script for clarity … */

//Signal a transaction start

StartTransaction( “Second Download” );

web_url(“testpage.asp_2″,

“URL=http://10.0.0.2/testanalyzer/testpage.asp”,

“Resource=0″,

“RecContentType=text/html”,

“Referer=http://10.0.0.2/”,

“Snapshot=t7.inf”,

“Mode=HTML”,

LAST);

web_link(“download 1000 bytes with 1 second delay”,

“Text=download 1000 bytes with 1 second delay”,

“Snapshot=t8.inf”,

LAST);

// Signal a transaction end.

EndTransaction( “Second Download”, LR_AUTO );

return 0;

}

When the test runs each transaction appears in the “Transaction Response Time” graph of LoadRunner but it’s value is aggregated for all locations. Using the StartTransaction / EndTransaction functions above you get the TRTs from each location in the data point graph aggregated only for virtual users running on that location. This allows you to see how each transaction performs from different emulated location.

Stay tuned as In a few weeks I’m planning to post another article detailing how to add bandwidth and latency to the mix and get a better view of how the network affects your users.

Continue reading...

Performance Engineering – Why so many companies don’t get it – Part 2

Part 2 (for part 1 click here)

Anyone who was ever part of a performance engineering process should be able to relate to the following story:

“…Version 3.5 of a critical application is scheduled for release in 6 weeks, the latest stable build (internal version) of the application finally made it to the hands of the performance engineering team 1 week ago. The team ran a few performance tests and found that this new version performs much slower than the previous 3.0 version. The team even identified potential tuning opportunities, but they require changes in the code. Trying to get time from the developers to address these issues was nearly impossible as they some are already working on a patch that adds more functionality to this version and some are already deployed at another project. The application ends up being deployed as is, clients begin complaining about the poor performance and the business unit points fingers at the performance engineering team for not delivering on its goal…”

Why does this happen?
Lack of goal commonality

The main reason behind this kind of story and others like it is the lack of goal commonality. The development team which has the biggest impact on application performance is measured on timely delivery of specified functionality and is rarely measured on the performance and responsiveness of those applications. While application performance is the goal of the performance engineering team, who can usually just verify the compliance of applications with their performance goals, but can rarely impact any changes that will actually improve performance. The performance engineering team can oversee performance improvement projects but the changes have to be driven through the development team, which as already established is not measured on performance, hence will be somewhat reluctant to spend time on these performance improving projects. There are other examples of lack of goal commonalities within IT (Network Ops and Application development is another common example), but none impact application performance (or lack there off) more than the above.
Performance vs.Functionality: Lack of goal commonality makes this a tough choice

But even if the development team isn’t measured on performance, don’t developers care about the performance and responsiveness of their applications?

Well off course they do.

I come from a development background and I can tell you first hand that developers want to take pride in the code they write, but when faced with the choice of spending time on performance improvements or on functionality enhancements, they are usually forced to choose the later.

Lack of tools and expertise

To make things worst, the lack of goal commonality prevents developers from being able to address performance problems even if they set the time for it, since for the most part developers are not equipped with the right tools, best practices and expertise to address potential performance issues. When development managers make their hiring decisions and development tool selections, they tend to make choices that will improve the team’s ability to deliver functionality fast and tend to focus less on tools and expertise that are performance oriented.

Performance later

There is a tendency in the development world to push performance to the last stage in the development life cycle, also known as “performance last” or “performance later”. While this approach may make sense initially, since you can’t test for performance before you have runable code that passed functional testing, it is one of the reasons behind many of the performance problems that exist in applications today. As we will see in future posts, many performance problems are a result of decisions made very early in the development process, such as platform selection, development language selection and even mundane things such as the type of user interface container selected for a specific web page.

In summary we identified the lack of goal commonality between application development teams and performance engineering groups as one of the reasons behind a sub optimal performance engineering practice. In the next post we will look into the lack of a clear goal as a fundamental roadblock in achieving efficient performance engineering.

In the mean time I am interested in your feedback, check within your company’s software specification documents, how many have performance requirements documented in the specifications? How many end up into the test plan? Curious to hear what you find.

Talk to you soon…

Continue reading...

Performance Engineering – Why so many companies don’t get it

The following is a series of posts that I originally wrote in www.excellingit.com but the issues raised are very relevant in this context as well. If anything, the time that past since I originally wrote this only reconfirmed the following statements.

Curious to hear your feedback on the following:

Performance Engineering – Why so many companies don’t get it

Many of you may read the headline and wonder, with so much money spent on performance engineering tools and personnel, surely most companies are getting it right. Well, most companies do get some performance engineering tasks right, but anyone familiar with the industry would agree that all or part of the following happens in almost every IT organization:

Applications are slower than initially expected
This story should sound familiar to anyone who ever dealt with deploying an enterprise application. The application goes through rigorous testing, first functional and later load and stress. Baselines are generated and average response times are predicted or estimated (and sometimes guestimated), only to find out that real end users, mostly remote but sometimes even local users experience a much slower application. It is relatively easy to verify the existence of this problem in your organization, simply compare the pre deployment response time baselines with the Application Performance Monitors (APM) statistics about the user experience post deployment. You are sure to find anything from a 2X up to over a 10X difference in response times, especially in applications that are deployed globally. Another way to validate the existence of this situation is by asking the IT department to predict how an application will perform once deployed globally, 99 times out of 100 you will not find a confident answer to that question.

Return on Investment is Hard to Show
In my line of business I get to work with a variety of organizations with different attitudes towards performance. In general there is a growing understanding that improved application performance has a positive impact on the financial bottom line. As a result more and more companies invest in performance engineering processes, tools and personnel in order to get from thinking about performance to actively addressing performance requirements (reaching the goal). However, even the most sophisticated organizations where performance is treated as a product with specific goals and objectives, even those cutting edge IT departments have a hard time showing the return on investment from their performance initiatives.

Performance Engineering becomes Load Testing
Effective performance engineering is hard to achieve. A practice that meets the performance engineering goal must address application performance from end to end. Very few organizations have the tools and expertise required to gain visibility into all the factors that impact application performance. As a result many organizations tend to focus on what can be measured and predominately that means focusing on load testing. Now don’t get me wrong, assessing an application’s ability to perform under load is critical to any successful deployment, but it is certainly not enough. Focusing on load testing is analogous to looking for a lost coin under the street lamp, load testing will highlight a set of potential performance problems, but many other may lurk in the dark and may have a bigger impact on performance than anything that comes up during the load test alone.

There are many other examples of how a suboptimal performance engineering practice manifests itself, in the 2nd part of this post we
will cover the reasons why most companies have a hard time implementing an efficient performance engineering practice.

Talk to you soon…

Continue reading...

Running mstest without Visual Studio.

In this post I will share a tip of how to install mstest in a standalone mode, i.e. without the Visual Studio.

Why would I want such a thing? Well, I am currently in the process of configuring our Continuous Integration (CI) server and I believe that such a server should have only the bare minimum necessary to do its job, nothing else. We develop our .NET code using Visual Studio 2008 Pro and use mstest as the unit test engine. So, these are the items I would like to have on my CI server:

• Windows Server 2003 – the OS
• Java, Tomcat, Hudson – my choice of the CI solution
• IIS with FrontPage server extensions – to perform unit tests of my WCF services hosted in IIS
• .NET v3.5
• msbuild
• mstest
• FxCop

Like I said, just the bare essentials. However, it turns out that mstest and Visual Studio are tightly coupled together. In fact, so tight, that googling for ways to break this coupling yielded no solutions. Not that other folks do not want to achieve the same goal as I do. The closest thing to qualify as the solution is posted by Christoph De Baene here. But, it requires us to install NUnit and Microsoft Team System NUnit Adapter from Exact Magic Software, which does the job of translating mstest attributes to nunit. Both tools are free, easy to install and NUnit is excellent, but, alas, we do not use it. (The reason is that it does not integrate with the Visual Studio, one needs TestDriven.NET for that, which would cost us more than $1000). So, I decided to divorce mstest from Visual Studio myself and the right tool for this is the excellent Process Monitor from SysInternals.

Without further ado I present my solution to how install mstest without the Visual Studio on a clean win2003 server.

Given:

• A win2003 server (the CIServer) with:
  • IIS with FrontPage server extensions (irrelevant for this post)
  • .NET framework v3.5
• My development machine (the DevMachine) with:
  • Visual Studio 2008 Pro installed in C:\Program Files\Microsoft Visual Studio 9.0 (VSINSTALL)
  • Microsoft SDK installed in C:\Program Files\Microsoft SDKs\Windows\v6.0A (MSSDK)

Isolating mstest from Visual Studio involves the following steps:

1. In order to build .NET code the CIServer must have a few more files, which it currently lacks:
   • sgen.exe – just to build any .NET code.
   • al.exe – I use PostSharp, so this is needed as well.

   All these files are found in the MSSDK directory on the DevMachine, so I just copied them to C:\WINDOWS\Microsoft.NET\Framework\v3.5 on the CIServer.
   (Of course, you copy them with the respective config files – sgen.exe.config and al.exe.config)

2. Copy the TeamTest directory to C:\Program Files\MSBuild\Microsoft\VisualStudio\v9.0 on the CIServer.They are found in the same directory on the DevMachine.
   This step is only needed if private accessors are used in the unit tests.
3. Put Microsoft.VisualStudio.QualityTools.Resource.dll and Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll in the GAC on the CIServer, because that is where they are on the DevMachine.
4. Create a stub directory for Visual Studio files, I named it VS2008Stub and created it under C:\ on the CIServer. This directory is populated by some files from the respective locations in the VSINSTALL directory on the DevMachine, namely these:
   • C:\VS2008Stub\Common7\IDE\MSTest.exe
   • C:\VS2008Stub\Common7\IDE\MSTest.exe.config
   • C:\VS2008Stub\Common7\IDE\VSTestHost.exe
   • C:\VS2008Stub\Common7\IDE\VSTestHost.exe.config
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.AgentObject.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.CommandLine.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Common.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.ControllerObject.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.ExecutionCommon.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.HostAdapters.ASPNETAgent.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.ObjectModel.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Tips.UnitTest.AssemblyResolver.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.TMI.dll
   • C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.SmartDevice.TestHostAdapter.dll
   • C:\VS2008Stub\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
   • C:\VS2008Stub\Common7\Packages\Debugger\msdia90.dll
5. Setup the registry by running the attached registermstest file (it is zipped, so you will have to unzip it first). It was created by concatenating three different keys exported from the DevMachine registry:
   • [HKLM\SOFTWARE\Microsoft\VisualStudio\9.0] – just the InstallDir value, which must be set to “C:\\VS2008Stub\\Common7\\IDE\\”
     Without it msbuild.exe will not build the unit tests. (Of course, the DevMachine has this value refer the VSINSTALL directory)
   • [HKCR\CLSID\{4C41678E-887B-4365-A09E-925D28DB33C2}] – This is for the msdia90.dll loaded by mstest.exe
   • [HKLM\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools] – needed by mstest.exe

That’s it. One can now run msbuild and mstest without Visual Studio on the CIServer.

Although, I must admit that I briefly had a problem with mstest, which failed with a mysterious error: Can’t save results of the run since ResultFilePath is null. Nothing that I tried helped to resolve the issue. Finally, I decided to debug mstest (see how frustrated a man can be!) and so I installed the Remote Debugger for Visual Studio on the CIServer, which is a very light install (though it does create the “C:\Program Files\Microsoft Visual Studio 9.0″ directory). However, just installing the remote debugger had eliminated the problem! I guess, the install did something that fixed it. To test this assumption, I uninstalled it (which completely removed the “C:\Program Files\Microsoft Visual Studio 9.0″ directory). Guess what? It still works! So, it remains a mystery… I wish I could provide a more definite answer, but I cannot.

As I mentioned before, I used the Process Monitor to come up with the described procedure. It is possible, that some pieces are still missing and that my unit tests have not hit these missing areas yet. Well, when the time comes I will fire up the Process Monitor again and complete any missing pieces. That is the price to be paid for divorcing mstest from the Visual Studio.

IMPORTANT UPDATE:

We had to abandon mstest as our testing solution, because it lacks certain advanced features required to implement complex testing scenarios:

• Combinatorial tests
• Dynamic test cases

While combinatorial tests may be simulated using data sources, the question is what for? There are mature solutions, letting do it out of the box. So, we have switched to MbUnit 3 maintained by Gallio. Much recommended.

UPDATE 2:

Apparently 64 bits machines require more registry settings (mine is 32 bits, so I was unaware of it). Below is the relevant comment from Dean Pendley, whose comment failed to show up due to some strange technical problems in our blog space. This is it:

I did have one issue the same as Alien in the comment where mstest reported

HKEY_LOCAL_MACHINE\\EnterpriseTools\QualityTools\TestTypes Key cannot be found!

After a little work I found that on 64 bit machines only, I think, you need a little more registry.  The key that you need is

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\9.0\EnterpriseTools

Hope this helps to those running 64 bits machines.

Continue reading...

More on how to add network conditions to a LoadRunner 9.5 Scenario

Last week I published a post (here) on how to add network conditions to a LoadRunner 9.5 scenario. For those of you who prefer a step by step video tutorial, the following video is an excellent one by David Berg who is Shunra’s product manager for VE Desktop.

Enjoy:

Amichai

Continue reading...

Adding Network Performance Analysis to a Quick Test Pro Script

This is the second post in the series that covers the use of QTP with VE Desktop. In the previous post I showed you how to add network conditions to a QTP script.

In this post I will expand the sample to automatically analyze the performance of transactions under network conditions. This leverages the built in packet capture and analysis capability within VE Desktop and leverages the VE Analyzer product for the reporting and analysis oprtion.

The process for adding the analysis is quite simple, it involves adding synchronization points at the end of each transaction as well as adding VE Desktop markers at the beginning and end of each transaction. The analysis itself can be triggered automatically at the end of the test.

First we need to add synchronization points at the end of each transaction so the playback will wait for the page to finish loading before ending the transaction. The easiest way to add synchronization points is during the recording of the QTP script, in this example the script waits for the “findFlights” button image to be visible before completing the “SignIn” transaction.

Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").WaitProperty "visible", true, 10000

More synchronization points need to be added, one for each transaction (for network analysis it is recommended to mark each none human intervention step in the script, i.e. from the “click the submit button” until the page loads).

Next we need to activate the packet capture before the test begins and we will also configure the analysis to run automatically at the end of the test, reminder in the previous post we defined the ctrl object which provides access to the VED client functionality:

'Activate the packet list and configure the auto analysis
ctrl.ActivatePacketList
ctrl.SetAutoAnalysisParams True, "HTTP", True

Finally we need to mark the beginning and end of each transaction so the VE Analyzer will be able to provide its analysis on a transaction be transaction basis. By default, if you don’t mark the transactions the analysis will be provided on the entire script, but as you will quickly find out, the ability to correlate the business logic to the analysis is very useful.

The marking of the transactions is done with the following functions:

ctrl.StartTransaction "SignIn"
' Transaction runs here
ctrl.StopTransaction 1

The final script looks like this:

'Load the COM interface for VE Desktop API
Set ctrl = CreateObject("ShunraVEAPI.ShunraVEDController")

'Just in case network conditions are already applied , stop the VED test
returnStatus = ctrl.StopTest()
If returnStatus  0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

' check out a license from the VED server
returnStatus = ctrl.LicenseCheckout(5)
If returnStatus  0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "License Checkout", text
End If

' Activate the packet list and configure the auto analysis and auto save
ctrl.ActivatePacketList
ctrl.SetAutoAnalysisParams True, "HTTP", True

' Activate the network conditions 50 msec latency, 0% packet loss and unlimited bandwidth (0)
returnStatus = ctrl.StartAdvanced (100,0,0)
If returnStatus  0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "Start Advanced", text
End If

'BEGINNING OF THE QTP ACTION NOW WITH VED MARKERS
ctrl.StartTransaction "SignIn"
Services.StartTransaction "SignIn"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("userName").Set "amichai"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").WebEdit("password").SetSecure "49c2a6682829d59b84f9eabf8c8d76a8557e"
Browser("Welcome: Mercury Tours").Page("Welcome: Mercury Tours").Image("Sign-In").Click 24,4
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").WaitProperty "visible", true, 10000
Services.EndTransaction "SignIn"
ctrl.StopTransaction 1

ctrl.StartTransaction "SearchFlight"
Services.StartTransaction "SearchFlight"
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").WebList("fromDay").Select "12"
Browser("Welcome: Mercury Tours").Page("Find a Flight: Mercury").Image("findFlights").Click 34,6
Browser("Welcome: Mercury Tours").Page("Select a Flight: Mercury").Image("reserveFlights").WaitProperty "visible", true, 10000
Services.EndTransaction "SearchFlight"
ctrl.StopTransaction 1

ctrl.StartTransaction "SignOff"
Services.StartTransaction "SignOff"
Browser("Welcome: Mercury Tours").Page("Select a Flight: Mercury").Link("SIGN-OFF").Click
Browser("Welcome: Mercury Tours").Page("Sign-on: Mercury Tours").Image("Login").WaitProperty "visible", true, 10000
Services.EndTransaction "SignOff"
ctrl.StopTransaction 1

'END OF THE QTP ACTION

'Stop the network impairments
returnStatus = ctrl.StopTest()
If returnStatus  0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "StopTest", text
End If

returnStatus = ctrl.LicenseCheckin()
If returnStatus  0 Then
	ctrl.GetLastError returnStatus, text
	Reporter.ReportEvent  micFail, "License Check In", text
End If

'Release the COM object
Set ctrl = Nothing

Continue reading...

Adding Network Analysis to a LoadRunner 9.5 Scenario

In a previous post I explained how to add network conditions to a LoadRunner scenario, answering the question, “How will this application perform in a remote site?”  The results from running a LoadRunner test with network conditions will highlight transactions that perform well under network conditions as well as pinpoint transactions that will degrade in performance under network conditions.

In this post I will show you how to generate analysis in less than three minutes that addresses the question that naturally follows. “Why does this transaction perform well locally, but slows down under network conditions?”

This analysis can be generated by installing VE Analyzer on the “remote” load generator and configuring it with the packet analysis settings (reminder: the “remote” load generator is any load generator that has the VE Desktop Professional WAN Emulation client installed on it. This WAN Emulation client enables us to simulate remote network conditions from a local load generator in the lab. Hence the double quotes in “remote” since the generator isn’t physically remote but is simulated to behave that way)

Follow these steps to setup the analysis on the “remote” load generator:

In the LoadRunner Controller Design screen click on the Load Generators button

Select the “remote” load generator

Click on details and select the WAN Emulation Tab:

Check the “Enable WAN Emulation” check box (if not already selected) and click on “WAN Emulation Settings”

See the previous post on how to setup the WAN Emulation parameters in the following screen.

Click on the additional settings button

First check the “Capture Packets” check box, this enables the packet capturing agent on the VE Desktop Client on the LoadGenerator. I recommend assigning at least 100 MB of buffer space for the capture so at least one iteration of the script will be captured.

For web based applications, select HTTP in the “Automatically run post-test analysis” option, for other applications select either TCP or UDP analysis.

Finally, select a folder on the load generator for automatically saving the test results. It is important to re-iterate that this folder is on the “remote” load generator, not on the LoadRunner Controller.

It is recommended to setup only 1 virtual user to run from this “remote” generator, so the analysis will focus on a single instance of the user. If load is needed as part of the scenario, it should be generated from a second load generator, in this case the analysis will show results of the marginal user.

Adding the analysis capabilities in this example doesn’t require any changes to the script and can be applied to the same scenario we presented in the previous post. Running this test results in an analysis report that is generated on the LoadGenerator. This report can be accessed from the VE Reperter that is installed on the “remote” load generator.

VE Reporter stores the reports based on the test names and the test run dates, so we are looking for the last test run for our test.

Once the report opens we are mainly interested in 4 reports:

The Application Efficiency Report shows us how chatty the application is (application chattiness is directly related to response time slow downs due to network latency) as well as how long it took to download various resources from the server. In this example, we see a resource that took 9 seconds to download, explaining why the entire action took 11 seconds for a remote user Vs, 0.6 seconds for a local user.

The second report is the Client Network Server Breakdown that shows how much time the transaction spent on the client, network and server.

The 3rd report is the Bandwidth Analysis which shows how much bandwidth was used by this single application user.

Finally, for a real deep dive we can use the Bounce Diagram that displays the entire communication between the client and the server as well as the delta times between consecutive packets, high delta times can point out to blocking events within the transaction.

In summary, in less than 3 minutes we can generate the above reports on any transaction that seems to be performing poorly over the Wide Area Network. This information can help the developers and network engineers to fine-tune the application and the network to improve the end user experience.

Feel free to suggest additional topics you would like to see covered in this series of posts.

Best,

Amichai

Continue reading...