It gets better!   For the beta program, HP is waiving all of their license fees!  (Amazon EC2 license/use fees still apply.)

And better yet!  Lots of reasons this is a win-win for HP LoadRunner customers:

• This makes LoadRunner significantly more available to testers.
• Using this delivery model, we can expect LoadRunner to be significantly less expensive  AND easier to deploy.
• Good-bye to  days of needing to own load testing software that you should be using regularly, but in reality may only use during specific phases of your SDLC.  Now you can pay for what you need, when you need it.
• Also gone are the days of installing and upgrading LoadRunner.  Have you ever tried to download a 1.2GB installer over a slow link?
• How about realizing that you need to uninstall a previous version before you can install the new one?  Now that HP and Amazon publish AMIs with the software pre-installed, all you need to do is spin-up the images you want and run them.

While my enthusiasm for this development is clear, it should be said that it’s not all, “moonlight and canoes.”

The Controller AMI in the beta program is limited to 250 virtual users.  Since HP hasn’t published their pricing yet, it’s not clear how larger tests will be accommodated.  For now, the beta testing will limit us to mid-sized tests.  Considering the actual technology (LR 9.5) is already stable and known to work well in virtual environments, there is little doubt that this solution will be massively scalable.

You will also need scripts to run in LoadRunner.  For existing or former LoadRunner customers, this is as easy as accessing your scripts from EC2.  For net-new LoadRunner customers, this will require some work.  Thankfully, the script generation environment, Virtual User Generator makes it easy to create simple scripts and there is an extensive community of users and consultants to help you develop the scripts you need.

Be sure to watch Mark Tomlinson’s video where he explains the benefit of testing applications for the cloud in the cloud.  The main benefit Mark demonstrates is the ability to omit your corporate firewall from the test environment.

One BIG benefit of testing in the cloud is eliminating the public network from your test environment:  If you are testing an application in the cloud from your on-premise lab, then you’re subjecting your test data to an unknown world of latency, packet loss and bandwidth limits.  Testing a cloud application from an on-premise lab can produce useless results as they  have no correlation to how your application will actually perform from your end user’s perspective.

Another benefit I would be remiss if I didn’t mention is that Shunra’s integration with HP LoadRunner, Shunra for HP Software already works well in clouds.  Shunra for HP runs inside of LoadRunner and lets you decide what network impairments to test.  You can import your production data, pick from pre-defined settings, or specify ad-hoc values to make each Load Generator emulate a different network location.  By doing this, you can conduct a performance test for all of your end user locations, entirely in a cloud environment, all the while paying as you go.  I may be a little biased, but my argument is that the functionality of Shunra for HP completes the picture for anyone performance testing applications that will be accessed remotely – and who isn’t?

As a techie who started performance testing using a green-screen and writing my own protocol stacks because the applications I was testing insisted on using proprietary protocols, I am excited to see professional quality performance testing moving into the cloud.  Although this innovative solution hasn’t created much new technology, it did make existing technology consumable in the cloud.  This is sure to challenge other vendors  to step up to the plate, making other products and services more cloud-enabled.

What do you think?  Are there any reasons you would not use this capability?  What are some other products or services that you hope will make their way into the cloud this year?

Which is why I was glad to stumble upon Shamus McGillicuddy article in www.searchenterpriseWAN.com titled “WAN engineers prepare networks as cloud computing adoption rises” that can be found here.

There are some good observations in the article, I especially appreciate the comment on how application architecture best practices such as minimizing application turns become key when developing applications for the Cloud as well as testing applications for network latency becomes critical as the enterprise is migrating more and more applications to the Cloud.

Many questions still remain though and the impact of Cloud computing on performance engineering is still to be determined. Some of the questions that are on my mind are: (and feel free to chime in with more questions or attempts at answers)

1. How do you manage the performance of an application in the Cloud?

2. What changes if any are needed to the service deployment model and version upgrades?

3. How do you conduct a performance test for an application or a service in the Cloud (internal clouds and external clouds)?

4. How do you troubleshoot a performance issue in the Cloud?

Something to think about…

Best,

Amichai Lesser

However, there is a problem with it. One cannot snoop WPF framework elements nested too deep. By too deep I mean somewhere around 60 layers. Trying to snoop at that nesting level crashes the Snoop and since it is injected into the snooped application process space, the application being snooped is crashed as well.  Which is very annoying.

First of all, there is nothing wrong with Snoop. The problem is inherent in WPF – one just cannot nest too many WPF framework elements. The limit is not 60, of course. It is 255 and is very easy to check. Try to write a simple WPF application with 256 nested panels – it is impossible. At some point the application will crash. Debugging it reveals an exception with the following message:

Layout recursion reached allowed limit to avoid stack overflow: ’255′. Either the tree contains a loop or is too deep.

This msdn forum post is precisely about this issue. Note, that a reply from a Microsoft guy Rob Relyea promises to increase the limit from 255 to 4096 in the upcoming .NET 3.5 “servicing release”. But, until then the problem exists.

So, why is Snoop affected at all? Simple, Snoop uses the WPF TreeView control to display the hierarchy of snooped WPF framework elements. The depth of a node in that tree is directly proportional to the nesting level of the respective snooped element. The following image is a snapshot of that tree created using the Snoop itself:

A snapshot of the Snoop property tree view.

(Actually, this is a slightly different version of the original Snoop. I have added the nesting level in the square brackets to the display of a node)

Inspecting the tree gives us the following formula for the nesting depth of a TreeViewItem object – 8 + 4*n, where n is the nesting depth of the respective snooped element. Hence snooping at the first framework element in a WPF application produces a node nested 12 levels deep, whereas snooping at an element at level 60 corresponds to a TreeViewItem at level 248 – dangerously close to the hard coded WPF nesting limit.

In reality, deep snooping of a complex WPF application almost always hits this limit, bringing down the Snoop code along with the snooped application.

So, how can one deal with this unfortunate situation? I see these ways:

• Be careful not to snoop too deeply – never works.
• Use the filter textbox to change the root node of the element TreeView. This is a good workaround, but from my own experience it is easy to forget oneself and snoop carelessly too deep – bang, you are toast.
• Forget about snooping, until the .NET 3.5 “servicing release” is actually released. The reply promising this release dates back to April 17th 2008. I have checked the state of the matters today (July 21st 2009) – the limit is still 255.
• Fix somehow the Snoop TreeView.

Since I love Snoop very much and use it extensively, I have chosen to fix this issue once and for all. The following items depict the fix:

• When a new node is added to the TreeView calculate its expected nesting level using this formula – rootLayoutDepth + levelLayoutDepth * n, where
  • rootLayoutDepth is the depth of the first TreeViewItem, calculated automatically somewhere in the beginning.
  • levelLayoutDepth is the depth difference between a node and its child, calculated automatically somewhere in the beginning and is assumed to be constant thereafter.
  • n is the zero based nesting level of the particular snooped element.
• If the expected nesting level is above 240, then a new TreeView root node is picked. I use the filtering feature of Snoop to setup a special reduce depth filter behind the scenes.

That’s it. There are, of course, many details to it, but the overall picture is captured pretty accurately by these two items.

A few final notes:

• The Mole visualizer does not have this problem, although it also displays the tree view of inspected elements. The reason is that Mole uses Windows Forms TreeView, which is a single control regardless of how many nodes it displays and how deep they are nested. The nodes are simply drawings on the control’s canvas.
• The described element nesting limit is not a regression from the Win32 world. Plain old Win32 windows have this limit as well, as described in this post.
• I have mentioned this issue some time ago in this msdn blog post. Cory Plotts responded to my post and he has added some code to the original Snoop source to better handle interop scenarios. Refer to his blog for details here.
• In addition to fixing the nesting issue, I have made the following changes to the original Snoop source code:
  • Fixed the property grid view – the last Snoop release has broken it.
  • Added the nesting level in the square brackets to the display of a node in the element tree.
  • Added a test application to the Snoop solution to make sure snooping at deeply nested elements works correctly.
  • Added a few more exception checks.

Last, but not least – the source code. Since I am not the owner of the Snoop application I am not putting the complete source code here, but rather the delta and the Release binaries. In order to compile the Snoop application with my fixes first the complete source code must be downloaded from here. Then, the delta containing my fixes should be copied over the Snoop source tree. Now build it – that’s it.

The release binaries are here.

Of the topics covered, WAN emulation is clearly closest to my heart, but please be sure to read up on Microsoft Vista support, the new RDP agent, and what is happening with LoadRunner’s Analysis API.

To read the full story, click here.

Do you plan to move to LoadRunner 9.5 or Performance Center 9.5 in the near future?  Will the integration with Shunra’s WAN emulation motivate you to upgrade or accelerate your upgrade schedule?

First off, VE Desktop for HP Software 5.0 is a new product.  It is in no way related to LoadRunner MWE.  HP/Mercury customers last saw MWE in LoadRunner 8.1 (July 2005) and the feature was removed from the LoadRunner 9.x releases.

While MWE was indeed “Powered by Shunra,” it was done so through an OEM relationship.  MWE was sold and supported by HP/Mercury.  VE Desktop for HP Software 5.0 development was a collaboration between Shunra and HP.  The product is the result of both companies learning from our experiences, designing, and co-developing a new solution that would exactly meet the needs of our customers.  VE Desktop for HP Software 5.0 can do everything LoadRunner MWE could, and much more.  VE Desktop for HP Software 5.0 is sold and supported by Shunra.

The top three issues we chose to address were:

1) Accuracy: Since the MWE did not support bandwidth emulation or jitter, the results of its emulations often did not fully reflect the real-world performance of applications being testing.  To address this, VE Desktop for HP Software supports bandwidth emulation (symmetric and asymmetric), jitter, imported network characteristics from Shunra’s VE Network Catcher, and a host of other network impairments aimed at precisely emulating network links.

2) Reporting: With MWE, there was no network specific reporting.  If a test was run with multiple locations emulated, the LoadRunner results would still show transaction response time and other metrics averaged across the whole test.  In order to understand how applications fared at individual locations, scripts would have to be modified so the script and transaction names reflected the location they were being run at.  This was very challenging to implement and maintain.  In VE Desktop for HP Software 5.0, there are no script changes required for such reporting.  For users of VE Desktop for HP Software, HP Analysis added the ability to sort results by “Emulated Location.”  HP also added the automatic import of network statistics on a per Load Generator basis.  Now users can understand their application performance by location and merge that data with network statistics like bandwidth utilization, latency and packet loss.  Finally, users can understand if the network is impacting their application performance and if so, how.

3) Performance Center: LoadRunner MWE had no compliment in Performance Center.  VE Desktop for HP Software 5.0 supports both LoadRunner 9.5 and Performance Center 9.5.  Within Performance Center, dynamic host allocation is fully supported.

VE Desktop for HP Software 5.0 provides a lot of other benefits beyond the direct improvements over MWE.  Some of those include a robust API and protocol analysis capabilities, but those are topics for other blog posts.

I look forward to your comments on the explanations above and any feedback you have about our latest addition to the Shunra family of products.