The FCC’s tool lets you measure download and upload speeds, as well as latency and jitter (the last two are important for video and voice applications).  Nobody talks about latency and jitter and that is a shame.  For high quality video conferencing, less than 20ms of jitter is best.  Latency less than 30 ms is excellent. 

If you want to test your Internet connection, you must choose what type of consumer are you, home or commercial, including street address and zip code.

The test include upload and download speeds and latency and jitter to the service provider. Results are comparable to speedtest.net, a site that also utilizes OOKLA technology.

The FCC site also provides an Iphone application to measure network performance on a WI-FI network using a smart phone.

If your Iphone is connected to a WI-FI network, your results will reflect the speed of your broadband connection. If you disable Wi-Fi and then perform the test, the results will reflect the performance on the 3G network provided by At&T.

I believe this to be a great step to forcing greater transparency in the Internet access space and eventually will force Service providers to increase speed and lower costs.

The report cites a global average speed of 1.7 MBPS with South Korea retaining the leadership position. As I mention in a previous post, the South Korean government is subsidizing deployment of fast speed internet in their quest to integrate the population from rural areas into their workforce. In the US, the reported average is 3.9 MBPS, declining from a year ago, with some exceptions, like Utah, Massachusetts or District of Columbia, all of those places showing a remarkable 15% increase from last year.

This report must help dispel the myth that Internet access is universally equal and unpredictable; in fact, it is quite simple to measure Internet access behavior patterns for individual locations or regions (associated with an Internet POP) and incorporate the methodology into the Performance Testing procedures.  Shunra provide the tools to measure the performance (VE Catcher) and to model the site locations by using VE Suite.

However, we may need to apply a sobriety test, where are those servers? And how far out is my virtual storage?  A recent article in Scientific Computing (http://www.scientificcomputing.com/articles-HPC-Cloud-Computing-Pie-in-the-sky-120109.aspx) reminds us that the flexibility of an elastic infrastructure comes with strings attached.

The author presents a case for testing adequate bandwidth, as well as measuring and testing network latency.  Virtual machines can introduce additional latency through the time-sharing nature of the underlying hardware. Since the service provider — and not the scientist — control the hardware, unanticipated sharing and reallocation of machines can significantly affect runtimes.

My conclusion from this article, and others, dealing with the latency issues induced by Cloud Computing, is to include a rigorous testing phase before committing to a migration. An end to end latency measurement and emulation are probably the best ways to ensure smooth adoption of this exciting new technology.

Which is why I was glad to stumble upon Shamus McGillicuddy article in www.searchenterpriseWAN.com titled “WAN engineers prepare networks as cloud computing adoption rises” that can be found here.

There are some good observations in the article, I especially appreciate the comment on how application architecture best practices such as minimizing application turns become key when developing applications for the Cloud as well as testing applications for network latency becomes critical as the enterprise is migrating more and more applications to the Cloud.

Many questions still remain though and the impact of Cloud computing on performance engineering is still to be determined. Some of the questions that are on my mind are: (and feel free to chime in with more questions or attempts at answers)

1. How do you manage the performance of an application in the Cloud?

2. What changes if any are needed to the service deployment model and version upgrades?

3. How do you conduct a performance test for an application or a service in the Cloud (internal clouds and external clouds)?

4. How do you troubleshoot a performance issue in the Cloud?

Something to think about…

Best,

Amichai Lesser

In the previous post http://www.shunra.com/shunrablog/index.php/2009/08/14/data-center-relocation-questions-and-answers-part-1/ I shared some of the questions that clients typically ask me during the performance analysis service in a data center relocation project. One of the most popular question, especially at the beginning of the project is “which applications are the most sensitive to network latency?”. Initially I thought of writing a post that categorizes and summarizes those applications, however, after reviewing the (long) list and reading through the analysis, I figured the community could benefit from some of the specific examples. I promise to try and summarize it all at the end.

However, there is a problem with it. One cannot snoop WPF framework elements nested too deep. By too deep I mean somewhere around 60 layers. Trying to snoop at that nesting level crashes the Snoop and since it is injected into the snooped application process space, the application being snooped is crashed as well.  Which is very annoying.

First of all, there is nothing wrong with Snoop. The problem is inherent in WPF – one just cannot nest too many WPF framework elements. The limit is not 60, of course. It is 255 and is very easy to check. Try to write a simple WPF application with 256 nested panels – it is impossible. At some point the application will crash. Debugging it reveals an exception with the following message:

Layout recursion reached allowed limit to avoid stack overflow: ’255′. Either the tree contains a loop or is too deep.

This msdn forum post is precisely about this issue. Note, that a reply from a Microsoft guy Rob Relyea promises to increase the limit from 255 to 4096 in the upcoming .NET 3.5 “servicing release”. But, until then the problem exists.

So, why is Snoop affected at all? Simple, Snoop uses the WPF TreeView control to display the hierarchy of snooped WPF framework elements. The depth of a node in that tree is directly proportional to the nesting level of the respective snooped element. The following image is a snapshot of that tree created using the Snoop itself:

A snapshot of the Snoop property tree view.

(Actually, this is a slightly different version of the original Snoop. I have added the nesting level in the square brackets to the display of a node)

Inspecting the tree gives us the following formula for the nesting depth of a TreeViewItem object – 8 + 4*n, where n is the nesting depth of the respective snooped element. Hence snooping at the first framework element in a WPF application produces a node nested 12 levels deep, whereas snooping at an element at level 60 corresponds to a TreeViewItem at level 248 – dangerously close to the hard coded WPF nesting limit.

In reality, deep snooping of a complex WPF application almost always hits this limit, bringing down the Snoop code along with the snooped application.

So, how can one deal with this unfortunate situation? I see these ways:

• Be careful not to snoop too deeply – never works.
• Use the filter textbox to change the root node of the element TreeView. This is a good workaround, but from my own experience it is easy to forget oneself and snoop carelessly too deep – bang, you are toast.
• Forget about snooping, until the .NET 3.5 “servicing release” is actually released. The reply promising this release dates back to April 17th 2008. I have checked the state of the matters today (July 21st 2009) – the limit is still 255.
• Fix somehow the Snoop TreeView.

Since I love Snoop very much and use it extensively, I have chosen to fix this issue once and for all. The following items depict the fix:

• When a new node is added to the TreeView calculate its expected nesting level using this formula – rootLayoutDepth + levelLayoutDepth * n, where
  • rootLayoutDepth is the depth of the first TreeViewItem, calculated automatically somewhere in the beginning.
  • levelLayoutDepth is the depth difference between a node and its child, calculated automatically somewhere in the beginning and is assumed to be constant thereafter.
  • n is the zero based nesting level of the particular snooped element.
• If the expected nesting level is above 240, then a new TreeView root node is picked. I use the filtering feature of Snoop to setup a special reduce depth filter behind the scenes.

That’s it. There are, of course, many details to it, but the overall picture is captured pretty accurately by these two items.

A few final notes:

• The Mole visualizer does not have this problem, although it also displays the tree view of inspected elements. The reason is that Mole uses Windows Forms TreeView, which is a single control regardless of how many nodes it displays and how deep they are nested. The nodes are simply drawings on the control’s canvas.
• The described element nesting limit is not a regression from the Win32 world. Plain old Win32 windows have this limit as well, as described in this post.
• I have mentioned this issue some time ago in this msdn blog post. Cory Plotts responded to my post and he has added some code to the original Snoop source to better handle interop scenarios. Refer to his blog for details here.
• In addition to fixing the nesting issue, I have made the following changes to the original Snoop source code:
  • Fixed the property grid view – the last Snoop release has broken it.
  • Added the nesting level in the square brackets to the display of a node in the element tree.
  • Added a test application to the Snoop solution to make sure snooping at deeply nested elements works correctly.
  • Added a few more exception checks.

Last, but not least – the source code. Since I am not the owner of the Snoop application I am not putting the complete source code here, but rather the delta and the Release binaries. In order to compile the Snoop application with my fixes first the complete source code must be downloaded from here. Then, the delta containing my fixes should be copied over the Snoop source tree. Now build it – that’s it.

The release binaries are here.

Of the topics covered, WAN emulation is clearly closest to my heart, but please be sure to read up on Microsoft Vista support, the new RDP agent, and what is happening with LoadRunner’s Analysis API.

To read the full story, click here.

Do you plan to move to LoadRunner 9.5 or Performance Center 9.5 in the near future?  Will the integration with Shunra’s WAN emulation motivate you to upgrade or accelerate your upgrade schedule?

First off, VE Desktop for HP Software 5.0 is a new product.  It is in no way related to LoadRunner MWE.  HP/Mercury customers last saw MWE in LoadRunner 8.1 (July 2005) and the feature was removed from the LoadRunner 9.x releases.

While MWE was indeed “Powered by Shunra,” it was done so through an OEM relationship.  MWE was sold and supported by HP/Mercury.  VE Desktop for HP Software 5.0 development was a collaboration between Shunra and HP.  The product is the result of both companies learning from our experiences, designing, and co-developing a new solution that would exactly meet the needs of our customers.  VE Desktop for HP Software 5.0 can do everything LoadRunner MWE could, and much more.  VE Desktop for HP Software 5.0 is sold and supported by Shunra.

The top three issues we chose to address were:

1) Accuracy: Since the MWE did not support bandwidth emulation or jitter, the results of its emulations often did not fully reflect the real-world performance of applications being testing.  To address this, VE Desktop for HP Software supports bandwidth emulation (symmetric and asymmetric), jitter, imported network characteristics from Shunra’s VE Network Catcher, and a host of other network impairments aimed at precisely emulating network links.

2) Reporting: With MWE, there was no network specific reporting.  If a test was run with multiple locations emulated, the LoadRunner results would still show transaction response time and other metrics averaged across the whole test.  In order to understand how applications fared at individual locations, scripts would have to be modified so the script and transaction names reflected the location they were being run at.  This was very challenging to implement and maintain.  In VE Desktop for HP Software 5.0, there are no script changes required for such reporting.  For users of VE Desktop for HP Software, HP Analysis added the ability to sort results by “Emulated Location.”  HP also added the automatic import of network statistics on a per Load Generator basis.  Now users can understand their application performance by location and merge that data with network statistics like bandwidth utilization, latency and packet loss.  Finally, users can understand if the network is impacting their application performance and if so, how.

3) Performance Center: LoadRunner MWE had no compliment in Performance Center.  VE Desktop for HP Software 5.0 supports both LoadRunner 9.5 and Performance Center 9.5.  Within Performance Center, dynamic host allocation is fully supported.

VE Desktop for HP Software 5.0 provides a lot of other benefits beyond the direct improvements over MWE.  Some of those include a robust API and protocol analysis capabilities, but those are topics for other blog posts.

I look forward to your comments on the explanations above and any feedback you have about our latest addition to the Shunra family of products.