The FCC’s tool lets you measure download and upload speeds, as well as latency and jitter (the last two are important for video and voice applications).  Nobody talks about latency and jitter and that is a shame.  For high quality video conferencing, less than 20ms of jitter is best.  Latency less than 30 ms is excellent. 

If you want to test your Internet connection, you must choose what type of consumer are you, home or commercial, including street address and zip code.

The test include upload and download speeds and latency and jitter to the service provider. Results are comparable to speedtest.net, a site that also utilizes OOKLA technology.

The FCC site also provides an Iphone application to measure network performance on a WI-FI network using a smart phone.

If your Iphone is connected to a WI-FI network, your results will reflect the speed of your broadband connection. If you disable Wi-Fi and then perform the test, the results will reflect the performance on the 3G network provided by At&T.

I believe this to be a great step to forcing greater transparency in the Internet access space and eventually will force Service providers to increase speed and lower costs.

While actual deployment may be month away, it is important to remember that the transport mechanism, open Internet over an IP Core will require substantial testing of the end user experience. IPTV is a protocol sensitive to network conditions, like latency and packet loss.  Any successful implementation will have to include significant bandwidth allocation testing, as well as load under latency, jitter and packet loss.

However this series of posts is about the day after a data center move. Now that the data center is remote, how does this paradigm shift impact the way we should develop, test, deploy, monitor and troubleshoot applications. I will try to cover as many topics as possible, but the main focus is still going to be around the role application performance management plays in this new paradigm.

I will start by covering the key reasons behind the performance impact that is experienced when applications are hosted in a remote data center? Those reasons are fairly intuitive, but it is important to understand them in depth in order to adequately plan for those new conditions. Two main things impact how applications perform when application servers are hosted in a remote data center vis a vis their application clients :

1. The performance of the network link between the client and the remote data center. This performance is defined by a set of network performance metrics that are application independent (for now we will ignore application aware networks, however the following basic concepts still hold in this scenario as well).

2. The application efficiency, specifically how efficient the application is when transferring data between the client and the remote server (and other tiers if applicable). This is an application attribute (and some time an attribute of a specific business process within the application). These attributes are application specific and are independent of any underlying network.

Let’s start with understanding the network performance metrics. Consider the following scenario:

An application is hosted in a NYC data center, with users in 2 places, some are in a NYC headquarters next to that NYC data center and some are in a remote branch that is located in San Francisco. The question is: “will the application perform the same for both type of users (local users in headquarters and remote users in SF)? In other words will the application be as responsive to the San Francisco user as it is to the NYC user?”

Well the obvious answer is NO, in most cases a NYC user will enjoy a faster more responsive application. What is less obvious is why? What is it about the network that causes remote users to experience a slower application than local users? The rest of this post will cover that question, future posts will address the application specific attributes. Once we cover that we will be ready to examine best practices in building applications for a remote data center.

When I ask this question during my training seminars, I get a variety of answers, many of them are the right ones, but I would like to address one wrong answer that keeps repeating itself for some reason.

Collisions – there is a general conception that collisions are common phenomena on the network which can explain any bad thing that happens to applications. The truth is that collisions are almost a thing of the past (on Enterprise LANs anyhow) and even when they happen they can’t explain why a remote user has a worst experience than a local user as both will experience a similar collision chance since collisions is a phenomenon that happens on local area Ethernet networks. If there are collisions on the Enterprise LAN it usually points to a configuration issue on a network device (like a duplex miss-match) but is still unrelated to the answer to our question.

Now to the right answers to the question, what is it about the Wide Area Network that causes applications to slow down:

There are 5 key conditions that predominately exist on Wide Area Networks and impact application performance, each in their own way:

1. Network Latency – the time it takes a packet to traverse from a source to the destination across the network, measured in milliseconds [msec]. A typical WAN link will introduce latency in the range of 10msec – 500 msec.

2. Bandwidth constraints – how fast can data be processed by the network link, measured in bits per second [bps, Kbps, Mbps, Gbps]

3. Bandwidth utilization (background traffic) – the percentage of bandwidth that is utilized by traffic that already exists on the link (background traffic).

4. Jitter – the deviation of the inter packet gap of sequential packets across a network link, it is a result of the deviation of the network latency and is sometimes used interchangeably with that standard deviation, measured in milliseconds [msec].

5. Packet Loss – the chance to drop a packet across an end to end network link, measured in %. Sometimes presented as the inverse metric called packet delivery rate.

The above are called network impairments, you can click on each one of the links to learn more about them and their causes.

Network impairments are performance conditions that inhibit the flow of data across a network. Each impairment type has an impact on the performance of business applications and network services. Some applications may be very sensitive to network impairments and some may be almost network agnostic. Sorting applications based on their network sensitivity is one of the important steps in performance engineering

In the next post we will discuss how application design can impact performance across the network. But in the mean time I would like to introduce a question for the group:

“We identified network latency as one of the key reasons that impact application performance; we also said that a typical WAN link will introduce 10 – 500 msec of latency. The question is, why does network latency have a big impact on application performance? surely a user doesn’t notice an increase of a few msec in response time, even 500 msec = ½ second goes by in a flinch. So think about it and let me know what you found based on your experience, why does network latency have such a big impact on application performance?”

Until next time,

Amichai Lesser

In alignment with global trends, more and more clinical trials take place outside of the United States, while the documentation and analysis is done in the US for submission to the FDA. So it is not out of the ordinary to find an Oracle Clinical user in Eastern Europe or in China submitting data to an Oracle Clinical Server hosted in the united states.

Unfortunately, as commonly found with an off the shelf software package that gets customized over time, this type of application can easily morph into a performance challenged application, especially for remote users. The following analysis shows an example of an Oracle Clinical transaction that completes in 8 seconds for local users, while extending to over a minute when users in E. Europe tried to use it.

Transaction Response Time Analysis for an Oracle Clinical Application Accessed by Local Users Vs. Remote Users in E. Europe

Obviously, that big jump in response time was a great cause for concern with my client’s management. Which is how I got the privilege to be asked to analyze the root cause of the poor performance.

The first thing I noticed was that one particular transaction (Connect to DB) had an extremely high jump in TRT from 8 seconds locally to over a minute when accessed remotely, so I focused on that one first.

Looking at the network fingerprint of the transaction you can observe the following things:

The transaction size (amount of data downloaded from the server required to complete the transaction) is extremely high (over 5 MB).

The transaction generates 34 HTTP calls. Some of those calls take disproportionally longer time than others (notice the Get jar file calls that each exceed 19 seconds marked below, click on the image for a larger view)

Oracle Clinical, Transaction Analysis for "Connect to DB"

When inspecting those JAR files, we saw that they were very heavy in size, over 1 MB each which is pretty big for basically a collection of Java classes that are zipped into a JAR file.

You can also observe in the next bounce diagram that most of the JAR files are downloaded in a serial fashion, blocking other objects from downloading in the mean time. Notice the delta time displayed on the right column

Bounce Diagram for an Oracle Clinical Transaction

What can be done to improve the performance of this application?

Oracle Clinical uses JAR files to package Java code that will run on the client. It is not uncommon for customized off the shelf applications to increase in code size over time until the code is very bloated.

There are several best practices that developers can follow to reduce the size of JAR files:

1. Rationalize the code – as applications develop over time, multiple classes and sometimes adjacent projects might reference similar libraries and other assets. If not careful those libraries and common assets end up packaged multiple times inside the JAR file, causing it to inflate in size.
2. Minify the code – a quick Google search for “reduce the size of JAR files” will reveal several free tools that can minimize the size of JAR files, usually through eliminating white spaces, comments, shortening variable names, etc.
3. Defer loading – chances are that not all the code in the JAR file is needed for the “connect to DB” transaction, which means that users that only want to perform a small transaction are penalized by the download time of code that will never get executed by them. Deferred loading is a design pattern that simply says “only download assets or code when it is needed” I wrote more on that design pattern in this previous post “http://www.shunra.com/shunrablog/index.php/2009/01/29/content-loading-when-being-lazy-pays-off/”

In summary, as more and more web 2.0 technologies hit main stream, new forms of performance pitfalls may present themselves, requiring us to consider application performance for remote users even for applications that traditionally were considered “remote user friendly” such as web based applications.

How was this Analysis Performed?

This analysis was used by integrating a HP LoadRunner script modeling an Oracle Clinical business user with Shunra VE Analyzer. So as the scripted transactions executed across a virtual WAN (simulated by the Shunra VE) the beginning and end of each transaction were marked by the VE Transaction Manager. At the end of the test, the VE Analyzer had sufficient data to generate the attached reports, enabling us to pin point the root cause of the performance problem. You can learn more on how to set this up by going through Shunra Certified Performance Engineering training, either on site or at the next Shunra University. Learn more about Shunra training here:  http://www.shunra.com/shunra-university-overview.php and here http://www.shunra.com/training-overview.php?keyword=services

Questions, comments, feel free to write me at amichai.lesser at Shunra dot com. Or comment on any of my posts.

BTW, we just created a new Application Performance Management Group on LinkedIn, feel free to look it up and join here  http://www.linkedin.com/groupRegistration?gid=2200667

Best,

Amichai Lesser

In the previous post http://www.shunra.com/shunrablog/index.php/2009/08/14/data-center-relocation-questions-and-answers-part-1/ I shared some of the questions that clients typically ask me during the performance analysis service in a data center relocation project. One of the most popular question, especially at the beginning of the project is “which applications are the most sensitive to network latency?”. Initially I thought of writing a post that categorizes and summarizes those applications, however, after reviewing the (long) list and reading through the analysis, I figured the community could benefit from some of the specific examples. I promise to try and summarize it all at the end.

In the next 2 posts, I will address the first 2 questions, the third one is probably a good topic for a book, so I will attempt to address that one over several future posts.

Should you really test all your applications for latency prior to a data center relocation?

Before jumping into a direct answer with examples of horror stories from past data center relocations that didn’t test all applications and the consequence of that excessive risk taking I want to first provide a conceptual best practice answer.

I recently received my Six Sigma Champion certification and one of the concepts in Six Sigma and in management for quality in general is “Don’t mass inspect, build quality into the process instead”. This concept was one of Deming’s teachings that revolutionized the Japanese manufacturing industry in the 1950s. So one would think that according to that best practice, we shouldn’t test all applications prior to a data center move, since that would be the equivalent of mass inspection. However, that would be a miss interpretation of that statement, an organization can shift from mass inspection to more statistical sampling only if quality is built into the process and when mission critical processes are in control. To complete the analogy, an organization can settle for sample testing a subset of applications only if application development processes have quality built into them, and specifically with a data center move, application development processes have network aware best practices embedded in them. I have yet to find an organization that reaches that level of maturity prior to a data center move. Usually the causation is reverse, first a firm relocates their data center, then IT realizes that they need to adopt a new application development paradigm since from now on applications will access remote servers in a remote data center and thus network aware best practices (like using VE Desktop as part of the development life cycle) are added.

So this brings us to the reasonable assumption that most companies will have a portfolio of applications that weren’t developed with a distributed WAN in mind and now have to adjust to operating in a remote data center. If that is the scenario your company is facing then mass inspection is your only way to mitigate the enormous risk associated with such a move.

Now if you don’t buy into the conceptual answer, consider the following:

1. In an average data center relocation project, we usually find that 20% – 40% of applications experience performance degradation ranging from visible to severe. (visible means that a user notices the slowdown in response time, severe means that an application stops being usable under latency conditions).
2. Many of the problems we uncover end up being show stoppers for the DCR project and need to go through remediation before the project can complete.
3. The mitigation path varies between applications and is very dependent on the type of problem that causes the performance degradation. Hence, a typical remediation solution may include WAN Acceleration for some applications (mainly server-to-server copy utilities and file services), terminal services (Citrix, remote desktop, VDI, etc.) for other applications, code fixes for some applications and architecture changes for the rest. Therefore engineering a solution based on a sample of applications will results in a sub optimal solution that doesn’t address all the performance problems that will manifest post the move.
4. When a large NY based insurance company relocated their servers from NY to Atlanta, GA without testing, they ended up rolling back the first phase of the relocation when several mission critical applications became unusable due to poor performance.

In summary, unless IT has built network aware best practices into the software development process, an organization planning a data center relocation must test all of its applications to analyze the performance impact that new added latency will have on application performance. More best practices on this topic can be found in my recently updated white paper that can be found at http://www.shunra.com/predicting-the-impact-of-data-center-moves-on-application-performance-whitepaper.php

The next post will cover the question “Which applications are the most vulnerable to the added network latency?” small example, any applications that are running executables from a shared network drive, will experience severe performance problems, that and more will be covered in the next post.

In the meantime, I am curious to hear your experience with data center relocations, what approach did your IT group take? Test all, sample test or cross your fingers and just move? Let me know.

P.S. as part of my Six Sigma Certification, I developed a dashboard and a flow chart for a data center relocation performance analysis, it needs some work before it can be shared, but if enough readers ask for it, I will try to clean it up and post it.

Till next time,

Amichai Lesser

Jitter:

Network jitter is the variation in the arrival rate of packets (more specifically, the inter packet gap of subsequent data packets as they arrive over a network). For most types of data applications large gap variations between arrival times (high jitter) are acceptable. For voice or video applications, relatively small jitter can cause perceptible disturbances in the recreated voice or video at the receiving end.
Because Wide Area Networks over IP generally rely on shared access to the network and can have many asynchronous sessions all sharing the same network they tend to introduce relatively high jitter.

Jitter can be reduced with buffers. Buffering some incoming packets and then outputting them at a more regular rate reduces jitter. This is fine for applications that are not interactive, such as streaming music or videos. For interactive applications such as VoIP conversations or video conferencing, this scheme will add network latency, which reduces the quality of the conversation.

Of the topics covered, WAN emulation is clearly closest to my heart, but please be sure to read up on Microsoft Vista support, the new RDP agent, and what is happening with LoadRunner’s Analysis API.

To read the full story, click here.

Do you plan to move to LoadRunner 9.5 or Performance Center 9.5 in the near future?  Will the integration with Shunra’s WAN emulation motivate you to upgrade or accelerate your upgrade schedule?

First off, VE Desktop for HP Software 5.0 is a new product.  It is in no way related to LoadRunner MWE.  HP/Mercury customers last saw MWE in LoadRunner 8.1 (July 2005) and the feature was removed from the LoadRunner 9.x releases.

While MWE was indeed “Powered by Shunra,” it was done so through an OEM relationship.  MWE was sold and supported by HP/Mercury.  VE Desktop for HP Software 5.0 development was a collaboration between Shunra and HP.  The product is the result of both companies learning from our experiences, designing, and co-developing a new solution that would exactly meet the needs of our customers.  VE Desktop for HP Software 5.0 can do everything LoadRunner MWE could, and much more.  VE Desktop for HP Software 5.0 is sold and supported by Shunra.

The top three issues we chose to address were:

1) Accuracy: Since the MWE did not support bandwidth emulation or jitter, the results of its emulations often did not fully reflect the real-world performance of applications being testing.  To address this, VE Desktop for HP Software supports bandwidth emulation (symmetric and asymmetric), jitter, imported network characteristics from Shunra’s VE Network Catcher, and a host of other network impairments aimed at precisely emulating network links.

2) Reporting: With MWE, there was no network specific reporting.  If a test was run with multiple locations emulated, the LoadRunner results would still show transaction response time and other metrics averaged across the whole test.  In order to understand how applications fared at individual locations, scripts would have to be modified so the script and transaction names reflected the location they were being run at.  This was very challenging to implement and maintain.  In VE Desktop for HP Software 5.0, there are no script changes required for such reporting.  For users of VE Desktop for HP Software, HP Analysis added the ability to sort results by “Emulated Location.”  HP also added the automatic import of network statistics on a per Load Generator basis.  Now users can understand their application performance by location and merge that data with network statistics like bandwidth utilization, latency and packet loss.  Finally, users can understand if the network is impacting their application performance and if so, how.

3) Performance Center: LoadRunner MWE had no compliment in Performance Center.  VE Desktop for HP Software 5.0 supports both LoadRunner 9.5 and Performance Center 9.5.  Within Performance Center, dynamic host allocation is fully supported.

VE Desktop for HP Software 5.0 provides a lot of other benefits beyond the direct improvements over MWE.  Some of those include a robust API and protocol analysis capabilities, but those are topics for other blog posts.

I look forward to your comments on the explanations above and any feedback you have about our latest addition to the Shunra family of products.