The FCC’s tool lets you measure download and upload speeds, as well as latency and jitter (the last two are important for video and voice applications).  Nobody talks about latency and jitter and that is a shame.  For high quality video conferencing, less than 20ms of jitter is best.  Latency less than 30 ms is excellent. 

If you want to test your Internet connection, you must choose what type of consumer are you, home or commercial, including street address and zip code.

The test include upload and download speeds and latency and jitter to the service provider. Results are comparable to speedtest.net, a site that also utilizes OOKLA technology.

The FCC site also provides an Iphone application to measure network performance on a WI-FI network using a smart phone.

If your Iphone is connected to a WI-FI network, your results will reflect the speed of your broadband connection. If you disable Wi-Fi and then perform the test, the results will reflect the performance on the 3G network provided by At&T.

I believe this to be a great step to forcing greater transparency in the Internet access space and eventually will force Service providers to increase speed and lower costs.

Given the application traffic requirements, the challenge is not going to be the amount of traffic (Throughput) , but rather the network conditions, or impairments, that are affecting how the application traffic is going to be transmitted over the network, eventually arriving at the corporate data center south of the nation’s capital.

Using VE Catcher 4.6, a model of the latency and packet loss characteristics was created, for a period of 10 days. Two separate devices where utilized, on one AT&T 3G network, the other, on the Sprint network, to ensure carrier availability (the project call for a 50% split among the networks).  A pilot site, south of the data center, was selected, and VE Catcher created two parallel recordings, one for device “A” and another for device in the alternative carrier.

Once the behavioral model has been created, Shunra’s Virtual Enterprise appliance will allow creating a “test-bed” where the application will be sending real traffic to the data center over an emulated link. The network conditions will be imported from VE Catcher and reflect the exact conditions observed at each of the 3G carriers.

The performance of the application, running over the emulated links will allow the testing team to predict the behavior of the application well before field deployment is completed, and make adjustments required by the application and also predict behavior under extreme conditions, like heavy network traffic, inclement weather or other events.

You may not always use WAN emulation, but when you do, use Shunra.

Shunra Virtual Enterprise is your guaranty to a successful deployment.

When do these issues become as important in LANs as they are in WANs? Probably when our LAN becomes huge – such as Cloud computing Data Centers.

These Data Centers are interconnected with each other and one of the main differences with our traditional Data Centers is that they are managed automatically by very complex and smart management systems. These systems are capable of automatically allocate hardware and software resources for our applications, maintain these resources and expand/decrease capacity of the infrastructure based on the application demands almost in real time.

So imagine that the system is scheduled to perform an OS upgrade on the managed infrastructure: it will download OS images on thousands of machines thus creating huge traffic congestion within the Data Center. What will happen to the system that knows only how to work on LAN and was never tested on coping with delays? Packet Loss? – it might go down partially or completely. Since it was not tested – how can we know? (Sounds a little like Skynet scenario in Terminator, doesn’t it?)

The importance of the management system performance becomes utmost in this case and it makes sense to test how the system will react to the issues that now might happen within it, doesn’t it?

This is where WAN emulation solution can help – it can create those conditions within the Data Center or between the Cloud Data Centers and the “unexpected” can become “expected” and fixed.

While actual deployment may be month away, it is important to remember that the transport mechanism, open Internet over an IP Core will require substantial testing of the end user experience. IPTV is a protocol sensitive to network conditions, like latency and packet loss.  Any successful implementation will have to include significant bandwidth allocation testing, as well as load under latency, jitter and packet loss.

However, we may need to apply a sobriety test, where are those servers? And how far out is my virtual storage?  A recent article in Scientific Computing (http://www.scientificcomputing.com/articles-HPC-Cloud-Computing-Pie-in-the-sky-120109.aspx) reminds us that the flexibility of an elastic infrastructure comes with strings attached.

The author presents a case for testing adequate bandwidth, as well as measuring and testing network latency.  Virtual machines can introduce additional latency through the time-sharing nature of the underlying hardware. Since the service provider — and not the scientist — control the hardware, unanticipated sharing and reallocation of machines can significantly affect runtimes.

My conclusion from this article, and others, dealing with the latency issues induced by Cloud Computing, is to include a rigorous testing phase before committing to a migration. An end to end latency measurement and emulation are probably the best ways to ensure smooth adoption of this exciting new technology.

The following posts will answer this question and more, but in order to get answers we need to first address additional questions:

Consider the remote user in Tokyo, he is accessing multiple applications that are all hosted in the remote NYC data center.

Will all these applications perform the same way once hosted in a remote data center?

The obvious answer is NO, some applications will perform well even when users are remote to the data center, while others will provide intermittent poor performance and some will always perform poorly for a remote user.

The answers to the next questions are less obvious:

• Why do some applications perform well in remote data centers while others fail miserably?
• What is it about the way applications are designed and architected in a remote data center that allows some applications to perform better than others?
• What are the key design flaws that cause applications to perform poorly in a remote data center?

Since there are a lot of different applications, there are also a lot of different answers to these questions. In the next couple of posts we will focus on enterprise-data-driven- transactional applications and their performance flaws over the network (ignoring for a moment back-end or desktop related bottlenecks). We will further limit this category to client server based applications, ignoring for a moment the complexity of N-Tier applications and multi web service based applications, these will be dealt with in future posts.

Over the years I was asked to analyze performance problems for many transactional applications and specifically analyze their performance degradation when users are remote to their data center, the following are the key application metrics that I found are related to performance degradation over the network:

1. The number of application turns per transaction (or how chatty the application is)
2. The transaction size (or how much data needs to be downloaded from the server to the client in order to complete each transaction)
3. The transaction efficiency factor (how much data a transaction downloads per application turn)
4. The blocking nature of object retrieval (can the transaction retrieve multiple objects concurrently or is each object download blocking other object requests from being processed)
5. The transaction redundancy metric (or how much of the same data is being retrieved by all the requests made by this transaction). It seems like this metric should always be zero, but you will be surprised how often this is the single reason behind performance problems.
6. The transaction initialization size (how much data does the transaction download initially Vs. sequential navigational steps)
7. The caching ratio (how much data is cached locally as a percentage of the overall data needed by the application)
8. The latency scale factor (How does the backend’s ability to scale change when network latency is added between front end clients and the backend)

Measuring and observing these metrics allows for the deep level analysis that is required to identify performance bottlenecks. This information also helps to point out to application developers and system engineers what needs to be changed in order to remediate application performance problems.

In the next couple of posts I will explain each of the above factors and describe how each of them impacts application performance, so sign up for the RSS feed to get notification on these future posts.

Until then…

Amichai Lesser

However this series of posts is about the day after a data center move. Now that the data center is remote, how does this paradigm shift impact the way we should develop, test, deploy, monitor and troubleshoot applications. I will try to cover as many topics as possible, but the main focus is still going to be around the role application performance management plays in this new paradigm.

I will start by covering the key reasons behind the performance impact that is experienced when applications are hosted in a remote data center? Those reasons are fairly intuitive, but it is important to understand them in depth in order to adequately plan for those new conditions. Two main things impact how applications perform when application servers are hosted in a remote data center vis a vis their application clients :

1. The performance of the network link between the client and the remote data center. This performance is defined by a set of network performance metrics that are application independent (for now we will ignore application aware networks, however the following basic concepts still hold in this scenario as well).

2. The application efficiency, specifically how efficient the application is when transferring data between the client and the remote server (and other tiers if applicable). This is an application attribute (and some time an attribute of a specific business process within the application). These attributes are application specific and are independent of any underlying network.

Let’s start with understanding the network performance metrics. Consider the following scenario:

An application is hosted in a NYC data center, with users in 2 places, some are in a NYC headquarters next to that NYC data center and some are in a remote branch that is located in San Francisco. The question is: “will the application perform the same for both type of users (local users in headquarters and remote users in SF)? In other words will the application be as responsive to the San Francisco user as it is to the NYC user?”

Well the obvious answer is NO, in most cases a NYC user will enjoy a faster more responsive application. What is less obvious is why? What is it about the network that causes remote users to experience a slower application than local users? The rest of this post will cover that question, future posts will address the application specific attributes. Once we cover that we will be ready to examine best practices in building applications for a remote data center.

When I ask this question during my training seminars, I get a variety of answers, many of them are the right ones, but I would like to address one wrong answer that keeps repeating itself for some reason.

Collisions – there is a general conception that collisions are common phenomena on the network which can explain any bad thing that happens to applications. The truth is that collisions are almost a thing of the past (on Enterprise LANs anyhow) and even when they happen they can’t explain why a remote user has a worst experience than a local user as both will experience a similar collision chance since collisions is a phenomenon that happens on local area Ethernet networks. If there are collisions on the Enterprise LAN it usually points to a configuration issue on a network device (like a duplex miss-match) but is still unrelated to the answer to our question.

Now to the right answers to the question, what is it about the Wide Area Network that causes applications to slow down:

There are 5 key conditions that predominately exist on Wide Area Networks and impact application performance, each in their own way:

1. Network Latency – the time it takes a packet to traverse from a source to the destination across the network, measured in milliseconds [msec]. A typical WAN link will introduce latency in the range of 10msec – 500 msec.

2. Bandwidth constraints – how fast can data be processed by the network link, measured in bits per second [bps, Kbps, Mbps, Gbps]

3. Bandwidth utilization (background traffic) – the percentage of bandwidth that is utilized by traffic that already exists on the link (background traffic).

4. Jitter – the deviation of the inter packet gap of sequential packets across a network link, it is a result of the deviation of the network latency and is sometimes used interchangeably with that standard deviation, measured in milliseconds [msec].

5. Packet Loss – the chance to drop a packet across an end to end network link, measured in %. Sometimes presented as the inverse metric called packet delivery rate.

The above are called network impairments, you can click on each one of the links to learn more about them and their causes.

Network impairments are performance conditions that inhibit the flow of data across a network. Each impairment type has an impact on the performance of business applications and network services. Some applications may be very sensitive to network impairments and some may be almost network agnostic. Sorting applications based on their network sensitivity is one of the important steps in performance engineering

In the next post we will discuss how application design can impact performance across the network. But in the mean time I would like to introduce a question for the group:

“We identified network latency as one of the key reasons that impact application performance; we also said that a typical WAN link will introduce 10 – 500 msec of latency. The question is, why does network latency have a big impact on application performance? surely a user doesn’t notice an increase of a few msec in response time, even 500 msec = ½ second goes by in a flinch. So think about it and let me know what you found based on your experience, why does network latency have such a big impact on application performance?”

Until next time,

Amichai Lesser

In alignment with global trends, more and more clinical trials take place outside of the United States, while the documentation and analysis is done in the US for submission to the FDA. So it is not out of the ordinary to find an Oracle Clinical user in Eastern Europe or in China submitting data to an Oracle Clinical Server hosted in the united states.

Unfortunately, as commonly found with an off the shelf software package that gets customized over time, this type of application can easily morph into a performance challenged application, especially for remote users. The following analysis shows an example of an Oracle Clinical transaction that completes in 8 seconds for local users, while extending to over a minute when users in E. Europe tried to use it.

Transaction Response Time Analysis for an Oracle Clinical Application Accessed by Local Users Vs. Remote Users in E. Europe

Obviously, that big jump in response time was a great cause for concern with my client’s management. Which is how I got the privilege to be asked to analyze the root cause of the poor performance.

The first thing I noticed was that one particular transaction (Connect to DB) had an extremely high jump in TRT from 8 seconds locally to over a minute when accessed remotely, so I focused on that one first.

Looking at the network fingerprint of the transaction you can observe the following things:

The transaction size (amount of data downloaded from the server required to complete the transaction) is extremely high (over 5 MB).

The transaction generates 34 HTTP calls. Some of those calls take disproportionally longer time than others (notice the Get jar file calls that each exceed 19 seconds marked below, click on the image for a larger view)

Oracle Clinical, Transaction Analysis for "Connect to DB"

When inspecting those JAR files, we saw that they were very heavy in size, over 1 MB each which is pretty big for basically a collection of Java classes that are zipped into a JAR file.

You can also observe in the next bounce diagram that most of the JAR files are downloaded in a serial fashion, blocking other objects from downloading in the mean time. Notice the delta time displayed on the right column

Bounce Diagram for an Oracle Clinical Transaction

What can be done to improve the performance of this application?

Oracle Clinical uses JAR files to package Java code that will run on the client. It is not uncommon for customized off the shelf applications to increase in code size over time until the code is very bloated.

There are several best practices that developers can follow to reduce the size of JAR files:

1. Rationalize the code – as applications develop over time, multiple classes and sometimes adjacent projects might reference similar libraries and other assets. If not careful those libraries and common assets end up packaged multiple times inside the JAR file, causing it to inflate in size.
2. Minify the code – a quick Google search for “reduce the size of JAR files” will reveal several free tools that can minimize the size of JAR files, usually through eliminating white spaces, comments, shortening variable names, etc.
3. Defer loading – chances are that not all the code in the JAR file is needed for the “connect to DB” transaction, which means that users that only want to perform a small transaction are penalized by the download time of code that will never get executed by them. Deferred loading is a design pattern that simply says “only download assets or code when it is needed” I wrote more on that design pattern in this previous post “http://www.shunra.com/shunrablog/index.php/2009/01/29/content-loading-when-being-lazy-pays-off/”

In summary, as more and more web 2.0 technologies hit main stream, new forms of performance pitfalls may present themselves, requiring us to consider application performance for remote users even for applications that traditionally were considered “remote user friendly” such as web based applications.

How was this Analysis Performed?

This analysis was used by integrating a HP LoadRunner script modeling an Oracle Clinical business user with Shunra VE Analyzer. So as the scripted transactions executed across a virtual WAN (simulated by the Shunra VE) the beginning and end of each transaction were marked by the VE Transaction Manager. At the end of the test, the VE Analyzer had sufficient data to generate the attached reports, enabling us to pin point the root cause of the performance problem. You can learn more on how to set this up by going through Shunra Certified Performance Engineering training, either on site or at the next Shunra University. Learn more about Shunra training here:  http://www.shunra.com/shunra-university-overview.php and here http://www.shunra.com/training-overview.php?keyword=services

Questions, comments, feel free to write me at amichai.lesser at Shunra dot com. Or comment on any of my posts.

BTW, we just created a new Application Performance Management Group on LinkedIn, feel free to look it up and join here  http://www.linkedin.com/groupRegistration?gid=2200667

Best,

Amichai Lesser

In the current economic environment where growth is tough to come by, Business Ethernet Service (aka Carrier Ethernet, Metro Ethernet) is bucking the trend. A recent analyst report confirms the uptick in growth of these services for Business Applications:

Source: Vertical Systems Group, March 2009

Service providers are reaching critical mass in making Business Ethernet services reliable and scalable which makes the cost savings they provide a serious consideration for Enterprise customers. One of the big issues holding back Enterprise deployments has been last-mile access. That issue has also seen significant improvement as carrier Ethernet footprints have grown significantly.

The Metro Ethernet Forum website provides a nice interactive services directory. You click on a global map and drill down to your locations to see which Service Providers offer Ethernet Services. I searched for Shunra’s HQ location in Philadelphia which resulted in 5 providers to choose from. I selected one provider and here’s what I got:

Here’s the link if you want to try it out:

http://metroethernetforum.org/page_loader.php?p_id=310

And here’s a list of leading Business Ethernet Service providers just published by Vertical Systems Group – my company happens to use Cogent with good success.

So, Enterprise-class Ethernet Business Services are increasingly available, they provide a more scalable and flexible service, and they offer significant bandwidth cost savings. The questions that remain:

• How will my existing applications perform over this new infrastructure?

• How much bandwidth should I buy at each location?

• What type of SLA (network latency, packet loss, availability) do I need to pay for to ensure adequate performance of my business-critical applications?

The ability to answer these questions proactively prior to contract negotiations ( and deployment! ) is the key to effective decision-making. This will ensure you unlock the full savings potential of these services and, more importantly, your end users receive a consistent level of service that ensures a successful service migration.

At Shunra we’ve seen cases where increases in network latency of only a few milliseconds can double Transaction Reponse Times (TRT) for some business transactions. We’ve seen cases where increases in network latency of only 15 ms can cause certain applications to fail completely. Applications that use Common Internet File System (CIFS) protocol, such as MS Sharepoint, is an example of an application prone to performance issues due to WAN changes.

The good news: Products are available that allow you to test your application performance in the comfort of your test lab or Data Center prior to service cut-over.  WAN Emulation solutions from companies like Shunra (http://shunra.com/products-overview.php?keyword=products) emulate the behavior of the Business Ethernet Services you’re considering. You can dial the bandwidth limits up and down; you can vary network latency and packet loss; essentially, program in various SLAs and see how your applications perform. You can even do Disaster Recovery testing to recreate automatic fail-over scenarios between service providers.

Here’s an example of an application vulnerable to an increase of only 2.5 ms in WAN latency, and how the problem was identified and resolved prior to cut-over:

http://www.shunra.com/shunrablog/index.php/2009/08/20/data-center-relocation-questions-and-answers-part-2/

Have you migrated some of your offices over to Business Ethernet Service already? Was the migration smooth or painful? Any pearls of wisdom you can offer to others to help them prepare for the change?

In the previous post http://www.shunra.com/shunrablog/index.php/2009/08/14/data-center-relocation-questions-and-answers-part-1/ I shared some of the questions that clients typically ask me during the performance analysis service in a data center relocation project. One of the most popular question, especially at the beginning of the project is “which applications are the most sensitive to network latency?”. Initially I thought of writing a post that categorizes and summarizes those applications, however, after reviewing the (long) list and reading through the analysis, I figured the community could benefit from some of the specific examples. I promise to try and summarize it all at the end.