Business Process Test Plan – Update employee record

However this series of posts is about the day after a data center move. Now that the data center is remote, how does this paradigm shift impact the way we should develop, test, deploy, monitor and troubleshoot applications. I will try to cover as many topics as possible, but the main focus is still going to be around the role application performance management plays in this new paradigm.

I will start by covering the key reasons behind the performance impact that is experienced when applications are hosted in a remote data center? Those reasons are fairly intuitive, but it is important to understand them in depth in order to adequately plan for those new conditions. Two main things impact how applications perform when application servers are hosted in a remote data center vis a vis their application clients :

1. The performance of the network link between the client and the remote data center. This performance is defined by a set of network performance metrics that are application independent (for now we will ignore application aware networks, however the following basic concepts still hold in this scenario as well).

2. The application efficiency, specifically how efficient the application is when transferring data between the client and the remote server (and other tiers if applicable). This is an application attribute (and some time an attribute of a specific business process within the application). These attributes are application specific and are independent of any underlying network.

Let’s start with understanding the network performance metrics. Consider the following scenario:

An application is hosted in a NYC data center, with users in 2 places, some are in a NYC headquarters next to that NYC data center and some are in a remote branch that is located in San Francisco. The question is: “will the application perform the same for both type of users (local users in headquarters and remote users in SF)? In other words will the application be as responsive to the San Francisco user as it is to the NYC user?”

Well the obvious answer is NO, in most cases a NYC user will enjoy a faster more responsive application. What is less obvious is why? What is it about the network that causes remote users to experience a slower application than local users? The rest of this post will cover that question, future posts will address the application specific attributes. Once we cover that we will be ready to examine best practices in building applications for a remote data center.

When I ask this question during my training seminars, I get a variety of answers, many of them are the right ones, but I would like to address one wrong answer that keeps repeating itself for some reason.

Collisions – there is a general conception that collisions are common phenomena on the network which can explain any bad thing that happens to applications. The truth is that collisions are almost a thing of the past (on Enterprise LANs anyhow) and even when they happen they can’t explain why a remote user has a worst experience than a local user as both will experience a similar collision chance since collisions is a phenomenon that happens on local area Ethernet networks. If there are collisions on the Enterprise LAN it usually points to a configuration issue on a network device (like a duplex miss-match) but is still unrelated to the answer to our question.

Now to the right answers to the question, what is it about the Wide Area Network that causes applications to slow down:

There are 5 key conditions that predominately exist on Wide Area Networks and impact application performance, each in their own way:

1. Network Latency – the time it takes a packet to traverse from a source to the destination across the network, measured in milliseconds [msec]. A typical WAN link will introduce latency in the range of 10msec – 500 msec.

2. Bandwidth constraints – how fast can data be processed by the network link, measured in bits per second [bps, Kbps, Mbps, Gbps]

3. Bandwidth utilization (background traffic) – the percentage of bandwidth that is utilized by traffic that already exists on the link (background traffic).

4. Jitter – the deviation of the inter packet gap of sequential packets across a network link, it is a result of the deviation of the network latency and is sometimes used interchangeably with that standard deviation, measured in milliseconds [msec].

5. Packet Loss – the chance to drop a packet across an end to end network link, measured in %. Sometimes presented as the inverse metric called packet delivery rate.

The above are called network impairments, you can click on each one of the links to learn more about them and their causes.

Network impairments are performance conditions that inhibit the flow of data across a network. Each impairment type has an impact on the performance of business applications and network services. Some applications may be very sensitive to network impairments and some may be almost network agnostic. Sorting applications based on their network sensitivity is one of the important steps in performance engineering

In the next post we will discuss how application design can impact performance across the network. But in the mean time I would like to introduce a question for the group:

“We identified network latency as one of the key reasons that impact application performance; we also said that a typical WAN link will introduce 10 – 500 msec of latency. The question is, why does network latency have a big impact on application performance? surely a user doesn’t notice an increase of a few msec in response time, even 500 msec = ½ second goes by in a flinch. So think about it and let me know what you found based on your experience, why does network latency have such a big impact on application performance?”

Until next time,

Amichai Lesser

Network capacity, or bandwidth, is the maximal number of bits a network connection or interface can carry at a given period of time. It is measured in bps (bits-per-second), Kbps, Mbps or Gbps. The greater the bandwidth, the greater the number of concurrent application sessions the link can serve (for a given transaction) and the greater the rate that each application session can consume from the network.

Bandwidth Utilization:

Bandwidth utilization is a measure of how much of the link’s maximum data rate is being used. Consider an intuitive notion of utilization – it may start by picturing the WAN circuit as a pipe of a certain diameter and then imagining that it is partly filled with something we call traffic. Bandwidth utilization is a factor of the number of concurrent application sessions across the link and the average rate used by each session. For example, if a T1 link (1544 Kbps) serves an average of 20 concurrent application sessions and each session uses 50 Kbps on average each way then we would say that the link is 64.7% utilized ((50 Kbps * 20 sessions)/1544 kbps = 64.7%).

Starting at 70% BU, network performance starts to degrade, 80% badly degraded, and 90+% “Flooded”. Smaller pipes, are subject to easier flooding, which will cause significant increased latency and jitter

Jitter:

Network jitter is the variation in the arrival rate of packets (more specifically, the inter packet gap of subsequent data packets as they arrive over a network). For most types of data applications large gap variations between arrival times (high jitter) are acceptable. For voice or video applications, relatively small jitter can cause perceptible disturbances in the recreated voice or video at the receiving end.
Because Wide Area Networks over IP generally rely on shared access to the network and can have many asynchronous sessions all sharing the same network they tend to introduce relatively high jitter.

Jitter can be reduced with buffers. Buffering some incoming packets and then outputting them at a more regular rate reduces jitter. This is fine for applications that are not interactive, such as streaming music or videos. For interactive applications such as VoIP conversations or video conferencing, this scheme will add network latency, which reduces the quality of the conversation.

• When networks get congested over a long period of time the router buffers get saturated which at some point leads to a situation where there is no room to store more pakets in the buffers, so new arriving packets get dropped
• Some routers implement a mechanism called random early detection, since IP doesn’t provide any explicit way to indicate that the network is congested, routers use loss for that purpose. Such loss is a way of communicating back to the users the need to scale back the offered load on the network. RED is becoming less common as new QoS techniques are used to implement rate control.
• Communications across wireless and cellular networks may encounter packet loss due to interference or a weak signal
• Hardware and cable errors are a common cause of packet loss

From the network perspective, loss can be categorized as random loss or loss due to congestion. The average packet loss rate for a network connection gives an overall sense of the quality of the connection. A connection with less than 1 percent average packet loss is considered a decent connection. But average loss doesn’t tell the whole story. There is importance to the type, or pattern, of packet loss. There are at least two kinds of packet loss that should be considered: ‘Random’ loss and ‘Burst’ loss. To explain the difference between them, let’s suppose we are trying to run 2 Voice over IP conversation over 2 links that have an average of 1 percent packet loss. Call A loses one packet in every 100 packets over the entire call (random loss) while Call B loses 100 packets in two incidents – at the beginning and the end of the call (burst loss). Which call would you rather have? That’s why it is important to consider not just the average packet loss but also the type of loss and information on any bursts of packet loss over time.

Network Latency/Network Delay

One-way network latency is defined as the amount of time it takes for a packet to traverse a particular network path, from a device that created the packet to the destination device. This is also known as end-to-end latency.

Network delay is composed of propagation, processing, serialization, and queuing delay.

Propagation delay is the time it takes the physical signal to traverse the path. This delay is usually fairly constant if there are no route changes (it’s more constant if the end-points are static connected via optical fiber and has more variation if one of the end-points is a moving airplane or satellite). Propagation delay is the result of pure physics where as

t(msec) = d(distance)/(2/3rds of the speed of light)

Processing and Serialization delay is the aggregate time it takes each hop on the route to process and transmit the packet. Depending on bit-rate (bandwidth) it may be a significant portion of overall delay or may be negligible. In modern networks where high bit-rate connections are becoming the norm, serialization delay is becoming more and more negligible. In any case, for given packet size and path, serialization delay is constant (unless affected by hardware compression) and varies during route changes and as packet length changes. Processing time in each hop depends on the services that this hop provides (bridging, routing, encryption, compression, tunneling, etc.) and on the speed of the device itself but generally in modern networks this is a number of a much lower order of magnitude than the rest of the latency factors.

Queuing delay is the time a packet spends in router queues. This time depends naturally on queue lengths: for an unloaded network it would be negligible; for a network that is heavily congested it could considerably contribute to the end to end delay. It is the most variable delay component in a typical modern network. Queuing delay changes based on the burstiness of the traffic, since a burst of traffic creates an increase in the queue depth. Queuing delay is the main reason for jitter, which is caused by the variation of latency over time.

Queuing delay is the time packets wait inside the devices’ queues.

Latency and bandwidth together determine the “speed” of a connection.

Latency can increase as the bandwidth utilization and traffic load changes but it will never go below the propagation delay (simple physics). As load increases, it is possible that latency will increase since buffers may begin to populate on the path between the sender and receiver.

All in all, network latency is the combination of all factors described above. The impact of the network delay on application performance is a topic of a future post.

Best,

Amichai