On February 17, the Chairman of the FCC (Federal Communication Commission), Julius Genachowski, proposed a decade long program to provide a minimum of 100 MBPS link access to every household in the US.  The Chairman said the FCC plan would set “ambitious but achievable goals” in remarks to the National Association of Regulatory Utility Commissioners conference.

Genachowski said speedier Internet service would help create jobs and economic growth.

Data shows that about 64 percent of U.S. households used a high-speed Internet service in 2009, the Commerce Department said on Tuesday. That is a 25 percent increase from 51 percent two years earlier.

Some internet providers already called this program “unrealistic”, given the current estimate of the average access speed at less than 4 MBPS. However, Verizon, the third-largest provider, and one that has a more advanced network than many competitors, said it has completed successful trials of 100 Mbps and higher through its fiber-optic FiOS network. Today, Verizon offers programs with 50 MBPS access, at a premium.

The FCC announcement should not come as s a surprise, given Google announcement from the beginning of February that it will pilot a deployment of 100 MBPS links to certain communities. Not surprising, the first newspaper to report this development was the Korean Times, the country with the most widely available high speed Internet service for home consumers.

Why 100 MBPS matters? Obviously, it will open the possibility for stay home Moms to join the work force by becoming part of corporate networks; will open great opportunities for home-schooling connected to on-line interactive schools and colleges and open a whole new world of entertainment and leisure activities, like interactive TV, collaborative gaming, real-time social interaction and more.

The key message is very clear – BIG pipes and FASTER applications are on the way.  We need to ensure this new environment can handle our demands.  Shunra technologies allow you to emulate the network environment, introducing real world impairments to test those media rich applications in a controlled lab environment. Shunra has been providing this capability for more than 10 years and now, more than ever, is the time to prepare.

Due to current events the timelines for testing and certification had become critical and the contractor responsible for the project was under enormous pressure to complete this engagement quickly. In conversations with the contractor prior to arriving onsite, I asked a series of questions intended to reveal the lab environment and methods of test traffic generation.  It became clear that some big action items that needed to done in order to achieve the requisite environment and deliver all of the desired results was still unfinished.

Some of these dependencies included….

• Reaching consensus among all the constituent groups as to what would constitute the physical test network as well as the specific impairments to define the characteristics of the circuits to be recreated within the Shunra scenarios.
• Confirmation of existing test scripts of the application for both the QTP and HP Performance Center traffic generation tools.
• Communication between the contractor (Project Manager), network equipment vendors, QTP and Performance Center engineers and Shunra to ensure that each step in the process was coordinated to proceed in an orderly sequence.

As these conversations took place it became evident that the project manager had never been exposed to the complexities associated with the range of technologies and the number of disparate groups required to compete the task. I cautioned him that to guarantee timely completion of all these steps a meeting should be scheduled to include all the participants prior to arriving onsite. Shunra offered our Professional Services to act as the Project Manager to broker to meeting and to coordinate the project execution. The key benefit to ensure that all parties would be fully prepared to execute their tasks when needed. This offer was declined. The meeting never was organized and the timelines of the engagement suffered accordingly.

Once onsite the network engineers and Shunra spent considerable time connecting and configuring the test lab to allow communication between all of the known test devices. After completion a member of the Performance Center group popped into the lab and informed us that some traffic needed to be initiated from end users on another network in a different building. This change mandated a complete reconfiguration effectively wasting hours of work. Only after completion of this task did the contractor learn that no test scripts had been created to exercise the application. Script creation could potentially add days or weeks to the completion of the project.

Meanwhile the representative from the Agency requesting this evaluation would appear every ten minutes asking whether the job was done. As the sweat poured off the brow of my contractor I could only shake my head and think…..Told you so…..

The lesson to be learned is that Shunra has a cadre of experienced professionals that can deal with the complexities of these engagements. In a proactive systemic process we can anticipate all contingencies and implement best practices honed over years of direct experience. With this expertise from our Professional Services group the scoping, timing, and delivery of test results are guaranteed.

The contractor is still in stasis awaiting his test scripts and that Federal representative is still poking his head asking for the results.

Oh well………

Many ISPs are already planning ahead by purchasing blocks of IPv6 addresses to ensure they’re prepared for the next generation of the Internet – especially since industry experts are claiming that IPv4 addresses will be diminished by 2012. Yet, despite all of the talk about IPv4 running out, is it really critical that organizations make the transition to IPv6 in the near term? According to the International Data Corporation (IDC), by the year 2012 there will be approximately 1.9 billion users accessing the Internet globally – a staggering figure considering this would account for nearly 30% of the world’s total population! Additionally, many analysts are claiming that by 2012 there will actually be more users accessing the Internet from mobile devices than computers.

Consequently, this is fundamentally changing how the Internet must scale to satisfy this increased demand.

Even though the IPv4 address space is rapidly being diminished, it seems that this is the only real driver that is prompting the IPv6 movement. Thus, it begs the question: what are the real benefits of migrating to IPv6 other than resolving the address space issue? Engineers from Internet Society (ISOC) conducted a survey about IPv6 and found that businesses are still skeptical about performing an IPv6 migration because there are no concrete business drivers that would entice an organization to make the move. What’s interesting to point out as well is that the survey found the main driver for IPv6 is almost entirely based on customer demand, not IPv4 depletion. Nevertheless, more and more businesses are gearing up for IPv6 because they believe that it’s the next major development in the Internet. This survey also found that even though IPv6 provides several advantages over IPv4; namely features like auto-configuration, mobility support, and built-in security features, more than half of the survey respondents indicated the main reason was attributed to address space.

Another growing concern for organizations is being able to support hybrid IPv4/IPv6 networks. The IETF has admitted that not making IPv6 backwards compatible with IPv4 was a significant oversight on their part and could complicate IPv6 rollouts. To this end, organizations need a way to test and deploy mobile applications, business services and various network configurations so as to ensure they can support IPv6 on the network. Fortunately, Shunra provides both software and hardware testing solutions that fully support IPv4 and IPv6. This enables network teams to fully model a hybrid network with both IP versions and all of the pertinent network characteristics that would define their network environment. This type of testing can even be leveraged by way of Shunra’s Professional Services team as a managed turnkey service – which many customers utilize because they typically derive significant value from this approach with a minimal upfront investment.

In closing, there seems to be a consensus amongst many industry experts that even though there isn’t a compelling business case to move to IPv6, this movement will inevitably occur and more importantly, if you’re not prepared the business could pay dire consequences.

Recently, I visited a company that develops and distributes software for Independent School District’s across USA. Their main product, originally developed for challenged kids, centers around dynamic computerized teaching and testing programs. This product proved to be so successful among this population that the company adapted it for other kids as well. Initially the product was designed to work in the classroom, hence a very simple architecture: 1 PC working as a learning manager.  This 1 PC managed the progress of individual students, as well as received and captured data back to computers across the LAN. However, because of such a high demand, the company decided to host a solution for multiple customers from a central location, specifically – a Cloud hosted server.

Since the original design was created with only a classroom setting in mind, testing and development only occurred over LAN. But now, with accessibility from different regions, including links from APAC countries to US, consideration of the WAN was critical. This required the need to test their solution in a new environment and see if any problems exist.  By analyzing this information, they gain a better understanding of how to fix any problems. With this challenge, the company needed a WAN emulator solution that would allow them to re-create the complex topology of the application served over WAN.

Once tested, they were able to easily reproduce the problems reported to the QA from the field. One issue that was noted – disconnections without the ability to reconnect, was leading to a loss of data and historical progress of the students in the program.  One of the key tools is tracking “task lists.”  One of the main advantages of the system is the ability to produce these new “task lists” for each student, based on historical data retrieved.  The new tasks matched the student’s progress in learning the subjects and fulfilling the tasks.

One other obstacle the company faced dealt with the USD security measures.  Since product was originally operating on a LAN environment, the security of outside access was a concern. As a result, certain ports were blocked and the company needed to understand how those ports will be affected from blocks in different directions of the link. Shunra VE WAN Emulator was able to reproduce such conditions by filtering application traffic based on protocol and ports, as well as IP addresses and blocking them in one direction – the problems were reproduced instantly. Now QA no longer has to rely on the reports generated from the field to try and fix the problems “in the dark”…  They now have a nice “flash light” to expose those reasons for the problem.

The impact of a broken link or just a simple delay in logging in can significantly impact students, especially the challenged ones.  Picture a child struggling to connect to the program, no sounds, no pictures… instead sitting with headphones and seeing on a screen: “Connecting…”  An unnerving experience for all of us, and even less for them (IMHO).  And what if after working for 20-30 minutes on a progressive task – they have to start over because computer lost all memory of what they were doing until now.

That sounds like a lot of clouds!  What we are simply trying to understand is where and how to deploy your application in the cloud, so it will perform well for your customers.

If you take the case of deploying an online store into the Amazon EC2 cloud, there are many things to consider that will impact your customer’s experience.  Two of the most important are where to deploy what, and how your store will function over the Internet.  To compound the issue, where you physically deploy machines within EC2 is not clear.  Amazon goes to great lengths to avoid you knowing where their zones are.

Fortunately, you cannot only find solutions to these problems, but with cloud computing the process gets a lot easier.  To break the problems down into a logical flow, you’ll first need to know the demographics of your customers.  Where are they?  How will they access the store?  What needs to happen for them to have a positive experience at your store?  If you can answer these questions, you’ll likely come up with some numbers that place percentages of your users in different regions around your target market (US, EMEA, global, etc.).  Further, you’ll know the top two browsers they’ll use and with what kind of connections they’ll access the internet (e.g. Firefox with a 768/256kbps DSL, or IE8 with dial-up – yes people still use dial-up!).  Finally, you’ll have some way to quantify a “positive user experience”.  That usually involves consistency (does it work?) and speed (how fast?).

Using the information above and HP LoadRunner with WAN emulation, you can quickly build a to-scale test in your cloud-based lab and understand what would happen if you hosted your application all in one place and serviced your entire customer base.  Odds are that the test will not meet all of your performance goals the first time you run it.  Most likely, you will need to adjust some things in your store (e.g. reduce chattiness and image resolution, optimize Web 2.0 use) and then begin the discussion about where to deploy what.  Can you keep all of your database servers in one place?  If you have to split your database servers, how will that affect performance?  Will you need to deploy web servers in every zone, will you be okay with one in each region, or will you need something in-between?

Again, not only can answers to these questions actually be answered, but by using cloud-based testing you can arrive at valid conclusions faster than ever.  To understand the network impact of deploying machines in different zones, network performance measurement tools like VE Network Catcher can run in each of the zones to measure network performance between zones and to customer representative endpoints.  Importing that data directly into HP LoadRunner with WAN emulation and using the flexible cloud environment, you can rapidly form and either dismiss or validate hypotheses about how your application should best be deployed.

The benefit of testing in this way is that you can experiment with many different configurations in rapid succession to find the one or few that are most appropriate for your needs.  Trial and error with a live application is not advisable.  And building a test lab with physical machines to simulate the same thing is not only significantly more expensive, but a lot slower process.  As I pointed out in my previous blog, the same ROI formula that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.

Now you know what I think.  What do you think??

Business Process Test Plan – Update employee record

However this series of posts is about the day after a data center move. Now that the data center is remote, how does this paradigm shift impact the way we should develop, test, deploy, monitor and troubleshoot applications. I will try to cover as many topics as possible, but the main focus is still going to be around the role application performance management plays in this new paradigm.

I will start by covering the key reasons behind the performance impact that is experienced when applications are hosted in a remote data center? Those reasons are fairly intuitive, but it is important to understand them in depth in order to adequately plan for those new conditions. Two main things impact how applications perform when application servers are hosted in a remote data center vis a vis their application clients :

1. The performance of the network link between the client and the remote data center. This performance is defined by a set of network performance metrics that are application independent (for now we will ignore application aware networks, however the following basic concepts still hold in this scenario as well).

2. The application efficiency, specifically how efficient the application is when transferring data between the client and the remote server (and other tiers if applicable). This is an application attribute (and some time an attribute of a specific business process within the application). These attributes are application specific and are independent of any underlying network.

Let’s start with understanding the network performance metrics. Consider the following scenario:

An application is hosted in a NYC data center, with users in 2 places, some are in a NYC headquarters next to that NYC data center and some are in a remote branch that is located in San Francisco. The question is: “will the application perform the same for both type of users (local users in headquarters and remote users in SF)? In other words will the application be as responsive to the San Francisco user as it is to the NYC user?”

Well the obvious answer is NO, in most cases a NYC user will enjoy a faster more responsive application. What is less obvious is why? What is it about the network that causes remote users to experience a slower application than local users? The rest of this post will cover that question, future posts will address the application specific attributes. Once we cover that we will be ready to examine best practices in building applications for a remote data center.

When I ask this question during my training seminars, I get a variety of answers, many of them are the right ones, but I would like to address one wrong answer that keeps repeating itself for some reason.

Collisions – there is a general conception that collisions are common phenomena on the network which can explain any bad thing that happens to applications. The truth is that collisions are almost a thing of the past (on Enterprise LANs anyhow) and even when they happen they can’t explain why a remote user has a worst experience than a local user as both will experience a similar collision chance since collisions is a phenomenon that happens on local area Ethernet networks. If there are collisions on the Enterprise LAN it usually points to a configuration issue on a network device (like a duplex miss-match) but is still unrelated to the answer to our question.

Now to the right answers to the question, what is it about the Wide Area Network that causes applications to slow down:

There are 5 key conditions that predominately exist on Wide Area Networks and impact application performance, each in their own way:

1. Network Latency – the time it takes a packet to traverse from a source to the destination across the network, measured in milliseconds [msec]. A typical WAN link will introduce latency in the range of 10msec – 500 msec.

2. Bandwidth constraints – how fast can data be processed by the network link, measured in bits per second [bps, Kbps, Mbps, Gbps]

3. Bandwidth utilization (background traffic) – the percentage of bandwidth that is utilized by traffic that already exists on the link (background traffic).

4. Jitter – the deviation of the inter packet gap of sequential packets across a network link, it is a result of the deviation of the network latency and is sometimes used interchangeably with that standard deviation, measured in milliseconds [msec].

5. Packet Loss – the chance to drop a packet across an end to end network link, measured in %. Sometimes presented as the inverse metric called packet delivery rate.

The above are called network impairments, you can click on each one of the links to learn more about them and their causes.

Network impairments are performance conditions that inhibit the flow of data across a network. Each impairment type has an impact on the performance of business applications and network services. Some applications may be very sensitive to network impairments and some may be almost network agnostic. Sorting applications based on their network sensitivity is one of the important steps in performance engineering

In the next post we will discuss how application design can impact performance across the network. But in the mean time I would like to introduce a question for the group:

“We identified network latency as one of the key reasons that impact application performance; we also said that a typical WAN link will introduce 10 – 500 msec of latency. The question is, why does network latency have a big impact on application performance? surely a user doesn’t notice an increase of a few msec in response time, even 500 msec = ½ second goes by in a flinch. So think about it and let me know what you found based on your experience, why does network latency have such a big impact on application performance?”

Until next time,

Amichai Lesser

In alignment with global trends, more and more clinical trials take place outside of the United States, while the documentation and analysis is done in the US for submission to the FDA. So it is not out of the ordinary to find an Oracle Clinical user in Eastern Europe or in China submitting data to an Oracle Clinical Server hosted in the united states.

Unfortunately, as commonly found with an off the shelf software package that gets customized over time, this type of application can easily morph into a performance challenged application, especially for remote users. The following analysis shows an example of an Oracle Clinical transaction that completes in 8 seconds for local users, while extending to over a minute when users in E. Europe tried to use it.

Transaction Response Time Analysis for an Oracle Clinical Application Accessed by Local Users Vs. Remote Users in E. Europe

Obviously, that big jump in response time was a great cause for concern with my client’s management. Which is how I got the privilege to be asked to analyze the root cause of the poor performance.

The first thing I noticed was that one particular transaction (Connect to DB) had an extremely high jump in TRT from 8 seconds locally to over a minute when accessed remotely, so I focused on that one first.

Looking at the network fingerprint of the transaction you can observe the following things:

The transaction size (amount of data downloaded from the server required to complete the transaction) is extremely high (over 5 MB).

The transaction generates 34 HTTP calls. Some of those calls take disproportionally longer time than others (notice the Get jar file calls that each exceed 19 seconds marked below, click on the image for a larger view)

Oracle Clinical, Transaction Analysis for "Connect to DB"

When inspecting those JAR files, we saw that they were very heavy in size, over 1 MB each which is pretty big for basically a collection of Java classes that are zipped into a JAR file.

You can also observe in the next bounce diagram that most of the JAR files are downloaded in a serial fashion, blocking other objects from downloading in the mean time. Notice the delta time displayed on the right column

Bounce Diagram for an Oracle Clinical Transaction

What can be done to improve the performance of this application?

Oracle Clinical uses JAR files to package Java code that will run on the client. It is not uncommon for customized off the shelf applications to increase in code size over time until the code is very bloated.

There are several best practices that developers can follow to reduce the size of JAR files:

1. Rationalize the code – as applications develop over time, multiple classes and sometimes adjacent projects might reference similar libraries and other assets. If not careful those libraries and common assets end up packaged multiple times inside the JAR file, causing it to inflate in size.
2. Minify the code – a quick Google search for “reduce the size of JAR files” will reveal several free tools that can minimize the size of JAR files, usually through eliminating white spaces, comments, shortening variable names, etc.
3. Defer loading – chances are that not all the code in the JAR file is needed for the “connect to DB” transaction, which means that users that only want to perform a small transaction are penalized by the download time of code that will never get executed by them. Deferred loading is a design pattern that simply says “only download assets or code when it is needed” I wrote more on that design pattern in this previous post “http://www.shunra.com/shunrablog/index.php/2009/01/29/content-loading-when-being-lazy-pays-off/”

In summary, as more and more web 2.0 technologies hit main stream, new forms of performance pitfalls may present themselves, requiring us to consider application performance for remote users even for applications that traditionally were considered “remote user friendly” such as web based applications.

How was this Analysis Performed?

This analysis was used by integrating a HP LoadRunner script modeling an Oracle Clinical business user with Shunra VE Analyzer. So as the scripted transactions executed across a virtual WAN (simulated by the Shunra VE) the beginning and end of each transaction were marked by the VE Transaction Manager. At the end of the test, the VE Analyzer had sufficient data to generate the attached reports, enabling us to pin point the root cause of the performance problem. You can learn more on how to set this up by going through Shunra Certified Performance Engineering training, either on site or at the next Shunra University. Learn more about Shunra training here:  http://www.shunra.com/shunra-university-overview.php and here http://www.shunra.com/training-overview.php?keyword=services

Questions, comments, feel free to write me at amichai.lesser at Shunra dot com. Or comment on any of my posts.

BTW, we just created a new Application Performance Management Group on LinkedIn, feel free to look it up and join here  http://www.linkedin.com/groupRegistration?gid=2200667

Best,

Amichai Lesser

Which is why I was glad to stumble upon Shamus McGillicuddy article in www.searchenterpriseWAN.com titled “WAN engineers prepare networks as cloud computing adoption rises” that can be found here.

There are some good observations in the article, I especially appreciate the comment on how application architecture best practices such as minimizing application turns become key when developing applications for the Cloud as well as testing applications for network latency becomes critical as the enterprise is migrating more and more applications to the Cloud.

Many questions still remain though and the impact of Cloud computing on performance engineering is still to be determined. Some of the questions that are on my mind are: (and feel free to chime in with more questions or attempts at answers)

1. How do you manage the performance of an application in the Cloud?

2. What changes if any are needed to the service deployment model and version upgrades?

3. How do you conduct a performance test for an application or a service in the Cloud (internal clouds and external clouds)?

4. How do you troubleshoot a performance issue in the Cloud?

Something to think about…

Best,

Amichai Lesser

In the current economic environment where growth is tough to come by, Business Ethernet Service (aka Carrier Ethernet, Metro Ethernet) is bucking the trend. A recent analyst report confirms the uptick in growth of these services for Business Applications:

Source: Vertical Systems Group, March 2009

Service providers are reaching critical mass in making Business Ethernet services reliable and scalable which makes the cost savings they provide a serious consideration for Enterprise customers. One of the big issues holding back Enterprise deployments has been last-mile access. That issue has also seen significant improvement as carrier Ethernet footprints have grown significantly.

The Metro Ethernet Forum website provides a nice interactive services directory. You click on a global map and drill down to your locations to see which Service Providers offer Ethernet Services. I searched for Shunra’s HQ location in Philadelphia which resulted in 5 providers to choose from. I selected one provider and here’s what I got:

Here’s the link if you want to try it out:

http://metroethernetforum.org/page_loader.php?p_id=310

And here’s a list of leading Business Ethernet Service providers just published by Vertical Systems Group – my company happens to use Cogent with good success.

So, Enterprise-class Ethernet Business Services are increasingly available, they provide a more scalable and flexible service, and they offer significant bandwidth cost savings. The questions that remain:

• How will my existing applications perform over this new infrastructure?

• How much bandwidth should I buy at each location?

• What type of SLA (network latency, packet loss, availability) do I need to pay for to ensure adequate performance of my business-critical applications?

The ability to answer these questions proactively prior to contract negotiations ( and deployment! ) is the key to effective decision-making. This will ensure you unlock the full savings potential of these services and, more importantly, your end users receive a consistent level of service that ensures a successful service migration.

At Shunra we’ve seen cases where increases in network latency of only a few milliseconds can double Transaction Reponse Times (TRT) for some business transactions. We’ve seen cases where increases in network latency of only 15 ms can cause certain applications to fail completely. Applications that use Common Internet File System (CIFS) protocol, such as MS Sharepoint, is an example of an application prone to performance issues due to WAN changes.

The good news: Products are available that allow you to test your application performance in the comfort of your test lab or Data Center prior to service cut-over.  WAN Emulation solutions from companies like Shunra (http://shunra.com/products-overview.php?keyword=products) emulate the behavior of the Business Ethernet Services you’re considering. You can dial the bandwidth limits up and down; you can vary network latency and packet loss; essentially, program in various SLAs and see how your applications perform. You can even do Disaster Recovery testing to recreate automatic fail-over scenarios between service providers.

Here’s an example of an application vulnerable to an increase of only 2.5 ms in WAN latency, and how the problem was identified and resolved prior to cut-over:

http://www.shunra.com/shunrablog/index.php/2009/08/20/data-center-relocation-questions-and-answers-part-2/

Have you migrated some of your offices over to Business Ethernet Service already? Was the migration smooth or painful? Any pearls of wisdom you can offer to others to help them prepare for the change?