What resonated well with me was a quote from Jim taken from “Alice in Wonderland” – “If you don’t know where you need to get to, then it doesn’t really matter which way you go”.

The full article is available here: http://www.networkworld.com/newsletters/frame/2011/013111wan1.html?page=1

Keep an eye out for additional articles by Jim Metzler around the Application Performance Engineering process in Network World, which should come out in the following weeks.

Talk to you soon,

Amichai Lesser

Losing site of the goal

What happens when IT departments lose site of the performance engineering goal? (Reminder in short the goal is to improve the end user’s quality of experience and productivity, while maintaining system costs within budget).

Well what happens is that each department gets lost in its own tactical goal:

1. The Capacity planning team focuses on efficient and accurate hardware provisioning
2. The load testing team focuses on test coverage and scale requirements
3. The network engineering team focuses on the speed and capacity of the pipes
4. The data base team focuses on the performance of the data bases
5. The server team focuses on the performance of the backend servers
6. Desktop team is focusing on the performance of the desktop clients
7.  …

What the organization ends up with is a set of local optimums, but in many cases those local optimums don’t amount to an optimal system. What’s missing in the above list is at least one department that is responsible for meeting the goal, it is very rare to find a team that oversees the end to end responsiveness and performance of the application across all its components from the end user’s perspective. It is even harder to find a team that is held accountable to end user performance.

But is it wrong for each team to improve its domain and make sure it is optimal? Well the counter intuitive answer is yes, it is wrong and for the following reasons:

Focusing on the wrong bottlenecks

Let’s consider the following transaction as an example: this transaction generates a time sheet report for global employees. This transaction is served by a client (web browser with java widgets) a few web servers behind a load balancer, a few application servers and a data base server. Now lets see what happens if there are performance issues with this transaction. Naturally each team will spend time in improving its own domain, so the data base team may index the employee’s data base to reduce the data base response time in half, the server team adds more web servers behind the load balancer to increase the application’s scalability and the network team adds more bandwidth to the data center router. All these steps sound like they should help, no? Well the realistic answer is that in some cases none of these steps help, in fact 3 negative things happen here:

1. The teams spent time and money on the wrong bottlenecks
2. The real bottleneck is still out there
3. Increasing the speed of none bottleneck components places more strain on the real bottleneck, slowing things down even further

In future posts I will give specific examples of several problems that can not be addressed in the realm of one IT department. Those problems usually result from interdependencies between the different systems (servers, networks and data bases). It takes a holistic and multi-disciplinary process to find the right bottleneck let along find a solution for the problem. It may sound complicated, but the concept is quite simple, when dealing with performance, it does little good to focus on local optimums, any optimization effort that is not spent on the actual bottleneck is counter productive and a waste of IT resources and money. Remember, the goal is to improve end user response time at the desktop, not optimize a specific component that is part of a bigger system.

Even though the concept is simple, the solution isn’t always as simple. In future posts, I will offer practical ways to find application performance bottlenecks, I have used them in many engagements and they haven’t failed me yet. But before we can talk about the solutions, it is important to understand the problems so in the next several posts we will cover a few basic performance engineering concepts.

Talk to you soon…

It’s a simple statement, but a powerful one that will change the existing application lifecycle management paradigm. At Shunra, we have been working hard, together with our partners, at developing and defining a new performance management model that completes the existing APM  frameworks. This post is a first in a series of posts that describe this model and the motivation behind it. In this first post, I will present the reasons why now, more than ever, such a model is needed – why the time is right. Future posts will dive deeper into the definition and building blocks of the application performance engineering model, explore where this model is applicable and explain the benefits it promises.

Application performance is still a problem.

Despite increased investment in application performance management and monitoring solutions, applications in production continue to fail to meet critical performance requirements. These failures create productivity loss for end users who cannot quickly accomplish the tasks the application was designed to facilitate. These failures result in lost revenue from eCommerce and other consumer services abandonment. And, these failures increase remediation costs as time and resources are required to isolate, diagnose and resolve performance problems in production.

Why now?

Application performance is still a problem – this is not a new concept, but it bears repeating. I could have written that sentence 5 or 10 years ago, and it would have held true then, just as it does now. So, why is the time right (or better said, why is the time right now) for a new and better approach to application performance management?

Today’s ever more complex enterprise environment and increasingly distributed application architectures (and user populations) are making the challenge of effectively managing application performance more difficult. New performance risks are arising from three Enterprise IT waves that are converging upon us today:

1)            The infrastructure paradigm shift – the economic downturn has accelerated the move towards data center centralization including, on an accelerating basis, Cloud migrations and associated cost benefits. Enterprise IT is expected to manage these complex projects to realize the promised savings, without impacting performance or end user productivity.

2)            The technology paradigm shift– several new technologies are redefining the way data is delivered to users including virtualization of servers and desktops, unified communications and Web 2.0.  As Web 2.0 technologies are rapidly being adopted in the enterprise, new smart browser-based capabilities are being introduced which blur the lines between where data resides and how and when it will be transferred across the network. In addition, and growing at an even more remarkable pace, is the mobile revolution. Organizations are increasingly challenged to deliver consumer-like applications, capable of handling enterprise grade data volumes, to an ever more demanding and mobile workforce.

3)            The reality of the “Virtual Remote Office” – the virtual office which was a forward looking concept just 10 years ago, is a reality for an increasingly growing workforce. In addition, the scope of the “virtual office” is expanding beyond the residential “home office” to include trains, planes, coffee shops and hotels. Today’s workforce is expected to be productive anywhere they can access a network connection, placing a tremendous dependency on the networks for how well applications will perform.

This “perfect storm” or confluence of Enterprise IT waves makes it difficult to deploy and gain confidence in the performance of applications. As application performance continues to become more dependent on the network, confidence in application performance will continue to decrease. This necessitates increased focus on how applications behave across a variety of networks including WAN, Internet, Mobile and Cloud.

What do companies do today and why isn’t it enough?

Traditional approaches to performance management include Application Performance Monitoring. This approach, however, offers only partial relief. Performance monitoring is a solid approach to discovering and diagnosing problems that result from the interaction of applications and networks in production. That is the key – performance monitoring is a reactive solution to discovering issues in production. And, because no application is perfect, performance monitoring is critical to performance management.

Discovering and fixing problems in production, though often required, is costly.  Remediation can require re-engineering or significant infrastructure investments that can be difficult to plan in a firefighting mode. There are time pressures and cost sensitivity issues since end users are being affected and application development and testing costs have already been allocated and incurred. Performance monitoring solutions help improve MTTR, but they cannot offset the time and resource costs associated with poor performance in production and poor end user experience.

What is missing?

More forward-thinking companies are moving away from the break-firefight-fix cycle and are beginning to augment their APM strategy with a more proactive, pre-production approach to performance management. This new approach leverages Application Performance Engineering best practices to reduce performance remediation costs, increase monetization and ensure end-user productivity. To be successful, this approach relies on a precise replica of the production environment for identifying poor performing business-process-steps in advance of deployment. This does not necessarily mean recreating the network hardware in the QA lab, but enabling a precise emulation of the hardware and production network conditions or impairments. With an APE-enabled test lab, these companies can effectively and reliably investigate performance optimization solutions, be confident in their network and performance decisions, and validate real-world application response times in advance of deploying to an end user community across networks – WAN, Web, Mobile and Cloud.

In my next post, I will dive deeper into defining the goal of Application Performance Engineering and the associated building blocks that make up this model.

In the mean time, I am looking for your feedback:

• What else is missing in how application performance is being managed?
• How do you view the application performance engineering goal?
• What do you think needs to be in the application performance engineering model?

The content in this post has been co-developed with Bill Varga, the COO of Shunra and Marty Brandwin, Director of Product Marketing at Shunra. Many parts in it were influenced by thoughts and comments from most of the executive team at Shunra and our partner community.

Looking forward to your comments,

Amichai Lesser

Below hear from Shunra customer Chris Thompson, about his thoughts on leveraging Shunra’s VE appliance within his organization to demonstrate that new applications will function well on the network, after testing the performance using Shunra:

The FCC’s tool lets you measure download and upload speeds, as well as latency and jitter (the last two are important for video and voice applications).  Nobody talks about latency and jitter and that is a shame.  For high quality video conferencing, less than 20ms of jitter is best.  Latency less than 30 ms is excellent. 

If you want to test your Internet connection, you must choose what type of consumer are you, home or commercial, including street address and zip code.

The test include upload and download speeds and latency and jitter to the service provider. Results are comparable to speedtest.net, a site that also utilizes OOKLA technology.

The FCC site also provides an Iphone application to measure network performance on a WI-FI network using a smart phone.

If your Iphone is connected to a WI-FI network, your results will reflect the speed of your broadband connection. If you disable Wi-Fi and then perform the test, the results will reflect the performance on the 3G network provided by At&T.

I believe this to be a great step to forcing greater transparency in the Internet access space and eventually will force Service providers to increase speed and lower costs.

On February 17, the Chairman of the FCC (Federal Communication Commission), Julius Genachowski, proposed a decade long program to provide a minimum of 100 MBPS link access to every household in the US.  The Chairman said the FCC plan would set “ambitious but achievable goals” in remarks to the National Association of Regulatory Utility Commissioners conference.

Genachowski said speedier Internet service would help create jobs and economic growth.

Data shows that about 64 percent of U.S. households used a high-speed Internet service in 2009, the Commerce Department said on Tuesday. That is a 25 percent increase from 51 percent two years earlier.

Some internet providers already called this program “unrealistic”, given the current estimate of the average access speed at less than 4 MBPS. However, Verizon, the third-largest provider, and one that has a more advanced network than many competitors, said it has completed successful trials of 100 Mbps and higher through its fiber-optic FiOS network. Today, Verizon offers programs with 50 MBPS access, at a premium.

The FCC announcement should not come as s a surprise, given Google announcement from the beginning of February that it will pilot a deployment of 100 MBPS links to certain communities. Not surprising, the first newspaper to report this development was the Korean Times, the country with the most widely available high speed Internet service for home consumers.

Why 100 MBPS matters? Obviously, it will open the possibility for stay home Moms to join the work force by becoming part of corporate networks; will open great opportunities for home-schooling connected to on-line interactive schools and colleges and open a whole new world of entertainment and leisure activities, like interactive TV, collaborative gaming, real-time social interaction and more.

The key message is very clear – BIG pipes and FASTER applications are on the way.  We need to ensure this new environment can handle our demands.  Shunra technologies allow you to emulate the network environment, introducing real world impairments to test those media rich applications in a controlled lab environment. Shunra has been providing this capability for more than 10 years and now, more than ever, is the time to prepare.

Since the January 27^th announcement, it has become public knowledge that also Google is about to release a smart tablet, as well as HP using a Windows 7 platform. So are we reaching critical mass? I believe we are reaching the point tablets cannot longer be ignored.  Before the end of the year you will be able to receive TV streams into your tablet, read E-books, use your favorite social network, including the latest addition, Goggle “Buzz”, order pizza for dinner and yes, check your email and place both video or voice conference calls.

On an interview with the Associated Press, Disney’s CEO, Robert Iger, as quoted at the blog TUAW, The Unofficial Apple Blog,  he depict different scenarios using IPAD to deploy games, media and gaming content, all of it oriented toward the non-technical audiences.

Apple is drawing heavily on the experience gained by both IPod Touch and Iphone, that  in only a couple  of  years had managed to become the industry standard for smart web appliances and smart phones. The primary audience is the millions of users who are too afraid or reluctant to use a laptop/netbook since they are too rigidly bound to an unfriendly Windows OS.  A typical tablet user doesn’t want to use a computer per se.

So, why this is relevant to this blog? Very simple, all the context, media, games, TV shows, music and E-books will be delivered over ever increasing congested networks, served from ever more complex infrastructures at all will require substantial testing. What better tool to test a distributed network that Shunra Virtual Enterprise?  Application providers will have to test their application before deploying them to the millions of users. What will be the user experience, how latency will impact an IPAD user watching 2011 Super Bowl or downloading the latest chapter of their favorite TV show?

The combination of testing for 3G readiness (see other posts in this blog) and end user experience assessment will be a key factor on the success of this game changer.

Shunra VE is the best way to prepare for this new wave of context and delivery mechanism.

Due to current events the timelines for testing and certification had become critical and the contractor responsible for the project was under enormous pressure to complete this engagement quickly. In conversations with the contractor prior to arriving onsite, I asked a series of questions intended to reveal the lab environment and methods of test traffic generation.  It became clear that some big action items that needed to done in order to achieve the requisite environment and deliver all of the desired results was still unfinished.

Some of these dependencies included….

• Reaching consensus among all the constituent groups as to what would constitute the physical test network as well as the specific impairments to define the characteristics of the circuits to be recreated within the Shunra scenarios.
• Confirmation of existing test scripts of the application for both the QTP and HP Performance Center traffic generation tools.
• Communication between the contractor (Project Manager), network equipment vendors, QTP and Performance Center engineers and Shunra to ensure that each step in the process was coordinated to proceed in an orderly sequence.

As these conversations took place it became evident that the project manager had never been exposed to the complexities associated with the range of technologies and the number of disparate groups required to compete the task. I cautioned him that to guarantee timely completion of all these steps a meeting should be scheduled to include all the participants prior to arriving onsite. Shunra offered our Professional Services to act as the Project Manager to broker to meeting and to coordinate the project execution. The key benefit to ensure that all parties would be fully prepared to execute their tasks when needed. This offer was declined. The meeting never was organized and the timelines of the engagement suffered accordingly.

Once onsite the network engineers and Shunra spent considerable time connecting and configuring the test lab to allow communication between all of the known test devices. After completion a member of the Performance Center group popped into the lab and informed us that some traffic needed to be initiated from end users on another network in a different building. This change mandated a complete reconfiguration effectively wasting hours of work. Only after completion of this task did the contractor learn that no test scripts had been created to exercise the application. Script creation could potentially add days or weeks to the completion of the project.

Meanwhile the representative from the Agency requesting this evaluation would appear every ten minutes asking whether the job was done. As the sweat poured off the brow of my contractor I could only shake my head and think…..Told you so…..

The lesson to be learned is that Shunra has a cadre of experienced professionals that can deal with the complexities of these engagements. In a proactive systemic process we can anticipate all contingencies and implement best practices honed over years of direct experience. With this expertise from our Professional Services group the scoping, timing, and delivery of test results are guaranteed.

The contractor is still in stasis awaiting his test scripts and that Federal representative is still poking his head asking for the results.

Oh well………

WAN Simulation methodology is a set of theoretical algorithms which are usually applied on a trace file. These algorithms aim to predict applications response times with different conditions and then applied to the captured trace file. A typical use case would be varying the conditions, such as available bandwidth and distance (latency), and then predicting the impact on response time. The main challenge with this approach is that those theoretical algorithms are based on assumptions.  Assumptions, which are in some cases very wrong and skew the results. Typically, with theoretical WAN Simulation you should expect around 70% accuracy.

On the other hand, when we say Emulation (or WAN Emulation) we mean that the application traffic will traverse in real time through a path that is identical to the path we have in mind. An appliance based WAN Emulator or software based WAN Emulation is usually present to create actual network elements in software, bring into account network impairment characteristics, such as distance (latency), bandwidth and loss. The accuracy of WAN Emulation can exceed 99% depending on the level of configuration. We can expect around 95% accuracy from a straight forward automatic configuration.

Another important difference is time. Theoretical WAN simulation may take days to run, but using WAN Emulation the duration is based on the time it takes the application to run.

The main reason why WAN Emulation triumphs is accuracy.

As more technologies are developed and deployed onto the network, this increased demand can significantly tax the network – requiring immensely more resources to manage and maintain.  This issue is certainly more of a concern now, considering that most providers host voice, video, and data services across several mediums (WiFi, Metro-Ethernet, Cable/DSL, etc.). Moreover, customers have also come to expect these on demand services to always be up and running whenever they need access.  This is exactly why MPLS has come to the fore-front because of how it can scale to address the demands of more complex networks. With the advent of IPv6, MPLS is now becoming the preferred transport mechanism to handle mobile backhaul and many other services over IPv6.  With all of these new technologies and applications coming to bear on the network at the same time, this will certainly require some extensive testing to ensure that the network can deliver all of these services reliably.

Case in point, this month at Shunra we finished an engagement with a large financial services company deploying a global MPLS network.  They had many concerns about this deployment, the most significant of which is that some of their end-users in various locations would experience a significant increase in latency.  Thus, this would cause disruptions in productivity and may even crash some of the applications – which are critical to the business. The Engineers also knew that MPLS was the way to go because they were deploying VoIP as well and they were concerned about capacity requirements for all of these applications.  As such, we worked with them to develop a test strategy to ensure they can deploy MPLS with confidence. In their environment, we recreated part of their MPLS production network in a virtual test bed and ran all of their critical applications. After many test iterations and diligent analysis, we were able to derive a deliverable that gave them considerable insight as to what the problems were with the applications and how to best address them – from both a network and application perspective.  Ultimately, this allowed the financial services company to make an informed decision about how to deploy MPLS across their network.

Will you be deploying MPLS across your network? If so, Shunra provides consulting services specifically for this type of testing.