What resonated well with me was a quote from Jim taken from “Alice in Wonderland” – “If you don’t know where you need to get to, then it doesn’t really matter which way you go”.

The full article is available here: http://www.networkworld.com/newsletters/frame/2011/013111wan1.html?page=1

Keep an eye out for additional articles by Jim Metzler around the Application Performance Engineering process in Network World, which should come out in the following weeks.

Talk to you soon,

Amichai Lesser

It’s a simple statement, but a powerful one that will change the existing application lifecycle management paradigm. At Shunra, we have been working hard, together with our partners, at developing and defining a new performance management model that completes the existing APM  frameworks. This post is a first in a series of posts that describe this model and the motivation behind it. In this first post, I will present the reasons why now, more than ever, such a model is needed – why the time is right. Future posts will dive deeper into the definition and building blocks of the application performance engineering model, explore where this model is applicable and explain the benefits it promises.

Application performance is still a problem.

Despite increased investment in application performance management and monitoring solutions, applications in production continue to fail to meet critical performance requirements. These failures create productivity loss for end users who cannot quickly accomplish the tasks the application was designed to facilitate. These failures result in lost revenue from eCommerce and other consumer services abandonment. And, these failures increase remediation costs as time and resources are required to isolate, diagnose and resolve performance problems in production.

Why now?

Application performance is still a problem – this is not a new concept, but it bears repeating. I could have written that sentence 5 or 10 years ago, and it would have held true then, just as it does now. So, why is the time right (or better said, why is the time right now) for a new and better approach to application performance management?

Today’s ever more complex enterprise environment and increasingly distributed application architectures (and user populations) are making the challenge of effectively managing application performance more difficult. New performance risks are arising from three Enterprise IT waves that are converging upon us today:

1)            The infrastructure paradigm shift – the economic downturn has accelerated the move towards data center centralization including, on an accelerating basis, Cloud migrations and associated cost benefits. Enterprise IT is expected to manage these complex projects to realize the promised savings, without impacting performance or end user productivity.

2)            The technology paradigm shift– several new technologies are redefining the way data is delivered to users including virtualization of servers and desktops, unified communications and Web 2.0.  As Web 2.0 technologies are rapidly being adopted in the enterprise, new smart browser-based capabilities are being introduced which blur the lines between where data resides and how and when it will be transferred across the network. In addition, and growing at an even more remarkable pace, is the mobile revolution. Organizations are increasingly challenged to deliver consumer-like applications, capable of handling enterprise grade data volumes, to an ever more demanding and mobile workforce.

3)            The reality of the “Virtual Remote Office” – the virtual office which was a forward looking concept just 10 years ago, is a reality for an increasingly growing workforce. In addition, the scope of the “virtual office” is expanding beyond the residential “home office” to include trains, planes, coffee shops and hotels. Today’s workforce is expected to be productive anywhere they can access a network connection, placing a tremendous dependency on the networks for how well applications will perform.

This “perfect storm” or confluence of Enterprise IT waves makes it difficult to deploy and gain confidence in the performance of applications. As application performance continues to become more dependent on the network, confidence in application performance will continue to decrease. This necessitates increased focus on how applications behave across a variety of networks including WAN, Internet, Mobile and Cloud.

What do companies do today and why isn’t it enough?

Traditional approaches to performance management include Application Performance Monitoring. This approach, however, offers only partial relief. Performance monitoring is a solid approach to discovering and diagnosing problems that result from the interaction of applications and networks in production. That is the key – performance monitoring is a reactive solution to discovering issues in production. And, because no application is perfect, performance monitoring is critical to performance management.

Discovering and fixing problems in production, though often required, is costly.  Remediation can require re-engineering or significant infrastructure investments that can be difficult to plan in a firefighting mode. There are time pressures and cost sensitivity issues since end users are being affected and application development and testing costs have already been allocated and incurred. Performance monitoring solutions help improve MTTR, but they cannot offset the time and resource costs associated with poor performance in production and poor end user experience.

What is missing?

More forward-thinking companies are moving away from the break-firefight-fix cycle and are beginning to augment their APM strategy with a more proactive, pre-production approach to performance management. This new approach leverages Application Performance Engineering best practices to reduce performance remediation costs, increase monetization and ensure end-user productivity. To be successful, this approach relies on a precise replica of the production environment for identifying poor performing business-process-steps in advance of deployment. This does not necessarily mean recreating the network hardware in the QA lab, but enabling a precise emulation of the hardware and production network conditions or impairments. With an APE-enabled test lab, these companies can effectively and reliably investigate performance optimization solutions, be confident in their network and performance decisions, and validate real-world application response times in advance of deploying to an end user community across networks – WAN, Web, Mobile and Cloud.

In my next post, I will dive deeper into defining the goal of Application Performance Engineering and the associated building blocks that make up this model.

In the mean time, I am looking for your feedback:

• What else is missing in how application performance is being managed?
• How do you view the application performance engineering goal?
• What do you think needs to be in the application performance engineering model?

The content in this post has been co-developed with Bill Varga, the COO of Shunra and Marty Brandwin, Director of Product Marketing at Shunra. Many parts in it were influenced by thoughts and comments from most of the executive team at Shunra and our partner community.

Looking forward to your comments,

Amichai Lesser

There are many challenges organizations face when it comes to administering and maintaining a dedicated pre-production or staging environment for which accurate performance testing can be conducted. The cost to manage and maintain infrastructure, along with personnel and facilities, can be fairly sizeable and are only a subset of the overall costs to be considered. So, in many cases performance testing can be rather expensive and this is exactly why virtualization can provide significant benefits because there is cost reduction across the board. A prime example is in many performance labs there are a variety of application performance tools typically utilized for testing – one such tool is HP LoadRunner or Performance Center. These tools are a primary part of a performance lab as they provide load generation capabilities and can accurately test applications under real world load and stress scenarios. However, these solutions require a significant amount of infrastructure and resources (A Controller to execute tests, LoadRunner Generators to produce user traffic, Virtual User Generator to record scripts, etc.) and this can make it very difficult to manage the environment when it has to scale to meet higher demand. In this case, virtualization saves time, effort and cost because resources can be allocated dynamically within the environment and any number of virtual machines can be leveraged when needed to handle these resource intensive applications. This is also enabling many organizations to architect and customize elegant configurations that more closely align with their testing requirements – which can minimize unnecessary infrastructure and resources. Yet, the prevalent issue many organizations still grapple with is how to execute performance tests that accurately depict the network for which the application will be deployed across.

The most pervasive approach that many organizations would take is to physically deploy hardware (remote load generators) in offices that they wanted to test an application from. This process was not only time-consuming, but also expensive, inaccurate and cumbersome to manage. For this reason, HP decided to form a partnership with Shunra to develop a seamless solution that provides this capability within the HP LoadRunner and Performance Center solutions – Shunra VE Desktop for HP. This solution aligns very well with the virtualization movement because it is simply a plug-in within the HP products that introduces the network into the existing test bed and can be leveraged across most virtual platforms. For LoadRunner 9.5 and later, there is simply a “WAN emulation” tab that can be accessed from within the Controller to introduce the latency, jitter, packet loss and bandwidth constraints directly into the test. With Performance Center 9.5 and later, this capability can easily be configured directly from the browser UI to allocate WAN emulation parameters across any number of desired load generators. A consequence of this is that organizations can leverage on-demand performance testing from a dynamic virtual environment that is agile, flexible and robust. This therefore eliminates the need to manage testing cycles across multiple time zones and remove any need for additional hardware. Additionally, all of the network metrics from each generator utilizing WAN emulation within the test will automatically be imported into the controller, which can save a significant amount of time when collating results and generating analysis reports. These reasons are precisely why numerous organizations have decided to improve their existing performance test environment with the VE Desktop for HP Software  solution. Not only does this solution address a strategic gap within the functionality of the HP solutions, it embraces virtualization as a means to more effectively administer performance testing. Overall, the VE Desktop for HP Software solution was co-developed with HP to considerably enhance the accuracy and value of these application performance test suites.

Recently, I visited a company that develops and distributes software for Independent School District’s across USA. Their main product, originally developed for challenged kids, centers around dynamic computerized teaching and testing programs. This product proved to be so successful among this population that the company adapted it for other kids as well. Initially the product was designed to work in the classroom, hence a very simple architecture: 1 PC working as a learning manager.  This 1 PC managed the progress of individual students, as well as received and captured data back to computers across the LAN. However, because of such a high demand, the company decided to host a solution for multiple customers from a central location, specifically – a Cloud hosted server.

Since the original design was created with only a classroom setting in mind, testing and development only occurred over LAN. But now, with accessibility from different regions, including links from APAC countries to US, consideration of the WAN was critical. This required the need to test their solution in a new environment and see if any problems exist.  By analyzing this information, they gain a better understanding of how to fix any problems. With this challenge, the company needed a WAN emulator solution that would allow them to re-create the complex topology of the application served over WAN.

Once tested, they were able to easily reproduce the problems reported to the QA from the field. One issue that was noted – disconnections without the ability to reconnect, was leading to a loss of data and historical progress of the students in the program.  One of the key tools is tracking “task lists.”  One of the main advantages of the system is the ability to produce these new “task lists” for each student, based on historical data retrieved.  The new tasks matched the student’s progress in learning the subjects and fulfilling the tasks.

One other obstacle the company faced dealt with the USD security measures.  Since product was originally operating on a LAN environment, the security of outside access was a concern. As a result, certain ports were blocked and the company needed to understand how those ports will be affected from blocks in different directions of the link. Shunra VE WAN Emulator was able to reproduce such conditions by filtering application traffic based on protocol and ports, as well as IP addresses and blocking them in one direction – the problems were reproduced instantly. Now QA no longer has to rely on the reports generated from the field to try and fix the problems “in the dark”…  They now have a nice “flash light” to expose those reasons for the problem.

The impact of a broken link or just a simple delay in logging in can significantly impact students, especially the challenged ones.  Picture a child struggling to connect to the program, no sounds, no pictures… instead sitting with headphones and seeing on a screen: “Connecting…”  An unnerving experience for all of us, and even less for them (IMHO).  And what if after working for 20-30 minutes on a progressive task – they have to start over because computer lost all memory of what they were doing until now.

However this series of posts is about the day after a data center move. Now that the data center is remote, how does this paradigm shift impact the way we should develop, test, deploy, monitor and troubleshoot applications. I will try to cover as many topics as possible, but the main focus is still going to be around the role application performance management plays in this new paradigm.

I will start by covering the key reasons behind the performance impact that is experienced when applications are hosted in a remote data center? Those reasons are fairly intuitive, but it is important to understand them in depth in order to adequately plan for those new conditions. Two main things impact how applications perform when application servers are hosted in a remote data center vis a vis their application clients :

1. The performance of the network link between the client and the remote data center. This performance is defined by a set of network performance metrics that are application independent (for now we will ignore application aware networks, however the following basic concepts still hold in this scenario as well).

2. The application efficiency, specifically how efficient the application is when transferring data between the client and the remote server (and other tiers if applicable). This is an application attribute (and some time an attribute of a specific business process within the application). These attributes are application specific and are independent of any underlying network.

Let’s start with understanding the network performance metrics. Consider the following scenario:

An application is hosted in a NYC data center, with users in 2 places, some are in a NYC headquarters next to that NYC data center and some are in a remote branch that is located in San Francisco. The question is: “will the application perform the same for both type of users (local users in headquarters and remote users in SF)? In other words will the application be as responsive to the San Francisco user as it is to the NYC user?”

Well the obvious answer is NO, in most cases a NYC user will enjoy a faster more responsive application. What is less obvious is why? What is it about the network that causes remote users to experience a slower application than local users? The rest of this post will cover that question, future posts will address the application specific attributes. Once we cover that we will be ready to examine best practices in building applications for a remote data center.

When I ask this question during my training seminars, I get a variety of answers, many of them are the right ones, but I would like to address one wrong answer that keeps repeating itself for some reason.

Collisions – there is a general conception that collisions are common phenomena on the network which can explain any bad thing that happens to applications. The truth is that collisions are almost a thing of the past (on Enterprise LANs anyhow) and even when they happen they can’t explain why a remote user has a worst experience than a local user as both will experience a similar collision chance since collisions is a phenomenon that happens on local area Ethernet networks. If there are collisions on the Enterprise LAN it usually points to a configuration issue on a network device (like a duplex miss-match) but is still unrelated to the answer to our question.

Now to the right answers to the question, what is it about the Wide Area Network that causes applications to slow down:

There are 5 key conditions that predominately exist on Wide Area Networks and impact application performance, each in their own way:

1. Network Latency – the time it takes a packet to traverse from a source to the destination across the network, measured in milliseconds [msec]. A typical WAN link will introduce latency in the range of 10msec – 500 msec.

2. Bandwidth constraints – how fast can data be processed by the network link, measured in bits per second [bps, Kbps, Mbps, Gbps]

3. Bandwidth utilization (background traffic) – the percentage of bandwidth that is utilized by traffic that already exists on the link (background traffic).

4. Jitter – the deviation of the inter packet gap of sequential packets across a network link, it is a result of the deviation of the network latency and is sometimes used interchangeably with that standard deviation, measured in milliseconds [msec].

5. Packet Loss – the chance to drop a packet across an end to end network link, measured in %. Sometimes presented as the inverse metric called packet delivery rate.

The above are called network impairments, you can click on each one of the links to learn more about them and their causes.

Network impairments are performance conditions that inhibit the flow of data across a network. Each impairment type has an impact on the performance of business applications and network services. Some applications may be very sensitive to network impairments and some may be almost network agnostic. Sorting applications based on their network sensitivity is one of the important steps in performance engineering

In the next post we will discuss how application design can impact performance across the network. But in the mean time I would like to introduce a question for the group:

“We identified network latency as one of the key reasons that impact application performance; we also said that a typical WAN link will introduce 10 – 500 msec of latency. The question is, why does network latency have a big impact on application performance? surely a user doesn’t notice an increase of a few msec in response time, even 500 msec = ½ second goes by in a flinch. So think about it and let me know what you found based on your experience, why does network latency have such a big impact on application performance?”

Until next time,

Amichai Lesser

In alignment with global trends, more and more clinical trials take place outside of the United States, while the documentation and analysis is done in the US for submission to the FDA. So it is not out of the ordinary to find an Oracle Clinical user in Eastern Europe or in China submitting data to an Oracle Clinical Server hosted in the united states.

Unfortunately, as commonly found with an off the shelf software package that gets customized over time, this type of application can easily morph into a performance challenged application, especially for remote users. The following analysis shows an example of an Oracle Clinical transaction that completes in 8 seconds for local users, while extending to over a minute when users in E. Europe tried to use it.

Transaction Response Time Analysis for an Oracle Clinical Application Accessed by Local Users Vs. Remote Users in E. Europe

Obviously, that big jump in response time was a great cause for concern with my client’s management. Which is how I got the privilege to be asked to analyze the root cause of the poor performance.

The first thing I noticed was that one particular transaction (Connect to DB) had an extremely high jump in TRT from 8 seconds locally to over a minute when accessed remotely, so I focused on that one first.

Looking at the network fingerprint of the transaction you can observe the following things:

The transaction size (amount of data downloaded from the server required to complete the transaction) is extremely high (over 5 MB).

The transaction generates 34 HTTP calls. Some of those calls take disproportionally longer time than others (notice the Get jar file calls that each exceed 19 seconds marked below, click on the image for a larger view)

Oracle Clinical, Transaction Analysis for "Connect to DB"

When inspecting those JAR files, we saw that they were very heavy in size, over 1 MB each which is pretty big for basically a collection of Java classes that are zipped into a JAR file.

You can also observe in the next bounce diagram that most of the JAR files are downloaded in a serial fashion, blocking other objects from downloading in the mean time. Notice the delta time displayed on the right column

Bounce Diagram for an Oracle Clinical Transaction

What can be done to improve the performance of this application?

Oracle Clinical uses JAR files to package Java code that will run on the client. It is not uncommon for customized off the shelf applications to increase in code size over time until the code is very bloated.

There are several best practices that developers can follow to reduce the size of JAR files:

1. Rationalize the code – as applications develop over time, multiple classes and sometimes adjacent projects might reference similar libraries and other assets. If not careful those libraries and common assets end up packaged multiple times inside the JAR file, causing it to inflate in size.
2. Minify the code – a quick Google search for “reduce the size of JAR files” will reveal several free tools that can minimize the size of JAR files, usually through eliminating white spaces, comments, shortening variable names, etc.
3. Defer loading – chances are that not all the code in the JAR file is needed for the “connect to DB” transaction, which means that users that only want to perform a small transaction are penalized by the download time of code that will never get executed by them. Deferred loading is a design pattern that simply says “only download assets or code when it is needed” I wrote more on that design pattern in this previous post “http://www.shunra.com/shunrablog/index.php/2009/01/29/content-loading-when-being-lazy-pays-off/”

In summary, as more and more web 2.0 technologies hit main stream, new forms of performance pitfalls may present themselves, requiring us to consider application performance for remote users even for applications that traditionally were considered “remote user friendly” such as web based applications.

How was this Analysis Performed?

This analysis was used by integrating a HP LoadRunner script modeling an Oracle Clinical business user with Shunra VE Analyzer. So as the scripted transactions executed across a virtual WAN (simulated by the Shunra VE) the beginning and end of each transaction were marked by the VE Transaction Manager. At the end of the test, the VE Analyzer had sufficient data to generate the attached reports, enabling us to pin point the root cause of the performance problem. You can learn more on how to set this up by going through Shunra Certified Performance Engineering training, either on site or at the next Shunra University. Learn more about Shunra training here:  http://www.shunra.com/shunra-university-overview.php and here http://www.shunra.com/training-overview.php?keyword=services

Questions, comments, feel free to write me at amichai.lesser at Shunra dot com. Or comment on any of my posts.

BTW, we just created a new Application Performance Management Group on LinkedIn, feel free to look it up and join here  http://www.linkedin.com/groupRegistration?gid=2200667

Best,

Amichai Lesser

Which is why I was glad to stumble upon Shamus McGillicuddy article in www.searchenterpriseWAN.com titled “WAN engineers prepare networks as cloud computing adoption rises” that can be found here.

There are some good observations in the article, I especially appreciate the comment on how application architecture best practices such as minimizing application turns become key when developing applications for the Cloud as well as testing applications for network latency becomes critical as the enterprise is migrating more and more applications to the Cloud.

Many questions still remain though and the impact of Cloud computing on performance engineering is still to be determined. Some of the questions that are on my mind are: (and feel free to chime in with more questions or attempts at answers)

1. How do you manage the performance of an application in the Cloud?

2. What changes if any are needed to the service deployment model and version upgrades?

3. How do you conduct a performance test for an application or a service in the Cloud (internal clouds and external clouds)?

4. How do you troubleshoot a performance issue in the Cloud?

Something to think about…

Best,

Amichai Lesser

Of the topics covered, WAN emulation is clearly closest to my heart, but please be sure to read up on Microsoft Vista support, the new RDP agent, and what is happening with LoadRunner’s Analysis API.

To read the full story, click here.

Do you plan to move to LoadRunner 9.5 or Performance Center 9.5 in the near future?  Will the integration with Shunra’s WAN emulation motivate you to upgrade or accelerate your upgrade schedule?

First off, VE Desktop for HP Software 5.0 is a new product.  It is in no way related to LoadRunner MWE.  HP/Mercury customers last saw MWE in LoadRunner 8.1 (July 2005) and the feature was removed from the LoadRunner 9.x releases.

While MWE was indeed “Powered by Shunra,” it was done so through an OEM relationship.  MWE was sold and supported by HP/Mercury.  VE Desktop for HP Software 5.0 development was a collaboration between Shunra and HP.  The product is the result of both companies learning from our experiences, designing, and co-developing a new solution that would exactly meet the needs of our customers.  VE Desktop for HP Software 5.0 can do everything LoadRunner MWE could, and much more.  VE Desktop for HP Software 5.0 is sold and supported by Shunra.

The top three issues we chose to address were:

1) Accuracy: Since the MWE did not support bandwidth emulation or jitter, the results of its emulations often did not fully reflect the real-world performance of applications being testing.  To address this, VE Desktop for HP Software supports bandwidth emulation (symmetric and asymmetric), jitter, imported network characteristics from Shunra’s VE Network Catcher, and a host of other network impairments aimed at precisely emulating network links.

2) Reporting: With MWE, there was no network specific reporting.  If a test was run with multiple locations emulated, the LoadRunner results would still show transaction response time and other metrics averaged across the whole test.  In order to understand how applications fared at individual locations, scripts would have to be modified so the script and transaction names reflected the location they were being run at.  This was very challenging to implement and maintain.  In VE Desktop for HP Software 5.0, there are no script changes required for such reporting.  For users of VE Desktop for HP Software, HP Analysis added the ability to sort results by “Emulated Location.”  HP also added the automatic import of network statistics on a per Load Generator basis.  Now users can understand their application performance by location and merge that data with network statistics like bandwidth utilization, latency and packet loss.  Finally, users can understand if the network is impacting their application performance and if so, how.

3) Performance Center: LoadRunner MWE had no compliment in Performance Center.  VE Desktop for HP Software 5.0 supports both LoadRunner 9.5 and Performance Center 9.5.  Within Performance Center, dynamic host allocation is fully supported.

VE Desktop for HP Software 5.0 provides a lot of other benefits beyond the direct improvements over MWE.  Some of those include a robust API and protocol analysis capabilities, but those are topics for other blog posts.

I look forward to your comments on the explanations above and any feedback you have about our latest addition to the Shunra family of products.