Application Performance Engineering Blog - Shunra Software » HP Performance Center

The Value of Leveraging Virtualization for Application Performance Testing

Liam McCamley — Fri, 12 Mar 2010 17:56:28 +0000

Virtualization has emerged as one of the leading technologies in today’s market; enabling businesses to more effectively scale operations to meet demand while significantly reducing costs at the same time. Everyone seems to understand what virtualization is, but it’s actually rather difficult to define because the term is used interchangeably to describe a plethora of different things. When I first tried to define what virtualization is in my own terms, I thought of it more as a technology for achieving some end – primarily server consolidation. However, after further investigation I realized that virtualization is really more of a concept than anything else. This certainly became evident in an article I read from the Virtualization Journal where the CTO of Citrix & Founder of XenSource, Simon Crosby, was being interviewed on the topic of virtualization. He stated that “virtualization is already widely used, but primarily for the first-order benefit, namely server consolidation. The second-order benefits of agility, availability and manageability of the IT stack are now becoming better understood,” Crosby continues, “and as a consequence virtualization has moved from a tactical tool for gaining immediate savings, to become a key strategic theme for every IT department.” Essentially, virtualization has become a business enabler for many – and that’s certainly apparent considering the number of organizations gearing up for cloud computing. This is also the case when it comes to how application performance testing can be leveraged within organizations today.

There are many challenges organizations face when it comes to administering and maintaining a dedicated pre-production or staging environment for which accurate performance testing can be conducted. The cost to manage and maintain infrastructure, along with personnel and facilities, can be fairly sizeable and are only a subset of the overall costs to be considered. So, in many cases performance testing can be rather expensive and this is exactly why virtualization can provide significant benefits because there is cost reduction across the board. A prime example is in many performance labs there are a variety of application performance tools typically utilized for testing – one such tool is HP LoadRunner or Performance Center. These tools are a primary part of a performance lab as they provide load generation capabilities and can accurately test applications under real world load and stress scenarios. However, these solutions require a significant amount of infrastructure and resources (A Controller to execute tests, LoadRunner Generators to produce user traffic, Virtual User Generator to record scripts, etc.) and this can make it very difficult to manage the environment when it has to scale to meet higher demand. In this case, virtualization saves time, effort and cost because resources can be allocated dynamically within the environment and any number of virtual machines can be leveraged when needed to handle these resource intensive applications. This is also enabling many organizations to architect and customize elegant configurations that more closely align with their testing requirements – which can minimize unnecessary infrastructure and resources. Yet, the prevalent issue many organizations still grapple with is how to execute performance tests that accurately depict the network for which the application will be deployed across.

The most pervasive approach that many organizations would take is to physically deploy hardware (remote load generators) in offices that they wanted to test an application from. This process was not only time-consuming, but also expensive, inaccurate and cumbersome to manage. For this reason, HP decided to form a partnership with Shunra to develop a seamless solution that provides this capability within the HP LoadRunner and Performance Center solutions – Shunra VE Desktop for HP. This solution aligns very well with the virtualization movement because it is simply a plug-in within the HP products that introduces the network into the existing test bed and can be leveraged across most virtual platforms. For LoadRunner 9.5 and later, there is simply a “WAN emulation” tab that can be accessed from within the Controller to introduce the latency, jitter, packet loss and bandwidth constraints directly into the test. With Performance Center 9.5 and later, this capability can easily be configured directly from the browser UI to allocate WAN emulation parameters across any number of desired load generators. A consequence of this is that organizations can leverage on-demand performance testing from a dynamic virtual environment that is agile, flexible and robust. This therefore eliminates the need to manage testing cycles across multiple time zones and remove any need for additional hardware. Additionally, all of the network metrics from each generator utilizing WAN emulation within the test will automatically be imported into the controller, which can save a significant amount of time when collating results and generating analysis reports. These reasons are precisely why numerous organizations have decided to improve their existing performance test environment with the VE Desktop for HP Software solution. Not only does this solution address a strategic gap within the functionality of the HP solutions, it embraces virtualization as a means to more effectively administer performance testing. Overall, the VE Desktop for HP Software solution was co-developed with HP to considerably enhance the accuracy and value of these application performance test suites.

Building Applications for a Remote Datacenter Part 2 Application Efficiency Metrics

Amichai Lesser — Thu, 05 Nov 2009 19:02:58 +0000

In the previous post we identified the Wide Area Network and the impairments that it introduces as a key reason for why a local user (let’s say in NYC) experiences a faster application than a user that is remote to his datacenter (let’s say in Tokyo). I also presented a question to the group: “We identified network latency as one of the key reasons that impact application performance to a remote datacenter; we also said that a typical WAN link will introduce 10 – 500 msec of latency. The question is, why does network latency impact application performance, surely a user doesn’t notice an increase of a few milliseconds in response time, even 500 milliseconds = ½ second goes by in a flinch. So why does network latency have such a big impact on application performance when the data center is remote?”

The following posts will answer this question and more, but in order to get answers we need to first address additional questions:

Consider the remote user in Tokyo, he is accessing multiple applications that are all hosted in the remote NYC data center.

Will all these applications perform the same way once hosted in a remote data center?

The obvious answer is NO, some applications will perform well even when users are remote to the data center, while others will provide intermittent poor performance and some will always perform poorly for a remote user.

The answers to the next questions are less obvious:

Why do some applications perform well in remote data centers while others fail miserably?
What is it about the way applications are designed and architected in a remote data center that allows some applications to perform better than others?
What are the key design flaws that cause applications to perform poorly in a remote data center?

Since there are a lot of different applications, there are also a lot of different answers to these questions. In the next couple of posts we will focus on enterprise-data-driven- transactional applications and their performance flaws over the network (ignoring for a moment back-end or desktop related bottlenecks). We will further limit this category to client server based applications, ignoring for a moment the complexity of N-Tier applications and multi web service based applications, these will be dealt with in future posts.

Over the years I was asked to analyze performance problems for many transactional applications and specifically analyze their performance degradation when users are remote to their data center, the following are the key application metrics that I found are related to performance degradation over the network:

The number of application turns per transaction (or how chatty the application is)
The transaction size (or how much data needs to be downloaded from the server to the client in order to complete each transaction)
The transaction efficiency factor (how much data a transaction downloads per application turn)
The blocking nature of object retrieval (can the transaction retrieve multiple objects concurrently or is each object download blocking other object requests from being processed)
The transaction redundancy metric (or how much of the same data is being retrieved by all the requests made by this transaction). It seems like this metric should always be zero, but you will be surprised how often this is the single reason behind performance problems.
The transaction initialization size (how much data does the transaction download initially Vs. sequential navigational steps)
The caching ratio (how much data is cached locally as a percentage of the overall data needed by the application)
The latency scale factor (How does the backend’s ability to scale change when network latency is added between front end clients and the backend)

Measuring and observing these metrics allows for the deep level analysis that is required to identify performance bottlenecks. This information also helps to point out to application developers and system engineers what needs to be changed in order to remediate application performance problems.

In the next couple of posts I will explain each of the above factors and describe how each of them impacts application performance, so sign up for the RSS feed to get notification on these future posts.

Until then…

Amichai Lesser

Hosted Load Testing II

Dave Berg — Fri, 30 Oct 2009 15:34:09 +0000

My friend Mark Tomlinson from HP recently wrote an informative blog about “Understanding the language of hosted load testing.” The blog touched on two competing approaches to application performance testing that were referenced as “behind the firewall” and “outside the firewall”. Behind the firewall testing usually means testing in a lab environment such as a Performance Test Lab. Outside the firewall testing means testing over some part of a live network. As you can imagine, and as Mark pointed out, both approaches have their pros and cons. Selectively borrowing from Mark and adding some of my own gives us the following:

Performance Test Lab testing

Pros: Controlled, repeatable, relatively easy to debug observed problems

Cons: Expensive to build and maintain, commonly implemented without real world network considerations

Live Network testing

Pros: Cheap, good for periodic sanity checks of performance

Cons: Puts non-revenue generating load on the production network, very difficult to debug, high volume testing is not an accurate depiction of end user performance

The good news is that developments in cloud-based testing and in HP LoadRunner have been able to improve the accuracy of Performance Test Lab testing while also reducing the complexity and overall cost. By moving HP LoadRunner load generators into a cloud-based testing environment, your testing can immediately benefit from the flexibility and scalability of the cloud.

One specific use case is for Peak Testing. Peak testing is simply another name for load testing. What “peak” implies is that it is at a scale that is not common. If you are running, or want to run a test that is not commonly run in your environment, then it is fair to assume that you’ll need to plan ahead to gather the necessary resources. Working in a cloud environment makes gathering and configuring extra resources on demand easy. Need an extra five load generators? No problem; just clone your existing one and wait for them to start. Need more RAM? No problem; just shut the machine down, configure more RAM, and restart the machine. Working with the latest version of HP LoadRunner with WAN emulation makes the story even more compelling. Need load generation from LA, London, and Tokyo? No problem; just configure the load generators to emulate those locations and they will. Sound easy? It is. Companies like Skytap already have thought this scenario through and have a great subscription-based model built for you to use today.

The real benefits of peak testing in the cloud are that you can quickly scale your lab up and down without the burden of maintaining a lab large enough to support your peak needs –saving time and money. It is the end of costly lab build-ups, running tests during maintenance windows in the middle of the night, sending load generators to all corners of the world, and putting test traffic on your production network!

Of course, there are a lot of other ways Cloud Computing can be used in a QA environment; validating new versions of test products and proof of concepts are two that immediately come to mind. In the end, the same ROI formulae that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.

Now you know what I think. What do you think??

P.S. Stay tuned for a coming blog on location-aware deployment testing in the cloud…

Analyzing and remediating latency sensitive applications part 1

Amichai Lesser — Thu, 20 Aug 2009 20:04:17 +0000

Examples of latency sensitive applications

In the previous post http://www.shunra.com/shunrablog/index.php/2009/08/14/data-center-relocation-questions-and-answers-part-1/ I shared some of the questions that clients typically ask me during the performance analysis service in a data center relocation project. One of the most popular question, especially at the beginning of the project is “which applications are the most sensitive to network latency?”. Initially I thought of writing a post that categorizes and summarizes those applications, however, after reviewing the (long) list and reading through the analysis, I figured the community could benefit from some of the specific examples. I promise to try and summarize it all at the end.

MS Access on a shared drive – it’s only a good idea if your users like to wait

The first example is MS Access, and specifically, MS Access that is hosted on a shared drive. It takes very little latency (even 2.5 msec) to “break” the performance of an MS Access application deployed in such a way. The following example is one of many MS Access deployments that I had the privilege of testing, the results always look the same. The names of the transactions have been removed to protect the IP of the clients.

The baseline response time of a report generation transaction was 19 seconds. Once tested with 2.5 msec of latency, the response time jumped to 45 seconds, the following analysis reveals why:

Generate MS Access reports – 0 msec latency run

Without latency, this transaction spends about 10 seconds accessing the mdb file on the shared drive and the rest of the time is spent on executing the query to generate the report.

Generate MS Access reports – 2.5 msec latency run

Once network latency is introduced, this transaction spends 24 seconds accessing the mdb file on the shared drive and the rest of the time is spent on executing the query to generate the report.

So it is clear that there is a big performance penalty for file access, as by default file access is done using CIFS which is a protocol that is known to be very latency sensitive.

The following chart shows over 2300 CIFS File requests and over 1100 CIFS directory requests just for that single transaction.

So what can be done to improve the performance of this application?

There are several options when it comes to mdb MS Access files. In this example, the mdb file contained both the code and the tables. The first remediation attempt should be to export the tables to a SQL data base and link the SQL tables to the MS Access forms, this should reduce the size of the mdb file and thus reduce the performance penalty associated with accessing the MS Access mdb file remotely.

A more beneficial approach but at the same time, one that requires more changes, would be to export the tables to a SQL data base and replace the MS Access forms with web based forms. This architecture will eliminate the file server access completely as the client would be accessing data via a browser.

How was the analysis performed?

The above test used VE Network Appliance, VE Analyzer, VE ClearSight, VE Transaction Manager and Wireshark. The baseline and latency tests were conducted by isolating a client workstation behind the network emulator and injecting 0 and 2.5 msec of latency. During the test we used the packet list within the VE Modeler to capture the transaction traffic and we timed the response time using VE Transaction Manager.

At the end of the test we used VE Analyzer to perform deep packet analysis. Part of the analysis reports include the transaction markers report which enabled us to isolate the 2 transaction runs in WireShark (baseline and latency). The last step was to generate the above IO graphs in WireShark. These charts demonstrate the difference in the timeline and transaction footprint between the baseline run and the latency run. VE ClearSight was used to analyze the SMB protocol (CIFS) and to generate the last chart.

Data Center Relocation Questions and Answers Part 1

Amichai Lesser — Fri, 14 Aug 2009 20:10:16 +0000

In my line of work, I get to assist many clients with analyzing the impact of an upcoming data center relocation on the performance of business applications. Many clients don’t know exactly what application performance to expect post the data center move. The main concern is that once all the applications start to operate across a Wide Area Network (WAN) connection to a remote data center with all the added network latency, the future performance of applications is unpredictable at best and potentially becomes so noticeable that it hinders the success of the data center relocation project itself. Therefore, clients often ask me, “Should we really test all the applications for latency? Which applications are the most vulnerable to the added network latency? Are there best practices when it comes to designing distributed applications that need to operate across network latency?”

In the next 2 posts, I will address the first 2 questions, the third one is probably a good topic for a book, so I will attempt to address that one over several future posts.

Should you really test all your applications for latency prior to a data center relocation?

Before jumping into a direct answer with examples of horror stories from past data center relocations that didn’t test all applications and the consequence of that excessive risk taking I want to first provide a conceptual best practice answer.

I recently received my Six Sigma Champion certification and one of the concepts in Six Sigma and in management for quality in general is “Don’t mass inspect, build quality into the process instead”. This concept was one of Deming’s teachings that revolutionized the Japanese manufacturing industry in the 1950s. So one would think that according to that best practice, we shouldn’t test all applications prior to a data center move, since that would be the equivalent of mass inspection. However, that would be a miss interpretation of that statement, an organization can shift from mass inspection to more statistical sampling only if quality is built into the process and when mission critical processes are in control. To complete the analogy, an organization can settle for sample testing a subset of applications only if application development processes have quality built into them, and specifically with a data center move, application development processes have network aware best practices embedded in them. I have yet to find an organization that reaches that level of maturity prior to a data center move. Usually the causation is reverse, first a firm relocates their data center, then IT realizes that they need to adopt a new application development paradigm since from now on applications will access remote servers in a remote data center and thus network aware best practices (like using VE Desktop as part of the development life cycle) are added.

So this brings us to the reasonable assumption that most companies will have a portfolio of applications that weren’t developed with a distributed WAN in mind and now have to adjust to operating in a remote data center. If that is the scenario your company is facing then mass inspection is your only way to mitigate the enormous risk associated with such a move.

Now if you don’t buy into the conceptual answer, consider the following:

In an average data center relocation project, we usually find that 20% – 40% of applications experience performance degradation ranging from visible to severe. (visible means that a user notices the slowdown in response time, severe means that an application stops being usable under latency conditions).
Many of the problems we uncover end up being show stoppers for the DCR project and need to go through remediation before the project can complete.
The mitigation path varies between applications and is very dependent on the type of problem that causes the performance degradation. Hence, a typical remediation solution may include WAN Acceleration for some applications (mainly server-to-server copy utilities and file services), terminal services (Citrix, remote desktop, VDI, etc.) for other applications, code fixes for some applications and architecture changes for the rest. Therefore engineering a solution based on a sample of applications will results in a sub optimal solution that doesn’t address all the performance problems that will manifest post the move.
When a large NY based insurance company relocated their servers from NY to Atlanta, GA without testing, they ended up rolling back the first phase of the relocation when several mission critical applications became unusable due to poor performance.

In summary, unless IT has built network aware best practices into the software development process, an organization planning a data center relocation must test all of its applications to analyze the performance impact that new added latency will have on application performance. More best practices on this topic can be found in my recently updated white paper that can be found at http://www.shunra.com/predicting-the-impact-of-data-center-moves-on-application-performance-whitepaper.php

The next post will cover the question “Which applications are the most vulnerable to the added network latency?” small example, any applications that are running executables from a shared network drive, will experience severe performance problems, that and more will be covered in the next post.

In the meantime, I am curious to hear your experience with data center relocations, what approach did your IT group take? Test all, sample test or cross your fingers and just move? Let me know.

P.S. as part of my Six Sigma Certification, I developed a dashboard and a flow chart for a data center relocation performance analysis, it needs some work before it can be shared, but if enough readers ask for it, I will try to clean it up and post it.

Till next time,

Amichai Lesser

What’s new in LoadRunner 9.5 – the rest of the story

Dave Berg — Wed, 25 Mar 2009 20:29:32 +0000

One of my favorite partners to work with is LoadTester. Not only are they competent and efficient, but they are nice and will tell you what they really think. Their latest update on HP’s latest release of LoadRunner 9.5 goes into a lot more detail than any readme file will, with perspective and colorful commentary around each of the improvements that might be important to you.

Of the topics covered, WAN emulation is clearly closest to my heart, but please be sure to read up on Microsoft Vista support, the new RDP agent, and what is happening with LoadRunner’s Analysis API.

To read the full story, click here.

Do you plan to move to LoadRunner 9.5 or Performance Center 9.5 in the near future? Will the integration with Shunra’s WAN emulation motivate you to upgrade or accelerate your upgrade schedule?

I Will be Teaching at the Next Shunra University in Philadelphia – April 14 – 16

Amichai Lesser — Wed, 11 Mar 2009 17:28:05 +0000

One of the highlights of my work at Shunra is being able to teach at the Shunra University. I get to work with a handful of great IT professionals as we drill down into the best practices and challenges of making applications perform well across networks. As much as I love teaching, the biggest value I get from this program comes from learning from my students, as do all the other participants. This program encourages Shunra users, staff and other IT professionals to share best practices and discuss daily challenges with their peers.

So here’s my shameless plug for the certificate program.

This 2 and half day certification program covers everything an IT engineer needs to know about Shunra’s technology, including:

Performance Engineering Best Practices
Deep Transaction Analysis and Performance Triage
Network Capacity Planning with Shunra VE Predictor, VE Profiler, VE Analyzer and HP LoadRunner
Advanced Network Topics
Network Aware Load Testing with VE Desktop for HP Software

Completing this program provides a SCPE certification (Shunra Certified Performance Engineer) which comes in very handy in today’s global performance world.

This year will be even more interesting as we will, for the first time, cover the new VE Desktop for HP Software. This product was co-developed by Shunra and HP to transform LoadRunner and Performance Center test beds into virtual network test beds for testing application performance from remote sites. The training session will provide expertise on combining load testing and network performance analysis in a single tool.

Another new addition to the curriculum is the Deep Transaction Analysis and Performance Triage which covers tips and tricks for effective triage and troubleshooting of application performance issues.

There are still some seats open which are sold on a first come first serve basis, so make sure to go here if you are interested in participating.

See you on the 14th at the DoubleTree Hotel in Center City, Philadelphia.

Amichai

VE Desktop for HP Software 5.0 versus LoadRunner MWE

Dave Berg — Fri, 06 Mar 2009 17:06:30 +0000

Earlier this week, Shunra announced the release of a new product, VE Desktop for HP Software 5.0. This product is embedded into HP’s LoadRunner 9.5 and Performance Center 9.5 products and from the surface, looks a lot like a previous feature in LoadRunner called Mercury WAN Emulation (MWE). Because of the obvious similarities, there have been a number of questions about the differences and any upgrade rights for legacy MWE customers. This blog will hopefully dispel any uncertainty and give you a clear understanding of how everything works.

First off, VE Desktop for HP Software 5.0 is a new product. It is in no way related to LoadRunner MWE. HP/Mercury customers last saw MWE in LoadRunner 8.1 (July 2005) and the feature was removed from the LoadRunner 9.x releases.

While MWE was indeed “Powered by Shunra,” it was done so through an OEM relationship. MWE was sold and supported by HP/Mercury. VE Desktop for HP Software 5.0 development was a collaboration between Shunra and HP. The product is the result of both companies learning from our experiences, designing, and co-developing a new solution that would exactly meet the needs of our customers. VE Desktop for HP Software 5.0 can do everything LoadRunner MWE could, and much more. VE Desktop for HP Software 5.0 is sold and supported by Shunra.

The top three issues we chose to address were:

1) Accuracy: Since the MWE did not support bandwidth emulation or jitter, the results of its emulations often did not fully reflect the real-world performance of applications being testing. To address this, VE Desktop for HP Software supports bandwidth emulation (symmetric and asymmetric), jitter, imported network characteristics from Shunra’s VE Network Catcher, and a host of other network impairments aimed at precisely emulating network links.

2) Reporting: With MWE, there was no network specific reporting. If a test was run with multiple locations emulated, the LoadRunner results would still show transaction response time and other metrics averaged across the whole test. In order to understand how applications fared at individual locations, scripts would have to be modified so the script and transaction names reflected the location they were being run at. This was very challenging to implement and maintain. In VE Desktop for HP Software 5.0, there are no script changes required for such reporting. For users of VE Desktop for HP Software, HP Analysis added the ability to sort results by “Emulated Location.” HP also added the automatic import of network statistics on a per Load Generator basis. Now users can understand their application performance by location and merge that data with network statistics like bandwidth utilization, latency and packet loss. Finally, users can understand if the network is impacting their application performance and if so, how.

3) Performance Center: LoadRunner MWE had no compliment in Performance Center. VE Desktop for HP Software 5.0 supports both LoadRunner 9.5 and Performance Center 9.5. Within Performance Center, dynamic host allocation is fully supported.

VE Desktop for HP Software 5.0 provides a lot of other benefits beyond the direct improvements over MWE. Some of those include a robust API and protocol analysis capabilities, but those are topics for other blog posts.

I look forward to your comments on the explanations above and any feedback you have about our latest addition to the Shunra family of products.