The following article builds a case for how performance engineering can help build more energy efficient mobile applications. Very interesting read, with strong reasons for why mobile performance engineering should be a critical part of mobile application development.

http://www.research.att.com/articles/featured_stories/2011_03/201102_Energy_efficient?fbid=dekBGE2Kx9k

I can picture how energy conservation will soon be added to the NFRs for mobile applications.

Enjoy,

What resonated well with me was a quote from Jim taken from “Alice in Wonderland” – “If you don’t know where you need to get to, then it doesn’t really matter which way you go”.

The full article is available here: http://www.networkworld.com/newsletters/frame/2011/013111wan1.html?page=1

Keep an eye out for additional articles by Jim Metzler around the Application Performance Engineering process in Network World, which should come out in the following weeks.

Talk to you soon,

Amichai Lesser

Losing site of the goal

What happens when IT departments lose site of the performance engineering goal? (Reminder in short the goal is to improve the end user’s quality of experience and productivity, while maintaining system costs within budget).

Well what happens is that each department gets lost in its own tactical goal:

1. The Capacity planning team focuses on efficient and accurate hardware provisioning
2. The load testing team focuses on test coverage and scale requirements
3. The network engineering team focuses on the speed and capacity of the pipes
4. The data base team focuses on the performance of the data bases
5. The server team focuses on the performance of the backend servers
6. Desktop team is focusing on the performance of the desktop clients
7.  …

What the organization ends up with is a set of local optimums, but in many cases those local optimums don’t amount to an optimal system. What’s missing in the above list is at least one department that is responsible for meeting the goal, it is very rare to find a team that oversees the end to end responsiveness and performance of the application across all its components from the end user’s perspective. It is even harder to find a team that is held accountable to end user performance.

But is it wrong for each team to improve its domain and make sure it is optimal? Well the counter intuitive answer is yes, it is wrong and for the following reasons:

Focusing on the wrong bottlenecks

Let’s consider the following transaction as an example: this transaction generates a time sheet report for global employees. This transaction is served by a client (web browser with java widgets) a few web servers behind a load balancer, a few application servers and a data base server. Now lets see what happens if there are performance issues with this transaction. Naturally each team will spend time in improving its own domain, so the data base team may index the employee’s data base to reduce the data base response time in half, the server team adds more web servers behind the load balancer to increase the application’s scalability and the network team adds more bandwidth to the data center router. All these steps sound like they should help, no? Well the realistic answer is that in some cases none of these steps help, in fact 3 negative things happen here:

1. The teams spent time and money on the wrong bottlenecks
2. The real bottleneck is still out there
3. Increasing the speed of none bottleneck components places more strain on the real bottleneck, slowing things down even further

In future posts I will give specific examples of several problems that can not be addressed in the realm of one IT department. Those problems usually result from interdependencies between the different systems (servers, networks and data bases). It takes a holistic and multi-disciplinary process to find the right bottleneck let along find a solution for the problem. It may sound complicated, but the concept is quite simple, when dealing with performance, it does little good to focus on local optimums, any optimization effort that is not spent on the actual bottleneck is counter productive and a waste of IT resources and money. Remember, the goal is to improve end user response time at the desktop, not optimize a specific component that is part of a bigger system.

Even though the concept is simple, the solution isn’t always as simple. In future posts, I will offer practical ways to find application performance bottlenecks, I have used them in many engagements and they haven’t failed me yet. But before we can talk about the solutions, it is important to understand the problems so in the next several posts we will cover a few basic performance engineering concepts.

Talk to you soon…

You have probably encountered a situation where pre-launch load tests did not correlate with the way it performed in production.  There are a lot of reasons why this can happen, but for now, I want to focus on 3^rd party content providers and Content Delivery Networks (CDN).  Almost everyone running a business today has some part of their application or Web presence farmed out to a 3^rd party.  Think about what happens when you go online to buy gifts for the upcoming Holidays.

As you browse for that perfect tie for your dad, the site is likely to be serving static content from Akamai or Limelight, adds from 3^rd party providers, and a shopping cart that may not be hosted by the company you are dealing with.  When you actually purchase that Tickle-me Elmo for your niece, the site processes your credit card by interacting with a financial institution and provides shipping tracking numbers by interacting with a major shipping company behind the scenes.  All that typically happens as you wait for the page to return in a few seconds if all goes as it should, but any of those 3^rd party providers have a strong likelihood to disrupt the page load and make the site look like it is not responding.

Increased delays from these and other 3^rd party providers do happen and you should definitely test for those scenarios, but how do you do that when you really do not have control over those sites. You definitely should perform load tests on the site as a whole to ensure that each 3^rd party provider will scale to your targeted peak traffic and perform within their SLA as expected. In addition, you should think about how slow your 3^rd party providers can get without affecting the performance of your site.

“How Slow Can They Go?” Yes, a corny statement, but to the point! You definitely need to identify how slow your 3^rd party providers can go before it starts to affect the performance of your sites.  The real dilemma is replicating the traffic from the 3^rd party site, which in the case of financial transactions can be very difficult plus your team can waste a great deal of time trying to do this. I’m suggesting that you don’t try to replicate that traffic, instead use the actual 3^rd party traffic from the 3^rd party site during the test. Here’s the catch, you impair the traffic coming from the 3^rd party site real time. Simply single out that traffic in your usability studies using a tool like Shunra’s VE Desktop to measure performance of the site with the 3^rd party adds impaired or by slowing down the response from your credit card processing partner using a Shunra VE Network Appliance on your backend systems during a maintenance window.

Now think about scalability load testing for a minute. I recommend that you do not implicitly trust 3^rd party and Content Delivery Network (CDN) provider’s scalability estimates and guarantees. You are probably scratching your head thinking that your CDN’s and other 3^rd party content providers have massive server and network infrastructures that are impossible to overload. While it is true that the major 3^rd party providers have massive infrastructures, the real question to ask yourself is this, “How much of that infrastructure is your content riding on?” Let’s just say that you actually do know how much of their infrastructure is dedicated to your cause, you still have to ask yourself the question, “Do I know for sure that it will handle the peak traffic?” Adding in impairments during a load test is easy using Shunra’s VE Suite or Shunra for HP Software which works seamlessly with Load Runner.

During my career I have load tested literally hundreds of different sites, and many of those sites had CDN and 3^rd party content provided by very large companies that did not hold up to the load during a stress test. Let’s face it, regardless of the size, the CDN or 3^rd party provider’s service can only scale to the level of infrastructure that is available to your site.  CDN content can fail to scale due to complex caching configurations that can get screwed up with one incorrect setting.

In more than one situation, I had a customer tell us that they were going to inform their CDN or 3^rd party providers to “be ready” for the peak traffic that our load test was going to generate. They were usually shocked when I asked, “Why?” Their answer was typically, “We need to be responsible partners and let them know anytime we are doing testing.”  Clearly, I would normally agree with that statement for many different kinds of testing, but not in the case of load testing against an established Service Level Agreement (SLA)!  Here is the bottom line, if the CDN or 3^rd Party has agreed to an SLA then that system should be ready to handle that load any time (barring timeframes specified in the SLA for maintenance windows, etc.).

In several load testing engagements, where the client let the CDN or 3^rd party know about the load test, there were “additional resources” stood up to support the load test. Surprisingly, the system scaled during those times, but failed to scale when no notice was given ahead of time. Does that sound suspicious to you? If the answer is yes, then we are on the same mental wavelength. If you encounter problems with your 3^rd party provider not scaling during your load test, and they will probably insist on “supporting” you during the next load test. Be sure that you ask your 3^rd party providers the following questions:

What did you do to support this load test Vs. the last one?

What did you change in your infrastructure that allowed it to scale successfully this time?

Did you add infrastructure or change a setting?

In one situation, I heard a 3^rd party provider tell my client that they stood up an extra bank of machines to support the testing that was happening that evening. My question to the customer was, “Does that sound strange to you?”

At the end of the day, we all need to admit to ourselves that even our monstrous 3^rd party CDN partners can slow your site down, so you really should test how your site performs to the end user and how the infrastructure scales when they do slow down. Try to find out how slow each 3^rd party provider can get before it impacts your site and ultimately the end user’s response time and build that into your Service Level Objectives and ultimately let that drive your Service Level Agreements with 3^rd party content providers.

Optimizing for mobile device cache -
http://www.yuiblog.com/blog/2010/07/12/mobile-browser-cache-limits-revisited/
http://www.stevesouders.com/blog/2010/07/12/mobile-cache-file-sizes/
http://www.browserscope.org/

Optimizing JavaFX code for mobile – http://weblogs.java.net/blog/2009/04/30/javafx-mobile-applications-performance-tuning
Mobile High Performance Presentation from Velocity – the book is also very good http://www.slideshare.net/firt/mobile-web-high-performance

Curious to hear what type of performance engineering challenges for mobile you are facing, please share…

Now that I’ve startled you out of your summer doldrums, let’s talk about how you can perform your tests most accurately. First, let’s talk about an all-too-common pitfall of not taking network latency into account when building the load test. Most teams will load test their site to ensure it will hold up to capacity, but all too often the team fails to take into account the implications that network latency will have on their load tests. As a result, testing teams will run load tests from a single location, often the same location that the Website is hosted or from within the lab itself.

If you just asked yourself, “What’s the problem with doing it that way?”, then you seriously need to read the next two paragraphs!

Unfortunately, our web sites have finite resources resulting in limited scalability. Yes, even cloud sites  occationally suffer from this affliction, here are a few examples from this year http://www.crn.com/software/225701829. Load testing without taking latency into account gives the site an unfair advantage during load tests, which does not exist in the real world. This is especially true when your internet traffic hits on Cyber Monday. Think about it this way, when your load test generators are on the same network or physically very close to the Website, then packet round trip time is less than 1 millisecond. Now contrast that with users across the internet with packet round trip times of 30 to 50 milliseconds (that’s if your user is in the same country as the web server) or even 100 milliseconds or higher going to different continents. That translates into network connections staying open for 30, 50, or 100 times longer with internet user traffic compared to your lab setting. This is a significant difference which results in longer HTTP sessions, longer page downloads, higher simultaneous HTTP and application server sessions. This can even impact database activity. Ultimately, means Internet user traffic consumes more site resources compared to testing without the correct network latency.

I’ve conducted numerous load tests in a previous role, and I have literally seen websites crumble under Internet traffic that was less than half the load tested traffic where no network emulation was used during the load testing. Now think about all the other parameters that you should be thinking about, like packet loss, jitter, and the impact that the end user’s bandwidth has on the test dynamics.

Now, let’s re-run your load test and add in end user network impairment (latency, packet loss, bandwidth, and jitter) into the mix using Shunra. Because the network latency that we just discussed now matches that of the end user, you can expect an accurate estimation of the site’s true scalability. With Shunra, you can replicate users coming from a variety of locations so you’ll be correctly replicating the latency of users from multiple geographies for even higher accuracy.

To bolster the accuracy of the network emulation, Shunra has a Network Catcher technology that records network latency and other impairments, and then allows you to use those conditions in products such as Shunra for HP Software while testing your systems. This is important because network latency changes throughout the day depending on internet traffic.  Thus internet latency at noon is not the same as it might be at 2AM, when many sites conduct their off hours load test.  With Shunra all the network guess-work is taken out of the load test process and you are left with a more accurate load test. That’s powerful! And it’s one of the reasons that I joined the team here at Shunra.

If you’re not familiar with LoadRunner in the Cloud, it’s one of HP’s latest offerings where they partner with Amazon AWS to offer HP LoadRunner in Amazon’s EC2.  The product is currently in beta and is doing very well.  To read my post about it and to get links to join the HP beta, go here.

As a long-time partner of HP, Shunra was invited to augment the HP/Amazon partnership by including Shunra for HP Software.  In our recent white paper with SAP and HP, we already showed how testing with WAN emulation is valuable for anyone testing distributed applications AND how it all works swimmingly in virtual environments, so the only hurdle was to package everything together.

That packaging is now done.  We worked with HP to build an Amazon Machine Image (AMI) that is based off of HP’s 32 bit Load Generator AMI and includes Shunra’s WAN emulation component by default, and we have defined a very fast and easy process to install our component on the Controller instance.  The Controller component installation for Shunra requires a 15MB download and approximately 90s to install.

Once the new Load Generator is approved internally by HP (in the next few days), it will be added to the list of AMIs available for the HP beta program (again, to request inclusion in the beta program, go here).  In the mean time, you can still take advantage of this offer by joining the HP beta program and contacting me directly.  After confirming your participation in the HP program, I’ll send you download and installation instructions for the Load Generator as well as the Controller components from Shunra for HP Software – both are required to use WAN emulation in your LoadRunner in the Cloud tests.

As I mentioned in my previous post, HP is waiving their license fees for this beta program.  In that light, Shunra has agreed to do the same.  The only fees you will be responsible for are from Amazon AWS.  From personal experience, I can say that the AWS usage fees are quite reasonable.

If you’re not planning to participate in this beta program, but would still like to learn more about application performance testing in the cloud, I encourage you to comment below or send me a message directly.  My contact information is listed at the bottom of this post.

My previous post about LoadRunner in the Cloud spoke about how Shunra™ for HP Software not only works well in clouds, but actually “completes the picture” for anyone conducting performance testing of applications that will be accessed remotely.    I am not alone with these statements.  SAP Labs, HP and Shunra are pleased to announce the availability of a new whitepaper, released today, June 7,  titled, “Application Performance Testing in a Virtual Environment.”

In this paper, we explain a best practice for performance testing SAP applications.  This best practice was conceived by SAP LABs and goes through four stages that take an SAP application from the initial steps of verifying functionality and accessibility, to how to tune the solution with WAN acceleration technology.  While the paper specifically focused on SAP applications, it’s clear that this best practice applies to almost any application that will be accessed remotely.

One of the key themes in the paper is virtualization.  As I have mentioned before, Shunra for HP Software already works well in clouds.  This paper took a scientific approach and compared tests using dedicated, physical machines with a completely virtual setup.  Specifically, HP LoadRunner was first run on a standalone machine, through a Shunra Appliance and then to the SAP application servers.  For comparison, HP LoadRunner with Shunra for HP was installed on virtual machines (controller plus load generators) and the SAP application servers were also installed on virtual machines.

When the six-hour load tests were run and the results were compared, the average transaction response time varied by only a few one thousandths of a second – well below any reasonable requirement for accuracy!

Whether you’re interested in using HP’s new LoadRunner in the Cloud offering, or simply trying to reap some of the many benefits of virtualization by moving your testing environment into a virtual server, this paper is a must read.

The new Virtual PC is based on new technologies that continue to evolve at an amazing space. The core consists of very powerful servers that can run dozens of sessions, each one supporting an individual “PC” with its own “memory”, “hard disk” and, naturally, CPU.

The secret sauce is a “Hardware Virtualization Layer” often called a “Hypervisor”, which is a program that allocates resources to each virtual machine, and also isolates them from its neighbors. As a result, completely separate computers are essentially created on the server hosting the virtual environment. The user, on the terminal or “virtual PC” receives only screen shots or sound bytes. All data is kept on the “virtual partition” on the server. As a result, the terminal can be a scaled-down version or an older version of a PC. However, local copies of the software are still needed on the terminals leaving the amount of actual savings debatable.

There are three main players in the field, VMware, Citrix and of course Microsoft. Each one with a particular flavor, however, they all share the same issue. At the end of the line there is a client and you must test the impact of the delivery mechanism, the network. Only testing with Shunra VE will ensure end-user performance and satisfaction.

The FCC’s tool lets you measure download and upload speeds, as well as latency and jitter (the last two are important for video and voice applications).  Nobody talks about latency and jitter and that is a shame.  For high quality video conferencing, less than 20ms of jitter is best.  Latency less than 30 ms is excellent. 

If you want to test your Internet connection, you must choose what type of consumer are you, home or commercial, including street address and zip code.

The test include upload and download speeds and latency and jitter to the service provider. Results are comparable to speedtest.net, a site that also utilizes OOKLA technology.

The FCC site also provides an Iphone application to measure network performance on a WI-FI network using a smart phone.

If your Iphone is connected to a WI-FI network, your results will reflect the speed of your broadband connection. If you disable Wi-Fi and then perform the test, the results will reflect the performance on the 3G network provided by At&T.

I believe this to be a great step to forcing greater transparency in the Internet access space and eventually will force Service providers to increase speed and lower costs.