Real-world Performance Testing Explained

I have heard every new generation of performance testers debate the same old things:

• What’s the difference between load and stress testing?
• What’s the definition of performance testing vs. load testing?
• What are all the different types of performance testing and when to conduct them?
• How is performance engineering different from performance testing?

Oddly, we rarely hear from customers the following question: “What’s the definition of real-world performance testing?”

In a few short explanations, I’d like to share my own ideas on this subject.

I’ll start by asking this – shouldn’t *all* testing be considered “real-world” testing, actually? I mean, if we spend days, weeks and months testing software and logging bugs – aren’t we assuming that those bugs (of any type) will impact end users in the real world? In this way, functional testing, unit testing, performance testing and even development itself is “real-world”. Perhaps we don’t think this way because it seems overly obvious or presumed in the nature of our work. Scott Barber recently posted a plea about making software development “Mission Focused” where I believe the mission he refers to is delivering software which will be used to impact something (for some purpose) in the real world.

Recently I’ve also seen numerous occurances of the “Real World” terminology over at uTest.com where the marketing department has gotten a hold of the “real-world” terminology and decided to use it as a competitive differentiation in their positioning.  Although they have a very compelling story about real testers in the real world conduting testing from “all over the world” I’m not sure that this is the only feasible definition we should accept; especially considering it probably came from the marketing department.  Is it good enough to just say it’s “real-world testing” just because it was done by a human being outside your company – in the crowd-sourced uTest.com ecosystem?  How is that different from the human beings who are testing software already?

Here’s the definition I would use to describe real-world testing: “any testing effort that defines a test criteria where a direct non-technical dependency is required for evaluation of the test result.” What this means to me is that if you have a test that is validating something internal to the application, and the specified pass/fail criteria does *not* include a non-technical (e.g. “human or living being”) dependency, then it is *not* a real world test.  Sure, it’s might be a perfectly valid technical test, but it’s not directly related to a component or function that will ever affect something or someone in the real world.  The operating idea is here is direct vs. indirect relation of the test case to the real world. For real-world performance testing, let’s consider the test design as: “any performance testing effort that includes test environment configuration and test execution constraints derived from the conditions of a non-technical entity which has dependency on the performance of the system under test.”  Like with mobile performance, if end-users have to wait…they will abandon and uninstall your app.

We often think of real world testing as being related to “the worst case scenario that could ever happen” in the form of a negative test case. Consider the o-rings which were cited as the indirect cause of the space shuttle disaster back in 1986.  Even though the o-rings may have been tested for their direct technical integration points or the dependencies that direct related to their functionality or performance as components, the complete test criteria perhaps didn’t include all the dependencies and conditions that would eventually affect the o-ring in the real world.  It’s not that we didn’t test the o-rings. We just didn’t conduct real-world testing of them to find exact conditions for their eminent failure.

Speaking of rocket science, I recently spoke with a very enlightened and experience architect at one of my customers who actually was a rocket scientist in a previous career. This sparked a unique conversation about the validity of arithmetical extrapolation to real-world results, which he stated [paraphrase] “was often required in the development and testing to build rockets. You couldn’t afford to test them in the real world, so we had to conduct models.” This makes sense. Rockets are expensive and executing a negative test could cause lots of blown-up rockets all over the place.

There’s also that scene in the movie Office Space where Tom Smykowski is explaining to “the Bobs” about exactly why software developers and testers are sometimes separated from the real world. Which may lead you to jump to a conclusion that people skills are a pre-requisite for conducting real-world testing.

Just a Second

1001, 1002, 1003, 1004….

That didn’t take much time, what could have happened? Actually when it comes to mobile websites, those few seconds can have a major impact on your revenue. Look what happens to mobile users when a minimal delay in response time is introduced. How many users will leave a mobile site with a 200 ms delayed response time? Not many. However, when the delay reaches 500 ms the effect is noticeable. Increase the delay to 1000 ms and the direct correlation is obvious. The higher the delay, the greater the number of users who are going to abandon ship. Does this affect your bottom line? You bet. Even worse, some of those users are never coming back.

Let’s look at Joshua Bixby’s results presented at Velocity Europe this November. Although it makes sense that users are going to desert poorly performing mobile sites and apps, most of his clients were not aware of the true impact on their KPIs. To quantify his hypothesis, he added various delays to HTML page response time to monitor the effect. With 500 ms of delay, page views and conversion rates are negatively impacted. At 1000 ms of delay, cart size is also affected.

Bounce rate, negligible at 200 ms, rises over 8% at 1000 ms. Those users may never return; Bixby pointed out that even after a few months, visitors who experienced long delays returned at lower rates. For iPad users, the correlation is even more obvious – their bounce rate increases significantly when performance degrades. He also examined the effect of delay during various steps in a mobile scenario. Bounce rate almost doubles when delay is about 2 seconds during any of the stages in the scenario.

How directly does this affect your key business metrics? There’s no question that more and more money is now spent using mobile devices. For a typical retailer just a year ago, for every $100 spent online only 50 cents was spent on a mobile. Today $7 out of every $100 spent online comes from a mobile source. With 14 times growth in just one year, the trend is obvious. Anyone selling or providing a service online no longer has a choice about whether to offer mobile access. But while a great app can improve your income, a poorly performing site can negatively impact not just your mobile, but also other online sales and even your company’s image. As users tweet and post comments about pathetic response time, the effects can quickly become viral. Will users say that the website is great but that the mobile app is terrible? No. They may just comment how the site took so long to respond that they checked out the competitor.

Mobile access is also changing. While most users are still accessing the full site, partially because many search engines point to these sites, an increasing number of users are going to the mobile site and mobile app. However, no matter how they access your information, when performance is poor users are not tolerant.

So what can you do to determine if your mobile performance is unsatisfactory? To obtain a true picture of the performance, Bixby recommends correlating performance and business metrics. The performance data includes real end-user monitoring, including data from site traffic analytics, latency and bandwidth checks. The business component includes data such as who is buying, what and how much they are purchasing, etc.

These decisions need to be taken at a policy level to have meaningful, long-term impacts to your business.  Once you commit to a company policy on performance, the next question is typically, “how do I know how my customers experience my site and how do I optimize that experience?”  That’s where Shunra comes in.  To improve customer experience on the mobile, Shunra’s Application Performance Engineering products deliver insights into application performance that include fast root cause analysis, location based SLA validation, and actionable information for improving and ensuring optimal end user experience. Ideally, you should set these targets early in the development process so that performance objectives are integrated throughout product development.

Shunra’s NetworkCatcher has an extensive library of network profiles based on millions of samples from across the globe that can be used to emulate various network conditions.  For customers with atypical networks, NetworkCatcher can measure the performance of your actual network environment.   Shunra’s PerformanceSuite is able to emulate almost any network and then provides rich reports and drill-down transaction analysis to help isolate and resolve the root causes of transaction problems in minutes. PerformanceSuite also helps determine whether any modifications to the application, network or infrastructure are required. Here’s an example of some of Shunra’s recommendations for optimization for a specific mobile site:

According to Bixby, even a 1 second delay can result in 16% decrease in customer satisfaction. The data he presented validates the assertion that mobile performance directly correlates to revenue. Shunra products help determine whether any modifications to the application, network or infrastructure are required. By testing early, adjustments can be made to performance well before your app goes live, saving time, effort and poor feedback. Testing and improving mobile performance both encourages customer satisfaction and helps to maintain a healthy bottom line.

SSL: Addressing Data Security in Application Performance Testing

Federal organizations alike expend great effort to ensure the security and integrity of their digital data. When it comes to application testing and the ability to ensure applications will perform when deployed to the real-world, it is necessary to bring real-world conditions into the test lab. But, how can this be done with confidence in data security? When we talk about bringing real-world conditions into the lab, there is a requirement to capture production network conditions, to emulate those conditions in the lab, and be able to decrypt data for analysis and reporting in the lab.

With over a decade of experience helping federal, state and local agencies address application and infrastructure performance concerns, Shunra has gained unique insight into the challenges of securely emulating real-world conditions in the test lab. Shunra’s NetworkCatcher captures real-world network conditions in secure environments by allowing SSL connections to be made via a public-private key pair. The public key is shared with the NetworkCatcher agent but the private key remains securely on the server. In the most recent edition of PerformanceSuite, Shunra introduced SSL support within the test lab. PerformanceSuite employs a private key that enables decryption and analysis of test traffic in the lab.

Federal and civilian organizations now have a way to securely discover real-world network conditions, emulate those conditions in the test lab, and analyze traffic down to the sub-component level for performance troubleshooting and optimization. To learn more about Shunra’s secure performance testing capabilities, please contact us : http://www.shunra.com/contact-us

Etsy’s Mobile App – Attack of the Clones

With the pre-holiday “how am I going to buy all those presents” panic in full swing, it was a relief to see that Etsy now has a mobile app for iOS. Considering how successful the Etsy website has been in marketing vintage and handicraft items, the accessibility of the mobile app should make finding gifts a breeze.

The Search feature made it easy to look for an obscure gift (yes, that cousin) – a Star Wars hat. So far so good, but the throughput quota was getting dangerously close to its limit before all the content loaded. Hmmmm. Typical users often leave a site after 3-4 seconds if they don’t get a response.

Since at Shunra we investigate performance issues and determine why response times are not up to snuff, Israel Nir decided to look for the cause of the tardy images. Using Shunra’s Mobile Performance Test the following transaction breakdown shows that Yoda’s chapeau appears three times. In other words, the same content is loading over and over. No wonder response time is poor!

Looking for something to keep Auntie warm was easy. She had sent a link via facebook that could be accessed in the mobile app to an item she really liked – a colorful knit hat. Transaction breakdown showed that the same image was downloaded four times, resulting in about 52 KB, 75% of it unnecessary.

Therefore, it was no surprise that Shunra’s Optimization report gave this transaction an “F” for downloading the same item more than once.

Transaction response time could be reduced for this image by a valuable second by modifying the mobile app’s behavior. For Etsy shoppers who often look at many items before selecting which ones to purchase, those cumulative seconds could result in frustrated users leaving the site. To prevent that, performing Shunra’s mobile performance test and implementing the recommendations can positively impact the shopping experience.

Thanks to Israeli Nir for the tip!

Can Mobile Performance Engineering Help Conserve Battery Life?

Can Mobile Performance Engineering Help Conserve Battery Life?

The following article builds a case for how performance engineering can help build more energy efficient mobile applications. Very interesting read, with strong reasons for why mobile performance engineering should be a critical part of mobile application development.

http://www.research.att.com/articles/featured_stories/2011_03/201102_Energy_efficient?fbid=dekBGE2Kx9k

I can picture how energy conservation will soon be added to the NFRs for mobile applications.

Enjoy,

Video over IP and Video Streaming – Managing Video Quality and Performance

I recently had the privilige to work with one of the world’s largest cable and media provider. They were challenged with managing the quality of their new adaptive streaming video player. This was such a great example to the value delivered from using Shunra on such projects, that I took the time to summarize it in this paper.

http://www.shunra.com/resources/case-studies/video-over-ip-and-video-streaming-managing-video-quality-and-performance

Enjoy,

Amichai Lesser

Mobile performance engineering rules for the iPhone

The following post was written by Israel Nir, Shunra’s head of research and analysis and Dror Vinkler, a lead developer on his team. It was developed as part of Shunra’s recent innovation around mobile performance engineering. The findings in this post became the basis for the new mobile performance engineering capabilities to be released in Shunra PerformanceSuite 7.0. Enjoy – Amichai Lesser

Web sites performance matters! It’s a well established fact that making your site load faster improves conversion rates, decreases abandonment and provides an enhanced user experience. To maintain realistic response times when connecting to websites with a mobile device such as an iPhone or an iPad, we must pay close attention to the behavior of mobile sites. The limited capabilities of such devices (weak CPU, restricted available RAM) combined with the inherent latency of mobile networks, provides further challenges in maintaining website response time.

In recent years much research focused on improving performance of websites. Companies as Yahoo, Google, AOL (and many more), and thought leaders such as Steve Souders,  Stoyan Stefanov and Maximiliano Firtman have developed tools and best practices, meant to improve websites’ load time. However, most of this research focused on desktop browsers, connected to the internet via links of relatively low latency and packet-loss rates.  We, at Shunra, decided to take another step forward, and investigate what best practices can be applied to web performance on mobile handhelds, starting with the iPhone.

Methodology

To establish which techniques  are most successful when trying to improve websites’ load time on the iPhone, we  downloaded the front pages of several sites, including all the referenced images, style sheets, scripts and other resources, and hosted them on our  local (Apache) server.  We did this to eliminate any dependency on the varying network conditions between our labs and the original web servers. When the referenced resources were loaded from domains other than the main html page, we created virtual domains on our local server and fixed the links accordingly.

Now, however, we had to recreate 3G network conditions in our own lab. Luckily, that’s exactly what Shunra products are made for. On the machine hosting the web server we installed Shunra PerformanceSuite vCat, a light weight driver able to emulate different network conditions, such as limited bandwidth, packet loss and added latency. In all of our tests we used the following parameters:

• Available bandwidth -  4096 Mb/s
• Round Trip – 150 ms
• 1% of all packets are randomly dropped

You’ll be lucky to get that on a real 3G network, but our results are even more significant when the simulated conditions are worse.

Each site was loaded on Mobile Safari one hundred times. In order to do so, we inserted a short javascript within each page that made Safari reload the page a few seconds after the OnLoad event was called. We collected the load times, modified the page itself (like combining style sheets, implementing domain sharding), and loaded it again one hundred times. We looked for changes that made the websites load faster in comparison to their original versions. Load times were measured by timing the OnLoad event, and by calculating the time difference between the first and last bytes sent by the server (as recorded in the server’s logs).

The astute reader might ask how we avoided caching on the iPhone. After all, reloading the same page when the iPhone cache is primed would be detrimental to our purpose. For this reason we defined a URL parameter that served as a counter. Each time the page was reloaded, the counter was incremented, so the page was not cached. On the server side, we rewrote all the links in the page (and in the referenced style sheets and scripts) to include the new value of the counter. That way, any referenced resource was reloaded from the server when the page was reloaded.  Don’t try this at home though; Mobile Safari has the lovely tendency to crash once it runs out of memory (which happens rather quickly for pages with many resources).

The fine print and some important caveats:

• All tests were carried on an iPhone 3GS running iOS 4.1. Results may vary as new features are added to Mobile Safari in later versions.
• We didn’t handle DNS caching, so pages load faster in our tests than they would in the real world.
• The recommended changes detailed below are not rules engraved in stone, but rather best practices that may or may not work for your site. We recommend using network emulation solutions such as Shunra PerformanceSuite, to fine tune your site’s mobile performance.

Our results

We tested the best practices detailed below on 17 websites that already had a customized (and usually lean) version for the iPhone. Our list comprised sites that one usually accesses on the go, such as travel oriented websites (hertz.com, hotels.com, united.com); news sites (espn.com, weather.com,  nbc.com) as well as others (target.com, dominos.com, urbanspoon.com). On average we obtained 17.6% improvement in load times, with a median of 18%. We had the most success with washingtonpost.com, reducing its load time by 32.6%.

We also tested our best practices for sites without an iPhone optimized version, where the improvements were even more significant. We were able to shave 44% of the loading time of britishairways.com. We reduced the load time of Shopzilla.com, which boasts a site optimized for desktop browsers (scoring a B according to ySlow), by 22%.

Best practices when developing a website for Mobile Safari

We are far from the first ones to research website performance. While working for Yahoo, Steve Souders came up with 14 rules for high performance websites, which later became the basis for YSlow. It’s no surprise that most of those rules are still valid when developing a site for Mobile Safari:

1. Use gzip. If you are going to implement only one thing out of this list, implement this one.
2. Reduce the number of HTTP requests. Remember that in mobile networks roundtrips are longer, so extra requests are costlier.
3. Put your script at the bottom. This rule and the following one allow Safari to download several resources in parallel.
4. Use domain sharding, that is, if rendering your site requires downloading of more than 4 resources (e.g. images, scripts) it’s worthwhile to put some of those under another domain. Mobile Safari is limited to downloading only 4 resources from a specific domain at the same time.
5. Configure caching. iOS 4 has no limit on cache size, so use it.
6. Avoid 404s. This is even more important on the iPhone than on desktop browsers, since upon encountering a 404, the iPhone will try to ask for the missing component a few more times before giving up. Therefore, the missing component will cause several HTTP requests.
7. Avoid redirects. Redirects are problematic for every mobile device, since the high latency means that the time it takes for the redirect to reach the device and for the new request to get back to the server is significantly longer. Try visiting msn.com from your iPhone, chances are that you’ll spend up to four seconds following redirects.

However, few of Souders’ original 14 rules are not applicable to Mobile Safari:

• Put stylesheets at the top. This is probably the most important difference between the iPhone and desktop browsers. Putting your stylesheets at the HEAD part of the HTML document has two major drawbacks on the iPhone. First, it prevents progressive rendering (which, ironically, is the whole reason it was put there in the original performance rules). Second, it blocks downloading of anything under the first javascript. So if your scripts are at the bottom, it’s not that big a deal, but it’s probably still better to avoid it.
• Choose over @import. We haven’t seen any noticeable difference in the way the iPhone handles these two ways to add a stylesheet.
• Specify a character set early. This rule was created since desktop browsers assume a default character set until one is specified, so links to components may be interpreted using the wrong character set, resulting in invalid requests and 404s. On the iPhone, however, no component is downloaded until the HEAD is received and parsed. Since a character set can only be specified in the HEAD, there’s no chance for mistakes caused by the iPhone assuming the default character set.

Finally, we identified some novel best practices that are relevant for the iPhone and (as far as we know) do not appear in prior lists. We acknowledge the fact that they are far from groundbreaking, but we still think they are worth considering when developing a website customized for the iPhone.

8. Put stylesheets at the bottom. As mentioned before, stylesheets in the HEAD prevent progressive rendering. But it’s worse than that. Nothing below the stylesheet will be rendered until the sheet is downloaded and parsed, even if it’s in the BODY.
9. Don’t put scripts in the HEAD. This is a lighter version of the original rule to put scripts at the bottom. The iPhone actually downloads components in the order it sees fit, so scripts and stylesheets are always downloaded first. As long as your scripts are above the stylesheets and not in the HEAD, they won’t block anything and will always be among the first things downloaded, so there’s no actual necessity to put them at the bottom.
10. Use HTML5 storage. The iPhone supports all three types of storage defined in HTML5: databases, local storage and manifest. All those methods can help drastically reduce the amount of data sent back and forth when loading a webpage.
11. Cache your redirects. While avoiding redirects completely could be difficult, using redirects that are cached by the iPhone could speed load time significantly the next time the user visits the site. With iOS 4.1, the iPhone only caches 301 redirects, but an updated list can be found at BrowserScope.
12. Design for progressive rendering. Some of the heavier sites that we’ve checked looked like a mess until the stylesheet was downloaded. So much so, that progressive rendering for those sights might actually confuse the user and create too much clutter. If you have a site with a lot of images, we recommend that you consider putting a block of very simple style rules inline in the HEAD section. The more complex those rules are, the bigger the HTML document will be and the longer component downloading will be blocked.
    A good compromise could be to inline only the rules that apply to the visible upper section of the site (above the fold) and concern the location of text and images.
13. Reference all images in the HTML. Some images, mainly background images, will only be referenced in the stylesheets, since they are not relevant in any other context. However, doing so means they are not downloaded until the stylesheet is downloaded. This might not be a big drawback for a site with a lot of images, since there are other things to download until the stylesheet is downloaded. But for a lean site with only a few images, this means that the stylesheet is practically “blocking” the download of those images. One solution is to place the images in a part of the HTML document which will be defined as invisible.
14. Be extra careful when pre-loading. While preloading might be useful to speed up your site, keep in mind that most users pay by the byte. So preloading a 6MB video clip, like we’ve seen when checking AOL’s homepage, might not be a good idea.
15. Strip all components that are not supported. Consider what kind of components are not supported by the iPhone (favicon, flash, etc.) and avoid using those.
16. Increase your server’s keep-alive timeout. It appears that the iPhone sometimes has a bug, causing it to use ports that were closed by the server. We haven’t been able to pinpoint what causes this bug, but it seems as if ports that were closed by the server less than 2-3 seconds ago are in danger of being used by the iPhone. This, of course, causes resets by the server and retransmits by the iPhone, and it can happen to any request done after the onload event (e.g. AJAX requests). Increasing the time until the server closes a port could prevent this, but it has other side effects (increased load on the server) that should be considered.

Being slow is not hip!

The following post was written by Israel Nir, Shunra’s head of research and analysis. It was developed as part of Shunra’s recent innovation around mobile performance engineering, which is being released soon as part of Shunra PerformanceSuite 7.0 and other related products and services. This new innovation provides capabilities for testing the performance of mobile applications under any mobile network conditions, analyze the performance and provide performance optimization and remediation suggestions, all in a matter of minutes. This post provides a sneak peek into these capabilities in the context of analyzing Hipmunk’s mobile service. Stay tuned for more on Shunra’s recent innovation around mobile performance engineering. – Amichai

I love Hipmunk.

If you didn’t get to play with it yet, Hipmunk is an amazing web service, making the dreaded task of searching for a flight a real joy. Hipmunk’s well designed chart-like user interface allows you to quickly compare flights according to their price, departure and arrival times and agony (which is a combination of the other factors).

Actually, quickly is not the right adverb to describe Hipmunk. It takes the service quite a few seconds to come up with the flight results. Sure, some of this sluggishness is due to the really laborious work on the server side, finding all the available flights and ranking them. However, having a less than optimal design for working over the network plays a big role in that as well, and luckily it’s easier to tackle this kind of issues than the server side ones. This is especially apparent when looking at Hipmunk’s recently released iPhone app. What a great opportunity to launch a dev version of Shunra’s up and coming Performance Suite 7.0 software and use some of its new analysis capabilities to drill down into the performance of this mobile application. My setup is simple, yet powerful, it includes a WiFi hotspot connected to a Shunra PerformanceSuite appliance that can play back any mobile network conditions, as well as capture the mobile transactions that are tested through it. My iPhone is connected to this WiFi network, yet it can experience any mobile network conditions introduced by the PerformanceSuite appliance. Finally, Shunra’s transaction manager allows me to demark the beginning and end of each mobile transaction, so my analysis can be focused and aware of the business logic of the application. The following chart shows one of the analysis screens provided when testing the Flight Search transaction.

There are quite a few artifacts to notice. You cannot see it in the diagram above, but when the app is launched, it first does a DNS query for hipmunk.com, and then for www.hipmunk.com. As far as I have seen, the app only issues HTTP requests to www.hipmunk.com so the first name resolution is redundant. You might laugh it off, after all, who cares about a redundant DNS query, but I find it as a sign for things to come, and boy do they come when searching for a flight.

Once you enter the flight details and press the “Search for flights” button, the app sends a POST request to the server. There’s a small room for improvement here, using a GET instead of a POST. While a GET request is (usually) contained in one packet, a POST requires two packets. This (almost) doubles your chances for a packet loss, which are already high in cellular networks.

Surprisingly, the server doesn’t reply with the results of the search request, but rather with a search identifier, which the app later uses to query for the actual results. I believe this procedure was designed to let the server calculate the results without keeping the session alive (though, interestingly, the TCP connection is kept alive). This however requires the client application to guess when to ask the server for actual results. It does so six seconds after the server’s response comes in (marked in the diagram as “client waiting”), which is a bit of a waste if the server successfully compiled the results faster than that.

Unfortunately, six seconds are also one second more than the server’s keep alive interval. By the time the app sends its request for the actual results, the server already terminated the TCP connection. This requires the client app to open a new TCP connection, taking an extra roundtrip. As a matter of fact, two roundtrips are wasted, since the client app tries to send the second request on the terminated TCP connection and only once that fails (after a roundtrip), it established the new TCP connection.

Finally, the client application queries the server for the actual flight results (again, over POST), and gets them back in a bplist file. A bplist file is an XML document that was saved in a binary format in order to save space (good!) and processing time (very good!). Hipmunk’s server also takes care of compressing the file before sending it to the client, and thus the once one megabyte long XML document, now weighs no more than 150KB. But wait a minute – why should the results be 1 megabyte long in the first place? Isn’t it too much data for the iPhone to display? Well, it seems to me the results contain a lot of redundant data, like airports’ latitude and longitude coordinates, or having the price of each flight specified three times. Moreover, the results contain the details of more than 100 flights, even though you can only fit eight flights at once on the iPhone’s screen. I believe that a better (and quicker) user experience would be to download and display the details of the few first flights, and to download the other flights in the background (deferred loading).

To add insult to injury, compressing the original XML document (without converting it to a binary format first), results with 75K bytes of data, half of what we get by compressing the bplist file. Now, according this blog post by the app developer, processing the textual (not binary) XML file was taking too long – probably he’s right, processing a 1MB XML file is quite a task, but he missed the network effect of his choice. Making the XML smaller, with fewer entries and then compressing it should be a better solution. If all else fails, I think the guys at Hipmunk can come with a better binary format than bplist.

To sum it up, in my humble opinion, Hipmunk can make its iPhone app run much faster with the following simple changes:

• Fetch fewer results. Hipmunk is known for making the task of choosing a flight simpler, so limiting our choice to 50 good options, instead of 100 should be easy enough.
• Improve the results file format, remove redundancies, use a compressed XML file with fewer results, while lazy loading the rest.
• Query the server sooner for the results, after let’s say 3 seconds, why wait? If the server completes its processing sooner. Better yet, if possible, serve the partial results as they become available.
• Try to eliminate the two stage process of fetching the results, or at the very least, extend the server’s keep alive duration to support this process.
• Eliminate unnecessary DNS queries, and try using GETs instead of POSTs when possible.

Hipmunk is a really cool service, implementing some of the above will make it friendlier and more responsive.

Do you have a mobile service that you would like us to analyze? Send us a comment, we would love to take a look at it.

3rd Party System Scalability? – “How Slow Can They Go”

Is your site dependent on performance of 3^rd party content providers? Don’t say no too quickly! Most sites have 3^rd party content in one way or another. Let’s talk about 3^rd party SLA’s and the notion of testing your site so that you know how slow your 3^rd party content can get before it significantly slows your site down.

You have probably encountered a situation where pre-launch load tests did not correlate with the way it performed in production.  There are a lot of reasons why this can happen, but for now, I want to focus on 3^rd party content providers and Content Delivery Networks (CDN).  Almost everyone running a business today has some part of their application or Web presence farmed out to a 3^rd party.  Think about what happens when you go online to buy gifts for the upcoming Holidays.

As you browse for that perfect tie for your dad, the site is likely to be serving static content from Akamai or Limelight, adds from 3^rd party providers, and a shopping cart that may not be hosted by the company you are dealing with.  When you actually purchase that Tickle-me Elmo for your niece, the site processes your credit card by interacting with a financial institution and provides shipping tracking numbers by interacting with a major shipping company behind the scenes.  All that typically happens as you wait for the page to return in a few seconds if all goes as it should, but any of those 3^rd party providers have a strong likelihood to disrupt the page load and make the site look like it is not responding.

Increased delays from these and other 3^rd party providers do happen and you should definitely test for those scenarios, but how do you do that when you really do not have control over those sites. You definitely should perform load tests on the site as a whole to ensure that each 3^rd party provider will scale to your targeted peak traffic and perform within their SLA as expected. In addition, you should think about how slow your 3^rd party providers can get without affecting the performance of your site.

“How Slow Can They Go?” Yes, a corny statement, but to the point! You definitely need to identify how slow your 3^rd party providers can go before it starts to affect the performance of your sites.  The real dilemma is replicating the traffic from the 3^rd party site, which in the case of financial transactions can be very difficult plus your team can waste a great deal of time trying to do this. I’m suggesting that you don’t try to replicate that traffic, instead use the actual 3^rd party traffic from the 3^rd party site during the test. Here’s the catch, you impair the traffic coming from the 3^rd party site real time. Simply single out that traffic in your usability studies using a tool like Shunra’s VE Desktop to measure performance of the site with the 3^rd party adds impaired or by slowing down the response from your credit card processing partner using a Shunra VE Network Appliance on your backend systems during a maintenance window.

Now think about scalability load testing for a minute. I recommend that you do not implicitly trust 3^rd party and Content Delivery Network (CDN) provider’s scalability estimates and guarantees. You are probably scratching your head thinking that your CDN’s and other 3^rd party content providers have massive server and network infrastructures that are impossible to overload. While it is true that the major 3^rd party providers have massive infrastructures, the real question to ask yourself is this, “How much of that infrastructure is your content riding on?” Let’s just say that you actually do know how much of their infrastructure is dedicated to your cause, you still have to ask yourself the question, “Do I know for sure that it will handle the peak traffic?” Adding in impairments during a load test is easy using Shunra’s VE Suite or Shunra for HP Software which works seamlessly with Load Runner.

During my career I have load tested literally hundreds of different sites, and many of those sites had CDN and 3^rd party content provided by very large companies that did not hold up to the load during a stress test. Let’s face it, regardless of the size, the CDN or 3^rd party provider’s service can only scale to the level of infrastructure that is available to your site.  CDN content can fail to scale due to complex caching configurations that can get screwed up with one incorrect setting.

In more than one situation, I had a customer tell us that they were going to inform their CDN or 3^rd party providers to “be ready” for the peak traffic that our load test was going to generate. They were usually shocked when I asked, “Why?” Their answer was typically, “We need to be responsible partners and let them know anytime we are doing testing.”  Clearly, I would normally agree with that statement for many different kinds of testing, but not in the case of load testing against an established Service Level Agreement (SLA)!  Here is the bottom line, if the CDN or 3^rd Party has agreed to an SLA then that system should be ready to handle that load any time (barring timeframes specified in the SLA for maintenance windows, etc.).

In several load testing engagements, where the client let the CDN or 3^rd party know about the load test, there were “additional resources” stood up to support the load test. Surprisingly, the system scaled during those times, but failed to scale when no notice was given ahead of time. Does that sound suspicious to you? If the answer is yes, then we are on the same mental wavelength. If you encounter problems with your 3^rd party provider not scaling during your load test, and they will probably insist on “supporting” you during the next load test. Be sure that you ask your 3^rd party providers the following questions:

What did you do to support this load test Vs. the last one?

What did you change in your infrastructure that allowed it to scale successfully this time?

Did you add infrastructure or change a setting?

In one situation, I heard a 3^rd party provider tell my client that they stood up an extra bank of machines to support the testing that was happening that evening. My question to the customer was, “Does that sound strange to you?”

At the end of the day, we all need to admit to ourselves that even our monstrous 3^rd party CDN partners can slow your site down, so you really should test how your site performs to the end user and how the infrastructure scales when they do slow down. Try to find out how slow each 3^rd party provider can get before it impacts your site and ultimately the end user’s response time and build that into your Service Level Objectives and ultimately let that drive your Service Level Agreements with 3^rd party content providers.

Tuning applications and web sites for mobile users

More and more of my clients recently asked me about performance engineering best practices for mobile applications. This came as no surprise as we observe the paradigm shift represented by more and more consumers performing more and more of their daily tasks via a mobile device. Here at Shunra we have been working with a best practice methodology for performance testing and optimizing mobile applications, which I will share in the near future, but in the mean time I wanted to share some of the useful resources that we incorporated into our methodology:

Optimizing for mobile device cache -
http://www.yuiblog.com/blog/2010/07/12/mobile-browser-cache-limits-revisited/
http://www.stevesouders.com/blog/2010/07/12/mobile-cache-file-sizes/
http://www.browserscope.org/

Optimizing JavaFX code for mobile – http://weblogs.java.net/blog/2009/04/30/javafx-mobile-applications-performance-tuning
Mobile High Performance Presentation from Velocity – the book is also very good http://www.slideshare.net/firt/mobile-web-high-performance

Curious to hear what type of performance engineering challenges for mobile you are facing, please share…