To encourage participation by attendees and a collaborative atmosphere, we held a few interactive polls throughout the webinar and gathered the results to share with you. In addition to sharing the recording, slides and poll results, we want to take a moment to reflect on the poll responses and solicit your feedback to this post.

The first poll combined three (3) questions designed to capture the OS, Connection Type, and Signal Strength our participants had at the start of our webinar. Our audience’s responses demonstrate some of the complexities mobile performance testing can introduce. For example, with just our audience, a testing team would have to plan for conservatively 80 different potential test cases to execute, covering 4 operating systems, 4 connection types, and 5 signal strengths. And by the way, two of these (Connection Type & Signal Strength) are constantly changing due to the nature of mobile.

We found it interesting that nearly 80% of the participants were either on an Android or iOS device, a bit higher than what we expected but certainly consistent with market trends. As for connection types being 58% 3G and 25% WiFi, this was consistent with our expectations. Signal strength came back a bit higher than expected, yet this was likely due to the fact that most people were stationary on the webinar.

1. What operating system does your mobile device use?

2. What is your connection type at this time?

3. How many signal bars do you have?

Our next poll delved a bit deeper into what options should be considered when testing mobile. Please note we did not specify mobile websites or apps; nor native, web, or hybrid app types. Participants had the opportunity to select as many of the listed considerations as they desired, and many did with >50% selecting all five (5).

Prior to the webinar, we thought only a small percentage (5-10%) would select all five (5) options, given most enterprise organizations do not recognize the significant performance risk all of these considerations play, not to mention the potential negative impact of these risks to their brand and customers.

4. When testing mobile, which options should be considered?

The third and final poll focused on where mobilization was being driven from within the organization. The results were not a huge surprise.We expected ‘Line of Business’ to be the majority of responses. The fact that ‘IT’ came back at 20% was the shocker… we think perhaps this was due to a reactive approach in IT bred by following ‘Line of Business’ requests for several years.

Of course, I do like the fact that we left “CUSTOMER” off the list of options, and only 6% of the participants responded with ‘Other’. I am really hopeful that more companies are being driven to deliver highly performing mobile applications to the market by their customers…ultimately, this is the end user many of us serve, and consumer demand for products and services is a significant business driver.

5. Who is driving mobilization at your company?

In closing, we enjoyed preparing for this webinar, sharing relevant and timely information, and describing key best practices. Hopefully, we’ve enabled you with a few key takeaways that you can apply in your career now and in the future. We hear stories daily of how best practices are not being implemented, resulting in poor performing mobile applications, driving customers away and reducing revenue…not to mention the poor reviews / feedback and negative impact of social media on organizations. Please take the slides, recorded webinar, and the results from the polls and share them with your peers. Also, we would appreciate your thoughts or comments within this blog; let us know how you would respond to the polls, what trends are you seeing in mobile, and what advice you would lend to mobile performance best practices.

• 100% global availability
• 3 second page load (first view) in the US and the UK (their main customer base)
• Best effort page load time in the Rest of the World

Challenge

The client, an innovative software firm in a highly technical sphere, had specific ideas about design, content and user experience. Their website was based on several systems, technologies, and information that needed to be centrally accessed. This structure included advanced form technologies for tracking, internal software platforms, CRM and content aggregation from external informational sites. Content included podcasts, blog posts, RSS feeds, white papers, case studies, streaming video, and traditional online content. A flexible design was required to accommodate large amounts of information and pages, while providing a targeted impact to the visitor. The long-term goal included a self-sustainable website that the client could maintain independently.

Project scopes and timelines were defined, and development began. As the project progressed, the developers shared iterative builds of the site with the client. Unleashed Technologies had informally spot-checked the site using smartphones their developers owned. However, that approach to testing was not comprehensive or reliable enough for the client:

• The network performance was not controllable. The speed and reliability of the network was dependent on the operator and volume of other subscribers using the network at the same time. This meant that worst, typical and best case comparison testing wasn’t possible. Also, when issues were found, they were very difficult to reproduce.
• Using phones with a live network prohibited developers from using most of their testing tools to identify and fix problems.

As a best practice, Unleashed Technologies had been using ySlow* throughout their development process. But the data was limited – by simply using a phone on the live network, the developers didn’t have access to the packet traces and therefore had no capability to analyze data directly from the mobile connection. It was impossible to know if changes they were making were helping or hurting performance.

To gain additional insight into performance, Unleashed Technologies leveraged Shunra’s Mobile Performance Test on the development site and shortly afterwards were examining concrete results and recommendations. The report compared performance on various types of mobile networks to a baseline WiFi connection. From the results it was obvious that improvements were required and precisely which improvements should be considered first.

Here’s one example of how the application performance improvements were addressed based on the Mobile Application Report. For an initial round of optimizations, the theory was that reducing the amount of data transferred and the number of requests needed would improve the application performance. This was validated using:

• Compression, partially using the Smush.it application to reduce size of other images by 58% (on average)
• CSS sprites: Combined the background images into a single image and used the CSS background-image and background-position properties to display the image segment
• CSS/JavaScript optimizations: The CSS and JavaScript were already being combined into a single file, so they focused on shrinking the size of that file. Then they minified some of the JavaScript using Crockford’s jsmin algorithm and confirmed that all CSS was being properly compressed.
• Delay-loaded the three banner images by using a temporary blank placeholder loaded inline using the data scheme, so that other files could load first.
• Offload theme images to the Content Delivery Network (CDN) to increase the number of parallel downloads.

Additional best practices can be found in Shunra’s blog Mobile performance engineering rules for the iPhone.

Although valuable gains in performance had been obtained using Shunra’s Mobile Performance Test, Unleashed Technologies knew they could deliver additional value and meet their SLOs by working with Shunra to leverage Shunra vCat* for demographic testing of the sessions that a user would experience. vCat’s Timing Transaction demonstrated a typical flow for a user browsing through the mobile site, and vCat’s mobile profiles, such as 3G, 4G, etc. to check performance using the client’s typical locations and metrics. Then vCat Analyt Mobile Performance Testics provided packet capture and detailed analysis.

Shunra’s vCat network virtualization capabilities were utilized between the third and fourth site iterations to provide insight into the critical metrics. Two profiles compared application performance differences when accessed over a “HSDPA 3G Weak” and a “HSDPA 3G Strong” connection. Then Shunra’s Analytics provided detailed breakdown of the statistics. When running with an optimistic mobile network (HSDPA Strong, an enhanced mobile 3G protocol), the home page still averaged about 20 seconds to fully load. Some images appeared after 5-6 seconds, but they weren’t functional for a few more seconds. However, after the recommendations from Shunra’s HTTP Analysis were implemented for the next iteration, the mobile home page load times were dramatically reduced:

The results on the weak connection were even more dramatic, where the home page load time improved from 70 to 20 seconds. This screen shot shows the data from the mobile site’s home page loading over a strong link, indicating how many resources are downloaded simultaneously:

Solution

Shunra’s Mobile Performance Test* reports delivered initial data and suggestions that were quickly implemented. Then Shunra’s vCat emulation and analytics were utilized to provide comprehensive in-depth testing, analysis and customized proposals. Using Shunra’s reports both parties were able to have quick, constructive conversations on performance-related topics that would not have been exposed unless there was a failure in production. In some cases, the optimization reports gave suggestions that had been intentionally ignored for design or business requirements. In other cases, the reports highlighted issues that had not been considered. And the report was used to validate and optimize quick course corrections at minimal cost.

The Bottom Line

By integrating application performance objectives early in the development of the mobile site, Unleashed Technologies took a proactive approach to ensuring its client’s success. With Shunra’s capabilities and a sound strategy to ensure performance, Unleashed Technologies positioned site performance as a top priority and was able to provide detailed measurements to validate their efforts. Using the results of Shunra’s in-depth analysis and optimizations, the mobile site development team shortened the project timeline and achieved performance level goals.

Shunra’s Mobile Application Performance Engineering methodology provides a proactive approach to mobile application performance that incorporates planning, testing and optimizing performance. This methodology demonstrates the winning combination of how Shunra technology can help any company deploy mobile applications in a unique and measurable way.

A Special thanks to Marty Brandwin, VP of Marketing at Shunra, Dave Berg, Senior Director of Product Management at Shunra, and to Unleashed Technologies for their help with this article.

In a recent Shunra webinar, attendees were asked which IT projects were up front for the coming year. No surprise, the leaders are virtualization, mobile app deployment and cloud migration. In other words, in addition to mobility, companies continue to focus on the consolidation of resources through virtualization of all types. Virtualization does not just mean the Cloud; it also refers to all methods that centralize administrative tasks while improving scalability and workloads. These include network, storage and server consolidations.

These results, and their impact on end user experience, align with Gary’s first projection:

#1 End user patience will continue to become shorter and shorter and thinner and thinner
Whereas just a year or two ago, an SLA of 6-7 seconds response time was par for the course, 3 seconds is now the norm for just about everything. Companies that have already tuned their applications to meet the 6-7 second goal are now required to adapt to the new benchmark of 3 seconds, which is what Gary is referring to as “thinner and thinner.” As Shunra’s Performance team reports, when determining performance goals, users now have zero tolerance for repeat work and recurring clicks. However, many business apps have not caught on and require repeat actions. In addition, users now expect instant confirmation that their work has completed successfully. For more ideas about designing business processes, see Gary’s 11th prediction.

#2 IT spending will continue to recover this year despite the Presidential election
Although election years are traditionally cautious (because of a corporate fear that a new administration may negatively impact our business, influence capital tax rates, etc.) this year should see the IT sector continue to grow. A different balance of spending will occur as EAAS -’Everything as a Service’ – is promised and ‘deliver anything anytime’ becomes the norm. This type of growth requires more external resources.

One leading indicator is the hiring of IT professionals. Despite the overall unemployment rate in the US over 8%, it’s dramatically lower in the IT sector. The talent is getting scooped up. However, more of these IT expenditures may be on non-US soil as multi-national companies avoid high taxes from revenue generated abroad as opposed to what would be due if this income is spent in the US.

#3 Many mobile apps will crash and burn because they have not undergone adequate design and performance testing
The consumption of mobile devices has exploded – it’s now the norm, not the trend. It’s hard to believe that only two years ago tablets were associated more with Moses than Steve. With the expectation that new devices and features have to keep up a dizzying pace, many apps are released well before they have undergone appropriate design and testing cycles with performance considerations. ‘Just get it out’ is the motto, but the impact on the back-end systems is not being properly assessed.

While many new apps look great on the client side, if the effect of mobile sessions on the back-end resources is not properly taken into consideration, those super cool graphics are not going to make up for slow response time. In fact, almost half of Shunra’s inbound requests relate to troubleshooting failed applications.

Mobile is driving both new and traditional applications to agile and quick development cycles, but cutting corners on performance can result in disastrous post-production results. Even shifting 10-15% of enterprise users on a specific app to mobile can overwhelm the back-end so that the app will perform slowly for all users. Different memory signatures and data throughput signatures are made on mobile apps that must be taken into account during mobile deployment.

#4 New opportunities are being driven by the rise of mCommerce
We just finished an interesting holiday shopping season. Overall retail sales were weak in early December, just slightly above the previous year’s results, and projections for December show a lower than expected rise in sales. However, while store traffic is decreasing, mobile sales are skyrocketing. PayPal reported a 397% increase in consumer use of PayPal Mobile on Cyber Monday; JoS. A. Bank’s mobile checkout increased more than 3000% on Black Friday; and Boston-based Rue La La reported an increase in mobile sales from 2% last January to 33% of all sales in this past holiday shopping season; and an estimated 87% of tablet owners did some holiday shopping from their mobile device. (This data was compiled and published by Mobile Commerce Daily, a daily trade publication dedicated to mobile commerce.)

Mobile Commerce is making significant inroads and Gary predicts that 2012 will be year that indoor Location Based Services (LBS) takes off. Developers are figuring out how to combine the carrier technologies, Wifi radio and GPS that are part of your smartphone but not available reliably indoors. Outdoors, an app such as Google Earth will first use the cell tower location, then the Wifi hotspots, and then GPS to pinpoint your bearings. Indoors, this approach is not always effective so companies are now working on intercepting the cell tower signal and using a series of repeaters to provide a signal.

In addition, red-line scanning and price search, which most stores now see as a threat, will become a business opportunity. The combination of scanning, location based services and new pricing strategies are set to have a big impact on the retail experience. The business upside is that using geo-correlated behavior, retailers can now provide pricing and promotion from within the store, enabling the so-called “bricks and clicks” business model. For example, when a shopper is scanning a TV model’s stats, the retailer can recommend a Blu-ray package. The physical store can also promote the ways that their shopping experience is more positive in ways such as availability, location, convenience, service, etc.

#5 Enterprise IT is forced to compete for delivery of service
In the good old days, the CIO was ‘The Expert’ on technology issues. When “Everything” is being delivered “as a service”, including platform as a service, software as a service and cloud-based services, the CIO no longer controls the infrastructure, or the expertise about the infrastructure.

Consolidation of IT resources is occurring among bigger global systems integrators who are taking a bigger slice of the Fortune 500 accounts. The CIO must now look at outside resources and consult with specialized personnel, and become an internal Systems Integrator.

#6 Your digital strategy will affect your ability to recruit and retain the right staff
With low unemployment in the IT sector, attracting top talent is more than challenging. One of the top three issues for a new hire in 2012, beyond salary and health benefits, may include IT resources that address the employee’s work/life balance and allow for self-provisioning.

IT resources require continuous adjustment when millions of employees now travel and work offsite and require the flexibility of different devices, connectivity and means of access to core systems. How does a company track the value of the IT investments, such as firewall improvements or the changes required in CRM to accommodate tablets? When creating a budget that aligns with the company’s digital strategy, consider the aim of the improvement. Is it increased productivity, which ideally can be seen on a daily basis, or cost reduction?

#7 Tech companies that are VC-backed will feel pressure to do transactions faster
When Venture Capital firms makes an investment, one consideration is early ROI versus long term investment. In the past few years, a number of the VC dollars have come from institutional investors, such as pension funds. Unfortunately, many of these funds have been burned by the real estate crashes, the economic turndown and various scandals.

Whereas a decade ago patience may have been considered a virtue, today there’s limited tolerance for returns. In fact a ‘sure thing’ today may be worth more than double that a year from now, but the “sure thing” is preferred as it is less risky and provides immediate return versus waiting. Just as end users now expect an application to react in half the response time, VCs are looking for results and ROI sooner than ever before. Startups and technology must prove itself sooner, or fail sooner.

For those investing in new technologies, because of the abundance of choice, it may be harder to predict whether a new technology is going to make it. Therefore, Gary advises to pick your partners wisely, taking into account the strength of customer referrals, history, and financial condition of the company before signing on the dotted line.

#8 IT Capital investment will continue to decrease, while consumption of resources will continue to increase
The continued pressure to virtualize desktop, servers and just about everything else, and to migrate to the cloud and use less expensive platforms (such as a tablet vs. a more expensive laptop), means that almost every end-user wants more than last year – more applications, resources and more access to resources. Startups and new companies may even go straight to the Cloud to consolidate facilities and save on energy costs. This has placed a lot of pressure on low cost facilities to deliver and may bring unrealistic expectations that have not taken operational and other costs into account.

Some companies even have a BYO support strategy – bring your own device. Although it may save the enterprise in capital costs, the support costs will rise as the IT team faces the challenging task of supporting all these various devices and configurations.

#9 Cloud services will prove an ROI in 2012
Although no studies of real cost reduction for cloud ROI are yet available, Gary predicts that 2012 will finally be the year that demonstrates ROI for Cloud investments. Companies that have kept their figures in-house are now more likely to reveal stats regarding the backend impact, especially as pressure mounts to prove the effectiveness of prior years’ investments. Useful data about the hidden costs of failure, such are rewriting apps, will then become public.

IT cannot just toss legacy applications into the Cloud and expect them to work perfectly where they have no real control. They must now design for Cloud, test for Cloud, and optimize for Cloud. The CIO now must assume the role of systems integrator and must have ‘the fixers’ lined up should a disaster occur.

#10 Attention on infrastructure will increase while inversely decreasing on applications
We are now experiencing a paradigm shift to virtualization – from hardware to software to services. In addition, consumerized end-user devices necessitate a continued investment in infrastructure. It will probably take a long time for enterprise apps to catch up in terms of required attention and investment. The silver lining is that opportunity is created for outsourcing (and crowdsourcing): IT is looking for vendors who can help. For example, through its community, SAP now has hundreds of mobile apps developed in the past year. SAP users, instead of waiting for SAP or their own IT department, can now to tap via 3rd party mobile applications into legacy systems and access specific functions.

#11 bonus prediction – Reduced form factor providing only the “essential engagement” will become more common.
With so much happening in the IT sphere, it was a real challenge to select only 10 predictions. Gary was not actually successful in this endeavor. Therefore he gives us one of his most interesting observations: Enterprise apps are going to be reexamined via a mobile device lens. In other words, how we design and use mobile devices is going to govern how we design and use almost all other applications!

Careful attention will be paid to those actions that you or your customers are performing on a regular basis and how data is accessed. When dealing with complex systems, such as a CRM application or a business process, just navigating through all the various functionality could take over 30 minutes. But a sales team member most frequently accesses a CRM application only to update customer records and contacts, set meetings and look at the probability of closing a sale. The 80/20 rule applies; about 80% of the team is utilizing about 20% of the available functionality.

That’s where the application and device form factor can impact and improve productivity, by narrowing selections to frequently used actions. This requires analysis of what the user needs to complete his/her tasks and how best to present the information. In a mobile app this analysis is crucial, as each action and screen in the workspace is highly valuable real estate. Therefore designers and developers have to determine the essential actions, flow and optimal presentation. As users continue to consume smartly designed mobile apps, they will no longer accept cumbersome enterprise functionality. To optimize the user experience, each action and state presented should be one that is repeatable and modularized, and this must also apply to other corporate experiences.

• What’s the difference between load and stress testing?
• What’s the definition of performance testing vs. load testing?
• What are all the different types of performance testing and when to conduct them?
• How is performance engineering different from performance testing?

Oddly, we rarely hear from customers the following question: “What’s the definition of real-world performance testing?”

In a few short explanations, I’d like to share my own ideas on this subject.

I’ll start by asking this – shouldn’t *all* testing be considered “real-world” testing, actually? I mean, if we spend days, weeks and months testing software and logging bugs – aren’t we assuming that those bugs (of any type) will impact end users in the real world? In this way, functional testing, unit testing, performance testing and even development itself is “real-world”. Perhaps we don’t think this way because it seems overly obvious or presumed in the nature of our work. Scott Barber recently posted a plea about making software development “Mission Focused” where I believe the mission he refers to is delivering software which will be used to impact something (for some purpose) in the real world.

Recently I’ve also seen numerous occurances of the “Real World” terminology over at uTest.com where the marketing department has gotten a hold of the “real-world” terminology and decided to use it as a competitive differentiation in their positioning.  Although they have a very compelling story about real testers in the real world conduting testing from “all over the world” I’m not sure that this is the only feasible definition we should accept; especially considering it probably came from the marketing department.  Is it good enough to just say it’s “real-world testing” just because it was done by a human being outside your company – in the crowd-sourced uTest.com ecosystem?  How is that different from the human beings who are testing software already?

Here’s the definition I would use to describe real-world testing: “any testing effort that defines a test criteria where a direct non-technical dependency is required for evaluation of the test result.” What this means to me is that if you have a test that is validating something internal to the application, and the specified pass/fail criteria does *not* include a non-technical (e.g. “human or living being”) dependency, then it is *not* a real world test.  Sure, it’s might be a perfectly valid technical test, but it’s not directly related to a component or function that will ever affect something or someone in the real world.  The operating idea is here is direct vs. indirect relation of the test case to the real world. For real-world performance testing, let’s consider the test design as: “any performance testing effort that includes test environment configuration and test execution constraints derived from the conditions of a non-technical entity which has dependency on the performance of the system under test.”  Like with mobile performance, if end-users have to wait…they will abandon and uninstall your app.

We often think of real world testing as being related to “the worst case scenario that could ever happen” in the form of a negative test case. Consider the o-rings which were cited as the indirect cause of the space shuttle disaster back in 1986.  Even though the o-rings may have been tested for their direct technical integration points or the dependencies that direct related to their functionality or performance as components, the complete test criteria perhaps didn’t include all the dependencies and conditions that would eventually affect the o-ring in the real world.  It’s not that we didn’t test the o-rings. We just didn’t conduct real-world testing of them to find exact conditions for their eminent failure.

Speaking of rocket science, I recently spoke with a very enlightened and experience architect at one of my customers who actually was a rocket scientist in a previous career. This sparked a unique conversation about the validity of arithmetical extrapolation to real-world results, which he stated [paraphrase] “was often required in the development and testing to build rockets. You couldn’t afford to test them in the real world, so we had to conduct models.” This makes sense. Rockets are expensive and executing a negative test could cause lots of blown-up rockets all over the place.

There’s also that scene in the movie Office Space where Tom Smykowski is explaining to “the Bobs” about exactly why software developers and testers are sometimes separated from the real world. Which may lead you to jump to a conclusion that people skills are a pre-requisite for conducting real-world testing.

That didn’t take much time, what could have happened? Actually when it comes to mobile websites, those few seconds can have a major impact on your revenue. Look what happens to mobile users when a minimal delay in response time is introduced. How many users will leave a mobile site with a 200 ms delayed response time? Not many. However, when the delay reaches 500 ms the effect is noticeable. Increase the delay to 1000 ms and the direct correlation is obvious. The higher the delay, the greater the number of users who are going to abandon ship. Does this affect your bottom line? You bet. Even worse, some of those users are never coming back.

Let’s look at Joshua Bixby’s results presented at Velocity Europe this November. Although it makes sense that users are going to desert poorly performing mobile sites and apps, most of his clients were not aware of the true impact on their KPIs. To quantify his hypothesis, he added various delays to HTML page response time to monitor the effect. With 500 ms of delay, page views and conversion rates are negatively impacted. At 1000 ms of delay, cart size is also affected.

Bounce rate, negligible at 200 ms, rises over 8% at 1000 ms. Those users may never return; Bixby pointed out that even after a few months, visitors who experienced long delays returned at lower rates. For iPad users, the correlation is even more obvious – their bounce rate increases significantly when performance degrades. He also examined the effect of delay during various steps in a mobile scenario. Bounce rate almost doubles when delay is about 2 seconds during any of the stages in the scenario.

How directly does this affect your key business metrics? There’s no question that more and more money is now spent using mobile devices. For a typical retailer just a year ago, for every $100 spent online only 50 cents was spent on a mobile. Today $7 out of every $100 spent online comes from a mobile source. With 14 times growth in just one year, the trend is obvious. Anyone selling or providing a service online no longer has a choice about whether to offer mobile access. But while a great app can improve your income, a poorly performing site can negatively impact not just your mobile, but also other online sales and even your company’s image. As users tweet and post comments about pathetic response time, the effects can quickly become viral. Will users say that the website is great but that the mobile app is terrible? No. They may just comment how the site took so long to respond that they checked out the competitor.

Mobile access is also changing. While most users are still accessing the full site, partially because many search engines point to these sites, an increasing number of users are going to the mobile site and mobile app. However, no matter how they access your information, when performance is poor users are not tolerant.

So what can you do to determine if your mobile performance is unsatisfactory? To obtain a true picture of the performance, Bixby recommends correlating performance and business metrics. The performance data includes real end-user monitoring, including data from site traffic analytics, latency and bandwidth checks. The business component includes data such as who is buying, what and how much they are purchasing, etc.

These decisions need to be taken at a policy level to have meaningful, long-term impacts to your business.  Once you commit to a company policy on performance, the next question is typically, “how do I know how my customers experience my site and how do I optimize that experience?”  That’s where Shunra comes in.  To improve customer experience on the mobile, Shunra’s Application Performance Engineering products deliver insights into application performance that include fast root cause analysis, location based SLA validation, and actionable information for improving and ensuring optimal end user experience. Ideally, you should set these targets early in the development process so that performance objectives are integrated throughout product development.

Shunra’s NetworkCatcher has an extensive library of network profiles based on millions of samples from across the globe that can be used to emulate various network conditions.  For customers with atypical networks, NetworkCatcher can measure the performance of your actual network environment.   Shunra’s PerformanceSuite is able to emulate almost any network and then provides rich reports and drill-down transaction analysis to help isolate and resolve the root causes of transaction problems in minutes. PerformanceSuite also helps determine whether any modifications to the application, network or infrastructure are required. Here’s an example of some of Shunra’s recommendations for optimization for a specific mobile site:

According to Bixby, even a 1 second delay can result in 16% decrease in customer satisfaction. The data he presented validates the assertion that mobile performance directly correlates to revenue. Shunra products help determine whether any modifications to the application, network or infrastructure are required. By testing early, adjustments can be made to performance well before your app goes live, saving time, effort and poor feedback. Testing and improving mobile performance both encourages customer satisfaction and helps to maintain a healthy bottom line.

With over a decade of experience helping federal, state and local agencies address application and infrastructure performance concerns, Shunra has gained unique insight into the challenges of securely emulating real-world conditions in the test lab. Shunra’s NetworkCatcher captures real-world network conditions in secure environments by allowing SSL connections to be made via a public-private key pair. The public key is shared with the NetworkCatcher agent but the private key remains securely on the server. In the most recent edition of PerformanceSuite, Shunra introduced SSL support within the test lab. PerformanceSuite employs a private key that enables decryption and analysis of test traffic in the lab.

Federal and civilian organizations now have a way to securely discover real-world network conditions, emulate those conditions in the test lab, and analyze traffic down to the sub-component level for performance troubleshooting and optimization. To learn more about Shunra’s secure performance testing capabilities, please contact us : http://www.shunra.com/contact-us

The Search feature made it easy to look for an obscure gift (yes, that cousin) – a Star Wars hat. So far so good, but the throughput quota was getting dangerously close to its limit before all the content loaded. Hmmmm. Typical users often leave a site after 3-4 seconds if they don’t get a response.

Since at Shunra we investigate performance issues and determine why response times are not up to snuff, Israel Nir decided to look for the cause of the tardy images. Using Shunra’s Mobile Performance Test the following transaction breakdown shows that Yoda’s chapeau appears three times. In other words, the same content is loading over and over. No wonder response time is poor!

Looking for something to keep Auntie warm was easy. She had sent a link via facebook that could be accessed in the mobile app to an item she really liked – a colorful knit hat. Transaction breakdown showed that the same image was downloaded four times, resulting in about 52 KB, 75% of it unnecessary.

Therefore, it was no surprise that Shunra’s Optimization report gave this transaction an “F” for downloading the same item more than once.

Transaction response time could be reduced for this image by a valuable second by modifying the mobile app’s behavior. For Etsy shoppers who often look at many items before selecting which ones to purchase, those cumulative seconds could result in frustrated users leaving the site. To prevent that, performing Shunra’s mobile performance test and implementing the recommendations can positively impact the shopping experience.

Thanks to Israeli Nir for the tip!

We started by loading the previous version, and looked at a waterfall chart in the Shunra vCat Analytics Deep Analysis Report of the app’s network behavior before an exercise:

And once one completes the exercise:

It’s clear that the previous version had several major flaws in the way it communicated over the network. The most obvious was a surplus of requests to the server, requests which could have easily been combined. For example, several times while the program is running, requests are being made to the URLs “[user-id]/friend”, “[user-id]/friend_request/sent” and “[user-id]/friend_request/received”. These three requests are always made in parallel, and each of them results in a short string. If they were to unite these requests, the length of the combined response would still be less than one packet. In a typical scenario of loading the program and doing a single exercise, there were 44 requests and responses, which we believe could easily be reduced to between 15-20 requests.

Combining all these requests will not only decrease the amount of packets that need to pass between the client and the server, it will also make the Gzip compression work more efficiently. That is, of course, if they were using Gzip. But it turns out, none of the responses is compressed. Even without combining requests, when most of the responses are less than 100 bytes long, compression would probably provide a nominal reduction, but since it involves practically no cost, why not do it?

Those two issues seem like obvious mistakes. After all, the first of Steve Souder’s famous 14 rules is “Make Fewer HTTP Requests”, and number 4 is “Gzip Components”. However, these are actually minor issues when you look closely at what is actually causing delays in this program. The biggest offender is much simpler to fix – 132 kilobytes of a png image that should have never been downloaded. That png image is your avatar, a small picture used to represent you in the program. We used the default avatar, and although it is shown as a 50×50 pixels image at most, its dimensions are 512×512, causing the download to take much longer than it should. Then again, why are we even downloading this image every time we finish an exercise? Surely, a much simpler solution would be to cache it, and issue a conditional request just to check whether the user has changed its picture. To add insult to injury, the image was not even downloaded in one request, but had to endure two redirects with no caching directive.

Another request that takes a lot of time to complete is a PUT request, which is sent after the user finishes an exercise, and is probably used to calculate how well the user did that particular workout.  This PUT request sends 223 kb of textual data from the user to the application’s server. The data is compressed, so it ends up only weighing 38 kb, but that’s still a lot of data, and it takes a lot of time to send it. The data sent looks like a list of timestamps, with a relative position or velocity at each marker (e.g. “1342092855.252760,0.259531,-0.613956,-0.708325,1342092855.252760\n 1342092855.260764,0.237418,-0.613956,-0.708325,1342092855.260764\n….”). However, it seems like there’s a lot of redundancy in this data, and the developers at FitFu might even be able to avoid sending it altogether and make the calculations on the iPhone instead of their server.

But enough about the previous version, let’s see what’s changed in version 1.7, which claims to reduce load times of almost every part of the program. Upon checking the new version’s network behavior, two major changes were visible:

First, some of the requests are now encrypted. So, it might seem as if the number of requests was reduced, but they are actually just “hidden” in the HTTPS connections, which we’ll discuss later.

Another big change is the vast increase in the number of HTTP requests made for analysis of the user’s behavior and usage profile. Although the encrypted data is what seems to take most of the time, those unencrypted analytics requests are not innocent and they should also be optimized. There are 25 requests issued during the app’s loading process alone, to three different analytics services: Google Analytics, Apsalar and Flurry. These could be reduced to 10 requests or less. After all, we’re just loading and signing in.

As for the encrypted data, we’ve used a proxy, Fiddler, to help us analyze the HTTPS requests. After decrypting the SSL connections and looking at the requests, it’s clear that not much has improved between versions. It’s good that personal data like the user’s email address is now transmitted securely, but a more significant improvement would have been to merge all those requests that have responses of 500 bytes and less. Every time the application downloads data from the server, there’s a group of 5-15 requests with 5-15 responses. Almost every time, one request and one response would have sufficed.

The ineffective method to download the user’s avatar is still in place. Once again, follow two redirects just to get the default avatar. Both redirects and the final response don’t have an Expires header, and the requests that are sent are not conditional requests. However, it seems that the app does some sort of caching on its own, since the request is not issued every time the app starts. The fact that they implemented their own caching mechanism is better than no caching at all, but one must wonder why not use the built-in caching mechanism of HTTP.

All in all, FitFu is a cool app that helps you stay in shape in the comfort of your own home. With a few small changes, it could be even better and reduce load times significantly:

• Reduce the number of packets sent to analytics services. The biggest offender is Apsalar, so getting them to reduce the number of packets their analytics service is sending seems like a good start.
• Reduce the number of HTTP requests sent by the application. This can be easily be achieved by combining all the requests that are not interdependent or rely on a user action. Since all the responses are quite small, combining them would probably reduce the number of packets sent from the server down to one or two.
• Save the user’s avatar image in the application, and only issue conditional requests. Also, the default avatar should be included in the application’s installation, have smaller dimensions, and not require 2 redirects.
• Try to reduce the amount of data sent back from the iPhone to FitFu’s server, if only to reduce the amount of data that’s sent for the sake of the user’s data plan.

Disclaimer: My boss used FitFu for three months and now has fabulous abs.  He also forced me to write this disclaimer.

We don’t use the browser’s caching system, so once you’ve brought the app down, unless we’ve changed something in the app, the most you have to download is 1K. So especially for international users, it’ really important to not make a bunch of extra traffic.

So we had to look at it – after all it’s always good to learn new optimization tricks. Imagine our surprise when we discovered that at least the iOS version is less than optimized. As a matter of fact, it’s wasteful. Sorting the downloaded objects by name, it immediately becomes obvious that the application downloads many pictures over and over again (click the images to see them in their full size, non-cropped, glory):

Here’s my friend Amichai, being downloaded four times:

In my case, just launching LinkedIn’s app makes it download about 150KB, out of which 50KB are redundant images. In other words, the amount of data downloaded by LinkedIn can be reduced by 33%. Closing the app and starting it again, causes it to re-download all these pictures, even though they all have a long-range expires header (2012).

The web based version of the application doesn’t behave in this way and performs much better, so it’s obviously a bug. A bug that could have been easily caught by using an analysis tool like the one I used above.

The following article builds a case for how performance engineering can help build more energy efficient mobile applications. Very interesting read, with strong reasons for why mobile performance engineering should be a critical part of mobile application development.

http://www.research.att.com/articles/featured_stories/2011_03/201102_Energy_efficient?fbid=dekBGE2Kx9k

I can picture how energy conservation will soon be added to the NFRs for mobile applications.

Enjoy,