In a recent Shunra webinar, attendees were asked which IT projects were up front for the coming year. No surprise, the leaders are virtualization, mobile app deployment and cloud migration. In other words, in addition to mobility, companies continue to focus on the consolidation of resources through virtualization of all types. Virtualization does not just mean the Cloud; it also refers to all methods that centralize administrative tasks while improving scalability and workloads. These include network, storage and server consolidations.

These results, and their impact on end user experience, align with Gary’s first projection:

#1 End user patience will continue to become shorter and shorter and thinner and thinner
Whereas just a year or two ago, an SLA of 6-7 seconds response time was par for the course, 3 seconds is now the norm for just about everything. Companies that have already tuned their applications to meet the 6-7 second goal are now required to adapt to the new benchmark of 3 seconds, which is what Gary is referring to as “thinner and thinner.” As Shunra’s Performance team reports, when determining performance goals, users now have zero tolerance for repeat work and recurring clicks. However, many business apps have not caught on and require repeat actions. In addition, users now expect instant confirmation that their work has completed successfully. For more ideas about designing business processes, see Gary’s 11th prediction.

#2 IT spending will continue to recover this year despite the Presidential election
Although election years are traditionally cautious (because of a corporate fear that a new administration may negatively impact our business, influence capital tax rates, etc.) this year should see the IT sector continue to grow. A different balance of spending will occur as EAAS -’Everything as a Service’ – is promised and ‘deliver anything anytime’ becomes the norm. This type of growth requires more external resources.

One leading indicator is the hiring of IT professionals. Despite the overall unemployment rate in the US over 8%, it’s dramatically lower in the IT sector. The talent is getting scooped up. However, more of these IT expenditures may be on non-US soil as multi-national companies avoid high taxes from revenue generated abroad as opposed to what would be due if this income is spent in the US.

#3 Many mobile apps will crash and burn because they have not undergone adequate design and performance testing
The consumption of mobile devices has exploded – it’s now the norm, not the trend. It’s hard to believe that only two years ago tablets were associated more with Moses than Steve. With the expectation that new devices and features have to keep up a dizzying pace, many apps are released well before they have undergone appropriate design and testing cycles with performance considerations. ‘Just get it out’ is the motto, but the impact on the back-end systems is not being properly assessed.

While many new apps look great on the client side, if the effect of mobile sessions on the back-end resources is not properly taken into consideration, those super cool graphics are not going to make up for slow response time. In fact, almost half of Shunra’s inbound requests relate to troubleshooting failed applications.

Mobile is driving both new and traditional applications to agile and quick development cycles, but cutting corners on performance can result in disastrous post-production results. Even shifting 10-15% of enterprise users on a specific app to mobile can overwhelm the back-end so that the app will perform slowly for all users. Different memory signatures and data throughput signatures are made on mobile apps that must be taken into account during mobile deployment.

#4 New opportunities are being driven by the rise of mCommerce
We just finished an interesting holiday shopping season. Overall retail sales were weak in early December, just slightly above the previous year’s results, and projections for December show a lower than expected rise in sales. However, while store traffic is decreasing, mobile sales are skyrocketing. PayPal reported a 397% increase in consumer use of PayPal Mobile on Cyber Monday; JoS. A. Bank’s mobile checkout increased more than 3000% on Black Friday; and Boston-based Rue La La reported an increase in mobile sales from 2% last January to 33% of all sales in this past holiday shopping season; and an estimated 87% of tablet owners did some holiday shopping from their mobile device. (This data was compiled and published by Mobile Commerce Daily, a daily trade publication dedicated to mobile commerce.)

Mobile Commerce is making significant inroads and Gary predicts that 2012 will be year that indoor Location Based Services (LBS) takes off. Developers are figuring out how to combine the carrier technologies, Wifi radio and GPS that are part of your smartphone but not available reliably indoors. Outdoors, an app such as Google Earth will first use the cell tower location, then the Wifi hotspots, and then GPS to pinpoint your bearings. Indoors, this approach is not always effective so companies are now working on intercepting the cell tower signal and using a series of repeaters to provide a signal.

In addition, red-line scanning and price search, which most stores now see as a threat, will become a business opportunity. The combination of scanning, location based services and new pricing strategies are set to have a big impact on the retail experience. The business upside is that using geo-correlated behavior, retailers can now provide pricing and promotion from within the store, enabling the so-called “bricks and clicks” business model. For example, when a shopper is scanning a TV model’s stats, the retailer can recommend a Blu-ray package. The physical store can also promote the ways that their shopping experience is more positive in ways such as availability, location, convenience, service, etc.

#5 Enterprise IT is forced to compete for delivery of service
In the good old days, the CIO was ‘The Expert’ on technology issues. When “Everything” is being delivered “as a service”, including platform as a service, software as a service and cloud-based services, the CIO no longer controls the infrastructure, or the expertise about the infrastructure.

Consolidation of IT resources is occurring among bigger global systems integrators who are taking a bigger slice of the Fortune 500 accounts. The CIO must now look at outside resources and consult with specialized personnel, and become an internal Systems Integrator.

#6 Your digital strategy will affect your ability to recruit and retain the right staff
With low unemployment in the IT sector, attracting top talent is more than challenging. One of the top three issues for a new hire in 2012, beyond salary and health benefits, may include IT resources that address the employee’s work/life balance and allow for self-provisioning.

IT resources require continuous adjustment when millions of employees now travel and work offsite and require the flexibility of different devices, connectivity and means of access to core systems. How does a company track the value of the IT investments, such as firewall improvements or the changes required in CRM to accommodate tablets? When creating a budget that aligns with the company’s digital strategy, consider the aim of the improvement. Is it increased productivity, which ideally can be seen on a daily basis, or cost reduction?

#7 Tech companies that are VC-backed will feel pressure to do transactions faster
When Venture Capital firms makes an investment, one consideration is early ROI versus long term investment. In the past few years, a number of the VC dollars have come from institutional investors, such as pension funds. Unfortunately, many of these funds have been burned by the real estate crashes, the economic turndown and various scandals.

Whereas a decade ago patience may have been considered a virtue, today there’s limited tolerance for returns. In fact a ‘sure thing’ today may be worth more than double that a year from now, but the “sure thing” is preferred as it is less risky and provides immediate return versus waiting. Just as end users now expect an application to react in half the response time, VCs are looking for results and ROI sooner than ever before. Startups and technology must prove itself sooner, or fail sooner.

For those investing in new technologies, because of the abundance of choice, it may be harder to predict whether a new technology is going to make it. Therefore, Gary advises to pick your partners wisely, taking into account the strength of customer referrals, history, and financial condition of the company before signing on the dotted line.

#8 IT Capital investment will continue to decrease, while consumption of resources will continue to increase
The continued pressure to virtualize desktop, servers and just about everything else, and to migrate to the cloud and use less expensive platforms (such as a tablet vs. a more expensive laptop), means that almost every end-user wants more than last year – more applications, resources and more access to resources. Startups and new companies may even go straight to the Cloud to consolidate facilities and save on energy costs. This has placed a lot of pressure on low cost facilities to deliver and may bring unrealistic expectations that have not taken operational and other costs into account.

Some companies even have a BYO support strategy – bring your own device. Although it may save the enterprise in capital costs, the support costs will rise as the IT team faces the challenging task of supporting all these various devices and configurations.

#9 Cloud services will prove an ROI in 2012
Although no studies of real cost reduction for cloud ROI are yet available, Gary predicts that 2012 will finally be the year that demonstrates ROI for Cloud investments. Companies that have kept their figures in-house are now more likely to reveal stats regarding the backend impact, especially as pressure mounts to prove the effectiveness of prior years’ investments. Useful data about the hidden costs of failure, such are rewriting apps, will then become public.

IT cannot just toss legacy applications into the Cloud and expect them to work perfectly where they have no real control. They must now design for Cloud, test for Cloud, and optimize for Cloud. The CIO now must assume the role of systems integrator and must have ‘the fixers’ lined up should a disaster occur.

#10 Attention on infrastructure will increase while inversely decreasing on applications
We are now experiencing a paradigm shift to virtualization – from hardware to software to services. In addition, consumerized end-user devices necessitate a continued investment in infrastructure. It will probably take a long time for enterprise apps to catch up in terms of required attention and investment. The silver lining is that opportunity is created for outsourcing (and crowdsourcing): IT is looking for vendors who can help. For example, through its community, SAP now has hundreds of mobile apps developed in the past year. SAP users, instead of waiting for SAP or their own IT department, can now to tap via 3rd party mobile applications into legacy systems and access specific functions.

#11 bonus prediction – Reduced form factor providing only the “essential engagement” will become more common.
With so much happening in the IT sphere, it was a real challenge to select only 10 predictions. Gary was not actually successful in this endeavor. Therefore he gives us one of his most interesting observations: Enterprise apps are going to be reexamined via a mobile device lens. In other words, how we design and use mobile devices is going to govern how we design and use almost all other applications!

Careful attention will be paid to those actions that you or your customers are performing on a regular basis and how data is accessed. When dealing with complex systems, such as a CRM application or a business process, just navigating through all the various functionality could take over 30 minutes. But a sales team member most frequently accesses a CRM application only to update customer records and contacts, set meetings and look at the probability of closing a sale. The 80/20 rule applies; about 80% of the team is utilizing about 20% of the available functionality.

That’s where the application and device form factor can impact and improve productivity, by narrowing selections to frequently used actions. This requires analysis of what the user needs to complete his/her tasks and how best to present the information. In a mobile app this analysis is crucial, as each action and screen in the workspace is highly valuable real estate. Therefore designers and developers have to determine the essential actions, flow and optimal presentation. As users continue to consume smartly designed mobile apps, they will no longer accept cumbersome enterprise functionality. To optimize the user experience, each action and state presented should be one that is repeatable and modularized, and this must also apply to other corporate experiences.

That didn’t take much time, what could have happened? Actually when it comes to mobile websites, those few seconds can have a major impact on your revenue. Look what happens to mobile users when a minimal delay in response time is introduced. How many users will leave a mobile site with a 200 ms delayed response time? Not many. However, when the delay reaches 500 ms the effect is noticeable. Increase the delay to 1000 ms and the direct correlation is obvious. The higher the delay, the greater the number of users who are going to abandon ship. Does this affect your bottom line? You bet. Even worse, some of those users are never coming back.

Let’s look at Joshua Bixby’s results presented at Velocity Europe this November. Although it makes sense that users are going to desert poorly performing mobile sites and apps, most of his clients were not aware of the true impact on their KPIs. To quantify his hypothesis, he added various delays to HTML page response time to monitor the effect. With 500 ms of delay, page views and conversion rates are negatively impacted. At 1000 ms of delay, cart size is also affected.

Bounce rate, negligible at 200 ms, rises over 8% at 1000 ms. Those users may never return; Bixby pointed out that even after a few months, visitors who experienced long delays returned at lower rates. For iPad users, the correlation is even more obvious – their bounce rate increases significantly when performance degrades. He also examined the effect of delay during various steps in a mobile scenario. Bounce rate almost doubles when delay is about 2 seconds during any of the stages in the scenario.

How directly does this affect your key business metrics? There’s no question that more and more money is now spent using mobile devices. For a typical retailer just a year ago, for every $100 spent online only 50 cents was spent on a mobile. Today $7 out of every $100 spent online comes from a mobile source. With 14 times growth in just one year, the trend is obvious. Anyone selling or providing a service online no longer has a choice about whether to offer mobile access. But while a great app can improve your income, a poorly performing site can negatively impact not just your mobile, but also other online sales and even your company’s image. As users tweet and post comments about pathetic response time, the effects can quickly become viral. Will users say that the website is great but that the mobile app is terrible? No. They may just comment how the site took so long to respond that they checked out the competitor.

Mobile access is also changing. While most users are still accessing the full site, partially because many search engines point to these sites, an increasing number of users are going to the mobile site and mobile app. However, no matter how they access your information, when performance is poor users are not tolerant.

So what can you do to determine if your mobile performance is unsatisfactory? To obtain a true picture of the performance, Bixby recommends correlating performance and business metrics. The performance data includes real end-user monitoring, including data from site traffic analytics, latency and bandwidth checks. The business component includes data such as who is buying, what and how much they are purchasing, etc.

These decisions need to be taken at a policy level to have meaningful, long-term impacts to your business.  Once you commit to a company policy on performance, the next question is typically, “how do I know how my customers experience my site and how do I optimize that experience?”  That’s where Shunra comes in.  To improve customer experience on the mobile, Shunra’s Application Performance Engineering products deliver insights into application performance that include fast root cause analysis, location based SLA validation, and actionable information for improving and ensuring optimal end user experience. Ideally, you should set these targets early in the development process so that performance objectives are integrated throughout product development.

Shunra’s NetworkCatcher has an extensive library of network profiles based on millions of samples from across the globe that can be used to emulate various network conditions.  For customers with atypical networks, NetworkCatcher can measure the performance of your actual network environment.   Shunra’s PerformanceSuite is able to emulate almost any network and then provides rich reports and drill-down transaction analysis to help isolate and resolve the root causes of transaction problems in minutes. PerformanceSuite also helps determine whether any modifications to the application, network or infrastructure are required. Here’s an example of some of Shunra’s recommendations for optimization for a specific mobile site:

According to Bixby, even a 1 second delay can result in 16% decrease in customer satisfaction. The data he presented validates the assertion that mobile performance directly correlates to revenue. Shunra products help determine whether any modifications to the application, network or infrastructure are required. By testing early, adjustments can be made to performance well before your app goes live, saving time, effort and poor feedback. Testing and improving mobile performance both encourages customer satisfaction and helps to maintain a healthy bottom line.

The Search feature made it easy to look for an obscure gift (yes, that cousin) – a Star Wars hat. So far so good, but the throughput quota was getting dangerously close to its limit before all the content loaded. Hmmmm. Typical users often leave a site after 3-4 seconds if they don’t get a response.

Since at Shunra we investigate performance issues and determine why response times are not up to snuff, Israel Nir decided to look for the cause of the tardy images. Using Shunra’s Mobile Performance Test the following transaction breakdown shows that Yoda’s chapeau appears three times. In other words, the same content is loading over and over. No wonder response time is poor!

Looking for something to keep Auntie warm was easy. She had sent a link via facebook that could be accessed in the mobile app to an item she really liked – a colorful knit hat. Transaction breakdown showed that the same image was downloaded four times, resulting in about 52 KB, 75% of it unnecessary.

Therefore, it was no surprise that Shunra’s Optimization report gave this transaction an “F” for downloading the same item more than once.

Transaction response time could be reduced for this image by a valuable second by modifying the mobile app’s behavior. For Etsy shoppers who often look at many items before selecting which ones to purchase, those cumulative seconds could result in frustrated users leaving the site. To prevent that, performing Shunra’s mobile performance test and implementing the recommendations can positively impact the shopping experience.

Thanks to Israeli Nir for the tip!

There are several nebulous concepts thrown around when Cloud as a topic is discussed. However, there is one thing for sure: for as many concepts there are discussed, many more risks, some quite significant, are identified. From the just published 2011/12 World Quality Report (Cap Gemini, Sogeti, HP), of all the risks associated with using a Cloud environment, performance is the second largest.

I find it reassuring that someone is finally putting a focus on performance, and starting to report on the real business and customer risks associated with poor performance, especially as we see an explosion of Cloud hosts (some fly by night) and an increase in Mobile users.

From the research, 37% of the respondents cited Performance as the greatest risk for using a Cloud environment. This sampling was cross geography (North America 39%, Europe 32%, Asia 18%, Rest of World 11%) and cross organization size (starting with 1-500 employee organizations to 10,000+). I am shocked this is not higher, since the risks of Security and Availability drive non-performance directly.

The study then moves into looking at the varied geographies and what plans exist among companies regionally to move applications to the Cloud. From the results, it appears rather consistent that companies worldwide plan to move ~42% of their applications to the Cloud over the next year. The amount of companies indicating a plan to move to the Cloud is much higher than I would have anticipated, and highlights the increased need for understanding the risks associated with migrating applications to the Cloud.

What comes next makes perfect sense to me. Actual problems encountered after migration to the Cloud include two large Performance aspects: Performance issues due to the network, and Performance issues due to servers. Well, after reading this report, I wonder how many of the organizations migrating ~42% of their applications to the Cloud in the coming year are looking for a better way to mitigate the “now known” risks of making the move?

Please leave a reply and submit your comments below. Seeking supporting or opposing views, as we seek to move the status quo, and look to minimize and mitigate the impact of the performance-based (and other) based risks through implementing Application Performance Engineering capabilities to mitigate the complex risks that moving to the Cloud introduces. I would be most interested if you could share a real example of what I see every day, as customers and businesses are significantly (negatively) impacted by performance, which often goes unreported or unnoticed.

The value of Application Performance Management (APM) is perceived as “less than fair.” Over 80% of large and mid-sized organizations worldwide have made multi-million dollar investments in APM solutions with the expectation that these capabilities would reduce their production outages, quickly pinpoint the precise root causes of issues during these incidents, and speed time to resolution. In a show-of-hands at Interop 2011 during the ‘Service Delivery Management’ panel session moderated by Jim Metzler, an industry-recognized expert in network technology and business applications, NONE of the attendees agreed that APM was working well at their organizations and only 2 agreed it was performing fair, putting the remaining 70+ people in the “less than fair category.”

It’s worth repeating – millions of dollars have been invested in APM. Yet, according to NIST, four out of every five dollars of the total cost of ownership of an application are spent and directly attributable to finding and fixing problems post-deployment.

If you are not familiar with APM or how it is specifically defined, Gartner and others have outlined APM capabilities as covering these five functional dimensions:

1. End-user experience monitoring
2. Application runtime architecture discovery and modeling
3. User-defined transaction profiling (also called Business Transaction Management)
4. Application component deep-dive monitoring
5. Application data analytics

As a panelist, along with my peers, we discussed this disconnect between expected and delivered value and speculated as to the reasons behind it. Here are a few of the observations we had.

1. Most APM solutions focus on object-based system monitoring, with a constant stream of alerts, and a problem-based environmental picture. This results in an overwhelmed (thousands of alerts per hour) Network Operations Center (NOC) which often only has a handful of 7×24 operators.
2. The complexity of an APM implementation into an enterprise prohibits widespread adoption of a solution across the infrastructure, application, and business unit ecosystem. When incidents happen, they can occur anywhere in the system and are often impacted through several layers of dependencies.
3. The focus of IT management and spending has been on implementation of new hardware technologies enabling larger throughput and capacity (e.g. 40Gig devices) rather than management of application performance. This seems to be occurring because the given roles and responsibilities of IT decision-makers are aligned more with Infrastructure rather than Applications.
4. The rise of Cloud and Mobile infrastructures is introducing new and unplanned-for performance risks. New skills and approaches to performance are required to manage the complexity of these new infrastructures.
5. Best practices of Application Performance Engineering (APE) are only now being introduced into organizations, and a proactive approach to building performance into the entire development lifecycle, before application deployment and before the point of performance monitoring, is not yet pervasive – the need is recognized, but budget and resources are not yet aligned with the need.

We think this show-of-hands needs to be viewed as a wake-up call:
Application Performance is something we all need to recognize as a major risk, and, in parallel, we need to accelerate awareness of the importance and value of proactively mitigating application performance issues prior to production.

Business success depends on this; we don’t have the luxury of time and must act now. As one of the APM industry leaders in the session stated, “Issues with the performance of business-critical applications can cause deterioration of an organization’s business performance. Slow or not readily available applications that support key business processes can cause revenue loss, and decline in customer satisfaction, employee productivity or brand reputation.”

Please leave a reply and submit your comments below. Seeking supporting or opposing views, as we seek to move the status quo, and look to maximize the value from the APM investments through implementing complimentary APE capabilities.

Losing site of the goal

What happens when IT departments lose site of the performance engineering goal? (Reminder in short the goal is to improve the end user’s quality of experience and productivity, while maintaining system costs within budget).

Well what happens is that each department gets lost in its own tactical goal:

1. The Capacity planning team focuses on efficient and accurate hardware provisioning
2. The load testing team focuses on test coverage and scale requirements
3. The network engineering team focuses on the speed and capacity of the pipes
4. The data base team focuses on the performance of the data bases
5. The server team focuses on the performance of the backend servers
6. Desktop team is focusing on the performance of the desktop clients
7.  …

What the organization ends up with is a set of local optimums, but in many cases those local optimums don’t amount to an optimal system. What’s missing in the above list is at least one department that is responsible for meeting the goal, it is very rare to find a team that oversees the end to end responsiveness and performance of the application across all its components from the end user’s perspective. It is even harder to find a team that is held accountable to end user performance.

But is it wrong for each team to improve its domain and make sure it is optimal? Well the counter intuitive answer is yes, it is wrong and for the following reasons:

Focusing on the wrong bottlenecks

Let’s consider the following transaction as an example: this transaction generates a time sheet report for global employees. This transaction is served by a client (web browser with java widgets) a few web servers behind a load balancer, a few application servers and a data base server. Now lets see what happens if there are performance issues with this transaction. Naturally each team will spend time in improving its own domain, so the data base team may index the employee’s data base to reduce the data base response time in half, the server team adds more web servers behind the load balancer to increase the application’s scalability and the network team adds more bandwidth to the data center router. All these steps sound like they should help, no? Well the realistic answer is that in some cases none of these steps help, in fact 3 negative things happen here:

1. The teams spent time and money on the wrong bottlenecks
2. The real bottleneck is still out there
3. Increasing the speed of none bottleneck components places more strain on the real bottleneck, slowing things down even further

In future posts I will give specific examples of several problems that can not be addressed in the realm of one IT department. Those problems usually result from interdependencies between the different systems (servers, networks and data bases). It takes a holistic and multi-disciplinary process to find the right bottleneck let along find a solution for the problem. It may sound complicated, but the concept is quite simple, when dealing with performance, it does little good to focus on local optimums, any optimization effort that is not spent on the actual bottleneck is counter productive and a waste of IT resources and money. Remember, the goal is to improve end user response time at the desktop, not optimize a specific component that is part of a bigger system.

Even though the concept is simple, the solution isn’t always as simple. In future posts, I will offer practical ways to find application performance bottlenecks, I have used them in many engagements and they haven’t failed me yet. But before we can talk about the solutions, it is important to understand the problems so in the next several posts we will cover a few basic performance engineering concepts.

Talk to you soon…

It’s a simple statement, but a powerful one that will change the existing application lifecycle management paradigm. At Shunra, we have been working hard, together with our partners, at developing and defining a new performance management model that completes the existing APM  frameworks. This post is a first in a series of posts that describe this model and the motivation behind it. In this first post, I will present the reasons why now, more than ever, such a model is needed – why the time is right. Future posts will dive deeper into the definition and building blocks of the application performance engineering model, explore where this model is applicable and explain the benefits it promises.

Application performance is still a problem.

Despite increased investment in application performance management and monitoring solutions, applications in production continue to fail to meet critical performance requirements. These failures create productivity loss for end users who cannot quickly accomplish the tasks the application was designed to facilitate. These failures result in lost revenue from eCommerce and other consumer services abandonment. And, these failures increase remediation costs as time and resources are required to isolate, diagnose and resolve performance problems in production.

Why now?

Application performance is still a problem – this is not a new concept, but it bears repeating. I could have written that sentence 5 or 10 years ago, and it would have held true then, just as it does now. So, why is the time right (or better said, why is the time right now) for a new and better approach to application performance management?

Today’s ever more complex enterprise environment and increasingly distributed application architectures (and user populations) are making the challenge of effectively managing application performance more difficult. New performance risks are arising from three Enterprise IT waves that are converging upon us today:

1)            The infrastructure paradigm shift – the economic downturn has accelerated the move towards data center centralization including, on an accelerating basis, Cloud migrations and associated cost benefits. Enterprise IT is expected to manage these complex projects to realize the promised savings, without impacting performance or end user productivity.

2)            The technology paradigm shift– several new technologies are redefining the way data is delivered to users including virtualization of servers and desktops, unified communications and Web 2.0.  As Web 2.0 technologies are rapidly being adopted in the enterprise, new smart browser-based capabilities are being introduced which blur the lines between where data resides and how and when it will be transferred across the network. In addition, and growing at an even more remarkable pace, is the mobile revolution. Organizations are increasingly challenged to deliver consumer-like applications, capable of handling enterprise grade data volumes, to an ever more demanding and mobile workforce.

3)            The reality of the “Virtual Remote Office” – the virtual office which was a forward looking concept just 10 years ago, is a reality for an increasingly growing workforce. In addition, the scope of the “virtual office” is expanding beyond the residential “home office” to include trains, planes, coffee shops and hotels. Today’s workforce is expected to be productive anywhere they can access a network connection, placing a tremendous dependency on the networks for how well applications will perform.

This “perfect storm” or confluence of Enterprise IT waves makes it difficult to deploy and gain confidence in the performance of applications. As application performance continues to become more dependent on the network, confidence in application performance will continue to decrease. This necessitates increased focus on how applications behave across a variety of networks including WAN, Internet, Mobile and Cloud.

What do companies do today and why isn’t it enough?

Traditional approaches to performance management include Application Performance Monitoring. This approach, however, offers only partial relief. Performance monitoring is a solid approach to discovering and diagnosing problems that result from the interaction of applications and networks in production. That is the key – performance monitoring is a reactive solution to discovering issues in production. And, because no application is perfect, performance monitoring is critical to performance management.

Discovering and fixing problems in production, though often required, is costly.  Remediation can require re-engineering or significant infrastructure investments that can be difficult to plan in a firefighting mode. There are time pressures and cost sensitivity issues since end users are being affected and application development and testing costs have already been allocated and incurred. Performance monitoring solutions help improve MTTR, but they cannot offset the time and resource costs associated with poor performance in production and poor end user experience.

What is missing?

More forward-thinking companies are moving away from the break-firefight-fix cycle and are beginning to augment their APM strategy with a more proactive, pre-production approach to performance management. This new approach leverages Application Performance Engineering best practices to reduce performance remediation costs, increase monetization and ensure end-user productivity. To be successful, this approach relies on a precise replica of the production environment for identifying poor performing business-process-steps in advance of deployment. This does not necessarily mean recreating the network hardware in the QA lab, but enabling a precise emulation of the hardware and production network conditions or impairments. With an APE-enabled test lab, these companies can effectively and reliably investigate performance optimization solutions, be confident in their network and performance decisions, and validate real-world application response times in advance of deploying to an end user community across networks – WAN, Web, Mobile and Cloud.

In my next post, I will dive deeper into defining the goal of Application Performance Engineering and the associated building blocks that make up this model.

In the mean time, I am looking for your feedback:

• What else is missing in how application performance is being managed?
• How do you view the application performance engineering goal?
• What do you think needs to be in the application performance engineering model?

The content in this post has been co-developed with Bill Varga, the COO of Shunra and Marty Brandwin, Director of Product Marketing at Shunra. Many parts in it were influenced by thoughts and comments from most of the executive team at Shunra and our partner community.

Looking forward to your comments,

Amichai Lesser

Good performance at peak loads should be something that we think about from design and development all the way through to the production environment.  It starts with understanding what users are going to do on the site, producing test scenarios that match that behavior, and applying those scenarios to your functionality and load testing efforts in the preproduction environment. Hopefully you are nodding your head as you just read my last sentence, but here is something that you may not have considered. Have you measured the Latency, Packet Loss, and bandwidth for your typical user population and recreated that in your preproduction environment?  “Why does it matter?”, you ask.  It can dramatically affect system performance and stability and always has an impact on end user performance.

Think about it this way, if you are load testing and functionality testing in the preproduction environment with lightning fast latency, near infinite bandwidth, and zero packet loss then you are gaming the system.  In the lab, your packet latency is less than 1 millisecond, but your users are going to be experiencing latencies of 30, 50, or even 100 milliseconds latency, which translates into keeping network resources tied up for much longer timeframes.  That leads to longer Web Session with more simultaneous HTTP connections and longer Application sessions, which can even affect the database. These are all infrastructure resources that are utilized differently using identical load scenarios and load levels, but have a dramatic impact on end user performance and reduce the scalability of the system.

Real Network Conditions Play a Big Role in System Scalability

Applying end user latency, loss, and bandwidth is easy with tools like Shunra’s VE Desktop, which can be used in functional testing to accurately apply the correct network conditions to boost accuracy with the latency that end users would experience in the real world. Shunra can go a step further and help you to determine the latency, loss, and bandwidth that you can plug right into the environment during testing using the Network Catcher tool.

Shunra also has a variety of options for emulating network conditions during load testing ranging from  appliances Like the STA to software. The appliance acts like a router on steroids applying network impairments dynamically to the network traffic during the test treating traffic from different URLs and IPs like they are in different cities.

If you are using HP’s LoadRunner load testing tool, Shunra has an elegant solution called Shunra for HP Software, which works seamlessly with the LoadRunner during the load tests to apply the latency, packet loss, and bandwidth during these critical scalability tests.

Regardless of your testing methods, be sure to include identical network conditions into your testing for the most accurate results.

You have probably encountered a situation where pre-launch load tests did not correlate with the way it performed in production.  There are a lot of reasons why this can happen, but for now, I want to focus on 3^rd party content providers and Content Delivery Networks (CDN).  Almost everyone running a business today has some part of their application or Web presence farmed out to a 3^rd party.  Think about what happens when you go online to buy gifts for the upcoming Holidays.

As you browse for that perfect tie for your dad, the site is likely to be serving static content from Akamai or Limelight, adds from 3^rd party providers, and a shopping cart that may not be hosted by the company you are dealing with.  When you actually purchase that Tickle-me Elmo for your niece, the site processes your credit card by interacting with a financial institution and provides shipping tracking numbers by interacting with a major shipping company behind the scenes.  All that typically happens as you wait for the page to return in a few seconds if all goes as it should, but any of those 3^rd party providers have a strong likelihood to disrupt the page load and make the site look like it is not responding.

Increased delays from these and other 3^rd party providers do happen and you should definitely test for those scenarios, but how do you do that when you really do not have control over those sites. You definitely should perform load tests on the site as a whole to ensure that each 3^rd party provider will scale to your targeted peak traffic and perform within their SLA as expected. In addition, you should think about how slow your 3^rd party providers can get without affecting the performance of your site.

“How Slow Can They Go?” Yes, a corny statement, but to the point! You definitely need to identify how slow your 3^rd party providers can go before it starts to affect the performance of your sites.  The real dilemma is replicating the traffic from the 3^rd party site, which in the case of financial transactions can be very difficult plus your team can waste a great deal of time trying to do this. I’m suggesting that you don’t try to replicate that traffic, instead use the actual 3^rd party traffic from the 3^rd party site during the test. Here’s the catch, you impair the traffic coming from the 3^rd party site real time. Simply single out that traffic in your usability studies using a tool like Shunra’s VE Desktop to measure performance of the site with the 3^rd party adds impaired or by slowing down the response from your credit card processing partner using a Shunra VE Network Appliance on your backend systems during a maintenance window.

Now think about scalability load testing for a minute. I recommend that you do not implicitly trust 3^rd party and Content Delivery Network (CDN) provider’s scalability estimates and guarantees. You are probably scratching your head thinking that your CDN’s and other 3^rd party content providers have massive server and network infrastructures that are impossible to overload. While it is true that the major 3^rd party providers have massive infrastructures, the real question to ask yourself is this, “How much of that infrastructure is your content riding on?” Let’s just say that you actually do know how much of their infrastructure is dedicated to your cause, you still have to ask yourself the question, “Do I know for sure that it will handle the peak traffic?” Adding in impairments during a load test is easy using Shunra’s VE Suite or Shunra for HP Software which works seamlessly with Load Runner.

During my career I have load tested literally hundreds of different sites, and many of those sites had CDN and 3^rd party content provided by very large companies that did not hold up to the load during a stress test. Let’s face it, regardless of the size, the CDN or 3^rd party provider’s service can only scale to the level of infrastructure that is available to your site.  CDN content can fail to scale due to complex caching configurations that can get screwed up with one incorrect setting.

In more than one situation, I had a customer tell us that they were going to inform their CDN or 3^rd party providers to “be ready” for the peak traffic that our load test was going to generate. They were usually shocked when I asked, “Why?” Their answer was typically, “We need to be responsible partners and let them know anytime we are doing testing.”  Clearly, I would normally agree with that statement for many different kinds of testing, but not in the case of load testing against an established Service Level Agreement (SLA)!  Here is the bottom line, if the CDN or 3^rd Party has agreed to an SLA then that system should be ready to handle that load any time (barring timeframes specified in the SLA for maintenance windows, etc.).

In several load testing engagements, where the client let the CDN or 3^rd party know about the load test, there were “additional resources” stood up to support the load test. Surprisingly, the system scaled during those times, but failed to scale when no notice was given ahead of time. Does that sound suspicious to you? If the answer is yes, then we are on the same mental wavelength. If you encounter problems with your 3^rd party provider not scaling during your load test, and they will probably insist on “supporting” you during the next load test. Be sure that you ask your 3^rd party providers the following questions:

What did you do to support this load test Vs. the last one?

What did you change in your infrastructure that allowed it to scale successfully this time?

Did you add infrastructure or change a setting?

In one situation, I heard a 3^rd party provider tell my client that they stood up an extra bank of machines to support the testing that was happening that evening. My question to the customer was, “Does that sound strange to you?”

At the end of the day, we all need to admit to ourselves that even our monstrous 3^rd party CDN partners can slow your site down, so you really should test how your site performs to the end user and how the infrastructure scales when they do slow down. Try to find out how slow each 3^rd party provider can get before it impacts your site and ultimately the end user’s response time and build that into your Service Level Objectives and ultimately let that drive your Service Level Agreements with 3^rd party content providers.

While the technology is still in its infancy, several providers are adopting distinctive approaches such as RF transmitters/receivers to forward the information from the metering devices upstream to the utilities, and back. Others are choosing to deploy 3G technologies. All solutions have similarities, for example several or up to several thousands communicate upstream to concentrator units, currently being deployed in utility poles, with the electric or phone wires.  In turn, the concentrators communicate with a “front end processing” unit, that properly parses the information, scans it for malware or security threats and then assimilates the information in the utility Information Systems applications, such as billing, power demand management systems, outage control and others. The Utilities can in turn, send instructions to individual meters, containing differential pricing – based on electric demand in real time – or command to a particular appliance when to commence a cycle, for example, re-charge an electric car or a laundry load only when the electric demand is off-peak and there is enough surplus to generate capacity.

A key factor in the adoption of the technology is the incorporation of the Home Appliance Network, a mechanism that allows the manipulation of home appliances by incorporating command consoles into them, and assigning them unique identifiers. This is where IP V6 comes into play. All the elements on the Smart Grid networks will communicate using IP based technology and the need to have millions of individual entities will force the migration into IPV6.

So where does Shunra become part of the solution? Actually in almost every step of the life cycle. Every smart meter communicates upstream with a concentrator, and the concentrator conveys feedback and instructions to every meter, either over a wireless, 3G or RF network. In turn, concentrators exchange data with the utilities, via an MPLS network supplied by the utilities or contracted by them into a carrier. Every step of the communication chain can, and should be emulated in a lab by Shunra technology. At the same time, every major utility will have to construct a complete mock-up lab for testing, training and development purpose. Shunra will provide the perfect replica of the network environment and incorporate it into the scripting and testing tools to certify your applications.