What resonated well with me was a quote from Jim taken from “Alice in Wonderland” – “If you don’t know where you need to get to, then it doesn’t really matter which way you go”.

The full article is available here: http://www.networkworld.com/newsletters/frame/2011/013111wan1.html?page=1

Keep an eye out for additional articles by Jim Metzler around the Application Performance Engineering process in Network World, which should come out in the following weeks.

Talk to you soon,

Amichai Lesser

Optimizing for mobile device cache -
http://www.yuiblog.com/blog/2010/07/12/mobile-browser-cache-limits-revisited/
http://www.stevesouders.com/blog/2010/07/12/mobile-cache-file-sizes/
http://www.browserscope.org/

Optimizing JavaFX code for mobile – http://weblogs.java.net/blog/2009/04/30/javafx-mobile-applications-performance-tuning
Mobile High Performance Presentation from Velocity – the book is also very good http://www.slideshare.net/firt/mobile-web-high-performance

Curious to hear what type of performance engineering challenges for mobile you are facing, please share…

It gets better!   For the beta program, HP is waiving all of their license fees!  (Amazon EC2 license/use fees still apply.)

And better yet!  Lots of reasons this is a win-win for HP LoadRunner customers:

• This makes LoadRunner significantly more available to testers.
• Using this delivery model, we can expect LoadRunner to be significantly less expensive  AND easier to deploy.
• Good-bye to  days of needing to own load testing software that you should be using regularly, but in reality may only use during specific phases of your SDLC.  Now you can pay for what you need, when you need it.
• Also gone are the days of installing and upgrading LoadRunner.  Have you ever tried to download a 1.2GB installer over a slow link?
• How about realizing that you need to uninstall a previous version before you can install the new one?  Now that HP and Amazon publish AMIs with the software pre-installed, all you need to do is spin-up the images you want and run them.

While my enthusiasm for this development is clear, it should be said that it’s not all, “moonlight and canoes.”

The Controller AMI in the beta program is limited to 250 virtual users.  Since HP hasn’t published their pricing yet, it’s not clear how larger tests will be accommodated.  For now, the beta testing will limit us to mid-sized tests.  Considering the actual technology (LR 9.5) is already stable and known to work well in virtual environments, there is little doubt that this solution will be massively scalable.

You will also need scripts to run in LoadRunner.  For existing or former LoadRunner customers, this is as easy as accessing your scripts from EC2.  For net-new LoadRunner customers, this will require some work.  Thankfully, the script generation environment, Virtual User Generator makes it easy to create simple scripts and there is an extensive community of users and consultants to help you develop the scripts you need.

Be sure to watch Mark Tomlinson’s video where he explains the benefit of testing applications for the cloud in the cloud.  The main benefit Mark demonstrates is the ability to omit your corporate firewall from the test environment.

One BIG benefit of testing in the cloud is eliminating the public network from your test environment:  If you are testing an application in the cloud from your on-premise lab, then you’re subjecting your test data to an unknown world of latency, packet loss and bandwidth limits.  Testing a cloud application from an on-premise lab can produce useless results as they  have no correlation to how your application will actually perform from your end user’s perspective.

Another benefit I would be remiss if I didn’t mention is that Shunra’s integration with HP LoadRunner, Shunra for HP Software already works well in clouds.  Shunra for HP runs inside of LoadRunner and lets you decide what network impairments to test.  You can import your production data, pick from pre-defined settings, or specify ad-hoc values to make each Load Generator emulate a different network location.  By doing this, you can conduct a performance test for all of your end user locations, entirely in a cloud environment, all the while paying as you go.  I may be a little biased, but my argument is that the functionality of Shunra for HP completes the picture for anyone performance testing applications that will be accessed remotely – and who isn’t?

As a techie who started performance testing using a green-screen and writing my own protocol stacks because the applications I was testing insisted on using proprietary protocols, I am excited to see professional quality performance testing moving into the cloud.  Although this innovative solution hasn’t created much new technology, it did make existing technology consumable in the cloud.  This is sure to challenge other vendors  to step up to the plate, making other products and services more cloud-enabled.

What do you think?  Are there any reasons you would not use this capability?  What are some other products or services that you hope will make their way into the cloud this year?

There are many challenges organizations face when it comes to administering and maintaining a dedicated pre-production or staging environment for which accurate performance testing can be conducted. The cost to manage and maintain infrastructure, along with personnel and facilities, can be fairly sizeable and are only a subset of the overall costs to be considered. So, in many cases performance testing can be rather expensive and this is exactly why virtualization can provide significant benefits because there is cost reduction across the board. A prime example is in many performance labs there are a variety of application performance tools typically utilized for testing – one such tool is HP LoadRunner or Performance Center. These tools are a primary part of a performance lab as they provide load generation capabilities and can accurately test applications under real world load and stress scenarios. However, these solutions require a significant amount of infrastructure and resources (A Controller to execute tests, LoadRunner Generators to produce user traffic, Virtual User Generator to record scripts, etc.) and this can make it very difficult to manage the environment when it has to scale to meet higher demand. In this case, virtualization saves time, effort and cost because resources can be allocated dynamically within the environment and any number of virtual machines can be leveraged when needed to handle these resource intensive applications. This is also enabling many organizations to architect and customize elegant configurations that more closely align with their testing requirements – which can minimize unnecessary infrastructure and resources. Yet, the prevalent issue many organizations still grapple with is how to execute performance tests that accurately depict the network for which the application will be deployed across.

The most pervasive approach that many organizations would take is to physically deploy hardware (remote load generators) in offices that they wanted to test an application from. This process was not only time-consuming, but also expensive, inaccurate and cumbersome to manage. For this reason, HP decided to form a partnership with Shunra to develop a seamless solution that provides this capability within the HP LoadRunner and Performance Center solutions – Shunra VE Desktop for HP. This solution aligns very well with the virtualization movement because it is simply a plug-in within the HP products that introduces the network into the existing test bed and can be leveraged across most virtual platforms. For LoadRunner 9.5 and later, there is simply a “WAN emulation” tab that can be accessed from within the Controller to introduce the latency, jitter, packet loss and bandwidth constraints directly into the test. With Performance Center 9.5 and later, this capability can easily be configured directly from the browser UI to allocate WAN emulation parameters across any number of desired load generators. A consequence of this is that organizations can leverage on-demand performance testing from a dynamic virtual environment that is agile, flexible and robust. This therefore eliminates the need to manage testing cycles across multiple time zones and remove any need for additional hardware. Additionally, all of the network metrics from each generator utilizing WAN emulation within the test will automatically be imported into the controller, which can save a significant amount of time when collating results and generating analysis reports. These reasons are precisely why numerous organizations have decided to improve their existing performance test environment with the VE Desktop for HP Software  solution. Not only does this solution address a strategic gap within the functionality of the HP solutions, it embraces virtualization as a means to more effectively administer performance testing. Overall, the VE Desktop for HP Software solution was co-developed with HP to considerably enhance the accuracy and value of these application performance test suites.

Since the January 27^th announcement, it has become public knowledge that also Google is about to release a smart tablet, as well as HP using a Windows 7 platform. So are we reaching critical mass? I believe we are reaching the point tablets cannot longer be ignored.  Before the end of the year you will be able to receive TV streams into your tablet, read E-books, use your favorite social network, including the latest addition, Goggle “Buzz”, order pizza for dinner and yes, check your email and place both video or voice conference calls.

On an interview with the Associated Press, Disney’s CEO, Robert Iger, as quoted at the blog TUAW, The Unofficial Apple Blog,  he depict different scenarios using IPAD to deploy games, media and gaming content, all of it oriented toward the non-technical audiences.

Apple is drawing heavily on the experience gained by both IPod Touch and Iphone, that  in only a couple  of  years had managed to become the industry standard for smart web appliances and smart phones. The primary audience is the millions of users who are too afraid or reluctant to use a laptop/netbook since they are too rigidly bound to an unfriendly Windows OS.  A typical tablet user doesn’t want to use a computer per se.

So, why this is relevant to this blog? Very simple, all the context, media, games, TV shows, music and E-books will be delivered over ever increasing congested networks, served from ever more complex infrastructures at all will require substantial testing. What better tool to test a distributed network that Shunra Virtual Enterprise?  Application providers will have to test their application before deploying them to the millions of users. What will be the user experience, how latency will impact an IPAD user watching 2011 Super Bowl or downloading the latest chapter of their favorite TV show?

The combination of testing for 3G readiness (see other posts in this blog) and end user experience assessment will be a key factor on the success of this game changer.

Shunra VE is the best way to prepare for this new wave of context and delivery mechanism.

WAN Simulation methodology is a set of theoretical algorithms which are usually applied on a trace file. These algorithms aim to predict applications response times with different conditions and then applied to the captured trace file. A typical use case would be varying the conditions, such as available bandwidth and distance (latency), and then predicting the impact on response time. The main challenge with this approach is that those theoretical algorithms are based on assumptions.  Assumptions, which are in some cases very wrong and skew the results. Typically, with theoretical WAN Simulation you should expect around 70% accuracy.

On the other hand, when we say Emulation (or WAN Emulation) we mean that the application traffic will traverse in real time through a path that is identical to the path we have in mind. An appliance based WAN Emulator or software based WAN Emulation is usually present to create actual network elements in software, bring into account network impairment characteristics, such as distance (latency), bandwidth and loss. The accuracy of WAN Emulation can exceed 99% depending on the level of configuration. We can expect around 95% accuracy from a straight forward automatic configuration.

Another important difference is time. Theoretical WAN simulation may take days to run, but using WAN Emulation the duration is based on the time it takes the application to run.

The main reason why WAN Emulation triumphs is accuracy.

Recently, I visited a company that develops and distributes software for Independent School District’s across USA. Their main product, originally developed for challenged kids, centers around dynamic computerized teaching and testing programs. This product proved to be so successful among this population that the company adapted it for other kids as well. Initially the product was designed to work in the classroom, hence a very simple architecture: 1 PC working as a learning manager.  This 1 PC managed the progress of individual students, as well as received and captured data back to computers across the LAN. However, because of such a high demand, the company decided to host a solution for multiple customers from a central location, specifically – a Cloud hosted server.

Since the original design was created with only a classroom setting in mind, testing and development only occurred over LAN. But now, with accessibility from different regions, including links from APAC countries to US, consideration of the WAN was critical. This required the need to test their solution in a new environment and see if any problems exist.  By analyzing this information, they gain a better understanding of how to fix any problems. With this challenge, the company needed a WAN emulator solution that would allow them to re-create the complex topology of the application served over WAN.

Once tested, they were able to easily reproduce the problems reported to the QA from the field. One issue that was noted – disconnections without the ability to reconnect, was leading to a loss of data and historical progress of the students in the program.  One of the key tools is tracking “task lists.”  One of the main advantages of the system is the ability to produce these new “task lists” for each student, based on historical data retrieved.  The new tasks matched the student’s progress in learning the subjects and fulfilling the tasks.

One other obstacle the company faced dealt with the USD security measures.  Since product was originally operating on a LAN environment, the security of outside access was a concern. As a result, certain ports were blocked and the company needed to understand how those ports will be affected from blocks in different directions of the link. Shunra VE WAN Emulator was able to reproduce such conditions by filtering application traffic based on protocol and ports, as well as IP addresses and blocking them in one direction – the problems were reproduced instantly. Now QA no longer has to rely on the reports generated from the field to try and fix the problems “in the dark”…  They now have a nice “flash light” to expose those reasons for the problem.

The impact of a broken link or just a simple delay in logging in can significantly impact students, especially the challenged ones.  Picture a child struggling to connect to the program, no sounds, no pictures… instead sitting with headphones and seeing on a screen: “Connecting…”  An unnerving experience for all of us, and even less for them (IMHO).  And what if after working for 20-30 minutes on a progressive task – they have to start over because computer lost all memory of what they were doing until now.

PCoIP is a type of display protocol used by remote desktops when doing desktop virtualization.  It delivers the desktop from the host server to the remote user, along with capturing mice and keyboard inputs.  PCoIP also delivers HD audio, USB and the user display as well. PCoIP will enable VMware View to deliver the true PC experience that end-users are seeking, capturing high-resolution graphics and rich multimedia, as well as streaming movies and high-definition audio capabilities. In most cases, the user cannot tell that their PC is not at their desktop anymore.

While PCoIP implementations today are all hardware-based, VMware’s implementation will be all software based (Soft PCoIP) and will not require any additional hardware on either end. VMware’s software implementation of PCoIP uses Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) over port 50002. The TCP port is used for session establishment and control.  The UDP port is used for optimal performance of media and streaming content. All traffic between the host server and remote user is encrypted with 128-bit Advanced Encryption Standard encryption.

VMware will use the software-only implementation of PC-over-IP to support users with 150-250ms latency. They’ll give them “basic” flash (think YouTube videos, not full-on Flash apps), voice over IP, remote printers, local storage, etc.

The implication is that you must plan and test for the deployment over a WAN environment that will produce realistic results, for example using Shunra Virtual Enterprise, to ensure user satisfaction and complete SLA compliance.

That sounds like a lot of clouds!  What we are simply trying to understand is where and how to deploy your application in the cloud, so it will perform well for your customers.

If you take the case of deploying an online store into the Amazon EC2 cloud, there are many things to consider that will impact your customer’s experience.  Two of the most important are where to deploy what, and how your store will function over the Internet.  To compound the issue, where you physically deploy machines within EC2 is not clear.  Amazon goes to great lengths to avoid you knowing where their zones are.

Fortunately, you cannot only find solutions to these problems, but with cloud computing the process gets a lot easier.  To break the problems down into a logical flow, you’ll first need to know the demographics of your customers.  Where are they?  How will they access the store?  What needs to happen for them to have a positive experience at your store?  If you can answer these questions, you’ll likely come up with some numbers that place percentages of your users in different regions around your target market (US, EMEA, global, etc.).  Further, you’ll know the top two browsers they’ll use and with what kind of connections they’ll access the internet (e.g. Firefox with a 768/256kbps DSL, or IE8 with dial-up – yes people still use dial-up!).  Finally, you’ll have some way to quantify a “positive user experience”.  That usually involves consistency (does it work?) and speed (how fast?).

Using the information above and HP LoadRunner with WAN emulation, you can quickly build a to-scale test in your cloud-based lab and understand what would happen if you hosted your application all in one place and serviced your entire customer base.  Odds are that the test will not meet all of your performance goals the first time you run it.  Most likely, you will need to adjust some things in your store (e.g. reduce chattiness and image resolution, optimize Web 2.0 use) and then begin the discussion about where to deploy what.  Can you keep all of your database servers in one place?  If you have to split your database servers, how will that affect performance?  Will you need to deploy web servers in every zone, will you be okay with one in each region, or will you need something in-between?

Again, not only can answers to these questions actually be answered, but by using cloud-based testing you can arrive at valid conclusions faster than ever.  To understand the network impact of deploying machines in different zones, network performance measurement tools like VE Network Catcher can run in each of the zones to measure network performance between zones and to customer representative endpoints.  Importing that data directly into HP LoadRunner with WAN emulation and using the flexible cloud environment, you can rapidly form and either dismiss or validate hypotheses about how your application should best be deployed.

The benefit of testing in this way is that you can experiment with many different configurations in rapid succession to find the one or few that are most appropriate for your needs.  Trial and error with a live application is not advisable.  And building a test lab with physical machines to simulate the same thing is not only significantly more expensive, but a lot slower process.  As I pointed out in my previous blog, the same ROI formula that explain and fuel the explosion of Cloud Computing for hosting applications are also relevant for using Cloud Computing to test your applications with.

Now you know what I think.  What do you think??