Application Performance Engineering Blog – Shunra Software

In my line of work, I get to assist many clients with analyzing the impact of an upcoming data center relocation on the performance of business applications. Many clients don’t know exactly what application performance to expect post the data center move. The main concern is that once all the applications start to operate across a Wide Area Network (WAN) connection to a remote data center with all the added network latency, the future performance of applications is unpredictable at best and potentially becomes so noticeable that it hinders the success of the data center relocation project itself. Therefore, clients often ask me, “Should we really test all the applications for latency? Which applications are the most vulnerable to the added network latency? Are there best practices when it comes to designing distributed applications that need to operate across network latency?”

In the next 2 posts, I will address the first 2 questions, the third one is probably a good topic for a book, so I will attempt to address that one over several future posts.

Should you really test all your applications for latency prior to a data center relocation?

Before jumping into a direct answer with examples of horror stories from past data center relocations that didn’t test all applications and the consequence of that excessive risk taking I want to first provide a conceptual best practice answer.

I recently received my Six Sigma Champion certification and one of the concepts in Six Sigma and in management for quality in general is “Don’t mass inspect, build quality into the process instead”. This concept was one of Deming’s teachings that revolutionized the Japanese manufacturing industry in the 1950s. So one would think that according to that best practice, we shouldn’t test all applications prior to a data center move, since that would be the equivalent of mass inspection. However, that would be a miss interpretation of that statement, an organization can shift from mass inspection to more statistical sampling only if quality is built into the process and when mission critical processes are in control. To complete the analogy, an organization can settle for sample testing a subset of applications only if application development processes have quality built into them, and specifically with a data center move, application development processes have network aware best practices embedded in them. I have yet to find an organization that reaches that level of maturity prior to a data center move. Usually the causation is reverse, first a firm relocates their data center, then IT realizes that they need to adopt a new application development paradigm since from now on applications will access remote servers in a remote data center and thus network aware best practices (like using VE Desktop as part of the development life cycle) are added.

So this brings us to the reasonable assumption that most companies will have a portfolio of applications that weren’t developed with a distributed WAN in mind and now have to adjust to operating in a remote data center. If that is the scenario your company is facing then mass inspection is your only way to mitigate the enormous risk associated with such a move.

Now if you don’t buy into the conceptual answer, consider the following:

1. In an average data center relocation project, we usually find that 20% – 40% of applications experience performance degradation ranging from visible to severe. (visible means that a user notices the slowdown in response time, severe means that an application stops being usable under latency conditions).
2. Many of the problems we uncover end up being show stoppers for the DCR project and need to go through remediation before the project can complete.
3. The mitigation path varies between applications and is very dependent on the type of problem that causes the performance degradation. Hence, a typical remediation solution may include WAN Acceleration for some applications (mainly server-to-server copy utilities and file services), terminal services (Citrix, remote desktop, VDI, etc.) for other applications, code fixes for some applications and architecture changes for the rest. Therefore engineering a solution based on a sample of applications will results in a sub optimal solution that doesn’t address all the performance problems that will manifest post the move.
4. When a large NY based insurance company relocated their servers from NY to Atlanta, GA without testing, they ended up rolling back the first phase of the relocation when several mission critical applications became unusable due to poor performance.

In summary, unless IT has built network aware best practices into the software development process, an organization planning a data center relocation must test all of its applications to analyze the performance impact that new added latency will have on application performance. More best practices on this topic can be found in my recently updated white paper that can be found at http://www.shunra.com/predicting-the-impact-of-data-center-moves-on-application-performance-whitepaper.php

The next post will cover the question “Which applications are the most vulnerable to the added network latency?” small example, any applications that are running executables from a shared network drive, will experience severe performance problems, that and more will be covered in the next post.

In the meantime, I am curious to hear your experience with data center relocations, what approach did your IT group take? Test all, sample test or cross your fingers and just move? Let me know.

P.S. as part of my Six Sigma Certification, I developed a dashboard and a flow chart for a data center relocation performance analysis, it needs some work before it can be shared, but if enough readers ask for it, I will try to clean it up and post it.

Till next time,

Amichai Lesser