Application Performance Engineering Blog – Shunra Software

I have heard every new generation of performance testers debate the same old things:

• What’s the difference between load and stress testing?
• What’s the definition of performance testing vs. load testing?
• What are all the different types of performance testing and when to conduct them?
• How is performance engineering different from performance testing?

Oddly, we rarely hear from customers the following question: “What’s the definition of real-world performance testing?”

In a few short explanations, I’d like to share my own ideas on this subject.

I’ll start by asking this – shouldn’t *all* testing be considered “real-world” testing, actually? I mean, if we spend days, weeks and months testing software and logging bugs – aren’t we assuming that those bugs (of any type) will impact end users in the real world? In this way, functional testing, unit testing, performance testing and even development itself is “real-world”. Perhaps we don’t think this way because it seems overly obvious or presumed in the nature of our work. Scott Barber recently posted a plea about making software development “Mission Focused” where I believe the mission he refers to is delivering software which will be used to impact something (for some purpose) in the real world.

Recently I’ve also seen numerous occurances of the “Real World” terminology over at uTest.com where the marketing department has gotten a hold of the “real-world” terminology and decided to use it as a competitive differentiation in their positioning.  Although they have a very compelling story about real testers in the real world conduting testing from “all over the world” I’m not sure that this is the only feasible definition we should accept; especially considering it probably came from the marketing department.  Is it good enough to just say it’s “real-world testing” just because it was done by a human being outside your company – in the crowd-sourced uTest.com ecosystem?  How is that different from the human beings who are testing software already?

Here’s the definition I would use to describe real-world testing: “any testing effort that defines a test criteria where a direct non-technical dependency is required for evaluation of the test result.” What this means to me is that if you have a test that is validating something internal to the application, and the specified pass/fail criteria does *not* include a non-technical (e.g. “human or living being”) dependency, then it is *not* a real world test.  Sure, it’s might be a perfectly valid technical test, but it’s not directly related to a component or function that will ever affect something or someone in the real world.  The operating idea is here is direct vs. indirect relation of the test case to the real world. For real-world performance testing, let’s consider the test design as: “any performance testing effort that includes test environment configuration and test execution constraints derived from the conditions of a non-technical entity which has dependency on the performance of the system under test.”  Like with mobile performance, if end-users have to wait…they will abandon and uninstall your app.

We often think of real world testing as being related to “the worst case scenario that could ever happen” in the form of a negative test case. Consider the o-rings which were cited as the indirect cause of the space shuttle disaster back in 1986.  Even though the o-rings may have been tested for their direct technical integration points or the dependencies that direct related to their functionality or performance as components, the complete test criteria perhaps didn’t include all the dependencies and conditions that would eventually affect the o-ring in the real world.  It’s not that we didn’t test the o-rings. We just didn’t conduct real-world testing of them to find exact conditions for their eminent failure.

Speaking of rocket science, I recently spoke with a very enlightened and experience architect at one of my customers who actually was a rocket scientist in a previous career. This sparked a unique conversation about the validity of arithmetical extrapolation to real-world results, which he stated [paraphrase] “was often required in the development and testing to build rockets. You couldn’t afford to test them in the real world, so we had to conduct models.” This makes sense. Rockets are expensive and executing a negative test could cause lots of blown-up rockets all over the place.

There’s also that scene in the movie Office Space where Tom Smykowski is explaining to “the Bobs” about exactly why software developers and testers are sometimes separated from the real world. Which may lead you to jump to a conclusion that people skills are a pre-requisite for conducting real-world testing.