Call (+1) 650-212-1212

Blog

Finding Out What You Don’t Know With Sensitivity Analysis

Sean Salleh 12 Jun 2013 Modeling methods

Remember Donald Rumsfeld’s speech from 2007 on the things we know and the things we don’t? His enumeration of known knowns, known unknowns and unknown unknowns garnered ratings from trivial to deeply meaningful. Fishing for truth in such murky waters is not just a governmental problem, but help (to a certain degree) is at hand. So Donald, how about a little sensitivity analysis to at least see what happens if the variables we know about deviate from expectations? And better still, how about some uncertainty or importance analysis to go with that?

Table with importance of energy How the Government Keeps Track of Importance (Relative Weight)

Our Old Friend “What-If” Modeling

If you’ve built your model with N perhaps slightly shaky variables and you want to check out different possibilities for outcomes, then of course you can simply set up a number of different scenarios. For each scenario, you choose different values for different variables, run the model and record the outcome. If you’ve made a basic or trial model for what-if modeling in a spread-sheet, you can do this any number of times by changing a value (or values) in the cells representing variables. Then (keeping careful not of what you changed where), you can scoot over to the result to see what the impact would be.

Sensitivity Analysis Done Backwards

If the sensitivity of a model turns out to be very high for a particular variable, you’ll know that the model is that much more unreliable in that case. The analysis in fact lets you go back and beef up the accuracy (and perhaps also the precision) of the estimated value for the variable, in order to have a more realistic model. You can also bounce the model around by using a wide range of values to see if there are any unexpected changes in extreme situations: for example, if a certain level of product sales suddenly opens the door to market leadership for that product.

Being More Certain About Uncertainty

Variable by variable sensitivity analysis is still a primitive approach when you want a global view of what’s really driving the results from your model. It’s better if you can compare sensitivities and see where the big changes in results will arise if a variable changes in value. Understanding modeling uncertainty like this can become an exercise in holism and probability, with the need to accept that you may gain additional insights rather than the absolute truth.

Chart of relative importance of variables Importance Chart in Analytica (Market Size will Cause the Biggest Variability in NPV)

Military, Marketing and Much More

If you accept the approximations in determining how much uncertainty there is in a model, the applications are highly varied. They can cover anything from military strategy to understanding which foods work best for dieting. Marketers dream of knowing what the one thing is that determines whether or not their product or service will be a hit. This case of the relative importance of marketing predictors is complex: it involves decisions on customer survey methods and the inter-correlated data they generate, with the consequent challenge of picking out the minimum set of design criteria that really influence customer satisfaction.

Table with market predictors Market Predictors - Courtesy of Flickr / PC Magazine

Not every dive into uncertainty or sensitivity analysis is crowned with success, however. Some subjects like global warming models are notorious for dividing opinion or simply leading people astray. Common sense still has a part to play, corresponding to the one item that Donald Rumsfeld’s taxonomy of certainties and uncertainties did not cover: the things we didn’t know that we already knew.

» Back

Sean Salleh

Sean Salleh is a data scientist with experience in guiding marketing strategy from building marketing mix models, forecasting models, scenario planning models, and algorithms. He is passionate about consumer technologies and resource management. He has master's degrees in Operations Research from University of California Irvine and Mathematics from Northeastern University.

Leave a Comment