Call (+1) 650-212-1212

Blog

Sensitivity Analysis: One at a Time or All Together?

Sean Salleh 15 Aug 2013 Risk and uncertainty

The whole is greater than the sum of the parts, say Aristotle and Gestalt psychology. The whole effect of combined factors of a model in terms of sensitivity may also be greater than the effect of any individual effect of a factor. In other words, if you’re trying to find out what contributes the most uncertainty to a model, you may be missing something if you only look at the effect of a single factor. Yet given that models and their variations need to be understandable to different audiences, how complicated should you make your sensitivity analysis anyway?

An optional representation of sensitivity as a tornado graph Image source: 8020world.com

What is Sensitivity Good For?

The short version is model understanding, model development, decision making, and communication. By varying one or more factors, deeper understanding can be developed of the relationship between input and output. The model can also be developed to become more correct, simpler, and more robust if data is missing or of poor quality. Priorities can be assigned in terms of data acquisition.  Decision making can be improved by identifying how and when optimal solutions change, or producing flexible recommendations that take account of changing circumstances. Communication of these decisions using the results of sensitivity analysis can be made more understandable or more credible.

One at a Time Variation in Sensitivity Analysis

The simplest way to approach sensitivity analysis is to vary each factor one at a time (OAT). The big advantage is that any changes (including model failure) observed can be ascribed to the change in that one factor. While one factor is being varied, the others are usually kept at their nominal or baseline values, so that each OAT change is effected with reference to a certain model ‘base space’.

Limitations of an OAT Approach

Varying one factor at a time does not allow for identification of correlation between variables. In some contexts, such correlation or interaction may be considerable. In business for instance, a supply chain is often an interlinked system of input factors with certain individual and collective constraints. Likewise, revenue models will depend on sales volume and sales pricing, but these two factors are also often interlinked. The elasticity of a market means that a business will sell more products at a lower price per product, subject again to certain constraints such as the need to make a per-product profit.

price elasticity curves Image source: commons.wikimedia.org

Combinations of Factors

Depending on the size of the model and the number of factors, testing different combinations of factors for sensitivity may be realistic. When running simulations of models using Monte Carlo techniques, Latin hypercube sampling may represent the best compromise between efficient coverage of different combinations and realistic assessment of their combined effect on sensitivity. Alternatively, sensitivity analysis can also be performed on sub-models of an overall model to see the effect of particular combinations of factors.

Models of Models

Moving up a level, different models can be run in response to the same problem or question, and yield different results. Some models may yield very different conclusions from the rest of the models. The sensitivity of overall conclusions to such models can in turn be assessed: for instance, by observing the effect of including or excluding the model in the overall spectrum of results obtained.

If you’d like to know how Analytica, the modeling software from Lumina, can help you do sensitivity analyses and automatically rank factors in order to sensitivity importance, then try a thirty day free evaluation of Analytica to see what it can do for you.

» Back

Sean Salleh

Sean Salleh is a data scientist with experience in guiding marketing strategy from building marketing mix models, forecasting models, scenario planning models, and algorithms. He is passionate about consumer technologies and resource management. He has master's degrees in Operations Research from University of California Irvine and Mathematics from Northeastern University.

Masoud Saatsaz

2017 07 09

Dear Sean Salleh,
I want to carry out an analysis of the sensitivity in a landfill suitability evaluation project performed by AHP method. In AHP all criteria weights should always equal to 1 . In order to meet above condition, is it needed that the weights of the other criteria tol be adjusted proportionally to one when the weight of a criterion is altered during One-At-a-Time approach?

best regards,

Masoud

Dale Rice

2017 07 10

In the AHP example, you’ll probably have an index, let’s say it is named Alternative, with one element for each alternative, and a set of weights W that is indexes by Alternative, with 0<=W<=1 and Sum(W,Alternative)=1. Your sensitivity analysis needs to twiddle one weight at a time, i.e., compute a WhatIf when W[@Alternative=1], and then compute a WhatIf when W[@Alternative=2] is twiddled, and so on.

The first challenge is to determine how much each weight should be twiddled. Let’s say you decide to twiddle by 10%. When doing so, don’t use 1.1 * W to increase by 10%, because that could result in weights greater than 1.0. Instead, use W - W*10% and W + (1-W)*10%. Let’s use an index:

Index Level := [‘Low’,‘High’]
And then the whatIf value that we can use is this:  W + Array(Level, [-10% * W, 10% * (1-W)] )

When we change the weight for the one alternative, we need to adjust all the others to retain the sum=1. So the full sensitivity Analysis would look like this:

Var w1 := W + Array(Level, [-10% * W, 10% * (1-W)] );

For a := Alternative Do (
  Var s := Sum( (a<>Alternative)*W, Alternative);
  Var wVec := if a=Alternative then w1 else if s=0 then 0 else W * (1-W1[Alternative=a]) / s;
  WhatIf( resultVar, W, wVec )
)

I computed wVec here so that the value for Alternative=a is w1, and the other values are scaled to preserve their original ratios while also preserving the sum=1.

Finally, there is one final challenge that arises when W has other indexes in addition to Alternative. In AHP, this would be the case when you have W matrices, indexed by Alternative and Criterion. The above would work, but would alter each Alternative-slice by 10% all-at-once, and hence you’d be calculating the sensitivity for what would happen if Alternative=1 became 10% more preferred simultaneously for every Criterion. That might be the sensitivity you want to compute, but another option would be to compute the sensitivity to each individual Alternative-Criterion weight, and hence you’ll need to iterate over Criterion as well like this:

Var w1 := W + Array(Level, [-10% * W, 10% * (1-W)] );

For a := Alternative Do (
  Var s := Sum( (a<>Alternative)*W, Alternative);
  For c := Criterion Do (
    Var wVec := If c<>Criterion then W
            Else if a=Alternative then w1
            Else if s=0 then 0
            Else W * (1-W1[Alternative=a,Criterion=c]) / s;
    WhatIf( resultVar, W, wVec )
  )
)

Dale Rice
Lumina Decision Systems

Leave a Comment