Chapter 5

A Review on Global Sensitivity

Analysis Methods

Bertrand Iooss and Paul Lemaître

5.1 Introduction

While building and using numerical simulation models, Sensitivity Analysis (SA)
methods are invaluable tools. They allow to study how the uncertainty in the output
of a model can be apportioned to different sources of uncertainty in the model
input (Saltelli et al. [77]). It may be used to determine the most contributing
input variables to an output behavior as the non-influential inputs, or ascertain
some interaction effects within the model. The objectives of SA are numerous; one
can mention model verification and understanding, model simplifying and factor
prioritization. Finally, the SA is an aid in the validation of a computer code, guidance
research efforts, or the justification in terms of system design safety.
There are many application examples, for instance Makowski et al. [58] analyze,
for a crop model prediction, the contribution of 13 genetic parameters on the
variance of two outputs. Another example is given in the work of Lefebvre et al.
[52] where the aim of SA is to determine the most influential input among a large
number (around 30), for an aircraft infrared signature simulation model. In nuclear
engineering field, Auder et al. [2] study the influential inputs on thermohydraulical
phenomena occurring during an accidental scenario, while Iooss et al. [37] and
Volkova et al. [92] consider the environmental assessment of industrial facilities.

B. Iooss ()
EDF R&D, 6 Quai Watier - 78401 Chatou, France
Institut de Mathématiques de Toulouse, 118 route de Narbonne - 31062 Toulouse, France
e-mail: [email protected]
P. Lemaître
EDF R&D, 6 Quai Watier - 78401 Chatou, France
INRIA Sud-Ouest, 351 cours de la libération - 33405 Talence, France

© Springer Science+Business Media New York 2015 101

G. Dellino, C. Meloni (eds.), Uncertainty Management in Simulation-Optimization
of Complex Systems, Operations Research/Computer Science Interfaces Series 59,
DOI 10.1007/978-1-4899-7547-8_5
102 B. Iooss and P. Lemaître

The first historical approach to SA is known as the local approach. The impact of
small input perturbations on the model output is studied. These small perturbations
occur around nominal values (the mean of a random variable for instance). This
deterministic approach consists in calculating or estimating the partial derivatives
of the model at a specific point. The use of adjoint-based methods allows to process
models with a large number of input variables. Such approaches are commonly
used in solving large environmental systems as in climate modeling, oceanography,
hydrology, etc. (Cacuci [9], Castaings et al. [13]).
From the late 1980s, to overcome the limitations of local methods (linearity and
normality assumptions, local variations), a new class of methods has been developed
in a statistical framework. In contrast to local sensitivity analysis, it is referred to
as “global sensitivity analysis” because it considers the whole variation range of
the inputs (Saltelli et al. [77]). Numerical model users and modelers have shown
large interests in these tools which take full advantages of the advent on computing
materials and numerical methods (see Helton [30], de Rocquigny et al. [19] and
Faivre et al. [23] for industrial and environmental applications). Saltelli et al. [78]
and Pappenberger et al. [67] emphasized the need to specify clearly the objectives
of a study before making a SA. These objectives may include:
– identify and prioritize the most influential inputs,
– identify non-influential inputs in order to fix them to nominal values,
– map the output behavior in function of the inputs by focusing on a specific
domain of inputs if necessary,
– calibrate some model inputs using some available information (real output
observations, constraints, etc.).
With respect to such objectives, first syntheses on the subject of SA were
developed (Kleijnen [43], Frey and Patil [25], Helton et al. [32], Badea and Bolado
[4], de Rocquigny et al. [19], Pappenberger et al. [67]). Unfortunately, between
heuristics, graphical tools, design of experiments theory, Monte Carlo techniques,
statistical learning methods, etc., beginners and non-specialist users can be found
quickly lost on the choice of the most suitable methods for their problem. The
aim of this chapter is to provide an educational synthesis of SA methods inside
an applicative methodological framework.
The model input vector is denoted X D .X1 ; : : : ; Xd / 2 Rd . For the sake of
simplicity, we restrict the study to a scalar output Y 2 R of the computer code (also
called “model”) f ./:

Y D f .X/ : (5.1)

In the probabilistic setting, X is a random vector defined by a probability distribution

and Y is a random variable. In the following, the inputs Xi (i D 1 : : : d ) are assumed
to be independent. More advanced works, listed in the last section, take into account
the dependence between components of X (see Kurowicka and Cooke [48] for an
introduction to this issue). Finally, this review focuses on the SA with respect to the
global variability of the model output, usually measured by its variance.
5 Global Sensitivity Analysis 103

Fig. 5.1 The flood example:

simplified model of a river
Q

Zm
Zu Ks

B
Zc
L

All along this chapter, we illustrate our discussion with a simple application
model that simulates the height of a river and compares it to the height of a dyke that
protects industrial facilities (Fig. 5.1). When the river height exceeds the one of the
dyke, flooding occurs. This academic model is used as a pedagogical example in de
Rocquigny [18] and Iooss [35]. The model is based on a crude simplification of the
1D hydro-dynamical equations of SaintVenant under the assumptions of uniform
and constant flowrate and large rectangular sections. It consists of an equation that
involves the characteristics of the river stretch:
0 10:6
B Q C
S D Z v C H  H d  Cb with H D @ q A ; (5.2)
Zm Zv
BKs L

where S is the maximal annual overflow (in meters), H is the maximal annual
height of the river (in meters) and the other variables (d D 8 inputs) are defined
in Table 5.1 with their probability distribution. Among the input variables of the
model, Hd is a design parameter. Its variation range corresponds to a design domain.
The randomness of the other variables is due to their spatio-temporal variability, our
ignorance of their true value or some inaccuracies of their estimation. We suppose
that the input variables are independent.
We also consider another model output: the associated cost (in million euros) of
the dyke,
h  1000
i 1  
Cp D 1 S>0 C 0:2 C 0:8 1  exp S 4 1 S0 C Hd 1 Hd >8 C 81 Hd 8 ; (5.3)
20

with 1 A .x/ the indicator function which is equal to 1 for x 2 A and 0 otherwise.
In this equation, the first term represents the cost due to a flooding (S > 0) which
104 B. Iooss and P. Lemaître

Table 5.1 Input variables of the flood model and their probability distributions
Input Description Unit Probability distribution
Q Maximal annual flowrate m3 /s Truncated gumbel G .1013; 558/ on Œ500; 3000
Ks Strickler coefficient – Truncated normal N .30; 8/ on Œ15; C1Œ
Zv River downstream level m Triangular T .49; 50; 51/
Zm River upstream level m Triangular T .54; 55; 56/
Hd Dyke height m Uniform U Œ7; 9
Cb Bank level m Triangular T .55; 55:5; 56/
L Length of the river stretch m Triangular T .4990; 5000; 5010/
B River width m Triangular T .295; 300; 305/

is 1 million euros, the second term corresponds to the cost of the dyke maintenance
(S  0) and the third term is the investment cost related to the construction of the
dyke. The latter cost is constant for a height of dyke less than 8 m and is growing
proportionally with respect to the dyke height otherwise.
The following section discusses the so-called screening methods, which are
qualitative methods for studying sensitivities on models containing several tens of
input variables. The most used quantitative measures of influence are described in
the third section. The fourth section deals with more advanced tools, which aim
to provide a subtle exploration of the model output behavior. Finally, a conclusion
provides a classification of these methods and a flowchart for practitioners. It also
discusses some open problems in SA.

5.2 Screening Techniques

Screening methods are based on a discretization of the inputs in levels, allowing a

fast exploration of the code behaviour. These methods are adapted to a large number
of inputs; practice has often shown that only a small number of inputs are influential.
The aim of this type of method is to identify the non-influential inputs with a small
number of model calls while making realistic hypotheses on the model complexity.
The model is therefore simplified before using other SA methods, more subtle but
more costly.
The most engineering-used screening method is based on the so-called “One At a
Time” (OAT) design, where each input is varied while fixing the others (see Saltelli
and Annoni [74] for a critique of this basic method). In this section, the choice has
been made to present the Morris method [65], which is the most complete and most
costly one. However, when the number of experiments has to be smaller than the
number of inputs, one can quote the usefulness of the supersaturated design (Lin
[56]), the screening by groups (Dean and Lewis [20]) and the sequential bifurcation
method (Bettonvil and Kleijnen [5]). When the number of experiments is of the same
order than the number of inputs, the classical theory of experimental design applies
(Montgomery [64]) for example with the so-called factorial fractional design.
5 Global Sensitivity Analysis 105

The method of Morris allows to classify the inputs in three groups: inputs
having negligible effects, inputs having large linear effects without interactions and
inputs having large non-linear and/or interaction effects. The method consists in
discretizing the input space for each variable, then performing a given number of
OAT design. Such designs of experiments are randomly chosen in the input space,
and the variation direction is also random. The repetition of these steps allows
the estimation of elementary effects for each input. From these effects, sensitivity
indices are derived.
Let us denote r the number of OAT designs (Saltelli et al. [78] propose to set
parameter r between 4 and 10). Let us discretize the input space in a d -dimensional
.i/
grid with n levels by input. Let us denote Ej the elementary effect of the j th
variable obtained at the i th repetition, defined as:

.i/ f .X.i/ C 4ej /  f .X.i/ /

Ej D (5.4)
4

where 4 is a predetermined multiple of .n1/

1
and ej a vector of the canonical base.
Indices are obtained as follows:
1 X .i/
r
– j D jE j (mean of the absolute value of the elementary effects),
r iD1 j
v !2
u r
u1 X 1 X .i/
r
– j D t .i/
Ej  E (standard deviation of the elementary
r iD1 r iD1 j
effects).
The interpretation of the indices is the following:
– j is a measure of influence of the j th input on the output. The larger j is, the
more the j th input contributes to the dispersion of the output.
– j is a measure of non-linear and/or interaction effects of the j th input. If j is
small, elementary effects have low variations on the support of the input. Thus
the effect of a perturbation is the same all along the support, suggesting a linear
relationship between the studied input and the output. On the other hand, the
larger j is, the less likely the linearity hypothesis is. Thus a variable with a large
j will be considered having non-linear effects, or being implied in an interaction
with at least one other variable.
Then, a graph linking j and j allows to distinguish the three groups.
Morris method is applied on the flood example [Eqs. (5.2) and (5.3)] with r D 5
repetitions, which require n D r.d C 1/ D 45 model calls. Figure 5.2 plots results
on the graph .j ; j /. This visualisation allows to make the following discussion:
– output S : inputs Ks , Zv , Q, Cb et Hd are influent, while other inputs have no
effects. In addition, the model output linearly depends on the inputs and there is
no input interaction (because j  j 8j ).
106 B. Iooss and P. Lemaître

Fig. 5.2 Results of Morris method (r D 5 with 4 levels): outputs S (left) and Cp (right)

– output Cp : inputs Hd , Q, Zv et Ks have strong influence with non-linear and/or

interaction effects (because j and j have the same order of magnitude). Cb
has an average influence while the other inputs have no influence.
Finally, after this screening phase, we have identified that three inputs (L, B and
Zm ) have no influence on the two model outputs In the following, we fix these three
inputs to their nominal values (which are the modes of their respective triangular
distributions).

5.3 Importance Measures

5.3.1 Methods Based on the Analysis of Linear Models

 
.i/ .i/
If a sample of inputs and outputs .Xn ; Yn / D X1 ; : : : ; Xd ; Yi is available,
iD1::n
it is possible to fit a linear model explaining the behaviour of Y given the values of
X, provided that the sample size n is sufficiently large (at least n > d ). Some global
sensitivity measures defined through the study of the fitted model are presented in
the following. Main indices are:
– Pearson correlation coefficient:
PN .i/
 E.Xj //.Yi  E.Y //
iD1 .Xj
.Xj ; Y / D v v : (5.5)
uN  2 u
uX uXN
t Xj  E.Xj / t
.i/
.Yi  E.Y //2
iD1 iD1
5 Global Sensitivity Analysis 107

It can be seen as a linearity measure between variable Xj and output Y . It equals

1 or 1 if the tested input variable has a linear relationship with the output. If Xj
and Y are independents, the index equals 0.
– Standard Regression Coefficient (SRC):
s
Var.Xj /
SRCj D ˇj (5.6)
Var.Y /

where ˇj is the linear regression coefficient associated to Xj . SRC2j represents a

share of variance if the linearity hypothesis is confirmed.
– Partial Correlation Coefficient (PCC):

b
PCCj D .Xj  Xj ; Y  Yj / b (5.7)

b
where Xj is the prediction of the linear model, expressing Xj with respect to
b
the other inputs and Yj is the prediction of the linear model where Xj is absent.
PCC measures the sensitivity of Y to Xj when the effects of the other inputs have
been canceled.
The estimation of these sensitivity indices is subject to an uncertainty estimation,
due to the limited size of the sample. Analytical formulas can be applied in order to
estimate this uncertainty (Christensen [15]).
These three indices are based on a linear relationship between the output and the
inputs. Statistical techniques allow to confirm the linear hypothesis, as the classical
coefficient of determination R2 and the predictivity coefficient Q2 (also called the
Nash-Sutcliffe model efficiency):
Pm
ŒYi  YO .Xp.i/ /2
p
Q D 1  PiD1
2
m p (5.8)
iD1 ŒYi  E.Y /
p 2

p
where .Xp.i/ ; Yi /iD1::m is a m-size test sample of inputs-output (not used for the
model fitting) and YO ./ is the predictor of the linear regression model. The value
of Q2 corresponds to the percentage of output variability explained by the linear
regression model (a value equals to 1 means a perfect fit). If the input variables
are independent, each SRC2j expresses the part of output variance explained by the
input Xj .
If the linear hypothesis is contradicted, one can use the same three importance
measures (correlation coefficient, SRC and PCC) than previously using a rank
transformation (Saltelli et al. [77]). The sample .Xn ; Yn / is transformed into a
sample .RnX ; RnY / by replacing the values by their ranks in each column of the
matrix. As importance measures, it gives the Spearman correlation coefficient
S , the Standardized Rank Regression Coefficient (SRRC) and the Partial Rank
Correlation Coefficient (PRCC). Of course, monotony hypothesis has to be validated
as in the previous case, with the determination coefficient of the ranks (R2 ) and the
predictivity coefficient of the ranks (Q2 ).
108 B. Iooss and P. Lemaître

These linear and rank-based measures are part of the so-called sampling-based
global sensitivity analysis method. This has been deeply studied by Helton and
Davis [31] who have shown the interest to use a Latin Hypercube Sample (Mc Kay
et al. [63]) in place of a Monte Carlo sample, in order to increase the accuracy of
the sensitivity indices.
These methods are now applied on the flood example [Eqs. (5.2) and (5.3)] with
the d D 5 inputs that have been identified as influent in the previous screening
exercise. A Monte Carlo sample of size n D 100 gives 100 model evaluations.
Results are the following:
– output S :
SRC2 .Q/ D 0:28; SRC2 .Ks / D 0:12; SRC2 .Zv / D 0:15; SRC2 .Hd / D
0:26; SRC2 .Cb / D 0:03 with R2 D 0:98;
SRRC2 .Q/ D 0:27; SRRC2 .Ks / D 0:12; SRRC2 .Zv / D 0:13;
SRRC2 .Hd / D 0:26; SRRC2 .Cb / D 0:02 with R2 D 0:95;
– output Cp :
SRC2 .Q/ D 0:25; SRC2 .Ks / D 0:16; SRC2 .Zv / D 0:18; SRC2 .Hd / D
0:00; SRC2 .Cb / D 0:07 with R2 D 0:70;
SRRC2 .Q/ D 0:26; SRRC2 .Ks / D 0:19; SRRC2 .Zv / D 0:18;
SRRC2 .Hd / D 0:06; SRRC2 .Cb / D 0:03 with R2 D 0:73.
For the output S , R2 is close to one, which shows a good fit of linear model on the
data. Analysis of regression residuals confirms this result. Variance-based sensitivity
indices are given using SRC2 . For the output Cp , R2 and R2 are not close to
one, showing that the relation is neither linear nor monotonic. SRC2 and SRRC2
indices can be used in a coarse approximation, knowing that it remains 30 % of non-
explained variance. However, using another Monte Carlo sample, sensitivity indices
values can be noticeably different. Increasing the precision of these sensitivity
indices would require a large increase of the sample size.

5.3.2 Functional Decomposition of Variance: Sobol’ Indices

When the model is non-linear and non-monotonic, the decomposition of the output
variance is still defined and can be used for SA. Let us have f .:/ a square-integrable
function, defined on the unit hypercube Œ0; 1d . It is possible to represent this
function as a sum of elementary functions (Hoeffding [33]):

X
d X
d
f .X/ D f0 C fi .Xi / C fij .Xi ; Xj / C    C f12:::d .X/ : (5.9)
iD1 i<j
5 Global Sensitivity Analysis 109

This expansion is unique under conditions (Sobol [83]):

Z 1
fi1 :::is .xi1 ; : : : ; xis /dxik D 0 ; 1  k  s; fi1 ; : : : ; is g  f1; : : : ; d g :
0

This implies that f0 is a constant.

In the SA framework, let us have the random vector X D .X1; : : : ; Xd / where
the variables are mutually independent, and the output Y D f .X/ of a deterministic
model f ./. Thus a functional decomposition of the variance is available, often
referred to as functional ANOVA (Efron and Stein [22]):

X
d X
d
Var.Y / D Di .Y / C Dij .Y / C    C D12:::d .Y / (5.10)
iD1 i<j

where Di .Y / D VarŒE.Y jXi /, Dij .Y / D VarŒE.Y jXi ; Xj /  Di .Y /  Dj .Y /

and so on for higher order interactions. The so-called “Sobol’ indices” or “variance-
based sensitivity indices” (Sobol [83]) are obtained as follows:

Di .Y / Dij .Y /
Si D ; Sij D ;  (5.11)
Var.Y / Var.Y /

These indices express the share of variance of Y that is due to a given input or input
combination.
The number of indices growths in an exponential way with the number d of
dimension: there are 2d  1 indices. For computational time and interpretation
reasons, the practitioner should not estimate indices of order higher than two.
Homma and Saltelli [34] introduced the so-called “total indices” or “total effects”
that write as follows:
X X X
ST i D Si C Sij C Sij k C : : : D Sl (5.12)
j ¤i j ¤i;k¤i;j <k l2#i

where #i are all the subsets of f1; : : : ; d g including i . In practice, when d is

large, only the main effects and the total effects are computed, thus giving a good
information on the model sensitivities.
To estimate Sobol’ indices, Monte Carlo sampling based methods have been
developed: Sobol [83] for first order and interaction indices and Saltelli [73] for fist
order and total indices. Unfortunately, to get precise estimates of sensitivity indices,
these
p methods are costly in terms of number of model calls (rate of convergence
in n where n is the sample size). In common practice, the value of 104 model
calls can be required to estimate the Sobol’ index of one input with an uncertainty
of 10 %. Using quasi-Monte Carlo sequences instead of Monte Carlo samples can
sometimes reduce this cost by a factor ten (Saltelli et al. [79]). The FAST method
(Cukier et al. [16]), based on a multi-dimensional Fourier transform, is also used to
110 B. Iooss and P. Lemaître

reduce this cost. Saltelli et al. [76] have extended this technique to compute total
Sobol’ indices and Tarantola et al. [90] have coupled FAST with a Random Balance
Design. Tissot and Prieur [91] have recently analyzed and improved these methods.
However, FAST remains costly, unstable and biased when the number of inputs
increases (larger than 10) (Tissot and Prieur [91]).
One advantage of using a Monte Carlo based method is that it provides error
made on indices estimates via random repetition (Iooss et al. [37]), asymptotic
formulas (Janon et al. [40]) or bootstrap methods (Archer et al. [1]). Thus, other
Monte Carlo based estimation formulas have been introduced and greatly improved
the estimation precision: Mauntz formulas (Sobol et al. [86], Saltelli et al. [74])
for estimating small indices, Jansen formula (Jansen [41], Saltelli et al. [74]) for
estimating total Sobol’ indices and Janon-Monod formula (Janon et al. [40]) for
estimating large first-order indices.
To illustrate the estimation of the Sobol’ indices on the flood exercise [Eqs. (5.2)
and (5.3)] with d D 5 random inputs, we use Saltelli [73] formula with a Monte
Carlo sampling. It has a cost N D n.d C 2/ in terms of model calls where n is the
size of an initial Monte Carlo sample. Here, n D 105 and we repeat 100 times the
estimation process to obtain confidence intervals (as boxplots) for each estimated
indices. Figure 5.3 gives the result of these estimates, which have finally required
N D 7  107 model calls.
For the output S , the first order indices are almost equal to the total indices, and
results seem very similar to those of SRC2 . The model is linear and the estimation
of Sobol’ indices is unnecessary in this case. For the output Cp , we obtain different
information than those provided by SRC2 and SRRC2 : the total effect of Q is about
50 % (twice than its SRC2 ), the effect of Hd is about 20 %, while Q and Ks have
non-negligible interaction effects. Second order Sobol’ index between Q and Ks is
worth 6 %.

5.3.3 Other Measures

From an independent and identically distributed sample (as a Monte Carlo one),
other techniques can be used for SA. For example, statistical testing based tech-
niques consist, for each input, to divide the sample into several sub-samples
(dividing the considered input into equiprobable stratas). Statistical tests can then
be applied to measure the homogeneity of populations between classes: common
means (CMN) based on a Fisher test, common median (CMD) based on a 2 -test,
common variances (CV) based on a Fisher test, common locations (CL) based on
the Kruskal-Wallis test, . . . (Kleijnen and Helton [45], Helton et al. [32]). These
methods do not require assumptions about the monotony of the output with respect
to the inputs but lacks of some quantitative interpretation.
The indices of Sect. 5.3.2 are based on the second-order moment (i.e. the
variance) of the output distribution. In some cases, variance poorly represents the
variability of the distribution. Some authors have then introduced the so-called
5 Global Sensitivity Analysis 111

Fig. 5.3 Estimation of Sobol’ indices on the flood example. Each boxplot corresponds to 100
independent estimates

moment independent importance measures, which do not require any computation

of the output moments. Two kinds of indices have been defined:
– The entropy-based sensitivity indices (Krzykacz-Hausmann [47], Liu et al. [57],
Auder and Iooss [3]),
– The distribution based sensitivity indices (Borgonovo [6], Borgonovo et al. [7])
which consider a distance or a divergence between the output distribution and the
output distribution conditionally to one or several inputs.
It has been shown that these indices can provide complementary information than
Sobol’ indices. However, some difficulties arise in their estimation procedure.

5.4 Deep Exploration of Sensitivities

In this section, the discussed methods provide additional sensitivity information

than just scalar indices. Moreover, for industrial computer codes with a high
computational cost (from several tens of minutes to days), the estimation of Sobol’
112 B. Iooss and P. Lemaître

indices, even with sophisticated sampling methods, are often unreachable. This
section also summarizes a class of methods for approximating the numerical model
to estimate Sobol’ indices at a low computational cost, while providing a deeper
view of the input variables effects.

5.4.1 Graphical and Smoothing Techniques

Beyond Sobol’ indices that only give a scalar value for the effect of an input Xi
on the output Y , the influence of Xi on Y along its domain of variation is also
of interest. In the literature, it is often referred to as main effects, but to avoid any
confusion with the indices of the first order, it is preferable to talk about main effects
visualization (or graph). The scatterplots (visualization of point cloud of any sample
simulations .Xn ; Yn / with the graphs of Y vs. Xi , i D 1; : : : ; d ) meets this goal, but
in a visual subjective manner. This is shown in Fig. 5.4, on the flood example and
using the 100-size sample of Sect. 5.3.1.
Based on parametric or non-parametric regression methods (Hastie and Tibshi-
rani [28]), the smoothing techniques aim to estimate the conditional moments of Y
0.75

0.75

0.75
0.65

0.65
0.65

500 1500 2500 49.5 50.5 55.2 55.6

Q Zv Cb
0.75

0.75
0.65

0.65

15 25 35 45 7.0 8.0 9.0

Ks Hd

Fig. 5.4 Scatterplots on the flood example with the five inputs Q, Zv , Cb , Ks , Hd and the output
Cp . Dotted curve is a local polynomial based smoother
5 Global Sensitivity Analysis 113

at first or higher order. SA is often restricted to the determination of the conditional

expectation at first and second orders (Santner et al. [80]), in order to obtain:
– main effects graphs, between Xi and E.Y jXi /  E.Y / on the whole variation
domain of Xi for i D 1; : : : ; d ;
– interaction effects graphs, between .Xi ; Xj / and E.Y jXi Xj /  E.Y jXi / 
E.Y jXj /  E.Y / on all the variation domain of .Xi ; Xj / for i D 1; : : : ; d and
j D i C 1; : : : ; d .
Storlie and Helton [87] conducted a fairly comprehensive review of non-
parametric smoothing methods that can be used for SA: moving averages, kernel
methods, local polynomials, smoothing splines, etc. In Fig. 5.4, the local polynomial
smoother is plotted for each cloud of points, thereby clearly identifying the mean
trend of the output versus each input.
Once these conditional expectations are modeled, it is easy to quantify their
variance by sampling, and thus to estimate Sobol’ indices [cf. Eq. (5.11)] of order
one, two, or even higher orders. Da Veiga et al. [17] discuss the theoretical properties
of local polynomial estimators of the conditional expectation and variance, and
then deduce the theoretical properties of the Sobol’ indices estimated by local
polynomials. Storlie and Helton [87] also discuss the efficiency of additive models
and regression trees to non-parametrically estimate E.Y jX1 ; : : : ; Xd /. This finally
leads to build an approximate model of f ./, which is called a “metamodel”. This
will be detailed in the following section.
In SA, graphical techniques can also be useful. For example, all the scatterplots
between each input variable and the model output can detect some trends in
their functional relation (see Fig. 5.4). However scatterplots do not capture some
interaction effects between the inputs. Cobweb plots (Kurowicka and Cooke [48]),
also called parallel coordinate plots, can be used to visualize the simulations as a set
of trajectories. In Fig. 5.5, the simulations leading to the 5 % largest values of the
model output S have been highlighted. This allows to immediately understand that
these simulations correspond to large values of the flowrate Q and small values of
the Strickler coefficient Ks .

Fig. 5.5 Cobweb plot of

10; 000 simulations of the
flood model
114 B. Iooss and P. Lemaître

5.4.2 Metamodel-Based Methods

The metamodel concept is frequently used to simulate the behavior of an experimen-

tal system or a long running computational code based on a certain number of output
values. Under the name of the response surface methodology, it was originally
proposed as a statistical tool, to find the operating conditions of a process at which
some responses were optimized (Box and Draper [8]). Subsequent generalizations
led to these methods being used to develop approximating functions of deterministic
computer codes (Downing et al. [21], Sacks et al. [71], Kleijnen and Sargent [46]). It
consists in generating a surrogate model that fits the initial data (using for example a
least squares procedure), which has good prediction capabilities and has negligible
computational cost. It is thus efficient for uncertainty and SA requiring several
thousands of model calculations (Iooss et al. [37]).
In practice, we focus on three main issues during the construction of a meta-
model:
– the choice of the metamodel that can be derived from any linear regression model,
non-linear parametric or non-parametric (Hastie et al. [29]). The most used
metamodels include polynomials, splines, generalized linear models, generalized
additive models, kriging, neural networks, SVM, boosting regression trees
(Simpson et al. [82], Fang et al. [24]). Linear and quadratic functions are
commonly considered as a first iteration. Knowledge on some input interaction
types may be also introduced in polynomials (Jourdan and Zabalza-Mezghani
[42], Kleijnen [44]). However, these kinds of models are not always efficient,
especially in simulation of complex and non-linear phenomena. For such models,
modern statistical learning algorithms can show much better ability to build
accurate models with strong predictive capabilities (Marrel et al. [59]);
– the design of (numerical) experiments. The main qualities required for an
experimental design are the robustness (ability to analyze different models),
the effectiveness (optimization of a criterion), the goodness of points repartition
(space filling property) and the low cost for its construction (Santner et al. [80],
Fang et al. [24]). Several studies have shown the qualities of different types of
experimental designs with respect to the predictivity metamodel (for example
Simpson et al. [82]);
– the validation of the metamodel. In the field of classical experimental design,
proper validation of a response surface is a crucial aspect and is considered with
care. However, in the field of numerical experiments, this issue has not been
deeply studied. The usual practice is to estimate global criteria (RMSE, absolute
error, . . . ) on a test basis, via cross-validation or bootstrap (Kleijnen and Sargent
[46], Fang et al. [24]). When the number of calculations is small and to overcome
problems induced by the cross validation process, Iooss et al. [38] have recently
studied how to minimize the size of a test sample, while obtaining a good estimate
of the metamodel predictivity.
5 Global Sensitivity Analysis 115

Some metamodel allows to directly obtain the sensitivity indices. For example,
Sudret [89] has shown that Sobol’ indices are a by-product result of the polynomial
chaos decomposition. The formulation of the kriging metamodel provides also
analytical formula for the Sobol’ indices, associated with interval confidence
coming from the kriging error (Oakley and O’Hagan [66], Marrel et al. [60], Le
Gratiet et al. [51]). A simplest idea, widely used in practice, is to apply an intensive
sampling technique (see Sect. 5.3.2) directly on the metamodel to estimate Sobol’
indices (Santner et al. [80], Iooss et al. [37]). The variance proportion not explained
by the metamodel [calculated by 1  Q2 , cf. Eq. (5.8)] gives us what is missing in
the SA (Sobol [84]). Storlie et al. [88] propose a bootstrap method for estimating
the impact of the metamodel error.
As in the previous section, we can be interested by visualizing main effects
(Schonlau and Welch [81]). These can be directly given by the metamodel (this is
the case with the polynomial chaos methods, kriging, additive models), or computed
by simulating the conditional expectation E.Y jXi /.
To illustrate our purpose, we use the flood example [Eqs. (5.2) and (5.3)].
A kriging metamodel is built on a 100-size Monte Carlo sample with inputs Q,
Ks , Zv , Hd , Cb and on the output Cp . The metamodel consists in a deterministic
term (simple linear model), and a corrective term modeled by a Gaussian stationary
stochastic process, with a generalized exponential covariance (see Santner et al. [80]
for more details). The technique for estimating the metamodel hyperparameters is
described in Roustant et al. [69]. The predictivity coefficient estimated by leave-
one-out is Q2 D 99 % compared with Q2 D 75 % obtained with a simple linear
model. The kriging metamodel is then used to estimate Sobol’ indices in the same
manner as in Sect. 5.3.2: Saltelli’s estimation formula, Monte Carlo sampling,
n D 105 , r D 100 repetitions. This requires N D 7  107 metamodel predictions. In
Table 5.2, we compare Sobol’ indices (averaged over 100 repetitions) obtained with
the metamodel to those obtained with the “real” flood model [Eqs. (5.2) and (5.3)].
Errors between these two estimates are relatively low: with only 100 simulations
with the true model, we were able to obtain precise estimates (errors < 15 %) of
first order and total Sobol’ indices.

Table 5.2 Sobol’ indices estimated by Monte Carlo

sampling (cost of N D 7  107 evaluations) using
the flood model and a metamodel fitted on N 0 D 100
calls of the flood model
Indices (in %) Q Ks Zv Hd Cb
Si model 35:5 15:9 18:3 12:5 3:8
Si metamodel 38:9 16:8 18:8 13:9 3:7
STi model 48:2 25:3 22:9 18:1 3:8
STi metamodel 45:5 21:0 21:3 16:8 4:3
116 B. Iooss and P. Lemaître

5.5 Synthesis and Conclusion

Although all SA techniques have not been listed, this review has illustrated the great
variety of available methods, positioning in terms of assumptions and kind of results.
Moreover, some recent improvements have not been explained, for example for the
Morris method (Pujol [68]).
A synthesis is provided in Fig. 5.6 which has several levels of reading:
– distinction between screening methods (identification of non-influential variables
among a large number) and more precise variance-based quantitative methods,
– positioning methods based on their cost in terms of model calls number (which
linearly depends in the number of inputs for most of the methods),
– positioning methods based on their assumptions about the model complexity and
regularity,
– distinction between the type of information provided by each method,
– identification of methods which require some a priori knowledge about the model
behaviour.
Based on the characteristics of the different methods, some authors (de Roc-
quigny et al. [19], Pappenberger et al. [67]) have proposed decision trees to help the
practitioner to choose the most appropriate method for its problem and its model.
Figure 5.7 reproduces the flowchart of de Rocquigny et al. [19]. Although useful to
fix some ideas, such diagrams are rather simple and should be used with caution.

Fig. 5.6 SA methods graphical synthesis

5 Global Sensitivity Analysis 117

Fig. 5.7 Decision diagram for the choice of a SA method (from de Rocquigny et al. [19])

Several issues about SA remain open. For instance, recent theoretical results
have been obtained on the asymptotical properties and efficiency of Sobol’ indices
estimators (Janon et al. [40]), but estimating total Sobol’ indices at low cost is a
problem of primary importance in applications (see Saltelli et al. [74] for a recent
review on the subject). SA for dependent inputs has also been discussed by several
authors (Saltelli and Tarantola [75], Jacques et al. [39], Xu and Gertner [93], Da
Veiga et al. [17], Gauchi et al. [27], Li et al. [54], Chastaing et al. [14]), but this
issue remains misunderstood.
This chapter has been focused on SA relative to the overall variability of model
output. In practice, one can be interested by other quantities of interest, such as the
output entropy (cf. Sect. 5.3.3), the probability that the output exceeds a threshold
(Saltelli et al. [77], Frey and Patil [25], Lemaître et al. [53]) or a quantile estimation
(Cannamela et al. [12]). This is an active area of research.
In many applications, the model output is not a single scalar but a vector or a
function (temporal, spatial, spatio-temporal, . . . ). Campbell et al. [11], Lamboni et
al. [50], Marrel et al. [61] and Gamboa et al. [26] have produced first SA results
on such problems. The case of functional inputs also receives a growing interest
(Iooss and Ribatet [36], Lilburne and Tarantola [55], Saint-Geours et al. [72]), but
its treatment in a functional statistical framework remains to be done.
In some situations, the computer code is not a deterministic simulator but a
stochastic one. This means that two model calls with the same set of input variables
leads to different output values. Typical stochastic computer codes are queuing
models, agent-based models, models involving partial differential equations applied
118 B. Iooss and P. Lemaître

to heterogeneous or Monte-Carlo based numerical models. For this type of codes,

Marrel et al. [62] have proposed a first solution for dealing with Sobol’ indices.
Finally, quantitative SA methods are limited to low-dimensional models, with no
more than a few tens of input variables. On the other hand, deterministic methods,
such as adjoint-based ones (Cacuci [10]), are well suited when the model includes
a large number of input variables. A natural idea is to use the advantages of both
methods. Recently introduced, Derivative-Based Sensitivity Measures (DGSM),
consists in computing the integral of the square model derivatives for each input
(Sobol and Kucherenko [85]). An inequality relation has been proved between
total Sobol’ indices and DGSM which allow to propose some interpretative results
(Lamboni et al. [49], Roustant et al. [70]). It opens the way to perform global SA in
high dimensional context.

Acknowledgements Part of this work has been backed by French National Research Agency
(ANR) through COSINUS program (project COSTA BRAVA no. ANR-09-COSI-015). We thank
Anne-Laure Popelin and Merlin Keller for providing the cobweb plot and the flood model figure.

References

1. Archer, G., Saltelli, A., Sobol, I.: Sensitivity measures, ANOVA-like techniques and the use of
bootstrap. J. Stat. Comput. Simul. 58, 99–120 (1997)
2. Auder, B., de Crécy, A., Iooss, B., Marquès, M.: Screening and metamodeling of computer
experiments with functional outputs. Application to thermal-hydraulic computations. Reliab.
Eng. Syst. Saf. 107, 122–131 (2012)
3. Auder, B., Iooss, B.: Global sensitivity analysis based on entropy. In: Martorell, S., Guedes
Soares, C., Barnett, J. (eds.), Safety, Reliability and Risk Analysis - Proceedings of the ESREL
2008 Conference, pp. 2107–2115. CRC Press, Valencia (2008)
4. Badea, A., Bolado, R.: Review of sensitivity analysis methods and experience. PAMINA 6t h
FPEC Project, European Commission (2008). http://www.ip-pamina.eu/downloads/pamina.
m2.1.d.4.pdf
5. Bettonvil, B., Kleijnen, J.P.C.: Searching for important factors in simulation models with many
factors: Sequential bifurcation. Eur. J. Oper. Res. 96, 180–194 (1996)
6. Borgonovo, E.: A new uncertainty importance measure. Reliab. Eng. Syst. Saf. 92, 771–784
(2007)
7. Borgonovo, E., Castaings, W., Tarantola, S.: Moment independent importance measures: new
results and analytical test cases. Risk Anal. 31, 404–428 (2011)
8. Box, G.E., Draper, N.R.: Empirical Model Building and Response Surfaces. Wiley Series in
Probability and Mathematical Statistics. Wiley, New York (1987)
9. Cacuci, D.G.: Sensitivity theory for nonlinear systems. I. Nonlinear functional analysis
approach. J. Math. Phys. 22, 2794 (1981)
10. Cacuci, D.G.: Sensitivity and Uncertainty Analysis - Theory. Chapman & Hall/CRC, Boca
Raton (2003)
11. Campbell, K., McKay, M.D., Williams, B.J.: Sensitivity analysis when model ouputs are
functions. Reliab. Eng. Syst. Saf. 91, 1468–1472 (2006)
12. Cannamela, C., Garnier, J., Iooss, B.: Controlled stratification for quantile estimation. Ann.
Appl. Stat. 2, 1554–1580 (2008)
5 Global Sensitivity Analysis 119

13. Castaings, W., Dartus, D., Le Dimet, F.-X., Saulnier, G.-M.: Sensitivity analysis and parameter
estimation for distributed hydrological modeling : potential of variational methods. Hydrol.
Earth Syst. Sci. Discuss. 13, 503–517 (2009)
14. Chastaing, G., Gamboa, F., Prieur, C.: Generalized Hoeffding-sobol decomposition for
dependent variables - Application to sensitivity analysis. Electron. J. Stat. 6, 2420–2448 (2012)
15. Christensen, R.: Linear Models for Multivariate, Time Series and Spatial Data. Springer,
New York (1990)
16. Cukier, H., Levine, R.I., Shuler, K.: Nonlinear sensitivity analysis of multiparameter model
systems. J. Comput. Phys. 26, 1–42 (1978)
17. Da Veiga, S., Wahl, F., Gamboa, F.: Local polynomial estimation for sensitivity analysis on
models with correlated inputs. Technometrics 51(4), 452–463 (2009)
18. de Rocquigny, E.: La maîtrise des incertitudes dans un contexte industriel - 1ère partie : une
approche méthodologique globale basée sur des exemples. Journal de la Société Française de
Statistique 147(3), 33–71 (2006)
19. de Rocquigny, E., Devictor, N., Tarantola, S. (eds.): Uncertainty in Industrial Practice. Wiley,
Chichester (2008)
20. Dean, A., Lewis, S., (eds.): Screening - Methods for Experimentation in Industry, Drug
Discovery and Genetics. Springer, New York (2006)
21. Downing, D.J., Gardner, R.H., Hoffman, F.O.: An examination of response surface method-
ologies for uncertainty analysis in assessment models. Technometrics 27(2), 151–163 (1985)
22. Efron, B., Stein, C.: The jacknife estimate of variance. Ann. Stat. 9, 586–596 (1981)
23. Faivre, R., Iooss, B., Mahévas, S., Makowski, D., Monod, H., (eds.): Analyse de sensibilité et
exploration de modèles. Éditions Quaé, Versailles (2013)
24. Fang, K.-T., Li, R., Sudjianto, A.: Design and Modeling for Computer Experiments. Chapman
& Hall/CRC, Boston (2006)
25. Frey, H.C., Patil, S.R.: Identification and review of sensitivity analysis methods. Risk Anal.
22, 553–578 (2002)
26. Gamboa, F., Janon, A., Klein, T., Lagnoux, A.: Sensitivity indices for multivariate outputs.
C. R. Acad. Sci. 351, 307–310 (2013)
27. Gauchi, J.P., Lehuta, S., Mahévas, S.: Optimal sensitivity analysis under constraints. In:
Procedia Social and Behavioral Sciences, vol. 2, pp. 7658–7659, Milan (2010)
28. Hastie, T., Tibshirani, R.: Generalized Additive Models. Chapman & Hall, London (1990)
29. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer,
New York (2002)
30. Helton, J.C.: Uncertainty and sensitivity analysis techniques for use in performance assesment
for radioactive waste disposal. Reliab. Eng. Syst. Saf. 42, 327–367 (1993)
31. Helton, J.C., Davis, F.J.: Latin hypercube sampling and the propagation of uncertainty in
analyses of complex systems. Reliab. Eng. Syst. Saf. 81, 23–69 (2003)
32. Helton, J.C., Johnson, J.D., Salaberry, C.J., Storlie, C.B.: Survey of sampling-based methods
for uncertainty and sensitivity analysis. Reliab. Eng. Syst. Saf. 91, 1175–1209 (2006)
33. Hoeffding, W.: A class of statistics with asymptotically normal distributions. Ann. Math. Stat.
19, 293–325 (1948)
34. Homma, T., Saltelli, A.: Importance measures in global sensitivity analysis of non linear
models. Reliab. Eng. Syst. Saf. 52, 1–17 (1996)
35. Iooss, B.: Revue sur l’analyse de sensibilité globale de modèles numériques. Journal de la
Société Française de Statistique 152, 1–23 (2011)
36. Iooss, B., Ribatet, M.: Global sensitivity analysis of computer models with functional inputs.
Reliab. Eng. Syst. Saf. 94, 1194–1204 (2009)
37. Iooss, B., Van Dorpe, F., Devictor, N.: Response surfaces and sensitivity analyses for an
environmental model of dose calculations. Reliab. Eng. Syst. Saf. 91, 1241–1251 (2006)
38. Iooss, B., Boussouf, L., Feuillard, V., Marrel, A.: Numerical studies of the metamodel fitting
and validation processes. Int. J. Adv. Syst. Meas. 3, 11–21 (2010)
39. Jacques, J., Lavergne, C., Devictor, N.: Sensitivity analysis in presence of model uncertainty
and correlated inputs. Reliab. Eng. Syst. Saf. 91, 1126–1134 (2006)
120 B. Iooss and P. Lemaître

40. Janon, A., Klein, T., Lagnoux, A., Nodet, M., Prieur, C.: Asymptotic normality and efficiency
of two sobol index estimators. ESAIM Probab. Stat. 18, 342–364 (2013)
41. Jansen, M.J.W.: Analysis of variance designs for model output. Comput. Phys. Commun. 117,
25–43 (1999)
42. Jourdan, A., Zabalza-Mezghani, I.: Response surface designs for scenario mangement and
uncertainty quantification in reservoir production. Math. Geol. 36(8), 965–985 (2004)
43. Kleijnen, J.P.C.: Sensitivity analysis and related analyses: a review of some statistical
techniques. J. Stat. Comput. Simul. 57, 111–142 (1997)
44. Kleijnen, J.P.C.: An overview of the design and analysis of simulation experiments for
sensitivity analysis. Eur. J.Oper. Res. 164, 287–300 (2005)
45. Kleijnen, J.P.C., Helton, J.C.: Statistical analyses of scatterplots to identify important factors
in large-scale simulations, 1: review and comparison of techniques. Reliab. Eng. Syst. Saf. 65,
147–185 (1999)
46. Kleijnen, J.P.C., Sargent, R.G.: A methodology for fitting and validating metamodels in
simulation. Eur. J.Oper. Res. 120, 14–29 (2000)
47. Krzykacz-Hausmann, B.: Epistemic sensitivity analysis based on the concept of entropy. In:
Proceedings of SAMO 2001, pp. 31–35. CIEMAT, Madrid (2001)
48. Kurowicka, D., Cooke, R.: Uncertainty Analysis with High Dimensional Dependence
Modelling. Wiley, Chichester (2006)
49. Lamboni, M., Iooss, B., Popelin, A.-L., Gamboa, F.: Derivative-based global sensitivity
measures: general links with sobol’ indices and numerical tests. Math. Comput. Simul. 87,
45–54 (2013)
50. Lamboni, M., Monod, H., Makowski, D.: Multivariate sensitivity analysis to measure global
contribution of input factors in dynamic models. Reliab. Eng. Syst. Saf. 96, 450–459 (2011)
51. Le Gratiet, L., Cannamela, C., Iooss, B.: A Bayesian approach for global sensitivity analysis
of (multifidelity) computer codes. SIAM/ASA J. Uncertain. Quantif. 2, 336–363 (2014)
52. Lefebvre, S., Roblin, A., Varet, S., Durand, G.: A methodological approach for statistical
evaluation of aircraft infrared signature. Reliab. Eng. Syst. Saf. 95, 484–493 (2010)
53. Lemaître, P., Sergienko, E., Arnaud, A., Bousquet, N., Gamboa, F., Iooss, B.: Density
modification based reliability sensitivity analysis. J. Stat. Comput. Simul. 85, 1200–1223
(2015)
54. Li, G., Rabitz, H., Yelvington, P.E., Oluwole, O.O., Bacon, F., Kolb, C.E., Schoendorf, J.:
Global sensitivity analysis for systems with independent and/or correlated inputs. J. Phys.
Chem. 114, 6022–6032 (2010)
55. Lilburne, L., Tarantola, S.: Sensitivity analysis of spatial models. Int. J. Geogr. Inf. Sci. 23,
151–168 (2009)
56. Lin, D.K.J.: A new class of supersaturated design. Technometrics 35, 28–31 (1993)
57. Liu, H., Chen, W., Sudjianto, A.: Relative entropy based method for probabilistic sensitivity
analysis in engineering design. ASME J. Mech. Des. 128, 326–336 (2006)
58. Makowski, D., Naud, C., Jeuffroy, M.H., Barbottin, A., Monod, H.: Global sensitivity analysis
for calculating the contribution of genetic parameters to the variance of crop model prediction.
Reliab. Eng. Syst. Saf. 91, 1142–1147 (2006)
59. Marrel, A., Iooss, B., Van Dorpe, F., Volkova, E.: An efficient methodology for modeling
complex computer codes with Gaussian processes. Comput. Stat. Data Anal. 52, 4731–4744
(2008)
60. Marrel, A., Iooss, B., Laurent, B., Roustant, O.: Calculations of the Sobol indices for the
Gaussian process metamodel. Reliab. Eng. Syst. Saf. 94, 742–751 (2009)
61. Marrel, A., Iooss, B., Jullien, M., Laurent, B., Volkova, E.: Global sensitivity analysis for
models with spatially dependent outputs. Environmetrics 22, 383–397 (2011)
62. Marrel, A., Iooss, B., Da Veiga, S., Ribatet, M.: Global sensitivity analysis of stochastic
computer models with joint metamodels. Stat. Comput. 22, 833–847 (2012)
63. McKay, M.D., Beckman, R.J., Conover, W.J.: A comparison of three methods for selecting
values of input variables in the analysis of output from a computer code. Technometrics 21,
239–245 (1979)
5 Global Sensitivity Analysis 121

64. Montgomery, D.C.: Design and Analysis of Experiments, 6th edn. Wiley, Hoboken (2004)
65. Morris, M.D.: Factorial sampling plans for preliminary computational experiments. Techno-
metrics 33, 161–174 (1991)
66. Oakley, J.E., O’Hagan, A.: Probabilistic sensitivity analysis of complex models: a Bayesian
approach. J. R. Stat. Soc. Ser. B 66, 751–769 (2004)
67. Pappenberger, F., Ratto, M., Vandenberghe, V.: Review of sensitivity analysis methods. In:
Vanrolleghem, P.A., (ed.) Modelling Aspects of Water Framework Directive Implementation,
pp. 191–265. IWA Publishing, London (2010)
68. Pujol, G. (2009). Simplex-based screening designs for estimating metamodels. Reliab. Eng.
Syst. Saf. 94, 1156–1160 (2009)
69. Roustant, O., Ginsbourger, D., Deville, Y.: DiceKriging, DiceOptim: Two R packages for the
analysis of computer experiments by kriging-based metamodeling and optimization. J. Stat.
Softw., 21 1–55 (2012)
70. Roustant, O., Fruth, J., Iooss, B., Kuhnt, S.: Crossed-derivative-based sensitivity measures for
interaction screening. 105, 105–118 (2014)
71. Sacks, J., Welch, W.J., Mitchell, T.J., Wynn, H.P.: Design and analysis of computer
experiments. Stat. Sci. 4, 409–435 (1989)
72. Saint-Geours, N., Lavergne, C., Bailly, J.-S., Grelot, F.: Analyse de sensibilité de sobol d’un
modèle spatialisé pour l’evaluation économique du risque d’inondation. Journal de la Société
Française de Statistique 152, 24–46 (2011)
73. Saltelli, A.: Making best use of model evaluations to compute sensitivity indices. Comput.
Phys. Commun. 145, 280–297 (2002)
74. Saltelli, A., Annoni, P.: How to avoid a perfunctory sensitivity analysis. Environ. Model.
Softw. 25, 1508–1517 (2010)
75. Saltelli, A., Tarantola, S.: On the relative importance of input factors in mathematical models:
Safety assessment for nuclear waste disposal. J. Am. Stat. Assoc. 97, 702–709 (2002)
76. Saltelli, A., Tarantola, S., Chan, K.: A quantitative, model-independent method for global
sensitivity analysis of model output. Technometrics 41, 39–56 (1999)
77. Saltelli, A., Chan, K., Scott, E.M. (eds.): Sensitivity Analysis. Wiley Series in Probability and
Statistics. Wiley, New York (2000)
78. Saltelli, A., Tarantola, S., Campolongo, F., Ratto, M.: Sensitivity Analysis in Practice: A Guide
to Assessing Scientific Models. Wiley, New York (2004)
79. Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Salsana, M.,
Tarantola, S.: Global Sensitivity Analysis - The Primer. Wiley, New York (2008)
80. Santner, T., Williams, B., Notz, W.: The Design and Analysis of Computer Experiments.
Springer, New York (2003)
81. Schonlau, M., Welch, W.J.: Screening the input variables to a computer model. In: Dean, A.,
Lewis, S., (eds.) Screening - Methods for Experimentation in Industry, Drug Discovery and
Genetics. Springer, New York (2006)
82. Simpson, T.W., Peplinski, J.D., Kock, P.N., Allen, J.K.: Metamodel for computer-based
engineering designs: Survey and recommandations. Eng. Comput. 17, 129–150 (2001)
83. Sobol, I.M.: Sensitivity estimates for non linear mathematical models. Math. Model. Comput.
Exp. 1, 407–414 (1993)
84. Sobol, I.M.: Theorems and examples on high dimensional model representation. Reliab. Eng.
Syst. Saf. 79, 187–193 (2003)
85. Sobol, I.M., Kucherenko, S.: Derivative based global sensitivity measures and their links with
global sensitivity indices. Math. Comput. Simul. 79, 3009–3017 (2009)
86. Sobol, I.M., Tarantola, S., Gatelli, D., Kucherenko, S.S., Mauntz, W.: Estimating the
approximation errors when fixing unessential factors in global sensitivity analysis. Reliab.
Eng. Syst. Saf. 92, 957–960 (2007)
87. Storlie, C.B., Helton, J.C.: Multiple predictor smoothing methods for sensitivity analysis:
Description of techniques. Reliab. Eng. Syst. Saf. 93, 28–54 (2008)
88. Storlie, C.B., Swiler, L.P., Helton, J.C., Salaberry, C.J.: Implementation and evaluation of
nonparametric regression procedures for sensitivity analysis of computationally demanding
models. Reliab. Eng. Syst. Saf. 94, 1735–1763 (2009)
122 B. Iooss and P. Lemaître

89. Sudret, B.: Uncertainty propagation and sensitivity analysis in mechanical models - Contri-
butions to structural reliability and stochastics spectral methods. Mémoire d’Habilitation à
Diriger des Recherches de l’Université Blaise Pascal - Clermont II (2008)
90. Tarantola, S., Gatelli, D., Mara, T.: Random balance designs for the estimation of first order
global sensitivity indices. Reliab. Eng. Syst. Saf. 91, 717–727 (2006)
91. Tissot, J.-Y., Prieur, C.: A bias correction method for the estimation of sensitivity indices based
on random balance designs. Reliab. Eng. Syst. Saf. 107, 205–213 (2012)
92. Volkova, E., Iooss, B., Van Dorpe, F.: Global sensitivity analysis for a numerical model of
radionuclide migration from the RRC “Kurchatov Institute” radwaste disposal site. Stoch.
Env. Res. Risk A. 22, 17–31 (2008)
93. Xu, C., Gertner, G.: Extending a global sensitivity analysis technique to models with correlated
parameters. Comput. Stat. Data Anal. 51, 5579–5590 (2007)