See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/316640799

Weights and importance in composite indicators: Closing the gap

Article  in  Ecological Indicators · September 2017

DOI: 10.1016/j.ecolind.2017.03.056

CITATIONS READS

28 315

4 authors:

William Becker Michaela Saisana

European Commission European Commission
48 PUBLICATIONS   217 CITATIONS    96 PUBLICATIONS   5,477 CITATIONS   

SEE PROFILE SEE PROFILE

Paruolo Paolo Ine Vandecasteele

European Commission European Commission
96 PUBLICATIONS   1,033 CITATIONS    45 PUBLICATIONS   668 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

iDRIP - illicit DRugs In the Pipe View project

European water resources View project

All content following this page was uploaded by William Becker on 04 May 2017.

The user has requested enhancement of the downloaded file.

 Ecological Indicators 80 (2017) 12–22

Contents lists available at ScienceDirect

Ecological Indicators
journal homepage: www.elsevier.com/locate/ecolind

Weights and importance in composite indicators: Closing the gap MARK

William Becker, Michaela Saisana, Paolo Paruolo, Ine Vandecasteele
European Commission, Joint Research Centre, Via Enrico Fermi, 2749, 21027 Ispra VA, Italy

A R T I C L E I N F O A B S T R A C T

Keywords: Composite indicators are very popular tools for assessing and ranking countries and institutions in terms of
Composite indicators environmental performance, sustainability, and other complex concepts that are not directly measurable.
Sensitivity analysis Because of the stakes that come with the media attention of these tools, a word of caution is warranted. One
Correlation common misconception relates to the effect of the weights assigned to indicators during the aggregation process.
Splines
This work presents a novel series of tools that allow developers and users of composite indicators to explore
Gaussian process
Smoothing
effects of these weights. First, the importance of each indicator to the composite is measured by the nonlinear
Optimisation Pearson correlation ratio, estimated by Bayesian Gaussian processes. Second, the effect of each indicator is
isolated from that of other indicators using regression analysis, and examined in detail. Finally, an optimisation
procedure is proposed which allows weights to be fitted to agree with pre-specified values of importance. These
three tools together give developers considerable insight into the effects of weights and suggest possibilities for
refining and simplifying the aggregation. The added value of these tools are shown on three case studies: the
Resource Governance Index, the Good Country Index, and the Water Retention Index.

1. Introduction covering a broad range of themes from happiness-adjusted income to

environmentally-adjusted income, from child development to informa-
Composite indicators (also known as synthetic indices or perfor- tion and communication technology development.
mance indices) are popular tools for assessing the performance of Even though considerable attention is given to the rankings of
countries/entities on human development, sustainability, perceived composite indicators, many subjective choices are made in their
corruption, innovation, competitiveness, or other complex phenomena construction: this has motivated studies which perform uncertainty
that are not directly measurable and not uniquely defined. Examples and sensitivity analysis on composite indicator assumptions (Saisana
include the Human Development Index (Jahan, 2015), the Sustainable et al., 2005). One important step is the aggregation of indicators, where
Society Index (Van de Kerk and Manuel, 2008), the Financial Secrecy typically the variables are combined in a weighted average to give the
Index (Cobham et al., 2013) and the Environmental Performance Index resulting value of the composite indicator. Apart from the decision of
(Hsu, 2016). Composite indicators are employed for many purposes, which kind of weighted average to use (e.g. arithmetic, geometric), the
including policy monitoring, communication to the public, and gen- developer must select values of weights to apply to each variable. The
erating rankings. values of these weights can have a large impact on the subsequent
The popularity of rankings owes to two main reasons. First, their rankings, which often goes unnoticed. Understanding the impact of
simplicity: they provide a summary picture of the multiple facets or weights on the variation of the composite indicator scores is therefore
dimensions of complex, multidimensional phenomena in a way that important.
facilitates evaluation and comparison. Second, rankings force institu- A possible misconception is that the weight assigned to a variable
tions and governments to question their standards; rankings are drivers can be directly interpreted as a measure of importance of the variable to
of behaviour and of change (Kelley and Simmons, 2015). Hence, it the resulting value of the composite indicator. Indeed, in common
comes at no surprise that over the past two decades there has been a approaches to composite indicator weighting such as budget allocation
turbulent growth of performance indices. Bandura (2011) provides a (expert input) and equal weighting, this appears to be the supporting
comprehensive inventory of over 400 country-level indexes monitoring logic. However this is rarely the case: different variances and correla-
complex phenomena from economic progress to educational quality. tions among variables, for instance, prevent the weights from corre-
Similarly, a more recent inventory by the United Nations (Yang, 2014) sponding to the variables’ importance. Two questions immediately
details 101 composite measures of human well-being and progress, arise: first, given a set of weights and a sample, what is the influence of

E-mail addresses: [email protected] (W. Becker), [email protected] (M. Saisana), [email protected] (P. Paruolo),
[email protected] (I. Vandecasteele).

http://dx.doi.org/10.1016/j.ecolind.2017.03.056
Received 23 February 2017; Received in revised form 30 March 2017; Accepted 30 March 2017
1470-160X/ © 2017 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/BY/4.0/).
W. Becker et al. Ecological Indicators 80 (2017) 12–22

each variable on the output? Second, how can weights be assigned to importance of each of the input variables xi with respect to the output y
reflect the desired importance? of the composite indicator. One approach is to measure the dependence
This article puts together, for the first time, tools that allow of y on xi. Consider the decomposition,
developers to examine in detail the effect of weights and to subse- yj = fi (x ji ) + εj , (2)
quently refine the composite indicator. First, building on an earlier
proposal of some of the present authors, see Paruolo et al. (2013), the where fi(xji) is an appropriate function, possibly nonlinear, that models
relative importance of each indicator is measured with the Pearson the conditional mean of y given a sample point xji—and εj is an error
correlation ratio, which is a variance-based measure that accounts for term which accounts for variation due to indicators other than xi. A
(possibly nonlinear) dependence between input variables and the well-known way to measure the linear dependence of y on xi is to use
composite indicator (see Section 2). Going beyond Paruolo et al. the coefficient of determination Ri2 : in sample, this can be computed as:
(2013), who use local-linear regression to estimate the correlation SSreg, i /SStot , (3)
ratio, the main tool used here is Bayesian Gaussian processes, which
have the advantage of providing confidence intervals on the Pearson
correlation ratio (see Section 3). This overall approach echoes the work
where (in the case of R ) SSreg, i = 2 n
∑ j =1 ˆ
( fi (x ji ) − y )2 is the sum of
n
squares explained by the linear regression, y : =n−1 ∑ j =1 yj is the sample
of Da Veiga et al. (2009) in the sensitivity analysis literature, but with
the additional advantage that it does not require the use of an ˆ ˆ ˆ
average, fi (x ji ) = β0 + β1 x ji is the linear fit for observation yj and
SStot = ∑j (yj − y )2 is the total sum of squares. Ri2 can hence be seen
independent sample from the marginal distribution of each indicator
to estimate the variance of the conditional expectation. as the ratio of the sum of squares explained by the linear regression of y
As a further step, to better understand the influence of variables on on xi, and the total sum of squares of y. Since this measure is based on
the composite indicator, an approach based on the correlated sensitivity linear regression, it does not account for any nonlinearities between y
analysis work of Xu and Gertner (2008a,b) is proposed in Section 4, and xi.
which uses additional regressions to decompose the influence of each In composite indicators, although the aggregation formula is often
variable into influence caused by correlation, and influence caused by linear, a nonlinearity in the relationship between y on xi can be
introduced by the correlation between variables. In such cases, Ri2
the composite indicator structure (aggregation and weights).
may underestimate the degree of dependence. The nonlinear measure
Finally, the issue of optimisation of weights is considered in Section
adopted here is the correlation ratio, Si, i = 1, 2, .. ., d also widely known
5. Although a linear solution to the optimisation problem is available in
as the first order sensitivity index, or main effect index (see e.g. Becker and
Paruolo et al. (2013) using linear regression, the proposal here is to
Saltelli, 2015). This measure is meant to measure the (possibly non-
extend it to nonlinear regression, to account for nonlinear main effects.
linear) influence of each variable on the composite indicator, and is a
This is done by numerically minimising the difference between
nonlinear generalisation of Ri2 , such that Ri2 equals Si when fi(xji) is
correlation ratios and their desired values. Given that this involves a
linear. It can be interpreted as the expected variance reduction of the
large number of regression fits, penalised splines are preferred over
composite indicator scores, if a given variable were fixed. It is defined
Gaussian processes because of their low computational cost.
as:
The various strands of methodology in this article are demonstrated
on three case studies in Section 6, which are chosen as examples related Vxi (E x∼ i (y xi ))
Si ≡ ηi2 : = ,
to sustainability and quality of life: the Resource Governance Index, the V(y ) (4)
Good Country Index, and the Water Retention index. The first two
where x∼i is the vector containing all the variables (x1, …, xd) except
indices were chosen because of their potentially high policy impact
variable xi. The term Ex∼ i (y xi ) is explicitly stated here with its subscript
owing to a considerable media presence. The latter index is chosen for
x∼i to emphasise that it is the expected value of y (the composite
academic purposes, in order to evidence the advantage of using
indicator), at a given value of xi, with the expectation taken over x∼i
penalised splines in studies with large number of entities (thousands
(hereafter the subscript x∼i is omitted to avoid cluttering the notation,
of drainage basins in this case).
such that Ex∼ i (y xi ) ≡ E (y xi )). In other words, it is conditional on xi, e.g.
The tools to perform these analyses in Matlab are available for free
with xi fixed at one value in its interval of variation. However, E (y xi ) is
download on the author's web page (Becker, 2017).
specified so that the value that y is conditioned on is not specified.
Therefore E (y xi ) is a function of xi. It is also known as the main effect of
2. Measures of importance xi, and is equivalent to the fi(xji) discussed previously; therefore it is
nothing more than a nonlinear regression fit on a scatter plot of y
Consider the case of a composite indicator y (output) calculated by against xi.
aggregating over d normalised input variables (indicators) {xi}id=1. The ˆ
Now let m ji: =fi (x ji ), corresponding to the fitted regression value (of
most common aggregation scheme is the weighted arithmetic average, y on xi) corresponding to the jth sample point yj. The correlation ratio Si
i.e. can then be estimated in sample as
d n n
yj = ∑ wi xji, j = 1, 2, ⋯, n Sˆi = ∑ (m ji − mi )2 / ∑ (yj − y )2
i =1 (1) j =1 j =1 (5)
n
where xji is the normalised score of individual j (e.g., country) based on where mi is the mean of the mji, i.e. mi : =n−1 ∑ j =1
m ji , mj : =mˆ (x ji ). Eq.
the value Xji of the ith raw variable i = 1, …, d and wi is the nominal (5) mimics (3).
d
weight assigned to the i-th variable, such that ∑i =1 wi = 1 and wi > 0 . In summary, this means that the correlation ratio, which is a
Now consider an importance measure I, which captures the nonlinear measure of dependence of y on xi, can be estimated simply
influence of each xi on y, and which is also normalised to sum to one by fitting a nonlinear regression to a scatter plot of y against xi, taking
over all d variables. The fundamental underlying principle of this paper the variance of the resulting curve, and standardising by the uncondi-
is that Ii ≠ wi , nor is I necessarily linearly related to w , although this fact tional variance of y.
is sometimes overlooked by developers. In fact, the importance of xi is
also strongly dependent on its (possibly nonlinear) correlations with 3. Estimating the main effect
other variables, which are in turn correlated with each other. Therefore
determining and isolating the effect of xi on y is by no means trivial. As discussed, the main effect E(y xi ) is simply the nonlinear
For any given composite indicator, one can define measures of regression fit of y against xi: therefore it can be estimated by various

13
W. Becker et al. Ecological Indicators 80 (2017) 12–22

nonlinear regression methods. This section briefly describes two 1996). The fitting of the resulting smoothing parameter is performed by
approaches which are used in this work for their particular advantages. cross-validation. The fact that penalised splines are only a short step
For conciseness, further technical details are left to the references. from linear regression means that they can exploit well-known proper-
ties to give fast order-n algorithms for the calculation of cross-
3.1. Gaussian processes validation measures, therefore the fitting of a penalised spline can be
very fast. A further useful property is that penalised splines allow very
Gaussian processes (GPs) are recognised as a highly flexible simple evaluation of derivatives of main effects. This property will be
approach to probabilistically modelling nonlinear data, and are widely illustrated in Section 6.1.
used in sensitivity analysis (Becker et al., 2011, 2013). GPs are a Finally, note that in the following work “penalised splines” will
“probabilistic distribution over functions”, in that a fit through data sometimes be referred to simply as “splines”. All splines used in this
points is expressed as a random function. The Gaussian process (GP) is investigation are penalised cubic splines.
based on the assumption that any points y1, y2, .. ., yn are distributed
joint-normally, specified by a mean function which returns the mean at 4. Separating correlation from aggregation
any point, and a covariance function which specifies the covariance
between any two points as a decreasing function of the distance in each It has been widely noted in the sensitivity analysis literature that in
variable xi between them. the case of correlated inputs, the correlation ratio (first order sensitivity
The specification for the GP includes a number of hyperparameters. index) Si captures both the effect of the variable xi alone, as well as that
In this work, a full Bayesian implementation of the GP is used, where due to other variables with which it is correlated (Xu and Gertner,
prior distributions are assigned to hyperparameters, and the resulting 2008a,b; Li et al., 2010; Mara and Tarantola, 2012). Therefore Si can be
posterior GP distribution is explored using Markov Chain Monte Carlo decomposed as follows:
methods – see Bishop (2006). In this framework the confidence
Si = Siu + Sic (6)
intervals of the GP also include the uncertainty in the estimation of
the hyperparameters. where Siurepresents the uncorrelated part, and Sic the correlated part.
Since the posterior distribution (fitted to the training data) of the GP This distinction allows a greater insight into the nature of the influence
is explored by MCMC, it is possible to estimate the correlation ratio at of the variable. In classical sensitivity analysis, which is concerned with
each set of hyperparameter values visited in the search (after burn in). understanding the effects of input uncertainties on the output of a
Since there are some thousands of MCMC steps, here only 50 samples model, the decomposition of the correlation ratio gives further infor-
are taken of posterior distribution to limit computational time, and mation on how uncertainty might be reduced, since strong correlations
therefore 50 sample values of the correlation ratio. This information imply that variables need to be treated in correlated groups rather than
enables the construction of distributions and confidence intervals over individually. In the context of composite indicators, it can potentially
Si. aid the construction and setting of weights: knowing for example that a
It should be noted that perhaps the main weakness of GPs is that the given variable has Sic ≈ Si implies that the value of the weight might
cost of training scales poorly with n, the number of training data, since have very little impact on the end result, because the impact of the
it involves the inversion of a n × n covariance matrix at a cost of the variable is entirely due to its correlation with other variables present in
order of n3. For this reason, although GPs are an excellent tool for the framework. Further, if Sic results to be negative, this indicates that
nonlinear regression (even allowing estimation of correlation ratio in there is a conceptual problem with the variable. This is caused by
the presence of discontinuities (Becker et al., 2013)), they cannot be negative correlations between variables, which are usually inadvisable
applied to large-n problems without discarding some of the data points. in composite indicators; an example of this occurred in the Sustainable
Finally, GPs naturally extend into higher dimensions, i.e. multi- Society Index—the conclusion was that in the presence of negative
variate regression. This property is used in the following section to correlations, the indicators should not actually be aggregated and
model dependence. In this work, the R package tgp is used for full- instead presented as a scoreboard (Saisana and Philippas, 2012).
Bayesian GPs (Gramacy and Taddy, 2010). In order to obtain the decomposition in (6), this paper follows Xu
and Gertner (2008a,b), who use a regression-based approach. However,
3.2. Penalised splines whereas Xu and Gertner assume a linear relationship between y and xi,
the approach here generalises to associations both between y and xi and
While Gaussian processes are the ideal tool for many of the between the xi themselves. The main advantage of the nonlinear
applications in this study, they encounter difficulties when there are a regression approach adopted herein is that no prior information about
very large number of data points (see 6.3 or a large number of the joint distribution of input variables is required, only a sample from
regressions, as are required in the optimisation of weights (see 5). In the distribution.
such cases, penalised splines offer a fast alternative. Penalised splines Of the terms in (6), Si can be readily estimated via a nonlinear
are an extension of linear parametric regression (linear in the para- regression using one of the regression methods discussed in the
meters), but also have the capabilities of nonparametric regression (i.e. previous section. The uncorrelated term Siu can be estimated by a
local polynomial regression), such as the flexibility to accommodate relatively straightforward procedure based on multivariate linear
nonlinear trends in the data (Ruppert et al., 2003). The basis function regression. First, the residuals ẑi of a regression of xi on x∼i are
which is the heart of the spline model is given by (x − κh )+p , where the obtained, i.e.,
“+” subscript denotes the positive part; in other words, for any number
u, u+ = u if u is positive, and equals zero otherwise. The κh parameter is ⎛ d ⎞
zˆi = xi − xˆi = xi − ⎜⎜β0 + ∑ βˆl xl⎟⎟
called the “knot” of the basis function. Splines are constructed by using ⎝ ⎠
l≠i (7)
a number k = 0, 1, 2, .. ., K of p-order spline basis functions with
different knots κh, combined with a p-order polynomial. In the applica- where the βl are coefficients of a multivariate linear regression, found
tions here a value of p = 3 is used (cubic splines). by the standard least-squares estimator. This step effectively removes
In order to avoid over-fitting, the estimation of the coefficients is the correlation between xi and x∼i by removing the (linear) depen-
constrained to limit the influence of the spline basis terms, resulting in a dence. Note that this implicitly assumes pairwise linear dependence
smoother fit: this results in the penalised spline model. The constraining between the input variables. From this point, the sensitivity of y to ẑi is
of regression parameters shares similarities with other approaches such estimated in exactly the same way as estimation of Si: a nonlinear
as ridge regression (see e.g. Hastie et al. (2001)), and LASSO (Tibshirani, regression of y on ẑi is performed, and the resulting fitted values

14
W. Becker et al. Ecological Indicators 80 (2017) 12–22

Fig. 1. Illustration of the weight-optimisation algorithm.

(Rasmussen and Williams, 2006).

In summary, the steps of the methodology described in this section
are as follows, repeated for each variable xi:

1. Estimate Si using a nonlinear regression approach, e.g. GPs or

splines (see previous section).
2. Perform a regression of xi on x∼i. This can be either linear (using
multivariate linear regression), or nonlinear (using a multivariate
Gaussian process). Denote this fitted regression as x̂i .
3. Get the residuals of this regression, ẑi , using zˆi = xi − xˆi .
4. Estimate Siu by a nonlinear regression of y on ẑi , using the same
approach as in Step 1.
5. Get Sic by the simple expression Sic = Si − Siu .

Notice then that there are three regression steps per variable: a first
univariate regression to estimate Si, a second multivariate regression
(either linear or nonlinear) to remove the dependence of xi on x∼i, and
a third univariate regression of y on the residuals (effectively the
“uncorrelated” xi) to obtain Siu .
Fig. 2. Correlation between variables of the four components of the Resource Governance
Index (in order: Institutional and Legal Setting, Reporting Practices, Safeguards and
Quality Controls, and Enabling Environment). Numbers represent Pearson correlation
coefficients; ellipses represent the strength and direction of the correlation. 5. Optimising weights

{yˆj(∼ i ) }nj =1 are used to estimate Siu as follows: Given a set of weights, the tools in the previous sections can be used
n n
to analyse in some detail the influence of each variable on the
∑ (yˆj(∼i) − y (∼i) )2 / ∑ (yj − y )2 composite indicator. A natural question which subsequently arises is,
j =1 j =1 (8) how can one adjust the weights of the composite indicator in order to
achieve the desired influence of each indicator? This question was
where y (∼ i )
is the mean of the ŷ j(∼ i ) .
Note that Xu and Gertner use y in addressed in Paruolo et al. (2013), in which a unique solution based on
place of y (∼ i ) : while in linear regression the two terms are equal by linear regression was found. However, when the main effects are
definition, in nonlinear regression they may differ slightly. For simpli- nonlinear, more accurate results will be achieved using nonlinear tools.
city, only penalised splines are used for the the regression of y on ẑi , Here, a simple numerical approach is proposed which searches for
however GPs or another approach could also be used. the weights that result in the desired importance using an optimisation
Sic may now be found simply by recalling that Sic = Si − Siu . Note ∼
algorithm. Let Si be the normalised correlation ratio of xi, i.e.
that, if the correlation between variables is negative, it may occur that ∼ d
Si = Si / ∑i =1 Si . The target normalised correlation ratio is denoted as
Sic is negative, and Si < Siu . In the case of composite indicators, this ∼ ∼
S *: it is assumed that S * = w , i.e. the weights assigned by developers
i i i
would indicate a conceptual problem as discussed.
reflect the intended importance. The normalisation of Si is done only in
Finally, if dependence between variables are thought to be non-
order to make it directly comparable with the target values (weights),
linear, the regression of xi on x∼i in (7) can also be performed with a
which also sum to one. In Paruolo et al. (2013) the normalisation was
nonlinear method. In reality, there is no reason to assume the relation-
done with respect to S1; in reality any value can be used since we only
ship between the input variables in a composite indicator is necessarily
want to make a relative comparison.
linear. To investigate the difference, multivariate Gaussian process
The optimisation problem is framed by defining an objective
regression is also used in the case studies in this paper (see Section 3.1).
function as the sum (over i = 1, 2, .. ., d) of squared differences
A further advantage of this approach is that the dependence need not be ∼ ∼
between the Si at a given set of weights, and the target Si*. The desired
pairwise, as the GP is able to model higher-order interactions. However,
weights wopt can be found by minimising this objective function, i.e.
as discussed, this can impose some limitations on sample size such that
samples of around 1000 points or over might become very slow to work d
∼ ∼
with. Still, many composite indicators have sample sizes lower than this wopt = argmin w ∑ (Si* − Si (w))2 .
limit, and there also exist approaches to apply GPs on large-N problems i =1 (9)

15
W. Becker et al. Ecological Indicators 80 (2017) 12–22

Fig. 3. GP main effect fits of the RGI (lines represent samples of the posterior distribution).

Table 1
Normalised Si of the Resource Governance Index. “Original” represents values before
weight optimisation; “Optimised” represents values with optimised weights with no
constraints; “Optimised (+)” represents values with optimised weights with the
constraint that all weights must be positive.

I. & L. S. Rep. Prac. S. & Q.C. En. Env.

Original 0.209 0.303 0.276 0.211

Optimised 0.200 0.400 0.200 0.200
Optimised (+) 0.187 0.372 0.255 0.185

where w = {wi}id=1. Of the many optimisation algorithms available, the

approach selected here is the well-known Nelder-Mead simplex search
Fig. 4. Estimates of Si (full bars) broken down into Si(u) (blue) and Si(c) (red), using both method, since it requires no knowledge of derivatives and tends to
linear and nonlinear dependence modelling. Error bars represent two standard deviations produce efficient convergence results (Lagarias et al., 1998). However,
in the estimate of Si, according to GP. if this does not result in a satisfactory solution, other approaches could
be used: for an overview on this very large topic of research see e.g.
(Nocedal and Wright, 2006). Note that, in contrast to the linear case
described in Paruolo et al. (2013), there is no theoretical result that a
unique solution exists or that an optimisation algorithm will converge
to this solution. However, the success of the procedure can be easily
judged by the resulting correlation ratios, and in our test cases a
solution has always been found. Fig. 1 shows an overview of the
optimisation process applied to the weights.
In practice, the optimised weights may result in negative weights
due to the correlation structure between indicators. It is not proposed
here that the optimised weights are blindly applied; rather that they are
used, among other considerations, as a learning tool to show how far
weights are from the “optimal” values, which indicators’ weights have
little impact on the results, and even which indicators could be
Fig. 5. RGI weights for each indicator, showing original weights, optimised weights, and
optimised weights with a positive constraint. removed. These ideas are illustrated in the following three case studies.
Weights can also be used as “scaling coefficients” taking the values 0.5
or 1.0, with the aim of arriving at aggregate scores that are balanced in
their underlying components (i.e., that indicators can explain a similar

16
W. Becker et al. Ecological Indicators 80 (2017) 12–22

Fig. 6. Shifts in rank between original RGI and weight-optimised RGI. Horizontal axis is ordered by weight-optimised rank. Selected countries labelled.

Table 2
Top ten countries in the RGI: comparison of original against optimised weights. Numbers
in brackets refer to the difference between the original rank and the optimised rank.

Rank Original weights Optimised Weights

1 Norway (-2) United States (Gulf of Mexico) (1)

2 United States (Gulf of Mexico) (1) Australia (Western Australia) (2)
3 United Kingdom (-2) Norway (-2)
4 Australia (Western Australia) (2) Peru (7)
5 Brazil (-6) United Kingdom (-2)
6 Mexico (-1) Timor-Leste (7)
7 Canada (Alberta) (-3) Mexico (-1)
8 Chile (0) Chile (0)
9 Colombia (-5) Trinidad and Tobago (1)
10 Trinidad and Tobago (1) Canada (Alberta) (-3)

amount of variance in their respective component). This approach has

been followed by the developers of the Global Innovation Index since
the 2012 edition (Dutta et al., 2014). The use of weights taking the
values of 0.5 or 1 is simple enough to be communicated to a larger
audience, while at the same time it can ensure statistical soundness in
the development of an index.

6. Case studies Fig. 7. Correlation between the seven components of the GCI (in order: Science,
Technology & Knowledge; Culture; International Peace and Security; World Order;
This section shows the various strands of methodology in the Planet and Climate; Prosperity and Equality; and Health and Wellbeing). Numbers
previous section applied to three existing composite indicators. To represent Pearson correlation coefficients; ellipses represent the strength and direction of
the correlation.
implement the tools described in the previous sections, codes were
written in Matlab. These are available for free download on the author's
web page (Becker, 2017), with the exclusion of the Gaussian process dollars in annual profits. The future of these countries depends on how
estimation of Si because this involves a complicated link with the R well they manage their natural resources.
programming language which is not easily transferred to another To evaluate the quality of governance in the extractive sector, the
computer. However, we can supply these codes with instructions on Resource Governance Index employs a 173-item questionnaire, the
request. answers to which are grouped into 45 indicators that are then mapped
into three (of the four) RGI dimensions: Institutional and Legal Setting,
Reporting Practices, and Safeguards and Quality Controls. The fourth
6.1. The Resource Governance Index dimension, Enabling Environment, consists of five additional indicators
that describe a country's broader governance environment; it uses data
The Resource Governance Index (RGI) is developed by the Revenue compiled from over 30 external sources. The weights are set equal to
Watch Institute in order to measure the transparency and accountability 0.2, except for Reporting Practices, which is deemed to be twice as
in the oil, gas and mining sectors in 58 countries (Quiroz and Lintzer, important as the other indicators and has a weight of 0.4.
2013). These nations produce 85 percent of the world's petroleum, 90 Examining first the correlations between variables, Fig. 2 shows that
percent of diamonds and 80 percent of copper, generating trillions of

17
W. Becker et al. Ecological Indicators 80 (2017) 12–22

Fig. 8. GP main effect fits of the GCI (lines represent samples of the posterior distribution).

Fig. 9. Estimates of Si (full bars) broken down into Si(u) (blue) and Si(c) (red), using both linear and nonlinear dependence modelling. Error bars represent two standard deviations in the
estimate of Si, according to GP.).

the four indicators are all correlated with each other positively and with correct amount. This problem is likely due to the fact that the correlated
values ranging from 0.82 to 0.41. Fig. 3 shows the GP fits on the RGI part of Si dominates the uncorrelated part in all four indicators, which
data. It is evident that the data is largely linear, but y shows some means that correlations largely dictate influence, rather than weights. A
nonlinearity with respect to Enabling Environment. The Si values, further point is that the linear and nonlinear estimates of Si(u) are not
decomposed into correlated and correlated parts, are shown in Fig. 4. very different; in this case at least, the linear estimate would have been
Here we must keep in mind that the influence (and therefore the sufficient.
correlation ratio) of Reporting Practices should be twice that of the Fig. 5 shows the weights for the four components of the RGI—In-
other three indicators, when clearly it is not (the confidence intervals stitutional and Legal Setting, Reporting Practices, Safeguards and
indicate that there is little uncertainty in these estimate). Instead, its Quality Controls, and Enabling Environment—that are obtained
correlation ratio is slightly higher than that of Safeguards and Quality through the optimisation procedure; the corresponding normalised Si
Controls. The other two indicators are less influential, but not by the values are given in Table 1. The unconstrained optimisation algorithm

18
W. Becker et al. Ecological Indicators 80 (2017) 12–22

Table 4
Top ten countries in the GCI: comparison of original against optimised weights. Numbers
in brackets refer to the difference between the original rank and the optimised rank.

Rank Original weights Optimised Weights

1 Ireland (-21) Sweden (5)
2 Finland (-7) Canada (10)
3 Switzerland (0) Switzerland (0)
4 Netherlands (-1) United Kingdom (3)
5 New Zealand (-15) Netherlands (-1)
6 Sweden (5) Australia (9)
7 United Kingdom (3) France (4)
8 Norway (-7) Germany (5)
9 Denmark (-7) Finland (-7)
10 Belgium (-3) Luxembourg (6)
Fig. 10. GCI weights for each indicator, showing original weights, optimised weights, and
optimised weights with a positive constraint.
to three of the RGI components and a weight of almost 1.0 to Reporting
Practices.. This is an interesting result in itself, because it suggests that
Table 3
Normalised Si of the GCI. “Orig.” represents values before weight optimisation; “Opt.” in fact three of the four RGI components could actually be omitted
represents values with optimised weights with no constraints; “Opt. (+)” represents except for Reporting Practices, given that the information about the
values with optimised weights with the constraint that all weights must be positive. other components is contained in it through correlations.
This case study has served to demonstrate that two strongly
S & T. Cult. I.P. & S. W.O. P. & C. P. & Eq. H. & Wb.
correlated variables can hardly have very different influence in the
Orig. 0.187 0.235 0.017 0.235 0.122 0.090 0.115 overall index, without resorting to the use of negative weights. By using
Opt. 0.143 0.143 0.143 0.143 0.143 0.143 0.143 the tools developed here, these issues become much clearer and allow
Opt. (+) 0.162 0.159 0.055 0.163 0.149 0.152 0.159 developers to make more informed decisions.
The changes in the RGI ranking as a result of the optimisation of
weights (with the constraint that all weights are positive and sum up to
is able to find weights (0.12, 1.30, −0.46, 0.05) that produce the target
one) is shown in Fig. 6. The chart labels the countries with the highest
correlation ratios, but it requires that the weight of Safeguards and
rank changes. Most notably, Algeria improves by 18 places, whilst
Quality Controls is negative. This is because the original influence of
South Africa drops by 31 places (out of 58). Table 2 also compares the
Safeguards and Quality Controls is higher than it should be due to the
original top ten countries in the ranking with the top ten resulting from
strong correlations to the other three components. Due to the correla-
optimising weights. The changes are not very large for most countries.
tions, it also requires a large change in its weight to give it the desired
Nevertheless, Brazil and Colombia would drop by six and five places,
importance. By constraining the weights to be positive (and to sum to
respectively, whilst Peru would gain seven places. The USA becomes
one), the alternative optimised weights offer a slight improvement over
the highest ranking country. One should be careful in interpreting these
the original weights in terms of correlation ratios, but not by a great
results however: they represent the ranking that would be obtained if
degree. The influence of Reporting Practices is brought closer to the
“importance” is directly interpreted as nonlinear dependence. Further,
target, but the influence of the Enabling Environment is slightly further
they ignore the many other substantial uncertainties involved in the
from the target than with the original weights. Moreover, the set of
construction of a composite indicator: choices of which indicators to
positively-constrained optimised weights results in setting zero weights

Fig. 11. Shifts in rank between original GCI and weight-optimised GCI. Horizontal axis is ordered by weight-optimised rank. Selected countries labelled.

19
W. Becker et al. Ecological Indicators 80 (2017) 12–22

Fig. 12. Water retention index as calculated for 2006 at the catchment level.

include, how to impute missing data, how to aggregate, and so on. All seven components are weighted equally with a weight of 1/7.
However, one might at least recognise that the rank shifts here Examining the GP fits (see Fig. 8) it is evident that while six of the
represent a credible lower bound on the uncertainty. seven components have a positive and fairly linear effect on the GCI,
International Peace and Security has a slightly negative main effect.
This is further reflected in Fig. 9, which shows the very low effect of
6.2. The Good Country Index
International Peace and Security on the composite. In fact, the
correlated part of Si is actually negative—as discussed earlier this
The Good Country Index (GCI) is developed by the Good Country
implies a problem with this component because, as is evident from
Party with a view to measure what a country contributes to the
Fig. 7, it is negatively correlated with the six remaining components
common good of humanity, and what it takes away (Anholt and
(see Fig. 7). A further conclusion from Fig. 9 is that three compo-
Govers, 2014), according to its developers’ normative framework and
nents—Planet and Climate, Prosperity and Equality, Health and Well-
world view. In total, 125 countries are included in the Index. In contrast
being—have relatively low Si values.
to the majority of similar composite indicators, the Good Country Index
The results of the (unconstrained) optimisation of weights are
does not measure what countries do at home; rather, it aims to start a
shown in Fig. 10. The most notable result here is that the weight of
global discussion about how countries can balance their duty to their
International Peace and Security is required to be negative due to the
own citizens with their responsibility to the wider world.
strong and negative correlations to other GCI components. The weight
The GCI builds upon 35 indicators that are produced by the United
of Culture is also required to be negative, for the reason that its effect
Nations and other international agencies, and a few by NGOs and other
must be reduced. Indeed, Culture has the strongest bivariate correla-
organisations. Most of the indicators used are direct measurements of
tions to other GCI components (0.78 with Science and Technology, and
world-friendly or world-unfriendly behaviour (such as signing of
0.65 with World Order), hence it needs a large change in its weight to
international treaties, pollution, acts of terrorism, wars) and some are
properly adjust its effect. This problem is further evidenced by the fact
rather indirect (such as Nobel prizes, exports of scientific journals). The that Culture has the largest Si(c) of all GCI components.
35 indicators are split in seven components of five indicators each. Table 3 shows that under the constraint that the weights are
These seven components, which closely mirror the dimensions of the positive, the influence of each component is fairly equal, apart from
United Nations Charter, are: Science, Technology & Knowledge; International Peace and Security, which is somewhat less than the
Culture; International Peace and Security; World Order; Planet and others. Moreover, Culture, International Peace and Security, and World
Climate; Prosperity and Equality; and Health and Wellbeing. A ranking Order, all have zero or near-zero weights. This shows that, if one is
is calculated for each of the seven components. The Good Country Index prepared to accept a slightly lessened influence of International Peace
is then calculated by taking the arithmetic average of the seven ranks.

20
W. Becker et al. Ecological Indicators 80 (2017) 12–22

Fig. 13. Spline fits to WRI data prior to optimisation.

Table 5 capacity for water retention, and is used as a tool to assess the impact of
Weights and normalised Si of the Water Retention Index. “Original” represents values land management scenarios at the European scale. It uses several data
before weight optimisation; “Opt” represents optimised weights with no constraints.
sets that are directly affected by changes in land use, allowing it to be
Weights Normalised Si re-calculated based on future land use projections.
The index is comprised of four parameters that take into account the
Original Opt Original Opt physical potential for water retention in the landscape. These include
interception by vegetation (Rv), storage in surface water bodies (Rwb),
Rv 0.200 0.170 0.231 0.200
Rgw 0.200 0.290 0.096 0.200 soil moisture (Rs), and percolation to groundwater stores (Rg w ). In
Slope 0.200 0.100 0.313 0.200 addition, the influence of average slope and soil sealing are taken into
Rs 0.200 0.140 0.238 0.200 account. The first four parameters and the slope together describe the
Rwb 0.200 0.280 0.122 0.200 physical capacity of the land to retain water, therefore reflecting the
maximal potential retention of water in the natural environment.
Lastly, the influence of artificially sealed areas are taken into account
and Security, one could remove the three components completely, and
as a multiplication factor to compute the final index, representing the
the result would be a fairly well-balanced composite indicator which
actual potential for water retention.
still captures the influence of the removed components simply by
The WRI is different from the previous two case studies in that it can
correlation.
be calculated at resolutions as high as the 1km2 level, across Europe.
The changes resulting from the optimisation of weights are shown in
However, the most relevant resolution for policy-making is to evaluate
Fig. 11. Most notably Sudan improves by 86 places (out of 125
the index at the level of catchment areas (drainage basins)—see Fig. 12.
countries). Instead, Ireland drops from first place to 21st. The largest
Across Europe, even this lower resolution results in a data set 4789
decline is experienced by South Africa, which loses 67 places. The
points, each with indicator values and resulting WRI values. This makes
comparison of the top ten performers before and after optimisation in
it somewhat challenging from a computational point of view, in terms
Table 4 shows that New Zealand loses 15 places, going from 5th to 20th
of estimating the correlation ratio and optimising the weights to reflect
place, while Canada and Australia enter the top ten with gains of ten
the desired importance. Here, the advantages of penalised splines come
and nine places respectively.
to the fore: since they can be fitted very efficiently, they bring such
large-n composite indicators within reach of the tools described here.
6.3. The Water Retention Index The indicators in the WRI are only weakly correlated; for this reason
the correlated and uncorrelated correlation ratios are not presented
The Water Retention Index (WRI) was developed to assess the here. Fig. 13 shows the spline fits to the WRI data. There are clearly
potential of a landscape to regulate the quantity of water passing strong skews in each indicator, and significantly nonlinear responses. In
through it (Vandecasteele et al., 2016). The index reflects the relative

21
W. Becker et al. Ecological Indicators 80 (2017) 12–22

this case, linear regression would not be an adequate tool. Table 5 W. Becker, A. Saltelli, in: Handbook of Design and Analysis of Experiments, Handbooks of
Modern Statistical Methods, Chapman & Hall/CRC, 2015. pp. 627-674.
shows the results of the weight-optimisation algorithm using penalised Becker, W., Rowson, J., Oakley, J., Yoxall, A., Manson, G., Worden, K., 2011. J. Biomech.
splines. Of the five indicators, it is clear that the normalised Si do not 44, 1499–1506.
reflect the desired equal importance, with the Slope variable having Becker, W., Worden, K., Rowson, J., 2013. Mech. Syst. Signal Process. 34, 57–75.
W. Becker, Matlab toolbox for analysis and adjustment of weights in composite indicators:
three times the influence of Rgw, for example. Using the optimisation http://dx.doi.org/10.13140/RG.2.2.27035.46883, 2017.
algorithm, the optimised weights constrained to positive values result Bishop, C., 2006. Pattern Recognition and Machine Learning. Springer, New York.
in all five indicators having equal importance to the overall index.. This A. Cobham, P. Jansky, J. Christensen, S. Eichenberger, Financial Secrecy Index 2013:
Methodology, Technical Report, The Tax Justice Network, 2013.
is largely due to the fact that there are no strong correlations between Da Veiga, S., Wahl, F., Gamboa, F., 2009. Technometrics 51, 452–463.
indicators, so weights do not need to be adjusted by large margins, as is S. Dutta, B. Lanvin, S. Wunsch-Vincent, Insead, World Intellectual Property Organization,
the case with the previous case studies (RGI and GCI). Johnson Graduate School of Management (Cornell University), The global innovation
index 2014: the human factor in innovation, 2014. OCLC: 916514895.
Gramacy, R.B., Taddy, M.A., 2010. J. Stat. Software 33, 1–48.
7. Conclusions Hastie, T., Tibshirani, R., Friedman, J., 2001. The elements of statistical learning. Data
mining, inference and prediction. Springer.
For developers and users of composite indicators, three exploratory Hsu, A., 2016. Environmental Performance Index, Technical Report. Yale University, New
Haven, CT.
tools have been presented herein which together allow for a detailed Jahan, S., 2015. Human Development Report, Technical Report. United Nations
analysis of the importance of weights assigned to the index compo- Development Programme, New York.
nents. First, one can determine the true influence of each component to Kelley, J.G., Simmons, B.A., 2015. Am. J. Political Sci. 59, 55–70.
Lagarias, J.C., Reeds, J.A., Wright, M.H., Wright, P.E., 1998. SIAM J. Optimiz. 9,
the overall index via the correlation ratio, which accounts for non- 112–147.
linearity in the data associations. This value comes with confidence Li, G., Rabitz, H., Yelvington, P.E., Oluwole, O.O., Bacon, F., Kolb, C.E., Schoendorf, J.,
intervals if the estimation follows a Bayesian Gaussian process. Second, 2010. J. Phys. Chem. A 114, 6022–6032.
Mara, T.A., Tarantola, S., 2012. Reliability Eng. Syst. Safety 107, 115–121.
one can decompose the correlation ratio into a part caused by Nocedal, J., Wright, S., 2006. Numerical Optimization. Springer Science & Business
correlations with other index components, and a part caused purely Media.
by the indicator itself. This decomposition allows one to identify Paruolo, P., Saisana, M., Saltelli, A., 2013. J. Royal Stat. Soc. Ser. A (Stat. Soc.) 176,
609–634.
indicators whose influence in the overall index is not greatly affected
Quiroz, J.C., Lintzer, M., 2013. The 2013 Resource Governance Index, Technical Report.
by their weight value (i.e. those indicators with a high correlated Si The Revenue Watch Institute.
compared to the uncorrelated part), and also whether correlations are Rasmussen, C., Williams, C., 2006. Gaussian Processes for Machine Learning. The MIT
Press, Cambridge, Massachusetts.
responsible for negative effects, among other things. This information
Ruppert, D., Wand, M., Carroll, R., 2003. Semiparametric regression, volume 12
does not emerge by examining the correlation matrix alone. Third, one Cambridge Univ Pr.
can estimate the values for the weights that would lead to the desired Saisana, M., Philippas, D., 2012. European Commission, Joint Research Centre, Institute
importance of the index components. The tools proposed here could for the Protection and the Security of the Citizen, Sustainable society index (SSI):
taking societies’ pulse along social, environmental and economic issues. Publications
also generalise to composite indicators with a large sample size (a few Office, Luxembourg OCLC: 847462966.
thousand of data points) by the use of penalised splines. The statistical Saisana, M., Saltelli, A., Tarantola, S., 2005. J. Royal Stat. Soc. A 168 (2), 307–323.
methods used here are coded in Matlab and available to download as a Tibshirani, R., 1996. J. Royal Stat. Soc. Ser. B (Methodological) 267–288.
Van de Kerk, G., Manuel, A.R., 2008. Ecol. Econ. 66, 228–242.
package at (Becker, 2017). I. Vandecasteele, I. Mar i Rivero, C. Baranzelli, W. Becker, I. Dreoni, C. Lavalle, O.
Batelaan, in: Conference proceedings of the 22nd International Sustainable
References Development Research Society Conference, International Sustainable Development
Research Society, Universidade Nova de Lisboa, Lisbon, Portugal, 2016.
Xu, C., Gertner, G.Z., 2008a. Reliability Eng. Syst. Safety 93, 1563–1573.
Anholt, S., Govers, R., 2014. The Good Country Index, Technical Report. The Good Xu, C., Gertner, G., 2008b. Reliability Eng. Syst. Safety 93, 1060–1071.
Country Party. Yang, L., 2014. An Inventory of Composite Measures of Human Progress, Technical
Bandura, R., 2011. Composite Indicators and Rankings: Inventory 2011, Technical Report. United Nations Development Programme Human Development Report Office.
Report. United Nations Development Programme - Office of Development Studies.

22

View publication stats