CMA-ES with Restarts for Solving CEC 2013

Benchmark Problems
Ilya Loshchilov

To cite this version:

Ilya Loshchilov. CMA-ES with Restarts for Solving CEC 2013 Benchmark Problems. IEEE Congress
on Evolutionary Computation, Jun 2013, Cancun, Mexico. �hal-00823880�

HAL Id: hal-00823880

https://inria.hal.science/hal-00823880v1
Submitted on 18 May 2013

HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est

archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents
entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non,
lished or not. The documents may come from émanant des établissements d’enseignement et de
teaching and research institutions in France or recherche français ou étrangers, des laboratoires
abroad, or from public or private research centers. publics ou privés.
 CMA-ES with Restarts for Solving CEC 2013
Benchmark Problems

Ilya Loshchilov
Laboratory of Intelligent Systems
École Polytechnique Fédérale de Lausanne, Switzerland
Email: [email protected]

Abstract—This paper investigates the performance of 6 ver- II presents the main principles of the CMA-ES algorithm.
sions of Covariance Matrix Adaptation Evolution Strategy (CMA- Section III describes the restart strategies of CMA-ES. Section
ES) with restarts on a set of 28 noiseless optimization problems IV explains the experimental procedure and comments the
(including 23 multi-modal ones) designed for the special session experimental results. Section V concludes the paper with a
on real-parameter optimization of CEC 2013. The experimental discussion and some perspectives for further research.
validation of the restart strategies shows that: i). the versions
of CMA-ES with weighted active covariance matrix update
outperform the original versions of CMA-ES, especially on ill- II. T HE (µ/µw , λ)-CMA-ES
conditioned problems; ii). the original restart strategies with in- The CMA-ES algorithm [8], [7] optimizes an objective
creasing population size (IPOP) are usually outperformed by the
bi-population restart strategies where the initial mutation step-
function f : x ∈ Rn → f (x) ∈ R by sampling λ candidate
size is also varied; iii). the recently proposed alternative restart solutions from a multivariate normal distribution. It exploits
strategies for CMA-ES demonstrate a competitive performance the best µ solutions out of the λ ones to adaptively estimate
and are ranked first w.r.t. the proportion of function-target pairs the local covariance matrix of the objective function, in order
solved after the full run on all 10-, 30- and 50-dimensional to increase the probability of successful samples in the next
problems. iteration. More formally, at iteration t, (µ/µw , λ)-CMA-ES
samples λ individuals (k = 1 . . . λ) according to
I. I NTRODUCTION
2
   
The Covariance Matrix Adaptation Evolution Strategy (t+1)
xk = N m(t) , σ (t) C(t) = m(t) +σ (t) ·N 0, C(t) , (1)
(CMA-ES) proposed by [8], [7] has become a standard for
continuous black-box evolutionary optimization. The main ad- where m(t) denotes the mean of a normally distributed random
vantage of CMA-ES over classical Evolution Strategies comes vector, C(t) is the covariance matrix and σ (t) is the mutation
from the use of correlated mutations instead of axis-parallel step-size.
ones. The adaptation of the covariance matrix C allows to
steadily learn appropriate mutation distribution and increase These λ individuals are evaluated and ranked. The mean of
the probability of repeating the successful search steps. the distribution is updated and set to the weighted sum of the
Pµ (t)
best µ individualsPas m(t+1) = i=1 wi xi:λ , with wi > 0 for
However, there are several properties of black-box opti- µ
i = 1 . . . µ and i=1 wi = 1, where index i : λ denotes the
mization problems which may lead to a premature convergence i-th best individual after the objective function. In the original
of CMA-ES, among the most common are multi-modality CMA-ES the information about the remaining (worst λ − µ)
and uncertainty. To increase the probability of finding the solutions is used only implicitly during the selection process.
global optima, IPOP-CMA-ES [2] and BIPOP-CMA-ES [4]
restart strategies for CMA-ES have been proposed. The IPOP- However, it has been shown in [11] that the information
CMA-ES was ranked first on the continuous optimization from the worst solutions also can be used to reduce the
benchmark at CEC 2005 [3]; and BIPOP-CMA-ES showed variance of the mutation distribution in unpromising direc-
the best results together with IPOP-CMA-ES on the black- tions. The corresponding active (µ/µI , λ)-CMA-ES algorithm
box optimization benchmark (BBOB) in 2009 and 2010 [1]. demonstrates a performance gain up to a factor of 2 without
Later, alternative restart strategies for CMA-ES proposed in loss of performance on any of tested functions in [11]. Later,
[12] demonstrated an even more competitive performance on the active update of (µ/µI , λ)-CMA-ES was extended to the
some of multi-modal functions during the BBOB 2012. The weighted case of (µ/µW , λ)-CMA-ES, where wi > wi+1 for
recently proposed weighted active covariance matrix update i = 1 . . . λ−1. This weighted active (µ/µW , λ)-CMA-ES (also
of CMA-ES [11], [9] is also a competitive alternative to the referred to as aCMA-ES) was implemented in the IPOP regime
original update procedure, it allows to substantially improve of restarts as IPOP-aCMA-ES and demonstrated improvements
the performance both on unimodal and multi-modal functions up to a factor of 2 on a set of noiseless and noisy functions
[9]. This paper focuses on analyzing the performance of the from the BBOB [9].
restart strategies of CMA-ES with the original and weighted More formally, the active CMA-ES only differs from
active covariance matrix updates on the CEC 2013 benchmark
the original CMA-ES in the adaptation of the covariance
test [10].
matrix C(t) . Like for CMA-ES, the covariance
Pµ matrixt is
The remainder of this paper is organized as follows. Section computed from the best µ solutions, C+
µ = i=1 wi
xi:λ −m
σt ×
(xi:λ −mt )T
σt . The main novelty is to exploit the worst solutions B. The IPOP-CMA-ES and IPOP-aCMA-ES
Pµ−1 T
to compute C−µ = i=0 wi+1 yλ−i:λ yλ−i:λ , where yλ−i:λ = As mentioned, [6] demonstrated that increasing the pop-
Ct (xλ−µ+1+i:λ −mt )
−1/2
xλ−i:λ −mt ulation size improves the performance of CMA-ES on multi-
× σt .
The covariance matrix
k C t −1/2 (xλ−i:λ −mt )k modal functions. The authors of [6] suggested a restart strategy
estimation of these worst solutions is used to decrease the for CMA-ES with successively increasing population size.
variance of the mutation distribution along these directions: Such an algorithm was later introduced in [2] as IPOP-CMA-
ES. IPOP-CMA-ES only aims at increasing the population
size λ. Each time at least one of the stopping criteria is met
Ct+1 = (1 − c1 − cµ + c− α− t
old )C + by the CMA-ES, it launches a new CMA-ES with population
t+1 t+1 T + (2)
+c1 pc pc + (cµ + c (1 − α−
−
old ))Cµ − c Cµ ,
− − size λ = ρiinc
restart
λdef ault , where irestart is the index of the
restart and λdef ault is the default population size. Factor ρinc
where pt+1 is adapted along the evolution path and coef- must be not too large to avoid ”overjumping” some possibly
c
optimal population size λ∗ ; in [2] it is set to ρinc = 2 that
ficients c1 , cµ , c− and α− old are defined such that c1 + cµ −
c− α− ≤ 1. The interested reader is referred to [7], [9] for a in certain cases allows to keep a potential loss in terms of
old
more detailed description of these algorithms. function evaluations (compared to the “oracle“ restart strategy
which would directly set the population size to the optimal
A potential issue of the active update is that the positive value λ∗ ) by about a factor of 2.
definiteness of the covariance matrix cannot be guaranteed
anymore, that may result in algorithmic instability. According The active version of IPOP-CMA-ES (IPOP-aCMA-ES)
to [12], this issue is not observed on the BBOB benchmark has been proposed in [9].
suite [5]. In our experiments with the CEC 2013 benchmark
suite this issue is also never observed. C. The BIPOP-CMA-ES and BIPOP-aCMA-ES
In BIPOP-CMA-ES [4] after the first single run with
III. R ESTART S TRATEGIES FOR CMA-ES default population size, the algorithm is restarted in one of
A. Preliminary Analysis two possible regimes and account the budget of function
evaluations spent in the corresponding regime. Each time the
The CMA-ES algorithm is a local search optimizer and its algorithm is restarted, the regime with smallest budget used so
default population size λdef ault has been tuned for unimodal far is used.
functions. On multi-modal functions, however, it can get stuck
in local optima and the convergence to global optima is not Under the first regime the population size is doubled as
guaranteed. Various approaches to increase the probability of λlarge = 2irestart λdef ault in each restart irestart and use
0 0
finding global optima have been proposed, many of them some fixed initial step-size σlarge = σdef ault . This regime
belong to i). niching approaches and ii). restart strategies. corresponds to the IPOP-CMA-ES.

A representative approach of the first category is the CMA- Under the second regime the CMA-ES is restarted with
0
ES with the fitness sharing [15], where the niche radius is some small population size λsmall and step-size σsmall , where
adapted during the search that allows to keep several running λsmall is set to
individual CMA-ES instances on a certain distance from each
other, and, thus, maintain some diversity. Another example
  U[0,1]2 
λlarge
is the NBC-CMA-ES algorithm [14] with the niching via λsmall = λdef ault 12 λdef ault
, (3)
Nearest-Better Clustering (NBC) which is employing a radius-
free basin identification method. In this approach, the niches
Here U [0, 1] denotes independent uniformly distributed
are dynamically identified and the corresponding points are
numbers in [0, 1] and λsmall ∈ [λdef ault , λ/2]. The initial step-
used to form populations for individual CMA-ES instances. 0 0
size is set to σsmall = σdef ault × 10 .
−2U[0,1]
According to [14], for very highly multi-modal functions,
the effort invested into the coordination of local searches In each restart, BIPOP-CMA-ES selects the restart regime
often does not pay off as it becomes almost impossible to with less function evaluations used so far. Since the second
identify enough basins of attraction to obtain an advantage regime uses a smaller population size, it is therefore launched
over uncoordinated restarts. more often.
The second category of restart strategies is not that different The active version of BIPOP-CMA-ES (BIPOP-aCMA-ES)
from the first one since restarts also can be viewed as a has been proposed in [12].
parallelized search, but rather in the time than in space [14].
A milestone paper [6] investigated the probability of reaching D. The NIPOP-aCMA-ES
the global optimum (and the overall number of function
evaluations needed to do so) w.r.t. the population size of The NIPOP-aCMA-ES [12] is an alternative restart strategy
CMA-ES. The analysis of empirical results demonstrated that, to the IPOP-aCMA-ES, where in addition to increasing of
indeed, this probability is very sensitive to the population size population size in each restart, the initial step-size is also
and that the default population size of CMA-ES is rather too decreased by some factor kσdec . In [12], this factor is set to
small. The restart strategies described in the following sections kσdec = 1.6 such that σ value after 9 restarts (the default
are inspired by an idea of exploring CMA-ES hyper-parameters maximum number of restarts in BIPOP-aCMA-ES) roughly
such as the population size and the initial step-size. corresponds to the minimum possible initial σ = 10−2 σdef ault
 F1 in 10-D, 30-D and 50-D F2 in 10-D, 30-D and 50-D F3 in 10-D, 30-D and 50-D F4 in 10-D, 30-D and 50-D
1 1 1 1
Proportion of function-target pairs

Proportion of function-target pairs

Proportion of function-target pairs

Proportion of function-target pairs

IPOP-CMA IPOP-CMA
0.9 0.9 BIPOP-CMA 0.9 BIPOP-CMA 0.9
IPOP-aCMA IPOP-aCMA
NIPOP-aCMA NIPOP-aCMA
0.8 0.8 BIPOP-aCMA 0.8 BIPOP-aCMA 0.8
NBIPOP-aCMA NBIPOP-aCMA
0.7 0.7 0.7 0.7
0.6 0.6 0.6 0.6

0.5 0.5 0.5 0.5

0.4 0.4 0.4 0.4

0.3 0.3 0.3 0.3

IPOP-CMA IPOP-CMA
0.2 BIPOP-CMA 0.2 0.2 0.2 BIPOP-CMA
IPOP-aCMA IPOP-aCMA
NIPOP-aCMA NIPOP-aCMA
0.1 BIPOP-aCMA 0.1 0.1 0.1 BIPOP-aCMA
NBIPOP-aCMA NBIPOP-aCMA
0 0 0 0
1 2 3 4 2 3 4 2 3 4 2 3 4
10 10 10 10 10 10 10 10 10 10 10 10 10
# evaluations / dimension # evaluations / dimension # evaluations / dimension # evaluations / dimension

Fig. 2. Empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/D) for 300 function-target pairs in
10[−1..4] (100 pairs for each of dimensions 10, 30 and 50) for F1, F2, F3.

0 The NBIPOP-aCMA-ES typically outperforms IPOP-

10
aCMA-ES, BIPOP-aCMA-ES and NIPOP-aCMA-ES on the
BBOB noiseless problems [13], especially in larger dimen-
sions.
All the above described algorithms can be viewed as some
default

search algorithms in the space of hyper-parameters λ and σ.

−1 The typical patterns of these search algorithms are shown in
10 Fig. 1.
σ/σ

IV. E XPERIMENTAL VALIDATION

The experimental validation investigates the performance
of 6 CMA-ES restart strategies: IPOP-CMA-ES, BIPOP-
−2 CMA-ES, IPOP-aCMA-ES, BIPOP-aCMA-ES, NBIPOP-
10
0 1 2 3 aCMA-ES, NBIPOP-aCMA-ES. We use the source code 1
10 10 10 10
provided by the authors of [12], which is based on the original
λ/λ MATLAB code 2 of CMA-ES provided by N. Hansen. Both
default
for IPOP and BIPOP versions the default parameter settings
are used as given in [9], [4], [12]. The initial step-size σ is
Fig. 1. An illustration of λ and σ hyper-parameters distribution for 9 restarts chosen according to the given search range [−100; 100] as
of IPOP-aCMA-ES (◦), BIPOP-aCMA-ES (◦ and · for 10 runs), NIPOP-
aCMA-ES () and NBIPOP-aCMA-ES ( and many △ for λ/λdef ault = 1, 0.6 · 200 = 120.
σ/σdef ault ∈ [10−2 , 100 ]). The first run of all algorithms corresponds to
the point with λ/λdef ault = 1, σ/σdef ault = 1.
For all functions and dimensions the maximum number of
function evaluations was set to 10000n.

used for BIPOP-aCMA-ES. This strategy represents an alterna- A. Results

tive to the BIPOP-aCMA-ES in the case if the restart strategy is The results individually for each function and problem
restricted to increasing of population size. It also outperforms dimension are given according to [10] in Tables II-XIX after
IPOP-aCMA-ES and is competitive with BIPOP-aCMA-ES on the maximum number of function evaluations.
the BBOB noiseless problems [13].
To assess the performance of the algorithms we use a
procedure similar to one used in BBOB framework: for each
E. The NBIPOP-aCMA-ES objective function we define a set of function-target pairs ∆ft
in the range [10−1 , 104 ]. The lower bound of 10−1 is chosen
In NBIPOP-aCMA-ES [12] as well as in BIPOP-aCMA- because for most of multi-modal functions the objective values
ES there are two restart regimes: below 10−1 are usually difficult to achieve. Fig. 2 and 3 depict
i). Double the population size and decrease the initial step-size the empirical cumulative distribution of running time of the
by kσdec = 1.6 (NIPOP-aCMA-ES). annotated algorithm individually on all objective functions.
ii). Launch CMA-ES with default population size λdef ault and Importantly, the results for all 3 problem dimensions and 51
σ 0 = σdef
0
ault × 10
−2U[0,1]
. runs are aggregated such that if the proportion of function-
target pairs equals to 1 after a given number of function
In contrast with BIPOP-aCMA-ES, where both regimes evaluations, then all 3 · 100 = 300 function-target pairs have
have the same budget, the budget is adapted here according been solved 51 times (i.e., 15300 problems solved) by the
to the performance of the regime: the best solutions x∗A and corresponding algorithm. For some functions, e.g., F20, the y-
x∗B found by regimes A and B are used as an estimate of axis is scaled to better illustrate the difference in performances.
the quality of the regimes. Thus, kbudget = 2 times larger
computation budget is allocated for regime A if it performs 1 https://sites.google.com/site/ppsnbipop/

better than B (i.e., if x∗A is better than x∗B ), and vice versa. 2 https://www.lri.fr/∼ hansen/cmaes inmatlab.html
 F5 in 10-D, 30-D and 50-D F6 in 10-D, 30-D and 50-D F7 in 10-D, 30-D and 50-D F8 in 10-D, 30-D and 50-D
1 1 1 1
Proportion of function-target pairs

Proportion of function-target pairs

Proportion of function-target pairs

Proportion of function-target pairs

IPOP-CMA IPOP-CMA
0.9 0.9 0.9 BIPOP-CMA 0.9 BIPOP-CMA
IPOP-aCMA IPOP-aCMA
NIPOP-aCMA NIPOP-aCMA
0.8 0.8 0.8 BIPOP-aCMA 0.8 BIPOP-aCMA
NBIPOP-aCMA NBIPOP-aCMA
0.7 0.7 0.7 0.7
0.6 0.6 0.6 0.6

0.5 0.5 0.5 0.5

0.4 0.4 0.4 0.4

0.3 0.3 0.3 0.3

IPOP-CMA IPOP-CMA
0.2 BIPOP-CMA 0.2 BIPOP-CMA 0.2 0.2
IPOP-aCMA IPOP-aCMA
NIPOP-aCMA NIPOP-aCMA
0.1 BIPOP-aCMA 0.1 BIPOP-aCMA 0.1 0.1
NBIPOP-aCMA NBIPOP-aCMA
0 0 0 0
1 2 3 4 2 3 4 2 3 4 2 3 4
10 10 10 10 10 10 10 10 10 10 10 10 10
# evaluations / dimension # evaluations / dimension # evaluations / dimension # evaluations / dimension
F9 in 10-D, 30-D and 50-D F10 in 10-D, 30-D and 50-D F11 in 10-D, 30-D and 50-D F12 in 10-D, 30-D and 50-D
1 1
Proportion of function-target pairs
0.8 0.9

Proportion of function-target pairs

Proportion of function-target pairs

Proportion of function-target pairs

IPOP-CMA IPOP-CMA IPOP-CMA
BIPOP-CMA 0.9 BIPOP-CMA BIPOP-CMA
0.85 IPOP-aCMA
0.75 IPOP-aCMA IPOP-aCMA 0.9
NIPOP-aCMA NIPOP-aCMA NIPOP-aCMA
0.8 0.8 BIPOP-aCMA
BIPOP-aCMA BIPOP-aCMA
0.7 NBIPOP-aCMA NBIPOP-aCMA NBIPOP-aCMA
0.7 0.75 0.8
0.65 0.6 0.7

0.6 0.5 0.65 0.7

0.4 0.6
0.55
0.6
0.3 0.55
0.5 IPOP-CMA
0.2 BIPOP-CMA 0.5
IPOP-aCMA 0.5
0.45 NIPOP-aCMA
0.1 BIPOP-aCMA 0.45
NBIPOP-aCMA
0.4 0 0.4 0.4
2 3 4 1 2 3 4 2 3 4 2 3 4
10 10 10 10 10 10 10 10 10 10 10 10 10
# evaluations / dimension # evaluations / dimension # evaluations / dimension # evaluations / dimension
F13 in 10-D, 30-D and 50-D F14 in 10-D, 30-D and 50-D F15 in 10-D, 30-D and 50-D F16 in 10-D, 30-D and 50-D
1 0.4 0.4 1
Proportion of function-target pairs

Proportion of function-target pairs

Proportion of function-target pairs

Proportion of function-target pairs

IPOP-CMA IPOP-CMA IPOP-CMA IPOP-CMA
BIPOP-CMA BIPOP-CMA BIPOP-CMA BIPOP-CMA
0.9 IPOP-aCMA 0.35 IPOP-aCMA 0.35 IPOP-aCMA 0.95 IPOP-aCMA
NIPOP-aCMA NIPOP-aCMA NIPOP-aCMA NIPOP-aCMA
BIPOP-aCMA BIPOP-aCMA BIPOP-aCMA BIPOP-aCMA
NBIPOP-aCMA 0.3 NBIPOP-aCMA 0.3 NBIPOP-aCMA 0.9 NBIPOP-aCMA
0.8

0.25 0.25 0.85

0.7
0.2 0.2 0.8
0.6
0.15 0.15 0.75
0.5
0.1 0.1 0.7

0.4 0.05 0.05 0.65

0.3 0 0 0.6
2 3 4 2 3 4 2 3 4 2 3 4
10 10 10 10 10 10 10 10 10 10 10 10
# evaluations / dimension # evaluations / dimension # evaluations / dimension # evaluations / dimension
F17 in 10-D, 30-D and 50-D F18 in 10-D, 30-D and 50-D F19 in 10-D, 30-D and 50-D F20 in 10-D, 30-D and 50-D
0.5 0.8
Proportion of function-target pairs
Proportion of function-target pairs

Proportion of function-target pairs

Proportion of function-target pairs

IPOP-CMA IPOP-CMA IPOP-CMA IPOP-CMA
BIPOP-CMA 0.48 BIPOP-CMA BIPOP-CMA BIPOP-CMA
IPOP-aCMA IPOP-aCMA IPOP-aCMA 0.605 IPOP-aCMA
0.55 NIPOP-aCMA NIPOP-aCMA 0.75 NIPOP-aCMA NIPOP-aCMA
BIPOP-aCMA 0.46 BIPOP-aCMA BIPOP-aCMA BIPOP-aCMA
NBIPOP-aCMA NBIPOP-aCMA NBIPOP-aCMA 0.6 NBIPOP-aCMA
0.44
0.5 0.7
0.42 0.595

0.45 0.4 0.65 0.59

0.38
0.585
0.4 0.6
0.36
0.58
0.34
0.35 0.55
0.32 0.575

0.3 0.3 0.5 0.57

2 3 4 2 3 4 2 3 4 2 3 4
10 10 10 10 10 10 10 10 10 10 10 10
# evaluations / dimension # evaluations / dimension # evaluations / dimension # evaluations / dimension
F21 in 10-D, 30-D and 50-D F22 in 10-D, 30-D and 50-D F23 in 10-D, 30-D and 50-D F24 in 10-D, 30-D and 50-D
0.4 0.4
Proportion of function-target pairs

Proportion of function-target pairs

Proportion of function-target pairs

Proportion of function-target pairs

IPOP-CMA IPOP-CMA IPOP-CMA IPOP-CMA

0.38 BIPOP-CMA BIPOP-CMA BIPOP-CMA 0.38 BIPOP-CMA
IPOP-aCMA IPOP-aCMA IPOP-aCMA IPOP-aCMA
NIPOP-aCMA 0.25 NIPOP-aCMA 0.25 NIPOP-aCMA NIPOP-aCMA
0.36 BIPOP-aCMA BIPOP-aCMA BIPOP-aCMA 0.36 BIPOP-aCMA
NBIPOP-aCMA NBIPOP-aCMA NBIPOP-aCMA NBIPOP-aCMA
0.34 0.34
0.2 0.2
0.32 0.32

0.3 0.15 0.15 0.3

0.28 0.28
0.1 0.1
0.26 0.26

0.24 0.24
0.05 0.05
0.22 0.22

0.2 0 0 0.2
2 3 4 2 3 4 2 3 4 2 3 4
10 10 10 10 10 10 10 10 10 10 10 10
# evaluations / dimension # evaluations / dimension # evaluations / dimension # evaluations / dimension
F25 in 10-D, 30-D and 50-D F26 in 10-D, 30-D and 50-D F27 in 10-D, 30-D and 50-D F28 in 10-D, 30-D and 50-D
0.4 0.4
Proportion of function-target pairs
Proportion of function-target pairs

Proportion of function-target pairs

Proportion of function-target pairs

IPOP-CMA IPOP-CMA IPOP-CMA

0.38 BIPOP-CMA 0.38 BIPOP-CMA 0.28 BIPOP-CMA 0.29
IPOP-aCMA IPOP-aCMA IPOP-aCMA
NIPOP-aCMA NIPOP-aCMA NIPOP-aCMA
0.36 BIPOP-aCMA 0.36 BIPOP-aCMA 0.26 BIPOP-aCMA 0.28
NBIPOP-aCMA NBIPOP-aCMA NBIPOP-aCMA
0.34 0.34 0.24 0.27

0.32 0.32 0.22 0.26

0.3 0.3 0.2 0.25

0.28 0.28 0.18 0.24

0.26 0.26 0.16 0.23

IPOP-CMA
0.24 0.24 0.14 0.22 BIPOP-CMA
IPOP-aCMA
NIPOP-aCMA
0.22 0.22 0.12 0.21 BIPOP-aCMA
NBIPOP-aCMA
0.2 0.2 0.1 0.2
2 3 4 2 3 4 2 3 4 2 3 4
10 10 10 10 10 10 10 10 10 10 10 10
# evaluations / dimension # evaluations / dimension # evaluations / dimension # evaluations / dimension

Fig. 3. Continuation of Fig. 2.

 TABLE I. C OMPUTATIONAL COMPLEXITY OF ALL 6 ALGORITHMS GIVEN FOR 10-, 30- AND 50- DIMENSIONAL S CHWEFEL’ S FUNCTION (F14).
T0 T1 (T2 - T1) / T0 for IPOP-CMA IPOP-aCMA BIPOP-CMA BIPOP-aCMA NIPOP-aCMA NBIPOP-aCMA
D=10 0.277 1.778 22.64 25.45 45.97 45.08 59.26 62.39
D=30 0.277 2.929 38.20 45.66 56.20 64.96 63.64 68.11
D=50 0.277 4.159 57.29 69.41 84.06 85.43 102.38 103.79

All 28 functions in 10-D, 30-D and 50-D All 28 functions in 10-D, 30-D and 50-D
Active covariance matrix update. The active versions of

Proportion of function-target pairs

Proportion of function-target pairs
1 0.64
IPOP-CMA
CMA-ES clearly outperform the original ones on unimodal ill- 0.9
BIPOP-CMA 0.62
0.8
conditioned functions F2, F3, F4. A substantial improvement 0.7
IPOP-aCMA
NIPOP-aCMA 0.6

is also observed on F5, F6, F7. The only function, where the 0.6 BIPOP-aCMA
0.58
NBIPOP-aCMA
original versions seem to perform better is F21 composition 0.5
0.56
0.4
function of functions F1, F3, F4, F5 and F6, i.e., on which the 0.3 0.54
IPOP-CMA
active versions actually perform better. This is an unexpected 0.2
0.52
BIPOP-CMA
IPOP-aCMA
NIPOP-aCMA
result and requires further analysis. 0.1

0 0.5
BIPOP-aCMA
NBIPOP-aCMA
1 2 3 4 3 4
10 10 10 10 10 10
# evaluations / dimension # evaluations / dimension
BIPOP vs IPOP. BIPOP-based algorithms outperform
IPOP-based algorithms on F9, F14, F16, F20, F21, F24, F25,
F26, F27, F28, and are outperformed by the latter on F11, Fig. 4. Empirical cumulative distribution of all function-target pairs solved
on all functions, dimensions and runs (in overall, 428400 pairs).
F12, F13, F14 and F15. While in some cases the difference is
minor, in overall, BIPOP-based algorithms perform better on
composition functions.
[2] A. Auger and N. Hansen. A Restart CMA Evolution Strategy With
NBIPOP and NIPOP vs BIPOP and IPOP. The alter- Increasing Population Size. In IEEE Congress on Evolutionary Com-
putation, pages 1769–1776. IEEE Press, 2005.
native restart strategies outperform the original ones on F9,
[3] N. Hansen. Compilation of results on the 2005 CEC benchmark function
F12, F16, F20, F24, F25, F26, F27, F28, and demonstrate a set. Online, May, 2006.
comparable performance on other functions. [4] N. Hansen. Benchmarking a BI-population CMA-ES on the BBOB-
2009 function testbed. In GECCO Companion, pages 2389–2396, 2009.
Computational Complexity. The results of experimental
[5] N. Hansen, A. Auger, S. Finck, and R. Ros. Real-Parameter Black-
runs on F14 Schwefel’s function are given in Table I according Box Optimization Benchmarking 2010: Experimental Setup. Technical
to [10]. The restart strategies where smaller population sizes Report RR-7215, INRIA, 2010.
are used (e.g., NBIPOP-aCMA-ES) spend more time on inter- [6] N. Hansen and S. Kern. Evaluating the CMA Evolution Strategy on
nal computations per function evaluation, and are typically up Multimodal Test Functions. In PPSN’04, pages 282–291, 2004.
to 2 times slower in terms of time than IPOP-CMA-ES. [7] N. Hansen, S. Müller, and P. Koumoutsakos. Reducing the time
complexity of the derandomized evolution strategy with covariance
matrix adaptation (CMA-ES). Evolutionary Computation, 11(1):1–18,
V. C ONCLUSION AND P ERSPECTIVES 2003.
[8] N. Hansen and A. Ostermeier. Completely Derandomized Self-
In this paper, we have compared the original and recently Adaptation in Evolution Strategies. Evol. Comput., 9(2):159–195, June
proposed restart strategies for CMA-ES on the CEC 2013 test 2001.
suite. The aggregated results depicted in Fig. 4 demonstrate [9] N. Hansen and R. Ros. Benchmarking a weighted negative covariance
a slightly better performance of the NBIPOP-aCMA-ES and matrix update on the BBOB-2010 noiseless testbed. In GECCO ’10:
NIPOP-aCMA-ES. A possible reason is that a smaller initial Proceedings of the 12th annual conference comp on Genetic and
evolutionary computation, pages 1673–1680, New York, NY, USA,
step-size is especially useful on composition functions where 2010. ACM.
the basins of attractions are relatively small. The results
[10] P. N. S. J. J. Liang, B-Y. Qu and A. G. Hernandez-Diaz. Problem Def-
also confirm some superiority of the active covariance matrix initions and Evaluation Criteria for the CEC 2013 Special Session and
update. Competition on Real-Parameter Optimization. Technical report, Com-
putational Intelligence Laboratory, Zhengzhou University, Zhengzhou
The main limitation of all tested approaches is that the China and Technical Report, Nanyang Technological University, 2013.
search in the hyper-parameter space of the population size and [11] G. A. Jastrebski and D. V. Arnold. Improving Evolution Strategies
initial step-size seems to be inefficient and some potentially through Active Covariance Matrix Adaptation. In IEEE Congress on
useful information from the restarts (e.g., the location of the Evolutionary Computation, pages 2814–2821, 2006.
best found solution) is not used. Another important limitation [12] I. Loshchilov, M. Schoenauer, and M. Sebag. Alternative Restart
inherited from the CMA-ES is a lack of functionality which Strategies for CMA-ES. In V. C. et al., editor, Parallel Problem Solving
from Nature (PPSN XII), LNCS, pages 296–305. Springer, 2012.
would allow to detect and exploit the separability of the
[13] I. Loshchilov, M. Schoenauer, and M. Sebag. Black-box Optimiza-
objective function. Thus, the algorithms which specifically tion Benchmarking of NIPOP-aCMA-ES and NBIPOP-aCMA-ES on
focus on separable and partially-separable functions will very the BBOB-2012 Noiseless Testbed. In T. Soule and J. H. Moore,
likely outperform the CMA-ES and its restarts strategies. The editors, Genetic and Evolutionary Computation Conference (GECCO
above-described issues need to be addressed in future work. Companion), pages 269–276. ACM Press, July 2012.
[14] M. Preuss. Niching the cma-es via nearest-better clustering. In
Proceedings of the 12th annual conference companion on Genetic and
R EFERENCES evolutionary computation, pages 1711–1718. ACM, 2010.
[15] O. M. Shir, M. Emmerich, and T. Bäck. Adaptive niche radii and
[1] A. Auger, S. Finck, N. Hansen, and R. Ros. BBOB 2010: Comparison
niche shapes approaches for niching with the cma-es. Evolutionary
Tables of All Algorithms on All Noiseless Functions. Technical Report
Computation, 18(1):97–126, 2010.
RR-7215, INRIA, 2010.
 TABLE II. IPOP-CMA-ES IN 10-D TABLE V. IPOP- ACMA-ES IN 10-D
Func. Best Worst Median Mean Std Func. Best Worst Median Mean Std
1 0.000 0.000 0.000 0.000 0.000 1 0.000 0.000 0.000 0.000 0.000
2 0.000 0.000 0.000 0.000 0.000 2 0.000 0.000 0.000 0.000 0.000
3 0.000 0.000 0.000 0.000 0.000 3 0.000 0.000 0.000 0.000 0.000
4 0.000 0.000 0.000 0.000 0.000 4 0.000 0.000 0.000 0.000 0.000
5 0.000 0.000 0.000 0.000 0.000 5 0.000 0.000 0.000 0.000 0.000
6 0.000 0.000 0.000 0.000 0.000 6 0.000 0.000 0.000 0.000 0.000
7 0.000 0.000 0.000 0.000 0.000 7 0.000 0.000 0.000 0.000 0.000
8 20.187 20.471 20.352 20.342 0.070 8 20.156 20.474 20.359 20.353 0.075
9 0.000 3.121 0.199 0.582 0.721 9 0.000 3.000 0.000 0.504 0.720
10 0.000 0.000 0.000 0.000 0.000 10 0.000 0.000 0.000 0.000 0.000
11 0.000 1.990 0.000 0.332 0.551 11 0.000 1.990 0.000 0.351 0.520
12 0.000 0.995 0.000 0.098 0.299 12 0.000 0.995 0.000 0.078 0.270
13 0.000 1.990 0.000 0.313 0.508 13 0.000 1.990 0.000 0.254 0.482
14 3.602 167.830 18.535 26.681 26.673 14 0.187 65.170 21.825 23.576 14.898
15 0.312 58.398 18.535 21.955 15.243 15 0.250 125.390 18.472 24.250 22.438
16 0.905 1.542 1.124 1.152 0.136 16 0.526 1.598 1.222 1.169 0.228
17 10.382 12.430 10.984 11.068 0.430 17 10.262 12.014 10.784 10.846 0.317
18 10.258 11.953 10.951 10.974 0.414 18 10.227 13.002 11.021 11.076 0.548
19 0.440 0.919 0.646 0.646 0.115 19 0.458 0.873 0.658 0.655 0.097
20 1.547 4.019 3.019 2.763 0.592 20 1.512 4.035 2.604 2.719 0.609
21 100.000 400.190 400.190 374.677 71.735 21 200.000 400.190 400.190 380.564 60.122
22 9.902 313.190 56.512 73.252 49.446 22 16.683 259.830 58.989 70.667 42.842
23 14.224 318.930 59.212 86.163 66.062 23 16.404 243.670 59.730 80.393 49.941
24 200.000 225.200 208.450 209.465 7.022 24 200.000 225.180 205.850 209.341 8.700
25 200.000 224.120 203.610 205.517 6.709 25 200.000 222.810 203.370 203.944 4.543
26 106.960 218.160 205.810 204.201 15.094 26 108.950 220.010 202.730 199.885 20.951
27 319.700 560.530 446.630 454.562 79.234 27 304.010 559.070 462.610 450.354 89.518
28 300.000 300.000 300.000 300.000 0.000 28 300.000 300.000 300.000 300.000 0.000

TABLE III. BIPOP-CMA-ES IN 10-D TABLE VI. BIPOP- ACMA-ES IN 10-D

Func. Best Worst Median Mean Std Func. Best Worst Median Mean Std
1 0.000 0.000 0.000 0.000 0.000 1 0.000 0.000 0.000 0.000 0.000
2 0.000 0.000 0.000 0.000 0.000 2 0.000 0.000 0.000 0.000 0.000
3 0.000 0.000 0.000 0.000 0.000 3 0.000 0.000 0.000 0.000 0.000
4 0.000 0.000 0.000 0.000 0.000 4 0.000 0.000 0.000 0.000 0.000
5 0.000 0.000 0.000 0.000 0.000 5 0.000 0.000 0.000 0.000 0.000
6 0.000 0.000 0.000 0.000 0.000 6 0.000 0.000 0.000 0.000 0.000
7 0.000 0.002 0.000 0.000 0.000 7 0.000 0.000 0.000 0.000 0.000
8 20.185 20.517 20.359 20.339 0.082 8 20.000 20.468 20.355 20.345 0.082
9 0.000 2.638 0.104 0.554 0.684 9 0.000 2.635 0.263 0.522 0.644
10 0.000 0.000 0.000 0.000 0.000 10 0.000 0.000 0.000 0.000 0.000
11 0.000 2.985 0.995 0.936 0.806 11 0.000 2.985 0.000 0.587 0.721
12 0.000 1.990 0.995 0.585 0.603 12 0.000 2.985 0.013 0.644 0.766
13 0.000 3.651 0.995 0.944 0.838 13 0.000 2.396 0.995 0.733 0.688
14 3.665 359.080 33.529 54.762 69.842 14 0.312 271.630 27.012 40.118 49.940
15 3.665 258.190 39.989 52.837 57.800 15 0.125 317.210 28.596 47.743 65.849
16 0.000 1.593 0.090 0.305 0.457 16 0.000 1.386 0.051 0.174 0.331
17 10.550 13.109 11.500 11.551 0.559 17 4.490 13.014 11.466 11.409 1.101
18 6.299 14.018 11.706 11.565 1.159 18 7.703 13.686 11.340 11.336 0.974
19 0.378 0.951 0.591 0.604 0.120 19 0.339 0.900 0.575 0.589 0.116
20 0.828 3.559 2.621 2.582 0.552 20 1.001 3.547 2.551 2.538 0.583
21 100.000 400.190 300.000 284.407 122.356 21 100.000 400.190 400.190 315.805 113.884
22 38.725 339.800 78.204 99.325 65.612 22 29.353 233.900 66.714 82.324 45.521
23 34.232 302.550 110.520 117.411 65.091 23 25.760 367.530 85.528 96.398 55.902
24 100.000 207.740 110.700 130.304 38.730 24 102.720 212.520 108.570 127.376 37.218
25 109.600 207.740 202.260 192.516 28.442 25 103.460 206.820 201.300 184.849 36.093
26 100.000 200.020 107.960 118.570 24.895 26 73.031 200.020 107.420 121.129 32.253
27 186.920 447.100 354.120 346.434 44.001 27 300.000 400.000 337.710 340.883 29.434
28 100.000 300.000 300.000 280.392 60.065 28 100.000 300.000 300.000 260.784 80.196

TABLE IV. NIPOP- ACMA-ES IN 10-D TABLE VII. NBIPOP- ACMA-ES IN 10-D
Func. Best Worst Median Mean Std Func. Best Worst Median Mean Std
1 0.000 0.000 0.000 0.000 0.000 1 0.000 0.000 0.000 0.000 0.000
2 0.000 0.000 0.000 0.000 0.000 2 0.000 0.000 0.000 0.000 0.000
3 0.000 0.000 0.000 0.000 0.000 3 0.000 0.000 0.000 0.000 0.000
4 0.000 0.000 0.000 0.000 0.000 4 0.000 0.000 0.000 0.000 0.000
5 0.000 0.000 0.000 0.000 0.000 5 0.000 0.000 0.000 0.000 0.000
6 0.000 0.000 0.000 0.000 0.000 6 0.000 0.000 0.000 0.000 0.000
7 0.000 0.000 0.000 0.000 0.000 7 0.000 0.000 0.000 0.000 0.000
8 20.189 20.470 20.357 20.353 0.064 8 20.000 20.520 20.353 20.339 0.090
9 0.000 1.782 0.000 0.254 0.457 9 0.000 1.503 0.000 0.232 0.440
10 0.000 0.000 0.000 0.000 0.000 10 0.000 0.000 0.000 0.000 0.000
11 0.000 0.995 0.000 0.267 0.441 11 0.000 1.990 0.000 0.364 0.506
12 0.000 0.995 0.000 0.078 0.270 12 0.000 2.985 0.000 0.238 0.542
13 0.000 1.026 0.000 0.254 0.439 13 0.000 2.836 0.001 0.484 0.676
14 3.602 336.200 142.300 140.243 97.669 14 6.892 356.120 76.816 114.997 92.377
15 3.727 332.870 109.320 129.231 96.480 15 18.597 659.330 151.310 158.161 117.317
16 0.000 1.529 1.121 1.055 0.314 16 0.011 1.369 0.054 0.120 0.263
17 10.310 11.968 10.980 11.006 0.382 17 10.333 12.390 11.369 11.334 0.545
18 10.346 11.439 10.824 10.849 0.257 18 7.956 16.995 11.071 11.288 1.276
19 0.059 0.953 0.679 0.658 0.159 19 0.010 0.876 0.518 0.525 0.139
20 1.563 3.606 2.479 2.417 0.455 20 1.198 3.795 2.761 2.726 0.650
21 100.000 400.190 400.190 350.538 87.857 21 100.000 200.000 200.000 152.941 50.410
22 21.790 375.240 98.857 146.533 110.690 22 36.355 451.250 141.820 175.131 114.655
23 18.237 506.830 180.540 196.642 117.469 23 24.453 512.870 129.180 174.230 122.831
24 100.000 206.400 108.070 149.495 49.703 24 100.000 202.240 107.870 119.885 32.220
25 100.000 207.430 200.000 196.967 19.858 25 100.000 205.770 200.060 176.972 39.918
26 49.144 200.020 100.990 123.776 43.711 26 100.000 246.650 105.970 111.035 24.986
27 300.000 547.980 325.980 350.654 67.183 27 172.980 360.600 311.620 316.684 29.556
28 109.340 300.000 300.000 292.859 35.738 28 100.000 300.000 300.000 249.020 88.029
 TABLE VIII. IPOP-CMA-ES IN 30-D TABLE XI. IPOP- ACMA-ES IN 30-D
Func. Best Worst Median Mean Std Func. Best Worst Median Mean Std
1 0.000 0.000 0.000 0.000 0.000 1 0.000 0.000 0.000 0.000 0.000
2 0.000 0.000 0.000 0.000 0.000 2 0.000 0.000 0.000 0.000 0.000
3 0.000 64.878 0.000 1.732 9.296 3 0.000 0.000 0.000 0.000 0.000
4 0.000 0.000 0.000 0.000 0.000 4 0.000 0.000 0.000 0.000 0.000
5 0.000 0.000 0.000 0.000 0.000 5 0.000 0.000 0.000 0.000 0.000
6 0.000 0.000 0.000 0.000 0.000 6 0.000 0.000 0.000 0.000 0.000
7 0.000 55.451 2.843 16.835 19.624 7 0.000 120.400 0.118 8.854 22.129
8 20.765 21.006 20.956 20.931 0.059 8 20.834 21.023 20.950 20.944 0.045
9 1.213 41.165 37.269 24.463 16.090 9 1.500 41.265 38.879 27.216 16.452
10 0.000 0.000 0.000 0.000 0.000 10 0.000 0.000 0.000 0.000 0.000
11 0.000 6.965 2.096 2.290 1.452 11 0.000 4.396 0.997 1.174 1.092
12 0.071 5.970 1.990 1.853 1.164 12 0.000 2.985 0.327 0.704 0.833
13 0.000 12.135 1.990 2.414 2.266 13 0.000 4.120 0.995 1.117 1.225
14 60.072 1277.400 185.840 287.008 272.130 14 54.523 1195.800 225.710 271.876 220.390
15 29.083 1055.100 344.610 337.708 241.796 15 60.736 1203.100 298.050 336.269 257.532
16 1.914 3.191 2.539 2.528 0.273 16 1.968 2.987 2.550 2.529 0.252
17 32.431 39.212 33.577 34.073 1.355 17 31.726 39.384 33.396 33.764 1.442
18 32.044 181.730 40.312 81.650 61.282 18 31.678 176.400 36.888 70.653 56.460
19 1.177 3.203 2.527 2.484 0.402 19 1.207 3.398 2.532 2.466 0.449
20 13.737 15.000 14.585 14.603 0.349 20 13.716 15.000 14.537 14.596 0.329
21 200.000 300.000 300.000 254.902 50.254 21 200.000 300.000 300.000 254.902 50.254
22 120.550 1483.100 420.510 502.379 309.407 22 99.474 1249.100 401.690 477.206 293.492
23 91.710 1869.600 517.520 576.071 350.245 23 119.140 1195.500 444.300 492.216 292.454
24 219.630 306.160 300.270 285.725 30.214 24 218.430 304.830 297.660 276.327 34.673
25 205.270 302.720 298.280 286.874 28.505 25 211.300 303.330 299.040 289.376 26.732
26 200.000 403.450 323.380 314.510 81.420 26 200.000 406.060 350.090 329.264 77.967
27 483.550 1326.300 1281.600 1141.729 290.392 27 380.180 1334.000 1272.900 1079.047 344.867
28 300.000 300.000 300.000 300.000 0.000 28 300.000 300.000 300.000 300.000 0.000

TABLE IX. BIPOP-CMA-ES IN 30-D TABLE XII. BIPOP- ACMA-ES IN 30-D

Func. Best Worst Median Mean Std Func. Best Worst Median Mean Std
1 0.000 0.000 0.000 0.000 0.000 1 0.000 0.000 0.000 0.000 0.000
2 0.000 0.000 0.000 0.000 0.000 2 0.000 0.000 0.000 0.000 0.000
3 0.000 3.638 0.000 0.082 0.509 3 0.000 0.000 0.000 0.000 0.000
4 0.000 0.000 0.000 0.000 0.000 4 0.000 0.000 0.000 0.000 0.000
5 0.000 0.000 0.000 0.000 0.000 5 0.000 0.000 0.000 0.000 0.000
6 0.000 0.000 0.000 0.000 0.000 6 0.000 0.000 0.000 0.000 0.000
7 0.000 46.223 1.156 9.426 13.302 7 0.000 67.819 0.027 2.727 10.937
8 20.799 21.017 20.936 20.935 0.051 8 20.797 21.040 20.950 20.939 0.055
9 0.231 12.008 6.419 6.489 2.380 9 1.305 9.197 5.168 5.214 1.949
10 0.000 0.000 0.000 0.000 0.000 10 0.000 0.000 0.000 0.000 0.000
11 0.000 6.965 2.985 3.082 1.519 11 0.000 5.970 2.985 3.142 1.426
12 0.000 5.970 1.990 2.410 1.430 12 0.000 5.970 2.985 2.810 1.575
13 0.000 6.853 1.990 2.391 1.474 13 0.995 6.495 1.990 2.646 1.437
14 51.251 4146.200 514.060 669.207 697.842 14 113.210 1461.700 429.630 495.496 277.647
15 56.319 2228.500 492.590 609.778 450.755 15 46.739 2217.100 465.140 544.527 416.219
16 0.002 2.826 0.042 0.775 1.143 16 0.000 2.914 0.060 0.940 1.203
17 33.612 40.999 36.002 36.343 1.770 17 32.490 38.849 35.802 35.702 1.712
18 32.257 172.370 41.362 54.364 33.956 18 32.241 178.520 37.687 58.053 43.411
19 1.265 3.309 2.497 2.395 0.418 19 1.350 3.275 2.291 2.285 0.323
20 12.392 15.000 14.344 14.237 0.636 20 11.551 15.000 14.155 14.015 0.770
21 100.000 300.000 200.000 200.000 28.284 21 200.000 300.000 200.000 213.725 34.754
22 113.670 2906.700 705.840 838.581 577.102 22 118.070 1347.500 636.750 662.775 301.895
23 189.780 2776.000 664.470 716.942 452.235 23 190.510 1403.400 649.110 702.919 312.690
24 117.230 300.540 161.900 180.398 50.160 24 120.200 238.170 213.840 186.292 41.916
25 214.860 302.750 224.960 231.191 21.156 25 210.660 301.460 225.560 226.475 13.732
26 111.940 205.460 148.750 163.826 32.299 26 118.910 203.450 154.730 168.368 30.175
27 378.550 660.750 513.360 503.581 71.345 27 400.000 909.700 513.390 516.864 93.743
28 100.000 300.000 300.000 292.157 39.208 28 100.000 300.000 300.000 284.314 54.305

TABLE X. NIPOP- ACMA-ES IN 30-D TABLE XIII. NBIPOP- ACMA-ES IN 30-D

Func. Best Worst Median Mean Std Func. Best Worst Median Mean Std
1 0.000 0.000 0.000 0.000 0.000 1 0.000 0.000 0.000 0.000 0.000
2 0.000 0.000 0.000 0.000 0.000 2 0.000 0.000 0.000 0.000 0.000
3 0.000 0.000 0.000 0.000 0.000 3 0.000 0.003 0.000 0.000 0.000
4 0.000 0.000 0.000 0.000 0.000 4 0.000 0.000 0.000 0.000 0.000
5 0.000 0.000 0.000 0.000 0.000 5 0.000 0.000 0.000 0.000 0.000
6 0.000 0.000 0.000 0.000 0.000 6 0.000 0.000 0.000 0.000 0.000
7 0.000 40.445 0.044 4.055 9.172 7 0.000 33.998 0.057 2.313 6.049
8 20.755 21.057 20.928 20.921 0.062 8 20.797 21.013 20.946 20.942 0.048
9 0.000 5.486 2.927 2.823 1.228 9 0.401 7.630 2.768 3.300 1.383
10 0.000 0.000 0.000 0.000 0.000 10 0.000 0.000 0.000 0.000 0.000
11 0.000 3.980 0.995 1.032 1.040 11 0.000 6.965 2.985 3.043 1.413
12 0.000 2.985 0.031 0.656 0.825 12 0.000 5.970 2.985 2.907 1.376
13 0.000 3.049 0.995 0.931 0.928 13 0.000 7.963 2.985 2.778 1.453
14 201.210 1328.000 712.620 716.645 244.372 14 278.650 2221.100 739.970 810.125 360.294
15 90.572 1334.200 668.950 670.256 280.430 15 282.650 1674.500 744.850 765.493 294.867
16 1.508 3.092 2.549 2.484 0.314 16 0.014 2.784 0.041 0.440 0.926
17 31.754 39.261 34.100 34.248 1.716 17 32.451 40.384 33.593 34.419 1.869
18 31.905 171.940 35.104 53.961 44.520 18 32.191 186.960 39.560 62.289 45.591
19 1.130 3.324 2.482 2.408 0.465 19 1.103 2.866 2.233 2.228 0.341
20 10.012 15.000 13.529 13.365 1.260 20 11.117 13.636 13.131 12.940 0.598
21 200.000 300.000 200.000 241.176 49.705 21 100.000 200.000 200.000 192.157 27.152
22 116.060 2326.200 530.650 572.759 341.016 22 129.850 2390.800 734.260 838.392 459.988
23 82.274 1546.600 632.300 667.436 326.261 23 188.220 1835.500 666.730 667.086 289.554
24 220.740 306.150 298.520 290.623 22.816 24 122.800 230.390 155.520 161.757 30.045
25 207.770 303.430 298.610 278.962 35.199 25 154.010 229.260 221.920 219.984 11.094
26 125.110 361.420 249.560 251.038 57.982 26 128.850 291.790 146.760 158.223 29.999
27 320.160 1329.300 639.970 870.294 422.247 27 350.550 606.710 471.890 468.925 73.770
28 300.000 300.000 300.000 300.000 0.000 28 100.000 300.000 300.000 268.627 73.458
 TABLE XIV. IPOP-CMA-ES IN 50-D TABLE XVII. IPOP- ACMA-ES IN 50-D
Func. Best Worst Median Mean Std Func. Best Worst Median Mean Std
1 0.000 0.000 0.000 0.000 0.000 1 0.000 0.000 0.000 0.000 0.000
2 0.000 0.000 0.000 0.000 0.000 2 0.000 0.000 0.000 0.000 0.000
3 0.000 173190.000 1.988 6506.587 27617.764 3 0.000 151.320 0.002 5.446 22.085
4 0.000 0.000 0.000 0.000 0.000 4 0.000 0.000 0.000 0.000 0.000
5 0.000 0.000 0.000 0.000 0.000 5 0.000 0.000 0.000 0.000 0.000
6 0.000 0.000 0.000 0.000 0.000 6 0.000 0.000 0.000 0.000 0.000
7 0.048 195.500 11.322 22.928 39.616 7 0.074 334.040 3.701 14.791 47.056
8 20.841 21.178 21.134 21.123 0.052 8 21.012 21.184 21.124 21.122 0.031
9 3.167 75.313 71.931 59.601 25.270 9 4.115 75.639 72.368 51.913 29.071
10 0.000 0.000 0.000 0.000 0.000 10 0.000 0.000 0.000 0.000 0.000
11 0.001 21.889 6.965 8.506 5.594 11 0.000 21.889 7.960 8.545 4.595
12 0.000 20.894 5.970 6.117 4.359 12 0.000 11.940 1.991 3.535 3.597
13 0.000 93.992 5.573 10.804 16.777 13 0.000 66.642 3.980 6.066 10.345
14 150.500 13329.000 780.840 1625.565 2921.834 14 155.470 11724.000 632.460 1138.034 2040.146
15 102.580 12866.000 801.890 1357.597 2387.453 15 115.590 11956.000 639.630 1067.475 1732.580
16 2.717 3.776 3.336 3.315 0.277 16 2.593 3.883 3.392 3.356 0.271
17 53.291 79.900 57.218 58.214 4.370 17 55.082 76.011 57.460 58.697 4.181
18 54.106 360.530 328.620 228.534 135.806 18 54.672 360.110 73.931 164.545 129.500
19 2.022 5.935 4.518 4.413 0.789 19 2.439 6.125 4.507 4.472 0.553
20 25.000 25.000 25.000 25.000 0.000 20 25.000 25.000 25.000 25.000 0.000
21 200.000 1122.200 200.000 516.822 408.086 21 200.000 1122.200 836.440 645.944 407.072
22 152.770 13113.000 1042.700 1825.791 2860.190 22 238.960 12547.000 832.130 1406.937 1989.910
23 164.350 13349.000 1133.400 2986.475 4190.174 23 225.600 12190.000 895.690 1201.597 1669.776
24 244.560 391.680 385.340 375.023 33.360 24 240.740 389.950 385.920 380.249 27.161
25 239.930 388.480 383.120 373.787 33.452 25 218.540 387.800 382.090 366.253 45.626
26 200.000 491.740 481.770 382.372 129.421 26 200.000 492.920 360.670 370.441 125.046
27 699.690 2200.700 2130.500 1936.220 454.537 27 664.700 2181.700 2128.900 2048.754 315.653
28 400.000 3400.600 400.000 1034.771 1222.556 28 400.000 3345.400 400.000 856.214 1067.395

TABLE XV. BIPOP-CMA-ES IN 50-D TABLE XVIII. BIPOP- ACMA-ES IN 50-D

Func. Best Worst Median Mean Std Func. Best Worst Median Mean Std
1 0.000 0.000 0.000 0.000 0.000 1 0.000 0.000 0.000 0.000 0.000
2 0.000 0.000 0.000 0.000 0.000 2 0.000 0.000 0.000 0.000 0.000
3 0.000 226180.000 0.044 8858.325 37582.750 3 0.000 1030.200 0.012 28.663 150.674
4 0.000 0.000 0.000 0.000 0.000 4 0.000 0.000 0.000 0.000 0.000
5 0.000 0.000 0.000 0.000 0.000 5 0.000 0.000 0.000 0.000 0.000
6 0.000 0.000 0.000 0.000 0.000 6 0.000 0.000 0.000 0.000 0.000
7 0.095 358.290 11.802 45.355 79.408 7 0.084 212.400 8.221 20.688 36.151
8 20.996 21.190 21.139 21.128 0.039 8 20.996 21.185 21.127 21.127 0.034
9 4.838 25.721 13.283 13.711 4.507 9 5.075 24.628 12.349 12.505 3.778
10 0.000 0.000 0.000 0.000 0.000 10 0.000 0.000 0.000 0.000 0.000
11 1.054 21.889 7.960 9.275 5.319 11 0.000 14.924 6.965 7.323 3.542
12 0.995 17.909 6.965 8.116 3.704 12 2.985 17.909 6.965 6.773 2.896
13 0.000 33.058 5.970 7.636 6.106 13 1.026 26.939 6.064 7.131 4.187
14 217.950 5743.500 1354.700 1697.035 1445.075 14 220.000 4419.200 1129.500 1253.627 857.811
15 159.450 4883.900 930.790 1305.709 1101.514 15 195.040 3854.700 1218.600 1399.566 923.669
16 0.003 3.801 0.069 1.562 1.662 16 0.005 3.916 0.418 1.673 1.696
17 54.361 94.885 61.199 61.619 5.778 17 54.955 68.581 58.751 60.136 3.608
18 54.870 359.620 74.722 138.883 117.742 18 54.805 360.920 81.181 158.085 123.762
19 3.099 5.892 4.264 4.347 0.560 19 2.587 5.346 4.294 4.240 0.572
20 22.118 25.000 23.573 23.527 0.642 20 22.080 25.000 23.663 23.752 0.626
21 200.000 836.440 200.000 224.958 124.767 21 200.000 1122.200 200.000 267.999 211.417
22 269.680 12181.000 1262.300 1765.215 1731.078 22 283.460 4940.200 1402.500 1628.679 1025.878
23 202.850 6981.000 1329.100 1922.464 1655.305 23 102.160 4956.800 1330.600 1752.374 1054.910
24 169.060 385.270 248.760 246.799 32.156 24 142.300 387.080 248.860 245.757 32.071
25 229.770 386.390 253.630 259.932 32.909 25 223.190 383.050 251.220 256.735 33.322
26 128.880 205.240 178.600 177.402 23.863 26 117.910 204.170 200.000 184.264 24.067
27 400.060 929.000 743.580 736.518 101.757 27 400.000 988.050 735.360 716.665 141.348
28 400.000 400.000 400.000 400.000 0.000 28 400.000 400.000 400.000 400.000 0.000

TABLE XVI. NIPOP- ACMA-ES IN 50-D TABLE XIX. NBIPOP- ACMA-ES IN 50-D
Func. Best Worst Median Mean Std Func. Best Worst Median Mean Std
1 0.000 0.000 0.000 0.000 0.000 1 0.000 0.000 0.000 0.000 0.000
2 0.000 0.000 0.000 0.000 0.000 2 0.000 0.000 0.000 0.000 0.000
3 0.000 832.750 0.000 18.773 116.751 3 0.000 866.850 0.000 18.166 121.348
4 0.000 0.000 0.000 0.000 0.000 4 0.000 0.000 0.000 0.000 0.000
5 0.000 0.000 0.000 0.000 0.000 5 0.000 0.000 0.000 0.000 0.000
6 0.000 0.000 0.000 0.000 0.000 6 0.000 0.000 0.000 0.000 0.000
7 0.035 207.120 1.910 11.714 37.728 7 0.039 19.629 3.646 4.971 5.724
8 20.963 21.183 21.117 21.111 0.044 8 20.969 21.186 21.131 21.119 0.045
9 2.038 11.924 7.054 6.882 1.819 9 2.022 12.466 7.058 7.220 2.286
10 0.000 0.000 0.000 0.000 0.000 10 0.000 0.000 0.000 0.000 0.000
11 0.000 5.970 1.990 1.991 1.305 11 0.998 11.940 5.970 5.505 2.959
12 0.000 5.970 0.995 1.366 1.289 12 0.007 9.950 4.975 5.371 2.540
13 0.000 3.982 0.995 1.481 1.009 13 0.995 29.051 6.965 7.595 5.468
14 450.280 2700.500 1204.200 1257.493 442.403 14 381.240 3312.300 1335.000 1375.403 566.544
15 365.230 2537.000 1331.800 1352.059 504.095 15 333.750 3413.000 1495.500 1553.688 548.191
16 2.580 4.025 3.386 3.370 0.296 16 0.018 3.864 0.044 0.878 1.441
17 53.824 67.491 57.287 57.737 2.373 17 54.741 66.344 56.737 57.369 2.726
18 55.423 356.310 106.450 193.660 134.807 18 55.241 352.130 104.030 133.647 100.310
19 2.571 5.272 4.544 4.467 0.521 19 3.040 5.370 4.422 4.458 0.593
20 19.790 25.000 22.969 22.985 1.370 20 18.746 24.587 22.738 22.547 1.175
21 200.000 1122.200 200.000 365.287 325.247 21 100.000 200.000 200.000 198.039 14.003
22 218.400 2165.900 895.340 1017.509 466.984 22 188.710 3858.600 1336.300 1448.353 601.295
23 253.900 3175.900 938.890 1186.484 690.262 23 477.380 4233.400 1492.400 1712.552 809.352
24 237.470 392.150 382.040 370.404 37.743 24 194.580 265.320 244.990 239.643 20.380
25 215.640 387.580 382.570 365.032 48.659 25 233.430 257.410 248.680 247.570 5.059
26 200.000 493.560 311.420 288.263 98.170 26 113.930 223.360 200.000 196.091 14.340
27 520.440 2183.300 2119.100 1898.799 520.000 27 390.610 878.060 777.220 727.829 144.098
28 400.000 3332.600 400.000 571.590 693.198 28 400.000 400.000 400.000 400.000 0.000