468 IEEE TRANSACTIONS ON ROBOTICS, VOL. 33, NO.

2, APRIL 2017

Multiobjective Optimization Based on Expensive

Robotic Experiments under Heteroscedastic Noise
Ryo Ariizumi, Member, IEEE, Matthew Tesch, Member, IEEE, Kenta Kato, Howie Choset, Member, IEEE,
and Fumitoshi Matsuno, Member, IEEE

Abstract—In many engineering problems, including those re-

lated to robotics, optimization of the control policy for multiple
conflicting criteria is required. However, this can be very challeng-
ing because of the existence of noise, which may be input dependent
or heteroscedastic, and restrictions regarding the number of eval-
uations owing to the costliness of the experiments in terms of time
and/or money. This paper presents a multiobjective optimization
algorithm for expensive-to-evaluate noisy functions for robotics.
We present a method for model selection between heteroscedas-
tic and standard homoscedastic Gaussian process regression tech-
niques to create suitable surrogate functions from noisy samples,
and to find the point to be observed at the next step. This algorithm
is compared against an existing multiobjective optimization algo-
rithm, and then used to optimize the speed and head stability of Fig. 1. Pareto-optimal points in objective function space in the case of biob-
the sidewinding gait of a snake robot. jective maximization. Points denoted by a black × mark are dominated by points
with a red circle.
Index Terms—Learning and adaptive systems, multiobjective
optimization (MOO), response surface method (RSM).
single-objective optimization problem by aggregating the mul-
I. INTRODUCTION tiple objectives into a single one using an aggregation method,
such as linear combination or Tchebycheff aggregation.
HE optimization of multiple conflicting objectives, or mul-
T tiobjective optimization (MOO), is commonly used for
problems in many fields, including engineering [1], economics,
If the user does not have a clear preference among the ob-
jectives, the better option would be to find the entire set of the
“best” tradeoff solutions, those which cannot be improved in
and finance [2]. In robotics, tradeoffs between locomotion speed
any objective without becoming worse in at least one other ob-
and energy efficiency appear very often. Other examples include
jective. These solutions are called noninferior or Pareto optimal
maximizing the stability of a camera mounted on the head of
(see Fig. 1). In addition, if a preference is given after generating
a snake robot and its locomotion speed [3], maximization of
the Pareto set, we can seek a compromise solution from the
the speed and stability of a quadruped robot [4], and the mini-
approximate Pareto set at hand.
mization of energy consumption and torque change in humanoid
In cases where the analytic form of the objective functions and
robots [5]. For such problems, one solution is to introduce a set
the mathematical model of the system are available, the problem
of scalars that expresses relative weights, or preferences, among
may be solved by searching for points that satisfy the Karush–
objectives. The MOO problem can then be converted into a
Kuhn–Tucker condition [6], or by solving the Hamilton–Jacobi–
Bellman equation [7]. Another popular approach is to use
Manuscript received November 4, 2015; revised July 13, 2016; accepted Oc-
tober 24, 2016. Date of publication December 28, 2016; date of current version evolutionary algorithms (EAs) [8], because they enable solu-
April 3, 2017. This paper was recommended for publication by Associate Editor tions to be searched for in multiple directions simultaneously.
J. Piater and Editor A. Billard upon evaluation of the reviewers’ comments. This If samples are cheap and the parameter and objective spaces are
work was supported in part by the JSPS Institutional Program for Young Re-
searcher Overseas Visits; in part by a JSPS Grant-in-Aid for JSPS fellows under very high dimensional, EAs are an effective MOO algorithm.
Grant 26.2946; and in part by the JSPS KAKENHI under Grant 26249024. However, in some robotic and other engineering problems,
R. Ariizumi is with the Department of Mechanical Science and Engineer- objective functions are accessible only through experiments and
ing, Graduate School of Engineering, Nagoya University, Nagoya, 464-8603,
Japan (e-mail: [email protected]). EA-based algorithms are, therefore, hard to apply. The chal-
M. Tesch and H. Choset are with the Robotics Institute, Carnegie Mellon lenges in these cases are twofold: first, since the observations
University, Pittsburgh, PA 15213 USA (e-mail: [email protected]; are costly in terms of time and/or money, the number of obser-
[email protected]).
K. Kato and F. Matsuno are with the Department of Mechanical Engineering vations must be severely limited; and second, noise in the ob-
and Science, Graduate School of Engineering, Kyoto University, Kyoto servations makes it difficult to extract useful information from
606-8501, Japan (e-mail: [email protected]; [email protected]). the samples. There are only a few, if any, methods that can be
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org. used in such cases; this likely prevents robotic researchers from
Digital Object Identifier 10.1109/TRO.2016.2632739 using MOO methods.
1552-3098 © 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications standards/publications/rights/index.html for more information.

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
ARIIZUMI et al.: MULTIOBJECTIVE OPTIMIZATION BASED ON EXPENSIVE ROBOTIC EXPERIMENTS UNDER HETEROSCEDASTIC NOISE 469

One promising strategy for expensive MOO is the response often not homoscedastic, and neglecting this sometimes results
surface method (RSM). In this type of algorithm, surrogate in poor function estimation. If the properties of the noise are
functions are constructed to fit the samples. These surrogates known a priori, this knowledge may be able to be coded into
are then used, in place of the unknown true objective functions, the kernel function of the GP regression model. However, since
to plan efficient experiments by balancing exploration and our target is the optimization of expensive functions, we cannot
exploitation. In [9], RSM is used to design the path of a mobile expect accurate prior knowledge. Therefore, we need a more
robot to monitor environments intelligently by making use of flexible framework. Other possible factors may include the ex-
noisy samples, and to this end, proposed an extension of the istence of non-Gaussian noise, and the difference in the rate of
upper confidence bound. A detailed explanation of RSM for change of the true objective functions (nonstationarity). There
single-objective optimization can be found in [10]. The efficient is some literature [31], [32] that deals with the nonstationarity
global optimization algorithm [11], an RSM-based single- problem; however, dealing with this problem is out of the scope
objective optimization method, is extended to MOO based on of this paper.
the aggregation method in [12] and [13]. Emmerich et al. [14] To take input-dependent noise into account, heteroscedas-
suggested using response surfaces to assist EAs, and proposed tic GP regression should be used in RSM. However, this kind
the expected improvement in hypervolume (EIHV) as the rank- of regression presents a difficult challenge because there is no
ing criterion. In [15], they applied a similar approach to [14], but analytical solution, and has been discussed extensively in the
used the lower confidence bound of the improvement. In [16], machine learning community [33], [34]. Among the available
the input to be evaluated in the next step is planned using dif- methods, we chose to use variational heteroscedastic Gaussian
ferent metrics, including approximate EIHV, and selecting four process (VHGP) regression [35] as it gives a reasonable result
or five points in each step. In [17], the authors proposed using with a relatively small computation. VHGP regression has al-
expected maximin fitness improvement, whose analytical form ready been used for RSM in the context of single-objective opti-
was also given for the two-input case. In [18], another statistical mization of a robot [36], but to our knowledge, we are the first to
measure based on the theory of random closed sets is proposed. use this in MOO. In [32], treed GP regression is used to make the
In terms of dealing with noise, most of the existing MOO model more flexible. Although this method would be applica-
methods have been evaluated for noiseless observations, as ble to problems with heteroscedastic noise with relatively small
in [12]. In the EA community, Teich [19] introduced the concept calculation cost, some information would be lost by partitioning
of probability of dominance into a well-known EA-based MOO, the search space into subregions, and training an independent
the strength Pareto evolutionary algorithm (SPEA) [20], to make GP regressor on each of them. Therefore, it would be better if
it robust to noise. Büche et al. [21] also extended SPEA to be we can do without partitioning the search area, in the case where
robust to noise and outliers, and implemented it in optimizing the number of experiments are strictly restricted. Also, note that
the combustion process of a gas turbine. However, these meth- there are other ways to deal with the difficulty of noise that is not
ods cannot be used for expensive optimization as they require modeled as homoscedastic Gaussian noise. In [37], a hyperprior
a considerable number of samples. Eskandari and Geiger [22] on a hyperparameter of the kernel function is introduced. This
proposed the stochastic Pareto genetic algorithm, which is an ex- can attenuate the effect of unmodeled noise and outliers. In [32]
tension of FastPGA [23], an EA for expensive MOO. However, and [38], Student’s t distribution is used, which is known to be
their method also depends on empirical means and variances more robust to outliers than the Gaussian distribution.
that require multiple evaluations for each input, which is not In this paper, we propose a MOO method for expensive noisy
suitable for the optimization of expensive objectives. Fieldsend objectives, particularly those with input-dependent noise. This
and Everson [24] proposed the rolling tide EA, which can handle method uses two GP regression methods to make surrogate
noise that varies in time or space; however, this method requires functions and plan the best experiment based on EIHV. These
too many evaluations to be used in robotic experiments. GP regression methods enable us to make good surrogates from
In the single-objective optimization of robots, the problem the data with input-dependent noise; however, the calculation
of noise has been addressed using RSM with Gaussian process of EIHV and model selection between these two is problematic,
(GP) regression [25], modeling uncertainty induced by obser- because in the heteroscedastic case, the predictive density is
vation noise [26]–[28]. Zuluaga et al. [29] proposed using GP not Gaussian and is, therefore, analytically intractable. In this
regression and the response surface to determine whether a point paper, the approximation of EIHV with reasonable calculation
is Pareto optimal. Independently from this, Tesch et al. [3] pro- cost, and a novel method to determine which regression method
posed using EIHV [14], [30] calculated based on the results to use at every step are also discussed. The effectiveness of the
of GP regression. Noiseless numerical examples exhibited the method is shown by numerical tests and robotic experiments.
superiority of using EIHV over the aggregation function-based The contents of this paper partially appeared in [39]. Compared
method proposed in [12]. However, the performance remained with our previous paper, this paper includes further numerical
insufficient for noisy robotic experiments using a snake robot, verification of the EIHV approximation, the lack of which was
and the authors had to take a mean of five runs per input to get a the primary weak point in [39]. Moreover, additional numerical
reliable result. One possible reason, aside from the overly large verifications are included, which make the efficacy and limita-
noise variance, is that the occurrence of homoscedastic noise tions of the proposed method clear. We also conduct new sets of
(i.e., input-independent noise) was assumed in their work. As is robotic experiments with a different robot to show the efficacy
verified later in this paper, the noise in robotic experiments is of the method in actual robotic problems.

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
470 IEEE TRANSACTIONS ON ROBOTICS, VOL. 33, NO. 2, APRIL 2017

The remainder of this paper is organized as follows. In is important to note that because the goodness of the model
Section II, the algorithm is explained in detail. In Sections III is not taken into consideration in the calculation of EIHV,
and IV, numerical and experimental validations are provided. this can lead to premature termination in cases where the
Section V concludes the paper. model is poor but confident in its prediction, especially at the

II. PROPOSED ALGORITHM Algorithm 1: Proposed Algorithm.

The problem we focus on in this paper is formulated as 1: Given
Ni : the number of initial sample points
maximize f (x) Nm : the maximum number of experiments
x
f : the vector-valued objective function evaluated
sub. to xi ∈ [ximin , ximax ]. (1) through expensive experiments/simulations
: a small positive constant
The objective function f is a vector-valued function, each of X: set of inputs at which f is evaluated
whose elements corresponds to one objective. We assume that Y : set of observed objective function values
on the observation of f , observation noise exists, whose level ref: the user-defined reference for hypervolume (HV)
may vary according to the input x. We further assume that the 2: X ← {x1 , x2 , · · · , xN i }: use Latin hypercube design
number of observations of f is severely limited, given that this 3: for k = 1 to Ni do
method requires significant time and/or money to observe its 4: y ← f (xk )
value, as is often the case with robotic experiments. Although 5: Y ← {Y, y}
constraints are important in optimization, we do not consider 6: end for
any constraints other than that the input vector should be in a 7: for j = 1 to Nm − Ni do
box region. Taking constraints into consideration is one of our 8: Perform regression (Section II-A)
directions for future work. 9: Model selection between Standard GP and VHGP
The proposed algorithm is shown in Algorithm 1. Lines 9–14 models (Section II-C)
are the primary contributions of this paper. 10: Search for max EIHV (Section II-A)
In Line 2, experimental planning for initial evaluations is 11: if max EIHV < HV then
made through a Latin hypercube [40], which is suitable for GP 12: return X, Y
regression, as suggested in [11]. Note that because GP regression 13: end if
is conducted for each objective independent of others, the choice 14: xnew ← argmaxx EIHV(x|ref, Y ) (Section II-B)
of a suitable experimental design method does not depend on the 15: y ← f (xnew )
number of objectives. In Line 8, to make the algorithm robust 16: X ← {X, xnew }, Y ← {Y, y}
to observation noise that may be heteroscedastic, we use both 17: end for
standard GP regression and heteroscedastic GP regression. For 18: return X, Y
the heteroscedastic case, we use VHGP regression [35]. This
method seeks the maximum of the lower bound of the marginal
likelihood instead of the marginal likelihood itself, which is beginning of the optimization. Nonetheless, we found this to be
analytically intractable in the heteroscedastic setup. a very rare occurrence, and we in fact did not observe it in our
However, the difficulty of calculating the marginal likelihood numerical/experimental tests.
causes another problem, because it is usually used not only After making another observation, add the result to the dataset
to tune hyperparameters, but also to select among different (Line 16) and, if the budget has not been completely consumed,
types of kernel functions (models). In this paper, we propose go back to Line 8. The return can be the approximated Pareto set
a novel model selection method based on leave-one-out (LOO) and Pareto front, instead of the whole set of inputs and outputs.
cross validation (Line 9) for deciding which GP—standard or In Section II-A, VHGP used in Lines 8 and 10, which is used
heteroscedastic—to use. This method is especially efficient in to handle heteroscedastic noise in samples, is explained. Line 14
cases where the sample size is small (less than about 40). In is explained in Section II-B, and Line 9 in Section II-C.
addition, because the resultant predictive distribution in the het-
A. Regression Method
eroscedastic case is not Gaussian, EIHV used at Line 14 also
does not have any closed form if heteroscedastic regression In this research, VHGP regression [35] and standard GP re-
is selected. Although the approximation can be calculated by gression are used. In this section, we review both of the regres-
Gauss–Hermite quadrature when the number of objectives is sion methods. The tuning of hyperparameters corresponds to
small, this is computationally very expensive and can make Line 8 of Algorithm 1, and the calculation of the predictive dis-
the procedure prohibitively slow. In this paper, an alternative tribution (5), (10), and (11) to Line 10. As a surrogate function,
approximation that is computationally much less expensive is the mean of the predictive distribution is used.
proposed in Section II-B. 1) Standard GP Regression [25]: Here, we briefly review
The lines 11–13 mean that the procedure is terminated if the standard, i.e., homoscedastic, GP regression.
maximum of EIHV becomes less than 100% of the current Consider the case where we are trying to fit the data D =
hypervolume, where   1 is a user-given positive value. It {(x1 , y1 ), . . . , (xN , yN )} by a function f . The input vector

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
ARIIZUMI et al.: MULTIOBJECTIVE OPTIMIZATION BASED ON EXPENSIVE ROBOTIC EXPERIMENTS UNDER HETEROSCEDASTIC NOISE 471

space is assumed to be a subset of RD , the output space is R, same as that of standard GP regression. In the heteroscedastic
and y and f are defined as the vectors whose ith components case, the marginal likelihood is as
are yi and fi = f (xi ), respectively. In standard GP regression, 
we assume p(y) = p(y|f , g)p(f |g)p(g)df dg
y = f (x) +  
= N (y|f , diag(eg 1 , . . . , eg n ))
f (x) ∼ GP(m(x), kf (x, x ))
 ∼ N (0, σn2 ) (2) × N (f |0, Kf )N (g|μ0 1, Kg )df dg (7)

where x ∼ P means that a random variable x is taken from a which indicates that our confidence in our regression is not an-
distribution (or a stochastic process) P;  is a noise term assumed alytically tractable, making it difficult to tune the hyperparam-
to be taken independently from the same Gaussian distribution eters. To optimize the hyperparameters in this case, VHGP re-
regardless of the input x; and kf is a user-defined kernel function gression maximizes the variational lower bound on the marginal
that expresses our prior knowledge of the latent function. The likelihood instead of marginal likelihood, with respect to the
mean function m(x) is set to be m(x) ≡ 0 to make the following variational parameters and the hyperparameters.
calculation concise; however, in real applications, this will also Define a function F as
be used to code our prior knowledge.
One of the most frequently used kernel functions is the F (q(f ), q(g)) = log p(y) − KL(q(f )q(g) p(f , g|y)) (8)
squared exponential kernel (SE kernel, Gaussian kernel)
  where q(f ) and q(g) are the variational probability densities, and
1 KL(· ·) is the Kullback–Leibler (KL) divergence. Because KL
kf (xi , xj ) = σf2 exp − (xi − xj )T M (xi − xj ) (3)
2 divergence is nonnegative, F gives the lower bound of the loga-
rithm of the marginal likelihood p(y). Therefore, we maximize
where M = diag(l1−2 , . . . , lD
−2
). Parameters σn , σf , and li (i = F instead of the marginal likelihood. To obtain the maximization
1, . . . , D) are called the hyperparameters and should be tuned of F , the dependence on q(f ) can be eliminated by assuming
based on the samples. that q(g) is fixed, and using the variational principle as the first
To tune the hyperparameters, we maximize the marginal like- step. This results in the optimal q(f ) as a function of q(g), and
lihood or evidence by substituting it back into F , F is transformed into what is
 
p(y) = p(y|f )p(f )df = N (y|f , σn2 I)N (f |0, K)df (4) called the marginalized variational bound

F (q(g)) = log Z(q(g)) − KL(q(g) p(g)) (9)

where I is the identity matrix of dimension N , and K is a kernel
matrix whose (i, j) component is k(xi , xj ). In the standard where Z(q(g)) is the normalizing constant of the optimal q(f ).
setting, the marginal likelihood and its gradient can be calculated This bound can be computed in closed form if q(g) is restricted
analytically. to be q(g) = N (g|μ, Σ). Furthermore, it can be shown that from
Once the hyperparameters are determined through the method the stationary equations ∂F/∂μ = 0 and ∂F/∂Σ = 0, μ and Σ
of maximum marginal likelihood, then the predictive distribu- reduce to be a function of a common n-by-n diagonal matrix Λ.
tion of y∗ at an unknown point x∗ is Therefore, F needs to be maximized with respect to these n
y∗ |x∗ ∼ N (kT∗ (K + σn2 I)−1 y, k∗∗ − kT∗ (K + σn2 I)−1 k∗ ) parameters, i.e., the diagonal elements of Λ. Simultaneously, F
(5) can be maximized with respect to the model hyperparameters.
These optimizations can be solved by, for example, the conjugate
where k∗ = [k(x1 , x∗ ) · · · k(xN , x∗ )]T and k∗∗ = k(x∗ , x∗ ). gradient method.
2) Variational Heteroscedastic Gaussian Process Regres- From the maximization of the lower bound of the marginal
sion [35]: Consider the case with input-dependent noise as likelihood, a variational predictive density will be obtained as

y = f (x) + (x)
q(y∗ |x∗ ) = p(y∗ |g∗ , f∗ )q(f∗ |x∗ )q(g∗ |x∗ )df∗ dg∗
f (x) ∼ GP(m(x), kf (x, x )) 
(x) ∼ N (0, e g (x)
) = N (y∗ |a∗ , c2∗ + eg ∗ )N (g∗ |μ∗ , σ∗2 )dg∗ (10)
g(x) ∼ GP(μ0 , kg (x, x )) (6)
where a∗ , c∗ , μ∗ , and σ∗ are determined by the kernel function,
where f is the latent objective function, g is the latent log noise the new input x∗ , and the training data D. Note that, though the
variance, and kf and kg are user-defined kernel functions that predictive distributions of f∗ and g∗ are Gaussian, the resultant
express our prior knowledge of latent functions, defined in the predictive distribution of y∗ is not Gaussian and is analytically
same manner as in standard GP regression. The main point of intractable. However, the mean and variance can be computed
this modeling is that it assumes that the noise level must also be analytically as
determined by a GP. If the noise is not dependent on the input x 2
and it can be written as (x) =  (const.), then this model is the Eq [y∗ |x∗ , D] = a∗ , Vq [y∗ |x∗ , D] = c2∗ + eμ ∗ +σ ∗ /2 . (11)

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
472 IEEE TRANSACTIONS ON ROBOTICS, VOL. 33, NO. 2, APRIL 2017

can then be defined as

I(y) = HV (Ỹ ∪ y) − HV (Ỹ ). (13)
If the predictive density is Gaussian, the closed form of the
EIHV is given by Emmerich et al. [30]. In the case where the
predictive density is expressed as (10), the EIHV will be

EI(x) = I(y)p(y|g, x)p(g|x)dydg

= EI(x|g)p(g|x)dg (14)

where

EI(x|g) = I(y)p(y|g, x)dy
Fig. 2. Surrogate function (thick line) interpolating the sampled points of an 
unknown underlying function with estimated uncertainty (three sigma interval)
shown in the gray area, and the EI for single-objective maximization (thick
= I(y)N (y|a, Σ)dy
dashed line).
Σ = diag(c21 + eg 1 , . . . , c2m + eg m ) (15)
where ck and gk correspond to the kth objective. For EI(x|g),
B. Expected Improvement in Hypervolume the closed form derived in [30] can be used, and because p(g|x)
Expected improvement (EI) is a popular statistical measure is a Gaussian density function, EI(x) can be calculated numer-
to make an efficient experimental plan for the next step, which ically by Gauss–Hermite quadrature if the number of objectives
automatically balances the tradeoff between exploration and m is small. However, even with the closed form [30], the cal-
exploitation without requiring a tuning parameter. To define EI, culation of EIHV is still time consuming, and the following
we first have to define the improvement. approximation of EIHV gives equivalent or better results with
In the single-objective case, the improvement at x with the much less computation, as will be shown by numerical examples
value y is the increase in the maximum sampled target value. in Section III-E2:

The expectation of improvement is ¯
EI(x) = I(y)N (y|a, Σ̄)dy (16)
 ∞
EI(x) = (y − max(Ỹ ))p(y|x)dy (12) where
m ax( Ỹ )
2 2
Σ̄ = diag(c21 + eμ 1 +σ 1 /2 , . . . , c2m + eμ m +σ m /2 ). (17)
where Ỹ is the set of sampled target values. Fig. 2 illustrates the
This is given by approximating the predictive density (10) by a
concept of EI. The dark colored area represents the probability
Gaussian distribution with the same mean and variance as the
for the sample at the point to give a better result than the current
true density calculated in (11), and can be calculated by the
best one. Since EI not only considers the probability that this
formula in [30]. In the limit of σ → 0 (i.e., in the limit of no
point is better but also by how much, the point with the highest
uncertainty in noise variance), (16) tends to be identical to (14).
probability to outperform the current best point does not always
Therefore, (16) is expected to give a good approximation in the
have the highest EI value. In general, between two points with
case where σ is small compared with |μ|. Note that the value of
equal predictive mean, higher predictive variance implies higher
EIHV itself is not so important for our purpose because we need
EI, and two points with equal predictive variance, higher predic-
only the maximizer of EIHV and not EIHV itself. Although
tive mean implies higher EI. In this figure, the rightmost peak
this approximation may not be as accurate as Gauss–Hermite
of the EI corresponds to its maximum, and therefore the input
quadrature, numerical examples in Section III-E2 show that the
that attains this will be used for the next experiment. In the case
discrepancy becomes small at the neighbor of the maximum of
where the predictive distribution p(y) is Gaussian, the analytic
EIHV, which implies that this approximation is sufficient for
form of EI can be obtained [11].
experiment planning.
In MOO, because the solution is not a single point but a whole
Pareto set of points, the improvement must capture the change
C. Model Selection
in the quality of this set. One metric that expresses the quality
of the set of solutions is the set’s hypervolume [41]. This is the Because in most cases we have little prior knowledge of the
volume in objective space that is Pareto-dominated by at least objective functions, we have to choose the best prior distribu-
one point in the Pareto subset of the set in question, at the same tion or model from multiple candidates. In particular, selection
time dominating a user-defined reference point, which basically between the standard GP model and the HGP model is impor-
defines the lower bounds of objective values. tant. As the HGP model is more complex, it is more likely to
Let HV (A) be the hypervolume of a set A; the improvement overfit than the standard GP model. The problem is not only that
in the case where the output of m objective functions is y ∈ Rm the HGP model is more prone to overfitting, but also that this

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
ARIIZUMI et al.: MULTIOBJECTIVE OPTIMIZATION BASED ON EXPENSIVE ROBOTIC EXPERIMENTS UNDER HETEROSCEDASTIC NOISE 473

Fig. 3. Two types of overfitting. The crosses are samples, the solid lines are
mean functions, and the colored areas show 2-σ intervals. (a) Common to both
regressions. (b) Special to HGP regression.

model sometimes results in a more problematic kind of overfit-

ting. There are two kinds of overfitting, the second of which is
specific to HGP regression.
1) The predictive mean is flat, except the vicinity of the
points on which the samples are drawn, and the predictive Fig. 4. Typical four patterns for regression in LOO cross validation. The red
variance becomes small only around the sampled points. point marked by ∗ is the point that is not used in the regression, the blue line is
the surrogate function, and the 3σ region is shown by the colored area. (a) Both
2) The predictive mean is flat even around the sampled a i and p i are small. (b) a i is small but p i is large. (c) a i is large but p i is small.
points, and the predictive variance is large around the (d) Both a i and p i are large.
sampled points.
In Fig. 3, we show typical examples of both kinds of over-
fitting. The crosses are samples, the solid lines are predictive accuracies of both predictions of the mean and the uncertainty
mean functions, and the colored areas illustrate predictive 2-σ are equally important, we need a metric that has the properties
intervals. Both types of overfitting cause the algorithm to be of both indicators, and that is normalized in some sense not to
inefficient, but the second type is more difficult to resolve. In put too much stress on only one of them.
the second type of overfitting, any variation from the mean is The proposed LOO method will choose the better model be-
attributed to the noise by adjusting the noise variance. This leads tween a VHGP model and a standard GP model in the following
to large EIHV values around sampled points, rather than uneval- manner. Choose a point xi that will serve as a test point and per-
uated regions, and leads to dense, noninformative experiments form the regression without it. Let ŷi\i be the predicted function
at a few fixed points. The standard GP model is free from the value at the training point xi , evaluated without using the data
second type of overfitting, although it is sometimes too simple of point xi . Similarly, σ̂i\i is the standard deviation (standard
to explain the data when used alone. Therefore, it is essential GP) or the mean of it (VHGP) at point xi , evaluated without
for our method to select between the standard GP model and the using xi . Let ai and pi be defined as
HGP model to make it useful for black-box objective functions.
|ŷi\i − yi |
However, selection between homoscedastic and het- ai = , pi = |ŷi\i − yi | (18)
eroscedastic models is quite challenging because of the lack of σ̂i\i
common analytically tractable metrics between these two mod- where | · | is absolute value and yi is the sampled value at xi . ai
els. In the standard GP regression setting, marginal likelihood is the absolute value of the standardized cross-validated resid-
can serve as a model selection metric, but in the heteroscedastic ual proposed in [11] and corresponds to the accuracy of the
case, it is not analytically tractable. The variational lower bound regression: If the regression is successful, ai may typically be
is also not a valid alternative for model selection between ho- less than about 3 (the sampled point is in the 3σ interval of the
moscedastic and heteroscedastic models because it is necessar- prediction). pi indicates how close the prediction is to the real
ily smaller than the marginal likelihood for the heteroscedas- value. These relations are illustrated in Fig. 4. These indicators
tic model (though it is equal to the marginal likelihood for are calculated both for standard GP regression (expressed by su-
the homoscedastic model). To overcome this difficulty, we pro- perscript “std”) and VHGP regression (expressed by superscript
pose using LOO validation. Instead, we could use Monte Carlo “vh”), and the sum of their ratio is taken as
(MC) to calculate the marginal likelihood and choose the model
(std) (std) (vh) (vh)
with the largest marginal likelihood; however, as shown in Sec- (std) ai pi (vh) ai pi
ri = (vh)
+ (vh)
, ri = (std)
+ (std)
. (19)
tion III, it is generally difficult to obtain good results by MC ai pi ai pi
calculation. The details on MC calculation are provided in the  
(std) (vh)
appendix. If ni=1 ri ≤ ni=1 ri , standard GP regression will fit bet-
Note that both of the ordinary squared forms of the cross- ter than VHGP regression.
validation error and the standardized residual proposed by Jones The computation requires that both the GP and HGP regres-
et al. [11] are not suitable for our application if used alone. As the sions are proportional to O(N 3 ). Therefore, LOO will require

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
474 IEEE TRANSACTIONS ON ROBOTICS, VOL. 33, NO. 2, APRIL 2017

TABLE I
TRUE HYPERVOLUME

MAT T3 T4 T6

True HV 5.1013 10.0444 44.6667 6.7989

O(N 4 ) calculations. However, note that in theory, LOO can

easily be parallelized, and this will reduce the calculation time Fig. 5. Graph of the test function MAT. (a) Objective 1. (b) Objective 2.
considerably. Regarding the calculation needed to find the max-
imizer of EIHV, it depends heavily on the properties of the
constructed surrogate functions. In this test, we only used 2-input-2-output functions for sim-
Another metric that would be suitable is the log pseudolike- plicity. Our method also works in higher dimensional input
lihood [25], which is defined as problems, although this will require more function evaluations
than 2-input cases. In future work, we will strive to elucidate the

n
relationship between the required number of experiments and
LLOO = log p(yi |y−i , θ) (20)
i=1
the dimensionality of the input space.
In this section, we first describe the test functions, and
where y−i is the targets except number i. In homoscedastic then the additive noise. The performance metric for compar-
regression, θ corresponds to f , and in heteroscedastic regres- ing MOO methods is explained in Section III-C. Settings that
sion to the pair of f and g. The model that has larger LLOO , or are commonly used throughout the section are explained in
equivalently, smaller −log LLOO should be selected. Although Section III-D, and the results are shown in Section III-E.
it requires numerical integration of (10) in our case, the compu-
tational burden is comparable to our proposed metric. Detailed A. Test Functions
examination on this metric compared to ours is left to our fu-
1) Test Function MAT: The first test function used in this
ture work, but empirically, these two metrics results in similar
research is the one proposed in [3], which is defined as
performances.
Note that some authors [38], [42] suggest that the problem f1 (x) = f1 (x1 , x2 ) = B(x1 , 2 + 0.5x2 )/20
of overfitting can be attenuated by restricting the intervals of
f2 (x) = f2 (x1 , x2 ) = B(0.4x1 , 5 + 0.1x2 )/10 (21)
hyperparameters. Although an implementation of this kind of
restriction would make our method more robust, its verification where the domain is [0, 10] × [0, 10], and B(x1 , x2 ) is the
is beyond the scope of this paper. Branin function [44]. The reference point for hypervolume cal-
culation is taken to be [0, 0]. The graphs of the functions are
shown in Fig. 5.
III. NUMERICAL TESTS
2) Test Function T3 : The test function T3 used in this re-
To test the efficacy of our method, we ran some numerical search is defined as
calculations using known 2-input-2-output functions: MAT, T3 ,
T4 , and T6 , whose true hypervolumes are shown in Table I. The f1 (x1 , x2 , . . . , xD ) = −x1
test function MAT was proposed in our previous research [3].  
f2 (x1 , x2 , . . . , xD ) = −g 1 − −f1 /g +f1 sin(−10πf1 )/g
We modified the T3 –T6 functions that appeared in [43] from 30-
input-2-output functions into 2-input-2-output functions, and 
D
also into maximization problems instead of minimizations. T5 g(x2 , x3 , . . . , xD ) = 1 + 9 xi /(D − 1) (22)
was omitted from those proposed in [43], given that it is a binary i=2
function and was out of the scope of this paper. In addition, as
where D is the dimensionality of the search space. The domain
a higher order example, we use the T3 function again, but with
is [0, 1]D . The graphs of the functions in the 2-D case are shown
ten-dimensional (10-D) inputs. We also added noise on obser-
in Fig. 6. The reference point for the hypervolume calculation
vations with known variance. In the test, we did not implement
is taken to be [−1, −10]. Note that the Pareto front and the true
Lines 11–13 of Algorithm 1, but this process consumed the en-
hypervolume are invariant to D.
tire budget in every trial. However, in practice, both would be
3) Test Function T4 : The test function T4 used in this re-
used to prevent extra experiments of less significance. We ob-
search is defined as
served that for the optimization of the function MAT through the
proposed method, max(EIHV ) became smaller than 0.01HV f1 (x1 , x2 ) = −x1
within 35 iterations in more than 75% of trials, which means that 
in most of the trials, the algorithm converged before the number f2 (x1 , x2 ) = −g 1 − −f1 /g
of iterations reached the maximum, and could be stopped earlier
if we also used Lines 11–13. g(x2 ) = 11 + x22 − 10 cos(4πx2 ). (23)

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
ARIIZUMI et al.: MULTIOBJECTIVE OPTIMIZATION BASED ON EXPENSIVE ROBOTIC EXPERIMENTS UNDER HETEROSCEDASTIC NOISE 475

Fig. 6. Graph of the test function T3 . (a) Objective 1. (b) Objective 2. Fig. 9. Comparison between the existing method (std. GP regression only)
and the method using VHGP regression only. The test function is MAT (hyper-
volume: 5.1013). (a) 5 initial evaluations. (b) 15 initial evaluations.

C. Performance Metric
For the selection metric of the optimization method, the hy-
pervolume indicator [41] was used. This is a common unary
indicator that has been examined closely [45]. We calculated
the true noiseless function values at the points of the resul-
tant approximated Pareto set. The hypervolume was calculated
based on these true function values, instead of the sampled val-
Fig. 7. Graph of the test function T4 . (a) Objective 1. (b) Objective 2.
ues, because hypervolume calculated based on noisy samples
can be under- or overestimated. Note that the number of points
in the approximated Pareto set varies from trial to trial, and is
not constant. We ran 60 trials for each setting and calculated the
empirical median, 25th percentile, and 75th percentile of the hy-
pervolume. In the tests, the approximated Pareto front is gener-
ated from the evaluated points. However, note that it is suggested
that constructing the approximated Pareto front from surrogate
functions would give a better approximation than constructing it
only from evaluations [46]. Therefore, the performance shown
in what follows can be understood as a lower limit.
Fig. 8. Graph of the test function T6 . (a) Objective 1. (b) Objective 2. If there is no noise on the observations, the hypervolume
will increase monotonically as the number of observations gets
larger. However, in our case, because the algorithm plans the
experiments and returns the approximated Pareto set based on
The domain is [0, 1] × [−5, 5]. The graphs of the functions are
the noisy samples, the corresponding estimated hypervolume
shown in Fig. 7. The reference point for hypervolume calculation
can decrease.
is taken to be [−1, −45].
4) Test Function T6 : The test function T6 used in this re-
search is defined as D. Common Settings
To solve the maximization problem of EIHV, we chose to
f1 (x1 , x2 ) = −1 + exp(−4x1 )sin6 (6πx1 ) first calculate EIHV at densely sampled points, and then used

the maximizer among these points as the starting point of the
f2 (x1 , x2 ) = −g 1 − (f1 /g)2
gradient method. Of course, other kinds of maximization meth-
1 ods, such as a gradient method with random restart or some
g(x2 ) = 1 + 9x24 . (24)
EAs, are applicable.
The domain is [0, 1] × [0, 1]. The graphs of the functions are Regarding the initial settings of the hyperparameters, we set
shown in Fig. 8. The reference point for hypervolume calculation all of the initial values of the hyperparameters at 1 for standard
is taken to be [−1, −10]. homoscedastic GP regression. For heteroscedastic regression,
we used the result of homoscedastic regression to set the initial
hyperparameter values.
B. Additive Noise
All numerical tests were repeated 60 times to make the results
We tested with two kinds of Gaussian noise; the first was statistically reliable. To illustrate the results, hypervolumes were
homoscedastic with variance r(x) = σ̄n2 (= const.), and the sec- plotted against the number of evaluations (Figs. 9 and 12–22).
ond was heteroscedastic with variance r(x) = {σn (sin( x ) + Because the distribution of hypervolumes after a fixed number
1)/2}2 . of evaluations is skewed, we used the median and the 25th/75th

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
476 IEEE TRANSACTIONS ON ROBOTICS, VOL. 33, NO. 2, APRIL 2017

percentiles instead of the mean and standard deviation, respec-

tively. Thick lines refer to the median, and the colored areas
refer to the region between the 25th and 75th percentiles.

E. Results
1) Need for Model Selection Between Two Kinds of GPs:
First, to show the need for the model selection discussed in
Section II-C, we compare the performances of two cases: 1)
standard GP regression only (existing method [3]) and 2) VHGP
regression only. The tests were performed with two different
settings for the number of initial points: 5 and 15. The test
function MAT was used, and homoscedastic noise (σ̄n = 0.15)
was added to observations. A maximum of 40 evaluations were
performed for each trial. The results are shown in Fig. 9.
In the case with five initial evaluations, the existing method
(standard GP regression only) clearly outperforms the other
(VHGP regression only). This is because the VHGP model is
more complex than the standard GP model, and tends to overfit Fig. 10. Slices of the negative log EIHV surface calculated by Gauss–Hermite
to the small size data. In the case with 15 initial points, both quadrature (14) and Gaussian approximation (16) [see (a) and (b)], and their
differences [see (c) and (d)]. The slices are axis-aligned [(a) and (c) with
methods work equally well. x2 = 2.0202 and (b) and (d) with x1 = 3.9394], and go through the point
The problem is that, in real problems, the necessary number of with maximum discrepancy between the methods. Homoscedastic noise (σ̄ n
initial evaluations for VHGP regression is likely unknown, and = 0.15) is used. (a) −log(EIH V ) after 40 evaluations. (b) −log(EIH V ) af-
ter 40 evaluations. (c) Discrepancy after 40 evaluations. (d) Discrepancy after
this in turn shows the need for model selection between standard 40 evaluations.
GP and VHGP. We show in the following that through the model
selection methods proposed in Section II-C, the results will be
at least as good as, and in many cases better than, the best of
methods 1) and 2).
2) Comparison Between Two EI Calculations, (14) and (16):
In this test, we compared the performances of the two calcula-
tions of EIHV in the case of VHGP regression: Gauss–Hermite
quadrature (14) and Gaussian approximation of the predictive
density (16). Here, only VHGP regression was used and not
standard GP regression.
The tests were done for MAT with two kinds of additive
noise: homoscedastic noise with σ̄n = 0.15, and sinus noise
with σn = 0.2, as explained in Section III-B. The number of
initial evaluations was set as 15. For Gauss–Hermite quadrature,
nine (3 × 3) nodes were used, which gave enough precision for
EIHV calculation.
Figs. 10 and 11 show axis-aligned slices of the negative log
EIHV surface after 40 evaluations, as calculated by each method,
and the corresponding slices of the discrepancy. The planes
shown are selected to go through the point with maximum error.
It can be seen that the discrepancy becomes large at the maxima Fig. 11. Slices of the negative log EIHV surface calculated by Gauss–Hermite
quadrature (14) and Gaussian approximation (16) [see (a) and (b)], and their
of the negative log EIHV, and at the points with small negative differences [see (c) and (d)]. The slices are axis-aligned with x1 = 7.7778 [see
log EIHV, the discrepancy becomes small. Because we need the (a) and (c)] and x2 = 1.5152 [see (b) and (d)]. Sinus noise (σ n =0.2) is used.
minimizer of the negative log EIHV (maximizer of the EIHV), (a) −log(EIH V ) after 40 evaluations. (b) −log(EIH V ) after 40 evaluations.
(c) Discrepancy after 40 evaluations. (d) Discrepancy after 40 evaluations.
the approximation (16) is considered appropriate for our pur-
pose, though it can be inaccurate in the area that we are not
interested in. Additionally, we observed that σ∗ in (10) are very
small compared with |μ∗ |; in fact, σ∗ /|μ∗ | is around 10−5 –10−10 each step in the case with sinus noise. The total time required
at the minimizer of the negative log EIHV, which verifies the for one trial was the integral of the curve, which was about 6 h
use of the approximation (16). where Gauss–Hermite quadrature was used, and about 44 min
Hypervolumes were plotted against the number of evalua- in the other case. From these results, it can be seen that the
tions in Fig. 12, and Fig. 13(a) shows the time consumption for calculation of EIHV through Gauss–Hermite quadrature is very

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
ARIIZUMI et al.: MULTIOBJECTIVE OPTIMIZATION BASED ON EXPENSIVE ROBOTIC EXPERIMENTS UNDER HETEROSCEDASTIC NOISE 477

Fig. 12. Comparison between two metric calculations: (14) and its approxi- Fig. 14. Comparison between LOO and MC as the model selection method.
mation (16). Because we use 15 initial evaluations, the graphs begin with 16 The number of initial evaluations is 15. The test function is MAT (hyper-
evaluations. The test function is MAT (hypervolume: 5.1013). (a) Homoscedas- volume: 5.1013). (a) Homoscedastic noise (σ̄ n = 0.15). (b) Sinusoidal noise
tic noise (σ̄ n = 0.15). (b) Sinusoidal noise (σ n = 0.2). (σ n = 0.2).

Fig. 13. Time needed to complete one step of the procedure. (a) Simplified
EI versus Complete EI. (b) LOO versus MC. Fig. 15. Comparison between the proposed method (model selection) and the
existing method (std. GP regression only) [3]. The number of initial evaluations
is 15. The test function is MAT (hypervolume: 5.1013). (a) Homoscedastic noise
(σ̄ n = 0.15). (b) Sinusoidal noise (σ n = 0.2).
time consuming given the insignificant improvements in accu-
racy.
Because our goal is to find the solution set with as few evalua-
tions as possible and not to reduce the calculation cost, a method
that requires a great amount of calculation can be used as long as
it contributes to reducing the number of necessary observations.
However, because the time-consuming calculation (14) did not
reduce the required number of samples, we concluded that the
approximation (16) should be applied instead.
3) Comparison Between Two Model Selection Methods: For
selecting between standard GP and VHGP, two methods are
compared: LOO as explained in Section II-C, and numerical
calculation of marginal likelihood by MC. See the Appendix for Fig. 16. Comparison between the proposed method (model selection) and
details on MC calculation. MAT was used as the test function. the method that uses VHGP regression only. The test function is MAT (hyper-
The results are shown in Fig. 14, and time required for each step volume: 5.1013). (a) Homoscedastic noise (σ̄ n = 0.15). (b) Sinusoidal noise
(σ n = 0.2).
of the procedure is shown in Fig. 13(b).
From Figs. 13(b) and 14, it can be seen that the performance
is slightly better if LOO is used, but required time is also less
with LOO if there are less than about 40 points. The poorer
performance of MC is attributed to the limited precision of the
calculation of marginal likelihood. Because precision through
MC is proportional to the square root of the sample size, it is
difficult to precisely calculate the marginal likelihood, which is
typically much smaller than 1. Moreover, we usually use the
logarithm of marginal likelihood, which amplifies the error as
log (p + Δp) − log (p) ≈ Δp/p > Δp (25) Fig. 17. Comparison between the proposed method and the existing
method [3]. The number of initial evaluations is 20. Test function is T3 (hyper-
where p < 1 is the marginal likelihood. Therefore, we need sev- volume: 10.0444). (a) Homoscedastic noise (σ̄ n = 0.1). (b) Sinusoidal noise
eral hundred times more samples to get ten times more precise (σ n = 0.2).

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
478 IEEE TRANSACTIONS ON ROBOTICS, VOL. 33, NO. 2, APRIL 2017

Fig. 18. Comparison between the proposed method and the method that uses Fig. 21. Comparison between the proposed method and the existing
VHGP regression only. Test function is T3 (hypervolume: 10.0444). (a) Ho- method [3]. The number of initial evaluations is 40. Test function is T6 (hy-
moscedastic noise (σ̄ n = 0.1). (b) Sinusoidal noise (σ n = 0.2). pervolume: 6.7989). (a) Homoscedastic noise (σ̄ n = 0.1). (b) Sinusoidal noise
(σ n = 0.1).

Fig. 19. Comparison between the proposed method and the existing
method [3]. The number of initial evaluations is 20. Test function is T4 (hyper-
volume: 44.6667). (a) Homoscedastic noise (σ̄ n = 0.2). (b) Sinusoidal noise
(σ n = 0.25).
Fig. 22. Comparison between the proposed method and the method that uses
VHGP regression only. Test function is T6 (hypervolume: 6.7989). (a) Ho-
moscedastic noise (σ̄ n = 0.1). (b) Sinusoidal noise (σ n = 0.1).

the other cases, it works at least as well as the others. In this

case, the method using VHGP regression exclusively performs
worse than the other two, even in the existence of heteroscedastic
noise. This is a somewhat counterintuitive result because VHGP
regression seemed appropriate for fitting samples wherein the
Fig. 20. Comparison between the proposed method and the method that uses noise level is actually a function of input and should, there-
VHGP regression only. Test function is T4 (hypervolume: 44.6667). (a) Ho- fore, provide a good surrogate. One possible explanation is that
moscedastic noise (σ̄ n = 0.2). (b) Sinusoidal noise (σ n = 0.25). VHGP regression has too much complexity in its noise model
to fit the samples of size 40 or so from T6 , overfitting the change
for the latent function because of the noise. This can lead to
estimation. However, as can be seen from Fig. 13(b), calculation a much flatter surrogate surface than the true latent function.
time for LOO grows faster than that for MC and will be much For the case of homoscedastic noise, the proposed method is
slower if there are many more evaluations. inferior to the existing method. This can be seen as a failure of
4) Comparison Among Three Methods (2-D Search Space): our model selection method, but we note that a similar trend is
Finally, the performance of the proposed method was tested observed, even when we used the maximization of the marginal
against that of the existing method and the method that only uses likelihood calculated by MC.
VHGP regression. In this test, all four test functions were used 5) 10-D Tests: For this test, the 10-D version of the T3 func-
to check the class of problems for which the proposed method tion, i.e., T3 with D = 10 in (22), is optimized. In each trial,
works efficiently. The number of initial experiments was 15 for 50 initial points and 100 points in total are sampled. The ad-
MAT, 20 for T3 (2-D) and T4 , and 40 for T6 . The results for ditive noise is the same for the 2-D case of T3 . The results are
MAT are shown in Figs. 15 and 16, T3 in Figs. 17 and 18, shown in Fig. 23. From the figures, it can be seen that both
T4 in Figs. 19 and 20, and T6 in Figs. 21 and 22. It can be seen the existing method and the proposed method perform well, in
that except for T6 , the proposed method outperforms the other that within only a small number of experiments, the hypervol-
two, or at least performs equally well. umes get larger compared with their initial values. However, our
For T6 , the proposed method is outperformed by the existing proposed method outperforms the existing one in terms of final
method in the case where the noise is homoscedastic, but in values.

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
ARIIZUMI et al.: MULTIOBJECTIVE OPTIMIZATION BASED ON EXPENSIVE ROBOTIC EXPERIMENTS UNDER HETEROSCEDASTIC NOISE 479

Fig. 23. Comparison between the proposed method and the existing method.
Test function is T3 with 10-D input (hypervolume: 10.0444). (a) Homoscedastic
noise (σ̄ n = 0.1). (b) Sinusoidal noise (σ n = 0.2).

Fig. 24. Snake robot.

IV. EXPERIMENTS
We conducted robotic experiments with a snake robot (Fig.
24), which moves via sidewinding locomotion. The objective
functions were set to be the speed and the stability of the robot
head. Here, head stability is roughly inversely proportional to the
amount of head motion, which is very important when operating
the robot. Again, in the experiments, we did not implement
Lines 11–13 of Algorithm 1.
The snake robot is often controlled by motions with a finite-
dimensional constrained control trajectory subspace (the same
as the gait model described in [47]), which is defined as
 Fig. 25. Resulting Pareto front in the case of (a) proposed method and
βeven + Aeven sin(θ), n = even (b) existing method [3] after 25, 30, 35, and 40 evaluations. Observed data
α(n, t) = are shown in × marks and the circles are the elements in the Pareto front.
βodd + Aodd sin(θ + δ), n = odd
θ = (ωs n + ωt t) (26)
where α(n, t) is the nth joint angle at time t. This can be seen 1) The speed: The net displacement of the head after running
as an extension of the serpenoid curve [48], which models the the snake for 15 s.
shape of a real snake well. There are seven free parameters 2) The head stability: The stability of the image from the
(βs, As, δ, ωs , and ωt ), and by tuning these parameters, we camera mounted on the head of the snake.
can command the snake to move via many kinds of motions, For head stability, we put an acceleration sensor on the head,
including slithering, sidewinding, and rolling in an arc. and calculated the level of vibration as follows. Let a(t) be the
Snake robots are expected to be useful for many applications, read of acceleration sensor, then the head stability is defined as
including pipe inspections and urban search and rescue opera-  T
tions. In practice, the operator would have to rely on the camera
(Head Stability) = − a(t) − E[a(t)] 2 dt (27)
image from the camera mounted on the head, which makes it 0
challenging to operate the robot from a distance. Although a
method that would provide information about the state of the where E[a(t)] is the mean of a(t) during one run.
robot based on the virtual chassis [49] has been proposed, it Although it is not clear only from the above definitions
remains difficult to operate the robot if the camera is mounted whether there is a tradeoff between the two objectives, the pro-
on a shaky base. At the same time, it would be advantageous posed method, along with other MOO methods, can be used for
to move the robot faster, which will amplify the movement of cases where the objectives do not actually conflict. Therefore,
the head camera. Therefore, we chose two objective functions if there is no reason to deny the existence of conflict between
as follows. objectives, it is better to turn to MOO.

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
480 IEEE TRANSACTIONS ON ROBOTICS, VOL. 33, NO. 2, APRIL 2017

Fig. 27. Hypervolume indicator calculated from the experimental results.

TABLE II
OBSERVED VALUE OF THE HEAD STABILITY
Fig. 26. Example of response surface of head stability and speed after 39
experiments, which was used to plan the 40th experiment. The color shows the
variance of the predictive distribution. (c) and (d) Variance as color maps, from Samp.1 Samp.2 Samp.3 Samp.4 Samp.5 st. dev.
which the nonuniformity of the uncertainty can be ascertained. (a) Head stability.
(b) Head speed. (c) Uncertainty in head stability estimation. (d) Uncertainty in x1 19.0062 15.2082 22.6125 16.6942 17.4333 2.8252
head stability estimation. x2 10.4526 9.4061 9.8783 9.6554 11.5510 0.8542
x3 7.0852 6.9981 6.4631 6.8942 7.1128 0.2644

The consideration of heteroscedasticity in the robotic exper-

iments is verified later in this section through another set of maps, from which it can be seen that the uncertainty is not
experiments, but essentially, it is appropriate to consider that uniform across the domain.
the larger the movement, the greater the noise level. Fig. 27 shows the plot of the hypervolumes. The thick lines
In these experiments, the model was restricted to the represent the median, and the colored area shows the region
sidewinding parameter space as defined in [47] (the phase shift δ between the 25th and 75th percentiles. Though comparison be-
is fixed as π/4, the offsets βs as 0, ωs = 3π/16, and ωt = 3π/5), tween hypervolumes is generally difficult because of the lack
and we introduced an additional phase shift φ for the head mod- of knowledge of the true objective functions, it is clear from
ule as this figure that the proposed method outperforms the existing
method. In Fig. 27, we also show the median of five random
α(1, t) = Aodd sin(θ + δ + φ). (28) searches as a dotted line. Note that we omit the error band for
random search to keep the figure from becoming cluttered. From
The ratio between Aodd and Aeven was also fixed as
this, it is clear that both the existing and proposed methods are
Aodd /Aeven = 3/8. The parameters that should be optimized are
much more efficient than the random search.
the amplitude Aodd and the phase shift φ. The domains of the
To verify the consideration of heteroscedasticity, we con-
parameters were set as Aodd ∈ [0.23, 1] and φ ∈ [0, 2π].
ducted another set of experiments. We chose three in-
The first 20 evaluations were planned for initialization
puts: x1 = [0.7177, 0.3569], x2 = [0.5634, 0.0538], and x3 =
through Latin hypercube design and the 20 subsequent eval-
[0.3767, 0.5500], and evaluated the head stability for each. We
uations were selected using the proposed method; the same
took five samples for each input, which are shown in Table II.
procedure was also performed for the existing method. The ini-
Each row corresponds to an input, and the last column shows
tial samples were shared by both methods, and in total, ten sets
the empirical standard deviation, which is the square root of the
of experiments were conducted for each method. Fig. 25(a) and
unbiased estimator of noise variances. It can be seen that there
(b) shows the resulting Pareto fronts for the proposed method
are large differences in the standard deviations. For example, the
and the existing method [3], respectively, for both of which
standard deviation for x1 is more than ten times larger than that
we used the same initial evaluations. By comparing the points
for x3 . In fact, the null hypothesis, that the variances between
that are the best in terms of speed, it is clear that the proposed
these two sets of samples are equal, is rejected by a two-sample
method found solutions better (i.e., faster) than the existing
F-test with a 1% significance level. These results clearly show
method. The proposed method outperforms the other in terms of
that the noise is actually heteroscedastic in this robotic example.
stability.
Fig. 26 shows the response surfaces used in planning for the
V. CONCLUSION
40th experiment with the proposed method. The crosses are
the samples, and the color shows the variance of the predictive In this paper, we proposed a Pareto optimization method that
density. In Fig. 26(c) and (d), the variance is shown as color can be used for the optimization of expensive noisy functions. By

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
ARIIZUMI et al.: MULTIOBJECTIVE OPTIMIZATION BASED ON EXPENSIVE ROBOTIC EXPERIMENTS UNDER HETEROSCEDASTIC NOISE 481

selecting between standard and heteroscedastic GPs using LOO, REFERENCES

the performance of the proposed method becomes better than [1] R. T. Marler and J. S. Arora, “Survey of multi-objective optimization
our previous method not only in the presence of heteroscedastic methods for engineering,” Struct. Multidisciplinary Optim., vol. 26, no. 6,
noise, but also in the case of homoscedastic noise. A model pp. 369–395, Apr. 2004.
[2] M. G. C. Tapia and C. A. Coello Coello, “Applications of multi-objective
selection method that can be used to select between standard evolutionary algorithms in economics and finance: A survey,” in Proc.
GP and heteroscedastic GP by LOO was also proposed. This IEEE Congr. Evol. Comput., 2007, pp. 532–539.
method often results in better results with less computation [3] M. Tesch, J. Schneider, and H. Choset, “Expensive multiobjective opti-
mization for robotics,” in Proc. IEEE Int. Conf. Robot. Autom., Karlsruhe,
than the numerical calculation of marginal likelihood by MC Germany, 2013, pp. 973–980.
sampling. [4] M. Oliveira, L. Costa, A. Rocha, C. Santos, and M. Ferreira, “Multi-
In this paper, we only used EIHV and did not compare with objective optimization of a quadruped robot locomotion using a genetic
algorithm,” in Soft Computing in Industrial Applications, vol. 96. Berlin,
other potentially powerful alternative metrics, such as proba- Germany: Springer, 2011, pp. 427–436.
bility of improvement or upper confidence bound. Compari- [5] G. Capi, M. Yokota, and K. Mitobe, “A new humanoid robot gait gen-
son between these methods in single-objective optimization in eration based on multiobjective optimization,” in Proc. IEEE/ASME Int.
Conf. Adv. Intell. Mechatronics, Monterey, CA, USA, 2005, pp. 450–454.
robotics can be found in [28], but for the multiobjective case [6] M. Ringkamp, S. Ober-Blöbaum, M. Dellnitz, and O. Schütze, “Handling
it remains to be done. Another direction of future research will high dimensional problems with multi-objective continuation methods via
include extending our method to be able to deal with changing successive approximation of the tangent space,” Eng. Optim., vol. 44 no. 9,
pp. 1117–1146, 2012.
environments. Since every single observation in one environ- [7] A. Kumar and A. Vladimirsky, “An efficient method for multiobjective
ment will give some information about the performance in all optimal control and optimal control subject to integral constraints,” J.
the other environments, in particular for similar environments, Comput. Math., vol. 28, no. 4, pp. 517–551, Apr. 2010.
[8] A. Zhou, B.-Y. Qu, H. Li, S.-Z. Zhao, P. N. Suganthan, and Q. Zhang,
it will be possible to accelerate the optimization procedure by “Multiobjective evolutionary algorithms: A survey of the state of the art,”
using the results from other environments. Our method will be Swarm Evol. Comput., vol. 1, no. 1, pp. 32–49, Mar. 2011.
useful in the case in which the function evaluation is expensive [9] R. Marchant and F. Ramos, “Bayesian optimisation for intelligent envi-
ronmental monitoring.” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.,
and the noise cannot be neglected; however, it is not so efficient Algarve, Portugal, 2012, pp. 2242–2249.
if the dimensionality of the search space is very large, as is the [10] E. Brochu, V. Cora, and N. de Freitas, “A tutorial on Bayesian optimization
case for hyper-redundant robots. Constructing an efficient MOO of expensive cost functions, with application to active user modelling and
hierarchical reinforcement learning,” Univ. British Columbia, Vancouver,
method that can be used for high-dimensional noisy objectives BC, Canada, Tech. Rep. TR-2009-23, 2009.
is another possible topic for future work. [11] D. R. Jones, M. Schonlau, and W. J. Welch, “Efficient global optimiza-
tion of expensive black-box functions,” J. Global Optim., vol. 13, no. 4,
pp. 455–492, Dec. 1998.
APPENDIX [12] J. Knowles, “ParEGO: A hybrid algorithm with on-line landscape approx-
imation for expensive multiobjective optimization problems,” IEEE Trans.
The marginal likelihood is calculated as Evol. Comput., vol. 10, no. 1, pp. 50–66, Feb. 2005.
 [13] Q. Zhang and H. Li, “Expensive multiobjective optimization by MOEA/D
p(y) = p(y|f , g)p(f )p(g)df dg (29) with Gaussian process model,” IEEE Trans. Evol. Comput., vol. 14, no. 3,
pp. 456–474, Jun. 2010.
[14] M. Emmerich, K. Giannakoglou, and B. Naujoks, “Single- and multi-
where p(y|f , g) is the likelihood function, and p(f ) and p(g) are objective evolutionary optimization assisted by Gaussian random field
metamodels,” IEEE Trans. Evol. Comput., vol. 10, no. 4, pp. 421–439,
the prior density functions. Given that the priors are Gaussian, Aug. 2006.
it is easy to draw samples from p(f ) and p(g). Let the samples [15] W. Ponweiser, T. Wagner, D. Biermann, and M. Vincze, “Multiobjec-
from the prior densities be f (i) and g(i) where i = 1, . . . , M . tive optimization on a limited budget of evaluations using model-assisted
S-Metric selection,” in Proc. 10th Parallel Problem Solving Nature, Dort-
Then, the approximation of the marginal likelihood p̂(y) can be mund, Germany, 2008, pp. 784–794.
obtained as [16] T. Akhtar and C. A. Shoemaker, “Multi objective optimization of com-
putationally expensive multi-modal functions with RBF surrogates and
1 
M
multi-rule selection,” J. Global Optim., vol. 64, pp. 17–32, 2016. [On-
p̂(y) = p(y|f (i) , g(i) ). (30) line]. Available: http://dx.doi.org/10.1007/s10898-015-0270-y
M i=1 [17] J. Svenson and T. Santner, “Multiobjective optimization of expensive-to-
evaluate deterministic computer simulator models,” Comput. Statist. Data
We use M ∈ [2 × 105 , 1 × 106 ] points, where the exact number Anal., vol. 94, pp. 250–264, Feb. 2016.
[18] B. Binois, D. Ginsbourger, and O. Roustant, “Quantifying uncertainty
differs according to the convergence of the calculation. Although on pareto fronts with Gaussian process conditional simulations,” in Proc.
it is well known that in some cases this simple approximation Learn. Intell. Optim. Conf., vol. 243, no. 2, pp. 386–394, 2014
can be inaccurate, it is easy to implement and gives reasonable [19] J. Teich, “Pareto-front exploration with uncertain objectives,” in Proc. 1st
Conf. Evol. Multi-Criterion Optim., Zurich, Switzerland, 2001, pp. 314–
results in our cases. Other much more sophisticated methods 328.
using Markov chain MC, such as those proposed in [50] and [20] E. Zitzler and L. Thiele, “Multiobjective evolutionary algorithms: A com-
[51], can also be used. However, we found that although they parative case study and the strength pareto approach,” IEEE Trans. Evol.
Comput., vol. 3, no. 4, pp. 257–271, Nov. 1999.
reduce the calculation time, they make no significant difference [21] D. Büche, P. Stoll, R. Dornberger, and P. Koumoutsakos, “Multiobjective
in hypervolume from the basic calculation (30). We also note that evolutionary algorithm for the optimization of noisy combustion pro-
it is often the case that the sampling used in MCMC calculation cesses,” IEEE Trans. Syst. Man, Cybern. C, vol. 32, no. 4, pp. 460–473,
Nov. 2002.
is ill conditioned, and the calculation becomes unreliable. In [22] H. Eskandari and C. D. Geiger, “Evolutionary multiobjective optimization
some cases, it even fails in drawing samples, in which cases in noisy problem environments,” J. Heuristics, vol. 15, no. 6, pp. 559–595,
naı̈ve MC also fails to return meaningful results. Dec. 2009.

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
482 IEEE TRANSACTIONS ON ROBOTICS, VOL. 33, NO. 2, APRIL 2017

[23] H. Eskandari and C. D. Geiger, “A fast pareto genetic algorithm approach [45] E. Zitzler, L. Thiele, M. Laumanns, C. M. Fonseca, and V. Grunert da Fon-
for solving expensive multiobjective optimization problems,” J. Heuris- seca, “Performance assessment of multiobjective Optimizers: An analysis
tics, vol. 14, no. 3, pp. 203–241, Jun. 2008. and review,” IEEE Trans. Evol. Comput., vol. 7, no. 2, pp. 117–132,
[24] J. E. Fieldsend and R. M. Everson, “The rolling tide evolutionary algo- Apr. 2003.
rithm: A multiobjective optimizer for noisy optimization problems,” IEEE [46] R. Calandra, J. Peters, and M. P. Deisenroth, “Pareto front modeling for
Trans. Evol. Comput., vol. 19, no. 1, pp. 103–117, Feb. 2015. sensitivity analysis in multi-objective Bayesian optimization,” in Proc.
[25] C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine NIPS Workshop Bayesian Optim., 2014, vol. 5.
Learning. Cambridge, MA, USA: MIT Press, 2006, [47] M. Tesch et al., “Parameterized and scripted gaits for modular snake
[26] D. Lizotte, T. Wang, M. Bowling, and D. Schuurmans, “Automatic gait robots,” Adv. Robot., vol. 23, no. 9, pp. 1131–1158, 2009.
optimization with Gaussian process regression,” in Proc. Int. Joint Conf. [48] S. Hirose, Biologically Inspired Robots: Snake-Like Locomotiors and Ma-
Artif. Intell., Hyderabad, India, 2007, pp. 944–949. nipulators. Oxford, U.K.: Oxford Univ. Press, 1993.
[27] M. Tesch, J. Schneider, and H. Choset, “Using response surfaces and [49] D. Rollinson and H. Choset, “Virtual chassis for snake robots,” in Proc.
expected improvement to optimize snake robot gait parameters,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., San Francisco, CA, 2011,
IEEE/RSJ Int. Conf. Intell. Robots Syst., San Francisco, CA, USA, 2011, pp. 221–226.
pp. 1069–1074. [50] S. Chib and I Jeliazkov, “Marginal likelihood from the Metropolis-
[28] R. Calandra, A. Seyfarth, J. Peters, and M. P. Deisenroth, “Bayesian Hastings output,” J. Amer. Statist. Assoc., vol. 96, no. 453, pp. 270–281,
optimization for learning gaits under uncertainty,” Ann. Math. Artif. Intell., Mar. 2001.
vol. 76, pp. 5–23, 2016. [51] S. Chib and I Jeliazkov, “Accept-reject Metropolis-Hastings sampling and
[29] M. Zuluaga, A. Krause, G. Sergent, and M. Püschel, “Active learning for marginal likelihood estimation,” Statistica Neerlandica, vol. 59, no. 1,
multi-objective optimization,” in Proc. Int. Conf. Mach. Learn., Atlanta, pp. 30–44, Feb. 2005.
GA, USA, 2013, pp. 462–470.
[30] M. Emmerich and J.-W. Klinkenberg, “The computation of the expected
improvement in dominated hypervolume of Pareto front approximations,”
Leiden Inst. Adv. Comput. Sci., Leiden, The Netherlands, Tech. Rep. 1,
2008.
[31] J.-A.M. Assael, Z. Wang, B. Shahriari, and N. Freitas, “Heteroscedastic
treed Bayesian optimization,” arXiv:1410.7172, 2014. Ryo Ariizumi (M’15) received the B.S., M.E., and
[32] M. A. Taddy, H. K. H. Lee, G. A. Gray, and J. D. Griffin, “Bayesian guided Ph.D. degrees in engineering from Kyoto University,
pattern search for robust local optimization,” Technometrics, vol. 51, no. 4, Kyoto, Japan, in 2010, 2012, and 2015, respectively.
pp. 389–401, 2009. He was a Postdoctoral Researcher with Kyoto Uni-
[33] P. Goldberg, C. Williams, and C. Bishop, “Regression with input- versity. He is currently an Assistant Professor with
dependent noise: A Gaussian process treatment,” in Proc. Adv. Neural Nagoya University, Nagoya, Japan. His research in-
Inf. Process. Syst., Denver, CO, USA, 1997, pp. 493–499. terests include control of redundant robots and opti-
[34] K. Kersting, C. Plagemann, P. Pfaff, and W. Burgard, “Most likely het- mization of robotic systems.
eroscedastic Gaussian processes regression,” in Proc. Int. Conf. Mach.
Learn., Corvallis, OR, USA, 2007, pp. 393–400.
[35] M. Lázaro-Gredilla and M. K. Titsias, “Variational heteroscedastic Gaus-
sian process regression,” in Proc. Int. Conf. Mach. Learn., Bellevue, WA,
USA, 2011, pp. 841–848.
[36] S. Kuindersma, R. Grupen, and A. Barto, “Variable risk control via
stochastic optimization,” Int. J. Robot. Res., vol. 32, no. 7, pp. 806–825,
Jun. 2013.
[37] R. Martinez-Cantin, N. Freitas, E. Brochu, J. Castellanos, and A. Doucet, Matthew Tesch (M’13) received the B.S. degree in
“A Bayesian exploration-exploitation approach for optimal online sensing engineering from Franklin W. Olin College of Engi-
and planning with a visually guided mobile robot.” Auton. Robots, vol. 27, neering, Needham, MA, USA, in 2007 and the M.S.
no. 2, pp. 93–103, 2009 and Ph.D. degrees in robotics from Carnegie Mellon
[38] R. Martinez-Cantin, “BayesOpt: A Bayesian optimization library for non- University, Pittsburgh, PA, USA, in 2011 and 2013,
linear optimization, experimental design and bandits,” J. Mach. Learn. respectively.
Res., vol. 15, no. 1, pp. 3735–3739, 2014. He is a Postdoctoral Researcher with Carnegie
[39] R. Ariizumi, M. Tesch, H. Choset, and F. Matsuno, “Expensive multi- Mellon University. His research focuses on machine
objective optimization for robotics with consideration of heteroscedastic learning methods for efficiently optimizing robotic
noise,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst., Chicago, IL, USA, systems from which data collection is time consum-
2014, pp. 2230–2235. ing or expensive. Application of this work has re-
[40] M. D. Mackay, W. J. Conover, and R. J. Beckman, “A comparison of three sulted in significant improvements of the locomotive capabilities of snake robots.
methods for selecting values of input variables in the analysis of output
from a computer code,” Technometrics, vol. 21 no. 2, pp. 239–245, May
1979.
[41] E. Zitzler and L. Thiele, “Multiobjective optimization using evolution-
ary algorithms—A comparative case study,” in Parallel Problem Solv-
ing From Nature—PPSN V. New York, NY, USA: Springer, Sep. 1998,
pp. 292–304. Kenta Kato received the B.S. and M.E. degrees in
[42] Z. Wang, F. Hutter, M. Zoghi, D. Matheson, and N. Freitas, “Bayesian mechanical engineering from Kyoto University, Ky-
optimization in a billion dimensions via random embeddings.” J. Artif. oto, Japan, in 2014 and 2016, respectively.
Intell. Res., vol. 55, pp. 361–387, 2016 He is with Mitsubishi Electric Corporation.
[43] E. Zitzler, K. Deb, and L. Thiele, “Comparison of multiobjective evo- His research interests include the semiautonomous
lutionary algorithms: Empirical results,” Evol. Comput., vol. 8, no. 2, control of rescue robots and machine learning, espe-
pp. 173–195, 2000. cially expensive multiobjective optimization.
[44] L. Dixon and G. Szego, “The global optimization problem: An introduc-
tion,” in Towards Global Optimization, vol. 2. Amsterdam, The Nether-
lands: North-Holland, 1978, pp. 1–15.

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.
ARIIZUMI et al.: MULTIOBJECTIVE OPTIMIZATION BASED ON EXPENSIVE ROBOTIC EXPERIMENTS UNDER HETEROSCEDASTIC NOISE 483

Howie Choset (M’92) received the Ph.D. degree in Fumitoshi Matsuno (M’94) received the Ph.D. (Dr.
mechanical engineering from California Institute of Eng.) degree in engineering from Osaka University,
Technology, Pasadena, CA, USA, in 1996. He is a Osaka, Japan, in 1986.
Professor of robotics with Carnegie Mellon Univer- In 1986, he joined the Department of Con-
sity, Pittsburgh, PA, USA. Motivated by applications trol Engineering, Osaka University. He became a
in confined spaces, he has created a comprehensive Lecturer and an Associate Professor in 1991 and
program in snake robots, which has led to basic re- 1992, respectively, in the Department of Systems En-
search in mechanism design, path planning, motion gineering, Kobe University. In 1996, he joined the
planning, and estimation. By pursuing the fundamen- Department of Computational Intelligence and Sys-
tals, this research program has made contributions to tems Science, Interdisciplinary Graduate School of
coverage tasks, dynamic climbing, and large-space Science and Engineering, Tokyo Institute of Tech-
mapping. He has already directly applied this body of work in challenging and nology, as an Associate Professor. In 2003, he became a Professor with the
strategically significant problems in diverse areas, such as surgery, manufactur- Department of Mechanical Engineering and Intelligent Systems, University of
ing, infrastructure inspection, and search and rescue. He directs the Undergrad- Electro-Communications. Since 2009, he has been a Professor with the Depart-
uate Robotics Minor Program at Carnegie Mellon University and teaches an ment of Mechanical Engineering and Science, Kyoto University, Kyoto, Japan.
overview course on robotics, which uses series of custom developed Lego Labs He is also a Vice President of NPO International Rescue Systems Institute and
to complement the course work. the Institute of Systems, Control and Information Engineers. His research in-
Prof. Choset’s students have received Best Paper Awards at the RIA in 1999 terests include robotics, control of distributed parameter systems and nonlinear
and ICRA in 2003; his group’s work has been nominated for best papers at ICRA systems, rescue support systems in fires and disasters, and swarm intelligence.
in 1997, IROS in 2003 and 2007, and CLAWAR in 2012 (Best Biorobotics Paper Dr. Matsuno has received many awards, including the Outstanding Paper
and Best Student Paper); they also received the best paper at IEEE Bio Rob in Award in 2001 and 2006, the Takeda Memorial Prize in 2001 from the Society of
2006, best video at ICRA 2011, and were nominated for the best video in ICRA Instrument and Control Engineers (SICE), and the Outstanding Paper Award in
2012. In 2002, the MIT Technology Review elected him as one of its top 100 2013 from the Information Processing Society of Japan. He is a Fellow Member
innovators under 35 in the world. In 2005, he authored a book titled Principles of the SICE, the Japan Society of Mechanical Engineers, the Robotics Society
of Robot Motion (MIT Press). of Japan, and a member of the ISCIE, among other organizations. He served as
the Cochair of the IEEE Robotics and Automation Society Technical Committee
on Safety, Security, and Rescue Robotics; the Chair of the Steering Committee
of the SICE Annual Conference; and is an Editor for Journal of Intelligent and
Robotic Systems, is an Associate Editor for Advanced Robotics and International
Journal of Control, Automation, and Systems among others, and is an Edito-
rial Board Member with the Conference Editorial Board of the IEEE Control
Systems Society.

Authorized licensed use limited to: Izmir Katip Celebi Univ. Downloaded on October 25,2023 at 08:17:45 UTC from IEEE Xplore. Restrictions apply.