Intelligent Optimization Principles Algorithms

And Applications Changhe Li download

https://ebookbell.com/product/intelligent-optimization-
principles-algorithms-and-applications-changhe-li-58439466

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Network Optimization In Intelligent Internet Of Things Applications

Principles And Challenges Payal Khurana Batra

https://ebookbell.com/product/network-optimization-in-intelligent-
internet-of-things-applications-principles-and-challenges-payal-
khurana-batra-58745762

Intelligent Optimization Modelling In Energy Forecasting Weichiang

Hong

https://ebookbell.com/product/intelligent-optimization-modelling-in-
energy-forecasting-weichiang-hong-55251802

Intelligent Optimization Of Mold Design And Process Parameters In

Injection Molding 1st Ed Mehdi Moayyedian

https://ebookbell.com/product/intelligent-optimization-of-mold-design-
and-process-parameters-in-injection-molding-1st-ed-mehdi-
moayyedian-7321250

Intelligent Optimization And Control Of Complex Metallurgical

Processes 1st Ed 2020 Min Wu

https://ebookbell.com/product/intelligent-optimization-and-control-of-
complex-metallurgical-processes-1st-ed-2020-min-wu-10806138
Configurable Intelligent Optimization Algorithm Design And Practice In
Manufacturing 1st Edition Fei Tao

https://ebookbell.com/product/configurable-intelligent-optimization-
algorithm-design-and-practice-in-manufacturing-1st-edition-fei-
tao-4931874

Learning And Intelligent Optimization 16th International Conference

Lion 16 Milos Island Greece June 510 2022 Revised Selected Papers
Dimitris E Simos

https://ebookbell.com/product/learning-and-intelligent-
optimization-16th-international-conference-lion-16-milos-island-
greece-june-510-2022-revised-selected-papers-dimitris-e-simos-49114524

Learning And Intelligent Optimization Second International Conference

Lion 2007 Ii Trento Italy December 812 2007 Selected Papers 1st
Edition Sameh Alshihabi

https://ebookbell.com/product/learning-and-intelligent-optimization-
second-international-conference-lion-2007-ii-trento-italy-
december-812-2007-selected-papers-1st-edition-sameh-alshihabi-2039740

Learning And Intelligent Optimization 5th International Conference

Lion 5 Rome Italy January 1721 2011 Selected Papers 1st Edition Olcay
Taner Yldz

https://ebookbell.com/product/learning-and-intelligent-
optimization-5th-international-conference-lion-5-rome-italy-
january-1721-2011-selected-papers-1st-edition-olcay-taner-yldz-2456114

Learning And Intelligent Optimization 15th International Conference

Lion 15 Athens Greece June 2025 2021 Revised Selected Papers Lecture
Notes In Computer Science 1st Ed 2021 Dimitris E Simos Editor

https://ebookbell.com/product/learning-and-intelligent-
optimization-15th-international-conference-lion-15-athens-greece-
june-2025-2021-revised-selected-papers-lecture-notes-in-computer-
science-1st-ed-2021-dimitris-e-simos-editor-37583830
Changhe Li · Shoufei Han
Sanyou Zeng · Shengxiang Yang

Intelligent
Optimization
Principles, Algorithms and
Applications
Intelligent Optimization
Changhe Li • Shoufei Han • Sanyou Zeng •
Shengxiang Yang

Intelligent Optimization
Principles, Algorithms and Applications
Changhe Li Shoufei Han
School of Artificial Intelligence School of Artificial Intelligence
Anhui University of Science Anhui University of Sciences
and Technology & Technology
Hefei, Anhui, China Hefei, Anhui, China

Sanyou Zeng Shengxiang Yang

School of Mechanical Engineering School of Computer Science
and Electronic Information and Informatics
China University of Geosciences De Montfort University
Wuhan, Hubei, China Leicester, UK

ISBN 978-981-97-3285-2 ISBN 978-981-97-3286-9 (eBook)

https://doi.org/10.1007/978-981-97-3286-9

Jointly published with China University of Geosciences Press, Hubei Province, P.R. China
The print edition is not for sale in Mainland China. Customers from Mainland China please order the
print book from: China University of Geosciences Press.

© China University of Geosciences Press 2024

This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or
information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publishers, the authors, and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publishers nor the authors
or the editors give a warranty, express or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publishers remain neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore

If disposing of this product, please recycle the paper.

To honor the memory of Lishan Kang for his
significant contribution to parallel computing
and evolutionary computation
Preface

Optimization problems widely exist in our world, manifesting themselves in various

domains, from the realms of science and engineering to our everyday routines. For
example, determining the ideal combination of breakfast foods, finding the most
efficient route to the office, organizing the optimal sequence for grocery shopping,
or managing room temperature to maximize comfort while minimizing energy
consumption, these scenarios all pose optimization questions that demand optimal
solutions. However, obtaining these optimal solutions can often be challenging, and
decision-making may not always come easily to us.
Generally speaking, optimization means the search for the best option among
different options. Broadly, the intelligence of intelligent optimization refers to
the usage of a priori and/or a posteriori knowledge about the problem, which is
expressed by implicit/explicit rules/models, to guide the search. More specifically,
it refers to the mechanism of probabilistic search based on these rules or models,
which are introduced in this book.
This book offers teaching and learning materials of intelligent optimization
regarding fundamental introduction, applications, and research topics for the fol-
lowing readers:
(1) Lecturers, graduate and undergraduate students. This book is devoted to
providing materials for beginning courses. It gives a thorough description of
intelligent optimization, including the fundamentals of optimization problems
and common intelligent optimization algorithms, especially evolutionary com-
putation algorithms.
(2) Users who wish to apply intelligent optimization methods to a particular
problem. This book also introduces applications to variant types of opti-
mization problems, such as the constrained optimization problem, the multi-
objective optimization problem, the dynamic optimization problem, the large-
scale optimization problem, the robust optimization problem, and the expensive
optimization problem. Particularly, we end this book with four real-world
applications, i.e., antenna design, vehicle routing, contamination source iden-
tification, and games.

vii
viii Preface

(3) Researchers in the domain of intelligent optimization. For each topic of this
book, we present the recent development and propose several challenging issues
that should be addressed in this domain. Potential solutions to these challenging
issues are also discussed.
The overall structure of this book contains four parts. Part I presents the
fundamentals of optimization from Chaps. 1 to 3. Part II presents evolutionary
computation algorithms from Chaps. 4 to 5. Part III presents optimization techniques
from Chaps. 6 to 8. Part IV presents advanced topics and applications. The four parts
are followed by the references with nearly 400 entries. We may have inevitably
missed some and please feel free to send us an email in case a really important one
is forgotten.
This book is distributed as part of the OFEC project, which is an open-source
project that aims to develop a common platform for students, users, and researchers
to quickly design and visualize their algorithms (see Appendix B). OFEC has
collected a plenty of algorithms and problems and is constantly maintained and
updated. The source code of the project and other materials of the book can be
obtained by following the link https://gitlab.com/ofec-release.

Hefei, China Changhe Li

May 2024
Acknowledgments

This book could not have been possible without the support of many people. The
authors thank the Intelligent Optimization and Learning team for their hard work on
the research. Furthermore, the authors thank Yong He, Min Wu, Chuanke Zhang,
Michalis Mavrovouniotis, Trung Thanh Nguyen, Carlos A. Coello Coello, Ming
Yang, Xin Yao, Kay Chen Tan, Boyang Qu, Jing Liang, Juan Zou, Qingfu Zhang,
Yew-Soon Ong, Jinliang Ding, Ponnuthurai Nagaratnam Suganthan, Miqing Li,
Juergen Branke, Xiaodong Li, Hui Wang, Aimin Zhou, Xinye Cai, Qunfeng Liu,
Ke Li, Yong Liu, Bob McKay, and Yuping Chen for their sincere support.
I would like to thank my wife, Fengxia Wang, for her endless love and care for
the family and children. I also would like to thank my parents and my parents-in-law
for their support and encouragement.
The genesis of this book is deeply rooted in the desire to honor the memory of
Lishan Kang. Lishan Kang made significant contributions to the fields of parallel
computing and evolutionary computation. His enduring scientific spirit serves as a
perpetual source of inspiration, motivating us to continually explore the frontiers of
knowledge.
The publication of this book was supported by the National Natural Science
Foundation of China under Grant 62076226, in part by the Hubei Provincial Natural
Science Foundation of China under Grant 2023AFA049.

ix
Contents

Part I Introduction and Fundamentals

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1 Optimization and Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Optimization Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.1 Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.2 Continuous Optimization Versus Discrete
Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Optimization Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.1 Deterministic Algorithms and Probabilistic
Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.2 Intelligent Optimization Techniques . . . . . . . . . . . . . . . . . . . . . . . 9
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1 Fitness Landscapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.1 Solution Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.2 Objective Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.3 Neighbourhood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.4 Global Optimum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.5 Local Optimum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Properties of Fitness Landscape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.1 Modality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.2 Ruggedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.3 Deceptiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.4 Neutrality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.5 Separability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2.6 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2.7 Domino Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.8 Property Control, Analysis, and Visualization . . . . . . . . . . . . 23

xi
xii Contents

2.3 Computational Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.3.1 Complexity Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.1.1 Time Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.1.2 Space Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.1.3 Ways of Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.1.4 Time Versus Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.2 P Versus NP Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3 Canonical Optimization Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.1 Numerical Optimization Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.1.1 Line Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.1.2 Steepest Descent Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.1.3 Newton Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.1.4 Conjugate Gradient Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.2 State Space Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.2.1 State Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2.1.1 The Shortest Path Problem . . . . . . . . . . . . . . . . . . . . 41
3.2.1.2 The Travelling Salesman Problem . . . . . . . . . . . . 42
3.2.2 Uninformed Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2.2.1 Breadth-First Search. . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2.2.2 Depth-First Search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.2.2.3 Depth-Limited Search . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.2.3 Informed Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.2.3.1 Greedy Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.2.3.2 A∗ Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.2.3.3 Monte-Carlo Tree Search . . . . . . . . . . . . . . . . . . . . . 52
3.3 Single-Solution-Based Random Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.3.1 Hill Climbing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.3.2 Simulated Annealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.3.3 Iterated Local Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.3.4 Variable Neighborhood Search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Part II Evolutionary Computation Algorithms

4 Basics of Evolutionary Computation Algorithms . . . . . . . . . . . . . . . . . . . . . . . 63
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.1.1 Biological Evolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.1.2 Origin of Evolutionary Algorithms . . . . . . . . . . . . . . . . . . . . . . . . 65
4.1.3 Basic Evolutionary Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.1.4 Developments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.1.5 Related Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.2 Solution Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.2.1 Binary Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.2.2 Integer Representation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Contents xiii

4.2.3 Real-Valued Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4.2.4 Tree Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.2.5 The Effect of Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.3 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.3.1 Parents Selection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.3.2 Survivor Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.3.3 Selection Pressure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.4 Reproduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.4.1 Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.4.2 Recombination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5 Popular Evolutionary Computation Algorithms . . . . . . . . . . . . . . . . . . . . . . . . 93
5.1 Genetic Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.1.1 Basic Principle and Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.1.2 Applications of Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . 95
5.2 Evolutionary Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.2.1 The Emergence of Evolutionary Programming . . . . . . . . . . . 96
5.2.2 The Classical Evolutionary Programming . . . . . . . . . . . . . . . . . 97
5.2.2.1 Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.2.2.2 Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.2.2.3 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.2.3 Framework and Parameter Settings . . . . . . . . . . . . . . . . . . . . . . . . 98
5.2.4 Recent Advances in Evolutionary Programming. . . . . . . . . . 99
5.3 Genetic Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.3.2 Genotype-Phenotype Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.3.2.1 Integer String Approach . . . . . . . . . . . . . . . . . . . . . . . 104
5.3.2.2 Gene Expression Programming . . . . . . . . . . . . . . . 105
5.3.3 Other Genome Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.3.3.1 Linear GP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.3.3.2 Graph-Based GP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.3.4 Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.3.4.1 Epistasis in GP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.3.4.2 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.3.4.3 All-or-Nothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.3.4.4 Non-functional Features of Algorithms . . . . . . 109
5.4 Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.4.1 The Rise of Particle Swarm Optimization . . . . . . . . . . . . . . . . . 110
5.4.2 Original Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . 111
5.4.3 Standard Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . 112
5.4.3.1 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.4.3.2 Population Topology . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.4.3.3 Confinement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.4.3.4 Velocity Updating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
xiv Contents

5.4.4 Recent Advances in Particle Swarm Optimization . . . . . . . . 114

5.4.4.1 Communication Topology . . . . . . . . . . . . . . . . . . . . 114
5.4.4.2 PSO with Diversity Maintenance . . . . . . . . . . . . . 115
5.4.4.3 Adaptive PSO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.4.4.4 Hybrid PSO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.5 Differential Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.5.1 Introduction of Differential Evolution . . . . . . . . . . . . . . . . . . . . . 116
5.5.1.1 Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.5.1.2 Crossover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.5.1.3 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.5.2 Framework and Parameter Settings . . . . . . . . . . . . . . . . . . . . . . . . 119
5.5.3 Some Advances in Differential Evolution . . . . . . . . . . . . . . . . . 120
5.5.3.1 Parameter Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.5.3.2 Adaptation of Mutation Operators . . . . . . . . . . . . 121
5.5.3.3 New Mutation Operators . . . . . . . . . . . . . . . . . . . . . . 121
5.6 Evolution Strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.6.1 Basic Evolution Strategy Paradigm . . . . . . . . . . . . . . . . . . . . . . . . 123
5.6.1.1 Parent Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.6.1.2 Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.6.1.3 Mutation Step Size Control . . . . . . . . . . . . . . . . . . . 124
5.6.1.4 Recombination. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.6.1.5 Survivor Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5.6.1.6 Canonical Self-Adaptation
Evolution Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5.6.2 Covariance Matrix Adaptation Evolution Strategy . . . . . . . 125
5.6.2.1 Reproduction with Covariance Matrix . . . . . . . 127
5.6.2.2 Covariance Matrix Adaptation . . . . . . . . . . . . . . . . 127
5.6.2.3 Complete Process of CMA-ES . . . . . . . . . . . . . . . 128
5.6.2.4 Niching CMA-ES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.7 Estimation of Distribution Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.7.1 Standard Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
5.7.2 Discrete Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5.7.2.1 Univariate Factorizations. . . . . . . . . . . . . . . . . . . . . . 132
5.7.2.2 Bivariate Factorizations . . . . . . . . . . . . . . . . . . . . . . . 133
5.7.2.3 Multivariate Factorizations. . . . . . . . . . . . . . . . . . . . 134
5.7.3 Continuous Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5.8 Ant Colony Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
5.8.1 Biological Inspiration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.8.2 ACO Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.8.2.1 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.8.2.2 Solution Construction . . . . . . . . . . . . . . . . . . . . . . . . . 137
5.8.2.3 Local Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5.8.2.4 Pheromone Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
5.8.3 ACO Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
5.8.3.1 Set of Update Solutions . . . . . . . . . . . . . . . . . . . . . . . 138
Contents xv

5.8.3.2 Pheromone Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

5.8.4 Recent Advances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5.8.4.1 Dynamic Optimization . . . . . . . . . . . . . . . . . . . . . . . . 141
5.8.4.2 Multi-objective Optimization . . . . . . . . . . . . . . . . . 141
5.8.4.3 Multimodal Optimization . . . . . . . . . . . . . . . . . . . . . 141
5.8.4.4 Parallel ACO Implementations . . . . . . . . . . . . . . . 141
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

Part III Optimization Techniques

6 Parameter Control and Policy Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
6.1 Parameter Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
6.1.1 Unary Parameter Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
6.1.1.1 Population Size Control . . . . . . . . . . . . . . . . . . . . . . . 150
6.1.1.2 Operator Parameter Control . . . . . . . . . . . . . . . . . . . 151
6.1.1.3 Problem-Related Parameter Control . . . . . . . . . . 151
6.1.2 Multi-parameter Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
6.1.3 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
6.2 Policy Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
6.2.1 Operator Selection Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
6.2.2 Hyper-heuristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
6.2.3 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
7 Exploitation Versus Exploration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
7.2 Exploitation and Exploration Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
7.2.1 Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
7.2.2 Single-Solution Meta-heuristics . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
7.2.3 Population-Based Meta-heuristics . . . . . . . . . . . . . . . . . . . . . . . . . 164
7.3 Enhancing Exploration and Exploitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
7.3.1 Exploration Enhancement Methods . . . . . . . . . . . . . . . . . . . . . . . 165
7.3.2 Exploitation Enhancement Methods . . . . . . . . . . . . . . . . . . . . . . . 167
7.4 Balancing Exploration and Exploitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
7.4.1 Explicit Differentiation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 167
7.4.2 Population Diversity-Driven Methods . . . . . . . . . . . . . . . . . . . . . 168
7.4.3 Non-overlapping Multi-population Methods . . . . . . . . . . . . . . 168
7.4.4 Space Partitioning-Based Methods . . . . . . . . . . . . . . . . . . . . . . . . 169
7.5 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
8 Multimodal Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
8.2 Niching Methods for Traditional EAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
8.2.1 Fitness Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
8.2.2 Restricted Tournament Selection . . . . . . . . . . . . . . . . . . . . . . . . . . 174
xvi Contents

8.2.3 Clearing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

8.2.4 Crowding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
8.3 Niching Methods of Emerging EAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
8.3.1 DE with Neighbor Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
8.3.2 PSO with Restricted Communication Topology . . . . . . . . . . 176
8.4 Other Evolutionary Niching Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
8.4.1 Multi-population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
8.4.2 Multi-objective Selection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
8.5 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

Part IV Advanced Topics and Applications

9 Multi-objective Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
9.1.1 Basic Concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
9.1.2 Properties of PF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
9.2 Multi-objective Evolutionary Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
9.2.1 Domination-Based Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
9.2.2 Indicator-Based Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
9.2.3 Decomposition-Based Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 187
9.3 Performance Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
9.3.1 C Indicator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
9.3.2 Generational Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
9.3.3 Maximum Spread . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
9.3.4 Spacing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
9.3.5 Inverted Generational Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
9.3.6 Hypervolume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
9.4 Visualization in the Objective Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
9.4.1 Visualization Using Original Values . . . . . . . . . . . . . . . . . . . . . . . 193
9.4.1.1 Scatterplot Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
9.4.1.2 Parallel Coordinate . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
9.4.1.3 Heat Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
9.4.2 Visualization Using Transformed Values . . . . . . . . . . . . . . . . . . 196
9.4.2.1 Radial Coordinate Visualization . . . . . . . . . . . . . . 196
9.4.2.2 Level Diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
9.4.2.3 Hyper-radial Visualization . . . . . . . . . . . . . . . . . . . . 197
9.4.3 Visualizing the Distribution Relation of Solution . . . . . . . . . 198
9.4.3.1 Distance and Distribution Charts . . . . . . . . . . . . . 198
9.4.3.2 Hyper-space Diagonal Counting . . . . . . . . . . . . . . 199
9.5 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
9.5.1 Many-objective Test Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
9.5.2 Many-Objective Optimization Algorithms . . . . . . . . . . . . . . . . 200
9.5.3 Performance Evaluation Methods . . . . . . . . . . . . . . . . . . . . . . . . . 200
9.5.4 Visualization of Many-Objective Optimization . . . . . . . . . . . 200
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Contents xvii

10 Constrained Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
10.2 Constraint-Handling Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
10.2.1 Penalty Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
10.2.1.1 Static Penalty Function . . . . . . . . . . . . . . . . . . . . . . . . 206
10.2.1.2 Dynamic Penalty Function . . . . . . . . . . . . . . . . . . . . 207
10.2.1.3 Adaptive Penalty Function . . . . . . . . . . . . . . . . . . . . 208
10.2.2 Separation of Objectives and Constraints . . . . . . . . . . . . . . . . . 208
10.2.2.1 Feasibility Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
10.2.2.2 ε-Constraint Methods . . . . . . . . . . . . . . . . . . . . . . . . . 209
10.2.2.3 Stochastic Ranking . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
10.2.2.4 Multi-objective Methods . . . . . . . . . . . . . . . . . . . . . . 211
10.2.3 Ensemble Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
10.3 Challenges and Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
11 Dynamic Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
11.1.1 Basics of Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
11.1.2 Characteristics of Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
11.2 Dynamism Handling Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
11.2.1 Difficulties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
11.2.2 Dynamism Handling Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
11.3 Benchmark Problems and Metrics for Dynamic
Single-Objective Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
11.3.1 Classical Problem Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
11.3.2 Performance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
11.4 Benchmark Problems and Metrics for Dynamic
Multi-objective Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
11.4.1 Classical Benchmark Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
11.4.2 Performance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
11.5 Dynamic Constrained Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
11.6 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
12 Robust Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
12.2 Robust Optimization Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
12.2.1 Uncertainties in Decision Variables. . . . . . . . . . . . . . . . . . . . . . . . 241
12.2.1.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
12.2.1.2 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
12.2.2 Uncertainties in Objective Function . . . . . . . . . . . . . . . . . . . . . . . 244
12.2.3 Uncertainties in Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
12.2.3.1 Min-Max Optimization Problem . . . . . . . . . . . . . 246
xviii Contents

12.2.3.2 Coevolution for Min-Max

Optimization Problem . . . . . . . . . . . . . . . . . . . . . . . . . 246
12.2.3.3 Limitation and Improvements. . . . . . . . . . . . . . . . . 247
12.2.4 Uncertainties Over Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
12.3 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
13 Large-Scale Global Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
13.1 Large-Scale Global Optimization Problems . . . . . . . . . . . . . . . . . . . . . . . . 253
13.2 Coevolution Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
13.2.1 Cooperative Coevolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
13.2.2 Static Grouping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
13.2.3 Dynamic Grouping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
13.2.3.1 Random Dynamic Grouping . . . . . . . . . . . . . . . . . . 256
13.2.3.2 Learning-Based Dynamic Grouping . . . . . . . . . . 256
13.3 Non-decomposition-Based Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
13.3.1 PSO-Based Algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
13.3.2 EDA-Based Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
13.3.3 DE-Based Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
13.4 Learning-Based Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
13.5 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
14 Expensive Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
14.1.1 Expensive Optimization Problems . . . . . . . . . . . . . . . . . . . . . . . . . 265
14.1.2 Surrogate-Assisted Evolutionary Algorithms . . . . . . . . . . . . . 266
14.2 Surrogate Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
14.2.1 Response Surface Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
14.2.2 Gaussian Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
14.2.3 Artificial Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
14.2.4 Radial Basis Function Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
14.2.5 Support Vector Machines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
14.2.6 Model Ensembles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
14.3 Model Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
14.3.1 Model Management in Offline SAEAs . . . . . . . . . . . . . . . . . . . . 272
14.3.2 Model Management in Online SAEAs . . . . . . . . . . . . . . . . . . . . 273
14.3.3 Infill Sampling Criterion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
14.4 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
15 Real-World Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
15.1 Antenna Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
15.1.1 Antenna Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
15.1.1.1 Frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
15.1.1.2 Frequency Bands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
Contents xix

15.1.1.3 Impedance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285

15.1.1.4 Radiation Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
15.1.1.5 Directivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
15.1.1.6 Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
15.1.1.7 Gain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
15.1.1.8 Beamwidths and Sidelobes. . . . . . . . . . . . . . . . . . . . 288
15.1.1.9 Voltage Standing Wave Ratio . . . . . . . . . . . . . . . . 289
15.1.1.10 Polarization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
15.1.2 Antenna Design Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
15.1.3 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
15.1.3.1 S-band Omnidirectional Antenna . . . . . . . . . . . . . 293
15.1.3.2 Circularly Polarized Antenna . . . . . . . . . . . . . . . . . 296
15.2 Vehicle Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
15.2.1 Basics of the Vehicle Routing Problem . . . . . . . . . . . . . . . . . . . . 299
15.2.1.1 Depot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
15.2.1.2 Customers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
15.2.1.3 Vehicles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
15.2.2 Variants of the Vehicle Routing Problem . . . . . . . . . . . . . . . . . . 300
15.2.2.1 Capacitated VRPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
15.2.2.2 VRPs with Time Windows . . . . . . . . . . . . . . . . . . . . 302
15.2.2.3 VRPs with Simultaneous Delivery
and Pickup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
15.2.2.4 VRPs with Uncertainties . . . . . . . . . . . . . . . . . . . . . . 302
15.2.3 Multi-objective VRP with Real-Time Traffic
Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
15.2.4 Algorithm Framework and Results . . . . . . . . . . . . . . . . . . . . . . . . 305
15.2.4.1 Solution Initialization . . . . . . . . . . . . . . . . . . . . . . . . . 306
15.2.4.2 Adaptive Local Search Strategy . . . . . . . . . . . . . . 307
15.2.4.3 Online Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
15.2.4.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 308
15.3 Contamination Source Identification in Water
Distribution Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
15.3.1 Contamination Source Identification Problem . . . . . . . . . . . . 310
15.3.1.1 Water Distribution System . . . . . . . . . . . . . . . . . . . . 311
15.3.1.2 Simulation-Optimization Model . . . . . . . . . . . . . . 311
15.3.2 Adaptive Multi-population Algorithm . . . . . . . . . . . . . . . . . . . . . 312
15.3.2.1 Generation of Multiple Populations . . . . . . . . . . 313
15.3.2.2 Cooperative Coevolutionary Search . . . . . . . . . . 313
15.3.2.3 Population Management . . . . . . . . . . . . . . . . . . . . . . 314
15.3.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
15.3.3.1 Experimental Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
15.3.3.2 Results and Discussions . . . . . . . . . . . . . . . . . . . . . . . 315
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
xx Contents

A Mathematical Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319

A.1 Mathematical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
A.1.1 Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
A.1.1.1 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
A.1.1.2 Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
A.1.1.3 Properties of Set Operations . . . . . . . . . . . . . . . . . . 320
A.1.1.4 Display Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
A.1.2 Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
A.1.2.1 Common Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
A.1.2.2 Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
A.1.2.3 The Limit of Function . . . . . . . . . . . . . . . . . . . . . . . . . 324
A.1.2.4 The Continuity of Function . . . . . . . . . . . . . . . . . . . 325
A.1.2.5 Derivative and Differentiation . . . . . . . . . . . . . . . . 326
A.1.2.6 Absolute Minimum and Maximum . . . . . . . . . . . 327
A.2 Mathematical Algebra. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
A.2.1 Vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
A.2.1.1 Vector Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
A.2.1.2 Dimension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
A.2.1.3 Systems of Linear Equations. . . . . . . . . . . . . . . . . . 331
A.2.2 Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
A.2.2.1 Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
A.2.2.2 Matrix of Linear System . . . . . . . . . . . . . . . . . . . . . . 334
A.3 Probability and Stochastics Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
A.3.1 Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
A.3.1.1 Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341

B Open Framework of Evolutionary Computation . . . . . . . . . . . . . . . . . . . . . . . 343

B.1 Features of OFEC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
B.1.1 Comprehensiveness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
B.1.2 Easy of Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
B.1.3 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344
B.2 Procedures of using OFEC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
B.2.1 Running OFEC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
B.2.2 Adding a Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346
B.2.2.1 Adding an Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 346
B.3 Components of OFEC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
B.3.1 Problem Component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
B.3.2 Algorithm Component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
B.4 Implemented Algorithms and Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
Acronyms

ABC artificial bee colony

ACO ant colony optimization
ACS ant colony system
AI artificial intelligence
ANN artificial neural network
APGA adaptive population size for genetic algorithm
AR axial ratio
AST abstract syntax tree
BOA Bayesian optimization algorithm
BFS breadth-first search
BWAS best-worst ant system
CBCC contribution-based cooperative co-evolution
CC cooperative co-evolution
cGA compact genetic algorithm
CEI constrained expected improvement
CEP classic evolutionary programming
CGP Cartesian genetic programming
COMIT combining optimizer with mutual information tree
CMA-ES covariance matrix adaptation evolution strategy
CMODE constrained multi-objective differential evolution
CMOP constrained multi-objective optimization problem
COP constrained optimization problem
CVRP capacitated vehicle routing problem
DCBG dynamic composition benchmark generator
DCMOEA dynamic constraint multi-objective genetic algorithm framework
DCMOP dynamic constrained multi-objective problem
DE differential evolution
DFS depth-first search
DI dependency identification
DOP dynamic optimization problem
DPDE co-evolutionary dual-population differential evolution

xxi
xxii Acronyms

DRPBG dynamic rotation peak benchmark generator

DTLZ Deb-Thiele-Laumanns-Zitzler benchmark
EA evolutionary algorithm
EAS elitism ant system
EC evolutionary computation
ECGA extended compact genetic algorithm
ECHT ensemble of constraint handling techniques
EDA estimation of distribution algorithm
EGNA estimation of Gaussian networks algorithm
EI expected improvement
EMNA estimation of multivariate normal algorithm
EOP expensive optimization problems
EP evolutionary programming
ES evolution strategy
FEP faster evolutionary programming
FER-PSO fitness Euclidean-distance ratio based particle swarm optimization
FIFO first-in-first-out
FSM finite-state-machine
GA genetic algorithm
GD generational distance
GDBG generalized dynamic benchmark generator
GEP gene expression programming
GL genetic learning
GLT Gu-Liu-Tan benchmark
GOP global optimization problem
GP Gaussian processes
GP genetic programming
GPM genotype-phenotype mapping
HV hypervolume
IBEA indicator-based evolutionary algorithm
IDEA iterated density evolutionary algorithm
IFEP improved faster evolutionary programming
IGD inverted generational distance
IPSOLS incremental particle swarm optimizer with local search
LCB lower confidence bound
LGP linear genetic programming
LIPS locally informed particle swarm model
LSGO large-scale global optimization
MaOP many-objective optimization problem
MIMIC mutual-information-maximization for input clustering
MLCC multilevel cooperative co-evolution
MLP multi-layer perception
MMAS max-min ant system
MMOP multimodal optimization problem
MOEA multi-objective optimization evolutionary algorithm
Acronyms xxiii

MOEA/D multi-objective optimization algorithm based on decomposition

MOP multi-objective optimization problem
MOVRPRTC multi-objective vehicle routing problem with real-time traffic con-
dition
MPB moving peak benchmark
MS maximum spread
MSE mean square error
NAS neural architecture search
NEA niching based evolutionary algorithm
NP non-deterministic polynomial time
NSGA-II non-dominated sorting genetic algorithm
PAES Pareto archived evolution strategy
PBI penalty-based boundary intersection
PBIL population-based incremental learning
PESA Pareto envelope-based selection algorithm
PF Pareto optimal front
PoI probability of improvement
PR polynomial regression
PRAM probabilistic rule-driven adaptive model
PS Pareto optimal set
PSO particle swarm optimization
RBF radial basis function model
RL reinforcement learning
RSM response surface method
RTS restricted tournament selection
SaDE self-adaptive differential evolution
SA simulated annealing
SAEA surrogate-assisted evolutionary algorithm
SGA standard genetic algorithm
SI swarm intelligence
SLL side lobe level
SLPSO self-learning particle swarm optimization
SPEA strength Pareto evolutionary algorithm
SPSO standard particle swarm optimization
SVM support vector machine
TCH Tchebycheff
TSP travelling salesman problem
UMDA univariate marginal distribution algorithm
VNS variable neighborhood search
VRP vehicle routing problem
VRPDP vehicle routing problem with simultaneous delivery and pickup
VRPTW vehicle routing problem with time window
WDS water distribution system
WFG walking fish group
WS weighted sum
ZDT Zitzler-Deb-Thiele benchmark
 Part I
Introduction and Fundamentals

This part introduces the basics of optimization, including the relationship between
optimization and machine learning, categories of optimization problems and opti-
mization algorithms, the fitness landscape, and computational complexity. Several
typical canonical optimization algorithms, i.e., numerical optimization algorithms,
state space search algorithms, and single-solution-based random search algorithms,
are introduced.
Chapter 1
Introduction

Abstract This chapter briefly introduces the background of optimization. Firstly,

the relationship between optimization and machine learning is discussed, and an
example of machine learning task is given. Secondly, the mathematical formulation
of an optimization problem is defined. Optimization problems are categorized into
continuous optimization problems and discrete optimization problems. Finally, opti-
mization algorithms are introduced and categorized into deterministic algorithms
and probabilistic algorithms, where several terms regarding intelligent optimization
are introduced.

1.1 Optimization and Machine Learning

Artificial intelligence (AI), which was coined in 1956, is a wide-ranging branch

of technology that mimics human intelligence to execute tasks. AI is an inter-
disciplinary science with multiple approaches in mathematics, computer science,
biology, neuroscience, psychology, sociology, linguistics, philosophy, and more.
Recent advances in machine learning, especially deep learning, has been success-
fully applied in games, transportation, medical treatment, industry, etc.
Maybe you have seen applications of machine learning somewhere in your
daily routine. Actually, it has been around for a long time in so many scenarios.
For example, when you find what you want on the Internet, it helps the search
engine judge which search results (advertisements at the same time) are suitable
for you; you do not need to label all spammers in your mailbox by hand since
it has helped you do most of the work (if you have never checked your spam
box, then check it, and you will find something you have never noticed); your
smartphone automatically identifies and ignores spam calling from auto-dialers; and
if you go to YouTube or some other video-sharing platforms, you can find that they
suggest something you are interested in and even automatically generates caption for
some videos. In addition, whenever you use facial-recognition payment and speech
recognition input, you are enjoying the benefits of machine learning.

© China University of Geosciences Press 2024 3

C. Li et al., Intelligent Optimization,
https://doi.org/10.1007/978-981-97-3286-9_1
4 1 Introduction

Fig. 1.1 Chirps per minute 35

as temperature changes
30

Temperature in Celsius
25

20

15

10

0
0 25 50 75 100 125 150 175
Cricket chirps per minute

Generally speaking, the target of application of a machine learning method is

to create a model to reflect the relationship between input features (weather for
tomorrow, all the pixels of your photo) and desired output (the rate of students being
late for the first class, who you are); then we can use this model to predict the result
according to new input features.
However, have you ever thought about how machine learning methods can do
such things? To answer this question, we have to get deeper into the inner world of
machine learning. Do not worry, we just need to take maybe the simplest machine
learning problem, a linear regression problem, for example.
Figure 1.1 shows the number of chirps of crickets per minute at different
temperatures in Celsius;1 one can say that the relationship is linear.2 Of course,
we can use an equation to represent this relationship

. y ' = wx + b (1.1)

where .y ' means temperature in Celsius, the label we predict; x means the number
of chirps of crickets per minute, the input feature;w is the slope of the line or the
weight of the feature; and b is y -intercept or the bias.
This equation is the model we need to build. w and b are its model parameters,
and now, our job is to find the right values for them. In supervised learning, we build
our model by examining a set of examples and trying to find a model that minimizes
loss. So, how to evaluate the loss? A simple and popular way is to use mean square
error (MSE). MSE is the average squared loss per example over the entire dataset.

1 https://developers.google.com/machine-learning/crash-course/descending-into-ml/linear-

regression
2 https://en.wikipedia.org/wiki/Dolbear%27s_law
1.2 Optimization Problems 5

To calculate MSE, we sum up all squared losses for each example and then divide
by the number of examples.

1 ⎲
MSE =
. (y − prediction(x))2 (1.2)
|D|
(x,y)∈D

where .D is a dataset containing labeled examples and .prediction(x) is a function

of the weights and bias with the input of x.
If we replace .prediciton(x) with our linear equation of Eq. (1.1), we get

1 ⎲
MSE =
. (y − wx − b)2 (1.3)
|D|
(x,y)∈D

Now we can examine this equation, all the x and y are known for us, and now,
the problem is very explicit to find the right values of w and b, which make MSE
minimal. The final problem here is a pure optimization problem. Once we solve this
problem, we can get the parameters needed in our model in Eq. (1.1), i.e., we have
built the model.
Generally speaking, as we have seen in the previous example, optimization lies
at the heart of most machine learning algorithms [1]. No matter how complex the
problems look, most machine learning problems can be reduced to optimization
problems. The modeler formulates the problem by selecting an appropriate family
of models. Then the model is typically trained by solving an optimization problem
that optimizes the parameters of the model given a selected loss function.

1.2 Optimization Problems

Optimization is very important for decision-makers to make good decisions before

taking actions. It exists widely in science and engineering as long as there are
multiple choices of solutions. To solve a real-world problem, the first step is to
establish a mathematical model to describe the problem.

1.2.1 Mathematical Formulation

In general, to find the best solution of an optimization problem, we first need to

construct a mathematical model to describe the problem. Normally, modelling an
optimization problem needs to identify three components, which are objectives,
variables, and constraints. They can be formulated as

. min or max fi (x), i = 1, · · · , M (1.4)

x x
6 1 Introduction

subject to

gj (x) ≤ 0,
. j = 1, · · · , N
hk (x) = 0, k = 1, · · · , P

where .fi (x) are objective functions to be minimized or maximized; they are
measures to quantify the performance of a solution.
In manufacturing, we want to maximize profit and minimize production cost,
while in data fitting, we want to minimize the difference error between observed
data and predicted results like Eq. (1.3). .x are variables represented for a solution.
The aim is to find solutions that have the best performance in terms of the objective
functions. In manufacturing, they may be the amount of each resource consumed
in production process, whereas in data fitting, they may be the parameters of a
model like the weights and bias defined in Eq. (1.3). .gj (x) ≤ 0 and .hk (x) = 0 are
called inequality constraints and equality constraints, respectively. They describe the
relationships among variables and define the possible range of values for variables.
A solution is feasible only if it satisfies all constraints; otherwise, the solution is
infeasible.
The term minimization problem refers to minimizing the objective function,
whereas the maximization problem refers to maximizing the objective function,
which can be defined as negating the minimized objective function. If both .N and
P are zero, the problem is an unconstrained optimization problem.
If M is larger than one, which means that there are more than one objectives
that need to be optimized synchronously, then the problem is called multi-objective
optimization problem. A simple example of a multi-objective optimization problem
is building a computer: we need to use limited money to balance performance,
appearance, comfort, and other conflicting indicators; the variables are configura-
tions of hardware components; the constraints may be the limited budget, or we
only want a 32-inch display.

1.2.2 Continuous Optimization Versus Discrete Optimization

The optimization problem can be divided into two types according to the type of
input: continuous optimization and discrete optimization. Here is an example to
help us understand continuous optimization and discrete optimization.
A chemical company has two factories .F1 and .F2 and three retailers .R1 , R2 , R3 .
Each factory .Fi produces .ai tons of a certain chemical product each week; each
retailer .Rj demands .bj tons of the product weekly. .cij is the cost of shipping one
ton of the product from factory .Fi to retailer .Rj . Figure 1.2 shows the problem.
The problem is to determine how much of the product to ship from each factory to
all retailers to satisfy all the requirements and minimize the total cost. The variables
1.2 Optimization Problems 7

Fig. 1.2 A transportation c11x11

problem R1 b1
c12x12
a1 F1
c13x13
c21x21 R2 b2
c22x22
a2 F2 c23x23
R3 b3

of the problem are .xij , .i = 1, 2, j = 1, 2, 3, where .xij is the amount of the product
shipped from .Fi to .Rj . The problem can be formulated as
⎲
. min cij xij (1.5)
ij

subject to

⎲
3
. xij ≤ ai , i = 1, 2
j =1

⎲
2
xij ≥ bj , j = 1, 2, 3
i=1

xij ≥ 0, i = 1, 2, j = 1, 2, 3

This is a continuous optimization problem, since the variables used in the

objective function are continuous variables, i.e., .xij to be chosen from real values
normally within a range.
But in some optimization problems, the variables make sense only if they
take integer values or binary values. Suppose that in the transportation problem
mentioned above, the factories produce cars rather than chemicals. In this case, .xij
would represent integers (i.e., the number of cars shipped) rather than real numbers.
We call this type of problems as discrete optimization problems; particularly, in this
example, the problem is also known as integer programming problems. For some
discrete optimization problems, discrete variables mean a set of objects to be chosen
or combinatorial structures, such as combinations, assignments, schedules, and
sequences. These problems are also known as combinatorial optimization problems.
Some problems contain both continuous and discrete variables. These problems
are referred to mixed-integer programming problems. For example, those factories
not only produce chemical materials weighed by real values but also produce
chemical equipment, which must be counted by integers. The problem turns into
a mixed-integer programming problem.
8 1 Introduction

1.3 Optimization Algorithms

Generally, optimization algorithms can be categorized into deterministic algorithms

and probabilistic algorithms in terms of whether random sources are used.

1.3.1 Deterministic Algorithms and Probabilistic Algorithms

A deterministic algorithm refers to an algorithm that will always produce the same
output given a particular input. It is often used in a problem where the objective
function is clear and the solution can be efficiently searched with a direction-guided
strategy. For example, the gradient information is normally used as the search
direction for continuous optimization problems that are smooth and particularly
differentiable. However, when the relationship between a solution and its objective
value is not obvious or calculating its objective value is very time-consuming or
the problem has multiple objectives or all possible solutions cannot be examined
by each due to the large number, particularly for problems where the number of
possible solutions increases exponentially as the number of variables increases, then
probabilistic algorithms come into play.
A probabilistic optimization algorithm refers to an algorithm that takes random
inputs during the search process, which are typically uniformly and independently
distributed. A typical characteristic of the probabilistic optimization algorithm is
that the algorithm may produce different outputs even when given the same input
due to randomness. However, we can get a stable output as long as the randomization
process is pseudo-random, which is usually driven by a random seed generator. The
randomness is typically used to help the algorithm discover new solutions and makes
the algorithm less sensitive to modeling errors. The algorithm normally produces a
satisfactory solution within an affordable amount of time. Furthermore, the injected
randomness can help the algorithm avoid trapping into local optima and eventually
find a global optimum.
Indeed, this randomization principle is known to be a simple yet effective way
to achieve good performance uniformly across many sorts of problems, particularly
for complex problems mentioned above. The probabilistic algorithm belongs to the
family of Monte Carlo method devised in the 1930s. Now, it has become one of
the most important research branches of optimization since it was proposed. Due
to the successful application of these intelligent optimization algorithms for solving
complicated optimization problems where deterministic algorithms fail, a variety of
probabilistic algorithms have been implemented in some commercial software, such
1.3 Optimization Algorithms 9

as TOMLAB,3 MIDACO,4 and IOSO,5 Most of the methods we will introduce in

the rest of this book are probabilistic optimization algorithms.

1.3.2 Intelligent Optimization Techniques

Intelligent optimization techniques have developed from the simple low-level

heuristic to the complex high-level metaheuristic and to the recent autonomous
search of hyper-heuristics.
A heuristic [4, 5] is a part of an optimization algorithm that uses the information
gathered by the algorithm so far to help decide which search direction to take
next. This is normally achieved by designing a heuristic function, which is usually
problem dependent. A heuristic function takes the goal into account and hence
provides a search direction for an algorithm. A heuristic can accelerate the search
process, which is very important for complex optimization problems, especially for
NP-hard problems that make it the only viable option in real-world scenarios.
For instance, if you are trying to solve the travelling salesman problem (TSP),
which is “Given a list of cities and the distances between each pair of cities, what is
the shortest route that visits each city and returns to the origin city?” Suppose now
that you are in a city, then your heuristic could be “next take the closest unvisited
city to the current city where you are.” Usually, this provides a very quick solution;
however, it can easily get stuck at the local optima since it does not examine all
possible choices from a global point of view.
As mentioned already, ◯ 1 heuristics use domain-specific knowledge and there-
fore cannot be easily used to solve problems that belong to other domains; ◯ 2
heuristics can easily get stuck in the local optima. Meta-heuristics aim to address
both of these two drawbacks. A metaheuristic is an algorithmic framework with
guidelines for designing heuristic optimization algorithms for solving very general
classes of problems. Generally, it combines heuristic optimization techniques with
randomness within its framework in an abstract and, hopefully, efficient way,
usually without utilizing deeper insight into their structures, i.e., by treating them
as black-box procedures [3]. Metaheuristic optimization algorithms can produce
good-enough solutions within a small enough computational time. They can often
take a good trade-off between computing time and solution quality. There are a
wide variety of metaheuristics developed for optimization problems of different
domains, which can be divided into single-solution-based methods and population-
based methods. Single-solution-based methods include the hill climbing, simulated
annealing, iterated local search, and variable neighborhood search. Population-

3 https://tomopt.com/tomlab/products/midaco/
4 http://www.midaco-solver.com/
5 http://iosotech.com/index.htm
10 1 Introduction

based metaheuristics include genetic algorithms, differential evolution, particle

swarm optimization, ant colony optimization, etc.
Experimental studies have shown that different metaheuristics perform differ-
ently on different types of problems. In addition, for the same type of problem,
different metaheuristics perform differently on different instances. Even different
metaheuristics perform differently at different stages of the search process on the
same instance. Unlike metaheuristics, a hyper-heuristic [2] framework aims to
search for a right heuristic or a sequence of heuristics within a search space of
heuristics. The idea is to automatically devise algorithms on low-level heuristics or
heuristic components in a way of selecting or generating heuristics with the purpose
of combining the strength and compensating for the weakness of known heuristics
(see Sect. 6.2.2 for more details).

References

1. Bennett, K.P., Parrado-Hernández, E.: The interplay of optimization and machine learning
research. J. Mach. Learn. Res. 7(46), 1265–1281 (2006)
2. Cowling, P., Kendall, G., Soubeiga, E.: A hyperheuristic approach to scheduling a sales summit.
In: International Conference on the Practice and Theory of Automated Timetabling, pp. 176–
190. Springer, Berlin (2000)
3. Glover, F.W., Kochenberger, G.A.: Handbook of Metaheuristics. Kluwer, Dordrecht (2003)
4. Michalewicz, Z., Fogel, D.B.: How to Solve It: Modern Heuristics. Springer Science & Business
Media, Berlin (2000)
5. Pearl, J.: Heuristics: Intelligent Search Strategies for Computer Problem Solving. Addison
Wesley, Reading (1984)
Chapter 2
Fundamentals

Abstract This chapter introduces the fundamentals of optimization. These funda-

mentals include the definition of fitness landscapes, properties of fitness landscapes,
and computational complexity. They help us more profoundly understand optimiza-
tion, especially in understanding the difficulties in solving optimization problems.
They will lay the foundation for learning and designing optimization methods.

2.1 Fitness Landscapes

The concept of fitness landscapes, originating from theoretical biology more than 90
years ago [15], has played an important role in optimization. It is a valuable tool for
analyzing the optimization process, especially for studying the relationship between
the performance of optimization algorithms and the characteristics of optimization
problems [12]. In practice, it attempts to draw an analogy with real landscapes to
gain a better understanding of how and where algorithms operate [7].
A fitness landscape is a mapping from every possible solution, which represents
a candidate solution of a problem to be solved, into a fitness value. More formally,
a fitness landscape consists of three main components: a set of solutions, a fitness
function that evaluates these solutions, and a neighborhood relationship between
the solutions (i.e., the definition of distance between solutions). The fitness function
determines how good a solution is in terms of the objective value. It is the
driving force behind the search. Neighborhood relations are crucial in the analysis
of the fitness landscape. Typically, the shape of a fitness landscape depends on
the representation/encoding (see Sect. 4.2) of solutions and the chosen distance
measures between solutions for a given problem. That is, a particular problem can
have different fitness landscapes. For example, the landscape of real-valued and
binary encoding for a continuous optimization problem varies significantly.
More specially, a fitness landscape describes a surface of fitness values over the
solution space where optimization methods are searched. When an optimization
problem is a minimization problem, searching for an optimal solution to the
problem can be interpreted as walking on the surface of its fitness landscape toward
the lowest valleys, with the ability to overcome other valleys [3]. Therefore, the

© China University of Geosciences Press 2024 11

C. Li et al., Intelligent Optimization,
https://doi.org/10.1007/978-981-97-3286-9_2
12 2 Fundamentals

10
20.0
8 15.0

10.0
6
5.0
4
0.0
0
2 2
4 0
6 2
2 8 6 4
1 4 7 10 13 16 8
10
10
1

(a) One dimensional problem (b) Two dimensional problem

Fig. 2.1 Fitness landscapes of one-dimensional (a) and two-dimensional (b) continuous optimiza-
tion problems

fitness landscape has been found to be effective for analyzing the behavior of the
optimization method in an optimization process and can help design an effective
optimization method. Figure 2.1 shows the fitness landscapes of two continuous
optimization problems in one dimension and two dimensions, respectively.

2.1.1 Solution Space

The first question when considering using an optimization method to solve a

problem is: What space does the optimization method to search in? The solution
space, also called decision space or search space, consists of all possible solutions
that connect with each other by a user-defined distance measure. For a practical
example, if we want to minimize the cost of manufacturing a car, all possible
solutions might represent all alternative ways the manufacturing process could be
configured. In this case, these solutions comprise the solution space.
We call the union of all possible solutions .x of an optimization problem as its
solution space .X. The representation of .x could be a vector of bit strings, real
values, integers, or even more complex structures depending on the problem and
the algorithm to be used. The solution space can be represented as

X = {x 1 , x 2 , · · · }
. (2.1)

Normally, the solution space is uncountably infinite for continuous optimization

problems and countably finite for discrete optimization problems. Note that a
countably finite solution space does not mean that it is easier for an algorithm to
search than an uncountably infinite solution space. For example, we consider the
TSP as a hard problem since there does not exist such an algorithm so far that
2.1 Fitness Landscapes 13

Fig. 2.2 Infeasible and Infeasible region

feasible regions of a
constrained optimization
problem

Feasible region

can solve it in polynomial time. However, the continuous sphere function with
an uncountably infinite solution space can be easily solved by a gradient descent
method since it is convex and unimodal.
For a constrained optimization problem, only parts of the solution space are
feasible (see the feasible region in Fig. 2.2). A solution is feasible only if it satisfies
all constraints, i.e., it is within the feasible region of the solution space; otherwise,
the solution is infeasible, i.e., it is within the infeasible region in the solution space.
Infeasible solutions cannot be deployed. We need to find the feasible solution with
the best fitness value (see Chap. 10 for methods for finding feasible solutions).

2.1.2 Objective Space

As we know, the main objective of optimization methods is to find a promising

solution (or a set of promising solutions; see the multimodal optimization in
Chap. 8 and multi-objective optimization in Chap. 9) as possible from the solution
space. Generally, we are interested in the solution with the best fitness value. This
evaluation is carried out by an objective function (or a set of objective functions in
the multi-objective optimization scenario).
The objective values corresponding to all possible solutions comprise the
objective space. The objective space .F can be represented as

F = {f (x 1 ), f (x 2 ), · · · }
. (2.2)

where f is the objective functions.

For unconstrained optimization problems, the objective function often is the only
guide for an algorithm to search in the search space. A solution’s fitness value is
normally positively correlated to its objective value for maximization problems.
Note that a surrogate model may be needed to describe the relationship between
the input .x and output f (see Chap. 14) in a situation where the relationship is
vague or the objective evaluation is time-consuming, especially for problems in the
14 2 Fundamentals

real world. For optimization problems with more than one objective function, the
comparison between solutions becomes complicated. Therefore, special methods
are needed to deal with the comparison difficulty (see Chap. 9). For constrained
optimization problems, the comparison between solutions is also complicated
since both objective values and constraint values need to be taken into account.
Constraint-handling techniques are normally needed to deal with constraints (see
Chap. 10) since solutions that violate any constraint are infeasible.

2.1.3 Neighbourhood

As a basic concept in topological space, a neighborhood of a solution .x is a set of

solutions containing that point where one can move any direction away from that
solution without leaving the set. Neighborhood solutions of .x are normally close to
.x in some sense because they share a certain amount of structure and can be easily

obtained by a neighborhood generating function with input of .x. The neighborhood

of a solution .x is often written as

Nτ (x) = {p|p ∈ X, p /= x, d(x, p) ≤ τ }

. (2.3)

where d is a distance function and .τ > 0 is the threshold of a neighborhood.

The neighborhood topology is used to define or describe the arrangement of
the elements of a communication network for a set of solutions. Some typical
topologies are ring network, global network, random network, and star network (see
the examples in Sect. 5.5). The neighborhood relations among solutions are often
defined on the basis of the distance function. Many distance measurement methods
can be used, such as Manhattan distance, Euclidean distance, Chebyshev distance,
Hamming distance, etc.
From the perspective of a search algorithm, the neighborhood topology depends
not only on the distance function with a given solution representation but also on
the way an algorithm searches in the solution space. Given a starting solution in
the search space, if the search of the algorithm follows a particular direction, the
algorithm cannot reach the space that is not in the search direction even if it is very
close to the given solution in terms of the distance function.

2.1.4 Global Optimum

For an optimization problem, a global optimum is an optimum of the whole solution

space. A global optimum is a solution that has the best objective value in the search
space, as shown in Fig. 2.3.

x ∗ is a global optimal solution, if ∀x ∈ X, f (x ∗ ) ≤ f (x)

. (2.4)
2.1 Fitness Landscapes 15

Global optimum

Fig. 2.3 The global optimum for a single objective minimization problem

Basin of attraction Local optima

(a) (b)

Fig. 2.4 Illustration for basin of attraction (a) and local optima (b)

Normally, the distribution and the number of optimal solutions of a problem are
strongly dependent on the type of the problem. In single-objective optimization
problems, there is one global optimum in general or multiple global optima in the
scenario of multimodal optimization (see Chap. 8). In multi-objective optimization
problems, there is a set of optimal solutions (widely known as Pareto-optimal
solutions; see Chap. 10), instead of a single optimal solution.

2.1.5 Local Optimum

Before introducing the definition of the local optimum, we introduce the concept of
the basin of attraction. The basin of attraction of an optimum is the region of all its
neighborhood solutions whose objective values change monotonically as they move
closer to the optimum, as shown in Fig. 2.4a.
16 2 Fundamentals

A local optimum is a solution that has the best fitness value within its neighbor-
hood in the search space, as shown in Fig. 2.4b.

x is a local optimal solution, if ∀p ∈ Nτ (x), f (x) ≤ f (p)

. (2.5)

In some interesting cases, the local optimum is not a single point but a set of
points on ridges or plateaus with the same objective value. The shape of ridges and
plateaus can be linearly or non-linearly distributed in the fitness landscape.
To find the global optimum of a problem is not an easy task due to many factors,
such as a huge number of local optima, comparison difficulty between solutions in
constrained or multi-objective situations, dynamic environments, etc. To design an
efficient optimization algorithm, we should ensure that the algorithm can converge
to the region of global optimum, i.e., carry out exploitation for local optima. To
achieve the objective, the algorithm should be able to avoid trapping into local
optima and continue to explore promising areas that are not exploited, i.e., carry out
exploration for global optima (see the balance between exploration and exploitation
in Chap. 7).

2.2 Properties of Fitness Landscape

The fitness landscape has played an important role in the theory of complex systems.
It has been found to be effective in optimization theory, shown to be powerful
for understanding the behavior of optimization algorithms, and to help improve
their performance. Therefore, by analyzing the structure of the underlying fitness
landscape, the behavior of optimization methods can be understood, and the design
of an effective algorithm becomes easier.
The structure of the fitness landscape significantly impacts the performance of
an optimization algorithm. Several properties associated with the structure of fitness
landscapes [8] will be discussed in detail as follows.

2.2.1 Modality

During landscape analysis in evolutionary theory, there has always been a tendency
to focus on the relationship between the problem difficulty and the landscape
modality. Generally speaking, more modality of the fitness landscape means more
difficulty of the problem. This is because modality biases search algorithms, as more
modality implies more candidate solutions with the same objective value, making it
more challenging for algorithms to determine the direction to be taken.
A landscape that has only one optimum is commonly called unimodal, as shown
in Fig. 2.5. On the contrary, a landscape with more than one optimum is known as
multimodal, as shown in Fig. 2.6. To some extent, the number of local optima in a
2.2 Properties of Fitness Landscape 17

140
1400

120
1200

100
1000

80
800

60
600

40
400

20
200

20
15
10 15 20
5 10
20 0 5
15 10 5 0
5 0 10 5
5 0 55 10 15 10
10 15 20 20 15

Fig. 2.5 An example for a 2-D unimodal shifted sphere function

400
40000

300
30000

200
20000

100
10000

00
3
3
2
2
1
1
0
0
3 1
3 2 1
2 11 0 2
1 2
2 33
3

Fig. 2.6 An example for a 2-D multimodal Schwefel’s function

18 2 Fundamentals

landscape significantly impacts the difficulty in finding the global optimum. A local
optimum is a point or region (a set of interconnected points with equal fitness) with
a fitness function value greater than those of all its nearest neighbors. Obviously, if
there are many local optima, it is difficult for a search algorithm to find all optima or
the global optimum since the algorithm needs to jump out many valleys, i.e., points,
which are in the basin of attraction of local optima, should cross all peaks along the
direction to the global optimum as shown in Fig. 2.4b. Therefore, it is a challenging
issue for search algorithms to address.
The number of optima does indeed have a significant impact on landscape
difficulty. However, for insightful analysis of landscape difficulty, it is crucial
to investigate the distribution, size, shape, and depth (or height) of the basins
of attraction. These factors may be more important in determining the level of
landscape difficulty. A landscape is deemed difficult when local optima possess
large basins of attraction, while the global optimum has a smaller basin of attraction.
An extreme scenario occurs when the global optimum is an isolated spike in a
plateau. Optima with basins of attraction of asymmetrical shapes are generally more
difficult to navigate than those with symmetrical shapes.
An objective-driven search algorithm inherently favors exploring local regions
that are easy to search. Solutions in such regions typically exhibit faster and larger
improvements in terms of the objective value compared to those in harder-to-search
regions. Consequently, solutions in the hard-search region are gradually removed
due to their poor fitness, leading the algorithm to converge in the easiest-to-search
region. However, if the easy-search region only contains local optima, the algorithm
is susceptible to premature convergence, wherein it becomes stuck in a local
optimum. Therefore, the modality of the fitness landscape presents fundamental
challenges for algorithms aiming to locate the global optimum when the search
space comprises local regions with varying levels of search difficulties.

2.2.2 Ruggedness

The search often takes place in a rugged landscape, which has a profound effect on
the problem difficulty and search speed of an algorithm. A rugged landscape can be
regarded as a multi-modality landscape with many sharp descents and ascends.
In simple cases, optimization methods can effectively track optimum solutions
using gradient information from fitness landscapes, particularly for objective func-
tions with continuous and low total variation. However, when objective functions
exhibit instability or frequent oscillations, as depicted in Fig. 2.7, it becomes more
challenging to determine the correct directions to proceed during the optimization
process. The ruggedness of these functions directly correlates with their difficulty
to optimize, with more rugged functions presenting greater challenges.
In short, ruggedness has to do with the level of variation in fitness values in
a fitness landscape, where the fitness values of neighborhood solutions have very
different fitness values. An extreme case is the needle-in-a-haystack problem, where
2.2 Properties of Fitness Landscape 19

3.0
3.0

2.5
2.5

2.0
2.0

1.5
1.5

1.0
1.0

0.5
0.5

40
40
20
20
40 0
40 20 0
20
00 20
20
20
20 40
40
40
40

Fig. 2.7 An example for ruggedness for a 2-D shifted Griewank function

each peak is like a needle without basin of attraction. The opposite of a very rugged
landscape would be a landscape with a single large basin/peak of attraction or a flat
landscape with no features. For the rugged landscape, a search algorithm is hardly
to get knowledge of where to go, causing the algorithm to be difficult to converge.

2.2.3 Deceptiveness

A typical fitness landscape should not only provide sufficient information to guide
the search for an algorithm, but the information should also guide the search in the
right direction, i.e., the direction toward the global optimum. In contrast, a fitness
landscape guides the optimization process far away from the optimum (and thus
also in the wrong direction), which is sometimes known as fitness deceptiveness
as shown in Fig. 2.8. The gradient of deceptive fitness landscapes usually leads the
search away from the global optimum.
Deceptiveness is intricately tied to the structure of optima within a landscape.
The relative positions of local optima to the global optimum and the presence of
isolation significantly influence the level of deceptiveness. For example, when an
algorithm discovers a region with superior fitness compared to others, it tends to
exploit this area under the assumption that it likely contains the global optimum.
However, in reality, the optimal solution may be distant from the currently searched
area [6]. Interestingly, certain optimization algorithms demonstrate an advantage
in solving problems with high levels of deceptiveness [2], whereas increased
20 2 Fundamentals

200
2000

150
1500

100
1000

50
500

6
4 6
2 4
0 2
44 22 2 0
0
0 2 4 2
2 44
6 4
6

Fig. 2.8 An example for deceptiveness for a 2-D rotated hybrid composition function

deceptiveness generally escalates problem complexity [14]. It is important to note

that deceptiveness holds meaning only in relation to a specific search algorithm, not
universally across all algorithms.

2.2.4 Neutrality

In a landscape where the fitness values of the neighborhood solutions of a

solution remain unchanged, the region surrounding that solution can be considered
neutral [9]. The size of the neutral area around the solution is called the neutral
degree of the solution. A neutral landscape does not imply a flat landscape across the
whole search space (i.e., the objective function is constant) but rather the presence
of neutrality in local areas, such as plateaus and ridges in the landscape mentioned
above.
Neutrality has a profound effect on the success of a search algorithm [12]. When
a population moves to a neutral area of a fitness landscape, where the fitness values
of all solutions are the same, as shown in Fig. 2.9, then this could be misunderstood
as stagnation in the search space or convergence in the objective space on a local
optimum. But in real case, the population is moving just across the neutral area. In
the other case, when the current best solution is located in a flat area of fitness
landscapes, and all adjacent solutions have the same fitness values, there is no
gradient information for algorithms to guide the search with a direction, as shown
in Fig. 2.10. For the two situations, they make the algorithm cannot move any more.
2.2 Properties of Fitness Landscape 21

100
10000

80
8000

60
6000

40
4000

20
2000

20
1520
1015
5 10
05
20
20 15
50
15 10 55 10 5
10 00 55 1510
10 15 20 15
10 15 20
20 20

Fig. 2.9 An example for neutrality, 3-D map for 2-D shifted and rotated Rosenbrock function

20
20

15
15

10
10

55

30
2030
1020
30
30 0 10
20
20 10 0
10
10 10
00 10
10 20
20
20
20 30
30
30 30

Fig. 2.10 An example for neutrality, 3-D map for 2-D shifted Ackley function

Interestingly, a study [1] of neutral landscapes found that neutrality has a

smoothing effect on problem difficulty where adding neutrality to a deceptive
landscape makes the problem easier and adding neutrality to an easy landscape
makes it harder on a binary landscape.
22 2 Fundamentals

2.2.5 Separability

Separability is a property of an objective function. If the objective function can be

written as a sum of a set of sub-objective functions with only one decision variable,
this type of objective function often is considered to be separable [5]. Therefore,
these decision variables of problems can be optimized independently of each other.
The concept of the separability can be defined as follows:

minf (x 1 , · · · , x n ) = (minf (x 1 , · · · ), · · · , minf (· · · , x n ))

. (2.6)

A fitness function is deemed as separable if and only if Eq. (2.6) holds.

Otherwise, it is called nonseparable. If a fitness function .f (x) is separable, the
decision variables .x 1 , · · · , x n are called independent, and the fitness function is
decomposable. A fitness function is called fully nonseparable if there is dependence
among all decision variables. Sometimes, the interaction among variables is also
referred to as epistasis, which is the degree of inter-dependency between variables
(also known as nonlinear separability). When variables in an optimization problem
are dependent on each other, it is impossible to find the optimum solutions by
tuning one variable without considering the changes of other variables. There are
partially separable fitness functions between separable and fully non-separable
fitness functions. For example, a fitness function is k-non-separable if at most
k decision variables are dependent. In general, the higher the degree of non-
separability, the greater difficulty will be for optimization algorithms.
Divide and conquer is a commonly used technique to solve partially separable
problems, especially for large-scale optimization problems (see Chap. 13). It divides
all variables into a set of groups by learning the linkage between variables [16].
Variables in different groups do not depend on each other and can be optimized
separably.

2.2.6 Scalability

In the dimension of the objective space, an increasing number of objective functions

can greatly impact the performance of an algorithm due to the contradictory gradient
for the same variables with different objectives and the exponential decrease
in selection pressure among solutions. In general, this is termed multi-objective
optimization problems (see Chap. 9). Likewise, in the dimension of the solution
space, the increase in decision variables also has an impact on the performance of an
algorithm due to the exponentially increased search space, which is often called the
“curse of dimensionality”. Generally, scalability refers to the solution space. Some
of the problems have the property of scalability. These problems are scalable if they
can scale dimensions without affecting the inherent difficulty of the problems.
2.2 Properties of Fitness Landscape 23

The difference between small-scale and large-scale problems in discrete or

continuous search spaces focuses on the number of decision variables. For large-
scale problems, the main difficulty is the slow search and the expensive runtime
requirement. Parallel and distributed computing can be used to solve the expensive
problem to some extent. The other way is to divide the large-scale problem into a set
of small-scale problems by the divide-and-conquer method mentioned in Sect. 2.2.5
using linkage learning methods between variables.
Note that increasing the number of decision variables normally exponentially
increases the number of optima for most problems. For some particular problems,
increasing the number of variables may change the problem property. For example,
the Rosenbrock function is unimodal in two dimensions but becomes multimodal
when the number of variables increases [11].

2.2.7 Domino Convergence

Domino convergence has begun to be considered since the emergence of litera-

ture [10]. In general, domino convergence usually describes the characteristic of
solutions that contribute to significantly different degrees to the total objective
values of optimization problems [13]. When these solutions are encoded as separate
variables, due to these variables contributing to significantly different degrees to the
total fitness, thus, they will be considered with different priorities in optimization
methods.
In a solution, some of the variables have a very important influence on the fitness.
Therefore, these variables play an important role in the optimization process and
usually converge fast. However, for other variables with a smaller contribution,
they will be ignored or have a small impact on the process. In the worst case,
variables with less contribution may look like noise and not be optimized at all. In
this situation, the search for optimization methods easily traps into local optima,
resulting in the global optimum being difficult to find. Usually, enhancing the
exploration of optimization methods will reduce the phenomenon of premature
convergence in a certain degree.

2.2.8 Property Control, Analysis, and Visualization

Some properties, such as modality, ruggedness, deceptiveness, and neutrality, can be

physically viewed in a fitness landscape, while separability, scalability, and domino
convergence cannot be easily viewed in the landscape. In addition, these properties
often interact with each other: changing one will have an effect on others. This
makes it difficult for algorithm designers to analyze factors related to algorithm
performance. To address this issue, an open framework was proposed to construct
continuous optimization problems by partitioning the solution space with the k-
24 2 Fundamentals

d tree [4]. In each subspace, the properties can be fine-tuned separately, such as
dimensionality, modality, the number of objectives, optima positions, shape and size
of basin of attraction, landscape structure (distribution of optima), inter-relationship
between decision variables, and domino convergence.
How to quantify the properties mentioned above is very important in fitness
landscape analysis. Normally, special techniques are needed to understand the
structure of the fitness landscape. In addition, how to visualize the fitness landscape
for discrete optimization problems or high-dimensional continuous optimization
problems while keeping the visual effect on these properties is also important yet a
challenging issue.

2.3 Computational Complexity

A problem that can be solved by a computer is called a computational problem.

To solve a computational problem by an algorithm, a certain amount of resources
is needed. Providing the bound of resource usage is an important role of compu-
tational complexity theory. Another task of computational complexity theory is to
analyze the complexity classes of problems by taking account of the type of the
computational problem (e.g., decision problems, function problems, optimization
problems, etc.), the model of computation (e.g., deterministic Turing machines,
Boolean circuits, quantum Turing machines, etc.), and the bound of resource usage.

2.3.1 Complexity Measures

Efficient algorithms are preferred in terms of time and storage usage. Time and
storage are the two most commonly used measures to quantify the amount of
resources used. The efficiency of an algorithm depends on its time complexity and
space complexity.

2.3.1.1 Time Complexity

The time complexity of an algorithm refers to the computational effort required

to execute this algorithm. The number of statements executed in an algorithm is
known as the statement frequency or time frequency, notes for .T (n), where n is the
scale of the problem. In general, the number of repeats of the basic operation of an
algorithm is a function .f (n), so the time complexity of the algorithm is denoted
as .T (n) = O(f (n)). When calculating the time complexity of an algorithm, one
should first find out the basic operation of this algorithm and then determine its
execution times in terms of the corresponding statements. Finally, find the highest
order of magnitude of .T (n) as the time complexity, i.e., the “Big O” notation. Note
2.3 Computational Complexity 25

that sorted by increasing order of magnitude, the commonly used time complexity
is 1, .lg n, n, .n lg n, .n2 , .n3 , .2n , and .n!.
When calculating the time complexity of an algorithm, one way is to look at how
many “for” loops there are, i.e., only one is .O(n), two is .O(n2 ), and so on. If the
upper bound of an algorithm is independent of the input size, it has a constant time,
denoted by .O(1). If the algorithm’s .T (n) = O(lg n), it has a logarithmic time, i.e.,
binary tree operations and binary search.

2.3.1.2 Space Complexity

An algorithm’s space complexity means the amount of memory space required by

the algorithm in its life cycle. The space required by the algorithm is made up of two
parts: ◯
1 fixed part, which is the space needed to store some data and variables (i.e.,
simple variables and constants, program size, etc.) that are independent of the size
of the problem, and ◯ 2 auxiliary part, which is the temporary or extra space required
by variables when the algorithm is running, and its size depends on the size of the
problem, such as recursive stack space and dynamic memory allocation.
Similar to the time complexity, the space complexity of an algorithm is defined
as .S(n), where n denotes the scale of the problem. In order to obtain the space
complexity, we have to know how much memory is needed to store values of
different data types (depending on the compiler). If an algorithm requires that all
input values have a fixed amount of space, then this space complexity is called
constant space complexity, denoted .S(n) = O(1). If the amount of space needed
by an algorithm increases with the input value, then the space complexity is called
linear space complexity.

2.3.1.3 Ways of Measures

The best, worst, and average case complexity are the three common ways of mea-
suring the time complexity (also applied to other complexity measures) regarding
different inputs with the same size. Take the quick sort algorithm as an example.
The best-case complexity of quit sort is .O(n lg(n)) when each pivot divides the set
into half. The worst-case complexity occurs when there is no division at all during
the whole sorting process, which takes .O(n2 ). The average-case complexity is also
.O(n lg(n)) by taking all possible permutations of n elements with equal probability.

2.3.1.4 Time Versus Space

Sometimes, we can optimize an algorithm for reducing both the time complexity
and the space complexity. However, in practice, it is not always possible to achieve
the two objectives. A different algorithm may take less time but at the expense of
requiring more memory space and vice versa.
26 2 Fundamentals

Therefore, we should balance the trade-off between space and time according
to our constraints. That is to say, if we have memory constraints, then we can take
algorithms requiring less memory at the cost of more running time. On the contrary,
if we have running-time constraints, then we can take algorithms using less time at
the cost of more memory. For example, our task needs to implement a lookup table.
We can implement it by storing all entries in the memory, and then a result can be
quickly obtained for each enquiry. Other than that, we can recalculate each entry
as needed for each enquiry to save memory but at the cost of increasing computing
time.

2.3.2 P Versus NP Problem

The P versus NP problem is one of the seven Millennium Prize Problems selected
by the Clay Mathematics Institute in 2000. It is also one of the hardest problems in
the field of computer science concerning the speed of a computer to finish a task.
The P and NP problem is the question of whether all so-called NP problems are
equal to the P problems. Although this question is still an open question, the latest
investigation shows that 99% of experts believe that P ./= NP.1
A problem in class P represents the problem that could be addressed in
polynomial time with respect to the size of the problem. The problem is also called
deterministic polynomial problem, the P problem, for short. To be specific, a P
problem means that a solution can be obtained by an algorithm running in an amount
of time that is polynomial to its problem scale.
NP problems denote the non-deterministic polynomial problems. If the solution
of a problem can be verified in polynomial time, then it could be regarded as
NP. Verifying a solution in polynomial time does not mean finding a solution in
polynomial time. For example, finding priming factors of a large number will try
many different candidates, but verifying these candidates can be achieved by simply
multiplying these factors.
NP-hard problems refer to problems that are at least as hard as NP problems,
i.e., all NP problems can be reduced to NP-hard problems. Nevertheless, it is worth
noting that not all NP-hard problems can be classified as NP problems, which means
that verifying a solution may not take polynomial time.
The NP-complete problem is a problem that is both NP and NP-hard. Therefore,
finding an effective algorithm for any NP-complete problem suggests that an
efficient algorithm could be found for all NP problems as a solution for any problem
belonging to this type could be transformed into a solution for any other member
of the type. The TSP is a classical NP-complete problem, which widely exists in
logistic scheduling. It is unknown whether there are polynomial-time algorithms
for NP-complete problems, since it is still one of the most important problems in

1 Guest Column: The Third P =? NP Poll, 2020.

References 27

theoretical computer science to determine whether these problems can be handled

or not.

References

1. Beaudoin, W., Verel, S., Collard, P., Escazut, C.: Deceptiveness and neutrality the ND family of
fitness landscapes. In: Proceedings of the 8th Annual Conference on Genetic and Evolutionary
Computation, pp. 507–514. ACM SIGEVO, Association for Computing Machinery, New York
(2006)
2. Goldberg, D.E.: Genetic algorithms and Walsh functions: Part II, deception and its analysis.
Complex Syst. 3(2), 153–171 (1989)
3. Jones, T.: Evolutionary algorithms, fitness landscapes and search. Ph.D. thesis, The University
of New Mexico, Albuquerque (1995)
4. Li, C., Nguyen, T.T., Zeng, S., Yang, M., Wu, M.: An open framework for constructing
continuous optimization problems. IEEE Trans. Cybern. 49(6), 2316–2330 (2018)
5. Li, X., Tang, K., Omidvar, M.N., Yang, Z., Qin, K.: Benchmark functions for the CEC’2013
special session and competition on large-scale global optimization. Technical report, RMIT
University, Cancun (2013)
6. Liepins, G.E., Vose, M.D.: Deceptiveness and genetic algorithm dynamics. In: Rawlins, G.J.
(ed.) Foundations of Genetic Algorithms, pp. 36–50. Elsevier, Amsterdam (1991)
7. Pitzer, E., Affenzeller, M.: A comprehensive survey on fitness landscape analysis. In: Fodor,
J., Klempous, R., Araujo, C.P.S. (eds.) Recent Advances in Intelligent Engineering Systems,
pp. 161–191. Springer, Berlin (2012)
8. Reeves, C., Rowe, J.E.: Genetic Algorithms: Principles and Perspectives: A Guide to GA
Theory. Springer Science & Business Media, Boston (2002)
9. Reidys, C.M., Stadler, P.F.: Neutrality in fitness landscapes. Appl. Math. Comput. 117(2–3),
321–350 (2001)
10. Rudnick, W.M.: Genetic algorithms and fitness variance with an application to the automated
design of artificial neural networks. Ph.D. thesis, Oregon Graduate Institute of Science &
Technology, Hillsboro (1992)
11. Shang, Y.W., Qiu, Y.H.: A note on the extended Rosenbrock function. Evol. Comput. 14(1),
119–126 (2006)
12. Smith, T., Husbands, P., O’Shea, M.: Fitness landscapes and evolvability. Evol. Comput. 10(1),
1–34 (2002)
13. Thierens, D., Goldberg, D.E., Pereira, A.G.: Domino convergence, drift, and the temporal-
salience structure of problems. In: 1998 IEEE International Conference on Evolutionary
Computation Proceedings. IEEE World Congress on Computational Intelligence, pp. 535–540.
IEEE, Anchorage (1998)
14. Weise, T., Niemczyk, S., Skubch, H., Reichle, R., Geihs, K.: A tunable model for multi-
objective, epistatic, rugged, and neutral fitness landscapes. In: Proceedings of the 10th Annual
Conference on Genetic and Evolutionary Computation, pp. 795–802. ACM SIGEVO, New
York (2008)
15. Wright, S.: The roles of mutation, inbreeding, crossbreeding, and selection in evolution. In:
Proceedings of the VI International Congress of Genetics, pp. 356–366. The International
Genetics Federation, The VI International Congress of Genetics, New York (1932)
16. Yang, M., Zhou, A., Li, C., Yao, X.: An efficient recursive differential grouping for large-scale
continuous problems. IEEE Trans. Evol. Comput. 25(1), 159–171 (2020)
Chapter 3
Canonical Optimization Algorithms

Abstract This chapter introduces canonical optimization algorithms, including

numerical optimization methods for continuous optimization problems and state
space search methods for discrete optimization problems. Several popular numerical
optimization methods based on line search are presented, e.g., steepest descent
method, Newton method, and conjugate gradient method. State space search
methods are categorized into uninformed search methods and informed search meth-
ods. Uninformed search methods include breadth-first search, depth-first search,
and depth-limited search. Informed search methods include greedy search, A*
search, and Monte-Carlo tree search. Several single-solution-based metaheuristic
search algorithms are also introduced in this chapter, e.g., hill climbing, simulated
annealing, iterated local search, and variable neighborhood search.

3.1 Numerical Optimization Algorithms

This section introduces several simple numerical optimization algorithms to solve

unconstrained continuous optimization problems. These algorithms assume that the
fitness landscape is like a valley. To reach the bottom of the valley from a certain
position, one firstly finds a direction to the bottom according to the gradient and
walks toward the direction for a step with a certain length, then recalculates the
direction, and walks forward one step again until reaches the bottom.

3.1.1 Line Search

Numerical optimization algorithms use the above idea to solve the problem in a
recursive way. Suppose that .x k is the point obtained at the k-th iteration and .d k is
the direction along which the fitness function .f (x) declines at point .x k . .αk > 0 is
the step size along .d k . Then the next point .x k+1 is obtained by

x k+1 = x k + αk d k
. (3.1)

© China University of Geosciences Press 2024 29

C. Li et al., Intelligent Optimization,
https://doi.org/10.1007/978-981-97-3286-9_3
30 3 Canonical Optimization Algorithms

where .f (x k+1 ) < f (x k ) must hold. Such an algorithm is also called a line search.
Line search starts from an initial solution .x 0 . It finds a direction along which the
fitness value will decline and decides the step size to calculate the next solution.
The procedure is repeated until the termination is met. Algorithm 3.1 shows the
procedures of the algorithm.

Algorithm 3.1: Line search

Input: Initial solution x 0
Output: Final solution x n
1 Set initial counter k = 0 while termination is not met do
2 Compute a descent direction d k ;
3 Choose αk to ‘loosely’ minimize f (x k+1 ) = f (x k + αk d k ) over α ∈ R+ ;
4 Update x k+1 = x k + αk d k and k = k + 1;

An important aspect in the algorithm is the termination criterion, which can be

defined as one of the following:
(1) The procedure is stopped when the computing resources (i.e., time) are
exhausted, that is, the maximum number of iterations is reached.
(2) The procedure is stopped when the gradient at .x k is smaller than a gradient
norm tolerance .ϵg , i.e., the algorithm has converged at a critical point.

. ‖∇f (x k )‖ < ϵg (3.2)

where .ϵg is a user-defined parameter.

(3) The procedure is stopped when the difference between .x k+1 and .x k is smaller
than a small constant value of .ϵx , i.e., the algorithm converges at an x point, and
the difference between .f (x k+1 ) and .f (x k ) is also smaller than a small constant
value of .ϵf

‖x k+1 − x k ‖ < ϵx
. (3.3)
‖f (x k+1 ) − f (x k )‖ < ϵf

where .ϵx and .ϵf are both user-defined parameters.

(4) The procedure is stopped when .f (x k+1 ) is larger than .f (x k ), i.e., the algorithm
diverges.
Another important aspect of Algorithm 3.1 is how to choose the step size .α.
Obviously, the optimum choice of .α is to make the function .φ(α) = f (x k + αd k )
smallest. The difficulty to solve this problem is the same as that to solve the current
minimization problem. Thus, we can only find an approximate .α that makes .φ(α)
smaller. It seems that .α just needs to satisfy the condition .f (x k+1 ) < f (x k ). But in
practice, we should take the other factor into account. For example, Fig. 3.1 shows
that although the selected .α satisfies the condition, the algorithm still traps at two
3.1 Numerical Optimization Algorithms 31

Fig. 3.1 Insufficient reduction in f

points, .x = −1 and .x = 1, where .x1 = 2. The key issue is that there is no sufficient
reduction in f .
There are some famous conditions to find such .α to obtain a sufficient reduction
in f :
(1) The Armijo condition

f (x k + αk d k ) ≤ f (x k ) + c1 αk d Tk ∇f (x k )
. (3.4)

where constant .c1 belonging to .(0, 1) generally is set to .c1 = 10−4 in practice.
.d k is the descent direction, so .d k ∇f (x k ) < 0 and the right-hand side of the
T

inequality is the subtractive function of .α. This condition guarantees that there
is some reduction in f as long as .α is not too small. There is an issue with
the Armijo condition. Any small .α that satisfies the Armijo condition cannot
make sufficient progress. To address this issue, we add another constraint to the
Armijo condition as follows:
(2) The Wolfe condition

f (x k + αk d k ) ≤ f (x k ) + c1 αk d Tk ∇f (x k )
. (3.5)
∇f (x k + αk d k )d Tk ≥ c2 d Tk ∇f (x k )

where .0 < c1 < c2 < 1, .c2 is a relatively large number compared with .c1 where
.c2 = 0.9 for Newton or quasi-Newton methods and .c2 = 0.1 for the nonlinear
conjugate gradient method. The second inequality guarantees that the slope of
Random documents with unrelated
content Scribd suggests to you:
Cassiterides yielded in abundance the ores of copper and tin, but
these had to be smelted and worked with all the accumulated results
of tentative skill before they yielded the copper or more useful
bronze. By whom or where this first knowledge was mastered is
unknown: the tendency is still to look to Asia, perhaps to Phœnicia,
or to the still undetermined cradle of the Aryans, wherever that may
prove to have been, for the birth of this phase of metallurgic art
which constituted so important a stage in early civilisation. Yet if the
ancient American missed it, it was not for want of opportunity.
Examples of the accidental fusion of copper by the sacrificial fires of
the Mound-Builders repeatedly occur in the mounds of the Ohio
valley. But no gifted native alchymist was prompt to read the lesson
and turn it to practical account.
Asia and Europe appear to have passed by a natural transition,
step by step, from their rudest stages of lithic art to polished stone,
and then to implements of metal. Some of the steps were doubtless
very slow. Worsaae believed that the use of bronze prevailed in
Denmark “five or six hundred years before the birth of Christ.”[76] In
Egypt it undoubtedly was known at a greatly earlier date. I still
incline to my early formed opinion, that gold was the first metal
worked. Found in nuggets, it could scarcely fail to attract attention.
It was easy to fashion into shape; and some of the small, highly
polished stone hammers seem fitter for this than any other work.[77]
The abundant gold ornaments of the New World at the time of the
discovery of Mexico and Peru accord with this idea. The like
attraction of the bright native copper, is proved by its employment
among the southern Indians for personal ornaments; and in this way
the economic use of the metals may have been first suggested.
From the working of gold nuggets, or of virgin copper, with the
hammer, to the smelting of the ores, was no trifling step; but that
knowledge once gained, the threshold of civilisation and true
progress had been reached. The history of the grand achievement is
embodied in the earliest myths both of the Old and the New World.
Tubal-cain, Dædalus, Hephæstus, Vulcan, Vœlund, Galant, the Luno
or the Celtic Fingal, and Wayland, the Saxon smith-god, are but
legendary variations of the first worker by whom the gift of
metallurgy was communicated to man; and so too the New World
has its Quetzalcoatl, or Vœlund of the Aztecs, the divine instructor of
their ancestors in the use of the metals. But whatever be the date of
this wise instructor, no share of the knowledge communicated by
him to that favoured race appears to have ever penetrated
northward of the Mexican Gulf.
It is vain to urge such dubious evidence as the fancied traces of a
mould-ridge, or the solitary example of a casting of uncertain age, in
proof of a knowledge of the furnace and the crucible among any
North American tribe. Everywhere in Europe the soil yields not only
its buried relics of gold, copper, and bronze, but also stone and
bronze moulds in which implements and personal ornaments were
cast. When the ingenious systematising of Danish archæologists had
familiarised the students of antiquity with the idea of a succession of
Stone, Bronze, and Iron periods in the history of Europe, the
question naturally followed, whether metallurgy did not begin, there,
as elsewhere, in the easy working of virgin copper. Dr. Latham
accordingly remarked, in his Ethnology of the British Islands, on the
supposition that no unalloyed copper relics had been found in
Britain: “Stone and bone first; then bronze, or copper and tin
combined; but no copper alone. I cannot get over this hiatus; cannot
imagine a metallurgic industry beginning with the use of alloys.” It
was a mistake, however, to assume that no copper relics had ever
been found. At first it had been taken for granted that all such
implements were of the familiar alloy. But so soon as the importance
of the distinction was recognised, examples of pure copper were
forthcoming. So early as 1822, Sir David Brewster described a large
axe of peculiar shape, and formed of copper, which was found in the
hard black till-clay at a depth of twenty feet under Ratho Bog, near
Edinburgh. This is no solitary example. The Scottish Museum of
Antiquities has other implements of pure copper; and Sir William
Wilde states in reference to the collections of the Royal Irish
Academy: “upon careful examination, it has been found that thirty of
the rudest, and apparently the very oldest celts, are of red, almost
unalloyed copper”; as is also the case with some other rudely
formed tools in the same collection.
It was a temporary advantage, doubtless, but a real loss, to the
Indian miners of Lake Superior that they found the native copper
there ready to hand, a pure ductile metal, probably regarded by
them as only a variety of stone which—unlike its rocky matrix,—they
could bend, or hammer into shape, without fracture. Its value as
such was widely appreciated. Copper tools, everywhere retaining the
specs, or larger crystals of silver, characteristic of the Lake Superior
veins, tell of the diffusion of the metal from that single source
throughout all the vast regions watered by the Mississippi and its
tributaries, and eastward by lake and river to the gulf of the St.
Lawrence and the mouth of the Hudson.
There was a time when this traffic must have been systematically
carried on; when the ancient miners of Lake Superior worked its rich
copper veins with industrious zeal; and when, probably as part of
the same aggressive energy, the valley of the Ohio was filling with a
settled population; its great earthworks were in process of
construction, and a native race entered on a course that gave
promise of social progress. But, from whatever cause, the work of
the old miners was abruptly terminated;[78] the race of the Mounds
vanished from the scenes of their ingenious toil; and rudest
barbarism resumed its sway over the whole northern continent. The
same Aryan race that, before the dawn of history, before the
Sanskrit-speaking people of India, or the Zends of Persia, entered on
their southern homes, spoke in its own cradleland the mother
tongue of Sanskrit, Greek, Celtic, and German, at length broke up
and went forth on its long wanderings. Whatever peoples it found
there; they were replaced by Celts, Romans, Greeks, Slavs, and
Teutons, who broke in upon the barbarism of prehistoric Europe;
displaced the older races, Allophylian, Neolithic, Iberian, Finnic, or by
whatever other name we may find it convenient to designate them;
but not without a considerable intermingling of the old blood with
that of the intruders. The sparsely settled continent gradually filled
up. Forests were cleared, swamps drained, rivers confined by
artificial banks and levées to their channels; and there grew up in
their new homes the Celtic, Classic, Slavic, and Teutonic tongues,
with all the varied culture and civilisation which they represent.
Agriculture, the special characteristic of the whole Aryan race,
flourished. They brought with them the cereals; and, with plenty, the
favoured race multiplied, till at length it has grown straitened within
the bounds of the old continent which it had made its own.
With the close of the fifteenth century one great cycle, that of
Europe’s mediæval era, came to an end; and then we trace the first
beginnings of that fresh scattering of the Aryan clan, and its new
western movement across the Ocean. It seems in a very striking
manner once more to repeat itself under our own eyes, as we look
abroad on the millions crowding from Europe and filling up the
western wilderness; hewing down the forests, cultivating the waste
prairies, and everywhere displacing the rude aborigines: yet here
also not without some interblending of the races; though the two
types, Aryan and pre-Aryan, meet under all the repellent influences
of high civilisation and the lowest barbarism. In the Canadian North-
West alone, the young province of Manitoba began its political
existence with a population of between 10,000 and 12,000 half-
breeds; in part at least, a hardy race of hunters and farmers; the
representatives of what is as certainly destined to constitute an
element in the new phases which the Aryan race already begins to
assume, under the diverse conditions of a new continent, as that
curious trace of Europe’s pre-Aryan people which attracted the
observant attention of Tacitus among the ancient Britons; and which
we are learning to recognise, with a new significance, as the
Melanochroi: the representatives of the old half-breed of Europe’s
prehistoric dawn.
[40] Gladstone, Juventus Mundi, pp. 474, 479.
[41] Indian Migrations, p. 24.
[42] Types of Mankind, p. 351.
[43] Early Man in Britain, p. 241.
[44] Ibid. p. 244.
[45] Alaska and its Resources, p. 237.
[46] Prehistoric Europe, p. 550.
[47] U.S. Geological Survey, 1872, p. 652. Report of
National Museum, 1887, p. 683, Fig. 11535.
[48] Report of National Museum, 1887, p. 678.
[49] Ancient Stone Implements of Great Britain, p. 140.
[50] Vide Prehistoric Man, 3d ed. vol. i. p. 180, Fig. 54.
[51] Report of the Peabody Museum, vol. ii., p. 38.
[52] Primitive Industry, p. 471.
[53] Ibid., p. 542.
[54] Primitive Industry, p. 547.
[55] Ibid., p. 545.
[56] Foster’s Prehistoric Races, p. 55.
[57] The Antiquity and Origin of the Trenton Gravel, p.
547.
[58] Primitive Industry, p. 481.
[59] Report of Washington National Museum, 1887-88,
pp. 677-702.
[60] Primitive Industry, p. 517.
[61] Indian Migrations as Evidence of Language (Horatio
Hale), p. 21.
[62] Indian Migrations, p. 22.
[63] Relations des Jésuites, 1660, p. 7. Quebec ed.
[64] Archæologia Americana, vol. ii.
[65] Brinton, Races and Peoples, p. 254.
[66] Bertram’s Travels through North and South Carolina,
Georgia, etc., 1791, p. 367.
[67] The League of the Iroquois, p. 2.
[68] Schoolcraft, History of the Indian Tribes, vol. ii., p.
78.
[69] Prehistoric Races of the United States, p. 259.
[70] Smithsonian Report, 1572, p. 353. The important
word not supplied here, it is obvious from the
context is absent by a mere typographical error.
[71] Ancient Monuments of the Mississippi Valley, vol. i.,
pp. 196-207.
[72] The North Americans of Antiquity, p. 95.
[73] Prehistoric Races of the United States, pp. 251-259.
[74] Antiquities of the Southern Indians, p. 225.
[75] Primitive Industry, pp. 411-422.
[76] Primæval Antiquities, p. 135.
[77] Prehistoric Annals of Scotland, 1st ed. 1851, p. 214;
2d ed. vol. i.
[78] Prehistoric Man, 3rd ed. vol. i., pp. 203-228.
 V

THE ÆSTHETIC FACULTY IN ABORIGINAL RACES

The ingenious arts of the palæolithic cave-dwellers of the Vézère
abundantly prove that it needed no wanderers from the cradlelands
of Old World civilisation to that strange Atlantis lying in the
engirdling ocean beyond the Pillars of Hercules, to engraft their
artistic capacities on the “Achoriens” by whom it was peopled. The
innate faculty for art has manifested itself in individuals and in
nations, among widely diverse Assyrian, Egyptian, Hellenic, Arabian,
and mediæval races, as in later Frank, Fleming, and Dane, with
unaccountable partiality. For its absence, or very subordinate
manifestation, is seen to be compatible with the highest intellectual
triumphs in other directions. The arts of Britain’s Allophyliæ manifest
no instinctive aim at a reproduction of familiar natural objects, such
as is characteristic of some races at a very primitive stage. Nor was
it till a recent generation that the stock from which Shakespeare and
Newton sprung put forth its first efforts at rivalling the great masters
of the Renaissance, or entering into competition with the skilled
painters of the Low Countries. It is otherwise with the nations of the
New World. The highest stage of civilisation attained there is a very
partial one. But among the various characteristics of the American
aborigines which invite attention, the very wide diffusion of an
aptitude for imitative art is one that merits careful study as typical of
American man. It is not, indeed, to be overlooked that if due
allowance be made for the narrow range in degrees of civilisation
among the races of the New World, the same diversity of racial
characteristics is observable there as elsewhere. The tendency,
moreover, of civilisation is to efface, or greatly to modify, such
distinctions. Civilised nations habitually borrow the arts and imitate
the social habits of neighbouring races, or accept some common
standard of intellectual and moral pre-eminence. Nevertheless, while
the capacity for imitative art is neither peculiar to the New World,
nor characteristic of all its diverse nationalities, it appears to be more
generally diffused among the races of America than elsewhere. It is
prevalent among tribes in nearly every condition, from the rude
Indian nomad, or the Eskimo, to the semi-civilised Zuñi, and the
skilled metallurgists and architects of Central America and Peru.
This development of a feeling for art in savage races is at all
times interesting as indicative of intellectual capacity and powers of
observation, even when manifested, as it frequently is, within a very
narrow range. It is by no means a general characteristic either of
savage or civilised man. Yet recent archæological discoveries prove it
to have been one of the earliest forms of intellectual activity among
the cave-dwellers of Europe’s palæolithic dawn. The most civilised
nations have differed widely in the manifestations of this æsthetic
faculty. The city of Dante was the Athens of the Middle Ages in art
as well as letters; while the land which gave birth to Shakespeare
can scarcely be said to have had a native school of painting or
sculpture till late in the eighteenth century. The like differences are
observable among barbarous nations. Races are met with, to whom
the drawing of a familiar object suggests no idea of the original;
while others, in nearly the same stage of savage life, habitually
practise the representation of natural objects in the decorative
details of their implements and articles of dress, and in the carvings
which furnish occupation for many leisure hours.
A special interest attaches to the disclosures of archæology
relative to the prehistoric races of Europe, owing to the evidence
thereby furnished of striking resemblances in their arts and
conditions of life to those of uncultured races of our own day. In
many respects it seems as though the present condition of some
existing races of America only repeats that of Europe’s infancy. But
so far as imitative art is concerned, the analogy fails when the more
recent contents of the barrows, cairns, and grave mounds of
prehistoric Europe are brought into comparison with those of the
New World. If, indeed, we leave behind us the age of cromlechs,
kistvaens, cairns, and barrows, and seek to estimate aright the
disclosures of artistic ability pertaining to the far more ancient men
of Europe’s Mammoth and Reindeer periods, it is otherwise. But,
before reviewing the wonderfully definite glimpses thereby furnished
of tribes of rude yet skilful hunters of that post-glacial world, it may
be of some help, in the comparisons which they suggest, to recall
impressions derived from a study of that Stone period in which the
natives of the British Islands appear to have approximated, in many
respects, to the Red Indian nomad of the American forest.
One little-heeded point of evidence of this correspondence, to
which I long since drew attention, is to be found in the traces of
artificial modification of the head-form in ancient British crania; a
comparison of which with skulls recovered from Indian grave
mounds helps to throw light on the habits and social life of the
British Islands in prehistoric times. In illustration of this I may refer
to an exploration, now of old date. In the early summer of 1851, I
learned of the accidental exposure of a stone cist, in trenching a
garden at Juniper Green, a few miles distant from Edinburgh, and
immediately proceeded to the spot. There, under a slightly elevated
knoll—the remains, in all probability, of the ancient barrow,—lay a
rude sarcophagus of unhewn sandstone, within which was a male
skeleton, still in good preservation. The body had been laid on its
left side, with the arms folded over the breast, and the knees drawn
up so as to touch the elbows. A flat water-worn stone formed the
pillow, from which the skull had rolled to the bottom of the cist.
Above the right shoulder stood a gracefully formed clay vase,
containing only a little sand and black dust, the remains, it may be
presumed, of food which it originally contained when deposited
there by affectionate hands, in some long-forgotten century. It was
recovered uninjured, and is now deposited, along with the skull, in
the Archæological Museum of Edinburgh.
The primitive grave, thus discovered within sight of the Scottish
capital, has a curious interest as a link connecting the present with a
remote past. But the special point which throws light on the habits
of the ancient race, is a parieto-occipital flattening, such as is of
common occurrence in skulls recovered from American ossuaries and
grave mounds. This feature is clearly traceable to the use of the
cradle-board in infancy. The mode of nursing the Indian papoose, by
bandaging it on a cradle-board, is specially adapted to the
vicissitudes of a nomad forest life. The infant is carried safely, slung
on the mother’s back, leaving her hands free; and in the pauses of
her journey, or when engaged in field work, it can be laid aside, or
suspended from the branch of a tree, without risk of injury. But one
result of this custom is that the soft bones of the skull are subjected
to a continuous pressure in one direction during the whole term of
suckling, which is necessarily protracted, among a nomad people,
much longer than is usual in settled communities; and to this cause
is undoubtedly traceable the occipital flattening of many skulls
recovered from European cists and barrows. Dr. L. A. Gosse, after
discussing in his Essai sur les déformations artificielles du Crâne
certain artificial modifications of the skull, of common occurrence
among the aborigines of the New World, thus proceeds: “Passant
dans l’ancient continent, ne tardons-nous pas à reconnaître que ce
berceau plat et solide y a produit des effets analogues. Les anciens
habitants de la Scandinavie et de la Calédonie devaient s’en servir, si
l’on en juge par la forme de leurs crânes.”[79]
Full-sized representations of the Juniper Green skull, and others
of the same type, are given in Crania Britannica.[80] Bateman also, in
his Ten Years’ Diggings in Celtic and Saxon Gravehills, concurs with
earlier writers in ascribing to the use of the cradle-board the
flattened occiput observed in skulls recovered from British barrows.
The employment, indeed, of the cradle-board among prehistoric
races of Northern Europe, and their nomad life of which it is so
characteristic a feature, may now be considered as generally
recognised. The implements and pottery recovered from graves of
the period show their constructors to have been, for the most part,
devoid of any knowledge of metals; or, at best, in the mere
rudimentary stage of metallurgic arts. But the Juniper Green cist,
that of the large Staffordshire barrow of Wotton Hill, that of
Roundway Hill, North Wilts, another of Green Lowe, Derbyshire, and
others described in the works above referred to, while all disclosing
evidence of correspondence in habits and social condition between
ancient races of the British Isles and the Indians of the New World,
also furnished characteristic examples of their fictile ware; and here
the analogy fails. There are, indeed, abundant specimens of broken
Indian pottery on many of their deserted village sites, which might
be mingled with the fragments of a like kind from early European
grave mounds without attracting special attention. Simple chevron
and saltier or herring-bone patterns, scratched with a pointed bone
on the soft clay, are common to both; and many of the more
elaborate linear and bead patterns of the primitive British potters
reappear with slight variation on the Indian ware. But besides those,
few ancient Indian village sites fail to yield fragments of pottery,
including clay tobacco pipes, ornamented with more or less rude
imitations of the human face and of familiar animals, such as the
beaver, the bear, the lynx, and the deer. Before my first visit to the
American continent, while still preoccupied with the arts of the
ancient British savage, and the more graceful devices of the
metallurgists of Europe’s Bronze period, I noted the prevalence of a
conventional ornamentation, consisting mainly of improvements on
what may be called the accidents of manufacture, or possibly of
linear decorations borrowed from patterns of the plaiter or knitter.[81]
No attempt appears to have been made by the old European
decorator at such imitations of familiar natural objects as are now
known to have been practised among the far more ancient cave-
dwellers of Europe, the contemporaries of the mammoth and the
woolly rhinoceros, and which are familiar to us in the primitive arts
of the New World. Objects recovered from the mounds of the Ohio
and the Mississippi valleys, as well as the diversified products of the
native artificers of Mexico and Peru, attract special attention by their
endless variety of imitative design; and similar skill and ingenuity are
apparent in the pottery, the plaited manufactures, the stone and
bone carvings, and even in many of the great animal mounds and
other earthworks of the North American continent. An observant
recognition of analogies, traceable in the rhetorical construction of
many American Indian holophrasms, appears to be only another
phase of this widely prevalent imitative faculty. At the same time,
whether we study the physical form or the intellectual characteristics
of native American races, it becomes more and more apparent that
the New World has been peopled from different centres, and still
presents essentially distinct types of race. It had its ferocious Caribs,
its Mexicans, with their revolting human sacrifices and other bloody
rites, and its stealthy, treacherous nomads, less courageous but not
less cruel. But it has also gentler races, in whom, as in the
Peruvians, the Zuñis, and others of the Pueblo Indians, the æsthetic
faculty predominates, and overlays with many a graceful
concomitant the utilitarian products of their industrial arts.
Whether barbarous or civilised nations are classified in
accordance with their linguistic affinities, both are found to manifest
other specialties according with the diverse families of speech. The
differences which separate the Aryan from the Semitic races are not
more marked than the intellectual and moral divergencies among
barbarous tribes. But while this is apparent on the American
continent, its diverse races appear to be characterised by a more
general aptitude for artistic imitation than is observable elsewhere,
except among the long-civilised nations of the Old World, whose
composite vocabularies reveal the sources of many borrowed arts.
The Peruvian potter sketched and modelled endless quaint devices in
clay; the Zuñian decorated his gracefully fashioned ware with highly
effective parti-coloured designs; and the old Mound-Builder wrought
in intaglio on his domestic and sepulchral vessels conventional flower
patterns; and in his miniature sculptures reproduced the fauna of an
area extending from the Ohio to the Gulf of Mexico. Native artificers
of widely different American races manifest this imitative faculty. Not
only is the Indian pipe sculptor found copying animate and
inanimate objects with an observant eye and a ready hand, but even
the linear patterns on pottery and straw basket-work are frequently
made to assume combinations obviously suggested by flowers and
other familiar objects of nature. The perception of such analogies,
and even the capacity for appreciating the linear or pictorial
representation of objects on a flat surface, varies greatly in different
races. Travellers have repeatedly described the manifestation by
savage races of an utter incapacity to comprehend pictured
representations. Mr. Oldfield, for example, tells how a large coloured
engraving of a native of New Holland was shown to some
Australians. “One declared it to be a ship, another a kangaroo, and
so on, not one of a dozen identifying the portrait as having any
connection with himself.”[82] The artistic faculty is unquestionably
hereditary. There are artistic families and artistic races. But if so, the
pictorial skill of the palæolithic cave-dwellers of Western Europe was
not transmitted to their successors. Guided not only by a comparison
of their tools and weapons with those of the Neolithic period, but
also by cranial and other physical evidence, we are led to assume
the absence of affinity between the men of the Perigord caves and
the greatly more modern races of Europe’s later Stone period; and
their lack of the imitative faculty, so characteristic of the elder race,
adds confirmation to this opinion.
Artistic sympathies, and a capacity for high achievements in
painting and sculpture, are neither the direct results of civilisation,
nor in many cases the product of culture and training. From the days
of Giotto, the shepherd boy, to those of Thorwaldsen, Wilkie, and
Turner, art-power is not only seen to be a direct and exceptional gift
of nature, but it is frequently the product of a singularly partial
intellectual development. Leonardo da Vinci and Michael Angelo are
examples of men of rare and comprehensive genius, who sought in
art the form in which to give expression to their many-sided powers.
But, on the other hand, instances are not rare of artists like
Thorwaldsen or Turner, who, except within the range of their own
special art, seemed exceptionally defective in the exercise of
ordinary mental powers. The same is true of races as of individuals.
Some show an aptitude for art wholly wanting in others, who
nevertheless equal or surpass them in more important qualities. The
æsthetic faculty may, indeed, be described as curiously capricious in
its manifestations. The Papuans of New Guinea and of New
Caledonia, a race of Negrillos, in some points presenting analogies to
the Australian, are nevertheless remarkable for a seemingly
instinctive ingenuity and aptitude for art. Mr. Wallace describes them
as contrasting with the Malay race in the habitual decoration of their
canoes, houses, and almost every domestic utensil, with elaborate
carving. The Fijians, who are allied to this Negrillo race, present in
many respects an unfavourable contrast to the true Polynesian. In
their physiognomy and whole physical aspect they are inferior to
other island races of the Pacific; and are further notable for repulsive
habits and a general condition of social and moral degradation. But
their ferocity and the cruel customs in which they so strikingly differ
from the Malays are vices of a vigorous race. They have frequently
been observed to indicate energy capable of being directed to useful
ends, as has been the case with the Maori cannibals of New Zealand,
and was seen of old in the Huns and the Northmen, whose
descendants are now among the most civilised races of the world. It
is obvious, at any rate, that the savage vices of the Fijians are
compatible with considerable skill in such arts as pertain to their
primitive condition. Their musical instruments are superior to those
of the Polynesians, and include the pan-pipe and others unknown in
the islands beyond their range. Their pottery exhibits great variety of
form; and some of the vessels combined in groups present a curious
correspondence to familiar examples of Peruvian art. Their fishing-
nets and lines are remarkable for neat and skilful workmanship, and
they carry on agriculture to a considerable extent. “Indeed,”
remarked the ethnologist of the United States Expedition, in
summing up the characteristics of the Fijians, “we soon began to
perceive that the people were in possession of almost every art
known to the Polynesians, and of many others besides. The highly-
finished workmanship was unexpected, everything being executed
until recently, and even now for the most part, without the use of
iron. In the collection of implements and manufactures brought
home by the Expedition, the observer will distinguish in the Fijian
division something like a school of arts for the other Pacific islands.”
All this has to be kept in view, in any attempt to gauge the
intellectual development, or determine the degree of civilisation, of
the palæolithic draughtsmen and carvers of the Garonne. One of the
scenes introduced by M. Figuier, in the fanciful illustrations of his
L’Homme primitif, represents a group of artists, such as, except for
their costume, might have been sketched from the students of the
École des Beaux Arts. Their mode of work was probably much more
akin to the intermittent labours of the Indian, whose elaborately
sculptured pipe is laid aside, and resumed again—often after
prolonged intervals,—before it receives the finishing touch. But
though the drawings and the carvings of those primitive artists alike
manifest remarkable skill and observant imitation, the former are the
objects of special interest. Their carvings appear to have been
executed, with rare exceptions, for the decoration of favourite
implements and weapons, in accordance with a practice common to
many diverse races and conditions of society. But the drawings have
no such motive. They more nearly correspond to the sketch or
drawing from nature of the modern artist; and furnish evidence of
peculiar attributes, strikingly distinguishing the race of that remote
age from most others that have succeeded them.
Certain it is that, so far as present evidence goes, the greatly
prolonged Neolithic period was characterised by no such artistic
feeling or imitative skill. Specimens of the ingenious handiwork of
the artificers of Europe’s later Stone age abound. We have numerous
relics from the kitchen middens of Denmark, the pile villages of
Switzerland, the crannoges of Scotland and Ireland, as well as the
varied contents of cromlechs, cists, cairns, and barrows, diligently
explored throughout Europe. But no such examples of carvings, or
graven representations of animals or other natural objects, have
been found. The “clay in the hands of the potter” is a familiar
symbol of plastic response to the will of the designer. It is, indeed,
easier for the practised modeller to fashion the clay into any desired
form, than to draw it, subject to rules of perspective, on a flat
surface. Linear devices and the representation of objects in intaglio,
or in low relief, are also accomplished with great facility on the soft
clay. Hence the art of diverse races, periods, and stages of progress,
finds its aptest illustration in fictile ware; and the imitative faculty of
widely different American races may be studied in their pottery. In
Mexico, apparently, we have to look for the northern school of
ceramic art. There, the aggressive races of the North first came in
contact with the civilisation of Central America; and the native
aptitude for imitative representation received a fresh impulse. The
Indian modeller learned to work skilfully in clay; and the variety of
design, combined with the quaint humour of the caricaturist,
displayed in many of the Mexican terra-cottas, serves to indicate this
class of work as specially significant in relation to the present
inquiry. The inventive fancy and skill of the Peruvian potter illustrates
in ampler variety the progress achieved by the races of the southern
continent. But this will more fitly come under review along with
other examples of modern native art. For no analogous traces of
contemporary modelling in clay furnish material for comparison with
the art of the Palæolithic era; though the skill of its bone and ivory
carvers was in no degree inferior to that of the Mexican or Peruvian
modeller. But the æsthetic aptitude of that old race of Europe’s
intellectual dawn is in some respects unique. In so far as their
ingenious arts furnish any evidence of true racial characteristics, the
men of the Neolithic era inherited none of their æsthetic feelings;
nor did the imitative faculty manifest itself with exceptional power
until the advent of the Aryan races brought with it the potentialities
of Hellenic inspiration.
The absence of nearly every trace of imitative art in the
prehistoric remains of Britain has already been noted. It made a
strong impression on my mind at an early stage of my archæological
researches; for this characteristic of European art extends over a
period of greatly prolonged duration, marked by the advent and
disappearance of races, dissimilar alike in physical and mental
characteristics. We have the laboriously finished implements of
neolithic art, the pottery of at least two distinct races seemingly prior
to the Celts, and then the graceful artistic productions of the Bronze
period, but still only the rarest traces of any effort at imitation. Long
before the imitative arts of the American continent were known to
me otherwise than from description, I remarked, of the archaic art of
the first British metallurgists: “The ornamentation consists, almost
without exception, only of improvements on the accidents of
manufacture. The incised decorations of the pottery appear, in many
cases, to have been produced simply by passing twisted cords round
the soft clay. More complicated designs, most frequently consisting
of chevron, saltire, or herring-bone patterns, where they are not
merely the results of a combination of such lines, have been
suggested, as I conceive, by the few and half-accidental patterns of
the industrious female knitter. In no single case is any attempt made
at the imitation of a leaf or flower, of animals, or any other simple
objects.”[83] At the date of those remarks the art of Europe’s
Palæolithic era was unknown; but with the arts of other primitive
races, and especially those of the American continent, in view, I then
added: “It is curious, indeed, and noteworthy, to find how entirely
every trace of imitative art is absent in British archaic relics; for it is
by no means an invariable characteristic of primitive arts.” Dr.
Hoffman, when commenting on aboriginal American art among the
Indians of California, adds: “I have not met with any attempts at
objective drawings or etchings which may be attributed to the
Tshuma Indians, who were the former occupants of the island;[84]
but ornamentations upon shells and bone beads, soap-stone pipes,
shell pendants, and other ornaments, seem to consist entirely of
straight or zigzag lines, cross-lines, circles,” etc. The earliest
examples of native metallurgy in Britain are to be found in the works
of the primitive goldsmith; but the same conventional ornamentation
which occurs on early pottery, is characteristic of the beautiful
personal ornaments of gold belonging, for the most part, to the first
period of working in metals. It is not till a late stage of the European
Bronze period that imitative art reappears, and zoomorphic
decorations become common.
The discovery in 1868, and subsequent years, of numerous
specimens of the artistic ability of the cave-men of palæolithic
Europe, revealed a singularly interesting phase of primitive history.
Remains of the so-called “Reindeer period” are now familiar to us
from many localities; for the range of this animal in palæolithic times
appears to have extended from the Baltic to the Pyrenees. But a
special interest was conferred on the first disclosures by the locality
itself, where the Vézère, an affluent of the river Dordogne, winds its
way through the cretaceous limestone, in which occur numerous
caves and rock-shelters, rich in remains of primitive art. In this
region of South-western France, where many historical and
legendary associations carry the fancy back to elder centuries, the
Dordogne unites with the Garonne at its estuary below Bordeaux.
The upper waters of the Dordogne form the boundary between
Limousin and Auvergne, and the Vézère is one of its highest
tributaries in Limousin. There, nearly in the latitude of Montreal, but
with the genial climate which, throughout the whole historic period,
has characterised Southern France, lie the caves of Cro-Magnon, La
Moustier, Gorge d’Enfer, Laugerie Haute and Basse, and La
Madelaine: the long-sealed art galleries of prehistoric Gaul. The
reindeer and the aurochs haunted its forests; the woolly rhinoceros
and the mammoth still frequented its glades; and the long-extinct
fossil horse was not only an object of the chase, but was possibly
already subdued to the companionship and service of man. Such, at
least, is the idea suggested by a scene graven on the portion of a
baton or staff, found by M. Lartet and Mr. Christy in La Madelaine
cave, which represents a man between two horses’ heads,
apparently walking past, with a staff or spear over his shoulder. Nor
were those man’s sole contemporaries.
The drawings of the ancient cave-men are of varying degrees of
merit, showing the efforts of the unskilled tyro as well as of the
practised artist. Some of the examples found at Laugerie Basse—as,
for instance, the assumed representation of an ibex, with its legs
folded as if sitting,—are the crude efforts of unpractised
draughtsmen, and would compare unfavourably with many examples
of graphic art, the work of modern Eskimo and Indian gravers and
draughtsmen. But other specimens—such as the mammoth from La
Madelaine cave, and the Alpine ibex and reindeers from Laugerie
Basse, in Southern France, and, still more, the remarkably spirited
drawing of the reindeer grazing, from the Kesserloch, near
Thayingen, sketched on a piece of reindeer horn,—evince powers of
observation, and a freedom of hand in sketching from nature, such
as would be found exceptional among pupils of our best training
schools of art. On this point my friend, Sir Noel Paton, writes me: “I
entirely concur in your view as to the immense superiority as works
of art of the engravings on horn and ivory found in the prehistoric
caves, over any modern work of the same kind which I have seen,
executed by the Eskimos or other savage, tribes of our own day. As
compared with the latter, the prehistoric productions are like the
swift and direct studies from nature of Landseer, compared with the
laboured scrawlings from memory of a rather dull schoolboy.”
I have elsewhere drawn attention to the fact that some of the
drawings of the Perigord cave-men and their palæolithic
contemporaries, especially, among the latter, the Kesserloch sketch
of a reindeer grazing, are left-hand drawings.[85] So far as this class
of evidence is of value, the examples from the caves in the valley of
the Vézère are exceptionally numerous. There, it may be, a family,
or possibly a tribe, dwelt, among whom left-handedness prevailed to
an unusual extent, along with a degree of skill and dexterity, such as
is frequently found to accompany the instinctive use of the left hand.
In this, as in other respects, the recovery of evidences of a well-
developed æsthetic faculty among the men of Europe’s Mammoth
and Reindeer period, furnishes materials for many suggestive
inferences; for we shall very imperfectly estimate the significance of
the primitive drawings so unexpectedly discovered, if we regard
them as no more than the pastimes of those ancient cave-men,
whose artistic ability they so unmistakably reveal. They are rather to
be classed with the picture-writing of the American aborigines—
including its most advanced Mexican stage abundantly illustrated in
Lord Kingsborough’s folios,—as one of the primitive supplements of
language among uncultured races. As such it is a form of visible
speech, and an important step in advance of the stage of gesture-
language. The historical value of the palæolithic drawings is
indisputable. They furnish a record, more trustworthy than any
written chronicle, of the strange conditions of life in a region familiar
to us throughout the whole historic period for its genial climate and
social civilisation. It is in this aspect, as a contemporary chronicling
of current events, that palæolithic art has its chief value. It furnishes
a graphic picturing of the habits of life, and of many of the attendant
circumstances of that remote period, recorded with such vivid
truthfulness, that we realise very definitely the character of its long-
extinct fauna, and, to some considerable extent, the occupations and
modes of life of the cave-men by whom they were hunted, and in
leisure hours were reproduced graven or carved, on bone, horn, or
ivory, or traced in free outline on slabs of schist or other soft stone.
Viewed simply as examples of imitative art among a people still
in the rudest Stone age, the drawings are significant and instructive.
They furnish evidence of observation and artistic capacity, and
consequently of intellectual powers capable of very different results
from anything that could be realised in the absence of all knowledge
of metallurgy, or of anything beyond the crudest appliances for
developing mechanical skill. The conditions of climate probably
forbade any attempt at agriculture. They were hunters, fowlers,
fishers, subsisting mainly, if not wholly, by the chase. They not only
successfully pursued the wild horse, the reindeer, and other swift-
footed herbivora, but assailed the cave-bear, the cave-lion, and other
formidable carnivora, as well as the huge rhinoceros and the
mammoth. They also made excursions to the sea-shore, and no
doubt left there shell mounds similar to those which have been
explored with such interesting results on the Danish coast; and
which have their New World equivalents on the seaboards of
Massachusetts, Georgia, and Florida, where at certain seasons the
Indians resorted to feast on the shell-fish. From their drawings and
carvings we not only learn this, but also that they were not
unfamiliar with the whale, the seal, and other marine fauna. The
presence of the whale and seal in the same latitude as the reindeer
need not surprise us. The occupation of Europe by palæolithic man
contemporary with the Elephas primigenius and other extinct
mammalia, belongs to an era when the relative levels of sea and
land, and the relations of the Atlantic coast-line to the ancient
continent, differed widely from their present conditions. If the genial
current of the Gulf Stream then reached the shores of Europe, its
influence extended over areas very diverse from those now affected
by it. But the range of the fauna of that Palæolithic era was a wide
one. Tusks of the mammoth and antlers of the reindeer occur in the
Scottish boulder-clay; and the discovery of skeletons of the whale far
inland in the carse of Stirling, accompanied in more than one case
by implements made of perforated stag’s horn, tells of the presence
of the Greenland whale on the ancient Scottish sea coast, while the
stag haunted its forests, and the Allophylian savage paddled his
canoe in estuaries marked for us now by old sea-margins that
preceded the last great rise of the land. Skulls and horns of the elk
occur in the Scottish peat-bogs, seemingly indistinguishable from
those of the Cervus alces, or North American moose.[86] As to the
reindeer, not only are its remains found in Scottish mosses and the
underlying marl, but they have been dug up in the ruined brochs, as
at Cill-Trolla, Sutherlandshire, and Keiss in Caithness. The favourite
haunts of the Greenland whale are in seas encumbered with floating
ice; and when they were stranded in the estuary of the Forth by a
tide rising on a shore-line now nearly thirty feet above the tide-mark
of the present day, the highlands of Scotland were capped with
perpetual snow, and great changes of level had still to occur. But
neither the whale nor the Eskimo retreated within the Arctic circle
because they could only be at home among polar ice and snow.
Remains of the whale in Scottish kitchen middens of greatly more
modern date show that it must have haunted the Scottish shores
when the temperature of the surrounding ocean differed little from
that of the present day. There is preserved in the Museum of the
Scottish Antiquaries a drinking-cup fashioned from the vertebra of a
whale, which was found in a weem, or subterranean dwelling, on the
Isle of Eday, Orkney, along with implements of stone, horn, bone,
bronze, and iron; and other evidences of the presence of the whale
in the Scottish seas are of frequent occurrence.
As to the ivory of the narwhal and the rostungr, or walrus, it was
in use by Scoto-Scandinavian carvers after the disappearance of the
reindeer from Scotland. A curious large sword, probably of the
fourteenth century, at Hawthornden, near Edinburgh, has the hilt
made of the narwhal’s tusk; and the famous Lewis chessmen, found
at Uig in the Isle of Lewis, as well as examples of chess and
tablemen recovered from time to time in other Scottish localities, are
all made of the walrus ivory, the “huel-bone” of Chaucer. But when
the whale haunted the shores to which the hunters of the Perigord
resorted, it is doubtful if Britain was an island. In that age of the
mammoth and the reindeer of the Pyrenees, when art flourished in
the valley of the Vézère, and men, scarcely less strange than the
long-extinct fauna on which they preyed, sheltered in their rock-
dwellings from the ice and snow, the relative levels of sea and land,
and the lines of the Atlantic coast, bore no relation to their present
aspect; for the old region of ice and snow was what is now familiar
to us as the vine-clad sunny land of France. All this we learn from
the archæological remains of those old times, and especially from
the carvings and gravings which, happily for us, were then executed,
whether for pastime or as actual records. Like many of the native
races of the American continent at the present day, the old cave-
dwellers employed their leisure time in carving in bone, horn, or
ivory; and like them too, as we believe, they applied their skill in
graphic art as a means of recording events and communicating facts
to others. The broad palinated antlers of the reindeer, prepared
sections of mammoth ivory, and slabs of schist, all furnished tablets
on which they not only delineated the objects of the chase, but
incidents and observations of daily experience. And if so, we have in
such drawings the germ of ideographic symbolism, and of
hieroglyphic writing. By just such a process of recording facts in a
form readily intelligible to others, the early dwellers in the Nile valley
originated the mode of object-drawing and ideographic chronicling,
from which hieroglyphic, demotic, and ultimately, phonetic writing
were evolved.
It is not solely by inference that we are led to surmise that the
ingenious draughtsmen of Southern France had a higher aim than
mere pastime in some, at least, of their graphic devices. The relics
recovered from the ancient caves include what appear to be tallies
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com