Mathematical Foundations

of Machine Learning

Seongjai Kim

Department of Mathematics and Statistics

Mississippi State University
Mississippi State, MS 39762 USA
Email: [email protected]

Updated: April 6, 2022

Seongjai Kim, Professor of Mathematics, Department of Mathematics and Statistics, Mississippi
State University, Mississippi State, MS 39762 USA. Email: [email protected].
Prologue

In organizing this lecture note, I am indebted by the following:

• S. R ASCHKA AND V. M IRJALILI, Python Machine Learning, 3rd Ed., 2019

[53].
• (Lecture note) http://fa.bianp.net/teaching/2018/eecs227at/, Dr. Fabian Pe-
dregosa, UC Berkeley
• (Lecture note) Introduction To Machine Learning, Prof. David Sontag,
MIT & NYU
• (Lecture note) Mathematical Foundations of Machine Learning, Dr. Justin
Romberg, Geoigia Tech

This lecture note will grow up as time marches; various core algorithms,
useful techniques, and interesting examples would be soon incorporated.

Seongjai Kim
April 6, 2022

i
ii
Contents

Title 2

Prologue i

Table of Contents vii

1 Introduction 1
1.1. Why and What in Machine Learning (ML)? . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.1. Inference problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2. Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.3. Machine learning examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2. Three Different Types of ML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.1. Supervised learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.2. Unsupervised learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.3. Python in 25 Minutes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2 Simple Machine Learning Algorithms for Classification 17

2.1. Binary Classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2. The Perceptron Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.1. The formal definition of an artificial neuron . . . . . . . . . . . . . . . . . . . . . 20
2.2.2. The perceptron learning rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.3. Problems with the perceptron algorithm . . . . . . . . . . . . . . . . . . . . . . . 24
2.3. Adaline: ADAptive LInear NEuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.1. Minimizing the cost function with the gradient descent method . . . . . . . . . 28
2.3.2. Convergence and optimization issues with the gradient descent method . . . . 29
Exercises for Chapter 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3 Gradient-based Methods for Optimization 35

3.1. Gradient Descent Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.1.1. The gradient descent method in 1D . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.1.2. A unifying principle: Surrogate minimization . . . . . . . . . . . . . . . . . . . . 41
3.1.3. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.1.4. The choice of step size and line search . . . . . . . . . . . . . . . . . . . . . . . . 44

iii
iv Contents

3.1.5. Optimizing optimization algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 46

3.2. Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.3. Variants of Optimization Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.3.1. Quasi-Newton methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.3.2. The stochastic gradient method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.4. The Levenberg–Marquardt Algorithm, for Nonlinear Least-Squares Problems . . . . . 61
3.4.1. The gradient descent method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.4.2. The Gauss-Newton method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.4.3. The Levenberg-Marquardt algorithm . . . . . . . . . . . . . . . . . . . . . . . . 64
Exercises for Chapter 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Project.1. Gaussian Sailing to Overcome Local Minima Problems . . . . . . . . . . . . . . . 67

4 Popular Machine Learning Classifiers 69

4.1. Logistic Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.1.1. The standard logistic sigmoid function . . . . . . . . . . . . . . . . . . . . . . . . 71
4.1.2. The logit function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.2. Classification via Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.2.1. The logistic cost function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.2.2. Gradient descent learning for logistic regression . . . . . . . . . . . . . . . . . . 79
4.2.3. Regularization: bias-variance tradeoff . . . . . . . . . . . . . . . . . . . . . . . . 80
4.3. Support Vector Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.3.1. Linear SVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.3.2. The method of Lagrange multipliers . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.3.3. Solution of the linear SVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.3.4. The inseparable case: soft-margin classification . . . . . . . . . . . . . . . . . . 91
4.3.5. Nonlinear SVM and kernel trick . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
4.3.6. Solving the dual problem with SMO . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.4. Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.4.1. Decision tree objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.4.2. Random forests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.5. k -Nearest Neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Exercises for Chapter 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

5 Quadratic Programming 113

5.1. Equality Constrained Quadratic Programming . . . . . . . . . . . . . . . . . . . . . . . 114
5.2. Direct Solution for the KKT System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.2.1. Symmetric factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.2.2. Range-space approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5.2.3. Null-space approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.3. Linear Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.3.1. Convergence theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.3.2. Graph theory: Estimation of the spectral radius . . . . . . . . . . . . . . . . . . 124
Contents v

5.3.3. Eigenvalue locus theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

5.3.4. Regular splitting and M-matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
5.4. Iterative Solution of the KKT System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.4.1. Krylov subspace methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.4.2. The transforming range-space iteration . . . . . . . . . . . . . . . . . . . . . . . 130
5.5. Active Set Strategies for Convex QP Problems . . . . . . . . . . . . . . . . . . . . . . . 131
5.5.1. Primal active set strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5.6. Interior-point Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5.7. Logarithmic Barriers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Exercises for Chapter 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

6 Data Preprocessing in Machine Learning 141

6.1. General Remarks on Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . 142
6.2. Dealing with Missing Data & Categorical Data . . . . . . . . . . . . . . . . . . . . . . . 144
6.2.1. Handling missing data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
6.2.2. Handling categorical data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
6.3. Feature Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6.4. Feature Selection: Selecting Meaningful Variables . . . . . . . . . . . . . . . . . . . . . 147
6.4.1. Ridge regression vs. LASSO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
6.4.2. Sequential backward selection (SBS) . . . . . . . . . . . . . . . . . . . . . . . . . 152
6.5. Feature Importance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Exercises for Chapter 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

7 Feature Extraction: Data Compression 157

7.1. Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
7.1.1. Computation of principal components . . . . . . . . . . . . . . . . . . . . . . . . 159
7.1.2. Dimensionality reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
7.1.3. Explained variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
7.2. Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
7.2.1. Interpretation of the SVD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
7.2.2. Properties of the SVD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
7.2.3. Computation of the SVD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
7.2.4. Application of the SVD to image compression . . . . . . . . . . . . . . . . . . . . 178
7.3. Linear Discriminant Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
7.3.1. Fisher’s LDA (classifier): two classes . . . . . . . . . . . . . . . . . . . . . . . . . 181
7.3.2. Fisher’s LDA: the optimum projection . . . . . . . . . . . . . . . . . . . . . . . . 183
7.3.3. LDA for multiple classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
7.3.4. The LDA: dimensionality reduction . . . . . . . . . . . . . . . . . . . . . . . . . . 196
7.4. Kernel Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
7.4.1. Principal components of the kernel PCA . . . . . . . . . . . . . . . . . . . . . . . 198
7.4.2. Further concerns with the kernel PCA . . . . . . . . . . . . . . . . . . . . . . . . 201
Exercises for Chapter 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
vi Contents

8 Cluster Analysis 207

8.1. Basics for Cluster Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
8.1.1. Quality of clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
8.1.2. Types of clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
8.1.3. Types of clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
8.1.4. Objective functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
8.2. K-Means Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
8.2.1. (Basic) K-Means clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
8.2.2. Solutions to initial centroids problem . . . . . . . . . . . . . . . . . . . . . . . . 223
8.2.3. Bisecting K-Means algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
8.2.4. Limitations of K-Means . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
8.2.5. K-Medoids algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
8.2.6. CLARA and CLARANS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
8.3. Hierarchical Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
8.3.1. AGNES: Agglomerative clustering . . . . . . . . . . . . . . . . . . . . . . . . . . 233
8.4. DBSCAN: Density-based Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
8.5. Self-Organizing Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
8.5.1. Kohonen networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
8.6. Cluster Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
8.6.1. Internal measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
8.6.2. External measures of cluster validity . . . . . . . . . . . . . . . . . . . . . . . . 261
Exercises for Chapter 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

9 Neural Networks and Deep Learning 265

9.1. Basics for Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
9.2. Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
9.2.1. Sigmoid neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
9.2.2. A simple network to classify handwritten digits . . . . . . . . . . . . . . . . . . 274
9.2.3. Implementing a network to classify digits [48] . . . . . . . . . . . . . . . . . . . 278
9.2.4. Toward deep learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
9.3. Back-Propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
9.3.1. Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
9.3.2. The cost function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
9.3.3. The four fundamental equations behind
back-propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
9.4. Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
9.4.1. Introducing convolutional networks . . . . . . . . . . . . . . . . . . . . . . . . . 293
9.4.2. CNNs in practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
Exercises for Chapter 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
Project.2. Development of New Machine Learning Algorithms . . . . . . . . . . . . . . . . . 303

Bibliography 311
Contents vii

Index 317
viii Contents
Chapter 1

Introduction

What are we “learning” in Machine Learning (ML)?

This is a hard question to which we can only give a somewhat fuzzy answer.
But at a high enough level of abstraction, there are two answers:

• Algorithms, which solve some kinds of inference problems

• Models for data sets.

These answers are so abstract that they are probably completely unsatis-
fying. But let’s (start to) clear things up, by looking at some particular
examples of “inference" and “modeling" problems.

Contents of Chapter 1
1.1. Why and What in Machine Learning (ML)? . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2. Three Different Types of ML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3. Python in 25 Minutes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

1
2 Chapter 1. Introduction

This Course
• This course will deal with some of actual ML algorithms.
• However, we will put an emphasis on basic mathematical concepts
on which these algorithms are built. In particular, to really under-
stand anything about ML, you need to have a very good grasp of
– calculus,
– linear algebra, and
– probabilistic inference (i.e. the mathematical theory of probabil-
ity and how to use it).
• We will cover the key parts of these branches of applied mathematics
in depth and in the context of ML.
• More specifically, the mathematical topics to be treated in this
course can be broken down into four basic subject areas:
– Representations for data and operators that map data to decisions,
estimates, or both. We will start with a thorough discussion of linear
representations; these are important/useful by themselves, and also
are used as building-blocks for nonlinear representations. Here is
where we will need a lot of linear algebra and its extension.
– Estimation. What does it mean to estimate a parameter for/from a
data set? We will try to put this question on as firm a mathematical
footing as we can, using the language of statistics.
– Modeling.
– Computing. Finally, we will get a look at how computations for
solving problems arising in ML are actually done. We will look at
some basic algorithms from optimization, and some algebraic tech-
niques from numerical linear algebra.
In particular, we will experiment examples in several chapters
in Python Machine Learning, 3rd Ed. [53].

Required Coding/Reading : Read Chapter 1, Python Machine Learning, 3rd Ed. Install
required packages, including scikit-learn.
1.1. Why and What in Machine Learning (ML)? 3

1.1. Why and What in Machine Learning (ML)?

1.1.1. Inference problems
Definition 1.1. Statistical inference is the process of using data
analysis to deduce properties of an underlying probability distribution.
Inferential statistical analysis infers properties of a population, for ex-
ample by testing hypotheses and deriving estimates.

Loosely speaking, inference problems take in data, then output

some kind of decision or estimate. The output of a statistical infer-
ence is a statistical proposition; here are some common forms of statistical
proposition.
• a point estimate
• an interval estimate
• a credible interval
• rejection of a hypothesis
• classification or clustering of the data points into discrete groups

Example 1.2. Inference algorithms can answer the following.

(a) Does this image have a tree in it?

(b) What words are in this picture?

(c) If I give you a recording of somebody speaking, can you produce text of
what they are saying?
4 Chapter 1. Introduction

Remark 1.3. What does a machine learning algorithm do?

Machine learning algorithms are not algorithms for performing in-
ference. Rather, they are algorithms for building inference algo-
rithms from examples. An inference algorithm takes a piece of data
and outputs a decision (or a probability distribution over the decision
space). A supervised machine learning algorithm takes many pieces of
data, each pre-labeled with a decision, and outputs a mapping from data
space to decision space.

1.1.2. Modeling
A second type of problem associated with machine learning (ML) might
be roughly described as:

Given a data set, how can I succinctly describe it (in a quantita-

tive, mathematical manner)?

One example is regression analysis. Most models can be broken into two
categories:

• Geometric models. The general problem is that we have example

data points
x 1 , x 2 , · · · , x N ∈ RD
and we want to find some kind of geometric structure that (ap-
proximately) describes them. Here is an example: given a set of vectors,
what (low dimensional) subspace comes closest to containing them?
• Probabilistic models. The basic task here is to find a probabil-
ity distribution that describes the data set {x n }. The classical name
for this problem is density estimation – given samples of a random
variable, estimate its probability density function (pdf). This gets ex-
tremely tricky in high dimensions (large values of D ) or when there are
dependencies between the data points. Key to solving these problems
is choosing the right way to describe your probability model.
1.1. Why and What in Machine Learning (ML)? 5

Note: In both cases above, having a concise model can go a tremen-

dous way towards simplifying the data – each data point in RD might be
described in many fewer than D parameters.
• As a rule, if you have a simple and accurate model, this is
tremendously helping in solving inference problems, because
there are fewer parameters to consider and/or estimate.
• The categories can also overlap with or complement each other. It is
often the case that the same model can be interpreted as a geometric
model or a probabilistic model.

1.1.3. Machine learning examples

• Classification: from data to discrete classes
– Spam filtering
– Object detection (e.g., face)
– Weather prediction (e.g., rain, snow, sun)
• Regression: predicting a numeric value
– Stock market
– Weather prediction (e.g., Temperature)
• Ranking: comparing items
– Web search (keywords)
– Given an image, find similar images
• Collaborative Filtering (e.g., Recommendation systems)
• Clustering: discovering structure in data
– Clustering points or images
– Clustering web search results
• Embedding: visualizing data
– Embedding images, words
• Structured prediction: from data to discrete classes
– Speech/image recognition
– Natural language processing
6 Chapter 1. Introduction

1.2. Three Different Types of ML

Example 1.4. Three different types of ML:

• Supervised learning: classification, regression

• Unsupervised learning: clustering
• Reinforcement learning: chess engine

Figure 1.1: Three different types of ML (by methods)

1.2. Three Different Types of ML 7

1.2.1. Supervised learning

Assumption. Given a data set {(xi , yi )}, ∃ a relation f : X → Y .

Supervised learning:
(
Given : Training set {(xi , yi ) | i = 1, · · · , N }
(1.1)
Find : bf : X → Y , a good approximation to f

Figure 1.2: Supervised learning and prediction.

Figure 1.3: Classification and regression.

8 Chapter 1. Introduction

Example 1.5. Regression

• Dataset: Consider a simple dataset: 10 points generated from a sin func-
tion, with noise

• Wanted: Find a regression model

f :X =R→Y =R (1.2)

• Question: How should we pick the hypothesis space, the set of possible
functions f ?
– Often, there exist infinitely many hypotheses
– Let’s select f from PM , polynomials of degree ≤ M .
– Which one is best?
1.2. Three Different Types of ML 9

• Error: misfit

– We measure the error using a loss function L (y, by ).

– For regression, a common choice is squared loss: for each (xi , yi ),

L (yi , f (xi )) = ((yi − f (xi ))2 .

– The empirical loss of the function f applied to the training data is

then:
N N
def 1 1X
X
EMS == L (yi , f (xi )) = (yi − f (xi ))2 , (1.3)
N N
i=1 i=1

which is the mean square error.

Figure 1.4: Learning curve.

10 Chapter 1. Introduction

Key Idea 1.6. Training and test performance. Assume that each
training and test example–label pair (x, y) is drawn independently at
random from the same (but unknown) population of examples and labels.
Represent this population as a probability distribution p(x, y), so that:

(xi , yi ) ∼ p(x, y).

Then, given a loss function L :

N
1X
• Empirical (training) loss = L (yi , f (xi )).
N
i=1
(Also called the empirical risk, R(f
b , DN ).)
• Expected (test) loss = E(x,y)∼p {L (y, f (x))}.
(Also called the risk R(f ).)

• Ideally, learning chooses the hypothesis that minimizes the risk.

(But this is impossible to compute!)
• The empirical risk is a good (unbiased) estimate of the risk
(by linearity of expectation).
• The principle of empirical risk minimization reads

f ∗ (DN ) = arg min R(f

b , DN ). (1.4)
f ∈F
1.2. Three Different Types of ML 11

Remark 1.7. Key Issues in Machine Learning

• How do we choose a hypothesis space?
– Often we use prior knowledge to guide this choice
– The ability to answer to the next two questions also affects the
choice
• How can we gauge the accuracy of a hypothesis on unseen test-
ing data?
– The previous example showed that choosing the hypothesis which
simply minimizes training set error (i.e., empirical risk mini-
mization) is not optimal.
– This question is a main topic in learning theory.
• How do we find the best hypothesis?
– This is an algorithmic question, at the intersection of mathemat-
ics, computer science, and optimization research.

Proposition 1.8. Occam’s Razor Principle (a.k.a. Law of parsimony):

“One should not increase, beyond what is necessary,

the number of entities required to explain anything.”

• William of Occam: A monk living in the 14-th century, England

• When many solutions are available for a given problem, we should
select the simplest one
• But what do we mean by simple?
• We will use prior knowledge of the problem to solve to define what
is a simple solution (Example of a prior: smoothness).
12 Chapter 1. Introduction

1.2.2. Unsupervised learning

Note:
• In supervised learning, we know the right answer beforehand when
we train our model, and in reinforcement learning, we define a mea-
sure of reward for particular actions by the agent.
• In unsupervised learning, however, we are dealing with unlabeled
data or data of unknown structure. Using unsupervised learn-
ing techniques, we are able to explore the structure of our data
to extract meaningful information without the guidance of a known
outcome variable or reward function.
• Clustering is an exploratory data analysis technique that allows
us to organize a pile of information into meaningful subgroups
(clusters) without having any prior knowledge of their group mem-
berships.

Figure 1.5: Clustering.

1.3. Python in 25 Minutes 13

1.3. Python in 25 Minutes

• As of the end of 2021, Python, R, and Java are most popular skills
when it comes to ML and data science jobs, with Python being the
leader.
• R was built as a statistical language and more functional, whereas
Python is more object-oriented. R is the choice of language for a quick
prototype, but for long term Python is the most preferred language.
• Many users with Matlab experience, in particular, will be capa-
ble of taking the step to using modules in scientific-computing-and-
graphics packages such as numpy, scipy, and matplotlib, because the
commands are sufficiently similar to Matlab.

Here we deal with one of popular optimization problems, a Python code for
solving it, followed by some essentials of Python programming.

Figure 1.6: Plots of the Rosenbrock function f (x, y) = (1 − x)2 + 100 (y − x 2 )2 .

Consider the Rosenbrock function in two variables defined by

f (x, y) = (1 − x)2 + 100 (y − x 2 )2 . (1.5)
The Rosenbrock function is commonly used when evaluating the perfor-
mance of optimization algorithms. Reasons for this include the fact that
• Its minimizer (b y ) = (1, 1) is found in curved valley, and so minimizing
x, b
the function is non-trivial, and
• The Rosenbrock function is included in the scipy.optimize package (as
rosen), as well as its gradient (rosen_der) and its Hessian (rosen_hess).
14 Chapter 1. Introduction
rosenbrock_opt.py
1 import numpy as np
2 from scipy import optimize as opt
3 import time
4

5 x0 = np.array([-1., 2.])
6

7 def rosen(xn):
8 return (1.-xn[0])**2+100*(xn[1]-xn[0]**2)**2
9

10 # default: method='BFGS'
11 t0 = time.time()
12 res1 = opt.minimize(rosen, x0, tol=1e-7,
13 options={'disp':True})
14 print res1.x
15 print 'Method = %s; E-time = %.4f\n' %('BFGS',time.time()-t0)
16

17 t0 = time.time()
18 res2 = opt.minimize(opt.rosen, x0, method='Newton-CG', tol=1e-7,
19 jac=opt.rosen_der, hess=opt.rosen_hess)
20 print 'Method = %s; E-time = %.4f' %('Newton-CG',time.time()-t0)
21 print res2

The result of rosenbrock_opt.py

1 Optimization terminated successfully.
2 Current function value: 0.000000
3 Iterations: 36
4 Function evaluations: 192
5 Gradient evaluations: 48
6 [0.99999552 0.99999104]
7 Method = BFGS; E-time = 0.0046
8

9 Method = Newton-CG; E-time = 0.0337

10 fun: 9.003538427961096e-20
11 jac: array([ 9.97554821e-08, -5.00779640e-08])
12 message: 'Optimization terminated successfully.'
13 nfev: 169
14 nhev: 148
15 nit: 148
16 njev: 316
17 status: 0
18 success: True
19 x: array([1., 1.])

Figure 1.7: rosenbrock_opt.py and its execution result.

1.3. Python in 25 Minutes 15

Note:
• A Python code is implemented to find the minimizer of the Rosenbrock
function (1.5), saved in the file rosenbrock_opt.py, as shown in Fig-
ure 1.7 (top).
• Under the Linux OS, you can execute it by giving the following com-
mand on the Terminal:
python rosenbrock_opt.py Enter
• Its result is attached in Figure 1.7 (bottom).
• Here are some remarks for the Python code.
– Lines 1-3: Python requires to import modules and packages to
use.
– Line 5: It shows how to make an array. Try np.ndarray or
np.zeros for a Matlab-like multi-dimensional array creation.
– Lines 7-8: It shows how to define (def) functions.
– Indentation: Leading whitespace (spaces and tabs) at the begin-
ning of a logical line is used to compute the indentation level of
the line, which in turn is used to determine the grouping of state-
ments.
– Line 10 (Comment): After #, the rest of the line is ignored by
Python.
– Lines 12-13 and 18-19: They call the built-in function minimize, a
method of optimize.
– Lines 14-15 and 20-21: They show how to make print for object-
oriented modules and data.

Example 1.9. Look at Lines 12-13 of rosenbrock_opt.py in Figure 1.7. The

algorithm converges in 36 iterations, as shown in the figure. Find the iter-
ates {x n | n = 0, 1, · · · , 36} and produce a figure in Python that superposes
the iterates (points) over a contour plot of the Rosenbrock function.
Hint : You may have to import matplotlib or pylab. PyLab is embedded
inside matplotlib1 and provides a Matlab-like experience for the user.

1
Matplotlib is the most popular plotting (data visualization) routine package for Python. Many users
with Matlab experience will be capable of taking the step to using import matplotlib.pyplot as plt, as the
commands are sufficiently similar to Matlab.
16 Chapter 1. Introduction

Some other Python essentials are shown below in Figure 1.8.

• Sequence datatypes: list, tuple, string >>> (1,2,3)*3

[Lists] are defined using square brackets (1, 2, 3, 1, 2, 3, 1, 2, 3)
(and commas) • Reference semantics
>>> li = ["abc", 14, 4.34, 23] >>> a = [1, 2, 3]
(Tuples) are defined using parentheses >>> b = a
>>> a.append(4)
(and commas) >>> b
>>> tu = (23, (4,5), 'a', 4.1, -7) [1, 2, 3, 4]
"Strings" are defined using quotes Be aware with lists and numpy arrays!
(", ’, or """) • numpy, range, and iteration
>>> st = 'Hello World' >>> range(8)
>>> st = "Hello World" [0, 1, 2, 3, 4, 5, 6, 7]
>>> st = """This is a multi-line >>> import numpy as np
string that uses triple quotes.""" >>> for k in range(np.size(li)):
• Retrieving elements ... li[k]
... <Enter>
>>> li[0] abc
abc 14
>>> tu[1],tu[2],tu[-2] 4.34
(4, 5) a 4.1 23
>>> st[25:36]
ng that use • array and deepcopy
• Slicing >>> from copy import deepcopy
>>> A = np.array([1,2,3])
>>> tu[1:4] >>> B = A
((4, 5), 'a', 4.1) >>> C = deepcopy(A)
• The + and ∗ operators >>> A *= 4
>>> [1, 2, 3]+[4, 5, 6,7] >>> B
[1, 2, 3, 4, 5, 6, 7] [ 4 8 12]
>>> "Hello" + " " + 'World' >>> C
Hello World [1 2 3]

Figure 1.8: Python essentials.

Chapter 2

Simple Machine Learning Algorithms

for Classification

In this chapter, we will make use of one of the first algorithmically described
machine learning algorithms for classification, the perceptron and adap-
tive linear neurons (adaline). We will start by implementing a perceptron
step by step in Python and training it to classify different flower species in
the Iris dataset.

Contents of Chapter 2
2.1. Binary Classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2. The Perceptron Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3. Adaline: ADAptive LInear NEuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Exercises for Chapter 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

17
18 Chapter 2. Simple Machine Learning Algorithms for Classification

2.1. Binary Classifiers

Definition 2.1. A binary classifier is a function which can decide
whether or not an input vector belongs to some specific class (e.g.,
spam/ham).
• Binary classification often refers to those classification tasks that
have two class labels. (two-class classification)
• It is a type of linear classifier, i.e. a classification algorithm that
makes its predictions based on a linear predictor function
combining a set of weights with the feature vector.
• Linear classifiers are artificial neurons.

Remark 2.2. Neurons are interconnected nerve cells that are involved
in the processing and transmitting of chemical and electrical signals.
Such a nerve cell can be described as a simple logic gate with binary
outputs;
• multiple signals arrive at the dendrites,
• they are integrated into the cell body,
• and if the accumulated signal exceeds a certain threshold, an output
signal is generated that will be passed on by the axon.

Figure 2.1: A schematic description of a neuron.

2.1. Binary Classifiers 19

Linear classifiers
As artificial neurons, they have the following characteristics:
• Inputs are feature values: x
• Each feature has a weight: w
• Weighted sum (integration) is the activation
X
activationw (x) = wj xj = w · x (2.1)
j

• Decision/output: If the activation is


Positive ⇒ class 1
Negative ⇒ class 2

Unknowns, in ML:

Training : w
Prediction : activationw (x)

Examples:
• Perceptron
• Adaline (ADAptive LInear NEuron)
• Support Vector Machine (SVM) ⇒ nonlinear decision boundaries, too
20 Chapter 2. Simple Machine Learning Algorithms for Classification

2.2. The Perceptron Algorithm

The perceptron is an algorithm for supervised learning of binary classi-
fiers.
• 1957: Perceptron algorithm is invented by Frank Rosenblatt, Cornell
Aeronautical Laboratory
– Built on work of Hebbs (1949)
– Improved by Widrow-Hoff (1960): Adaline
• 1960: Perceptron Mark 1 Computer – hardware implementation
• 1969: Minksky & Papert’s book shows perceptrons limited to linearly
separable data, and Rosenblatt dies in boating accident
• 1970’s: Learning methods for two-layer neural networks

2.2.1. The formal definition of an artificial neuron

Definition 2.3. We can pose the perceptron as a binary classifier,
in which we refer to our two classes as 1 (positive class) and −1 (negative
class) for simplicity.
• Input values: x = (x1 , x2 , · · · , xm )T
• Weight vector: w = (w1 , w2 , · · · , wm )T
• Net input: z = w1 x1 + w2 x2 + · · · + wm xm
• Activation function: φ(z), defined by

1 if z ≥ θ,
φ(z) = (2.2)
−1 otherwise,

where θ is a threshold.

For simplicity, we can bring the threshold θ in (2.2) to the left side of the
equation; define a weight-zero as w0 = −θ and x0 = 1 and reformulate as

1 if z ≥ 0,
φ(z) = z = w T x = w0 x0 + w1 x1 + · · · + wm xm . (2.3)
−1 otherwise,
In the ML literature, the negative threshold (w0 = −θ) is called the bias
unit, while x0 (that always has value 1) is the bias or intercept.
2.2. The Perceptron Algorithm 21

2.2.2. The perceptron learning rule

The whole idea behind the Rosenblatt’s thresholded perceptron model is
to use a reductionist approach to mimic how a single neuron in the brain
works: it either fires or it doesn’t.
Summary 2.4. Thus, Rosenblatt’s initial perceptron rule is fairly sim-
ple and can be summarized by the following steps:
1. Initialize the weights to 0 or small random numbers.
2. For each training sample x (i) ,
y (i) (:= φ(w T x (i) )).
(a) Compute the output value b
(b) Update the weights.

The update of the weight vector w can be more formally written as:

w := w + ∆w, ∆w = η (y (i) − b
y (i) ) x (i) , (2.4)

where η is the learning rate, 0 < η < 1, y (i) is the true class label of the i -th
training sample, and by (i) denotes the predicted class label. (See (2.15) below
for the update with the Gradient descent method.)

Remark 2.5. A simple thought experiment for the perceptron

learning rule:
• Let the perceptron predict the class label correctly. Then y (i) − b
y (i) = 0
so that the weights remain unchanged.
• Let the perceptron make a wrong prediction. Then
(i) (i)
∆wj = η (y (i) − b
y (i) ) xj = ±2 η xj

so that the weight wj is pushed towards the direction of the positive

or negative target class, respectively.
22 Chapter 2. Simple Machine Learning Algorithms for Classification

Note: It is important to note that the convergence of the perceptron

is only guaranteed if the two classes are linearly separable and the
learning rate is sufficiently small. If the two classes can’t be sep-
arated by a linear decision boundary, we can set a maximum number
of passes over the training dataset (epochs) and/or a threshold for the
number of tolerated misclassifications.
Definition 2.6. (Linearly separable dataset). A dataset {(x (i) , y (i) )}
is linearly separable if there exist w
b and γ such that
T
y (i) w
b x (i) ≥ γ > 0, ∀ i, (2.5)

where γ is called the margin.

Figure 2.2: Linearly separable dataset.

Definition 2.7. (More formal/traditional definition). Let X and Y

be two sets of points in an m-dimensional Euclidean space. Then X and Y
are linearly separable if there exist m +1 real numbers w1 , w2 , · · · , wm , k
Xm
such that every point x ∈ X satisfies wj xj > k and every point y ∈ Y
j=1
m
X
satisfies wj yj < k .
j=1
2.2. The Perceptron Algorithm 23

Theorem 2.8. Assume the data set D = {(x (i) , y (i) )} is linearly separable
with margin γ, i.e.,
T
∃ w,
b kw
b k = 1, y (i) w
b x (i) ≥ γ > 0, ∀ i. (2.6)

Suppose that kx (i) k ≤ R, ∀ i , for some R > 0. Then, the maximum num-
ber of mistakes made by the perceptron algorithm is bounded by R 2 /γ 2 .

Proof. Assume the perceptron algorithm makes yet a mistake for (x (`) , y (`) ).
Then
kw (`+1) k2 = kw (`) + η (y (`) − b
y (`) )x (`) k2
T
= kw (`) k2 + kη (y (`) − b
y (`) )x (`) k2 + 2η (y (`) − b
y (`) )w (`) x (`) (2.7)
≤ kw (`) k2 + kη (y (`) − b
y (`) )x (`) k2 ≤ kw (`) k2 + (2η R)2 ,

where we have used

T
(y (`) − b
y (`) )w (`) x (`) ≤ 0. (2.8)
(See Exercise 1.) The inequality (2.7) implies

kw (`) k2 ≤ ` · (2η R)2 . (2.9)

(Here we have used kw (0) k = 0.) On the other hand,

T T T T
b w (`+1) = w
w b w (`) + η (y (`) − b
y (`) )w
b x (`) ≥ w
b w (`) + 2η γ ,

which implies
T
b w (`) ≥ ` · (2η γ )
w (2.10)
and therefore
kw (`) k2 ≥ `2 · (2η γ )2 . (2.11)
It follows from (2.9) and (2.11) that ` ≤ R 2 /γ 2 .

Properties of the perceptron algorithm

• Separability: some weights get the training set perfectly correct.
• Convergence: if the training is linearly separable, perceptron will
eventually converge.
24 Chapter 2. Simple Machine Learning Algorithms for Classification

2.2.3. Problems with the perceptron algorithm

Inseparable Datasets

• Noise: if the data isn’t linearly

separable, no guarantees of con-
vergence or accuracy
• Frequently the training data is
linearly separable! Why?
– For example, when the num-
ber of examples is much
smaller than the number of
features
– Perceptron can significantly
overfit the data

Averaged perceptron is an
algorithmic modification that
helps with both issues
– Average the weight vectors
across, either all or a last
part of iterations Figure 2.3
2.2. The Perceptron Algorithm 25

Optimal Separator?

Quesiton. Which of these linear separators is optimal?

Figure 2.4

Example 2.9. Support Vector Machine (Cortes & Vapnik, 1995) chooses
the linear separator with the largest margin.

Figure 2.5

We will consider the SVM in Section 4.3.

26 Chapter 2. Simple Machine Learning Algorithms for Classification

How Multi-class Classification?

Figure 2.6: Classification for three classes.

One-versus-all (one-versus-rest) classification

Learning: learn 3 classifiers

• − vs {◦, +} ⇒ weights w −
• + vs {◦, −} ⇒ weights w +
• ◦ vs {+, −} ⇒ weights w ◦

Prediction: for a new data sample

x,

Figure 2.7: Three weights: w − , w + , and w ◦ . y = arg max φ(w Ti x).

b
i ∈{−,+,◦}

OvA (OvR) is readily applicable for classification of general n classes, n ≥ 2.

2.3. Adaline: ADAptive LInear NEuron 27

2.3. Adaline: ADAptive LInear NEuron

• (Widrow & Hoff, 1960)
• Weights are updated based on linear activation: e.g.,

φ(w T x) = w T x.

That is, φ is the identity function.

• Adaline algorithm is particularly interesting because it illustrates the
key concept of defining and minimizing continuous cost functions,
which will lay the groundwork for understanding more ad-
vanced machine learning algorithms for classification, such as lo-
gistic regression and support vector machines, as well as regression
models.
• Continuous cost functions allow the ML optimization to incorporate
advanced mathematical techniques such as calculus.

Figure 2.8: Perceptron vs. Adaline

28 Chapter 2. Simple Machine Learning Algorithms for Classification

2.3.1. Minimizing the cost function with the gradient de-

scent method
Key Idea 2.10. One of the key ingredients of supervised machine learn-
ing algorithms is to define an objective function that is to be optimized
during the learning process. This objective function is often a cost func-
tion that we want to minimize.

Definition 2.11. In the case of Adaline, we can define the cost function
J to learn the weights as the Sum of Squared Errors (SSE) between
the calculated outcomes and the true class labels:
1 X (i)
2
J (w) = y − φ(w T x (i) ) . (2.12)
2
i

Algorithm 2.12. The Gradient Descent Method uses −∇J (w) for
the search direction (update direction):

w := w + ∆w = w − η∇J (w), (2.13)

where η > 0 is the step length (learning rate).

Computation of ∇J (w) and ∆w :

The partial derivatives of the cost function J read
∂J (w) X
(i)
=− y (i) − φ(w T x (i) ) xj . (2.14)
∂ wj
i

(Here we have used φ = I.) Thus

X
∆w = −η∇J (w) = η y (i) − φ(w T x (i) ) x (i) .


(2.15)
i

Compare the above Widrow-Hoff rule with the perceptron rule in (2.4) on
p. 21.
2.3. Adaline: ADAptive LInear NEuron 29

2.3.2. Convergence and optimization issues with the gra-

dient descent method
• Depending on choices of certain algorithmic parameters, the gradi-
ent descent method may fail to converge to the the global minimizer.
• Data characteristics often determines both successability and speed
of convergence; data preprocessing operations may improve conver-
gence.
• For large-scale data, the gradient descent method is computationally
expensive; a popular alternative is the stochastic gradient descent
method.

♦ Hyperparameters
Definition 2.13. In ML, a hyperparameter is a parameter whose
value is set before the learning process begins. Thus it is an algorithmic
parameter. Examples are
• The learning rate (η)
• The number of maximum epochs/iterations (n_iter)

Figure 2.9: Well-chosen learning rate vs. a large learning rate

Hyperparameters must be selected to optimize the learning process:

• to converge fast to the global minimizer,
• avoiding overfit.
30 Chapter 2. Simple Machine Learning Algorithms for Classification

♦ Standardization: a feature scaling preprocessing

Definition 2.14. Gradient descent is one of the many algorithms that
benefit from feature scaling. Here, we will consider a feature scaling
method called standardization, which gives each feature of the data
the property of a standard normal distribution.
For example, to standardize the j -th feature, we simply need to subtract
the sample mean µj from every training sample and divide it by its stan-
dard deviation σj :
(i)
(i)
xj − µj
xj =
e . (2.16)
σj
Then,
(i)
{e
xj | i = 1, 2, · · · , n} ∼ N (0, 1). (2.17)

Figure 2.10: Standardization, which is one of data normalization techniques.

The gradient method has a tendency to converge faster with the

standardized data.
2.3. Adaline: ADAptive LInear NEuron 31

♦ Stochastic gradient descent method

Note: Earlier, we learned how to minimize a cost function by taking
a step into the opposite direction of a gradient that is calculated from
the whole training set; this is why this approach is sometimes also
referred to as batch gradient descent. Now imagine we have a very
large dataset with millions of data points, which is not uncommon in
many machine learning applications. Running with the gradient descent
method can be computationally quite expensive in such scenarios since
we need to reevaluate the whole training dataset each time we take one
step towards the global minimum. A popular alternative to the batch
gradient descent algorithm is the stochastic gradient descent (SGD).

Algorithm 2.15. The SGD method updates the weights incrementally

for each training sample:

Given a training set D = {(x (i) , y (i) ) | i = 1, 2, · · · , n}

1. For i = 1, 2, · · · , n
(2.18)
w = w + η y (i) − φ(w T x (i) ) x (i) ;
2. If not convergent, shuffle D and goto 1;

• The SGD method updates the weights based on a single training ex-
ample, while the gradient descent method uses the sum of the accu-
mulated errors over all samples.
• The SGD method typically reaches convergence much faster be-
cause of the more frequent weight updates.
• Since each search direction is calculated based on a single training
example, the error surface is smoother (not noisier) than in the gradi-
ent descent method, which can also have the advantage that the SGD
method can escape shallow local minima more readily.
• To obtain accurate results via the SGD method, it is important to
present it with data in a random order, which is why we want
to shuffle the training set for every epoch to prevent cycles.
• In the SGD method, the learning rate η is often set adaptively, de-
creasing over iteration k . For example, ηk = c1 /(k + c2 ).
32 Chapter 2. Simple Machine Learning Algorithms for Classification

♦ Mini-batch learning
Definition 2.16. A compromise between the batch gradient descent and
the SGD is the so-called mini-batch learning. Mini-batch learning can
be understood as applying batch gradient descent to smaller subsets of
the training data – for example, 32 samples at a time.

The advantage over batch gradient descent is that convergence is reached

faster via mini-batches because of the more frequent weight updates. Fur-
thermore, mini-batch learning allows us to replace the for-loop over the
training samples in stochastic gradient descent by vectorized operations
(vectorization), which can further improve the computational efficiency of
our learning algorithm.

Required Coding/Reading : Read Chapter 2 and experiment all examples in it, Python
Machine Learning, 3rd Ed.
2.3. Adaline: ADAptive LInear NEuron 33

Exercises for Chapter 2

2.1. Verify (2.8).

Hint : We assumed that the parameter w (`) gave a mistake on x (`) . For example, let
T
w (`) x (`) ≥ 0. Then we must have (y (`) − b
y (`) ) < 0. Why?
2.2. Experiment all the examples on pp. 38–51, Python Machine Learning, 3rd Ed.. Through
the examples, you will learn
(a) Gradient descent rule for Adaline,
(b) Feature scaling techniques, and
(c) Stochastic gradient descent rule for Adaline.

To get the Iris dataset, you have to use some lines on as earlier pages from 29.
2.3. Perturb the dataset (X ) by a random Gaussian noise Gσ of an observable σ (so as for
Gσ (X ) not to be linearly separable) and do the examples in Exercise 2 again.
34 Chapter 2. Simple Machine Learning Algorithms for Classification
Chapter 3

Gradient-based Methods for

Optimization

Optimization is the branch of research-and-development that aims to solve

the problem of finding the elements which maximize or minimize a given
real-valued function, while respecting constraints. Many problems in engi-
neering and machine learning can be cast as optimization problems, which
explains the growing importance of the field. An optimization problem is
the problem of finding the best solution from all feasible solutions.
In this chapter, we will discuss details about
• Gradient descent method,
• Newton method, and
• Their variants.

Contents of Chapter 3
3.1. Gradient Descent Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.2. Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.3. Variants of Optimization Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.4. The Levenberg–Marquardt Algorithm, for Nonlinear Least-Squares Problems . . . . . 61
Exercises for Chapter 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Project.1. Gaussian Sailing to Overcome Local Minima Problems . . . . . . . . . . . . . . . 67

35
36 Chapter 3. Gradient-based Methods for Optimization

3.1. Gradient Descent Method

The first method that we will de-

scribe is one of the oldest meth-
ods in optimization: gradient de-
scent method, a.k.a steepest de-
scent method. The method was
suggested by Augustin-Louis Cauchy
in 1847 [40]. He was a French math-
ematician and physicist who made
pioneering contributions to mathe-
matical analysis. Motivated by the
need to solve “large" quadratic prob-
lems (6 variables) that arise in As-
tronomy, he invented the method of
gradient descent. Today, this method
is used to comfortably solve problems
with thousands of variables.
Figure 3.1: Augustin-Louis Cauchy

Problem 3.1. (Optimization Problem). Let Ω ⊂ Rd , d ≥ 1. Given a

real-valued function f : Ω → R, the general problem of finding the value
that minimizes f is formulated as follows.

min f (x). (3.1)

x ∈Ω

In this context, f is the objective function (sometimes referred to as

loss function or cost function). Ω ⊂ Rd is the domain of the function
(also known as the constraint set).
3.1. Gradient Descent Method 37

Example 3.2. (Rosenbrock function). For example, the Rosenbrock

function in the two-dimensional (2D) space is defined as1

f (x, y) = (1 − x)2 + 100 (y − x 2 )2 . (3.2)

Figure 3.2: Plots of the Rosenbrock function f (x, y) = (1 − x)2 + 100 (y − x 2 )2 .

Note: The Rosenbrock function is commonly used when evaluating the

performance of an optimization algorithm; because
• its minimizer x = np.array([1.,1.]) is found in curved valley, and
so minimizing the function is non-trivial, and
• the Rosenbrock function is included in the scipy.optimize pack-
age (as rosen), as well as its gradient (rosen_der) and its Hessian
(rosen_hess).

1
The Rosenbrock function in 3D is given as f (x, y, z) = [(1 − x)2 + 100 (y − x 2 )2 ] + [(1 − y)2 + 100 (z − y 2 )2 ],
which has exactly one minimum at (1, 1, 1). Similarly, one can define the Rosenbrock function in
general N -dimensional spaces, for N ≥ 4, by adding one more component for each enlarged dimen-
NX−1
(1 − xi )2 + 100(xi+1 − xi2 )2 , where x = [x1 , x2 , · · · , xN ] ∈ RN . See Wikipedia
 
sion. That is, f (x) =
i=1
(https://en.wikipedia.org/wiki/Rosenbrock_function) for details.
38 Chapter 3. Gradient-based Methods for Optimization

Remark 3.3. (Gradient). The gradient ∇f is a vector (a direction to

move) that is
• pointing in the direction of greatest increase of the function, and
• zero (∇f = 0) at local maxima or local minima.

The goal of the gradient descent method is to address directly the process
of minimizing the function f , using the fact that −∇f (x) is the direction of
steepest descent of f at x . Given an initial point x 0 , we move it to the
direction of −∇f (x 0 ) so as to get a smaller function value. That is,

x 1 = x 0 − γ ∇f (x 0 ) ⇒ f (x 1 ) < f (x 0 ).

We repeat this process till reaching at a desirable minimum. Thus the

method is formulated as follows.

Algorithm 3.4. (Gradient descent method). Given an initial point

x 0 , find iterates x n+1 recursively using

x n+1 = x n − γ ∇f (x n ), (3.3)

for some γ > 0. The parameter γ is called the step length or the learn-
ing rate.

To understand the basics of gradient descent (GD) method thoroughly, we

start with the algorithm for solving

• unconstrained minimization problems

• defined in the one-dimensional (1D) space.
3.1. Gradient Descent Method 39

3.1.1. The gradient descent method in 1D

Problem 3.5. Consider the minimization problem in 1D:

min f (x), x ∈ S, (3.4)

x

where S is a closed interval in R. Then its gradient descent method reads

xn+1 = xn − γ f 0 (xn ). (3.5)

Picking the step length γ : Assume that the step length was chosen to
be independent of n, although one can play with other choices as well. The
question is how to select γ in order to make the best gain of the method. To
turn the right-hand side of (3.5) into a more manageable form, we invoke
Taylor’s Theorem:2
ˆ x+t
0
f (x + t) = f (x) + t f (x) + (x + t − s) f 00 (s) ds. (3.6)
x
00
Assuming that |f (s)| ≤ L , we have
t2
f (x + t) ≤ f (x) + t f 0 (x) + L.
2
Now, letting x = xn and t = −γ f 0 (xn ) reads
f (xn+1 ) = f (xn − γ f 0 (xn ))
1
≤ f (xn ) − γ f 0 (xn ) f 0 (xn ) + L [γ f 0 (xn )]2 (3.7)
2
0 2
 L 2
= f (xn ) − [f (xn )] γ − γ .
2
The gain (learning) from the method occurs when
L 2 2
γ− γ >0 ⇒ 0<γ< , (3.8)
2 L
and it will be best when γ − L2 γ 2 is maximal. This happens at the point
1
γ= (3.9)
L
2
Taylor’s Theorem with integral remainder: Suppose f ∈ C n+1 [a, b] and x0 ∈ [a, b]. Then, for every
n ˆ x
X f (k ) (x0 ) 1
x ∈ [a, b], f (x) = (x − x0 )k + Rn (x), Rn (x) = (x − s)n f (n+1) (s) ds.
k! n! x0
k =0
40 Chapter 3. Gradient-based Methods for Optimization

It follows from (3.7) and (3.9) that

1 0
f (xn+1 ) ≤ f (xn ) − [f (xn )]2 . (3.10)
2L

Remark 3.6. (Convergence of gradient descent method).

Thus it is obvious that the method defines a sequence of points {xn } along
which {f (xn )} decreases.
• If f is bounded from below and the level sets of f are bounded, {f (xn )}
converges; so does {xn }. That is, there is a point b
x such that

lim xn = b
x. (3.11)
n→∞

• Now, we can rewrite (3.10) as

[f 0 (xn )]2 ≤ 2L [f (xn ) − f (xn+1 )]. (3.12)

Since f (xn ) − f (xn+1 ) → 0, also f 0 (xn ) → 0.

• When f 0 is continuous, using (3.11) reads

f 0 (b
x ) = lim f 0 (xn ) = 0, (3.13)
n→∞

which implies that the limit b

x is a critical point.
• The method thus generally finds a critical point but that could still
be a local minimum or a saddle point. Which it is cannot be decided
at this level of analysis.
3.1. Gradient Descent Method 41

3.1.2. A unifying principle: Surrogate minimization

Now, we aim to solve an optimization problem as in (3.1):

min f (x). (3.14)

x ∈Ω

Key Idea 3.7. Start at an initial estimate x 0 and successively minimize

an approximating function Qn (x) [36]:

x n+1 = arg min Qn (x). (3.15)

x ∈Ω

We will call Qn a surrogate function. It is also known as a merit

function. A good surrogate function should be:
• Easy to optimize.
• Flexible enough to approximate a wide range of functions.

Gradient descent method: Approximates the objective function near x n

with a quadratic surrogate of the form
1
Qn (x) = c n + G n · (x − x n ) + (x − x n )T (x − x n ), (3.16)
2γ
which coincides with f in its value and first derivative, i.e.,
Qn (x n ) = f (x n ) ⇒ c n = f (x n ),
(3.17)
∇Qn (x n ) = ∇f (x n ) ⇒ G n = ∇f (x n ).

The gradient descent method thus updates its iterates minimizing the fol-
lowing surrogate function:
1
Qn (x) = f (x n ) + ∇f (x n ) · (x − x n ) + kx − x n k2 . (3.18)
2γ
Differentiating the function and equating to zero reads

x n+1 = arg min Qn (x) = x n − γ ∇f (x n ). (3.19)

x
42 Chapter 3. Gradient-based Methods for Optimization

Remark 3.8. In theory, the step length γ can be found as in (3.9):

1
γ= , where L = max k∇2 f (x)k. (3.20)
L x

Here || · || denotes an induced matrix norm and ∇2 f (x) is the Hessian

of f defined by
 2 
∂ f ∂ 2f ∂ 2f
 ∂x 2 ∂x ∂x · · · ∂x ∂x 
 1 1 2 1 d
 2 2 2
 ∂ f ∂ f ∂ f 

···
∂ x2 ∂ x1 ∂ x2 2 ∂ x2 ∂ xd  ∈ Rd ×d .
 
∇2 f = 
 . (3.21)
 .. .. . .. .
.. 

 . 
 2 
 ∂ f ∂ 2f ∂ 2f 
···
∂ xd ∂ x1 ∂ xd ∂ x2 ∂ xd 2
However, in practice, the computation of the Hessian (and L ) can be ex-
pensive.

We can implement the full gradient descent algorithm as follows. The algo-
rithm has only one free parameter: γ.

Algorithm 3.9. (The Gradient Descent Algorithm).

input: initial guess x 0 , step size γ > 0;

for n = 0, 1, 2, · · · do
x n+1 = x n − γ ∇f (x n ); (3.22)
end for
return x n+1 ;
3.1. Gradient Descent Method 43

3.1.3. Examples
Here we examine the convergence of gradient descent on three examples:
a well-conditioned quadratic, an badly-conditioned quadratic, and a non-
convex function, as shown by Dr. Fabian Pedregosa, UC Berkeley.

γ = 0.2
Figure 3.3: On a well-conditioned quadratic function, the gradient descent converges in a
few iterations to the optimum

γ = 0.02
Figure 3.4: On a badly-conditioned quadratic function, the gradient descent converges and
takes many more iterations to converge than on the above well-conditioned problem. This
is partially because gradient descent requires a much smaller step size on this problem
to converge.

γ = 0.02
Figure 3.5: Gradient descent also converges on a badly-conditioned non-convex problem.
Convergence is slow in this case.
44 Chapter 3. Gradient-based Methods for Optimization

3.1.4. The choice of step size and line search

Note: The convergence of the gradient descent method can be extremely

sensitive to the choice of step size. It often requires to choose the
step size adaptively: the step size would better be chosen small in re-
gions of large variability of the gradient, while in regions with small
variability we would like to take it large.

Strategy 3.10. Backtracking line search procedures allow to select

a step size depending on the current iterate and the gradient. In this
procedure, we select an initial (optimistic) step size γn and evaluate the
following inequality (known as sufficient decrease condition):
γn
f (x n − γn ∇f (x n )) ≤ f (x n ) − k∇f (x n )k2 . (3.23)
2
If this inequality is verified, the current step size is kept. If not, the step
size is divided by 2 (or any number larger than 1) repeatedly until (3.23)
is verified. To get a better understanding, refer to (3.10) on p. 40, with
(3.9).

The gradient descent algorithm with backtracking line search then becomes
Algorithm 3.11. (The Gradient Descent Algorithm, with Back-
tracking Line Search).
input: initial guess x 0 , step size γ > 0;
for n = 0, 1, 2, · · · do

initial step size estimate γn ;
 while (TRUE) do
if f (x n − γn ∇f (x n )) ≤ f (x n ) − γ2n k∇f (x n )k2




 break; (3.24)
 else γn = γn /2;
end while
x n+1 = x n − γn ∇f (x n );
end for
return x n+1 ;
3.1. Gradient Descent Method 45

The following examples show the convergence of gradient descent with the
aforementioned backtracking line search strategy for the step size.

Figure 3.6: On a well-conditioned quadratic function, the gradient descent converges in a

few iterations to the optimum. Adding the backtracking line search strategy for the step
size does not change much in this case.

Figure 3.7: In this example we can clearly see the effect of the backtracking line search
strategy: once the algorithm in a region of low curvature, it can take larger step sizes. The
final result is a much improved convergence compared with the fixed step-size equivalent.

Figure 3.8: The backtracking line search also improves convergence on non-convex prob-
lems.

See Exercise 1 on p. 65.

46 Chapter 3. Gradient-based Methods for Optimization

3.1.5. Optimizing optimization algorithms

Remark 3.12. (Multiple local minima problem). Although you can

choose the step size smartly, there is no guarantee for your algorithm
to converge to the desired solution (the global minimum), particularly
when the objective is not convex.
Here, we consider the so-called Gaussian homotopy continuation
method [45], which may overcome the local minima problem for cer-
tain classes of optimization problems.
• The method begins by trying to find a convex approximation of an
optimization problem, using a technique called Gaussian smooth-
ing.
• Gaussian smoothing converts the cost function into a related func-
tion, each of whose values is a weighted average of all the sur-
rounding values.
• This has the effect of smoothing out any abrupt dips or ascents in
the cost function’s graph, as shown in Figure 3.9.
• The weights assigned the surrounding values are determined by a
Gaussian function, or normal distribution.

Figure 3.9: Smooth sailing, through a Gaussian smoothing.

However, there will be many ways to incorporate Gaussian smoothing; a

realization of the method will be challenging, particularly for ML optimiza-
tion. See Project.1 (p. 67).
3.2. Newton’s Method 47

3.2. Newton’s Method

Scheme 3.13. The Newton’s method is an iterative method to solve
the unconstrained optimization problem in (3.1), p. 36, when f is twice
differentiable. In Newton’s method, we approximate the objective with a
quadratic surrogate of the form
1
Qn (x) = c n + G n · (x − x n ) + (x − x n )T H n (x − x n ). (3.25)
2γ
Compared to gradient descent, the quadratic term is not fixed to be the
identity but instead incorporates an invertible matrix H n .

A reasonable condition to impose on this surrogate function is that at x n it

coincides with f at least in its value and first derivatives, as in (3.17). An
extra condition the method imposes is that

H n = ∇2 f (x n ), (3.26)

where ∇2 f is the Hessian of f defined as in (3.21). Thus the Newton’s

method updates its iterates minimizing the following surrogate function:
1
Qn (x) = f (x n ) + ∇f (x n ) · (x − x n ) + (x − x n )T ∇2 f (x n ) (x − x n ). (3.27)
2γ

We can find the optimum of the function deriving and equating to zero. This
way we find (assuming the Hessian is invertible)
−1
x n+1 = arg min Qn (x) = x n − γ ∇2 f (x n )

∇f (x n ). (3.28)
x

Remark 3.14. Where applicable, Newton’s method converges much

faster towards a local maximum or minimum than the gradient descent.
In fact, every local minimum has a neighborhood such that, if we start
within this neighborhood, Newton’s method with step size γ = 1 con-
verges quadratically assuming the Hessian is invertible and Lipschitz
continuous.
48 Chapter 3. Gradient-based Methods for Optimization

Remark 3.15. The Newton’s method can be seen as to find the critical
b such that ∇f (x
points of f , i.e., x b) = 0. Let
x n+1 = x n + ∆x. (3.29)

Then
∇f (x n+1 ) = ∇f (x n + ∆x) = ∇f (x n ) + ∇2 f (x n ) ∆x + O(|∆x |2 ).

Truncating high-order terms of ∆x and equating the result to zero reads

−1
∆x = − ∇2 f (x n ) ∇f (x n ). (3.30)

Implementation of Newton’s method : Only the difference from the gra-

dient descent algorithm is to compute the Hessian matrix ∇2 f (x n ) to be ap-
plied to the gradient.

Algorithm 3.16. (Newton’s method).

input: initial guess x 0 , step size γ > 0;

for n = 0, 1, 2, · · · do
 −1
x n+1 = x n − γ ∇2 f (x n ) ∇f (x n ); (3.31)
end for
return x n+1 ;
3.2. Newton’s Method 49

For the three example functions in Section 3.1.3, the Newton’s method per-
forms better as shown in the following.

γ=1
Figure 3.10: In this case the approximation is exact and it converges in a single iteration.

γ=1
Figure 3.11: Although badly-conditioned, the cost function is quadratic; it converges in a
single iteration.

γ=1
Figure 3.12: When the Hessian is close to singular, there might be some numerical insta-
bilities. However, it is better than the result of the gradient descent method in Figure 3.5.
50 Chapter 3. Gradient-based Methods for Optimization

Claim 3.17. The Hessian (or Hessian matrix) describes the local
curvature of a function. The eigenvalues and eigenvectors of the Hes-
sian have geometric meaning:
• The first principal eigenvector (corresponding to the largest eigen-
value in modulus) is the direction of greatest curvature.
• The last principal eigenvector (corresponding to the smallest eigen-
value in modulus) is the direction of least curvature.
• The corresponding eigenvalues are the respective amounts of these
curvatures.
The eigenvectors of the Hessian are called principal directions, which
are always orthogonal to each other. The eigenvalues of the Hessian
are called principal curvatures and are invariant under rotation and
always real-valued.

Observation 3.18. Let a Hessian matrix H ∈ Rd ×d be positive definite

and its eigenvalue-eigenvector pairs be given as {(λj , uj )}, j = 1, 2, · · · , d .
Then, given a vector v ∈ Rd , it can be expressed as
d
X
v= ξj u j ,
j=1

and therefore
d
−1
X 1
H v = ξj uj , (3.32)
λj
j=1

where components of v in leading principal directions of H have been

diminished with larger factors. Thus the angle measured from H −1 v to
the least principal direction of H becomes smaller than the angle
measured from v . It is also true when v is the gradient vector (in fact,
the negation of the gradient vector).
3.2. Newton’s Method 51

Note: The above observation can be rephrased mathematically as fol-

lows. Let ud be the least principal direction of H . Then

angle(ud , H −1 v) < angle(ud , v), ∀ v, (3.33)

where  a·b 
angle(a, b) = arccos .
ka k kb k
This implies that by setting v = −∇f (x n ), the adjusted vector H −1 v is a
rotation (and scaling) of the steepest descent vector towards the least
curvature direction.

Figure 3.13: The effect of the Hessian inverse H −1 .

Claim 3.19. The net effect of H −1 is to rotate and scale the gradi-
ent vector to face towards the minimizer by a certain degree, which may
make the Newton’s method converge much faster than the gradient de-
scent method.

Example 3.20. One can easily check that at each point (x, y) on the ellip-
soid
(x − h)2 (y − k )2
z = f (x, y) = + , (3.34)
a2 b2
 −1
the vector − ∇2 f (x, y) ∇f (x, y) is always facing towards the minimizer
(h, k ). See Exercise 2.
52 Chapter 3. Gradient-based Methods for Optimization

3.3. Variants of Optimization Algorithms

Note: The central issue with Newton’s method is that we need to be able
to compute the inverse Hessian matrix.
• For ML applications, the dimensionality of the problem can be of
the order of thousands or millions; computing the Hessian or its
inverse is often impractical.
• Because of these reasons, Newton’s method is rarely used in prac-
tice to optimize functions corresponding to large problems.
• Luckily, the above algorithm can still work even if the Hessian is
replaced by a good approximation.

3.3.1. Quasi-Newton methods

In quasi-Newton methods, the Hessian matrix of second derivatives doesn’t
need to be evaluated directly. Instead, the Hessian matrix is approxi-
mated using updates specified by gradient evaluations (or approximate
gradient evaluations).

The BFGS algorithm (1970)

Note: One of the most popular quasi-Newton methods is the BFGS

algorithm, which is named after Charles George Broyden [9], Roger
Fletcher [17], Donald Goldfarb [20], and David Shanno [61].

Key Idea 3.21. As a byproduct of the optimization, we observe many

gradients. Can we use these gradients to iteratively construct an esti-
mate of the Hessian?
3.3. Variants of Optimization Algorithms 53

Derivation of BFGS algorithm :

• At each iteration of the method, we consider the surrogate function:

1
Qn (x) = c n + G n · (x − x n ) + (x − x n )T H n (x − x n ), (3.35)
2
where in this case H n is an approximation to the Hessian matrix,
which is updated iteratively at each stage.
• A reasonable thing to ask to this surrogate is that its gradient coin-
cides with ∇f at the last two iterates x n+1 and x n :
∇Qn+1 (x n+1 ) = ∇f (x n+1 ),
(3.36)
∇Qn+1 (x n ) = ∇f (x n ).
From the definition of Qn+1 , we have

∇Qn+1 (x n+1 ) − ∇Qn+1 (x n ) = G n+1 − ∇Qn+1 (x n ) = −H n+1 (x n − x n+1 ).

Thus we reach at the following condition on H n+1 :

H n+1 (x n+1 − x n ) = ∇f (x n+1 ) − ∇f (x n ), (3.37)

which is the secant equation.

• Let s n = x n+1 − x n and y n = ∇f (x n+1 ) − ∇f (x n ) . Then, it is required to sat-
isfy the curvature condition

y n · s n > 0. (3.38)

• If the function f is not strongly convex, then the above curvature condition
has to be enforced explicitly. In order to maintain the symmetry and
positive definiteness of H n+1 , the update formula can be chosen as3

H n+1 = H n + αuuT + β vv T . (3.39)

• Imposing the secant condition H n+1 s n = y n and with (3.39), we get the
update equation of H n+1 :
y n y Tn (H n s n )(H n s n )T
H n+1 = Hn + − . (3.40)
y n · sn sn · Hnsn
3
Rank-one matrices: Let A be an m × n matrix. Then rank(A ) = 1 if and only if there exist column vectors
v ∈ Rm and w ∈ Rn such that A = vw T .
54 Chapter 3. Gradient-based Methods for Optimization

• Let B n = H −1
n , the inverse of H n . Then, applying the Sherman-Morrison
formula, we can update B n+1 = H − 1
n+1 as follows.

 s n y Tn   y n s Tn  s n s Tn
B n+1 = I− Bn I − + . (3.41)
y n · sn y n · sn y n · sn

See Exercise 4.

Now, we are ready to summarize the BFGS algorithm.

Algorithm 3.22. (The BFGS algorithm). The n-th step:
1. Obtain the search direction: p n = B n (−∇f (x n )).
2. Perform line-search to find an acceptable stepsize γn .
3. Set s n = γn p n and update x n+1 = x n + s n .
4. Get y n = ∇f (x n+1 ) − ∇f (x n ).
5. Update B = H −1 :
 s n y Tn   y n s Tn  s n s Tn
B n+1 = I− Bn I − + .
y n · sn y n · sn y n · sn

The resulting algorithm is a method which combines the low-cost of gra-

dient descent with the favorable convergence properties of Newton’s
method.
3.3. Variants of Optimization Algorithms 55

Examples, with the BFGS algorithm

Figure 3.14: BFGS, on the well-conditioned quadratic objective function.

Figure 3.15: On the badly-conditioned quadratic problem, the BFGS algorithm quickly
builds a good estimator of the Hessian and is able to converge very fast towards the opti-
mum. Note that this, just like the Newton method (and unlike gradient descent), BFGS
does not seem to be affected (much) by a bad conditioning of the problem.

Figure 3.16: Even on the ill-conditioned nonconvex problem, the BFGS algorithm also
converges extremely fast, with a convergence that is more similar to Newton’s method
than to gradient descent.
56 Chapter 3. Gradient-based Methods for Optimization

3.3.2. The stochastic gradient method

The stochastic gradient method (SGM), introduced by Robbins and Monro
in 1951 [54], is one of the most widely-used methods for large-scale opti-
mization and is one of the main methods behind the current AI revolution.
Note: The SGM was considered earlier in Section 2.3.2, as a variant
of the gradient descent method for Adaline classification. Here we will
discuss it in details for more general optimization problems.

The stochastic gradient method (a.k.a. stochastic gradient descent or

SGD) can be used to solve optimization problems in which the objective
function is of the form
f (x) = E[fi (x)],
where the expectation is taken with respect to i . The most common case is
when i can take a finite number of values, in which the problem becomes
m
1 X
min f (x), f (x) = fi (x). (3.42)
x ∈Rp m
i=1

The SGM can be motivated as an approximation to gradient descent in

which at each iteration we approximate the gradient as

∇f (x n ) ≈ ∇fi (x n )
3.3. Variants of Optimization Algorithms 57

We can write the full stochastic gradient algorithm as follows. The algo-
rithm has only one free parameter: γ.
Algorithm 3.23. (Stochastic Gradient Descent).

input: initial guess x 0 , step size sequence γn > 0;

for n = 0, 1, 2, · · · do
Choose i ∈ {1, 2, · · · , m} uniformly at random;
(3.43)
x n+1 = x n − γn ∇fi (x n );
end for
return x n+1 ;

The SGD can be much more efficient than gradient descent in the case in
which the objective consists of a large sum, because at each iteration we
only need to evaluate a partial gradient and not the full gradient.
Example 3.24. A least-squares problem can be written in the form accept-
able by SGD since
m
1 2 1 X T
kA x − b k = (Ai x − b i )2 , (3.44)
m m
i=1

where Ai is the i -th row of A .

58 Chapter 3. Gradient-based Methods for Optimization

Step size for the SGD

• The choice of step size is one of the most delicate aspects of the SGD.
For the SGD, the backtracking line search is not an option since it
would involve to evaluate the objective function at each iteration, which
destroys the computational advantage of this method.
• Two popular step size strategies exist for the SGD: constant step size and
decreasing step size.
(a) Constant step size: In the constant step size strategy,

γn = γ

for some pre-determined constant γ.

The method converges very fast to neighborhood of a local minima
and then bounces around. The radius of this neighborhood will de-
pend on the step size γ [37, 43].
(b) Decreasing step size: One can guarantee convergence to a local
minimizer choosing a step size sequence that satisfies
∞
X ∞
X
γn = ∞ and γn2 < ∞. (3.45)
n=1 n=1

The most popular sequence to verify this is

C
γn = , (3.46)
n
for some constant C . This is often referred to as a decreasing step-
size sequence, although in fact the sequence does not need to be
monotonically decreasing.
3.3. Variants of Optimization Algorithms 59

Examples, with SGD

γ = 0.2
Figure 3.17: For the well-conditioned convex problem, stochastic gradient with constant
step size converges quickly to a neighborhood of the optimum, but then bounces around.

Figure 3.18: Stochastic Gradient with decreasing step sizes is quite robust to the choice of
step size. On one hand there is really no good way to set the step size (e.g., no equivalent
of line search for Gradient Descent) but on the other hand it converges for a wide range of
step sizes.
60 Chapter 3. Gradient-based Methods for Optimization

Convergence of the SGD

Quesiton. Why does the SGD converge, despite its update being a very
rough estimate of the gradient?

To answer this question, we must first understand the unbiasedness prop-

erty of its update.

Definition 3.25. (Unbiasedness of the SGD update). Let En denote

the expectation with respect to the choice of random sample (i ) at itera-
tion n. Then since the index i is chosen uniformly at random, we have
m
X
En [∇fin (x n )] = ∇fi (x n )P(in = i)
i=1
m (3.47)
1 X
= ∇fi (x n ) = ∇f (x n )
m
i=1

This is the crucial property that makes SGD work. For a full proof, see
e.g. [7].
3.4. The Levenberg–Marquardt Algorithm, for Nonlinear Least-Squares Problems 61

3.4. The Levenberg–Marquardt Algorithm, for

Nonlinear Least-Squares Problems
The Levenberg-Marquardt algorithm (LMA), a.k.a. the damped least-
squares (DLS) method, is used for the solution of nonlinear least squares
problems which arise especially in curve fitting.
In fitting a function b y (x; p) of an independent variable x and a parameter
n
vector p ∈ R to a set of m data points (xi , yi ), it is customary and conve-
nient to minimize the sum of the weighted squares of the errors (or
weighted residuals) between the measured data yi and the curve-fit func-
tion b
y (xi ; p). m h
X yi − b
y (xi ; p) i2
f (p) =
ηi
i=1 (3.48)
= (y − y b(p))T W (y − y b(p))
= y T W y − 2y T W y b(p)T W y
b(p) + y b(p),
where ηi is the measurement error for yi and the weighting matrix W is
defined as
W = diag{1/ηi2 } ∈ Rm×m .
However, more formally, W can be set to the inverse of the measurement
error covariance matrix; more generally, the weights can be set to pursue
other curve-fitting goals.
Note: The goodness-of-fit measure in (3.48) is called the chi-squared
error criterion because the sum of squares of normally-distributed
variables is distributed as the χ-squared distribution.

If the function b
y (x; p) is nonlinear in the model parameters p , then the
minimization of the χ-squared function f with respect to the parameters
must be carried out iteratively:

p := p + ∆p. (3.49)

The goal of each iteration is to find the parameter update ∆p that reduces
f . We will begin with the gradient descent method and the Gauss-Newton
method.
62 Chapter 3. Gradient-based Methods for Optimization

3.4.1. The gradient descent method

Recall: The gradient descent method is a general minimization method

which updates parameter values in the steepest “downhill” direction: the
direction opposite to the gradient of the objective function.
• The gradient descent method converges well for problems with sim-
ple objective functions.
• For problems with thousands of parameters, gradient descent meth-
ods are sometimes the only workable choice.

The gradient of the objective function with respect to the parameters is

∂ ∂
b(p))T W
f = 2(y − y (y − y
b(p))
∂p ∂p
h ∂y
b(p) i
T (3.50)
= −2(y − yb(p)) W
∂p
= −2(y − yb(p))T W J,

where J = ∂ y∂(p) ∈ Rm×n is the Jacobian matrix. The parameter update ∆p

b
p
that moves the parameters in the direction of steepest descent is given by

∆p gd = γ J T W (y − y
b(p)), (3.51)

where γ > 0 is the step length.

3.4. The Levenberg–Marquardt Algorithm, for Nonlinear Least-Squares Problems 63

3.4.2. The Gauss-Newton method

The Gauss-Newton method is a method for minimizing a sum-of-
squares objective function.
• It assumes that the objective function is approximately quadratic
in the parameters near the optimal solution [6].
• For moderately-sized problems, the Gauss-Newton method typically
converges much faster than gradient-descent methods [44].

The function evaluated with perturbed model parameters may be locally

approximated through a first-order Taylor series expansion.
h ∂y
b(p) i
b(p + ∆p) ≈ y
y b(p) + ∆p = y
b(p) + J∆p. (3.52)
∂p
Substituting the approximation into (3.48), p. 61, we have
f (p + ∆p) ≈ y T W y − 2y T W y b(p)T W y
b(p) + y b(p)
(3.53)
b(p))T W J∆p + (J∆p)T W J∆p.
−2(y − y

The parameter update ∆p can be found from ∂ f /∂ ∆p = 0:

∂
f (p + ∆p) ≈ −2(y − y
b(p))T W J + 2(J∆p)T W J = 0, (3.54)
∂ ∆p
and therefore the resulting normal equation for the Gauss-Newton update
reads
[J T W J] ∆p gn = J T W (y − y
b(p)). (3.55)

Note: The approximation for f in (3.53) is quadratic in the parameter

perturbation ∆p . So, the Hessian of the chi-squared fit criterion is
approximately J T W J ∈ Rn×n . Here, we require m ≥ n; otherwise, the
approximate Hessian J T W J must be singular.
64 Chapter 3. Gradient-based Methods for Optimization

3.4.3. The Levenberg-Marquardt algorithm

The Levenberg-Marquardt algorithm adaptively varies the parameter
updates between the gradient descent and the Gauss-Newton methods:

[J T W J + λI] ∆p lm = J T W (y − y
b(p)), (3.56)

where λ ≥ 0 is the damping parameter.

Remark 3.26. Small values of λ result in a Gauss-Newton update and
large values of it result in a gradient descent update. The damping pa-
rameter λ is often initialized to be large so that first updates are small
steps in the steepest-descent direction.
• If any iteration happens to result in a worse approximation f (p +
∆p lm ) > f (p), then λ is increased.
• Otherwise, as the solution improves, λ is decreased, the Levenberg-
Marquardt method approaches the Gauss-Newton method, and the
solution typically accelerates to the local minimum [42, 44].

Acceptance of the step: There have been many variations of the Levenberg-
Marquardt method, particularly for acceptance criteria. For example, at the
k -th step, we first compute
f (p) − f (p + ∆p lm )
ρk (∆p lm ) =
(y − y b)T W (y − y b) − (y − y b − J∆p lm )T W (y − yb − J∆p lm )
(3.57)
f (p) − f (p + ∆p lm )
= , (using (3.56))
∆p Tlm (λk ∆p lm + J T W (y − yb(p))

and then the step is accepted when ρk (∆p lm ) > ε0 , for some threshold ε0 > 0
(e.g., ε0 = 0.1). An example implementation reads
Initialize p 0 and λ0 ; (e.g. λ0 = 0.01)

 Get ∆p lm from (3.56);

 Get ρk from (3.57);

 If ρk > ε0 : (3.58)

 p k +1 = p k + ∆p lm ; λk +1 = λk · max[1/3, 1 − (2ρk )3 )]; νk = 2;
otherwise: λk +1 = λk νk ; νk +1 = 2νk ;
3.4. The Levenberg–Marquardt Algorithm, for Nonlinear Least-Squares Problems 65

Exercises for Chapter 3

3.1. (Gradient descent method). Implement the gradient descent algorithm (3.22) and
the gradient descent algorithm with backtracking line search (3.24).

(a) Compare their performances with the Rosenbrock function in 2D (3.2).

(b) Find an effective strategy for initial step size estimate for (3.24).

3.2. (Net effect of the inverse Hessian matrix). Verify the claim in Example 3.20.
3.3. (Newton’s method). Implement a line search version of the Newton’s method (3.31)
with the Rosenbrock function in 2D.

(a) Recall the results in Exercise 1. With the backtracking line search, is the New-
ton’s method better than the gradient descent method?
(b) Now, we will approximate the Hessian matrix by its diagonal. That is,

∂ 2f ∂ 2f ∂ 2f
   
0
 ∂x 2  def  ∂ x
2 ∂x ∂y 
Dn =  (x n ) ≈ ∇2 f (x n ) ==  (x n ). (3.59)
 ∂ 2f   ∂ 2f ∂ 2f 
0
∂y 2 ∂y ∂x ∂y 2

How does the Newton’s method perform when the Hessian matrix is replaced by
Dn ?

3.4. (BFGS update). Consider H n+1 and B n+1 in (3.40) and (3.41), respectively:

y n y Tn (H n s n )(H n s n )T
H n+1 = H n + − ,
y n · sn sn · Hnsn
 s n y Tn   y n s Tn  s n s Tn
B n+1 = I− Bn I − + .
y n · sn y n · sn y n · sn

(a) Verify the the secant condition H n+1 s n = y n .

(b) Verify H n+1 B n+1 = I, assuming that H n B n = I.
66 Chapter 3. Gradient-based Methods for Optimization

3.5. (Curve fitting). Consider a set of data consisting of four points

1 2 3 4
xi 0.0 1.0 2.0 3.0
yi 1.1 2.6 7.2 21.1

Fit the data with a fitting function of the form

y (x, p) = a e bx , where p = [a, b],

b (3.60)

by minimizing the sum of the square-errors:

(a) Implement the three algorithms introduced in Section 3.4: the gradient descent
method, the Gauss-Newton method, and the Levenberg-Marquardt method.
(b) Ignore the weight vector W , i.e., set W = I .
(c) For each method, set p 0 = [a0 , b0 ] = [1.0, 0.8].
(d) Discuss how to choose γ for the gradient descent and λ for the Levenberg-Marquardt.

Hint : The Jacobian for this example must be in R4×2 ; more precisely,
 
1 0
∂  eb a eb 
J= b(x, p) = 
y  e 2b 2a e 2b  ,

∂p
e 3b 3a e 3b

b(x, p) = [a, a e b , a e 2b , a e 3b ]T from (3.60) and {xi }.

because we have y
3.4. The Levenberg–Marquardt Algorithm, for Nonlinear Least-Squares Problems 67

Project.1. Gaussian Sailing to Overcome Local

Minima Problems
A Gaussian smoothing for a 2D function f (x, y) can be achieved by storing
the function to a 2D-array A , and applying a built-in function in scipy:
scipy.ndimage.filters.gaussian_filter
which requires a parameter σ (standard deviation for Gaussian kernel).
Alternatively, one can employ an averaging operator; for example, apply a
few iterations of the following convolution
 
1 cw 1
def 1 cw cw2 cw  ,
S ∗ A, S == 2
(3.61)
(2 + cw )
1 cw 1

for cw ≥ 0. Since    
1 cw 1 1
2
cw cw cw  = cw  [1 cw 1],
1 cw 1 1
the convolution smoothing can be implemented easily and conveniently as
  
1
1  
S ∗A = cw  ∗ [1 cw 1] ∗ A  ,
(2 + cw )2
1

which is a horizontal filtering followed by a vertical filtering and finally a

scaling with the factor 1/(2 + cw )2 .
In this project, you will explore the gradient descent method with line search
(3.24) for the computation of the global minimizer of multiple local minima
problems.
68 Chapter 3. Gradient-based Methods for Optimization

Tasks to do:
1. Go to http://www.sfu.ca/∼ssurjano/optimization.html (Virtual Library
of Simulation Experiments) and select three functions of your inter-
ests having multiple local minima. (e.g., two of them are the Ackley
function and Griewank function.)
2. Store each of the functions in a 2D-array A which has dimensions large
enough.
3. Compute

| ∗ S ∗{z· · · ∗ S} ∗A ,
Aσ = gaussian_filter(A , σ ), or At = S
t -times

which can be considered as a convex/smooth approximation of the orig-

inal function. You can use it for the estimation of f and its derivatives
at x n .
4. Design a set of σ/t -values T (including “0” as the last entry) so that
given an initial point x 0 , the Gaussian homotopy continuation method
discussed in Section 3.1.5 can locate the global minimum, while the al-
gorithm (3.24) can find only a local minimum, for each of the functions.

Report. Submit hard copies of your experiences.

• Attach a “summary” or “conclusion” page at the beginning of report.
• Your work process.
• Your code.
• Figures; for each of the functions, include a figure that shows move-
ment of the minimizers for all σ’s or t ’s in T .
• Discuss pros and cons of the Gaussian sailing strategy.

You may work in a group of two people; however, you must report individu-
ally.
Chapter 4

Popular Machine Learning Classifiers

In this chapter, we will take a tour through a selection of popular and pow-
erful machine learning algorithms that are commonly used in academia as
well as in the industry. While learning about the differences between sev-
eral supervised learning algorithms for classification, we will also develop
an intuitive appreciation of their individual strengths and weaknesses.
The topics that we will learn about throughout this chapter are as follows:

• Introduction to the concepts of popular classification algorithms such as

logistic regression, support vector machines (SVM), and decision trees
• Questions to ask when selecting a machine learning algorithm
• Discussions about the strengths and weaknesses of classifiers with lin-
ear and nonlinear decision boundaries

Contents of Chapter 4
4.1. Logistic Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.2. Classification via Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.3. Support Vector Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.4. Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.5. k -Nearest Neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Exercises for Chapter 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

69
70 Chapter 4. Popular Machine Learning Classifiers

Choosing a classification algorithm

Choosing an appropriate classification algorithm for a particular problem
task requires practice:

• Each algorithm has its own quirks/characteristics and is based on cer-

tain assumptions.
• No Free Lunch theorem: no single classifier works best across all
possible scenarios.
• In practice, it is always recommended that you compare the perfor-
mance of at least a handful of different learning algorithms to select
the best model for the particular problem.

Eventually, the performance of a classifier, computational power as well as

predictive power, depends heavily on the underlying data that are available
for learning. The five main steps that are involved in training a machine
learning algorithm can be summarized as follows:

1. Selection of features.
2. Choosing a performance metric.
3. Choosing a classifier and optimization algorithm.
4. Evaluating the performance of the model.
5. Tuning the algorithm.

Required Coding/Reading : Read Chapter 3 and experiment all examples in it, Python
Machine Learning, 3rd Ed.
4.1. Logistic Function 71

4.1. Logistic Function

A logistic function (or logistic curve) is a common “S" shape (sigmoid
curve), with equation:
L
f (x) = , (4.1)
1+ e −k (x −x0 )
where L denotes the curve’s maximum value, x0 is the sigmoid’s midpoint,
and k is the logistic growth rate or steepness of the curve. In statistics, the
logistic model is a widely used statistical model that uses a logistic func-
tion to model a binary dependent variable; many more complex extensions
exist.

4.1.1. The standard logistic sigmoid function

Setting L = 1, k = 1, and x0 = 0 gives the standard logistic sigmoid
function:
1
s(x) = −x
. (4.2)
1+e

Figure 4.1: Standard logistic sigmoid function s(x) = 1/(1 + e −x ).

72 Chapter 4. Popular Machine Learning Classifiers

Remark 4.1. (The standard logistic sigmoid function):

1 ex
s(x) = =
1 + e −x 1 + e x
• The standard logistic function is the solution of the simple first-
order non-linear ordinary differential equation
d 1
y = y(1 − y), y(0) = . (4.3)
dx 2
It can be verified easily as

0 e x (1 + e x ) − e x · e x ex
s (x) = = = s(x)(1 − s(x)). (4.4)
(1 + e x )2 (1 + e x )2

• s 0 is even: s 0 (−x) = s 0 (x).

• Rotational symmetry about (0, 1/2):
1 1 2 + e x + e −x
s(x) + s(−x) = + = ≡ 1. (4.5)
1 + e −x 1 + e x 2 + e x + e −x
ˆ ˆ
ex
• s(x) dx = x
dx = ln(1 + e x ), which is known as the softplus
1+e
function in artificial neural networks. It is a smooth approximation
of the the rectifier (an activation function) defined as

f (x) = x + = max(x, 0). (4.6)

Figure 4.2: The rectifier and softplus function.

4.1. Logistic Function 73

4.1.2. The logit function

To explain the idea behind logistic regression as a probabilistic
model, we may first introduce the odds ratio, which is the odds in favor of
a particular event.

• Let p stand for the probability of the particular event (that has class label
y = 1). Then the odds ratio of the particular event is defined as
p
.
1−p

• We can then define the logit function, which is simply the logarithm of
the odds ratio (log-odds):
p
logit(p) = ln . (4.7)
1−p

• The logit function takes input values in the range 0 to 1 and transforms
them to values over the entire real line, which we can use to express a
linear relationship between feature values and the log-odds:

logit(p(y = 1|x)) = w0 x0 + w1 x1 + · · · + wm xm = w T x, (4.8)

where p(y = 1|x) is the conditional probability that a particular sample

belongs to class 1 given its features x .

Remark 4.2. What we are actually interested in is

predicting the probability

that a certain sample belongs to a particular class, which is the inverse

form of the logit function:

p(y = 1|x) = logit−1 (w T x). (4.9)

74 Chapter 4. Popular Machine Learning Classifiers

Self-study 4.3. Find the inverse of the logit function

p
logit(p) = ln .
1−p

Solution.
1
Ans: logit−1 (z) = , the standard logistic sigmoid function.
1 + e −z

Figure 4.3: The standard logistic sigmoid function, again.

Note: When the standard logistic sigmoid function is adopted as an ac-

tivation function, the prediction may be considered as the probability
that a certain sample belongs to a particular class. Now, can you see
why the logistic sigmoid function is most popular for the activa-
tion function?
4.2. Classification via Logistic Regression 75

4.2. Classification via Logistic Regression

In logistic regression, the activation function simply becomes the sig-
moid function, while Adaline uses the identity function φ(z) = z . The
difference between Adaline and logistic regression is illustrated in the fol-
lowing figure:

Figure 4.4: Adaline vs. Logistic regression.

The output of the sigmoid function is then interpreted as the probability of

a particular sample belonging to class 1,

φ(z) = p(y = 1|x; w),

given its features x parameterized by the weights w . The predicted proba-

bility φ(z) can then simply be converted into a binary outcome via a thresh-
old function: 
1, if φ(z) ≥ 0.5 (or, z ≥ 0)
y= (4.10)
0, otherwise
b
76 Chapter 4. Popular Machine Learning Classifiers

Remark 4.4. (logistic regression). It can be applied not only for

classification (class labels) but also for class-membership probabil-
ity. For example, logistic regression is used in weather forecasting (to
predict the chance of rain). Similarly, it can be used to predict the chance
that a patient has a particular disease given certain symptoms, which is
why logistic regression enjoys great popularity in the field of medicine.

4.2.1. The logistic cost function

The logistic regression incorporates a cost function in which the the likeli-
hood is maximized.1
Definition 4.5. The binomial distribution with parameters n and
p ∈ [0, 1] is the discrete probability distribution of the number of suc-
cesses in a sequence of n independent experiments, each asking a
success–failure question, with probability of success being p . The proba-
bility of getting exactly k successes in n trials is given by the probability
mass function
f (k , n, p) = P(k ; n, p) = n Ck p k (1 − p)n−k . (4.11)

Definition 4.6. (Likelihood). Let X1 , X2 , · · · , Xn have a joint density

function f (X1 , X2 , · · · , Xn |θ). Given X1 = x1 , X2 = x2 , · · · , Xn = xn observed,
the function of θ defined by
L (θ) = L (θ|x1 , x2 , · · · , xn ) = f (x1 , x2 , · · · , xn |θ) (4.12)

is the likelihood function, or simply the likelihood.

Note: The likelihood describes the joint probability of the observed

data as a function of the parameters of the chosen statistical model. That
is, the likelihood is a measurement of the support provided by the
data for each possible value of the parameter θ.

1
Note that the Adaline minimizes the sum-squared-error (SSE) cost function defined as J (w) =
1X
2
φ(z (i) ) − y (i) , where z (i) = w T x (i) , using the gradient descent method; see Section 2.3.1.
2
i
4.2. Classification via Logistic Regression 77

Assume that the individual samples in our dataset are independent of one
another. Then we can define the likelihood L as
L (w) = P(y |x; w) = Πni=1 P(y (i) |x (i) ; w)

y (i)
1−y (i) (4.13)
= Πni=1 (i)
φ(z ) (i)
1 − φ(z ) ,

where z (i) = w T x (i) . In practice, it is easier to maximize the (natural) log of

this equation, which is called he log-likelihood function:
n
X
y (i) ln φ(z (i) ) + (1 − y (i) ) ln 1 − φ(z (i) )



`(w) = ln(L (w)) = . (4.14)
i=1

Note:
• Firstly, applying the log function reduces the potential for numerical
underflow, which can occur if the likelihoods are very small.
• Secondly, we can convert the product of factors into a summation of
factors, which makes it easier to obtain the derivative of this func-
tion via the addition trick, as you may remember from calculus.
• We can adopt the negation of the log-likelihood as a cost function J
that can be minimized using gradient descent.

Now, we define the logistic cost function to be minimized:

X n
 (i)
−y ln φ(z (i) ) − (1 − y (i) ) ln 1 − φ(z (i) ) ,



J (w) = (4.15)
i=1

where z (i) = w T x (i) .

Note: Looking at the equation, we can see that the first term becomes
zero if y (i) = 0, and the second term becomes zero if y (i) = 1.
78 Chapter 4. Popular Machine Learning Classifiers

Figure 4.5: Plot of J (w), when n = 1 (one single-sample):


− ln(φ(z)), if y = 1,
J (w) =
− ln(1 − φ(z)), if y = 0.

Observation 4.7. We can see that

• (Solid curve, in blue). If we correctly predict that a sample be-
longs to class 1, the cost approaches 0.
• (Dashed curve, in orange). If we correctly predict y = 0, the cost
also approaches 0.
• However, if the prediction is wrong, the cost goes towards infinity.
• Here, the main point is that we penalize wrong predictions with
an increasingly larger cost, which will enforce the model to fit the
sample.
• For general n ≥ 1, it would try to fit all the samples in the training
dataset.
4.2. Classification via Logistic Regression 79

4.2.2. Gradient descent learning for logistic regression

Let’s start by calculating the partial derivative of the logistic cost function
with respect to the j –th weight, wj :
n 
∂φ(z (i) )

∂J (w) X (i) 1 (i) 1
= −y + (1 − y ) , (4.16)
∂ wj φ(z (i) ) 1 − φ(z (i) ) ∂ wj
i=1

where, using z (i) = w T x (i) and (4.4),

∂φ(z (i) ) 0 (i) ∂ z
(i)
(i)
= φ (z ) = φ(z (i) ) 1 − φ(z (i) ) xj .
∂ wj ∂ wj
Thus, if follows from the above and (4.16) that
n
∂J (w) X
−y 1 − φ(z (i) ) + (1 − y (i) )φ(z (i) ) xj(i)
 (i)

=
∂ wj
i=1
Xn
 (i)  (i)
= − y − φ(z (i) ) xj
i=1

and therefore
n
X
y (i) − φ(z (i) ) x (i) .
 
∇J (w) = − (4.17)
i=1

Algorithm 4.8. Gradient descent learning for logistic regression is for-

mulated as
w := w + ∆w, (4.18)
where η > 0 is the step length (learning rate) and
n
X
∆w = −η∇J (w) = η y (i) − φ(z (i) ) x (i) .
 
(4.19)
i=1

Note: The above gradient descent rule for logistic regression is equal to
that of Adaline; see (2.15) on p. 28. Only the difference is the activation
function φ(z).
80 Chapter 4. Popular Machine Learning Classifiers

4.2.3. Regularization: bias-variance tradeoff

• Overfitting is a common problem in ML, where a model performs

well on training data but does not generalize well to unseen data (test
data).
– Due to a high variance, from randomness in the training data.
– Variance measures the consistency (or variability) of the model
prediction for a particular sample instance.
• Similarly, our model can also suffer from underfitting, which means
that our model is not complex enough to capture the pattern in the
training data well and therefore also suffers from low performance on
unseen data.
– Due to a high bias.
– Bias is the measure of the systematic error that is not due to ran-
domness.

Figure 4.6
4.2. Classification via Logistic Regression 81

Regularization
• One way of finding a good bias-variance tradeoff.
• It is useful to prevent overfitting, also handling
– collinearity (high correlation among features)
– filter-out noise from data
– multiple minima problem
• The concept behind regularization is to introduce additional in-
formation (bias) to penalize extreme parameter (weight) values.
• The most common form of regularization is so-called L 2 regularization
(sometimes also called L 2 shrinkage or weight decay):
m
λ 2 λX
kw k = wj2 , (4.20)
2 2
j=1

where λ is the regularization parameter.

The cost function for logistic regression can be regularized by adding a sim-
ple regularization term, which will shrink the weights during model train-
ing: for z (i) = w T x (i) ,

n
X  (i)
 λ
−y ln φ(z (i) ) − (1 − y (i) ) ln 1 − φ(z (i) ) + kw k2 .


J (w) = (4.21)
2
i=1

Note:
• Regularization is another reason why feature scaling such as stan-
dardization is important. For regularization to work properly, we
need to ensure that all our features are on comparable scales. See
§ 6.3 for more details of feature scaling.
• Via the regularization parameter λ, we can then control how well we
fit the training data while keeping the weights small. By increasing
the value of λ, we increase the regularization strength.
82 Chapter 4. Popular Machine Learning Classifiers

4.3. Support Vector Machine

• Support vector machine (SVM), developed in 1995 by Cortes-
Vapnik [12], can be considered as an extension of the Percep-
tron/Adaline, which maximizes the margin.
• The rationale behind having decision boundaries with large margins
is that they tend to have a lower generalization error, whereas
models with small margins are more prone to overfitting.

4.3.1. Linear SVM

Figure 4.7: Linear support vector machine.

To find an optimal hyperplane that maximizes the margin, let’s begin with
considering the positive and negative hyperplanes that are parallel to the
decision boundary:
w0 + w T x + = 1,
(4.22)
w0 + w T x − = −1.
where w = [w1 , w2 , · · · , wd ]T . If we subtract those two linear equations from
each other, then we have
w · (x + − x − ) = 2
and therefore
w 2
· (x + − x − ) = . (4.23)
kw k kw k
4.3. Support Vector Machine 83

Note: w = [w1 , w2 , · · · , wd ]T is a normal vector to the decision boundary

(a hyperplane) so that the left side of (4.23) is the distance between the
positive and negative hyperplanes.

Maximizing the distance (margin) is equivalent to minimizing its reciprocal

1
2
kw k, or minimizing 12 kw k2 .

Problem 4.9. The linear SVM is formulated as

1
min kw k2 , subject to
w,w0 2
(4.24)
w0 + w T x (i) ≥ 1 if y (i) = 1,


w0 + w T x (i) ≤ −1 if y (i) = −1.

Remark 4.10. The constraints in Problem 4.9 can be written as

y (i) (w0 + w T x (i) ) − 1 ≥ 0, ∀ i. (4.25)

• The beauty of linear SVM is that if the data is linearly separable,

there is a unique global minimum value.
• An ideal SVM analysis should produce a hyperplane that completely
separates the vectors (cases) into two non-overlapping classes.
• However, perfect separation may not be possible, or it may result
in a model with so many cases that the model does not classify cor-
rectly.

Note: Constrained optimization problems such as (4.24) are typically

solved using the method of Lagrange multipliers.
84 Chapter 4. Popular Machine Learning Classifiers

4.3.2. The method of Lagrange multipliers

In this subsection, we briefly consider Lagrange’s method to solve the
problem of the form
min / max f (x) subj.to g(x) = c. (4.26)
x

Figure 4.8: The method of Lagrange multipliers in R2 : ∇f // ∇g , at optimum.

Strategy 4.11. (Method of Lagrange multipliers). For the maxi-

mum and minimum values of f(x) subject to g(x) = c,
(a) Find x and λ such that

∇f (x) = λ∇g(x) and g(x) = c.

(b) Evaluate f at all these points, to find the maximum and minimum.

Self-study 4.12. Use the method of Lagrange multipliers to find the ex-
treme values of f (x, y) = x 2 + 2y 2 on the circle x 2
+ y 2 = 1.
     2x = 2x λ 1
2x 2x
Hint : ∇f = λ∇g =⇒ =λ . Therefore, 4y = 2y λ 2
4y 2y  2 2
x +y =1 3
From 1 , x = 0 or λ = 1.

Ans: min: f (±1, 0) = 1; max: f (0, ±1) = 2

4.3. Support Vector Machine 85

Lagrange multipliers – Dual variables

For simplicity, consider

minx x 2 subj.to x ≥ 1. (4.27)

Rewriting the constraint

x − 1 ≥ 0,

introduce Lagrangian (objective):

L(x, α) = x 2 − α (x − 1). (4.28)

Now, consider
minx maxα L(x, α) subj.to α ≥ 0.
Figure 4.9: minx x 2 subj.to x ≥ 1.
(4.29)

Claim 4.13. The minimization problem (4.27) is equivalent to the max-

min problem (4.29).

Proof. 1 Let x > 1. ⇒ maxα≥0 {−α(x − 1)} = 0 and α∗ = 0. Thus,

L(x, α) = x 2 . (original objective)

2 Let x = 1. ⇒ maxα≥0 {−α(x − 1)} = 0 and α is arbitrary. Thus, again,

L(x, α) = x 2 . (original objective)

3 Let x < 1. ⇒ maxα≥0 {−α(x − 1)} = ∞. However, minx won’t make this
happen! (minx is fighting maxα ) That is, when x < 1, the objective L(x, α)
becomes huge as α grows; then, minx will push x % 1 or increase it to become
x ≥ 1. In other words, minx forces maxα to behave, so constraints will
be satisfied.

Now, the goal is to solve (4.29). In the following, we will define the dual
problem of (4.29), which is equivalent to the primal problem.
86 Chapter 4. Popular Machine Learning Classifiers

Recall: The min-max problem in (4.29), which is equivalent to the (orig-

inal) primal problem:

min max L(x, α) subj.to α ≥ 0, (Primal) (4.30)

x α

where
L(x, α) = x 2 − α (x − 1).

Definition 4.14. The dual problem of (4.30) is formulated by swap-

ping minx and maxα as follows:

max min L(x, α) subj.to α ≥ 0, (Dual) (4.31)

α x

The term minx L(x, α) is called the Lagrange dual function and the
Lagrange multiplier α is also called the dual variable.

How to solve it. For the Lagrange dual function minx L(x, α), the minimum
occurs where the gradient is equal to zero.
d α
L(x, α) = 2x − α = 0 ⇒ x = .
dx 2
Plugging this to L(x, α), we have
 α 2 α  α2
L(x, α) = −α −1 =α− .
2 2 4

We can rewrite the dual problem (4.31) as

h α2 i
max α − . (Dual) (4.32)
α≥0 4

⇒ the maximum is 1 when α∗ = 2 (for the dual problem).

Plugging α = α∗ into the Lagrangian objective, we have

L(x, α) = x 2 − 2(x − 1) = (x − 1)2 + 1. (4.33)

⇒ the minimum is 1 when x ∗ = 1 (for the primal problem).

4.3. Support Vector Machine 87

4.3.3. Solution of the linear SVM

Recall: The linear SVM formulated in Problem 4.9:
1
min kw k2 , subj.to
w,w0 2 (Primal) (4.34)
y (i) (w0 + w T x (i) ) − 1 ≥ 0, ∀ i.

To solve the problem, let’s begin with its Lagrangian:

N
1 2
X
L([w, w0 ], α) = kw k − αi [y (i) (w0 + w T x (i) ) − 1], (4.35)
2
i=1

where α = [α1 , α2 , · · · , αN ]T , the dual variable (Lagrange multipliers).

Definition 4.15. In optimization, the Karush-Kuhn-Tucker (KKT)

conditions [30, 35] are first derivative tests (a.k.a. first-order nec-
essary conditions) for a solution in nonlinear programming to be opti-
mized, provided that some regularity conditions are satisfied.

Note: Allowing inequality constraints, the KKT approach to nonlinear

programming generalizes the method of Lagrange multipliers, which al-
lows only equality constraints.
88 Chapter 4. Popular Machine Learning Classifiers

Writing the KKT conditions, starting with Lagrangian stationarity, where

we need to find the first-order derivatives w.r.t. w and w0 :

N
X N
X
(i) (i)
∇w L([w, w0 ], α) = w− αi y x =0 ⇒ w= αi y (i) x (i) ,
i=1 i=1
N N
∂ X X
L([w, w0 ], α) = − αi y (i) = 0 ⇒ αi y (i) = 0,
∂ w0 (4.36)
i=1 i=1

αi ≥ 0, (dual feasibility)
αi [y (i) (w0 + w T x (i) ) − 1] = 0, (complementary slackness)
y (i) (w0 + w T x (i) ) − 1 ≥ 0. (primal feasibility)

Using the KKT conditions (4.36), we can simplify the Lagrangian:

N N N
1 2
X
(i)
X
(i) T (i)
X
L([w, w0 ], α) = kw k − α i y w0 − αi y w x + αi
2
i=1 i=1 i=1
N
1 X
= kw k2 − 0 − w T w + αi (4.37)
2
i=1
N
1 X
= − kw k2 + αi .
2
i=1

Again using the first KKT condition, we can rewrite the first term.
N N
1 2 1 X (i) (i)
 X
(j) (j)

− kw k = − αi y x · αj y x
2 2
i=1 j=1
N N (4.38)
1 XX
= − αi αj y (i) y (j) x (i) · x (j) .
2
i=1 j=1

Plugging (4.38) into the (simplified) Lagrangian (4.37), we see that the La-
grangian now depends on α only.
4.3. Support Vector Machine 89

Problem 4.16. The dual problem of (4.34) is formulated as

N N N
hX 1 XX (i) (j) (i) (j)
i
max αi − αi αj y y x · x , subj.to
α 2
 i=1 i=1 j=1 (4.39)
α ≥ 0, ∀ i,
Pi N (i)
i=1 αi y = 0.

Remark 4.17. (Solving the dual problem).

• We can solve the dual problem (4.39), by using either a generic
quadratic programming solver or the Sequential Minimal Opti-
mization (SMO), which we will discuss in § 4.3.6, p. 100.
• For now, assume that we solved it to have α∗ = [α1∗ , · · · , αn∗ ]T .
• Then we can plug it into the first KKT condition to get
N
X
∗
w = αi∗ y (i) x (i) . (4.40)
i=1

• We still need to get w0∗ .

Remark 4.18. The objective function L(α) in (4.39) is a linear com-

bination of the dot products of data samples {x (i) · x (j) }, which will be
used when we generalize the SVM for nonlinear decision boundaries;
see § 4.3.5.
90 Chapter 4. Popular Machine Learning Classifiers

Support vectors
Assume momentarily that we have w0∗ . Consider the complementary slack-
ness KKT condition along with the primal and dual feasibility conditions:

αi∗ [y (i) (w0∗ + w ∗T x (i) ) − 1] = 0



 αi∗ > 0 ⇒ y (i) (w0∗ + w ∗T x (i) ) = 1
 α∗ < 0 (can’t happen)

(4.41)
i
⇒

 y (w0∗ + w T x (i) ) − 1 > 0 ⇒ αi∗ = 0
(i)
 (i) ∗

y (w0 + w T x (i) ) − 1 < 0 (can’t happen).
We define the optimal (scaled) scoring function:

f ∗ (x (i) ) = w0∗ + w ∗T x (i) . (4.42)

Then
αi∗ > 0 ⇒ y (i) f ∗ (x (i) ) = scaled margin = 1,

(4.43)
y (i) f ∗ (x (i) ) > 1 ⇒ αi∗ = 0.

Definition 4.19. The examples in the first category, for which the
scaled margin is 1 and the constraints are active, are called support
vectors. They are the closest to the decision boundary.

Finding the optimal value of w0

To get w0∗ , use the primal feasibility condition:

y (i) (w0∗ + w ∗T x (i) ) ≥ 1 and min y (i) (w0∗ + w ∗T x (i) ) = 1.

i

If you take a positive support vector (y (i) = 1), then

w0∗ = 1 − min w ∗T x (i) . (4.44)

i:y (i) =1

Here, you’d better refer to Summary of SVM in Algorithm 4.22, p. 95.

4.3. Support Vector Machine 91

4.3.4. The inseparable case: soft-margin classification

When the dataset is inseparable, there would be no separating hyperplane;
there is no feasible solution to the linear SVM.

Figure 4.10: Slack variable: ξi .

Let’s fix our SVM so it can accommodate the inseparable case.

• The new formulation involves the slack variable; it allows some
instances to fall off the margin, but penalize them.
• So we are allowed to make mistakes now, but we pay a price.

Note: The motivation for introducing the slack variable ξ was that the
linear constraints need to be relaxed for nonlinearly separable data to
allow the convergence of the optimization in the presence of misclassi-
fications, under appropriate cost penalization. Such strategy of SVM is
called the soft-margin classification.
92 Chapter 4. Popular Machine Learning Classifiers

Recall: The linear SVM formulated in Problem 4.9:

1
min kw k2 , subj.to
w,w0 2 (Primal) (4.45)
y (i) (w0 + w T x (i) ) − 1 ≥ 0, ∀ i.

Let’s change it to this new primal problem:

Problem 4.20. (Soft-margin classification). The SVM with the slack

variable is formulated as
N
1 2
X
min kw k + C ξi subj.to
w,w0 ,ξ 2
 (i) i=1 (Primal) (4.46)
T (i)
y (w0 + w x ) ≥ 1 − ξi ,
ξi ≥ 0.

Via the variable C , we can then con- variance trade-off, as illustrated

trol the penalty for misclassification. in the following figure:
Large values of C correspond to
large error penalties, whereas we
are less strict about misclassification
errors if we choose smaller values
for C . We can then use the C pa-
rameter to control the width of the
margin and therefore tune the bias-
Figure 4.11: Bias-variance trade-off, via C .

The constraints allow some slack of size ξi , but we pay a price for it in the
objective. That is,
if y (i) f (x (i) ) ≥ 1, then ξi = 0 and penalty is 0. Otherwise, y (i) f (x (i) ) = 1 − ξi
and we pay price ξi > 0

.
4.3. Support Vector Machine 93

Going on a bit for soft-margin classification

We rewrite the penalty another way:
if y (i) f (x (i) ) ≥ 1
 
0,
ξi = (i) (i) = [1 − y (i) f (x (i) )]+
1 − y f (x ), otherwise

Thus the objective function for soft-margin classification becomes

N
1 X
min kw k2 + C [1 − y (i) f (x (i) )]+ . (4.47)
w,w0 ,ξ 2
i=1

The Dual for soft-margin classification

Form the Lagrangian of (4.46):
N N
1 X X
L([w, w0 ], ξ , α, r) = kw k2 + C ξi − r i ξi
2
i=1 i=1
N (4.48)
X
− αi [y (i) (w0 + w T x (i) ) − 1 + ξi ],
i=1

where αi ’s and ri ’s are Lagrange multipliers (constrained to be ≥ 0). After

some work, the dual turns out to be
Problem 4.21. The dual problem of (4.45) is formulated as
N N N
hX 1 XX (i) (j) (i) (j)
i
max αi − αi αj y y x · x , subj.to
α 2
 i=1 i=1 j=1 (4.49)
0 ≤ α ≤ C, ∀ i,
PN i (i)
i=1 αi y = 0.

So the only difference from the original problem’s Lagrangian (4.39) is that
0 ≤ αi was changed to 0 ≤ αi ≤ C . Neat!

See § 4.3.6, p. 100, for the solution of (4.49), using the SMO algorithm.
94 Chapter 4. Popular Machine Learning Classifiers

Algebraic expression for the dual problem :

Let    
y (1) x (1) 1
 y (2) x (2)  1
Z =  ∈ R N ×m , 1 =  ..  ∈ RN .
   
..
 .  .
y (N) x (N) 1

Then dual problem (4.49) can be written as

1
max α · 1 − αT ZZ T α subj.to α · y = 0.
 
(4.50)
0≤α≤C 2

Note:
• G = ZZ T ∈ RN×N is called the Gram matrix. That is,

Gij = y (i) y (j) x (i) · x (j) . (4.51)

• The optimization problem in (4.49) or (4.50) is a typical form of

quadratic programming (QP) problems.
• The dual problem (4.50) admits a unique solution; see Exercise 5.
4.3. Support Vector Machine 95

Algorithm 4.22. (Summary of SVM)

• Training
– Compute Gram matrix: Gij = y (i) y (j) x (i) · x (j)
– Solve QP to get α∗ (Chapter 5, or § 4.3.6)
– Compute the weights: w ∗ = Ni=1 αi∗ y (i) x (i) (4.40)
P

– Compute the intercept: w0∗ = 1 − mini:y (i) =1 w ∗T x (i) (4.44)

• Classification (for a new sample x )

– Compute ki = x · x (i) for support vectors x (i)
– Compute f (x) = w0∗ + i αi y (i) ki (:= w0∗ + w ∗T x)
P
(4.24)
– Test sign(f (x)).

Why are SVMs popular in ML?

• Margins
– to reduce overfitting
– to enhance classification accuracy
• Feature expansion
– mapping to a higher-dimension
– to classify inseparable datasets
• Kernel trick
– to avoid writing out high-dimensional feature vectors
96 Chapter 4. Popular Machine Learning Classifiers

4.3.5. Nonlinear SVM and kernel trick

Note: A reason why SVMs enjoy high popularity among machine learn-
ing practitioners is that it can be easily kernelized to solve nonlinear
classification problems incorporating linearly inseparable data. The
basic idea behind kernel methods is to create nonlinear combinations
of the original features to project them onto a higher-dimensional space
via a mapping φ where it becomes linearly separable.

For example, for the inseparable data set in Figure 4.12, we define

φ(x1 , x2 ) = (z1 , z2 , z2 ) = (x1 , x2 , x12 + x22 ).

Figure 4.12: Inseparable dataset, feature expansion, and kernel SVM.

To solve a nonlinear problem using an SVM, we would a transform the

training data onto a higher-dimensional feature space via a mapping φ and
b train a linear SVM model to classify the data in this new feature space.
Then, we can c use the same mapping function φ to transform new, unseen
data to classify it using the linear SVM model.
4.3. Support Vector Machine 97

Kernel trick
Recall: the dual problem to the soft-margin SVM given in (4.49):
N N N
hX 1 XX (i) (j) (i) (j)
i
max αi − αi αj y y x · x , subj.to
α 2
 i=1 i=1 j=1 (4.52)
0 ≤ α ≤ C, ∀ i,
PN i (i)
i=1 αi y = 0.

Observation 4.23. The objective is a linear combination of dot prod-

ucts {x (i) · x (j) }. Thus,
• If the kernel SVM transforms the data samples through φ, the dot
product x (i) · x (j) in the objective must be replaced by φ(x (i) ) · φ(x (j) ).
• The dot product φ(x (i) ) · φ(x (j) ) is performed in a higher-dimension,
which may be costly.

Definition 4.24. In order to save the expensive step of explicit

computation of this dot product (in a higher-dimension), we define a
so-called kernel function:

K(x (i) , x (j) ) ≈ φ(x (i) ) · φ(x (j) ). (4.53)

One of the most widely used kernels is the Radial Basis Function
(RBF) kernel or simply called the Gaussian kernel:

(i) (j) kx (i) − x (j) k2 


(i) (j) 2


K(x , x ) = exp − = exp −γkx − x k , (4.54)
2σ 2
where γ = 1/(2σ 2 ). Occasionally, the parameter γ plays an important role
in controlling overfitting.

Note: Roughly speaking, the term kernel can be interpreted as a

similarity function between a pair of samples. The minus sign inverts
the distance measure into a similarity score, and, due to the exponential
term, the resulting similarity score will fall into a range between 1 (for
exactly similar samples) and 0 (for very dissimilar samples).

This is the big picture behind the kernel trick.

98 Chapter 4. Popular Machine Learning Classifiers

(Kernel SVM algorithm): It can be summarized as in Algorithm 4.22,

p. 95; only the difference is that dot products x (i) ·x (j) and x ·x (i) are replaced
by K(x (i) , x (j) ) and K(x, x (i) ), respectively.

Common kernels

• Polynomial of degree exactly k (e.g. k = 2):

K(x (i) , x (j) ) = (x (i) · x (j) )k (4.55)

• Polynomial of degree up to k : for some c > 0,

K(x (i) , x (j) ) = (c + x (i) · x (j) )k (4.56)

• Sigmoid:
K(x (i) , x (j) ) = tanh(a x (i) · x (j) + b) (4.57)
• Gaussian RBF:
(i) (j) kx (i) − x (j) k2 

K(x , x ) = exp − (4.58)
2σ 2
• And many others: Fisher kernel, graph kernel, string kernel, ...
very active area of research!

Example 4.25. (Quadratic kernels). K(x, z) = (c + x · z)2

 m
X  m
X 
2
(c + x · z) = c + xj zj c + x` z`
j=1 `=1
m
X m
XX m
= c 2 + 2c xj zj + xj zj x` z` (4.59)
j=1 j=1 `=1
m m
2
X √ √ X
= c + ( 2cxj )( 2czj ) + (xj x` )(zj z` ).
j=1 j,`=1

So, the corresponding feature expansion is given by

√ √
φ([x1 , · · · , xm ]) = [x12 , x1 x2 , · · · , xm xm−1 , xm2 , 2cx1 , · · · , 2cxm , c], (4.60)
2
which is in Rm +m+1
. Q: How is this feature expansion meaningful?
Although not expressible by φ(x) · φ(z) [ever, or in finite dimensions], the
kernel K(x, z) must be formulated meaningfully!
4.3. Support Vector Machine 99

Summary: Linear classifiers & their variants

• Perceptrons are a simple, popular way to learn a classifier
• They suffer from inefficient use of data, overfitting, and lack of ex-
pressiveness

• SVMs can fix these problems using 1 margins and

2 feature expansion (mapping to a higher-dimension)
• In order to make feature expansion computationally feasible,
we need the 3 kernel trick, which avoids writing out high-
dimensional feature vectors
• SVMs are popular classifiers because they usually achieve good error
rates and can handle unusual types of data
100 Chapter 4. Popular Machine Learning Classifiers

4.3.6. Solving the dual problem with SMO

SMO (Sequential Minimal Optimization) is a type of coordinate ascent
algorithm, but adapted to SVM so that the solution always stays within the
feasible region.

Recall: The dual problem of the slack variable primal, formulated in

(4.49):
N N N
hX 1 XX (i) (j) (i) (j)
i
max αi − αi αj y y x · x , subj.to
α 2
 i=1 i=1 j=1 (4.61)
0 ≤ αi ≤ C, ∀ i,
PN (i)
i=1 αi y = 0.

Quesiton. Start with (4.61). Let’s say you want to hold α2 , · · · , αN fixed
and take a coordinate step in the first direction. That is, change α1 to
maximize the objective in (4.61). Can we make any progress? Can we
get a better feasible solution by doing this?
PN
Turns out, no. Let’s see why. Look at the constraint in (4.61), i=1 αi y (i) = 0.
This means
XN N
X
(1) (i) (1)
α1 y = − αi y ⇒ α1 = − y αi y (i) .
i=2 i=2

So, since α2 , · · · , αN are fixed, α1 is also fixed.

Thus, if we want to update any of the αi ’s, we need to update at least
2 of them simultaneously to keep the solution feasible (i.e., to keep the
constraints satisfied).
4.3. Support Vector Machine 101

• Start with a feasible vector α.

• Let’s update α1 and α2 , holding α3 , · · · , αN fixed. What values of α1 and
α2 are we allowed to choose?
X N
(1) (2)
• The constraint is: α1 y + α2 y = − αi y (i) =: ξ .
i=3

Figure 4.13

• We are only allowed to choose α1 and α2 on the line; when we pick α2 ,

we get α1 automatically from
1
α1 = (1)
(ξ − α2 y (2) ) = y (1) (ξ − α2 y (2) ). (4.62)
y

• (Optimization for α2 ): The other constraints in (4.61) says 0 ≤

α1 , α2 ≤ C . Thus, α2 needs to be within [L , H] on the figure (∵ α1 ∈
[0, C]). To do the coordinate ascent step, we will optimize the objective
over α2 , keeping it within [L , H]. Using (4.62), (4.61) becomes
N N N
h
(1) (2)
X 1 XX (i) (j) (i) (j)
i
max y (ξ − α2 y ) + α2 + αi − αi αj y y x · x , (4.63)
α2 ∈[L ,H] 2
i=3 i=1 j=1

of which the objective is quadratic in α2 . This means we can just set

its derivative to 0 to optimize it, and to get α2 .
• Move to the next iteration of SMO.

Note: There are heuristics to choose the order of αi ’s chosen to update.

102 Chapter 4. Popular Machine Learning Classifiers

4.4. Decision Trees

Decision tree classifiers are attractive models if we care about inter-
pretability. As the name decision tree suggests, we can think of this model
as breaking down our data by making decision based on asking a series of
questions. Decision tree was invented by a British researcher, William Bel-
son, in 1959 [1].

Note: Decision trees are commonly used in operations research,

specifically in decision analysis, to help identify a strategy most likely
to reach a goal, but are also a popular tool in ML.

Figure 4.14: A decision tree to decide upon an activity on a particular day.

Key Idea 4.26. (Decision tree).

• Start at the tree root
• Split the data so as to result in the largest Information Gain (IG)
• Repeat the splitting at each child node until the leaves are pure
(This means the samples at each node all belong to the same class)

• In practice , this can result in a very deep tree with many nodes,
which can easily lead to overfitting
(We typically set a limit for the maximal depth of the tree)
4.4. Decision Trees 103

4.4.1. Decision tree objective

Decision tree also needs to incorporate an objective function, to be optimized
via the tree learning algorithm. Here, the objective function is to maximize
the information gain at each split, which we define as follows:
m
X Nj
IG(DP , f ) = I(DP ) − I(Dj ), (4.64)
NP
j=1

where
f : the feature to perform the split
DP : the parent dataset
Dj : the dataset of the j -th child node
I : the impurity measure
NP : the total number of samples at the parent note
Nj : the number of samples in the j -th child node

The information gain is simply the difference between the impurity of the
parent node and the sum of the child node impurities — the lower the
impurity of the child nodes, the larger the information gain.
However, for simplicity and to reduce the combinatorial search space, most
libraries implement binary decision trees, where each parent node is
split into two child nodes, DL and DR :
NL NR
IG(DP , f ) = I(DP ) − I(DL ) − I(DR ). (4.65)
NP NP
104 Chapter 4. Popular Machine Learning Classifiers

Impurity measure?
Commonly used in binary decision trees:
• Entropy c
X
IH (t) = − p(i |t) log2 p(i |t) (4.66)
i=1

• Gini impurity
c
X c
X
p(i |t)2


IG (t) = p(i |t) 1 − p(i |t) = 1 − (4.67)
i=1 i=1

• Classification error
IE (t) = 1 − max{p(i |t)} (4.68)
i

where p(i |t) denotes the proportion of the samples that belong to class
i for a particular node t .

Mind simulation
• The entropy is maximal, if we have a uniform class distribution; it is
0, if all samples at the node t belong to the same class.
e.g., when c = 2:
IH (t) = 0, if p(i = 1|t) = 1 or p(i = 0|t) = 0
IH (t) = 1, if p(i = 1|t) = p(i = 0|t) = 0.5
⇒ We can say that the entropy criterion attempts to maximize the mutual
information in the tree.
• Intuitively, the Gini impurity can be understood as a criterion to min-
imize the probability of misclassification.
• The Gini impurity is maximal, if the classes are perfectly mixed.
e.g., when c = 2:
Pc
IG (t) = 1 − i=1 0.52 = 0.5
⇒ In practice, both Gini impurity and entropy yield very similar results.
• The classification error is less sensitive to changes in the class prob-
abilities of the nodes.
⇒ The classification error is a useful criterion for pruning, but not recom-
mended for growing a decision tree.
4.4. Decision Trees 105

Figure 4.15: A decision tree result with Gini impurity measure, for three classes with two
features (petal length, petal width). Page 97, Python Machine Learning, 3rd Ed..

Quesiton. How can the decision tree find questions such as

petal width <= 0.75 petal length <= 4.75 ... ?
106 Chapter 4. Popular Machine Learning Classifiers

Algorithm 4.27. (Decision tree split rule).

1. For each and every feature in DP , fj(i) :
◦ make a question to split Dp into DL and DR
(e.g. fj(k ) ≤ fj(i) , for which k ’s?)
◦ compute the impurities: I(DL ) and I(DR )
◦ compute the information gain, using (4.65):
(i) NL NR
IG(DP , fj ) = I(DP ) − I(DL ) − I(DR ).
NP NP

2. Let
(i)
fq(p) = arg max IG(DP , fj ). (4.69)
i,j

3. Then, the best split question (at the current node) is

fq(k ) ≤ fq(p) , for which k ’s ? (4.70)

The maximum in (4.69) often happens when one of the child impurities
is zero or very small.
4.4. Decision Trees 107

4.4.2. Random forests

Random forests (or random decision forests) are an ensemble learn-
ing method for classification, regression, and other tasks that operates
by constructing a multitude of decision trees at training time and out-
putting the class that is the mode of the classes (classification) or mean
prediction (regression) of the individual trees [26].

• Random forests have gained huge popularity in applications of ML

during the last decade due to their good classification performance,
scalability, and ease of use.
• The idea behind a random forest is to average multiple (deep) de-
cision trees that individually suffer from high variance, to build a
more robust model that has a better generalization performance and
is less susceptible to overfitting.

The random forest algorithm can be summarized in four simple steps:

1. Draw a random bootstrap sample of size n

(Randomly choose n samples from the training set with replacement).
2. Grow a decision tree from the bootstrap sample.
3. Repeat Steps 1-2 k times.
4. Aggregate the prediction by each tree to assign the class label by ma-
jority vote.

Note: In Step 2, when we are training the individual decision trees:

instead of evaluating all features to determine the best split at each
node, we can consider a random (without replacement) subset of those
(of size d ).
108 Chapter 4. Popular Machine Learning Classifiers

Remark 4.28. A big advantage of random forests is that we don’t

have to worry so much about choosing good hyperparameter val-
ues.

• A smaller n increases randomness of the random forest; the bigger n is,

the larger the degree of overfitting becomes.
Default n = size(the original training set), in most implementations
√
• Default d = M , where M is the number of features in the training set
• The only parameter that we really need to care about in practice is

the number of trees k (Step 3).

Typically, the larger the number of trees, the better the performance
of the random forest classifier at the expense of an increased computa-
tional cost.
4.5. k -Nearest Neighbors 109

4.5. k -Nearest Neighbors

The k -nearest neighbor (k -NN) classifier is a typical example of a lazy
learner. It is called lazy not because of its apparent simplicity, but because
it doesn’t learn a discriminative function from the training data, but memo-
rizes the training dataset instead; analysis of the training data is delayed
until a query is made to the system.

Algorithm 4.29. (k -NN algorithm). The algorithm itself is fairly

straightforward and can be summarized by the following steps:
1. Choose the number of k and a distance metric.
2. Find the k -nearest neighbors of the sample that we want to classify.
3. Assign the class label by majority vote.

Figure 4.16: Illustration for how a new data point (?) is assigned the triangle class label,
based on majority voting, when k = 5.

Based on the chosen distance metric, the k -NN algorithm finds the k sam-
ples in the training dataset that are closest (most similar) to the point that
we want to classify. The class label of the new data point is then determined
by a majority vote among its k nearest neighbors.
110 Chapter 4. Popular Machine Learning Classifiers

k -NN: pros and cons

• Since it is memory-based, the classifier immediately adapts as we
collect new training data.
• The computational complexity for classifying new samples grows
linearly with the number of samples in the training dataset in the
worst-case scenario.a
• Furthermore, we can’t discard training samples since no training
step is involved. Thus, storage space can become a challenge if we
are working with large datasets.
a
J. H. Friedman, J. L. Bentley, and R.A. Finkel (1977). An Algorithm for Finding Best Matches in
Logarithmic Expected Time, ACM transactions on Mathematical Software (TOMS), 3, no. 3, pp. 209–
226. The algorithm in the article is called the KD-tree.

k -NN: what to choose k and a distance metric?

• The right choice of k is crucial to find a good balance between over-
fitting and underfitting.
(For sklearn.neighbors.KNeighborsClassifier, default n_neighbors = 5.)
• We also choose a distance metric that is appropriate for the features
in the dataset. (e.g., the simple Euclidean distance, along with data
standardization)
• Alternatively, we can choose the Minkowski distance:
m
X 1/p
def p
d(x, z) = kx − z kp == |xi − zi | . (4.71)
i=1

(For sklearn.neighbors.KNeighborsClassifier, default p = 2.)

Remark 4.30. The k -NN algorithm is very susceptible (wide open) to

overfitting due to the curse of dimensionality.a
Since regularization is not applicable for k -NN, we can use feature se-
lection and dimensionality reduction techniques to help us avoid the
curse of dimensionality and avoid overfitting. This will be discussed in
more details later.
a
The curse of dimensionality describes the phenomenon where the feature space becomes increas-
ingly sparse for an increasing number of dimensions of a fixed-size training dataset. Intuitively, we
can think of even the closest neighbors being too far away in a high-dimensional space to give a good
estimate.
4.5. k -Nearest Neighbors 111

Exercises for Chapter 4

4.1. For this problem, you would modify the code used for Problem 2.2 in Chapter 2. For
the standardized data (XSD ),

(a) Apply the logistic regression gradient descent (Algorithm 4.8).

(b) Compare the results with that of Adaline descent gradient.
4.2. (Continuation of Problem 4.1). Perturb the standardized data (XSD ) by a random
Gaussian noise Gσ of an observable σ (so as for Gσ (XSD ) not to be linearly separable).

(a) Apply the logistic regression gradient descent (Algorithm 4.8) for the noisy data
Gσ (XSD ).
(b) Modify the code for the logistic regression with regularization (4.21) and apply
the resulting algorithm for Gσ (XSD ).
(c) Compare their performances
4.3. Verify the formulation in (4.49), which is dual to the minimization of (4.48).
4.4. Experiment examples on pp. 84–91, Python Machine Learning, 3rd Ed., in order to
optimize the performance of kernel SVM by finding a best kernel and optimal hyper-
parameters (gamma and C ).
Choose one of Exercises 5 and 6 below to implement and experiment. The experiment
will guide you to understand how the LM software has been composed from scratch.
You may use the example codes thankfully shared by Dr. Jason Brownlee, who is
the founder of machinelearningmastery.com.

4.5. Implement a decision tree algorithm that incorporates the Gini impurity measure,
from scratch, to run for the data used on page 96, Python Machine Learning, 3rd Ed..
Compare your results with the figure on page 97 of the book. You may refer to
https://machinelearningmastery.com/implement-decision-tree-algorithm-scratch-python/
4.6. Implement a k -NN algorithm, from scratch, to run for the data used on page 106,
Python Machine Learning, 3rd Ed.. Compare your results with the figure on page
103 of the book. You may refer to
https://machinelearningmastery.com/tutorial-to-implement-k-nearest-neighbors-in-python-from-
scratch/
112 Chapter 4. Popular Machine Learning Classifiers
Chapter 5

Quadratic Programming
Quadratic programming (QP) is the process of solving a constrained
quadratic optimization problem. That is, the objective f is quadratic and
the constraints are linear in several variables x ∈ Rn . Quadratic program-
ming is a particular type of nonlinear programming. Its general form is
1 T
minn f (x) := x A x − x T b, subj.to
x ∈R 2
Cx = c, (5.1)
Dx ≤ d,
where A ∈ Rn×n is symmetric, C ∈ Rm×n , D ∈ Rp ×n , b ∈ Rn , c ∈ Rm , and
d ∈ Rp .

In this chapter, we will study various meth-

ods for solving the QP problem (5.1), involv-
ing
• the method of Lagrange multipliers,
• direct solution methods, and
• iterative solution methods.

Contents of Chapter 5
5.1. Equality Constrained Quadratic Programming . . . . . . . . . . . . . . . . . . . . . . . 114
5.2. Direct Solution for the KKT System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.3. Linear Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.4. Iterative Solution of the KKT System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.5. Active Set Strategies for Convex QP Problems . . . . . . . . . . . . . . . . . . . . . . . 131
5.6. Interior-point Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5.7. Logarithmic Barriers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Exercises for Chapter 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

113
114 Chapter 5. Quadratic Programming

5.1. Equality Constrained Quadratic Program-

ming
If only equality constraints are imposed, the QP (5.1) reduces to
1 T
minn f (x) := x A x − x T b, subj.to
x ∈R 2 (5.2)
Cx = c,

where A ∈ Rn×n , C ∈ Rm×n , m ≤ n. For the time being we assume that C

has full row rank m and m < n.

To solve the problem, let’s begin with its Lagrangian:

1 T
L(x, λ) = x A x − x T b − λ(c − Cx), (5.3)
2
where λ ∈ Rm is the the associated Lagrange multiplier. Then, we have
∇x L(x, λ) = A x − b + C T λ. (5.4)

The KKT conditions (first-derivative tests) for the solution x ∈ Rn of

(5.2) give rise to the following linear system
A CT x
    
b
= , (5.5)
C 0 λ c
| {z }
:=K

where the second row is the primal feasibility.

Let Z ∈ Rn×(n−m) be a matrix whose columns span the null space of C , N (C),
i.e.,
CZ = 0 or Span(Z) = N (C). (5.6)

Definition 5.1. The matrix K in (5.5) is called the KKT matrix and
the matrix Z T AZ is referred to as the reduced Hessian.

Note: Now, a question is: “Is the KKT matrix nonsingular?"

5.1. Equality Constrained Quadratic Programming 115

Definition 5.2. Let A be a symmetric matrix. We say that A is (pos-

itive) semidefinite, and we write A  0, if all eigenvalues of A are
nonnegative. We say that A is (positive) definite, and write A  0, if
all eigenvalues of A are positive.

Lemma 5.3. Assume that A ∈ Rn×n is symmetric and (positive) semidef-

inite (A  0) and C ∈ Rm×n has full row rank m ≤ n. Then the following
are equivalent.
(a) N (A ) ∩ N (C) = {0}.
(b) Cx = 0, x = 6 0 ⇒ x T A x > 0.
(c) Z T AZ is positive definite ( 0), where Z ∈ Rn×(n−m) is a matrix for
which Span(Z) = N (C).
(d) A + C T QC  0 for some Q  0.

Proof. See Exercise 1.

Proposition 5.4. For a symmetric matrix A , the following are equiva-

lent.
1. A  0.
2. A = U T U for some U .
3. x T A x ≥ 0 for all x ∈ Rn .
4. All principal minors of A are nonnegative.
5. There exist x 1 , x 2 , · · · , x k ∈ Rn such that
Xk
A= x i x Ti . (5.7)
i=1

Definition 5.5. For A , B ∈ Rn×n , we define the dot product of matri-

ces as Xn X n
A ·B = Aij Bij = tr(A T B). (5.8)
i=1 j=1
116 Chapter 5. Quadratic Programming

Proposition 5.6.
• If A , B  0, then A · B ≥ 0, and A · B = 0 implies AB = 0.
• A symmetric matrix A is semidefinite if A · B ≥ 0 for every B  0.

Theorem 5.7. (Existence and uniqueness). Assume that C ∈ Rm×n

has full row rank m ≤ n and that the reduced Hessian Z T AZ is positive
definite. Then, the KKT matrix K is nonsingular and therefore the
KKT system (5.5) admits a unique solution (x ∗ , λ∗ ).

Proof. Suppose that x ∈ N (A ) ∩ N (C), x 6= 0. ⇒ A x = Cx = 0 and therefore

A CT x
    
x
K = = 0, (5.9)
0 C 0 0

which implies the KKT matrix K is singular.

Now, we assume that the KKT matrix is singular. That is, there are
x ∈ Rn , z ∈ Rm , not both zero, such that
T
    
x A C x
K = = 0,
z C 0 z

which implies
A x + C T z = 0 and Cx = 0. (5.10)
It follows from the above equations that

0 = x T A x + x T C T z = x T A x,

which contradicts (b) in Lemma 5.3, unless x = 0.

In the case (i.e., x = 0), we must have z 6= 0. But then C T z = 0 contracts the
assumption that C has full row rank.

Note: More generally, the nonsingularity of the KKT matrix is equiva-

lent to each of statements in Lemma 5.3.
5.1. Equality Constrained Quadratic Programming 117

Theorem 5.8. (Global minimizer of (5.2)). Let the assumptions in

Theorem 5.7 be satisfied and (x ∗ , λ∗ ) the unique solution of the KKT sys-
tem (5.5). Then x ∗ is the unique global solution of the QP (5.2).

Proof. When m = n, the theorem is trivial; we may assume m < n.

Let F = {x ∈ Rn | Cx = c }, the feasible set.
Clearly, x ∗ is a solution of (5.2), i.e., x ∗ ∈ F.
Let x ∈ F be another feasible point and p := x ∗ − x 6= 0.
Then x = x ∗ − p and
1 ∗
f (x) = (x − p)T A (x ∗ − p) − (x ∗ − p)T b
2 (5.11)
1 T
= p A p − p T A x ∗ + p T b + f (x ∗ ).
2
Now, (5.5) implies that A x ∗ = b − C T λ∗ and thus

p T A x ∗ = p T (b − C T λ∗ ) = p T b − p T C T λ∗ = p T b,

where we have used Cp = C(x ∗ − x) = c − c = 0. Hence, (5.11) reduces to

1 T
f (x) = p A p + f (x ∗ ). (5.12)
2
Since p ∈ N (C), we can write p = Zy , for some nonzero y ∈ Rn−m , and
therefore
1
f (x) = y T Z T AZy + f (x ∗ ). (5.13)
2
Since Z T AZ  0, we deduce f (x) > f (x ∗ ); consequently, x ∗ is the unique global
minimizer of (5.2).
118 Chapter 5. Quadratic Programming

Theorem 5.9. Let the assumptions in Theorem 5.7 be satisfied. Then

the KKT matrix K has exactly n positive and m negative eigenvalues.
Proof. From Lemma 5.3, A + C T C  0; also see (5.89), p. 139. Therefore there exists a
nonsingular matrix R ∈ Rn×n such that
R T (A + C T C)R = I. (5.14)
Let CR = UΣV1T be the singular value decomposition of CR , where U ∈ Rm×m ,
Σ = diag(σ1 , σ2 , · · · , σm ), and V1 ∈ Rn×m . Let V2 ∈ Rn×(n−m) such that
V = [V1 V2 ]
is orthogonal, and define
S = [Σ 0] ∈ Rm×n .
Then, we have
CR = USV T (5.15)
and therefore
V T R T (A + C T C)RV = V T R T ARV + (CRV)T CRV = I.
It follows from (5.15) that S = U T CRV and therefore
S T S = (U T CRV )T U T CRV = (CRV)T CRV .
Thus we have Λ := V T R T ARV = I − S T S is diagonal; we can write
Λ = V T R T ARV = diag(1 − σ12 , 1 − σ22 , · · · , 1 − σm
2
, 1, · · · , 1). (5.16)
Now, applying a congruence transformation to the KKT matrix gives
Λ S
 T T  T
   T

V R 0 A C RV 0
= (5.17)
0 UT C 0 0 U S 0
and the inertia of the KKT matrix is equal to the inertia of the matrix on the right.1
Applying a permutation to the matrix on the right of (5.17) gives a block diagonal matrix
with n diagonal blocks
 
λi σ i  
, i = 1, 2, · · · , m; λi , i = m + 1, , · · · , n, (5.18)
σi 0
where λi are as in (5.16). The eigenvalues of the 2 × 2-blocks are
p
λi ± λ2i + 4σi2
, i = 1, 2, · · · , m,
2
i.e., one eigenvalue is positive and one is negative. So we can conclude that there are
m + (n − m) = n positive eigenvalues and m negative eigenvalues.
1
Sylvester’s law of inertia is a theorem in matrix algebra about certain properties of the coefficient
matrix of a real quadratic form that remain invariant under a change of basis. Namely, if A is the symmetric
matrix that defines the quadratic form, and S is any invertible matrix such that D = SAS T is diagonal, then
the number of negative elements in the diagonal of D is always the same, for all such S ; and the same goes
for the number of positive elements.
5.2. Direct Solution for the KKT System 119

5.2. Direct Solution for the KKT System

Recall: In (5.5), the KKT system for the solution x ∈ Rn of (5.2) reads
A CT x
    
b
= , (5.19)
C 0 λ c
| {z }
:=K

where the second row is the primal feasibility.

For direct solutions of the KKT system (5.19), this section considers sym-
metric factorization, the range-space approach, and the null-space approach.

5.2.1. Symmetric factorization

A method to solve the KKT system (5.19) is to provide a symmetric fac-
torization of the KKT matrix:

PKP T = LDL T , (5.20)

where P is a permutation matrix (appropriately chosen), L is lower triangu-

lar with diag(L ) = I, and D is block diagonal. Based on (5.20), we rewrite the
KKT system (5.19) as
         
b x x x
P = PK = PKP T P = LDL T P .
c λ λ λ

Thus it can be solved as follows.

 
b
solve L y 1 = P
c
solve Dy 2 = y 1
(5.21)
solve L T y 3 = y 2
 ∗
x T
set ∗ = P y3
λ
120 Chapter 5. Quadratic Programming

In python, scipy.linalg.ldl is available.

ldl_test.py
1 import numpy as np
2 from scipy.linalg import ldl
3

4 A = np.array([[ 2, -1, 0],

5 [-1, 0, -1],
6 [ 0, -1, 4]])
7 L0,D,P = ldl(A, lower=1) # Use the default: the lower matrix
8

9 print('L0=\n',L0)
10 print('D=\n',D)
11 print('P=\n',P)
12 print('L0*D*L0^T=\n',L0.dot(D).dot(L0.T),'\n#-----------')
13

14 P_L0 = L0[P,:] # the row permutation of 'L0'

15 print('P*L0=\n',P_L0)

Result • As one can see from the result,

1 L0= P*L0 is a lower triangular matrix.
2 [[ 1. 0. 0. ]
3 [-0.5 -0.25 1. ] The output L0 of Python function
4 [ 0. 1. 0. ]] ldl is permuted as
5 D=
6 [[ 2. 0. 0. ] L0 = P T L . (5.22)
7 [ 0. 4. 0. ]
8 [ 0. 0. -0.75]]
9 P= • Reference: [10] J.R. Bunch and
10 [0 2 1]
L. Kaufman, Some stable methods
11 L0*D*L0^T=
12 [[ 2. -1. 0.] for calculating inertia and solving
13 [-1. 0. -1.] symmetric linear systems, Math.
14 [ 0. -1. 4.]]
15 #-----------
Comput. 31, 1977, pp.163-179.
16 P*L0=
17 [[ 1. 0. 0. ]
18 [ 0. 1. 0. ]
19 [-0.5 -0.25 1. ]]
5.2. Direct Solution for the KKT System 121

5.2.2. Range-space approach

Recall: The KKT system for the solution x ∈ Rn of (5.2) is given by

A CT x
    
b
= , (5.23)
C 0 λ c
| {z }
:=K

where the second row is the primal feasibility.

The range-space approach applies, when A ∈ Rn×n is symmetric posi-

tive definite. Block Gauss elimination of the primal variable x leads to the
Schur complement system
CA −1 C T λ = CA −1 b − c, (5.24)
where S := CA −1 C T ∈ Rm×m is the Schur complement. See Exercise 2.

Note: Once the optimal Lagrange multipliers λ∗ is determined from

(5.24), the minimizer x ∗ can be obtained by solving the first equation
of the KKT system
A x = b − C T λ∗ . (5.25)

Theorem 5.10. Suppose that A ∈ Rn×n is symmetric positive definite,

C ∈ Rm×n , m ≤ n, such that rank(C) = m. Let S = CA −1 C T be the Schur
complement associated with the KKT-matrix. Then S is symmetric
positive definite on Rm .

Proof. See Exercise 3.

Remark 5.11. The range-space approach is particularly effective, when

• The matrix A is well conditioned and efficiently invertible.
(e.g., diagonal or block-diagonal)
• Its inverse A −1 is known explicitly.
(e.g., by means of a quasi-Newton updating formula)
• The number of equality constraints (m) is small.

Note that C ∈ Rm×n and it can be considered as a map C : Rn → Rm .

122 Chapter 5. Quadratic Programming

5.2.3. Null-space approach

The null-space approach does not require regularity of A and thus
has a wider range of applicability than the range-space approach. The
method begins with some assumptions.

1. Assume that C ∈ Rm×n has full row rank m.

2. Assume that Z T AZ  0, where Z ∈ Rn×(n−m) is a matrix for which Span(Z) = N (C) and
CZ = 0; see (5.6).

3. Let Y ∈ Rn×m be a matrix such that [Y Z] ∈ Rn×n is nonsingular.

4. Partition the vector x ∈ Rn according to
x = Y w Y + Zw Z , (5.26)

where w Y ∈ Rm , w Z ∈ Rn−m .

5. Substitute (5.26) into the second equation of (5.19) to have

CZ w Z = c ⇒ CY w Y = c,
Cx = CY w Y + |{z} (5.27)
=0

i.e. Y w Y is a particular solution of Cx = c .

6. Furthermore, w Y is well determined by (5.27). ∵ Since C ∈ Rm×n has full row rank
m and [Y Z] ∈ Rn×n is nonsingular, the product C[Y Z] = [CY 0] ∈ Rm×n has full row
rank m and therefore CY ∈ Rm×m is nonsingular.

7. On the other hand, substituting (5.26) into the first equation of (5.19), we get
A x + C T λ = AY w Y + AZw Z + C T λ = b. (5.28)

Multiplying Z T and observing Z T C T = (CZ)T = 0 yield

Z T AZw Z = Z T b − Z T AY w Y . (5.29)

The reduced KKT system (5.29) can be solved easily e.g. by a Cholesky factoriza-
tion of the reduced Hessian Z T AZ ∈ R(n−m)×(n−m) .

8. Once w Y and w Z have been computed as solutions of (5.27) and (5.29), respectively,
x ∗ is obtained from (5.26).
9. When Lagrange multipliers λ∗ is to be computed, we multiply (5.28) by Y T and solve
the resulting equation:
(CY )T λ∗ = Y T b − Y T A x ∗ . (5.30)
5.3. Linear Iterative Methods 123

5.3. Linear Iterative Methods

Consider a linear algebraic system

A x = b, (5.31)

for which we assume that A ∈ Rn×n is invertible.

Key Idea 5.12. Iterative methods for solving (5.31):

• Linear iterative methods begin with splitting the matrix A by

A = M − N, (5.32)

for some invertible matrix M .

• Then, the linear system equivalently reads

Mx = Nx + b. (5.33)

• Associated with the splitting is an iterative method

Mx k = Nx k −1 + b, (5.34)

or, equivalently,

x k = M −1 (Nx k −1 + b) = x k −1 + M −1 (b − A x k −1 ), (5.35)

for an initial value x 0 .

Note: Methods differ for different choices of M .

• M must be easy to invert (efficiency), and
• M −1 ≈ A −1 (convergence).
124 Chapter 5. Quadratic Programming

5.3.1. Convergence theory

• Let
ek = x − xk .

• It follows from (5.33) and (5.34) that the error equation reads

M e k = N e k −1 (5.36)

or, equivalently,
e k = M −1 N e k −1 . (5.37)

• Since
ke k k ≤ kM −1 N k · ke k −1 k ≤ kM −1 N k2 · ke k −2 k
(5.38)
≤ · · · ≤ kM −1 N kk · ke 0 k,
a sufficient condition for the convergence is

kM −1 N k < 1. (5.39)

Definition 5.13. Let σ (B) be the spectrum, the set of eigenvalues of

the matrix B , and ρ(B) denote the spectral radius defined by

ρ(B) = max |λi |.

λi ∈σ (B)

Theorem 5.14. The iteration converges if and only if

ρ(M −1 N) < 1. (5.40)

5.3.2. Graph theory: Estimation of the spectral radius

Definition 5.15. A permutation matrix is a square matrix in which
each row and each column has one entry of unity, all others zero.
5.3. Linear Iterative Methods 125

Definition 5.16. For n ≥ 2, an n × n complex-valued matrix A is re-

ducible if there is a permutation matrix P such that
 
A11 A12
PAP T = ,
0 A22

where A11 and A22 are respectively r × r and (n − r) × (n − r) submatrices,

0 < r < n. If no such permutation matrix exists, then A is irreducible.

The geometrical interpretation of the concept of the irreducibility by means

of graph theory is useful.
Geometrical interpretation of irreducibility

in the plane, which we will call

nodes or nodal points.
• For any nonzero entry aij of A , we
−→
connect Pi to Pj by a path Pi Pj , di-
Figure 5.1: The directed paths for nonzero aii
and aij .
rected from the node Pi to the node
Pj ; a nonzero aii is joined to itself
by a directed loop, as shown in Fig-
ure 5.1.
• In this way, every n × n matrix
A can be associated a directed
graph G(A ). For example, the ma-
trix
Figure 5.2: The directed graph G(A ) for A in  
(5.41). 2 −1 0
A =  −1 2 −1  (5.41)
• Given A = [aij ] ∈ Cn×n , consider n 0 −1 2
distinct points
has a directed graph shown in Fig-
P1 , P2 , · · · , Pn ure 5.2.
126 Chapter 5. Quadratic Programming

Definition 5.17. A directed graph is strongly connected if, for any

ordered pair of nodes (Pi , Pj ), there is a directed path of a finite length
−→ −→ −→
Pi Pk1 , Pk1 Pk2 , · · · , Pkr −1 Pkr =j ,

connecting from Pi to Pj .

The theorems to be presented in this subsection can be found in [66] along

with their proofs.
Theorem 5.18. An n × n complex-valued matrix A is irreducible if and
only if its directed graph G(A ) is strongly connected.

5.3.3. Eigenvalue locus theorem

For A = [aij ] ∈ Cn×n , let n
X
Λi := |aij | (5.42)
j=1
j 6= i

Theorem 5.19. (Eigenvalue locus theorem) Let A = [aij ] be an irre-

ducible n × n complex matrix. Then,
1. (Gerschgorin [18]) All eigenvalues of A lie in the union of the disks
in the complex plane

|z − aii | ≤ Λi , 1 ≤ i ≤ n. (5.43)

2. (Taussky [63]) In addition, assume that λ, an eigenvalue of A , is a

boundary point of the union of the disks |z − aii | ≤ Λi . Then, all the
n circles |z − aii | = Λi must pass through the point λ, i.e., |λ − aii | = Λi
for all 1 ≤ i ≤ n.
5.3. Linear Iterative Methods 127

Example 5.20. For example, for

 
2 −1 0
A = −1
 2 −1 
0 −1 2

Λ1 = 1, Λ2 = 2, and Λ3 = 1. Since aii = 2, for i = 1, 2, 3,

|λ − 2| < 2

for all eigenvalues λ of A .

Positiveness
Definition 5.21. An n × n complex-valued matrix A = [aij ] is diagonally
dominant if
n
X
|aii | ≥ Λi := |aij |, (5.44)
j=1
j 6= i

for all 1 ≤ i ≤ n. An n × n matrix A is irreducibly diagonally dominant if

A is irreducible and diagonally dominant, with strict inequality holding
in (5.44) for at least one i .

Theorem 5.22. Let A be an n × n strictly or irreducibly diagonally

dominant complex-valued matrix. Then, A is nonsingular. If all the di-
agonal entries of A are in addition positive real, then the real parts of all
eigenvalues of A are positive.

Corollary 5.23. A Hermitian matrix satisfying the conditions in Theo-

rem 5.22 is positive definite.
128 Chapter 5. Quadratic Programming

5.3.4. Regular splitting and M-matrices

Definition 5.24. For n × n real matrices, A , M , and N , A = M − N is a
regular splitting of A if M is nonsingular with M −1 ≥ 0, and N ≥ 0.

Theorem 5.25. If A = M − N is a regular splitting of A and A −1 ≥ 0,

then
−1 ρ(A −1 N)
ρ(M N) = < 1. (5.45)
1 + ρ(A −1 N)
Thus, the matrix M −1 N is convergent and the iterative method of (5.34)
converges for any initial value x 0 .

Definition 5.26. An n × n real matrix A = [aij ] with aij ≤ 0 for all i 6= j is

an M-matrix if A is nonsingular and A −1 ≥ 0.

Theorem 5.27. Let A = (aij ) be an n × n M -matrix. If M is any n × n

matrix obtained by setting certain off-diagonal entries of A to zero, then
A = M − N is a regular splitting of A and ρ(M −1 N) < 1.

Theorem 5.28. Let A be an n × n real matrix with A −1 > 0, and A =

M1 − N1 = M2 − N2 be two regular splittings of A . If N2 ≥ N1 ≥ 0, where
neither N2 − N1 nor N1 is null, then

1 > ρ(M2−1 N2 ) > ρ(M1−1 N1 ) > 0. (5.46)

5.4. Iterative Solution of the KKT System 129

5.4. Iterative Solution of the KKT System

Recall: The KKT system for the solution x ∈ Rn of (5.2) is given by
A CT x
    
b
= , (5.47)
C 0 λ c
| {z }
:=K

where the second row is the primal feasibility.

When the direct solution of the KKT system (5.47) is computationally

too costly, the alternative is to use an iterative method. An iterative
solver can be applied
• either to the entire KKT system
• or to the special structure of the KKT matrix, as in the range-
space and null-space approach, and based on regular splittinga of
specifically transformed matrices of K .
a
For n × n real matrices, A , M , and N , A = M − N is a regular splitting of A if M is nonsingular with
−1
M ≥ 0, and N ≥ 0.

The transforming null-space iteration does not require regularity of A

and therefore has a wider range of applicability than the transforming
range-space iteration. Here we will deal with the transforming range-
space iteration only for simplicity.

5.4.1. Krylov subspace methods

The KKT matrix K ∈ R(n+m)×(n+m) is indefinite; if C has full row rank m,
then K has exactly n positive and m negative eigenvalues, as shown in The-
orem 5.9. Therefore, for iterative methods for the solution of entire KKT
system, appropriate candidates are Krylov subspace methods like

• GMRES (Generalized Minimum RESidual) and

• QMR (Quasi Minimum Residual).
130 Chapter 5. Quadratic Programming

5.4.2. The transforming range-space iteration

Assumption. The matrix A ∈ Rn×n is symmetric positive definite (SPD)

and A has an easily invertible SPD approximation A b −1 A ∼ I .
b such that A

1. We choose L ∈ R(n+m)×(n+m) as a lower triangular block matrix

 
I 0
L= − 1 , (5.48)
−C A
b I

which gives rise to the regular splitting of LK :

" # " #
b CT
A A b −1 A ) 0
b (I − A
LK =
0 Sb − C(I − A b −1 A ) 0 =: M1 − M2 , (5.49)

b −1 C T ∈ Rm×m . (Note M2 = M1 − LK ∼ 0.)

b = −C A
where S
2. Let
ψ := (x, λ)T , β := (b, c)T .
Then the KKT system (5.47) gives LK ψ = (M1 − M2 ) ψ = L β so that
M1 ψ = M2 ψ + L β
(5.50)
= (M1 − LK ) ψ + L β = M1 ψ + L (β − K ψ ).

3. Given an initialization ψ 0 ∈ R(n+m)×(n+m) , we compute ψ k +1 by means of

the transforming range-space iteration
ψ k +1 = (I − M1−1 LK ) ψ k + M1−1 L β
(5.51)
= ψ k + M1−1 L (β − K ψ k ), k ≥ 0.

Implementation of (5.51):
(1) T (2)
compute r k " k , r k ) := β −#K ψ k ;
= (r
(1)
rk
compute L r k = b −1 r (1) + r (2) ;
−C A k k
(5.52)
solve M1 ∆ψ k = L rk ;
set ψ k +1 = ψ k + ∆ψ k ;
5.5. Active Set Strategies for Convex QP Problems 131

5.5. Active Set Strategies for Convex QP Prob-

lems
Recall: Quadratic programming formulated in a general form (5.1):
1 T
minn f (x) := x A x − x T b, subj.to
x ∈R 2
Cx = c, (5.53)
Dx ≤ d,

where A ∈ Rn×n , C ∈ Rm×n , D ∈ Rp ×n , and b ∈ Rn , c ∈ Rm , d ∈ Rp .

Here, we assume A is SPD.

Definition 5.29. The inequality constraint in (5.53) can be written as

gi (x) ≤ 0, i = 1, 2, · · · , p.

Given a point x in the feasible region, a constraint gi (x) ≤ 0 is called

active at x if gi (x) = 0 and inactive if gi (x) 6= 0. The active set at
x is made up of those constraints that are active at the current point.
(Equality constraints are always active.)

Note: The active set is particularly important in optimization theory,

because it determines which constraints will influence the final result of
optimization. For example,
• For linear programming problem, the active set gives the hyper-
planes that intersect at the solution point.
• In quadratic programming, the solution is not necessarily on one
of the edges of the bounding polygon; an estimation of the active
set gives us a subset of inequalities to watch while searching the
solution, which reduces the complexity of the search [49].
132 Chapter 5. Quadratic Programming

5.5.1. Primal active set strategy

We rewrite the matrices
  C and D in the form
 
C1 D1
C =  ...  , C i ∈ Rn ; D =  ...  , D i ∈ Rn . (5.54)
   
Cm Dp

Then the inequality constraints in (5.53) can be equivalently stated as

D Ti x ≤ di , i = 1, 2, · · · , p. (5.55)

(C i and D i are row vectors; we will deal with them like column vectors.)
The primal active set strategy is an iterative procedure:
• Given a feasible iterate x k , k ≥ 0, we determine its active set

Iac (x k ) ⊂ {1, 2, · · · , p } (5.56)

and consider the corresponding constraints as equality constraints,

whereas the remaining inequality constraints are disregarded.
• Setting
p = x k − x, r k = A x k − b, (5.57)
we find
1 T
f (x) = f (x k − p) = p A p − r Tk p + g, (5.58)
2
where g = 12 x Tk A x k − b T x k .
• Then the equality constrained QP problem to be solved for the (k + 1)-st
iteration step is:
1 
p k = arg minn p T A p − r Tk p , subj.to
p ∈R 2
Cp = 0
(5.59)
D Ti p = 0, i ∈ Iac (x k ).

• The new iterate is then obtained according to

x k +1 = x k − αk p k , (5.60)

where αk is chosen such that x k +1 stays feasible.

5.5. Active Set Strategies for Convex QP Problems 133

Remark 5.30. (Determination of αk ). The parameter can be deter-

mined as follows.
• For each i ∈ Iac (x k ), we have
D Ti x k +1 = D Ti x k − αk D Ti p k = D Ti x k ≤ di . (5.61)

• If D Ti p k ≥ 0 for some i 6∈ Iac (x k ), it follows that

D Ti x k +1 = D Ti x k − αk D Ti p k ≤ D Ti x k ≤ di . (5.62)

• On the other hand, if D Ti p k < 0 for some i 6∈ Iac (x k ), we have

di − D Ti x k
D Ti x k +1 = D Ti x k − αk D Ti p k ≤ di ⇐⇒ αk ≤ . (5.63)
−D Ti p k

• Consequently, in order to guarantee feasibility, we choose

D Ti x k − di
bk ), where α
αk := min(1, α bk := min . (5.64)
i 6∈Iac (x k ); D Ti p k <0 D Ti p k

Remark 5.31. (Update for Iac (x k +1 )). Let’s begin with defining the set
of blocking constraints:

def
n D Ti x k − di o
Ibl (p k ) == i 6∈ Iac (x k ) | D Ti p k < 0, ≤1 . (5.65)
D Ti p k
Then we specify Iac (x k +1 ) by adding the most restrictive blocking con-
straint to Iac (x k ):

def
n D Tj x k − dj o
Iac (x k +1 ) == Iac (x k ) ∪ j ∈ Ibl (p k ) | =α
bk . (5.66)
D Tj p k

For such a j (the index of the newly added constraint), we clearly have

D Tj x k +1 = D Tj x k − αk D Tj p k = D Tj x k − α
bk D Tj p k = dj , (5.67)

and therefore Iac (x k +1 ) contains active constraints only.

134 Chapter 5. Quadratic Programming

5.6. Interior-point Methods

Interior-point methods are iterative schemes where the iterates approach
the optimal solution from the interior of the feasible set. For simplicity, we
consider inequality-constrained quadratic programming problems of
the form
1 T
minn Q(x) := x A x − x T b, subj.to
x ∈R 2 (5.68)
Dx ≤ d,
where A ∈ Rn×n is SPD, D ∈ Rp ×n , and b ∈ Rn and d ∈ Rp .

Note: Its Lagrangian reads

p
def
X
L(x, µ) == Q(x) + µi (D Ti x − di ), (5.69)
i=1

where µ = (µ1 , µ2 , · · · , µp )T and D = [D 1 , · · · , D p ]T , and therefore

∇x L(x, µ) = ∇x Q(x) + D T µ = A x − b + D T µ. (5.70)

Thus the KKT conditions for (5.68) are stated as

A x + D T µ − b = 0,
Dx − d ≤ 0,
(5.71)
µi (Dx − d)i = 0, i = 1, 2, · · · , p,
µi ≥ 0, i = 1, 2, · · · , p.
{µi (Dx − d)i = 0} is the complementary slackness.
By introducing a slack variable z = d − Dx , the above conditions can be
equivalently formulated as follows:
A x + DT µ − b = 0,
Dx + z − d = 0,
(5.72)
µi zi = 0, i = 1, 2, · · · , p,
µi , zi ≥ 0, i = 1, 2, · · · , p.

The interior-point method begins with replacing µi zi = 0 by µi zi = θ > 0

for all i = 1, 2, · · · , p , and enforces θ & 0.
5.6. Interior-point Methods 135

Equation (5.72) can be rewritten as a constrained system of nonlinear equa-

tions. We define the nonlinear map 
A x + DT µ − b

def
F(x, µ, z) ==  Dx + z − d  , (5.73)
ZMe
where
Z = diag(z1 , · · · , zp ), M = diag(µ1 , · · · , µp ), e = (1, · · · , 1)T .

Definition 5.32. (Central path). The set of points (x τ , µτ , z τ ), τ > 0,

satisfying  
0
F(x τ , µτ , z τ ) =  0  , z, µ ≥ 0, (5.74)
τe
is called the central path.

Newton’s method:
• Given a feasible iterate (x, µ, z) = (x k , µk , z k ), we introduce a duality
measure θ: p
1X zT µ
θ := zi µi = . (5.75)
p p
i=1

• The idea is to apply Newton’s method to (5.73) to compute

(x σθ , µσθ , z σθ ) on the central path, where σ ∈ [0, 1] is an algorithm pa-
rameter.
• The Newton increments (∆x, ∆µ, ∆z) solve the linear system
∆x
   
0
∇F(x, µ, z) ∆µ = −F(x, µ, z) +  0  , (5.76)
∆z σθe

where A DT 0
 
def
∇F(x, µ, z) == D 0 I .
0 Z M

• The new iterate (x k +1 , µk +1 , z k +1 ) is then determined by means of

(x k +1 , µk +1 , z k +1 ) = (x k , µk , z k ) + α(∆x, ∆µ, ∆z), (5.77)

with α chosen such that the new iterate stays feasible.

136 Chapter 5. Quadratic Programming

5.7. Logarithmic Barriers

Recall: The inequality-constrained quadratic programming prob-
lem in (5.68):
1 T
minn Q(x) := x A x − x T b, subj.to
x ∈R 2 (5.78)
Dx ≤ d,

where A ∈ Rn×n is SPD, D ∈ Rp ×n , and b ∈ Rn and d ∈ Rp .

Algorithms based on barrier functions are iterative methods where the it-
erates are forced to stay within the interior of the feasible set:
def
F int == {x ∈ Rn | D Ti x − di < 0, 1 ≤ i ≤ p }. (5.79)

Barrier functions commonly have the following properties:

• They are smooth within F int .
• They approach ∞ as x approaches the boundary of F int .
• They are infinite outside F int .

Definition 5.33. (Logarithmic barrier function). For the QP prob-

lem (5.78), the objective functional
p
def
X
Bβ (x) == Q(x) − β log(di − D Ti x), β > 0, (5.80)
i=1

is called the logarithmic barrier function.a The parameter β is re-

ferred to as the barrier parameter.
a
The function log stands for the natural logarithm (ln), as commonly accepted in the literature of
computational algorithms.
5.7. Logarithmic Barriers 137

Theorem 5.34. (Properties of the logarithmic barrier function)

[68, (Wright,1992)]. Assume that the set S of solutions of (5.78) is
nonempty and bounded and that the interior F int of the feasible set is
nonempty. Let {βk } be a decreasing sequence of barrier parameters with

βk → 0, as k → ∞. (5.81)
Then there holds:
1. For any β > 0, the logarithmic barrier function Bβ (x) is convex in
F int and attains a minimizer x β ∈ F int .
2. There is a unique minimizer; any local minimizer x β is also a
global minimizer of Bβ (x).

3. If {x βk | k ∈ N} is a sequence of minimizers, then there exists N0 ⊂ N

such that
x βk → x ∗ ∈ S , k ∈ N 0 .

4. If Q ∗ is the optimal value of the objective functional Q in (5.78), then

for any sequence {x βk } of minimizers,
Q(x βk ) → Q ∗ , Bβk (x βk ) → Q ∗ , as k → ∞. (5.82)

Objective: In the following:

We will have a closer look at the relation between a minimizer of Bβ (x)
and the solution of the KKT system, a point (x, µ) satisfying the KKT
conditions for (5.78).
138 Chapter 5. Quadratic Programming

Recall: The KKT conditions for (5.78) are given in (5.71), p. 134:
∇x Q(x) + D T µ = A x − b + D T µ = 0,
Dx − d ≤ 0,
(5.83)
µi (Dx − d)i = 0, i = 1, 2, · · · , p,
µi ≥ 0, i = 1, 2, · · · , p.

If x β is a minimizer of Bβ (x), we obviously have

p
X β
∇x Bβ (x β ) = ∇x Q(x β ) + T
D i = 0. (5.84)
i=1
di − D i x β

Definition 5.35. Perturbed (or, approximate) complementarity is

the vector z β ∈ Rp having its components
β
(z β )i = zβ ,i := , 1 ≤ i ≤ p. (5.85)
di − D Ti x β

In terms of the perturbed complementarity, (5.84) can be stated as

p
X
∇x Q(x β ) + zβ ,i D i = 0. (5.86)
i=1

Rewrite the first of the KKT conditions (5.83) as

p
X
∇x Q(x) + µi D i = 0. (5.87)
i=1

• Obviously, (5.87) looks the same as (5.86).

• Apparently, the 2nd and the 4th KKT conditions in (5.83) are satis-
fied by (x, µ) = (x β , z β ).
• However, the 3rd KKT condition does not hold true, because it fol-
lows readily from (5.85) that
zβ ,i (di − D Ti x β ) = β > 0, 1 ≤ i ≤ p. (5.88)

As β → 0, the minimizer x β and the associated z β come closer and

closer to satisfying the 3rd KKT condition. This is why z β is called
perturbed (approximate) complementarity.
5.7. Logarithmic Barriers 139

Exercises for Chapter 5

5.1. Recall the KKT matrix  

A CT
K = .
C 0
Here, we assume that A ∈ Rn×n is symmetric and (positive) semidefinite (A  0) and
C ∈ Rm×n has full row rank m ≤ n. Show the following are equivalent.

(a) N (A ) ∩ N (C) = {0}.

(b) Cx = 0, x 6= 0 ⇒ x T A x > 0.
(c) Z T AZ is positive definite ( 0), where Z ∈ Rn×(n−m) is a matrix for which Span(Z) =
N (C).
(d) A + C T QC  0 for some Q  0.
When (c) is considered, you may assume m < n.
Hint : (a)⇔(b): Use contradictions. For example, N (A ) ∩ N (C) 6= {0}. ⇒ ∃x ∈
N (A ) ∩ N (C), x 6= 0. ⇒ A x = 0 and Cx = 0 and therefore x T A x = 0, which is a
contradiction; this implies (a)⇐(b).
(b)⇐(c): Let Cx = 0, x = 6 0. ⇒ x must have the form x = Zy for some y 6= 0. (why?) ⇒
T T T
x A x = y Z PZy > 0.
(b)⇔(d): If (b) holds, then
x T (A + C T C)x = x T A x + kCx k2 > 0. (5.89)
⇒ (d) holds with Q = I. On the other hand, if (d) holds for some Q  0, then
x T (A + C T QC)x = x T A x + x T C T QCx > 0, x 6= 0. (5.90)
When Cx = 0, x T A x must be positive.
Now, you should fill out missing gaps: (a)⇒(b) and (b)⇒(c).
5.2. Derive the Schur complement system (5.24) from the KKT system (5.19).
5.3. Suppose that A ∈ Rn×n is symmetric positive definite, C ∈ Rm×n , m ≤ n, such that
rank(C) = m. Let S = CA −1 C T be the Schur complement associated with the KKT-
matrix, as in (5.24). Show that S is symmetric positive definite on Rm .
Hint : You may begin with claiming that C T x =6 0 for every nonzero x ∈ Rm . (Figure
out why)
5.4. Verify the splitting (5.49).
5.5. Recall the dual problem of linear SVM in (4.50):
 1 T 
max α · 1 − α G α subj.to α · y = 0, (5.91)
0≤α≤C 2
where G = ZZ T and Gij = y (i) y (j) x (i) · x (j) .
(a) Ignoring momentarily the inequality constraints on the dual problem, 0 ≤ α ≤ C ,
prove that the problem has a unique solution. Hint : Formulate the KKT system
for the problem and check if it satisfies a statement in Lemma 5.3. You may use
the fact that G = ZZ T  0.
140 Chapter 5. Quadratic Programming

(b) Now, considering the inequality constraints, discuss why the problem is yet ad-
mitting a unique solution α∗ .
5.6. Consider the following equality-constrained QP problem

min 3x 2 + 2y 2 + 2z 2 − 2yz − 8x − 3y − 3z, subj.to

(x,y,z)∈R3
x + z = 1, (5.92)
y + z = 3.

Begin with pencil-and-paper:

(a) Formulate the KKT system for (5.92) of the form (5.5) by identifying A ∈ R3×3 ,
C ∈ R2×3 , and vectors b ∈ R3 and c ∈ R2 .
(b) Solve the QP problem (5.92) by the null-space approach. In particular, specify
the matrices Y and Z and compute w Y and w Z .
Now, use your computer:
(c) Implement the null-space algorithm presented in Section 5.2.3 to find the mini-
mizer of (5.92) numerically.
(d) Implement the range-space (Schur complement) method presented in Section 5.2.2
to find the minimizer of (5.92) numerically.
5.7. Now, consider the following inequality-constrained QP problem

min Q(x) := (x1 − 1)2 + 2(x2 − 2)2 + 2(x3 − 2)2 − 2x1 x2 , subj.to
x ∈R3
x1 + x2 − 3x3 ≤ 0, (5.93)
4x1 − x2 + x3 ≤ 1.

Implement the interior-point method with the Newton’s iterative update, pre-
sented in Section 5.6, to solve the QP problem (5.93).
(a) As in Exercise 6, you should first formulate the KKT system for (5.93) by identi-
fying A ∈ R3×3 , D ∈ R2×3 , and vectors b ∈ R3 and d ∈ R2 .
(b) Choose a feasible initial value (x 0 , µ0 , z 0 ).
(c) Select the algorithm parameter σ ∈ [0, 1] appropriately for Newton’s method.
(d) Discuss how to determine α in (5.77) in order for the new iterate (x k +1 , µk +1 , z k +1 )
to stay feasible.
Chapter 6

Data Preprocessing in Machine

Learning

Data preprocessing (or, data preparation) is a data mining technique,

which is the most time consuming (often, the most important) step in
machine learning.

Contents of Chapter 6
6.1. General Remarks on Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . 142
6.2. Dealing with Missing Data & Categorical Data . . . . . . . . . . . . . . . . . . . . . . . 144
6.3. Feature Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6.4. Feature Selection: Selecting Meaningful Variables . . . . . . . . . . . . . . . . . . . . . 147
6.5. Feature Importance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Exercises for Chapter 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

141
142 Chapter 6. Data Preprocessing in Machine Learning

6.1. General Remarks on Data Preprocessing

Data preprocessing is a data mining technique.
• It involves transforming raw data into a understandable and
more tractable format.
• Real-world data is often incomplete, redundant, inconsistent,
and/or lacking in certain behaviors or trends, and is likely to contain
many errors.
• Data preprocessing is a proven method of resolving such issues.
• Often, data preprocessing is the most important phase of a ma-
chine learning project, especially in computational biology.

Summary 6.1. Different steps involved for data preprocessing can be

summarized as follows.
1. Data Cleaning: In this first step, the primary focus is on handling
missing data, noisy data, detection and removal of outliers, and min-
imizing duplication and computed biases within the data.
2. Data Integration: This process is used when data is gathered from
various data sources and data are combined to form consistent data.
3. Data Transformation: This step is used to convert the raw data
into a specified format according to the need of the model.
(a) Normalization – Numerical data is converted into the specified
range (e.g., feature scaling → ∼ N (0, 1)).
(b) Aggregation – This method combines some features into one.
4. Data Reduction: Redundancy within the data can be removed and
efficiently organize the data.

The more disciplined you are in your handling of data, the more
consistent and better results you are likely to achieve.
6.1. General Remarks on Data Preprocessing 143

Remark 6.2. Data preparation is difficult because the process is not

objective, and it is important because ML algorithms learn from data.
Consider the following.
• Preparing data for analysis is one of the most important steps in
any data-mining project – and traditionally, one of the most time
consuming.
• Often, it takes up to 80% of the time.
• Data preparation is not a once-off process; that is, it is iterative
as you understand the problem deeper on each successive pass.
• It is critical that you feed the algorithms with the right data for
the problem you want to solve. Even if you have a good dataset, you
need to make sure that it is in a useful scale and format and that
meaningful features are included.

Questions in ML, in practice

• What is the best form of the data to describe the problem?
(It is difficult to answer, because it is not objective.)
• Can we design effective methods and/or smart algorithms for AI-
driven/automated data preparation (ADP)?
• What would reduce the generalization error?
144 Chapter 6. Data Preprocessing in Machine Learning

6.2. Dealing with Missing Data & Categorical

Data
6.2.1. Handling missing data
Software: [pandas.DataFrame].isnull().sum() > 1

For missing values, three different steps can be executed.

• Removal of samples (rows) or features (columns):
– It is the simplest and efficient method for handling the missing
data.
– However, we may end up removing too many samples or features.
Software: pandas.dropna
• Filling the missing values manually:
– This is one of the best-chosen methods.
– But there is one limitation that when there are large data set, and
missing values are significant.
• Imputing missing values using computed values:
– The missing values can also be occupied by computing mean, me-
dian, or mode of the observed given values.
– Another method could be the predictive values that are computed
by using any ML or Deep Learning algorithms.
– But one drawback of this approach is that it can generate bias
within the data as the calculated values are not accurate concern-
ing the observed values.
Software: from sklearn.preprocessing import Imputer
6.2. Dealing with Missing Data & Categorical Data 145

6.2.2. Handling categorical data

It is common that real-world datasets contain one or more categorical fea-
ture columns. These categorical features must be effectively handled to fit
in numerical computing libraries.

When we are talking about categorical data, we should further distin-

guish between ordinal features and nominal features.
• Mapping ordinal features: e.g.,
   
M 0
size : L
  ←→ 1 .

XL 2

Software: pandas.DataFrame
• Encoding nominal features: one-hot-encoding, e.g.,
     
blue 0 1 0 0
color :  green  ←→ 1 ←→ 0 1 0 .
red 2 0 0 1

Software: from sklearn.preprocessing import OneHotEncoder

• Encoding class labels:
   
label1 0
label : ←→ .
label2 1
Software: from sklearn.preprocessing import LabelEncoder
146 Chapter 6. Data Preprocessing in Machine Learning

6.3. Feature Scaling

Feature scaling is a method used to standardize the range of independent
variables or features of data. It is one of data normalization methods1
and is generally performed during the data preprocessing step.
Note: There are some scale-invariant algorithms such as decision trees
and random forests. However, most of other algorithms perform better
with feature scaling.

There are two common approaches to bring different features onto the same
scale:
• min-max scaling (normalization):
(i)
(i)
xj − xj,min
xj,norm = ∈ [0, 1], (6.1)
xj,max − xj,min
where xj,min and xj,max are the minimum and maximum of the j -th feature
column (in the training dataset), respectively.
• standardization:
(i)
(i)
xj − µj
xj,std = , (6.2)
σj
where µj is the sample mean of the j -th feature column and σj is the
corresponding standard deviation.2

Remark 6.3. Standardization is more practical for many ML methods,

especially for optimization algorithms such as gradient descent.
• Reason 1 : For many linear models such as the logistic regression
and the SVM, we can easily initialize the weights to 0 or small
random values close to 0.
• Reason 2 : It makes regularization perform more effectively; see
Section 4.2.3 for regularization.
1
In a broad sense, data normalization is a process of reorganizing data, by cleaning and adjusting data
values measured on different scales to a notionally common scale; its intention is to bring the entire
probability distributions of adjusted values into alignment.
2
As a rule of thumb, if all your variables are measured on the same scale and have the same unit, it might
be a good idea not to scale the variables (as for PCA based on the covariance matrix).
6.4. Feature Selection: Selecting Meaningful Variables 147

6.4. Feature Selection: Selecting Meaningful Vari-

ables
Remark 6.4. If we observe that a model performs much better on a
training dataset than on the test dataset, it is a strong indicator of over-
fitting.
• Overfitting means the model fits the parameters too closely with re-
gard to the particular observations in the training dataset, but does
not generalize well to new data. (The model has a high variance.)
• The reason for the overfitting is that our model is too complex for
the given training data.

Common solutions to reduce the generalization error (via bias-variance

tradeoff) are listed as follows:
• Collect more training data (often, not applicable)
• Introduce regularization (penalty for complexity)
• Choose a simpler model with fewer parameters
• Reduce the dimensionality by feature selection

Feature selection (a.k.a. variable selection): Its objective is four-

fold:
• enhancing generalization by reducing overfitting/variance,
• providing faster and more cost-effective predictors,
• reducing training time, and
• providing a better understanding of the underlying process that gen-
erated the data.

Recall: The curse of dimensionality describes the phenomenon

where the feature space becomes increasingly sparse for an increasing
number of dimensions of a fixed-size training dataset.
148 Chapter 6. Data Preprocessing in Machine Learning

Methods for “automatic” feature selection

• Filter methods: Filter methods suppress the least interesting fea-
tures, after assigning a scoring to each feature and ranking the fea-
tures. The methods consider the feature independently, or with regard
to the dependent variable.
Examples: Chi-squared test & correlation coefficient scores.
• Wrapper methods: Wrapper methods evaluate subsets of features
which allows, unlike filter approaches, to detect the possible interac-
tions between features. They prepare different combinations of fea-
tures, to evaluate and compare with other combinations. The two main
disadvantages of these methods are:
– Increasing overfitting risk when the number of observations is in-
sufficient
– Significant computation time when the number of variables is
large
Example: The recursive feature elimination algorithm
• Embedded methods: Embedded methods have been recently pro-
posed that try to combine the advantages of both previous methods.
They learn which features contribute the best to the accuracy of the
model while the model is being created. The most common types of
embedded feature selection methods are regularization methods.
Examples: ridge regressiona , LASSOb , & elastic net regularizationc
a
The ridge regression (a.k.a. Tikhonov regularization) is the most commonly used method of
regularization of ill-posed problems. In machine learning, ridge regression is basically a regularized lin-
ear regression model: minw Q(X , y; w) + λ2 kw k22 , in which the regularization parameter λ should be learned
as well, using a method called cross validation. It is related to the Levenberg-Marquardt algorithm for
non-linear least-squares problems.
b
LASSO (least absolute shrinkage and selection operator) is a regression analysis method that per-
forms both variable selection and regularization in order to enhance the prediction accuracy and inter-
pretability of the statistical model it produces. It includes an L 1 penalty term: minw Q(X , y; w) + λkw k1 .
It was originally developed in Geophysics [57, (Santosa-Symes, 1986)], and later independently redis-
covered and popularized in 1996 by Robert Tibshirani [64], who coined the term and provided further
insights into the observed performance.
c
The elastic net regularization is a regularized regression method that linearly combines the L 1
and L 2 penalties of the LASSO and ridge methods, particularly in the fitting of linear or logistic regres-
sion models.

We will see how L 1 -regularization can reduce overfitting (serving as a fea-

ture selection method).
6.4. Feature Selection: Selecting Meaningful Variables 149

6.4.1. Ridge regression vs. LASSO

Pm Pm
Figure 6.1: L 2 -regularization (kw k22 = i=1 wi2 ) and L 1 -regularization (kw k1 = i=1 |wi |).

Observation 6.5. When an L p -penalty term is involved (p = 1, 2), the

minimization problem can be written as follows:

w ∗ = arg min Q(X , y; w) + λRp (w), (6.3)

w

where
1
Rp (w) := kw kpp , p = 1, 2. (6.4)
p
• Regularization can be considered as adding a penalty term to the
cost function to encourage smaller weights; or in other words, we
penalize large weights.
• Thus, by increasing the regularization strength (λ ↑),
– we can shrink the weights towards zero, and
– decrease the dependence of our model on the training data.
• The minimizer w ∗ keeps a balance between Q(X , y; w) and λRp (w);
the weight coefficients cannot fall outside the shaded region, which
shrinks as λ increases.
• The minimizer w ∗ must be the point where the L p -ball intersects
with the minimum-valued contour of the unpenalized cost function.
150 Chapter 6. Data Preprocessing in Machine Learning

LASSO (L 1 -regularization). In the right figure, the L 1 -ball touches the

minimum-valued contour of the cost function at w1 = 0; the optimum
is more likely located on the axes, which encourages sparsity (zero
entries in w ∗ ).

Remark 6.6. LASSO (L 1 -regularization).

• We can enforce sparsity (more zero entries) by increasing the reg-
ularization strength λ.
• A sparse model is a model where many of the weights are 0 or close
to 0. Therefore L 1 -regularization is more suitable to create desired
0-weights, particularly for sparse models.

Remark 6.7. In general, regularization can be understood as adding

bias and preferring a simpler model to reduce the variance (over-
fitting), in the absence of sufficient training data, in particular.
• L 1 -regularization encourages sparsity.
• We can enforce sparsity (more zero entries) by increasing the reg-
ularization strength λ.
• Thus it can reduce overfitting, serving as a feature selection
method.

• L 1 -regularization may introduce oscillation, particularly when the

regularization strength λ is large.
• A post-processing operation may be needed to take into account
oscillatory behavior of L 1 -regularization.
6.4. Feature Selection: Selecting Meaningful Variables 151

Example 6.8. Consider a model consisting of the weights w =

(w1 , · · · , wm )T and
m m
X 1X 2
R1 (w) = |wi |, R2 (w) = wi . (6.5)
2
i=1 i=1

Let us minimize Rp (w), p = 1, 2, using gradient descent.

Solution. The gradients read

∇w R1 (w) = sign(w), ∇w R2 (w) = w, (6.6)

where 
1, if wi > 0

sign(wi ) = −1, if wi < 0
0, if wi = 0.


Thus the gradient descent becomes

R1 : w k +1 = w k − λ sign(w k ),
(6.7)
R2 : w k +1 = w k − λ w k = (1 − λ) w k = (1 − λ)k +1 w 0 .

• The L 2 -gradient is linearly decreasing towards 0 as the weight goes

towards 0. Thus L 2 -regularization will move any weight towards 0,
but it will take smaller and smaller steps as a weight approaches 0.
(The model never reaches a weight of 0.)
• In contrast, L 1 -regularization will move any weight towards 0 with
the same step size λ, regardless the weight’s value.
• The iterates for minimizing R1 may oscillate endlessly near 0.
(e.g., w0 = 0.2 and λ = 0.5 ⇒ w1 = w3 = · · · = −0.3 and w2 = w4 = · · · = 0.2)
• The oscillatory phenomenon may not be severe for real-world problems
where R1 is used as a penalty term for a cost function.
• We may need a post-processing to take account of oscillation, when
λ is set large.
152 Chapter 6. Data Preprocessing in Machine Learning

6.4.2. Sequential backward selection (SBS)

The idea behind the sequential backward selection (SBS) algorithm is
quite simple:
• SBS sequentially removes features from the full feature subset until
the new feature subspace contains the desired number of features.
• In order to determine which feature is to be removed at each stage,
we need to define the criterion function C, e.g., performance of the
classifier after the removal of a particular feature.
• Then, the feature to be removed at each stage can simply be defined
as the feature that maximizes this criterion; or in more intuitive
terms, at each stage we eliminate the feature that causes the least
performance loss after removal.

Based on the preceding definition of SBS, we can outline the algorithm

in four simple steps:
1. Initialize the algorithm with k = d , where d is the dimensionality of
the full feature space Fd .
2. Determine the feature bf such that
bf = arg max C (Fk − f ).
f ∈Fk

3. Remove the feature bf from the feature set:

Fk −1 = Fk − bf ; k = k − 1;

4. Terminate if k equals the number of desired features; otherwise, go

to step 2.
6.5. Feature Importance 153

6.5. Feature Importance

The concept of feature importance is straightforward: it is the increase
in the model’s prediction error after we permuted the feature’s values,
which breaks the relationship between the feature and the true outcome.
• A feature is “important" if shuffling its values increases the model
error, because in this case the model relied on the feature for the
prediction.
• A feature is “unimportant" if shuffling its values leaves the model
error unchanged, because in this case the model ignored the feature
for the prediction.

The permutation feature importance measurement was introduced by

Breiman (2001) [8] for random forests. Based on this idea, Fisher, Rudin,
and Dominici (2018) [15] proposed a model-agnostic version of the feature
importance and called it model reliance.

Algorithm 6.9. Permutation feature importance (FI).

input: Trained model f , feature matrix X , target vector y ,
error measure L(f , X , y);
1. Estimate the original model error εorig = L(f , X , y);
2. For each feature j = 1, 2, · · · , d ; do:

Permute feature j in the data X to get X (j) ;
 Estimate error ε(j) = L(f , X (j) , y);

Calculate permutation FI: FI(j) = ε(j) /εorig (or, ε(j) − εorig );
3. Sort features by descending FI;
154 Chapter 6. Data Preprocessing in Machine Learning

Should we compute FI on training or test data?

To answer the question, you need to decide whether

• you want to know how much the model relies on each feature for
making predictions (→ training data) or
• how much the feature contributes to the performance of the model
on unseen data (→ test data).

There is no research addressing the question of training vs. test data; more
research and more experience are needed to gain a better understanding.
6.5. Feature Importance 155

Exercises for Chapter 6

First, read pp. 135-143, Python Machine Learning, 3rd Ed..

6.1. On pp. 135-143, the sequential backward selection (SBS) is implemented as a
feature selection method and experimented with a k -NN classifier (n_neighbors=5),
using the wine dataset.

(a) Perform the same experiment with the k -NN classifier replaced by the support
vector machine (soft-margin SVM classification).
(b) In particular, analyze accuracy of the soft-margin SVM and plot the result
as in the figure on p. 139.
6.2. On pp. 141-143, the permutation feature importance is assessed from the ran-
dom forest classifier, using the wine dataset.

(a) Discuss whether or not you can derive feature importance for a k -NN classifier.
(b) Assess feature importance with the logistic regression classifier, using the
same dataset.
(c) Based on the computed feature importance, analyze and plot accuracy of the
logistic regression classifier for k_features = 1, 2, · · · , 13.
156 Chapter 6. Data Preprocessing in Machine Learning
Chapter 7

Feature Extraction: Data Compression

There are two main categories of dimensionality reduction techniques: feature selec-
tion and feature extraction. Via feature selection, we select a subset of the original
features, as we have considered in Chapter 6; whereas in feature extraction, we derive
information from the feature set to construct a new feature subspace.
In this chapter, we will study three fundamental techniques for dimensionality reduc-
tion:
• Principal component analysis (PCA), for unsupervised data compression
• Linear discriminant analysis (LDA), as a supervised dimensionality reduction
technique for maximizing class separability
• Kernel principal component analysis (KPCA), for nonlinear dimensionality re-
duction
In the context of dimensionality reduction, feature extraction can be understood as an
approach to data compression with the goal of maintaining most of the relevant infor-
mation. In practice, feature extraction is not only used to improve storage space or the
computational efficiency of the learning algorithm, but can also improve the predictive per-
formance by reducing the curse of dimensionality – especially when we are working with
non-regularized models.

Contents of Chapter 7
7.1. Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
7.2. Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
7.3. Linear Discriminant Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
7.4. Kernel Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Exercises for Chapter 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204

157
158 Chapter 7. Feature Extraction: Data Compression

7.1. Principal Component Analysis

• Principal component analysis (PCA) (a.k.a. orthogonal linear
transformation) was invented in 1901 by K. Pearson [50], as an ana-
logue of the principal axis theorem in mechanics; it was later indepen-
dently developed and named by H. Hotelling in the 1930s [27, 28].
• The PCA is a statistical procedure that uses an orthogonal transfor-
mation to convert a set of observations of possibly correlated variables
into a set of linearly uncorrelated variables called principal compo-
nents.
• The orthogonal axes of the new subspace can be interpreted as the
directions of maximum variance given the constraint that the new
feature axes are orthogonal to each other:

Figure 7.1: Principal components.

• As an unsuperviseda linear transformation technique, the PCA is

widely used across various fields – in ML, most prominently for fea-
ture extraction and dimensionality reduction.
• PCA identifies patterns in data based on the correlation between
features.
• The PCA directions are highly sensitive to data scaling, and we
need to standardize the features prior to PCA if the features were
measured on different scales and we want to assign equal impor-
tance to all features.b
a
The PCA is a unsupervised technique, because it does not use any class label information.
b
See § 6.3. Feature Scaling, on p. 146.
7.1. Principal Component Analysis 159

7.1.1. Computation of principal components

• Consider a data matrix X ∈ RN×d :

– each of the N rows represents a different data point,
– each of the d columns gives a particular kind of feature, and
– each column has zero empirical mean (e.g., after standardization).
• Our goal is to find an orthogonal weight matrix W ∈ Rd ×d such that

Z = X W, (7.1)

where Z ∈ RN×d is call the score matrix. Columns of Z represent

principal components of X .

First weight vector w 1 : the first column of W :

In order to maximize variance of z 1 , the first weight vector w 1 should satisfy
w 1 = arg max kz 1 k2 = arg max kX w k2
kw k=1 kw k=1
wT XT Xw (7.2)
= arg max w T X T X w = arg max ,
kw k=1 w 6=0 wT w

where the quantity to be maximized can be recognized as a Rayleigh quo-

tient.

Theorem 7.1. For a positive semidefinite matrix (such as X T X ), the

maximum of the Rayleigh quotient is the same as the largest eigenvalue
of the matrix, which occurs when w is the corresponding eigenvector, i.e.,

wT XT Xw v1
w 1 = arg max = , (X T X )v 1 = λ1 v 1 , (7.3)
w 6=0 wT w kv 1 k
where λ1 is the largest eigenvalue of X T X ∈ Rd ×d .

Example 7.2. With w 1 found, the first principal component of a data

vector x (i) can then be given as a score z1(i) = x (i) · w 1 .
160 Chapter 7. Feature Extraction: Data Compression

Further weight vectors w k :

The k -th weight vector can be found by 1 subtracting the first (k − 1) prin-
cipal components from X :
k −1
X
Xk := X −
b X w i w Ti , (7.4)
i=1

and then 2 finding the weight vector which extracts the maximum variance
from this new data matrix
w k = arg max kX bk w k2 ,
kw k=1
(7.5)

which turns out to give the remaining (normalized) eigenvectors of X T X .

Remark 7.3. Using singular value decomposition, we can see that

the transformation matrix W is the stack of eigenvectors of X T X , i.e.,

W = [w 1 |w 2 | · · · |w d ], (X T X ) w j = λj w j , w Ti w j = δij , (7.6)

where λ1 ≥ λ2 ≥ · · · ≥ λd ≥ 0.

• With W found, a data vector x (new or old) is transformed to a

d -dimensional row vector of principal components

z = xW , (7.7)

of which components zj , j = 1, 2, · · · , d , are decorrelated.

• For the data vector x T considered as a column vector, we have

zT = W T xT , (7.8)

where W T is called the whitening transformation or sphering

transformation.

Remark 7.4. While the weight matrix W ∈ Rd ×d is the collection of

eigenvectors of X T X , the score matrix Z ∈ RN×d is the stack of eigenvec-
tors of XX T , scaled by the square-root of eigenvalues:
p p p
Z = [ λ1 u1 | λ2 u2 | · · · | λd ud ], (XX T ) uj = λj uj , uTi uj = δij . (7.9)

See (7.15) and § 7.2.

7.1. Principal Component Analysis 161

7.1.2. Dimensionality reduction

The transformation Z = XW maps a data vector x (i) ∈ Rd to a new space
of d variables which are now uncorrelated. However, not all the principal
components need to be kept.
Keeping only the first k principal components, produced by using only the
first k eigenvectors of X T X (k  d ), gives the truncated transformation:
Zk = X Wk : x (i) ∈ Rd 7→ z (i) ∈ Rk , (7.10)
where Zk ∈ RN×k and Wk ∈ Rd ×k . Let
Xk := Zk WkT . (7.11)

Quesitons. How can we choose k ? &

Is the difference kX − Xk k (that we truncated) small ?

Remark 7.5. The principal components transformation can also be as-

sociated with the singular value decomposition (SVD) of X :
X = U ΣV T , (7.12)

where
U : n × d orthogonal (the left singular vectors of X .)
Σ : d × d diagonal (the singular values of X .)
V : d × d orthogonal (the right singular vectors of X .)

• The matrix Σ explicitly reads

Σ = diag(σ1 , σ2 , · · · , σd ), (7.13)

where σ1 ≥ σ2 ≥ · · · ≥ σd ≥ 0.
• In terms of this factorization, the matrix X T X reads

X T X = (U ΣV T )T U ΣV T = V ΣU T U ΣV T = VΣ2 V T . (7.14)

• Comparing with the eigenvector factorization of X T X , we have

– the right singular vectors V ∼ = the eigenvectors of X T X ⇒ V ∼
= W
– the square of singular values of X are equal to the eigenvalues of X T X
⇒ σj2 = λj , j = 1, 2, · · · , d .
162 Chapter 7. Feature Extraction: Data Compression

Further considerations for the SVD

• Using the SVD, the score matrix Z reads

Z = XW = U ΣV T W = U Σ, (7.15)

and therefore each column of Z is given by one of the left singular vectors
of X multiplied by the corresponding singular value. This form is also the
polar decomposition of Z . See (7.9) on p. 160.
• As with the eigen-decomposition, the SVD, the truncated score matrix
Zk ∈ RN ×k can be obtained by considering only the first k largest singular
values and their singular vectors:
Zk = XWk = U ΣV T Wk = U Σk , (7.16)

where
Σk := diag(σ1 , · · · , σk , 0, · · · , 0). (7.17)

• Now, using (7.16), the truncated data matrix reads

Xk = Zk WkT = U Σk WkT = U Σk W T = U Σk V T . (7.18)

Claim 7.6. It follows from (7.12) and (7.18) that

kX − Xk k2 = kU ΣV T − U Σk V T k2
= kU(Σ − Σk )V T k2 (7.19)
= kΣ − Σk k2 = σk +1 ,

where k · k2 is the induced matrix L 2 -norm.

Remark 7.7. Efficient algorithms exist to calculate the SVD of X with-

out having to form the matrix X T X , so computing the SVD is now the
standard way to carry out the PCA. See [22, 67].
7.1. Principal Component Analysis 163

7.1.3. Explained variance

Note: Since we want to reduce the dimensionality of our dataset by com-

pressing it onto a new feature subspace, we only select the subset of the
eigenvectors (principal components) that contains most of the informa-
tion (variance). The eigenvalues define the magnitude of the eigenvec-
tors, so we have to sort the eigenvalues by decreasing magnitude; we are
interested in the top k eigenvectors based on the values of their corre-
sponding eigenvalues.

Definition 7.8. Let λj be eigenvalues of X T X : (X T X )v j = λj v j . Define the

explained variance ratio of the eigenvalues as
λi
evr(λi ) = Pd , i = 1, 2, · · · , d, (7.20)
λ
j=1 j

and cumulative explained variance as

k
X k
X d
.X
cev(λk ) = evr(λi ) = λi λj , k = 1, 2, · · · , d. (7.21)
i=1 i=1 j=1

Then, we may choose k satisfying

cev(λk −1 ) < ε and cev(λk ) ≥ ε, (7.22)

for a tolerance ε.

Figure 7.2: evr and cev for the wine dataset.

164 Chapter 7. Feature Extraction: Data Compression

7.2. Singular Value Decomposition

Here we will deal with the SVD in detail.

Theorem 7.9. (SVD Theorem). Let A ∈ Rm×n with m ≥ n. Then we

can write
A = U Σ VT, (7.23)
where U ∈ Rm×n and satisfies U T U = I, V ∈ Rn×n and satisfies V T V = I,
and Σ = diag(σ1 , σ2 , · · · , σn ), where

σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0.

Remark 7.10. The matrices are illustrated pictorially as

   
  
   
 A  =  U   Σ   VT  ,
   
    (7.24)
   

where
U : m × n orthogonal (the left singular vectors of A .)
Σ : n × n diagonal (the singular values of A .)
V : n × n orthogonal (the right singular vectors of A .)

• For some r ≤ n, the singular values may satisfy

σ ≥ σ2 ≥ · · · ≥ σr > σr+1 = · · · = σn = 0. (7.25)

|1 {z }
nonzero singular values

In this case, rank(A ) = r .

• If m < n, the SVD is defined by considering A T .
7.2. Singular Value Decomposition 165

Proof. (of Theorem 7.9) Use induction on m and n: we assume that the
SVD exists for (m − 1) × (n − 1) matrices, and prove it for m × n. We assume
6 0; otherwise we can take Σ = 0 and let U and V be arbitrary orthogonal
A =
matrices.

• The basic step occurs when n = 1 (m ≥ n). We let A = UΣV T with

U = A /||A ||2 , Σ = ||A ||2 , V = 1.
• For the induction step, choose v so that

||v||2 = 1 and ||A ||2 = ||A v||2 > 0.

Av
• Let u = ||A v||2 , which is a unit vector. Choose Ũ , Ṽ such that

U = [u Ũ] ∈ Rm×n and V = [v Ṽ] ∈ Rn×n

are orthogonal.
• Now, we write
" #
uT uT A v uT A Ṽ
 
T
U AV = · A · [v Ṽ ] =
Ũ T Ũ T A v Ũ T A Ṽ

Since

T (A v)T (A v) ||A v||22

u Av = = = ||A v||2 = ||A ||2 ≡ σ ,
||A v||2 ||A v||2
Ũ T A v = Ũ T u||A v||2 = 0,

we have
     T
T σ 0 1 0 σ 0 1 0
U AV = = ,
0 U1 Σ1 V1T 0 U1 0 Σ1 0 V1

or equivalently
      T
1 0 σ 0 1 0
A= U V . (7.26)
0 U1 0 Σ1 0 V1

Equation (7.26) is our desired decomposition.

166 Chapter 7. Feature Extraction: Data Compression

7.2.1. Interpretation of the SVD

Algebraic interpretation of the SVD
Let rank(A ) = r . let the SVD of A be A = U Σ V T , with
U = [u1 u2 · · · un ],
Σ = diag(σ1 , σ2 , · · · , σn ),
V = [v1 v2 · · · vn ],

and σr be the smallest positive singular value. Since

A = U Σ V T ⇐⇒ AV = UΣV T V = UΣ,

we have
AV = A [v1 v2 ··· vn ] = [A v1 A v2 ··· A vn ]
 
σ1
 ... 
 
= [u1 ··· ur ··· un ] 

σr

 (7.27)

 ... 

0
= [σ1 u 1 ··· σr ur 0 ··· 0].

Therefore,

A vj = σj uj , j = 1, 2, · · · , r
A = U Σ VT ⇔ (7.28)
A vj = 0, j = r + 1, · · · , n

Similarly, starting from A T = V Σ U T ,

 T
A uj = σj vj , j = 1, 2, · · · , r
A = V ΣU ⇔
T T
(7.29)
A T uj = 0, j = r + 1, · · · , n
7.2. Singular Value Decomposition 167

Summary 7.11. It follows from (7.28) and (7.29) that

• (v j , σj2 ), j = 1, 2, · · · , r , are eigenvector-eigenvalue pairs of A T A .

A T A vj = A T (σj uj ) = σj2 vj , j = 1, 2, · · · , r. (7.30)

So, the singular values play the role of eigenvalues.

• Similarly, we have

AA T uj = A (σj v j ) = σj2 uj , j = 1, 2, · · · , r. (7.31)

• Equation (7.30) gives how to find the singular values {σj } and the
right singular vectors V , while (7.28) shows a way to compute the
left singular vectors U .
• (Dyadic decomposition) The matrix A ∈ Rm×n can be expressed as
n
X
A= σj uj v Tj . (7.32)
j=1

When rank(A ) = r ≤ n,
r
X
A= σj uj v Tj . (7.33)
j=1

This property has been utilized for various approximations and ap-
plications, e.g., by dropping singular vectors corresponding to small
singular values.
168 Chapter 7. Feature Extraction: Data Compression

Geometric interpretation of the SVD

The matrix A maps an orthonormal basis

B1 = {v1 , v2 , · · · , vn }

of Rn onto a new “scaled” orthogonal basis

B2 = {σ1 u1 , σ2 u2 , · · · , σr ur }

for a subspace of Rm :
A
B1 = {v1 , v2 , · · · , vn } −
→ B2 = {σ1 u1 , σ2 u2 , · · · , σr ur } (7.34)

Consider a unit sphere S n−1 in Rn :

n n
X o
n −1
S = x xj2 =1 .
j=1

Then, ∀x ∈ S n−1 ,
x = x1 v1 + x2 v2 + · · · + xn vn
A x = σ1 x1 u1 + σ2 x2 u2 + · · · + σr xr ur (7.35)
= y1 u1 + y2 u2 + · · · + yr ur , (yj = σj xj )

So, we have
yj
yj = σj xj ⇐⇒ xj =
σj
n r
X yj2 (7.36)
xj2
P
= 1 (sphere) ⇐⇒ = α ≤ 1 (ellipsoid)
j=1 j=1
σj2
7.2. Singular Value Decomposition 169

Example 7.12. We build the set A (S n−1 ) by multiplying one factor of A =

UΣV T at a time. Assume for simplicity that A ∈ R2×2 and nonsingular. Let
 
3 −2
A = = UΣV T
−1 2
   
−0.8649 0.5019 4.1306 0 −0.7497 0.6618
=
0.5019 0.8649 0 0.9684 0.6618 0.7497

Then, for x ∈ S 1 ,
A x = UΣV T x = U Σ(V T x)

In general,

• V T : S n−1 → S n−1 (rotation in Rn )

• Σ : e j 7→ σj e j (scaling from S n−1 to Rn )
• U : Rn → Rm (rotation)
170 Chapter 7. Feature Extraction: Data Compression

7.2.2. Properties of the SVD

Theorem 7.13. Let A ∈ Rm×n with m ≤ n. Let A = UΣV T be the SVD of

A , with
σ1 ≥ · · · ≥ σr > σr+1 = · · · = σn = 0.
Then, 

 rank(A ) = r

 Null(A )


 = Span{vr+1 , · · · , vn }
Range(A ) = Span{u1 , · · · , ur } (7.37)

 r
X

A = σi ui vTi



i=1

and 

 ||A ||2 = σ1 (See Exercise 2.)




 ||A ||2F = σ12 + · · · + σr2 (See Exercise 3.)
||A x||2


min = σn (m ≥ n) (7.38)
x6=0 ||x||2
σ1


κ2 (A ) = ||A ||2 · ||A −1 ||2 =





 σn
( when m = n, & ∃A −1 )


Theorem 7.14. Let A ∈ Rm×n , m ≥ n, rank(A ) = n, with singular values

σ1 ≥ σ2 ≥ · · · σn > 0.

Then
||(A T A )−1 ||2 = σn−2 ,
||(A T A )−1 A T ||2 = σn−1 ,
(7.39)
||A (A T A )−1 ||2 = σn−1 ,
||A (A T A )−1 A T ||2 = 1.

Definition 7.15. (A T A )−1 A T is called the pseudoinverse of A , while

A (A T A )−1 is called the pseudoinverse of A T . Let A = UΣV T be the SVD
of A . Then def
(A T A )−1 A T = V Σ−1 U T == A + . (7.40)
7.2. Singular Value Decomposition 171

Theorem 7.16. Let A ∈ Rm×n with rank(A ) = r > 0. Let A = UΣV T be

the SVD of A , with singular values

σ1 ≥ · · · ≥ σr > 0.

Define, for k = 1, · · · , r − 1,
k
X
Ak = σj uj v Tj (sum of rank-1 matrices).
j=1

Then, rank(Ak ) = k and

||A − Ak ||2 = min{||A − B ||2 | rank(B) ≤ k }
= σk +1 ,
(7.41)
||A − Ak ||F = min{||A − B ||2F | rank(B) ≤ k }
2

= σk2+1 + · · · + σr2 .

That is, of all matrices of rank ≤ k , Ak is closest to A .

Note: The matrix Ak can be written as

Ak = U Σk V T , (7.42)

where Σk = diag(σ1 , · · · , σk , 0, · · · , 0).

Corollary 7.17. Suppose A ∈ Rm×n has full rank; rank(A ) = n. Let

σ1 ≥ · · · ≥ σn be the singular values of A . Let B ∈ Rm×n satisfy

||A − B ||2 < σn .

Then B also has full rank.

172 Chapter 7. Feature Extraction: Data Compression

Full SVD

• For A ∈ Rm×n ,
A = UΣV T ⇐⇒ U T AV = Σ,
where U ∈ Rm×n and Σ, V ∈ Rn×n .
• Expand
U → Ũ = "[U U#2 ] ∈ Rm×m , (orthogonal)
Σ
Σ → Σ̃ = ∈ Rm×n ,
O
where O is an (m − n) × n zero matrix.
• Then, " #
Σ
Ũ Σ̃V T = [U U2 ] V T = UΣV T = A (7.43)
O
7.2. Singular Value Decomposition 173

7.2.3. Computation of the SVD

For A ∈ Rm×n , the procedure is as follows.

1. Form A T A (A T A – covariance matrix of A ).
2. Find the eigen-decomposition of A T A by orthogonalization process,
i.e., Λ = diag(λ1 , · · · , λn ),

A T A = V ΛV T ,

where V = [v1 ··· vn ] is orthogonal, i.e., V T V = I.

3. Sort the eigenvalues according to their magnitude and let

p
σj = λj , j = 1, 2, · · · , n.

4. Form the U matrix as follows,

1
uj = A vj , j = 1, 2, · · · , r.
σj
If necessary, pick up the remaining columns of U so it is orthogonal.
(These additional columns must be in Null(AA T ).)
 T 
v1
5. A = UΣV T = [u1 · · · ur · · · un ] diag(σ1 , · · · , σr , 0, · · · , 0)  .. 
.


vTn

Lemma 7.18. Let A ∈ Rn×n be symmetric. Then (a) all the eigenvalues
of A are real and (b) eigenvectors corresponding to distinct eigenvalues
are orthogonal.

Proof. See Exercise 4.

174 Chapter 7. Feature Extraction: Data Compression
 
1 2
Example 7.19. Find the SVD for A =  −2 1 .
3 2
Solution.
 
T 14 6
1. A A = .
6 9

2. Solving det(A T A − λI) = 0 gives the eigenvalues of A T A

λ1 = 18 and λ2 = 5,

of which corresponding eigenvectors are

√3 − √213
" # " # " #
3 −2 13
ṽ1 = , ṽ2 = . =⇒ V =
2 3 √2 √3
13 13

√ √ √ √ √
3. σ1 = λ1 = 18 = 3 2, σ2 = λ2 = 5. So
√ 
18 √0
Σ=
0 5

√7
   
" # 7
√3 234
1 4
4. u1 = √1 A 13 √1 √1  −4  =  − 234
√
σ1 A v1 = =
   
18 √2 18 13

13
13 √13
234
   
" # 4 √4
−2
√ 65
1 √1 A 13 √1 √1 √7
u2 = σ2 A v2 = = 7= .
   
5 √3 5 13 65
13
0 0
 
√7 √4
234 65 √ " √3 √2
#
 4 7
 18 √0 13 13
5. A = UΣV T =  − √234 √
65

2 3
  0 5 − √13 √
√13 0 13
234
7.2. Singular Value Decomposition 175

Example 7.20. Find the pseudoinverse of A ,

A + = (A T A )−1 A T = VΣ−1 U T ,
 
1 2
when A = −2 1 .

3 2
Solution. From Example 7.19, we have
 
√7 √4 √ " #
234 65
18 √0 √3 √2
4 √7 
A = UΣV T =  − √234 13 13

65  0 5 − √213 √3
√13 0 13
234

Thus,
" #" #" #
√3 − √213 √1 0 √7 4
− √234 √13
A + = V Σ−1 U T = 13 18 234 234
√2 √3 0 √1 √4 √7 0
13 13 5 65 65
1 4 1
 
− 30 − 15 6
= 11 13 1
45 45 9
176 Chapter 7. Feature Extraction: Data Compression

Computer implementation [21]

Figure 7.3: A two-phase procedure for the SVD: A = UΣV T .

Algorithm 7.21. [Golub and Reinsch, 1970]a Let A ∈ Rm×n .

• Phase 1: It constructs two finite sequences of Householder
transformations to find an upper bidiagonal matrix:

Pn · · · P1 A Q1 · · · Qn−2 = B (7.44)

• Phase 2: It is to iteratively diagonalize B using the QR method.

a
G. Golub and C. Reinsch. Singular value decomposition and least squares solutions. Numer. Math.,
14 (1970), pp. 403–420.

Golub-Reinsch SVD algorithm

• It is extremely stable.
• Computational complexity:
– Computation of U , V , and Σ: 4m2 n + 8mn2 + 9n3 .
– Computation of V and Σ: 4mn2 + 8n3 .
• Phases 1 & 2 take O(mn2 ) and O(n2 ) flops, respectively.
(when Phase 2 is done with O(n) iterations)

• Python: U,S,V = numpy.linalg.svd(A)

• Matlab/Maple: [U,S,V] = svd(A)
• Mathematica: {U,S,V} = SingularValueDecomposition[A]
7.2. Singular Value Decomposition 177

Numerical rank
In the absence of round-off errors and uncertainties in the data, the SVD
reveals the rank of the matrix. Unfortunately the presence of errors makes
rank determination problematic. For example, consider
 
1/3 1/3 2/3
2/3 2/3 4/3
 
A = 1 / 3 2/ 3 3 /3 
 (7.45)
2/5 2/5 4/5
3/5 1/5 4/5
• Obviously A is of rank 2, as its third column is the sum of the first two.
• Matlab “svd" (with IEEE double precision) produces
σ1 = 2.5987, σ2 = 0.3682, and σ3 = 8.6614 × 10−17 .

• What is the rank of A , 2 or 3? What if σ3 is in O(10−13 )?

• For this reason we must introduce a threshold T . Then we say that A
has numerical rank r if A has r singular values larger than T , that is,
σ1 ≥ σ2 ≥ · · · ≥ σr > T ≥ σr+1 ≥ · · · (7.46)

In Matlab
• Matlab has a “rank" command, which computes the numerical rank
of the matrix with a default threshold
T = 2 max{m, n}  ||A ||2 (7.47)

where  is the unit round-off error.

• In Matlab, the unit round-off error can be found from the parameter
“eps"
eps = 2−52 = 2.2204 × 10−16 .
• For the matrix A in (7.45),
T = 2 · 5 · eps · 2.5987 = 5.7702 × 10−15

and therefore rank(A)=2.

See Exercise 5.
178 Chapter 7. Feature Extraction: Data Compression

7.2.4. Application of the SVD to image compression

• A ∈ Rm×n is a sum of rank-1 matrices (dyadic decomposition):

V = [v1 , · · · , vn ], U = [u1 , · · · , un ],
n
T
X (7.48)
A = UΣV = σi ui vTi , ui ∈ Rm , v i ∈ Rn .
i=1

• The approximation k
X
T
Ak = UΣk V = σi ui vTi (7.49)
i=1

is closest to A among matrices of rank≤ k , and

||A − Ak ||2 = σk +1 . (7.50)

• It only takes (m + n) · k words to store u1 through uk , and σ1 v1 through

σk vk , from which we can reconstruct Ak .

Image compression using k singular values

peppers_SVD.m
1 img = imread('Peppers.png'); [m,n,d]=size(img);
2 [U,S,V] = svd(reshape(im2double(img),m,[]));
3 %%---- select k <= p=min(m,n)
4 k = 20;
5 img_k = U(:,1:k)*S(1:k,1:k)*V(:,1:k)';
6 img_k = reshape(img_k,m,n,d);
7 figure, imshow(img_k)

The “Peppers" image is in [270, 270, 3] ∈ R270×810 .

7.2. Singular Value Decomposition 179

Original (k = 270) k =1 k = 10

k = 20 k = 50 k = 100

Peppers: Singular values

Peppers: Storage: It requires (m + n) · k words.For example, when k = 50,

(m + n) · k = (270 + 810) · 50 = 54,000 , (7.51)

which is approximately a quarter the full storage space

270 × 270 × 3 = 218,700 .

180 Chapter 7. Feature Extraction: Data Compression

7.3. Linear Discriminant Analysis

Linear discriminant analysis is a method to find a linear combination
of features that characterizes or separates two or more classes of objects
or events.
• The LDA is sometimes also called Fisher’s LDA. Fisher initially for-
mulated the LDA for two-class classification problems in 1936 [16], and
later generalized for multi-class problems by C. Radhakrishna Rao under
the assumption of equal class covariances and normally distributed
classes in 1948 [52].
• The LDA may be used as a linear classifier, or, more commonly, for
dimensionality reduction (§ 7.3.4) for a later classification.
• The general concept behind the LDA is very similar to PCA.1

LDA objective
• The LDA objective is to perform dimensionality reduction.
– So what? PCA does that, too!
• However, we want to preserve as much of the class discriminatory
information as possible.
– OK, this is new!
LDA
• Consider a pattern classification problem, where we have c classes.
• Suppose each class has Nk samples in Rd , where k = 1, 2, · · · , c .
• Let Xk = {x (1) , x (2) , · · · , x (Nk ) } be the set of d -dimensional samples for
class k .
• Let X ∈ Rd ×N be the data matrix, stacking all the samples from P all
classes, such that each column represents a sample, where N = k Nk .
• The LDA seeks to obtain a transformation of X to Z through projecting
the samples in X onto a hyperplane with dimension c − 1.

1
In PCA, the main idea is to re-express the available dataset to extract the relevant information by re-
ducing the redundancy and to minimize the noise. While (unsupervised) PCA attempts to find the
orthogonal component axes of maximum variance in a dataset, the goal in the (supervised) LDA is to find
the feature subspace that optimizes class separability.
7.3. Linear Discriminant Analysis 181

7.3.1. Fisher’s LDA (classifier): two classes

Let us define a transformation of samples x onto a line [(c − 1)-space, for
c = 2]:
z = w T x = w · x, (7.52)
where w ∈ Rd is a projection vector.
Of all the possible lines, we would like to select the one that maximizes the
separability of the scalars {z }.
• In order to find a good projection vector, we need to define a measure
of separation between the projections.
• The mean vector of each class in x and z feature space is
1 X 1 X 1 X T
µk = x, µek = z= w x = w T µk , (7.53)
Nk Nk Nk
x ∈Xk x ∈Xk x ∈Xk

i.e., projecting x to z will lead to projecting the mean of x to the mean

of z .
• We could then choose the distance between the projected means
as our objective function:

Jb(w) = |e e2 | = |w T (µ1 − µ2 )|.

µ1 − µ (7.54)

the projected means is not a

very good measure, since it
does not take into account the
sample distribution within the
classes.
• The maximizer w ∗ of (7.54) must
be parallel to (µ1 − µ2 ):
Figure 7.4: The x1 -axis has a larger dis- w ∗ // (µ1 − µ2 );
tance between means, while the x2 -axis
yields a better class separability. the projection to a parallel line of
(µ1 − µ2 ) is not an optimal trans-
• However, the distance between formation.
182 Chapter 7. Feature Extraction: Data Compression

Fisher’s LDA
The solution proposed by Fisher is to maximize a function that repre-
sents the difference between the means, normalized by a measure of the
within-class variability, or the so-called scatter.

• For each class k , we define the scatter (an equivalent of the variance)
as 2
X
sk =
e ek )2 , z = w T x.
(z − µ (7.55)
x ∈Xk

• The quantity e sk2 measures the variability within class Xk after project-
ing it on the z -axis.
• Thus, e s22 measures the variability within the two classes at hand
s12 + e
after projection; it is called the within-class scatter of the projected
samples.
• Fisher’s linear discriminant is defined as the linear function w T x that
maximizes the objective function:

∗ e1 − µ
(µ e2 )2
w = arg max J (w), where J (w) = . (7.56)
w s12 + e
e s22

• Therefore, Fisher’s LDA searches for a projection where samples from

the same class are projected very close to each other; at the same time,
the projected means are as farther apart as possible.

Figure 7.5: Fisher’s LDA.

7.3. Linear Discriminant Analysis 183

7.3.2. Fisher’s LDA: the optimum projection

Rewrite the Fisher’s objective function:
e1 − µ
(µ e2 )2
J (w) = , (7.57)
s12 + e
e s22

where 1 X X
µk = x, ek = w T µk ,
µ sk2 =
e ek )2 .
(z − µ
Nk
x ∈Xk x ∈Xk

• In order to express J (w) as an explicit function of w , we first define a

measure of the scatter in the feature space x :
X
Sw = S1 + S2 , for Sk = (x − µk )(x − µk )T , (7.58)
x ∈Xk

where Sw ∈ Rd ×d is called the within-class scatter matrix of samples

x , while Sk is the covariance matrix of class Xk .
Then, the scatter of the projection z can then be expressed as
X X
sk2 =
e ek )2 =
(z − µ (w T x − w T µk )2
xX∈Xk x ∈Xk
= w (x − µk )(x − µk )T w
T
(7.59)
x ∈Xk 
X
T
= w T
(x − µk )(x − µk ) w = w T Sk w.
x ∈Xk

Thus, the denominator of the objective function gives

s12 + e
e s22 = w T S1 w + w T S2 w = w T Sw w =: S
ew , (7.60)

where S
ew is the within-class scatter of projected samples z .
• Similarly, the difference between the projected means (in z -space) can be
expressed in terms of the means in the original feature space (x -space).

e2 )2 = (w T µ1 − w T µ2 )2 = w T (µ1 − µ2 )(µ1 − µ2 )T w
e1 − µ
(µ
(7.61)
| {z }
=:Sb
T
= w Sb w =: S
eb ,
184 Chapter 7. Feature Extraction: Data Compression

where the rank-one matrix Sb ∈ Rd ×d is called the between-class scat-

ter matrix of the original samples x , while S eb is the between-class
scatter of the projected samples z .
• Since Sb is the outer product of two vectors, rank(Sb ) ≤ 1.
We can finally express the Fisher criterion in terms of Sw and Sb as
e1 − µ
(µ e2 )2 w T Sb w
J (w) = = . (7.62)
s12 + e
e s22 w T Sw w

Hence, J (w) is a measure of the difference between class means (en-

coded in the between-class scatter matrix), normalized by a measure
of the within-class scatter matrix.

• To find the maximum of J (w), we differentiate it with respect to w and

equate to zero. Applying some algebra leads (Exercise 6)

Sw−1 Sb w = J (w) w. (7.63)

Note that Sw−1 Sb is a rank-one matrix.

Equation (7.63) is a generalized eigenvalue problem:

Sw−1 Sb w = λ w ⇐⇒ Sb w = λ Sw w; (7.64)

the maximizer w ∗ of J (w) is the eigenvector associated with the nonzero

eigenvalue λ∗ = J (w).

Summary 7.22. Finding the eigenvector of Sw−1 Sb associated with the

largest eigenvalue yields
∗ w T Sb w
w = arg max J (w) = arg max T . (7.65)
w w w Sw w

This is known as Fisher’s Linear Discriminant, although it is not a discrim-

inant but a specific choice of direction for the projection of the data down to
one dimension.
7.3. Linear Discriminant Analysis 185

Fisher’s LDA: an example

We will compute the Linear Discriminant projection for the following two-
dimensional dataset of two classes (c = 2).
lda_Fisher.m
1 m=2; n=5;
2

3 X1=[2,3; 4,3; 2,1; 3,4; 5,4];

4 X2=[7,4; 6,8; 7,6; 8,9; 10,9];
5

6 Mu1 = mean(X1)'; % Mu1 = [3.2,3.0]

7 Mu2 = mean(X2)'; % Mu2 = [7.6,7.2]
8

9 S1 = cov(X1,0)*n;
10 S2 = cov(X2,0)*n;
11 Sw = S1+S2; % Sw = [20,13; 13,31]
12

13 Sb = (Mu1-Mu2)*(Mu1-Mu2)'; % Sb = [19.36,18.48; 18.48,17.64]

14

15 invSw_Sb = inv(Sw)*Sb;
16 [V,L] = eig(invSw_Sb); % V1 = [ 0.9503,0.3113]; L1 = 1.0476
17 % V2 = [-0.6905,0.7234]; L2 = 0.0000

Figure 7.6: A synthetic dataset and Fisher’s LDA projection.

186 Chapter 7. Feature Extraction: Data Compression

7.3.3. LDA for multiple classes

• Now, we have c -classes instead of just two.
• We are now seeking (c − 1) projections [z1 , z2 , · · · , zc −1 ] by means of (c − 1)
projection vectors w k ∈ Rd .
• Let W = [w 1 |w 2 | · · · |w c −1 ], a collection of column vectors, such that

zk = w Tk x =⇒ z = W T x ∈ Rc −1 . (7.66)

• If we have N sample (column) vectors, we can stack them into one ma-
trix as follows.
Z = WTX, (7.67)
where X ∈ Rd ×N , W ∈ Rd ×(c −1) , and Z ∈ R(c −1)×N .

Recall: For the two classes case, the within-class scatter was com-
puted as
Sw = S1 + S2 .

This can be generalized in the c -classes case as:

c
X X
Sw = Sk , Sk = (x − µk )(x − µk )T , (7.68)
k =1 x ∈Xk

where µk = N1k x ∈Xk x , where Nk is the number of data samples in class

P

Xk , and Sw ∈ Rd ×d .

Recall: For the two classes case, the between-class scatter was com-
puted as
Sb = (µ1 − µ2 )(µ1 − µ2 )T .

For c -classes case, we will measure the between-class scatter with

respect to the mean of all classes as follows:
c
X
T 1X
Sb = Nk (µk − µ)(µk − µ) , µ= x, (7.69)
N
k =1 ∀x

where rank(Sb ) = c − 1.
7.3. Linear Discriminant Analysis 187

Definition 7.23. As an analogue to (7.65), we may define the LDA

optimization, for c classes case, as follows.

∗ W T Sb W
W = arg max J (W ) = arg max T . (7.70)
W W W Sw W

Recall: For two-classes case, when we set ∂J∂ w(w) = 0, the optimization
problem is reduced to the eigenvalue problem

Sw−1 Sb w ∗ = λ∗ w ∗ , where λ∗ = J (w ∗ ).

For c -classes case, we have (c − 1) projection vectors. Hence the eigenvalue

problem can be generalized to the c -classes case:

Sw−1 Sb w ∗k = λ∗k w ∗k , λ∗k = J (w ∗k ), k = 1, 2, · · · , c − 1. (7.71)

Thus, it can be shown that the optimal projection matrix

W ∗ = [w ∗1 |w ∗1 | · · · |w ∗c −1 ] ∈ Rd ×(c −1) (7.72)

is the one whose columns are the eigenvectors corresponding to the

largest eigenvalues of the following generalized eigenvalue problem:

Sw−1 Sb W ∗ = λ W ∗ , λ = J (W ∗ ), (7.73)
where Sw−1 Sb ∈ Rd ×d .
188 Chapter 7. Feature Extraction: Data Compression

Illustration – 3 classes

• Let us generate a dataset for each class to illustrate the LDA transfor-
mation.
• For each class:
– Use the random number generator to generate a uniform stream of
500 samples that follows U (0, 1).
– Using the Box-Muller approach, convert the generated uniform stream
to N (0, 1).
– Then use the method of eigenvalues and eigenvectors to manip-
ulate the standard normal to have the required mean vector and
covariance matrix .
– Estimate the mean and covariance matrix of the resulted dataset.

Figure 7.7: Generated and manipulated dataset, for 3 classes.

7.3. Linear Discriminant Analysis 189
lda_c3.m
1 close all;
2 try, pkg load statistics; end % for octave
3

4 %% uniform stream
5 U = rand(2,1000); u1 = U(:,1:2:end); u2 = U(:,2:2:end);
6

7 %% Box-Muller method to convert to N(0,1)

8 X = sqrt((-2).*log(u1)).*(cos(2*pi.*u2)); % 2 x 500
9 clear u1 u2 U;
10

11 %% manipulate for required Mean and Cov

12 Mu = [5;5];
13

14 Mu1= Mu +[-3;7]; Cov1 =[5 -1; -3 3];

15 X1 = denormalize(X,Mu1,Cov1);
16 Mu2= Mu +[-3;-4]; Cov2 =[4 0; 0 4];
17 X2 = denormalize(X,Mu2,Cov2);
18 Mu3= Mu +[7; 5]; Cov3 =[4 1; 3 3];
19 X3 = denormalize(X,Mu3,Cov3);
20

21 %%Begin the comptation of the LDA Projection Vectors

22 % estimate mean and covariance
23 N1 = size(X1,2); N2 = size(X2,2); N3 = size(X3,2);
24 Mu1 = mean(X1')'; Mu2 = mean(X2')'; Mu3 = mean(X3')';
25 Mu = (Mu1+Mu2+Mu3)/3.;
26

27 % within-class scatter matrix

28 S1 = cov(X1'); S2 = cov(X2'); S3 = cov(X3');
29 Sw = S1+S2+S3;
30

31 % between-class scatter matrix

32 Sb1 = N1 * (Mu1-Mu)*(Mu1-Mu)'; Sb2 = N2 * (Mu2-Mu)*(Mu2-Mu)';
33 Sb3 = N3 * (Mu3-Mu)*(Mu3-Mu)';
34 Sb = Sb1+Sb2+Sb3;
35

36 % computing the LDA projection

37 invSw_Sb = inv(Sw)*Sb; [V,D] = eig(invSw_Sb);
38 w1 = V(:,1); w2 = V(:,2);
39 if D(1,1)<D(2,2), w1 = V(:,2); w2 = V(:,1); end
40

41 lda_c3_visualize;

Figure 7.8: lda_c3.m

190 Chapter 7. Feature Extraction: Data Compression

denormalize.m
1 function Xnew = denormalize(X,Mu,Cov)
2 % it manipulates data samples in N(0,1) to something else.
3

4 [V,D] = eig(Cov); VsD = V*sqrt(D);

5

6 Xnew = zeros(size(X));
7 for j=1:size(X,2)
8 Xnew(:,j)= VsD * X(:,j);
9 end
10

11 %Now, add "replicated and tiled Mu"

12 Xnew = Xnew +repmat(Mu,1,size(Xnew,2));

lda_c3_visualize.m
1 figure, hold on; axis([-10 20 -5 20]);
2 xlabel('x_1 - the first feature','fontsize',12);
3 ylabel('x_2 - the second feature','fontsize',12);
4 plot(X1(1,:)',X1(2,:)','ro','markersize',4,"linewidth",2)
5 plot(X2(1,:)',X2(2,:)','g+','markersize',4,"linewidth",2)
6 plot(X3(1,:)',X3(2,:)','bd','markersize',4,"linewidth",2)
7 hold off
8 print -dpng 'LDA_c3_Data.png'
9

10 figure, hold on; axis([-10 20 -5 20]);

11 xlabel('x_1 - the first feature','fontsize',12);
12 ylabel('x_2 - the second feature','fontsize',12);
13 plot(X1(1,:)',X1(2,:)','ro','markersize',4,"linewidth",2)
14 plot(X2(1,:)',X2(2,:)','g+','markersize',4,"linewidth",2)
15 plot(X3(1,:)',X3(2,:)','bd','markersize',4,"linewidth",2)
16

17 plot(Mu1(1),Mu1(2),'c.','markersize',20)
18 plot(Mu2(1),Mu2(2),'m.','markersize',20)
19 plot(Mu3(1),Mu3(2),'r.','markersize',20)
20 plot(Mu(1),Mu(2),'k*','markersize',15,"linewidth",3)
21 text(Mu(1)+0.5,Mu(2)-0.5,'\mu','fontsize',18)
22

23 t = -5:20; line1_x = t*w1(1); line1_y = t*w1(2);

24 plot(line1_x,line1_y,'k-',"linewidth",3);
25 t = -5:10; line2_x = t*w2(1); line2_y = t*w2(2);
26 plot(line2_x,line2_y,'m--',"linewidth",3);
27 hold off
28 print -dpng 'LDA_c3_Data_projection.png'
29

30 %Project the samples through w1

31 wk = w1;
32 z1_wk = wk'*X1; z2_wk = wk'*X2; z3_wk = wk'*X3;
7.3. Linear Discriminant Analysis 191

33

34 z1_wk_Mu = mean(z1_wk); z1_wk_sigma = std(z1_wk);

35 z1_wk_pdf = mvnpdf(z1_wk',z1_wk_Mu,z1_wk_sigma);
36

37 z2_wk_Mu = mean(z2_wk); z2_wk_sigma = std(z2_wk);

38 z2_wk_pdf = mvnpdf(z2_wk',z2_wk_Mu,z2_wk_sigma);
39

40 z3_wk_Mu = mean(z3_wk); z3_wk_sigma = std(z3_wk);

41 z3_wk_pdf = mvnpdf(z3_wk',z3_wk_Mu,z3_wk_sigma);
42

43 figure, plot(z1_wk,z1_wk_pdf,'ro',z2_wk,z2_wk_pdf,'g+',...
44 z3_wk,z3_wk_pdf,'bd')
45 xlabel('z','fontsize',12); ylabel('p(z|w1)','fontsize',12);
46 print -dpng 'LDA_c3_Xw1_pdf.png'
47

48 %Project the samples through w2

49 wk = w2;
50 z1_wk = wk'*X1; z2_wk = wk'*X2; z3_wk = wk'*X3;
51

52 z1_wk_Mu = mean(z1_wk); z1_wk_sigma = std(z1_wk);

53 z1_wk_pdf = mvnpdf(z1_wk',z1_wk_Mu,z1_wk_sigma);
54

55 z2_wk_Mu = mean(z2_wk); z2_wk_sigma = std(z2_wk);

56 z2_wk_pdf = mvnpdf(z2_wk',z2_wk_Mu,z2_wk_sigma);
57

58 z3_wk_Mu = mean(z3_wk); z3_wk_sigma = std(z3_wk);

59 z3_wk_pdf = mvnpdf(z3_wk',z3_wk_Mu,z3_wk_sigma);
60

61 figure, plot(z1_wk,z1_wk_pdf,'ro',z2_wk,z2_wk_pdf,'g+',...
62 z3_wk,z3_wk_pdf,'bd')
63 xlabel('z','fontsize',12); ylabel('p(z|w2)','fontsize',12);
64 print -dpng 'LDA_c3_Xw2_pdf.png'
192 Chapter 7. Feature Extraction: Data Compression

Figure 7.9: w ∗1 (solid line in black) and w ∗2 (dashed line in magenta).

• w ∗1 = [0.85395, 0.52036]T , w ∗2 = [−0.62899, 0.77742]T .

• Corresponding eigenvalues read

λ1 = 3991.2, λ2 = 1727.7.
7.3. Linear Discriminant Analysis 193

Figure 7.10: Classes PDF, along the first projection vector w ∗1 ; λ1 = 3991.2.

Figure 7.11: Classes PDF, along the second projection vector w ∗2 ; λ2 = 1727.7.

Apparently, the projection vector that has the highest eigenvalue pro-
vides higher discrimination power between classes.
194 Chapter 7. Feature Extraction: Data Compression

LDA vs. PCA

Figure 7.12: PCA vs. LDA.

The (supervised) LDA classifier must work better than the (unsuper-
vised) PCA, for datasets in Figures 7.9 and 7.12.

Recall: Fisher’s LDA was generalized under the assumption of equal class
covariances and normally distributed classes.
However, even if one or more of those assumptions are (slightly) violated,
the LDA for dimensionality reduction can still work reasonably well.
7.3. Linear Discriminant Analysis 195

Limitations of the LDA (classifier)

• The LDA produces at most (c − 1) feature projections.
– If the classification error estimates establish that more features
are needed, some other method must be employed to provide those
additional features.
• The LDA is a parametric method, since it assumes unimodal Gaus-
sian likelihoods.
– If the distributions are significantly non-Gaussian, the LDA pro-
jections will not be able to preserve any complex structure of the
data, which may be needed for classification.
• The LDA will fail when the discriminatory information is not in the
mean but rather in the variance of the data.
196 Chapter 7. Feature Extraction: Data Compression

7.3.4. The LDA: dimensionality reduction

Let X ∈ RN×d be the data matrix, in which each row represents a sample.

We summarize the main steps that are required to perform the LDA for
dimensionality reduction.
1. Standardize the d -dimensional dataset (d is the number of features).
2. For each class j , compute the d -dimensional mean vector µj .
3. Construct the within-class scatter matrix Sw (7.68) and the
between-class scatter matrix Sb (7.69).
4. Compute the eigenvectors and corresponding eigenvalues of the ma-
trix Sw−1 Sb (7.71).
5. Sort the eigenvalues by decreasing order to rank the corresponding
eigenvectors.
6. Choose the k eigenvectors that correspond to the k largest eigenvalues
to construct a transformation matrix

W = [w 1 |w 2 | · · · |w k ] ∈ Rd ×k ; (7.74)

the eigenvectors are the columns of this matrix.

7. Project the samples onto a new feature subspace: X → Z := XW .

Remark 7.24.
• rank(Sw−1 Sb ) ≤ c − 1; we must have k ≤ c − 1.
• The projected feature Zij is x (i) · w j in the projected coordinates and
(x (i) · w j ) w j in the original coordinates.

Required Coding/Reading : pp. 159-169, Python Machine Learning, 3rd Ed.

7.4. Kernel Principal Component Analysis 197

7.4. Kernel Principal Component Analysis

The kernel principal component analysis (kernel PCA) [59] is an ex-
tension of the PCA using kernel techniques and performing the originally
linear operations of the PCA in a reproducing kernel Hilbert space.

Recall: (PCA). Consider a data matrix X ∈ RN×d :

◦ each of the N rows represents a different data point,
◦ each of the d columns gives a particular kind of feature, and
◦ each column has zero empirical mean (e.g., after standardization).

• The goal of the standard PCA is to find an orthogonal weight matrix

Wk ∈ Rd ×k such that
Zk = X Wk , k ≤ d, (7.75)
where Zk ∈ RN×k is call the truncated score matrix and Zd = Z .
Columns of Z represent the principal components of X .
• (Remark 7.3, p. 160). The transformation matrix Wk turns out to be
the collection of eigenvectors of X T X :
Wk = [w 1 |w 2 | · · · |w k ], (X T X ) w j = λj w j , w Ti w j = δij , (7.76)

where λ1 ≥ λ2 ≥ · · · ≥ λk ≥ 0.
• (Remark 7.4, p. 160). The matrix Zk ∈ RN×k is scaled eigenvectors of
XX T :
p p p
Zk = [ λ1 u1 | λ2 u2 | · · · | λk uk ], (XX T ) uj = λj uj , uTi uj = δij . (7.77)

• A data (row) vector x (new or old) is transformed to a k -

dimensional row vector of principal components
z = xWk ∈ R1×k . (7.78)

• (Remark 7.5, p. 161). Let X = U ΣV T be the SVD of X , where

Σ = diag(σ1 , σ2 , · · · , σd ), σ1 ≥ σ2 ≥ · · · ≥ σd ≥ 0.

Then,
V∼
= W; σj2 = λj , j = 1, 2, · · · , d. (7.79)
198 Chapter 7. Feature Extraction: Data Compression

7.4.1. Principal components of the kernel PCA

1
Note: Let C = N
X T X , the covariance matrix of X . Then,
N N
1 X (i) (i) T 1 X (i) (i)
C= x x ∈ Rd ×d , Cjk = xj xk . (7.80)
N N
i=1 i=1

Here, we consider x (i) as a column vector (when standing alone), while it

lies in X as a row.

• The kernel PCA is a generalization of the PCA, where the dataset X

is transformed into a higher dimensional space (by creating non-
linear combinations of the original features):

φ : X ∈ RN×d → φ(X ) ∈ RN×p , d < p, (7.81)

and the covariance matrix is computed via outer products between

such expanded samples:
N
1X 1
C= φ(x (i) )φ(x (i) )T = φ(X )T φ(X ) ∈ Rp ×p . (7.82)
N N
i=1

• To obtain the eigenvectors – the principal components – from the co-

variance matrix, we should solve the following equation:
Cv = λv. (7.83)

Let, for λ1 ≥ λ2 ≥ · · · ≥ λk ≥ · · · ≥ λp ≥ 0,
Vk = [v 1 |v 2 | · · · |v k ] ∈ Rp ×k , Cv j = λj v j , v Ti v j = δij . (7.84)

Then, the score matrix Zk (principal components) for the kernel

PCA reads
Zk = φ(X )Vk ∈ RN ×k , (7.85)
which is an analogue to (7.75).

However, it is computationally expensive or impossible to solve the

eigenvalue problem (7.83), when p is large or infinity.
7.4. Kernel Principal Component Analysis 199

Claim 7.25. Let v be an eigenvector of C as in (7.83). Then it can be

expressed as linear combination of features:
N
X
v= αi φ(x (i) ). (7.86)
i=1

Proof. Since Cv = λv , we get

N
1X
φ(x (i) )φ(x (i) )T v = λv
N
i=1

and therefore
N N
1 X (i) (i) T 1 X
v= φ(x )φ(x ) v = [φ(x (i) ) · v] φ(x (i) ), (7.87)
λN λN
i=1 i=1

where φ(x (i) ) · v is a scalar and αi := (φ(x (i) ) · v)/(λN).

Note:
• The above claim means that all eigenvectors v with λ 6= 0 lie in the
span of φ(x (1) ), · · · , φ(x (N) ).
• Thus, finding the eigenvectors in (7.83) is equivalent to finding the
coefficients α = (α1 , α2 , · · · , αN )T .
200 Chapter 7. Feature Extraction: Data Compression

How to find α

• Let Cv j = λj v j with λj 6= 0. Then, (7.86) can be written as

N
X
vj = α`j φ(x (`) ) = φ(X )T αj . (7.88)
`=1

• By substituting this back into the equation and using (7.82), we get
1
Cv j = λj v j ⇒ φ(X )T φ(X )φ(X )T αj = λj φ(X )T αj . (7.89)
N
and therefore
1
φ(X )φ(X )T φ(X )φ(X )T αj = λj φ(X )φ(X )T αj . (7.90)
N
• Let K be the similarity (kernel) matrix:
def
K == φ(X )φ(X )T ∈ RN ×N , (7.91)

where
Kij := K(x (i) , x (j) ) = φ(x (i) )T φ(x (j) ), (7.92)
where K is called the kernel function.a
• Then, (7.90) can be rewritten as
K 2 αj = (N λj ) K αj . (7.93)

• We can remove a factor of K from both sides of the above equation:b

K αj = µ j αj , µj = N λj . (7.94)

which implies that αj are eigenvectors of K .

a
As for nonlinear SVM, the most commonly used kernels are the polynomial kernel, the hyperbolic
tangent (sigmoid) kernel, and the Gaussian Radial Basis Function (RBF) kernel. See (4.55)-(4.58), p. 98.
b
This will only affects the eigenvectors with zero eigenvalues, which will not be a principle component
anyway.

Note: There is a normalization condition for the αj vectors:

√
kv j k = 1 ⇐⇒ kαj k = 1/ µj .
"
1 = v Tj v j = (φ(X )T αj )T φ(X )T αj = αTj φ(X )φ(X )T αj ⇐ (7.88) (7.95)
∵
= αTj K αj = αTj (µj αj ) = µj kαj k2 ⇐ (7.94)
7.4. Kernel Principal Component Analysis 201

7.4.2. Further concerns with the kernel PCA

Remark 7.26. Let the eigenvalue-eigenvector pairs of the kernel ma-
trix K be given as

K αj = µj αj , j = 1, 2, · · · , N; αTi αj = δij . (7.96)

Referring (7.77) derived for the standard PCA, we may conclude that the
k principal components for the kernel PCA are
√ √ √
Ak = [ µ1 α1 | µ2 α2 | · · · | µk αk ] ∈ RN×k . (7.97)

Remark 7.27. It follows from (7.85), (7.88), and (7.95)-(7.96) that for a
new point x , its projection onto the principal components is:
N N
1 X
(`) 1 X
T
zj = φ(x) v j = √ φ(x) T
α`j φ(x ) = √ α`j φ(x)T φ(x (`) )
µj µj
`=1 `=1
(7.98)
N
1 X 1
= √ α`j K(x, x (`) ) = √ K(x, X )T αj .
µj µj
`=1

Construction of the kernel matrix K

• The kernel trick is to avoid calculating the pairwise dot products of

the transformed samples φ(x) explicitly by using a kernel function.
• For a selected kernel function K,
 
K(x (1) , x (1) ) K(x (1) , x (2) ) · · · K(x (1) , x (N) )
 K(x (2) , x (1) ) K(x (2) , x (2) ) · · · K(x (2) , x (N) ) 
K =  ∈ RN×N . (7.99)
 
.. .. ... ..
 . . . 
K(x , x ) K(x , x (2) )
(N) (1) (N) (N)
· · · K(x , x ) (N)
202 Chapter 7. Feature Extraction: Data Compression

Normalizing the feature space

• In general, φ(x (i) ) may not be zero mean.

• Thus K = φ(X )φ(X )T would better be normalized before start finding its
eigenvectors and eigenvalues.
• Centered features:
N
(i) 1 X
(i)
φe(x ) = φ(x ) − φ(x (k ) ), ∀ i. (7.100)
N
k =1

• The corresponding kernel is

e (x (i) , x (j) ) = φe(x (i) )T φe(x (j) )
K
N N

(i) 1X (k )
T 
(j) 1X (k )

= φ(x ) − φ(x ) φ(x ) − φ(x )
N N
k =1 k =1
N N N
1X 1X 1 X
(i)
= K(x , x ) − (j) (i)
K(x , x ) − (k )
K(x (k ) (j)
,x ) + 2 K(x (k ) , x (`) ).
N N N
k =1 k =1 k ,`=1
(7.101)
• In a matrix form
e = K − K1 − 1 K + 1 K1 ,
K (7.102)
1 /N 1/N 1/N 1 /N

where 11/N is an N × N matrix where all entries are equal to 1/N .

7.4. Kernel Principal Component Analysis 203

Summary 7.28. (Summary of the Kernel PCA).

• Pick a kernel function K.
• For data X ∈ RN×d , construct the kernel matrix

K = [K(x (i) , x (j) )] ∈ RN ×N . (7.103)

• Normalize the kernel matrix K :

e = K − K1 − 1 K + 1 K1 .
K (7.104)
1 /N 1/N 1 /N 1/N

• Solve an eigenvalue problem:

e αj = µj αj ,
K αTi αj = δij . (7.105)

• Then, the k principal components for the kernel PCA are

Ak = [µ1 α1 |µ2 α2 | · · · |µk αk ] ∈ RN×k , k ≤ N. (7.106)

• For any data point x (new or old), we can represent it as

N
X N
X
(`)
T
zj = φ(x) v j = φ(x) T
α`j φ(x ) = α`j K(x, x (`) ), j = 1, 2, · · · , k .
`=1 `=1
(7.107)

Note: Formulas in (7.106)-(7.107) are alternatives of (7.97)-(7.98).

Properties of the KPCA

• With an appropriate choice of kernel function, the kernel PCA can give a
good re-encoding of the data that lies along a nonlinear manifold.
• The kernel matrix is in (N × N)-dimensions, so the kernel PCA will have
difficulties when we have lots of data points.
204 Chapter 7. Feature Extraction: Data Compression

Exercises for Chapter 7

7.1. Read pp. 145–158, Python Machine Learning, 3rd Ed., about the PCA.
(a) Find the optimal number of components k ∗ which produces the best classifica-
tion accuracy (for logistic regression), by experimenting the example code with
n_components = 1, 2, · · · , 13.
(b) What is the corresponding cumulative explained variance?
7.2. Let A ∈ Rm×n . Prove that ||A ||2 = σ1 , the largest singular value of A . Hint: Use the
following
||A v 1 ||2 σ1 ||u1 ||2
= = σ1 =⇒ ||A ||2 ≥ σ1
||v 1 ||2 ||v 1 ||2
and arguments around Equations (7.35) and (7.36) for the opposite directional in-
equality.
7.3. Recall that the Frobenius matrix norm is defined by
m X
X n 1/2
||A ||F = |aij |2 , A ∈ Rm×n .
i=1 j=1

Show that ||A ||F = (σ12 + · · · + σk2 )1/2 , where σj are nonzero singular values of A . Hint:
You may use the norm-preserving property of orthogonal matrices. That is, if U is
orthogonal, then ||UB ||2 = ||B ||2 and ||UB ||F = ||B ||F .
7.4. Prove Lemma 7.18. Hint: For (b), let A v i = λi v i , i = 1, 2, and λ1 6= λ2 . Then

(λ1 v 1 ) · v 2 = (A v 1 ) · v 2 = v 1 · (A v 2 ) = v 1 · (λ2 v 2 ).
| {z }
∵ A is symmetric

For (a), you may use a similar argument, but with the dot product being defined for
complex values, i.e.,
u · v = uT v,
where v is the complex conjugate of v .
7.5. Use Matlab to generate a random matrix A ∈ R8×6 with rank 4. For example,
A = randn(8,4);
A(:,5:6) = A(:,1:2)+A(:,3:4);
[Q,R] = qr(randn(6));
A = A*Q;

(a) Print out A on your computer screen. Can you tell by looking if it has (numerical)
rank 4?
(b) Use Matlab’s “svd" command to obtain the singular values of A . How many are
“large?" How many are “tiny?" (You may use the command “format short e" to get
a more accurate view of the singular values.)
(c) Use Matlab’s “rank" command to confirm that the numerical rank is 4.
7.4. Kernel Principal Component Analysis 205

(d) Use the “rank" command with a small enough threshold that it returns the value
6. (Type “help rank" for information about how to do this.)

7.6. Verify (7.63). Hint : Use the quotient rule for ∂J∂ w(w) and equate the numerator to zero.
7.7. Try to understand the kernel PCA more deeply by experimenting pp. 175–188, Python
Machine Learning, 3rd Ed.. Its implementation is slightly different from (but equiv-
alent to) Summary 7.28.

(a) Modify the code, following Summary 7.28, and test if it works as expected as in
Python Machine Learning, 3rd Ed..
(b) The datasets considered are transformed via the Gaussian radial basis function
(RBF) kernel only. What happens if you use the following kernels?
K1 (x (i) , x (j) ) = (a1 + b1 x (i) · x (j) )2 (polynomial of degree up to 2)
K2 (x (i) , x (j) ) = tanh(a2 + b2 x (i) · x (j) ) (sigmoid)

Can you find ai and bi , i = 1, 2, appropriately?

206 Chapter 7. Feature Extraction: Data Compression
Chapter 8

Cluster Analysis

Cluster analysis or clustering is the task of finding groups of objects such

that the objects in a group will be similar (or related) to one another and dif-
ferent from (or unrelated to) the objects in other groups. It is a main task of
exploratory data mining, and a common technique for statistical data
analysis, used in many fields, including machine learning, pattern recog-
nition, image analysis, information retrieval, bioinformatics, data compres-
sion, and computer graphics.

History . Cluster analysis was originated in anthropology by Driver

and Kroeber in 1932 [13] and introduced to psychology by Zubin in 1938
[70] and Robert Tryon in 1939 [65] and famously used by Cattell begin-
ning in 1943 [11] for trait theory classification in personality psychology.

Contents of Chapter 8
8.1. Basics for Cluster Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
8.2. K-Means Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
8.3. Hierarchical Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
8.4. DBSCAN: Density-based Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
8.5. Self-Organizing Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
8.6. Cluster Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
Exercises for Chapter 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

207
208 Chapter 8. Cluster Analysis

8.1. Basics for Cluster Analysis

Figure 8.1: Intra-cluster distance vs. inter-cluster distance.

Applications of Cluster Analysis

• Understanding
– group related documents or browsings
– group genes/proteins that have similar functionality, or
– group stocks with similar price fluctuations

• Summarization
– reduce the size of large data sets

Not Cluster Analysis

• Supervised classification – Uses class label information
• Simple segmentation – Dividing students into different registration
groups alphabetically, by last name
• Results of a query – Groupings are a result of an external specification

Clustering uses only the data:

– unsupervised learning
– to discover hidden structures in data
8.1. Basics for Cluster Analysis 209

8.1.1. Quality of clustering

• A good clustering method will produce high quality clusters with
– high intra-class similarity
– low inter-class similarity

• The quality of a clustering result depends on both the similarity mea-

sure and its implementation
• The quality of a clustering method is also measured by its ability to
discover some or all of the hidden patterns

Measuring the Quality of Clustering

• Dissimilarity/Similarity/Proximity metric: Similarity is ex-

pressed in terms of a distance function d(i, j)
• The definitions of distance functions are usually very different for
interval-scaled, boolean, categorical, ordinal ratio, and vector vari-
ables.
• There is a separate “quality” function that measures the “goodness” of
a cluster.
• Weights should be associated with different variables based on appli-
cations and data semantics.
• It is hard to define “similar enough” or “good enough”
– the answer is typically highly subjective
210 Chapter 8. Cluster Analysis

Notion of a Cluster can be Ambiguous

Figure 8.2: How many clusters?

The answer could be:

Figure 8.3: Two clusters.

Figure 8.4: Four clusters.

Figure 8.5: Six clusters.

8.1. Basics for Cluster Analysis 211

Similarity and Dissimilarity Between Objects

• Distances are normally used to measure the similarity or dissimilar-

ity between two data objects
• Some popular ones include: Minkowski distance

1/p
d(i, j) = kx i − x j kp = |xi1 − xj1 |p + |xi2 − xj2 |p · · · |xid − xjd |p , (8.1)

where x i , x j ∈ Rd , two d -dimensional data objects.

– When p = 1, it is Manhattan distance
– When p = 2, it is Euclidean distance
• Also, one can use weighted distance, parametric Pearson product mo-
ment correlation, or other dissimilarity measures

• Various similarity measures have been studied for

– Binary variables
– Nominal variables & ordinal variables
– Ratio-scaled variables
– Variables of mixed types
– Vector objects
212 Chapter 8. Cluster Analysis

8.1.2. Types of clusters

• Center-based clusters
• Contiguity/connectivity-based clusters
• Density-based clusters
• Conceptual clusters
• Described by an Objective Function

Note: (Well-separated clusters). A cluster is a set of objects such that

an object in a cluster is closer (more similar) to every/some of points
in the cluster, than any points not in the cluster.

Center-based Clusters

• The center of a cluster is often

– a centroid, the average of all the points in the cluster, or
– a medoid, the most representative point of a cluster.
• A cluster is a set of objects such that an object in a cluster is closer (more
similar) to the “center” of a cluster, than to the center of any other
clusters.

Figure 8.6: Well-separated, 4 center-based clusters.

8.1. Basics for Cluster Analysis 213

Contiguity-based Clusters

• Contiguous cluster (nearest neighbor or transitive)

• A cluster is a set of points such that a point in a cluster is closer (or
more similar) to one or more other points in the cluster, than to any
points not in the cluster.

Figure 8.7: 8 contiguous clusters.

Density-based Clusters

• A cluster is a dense region of points, which is separated by low-density

regions, from other regions of high density.
• Used when the clusters are irregular or intertwined, and when noise
and outliers are present.

Figure 8.8: 6 density-based clusters.

214 Chapter 8. Cluster Analysis

Conceptual Clusters

• Points in a cluster share some general property.

– Conceptual clusters are hard to detect, because they are often none
of the center-based, contiguity-based, or density-based.
– Points in the intersection of the circles belong to both.

Figure 8.9: Conceptual clusters

Clusters Defined by an Objective Function

• Find clusters that minimize or maximize an objective function.

• Enumerate all possible ways of dividing the points into clusters and
evaluate the “goodness” of each potential set of clusters by using the
given objective function. (NP-hard)
• Can have global or local objectives. Typically,
– Partitional clustering algorithms have global objectives
– Hierarchical clustering algorithms have local objectives
8.1. Basics for Cluster Analysis 215

8.1.3. Types of clustering

• Partitional clustering
• Hierarchical clustering (agglomerative; divisive)
• Density-based clustering (DBSCAN)

Partitional Clustering
Divide data objects into non-overlapping subsets (clusters) such that each
data object is in exactly one subset

Figure 8.10: Original points & A partitional clustering

Examples are

• K-Means, Bisecting K-Means

• K-Medoids (PAM: partitioning around medoids)
• CLARA, CLARANS (Sampling-based PAMs)
216 Chapter 8. Cluster Analysis

Hierarchical Clustering
A set of nested clusters, organized as a hierarchical tree

Figure 8.11: Dentrogram and Nested cluster diagram.

Other Distinctions Between Sets of Clusters

• Exclusive (hard) vs. Non-exclusive (soft)

– In non-exclusive clusterings, points may belong to multiple clus-
ters.
• Fuzzy vs. Non-fuzzy
– In fuzzy clustering, a point belongs to every cluster with some
membership weight between 0 and 1
– Membership weights must sum to 1
– Probabilistic clustering has similar characteristics
• Partial vs. Complete
– In some cases, we only want to cluster some of the data
• Homogeneous vs. Heterogeneous
– Cluster of widely different sizes, shapes, and densities
8.1. Basics for Cluster Analysis 217

8.1.4. Objective functions

Global objective function

• Typically used in partitional clustering

– K-Means minimizes the Sum of Squared Errors (SSE):
K X
X
SSE = kx − µi k22 , (8.2)
i=1 x ∈Ci

where x is a data point in cluster Ci and µi is the center for cluster

Ci as the mean of all points in the cluster.

• Mixture models: assume that

the dataset is a “mixture” of a
number of parametric statistical
distributions
(e.g., Gaussian mixture models).

Figure 8.12: A two-component Gaussian

mixture model: data points, and equi-
probability surfaces of the model.

Local objective function

• Hierarchical clustering algorithms typically have local objectives

• Density-based clustering is based on local density estimates
• Graph based approaches: Graph partitioning and shared nearest
neighbors

We will consider the objective functions when we talk about individual

clustering algorithms.
218 Chapter 8. Cluster Analysis

8.2. K-Means Clustering

• Given
– X , a dataset of N objects
– K , the number of clusters to form
• Organize the objects into K partitions (K ≤ N ), where each partition
represents a cluster
• The clusters are formed to optimize an objective partitioning cri-
terion:
– Objects within a cluster are similar
– Objects of different clusters are dissimilar

8.2.1. (Basic) K-Means clustering

• Partitional clustering approach
• Each cluster is associated with a centroid (mean)
• Each point is assigned to the cluster with the closest centroid
• Number of clusters, K , must be specified

Algorithm 8.1. Lloyd’s algorithm (a.k.a. Voronoi iteration):

(Lloyd, 1957) [41]
1. Select K points as the initial centroids;
2. repeat
3. Form K clusters by assigning all points to the closest centroid;
4. Recompute the centroid of each cluster;
5. until (the centroids don’t change)
8.2. K-Means Clustering 219

iteration=1 iteration=2

iteration=3 iteration=15

Figure 8.13: Lloyd’s algorithm: The Voronoi diagrams and the centroids (•) of the current
points at iterations 1, 2, 3, and 15.
220 Chapter 8. Cluster Analysis

K-Means Clustering – Details

• Initial centroids are often chosen randomly.

– Clusters produced vary from one run to another.
• The centroid is (typically) the mean of the points in the cluster.
• “Closeness” is measured by Euclidean distance, cosine similarity,
correlation, etc..
• K-Means will converge typically in the first few iterations.
– Often the stopping condition is changed to “until (relatively few
points change clusters)” or some measure of clustering doesn’t
change.
• Complexity is O(N ∗ d ∗ K ∗ I), where
N : the number of points
d : the number of attributes
K : the number of clusters
I: the number of iterations

Evaluating K-Means Clusters

• Most common measure is Sum of Squared Error (SSE):

XK X
SSE = kx − µi k22 , (8.3)
i=1 x ∈Ci

where x is a data point in cluster Ci and µi is the center for cluster Ci .

• Given sets of clusters, we can choose the one with the smallest error.
• One easy way to reduce SSE is to increase K , the number of clusters.
– A good clustering with smaller K can have a lower SSE than a poor
clustering with higher K .

K-Means is a heuristic to minimize SSE.

8.2. K-Means Clustering 221

Figure 8.14: K-Means clustering example.

222 Chapter 8. Cluster Analysis

Problems with Selecting Initial Points

The chance of selecting one centroid from each cluster is small.

• Chance is relatively small when K is large
• If clusters are the same size, n, then
# of ways to select a centroid from each cluster
P =
# of ways to select K centroids
K !nK K!
= = .
(Kn)K K K

• For example, if K = 5 or 10, then probability is:

5!/55 = 0.0384, 10!/1010 = 0.00036.

• Sometimes the initial centroids will readjust themselves in “right”

way, and sometimes they don’t.

Figure 8.15: Importance of choosing initial centroids.

8.2. K-Means Clustering 223

8.2.2. Solutions to initial centroids problem

• Multiple runs
– Helps, but probability is not on your side
• Sample and use hierarchical clustering to determine initial cen-
troids
• Select more than K initial centroids and then, among these initial
centroids
– Select most widely separated
• Post-processing
• Bisecting K-Means
– Not as susceptible to initialization issues

Pre-processing and Post-processing

• Pre-processing
– Normalize the data
– Eliminate outliers
• Post-processing
– Eliminate small clusters that may represent outliers
– Split “loose” clusters, i.e., clusters with relatively high SSE
– Merge clusters that are “close” and that have relatively low SSE

* Can use these steps during the clustering process – ISODATA

224 Chapter 8. Cluster Analysis

8.2.3. Bisecting K-Means algorithm

A variant of K-Means that can produce a partitional or a hierarchical
clustering

1. Initialize (the list of clusters) to contain the cluster of all points.

2. repeat
3. Select a cluster from the list of clusters
4. for i = 1 to iter _runs do
5. Bisect the selected cluster using basic K-Means
6. end for
7. Add the two clusters from the bisection with the lowest
SSE to the list of clusters.
8. until (the list of clusters contains K clusters)

Figure 8.16: Bisecting K-Means algorithm, with K = 4.

8.2. K-Means Clustering 225

8.2.4. Limitations of K-Means

• K-Means has problems when clusters are of differing
– sizes, densities, and non-globular shapes
• K-Means has problems when the data contains outliers

Figure 8.17: K-Means with 3 clusters of different sizes.

Figure 8.18: K-Means with 3 clusters of different densities.

Figure 8.19: K-Means with 2 non-globular clusters.

226 Chapter 8. Cluster Analysis

Overcoming K-Means Limitations

• Use a larger number of clusters

• Several clusters represent a true cluster

Figure 8.20: Unequal-sizes.

Figure 8.21: Unequal-densities.

8.2. K-Means Clustering 227

Figure 8.22: Non-spherical shapes.

Overcoming K-Means Outlier Problem

• The K-Means algorithm is sensitive to outliers.

– Since an object with an extremely large value may substantially
distort the distribution of the data.
• Solution: Instead of taking the mean value of the object in a cluster as
a reference point, medoids can be used, which is the most centrally
located object in a cluster.
228 Chapter 8. Cluster Analysis

8.2.5. K-Medoids algorithm

The K-Medoids algorithm (or PAM algorithm) is a clustering al-
gorithm similar to the K-Means algorithm. Both the K-Means and
K-Medoids algorithms are partitional (breaking the dataset up into
groups) and both attempt to minimize the distance between points la-
beled to be in a cluster and a point designated as the center of that clus-
ter. K-Medoids chooses data points as centers (medoids) and can
be used with arbitrary distances, while K-Means only minimizes the
squared Euclidean distances from cluster means. The PAM method was
proposed by (Kaufman & Rousseeuw, 1987) [31] for the work with L 1 -
norm and other distances.

• Find representative objects, called medoids, in clusters.

• PAM (partitioning around medoids) starts from an initial set of
medoids and iteratively replaces one of the medoids by one of the non-
medoids if it improves the total distance of the resulting clustering.
• PAM works effectively for small data sets, but does not scale well for
large data sets.
• CLARA (Clustering LARge Applications): sampling-based method
(Kaufmann & Rousseeuw, 1990) [32]
• CLARANS: CLARA with randomized search (Ng & Han, 1994) [46]

PAM (Partitioning Around Medoids) : Use real objects to represent the

clusters (called medoids).

Initialization: select K representative objects;

Associate each data point to the closest medoid;
while (the cost of the configuration decreases) :

For each medoid m an each non-medoid data point o :

 swap m and o;

 associate each data point to the closest medoid;
 recompute the cost (sum of distances of points to their medoid);
If the total cost of the configuration increased, undo the swap;
8.2. K-Means Clustering 229

Pros and cons of PAM

• PAM is more robust than K-Means in the presence of noise and out-
liers, because a medoid is less influenced by outliers or other extreme
values than a mean.
• PAM works efficiently for small data sets but does not scale well for
large data sets.
• The run-time complexity of PAM is O(K (N − K )2 ) for each iteration,
where N is the number of data points and K is the number of clusters.

⇒ CLARA (Clustering LARge Applications): sampling-based method

(Kaufmann & Rousseeuw, 1990) [32]

PAM finds the best K medoids among a given data, and CLARA finds the
best K medoids among the selected samples.

8.2.6. CLARA and CLARANS

CLARA (Clustering LARge Applications)

• Sampling-based PAM (Kaufmann & Rousseeuw, 1990) [32]

• It draws multiple samples of the dataset, applies PAM on each sam-
ple, and gives the best clustering as the output.

⊕ Strength: deals with larger data sets than PAM.

Weakness:
– Efficiency depends on the sample size.
– A good clustering based on samples will not necessarily represent
a good clustering of the whole data set, if the sample is biased.
• Medoids are chosen from the sample:
The algorithm cannot find the best solution if one of the best K-
Medoids is not among the selected samples.
230 Chapter 8. Cluster Analysis

CLARANS (“Randomized” CLARA)

• CLARANS (CLARA with Randomized Search) (Ng & Han; 1994,2002)
[46, 47]
• CLARANS draws sample of neighbors dynamically.
• The clustering process can be presented as searching a graph where
every node is a potential solution, that is, a set of K medoids.
• If the local optimum is found, CLARANS starts with new randomly
selected node in search for a new local optimum.
• Finds several local optimums and output the clustering with the best
local optimum.

⊕ It is more efficient and scalable than both PAM and CLARA; handles
outliers.
⊕ Focusing techniques and spatial access structures may further im-
prove its performance; see (Ng & Han, 2002) [47] and (Schubert &
Rousseeuw, 2018) [60].

Yet, the computational complexity of CLARANS is O(N 2 ), where N is

the number of objects.
The clustering quality depends on the sampling method.
8.3. Hierarchical Clustering 231

8.3. Hierarchical Clustering

Hierarchical clustering can be divided into two main types: agglomerative
and divisive.

Figure 8.23: AGNES and DIANA

Agglomerative hierarchical clustering

(a.k.a. AGNES: Agglomerative Nesting). It works in a bottom-up manner.

• Each object is initially considered as its own singleton cluster (leaf).

• At each iteration, the two closest clusters are merged into a new
bigger cluster (nodes).
• This procedure is iterated until all points are merged into a single
cluster (root).
• The result is a tree which can be plotted as a dendrogram.

Divisive hierarchical clustering

(a.k.a. DIANA: Divisive Analysis). It works in a top-down manner; the
algorithm is an inverse order of AGNES.
• It begins with the root, where all objects are included in a single cluster.
• Repeat: the most heterogeneous cluster is divided into two.
• Until: all objects are in their own cluster.

Note: Agglomerative clustering is good at identifying small clusters,

while divisive hierarchical clustering is good for large clusters.
232 Chapter 8. Cluster Analysis

Complexity

• The optimum cost is O(N 2 ), because it uses the proximity matrix. (N is

the number of points)
• In practice, O(N 3 ) in many cases.
– There are O(N) steps and at each step the proximity matrix of size
O(N 2 ) must be updated and searched.
– Complexity can be reduced to O(N 2 log(N)) for some approaches.

Limitations

• Greedy: Once a decision is made to combine two clusters, it cannot be

undone.
• No global objective function is directly minimized.
• Most algorithms have problems with one or more of:
– Sensitivity to noise and outliers
– Difficulty handling different sized clusters and convex shapes
– Chaining, breaking large clusters

Hierarchical Clustering vs. K-Means

• Recall that K-Means or K-Medoids requires

– The number of clusters K
– An initial assignment of data to clusters
– A distance measure between data d(x i , x j )
• Hierarchical clustering requires only a similarity measure between
groups/clusters of data points.
8.3. Hierarchical Clustering 233

8.3.1. AGNES: Agglomerative clustering

Quesiton. How do we measure the similarity (or dissimilarity) between

two groups of observations?

A number of different cluster agglomeration methods (i.e, linkage meth-

ods) have been developed to answer to the question. The most popular
choices are:

• Single linkage
• Complete linkage
• Group linkage
• Centroid linkage
• Ward’s minimum variance

1. Single linkage: the similarity of the closest pair

dSL (G, H) = min d(i, j). (8.4)

i ∈G,j ∈H

• Single linkage can produce “chaining”, where a sequence of

close observations in different groups cause early merges of those
groups
• It tends to produce long “loose” clusters.
2. Complete linkage: the similarity of the furthest pair

dCL (G, H) = max d(i, j). (8.5)

i ∈G,j ∈H

• Complete linkage has the opposite problem; it might not merge

close groups because of outlier members that are far apart.
• It tends to produce more compact clusters.
234 Chapter 8. Cluster Analysis

3. Group average: the average similarity between groups

1 XX
dGA (G, H) = d(i, j). (8.6)
NG NH
i ∈G j ∈H

– Group average represents a natural compromise, but depends

on the scale of the similarities. Applying a monotone transforma-
tion to the similarities can change the results.
4. Centroid linkage: It computes the dissimilarity between the cen-
troid for group G (a mean vector of length d variables) and the centroid
for group H .
5. Ward’s minimum variance: It minimizes the total within-cluster
variance. More precisely, at each step, the method finds the pair of
clusters that leads to minimum increase in total within-cluster
variance after merging. It uses the squared error (as an objective
function).
8.3. Hierarchical Clustering 235

AGNES

• Each level of the resulting tree is a segmentation of data

• The algorithm results in a sequence of groupings
• It is up to the user to choose a natural clustering from this se-
quence

Dendrogram

• Agglomerative clustering is monotonic

– The similarity between merged clusters is monotone decreasing
with the level of the merge.
• Dendrogram: Plot each merge at the (negative) similarity between
the two merged groups
• Provides an interpretable visualization of the algorithm and data
• Useful summarization tool, part of why hierarchical clustering is
popular

Figure 8.24: Six observations and a dendrogram showing their hierarchical clustering.

Remark 8.2.
• The height of the dendrogram indicates the order in which the clus-
ters were joined; it reflects the distance between the clusters.
• The greater the difference in height, the more dissimilarity.
• Observations are allocated to clusters by drawing a horizontal line
through the dendrogram. Observations that are joined together below
the line are in the same clusters.
236 Chapter 8. Cluster Analysis

Single Link

Figure 8.25: Single link clustering of six points.

• Pros: Non-spherical, non-convex clusters

• Cons: Chaining

Complete Link

Figure 8.26: Complete link clustering of six points.

• Pros: more robust against noise (no chaining)

• Cons: Tends to break large clusters; biased towards globular clusters
8.3. Hierarchical Clustering 237

Average Link

Figure 8.27: Average link clustering of six points.

• Compromise between single and complete links

Ward’s Minimum Variance Method

• Similarity of two clusters is based on the increase in squared error

when two clusters are merged
• Less susceptible to noise and outliers
• Biased towards globular clusters.
• Hierarchical analogue of K-Means; it can be used to initialize K-
Means.
238 Chapter 8. Cluster Analysis

8.4. DBSCAN: Density-based Clustering

In density-based clustering:

• Clusters are defined as areas of higher density than the remainder

of the data set. (core points)
• Objects in sparse areas – that are required to separate clusters – are
usually considered to be noise and border points.
• The most popular density-based clustering method is
– DBSCANa (Ester, Kriegel, Sander, & Xu, 1996) [14].
a
Density-Based Spatial Clustering of Applications with Noise (DBSCAN).

DBSCAN

• Given a set of points in some space, it groups together points that are
closely packed together (points with many nearby neighbors), marking
as outliers points that lie alone in low-density regions (whose nearest
neighbors are too far away).
• It is one of the most common clustering algorithms and also most
cited in scientific literature.
• In 2014, the algorithm was awarded the test of time awarda at the
leading data mining conference,
KDD 2014: https://www.kdd.org/kdd2014/.
a
The test of time award is an award given to algorithms which have received substantial attention in
theory and practice.
8.4. DBSCAN: Density-based Clustering 239

Preliminary for DBSCAN

• Consider a set of points in some space to be clustered.

• Let ε be a parameter specifying the radius of a neighborhood with
respect to some point.
• For the purpose of DBSCAN clustering, the points are classified as
core points, reachable points, and outliers, as follows:
– A point p is a core point if at least minPts points are within
distance ε of it (including p itself).
– A point q is directly reachable from p if point q is within dis-
tance ε from core point p .
(Points are only said to be directly reachable from core points.)
– A point q is reachable from p if there is a path p1 , · · · , pn with
p1 = p and pn = q, where each pi+1 is directly reachable from pi .
(Note that this implies that all points on the path must be core
points, with the possible exception of q.)
– All points not reachable from any other points are outliers or
noise points.
• Now, a core point forms a cluster together with all points (core or
non-core) that are reachable from it.
• Each cluster contains at least one core point; non-core points can
be part of a cluster, but they form its “edge”, since they cannot be used
to reach more points.

A reachable point is also called a border point.

240 Chapter 8. Cluster Analysis

Illustration of DBSCAN

Figure 8.28: Illustration of DBSCAN, with minPts = 4.

• Point A and 5 other red points are core points. They are all reachable
from one another, so they form a single cluster.
• Points B and C are not core points, but are reachable from A (via
other core points) and thus belong to the cluster as well.
• Point N is a noise point that is neither a core point nor directly-
reachable.
• Reachability is not a symmetric relation since, by definition, no
point may be reachable from a non-core point, regardless of distance
(so a non-core point may be reachable, but nothing can be reached from
it).

Definition 8.3. Two points p and q are density-connected if there

is a point c such that both p and q are reachable from c . Density-
connectedness is symmetric.

A cluster satisfies two properties:

1. All points within the cluster are mutually density-connected.
2. If a point is density-reachable from any point of the cluster, then it
is part of the cluster as well.
8.4. DBSCAN: Density-based Clustering 241

DBSCAN: Pseudocode
DBSCAN
1 DBSCAN(D, eps, MinPts)
2 C=0 # Cluster counter
3 for each unvisited point P in dataset D
4 mark P as visited
5 NP = regionQuery(P, eps) # Find neighbors of P
6 if size(NP) < MinPts
7 mark P as NOISE
8 else
9 C = C + 1
10 expandCluster(P, NP, C, eps, MinPts)
11

12 expandCluster(P, NP, C, eps, MinPts)

13 add P to cluster C
14 for each point Q in NP
15 if Q is not visited
16 mark Q as visited
17 NQ = regionQuery(Q, eps)
18 if size(NQ) >= MinPts
19 NP = NP joined with NQ
20 if Q is not yet member of any cluster
21 add Q to cluster C

Figure 8.29: Original points (left) and point types of DBSCAN clustering with eps=10 and
MinPts=4 (right): core (green), border (blue), and noise (red).
242 Chapter 8. Cluster Analysis

DBSCAN Clustering

• Resistant to Noise
• Can handle clusters of different shapes and sizes
• Eps and MinPts depend on each other and can be hard to specify

When DBSCAN Does NOT Work Well

• Varying densities
• High-dimensional data

Figure 8.30: Original points.

Figure 8.31: DBSCAN clustering. For both cases, it results in 3 clusters.

8.4. DBSCAN: Density-based Clustering 243

Some Other Clustering Algorithms

• Center-based Clustering
– Fuzzy c-means
• Mixture Models
– Expectation-maximization (EM) algorithm
• Hierarchical
– CURE (Clustering Using Representatives): shrinks points toward
center
– BIRCH (balanced iterative reducing and clustering using hierar-
chies)
• Graph-based Clustering
– Graph partitioning on a sparsified proximity graph
– SNN graph (Shared nearest-neighbor)
• Spectral Clustering
– Reduce the dimensionality using the spectrum of the similarity,
and cluster in this space
• Subspace Clustering
• Data Stream Clustering
244 Chapter 8. Cluster Analysis

8.5. Self-Organizing Maps

• Self-organization refers to a process in which the internal organiza-
tion of a system increases automatically without being guided or man-
aged by an outside source.
• This process is due to local interaction with simple rules.
• Local interaction gives rise to global structure.

We can interpret emerging global structures as learned structures,

which in turn appear as clusters of similar objects.

Note: The SOM acts as a unsupervised clustering algorithm and a

powerful visualization tool as well.
• It considers a neighborhood structure among the clusters.
⊕ The SOM is widely used in many application domains, such as econ-
omy, industry, management, sociology, geography, text mining, etc..
⊕ Many variants have been suggested to adapt the SOM to the pro-
cessing of complex data, such as time series, categorical data, nominal
data, dissimilarity or kernel data.
However, the SOM has suffered from a lack of rigorous results on
its convergence and stability.
8.5. Self-Organizing Maps 245

Game of Life : – Most famous example of self-organization.

Simple local rules (en.wikipedia.org/wiki/Conway’s_Game_of_Life):

• Any live cell with fewer than two live neighbors dies, as if caused
by under-population.
• Any live cell with two or three live neighbors lives on to the next
generation.
• Any live cell with more than three live neighbors dies, as if by
overcrowding.
• Any dead cell with exactly three live neighbors becomes a live
cell, as if by reproduction.

Figure 8.32: Still life, oscillator, and spaceship.

246 Chapter 8. Cluster Analysis

SOM Architecture

• A feed-forward neural networka architecture based on competi-

tive learningb , invented by Teuvo Kohonen in 1982 [33].
– Does not depend on a priori selection of number of clusters to
search for
– Will find the appropriate number of clusters for given the set of
instances.
• Neurobiological studies indicate that different sensory inputs (motor,
visual, auditory, etc.) are mapped onto corresponding areas of the cere-
bral cortex in an orderly fashion.
• Our interest is in building artificial topographic maps that learn
through self-organization in a neurobiologically inspired manner.
• We shall follow the principle of topographic map formation: “The
spatial location of an output neuron in a topographic map corresponds
to a particular domain or feature drawn from the input space”.
A feed-forward neural network is an artificial neural network wherein connec-
a

tions between the nodes do not form a cycle. It is the first and simplest type of artificial
neural network devised. In this network, the information moves in only one direction,
forward, from the input nodes, through the hidden nodes (if any) and to the output
nodes.
b
One particularly interesting class of unsupervised system is based on competitive
learning, in which the output neurons compete amongst themselves to be activated,
with the result that only one is activated at any one time. This activated neuron is
called a winner-takes-all neuron or simply the winning neuron. Such competi-
tion can be induced/implemented by having lateral inhibition connections (nega-
tive feedback paths) between the neurons. The result is that the neurons are forced to
organize themselves.
8.5. Self-Organizing Maps 247

Some More Details on the SOM

• The principal goal of an SOM is to transform an incoming signal

pattern of arbitrary dimension into a one or two dimensional dis-
crete map, and to perform this transformation adaptively in a topo-
logically ordered fashion.
– We therefore set up our SOM by placing neurons at the nodes of a
one or two dimensional lattice.
– Higher dimensional maps are also possible, but not so common.
• The neurons become selectively tuned to various input patterns
(stimuli) or classes of input patterns during the course of the com-
petitive learning.
• The locations of the neurons so tuned (i.e. the winning neurons) be-
come ordered and a meaningful coordinate system for the input
features is created on the lattice.
– The SOM thus forms the required topographic map of the input
patterns.
• We can view this as a non-linear generalization of principal com-
ponent analysis (PCA).

Versions of the SOM

• Basic version: a stochastic process

• Deterministic version:
– For industrial applications, it can be more convenient to use a
deterministic version of the SOM, in order to get the same results
at each run of the algorithm when the initial conditions and the
data remain unchanged (repeatable!).
– To address this issue, T. Kohonen has introduced the batch SOM
in 1995.
248 Chapter 8. Cluster Analysis

8.5.1. Kohonen networks

We shall concentrate on the particular kind of the SOM known as
a Kohonen network.
• This SOM has a feed-forward structure with a single computa-
tional layer arranged in rows and columns.
• Each neuron is fully connected to all the source nodes in the input
layer

Figure 8.33: Kohonen network.

Data Types for the Kohonen SOM

Originally, the SOM algorithm was defined for data described by numer-
ical vectors which belong to a subset X of an Euclidean space (typically
Rd ). For some results, we need to assume that the subset is bounded and
convex. Two different settings have to be considered from theoretical
and practical points of view:
• Continuous setting: the input space X ⊂ Rd is modeled by a proba-
bility distribution with a density function f ,
• Discrete setting: the input space X comprises N data points

x 1 , x 2 , · · · , x N ∈ Rd .

Here the discrete setting means a finite subset of the input space.
8.5. Self-Organizing Maps 249

Components of Self Organization

The self-organization process involves four major components:

• Initialization: All the connection weights are initialized with small
random values.
• Competition: For each input pattern, the neurons compute their re-
spective values of a discriminant function which provides the basis
for competition.
– The particular neuron with the smallest value of the discriminant
function is declared the winner.
• Cooperation: The winning neuron determines the spatial lo-
cation of a topological neighborhood of excited neurons, thereby
providing the basis for cooperation among neighboring neurons.
(smoothing the neighborhood of the winning neuron)
• Adaptation: The excited neurons decrease their individual values of
the discriminant function in relation to the input pattern through suit-
able adjustment of the associated connection weights, such that the
response of the winning neuron to the subsequent application of a sim-
ilar input pattern is enhanced.
(making the winning neuron look more like the observation)
250 Chapter 8. Cluster Analysis

Neighborhood Structure

• Let us take K units on a regular lattice

(string-like for 1D, or grid-like for 2D).
• If K = {1, 2, · · · , K } and t is the time, a neighborhood function h(t)
is defined on K × K. It has to satisfy the following properties:
– h is symmetric with hkk = 1,
– hkl depends only on the distance dist(k , l) between units k and l on
the lattice and decreases with increasing distance.
• Several choices are possible for h .
– The most classical is the step function; equal to 1 if the distance
between k and l is less than a specific radius (this radius can de-
crease with time), and 0 otherwise.
– Another very classical choice is a Gaussian-shaped function
 dist2 (k , l) 
hkl (t) = exp , (8.7)
2σ 2 (t)

where σ 2 (t) can decrease over time to reduce the intensity and
the scope of the neighborhood relations. This choice is related to
the cooperation process.
8.5. Self-Organizing Maps 251

The Stochastic SOM

• Initialization: A prototype mk ∈ Rd is attached to each unit k , whose
initial values are chosen at random and denoted by
m(0) = [m1 (0), m2 (0), · · · , mK (0)].

• Sampling: At time t , a data point x is randomly drawn (according

to the density function f or from the finite set X )
• Matching: The best matching unit is defined by
c t (x) = arg min ||x − mk (t)||2 (8.8)
k ∈K

• Updating: All the prototypes are updated via

mk (t + 1) = mk (t) + ε(t)hkc t (x) (t)(x − mk (t)), ∀ k ∈ K, (8.9)

where ε(t) is a learning rate (0 < ε(t) < 1, constant or decreasing).

• Continuation: Keep returning to the sampling step until the feature
map stops changing.

• After learning, cluster Ck is defined as the set of inputs closer to mk

than to any other one.
• The result is a data space partition, called Voronoï tessellation, with
a neighborhood structure between the clusters.

Figure 8.34: Cluster Ck in the Voronoï diagram.

• The Kohonen map is the representation of the prototypes or of the

cluster contents displayed according to the neighborhood struc-
ture.
252 Chapter 8. Cluster Analysis

Properties of the Kohonen Maps

• The quantization property: the prototypes represent the data

space as accurately as possible, as do other quantization algorithms.
– To get a better quantization, the learning rate ε(t) decreases with
time as well as the scope of the neighborhood function h .
• The self-organization property, that means that the prototypes
preserve the topology of the data: close inputs belong to the same
cluster (as do any clustering algorithms) or to neighboring clus-
ters.

Theoretical Issues

• The algorithm is easy to define and to use, and a lot of practical stud-
ies confirm that it works. However, the theoretical study of its conver-
gence when t tends to ∞ remains without complete proof and provides
open problems. The main question is to know if the solution obtained
from a finite sample converges to the true solution that might be ob-
tained from the true data distribution.
• When t tends to ∞, the Rd -valued stochastic processes [mk (t)]k =1,2,··· ,K
can present oscillations, explosion to infinity, convergence in distribu-
tion to an equilibrium process, convergence in distribution or almost
sure to a finite set of points in Rd , etc.. Some of the open questions are:
– Is the algorithm convergent in distribution or almost surely, when
t tends to ∞?
– What happens when ε(t) is constant? when it decreases?
– If a limit state exists, is it stable?
– How to characterize the organization?
8.6. Cluster Validation 253

8.6. Cluster Validation

• For supervised classification (= using class label), we have a variety of
measures to evaluate how good our model is.
◦ Accuracy, precision, recall
• For cluster analysis (= unsupervised), the analogous question is:
How to evaluate the “goodness” of the resulting clusters?

• But “clusters are in the eye of the beholder”!

• Then, why do we want to evaluate them?
◦ To avoid finding patterns in noise
◦ To compare clustering algorithms
◦ To compare two sets of clusters
◦ To compare two clusters

Aspects of Cluster Validation

1. Determining the clustering tendency of a set of data, i.e., distin-
guishing whether non-random structure actually exists in the data
(e.g., to avoid overfitting).
2. External Validation: Compare the results of a cluster analysis to
externally known class labels (ground truth).
3. Internal Validation: Evaluating how well the results of a cluster
analysis fit the data without reference to external information – use
only the data.
4. Compare clusterings to determine which is better.
5. Determining the “correct” number of clusters.
For 2, 3, and 4, we can further distinguish whether we want to evalu-
ate the entire clustering or just individual clusters.
254 Chapter 8. Cluster Analysis

Measures of Cluster Validity

Numerical measures for judging various aspects of cluster validity are

classified into the following three types.
• External Index: Used to measure the extent to which cluster labels
match externally supplied class labels.
– Entropy, Purity, Rand index
• Internal Index: Used to measure the goodness of a clustering struc-
ture without respect to external information.
– Sum of Squared Error (SSE), Silhouette coefficient
• Relative Index: Used to compare 2 different clusterings or clusters.
– Often an external or internal index is used for this function, e.g.,
SSE or entropy
Sometimes these are referred to as criteria instead of indices
• However, for some researchers, a criterion is the general strategy
and an index is the numerical measure itself that implements the
criterion.

Definition 8.4. The correlation coefficient ρX ,Y between two random

variables X and Y with expected values µX and µY and standard devia-
tions σX and σY is defined as
cov(X , Y ) E[(X − µX )(Y − µY )]
ρX ,Y = corr(X , Y ) = = ∈ [−1, 1], (8.10)
σX σY σX σY
where if X and Y are independent, ρX ,Y = 0. (The reverse may not be
true.)
8.6. Cluster Validation 255

Measuring Cluster Validity Via Correlation

• Two matrices
– Proximity matrixa (P ∈ RN×N )
– Incidence matrix (I ∈ RN×N )
* One row and one column for each data point
* An entry is 1 if the associated pair of points belong to the same cluster
* An entry is 0 if the associated pair of points belongs to different clusters

• Compute the correlation between the two matrices

– Since the matrices are symmetric, only the correlation between
N(N − 1)/2 entries needs to be calculated.

• High correlation indicates that points that belong to the same clus-
ter are close to each other.
Example: For K-Means clusterings of two data sets, the correlation
coefficient are:

Figure 8.35: ρP,I = −0.924. Figure 8.36: ρP,I = −0.581.

• Not a good measure for some density- or contiguity-based clus-

ters (e.g., single link HC).
A proximity matrix is a square matrix in which the entry in cell (i, j) is some
a

measure of the similarity (or distance) between the items to which row i and column j
correspond.
256 Chapter 8. Cluster Analysis

Using Similarity Matrix for Cluster Validation

Order the similarity matrix with respect to cluster labels and inspect
visually.

Figure 8.37: Clusters are so crisp!

Figure 8.38: Clusters in random data are not so crisp.

8.6. Cluster Validation 257

Figure 8.39: Similarity matrix for cluster validation, for DBSCAN.

8.6.1. Internal measures

• (Average) SSE is good for comparing two clusterings or two clusters.
XK X
SSE = ||x − µi k2 . (8.11)
i=1 x ∈Ci

• It can also be used to estimate the number of clusters

10 clusters

Figure 8.40: Estimation for the number of clusters.

258 Chapter 8. Cluster Analysis

Cohesion and Separation

• Cluster cohesion: Measure how closely related are objects in a cluster

– Example: Within-cluster sum of squares (WSS=SSE)
K X
X
WSS = ||x − µi k2 . (8.12)
i=1 x ∈Ci

• Cluster separation: Measures how distinct or well-separated a clus-

ter is from other clusters
– Example: Between-cluster sum of squares (BSS)
K
X
BSS = |Ci | ||µ − µi k2 . (8.13)
i=1

• Total sum of squares: TSS = WSS + BSS

– TSS is a constant for a given data set (independently of the number
of clusters)
– Example: a cluster {1, 2, 4, 5} can be separated into two clusters
{1, 2} ∪ {4, 5}. It is easy to check the following.
* 1 cluster: TSS = WSS + BSS = 10 + 0 = 10.
* 2 clusters: TSS = WSS + BSS = 1 + 9 = 10.

Silhouette Coefficient

• Silhouette coefficient combines ideas of both cohesion and separa-

tion, but for individual points. For an individual point i :
– Calculate a(i) = average distance of i to all other points in its cluster
– Calculate b(i) = min {average distance of i to points in another cluster}
– The silhouette coefficient for the point i is then given by
s(i) = 1 − a(i)/b(i). (8.14)
– Typically, s(i) ∈ [−1, 1].
– The closer to 1, the better.
• Can calculate the average silhouette width for a cluster or a clus-
tering
8.6. Cluster Validation 259

Selecting K with Silhouette Analysis on K-Means Clustering

Figure 8.41: n_clusters = 2; Figure 8.44: n_clusters = 5;

average silhouette score = 0.705. average silhouette score = 0.564.

Figure 8.42: n_clusters = 3; Figure 8.45: n_clusters = 6;

average silhouette score = 0.588. average silhouette score = 0.450.

• The silhouette plot shows that

(n_clusters = 3, 5, and 6) are
bad picks for the data, due to
– the presence of clusters with below
average silhouette scores
– wide fluctuations in the size of the
silhouette plots
Figure 8.43: n_clusters = 4;
• Silhouette analysis is ambiva-
average silhouette score = 0.651.
lent in deciding between 2 and
4.
• When n_clusters = 4, all the
silhouette subplots are more or
less of similar thickness.
https://scikit-learn.org
260 Chapter 8. Cluster Analysis

Another way of Picking K with Silhouette Analysis

10 clusters

Figure 8.46: Average silhouette coefficient vs. number of clusters.

8.6. Cluster Validation 261

8.6.2. External measures of cluster validity

• Entropya
– For cluster j , let pij be the probability that a member of cluster j
belongs to class i , defined as
pij = nij /Nj , (8.15)

where Nj is the number of points in cluster j and nij is the number

of points of class i in cluster j .
– The entropy of each cluster j is defined as
L
X
ej = − pij log2 pij , (8.16)
i=1

where L is the number of classes and

– The total entropy is calculated as the sum of entropies
PK of each clus-
ter weighted by the size of each cluster: for N = j=1 Nj ,
K
1X
e= Nj ej . (8.17)
N
j=1

• Purity
– The purity of cluster j is given by

purityj = max pij . (8.18)

i

– The overall purity of a clustering is

K
1X
purity = Nj purityj . (8.19)
N
j=1
a
The concept of entropy was introduced earlier in § 4.4.1. Decision tree objectives, when we defined
impurity measures. See (4.66) on p. 104.

Other measures: Precision, Recall, F-measure, Rand

262 Chapter 8. Cluster Analysis

Final Comment on Cluster Validity

The following is a claim in an old book by (Jain & Dubes, 1988) [29].
However, today, the claim is yet true.

“The validation of clustering structures is the most difficult and frus-

trating part of cluster analysis. Without a strong effort in this direction,
cluster analysis will remain a black art accessible only to those true be-
lievers who have experience and great courage.”
8.6. Cluster Validation 263

Exercises for Chapter 8

8.1. We will experiment the K-Means algorithm following the first section of Chapter 11,
Python Machine Learning, 3rd Ed., in a little bit different fashion.
(a) Make a dataset of 4 clusters (modifying the code on pp. 354–355).
(b) For K = 1, 2, · · · , 10, run the K-Means clustering algorithm with the initialization
init=’k-means++’.
(c) For each K , compute the within-cluster SSE (distortion) for an elbow analysis
to select an appropriate K . Note: Rather than using inertia_ attribute, imple-
ment a function for the computation of distortion.
(d) Produce silhouette plots for K = 3, 4, 5, 6.
8.2. Now, let’s experiment DBSCAN, following Python Machine Learning, 3rd Ed., pp. 376–
381.
(a) Produce a dataset having three half-moon-shaped structures each of which con-
sists of 100 samples.
(b) Compare performances of K-Means, AGNES, and DBSCAN.
(Set n_clusters=3 for K-Means and AGNES.)
(c) For K-Means and AGNES, what if you choose n_clusters much larger than 3 (for
example, 9, 12, 15)?
(d) Again, for K-Means and AGNES, perform an elbow analysis to select an appro-
priate K .
264 Chapter 8. Cluster Analysis
Chapter 9

Neural Networks and Deep Learning

Deep learning is a family of machine learning methods based on learning

data representations, as opposed to task-specific algorithms. Learning
can be supervised, semi-supervised, or unsupervised [3, 5, 58].
• Deep learning allows computational models that are composed of mul-
tiple processing layers to learn representations of data with multi-
ple levels of abstraction.
• These methods have dramatically improved the state-of-the-art in
speech recognition, visual object recognition, object detection,
and many other domains such as drug discovery and genomics.
• Deep learning discovers intricate structure in large data sets by
using the back-propagation algorithm to indicate how a machine
should change its internal parameters that are used to compute the
representation in each layer.
• Convolutional neural nets have brought about breakthroughs in
processing images, video, speech and audio, whereas recurrent neu-
ral nets have shone light on sequential data such as text and speech.

Contents of Chapter 9
9.1. Basics for Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
9.2. Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
9.3. Back-Propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
9.4. Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
Exercises for Chapter 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
Project.2. Development of New Machine Learning Algorithms . . . . . . . . . . . . . . . . . 303

265
266 Chapter 9. Neural Networks and Deep Learning

9.1. Basics for Deep Learning

Conventional Machine Learning
• Limited in their ability to process data in their raw form
• Feature!!
– Coming up with features is difficult, time-consuming, requires ex-
pert knowledge.
– When working applications of learning, we spend a lot of time tun-
ing the features.

Feature Learning
−→ −→
Representation Algorithm

Examples of features: histogram of oriented gradients (HOG), scale-invariant

feature transform (SIFT), etc.

Representation Learning

• A machine be fed with raw data; automatically discover representa-

tions
• Deep-learning methods are representation-learning methods with mul-
tiple levels of representation/abstraction
– Simple but non-linear modules ⇒ higher and abstract representa-
tion
– With the composition of enough such transformations, very com-
plex functions can be learned.
• Key aspects
– Layers of features are not designed by human engineers.
– Learn features from data using a general-purpose learning pro-
cedure.
9.1. Basics for Deep Learning 267

Advances in Deep Learning

• Image recognition [19, 23, 34]

• Speech recognition [23, 56]
• Natural language understanding [19, 62, 69]
– Machine translation
– Image 2 text
– Sentiment analysis
– Question answering (IBM’s Watson, 2011, defeated legendary Jeop-
ardy champions Brad Rutter and Ken Jennings, winning the first
place prize of $1 million)
• Many other domains
– Predicting the activity of potential drug molecules
– Analyzing particle accelerator data
– Reconstructing brain circuits
– Predicting the effects of mutations in non-coding DNA on gene ex-
pression and disease
• Deep learning has provided breakthrough results in speech recogni-
tion and image classification.
268 Chapter 9. Neural Networks and Deep Learning

Why/What about Deep Learning?

• Why is it generally better than other methods on image, speech, and

certain other types of data? Short answers:
– Deep learning means using a neural network with several lay-
ers of nodes between input and output
– The series of layers between input & output do feature identifica-
tion and processing in a series of stages, just as our brains seem
to.
• Multi-layer neural networks have been more than 30 years (Rina Dechter,
1986). What is actually new?
– We have always had good algorithms for learning the weights in
networks with 1 hidden layer; but these algorithms are not good at
learning the weights for networks with more hidden layers
– what’s new is: algorithms for training many-layer networks
9.1. Basics for Deep Learning 269

Back-propagation to Train Multi-layer Architectures

• Nothing more than a practical application of the chain rule for

derivatives
• Feed-forward neural networks
– Non-linear function: max(z, 0) (ReLU), tanh(z), 1/(1 + e −z )
• Forsaken because poor local minima
• Revived around 2006 by unsupervised learning procedures with unla-
beled data
– Local minima are rarely a problem, in practice
– CIFAR (Canadian Institute for Advanced Research): [4, 24, 25, 39]
– Recognizing handwritten digits or detecting pedestrians
– Speech recognition by GPUs, with 10 or 20 times faster (Raina et
al., 2009) [51] and (Bengio, 2013) [2].
• Convolutional neural network (CNN)
– Widely adopted by computer-vision community (LeCun et al., 1989)
[38]

AI vs. ML vs. Deep Learning

• Artificial intelligence (AI): Human intelligence exhibited by machines

• Machine learning (ML): An approach to achieve AI
• Deep learning (DL): A technique for implementing ML
– Feature/Representation-learning
– Multi-layer neural networks (NN)
– Back-propagation
(In the 1980s and 1990s, researchers did not have much luck, except
for a few special architectures.)
– New ideas (enable learning in deep NNs; since 2006)
270 Chapter 9. Neural Networks and Deep Learning

9.2. Neural Networks

Recall: In 1957, Frank Rosenblatt invented the perceptron algorithm:
• For input values: x = (x1 , x2 , · · · , xd )T ,
• Learn weight vector: w = (w1 , w2 , · · · , wd )T
• Get the net input z = w1 x1 + w2 x2 + · · · + wd xd = w · x
• Classify, using the activation function

1 if z ≥ θ,
φ(z) = z = w · x, (9.1)
−1 otherwise,

or, equivalently,

1 if z ≥ 0,
φ(z) = z = b + w · x, (9.2)
−1 otherwise,

where b = −θ is the bias. (See (2.2) and (2.3), p. 20.)

Figure 9.1: Perceptron: The simplest artificial neuron.

Example: Going to a cheese festival . Suppose the weekend is coming

up, and you’ve heard that there will be a cheese festival. You like cheese,
and are trying to decide whether or not to go to the festival. You might
make your decision by weighing up three factors:
• Is the weather good? (x1 ∈ [0, 1])
• Does your boyfriend or girlfriend want to accompany you? (x2 ∈ {0, 1})
• Is the festival near public transit? (x3 ∈ [0, 1])

For example, choose w1 = 6, w2 = 2, w3 = 2, and b = −θ = −5.

With these choices, the perceptron implements the desired decision-making
model. By varying the weights and the threshold, we can get different mod-
els of decision-making.
9.2. Neural Networks 271

Perceptron is the simplest artificial neuron that makes decisions by

weighting up evidence.

Complex Network of Perceptrons

• Obviously, the perceptron is not a complete model of human decision-

making!
• What the example illustrates is how a perceptron can weigh up differ-
ent kinds of evidence in order to make decisions.
• It should seem plausible that a complex network of perceptrons could
make quite subtle decisions:

Figure 9.2: A complex network of perceptrons.

An Issue on Perceptron Networks

• A small change in the weights or bias of any single perceptron in the

network can sometimes cause the output of that perceptron to com-
pletely flip, say from 0 to 1.
• That flip may then cause the behavior of the rest of the network to
completely change in some very complicated way.
• We can overcome this problem by introducing a new type of artificial
neuron called a sigmoid neuron.
272 Chapter 9. Neural Networks and Deep Learning

9.2.1. Sigmoid neural networks

Recall: The logistic sigmoid function is defined as

1
σ (z) = . (9.3)
1 + e −z

Figure 9.3: Standard logistic sigmoid function σ (z) = 1/(1 + e −z ).

Sigmoid Neural Networks

• They are built with sigmoid neurons.
• The output of a sigmoid neuron with inputs x , weights w , and bias b is
1
σ (z) = , (9.4)
1 + exp(−b − w · x)
which we considered as the logistic regression model in Section 4.2.
• Advantages of the sigmoid activation:
– It allows calculus to design learning rules. (σ 0 = σ (1 − σ ))
– Small changes in weights and bias produce a corresponding small
change in the output.
X ∂ output ∂ output
∆output ≈ ∆wj + ∆b. (9.5)
∂ wj ∂b
j

Figure 9.4: ∆output is a linear combination of ∆wj and ∆b .

9.2. Neural Networks 273

The architecture of (sigmoid) neural networks

• The leftmost layer is called the input layer, and the neurons within
the layer are called input neurons.
• The rightmost layer is the output layer.
• The middle layers are called hidden layers.

• The design of the input and output layers in a network is often straight-
forward. For example, for the classification of handwritten digits:
– If the images are in 28 × 28 grayscale pixels, then we’d have 784(=
28 × 28) input neurons.
– It is heuristic to set 10 neurons in the output layer. (rather than 4,
where 24 = 16 ≥ 10)
• There can be quite an art to the design of the hidden layers. In par-
ticular, it is not possible to sum up the design process for the hidden
layers with a few simple rules of thumb. Instead, neural networks re-
searchers have developed many design heuristics for the hidden layers,
which help people get the behavior they want out of their nets. For
example, such heuristics can be used to help determine how to trade
off the number of hidden layers against the time required to train the
network. We’ll meet several such design heuristics later in this chapter.
274 Chapter 9. Neural Networks and Deep Learning

9.2.2. A simple network to classify handwritten digits

• The problem of recognizing handwritten digits has two components:
segmentation and classification.

=⇒
Figure 9.5: Segmentation.

• We’ll focus on algorithmic components for the classification of individ-

ual digits.

MNIST data set : A modified subset of two data sets collected by NIST
(US National Institute of Standards and Technology):
• Its first part contains 60,000 images (for training)
• The second part is 10,000 images (for test), each of which is in 28 × 28
grayscale pixels

A Simple Feed-forward Network

Figure 9.6: A sigmoid network having a hidden layer.

9.2. Neural Networks 275

What the Neural Network Will Do

• Let’s concentrate on the first output neuron, the one that is trying
to decide whether or not the input digit is a 0.
• It does this by weighing up evidence from the hidden layer of neurons.

• What are those hidden neurons doing?

• Let’s suppose for the sake of argument that the first neuron in the
hidden layer may detect whether or not an image like the following is
present

It can do this by heavily weighting input pixels which overlap with the
image, and only lightly weighting the other inputs.
• Similarly, let’s suppose that the second, third, and fourth neurons in
the hidden layer detect whether or not the following images are present

• As you may have guessed, these four images together make up the 0
image that we saw in the line of digits shown in Figure 9.5:

• So if all four of these hidden neurons are firing, then we can conclude
that the digit is a 0.
276 Chapter 9. Neural Networks and Deep Learning

Learning with Gradient Descent

• Data set {(x (i) , y (i) )}, i = 1, 2, · · · , N

(e.g., if an image x (k ) depicts a 2, then y (k ) = (0, 0, 1, 0, 0, 0, 0, 0, 0, 0)T .)
• Cost function
1 X (i)
C(W , B) = ||y − a(x (i) )||2 , (9.6)
2N
i

where W denotes the collection of all weights in the network, B all the
biases, and a(x (i) ) is the vector of outputs from the network when x (i) is
input.
• Gradient descent method
∆W
     
W W
← + , (9.7)
B B ∆B

where
∆W
   
∇W C
= −η .
∆B ∇B C

Note: To compute the gradient ∇C , we need to compute the gradients

∇Cx (i) separately for each training input, x (i) , and then average them:
1X
∇C = ∇Cx (i) . (9.8)
N
i

Unfortunately, when the number of training inputs is very large, it

can take a long time, and learning thus occurs slowly. An idea called
stochastic gradient descent can be used to speed up learning.
9.2. Neural Networks 277

Stochastic Gradient Descent

The idea is to estimate the gradient ∇C by computing ∇Cx (i) for a small
sample of randomly chosen training inputs. By averaging over this
small sample, it turns out that we can quickly get a good estimate of the
true gradient ∇C ; this helps speed up gradient descent, and thus learning.

• Pick out a small number of randomly chosen training inputs (m  N):

(1) (2) (m)
x e , ··· , x
e ,x e ,

which we refer to as a mini-batch.

• Average ∇Cxe(k ) to approximate the gradient ∇C . That is,
m
1 X def 1
X
∇Cxe(k ) ≈ ∇C == ∇Cx (i) . (9.9)
m N
k =1 i

• For classification of handwritten digits for the MNIST data set, you
may choose: batch_size = 10.

Note: In practice, you can implement the stochastic gradient descent as

follows. For an epoch,
• Shuffle the data set
• For each m samples (selected from the beginning), update (W , B) us-
ing the approximate gradient (9.9).
278 Chapter 9. Neural Networks and Deep Learning

9.2.3. Implementing a network to classify digits [48]

network.py
1 """
2 network.py (by Michael Nielsen)
3 ~~~~~~~~~~
4 A module to implement the stochastic gradient descent learning
5 algorithm for a feedforward neural network. Gradients are calculated
6 using backpropagation. """
7 #### Libraries
8 # Standard library
9 import random
10 # Third-party libraries
11 import numpy as np
12

13 class Network(object):
14 def __init__(self, sizes):
15 """The list ``sizes`` contains the number of neurons in the
16 respective layers of the network. For example, if the list
17 was [2, 3, 1] then it would be a three-layer network, with the
18 first layer containing 2 neurons, the second layer 3 neurons,
19 and the third layer 1 neuron. """
20

21 self.num_layers = len(sizes)
22 self.sizes = sizes
23 self.biases = [np.random.randn(y, 1) for y in sizes[1:]]
24 self.weights = [np.random.randn(y, x)
25 for x, y in zip(sizes[:-1], sizes[1:])]
26

27 def feedforward(self, a):

28 """Return the output of the network if ``a`` is input."""
29 for b, w in zip(self.biases, self.weights):
30 a = sigmoid(np.dot(w, a)+b)
31 return a
32

33 def SGD(self, training_data, epochs, mini_batch_size, eta,

34 test_data=None):
35 """Train the neural network using mini-batch stochastic
36 gradient descent. The ``training_data`` is a list of tuples
37 ``(x, y)`` representing the training inputs and the desired
38 outputs. """
39

40 if test_data: n_test = len(test_data)

41 n = len(training_data)
42 for j in xrange(epochs):
43 random.shuffle(training_data)
44 mini_batches = [
45 training_data[k:k+mini_batch_size]
46 for k in xrange(0, n, mini_batch_size)]
9.2. Neural Networks 279

47 for mini_batch in mini_batches:

48 self.update_mini_batch(mini_batch, eta)
49 if test_data:
50 print "Epoch {0}: {1} / {2}".format(
51 j, self.evaluate(test_data), n_test)
52 else:
53 print "Epoch {0} complete".format(j)
54

55 def update_mini_batch(self, mini_batch, eta):

56 """Update the network's weights and biases by applying
57 gradient descent using backpropagation to a single mini batch.
58 The ``mini_batch`` is a list of tuples ``(x, y)``, and ``eta``
59 is the learning rate."""
60 nabla_b = [np.zeros(b.shape) for b in self.biases]
61 nabla_w = [np.zeros(w.shape) for w in self.weights]
62 for x, y in mini_batch:
63 delta_nabla_b, delta_nabla_w = self.backprop(x, y)
64 nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]
65 nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]
66 self.weights = [w-(eta/len(mini_batch))*nw
67 for w, nw in zip(self.weights, nabla_w)]
68 self.biases = [b-(eta/len(mini_batch))*nb
69 for b, nb in zip(self.biases, nabla_b)]
70

71 def backprop(self, x, y):

72 """Return a tuple ``(nabla_b, nabla_w)`` representing the
73 gradient for the cost function C_x. ``nabla_b`` and
74 ``nabla_w`` are layer-by-layer lists of numpy arrays, similar
75 to ``self.biases`` and ``self.weights``."""
76 nabla_b = [np.zeros(b.shape) for b in self.biases]
77 nabla_w = [np.zeros(w.shape) for w in self.weights]
78 # feedforward
79 activation = x
80 activations = [x] #list to store all the activations, layer by layer
81 zs = [] # list to store all the z vectors, layer by layer
82 for b, w in zip(self.biases, self.weights):
83 z = np.dot(w, activation)+b
84 zs.append(z)
85 activation = sigmoid(z)
86 activations.append(activation)
87 # backward pass
88 delta = self.cost_derivative(activations[-1], y) * \
89 sigmoid_prime(zs[-1])
90 nabla_b[-1] = delta
91 nabla_w[-1] = np.dot(delta, activations[-2].transpose())
92

93 for l in xrange(2, self.num_layers):

 280 Chapter 9. Neural Networks and Deep Learning

94 z = zs[-l]
95 sp = sigmoid_prime(z)
96 delta = np.dot(self.weights[-l+1].transpose(), delta) * sp
97 nabla_b[-l] = delta
98 nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())
99 return (nabla_b, nabla_w)
100

101 def evaluate(self, test_data):

102 test_results = [(np.argmax(self.feedforward(x)), y)
103 for (x, y) in test_data]
104 return sum(int(x == y) for (x, y) in test_results)
105

106 def cost_derivative(self, output_activations, y):

107 """Return the vector of partial derivatives \partial C_x /
108 \partial a for the output activations."""
109 return (output_activations-y)
110

111 #### Miscellaneous functions

112 def sigmoid(z):
113 return 1.0/(1.0+np.exp(-z))
114

115 def sigmoid_prime(z):

116 return sigmoid(z)*(1-sigmoid(z))

The code is executed using

Run_network.py
1 import mnist_loader
2 training_data, validation_data, test_data = mnist_loader.load_data_wrapper()
3

4 import network
5 n_neurons = 20
6 net = network.Network([784 , n_neurons, 10])
7

8 n_epochs, batch_size, eta = 30, 10, 3.0

9 net.SGD(training_data , n_epochs, batch_size, eta, test_data = test_data)

len(training_data)=50000, len(validation_data)=10000, len(test_data)=10000

9.2. Neural Networks 281

Validation Accuracy
Validation Accuracy
1 Epoch 0: 9006 / 10000
2 Epoch 1: 9128 / 10000
3 Epoch 2: 9202 / 10000
4 Epoch 3: 9188 / 10000
5 Epoch 4: 9249 / 10000
6 ...
7 Epoch 25: 9356 / 10000
8 Epoch 26: 9388 / 10000
9 Epoch 27: 9407 / 10000
10 Epoch 28: 9410 / 10000
11 Epoch 29: 9428 / 10000

Accuracy Comparisons

• scikit-learn’s SVM classifier using the default settings: 9435/10000

• A well-tuned SVM: 98.5%
• Well-designed (convolutional) NN: 9979/10000 (only 21 missed!)

Note: For well-designed neural networks, the performance is close

to human-equivalent, and is arguably better, since quite a few of
the MNIST images are difficult even for humans to recognize with confi-
dence, e.g.,

Figure 9.7: MNIST images difficult even for humans to recognize.

Moral of Neural Networks

• Let all the complexity be learned, automatically, from data
• Simple algorithms can perform well for some problems:
(sophisticated algorithm) ≤ (simple learning algorithm + good training data)
282 Chapter 9. Neural Networks and Deep Learning

9.2.4. Toward deep learning

While the neural network gives impressive performance, that performance
is somewhat mysterious. The weights and biases in the network were dis-
covered automatically. And that means we don’t immediately have an ex-
planation of how the network does and what it does.

Intuition for Deep Neural Networks

Assume we want to determine whether an image shows a human face or
not.

3 Suppose we are going to design a network by hand, choosing appropriate

weights and biases (not using a learning algorithm).
3 Then, a heuristic we could use is to decompose the problem into
sub-problems:

1. Does the image have an eye in the top left?

2. Does it have an eye in the top right?
3. Does it have a nose in the middle?
4. Does it have a mouth in the bottom middle?
5. Is there hair on top?
...

3 Of course, this is just a rough heuristic, and it suffers from many deficien-
cies.

• Maybe the person is bald, so they have no hair.

• Maybe we can only see part of the face, or the face is at an angle, so
some of the facial features are obscured.

3 Still, the heuristic suggests that if we can solve the sub-problems using
neural networks, then we can build a neural network for face-detection, by
combining the networks for the sub-problems.
9.2. Neural Networks 283

Figure 9.8: A possible architecture for face-detection.

3 It is also plausible that the sub-problems/sub-networks can be decom-

posed. For example, for the question: Is there an eye in the top left?
1. Is there an eyebrow?
2. Are there eyelashes?
3. Is there an iris?
...

3 Thus, the sub-network can now be decomposed.

Figure 9.9: A sub-network is decomposed.

284 Chapter 9. Neural Networks and Deep Learning

3 Those questions too can be broken down, further and further, through
multiple layers.

• Ultimately, we’ll be working with sub-networks that answer questions

so simple they can easily be answered at the level of single pixels.
• Those questions might, for example, be about the presence or absence
of very simple shapes at particular points in the image.
• Such questions can be answered by single neurons connected to the raw
pixels in the image.

Summary: Deep Neural Networks

• A network breaks down a complicated question into a series of sim-

ple questions answerable at the level of single pixels.
• It does this through a series of many layers, with early layers an-
swering very simple and specific questions about the input image, and
later layers building up a hierarchy of ever more complex and abstract
concepts.
• Networks with this kind of many-layer structure – two or more hidden
layers - are called deep neural networks.

Final Remarks

• Researchers in the 1980s and 1990s tried using stochastic gradient de-
scent and back-propagation to train deep networks.
• Unfortunately, except for a few special architectures, they didn’t have
much luck.
• The networks would learn, but very slowly, and in practice often too
slowly to be useful.
• Since 2006, a set of new techniques has been developed that enable
learning in deep neural networks.
9.3. Back-Propagation 285

9.3. Back-Propagation
• In the previous section, we saw an example of neural networks that
could learn their weights and biases using the stochastic gradient de-
scent algorithm.
• In this section, we will see how to compute the gradient, more pre-
cisely, the derivatives of the cost function with respect to weights and
biases in all layers.
• Back-propagation is a practical application of the chain rule for
the computation of derivatives.
• The back-propagation algorithm was originally introduced in the 1970s,
but its importance was not fully appreciated until a famous 1986 paper
by Rumelhart-Hinton-Williams [55], in Nature.

9.3.1. Notations
Let’s begin with notations which let us refer to weights, biases, and activa-
tions in the network in an unambiguous way.
wjk` : the weight for the connection from the k -th neuron in the (`− 1)-th
layer to the j -th neuron in the `-th layer
`
bj : the bias of the j -th neuron in the `-th layer
`
aj : the activation of the j -th neuron in the `-th layer

Figure 9.10: The weight wjk` .

286 Chapter 9. Neural Networks and Deep Learning

Figure 9.11: The bias bj` and activation aj` .

With these notations, the activation aj` reads

X 
` ` `−1 `
aj = σ wjk ak + bj , (9.10)
k

where the sum is over all neurons k in the (` − 1)-th layer. Denote the
weighted input by X
zj` := wjk` ak`−1 + bj` . (9.11)
k

Now, define
W ` = [wjk` ] : the weight matrix for layer `
b ` = [bj` ] : the bias vector for layer `
(9.12)
z ` = [zj` ] : the weighted input vector for layer `
a ` = [aj` ] : the activation vector for layer `

Then, (9.10) can be rewritten (in a vector form) as

a ` = σ (z ` ) = σ (W ` a `−1 + b ` ). (9.13)
9.3. Back-Propagation 287

9.3.2. The cost function

With the notations, the quadratic cost function (9.6) has the form
1 X
C= ||y(x) − a L (x)||2 , (9.14)
2N x

where N is the total number of training examples, y(x) is the corresponding

desired output for the training example x , and L denotes the number of
layers in the network.

Two Assumptions for the Cost Function

1. The cost function can be written as an average

1X
C= Cx , (9.15)
N x

over cost functions Cx for individual training examples x .

2. The cost function can be written as a function of the outputs from the
neural network (a L ).

Thus, the cost function in (9.14) satisfies the assumptions, with

1 L 2 1X
Cx = ||y(x) − a (x)|| = (yj (x) − ajL (x))2 . (9.16)
2 2
j

Remark 9.1. The reason we need the first assumption is because what
back-propagation actually lets us do is compute the partial deriva-
tives ∂ Cx /∂ wjk` and ∂ Cx /∂ bj` for a single training example. We then can
recover ∂ C /∂ wjk` and ∂ C /∂ bj` by averaging over training examples.

• With this assumption in mind, we’ll suppose the training example x

has been fixed, and drop the x subscript, writing the cost Cx as C .
• We may focus on computing the partial derivatives for a single ex-
ample (fixed), as given in (9.16).
288 Chapter 9. Neural Networks and Deep Learning

The Hadamard product

A frequently used algebraic operation is the element-wise product of two
vectors/matrices, which is called the Hadamard product or Schur prod-
uct, and defined as
(c d)j = cj dj . (9.17)
For example,        
1 3 1·3 3
= = . (9.18)
2 4 2·4 8
In Numpy, A*B denotes the Hadamard product of A and B, while A.dot(B) or
A@B produces the regular matrix-matrix multiplication.

9.3.3. The four fundamental equations behind

back-propagation
Back-propagation is about understanding how changing the weights and
biases in a network changes the cost function, which means comput-
ing the partial derivatives ∂ C /∂ wjk` and ∂ C /∂ bj` .

Definition 9.2. Define the learning error (or, error) of neuron j in

layer ` by
def ∂ C
δj` == . (9.19)
∂ zj`

Back-propagation will give us a way of computing δ ` = [δj` ] for every layer

`, and then relating those errors to the quantities of real interest, ∂ C /∂ wjk`
and ∂ C /∂ bj` .
9.3. Back-Propagation 289

Theorem 9.3. Suppose that the cost function C satisfies the two as-
sumptions in Section 9.3.2 so that it represents the cost for a single train-
ing example. Assume the network contains L layers, of which the feed-
forward model is given as in (9.10):
X
` ` `
aj = σ (zj ), zj = wjk` ak`−1 + bj` ; ` = 2, 3, · · · , L .
k

Then, ∂C 0 L
(a) δjL = σ (zj ),
∂ ajL
X
`
(b) δj = wkj`+1 δk`+1 σ 0 (zj` ), ` = L − 1, · · · , 2,
k
∂C (9.20)
(c) `
= δj` , ` = 2, · · · , L ,
∂ bj
∂C
(d) `
= ak`−1 δj` , ` = 2, · · · , L .
∂ wjk

Proof. Here, we will prove (b) only; see Exercise 1 for the others. Using the
definition (9.19) and the chain rule, we have
` ∂ C X ∂ C ∂ zk`+1 X ∂ zk`+1 `+1
δj = ` = `+1 ∂ z `
= δ
` k
(9.21)
∂ zj k
∂ z k j k
∂ zj

Note X X
zk`+1 = wki`+1 ai` + bk`+1 = wki`+1 σ (zi` ) + bk`+1 . (9.22)
i i

Differentiating it, we obtain

∂ zk`+1 X `+1 ∂σ (zi` )
`
= wki `
= wkj`+1 σ 0 (zj` ). (9.23)
∂ zj i
∂ zj

Substituting back into (9.21), we complete the proof.

Note: (9.20) can be written in a vector form as
(a) δ L = ∇a L C σ 0 (z L ),
(b) δ ` = ((W `+1 )T δ `+1 ) σ 0 (z ` ), ` = L − 1, · · · , 2,
(9.24)
(c) ∇b ` C = δ ` , ` = 2, · · · , L ,
(d) ∇W ` C = δ ` (a `−1 )T , ` = 2, · · · , L .
290 Chapter 9. Neural Networks and Deep Learning

Remarks 9.4.
• The four fundamental equations satisfy independently of choices of
the cost function C and the activation σ.
• A consequence of (9.24.d) is that if a `−1 is small (in modulus), gra-
dient term ∂ C /∂ W ` will also tend to be small. In this case, we’ll
say the weight learns slowly, meaning that it’s not changing much
during gradient descent.
• In other words, a consequence of (9.24.d) is that weights output
from low-activation neurons learn slowly.
• The sigmoid function σ becomes very flat when σ (zjL ) is approxi-
mately 0 or 1. When this occurs we will have σ 0 (zjL ) ≈ 0. So, a weight
in the final layer will learn slowly if the output neuron is either low
activation (≈ 0) or high activation (≈ 1). In this case, we usually say
the output neuron has saturated and, as a result, the weight is
learning slowly (or stopped).
• Similar remarks hold also in other layers and for the biases as well.
• Summing up, weights and biases will learn slowly if either the in-
neurons (upwind) are in low-activation or the out-neurons
(downwind) have saturated (either high- or low-activation).

Designing Activation Functions

The four fundamental equations can be used to design activation functions
which have particular desired learning properties. For example, sup-
pose we were to choose a (non-sigmoid) activation function σ so that σ 0 is
always positive, and never gets close to zero. That would prevent
the slow-down of learning that occurs when ordinary sigmoid neurons
saturate. Learning accuracy and efficiency can be improved by finding
more effective cost and activation functions.
9.3. Back-Propagation 291

The back-propagation algorithm

Let’s summarize back-propagation in the form of an algorithm:
1. Input x : Set the corresponding activation a 1 for the input layer.
2. Feed-forward:

z ` = W ` a `−1 + b ` ; a ` = σ (z ` ); ` = 2, 3, · · · , L

3. Output error δ L :
δ L = ∇a L C σ 0 (z L );
4. Back-propagate the error:

δ ` = ((W `+1 )T δ `+1 ) σ 0 (z ` ); ` = L − 1, · · · , 2

5. The gradient of the cost function:

∇b ` C = δ ` ; ∇W ` C = δ ` (a `−1 )T ; ` = 2, · · · , L

An SDG Learning Step, Based on a Mini-batch

1. Input a set of training examples of size m;

2. Initialize: ∆W = 0 and ∆B = 0;
3. For each training example x :

(a) Apply the back-propagation algorithm to find

∇b ` Cx = δ x,` ; ∇W ` Cx = δ x,` (a x,`−1 )T ; ` = 2, · · · , L

(b) Update the gradient:

∆B = ∆B + [∇b 2 Cx | · · · |∇b L Cx ];
∆W = ∆W + [∇W 2 Cx | · · · |∇W L Cx ];

4. Gradient descent: Update the biases and weights

η η
B = B − ∆B; W = W − ∆W ;
m m

See lines 55–69 in network.py shown in Section 9.2.3, p. 278.

292 Chapter 9. Neural Networks and Deep Learning

9.4. Deep Learning

In this section, we will consider deep neural networks; the focus is on un-
derstanding core fundamental principles behind them, and applying those
principles in the simple, easy-to-understand context of the MNIST problem.
Example 9.5. Consider neural networks for the classification of handwrit-
ten digits, as shown in the following images:

Figure 9.12: A few images in the MNIST data set.

A neural network can be built, with three hidden layers, as follows:

Figure 9.13

• Let each hidden layers have 30 neurons:

– n_weights = 282 · 30 + 30 · 30 + 30 · 30 + 30 · 10 = 25, 620
– n_biases = 30 + 30 + 30 + 10 = 100
• Optimization is difficult
– The number of parameters to teach is huge (low efficiency)
– Multiple local minima problem (low solvability)
– Adding hidden layers is not necessarily improving accuracy

In fully-connected networks, deep neural networks have been

hardly practical, except for some special applications.
9.4. Deep Learning 293

9.4.1. Introducing convolutional networks

Remarks 9.6. The neural network exemplified in Figure 9.13 can pro-
duce a classification accuracy better than 98%, for the MNIST hand-
written digit data set. But upon reflection, it’s strange to use net-
works with fully-connected layers to classify images.
• The network architecture does not take into account the spatial
structure of the images.
• For instance, it treats input pixels which are far apart and close
together, on exactly the same footing.

What if, instead of starting with a network architecture which is tabula

rasa (blank mind), we used an architecture which tries to take advan-
tage of the spatial structure? Could it be better than 99%?

Here, we will introduce convolutional neural networks (CNN), which

use a special architecture which is particularly well-adapted to classify im-
ages.
• The architecture makes convolutional networks fast to train.
• This, in turn, helps train deep, many-layer networks.
• Today, deep CNNs or some close variants are used in most neural net-
works for image recognition.
• CNNs use three basic ideas: (a) local receptive fields, (b) shared
weights and biases, and (c) pooling.
294 Chapter 9. Neural Networks and Deep Learning

(a) Local Receptive Fields

Figure 9.14: Illustration for local receptive fields.

• In CNNs, the geometry of neurons (units) in the input layer is

exactly the same as that of images (e.g., 28 × 28).
(rather than a vertical line of neurons as in fully-connected networks)
• As per usual, we’ll connect the input neurons (pixels) to a layer of
hidden neurons.
– But we will not connect fully from every input pixel to every hid-
den neuron.
– Instead, we only make connections in small, localized regions of
the input image.
– For example: Each neuron in the first hidden layer will be con-
nected to a small region of the input neurons, say, a 5 × 5 region
(Figure 9.14).
• That region in the input image is called the local receptive field for
the hidden neuron.
We slide the local receptive field across the entire input image. For each
local receptive field, there is a different hidden neuron in the first hidden
layer.

Figure 9.15: Two of local receptive fields, starting from the top-left corner.
(Geometry of neurons in the first hidden layer is 24 × 24.)
9.4. Deep Learning 295

Note: We have seen the local receptive field being moved by one pixel at
a time (stride_length=1). In fact, sometimes a different stride length
is used. For instance, we might move the local receptive field 2 pixels to
the right (or down), in which case we would say a stride length of 2 is
used.

(b) Shared Weights and Biases

Recall that each hidden neuron has a bias and 5 × 5 weights connected to
its corresponding local receptive field. In CNNs, we use the same weights
and bias for each of the 24 × 24 hidden neurons. In other words, for the
(j, k )-th hidden neuron, the output is:
 X4 X 4 
σ b+ wp,q aj+p,k +q , (9.25)
p=0 q=0

where σ is the neural activation function (e.g., the sigmoid function), b is

the shared value for the bias, and wp,q is a 5 × 5 array of shared weights.1

• We sometimes call the map from the input layer to the hidden layer a
feature map.
– Suppose the weights and bias are such that the hidden neuron
can pick out a feature (e.g., a vertical edge) in a particular local
receptive field.
– That ability is also likely to be useful at other places in the image.
– And therefore it is useful to apply the same feature detector
everywhere in the image.
• We call the weights and bias defining the feature map the shared
weights and the shared bias, respectively.
• The shared weights and bias defines clearly a kernel or filter.

To put it in slightly more abstract terms, CNNs are well adapted to the
translation invariance of images.2
1
The weighting in (9.25) is just a form of convolution; we may rewrite it as a 1 = σ (b + w ∗ a 0 ). So the
network is called a convolutional network.
2
Move a picture of a cat a little ways, and it’s still an image of a cat.
296 Chapter 9. Neural Networks and Deep Learning

Remark 9.7.
• The network structure we have considered so far can detect just a
single localized feature.
• To do more effective image recognition, we’ll need more than
one feature map.
• Thus, a complete convolutional layer consists of several different
feature maps:

Figure 9.16: A convolutional network, consisting of 3 feature maps.

Modern CNNs are often built with 10 to 50 feature maps, each associated
to a r × r local receptive field: r = 3 ∼ 9.

Figure 9.17: The 20 images corresponding to 20 different feature maps, which are actually
learned when classifying the MNIST data set (r = 5).

The number of parameters to learn

A big advantage of sharing weights and biases is that it greatly reduces the
number of parameters involved in a convolutional network.
• Convolutional networks: (5 × 5 + 1) ∗ 20 = 520
• Fully-connected networks: (28 × 28 + 1) ∗ 20 = 15, 700
9.4. Deep Learning 297

(c) Pooling

• CNNs also contain pooling layers, in addition to the convolutional

layers just mentioned.
• Pooling layers are usually used right after convolutional layers.
• What they do is to simplify the information in the output from the
convolutional layer.

Figure 9.18: Pooling: summarizing a region of 2 × 2 neurons in the convolutional layer.

From Figure 9.16: Since we have 24 × 24 neurons output from the convo-
lutional layer, after pooling we will have 12 × 12 neurons for each feature
map:

Figure 9.19: A convolutional network, consisting of 3 feature maps and pooling.

Types of pooling

1. max-pooling: simply outputs the maximum activation in the 2 × 2

input neurons.
2. L 2 -pooling: outputs the L 2 -average of the 2 × 2 input neurons.
...
298 Chapter 9. Neural Networks and Deep Learning

9.4.2. CNNs in practice

Figure 9.20: A simple CNN of three feature maps, to classify MNIST digits.

• To form a complete CNN by putting all these ideas, we need to add

some extra layers, below the convolution-pooling layers.
• Figure 9.20 shows a CNN that involves an extra layer of 10 output neu-
rons, for the 10 possible values for MNIST digits (’0’, ’1’, ’2’, etc.).
• The final layer of connections is a fully-connected layer. For example:
– Let filter_shape =(20, 1,5,5), poolsize =(2,2)
(20 feature maps; 1 × 5 × 5 kernel; 2 × 2 pooling)
– Then, the number of parameters to teach:
(52 + 1) · 20 + (20 · 122 ) · 10 = 29, 300.
– Classification accuracy for the MNIST data set . 99%
• Add a second convolution-pooling layer:
– Its input is the output of the first convolution-pooling layer.
– Let filter_shape =(40, 20,5,5), poolsize =(2,2)
– The output of the second convolution-pooling layer: 40 × 5 × 5
– Then, the number of parameters to teach:
(52 + 1) · 20 + (52 + 1) · 40 + (40 · 52 ) · 10 = 11, 560
– Classification accuracy for the MNIST data set & 99%
• Add a fully-connected layer (up the output layer):
– Let choose 40 neurons: =⇒ 41, 960 parameters
– Classification accuracy ≈ 99.5%
• Use an ensemble of networks
9.4. Deep Learning 299

– Using 5 CNNs, classification accuracy = 99.67% (33 missed!)

300 Chapter 9. Neural Networks and Deep Learning

Figure 9.21: The images missed by an ensemble of 5 CNNs. The label in the top right is
the correct classification, while in the bottom right is the label classified output.

Remarks 9.8. Intuitively speaking:

• The use of translation invariance by the convolutional layer will
reduce the number of parameters it needs to get the same perfor-
mance as the fully-connected model.
• The filters try to detect localized features, producing feature
maps.
• Pooling simplifies the information in the output from the convolu-
tional layer.
• That, in turn, will result in faster training for the convolutional
model, and, ultimately, will help us build deep networks using
convolutional layers.
• Fully-connected hidden layers try to collect information for
more widely formed features.
9.4. Deep Learning 301

Exercises for Chapter 9

9.1. Complete proof of Theorem 9.3. Hint : The four fundamental equations in (9.20) can
be obtained by simple applications of the chain rule.
9.2. The core equations of back-propagation in a network with fully-connected layers are
given in (9.20). Suppose we have a network containing a convolutional layer, a max-
pooling layer, and a fully-connected output layer, as in the network shown in Fig-
ure 9.20. How are the core equations of back-propagation modified?
9.3. (Designing a deep network). First, download a CNN code (including the MNIST
data set) by accessing to
https://github.com/mnielsen/neural-networks-and-deep-learning.git
or ‘git clone’ it. In the ‘src’ directory, there are 8 python source files:
conv.py mnist_average_darkness.py mnist_svm.py network2.py
expand_mnist.py mnist_loader.py network.py network3.py

On lines 16–22 in Figure 9.22 below, I put a design of a CNN model, which involved 2
hidden layers, one for a convolution-pooling layer and the other for a fully-connected
layer. Its test accuracy becomes approximately 98.8% in 30 epochs.

(a) Set ‘GPU = False’ in network3.py, if you are NOT using a GPU.
(Default: ‘GPU = True’, set on line 50-some of network3.py)
(b) Modify Run_network3.py appropriately to design a CNN model as accurate as
possible. Can your network achieve an accuracy better than 99.5%? Hint : You
may keep using the SoftmaxLayer for the final layer. ReLU (Rectified Linear
Units) seems comparable with the sigmoid function (default) for activation. The
default p_dropout=0.0. You should add some more layers, meaningful, and tune
well all of hyperparameters: the number of feature maps for convolutional layers,
the number of fully-connected neurons, η, p_dropout, etc..
(c) Design an ensemble of 5 such networks to further improve the accuracy. Hint :
Explore the function ‘ensemble’ defined in conv.py.
302 Chapter 9. Neural Networks and Deep Learning
Run_network3.py
1 """ Run_network3.py:
2 -------------------
3 A CNN model, for the MNIST data set,
4 which uses network3.py written by Michael Nielsen.
5 The source code can be downloaded from
6 https://github.com/mnielsen/neural-networks-and-deep-learning.git
7 or 'git clone' it
8 """
9 import network3
10 from network3 import Network, ReLU
11 from network3 import ConvPoolLayer, FullyConnectedLayer, SoftmaxLayer
12

13 training_data, validation_data, test_data = network3.load_data_shared()

14

15 mini_batch_size = 10
16 net = Network([
17 ConvPoolLayer(image_shape=(mini_batch_size, 1, 28, 28),
18 filter_shape=(20, 1, 5, 5),
19 poolsize=(2, 2), activation_fn=ReLU),
20 FullyConnectedLayer(
21 n_in=20*12*12, n_out=100, activation_fn=ReLU, p_dropout=0.0),
22 SoftmaxLayer(n_in=100, n_out=10, p_dropout=0.5)], mini_batch_size)
23

24 n_epochs, eta = 30, 0.1

25 net.SGD(training_data, n_epochs, mini_batch_size, eta, \
26 validation_data, test_data)

Figure 9.22: Run_network3.py

9.4. Deep Learning 303

Project.2. Development of New Machine Learn-

ing Algorithms
Here we present three project problems, which require to develop new algo-
rithms for machine learning.

1. A new quasi-Newton optimization method using partial information

of Hessian
2. An effective preprocessing technique for filling missing data in ma-
chine learning
3. Membership score machines for machine learning

• Choose one of them to carry out research.

• Through group discussion, you should do survey, implementation,
and performance analysis for the subject selected.
• Each group will make one or two progress presentations.
• Make the final group presentation at the end of the semester.
• Write down your experiences LATEX, for 5 to 10 pages.
304 Chapter 9. Neural Networks and Deep Learning

1. A new quasi-Newton optimization method using partial informa-

tion of Hessian
Consider a unconstrained optimization problem of the form
min f (x), (9.26)
x ∈Rd
and Newton’s method −1
x n+1 = arg min Qn (x) = x n − γ ∇2 f (x n )

∇f (x n ), (9.27)
x
where 
∂ 2f ∂ 2f ∂ 2f

 ∂x 2 ∂x ∂x ···
 1 1 2 ∂ x1 ∂ xd 

 2 2 2
 ∂ f ∂ f ∂ f 

···
∂ x2 2 ∂ x2 ∂ xd  ∈ Rd ×d .
 
∇2 f =  ∂x ∂x
 2. 1 (9.28)
 .. .. ... .. 
 . .  
 2 
 ∂ f ∂ 2f ∂ 2f 
···
∂ xd ∂ x1 ∂ xd ∂ x2 ∂ xd 2

Known . (Remark 3.14). Where applicable, Newton’s method con-

verges much faster towards a local extremum than gradient descent.
Every local minimum has a neighborhood such that, if we start within
this neighborhood, Newton’s method with step size γ = 1 converges
quadratically assuming the Hessian is invertible and Lipschitz contin-
uous.

Issues . The central issue with Newton’s method is that we need to be

able to compute the inverse Hessian matrix.
• For ML applications, the dimensionality of the problem can be of
the order of thousands or millions; computing the Hessian or its
inverse is often impractical.
• Because of these reasons, Newton’s method is rarely used in prac-
tice to optimize functions corresponding to large problems.
• Luckily, the above algorithm can still work even if the Hessian is
replaced by a good approximation.
• Various “quasi-Newton" methods have been developed so as to ap-
proximate and update the Hessian matrix (without evaluating the
second derivatives).
9.4. Deep Learning 305

Objectives. The group solving this problem is required to perform tasks

including the following.
• Algorithm Development:
– Note the Newton’s search direction is −H −1 ∇f (x n ), where H = ∇2 f (x n ).
– Select a set of k components (k  d ) from ∇f ∈ Rd which would
dominate the search direction. Then, for some permutation P ,
" #
∇f (x n )
c
P ∇f (x n ) = (9.29)
ff (x n ) .
∇

– Construct H b n ) ∈ Rk ×k (using finite differences) to solve

b = H(x

H
bqb=∇
cf (x n ). (9.30)

– Find a scaling factor σ > 0 such that

 
σ q
−P T f
b
(9.31)
∇f (x n )

is the final search direction.

– Suggestions: For d ≥ 10, k = 2 ∼ 5 and

∼ ||∇
cf (x n )||
σ = . (9.32)
||q
b||
• Comparisons:
– Implement (or download codes for) the original Newton’s method
and one of quasi-Newton methods (e.g., BFGS).
– Let’s call our method the partial Hessian (PH)-Newton method.
Compare the PH-Newton with those known methods for: the num-
ber of iterations, the total elapsed time, convergence behavior, and
stability/robustness.
– Test with e.g. the Rosenbrock function defined on Rd , d ≥ 10, with
various initial points x 0 .
306 Chapter 9. Neural Networks and Deep Learning

2. An effective preprocessing technique for filling missing data in

machine learning
(From Remark 6.2). Data preparation is difficult because the process is
not objective, and it is important because ML algorithms learn from data.
Consider the following.

• Preparing data for analysis is one of the most important steps in any
data-mining project – and traditionally, one of the most time consum-
ing.
• Often, it takes up to 80% of the time.
• Data preparation is not a once-off process; that is, it is iterative as
you understand the problem deeper on each successive pass.

Known . For missing values, three different steps can be executed.

• Removal of samples (rows) or features (columns):
It is the simplest and efficient method for handling the missing data.
However, we may end up removing too many samples or features.
• Filling the missing values manually:
This is one of the best-chosen methods.
But there is one limitation that when there are large data set,
and missing values are significant.
• Imputing missing values using computed values:
The missing values can also be occupied by computing mean, me-
dian, or mode of the observed given values. Another method could
be the predictive values that are computed by using any ML or Deep
Learning algorithm.
But one drawback of this approach is that it can generate bias
within the data as the calculated values are not accurate concerning
the observed values.
9.4. Deep Learning 307

Objectives.

• Algorithm Development:

– Think a good strategy or two for filling the missing values, if it

is to be done manually.
– What information available from the dataset is useful and help us
build a good strategy?
– Suggestions: Use near-values to interpolate; try to employ the
concept of feature importance, if available.
• Comparisons:

– For the Wine dataset, for example, erase r % data values in random;
r = 5, 10, 20.
– Compare your new filling strategy with 1 the simple sample re-
moval method and 2 the imputation strategy using mean, me-
dian, or mode.
– Perform accuracy analysis for various classifiers, e.g., logistic re-
gression, support vector machine, and random forests.
308 Chapter 9. Neural Networks and Deep Learning

3. Membership score machines

for machine learning
For the basic idea of principal component analysis (PCA), see § 7.1.
Known . • PCA is a statistical procedure that uses an orthogonal trans-
formation to convert a set of observations of possibly correlated variables
into a set of values of linearly uncorrelated variables called principal
components.
• As an unsupervised linear transformation technique, PCA is widely
used across various fields – in ML, most prominently for feature ex-
traction and dimensionality reduction.
• The orthogonal axes of the new subspace can be interpreted as the
directions of maximum variance given the constraint that the new
feature axes are orthogonal to each other, as illustrated in the following
figure:

Figure 9.23: Principal components.

• It renders a set of components that are uncorrelated.

Issues . • Not good when working with fine-grained classes.

• PCA assumes a linear transformation; it fails finding directions that
require a nonlinear transformation =⇒ Kernel PCA
9.4. Deep Learning 309

Objectives.

• Algorithm Development:

– This project will study a new probabilistic classifier, called the mem-
bership score machines (MSM), by introducing membership
score functions of the form
d
−r 2 /σ 2 2
X (µj − xi )2
ϕ(x) = e , r = . (9.33)
j=1
λ2j

– We will try to apply MSM for fine-grained classes.

– Suggestions:
• Comparisons:
310 Chapter 9. Neural Networks and Deep Learning
Bibliography

[1] W. B ELSON, Matching and prediction on the principle of biological classification,

JRSS, Series C, Applied Statistics, 8 (1959), pp. 65–75.

[2] Y. B ENGIO, Deep learning of representations: Looking forward, in Statistical Lan-

guage and Speech Processing, A. H. Dediu, C. Martín-Vide, R. Mitkov, and B. Truthe,
eds., vol. 7978 of Lecture Notes in Computer Science, Springer, Berlin, Heidelberg,
2013, pp. 1–37.

[3] Y. B ENGIO, A. C OURVILLE , AND P. V INCENT, Representation learning: A review and

new perspectives, IEEE Transactions on Pattern Analysis and Machine Intelligence,
35 (2013), pp. 1798–1828.

[4] Y. B ENGIO, P. L AMBLIN, D. P OPOVICI , AND H. L AROCHELLE, Greedy layer-wise

training of deep networks, in NIPS’06 Proceedings of the 19th International Confer-
ence on Neural Information Processing Systems, NIPS, 2006, pp. 153–160.

[5] Y. B ENGIO, Y. L E C UN, AND G. H INTON, Deep learning, Nature, 521 (2015), pp. 436–
444.

[6] A. B JÖRCK, Numerical Methods for Least Squares Problems, SIAM, Philadelphia,
1996.

[7] L. B OTTOU, F. E. C URTIS, AND J. N OCEDAL, Optimization methods for large-scale

machine learning, SIAM Rev., 60 (2018), pp. 223–311.

[8] L. B REIMAN, Random forests, Machine Learning, 45 (2001), pp. 5–32.

[9] C. G. B ROYDEN, The convergence of a class of double-rank minimization algorithms

1. General considerations, IMA Journal of Applied Mathematics, 6 (1970), pp. 76–90.

[10] J. B UNCH AND L. K AUFMAN, Some stable methods for calculating inertia and solving
symmetric linear systems, Math. Comput., 31 (1977), pp. 163–179.

[11] R. B. C ATTELL, The description of personality: Basic traits resolved into clusters, Jour-
nal of Abnormal and Social Psychology., 38 (1943), pp. 476–506.

[12] C. C ORTES AND V. N. VAPNIK, Support-vector networks, Machine Learning, 20 (1995),

pp. 273–297.

311
312 BIBLIOGRAPHY

[13] H. E. D RIVER AND A. L. K ROEBER, Quantitative expression of cultural relation-

ships, in University of California Publications in American Archaeology and Ethnol-
ogy, vol. Quantitative Expression of Cultural Relationships, 1932, pp. 211–256.

[14] M. E STER , H.-P. K RIEGEL , J. S ANDER , AND X. X U, A density-based algorithm for

discovering clusters in large spatial databases with noise, in Proceedings of the Second
International Conference on Knowledge Discovery and Data Mining, KDD-96, AAAI
Press, 1996, pp. 226–231.

[15] A. F ISHER , C. R UDIN, AND F. D OMINICI, Model class reliance: Variable importance
measures for any machine learning model class, from the "rashomon" perspective,
(2018).

[16] R. A. F ISHER, The use of multiple measurements in taxonomic problems, Annals of

Eugenics, 7 (1936), pp. 179–188.

[17] R. F LETCHER, A new approach to variable metric algorithms, The Computer Journal,
13 (1970), pp. 317–322.

[18] S. G ERSCHGORIN, Über die abgrenzung der eigenwerte einer matrix, Izv. Akad. Nauk
SSSR Ser. Mat., 7 (1931), pp. 746–754.

[19] X. G LOROT, A. B ORDES, AND Y. B ENGIO, Deep sparse rectifier neural networks,
in Proceedings of the Fourteenth International Conference on Artificial Intelligence
and Statistics, G. Gordon, D. Dunson, and M. Dudík, eds., vol. 15 of Proceedings
of Machine Learning Research, Fort Lauderdale, FL, USA, 11–13 Apr 2011, PMLR,
pp. 315–323.

[20] D. G OLDFARB, A family of variable metric methods derived by variational means,

Mathematics of Computation, 24 (1970), pp. 23–26.

[21] F. G OLUB AND C. V. L OAN, Matrix Computations, 3rd Ed., The Johns Hopkins Uni-
versity Press, Baltimore, 1996.

[22] B. G ROSSER AND B. L ANG, An o(n2 ) algorithm for the bidiagonal svd, Lin. Alg. Appl.,
358 (2003), pp. 45–70.

[23] G. H INTON, , , G. D AHL , A.- R . M OHAMED, N. J AITLY, A. S ENIOR , V. VANHOUCKE ,

B. K INGSBURY, AND T. S AINATH, Deep neural networks for acoustic modeling in
speech recognition, IEEE Signal Processing Magazine, 29 (2012), pp. 82–97.

[24] G. E. H INTON, S. O SINDERO, AND Y.-W. T EH, A fast learning algorithm for deep
belief nets, Neural Comput., 18 (2006), pp. 1527–1554.

[25] G. E. H INTON AND R. S ALAKHUTDINOV, Reducing thedimensionality of data with

neural networks, Science, 313 (2006), pp. 504–507.

[26] T. K. H O, Random decision forests, in Proceedings of the 3rd International Conference

on Document Analysis and Recognition, Montreal, QC, 1995, pp. 278–282.
BIBLIOGRAPHY 313

[27] H. H OTELLING, Analysis of a complex of statistical variables into principal compo-

nents, Journal of Educational Psychology, 24 (1933), pp. 417–441 and 498–520.
[28] , Relations between two sets of variates, Biometrika, 28 (1936), pp. 321–377.
[29] A. K. J AIN AND R. C. D UBES, Algorithms for Clustering Data, Prentice-Hall, Inc.,
Upper Saddle River, NJ, USA, 1988.
[30] W. K ARUSH, Minima of functions of several variables with inequalities as side con-
straints, M.Sc. Dissertation, Department of Mathematics, Univ. of Chicago, Chicago,
Illinois, 1939.
[31] L. K AUFMAN AND P. R OUSSEEUW, Clustering by means of medoids, in Statistical
Data Analysis Based on the L 1 -Norm and Related Methods, Y. Dodge, ed., North-
Holland, 1987, pp. 405–416.
[32] L. K AUFMAN AND P. J. R OUSSEEUW, Finding Groups in Data: An Introduction to
Cluster Analysis, John Wiley & Sons, Inc., 1990.
[33] T. K OHONEN, Self-organized formation of topologically correct feature maps, Biologi-
cal Cybernetics, 43 (1982), pp. 59–69.
[34] A. K RIZHEVSKY, I. S UTSKEVER , AND G. E. H INTON, Imagenet classification with
deep convolutional neural networks, Commun. ACM, 60 (2017), pp. 84–90.
[35] H. W. K UHN AND A. W. T UCKER, Nonlinear programming, in Proceedings of 2nd
Berkeley Symposium, Berkeley, CA, USA, 1951, University of California Press.,
pp. 481–492.
[36] K. L ANGE , D. R. H UNTER , AND I. YANG, Optimization transfer using surrogate objec-
tive functions, Journal of Computational and Graphical Statistics, 9 (2000), pp. 1–20.
[37] R. L EBLOND, F. P EDREGOSA , AND S. L ACOSTE -J ULIEN, Improved asynchronous par-
allel optimization analysis for stochastic incremental methods, CoRR, abs/1801.03749
(2018).
[38] Y. L E C UN, B. E. B OSER , J. S. D ENKER , D. H ENDERSON, R. E. H OWARD, W. E.
H UBBARD, AND L. D. J ACKEL, Handwritten digit recognition with a back-propagation
network, in Advances in Neural Information Processing Systems 2, D. S. Touretzky,
ed., Morgan-Kaufmann, 1990, pp. 396–404.
[39] Y. L E C UN, S. C HOPRA , R. H ADSELL , F. J. H UANG, AND ET AL ., A tutorial on energy-
based learning, in PREDICTING STRUCTURED DATA, MIT Press, 2006.
[40] C. L EMARECHAL, Cauchy and the gradient method, Documenta Mathematica, Extra
Volume, (2012), pp. 251–254.
[41] S. P. L LOYD, Least square quantization in pcm, Bell Telephone Laboratories Report,
1957. Published in journal: S. P. Lloyd (1982). Least squares quantization in PCM.
IEEE Transactions on Information Theory. 28 (2): pp. 129–137.
314 BIBLIOGRAPHY

[42] M. L OURAKIS, A brief description of the Levenberg-Marquardt algorithm implemented

by levmar, Technical Report, Institute of Computer Science, Foundation for Research
and Technology – Hellas, 2005.

[43] S. M ANDT, M. D. H OFFMAN, AND D. M. B LEI, Stochastic gradient descent as approx-

imate Bayesian inference, Journal of Machine Learning Research, 18 (2017), pp. 1–35.

[44] D. M ARQUARDT, An algorithm for least-squares estimation of nonlinear parameters,

Journal of the Society for Industrial and Applied Mathematics, 11 (1963), pp. 431–441.

[45] H. M OBAHI AND J. W. F ISHER III, On the link between gaussian homotopy continua-
tion and convex envelopes, in Energy Minimization Methods in Computer Vision and
Pattern Recognition. Lecture Notes in Computer Science, vol. 8932, X.-C. Tai, E. Bae,
T. F. Chan, and M. Lysaker, eds., Hong Kong, China, 2015, Springer, pp. 43–56.

[46] R. T. N G AND J. H AN, Efficient and effective clustering methods for spatial data
mining, in Proceedings of the 20th International Conference on Very Large Data
Bases, VLDB ’94, San Francisco, CA, USA, 1994, Morgan Kaufmann Publishers Inc.,
pp. 144–155.

[47] , CLARANS: A method for clustering objects for spatial data mining, IEEE Trans-
actions on Knowledge and Data Engineering, 14 (2002), pp. 1003–1016.

[48] M. N IELSEN, Neural networks and deep learning. (The online book can be found at
http://neuralnetworksanddeeplearning.com), 2013.

[49] J. N OCEDAL AND S. J. W RIGHT, Numerical Optimization (2nd Ed.), Springer-Verlag,

Berlin, New York, 2006.

[50] K. P EARSON, On lines and planes of closest fit to systems of points in space, Philosoph-
ical Magazine, 2 (1901), pp. 559–572.

[51] R. R AINA , A. M ADHAVAN, AND A. Y. N G, Large-scale deep unsupervised learning us-

ing graphics processors, in Proceedings of the 26th Annual International Conference
on Machine Learning, ICML ’09, New York, NY, USA, 2009, ACM, pp. 873–880.

[52] C. R. R AO, The utilization of multiple measurements in problems of biological classi-

fication, Journal of the Royal Statistical Society. Series B: Statistical Methodology, 10
(1948), pp. 159–203.

[53] S. R ASCHKA AND V. M IRJALILI, Python Machine Learning, 3rd Ed., Packt Publishing
Ltd., Birmingham, UK, 2019.

[54] H. R OBBINS AND S. M ONRO, A stochastic approximation method, Ann. Math. Statist.,
22 (1951), pp. 400–407.

[55] D. E. R UMELHART, G. E. H INTON, AND R. J. W ILLIAMS, Learning representations

by back-propagating errors, Nature, 323 (1986), pp. 533–536.
BIBLIOGRAPHY 315

[56] T. N. S AINATH , B. K INGSBURY, G. S AON, H. S OLTAU, M. A. R., G. D AHL , AND

B. R AMABHADRAN, Deep convolutional neural networks for large-scale speech tasks,
Neural Netw., 64 (2015), pp. 39–48.

[57] F. S ANTOSA AND W. W. S YMES, Linear inversion of band-limited reflection seismo-

grams, SIAM Journal on Scientific Computing, 7 (1986), pp. 1307–1330.

[58] J. S CHMIDHUBER, Deep learning in neural networks: An overview, Neural Networks,

61 (2015), pp. 85–117.

[59] B. S CHÖLKOPF , A. S MOLA , AND K.-R. M ÜLLER, Kernel principal component analy-
sis, in Artificial Neural Networks — ICANN’97, EDITORS, ed., vol. 1327 of Lecture
Notes in Computer Science, Springer, Berlin, Heidelberg, 1997, pp. 583–588.

[60] E. S CHUBERT AND P. J. R OUSSEEUW, Faster k-medoids clustering: Improving the

PAM, CLARA, and CLARANS algorithms, CoRR, abs/1810.05691 (2018).

[61] D. F. S HANNO, Conditioning of quasi-Newton methods for function minimization,

Mathematics of Computation, 24 (1970), pp. 647–656.

[62] I. S UTSKEVER , O. V INYALS, AND Q. V. L E, Sequence to sequence learning with neural

networks, in Advances in Neural Information Processing Systems 27, Z. Ghahramani,
M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, eds., Curran Associates,
Inc., 2014, pp. 3104–3112.

[63] O. T AUSSKY, Bounds for characteristic roots of matrices, Duke Math. J., 15 (1948),
pp. 1043–1044.

[64] R. T IBSHIRANI, Regression shrinkage and selection via the LASSO, Journal of the
Royal Statistical Society. Series B (methodological), 58 (1996), pp. 267–288.

[65] R. C. T RYON, Cluster Analysis: Correlation Profile and Orthometric (factor) Analysis
for the Isolation of Unities in Mind and Personality, Edwards Brothers, 1939.

[66] R. VARGA, Matrix Iterative Analysis, 2nd Ed., Springer-Verlag, Berlin, Heidelberg,
2000.

[67] P. R. W ILLEMS, B. L ANG, AND C. V ÖMEL, Computing the bidiagonal SVD using
multiple relatively robust representations, SIAM Journal on Matrix Analysis and Ap-
plications, 28 (2006), pp. 907–926.

[68] M. H. W RIGHT, Interior methods for constrained optimization, Acta Numerica, 1

(1992), pp. 341–407.

[69] K. X U, J. L. B A , R. K IROS, K. C HO, A. C OURVILLE , R. S ALAKHUTDINOV, R. S.

Z EMEL , AND Y. B ENGIO, Show, attend and tell: Neural image caption generation
with visual attention, in Proceedings of the 32Nd International Conference on Inter-
national Conference on Machine Learning - Volume 37, ICML’15, 2015, pp. 2048–
2057.
316 BIBLIOGRAPHY

[70] J. Z UBIN, A technique for measuring like-mindedness, The Journal of Abnormal and
Social Psychology, 33 (1938), pp. 508–516.
Index

active set, 131 centroid linkage, 234

Adaline, 27 chain rule, 269
adaline, 17 chi-squared error criterion, 61
adaptive linear neurons, 17 CIFAR, 269
adaptive step size, 44 CLARA, 228, 229
agglomerative clustering, 231, 233 CLARANS, 230
AGNES, 231, 235 classification, 3
AI-driven data preparation, 143 classification error, 104
algorithmic parameter, 29 cluster analysis, 207
approximate complementarity, 138 cluster cohesion, 258
artificial topographic maps, 246 cluster separation, 258
automated data preparation, 143 cluster validation, 253
averaged perceptron, 24 clustering, 3, 12, 207
CNN, 269, 293
back-propagation, 285 competitive learning, 246
backtracking line search, 44 complementary slackness, 88, 134
barrier functions, 136 complete linkage, 233
barrier parameter, 136 constraint set, 36
batch gradient descent, 31 convergence, iterative methods, 124
best split, 106 convolution, 295
between-class scatter, 184, 186 convolutional neural network, 269
between-class scatter matrix, 184 convolutional neural networks, 293
BFGS algorithm, 52 core points, 239
bias, 80, 270 correlation, 254
bias unit, 20 cost function, 28
binary classifier, 18 covariance matrix, 173, 183
binary decision tree, 103 criterion function, 152
binomial distribution, 76 critical point, 40, 48
bisecting K-Means algorithm, 224 cumulative explained variance, 163
bootstrap sample, 107 curse of dimensionality, 110, 147
border point, 239 curvature condition, 53
categorical data, 145 damped least-squares, 61
Cauchy, Augustin-Louis, 36 damping parameter, 64
central path, 135 data compression, 157
centroid, 212 data matrix, 159, 197

317
318 INDEX

data normalization, 30, 146 feature map, 295, 296

data preparation, 141, 143, 306 feature scaling, 30, 142, 146
data preprocessing, 29, 141, 142 feature selection, 150, 157
DBSCAN, 238 feature selection, automatic, 148
decision tree, 102 feed-forward neural network, 246
decreasing step-size sequence, 58 filter methods, 148
deep learning, 265 first-order necessary conditions, 87
deep neural network, 282, 284 Fisher’s LDA, 180
deep neural networks, 292 fully-connected networks, 292
definite, 115
dendrogram, 235 Gauss-Newton method, 63
Gaussian homotopy continuation, 46
denormalize.m, 190
Gaussian kernel, 97
density estimation, 4
Gaussian smoothing, 46
density-based clustering, 238
generalization error, 143, 147
density-connected, 240
generalized eigenvalue problem, 184
diagonal dominance, 127
Gini impurity, 104
DIANA, 231
GMRES, 129
dimensionality reduction, 157
Golub-Reinsch SVD algorithm, 176
directed graph, 125
goodness-of-fit measure, 61
distance functions, 209
gradient descent algorithm, 42, 44
divisive clustering, 231
gradient descent method, 28, 36, 276
DLS, 61
Gram matrix, 94
domain, 36
graph theory, 124
dot product of matrices, 115
greatest curvature, 50
dual feasibility, 88
group average, 234
dual problem, 86, 89, 93, 97, 100
dual variable, 86 Hadamard product, 288
dyadic decomposition, 167 Hessian, 42, 47, 50
Hessian matrix, 50
eigenvalue locus theorem, 126 hidden layers, 273
elastic net regularization, 148 hierarchical clustering, 216, 231
elbow analysis, 263 high bias, 80
element-wise product, 288 high variance, 80
embedded methods, 148 Householder transformation, 176
empirical loss, 9 hyperparameter, 29
empirical risk, 10 hypothesis space, 8
entropy, 104, 261
error, 288 image compression, 178
Euclidean distance, 211 import, in Python, 15
explained variance ratio, 163 impurity measure, 103
incidence matrix, 255
feasible solution, 35 indentation, in Python, 15
feature expansion, 99 induced matrix norm, 42
feature extraction, 157 inertia of matrix, 118
feature importance, 153 inference, statistical, 3
INDEX 319

information gain, 102, 103 likelihood function, 76

input layer, 273 linear algebraic system, 123
input neurons, 273 linear classifier, 19
interior-point method, 134, 140 linear discriminant analysis, 157, 180
internal measures, 257 linear iterative method, 123
irreducible matrix, 125 linear separators, 25
linear SVM, 83, 87, 92
Jacobian matrix, 62 linearly separable, 22
K-Medoids algorithm, 228 Lloyd’s algorithm, 218
k-nearest neighbor, 109 local minima problem, 46
Karush-Kuhn-Tucker conditions, 87 local receptive field, 294
KD-tree, 110 log-likelihood function, 77
kernel function, 97, 200 logarithmic barrier function, 136
kernel matrix, 200 logistic cost function, 77
kernel method, 96 logistic curve, 71
kernel PCA, 197 logistic function, 71
kernel PCA, summary, 203 logistic model, 71
kernel principal component analysis, 157, logistic regression, 75, 272
197 logistic sigmoid function, 272
kernel SVM, 96 logit, 73
kernel trick, 97, 99, 201 loss function, 9
KKT conditions, 87, 114, 134
KKT matrix, 114 M-matrix, 128
Kohonen network, 248 machine learning algorithm, 4
Manhattan distance, 211
L2-pooling, 297 margin, 22
Lagrange dual function, 86 Matplotlib, in Python, 15
Lagrange multipliers, 84 max-pooling, 297
Lagrangian, 87, 114 mean square error, 9
Lagrangian (objective), 85 medoid, 212
largest margin, 25 membership score functions, 309
LASSO, 148 membership score machines, 308, 309
lateral inhibition connections, 246 membership weight, 216
law of parsimony, 11 merit function, 41
lazy learner, 109 min-max scaling, 146
lda_c3.m, 189 mini-batch, 277
lda_c3_visualize.m, 190 mini-batch learning, 32
lda_Fisher.m, 185 minimize, in Python, 15
ldl_test.py, 120 Minkowski distance, 110, 211
learning error, 288 missing data, 144
learning rate, 21, 38 mixture model, 217
least curvature, 50 model reliance, 153
left singular vectors, 161, 164 multiple local minima, 46
Levenberg-Marquardt algorithm, 61, 64
likelihood, 76, 77 neighborhood function, 250
320 INDEX

network.py, 278 primal active set strategy, 132

neural network, 268 primal feasibility, 88
neuron, 18 principal component analysis, 157, 158
Newton’s method, 47, 135 principal curvatures, 50
no free lunch theorem, 70 principal directions, 50
nodal point, 125 principle of topographic map formation,
nominal features, 145 246
nonlinear least squares problems, 61 print, in Python, 15
normalization, 142, 146 probabilistic clustering, 216
normalization condition, 200 projection vector, 181
np.ndarray, in Python, 15 proximity matrix, 255
null-space approach, 122, 140 pseudoinverse, 170, 175
numerical rank, 177 purity, 261
pylab, in Python, 15
objective function, 28, 36
Occam’s razor principle, 11 QMR, 129
odds ratio, 73 QR method, 176
one-hot-encoding, 145 quadratic programming, 94, 113
one-versus-all, 26 quadratic programming, equality con-
one-versus-rest, 26 strained, 114
optimization, 35 quadratic programming, general form,
optimization problem, 35, 36 113, 131
optimize, in Python, 15 quadratic programming, inequality-
ordinal features, 145 constrained, 134, 136
orthogonal linear transformation, 158 quasi-Newton methods, 52
outlier problem, K-Means, 227
outliers, 239 radial basis function, 97
output layer, 273 random decision forests, 107
overfitting, 80, 147 random forests, 107
range-space approach, 121
PAM algorithm, 228 rank-one matrix, 53, 184
partitional clustering, 215 Rayleigh quotient, 159
PCA, 158 reachable points, 239
Peppers image, 178 rectified linear units, 301
peppers_SVD.m, 178 rectifier, 72
perceptron, 17, 20, 270 reduced Hessian, 114
permutation feature importance, 153, 155 reduced KKT system, 122
permutation matrix, 124 reducible matrix, 125
perturbed complementarity, 138 regression analysis, 4
polar decomposition, 162 regression example, 8
pooling, 297 regular splitting, 128–130
pooling layers, 297 regularization, 81
positive definite, 115 ReLU, 301
positive semidefinite, 115 representation learning, 266
positive semidefinite matrix, 159 ridge regression, 148
INDEX 321

right singular vectors, 161, 164 SPD, 130

risk, 10 spectral radius, 124
Rosenbrock function, 13, 37 spectrum, 124
rosenbrock_opt.py, 14 sphering transformation, 160
rotational symmetry, 72 standard logistic sigmoid function, 71
Run_network.py, 280 standardization, 30, 81, 146
Run_network3.py, 302 steepest descent, 38
steepest descent method, 36
scatter, 182 step length, 28, 38, 79
Schur complement, 121, 139 stochastic gradient descent, 29, 31, 56, 57,
Schur product, 288 277
score matrix, 159 stochastic gradient method, 56
search direction, 28 stride length, 295
secant condition, 65 strongly connected, 126
secant equation, 53 sufficient decrease condition, 44
self-organization map, 244 sum of squared errors, 28
semidefinite, 115 sum-squared-error, 76
sequential backward selection, 152 summary of SVM, 95
Sequential Minimal Optimization, 89 supervised learning, 7
sequential minimal optimization, 100 support vector machine, 25, 82
set of blocking constraints, 133 support vectors, 90
SGD, 56 surrogate function, 41
shared bias, 295 SVD theorem, 164
shared weights, 295 SVD, algebraic interpretation, 166
Sherman-Morrison formula, 54 SVD, geometric interpretation, 168
sigmoid function, 71, 272 SVM summary, 95
sigmoid neural networks, 272 Sylvester’s law of inertia, 118
sigmoid neuron, 271 symmetric factorization, 119
sigmoid neurons, 272 symmetric positive definite, 130
silhouette coefficient, 258
silhouette plots, 263 Taylor’s Theorem, with integral remain-
similarity function, 97 der, 39
similarity matrix, 200, 256 the test of time award, 238
single linkage, 233 Tikhonov regularization, 148
singular value decomposition, 118, 160, transforming null-space iteration, 129
161, 164 transforming range-space iteration, 129,
singular values, 161, 164 130
slack variable, 91, 134 translation invariance, 295, 300
SMO, 89, 100 truncated score matrix, 162, 197
soft-margin classification, 91
SoftmaxLayer, 301 unbiasedness property, 60
softplus function, 72 underfitting, 80
SOM, 244 unit round-off error, 177
sparse model, 150 unsupervised learning, 12, 208
sparsity, 150 upper bidiagonal matrix, 176
322 INDEX

variable selection, 147 whitening transformation, 160

variance, 80 William of Occam, 11
vectorization, 32 winner-takes-all neuron, 246
Voronoï tessellation, 251 winning neuron, 246
Voronoi iteration, 218 within-class scatter, 182, 183, 186
within-class scatter matrix, 183
Ward’s method, 237 within-class variability, 182
Ward’s minimum variance, 234 wrapper methods, 148