I©;.

•r-
i
1 T
I
i
V
t
f:
#
a 19
\
L \ L
S'?'
J
-t p?
M 3 *5»
FiKa
m %
fs k m.
mm ’y
•r/f/.
a/
■rr
Q !^t/
.ilL^
to IJ5 (.
fM ■K
wm .y/ ( s
0 %
m
u
■\ Al /K
$
( A /m
/
"V?
•)>5,
1^ V.
A /
/d
/
S3liVllil]HlVllilQ]llddVNV
^ 9NINIIV]] IMM
MACHINE LEARNING

An Applied Mathematics Introduction

First published 2019 by Panda Ghana Publishing
pandaohana.com

adminSpandaohana.com

First printing 2019

©2019 Paul Wilmott

All rights reserved. This book or any portion thereof may not be reprinted
or reproduced or utilized in any form or by any electronic, mechanical or
other means, now known or hereafter invented, including photocopying,
scanning or recording, or in any information and retrieval system, without
the express written permission of the publisher.
The publisher and the author of this book shall not be liable in the event of
incidental or consequential damages in connection with, or arising out of
reference to or reliance on any information in this book or use of the
suggestions contained in any part of this book. The publisher and the
author make no representation or warranties with respect to the accuracy
or completeness of the contents of this book and specifically disclaim all
warranties, including without limitation warranties of fitness for a particular
purpose. The methods contained herein may not be suitable for every
situation. The information in this book does not constitute advice.

ISBN 978-1-9160816-0-4

Library of Congress Control Number: 2019938983

MACHINE LEARNING

An Applied Mathematics
Introduction

First Edition

Paul Wilmott
WWW.wilmott.com

Panda Ghana Publishing

Other books by Paul Wilmott

Option Pricing: Mathematical Models and Computation. (With

J.N.Dewynne and S.D.Howison.) Oxford Financial Press (1993)
Mathematics of Financial Derivatives: A Student Introduction. (With
J.N.Dewynne and S.D.Howison.) Cambridge University Press (1995)
Mathematical Models in Finance. (Ed. with S.D.Howison and F.P.Kelly.)
Chapman and Hall (1995)
Derivatives: The Theory and Practice of Financial Engineering. John Wiley
and Sons(1998)
Paul Wilmott On Quantitative Finance. John Wiley and Sons (2000)
Paul Wilmott Introduces Quantitative Finance. John Wiley and Sons
(2001)
New Directions in Mathematical Finance. (Ed. with H.Rasmussen.) John
Wiley and Sons (2001)
Best of Wilmott 1. (Ed.) John Wiley and Sons (2004)
Exotic Option Pricing and Advanced Levy Models. (Ed. with W.Schoutens
and A. Kyprianou) John Wiley and Sons (2005)
Best of Wilmott 2. (Ed.) John Wiley and Sons (2005)
Frequently Asked Questions in Quantitative Finance. John Wiley and Sons
(2006)
Paul Wilmott On Quantitative Finance, second edition. John Wiley and
Sons (2006)
Paul Wilmott Introduces Quantitative Finance, second edition. John Wiley
and Sons (2007)
Frequently Asked Questions in Quantitative Finance, second edition. John
Wiley and Sons (2009)
The Money Formula: Dodgy Finance, Pseudo Science, and How
Mathematicians Took Over the Markets. (With David Orrell). John Wiley
and Sons (2017)
To JRO, from your fearless student
Contents

Prologue XI

1 Introduction 1
1,1 The Topic At Hand 2
1.2 Learning Is Key 3
1.3 A Little Bit Of History 4
1.4 Key Methodologies Covered In This Book 6
1.5 Classical Mathematical Modelling . . . . 9
1.6 Machine Learning Is Different 11
1.7 Simplicity Leading To Complexity . . . . 12

2 General Matters 17
2.1 Jargon And Notation 17
2.2 Scaling 18
2.3 Measuring Distances 19
2.4 Curse Of Dimensionality . . . . 20
2.5 Principal Components Analysis . 21
2,6 Maximum Likelihood Estimation 22
2.7 Confusion Matrix 26
2.8 Cost Functions 28
2.9 Gradient Descent 33
2.10 Training, Testing And Validation 35
2.11 Bias And Variance 37
2.12 Lagrange Multipliers 44
2.13 Multiple Classes 45
2.14 Information Theory And Entropy 47
2.15 Natural Language Processing . . 50
2.16 Bayes Theorem 51
2,17 What Follows 53

VI I
VIII CONTENTS

3 K Nearest Neighbours 55
3.1 Executive Summary 55
3.2 What Is It Used For? 55
3.3 Flow It Works 56
3.4 The Algorithm 58
3.5 Problems With KNN 58
3.6 Example: Heights and weights 59
3.7 Regression 62

4 K Means Clustering 65
4.1 Executive Summary 65
4.2 What Is It Used For? . 65
4.3 What Does K Means Clustering Do? . . 67
4.4 Scree Plots 71
4.5 Example: Crime in England, 13 dimensions 72
4.6 Example: Volatility 75
4.7 Example: Interest rates and inflation . . . 77
4.8 Example: Rates, inflation and GDP . . . 80
4.9 A Few Comments 81

5 NaVve Bayes Classifier 83

5.1 Executive Summary . . . . 83
5.2 What Is It Used For? . . . . . 83
5.3 Using Bayes Theorem . . . 84
5.4 Application Of NBC . . . . 84
5.5 In Symbols 85
5.6 Example: Political speeches 86

6 Regression Methods 91
6.1 Executive Summary 91
6.2 What Is It Used For? 91
6.3 Linear Regression In Many Dimensions 92
6.4 Logistic Regression 93
6.5 Example: Political speeches again . . 95
6.6 Other Regression Methods 96

7 Support Vector Machines 99

7.1 Executive Summary . . . . 99
7.2 What Is It Used For? . . . 99
7.3 Hard Margins 100
7.4 Example: Irises 102
7.5 Lagrange Multiplier Version 104
7.6 Soft Margins 106
7.7 Kernel Trick 108
CONTENTS IX

8 Self-Organizing Maps 113

8.1 Executive Summary 113
8.2 What Is It Used For? 113
8.3 The Method 114
8.4 The Learning Algorithm 116
8.5 Example: Grouping shares 119
8.6 Example: Voting in the House of Commons 124

9 Decision Trees 127

9.1 Executive Summary 127
9.2 What Is It Used For? 127
9.3 Example: Magazine subscription 129
9.4 Entropy 134
9.5 Overfitting And Stopping Rules . 137
9.6 Pruning 137
9.7 Numerical Features 138
9.8 Regression 139
9.9 Looking Ahead 144
9.10 Bagging And Random Forests 145

10 Neural Networks 147

10.1 Executive Summary 147
10.2 What Is It Used For? 147
10.3 A Very Simple Network 147
10.4 Universal Approximation Theorem 149
10.5 An Even Simpler Network 150
10.6 The Mathematical Manipulations In Detail . . . . . 151
10.7 Common Activation Functions 154
10.8 The Goal 156
10.9 Example: Approximating a function 157
10.10 Cost Function 158
10.11 Backpropagation 159
10.12 Example: Character recognition 162
10.13 Training And Testing 164
10.14 More Architectures 168
10.15 Deep Learning 170

11 Reinforcement Learning 173

11.1 Executive Summary 173
11.2 What Is It Used For? 173
11.3 Going Offroad In Your Lamborghini 400 GT 174
11.4 Jargon 175
11.5 A First Look At Blackjack 176
11.6 The Classical MDP Approach In O&Xs . . 177
X CONTENTS

11.7 More Jargon 179

11.8 Example: The multi-armed bandit 180
11.9 Getting More Sophisticated 1 . . . 183
11.10 Example: A maze 186
11.11 Value Notation 190
11.12 The Bellman Equations 192
11.13 Optimal Policy 193
11.14 The Role Of Probability 194
11.15 Getting More Sophisticated 2 . . . 195
11.16 Monte Carlo Policy Evaluation . . 195
11.17 Temporal Difference Learning . . . 198
11.18 Pros And Cons: MC v TD . . . . 200
11.19 Finding The Optimal Policy . . . . 200
11.20 Sarsa 201
11.21 Q Learning 203
11.22 Example: Blackjack 204
11.23 Large State Spaces 215

Datasets 217

Epilogue 221

Index 223
Prologue

This book aims to teach you the most important mathematical foundations
of as many machine-learning techniques as possible. I also leave out what I
judge to be the unnecessary boring bits. It is also meant to be orthogonal
to most other books on and around this subject.
If you've already bought this book then you may want to skip the next
few paragraphs. In them I explain who the book is for. If you have bought
the book and it turns out you are the wrong audience then... Oops.
There are several different types of reader of machine-learning textbooks.
And different quantities of those.
Judging by what's available on Amazon I reckon that there are an awful
lot of people out there who Just want to see the relevant lines of code. Often
in Python or R. There are plenty of books that wil l show you a surprisingly
compact coded-up representation of the solution of a problem. I'm not going
to worry about these people. You are well catered for already.
Then there are quite a lot of books that describe the uses of machine
learning techniques. But this time not much code, and very elementary
mathematics. These are great for getting an overview of the subject. But
they lack in detail. You can spot them by the dodgy typeface they use
for their mathematics. And that they are self published, although there is
absolutely nothing wrong with that at all.
And then there are the specialist books. Each devoted to a narrow field,
but extremely deep. I can't fault their typefaces. But they are a little bit
scary. And usually very, very expensive.
No, I'm not trying to compete with any of the above.
I am aiming for a small audience, not well catered for. An audience,
probably shrinking as I type. An audience of applied mathematicians. People
who like doing mathematics, solving problems, perhaps old hands at classical
techniques and keen to know more about this new data-driven revolution
that they are on the edges of. You know who you are. Yes, you at the back,
don't try sneaking out, sit down here at the front and let’s learn this subject
together.. .

XI
XI I PROLOGUE

A Word From The Machine

But first, I think we need to hear what The Machine has to say about
this bookT (It is very insightful. I particularly liked the endorsement at the
end.)

These volumes are meant to be in multiple sections. You can read them
in this book at each of the four different places, from beginning to end. The
main point you want to make here is that both the book and the text on as
many of the techniques that you will use will be well-understood. But what
will be done to make them all accessible, and that they won’t be too hard to
understand? In fact, each of the techniques will be quite easy to understand,
and the material will be very well organized so that you can do what you
want in the most short amount of time (i.e., in less than 40 pages). A few
basic examples of different types of machine-learning can be found in the
book, but I'll focus on the main points of the book. If you're interested in
reading more about this subject, then see my book.

Acknowledgements

I would like to thank my co-author on The Money Formula (available in

all good book shops), David Orrell, for his fresh pair of eyes in typo spotting.
And Liam Larkin, the designer of my magazine (“The world's most expensive
magazine," according to Esquire), for the cover.
I am especially indebted to Thijs van den Berg who got me into this
subject. He also made many suggestions for additions and corrections to
this book, 82.1% of which I implemented. He and I give training courses
on machine learning, and so I'd also like to thank the many delegates on
those courses for all their comments and questions that helped me to explain
things as well as possible.
Finally, I must mention my wife, my children, Her Majesty's Revenue and
Customs, the IRS, “Game of Thrones," Sky News and Brexit, without whom
this book would have been finished in half the time.

'^Text generated by https://colab.reseaxch.google.com/github/

ilopezir/gpt-2/blob/master/gpt-2-playground_.ipynb
 XIII

About the Author

Professionally speaking...
Paul Wilmott studied mathematics at St Catherine's College, Oxford,
where he also received his D.Phil. He is the author of Paul Wilmott In
troduces Quantitative Finance (Wiley 2007), Paul Wilmott On Quantitative
Finance (Wiley 2006), Frequently Asked Questions in Quantitative Finance
(Wiley 2009), The Money Formula: Dodgy Finance, Pseudo Science, and
Flow Mathematicians Took Over the Markets (with David Orrell) (Wiley
2017) and other financial textbooks. He has written over 100 research ar
ticles on finance and mathematics. Paul Wilmott was a founding partner
of the volatility arbitrage hedge fund Caissa Capital which managed $170
million. His responsibilities included forecasting, derivatives pricing, and risk
management.
Paul is the proprietor of www.wilmott.com, the popular quantitative
finance community website, and the quant magazine Wilmott. He is the cre
ator of the Certificate in Quantitative Finance, cqf.com, and the President
of the CQF Institute, cqfinstitute.org.

On the other hand...

Paul was a professional juggler with the Dab Hands troupe, and has been
an undercover investigator for Channel 4. He also has three half blues from
Oxford University for Ballroom Dancing. At the age of 41 he finally won a
sandcastle-building competition. He makes his own cheese. Its flavour has
been described as “challenging.”
Paul was the first man in the UK to get an online divorce. He was
an expert on a TV show, tasked with forecasting a royal baby name and
the winner of the Eurovision Song Contest among other things. He got
everything wrong.
He played bridge for his school's D team. There was no E team.
And he plays the ukulele. Obviously.
Chapter 1

Introduction

Hi, my name is Paul Wilmott. (This chapter is called Introduction, after

all.) I am an applied mathematician. For many years I worked in academia,
researching this and that, supervising students, writing papers for learned
journals. My early research was in the field of applicable/industrial math
ematics. That meant speaking to people from industry and turning their
problems, often about some physical process, into mathematical models.
These mathematical models could then be solved to give some insight into
the problems they had posed. This had many advantages over, say, building
experiments, not least that we were much cheaper. After a few years of this
one builds up a skill set of techniques and principles that help with modelling
Just about anything. Sometimes you’d have physical laws, such as conserva
tion of mass, on which to base your models, although often not. If you have
such solid physical laws then this should lead to a quantitatively good model.
Otherwise you might end up with a model that is only qualitatively good.
The latter could be thought of as a toy model. (Ok, I guess sometimes the
models were even bad, but we tended not to publish those. Usually.)
I'm telling you all this because it very much influences the way that I
see this relatively young field of machine learning. I bring to it some worries
and some scepticism. I will be sprinkling my concerns throughout this book,
although I do hope not to spoil the party too much. I'll come back to
this a bit later in this chapter. And I am going to be contrasting classical
mathematical modelling and machine learning. For two fields that are trying
to achieve the same goal it's surprising how different they are.
I also bring my philosophy as an applied mathematician who likes to
solve real-world problems. This means writing down some mathematical
model as quickly as possible without having to worry too much about proving
everything at every step of the way. Please note that just because something
is not proved does not mean that it is wrong. It just means getting to the
meat as quickly as possible without having to eat the vegetables. (Having

1
2 CHAPTER 1. INTRODUCTION

said all that I am sure that some readers will take it upon themselves to find
as many errors in this book as they can. If you do find any then email me
at paulOwilmott.com in the first instance before reviewing me on Amazon.
I'm very approachable.)
This book does not include any code whatsoever. This is deliberate for
two reasons. Mainly it's because I am a useless programmer. But also the
upside is that I do have to teach the methods with enough detail that you
can implement them yourself. I cannot take any shortcuts such as giving
a no-math description of a technique followed by saying "Now write these
two lines of code, press enter, and look at the pretty pictures." There are
many programming packages you can use to implement the methods that
I describe and you don't need to know what they are doing to use them.
That's fine. It's like my feeble understanding of a carburettor doesn't stop me
enjoying my Jensen Interceptor Convertible. But equally if I did understand
the carburettor it would probably save me quite a bit of time and expense.
Finally, there are plenty of examples in this book, almost always using
real data. But this is not a research-level tome, and so the examples are for
illustration only. If it were a book describing the cutting edge of research
then there would be a heck of a lot more cleaning and checking of the data
and a lot more testing and validation of the models. But there'd also be
a lot less explanation of the basics, and perhaps less mentioning of where
things go wrong.
But now let's actually talk about the topic at hand.

1.1 The Topic At Hand

A definition from Wikipedia:

Machine learning is a field of computer science that uses statistical tech

niques to give computer systems the ability to ‘learn' (e.g., progressively
improve performance on a specific task) with data, without being explicitly
programmed.

Where to start. .. with computers? Or with statistics? Or data?

This is not the place for a detailed history of machine learning. I could
never do as well as google. (I also have this personal prejudice that any
mathematician who knows about the history of their subject isn't a proper
mathematician.) If you want some pithy paragraphs for a dinner-party con
versation (before changing the subject to something more spicey, like religion
or politics) then.. .
• The field of probability and statistics goes back to 1654, and a question
of gambling debated by the French mathematicians Blaise Pascal and
1.2. LEARNING IS KEY 3

Pierre de Fermat. Their problem was to figure out how to allocate a bet
in a game of dice (called Points) if the game were to be suspended part
way through, one side of the game doing better than the other. Thus
was invented/discovered the concept of mathematical expectations.

• Who invented the first calculator is disputed. It might have been

Pascal again or it might have been Wilhelm Schickard around 1623.
Either way it was little more than an adding machine. However the
first programmable computer was invented, but never built, by Charles
Babbage, often called the father of the computer, in 1837. His com
puter was intended to use cards with holes punched in them for the
programming. Such a system of programming existed into surprisingly
recent times. To readers of a certain age this will be a familiar, if
painful, memory.

• However, before these came data collection. And machine learning

uses data. Lots and lots of it. Historically data was often in the form
of surveys. The famous Domesday Book of the late 11th Century was a
survey, ordered by William the Conqueror, of England and Wales. One
of its purposes was to aid in the collection of taxes. Plus pa change,
plus c’est la mime chose.

Take data and statistical techniques, throw them at a computer and what
you have is machine learning.

1.2 Learning Is Key

Except that there's more to it than that. And there's a clue in the
title of the subject, in the word "learning.” In traditional modelling you
would sit down with a piece of paper, a pencil and a single malt and. . .tell
the algorithm everything you know about, say, chess. You’d write down
some code like “IF White Queen something something Black Bishop THEN
something something." (I haven't played chess for forty years so apologies for
a lack of precision here.) And you’d have many, many lines of IF, AND and
OR, etc. code telling the programme what to do in complex situations, with
nuances in positions. Or .. . you'd write down some differential equation that
captures how you believe that inflation will respond to a change in interest
rates. As inflation rises so a central bank responds by increasing interest
rates. Increasing interest rates causes.. .
You might eyeball the data a bit, maybe do a smal l amount of basic
statistics, but mainly you’d just be using your brain.
Whatever the problem you would build the model yourself.
4 CHAPTER 1. INTRODUCTION

In machine learning your role in modelling is much more limited. You will
decide on a framework, whether it be a neural network or a support vector
machine, for example, and then the data does the rest. (It won't be long
now before the machine even does that for you.) The algorithm learns.

1.3 A Little Bit Of History

As I said. I'm not going to give you much history in this book. That
would be pointless, history is always being added to and being reassessed,
and this field is changing very rapidly. But I will tell you the two bits of
machine-learning history that I personally find interesting.
First let me introduce Donald Michie. Donald Michie had worked on
cyphers and code cracking with his colleague Alan Turing at Bletchley Park
during the Second World War. After the war he gained a doctorate and in the
early 1960s, as a professor in Edinburgh, turned his attention to the problem
of training — a word you will be hearing quite a lot of — a computer to play
the game of Noughts and Crosses, a.k.a. Tic Tac Toe. Well, not so much
a computer as an array of matchboxes, sans matches. Three hundred and
four matchboxes, laid out to represent the stages of a game of Noughts and
Crosses. I won't explain the rules of O&Xs,just like further down I won't be
explaining the rules of Go, but each stage is represented by a grid on which
one player has written an X alternating with the other player writing an O
with each player's goal being to get three of their symbols in a row. (Damn,
I did explain the rules. But really do not expect this for Go. Mainly because
I don't fully understand the game myself.)

Figure 1.1: A matchbox representing a state in Noughts and Crosses.

In Figure 1.1 we see a sketch of what one of Professor Michie's match

boxes might have looked like. In this it is X's turn to play. There is clearly
a winning move here. (Actually two of them.) I know that, and you know
that, but if the computer were playing would it also know? Traditionally one
might programme a computer with all the possible states of the game and
the best next move in each case. That's doable for O&cXs but most definitely
is not with any non-trivial game. What Professor Michie did instead was to
1.3. A LITTLE BIT OF HISTORY 5

fill each of his 304 matchboxes with coloured beads, each colour representing
one of the empty cells in the grid. (He took advantage of symmetries or there
would have been even more than 304 matchboxes.) When it's our move we
take the matchbox representing the current state, shake it, and remove a
random bead. It’s colour tells us where to place our X. This is followed by
our opponent's play. The game continues, and we move from state to state
and matchbox to matchbox. And if we win then each matchbox used in that
game gets rewarded with extra beads of its chosen colour, and if we lose then
a bead of that colour gets taken away. Eventually the matchboxes fill up with
beads representing the most successful action at each state. The machine,
and Michie called it MENACE, for Machine Educable Noughts And Crosses
Engine, has learned to play Noughts and Crosses. There’s some jargon in
the above: Training; State; Action; Reward; Learn. We'll be hearing more
of these later. And we shall return to Donald's matchboxes in the chapter
on reinforcement learning.
The next thing that I found interesting was the story of how Google
DeepMind programmed a computer to learn to play Go. And it promptly
went on to beat Lee Sedol, a 9-dan professional Go player. (No, I didn't
know you could become a 9th dan by sitting on your butt either. There's
hope for us all.) AlphaGo used a variety of techniques, among them neural
networks. Again it learned to play without explicit programming of opti
mal play. Curiously not all of that play was against human opponents... it
also played against itself. To show how difficult Go is, and consequently
how difficult was the task for Google DeepMind, if you used matchboxes to
represent states a la Donald Michie then you’d need a lot more than 304
of them, you’d probably have to fill the known universe with matchboxes.
I would strongly recommend watching the creatively titled movie "The Al
phaGo Movie" to see a better description than the above and also to watch
the faces of everyone involved. Lee Sedol started moderately confident, be
came somewhat gutted, and finally resigned (literally and psychologically).
Most of the programmers were whooping for Joy as the games progressed.
Although there was one pensive-looking one who looked like he was thinking
"Oh.. . my.. . God.. . what.. . have.. . we.. . done?"
At times the computer made some strange moves, moves that no profes
sional Go player would have considered. In the movie there was discussion
about whether this was a programming error, lack of sufficient training, or a
stroke of computer genius. It was always the last of these. It has been said
that the game of Go has advanced because of what humans have learned
from the machine. (Did a chill Just go down your spine? It did mine.)
In this book I am going to guide you through some of the most popular
machine-learning techniques, to the level at which you can start trying them
out for yourself, and there'l l be an overview of some of these for the rest of
this chapter.
6 CHAPTER 1. INTRODUCTION

The book is not meant to be rigorous, it is meant to get you interested

and sitting at your PC with a spreadsheet (or whatever) open. It's the book
I wish I’d read when starting out in this field.
I'll also give some warnings. The warnings are inspired by how easy it is
to get carried away by machine learning, because it's so much gosh-darned
fun! Overconfidence is not good when you are dealing with a topic that
judging by current evidence seems to be taking over the planet.

1.4 Key Methodologies Covered In This Book

There are three main categories of machine learning. Two of these cate
gories concern how much help we give the machine. And the third is about
teaching a machine to do things.

Principal categories

• Supervised Learning: In supervised learning you are given both inputs

and outputs. The input would be a vector, i.e. typically more than one
variable. For example, we start with pictures of dogs that have been
digitized and turned into vectors representing the colours in the pixels.
These pictures have each been classified as Flound, Toy, Terrier, etc.
We train our algorithm on these pictures and then we would want it to
correctly classify any new picture we give it. We can also use supervised
learning for regression, predicting numerical values for outputs for any
new data points we input (e.g. input temperature and time of day, and
predict electricity usage). Supervised learning includes many different
algorithms, some mentioned below.

• Unsupervised Learning: Unsupervised learning is when the data is

not labelled. This means that we only have inputs, not outputs. In
a sense the algorithm is on its own. The algorithm finds relation
ships or patterns for you. We show the computer digitized pictures
of dogs, and the computer groups or clusters them together according
to whatever features it deems most important. Perhaps it will come
up with Hound, Toy, Terrier,... or perhaps black, brown, white,...or
something completely different that humans don't even notice. Unsu
pervised learning might at first seem strange, but you’ll understand it
if I said to you "Here are the contents of my desk drawer. Please sort
them out." Unsupervised learning includes many different algorithms,
some mentioned below.
1.4. KEY METHODOLOGIES COVERED IN THIS BOOK 7

• Reinforcement Learning: In reinforcement learning an algorithm

learns to do something by being rewarded for successful behaviour
and/or being punished for unsuccessful behaviour. (And we're back
with dogs again!) The above-mentioned MENACE was a simple ex
ample of this, the rewards and punishments being the addition or sub
traction of beads to or from the matchboxes. Computers learning to
play board games or video games from the 1980s seem to get a lot of
publicity. That would usually involve reinforcement learning.

Principal techniques

I’m going to describe many of the most important techniques within

the above principal categories. And the order I've chosen to explain them
is governed mainly by their mathematical complexity. Most of them can
be implemented in just a few lines of Python code, but my Job is not to
give you those few lines but to explain what that code is doing. Below are
the supervised and unsupervised techniques we'l l be seeing. Reinforcement
learning is treated separately, right at the end of the book.
I personally don't necessarily see some of these techniques as machine
learning, they share too many similarities to classical statistical methods.
But I’ve included them here anyway because other people think they are
machine learning and they do share the fundamental characteristic of using
lots and lots of data.

• K Nearest Neighbours: K nearest neighbours is a supervised-learning

technique in which we measure distances between any new data point
(in any number of dimensions) and the nearest K of our already-
classified data and thus conclude to which class our new data point
belongs. A silly example: We are at a party, and out of the five people
standing closest to us three are wearing glasses, so we are probably
glass wearers too. The method can also be used for regression.
• K Means Clustering: K means clustering is an unsupervised-learning
technique. We have lots of data in vector form (representing many di
mensions of information about each point), and we have K, a number,
points or centroids in the space. Each data point is associated with its
nearest centroid, and that defines to which category each data point
belongs. Then we find the optimal position for these centroids that
gives us the best division into the K classes. And finally we play around
with the number K to see what effect it has on the division of the data
into categories.

• NaTve Bayes Classifier: Naive Bayes classifier is a supervised-learning

technique that uses Bayes Theorem to calculate the probability of new
8 CHAPTER 1. INTRODUCTION

data points being in different classes. The naive bit refers to the
assumptions made, which are rarely true in practice but that doesn't
usually seem to matter.

• Regression Methods: Regression methods are supervised-learning

techniques that try to explain a numerical dependent variable in terms
of independent variables. You'll probably know of linear regression at
least. But you can go much further with more complicated types of
regression.

• Support Vector Machines: A support vector machine is another

supervised-learning technique, one that divides data into classes ac
cording to which side of a hyperplane in feature space each data point
lies.

• Self-organizing Maps: A self-organizing map is an unsupervised-

learning technique that turns high-dimensional data into nice, typ
ically two-dimensional, pictures for visualizing relationships between
data points. Imagine a chess board and into the squares you put data
points with similar characteristics. You don’t specify those character
istics, they are found as part of the algorithm.

• Decision Trees: This is a supervised-learning technique. A decision

tree is just a flowchart. “How many legs does it have? Two, four,
more than four?” Four. Next, “Does it have a tail? Yes or no." Yes.
And so on. Like a game of 20 Questions. But can the machine learn
and improve on how humans might classify? Is there a best order in
which to ask the questions so that the classifying is done quickly and
accurately? It can also be used for regression.

• Neural Networks: A neural network (sometimes with artificial in

front) is a type of machine learning that is meant to mimic what
happens in the brain. An input vector is transformed, typically by mul
tiplying by a matrix. So far, so linear. Then the result is put into some
non-linear activation function. That's one layer. But that result then
usually goes into another layer, where it undergoes similar transforma
tions, and so on. In this form it's really Just a non-linear function of
the original data. But it can get more complicated by having the data
not Just going in one direction but also looping back on itself. The
parameters in the neural network are found by training. It can be used
for either supervised or unsupervised learning.
1.5. CLASSICAL MATHEMATICAL MODELLING 9

1.5 Classical Mathematical Modelling

I want to explain, briefly, how machine learning is different from classical

mathematical modelling. And I must stress that I'm not talking about statis
tical modelling or econometrics which share some similarities with machine
learning. This is to some extent a personal view of my experience in applied
mathematical modelling.
There's a well-recognised, and well-trodden, path that(we) mathematical
modellers go along when faced with a new problem. It goes something like
this...

First decide on your variables

Variables come in two main types, independent and dependent. Indepen

dent variables represent the domain in which your problem is specified. A
common group of independent variables would be spatial and time coordi
nates, X, y, z, and t.
Contrast this with the dependent variables, these are the things you are
solving for. You might have a problem in heat flow in which case your depen
dent variable would probably be temperature. As another example consider
the oscillation of a 'cello string (guitar string for you in the cheap seats).
The dependent variable would be distance by which the string is perturbed
from rest. (So here you’d have a dependent variable that is a distance from
rest, but also an independent variable that's a distance, distance along the
string, as well as time.)
You really want the minimum of variables to specify the system and its
behaviour. Einstein said something along the lines of "Make things as simple
as possible, and no simpler." That's probably Rule Number 1 of modelling.
Rule Number 2 would be “If you are going to make things more complicated
then do so one new feature at a time.” Already that's a big difference to
machine learning in which one tends to throw in everything at the start,
including the kitchen sink.
The goal at this stage in classical mathematical modelling is to get to
the first minimally interesting non-triviai problem.

Figure out what are the main drivers in your system

What drives your system? Are there forces pushing and pulling some
thing? Are molecules bouncing off each other at random? Do buyers and
sellers move the price of milk up and down?
If you are lucky you'l l have some decent principles to base your model
10 CHAPTER 1. INTRODUCTION

on, such as conservation of something. The 'cello string moves according to

Newton’s second law, the force caused by the bend in the string makes the
string move.
Or maybe the physics is too complicated but the statistics is robust, such
as in a coin toss.

(There's also the category of simply feeling right, such as the inverse
square law of gravity.)
But maybe you have to fall back on some common sense or observation.
To model the dynamical system of lions and gazelles (the former like to
eat the latter) then you would say that the more lions there are the more
they breed, ditto for gazelles. And the more gazelles there are the more
food for lions, which is great for the lions but not so much for the gazelles.
From this it is possible to write down a nice differential-equation model.
The 'cello-string model will give accurate behaviour for the string, but while
the lion-gazelle model is usefully explanatory of certain effects it will not
necessarily be quantitatively accurate. The latter is a toy model.
Your model will probably need you to give it values for some parameters.
Again if you are lucky you might be able to get these from experiment; the
acceleration due to gravity, or the viscosity of a fluid. And perhaps those
parameters will stay constant. But maybe the parameters will be difficult to
measure, unstable, or perhaps not even exist, the volatility of a stock price
for example.
These are issues that the mathematical modeller needs to address and
will make the difference between a model with reliable outputs and one that
is merely useful for explanation of phenomena.

Decide on your type of mathematics

There are many branches of mathematics, some are more easily employed
than others when it comes to practical problems.
Discrete mathematics, or continuous mathematics? Will you be using
calculus, perhaps ordinary or partial differential equations? Or perhaps the
model is going to use probabilistic concepts. If you are trying to figure out
how to win at Blackjack then clearly you'll be working with discrete values
for the cards and probability will play a major role. As will optimization.
Change to poker and game theory becomes important.
Mathematicians do suffer from the classic “If all you have is a hammer
every problem becomes a nail." This can affect the evolution of problem
solutions, as the problem becomes known as, say, a problem in subject X,
when subject Y might also be useful.
1.6. MACHINE LEARNING IS DIFFERENT 11

And the answer is...

Setting out a problem and a model is nice. But you’ll almost always want
to find its solution. If lucky, again, you'l l get a formula using a pencil and pa
per. More often than not, especially if it’s a real-world, often messy, problem
then you’ll have to do some number crunching. There are many numerical
methods, many for each branch of mathematical modelling. Numerical so
lution is different from modelling, don’t confuse the two. Numerical analysis
is a subject in its own right.

1.6 Machine Learning Is Different

Machine learning could not be more different if it tried. Almost nothing

in the above carries over to machine learning.
One big difference is the role of data. In classical mathematical modelling
you might not have any data. Say you have a problem in fluid mechanics then
you build your model on conservation of mass and momentum (I say "your
model," but of course it’s due to Euler, Navier and Stokes). You’d come
up with a model while sitting in your office with nothing more than a pencil
and some paper. And the whisky. There would be but a few parameters
(fluid density and viscosity) and you are done. You then have to solve for
your particular geometry. If this were a machine-learning problem then you
show the machine a great deal of data, perhaps in the form of movies of flow
past various shapes, and let it do its thing. All being well the machine would
learn and somewhere deep inside it would have indirectly hit on something
like the Navier-Stokes equation. You certainly wouldn’t have specified any
drivers, conservation laws, etc.
Sherlock Holmes would no doubt embrace machine learning, "It is a
capital mistake to theorise before one has data."
In classical modelling, parameters often have some meaning or nice inter
pretation. They certainly do if they represent something physical. But how
can you interpret parameters in the middle hidden layer of a neural network?
They are Just numbers that happen to give you the best fit of your model to
data.

And the role of type of mathematics would become the choice of machine
learning scheme. Is the problem best approached via supervised or unsuper
vised learning? Would it be a self-organizing map or K means clustering
that gave the most reliable results?
Very often the building blocks of a machine-learning algorithm are very
simple. A neural network might involve matrix multiplications and a few
simple non-linear functions. Simple models can lead to complex behaviour.
But is it the right simple model and a quantitatively useful behaviour?
12 CHAPTER 1. INTRODUCTION

1.7 Simplicity Leading To Complexity

Let's remind ourselves about a few simple mathematical concepts that

lead to complex structured behaviour.

Cellular automata

A cellular automaton is typically made up of a grid of cells, each cell

being in one of several states. The cells change state according to some
rules. Although dating back to the 1940s with Stanislaw Ulam and John von
Neumann at the Los Alamos National Laboratory, the most famous cellular
automaton is probably what is known as The Game of Life designed by John
Conway in the 1970s. In this each cell is either alive or dead, and is meant
to represent a population of living creatures. See Figure 1.2 for an example.

Figure 1.2: A snapshot in The Game of Life.

The rules are:

• Any live cell with fewer than two live neighbours dies, as if caused by
underpopulation

• Any live cell with two or three live neighbours lives on to the next
generation.

• Any live cell with more than three live neighbours dies, as if by over
population

• Any dead cell with exactly three live neighbours becomes a live cell,
as if by reproduction
1.7. SIMPLICITY LEADING TO COMPLEXITY 13

A nice video of The Game of Life on a torous can be found here:

https://www.youtube.com/watch?v=lxIeaotWIks.
And a three-dimensional version here:

https://www.youtube.com/watch?v=iiEQg-SHYlg.
I worked on a cellular automaton model in the early 1990s, a model
representing the separation one finds in granular material when they are of
different sizes. This model was created because of the lack of progress we
had made in trying to develop a more classical continuum model.

Fractals

Technically to understand fractals you need to know about Hausdorff

Dimension versus Topological Dimension. But we can get part way there by
considering the length of the coastline of Great Britain. It’s fairly obvious
that the shorter the ruler that you use for measuring then the longer the
measured coastline. The shorter the ruler the more nooks and crannies you
will get into. This is completely different for a classical two-dimensional
shape, such as a circle, for which the length of the boundary converges as
the ruler shrinks. Lewis Fry Richardson measured various coastlines and
found a power law approximation to the measured length. From this follows
the idea of the fractal dimension of a coastline. In the case of the west coast
of Britain he found it to be 1.25, so somewhere between a line and an area.
Simple algorithms can be used to generate complex fractal shapes. Here
is an algorithm for generating the Barnsley Fern, Figure 1.3.

0.00 0.00 X
f\{-c y) = 0.00 0.16 y

0.85 0.04 X 0.00

h{x,y) -0.04 0.85 1.60
y

0.20 -0.26 X 0.00

h{x,y) 0.23 0.22 1.60
y

-0.15 0.28 X 0.00

h{x,y) 0.26 0.24
+
0.44 ■
y

To generate a fern simply choose a starting x and y, say (0,0), and

place a dot at that point. Now pick one of the above four transformations
at random and input the x and y. You pick the first transformation with
probability 1%, the second with probability 85%, and the last two each with
7%. This gives you a new position for a dot. Now take the new x and y and
one of the four transformations.. . repeat until you have filled in the shape
sufficiently.
14 CHAPTER 1. INTRODUCTION

-e-

-3 -2 -1 0 1 2 3

Figure 1.3: Computer-generated fern.

Chaos

And then there's everybody's favourite, chaos. A simple nonlinearity

can lead to extremely complex and unpredictable behaviour despite perfect
determinism. Take for example a basic population model (kno\«n as the
Logistic Map)
Xn+\ — Aa::„(l

where x„ is the number in the population as a function of the generation n.

This can be thought of as a model for population dynamics because it has
two key features: For small populations the size of the next generation is
proportional to the current population, with a growth coefficient of A; If the
population gets too big then the growth rate shrinks because of competition
for resources.
Depending on the size of A you can get converging to zero, converg
ing to some other value, oscillating between two or more values, or chaos.
A primary feature of chaos is sensitive dependence on initial conditions, the
Butterfly Effect. The departure of two paths with initially close initial condi
tions is governed by the Lyapunov Exponent, and is the reason why chaotic
systems are unpredictable too far into the future.
1.7. SIMPLICITY LEADING TO COMPLEXITY 15

See Figure 1.4 for an example plot of Xn with A = 3.7653.

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
0 10 20 30 40 50 60 70 80 90 100

Figure 1.4: Chaos,

And your point is, Paul?

Those last few pages were rather tangential to our topic. But let me
reassure you that you have not been charged for them.
I mention all of the above as both an encouragement and as a warning to
anyone embarking on machine learning. On a positive note we see complex
behaviour arising out of the simplest of models. And some of the machine
learning you will be seeing can be very simple indeed. The building blocks
of basic neural networks are just matrix multiplication and a fairly bland
activation function.
On the other hand once we have represented something quite compli
cated, such as millions of data points, with something simple, how do we
ever know that it’s the right simple model?
I recall a great deal of excitement in my mathematics department in the
late 1980s because a group had discovered they could use dynamical systems
(chaos) to model the foreign exchange markets. As I recall the excitement
soon faded once they learned more about how the currency markets actually
worked, and that their forecasts were essentially useless. (I could tell by
looking at their shoes that they hadn't struck it rich.)
16 CHAPTER 1. INTRODUCTION

Further Reading

If you want to learn about a wide variety of mathematical-modelling

methods then you can't do much better that reading all of Professor Jim
Murray's Mathematical Biology, published by Springer and now in two vol
umes. To date it is the only mathematics book 1 have ever read in bed.
For more about cellular automata see Stephen Wolfram's A New Kind of
Science, published in 2002 by Wolfram Media.
For fractals anything by Benoit Mandelbrot, for example on the financial
markets The (Mis)Behaviour of Markets: A Fractal View of Risk, Ruin and
Reward published by Profile Books.
For chaos there is the classic Chaos: Making a New Science by James
Gleick, published by Penguin.
Chapter 2

General Matters

There are some issues that are common to many branches of machine learn
ing. In this chapter I put these al l together for some brief discussion. More
often than not in later chapters I will simply refer you to this chapter for
further consideration.

2.1 Jargon And Notation

Every technical subject has its own Jargon, and that includes machine
learning. Important Jargon to get us started is as follows. (Every chapter
will introduce more.)
One small complaint here: Some rather simple things I've grown up with
in applied mathematics seem to be given fancy names when encountered in
machine learning. I'll say no more.

• Data (points, sets): To train our machine-learning algorithm we will

need lots of data to input. Each data point could also be called an
example or a sample. If we have a supervised-learning problem each
input (vector) will have an associated output (vector). You can think
of these as the independent and associated dependent variables.

• Feature (vector): A feature is a characteristic of the data. If you are

from statistics then think of it as an explanatory variable. Although in
machine learning we will often have many more explanatory variables
than in classical stats models.

It might be numerical (height of tree 6.3m) or it might be descriptive

(eye colour blue). Often if it is descriptive then you will stil l have to
give it a numerical label in order to do mathematical manipulations.

17
18 CHAPTER 2. GENERAL MATTERS

For example, you might want to group cats into different breeds, and
the features might be length of fur (numerical) or whether or not it
has a tail (descriptive but easily represented by value zero or one). (No
tail? If you are wondering, the Manx cat is tailless.) Each data point
will often be represented by a feature vector, each entry in the vector
representing a feature.

• Classification: We will often be dividing our data into different classes

or categories. In supervised learning this is done a priori. (Is the wash
ing machine reliable or not?) In unsupervised learning it happens as
part of the learning process. Sometimes we represent classes via vec
tors. Classifying sandwiches we might represent ham as (1,0,... ,0),
cheese as (0,1,...,0), and so on. (Although I tend to use column
vectors in this book.) When only one of the entries in the feature
vector is 1 and the rest 0 it is called one-hot encoding.

• Regression: You want to output a numerical forecast? Then you

need to do a regression. This contrasts with the above classification
problems where the result of an algorithm is a class.

I have tried to make my notation consistent throughout this book, and

also to align with common conventions as much as possible.

I have tended to use the superscript to denote the n'*' out of a

total of N data points. That means I would have, say, N flowers to
categorize, and a general flower would have the label n. The class,
output or value of the individual will be

Vectors are bold face, and are column vectors. The vector of features
of the n**’ individual wil l be

I use subscript rn to denote the m* feature out of a total of M features.

A feature might be salary, or number of words in an email.

When I have categories, classes or clusters I will use K to be the total

number of categories, etc. and k for one of them.

I keep jargon to a minimum but hope to include al l that is important.

2.2 Scaling

The first part of many algorithms that you'll be seeing is a transformation

and a scaling of the data. We are often going to be measuring distances
2.3. MEASURING DISTANCES 19

between data points so we must often make sure that the scales for each
feature are roughly the same.
Suppose we were characterising people according to their salary and num
ber of pets. If we didn't scale then the salary numbers, being in the thou
sands, would outweigh the number of pets. Then the number of pets a
person had would become useless, which would be silly if one was trying to
predict expenditure on dog food.
All we have to do is adjust entries for each feature by adding a number
and multiplying by a number so as to make sure that the scales match.
This is easy to do. And there are a couple of obvious ways (a second
is in the parentheses below). With the original unsealed data take one of
the features, say the top entry in the feature vector, measure the mean and
standard deviation (or minimum and maximum) across all of the data. Then
translate and rescale the first entries in all of the feature vectors to have a
mean of zero and a standard deviation of one (or a minimum of zero and a
maximum of one). Then do the same for the second entry, and so on.
Just to be clear we use the same translation and scaling for the same
feature across all data points. But it will be a different translation and scaling
for other features.
This scaling and translation is a common first step in many machine
learning algorithms. I don’t always mention the rescaling at the start of each
chapter so please take it as read.
And don’t forget that when you have a new sample for prediction you
must scale all its features first, using the in-sample scaling.
One small warning though, if you have outliers then they can distort
the rest of the data. For example one large value for one feature in one
sample can, after rescaling, cause the values for that feature for the rest of
the samples to be tiny and thus appear to be irrelevant when you measure
distances between feature vectors. Flow important this effect is will depend
on what type of rescaling you use. Common sense will help determine if and
how you should rescale.

2.3 Measuring Distances

As I Just said, we’l l often be working with vectors and we will want to
measure the distances between them. The shorter the distance between two
feature vectors the closer in character are the two samples they represent.
There are many ways to measure distance.

Euclidean distance The classic measurement, using Pythagoras, just

square the differences between vector entries, sum these and square root.
20 CHAPTER 2. GENERAL MATTERS

This would be the default measure, as the crow flies. It’s the norm.

Manhattan distance The Manhattan distance is the sum of the absolute

values of the differences between entries in the vectors. The name derives
from the distance one must travel along the roads in a city laid out in a grid
pattern. This measure of distance can be preferable when dealing with data
in high dimensions. This is the norm.

Chebyshev distance Take the absolute value of the differences between

all the entries in the two vectors and the maximum of these numbers is the
Chebyshev distance. This can be the preferred distance measure when you
have many dimensions and most of them just aren't important in classifica
CX)
tion. This is the L norm.

Cosine similarity In some circumstances two vectors might be similar if

they are pointing in the same direction even if they are of different lengths.
A measure of distance for this situation is the cosine of the angle between
the two vectors. Just take the dot product of the two vectors and divide by
the two lengths. This measure is often used in Natural Language Processing,
e.g. with Word2vec, mentioned briefly later.

2.4 Curse Of Dimensionality

In any problem involving data in high dimensions in which one is mea

suring distances one has to be careful not to be struck by the curse of
dimensionality. Sounds frightening. Suppose you are working in M dimen
sions, that is your data has M features. And suppose you have N data
points. Having a large number of data points is good, the more the merrier.
But what about number of features?
You might think that the more features you have the better because it
means that you can be more precise in classification of data. Unfortunately
it doesn’t quite work like that. Think how those N data points might be
distributed in M dimensions. Suppose that the numerical data for each
feature is either zero or one, to keep things simple. There will therefore be
2^"^ possible combinations. If N is less than this value then you run the risk
of having every data point being on a different corner of the M-dimensional
hypercube. The consequence of this is that the distances on which you base
your analysis become somewhat meaningless. If everyone is unique then
classification becomes meaningless.
2.5. PRINCIPAL COMPONENTS ANALYSIS 21

There are only seven billion people on the planet but certain companies
are trying to collect so much data on each of them/us that you can imagine
the curse of dimensionality becoming an issue. The solution? Don't collect
so much data, Mark.
In practice things are not usually as bad as all that, this is very much a
worst-case scenario. It is certainly possible that there is some relationship
between features so that your 13-dimensional, say, problem might only be
effectively in six dimensions. Sometimes you might want to consider doing
some preparatory analysis on your features to see if the dimension can be
reduced. Which leads us onto . . .

2.5 Principal Components Analysis

Principal Components Analysis(PCA) is a technique for finding common

movements in data. For example you might have data for different crimes in
different places (and we will have this data later). It seems reasonable that
a place in which there are a large number of “Violence against the person
- with injury” cases might also have a large number of “Violence against
the person - without injury" cases. You might think about throwing away
one of these features (a strange word for a crime!) in order to reduce your
problem's dimensions. You could do that or you could do something a bit
more sophisticated by using PCA.
PCA involves first finding the correlation matrix for all of your original
features and then finding the eigenvectors and eigenvalues of this matrix.
The eigenvector associated with the largest eigenvalue is the first principal
component, and so on. From the sizes of the eigenvalues you get a measure
of how much each eigenvalue contributes to the overal l behaviour of your
data. The larger the eigenvalue then the more it contributes. I'm going to
assume that my readers know about eigenvalues and eigenvectors.
Now all you have to do is write the feature vector for each data point
as the sum of the principal component vectors. That's easy, and unique,
because the principal component vectors will almost certainly be orthogonal
if you’ve used enough real data. In this way your problem is recast in terms
of the principal components, and then, rather than throwing out all the “Vi
olence against the person - without injury" cases you would instead throw
out the principal component with the smallest eigenvalue. Or rather you
only keep those components that give a total contribution of 95%, or 99%,
or whatever level you deem suitable. And how do we know those percent
ages? Easy just divide each component's eigenvalue by the sum of all the
eigenvalues and that gives you the fraction explained by that component.
22 CHAPTER 2. GENERAL MATTERS

2.6 Maximum Likelihood Estimation

Maximum Likelihood Estimation (MLE) is a common method for esti

mating parameters in a statistical/probabilistic model. In words, you simply
find the parameter (or parameters) that maximizes the likelihood of observing
what actually happened. Let's see this in a few classical examples.

Example: Taxi numbers

You arrive at the train station in a city you've never been to before. You
go to the taxi rank so as to get to your final destination. There is one taxi,
you take it. While discussing European politics with the taxi driver you notice
that the cab’s number is 1234. How many taxis are in that city?
To answer this we need some assumptions. Taxi numbers are positive
integers, starting at 1, no gaps and no repeats. We'll need to assume that we
are equally likely to get into any cab. And then we introduce the parameter
N as the number of taxis. What is the MLE for N7
Well, what is the probability of getting into taxi number 1234 when there
are N taxis? It is

^ for N > 1234 and zero otherwise.

N

What value of N maximizes this expression? Obviously it is W = 1234.

That is the MLE for the parameter N. It looks a bit disturbing because it
seems a coincidence that you happened to get into the cab with the highest
number. But then the probability of getting into any cab is equally likely. It
is also disturbing that if there are N taxis then the average cab number is
{N + l)/2, and we somehow feel that should play a role.

Example: Coin tossing

Suppose you toss a coin n times and get h heads. What is the probability,
p, of tossing a head next time?
The probability of getting h heads from n tosses is, assuming that the
tosses are independent.
n! n— h
n
n-h
p^{i-p)
h\{n-h)\ \h
Applying MLE is the same as maximizing this expression with respect to p.
This likelihood function (without the coefficient in the front that is indepen
dent of p) is shown in Figure 2.1 for n = 100 and h = 55. There is a very
obvious maximum.
2.6. MAXIMUM LIKELIHOOD ESTIMATION 23

Likelihood (scaled)
1.4E-30

1.2E-30

lE-30

8E-31

6E-31

4E-31

2E-31
P
0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Figure 2.1; Likelihood versus probability p.

Often with MLE when multiplying probabilities, as here, you will take
the logarithm of the likelihood and maximize that. This doesn’t change the
maximizing value but it does stop you from having to multiply many small
numbers, which is going to be problematic with finite precision. (Look at
the scale of the numbers on the vertical axis in the figure.) Since the first
part of this expression is independent of p we maximize

hlnp {n — h)ln(l — p) (2.1)

with respect to p. See Figure 2.2.

Log likelihood
P
0

0.5 0.6 0.7 0,8 0.9 1

-50

-100

-150

-200

-250

-300

Figure 2.2; Log likelihood versus probability p.

This just means differentiating with respect top and setting the derivative
equal to zero. This results in
h
p=-
n

which seems eminently reasonable.

24 CHAPTER 2. GENERAL MATTERS

Example: A continuous distribution and more than one

parameter

For the final example let’s say we have a hat ful l of random numbers
drawn from a normal distribution but with unknown mean and standard
deviation, that's two parameters. The probability of drawing a number x
from this hat is

1
p{x) exp
2cr2
V^cr

where p, is the mean and a the standard deviation which are both to be
estimated. The log likelihood is then the logarithm of this, i.e.

1
ln(p(a;)) = —— 1ii(27t) — In cr — 20-2
{x -pf.

If draws are independent then after N draws, .r„, the likelihood is just

p{xi)p{x2)...p{xN) = Y[p{Xn)-
n=l

And so the log-likelihood function is

N
/N
In ^ln(p(x„)).
\n=l n=l

This gives us the nice log likelihood of

N
1
-Nina -
2o-2 E( Xn (2.2)
n=l

for the normal distribution. (I've dropped a constant at the front because it
doesn't affect the position of the maximum.)
An example is plotted in Figure 2.3. For this plot I generated ten random
xs from a normal distribution with mean of 0.1 and standard deviation of
0.2. These ten numbers go into the summation in expression (2.2), and I've
plotted that expression against /n and a. The maximum should be around
p = 0.1 and a = 0.2. (It will be at exactly whatever is the actual mean
and standard deviation of the ten random numbers I generated, as I'll show
next.)
2.6. MAXIMUM LIKELIHOOD ESTIMATION 25

Log likelihood

11

1 0.18

0.12

^ ^ m
^ in lO 0.06 £
00 cn
o d d _ (N
ro 0
fN S
(N
m
O d rM fN
o o rsl
o
d d
Sigma

Figure 2.3: Log likelihood for the normal distribution.

To find the maximum likelihood estimate for /r you just differentiate with
respect to jj, and set this to zero. This gives

N
1

N
n=l

I.e. ij, is the simple average. And differentiating with respect to a gives

N
1
a =
N
(2.3)
n—1

And this again looks as expected.

You could also use this technique to decide among different distributions.
Suppose you have three hats full of random numbers. For each hat you know
the distribution and parameters. Someone picks one of the hats, you don't
know which one it is, and they start drawing out numbers from it. You could
use MLE to tel l you which hat is the one most likely to have been chosen.
All of the above has been done analytically, i.e. finding the parameters
that maximize the likelihood. But that won’t be happening when we use
MLE in machine learning for more complicated problems. Then we’l l need
numerical methods.
26 CHAPTER 2. GENERAL MATTERS

2.7 Confusion Matrix

A confusion matrix is a simple way of understanding how well an algorithm

is doing at classifying data. It is really just the idea of false positives and
false negatives.
Let’s take a simple example.
An algorithm looks at pictures of fruit in order to classify them as apples,
pears, oranges, etc. Most it gets right, some it gets wrong. Can we quantify
this?
There are 100 items of fruit. Let's focus on how wel l our algorithm
identifies apples.

ACTUAL CLASS

Apple Non apple

11 9
Q Apple
K- TP FP
53
Uj U 5 75
ce
a. Non apple
FN TN

Figure 2.4: (Mis)identification of apples. TN = True Positives, FP = False

Positives, etc.

Sixteen of the 100 are apples. The rest are a mix of pears, oranges,
bananas etc. Our algorithm labels each item as whatever fruit it thinks it is.
It thinks there are 20 apples. Unfortunately the algorithm had both identified
as apples some that weren’t and some that were apples it missed. Only 11 of
the apples did it correctly identify. So it had some false positives and some
false negatives. This is shown in Figure 2.4.
In order to interpret these numbers they are often converted into several
rates:

TP+TN
• Accuracy rate: where Total = TP + TN + FP + FN. This
measures the fraction of times the classifier is correct

• Error rate: 1 —
TP+TN
Total
2.7. CONFUSION MATRIX 27

TP
• True positive rate or Recall: jp^p|\|
FP
• False positive rate:■ TN+FP

FP
• True negative rate or Specificity: 1 — TNTFP

TP
• Precision: If the algorithm predicts apple how often is it
TP+FP-
correct?

• Prevalence: TP+FN . What is the fraction of apples?

Total

Which of these measures is best depends on the context. For example

if you are testing for cancer then you will want a small false negative rate
and aren't too concerned about high false positives. On the other hand you
don't want to go around predicting everyone has cancer.
It is useful to compare numbers with the null error rate. That means how
often would you be wrong if you always predicted the largest class. In the
figure we have non apple being the largest actual category, so if we called
11+5
every fruit a non apple then we would be wrong 11+5+9+75 of the time. The
null error rate is particularly useful for comparison if one class is much larger
than the other.

Considering that there are only four numbers here (three if you scale to
have 100, say, as the total) there are an awful lot of ways they are sliced and
diced to give different interpretative measures. On the Wikipedia page for
Confusion Matrix I counted 13 different measures. Here's a good one, the
Matthews correlation coefficient, defined by

TP X TN - FP X FN

V(TP + FP)(TP + FN)(TN + FP)(TN + FN)’

This is a type of correlation that measures how good the classifier is. The
number it yields is between plus and minus one. Plus one means perfect pre
diction, zero means no better than random. This measure is good when you
have unbalanced numbers, with one category being much larger or smaller
than others.

Receiver operating characteristic

Yet another way of looking at how well an algorithm is classifying is the

receiver operating characteristic or ROC (curve). This is plotted, an example
is shown in Figure 2.5, as follows. Suppose there is a threshold/parameter
28 CHAPTER 2. GENERAL MATTERS

in your algorithm that determines whether you have an apple or a non ap

ple. Plot the true positive rate against the false positive rate as the thresh
old/parameter varies. Points above the 45° diagonal are good, and the
further into the top left of the plot the better.

0.2 0.4 0.6 0.8 1

False positive rate

Figure 2.5: Receiver operating characteristic.

The area under the ROC curve (the AUC) is then a measure of how
good different algorithms are. The closer to one (the maximum possible)
the better. If you enter Kaggle competitions they often use the AUC to rank
competitors. (If you do enter Kaggle competitions after reading this book,
and I hope you will, don’t forget my 10% cut of winnings. You did read the
small print didn't you?)

2.8 Cost Functions

In machine learning a cost function or loss function is used to represent

how far away a mathematical model is from the real data. One adjusts the
mathematical model, usually by varying parameters within the model, so as
to minimize the cost function. This is then interpreted as giving the best
model, of its type, that fits the data.
Cost functions can take many forms. They are usually chosen to have
properties that make some mathematical sense for the problem under inves
tigation and also to have some tractability.
2.8. COST FUNCTIONS 29

Let's look at an example. Suppose we have data for age and salary of
a sample of lawyers then we might want to look for a linear relationship
between the two. Old lawyers earn more perhaps. We might represent the
th
n lawyer's age by and their salary by And we'll have N of them.
See Figure 2.6, although the numbers here are made up. Note that I haven't
bothered to scale the features, the xs. That’s because I’m not measuring
any distances between feature vectors here.

250

200

••
150

100

50

X
0

20 25 30 35 40 45 50 55 60 65 70

Figure 2.6: Fit a straight line through these points.

We want to find a relationship of the form

y = 9o-\-0ix, (2.4)

where the 9s are the parameters that we want to find to give us the best
fit to the data. (Shortly I’ll use 0 to represent the vector with entries being
these 6s.) Call this linear function A@(x), to emphasise the dependence on
both the variable x and the two parameters, 9o and 0|.
We want to measure how far away the data, the ?y^”^s, are from the
function ho{x). A common way to do this is via the quadratic cost function
N
1 2
(n)
.]{e) = 2N
n=l
-y
) (2.5)

This is just the sum of the squares of the vertical distances between the points
and the straight line. The constant in front, the makes no difference
to the minimization but is there by convention, and can also be important
when N is large, to scale quantities.
We want the parameters that minimize (2.5). This is called ordinary least
squares (OLS).
This is a popular choice for the cost function because obviously it will be
zero for a perfect, straight line, fit, but it also has a single minimum. This
30 CHAPTER 2. GENERAL MATTERS

minimum is easily found analytically, simply differentiate (2.5) with respect

to both 9s and set the results to zero:

N N

in) n) in)
^(00 +
n=l
-y
)= n=l
[eo + 9ix^ -y
) = 0.

This can be trivially solved for the two 9s:

(Ey) {H^V)
'0 =

and

91
^(E-^-y)-(E-t)(E?v)
Nj2x^-iY:xf

I have dropped all the (n) superscripts in these to make them easier to read,
it’s obvious what is meant.
The cost function as a function of 9q and 9\ is shown in Figure 2.7
using the same data as in Figure 2.6. Actually it’s the logarithm of the cost
function. This makes it easier to see the minimum. Without taking logs the
wings in this plot would dominate the picture. And the fitted line is shown
in Figure 2.8.

Log of cost function

6
4
2
0
o
o
O CO
• O
KD

tb
rv
o

m oi
o
o
ro
theta_0 «X) ro

o I
ro
fNJ
(U
o
(N

Figure 2.7: The logarithm of the cost function as a function of the two
parameters.
2.8. COST FUNCTIONS 31

250

y = 3.3667x-5.2956
R^ = 0.9723
200

150

.*■
100

50

X
0

20 25 30 35 40 45 50 55 60 65 70

Figure 2.8: The linear fit.

Many machine learning methods fit models by looking for model param
eters that minimize some cost function.
Later we'll see cost functions in higher dimensions, i.e. more explanatory
features (as well as the lawyer’s age we could also include the annual fee for
the school he went to, etc.).
In some problems minimizing a cost function and maximizing a likelihood
can lead to the same or similar mathematical optimization problems. Often
this is when some aspect of the model is Gaussian. And, of course, applying
MLE really requires there to be a probabilistic element to your problem,
whereas the cost-function approach does not.
Other cost functions are possible, depending on the type of problem you
have. For example if we have a classification problem — "This is an apple"
instead of "0.365" — then we won’t be fitting with a linear function. If it's a
binary classification problem then apples would be labelled 1, and non apples
would be 0, say. And every unseen fruit that we want to classify is going to
be a number between zero and one. The closer to one it is the more likely
it is to be an apple.
In such a classification problem we would want to fit a function that
ranged from zero to one, to match the values for the class labels. And
linear, as we’ l l see later, would be a silly choice. Typically we fit an S-shaped
sigmoidal function. Probably the most common example would be
1
ho{x)
I -|_ g-Oa-OiX ■

The quadratic cost function is just not suitable in this situation. Instead
we often see the following.
N
1
(n
j{e) = - N ^ In + (1 - y
n=l
32 CHAPTER 2. GENERAL MATTERS

This looks a bit strange at first, it's not immediately clear where this
has come from. But if you compare it with (2.1) then you’ll start to see
there might be a link between minimizing this cost function and maximizing
the likelihood of something (there's a sign difference between the two). The
analogy is simply that the ys represent the class of the original data (the
head/tail or apple/non apple) and the hg is like a probability (for y being 1
or 0). More anon.
This cost function is also called the cross entropy between the two proba
bility distributions, one distribution being the ys, the empirical probabilities,
and the other being he, the fitted function. We'll see more about the cross
entropy in a couple of sections' time.

Regularization

Sometimes one adds a regularization term to the cost function. Here's

an example applied to OLS in a single variable:
2

j{e) -y
in)
^ + X&1 ■
The addition of the second, regularization, term on the right encourages the
optimization to find simpler models. Note that it doesn't include the 9q
parameter. The larger the 0i the greater the change to the forecast for a
small change in the independent variable. So we might want to reduce it.
It's not easy to appreciate the importance of the regularization term
with a simple one-dimensional linear regression. Instead think of fitting a
polynomial to some data. If you fit with a high-order polynomial you will get
a lot of rapid wiggling of the fitted line. Is that real, or are you over fitting?
The regularization term reduces the size of all the coefficients, except for the
first one representing a constant level, thus reducing such possibly irrelevant
wiggles.
Back to the linear regression in one dimension, the minimum is now at

9o =
(E^)(A + E-^^)-(E^)ilZxy)
2
^(A + Ea;^)-(Ea;)
and
-^(Ea:y)-(EE(Ey)
1

A little warning, just because I've Just given a few explicit solutions for
where the minimum is don’t expect this to happen much in this book. More
often than not we'll be minimizing cost functions etc. numerically.
The idea of regularization is easily generalized to higher dimensions.
2.9. GRADIENT DESCENT 33

2.9 Gradient Descent

So you want to find the parameters that minimize a loss function. And
almost always you are going to have to do this numerically. We'll even
see an example later where there is technically an explicit solution for the
minimizing parameters but since it involves matrix inversion you might as
well go straight into a numerical solution anyway.

O
U

These
will converge to
here

Figure 2.9: Illustrating gradient descent. Change 0 in the direction of the

slope, a distance proportional to the slope and a parameter (3.

If we have a nice convex function then there is a numerical method that

will converge to the solution, it is called gradient descent. For gradient
descent to work straight out of the box we really ought to have a cost
function with a single minimum, the global minimum, otherwise we are likely
to converge to a local minimum. But things can be done even if there are
local minima, they’l l get a mention shortly.
The method is illustrated in Figure 2.9.
The scheme works as follow. Start with an initial guess for each parameter
9k. Then move 0k in the direction of the slope:

New 9k = Old 0k - fi dJ/d0k-

Update all 9k simultaneously, and repeat until convergence.
Flere /? is a learning factor that governs how far you move. If ^ is too
small it will take a long time to converge. If too large it will overshoot and
might not converge at all. It is clear, as I mentioned, that this method might
converge to only a local minimum if there is more than one such.
34 CHAPTER 2. GENERAL MATTERS

The loss function J is a function of ail of the data points, i.e. their actual
values and the fitted values, in the above description of gradient descent we
have used al l of the data points simultaneously. This is called batch gradient
descent. But rather than use all of the data in the parameter updating we
can use a technique called stochastic gradient descent. This is like batch
gradient descent except that you only update using one of the data points
each time. And that data point is chosen randomly. Hence the stochastic.
Let me write the cost function as

j{0) = j2uo),
n=l

with the obvious interpretation. So in the case of OLS, Equation (2.5), we

would have
1
Jn{0) 2N
-y

Jn{0) is the contribution of data point n to the cost function.

Stochastic gradient descent means pick an n at random and then update
according to
New 6k = Old 6k - P dJn/dOk.
Repeat, picking another data point at random, etc.
There are several reasons why we might want to use stochastic gradient
descent instead of batch. For example;

• Since the data points used for updating are chosen at random the
convergence will not be as smooth as batch gradient descent. But
surprisingly this can be a good thing. If your loss function has lo
cal minima then the bouncing around can bounce you past a local
minimum and help you converge to the global minimum.

• It can be computationally more efFicient. If you have a very large

dataset it is likely that many of the data points are similar so you
don't need to use all of them. If you did it would take longer to
update the parameters without adding much in terms of convergence.

And then there’s mini batch gradient descent in which you use subsets of
the full dataset, bigger than 1 and smaller than n, again chosen randomly.
Finally if you are using a stochastic version of gradient descent you could
try smoothing out the bouncing around, if it's not being helpful, by taking
an average of the new update and past updates, leading to an exponentially
weighted updating. This is like having momentum in your updating and can
speed up convergence. You’ll need another parameter to control the amount
of momentum.
2.10. TRAINING, TESTING AND VALIDATION 35

2.10 Training, Testing And Validation

Most machine-learning algorithms need to be trained. That is, you give

them data and they look for patterns, or best fits, etc. They know they are
doing well when perhaps a loss function has been minimized, or the rewards
have been maximized. You'l l see all of this later. But you have to be careful
that the training is not overfitting. You don't want an algorithm that will
give perfect results using the data you've got. No, you want an algorithm
that will do well when you give it data it hasn't seen before.
When doing the training stage of any algorithm you could use all of the
data. But that doesn’t tell you how robust the model is. To do this properly
there are several things you should try. First of al l , divide up the original data
into training and test sets. Do this randomly, maybe use 75% for training
and then the remaining 25% for testing. How well does the trained algorithm
do on the test data?
Another thing you can do if there is any time dependence in your data
— maybe it was collected over many years — instead of taking 25% out at
random you split the data into two groups, before and after a certain date.
Now train these two data sets independently. Do you get similar results?
If you do then that's great, the model is looking reliable. If you don’t then
there are three, at least, explanations. One is that the model is fine but there
are regions in one of the two halves that are just not reached in the other
half. Second, there might have been a regime shift. That is, the world has
changed, and you need a time-dependent model. Or maybe you are simply
barking up the wrong tree with whatever algorithm you are using.

Epochs

In some machine-learning methods one uses the same training data many
times, as the algorithm gradually converges, for example, in stochastic gradi
ent descent. Each time the whole training set of data is used in the training
that is called an epoch or iteration. Typically you won’t get decent results
until convergence after many epochs.
One sees a decreasing error as the number of epochs increases, as shown
in Figure 2.10. But that does not mean that your algorithm is getting better,
it could easily mean that you are overfitting.
36 CHAPTER 2. GENERAL MATTERS

0 5 10 15 20

Epochs

Figure 2,10; Error versus epochs during training.

This can happen if the algorithm has seen the training data too many
times, i.e. there have been too many epochs. To test for this you introduce
the test set of data, the data that you have held back. All being well you wi
get results like in Figure 2.11. The test set is not as good as the training set,
obviously, but both are heading in the right direction. You can stop training
when the errors have levelled out consistently. There is a caveat to this, if
the test error is much bigger than the training error then that could also be
a sign of overfitting.

Training

P Test Case 1
LU

0 5 10 IS 20
Epochs

Figure 2.11: Training and testing are looking good. Stop training when the
error levels out.

On the other hand if you get results like in Figure 2.12 where the test-set
error begins to rise again then you have overfitted.
2.11. BIAS AND VARIANCE 37

Training

Test Case 2

0 5 10 15 20

Epochs
Figure 2.12: Looks like we have overfitting.

To help avoid overfitting sometimes we divide up our original data into

three sets. The third data set is called the validation set. We train the data
on the training set as before. Then after a few epochs we use the validation
set to see how the algorithm copes with out-of-sample data. We keep doing
this with more and more epochs. If we see the error for the validation set
rising then we stop at that number of epochs. But we can't be sure that the
algorithm hasn't learned to cope with new samples because our algorithm
knows about the validation set since we used it to decide when to stop. So
now we bring in the test data and see how the algorithm copes with that.
Note that we don't always plot error versus epoch. There'll be times
when we plot error versus a model parameter. This can help us choose the
best parameters for our model.

2.11 Bias And Variance

Suppose there is a relationship between an independent variable, x, and

a dependent variable, y, given by

y = fix)+ e.

Flere e is an error term, e has mean of zero (if it wasn't zero it could be
absorbed into f{x)) and variance cr^. The error term, whose variance could
also be x dependent, captures either genuine randomness in the data or noise
due to measurement error.
And suppose we find, using one of our machine-learning techniques, a
deterministic model for this relationship:

y = fix)-
38 CHAPTER 2. GENERAL MATTERS

Now / and / won't be the same. The function our algorithm finds, /, is
going to be limited by the type of algorithm we use. And it will have been
fitted using a lot of real training data. That fitting will probably be trying
to achieve something like minimizing
N

-y
n=\

The more complex, perhaps in terms of parameters, the model then the
smaller this error will be. Smaller, that is, for the training set.
Now along comes a new data point, x', not in the training set, and we
want to predict the corresponding y'. We can easily see how much is the
error in our prediction.
The error we will observe in our model at point x' is going to be

]{x')- fix')- e. (2.6)

There is an important subtlety here. Expression (2.6) looks like it only has
the error due to the e. But it's also hiding another, possibly more important
and definitely more interesting, error, and that is the error due to what is
in our training data. A robust model would give us the same prediction
whatever data we used for training our model.
So let's look at the average error, the mean of (2.6), given by

E /(.t') /(•^')

where the expectation E[-\ is taken over random samples of training data
(having the same distribution as the training data).
This is the definition of the bias,

Bias(/(x')) = E /(xO - fix').

We can also look at the mean square error. And since fix') and e are
independent we easily find that
2

E (/(.x-O - Z(x')- e)2 (Bias(/(.x'))) +Var(,/V))+ ^^ (2.7)

where
2

Var(/(x')) = E ffx')'^ E fix') .

Expression (2.7) is the mean square error for our new prediction.
This shows us that there are two important quantities, the bias and the
variance, that will affect our results and that we can control to some extent
by tweaking our model. The squared error (2.7) is composed of three terms.
2.11. BIAS AND VARIANCE 39

the bias of the method, the variance of the method and a term that we are
stuck with (the a^).
A good model would have low bias and low variance as illustrated in the
top left of Figure 2.13.

Low bias and low variance High bias and low variance

••
#•

•: «

Low bias and high variance High bias and high variance

Figure 2.13: Examples of low and high bias and variance.

Bias is how far away the trained model is from the correct result on
average. Where “on average" means over many goes at training the model,
using different data. And variance is a measure of the magnitude of that
error.

Unfortunately, we often find that there is a trade-off between bias and

variance. As one is reduced, the other is increased. This is the matter of
over- and underfitting.
Overfitting is when we train our algorithm too wel l on training data,
perhaps having too many parameters for fitting. An analogy is the two-year
old playing with his first jigsaw puzzle. After many attempts, and quite a
few tears, he finally succeeds. Thereafter he finds it much easier to solve
the puzzle. But has he really learned how to do jigsaw puzzles or has he just
memorized the one picture?
Underfitting is when the algorithm is too simple or not sufficiently trained.
It might work on average in some sense but fails to capture nuances of the
data.
40 CHAPTER 2. GENERAL MATTERS

Let’s see how this works with a simple example. We’ll work with the
data in Figure 2.14. This shows the relationship between y and x, including
the random component e. The function f{x) is just a shifted sine wave,
and there’s a uniformly distributed random variable added to it. Each dot
represents a sample, and the full training set is shown.

« • Full training set

•• ^—Actual behaviour, f(x)

••
/•

^ .•

••

Figure 2.14: The true relationship between y and x (a shifted sine wave),
also showing the random e.

Let’s start with a simple model for the relationship between x and y, i.e.
a simple f{x).
It doesn’t get any simpler than a constant. So in Figure 2.15 I have shown
our first model, it is just a constant, the dashed line. I have chosen for this
simple model the average y value for a subset of all the data. It doesn’t
actually matter which subset of the data because if I changed the subset the
average wouldn’t change all that much. This is clearly very underfitted, to
say the least.
Along comes an unseen sample, represented by the hollow dot in the
figure. But our forecast is the solid dot. There are three sources of error,
the error due to the e, the variance and the bias.
The e we are stuck with, we can't forecast that, although it is in the
training data and will therefore be implicitly within the model. The variance
(of the model) is tiny. As I said, if I used a different subset of the training
data then the model, here the average, would hardly change at all. So most
of the model error is the bias.
2.11. BIAS AND VARIANCE 41

epsilon

almost entirely bias

••
• Full training set m
Actual behaviour,f(x)
O Unseen sample
• Forecast
##
— — A simple model, underfitted

Figure 2.15: Underfitted model with high bias.

For an underfitted model the mean squared error will be dominated by

the bias and the e.
In the next two figures we see what happens when we use a more com
plicated model and two different subsets of the full set of training data. In
both cases I have used Excel's smooth-fitting option with a subset of the
data, so that’s two different models and thus two different forecasts. The
forecasts move about mainly because of the variance in the model.

Subset 1

t/

■Actual behaviour, f{x)

O Unseen sample
• Forecast

Fitted to Subset 1

Figure 2.16: A complicated model (over)fitted to one subset of the training

data.
42 CHAPTER 2. GENERAL MATTERS

Subset 2

^—Actual behaviour, f(x)

O Unseen sample

• Forecast

Fitted to Subset 2

Figure 2.17: The same model (over)fitted to a different subset.

Finally, two more plots, fitted to the same subsets but using a simpler
model, just a cubic polynomial. You see much less bias and variance now.

Subset 1

O 9
9

i
9 X

m
9 Subset 1

Actual behaviour, f(x)

O Unseen sample
9 Forecast

Poly. (Subset 1)

Figure 2.18: A good fit using a model that is not too simple or too complex.
2.11. BIAS AND VARIANCE 43

Subset 2

o%

^ «

A Subset 2 ■••■I*.
« «•
— Actual behaviour, f(x)
O Unseen sample
• Forecast

Poly. (Subset 2)

Figure 2.19: Same model, different dataset.

The trade off between bias and variance is shown in Figure 2.20.

Errors, and a balanced model

epsilon''2
Bias''2

Var
«
Total error
to
#
I
O I
$
I
A balanced #
#
model

^ «»>•

T T T T

<- Simple Model complexity Complex ->

Figure 2.20: The various error terms.

44 CHAPTER 2. GENERAL MATTERS

2.12 Lagrange Multipliers

In using most machine-learning techniques there comes a point where

we need to find some parameters. More often than not this must be done
via an optimization problem. And sometimes our optimization problem is
constrained, "Find the parameters that maximize.. . subject to the constraint

Such constrained optimization problems can often be addressed via the

method of Lagrange multipliers.
Suppose we want to find the maximum of the function /(xi,X2) then
we would differentiate with respect to the two variables, and find the values
of the variables that make these two expressions zero. (This might give
us a maximum, minimum or saddle point, so we'd have a tiny bit of extra
work to do to see which one is the maximum.) But what if we want to
maximize /(xi,X2) subject to the constraint g(xi,X2) = 0, representing a
curve in xi, X2 space? We can visualize this problem by thinking of Xi and
X2 being latitude and longitude and / being the height of a hill. The curve
,g(xi,X2) = 0 is a path along the hillside as seen from above. So you walk
along this path until you reach the highest point.
The method of Lagrange multipliers says that we can find the solution
by looking at the Lagrangian

L = /(xi,X2)- A.g(xi,X2) (2.8)

and finding its stationary points with respect to xj, X2 and A.

Figure 2.21: The contour map of the function /(.Xi,xg) and the line
g{xi,X2) = 0.
2.13. MULTIPLE CLASSES 45

To explain why this works look at the contour map for our hill in Figure
2.21. Here we can see where the greatest value of / along the path is also
where, in the view from above, the path and a contour touch. Geometrically
this means that the vector that is orthogonal to the contour is in the same
direction as the vector that is orthogonal to the constraint or mathematically
df df \ oc
dg dg \
dx\ ’ 8x2/ dxi ’ 8x2)
But differentiating (2.8) with respect to x\ and X2 gives us this, with A being
the constant of proportionality. And differentiating (2.8) with respect to A
is just the constraint again.
Although I've explained the method in two dimensions it is applicable
in any number, and also with any number of constraints, just add to the
Lagrangian terms linear in the constraints, each having its own A.
The method can be generalized to inequality constraints. So we would
now want to maximize f{xi,X2) subject to g{xi,x,2) < 0. We can still work
with (2.3) but now
A >0

and

^g{xi,X2) = 0.
This last expression is called complementary slackness and says that either
g{xi,X2) < 0 and the constraint is satisfied so we don't need the A, or
g{xi,X2) — 0 and we are on the boundary of the permitted region and we
are back with the earlier version of Lagrange multipliers. This generalization
goes by the name of Karush-Kuhn-Tucker (KKT).
Be careful that Lagrange multipliers and KKT only give necessary con
ditions for a solution. So you will need to check that any solution they give
actually works.

2.13 Multiple Classes

Many of the techniques we shall be learning classify data into two groups,
an email is or isn't spam. For example the Support Vector Machine in its
simplest version divides sample data according to which side of a hyperplane
they lie. But what if we have more than one class, we have spam, not spam,
and emails from the mother-in-law?
There are three techniques commonly used: One-versus-one classifica
tion; One-versus-all classification; Softmax function.
But first let me tell you something that is not, and should not be, used.
Suppose you want to determine who a painting is by. You might show
your algorithm many paintings by various artists. You might want to use
46 CHAPTER 2. GENERAL MATTERS

numerical labels for each artist. Paintings by van Gogh might be labelled as
1, paintings by Monet as 2, by Cassatt as 3, and so on. And then you show
it a painting by an unknown artist. It gives you its numerical output. It is
2.7. Who is that painting by? Well, it seems to think that it's quite likely
by Cassatt but possibly by Monet. But that would be nonsense. There is
no ordering of artists such that you can say that Monet is halfway between
van Gogh and Cassatt. You would only have a single, scalar, output if the
classes could be ordered in such a way that the associated numbers are
meaningful. E.g. Which rides is a person allowed to go on at Disneyland?
There might be three types of ride, ones for toddlers, ones for children and
ones for adults. That’s three classes but, according to the signs I’ve seen on
rides, they correspond exactly to a person’s height. So that would be fine.
Generally however you don’t want to use a scalar to represent more than
two classes. But you still might want to use numbers for classification. What
can you do? Here are some common methods.

One versus one

If we have K classes, A, B, C, ... , then we will set up and train K{K —

l)/2 binary classifiers using subsets of the training data: A vs B; B vs C; C vs
A; .... Our new, unclassified, sample is then tested against each of these,
gets a vote each time (for example resulting in (A,B,A)), and the chosen
class is the one with the most votes (A).

One versus all

The one-vs-all technique requires classifiers to produce a degree of confi

dence for any decision, and not simply a class label . This limits its usefulness.
But it works like this. If we have K classes then we train K binary classifiers:
A vs not-A; B vs not-B; C vs not-C; .... The winning class is the one that
has the highest confidence score.

Softmax function

Your classification algorithm could be trained on feature vectors in which

one entry is one and all other entries are zero, the one-hot encoding men
tioned earlier. But then the output vector for a new sample would be a
K-dimensional vector but typically the entries could be anything. You can
turn these outputs into something like a probability by using the softmax
function. The softmax function takes an array of values {zi,Z2, ■ ■ ■ ,z^),
2.14. INFORMATION THEORY AND ENTROPY 47

whether positive or negative, and turns them into a new array with values
between zero and one, and such that they sum to one:

K
Ek=\

This gives you a probabilistic interpretation of which class a new sample is

from. The entry with the highest value gives you its class, but you also get
an idea of the confidence in that result.

2.14 Information Theory And Entropy

In its original form information theory concerned the transmission of mes

sages. If you have a crackly telephone line then messages might be corrupted
between the transmitter and the receiver. A simple Morse code message of
dots and dashes could very easily be messed up as some dots turn to dashes
and vice versa. If there is some probability of corruption of the data then
how much information from the original message gets through and conversely
how much longer must a message be to ensure, to some given accuracy, that
the correct message is received?
Some of information theory is going to be important to us. These are
the concepts of self information or surprisal (me too, I wasn’t expecting such
a word), and information entropy, or simply entropy.
If the sun rises in the East there is no information content, the sun always
rises in the East. If you toss an unbiased coin then there is information in
whether it lands heads or tails up. If the coin is biased towards heads then
there is more information if it lands tails.
Surprisal associated with an event is the negative of the logarithm of the
probability of the event
log2(p).

People use different bases for the logarithm, but it doesn't make much
difference, it only makes a scaling difference. But if you use base 2 then
the units are the familiar bits. If the event is certain, so that p = 1, the
information associated with it is zero. The lower the probability of an event
the higher the surprise, becoming infinity when the event is impossible.
But why logarithms? The logarithm function occurs naturally in infor
mation theory. Consider for example the tossing of four coins. There are 16
possible states for the coins, HHHH, HHHT, ... , TTTT. But only four bits
of information are needed to describe the state. HTHH could be represented
by 0100.
4 = log2(16) = ~log2(l/16).
48 CHAPTER 2. GENERAL MATTERS

Going back to the biased coin, suppose that the probability of tossing
heads is 3/4 and 1/4 of tossing tails. If I toss heads then that was almost
expected, there's not that much information. Technically it's — log2(0.75) =
0.415 bits. But if I toss tails then it is -log2(0.25) = 2 bits.
This leads naturally to looking at the average information, this is our
entropy:

^plog2(p),
where the sum is taken over all possible outcomes. (Note that when there
are only two possible outcomes the formula for entropy must be the same
when p is replaced by 1 — p. And this is true here.)
For an unbiased coin the entropy is easily found to be 1. For a coin with
zero probability of either heads or tails then the entropy is zero. For the
75:25 coin it is 0.811. Entropy is a measure of uncertainty, but uncertainty
linked to information content rather than the uncertainty associated with
betting on the outcome of the coin toss.

2
\
\
1.5 \

0.5

0
0.1 0.2 .o.e—' 0.4 0.5 1

log^2(p)
-0.5
P log_2(p)-(l-p) log_2(l-p)
Expected winnings for 0.5 bet
-1
—--•Standard deviation of winnings

Figure 2.22: The information function, the entropy, expected winnings and
standard deviation. See text for details.

In Figure 2.22 I plot four quantities for the coin tossing, all against p
the probability of heads. There is the information function, the — log2(p),
the entropy plog2(p) — (1 — p)log2(l — p) and for comparison a couple of
lines usually associated with coin tossing. One is the expected winnings for
betting $0.5 on heads, p — 0.5, and the standard deviation of the winnings
■y/p(r^^p)2~+~(r^T^))p2. You can see that the standard deviation, the risk,
in the coin toss is qualitatively similar to the entropy.
Entropy is going to be important when we come to decision trees, it will
tel l us how to decide what order in which to ask questions so as to minimize
entropy, or, as explained later, maximize information gain.
2.14. INFORMATION THEORY AND ENTROPY 49

Cross entropy again

Suppose you have a model for the probability of discrete events, call
this where the index k just means one of K possibilities. The sum of
these probabilities must obviously be one. And suppose that you have some
empirical data for the probabilities of those events, With the sum again
being one. The cross entropy is defined as

fc

It is a measure of how far apart the two distributions are.

Let me give you an example.
Suppose that you have a machine-learning algorithm that is meant to
tell you whether a fruit is a passion fruit, orange or guava. As a test you
input the features for an orange. And the algorithm is going to output three
numbers, perhaps thanks to the softmax function, which can be interpreted
as the probabilities of the fruit in question (the orange) being one of P, 0 or
G. Will it correctly identify it as an orange?
The model probabilities come out of the algorithm as

= 0.13, = 0.69 and p^ = 0.18.

Ok, it's done quite well. It thinks the fruit is most likely to be an orange.
But it wasn't 100% sure. Empirically we know that

Pp = 0, pg = 1 and p^ = 0,
because it definitely is an orange. The cross entropy is thus

(0 X ln(0.13) + 1 X ln(0.69) -L 0 x ln(0.18)) = 0.371.

The cross entropy is minimized when the model probabilities are the same
as the empirical probabilities. To see this we can use Lagrange multipliers.
Write
/
L = -J]pfln(pH-A • (2.9)
k \ fc
The second term on the right is needed because the sum of the model
probabilities is constrained to be one. Now differentiate (2.9) with respect
to each model probability, and set the results to zero:
dL Pi - A = 0.
M
dp^ Pk

But since the sums of the two probabilities must be one we find that A = —1
and

Pk = Pk-
50 CHAPTER 2. GENERAL MATTERS

Because the cross entropy is minimized when the model probabilities are
the same as the empirical probabilities we can see that cross entropy is a
candidate for a useful cost function when you have a classification problem.
If you take another look at the sections on MLE and on cost functions,
and compare with the above on entropy you'll find a great deal of overlap
and similarities in the mathematics. The same ideas keep coming back in
different guises and with different justifications and uses.

2.15 Natural Language Processing

Some of the methods we will look at are often used for Natural Language
Processing (NLP). NLP is about how an algorithm can understand, interpret,
respond to or classify text or speech.
NLP is used for

• Text/sentiment classification: Spam emails versus non spam is the

example often given. And is a movie review positive or negative?

• Answering questions: Determining what a question is about and then

searching for an answer

• Speech recognition: "Tell us in a few words why you are contacting

BT today"

We will later give a couple of examples.

Text is obviously different from numeric data and this brings with it some
special problems. Text also contains nuances that a native speaker might
understand intuitively but which are difficult for an algorithm to appreciate.
No kidding.
Here are a few of the pre-processing techniques we might need to employ
and issues we need to deal with when we have text.

• Tokenize: Break up text into words plus punctuation (commas, full

stops, exclamation marks, question marks, ellipses, . ..)

• Stop words: Some words such as a, the, and, etc. might not add any
information and can be safely removed from the raw text

• Stemming: One often reduces words to their root form, i.e. cutting
off their endings. Meaning might not be lost and analytical accuracy
might be improved. For example, all of love, loves, loving, loved, lover,
. . . would become lov
2.16. BAYES THEOREM 51

Lemmatize: Lemmatizing is a more sophisticated version of stem

ming. It takes into account irregularity and context. So that eat,
eating, ate, ... are treated as all being one word

Categorize: Identify words as nouns, verbs, adjectives, etc.

‘N’-grams: Group together multiple words into phrases such as "red

flag,” "jump the shark," etc.

Multiple meanings: Sometimes one needs to look at surrounding

words to get the correct meaning. "The will of the people" or “I will
have pizza for lunch" or "In his will he left all his money to his cat.”
Some words might be spelled the same, and have the same root, but
be subtly different, for example, read and read

Word2vec

Word2vec are methods that assign vectors to words. This requires a

large number of dimensions to capture a decent vocabulary. Words that
have something in common end up, after training, near each other. Rather
cleverly, combinations of words can be represented by simple vector opera
tions, such as addition and subtraction. For example, fish -F chips H- mushy
peas might result in a vector close to the dinner vector.

2.16 Bayes Theorem

Bayes Theorem relates probabilities of events given important information

about the events. In symbols we have

P{B\A)P{A)
P{A\B) =
P{B)

where P{A\B) is the probability of A being true/having happened given that

B is true/happened. But P{A) is the probability of A being true/having
happened without any information/independently of B. Thus P{A\B) is a
conditional probability.
It's easy to see how this works in the following sketch, Figure 2.23.
52 CHAPTER 2. GENERAL MATTERS

Figure 2.23: Demonstration of Bayes Theorem.

There are two ellipses here, with an overlapping area, and dots inside
them. Imagine that the left ellipse is red, the right blue, and the overlapping
area is thus purple. There are a total of 22 dots, ten in Ellipse A, 15 in
Ellipse B, and thus three in both ellipses, in the purple area.
Assuming a dot is chosen at random, the probability that a dot is in
Ellipse A is
10

22

The probability of being in B is

15
P{B) 22

The probability that a dot is in both A and B, called the intersection of A

and B, i.e. in the purple region, is written as

P{AnB)= 22
—.

The conditional probability of A given B is

3
P{A\B) 15

That is, out of the 15 in B there are three that are also in A. This is equal
to
3/22 _ P{AnB)
lb122 ^ P{B)
That's a simple example of

P{Ar\B)
P{A\B)
P{B) ■
But symmetrically

P{B n A)
P{B\A)=
P{A) ■
2.17. WHAT FOLLOWS 53

It's obvious that P{B n A)= P{A n B) and so

P{A\B) P{B\A)
P{A) ” P(B) ■

This is just another, symmetrical, way of writing Bayes Theorem.

2.17 What Follows

Having set the scene I am now going to walk you through some of the
most important machine-learning techniques, method by method, chapter by
chapter. I have chosen to present them in order of increasing difficulty. The
first methods can be done in a simple spreadsheet, as long as your datasets
aren't too large. As the methods get more difficult to implement then you
might have to move on to proper programming for anything other than trivial
examples. Although I am not going to cover programming in this book there
are very many books out there which do this Job wonderfully. In particular you
will find many books on machine learning using the programming language
du jour, Python.
All of the methods have worked illustrative examples. And they use real
data, not made up. Having said that, none of the results, such as they are,
come with any warranty or guarantee. Despite using real data the examples
are illustrative. Sometimes I've violated good-practice guidelines, for example
not worrying about the curse of dimensionality, in pursuit of a good story. In
some cases the end results seem encouraging, perhaps even useful. Please
also take those with a pinch of salt. Equally some of the results are perhaps
unexpected. I have left those as a lesson in how machine learning can throw
up the unexpected. Whether the unexpected results are meaningful though
would require further analysis.
My primary goal is to get you up and running with some machine learning
as soon as possible.
As the Ramones would say, "Hey, ho, let's go!"

Further Reading

Most of these topics are covered in depth in many decent books on

numerical analysis and statistics/probability, but not all in one book as far
as I know.
I would recommend that you take a look at Claude Shannon's classic
paper introducing information theory from 1948 "A Mathematical Theory of
54 CHAPTER 2. GENERAL MATTERS

Communication," The Bell System Technical Journal, \/o\. 27, pp. 379-423,
623-656, July, October, available online.
Chapter 3

K Nearest Neighbours

3.1 Executive Summary

K nearest neighbours is a supervised-learning technique. We take clas

sified data represented by a vector of features. Think of coloured dots in
space, the dimension of that space being the number of features and the
colours being the categories. We want to classify a new dot, i.e. figure out
its colour, given where it is in space.
We measure distances between the new data point and the nearest K of
our already-classified data and thus conclude to which class our new data
point belongs by majority voting.
Instead of having classes we can associate a number with each data point
and so use the method for regression.

3.2 What Is It Used For?

K nearest neighbours is used for

• Classifying samples according to their numerical features

• Performing a regression of numerical values based on the features of

the sample

• Example: Predict whether a spouse will like a Christmas present based

on price, size, — of the present, not of the spouse — Amazon rating.
etc.

• Example: Forecast the sales of a mathematics text book based on

features such as price, length, number of university courses in the
area, etc.

55
56 CHAPTER 3. K NEAREST NEIGHBOURS

• Example: Credit scoring, classify as good or bad risk, or on a scale,

based on features such as income, value of assets, etc.

• Example: Predict movie rating (number of stars) based on features

such as amount of action sequences, quantity of romance, budget.
etc.

• Example: Classify risk of epilepsy by analysing EEC signals

3.3 How It Works

K nearest neighbours (KNN) is perhaps the easiest machine-learni ng

technique to grasp conceptually. Although really there is no learning at all.
It can be used for classification, determining into which group an individ
ual belongs. Or it can be used for regression, predicting for a new individual
a numerical value based on the values for similar individuals.
We start off with N points of already-classified data, making this a
supervised-learning technique. Each data point has M features, so each
point is represented by a vector with M entries. Each entry represents a
feature. The first entry in the vector might be the number of words in an
email, the second entry might be the number of exclamation marks, the
third might be number of spelling mistakes, and so. And then each vector
is classified as spam or not spam. The number of classes does not have to
be two.
When we have a new data point to classify we simply look at the K
original data points nearest to our new point and those tell us how to classify
our new point.
In Figure 3.1 we see = 20 data points with M = 2 features (repre
sented by the two axes) classified into two groups, circles and triangles. Note
that there could be any number of features/dimensions, I have two here just
so I can easily draw the pictures (you'll be reading that a lot). And there
could be any number of groups.
We add a new point, for the moment represented by a square in the
figure. Should that new point be classified as a circle or a triangle?
Suppose we decide that AT = 3, that we will look at the three nearest
neighbours, then from Figure 3.2 we see that two of the nearest neighbours
are circles and only one is a triangle. Thus we classify our square as being a
circle (if you see what I mean). This is simple majority voting.
Of course, this example is only in two dimensions, so we can draw the
pictures. The two dimensions Just means that each data point has two
features. In practice there might be many more features and thus we'd be
working with hyperspheres to determine the nearest neighbours.
3.3. HOW IT WORKS 57

A A
A

A A
#

•• A 4

O?^
A
4®

Figure 3.1: Does the square belong with the circles or the triangles?

A A
A

A
m

Figure 3.2: Finding the K nearest neighbours.

Choosing K

A big question is how to choose K. If it is small then you will get low
bias but high variance (see Chapter 2). A large K gives the opposite, high
bias but low variance.
58 CHAPTER 3. K NEAREST NEIGHBOURS

3.4 The Algorithm

The idea is almost too simple to put into mathematics!

The K Nearest Neighbours Algorithm

Step 0: Scaling

As mentioned in Chapter 2, the first thing we do is to scale all features

so that they are of comparable sizes. Perhaps scale so that the mean
number of words in an email is zero with a standard deviation of one.
Do this for all features.

Step 1: A new data point

We now take our new, unclassified data point, and measure the dis
tance from it to all of the other data points. We look at the K points
with the smallest distance. From those K points the classification
with the most number of appearances determines the classification
of our new point.

There is no real learning, as such, done in KNN. An algorithm in which

a model is generalized only as new data points are added is called lazy. The
opposite, in which an algorithm is trained before being used on new samples,
is called eager. Lazy methods are useful when there is new data appearing all
the time. In such a situation eager models would quickly become outdated.

3.5 Problems With KNN

Computer issues In this book I rarely talk about hardcore computer issues
like memory and calculation speed. But I will make an observation here about
the K nearest neighbours method. Although the non-existent learning part
of the algorithm is lightning fast(!) the classification of new data points can
be quite slow. You have to measure the distance between the new point and
al l of the already-classified points.

Skewed data If there are many more data points of one class than others
then the data is skewed. In the above figures that might mean many more
circles than triangles. The circles would tend to dominate by sheer numbers.
3.6. EXAMPLE: HEIGHTS AND WEIGHTS 59

It is therefore common to weight the number of K nearest data points by

either the inverse of the distance from our unclassified point or by a decaying
exponential of the distance.

3.6 Example: Heights and weights

You can easily find data for men's and women's heights and weights
online. In Figure 3.3 I show sample data, just 500 points for each gender.
Note that the data has been shifted and scaled so as to have a mean of zero
and a standard deviation of one for each of height and weight. Obviously I've
used the same transformation for both genders. Because of the high degree
of correlation I could have done a simple PCA which would have had the
effect of spreading out the data. I haven't done that here for two reasons.
First, because I want to focus on the KNN method. And second, because in
higher-dimensional problems it might not be so obvious that there are any
patterns. I've also sketched a couple of ellipses to help you see roughly where
the different data points are.

• Men
0)
5
Women ~o 2
O)

♦ PW
TO

1/1

&3x

-3 -2 n 1> 2 3

Scaled height

OQ

PW

-2
o °
£13

-3

Figure 3.3: Fleights and weights for 500 men, 500 women, and one PW, all
scaled.

We can use KNN on a new sample to decide whether they are male or
female. The new sample is the diamond in the plot (it's me!). You can see
that I'm probably female, at least for small K. Maybe it's time to stop the
diet.
60 CHAPTER 3. K NEAREST NEIGHBOURS

When K = I the male/female regions are as shown in Figure 3.4. You

can see several islands, pockets of males within what looks like an otherwise
female zone and vice versa.
3

JZ
oo

. Ven

Womon

3 ■2. 3
d*
S: teled height
T

.■M
mii -2

-3

Figure 3.4: Male and female regions when K = 1.

When = 21 the boundary has been smoothed out and there is just a
single solitary island left. See Figure 3.5.
3
x:

: •
• Men I '
. : ■

WomcT I
m
*
*
m
-

r-\
• -

•3 ■1
*v4

-2

-3

Figure 3.5: Male and female regions when K = 21.

As an extreme case take K = 101. See Figure 3.6.

3.6. EXAMPLE: HEIGHTS AND WEIGHTS 61

f =- 5
*

w.
S*
Wniic-;
e
m
m

>
-3 K 3

height
:*41
•a
^9 m
-2
3' ii'i>

•3

Figure 3.6: Male and female regions when K = 101.

Figures 3.4 and 3.6 nicely show the differences between bias and variance.
The former figure shows high variance but low bias. It's a complicated model
because it only looks at the nearest neighbour. Yes, I know that sounds like
it's a simpler model but it’s really not. The model changes a lot as we
move around because the nearest neighbours change a lot. The latter figure
shows high bias and low variance. It's a simpler model because the nearest
neighbours don't change so much as we look at new samples. It has a high
bias and low variance. In the extreme case in which we use the same number
of neighbours as data points then every prediction would be the same.
We can plot misspecification error as a function of the number of neigh
bours, K. I have done this in Figure 3.7 using out-of-sample data. It looks
like it' = 5 is optimal.

11.0%

10.5% ,

10.0%

o
^ 9.5%
UJ

9.0%

•••
8.5%

K
8.0%
0 20 40 60 100

Figure 3.7: Error versus K.

62 CHAPTER 3. K NEAREST NEIGHBOURS

We can also get a probabilistic interpretation of the classification. In

Figure 3.8 is shown the probability of being male. This uses K = 5 and the
obvious calculation.

0.8-1

lBO.6-0.8

5S0.40.6

15 0.2-0.4

0-0.2

•a- o o <xs <r> in co rNj ^D cri m <.£> 0 '?

CO cq CO Ln CO CO in
cs
rH O 9 O odd rs

Figure 3.8: Probability of being male.

3.7 Regression

I’ve explained how KNN can be used for classification but it can also
easily be used for regression.
The idea is simple. Instead of samples being labelled with classes, such
as type of fruit, the labels are numbers. The features might be age, IQ, and
height with the labels being salary. Your goal is to find the salary for a new
sample. You tell me your age, IQ and height and I’ll tell you how much you
earn.

This regression can be done in a variety of ways.

• The simplest method is to just average the numbers (salaries) of the

nearest K neighbours

• You could get more sophisticated by weighting the average, perhaps

weighted by the inverse of the distances to the new data point. This
3.7. REGRESSION 63

inverse weighting might be a bit too extreme, if you have a new data
point that is exactly the same as one in your training set then it
will give exactly the same label, no room for perhaps a confidence
interpretation.

• Or exponentially weighted by distance

• Or using a Gaussian function of the distance, a Gaussian kernel

• Or by fitting a hyperplane to the K data points. This is a local linear

regression. Figure 3.9 shows an example, with one independent variable
for ease of plotting. Here K = 5. The bold dots are the nearest five
points to our new sample. The faint dots represent the ones we are
ignoring in the linear regression

• Unused data
»
• Data used in the regression
Linear(Data used in the regression)

New

sample
X

Figure 3.9: Linear regression on K 5 nearest neighbours.

The weighting might require another parameter, as well as the K, such

as the rate of decay in the exponential examples.
These methods generally go by the name of kernel smoothing.
Oh, and don't forget that the weights must add up to one!

Further Reading

There are not many books devoted to KNN in general, perhaps because
it is so straightforward. Here are a couple you might want to look at.
Algorithms for Data Science Hardcover by Brian Steele, John Chandler
and Swarna Reddy, published by Springer in 2017, covers the method, algo
rithms and also has exercises and solutions.
64 CHAPTER 3. K NEAREST NEIGHBOURS

If you want to read something with depth then you should look at Lectures
on the Nearest Neighbor Method (Springer Series in the Data Sciences) by
Gerard Biau and Luc Devroye, published in 2015. This book covers the
methods and provides a rigorous basis for them.
There are however many books and research papers that cover the ap
plication of KNN to specific problems.
Chapter 4

K Means Clustering

4.1 Executive Summary

K means clustering is an unsupervised-learning technique. We have many

individual data points each represented by vectors. Each entry in the vector
represents a feature. But these data points are not labelled or classified a
prion.
Our goal is to group these data points in a sensible way. Each group will
be associated with its centre of mass. But how many groups are there and
where are their centres of mass?

4.2 What Is It Used For?

K means clustering is used for

Grouping together unlabelled data points according to similarities in

their features

Example: Classification of customers according to their purchase his

tory, each feature might be expenditure on different types of goods

Example; Optimal placement of car parks in a city

Example: Optimizing the neck size and arm length of shirts

Example: Grouping together similar images, without them being pre

viously classified

K means clustering (KMC) is a very simple method for assigning indi

vidual data points to a collection of groups or clusters. Which cluster each

65
66 CHAPTER 4. K MEANS CLUSTERING

data point is assigned to is governed simply by its distance from the centres
of the clusters.
Since it's the machine that decides on the groups this is an example of
unsupervised learning. KMC is a very popular clustering technique. It is
highly intuitive and visual, and extremely easy to programme.
The technique can be used in a variety of situations:

• Looking for structure within datasets. You have unclassified data but
you suspect that the data fall naturally into several categories.

You might have data for car models, such as prices, fuel efficiency,
wheel size, speaker wattage, etc. You find that there are two natural
clusters, and they turn out to be cars that appeal to people with no
taste and cars that appeal to everyone else. The data might look like
in Figure 4.1. Here there are two obvious distinct groups.

••

.•
>1*

Figure 4.1: Two fairly distinct groups.

• Dividing up data artificially even if there is no obvious grouping.

The data in Figure 4.2 might represent family income on the horizontal
axis with the vertical axis being the number of people in the household.
A manufacturer of cutlery wants to know how many knives and forks to
put into his presentation boxes and how fancy they should be. There
might not be obvious groups but that doesn't necessarily matter.
4.3. WHAT DOES K MEANS CLUSTERING DO? 67

• •• •

•••

•
••

Figure 4.2: Flow would you group this data?

4.3 What Does K Means Clustering Do?

We have a dataset of N individuals, each having an associated M-

dimensional vector representing M features, so for n = 1 to N. Each
entry in the vectors represents a different numerical quantity. For example
each vector could represent an individual household, with the first element
in each vector being an income, the second the number of cars, . . . , the
being the number of guns.
We are going to group these data points into K clusters.
We will pick a number for K, say three. So we will have three clusters.
Each of these three clusters will have a centre, a point in the M-dimensional
feature space. And each of the N individual data points is associated with the
centre that is closest to it. (Like houses and postboxes. The postbox is like
the centre for the cluster. And each house is associated with one postbox.)
The goal of the method is to find the best positions for the centres of these
clusters. (And so you could use KMC to tell you where is best to put the
postboxes.)
In Figure 4.1 there are clearly two distinct groups. If I asked you to put
dots in the picture representing the centres of the two groups where would
you put them? In this example it would be quite easy to do a reasonable job
by hand. But in more than two or three dimensions? Not so easy. We need
the machine to do it for us.
68 CHAPTER 4. K MEANS CLUSTERING

Mathematically, the idea is to minimize the intra-cluster (or within-

cluster) variance. And the intra-cluster variance is just a measure of how
far each individual point is to its nearest centre. (How far is the house from
the nearest postbox.)
Typically you might then choose a different K and see what effect that
has on distances.
The algorithm is really simple. It involves first guessing the centres and
then iterating until convergence.

The K Means Algorithm

Step 0: Scaling
As explained in Chapter 2 we first scale our data, since we are going
to be measuring distances.
Now we start on the iteration part of the algorithm.

Step 1: Pick some centres

We need to seed the algorithm with centres for the K clusters. Either
pick K of the N vectors to start with, or just generate K random
vectors. In the latter case they should have the same size properties
as the scaled datasets, either in terms of mean and standard deviation
or minimum and maximum. Call these centres for A; = 1 to K.
See the two diamonds in Figure 4.3.

.* o

»»•

Figure 4.3: First pick some centres.

4.3. WHAT DOES K MEANS CLUSTERING DO? 69

Step 2: Find distances of each data point to the centres

Now for each data point (vector measure its distance from the
centres of each of the K clusters. We’ve discussed measuring dis
tances in Chapter 2. The measure we use might be problem depen
dent. If we have the house/postbox problem then we'd probably use
the Manhattan distance (unless you expect people to walk through
each other’s back gardens). But often we'd use the obvious Euclidean
distance:

Distance^”’''^ (u)

\ i:{
for A: = 1 to K.
m=l

Each data point, that is each n, is then associated with the nearest
cluster/centre:
argmin Distance^”’^\
k

This is easily visualized as follows. Suppose K is two, there are thus

two clusters and two corresponding centres. Let’s call them the red
cluster and the blue cluster. We take the first data point and measure
its distance from each of the two centres. That’s two distances. The
smaller of these turns out to be the distance to the blue centre. So
we paint our first data point blue. Repeat for all of the other data
points, so each data point gets coloured. See Figure 4.4 in which I
have drawn a line dividing the two groups of dots.

.* o

Figure 4.4: Assign each data point to its nearest center.

70 CHAPTER 4. K MEANS CLUSTERING

Step 3: Find the K centroids

Now take all of the data points associated with the first centre and
calculate the centroid, its centre of mass. In the colourized version,
just find the centroid of all the red dots. Do the same with all the
blue dots. You will find K centroids. These will be the cluster centres
for the next iteration.
Go back to Step 2 and repeat until convergence. See Figures 4.5 and
4.6.

Figure 4.5: Update the centroid.

Figure 4.6: Repeat until converged.

4.4. SCREE PLOTS 71

4.4 Scree Plots

Adding up all the squared distances to the nearest cluster gives us a

measure of total distance or an error. This error is a decreasing function of
the number of clusters, K. In the extreme case K = N you have one cluster
for each data point and this error will be zero.
If you plot this error against K you will get a scree plot. It will be one
of the two types of plots shown in Figure 4.7. If you get the plot with an
elbow where the error falls dramatically and then levels off then you probably
have data that falls into nice groupings (the triangles in Figure 4.7). In this
example there is a big decrease in error going from K = 2 to K = ?>, but
there isn't so much of a drop for 7F = 3 to AT = 4. The number of clusters
is obvious; it is three in this plot.

Error

m
'A.
A
& I
0 1 2 3 4 5 6 7 8

Number of Clusters

Figure 4.7: Two ways that error could decrease with number of clusters. The
triangles are an example with an elbow.

If the error only falls gradually (the circles in Figure 4.7) then there is no
obvious best K. Here we don't see large a large drop in error followed by
a flattening out. That doesn't mean that there isn’t a natural grouping of
the data, for example you could have data like in Figure 4.8, but it will be
harder to find that grouping.
72 CHAPTER 4. K MEANS CLUSTERING

*• 1 » %

>« •

Figure 4.8: There is an obvious pattern here but it will take some work — a
change of coordinate system, for example — to find it.

Note that while convergence is typically quite rapid, the algorithm might
converge to a local minimum distance. So one would usually repeat several
times with other initial centres.

4.5 Example: Crime in England, 13 dimensions

For our first real example 1 am going to take data for crime in England.
The data I use is for a dozen different categories of crime in each of 342
local authorities, together with population and population density data. The
population data is only for scaling the numbers of crimes, so we shall therefore
be working in 13 dimensions. Don't expect many 13-dimensional pictures.
The raw data looks like in Figure 4.9. The full list of crimes is:
Burglary in a building other than a dwelling
Burglary in a dwelling
Criminal damage
Drug offences
Fraud and forgery
Offences against vehicles
Other offences
Other theft offences
Robbery
Sexual offences
Violence against the person - with injury
Violence against the person - without injury
4.5. EXAMPLE: CRIME IN ENGLAND, 13 DIMENSIONS 73

Burglary in
a building Offences Population
other than Burglary in Criminal Drug Fraud and against per Square
Local Authority a dwelling a dwelling damage offences vehicles Population Mile
Adur 280 120 708 1S8 68 382 S8S00 3610
Allerdale 323 126 1356 392 79 394 96100 198
Alnwick 94 33 215 25 11 71 31400 75
Amber Valley 498 367 1296 241 195 716 116600 1140
Arun 590 299 1806 471 194 819 140800 1651
Ashficid 784 504 1977 352 157 823 107900 2543
Ashford 414 226 1144 196 162 99900 446
Aylesbury Vale 696 377 1490 502 315 833 157900 453

Babcrgh 398 179 991 137 152 448 79500 346

Barking & Dagenham 639 1622 2353 1071 1194 3038 155600 11862
Barnet 1342 3550 2665 1198 1504 4104 331500 9654

Barnsley 1332 860 3450 1220 322 1661 228100 1803

Barrow-in-Furness 190 134 1158 179 59 227 70400 2339
Basildon 756 1028 1906 680 281 1615 164400 3874

Basingstoke & Deane 1728 598 426 930 182 1159 147900 605

Figure 4.9: A sample of the crime data.

The crime numbers are first scaled with population in the local author
ities. And then there is the second translation and scaling as discussed in
Chapter 2.
A straightforward application of iC-means clustering results in the fol
lowing scree plot, shown in Figure 4.10.

5000

4500

4000

3500

3000
O
J- 2500
2000

1500

1000

500

0 1 2 3 4 5 6 7 8

Number of clusters

Figure 4.10: The scree plot for crime in England.

There is a moderately convincing elbow at around three clusters. And

those three clusters are particularly interesting. The results are shown in the
following table, Table, 4.1, in original variables but with the crime numbers
scaled per head of population.
74 CHAPTER 4. K MEANS CLUSTERING

Cluster 1 Cluster 2 Cluster 3

Number in cluster 1 273

Burglary in a building other than a dwelling 0.0433 0.0059 0.0046

Burglary in a dwelling 0.0077 0.0079 0.0030

Criminal damage 0.0398 0.0156 0.0114

Drug offences 0.1446 0.0070 0.0029

Fraud and forgery 0.1037 0.0042 0.0020

Offences against vehicles 0.0552 0.0125 0.0060

Other offences 0.0198 0.0018 0.0009
Other theft offences 0.6962 0.0313 0.0154

Robbery 0.0094 0.0033 0.0004

Sexual offences 0.0071 0.0015 0.0008

Violence against the person - with injury 0.0560 0.0098 0.0053

Violence against the person - without injury 0.0796 0.0128 0.0063

Population per Square Mile 4493 10952 1907

Table 4.1: Table of cluster results.

Cluster 1 has precisely one point. It is the City of London. In this data it
appears as an outlier because the population figures for each local authority
are people who live there. And not many people live in the City. We thus
can’t tell from this analysis how safe the City really is, since many of the
crimes probably happen to non residents. For example, Burglary from a
Dwelling is similar in both the City and in Cluster 2.
The other two clusters clearly represent dangerous local authorities (Clus
ter 2) and safer ones (Cluster 3). And what stands out to me, as someone
who spends most of his time in the countryside, is that the safe places are
the least densely populated. Phew.
This example also nicely illustrates an important point, specifically the
effect of scaling. Although there is nothing wrong in having a cluster con
taining a small number of data points, here there could possibly be an issue.
Outliers can easily mess up the scaling. Having a large value for a feature
in a small number of samples will usually cause that feature for the remaining
samples to be rescaled to pretty small numbers. Although this will depend on
the type of rescaling you use. And so when it comes to measuring distances
this feature will not fully participate. In the present case what I should now
do, if this were a research paper, is to remove the outlier, the City of London,
and redo the calculations.
The above is quite a high-dimensional problem, 13 features meaning 13
dimensions, compared to relatively few, 342, training points. We might have
expected to suffer from the curse of dimensionality mentioned in Chapter 2.
From the results it looks like, luckily, we didn't. The reason for this might be
the common one that the different features don’t seem to be independent.
Theft, robbery, burglary are very similar. Fraud and forgery, and sexual
offences are not.
4.6. EXAMPLE: VOLATILITY 75

If you wanted to reduce the dimensions early on, before using K means,
then you could use Principal Components Analysis. Or, as possibly here,
don't use features that common sense says are very closely related.
We are now going to do a few financial/economic examples. There is an
abundance of such data, for different financial or economic quantities and
in vast amounts. Some of it you might have to pay for, for example the
so-called tick data used in high frequency trading, but my examples all use
the free stuff.

Warning I am going to be using time-series data below. Time-series data

is not necessarily suitable for K means clustering analysis because the time
component is lost, or rather it is not utilized. Nevertheless in the following
examples we shall find some interesting results.

4.6 Example: Volatility

Of course, we can do problems in any number of dimensions. So let’s

start our financial examples with a one-dimensional example.
This problem, like several in this book, uses financial data. Finan
cial data is particularly easy to get hold of. For example, take a look at
finance. yahoo.com for many share and index prices. I'm going to be using
the S&P500 index now. I first downloaded the index historical time series
going back to 1950. From this I calculated in Excel a 30-day volatility. Now
if you are from finance you'll know what I mean. But if you aren't then this
is not the place to go into too many details. (I can give you the names of
a few excellent technical books.) Suffice to say from the column of SP500
daily closing prices you calculate a column of daily returns, and then calcu
late the standard deviation of a window of these returns. Volatility is then
just a scaled version of this. In Figure 4.11 is a plot of that volatility.
76 CHAPTER 4. K MEANS CLUSTERING

0.9

0.8
O
>
0.7

0,6

0.5

0.4

0.3

0.2

0.1

12/04/1949 20/12/1962 28/08/1976 07/05/1990 14/01/2004 22/09/2017

Figure 4,11: Thirty-day SPX volatility.

Remember that this KMC analysis completely throws away any time
dependence in the behaviour of volatility. However I do have some motivation
for using K means on this data and that is that there is a model used in
finance in which volatility jumps from one level to another, from regime to
regime. And the volatility in this plot seems to behave a bit like this. You
see that often the volatility is pretty low, sometimes kind of middling, and
occasionally very large.
That's not exactly what we have here. Here there is a continuum of
volatility levels. But I'm not going to worry about that. I'm going to work
with three clusters, K = 3. Remember this is Just an illustrative example.
Let's see how it goes. I will find those three clusters and then take this model
a little bit further.
The algorithm very quickly finds the following clusters, see Table 4.2.

Cluster 1 Cluster 2 Cluster 3

Number in cluster 586 246 24

SPX volatility 9.5% 18.8% 44.3%

Table 4.2: Volatility clusters in the S&P500 Index.

How volatility moves from cluster to cluster is shown in Figure 4.12.

4.7. EXAMPLE: INTEREST RATES AND INELATION 77

0.9

I
0,8
SPX vot. •SPX cluster

0.7

0.6

0.5

0,4

0.3

0.2

0.1

12/04/1949 20/12/1962 28/08/1976 07/05/1990 14/01/2004 22/09/2017

Figure 4.12: The original volatility and its clusters.

We can take this a step further and use it to give an idea of the likeli
hood of jumping from one volatility regime to another. And this is where I
cunningly bring back a very simple, weak time dependence.
Rather easily one finds the following matrix of probabilities, Table 4.3.

To:
Erom: Cluster 1 Cluster 2 Cluster 3
Cluster 1 84% 16% 0%
Cluster 2 38% 57% 5%
Cluster 3 0% 54% 46%

Table 4.3: Transition probabilities for a Jump volatility model.

We interpret this as, for example, the probability of Jumping from Cluster
1 to Cluster 2 is 16% every 30 days.

4.7 Example: Interest rates and inflation

Continuing with a financial theme, or rather economic, let's look at base

rates and inflation. This is motivated by how central banks supposedly use
base rates to control inflation. But again, this example is for illustrative
purposes. (I will eventually stop saying this.) There will certainly be an
important time component to these two variables, but I'm going to ignore it
78 CHAPTER 4. K MEANS CLUSTERING

in this KMC analysis.

I found data, shown in Figure 4.13, for United Kingdom interest rates
and inflation going back to 1751.

40
Interest Rate
30
~—Inflation

20
f.
f
r

10
I 1 i si! ill III. ll.
I
i
i i
hi Vi^ V
[NJ
U(5*v» V 1900 s' ^ 1950 2000

-10
$
j.y I I
-20 I
I

-30

Figure 4.13; Inflation and interest rates going back to 1751.

It seems unlikely that something simple like K means clustering will do a

good job with all of this data so I restricted myself to looking at data since
1945. In Figure 4.14 I have plotted inflation against the interest rate. I have
also Joined the dots to show how the data evolves, but, of course, this is not
modelled by the simple K means algorithm.

30

25

%
S
20

c
4^^
I
015
I
\
CD

^10 x:

>
5

0 u 5 10 15 20

-5 Interest Rate

Figure 4.14: Inflation versus interest rates, joined chronologically.

4.7. EXAMPLE: INTEREST RATES AND INFLATION 79

Even though the interest rate and inflation numbers are ballpark the same
order of magnitude I still translated and scaled them first.
With four clusters we find the results in Table 4.4.

Cluster 1 Cluster 2 Cluster 3 Cluster 4

No. in cluster 25 30 11 7
Description V. Low Base Rate, 'Normal Economy’ High Base Rate. High Base Rate.
Normal Inflation 'Normal Economy' Medium Inflation High Inflation
Interest rate Tm% 6.15% 11.65% IT77%
Inflation TTWo 6'.'89% 16.54%

Table 4.4: Clusters in interest rates and inflation. (In original scales.)

The centres of the clusters and the original data are shown in Figure
4.15. I have here plotted in the scaled quantities, and with the same length
of axes, to make it easier to see (and draw) the lines dividing the nearest
cluster.

5 • Data O Cluster Centres

O

High Base
Rate, High
3
Inflation .

.o •
2

\ ■ -High Base Rate,

.\' O Medium
67^ —•\. :—Inflatien
-3 -2 2 3 4
:6 - t
NormalX
Y Interest Rate
w* 1 v.V^ Economy
Very Low Bise
Rate, Nornfel
Inflation
-2

Figure 4.15: Inflation versus interest rate showing the four clusters. (Scaled
quantities.) This is a Voronoi diagram (see text).

How the cluster evolves through time is shown in Figure 4.16.

80 CHAPTER 4. K MEANS CLUSTERING

O)

£ 3
13
C

01
V, 2
=5

0
1940 1960 1980 2000 2020

Figure 4.16: Evolution of the cluster number.

Again we can look at the probability of jumping from one cluster to

another. The results are given in Table 4.5.

To:
From: Cluster 1 Cluster 2 Cluster 3 Cluster 4
Cluster 1 88.9% 11.1% 0.0% 0.0%
Cluster 2 5.6% 86.1% 8.3% 0.0%
Cluster 3 0.0% 25.0% 58.3% 16.7%
Cluster 4 0.0% 0.0% 33.3% 66.7%

Table 4.5: Transition probabilities from cluster to cluster.

An interesting experiment that you can do is to shuffle the data randomly.

In this example it means keeping the interest rate data, say, fixed and reorder
the inflation data randomly. If there was any structure to the data initially
then it will probably be lost during this shuffling.

4.8 Example: Interest rates, inflation and GDP growth

Take the data in the previous example and add to it data for GDP growth.
We now have a three-dimensional problem. (And now I’ve used quarterly
data for this.)
With six clusters I found that three of the clusters that I had earlier
4.9. A FEW COMMENTS 81

did not change much but the normal economy broke up into three separate
clusters. See Figure 4.17 for the results plotted in two dimensions.

40

O Cluster centres - Data

35

30

c 25
O

nj
20
High Base Rate, High Inflation
c •D
15

10
Normal’.EconorhvJ High.Base Rate,

5
I .*.•
..Megliui^lnflation •
..j t • *• *.
0
0 to
t •
Very LQiW Bast flate, 4 .
8 10 12 14 16 18

Norpgal Inflation
Interest Rate
-10

Figure 4.17: Inflation versus interest rate. (The GDP-growth axis would
come out of the page. Original scaling.)
The clusters are in Table 4.6.

Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6

Number in cluster 31 w 23 31 17 38
Interest rate 0.50% 5.77% 5.89% 7.44% 12.11% 12.56%
Inflation 2.24% T37% 5.59% 5l7% 18.89% 6.18%
GDP growth 0.48% 0.71% -0.63% 2.25% -0-51% 0.33%

Table 4.6; Clusters for interest rates, inflation and GDP growth.

4.9 A Few Comments

• Voronoi diagrams: Figure 4.15 is a Voronoi diagram. This is a par

titioning of a space into regions depending on the distance, usually
Euclidean but not necessarily, from a set of points. The use of Voronoi
diagrams dates back to Descartes in the 17th century. An interesting
use of them was by John Snow in 1854 to show how most people dying
in a cholera outbreak were nearest to a particular water pump.
82 CHAPTER 4. K MEANS CLUSTERING

Choosing K: If you are lucky then it will be clear from the error-
versus-ZsT plot what is the optimal number of clusters. Or perhaps the
number will be obvious, something about the nature of the problem
will be a clue. Not quite so convincing but it helps plausibility if you
can give each cluster a name, like I did in the first interest rate example.

There is a small tweak you must do to the distance measure if you

are going to do one of examples I mentioned right at the start of this
chapter. Specifically with reference to the neck sizes of shirts. Can
you guess what it is? Clue: If you have a 17-inch neck you aren't going
to be able to wear a 15-inch shirt.

Further Reading

For KMC implemented using the programming language R, see Applied

Unsupervised Learning with R: Uncover previously unknown patterns and
hidden relationships with k-means clustering, heirarchical clustering and RCA
by Bradford Tuckfield, published by Packt in 2019.
There are many books in the “For Dummies" series that cover differ
ent aspects of machine learning. KMC is found in Predictive Analytics Por
Dummies, 2nd Edition by Anasse Bari, published by John Wiley & Sons in
2016.

If you want to read about modelling volatility as a jump process then go

to Ito33.fr and download their technical publications.
1 did promise I could suggest a few excellent books on quantitative finance
if you want to learn about that subject. I can wholeheartedly recommend
Paul Wilmott Introduces Quantitative Finance, Paul Wilmott On Quantita
tive Finance (in three amazing volumes) and Frequently Asked Questions in
Quantitative Finance. All by Paul Wilmott and published by John Wiley &
Sons.
Chapter 5

NaYve Bayes Classifier

5.1 Executive Summary

Naive Bayes classifier is a supervised-learning technique. We have sam

ples of data representative of different classes. Then using Bayes Theorem
we calculate the probability of new data being in each class.

5.2 What Is It Used For?

Naive Bayes classifier is used for

Classifying data, often text, and analysing sentiment

Example: Natural Language Processing (NLP), getting the computer

to understand humans

Example; Email spam detection

Example: Classify whether a piece of news is good or bad

Example: Predict in which direction tweets on twitter are going to

influence an election or referendum

Example: Determine if said tweets are from a Russian bot

Naive Bayes Classifier (NBC) is sometimes explained using examples of

book reviews. By analysing the statistics of a review and comparing with
a catalogue of other reviews the algorithm learns which words appear in
good reviews and which in bad reviews. It is another example of supervised
learning since the training examples are already classified.

83
84 CHAPTER 5. NAIVE BAYES CLASSIEIER

First we need a large collection of good reviews, and a large collection of

bad reviews. We count up incidences of certain words appearing in each of
these categories. And thus the probability of each word appearing in each
class. Then if we are given a new review a straightforward use of Bayes
Theorem will give us the probability that the new review is either good or
bad.
Of course we are not restricted to just two classes, good or bad, nor does
it have to be book reviews. As my example will show.
The reason that the method is called naive is that the assumptions are
usually quite unrealistic. The main assumption being that words are inde
pendent of each other, which is clearly not the case. Flowever one always
reads that this doesn’t seem to matter all that much in practice.

5.3 Using Bayes Theorem

In our NBC example shortly we shall be applying Bayes Theorem to

political speeches. An example of Bayes Theorem would be

P(Politician is left wingjUses the word ‘‘Comrade'') =

P(Uses the word “Comrade"|Politician is left wing)7^(Politician is left wing)
f^(Uses the word “Comrade")

So if we hear a politician using the word "Comrade' this won’t be

something my American readers will have experienced — then we could
calculate the probability of him being left wing by studying the speeches
of left-wing politicians, and knowing how often politicians generally use the
word, and the probability of any random politician being left wing.

5.4 Application Of NBC

But we are going to apply NBC not Just to a single word but to whole
phrases and ultimately entire speeches. And also we don’t calculate the exact
probability of a politician being left wing. Instead we compare probabilities
for being left wing and right wing. You will see when I go through the details
in a moment.

Suppose we hear a politician saying “Property is theft. Comrade" and

want to know whether he is left or right wing. We’d like to know

F(Left|Says, “Property is theft. Comrade"). (5.1)

And similar for being right wing.

5.5, IN SYMBOLS 85

We use Bayes Theorem as follows. We first write (5.1) as

P(Left|Says, "Property is theft, Comrade") =

P(Says, "Property is theft, Comrade”|Left)P(Left)
P(Says, "Property is theft. Comrade")
And we’d write something similar for the probability of being right wing.
We compare for Left and Right and the larger determines which way our
politician swings. But note that we don't need to calculate the denominator
in the above. The denominator does not depends on the classes, whether
left or right. And so all we need to do is calculate the numerator for left and
right and then compare, to see which is the larger.
We just have to estimate

P("Property is theft. Comrade”|Left)P(Left)

and

P("Property is theft. Comrade”|Right)P(Right).

Now comes the naive part. We shall assume that

P(Says, “Property is theft. Comrade"|Left) =

P(Says, “Property” I Left) x P(Says, "is"|Left) x

P(Says, "theft" I Left) X P(Says, “Comrade"|Left).
That is, all the words are independent. (Which is clearly not true for most
sentences.) From training data we would know how often politicians of
different persuasions use the words “property,” "theft," etc. (Not so much
the "is.” That’s a stop word that we would drop.) Thus each of those
probabilities, for both class of politician, is easily found from many speeches
of many politicians.
Let’s write all this in symbols.

5.5 In Symbols

The text, or whatever, we want to classify is going to be written as x, a

vector consisting of entries Xm for 1 < m < M, so that there are M words
in the text. We want to find

F(Cfc|x) (5.2)
for each of the K classes (political persuasions) Ck- And whichever prob
ability is largest gives us our decision (that's just maximum likelihood as
mentioned in Chapter 2).
86 CHAPTER 5. NAIVE BAYES CLASSIFIER

Bayes tells us that (5.2) can be written as

P{Ck) P(x|Cfe)
P(x)
The denominator we can ignore because it is common to all classes (and we
are only comparing the numbers for each class, the exact number doesn't
matter).
If the features are all independent of each other then this simplifies — if
they’re not then this is much harder — and the numerator becomes
M

P{Ck) n P{^m\Ck).
m=l

This is what we must compare for each class. The term P{xm\Ck) we will
get from the data, just look at the speeches of other politicians and look at
the probabilities of words, the x^s, appearing and which political direction
they lean.
Finally, because we are here multiplying potentially many small numbers
we usually take the logarithm of this expression. This doesn't change which
class gives the maximum likelihood just makes the numbers more manage
able:
M

ln(P(Cfe))+ ^lri(P(x™|Cfc)).
m=l

In the terminology of Bayes Theorem the leading term, the ln(P(C'fe)),

is known as the prior (probability). It is the probability estimated before
taking into account any further information. Here we would typically get this
probability from knowledge of the population as a whole, here the probability
of a politician being in each class, or from our training set.

5.6 Example: Political speeches

I am going to work with the speeches or writings of some famous politi

cians or people with political connections. And then I will use this training
set to classify another speech.
The usual caveats apply here. This is not research-level material so I have
not gone to the lengths that I would if this were for an academic journal.
On the other hand it is not data that I have made up just for this book to
give nice results. This is very much warts and al l.
I have taken eight speeches/writings. For example one is:

A spectre is haunting Europe-the spectre of Communism. All the Powers

of old Europe have entered into a holy alliance to exorcise this spectre:
5.6. EXAMPLE: POLITICAL SPEECHES 87

Pope and Czar, Metternich and Guizot, French Radicals and German police-
spies. ..

This is immediately recognisable as the start of the Communist Manifesto

by Marx and Engels.
I have also used the opening thousand or so words from each of

• Churchill: "Beaches” speech

• JFK; Inaugural address

• Benn: Maiden speech as MB

• Thatcher: “The Lady’s not for turning" speech

• May: Syria strikes speech

• Corbyn: Post-Brexit-Referendum speech

• Trump: State of the Union speech

And I have classified them as either Left or Right.

Here’s a selection of words used by the left and their probabilities and
log probabilities:

Word Prob. Log Prob.

ablaze 0.013% -8.983

able 0.050% -7.597
abolish 0.025% -8.290
abolished 0.013% -8.983

abolishing 0.019% -8.578

abolition 0.088% -7.037
about 0.094% -6.968
above 0.019% -8.578
abroad 0.019% -8.578
absence 0.013% -8.983
absolute 0.031% -8.067

absolutely 0.013% -8.983

Table 5.1: Left-wing words.

88 CHAPTER 5. NAIVE BAYES CLASSIFIER

And by the right:

Word Prob. Log Prob.

add 0.029% -8.130

addiction 0.015% -8.824
additional 0.015% -8.824
address 0.015% -8.824
administration 0.081% -7.119
admits 0.015% -8.824
adopt 0.015% -8.824
advance 0.015% -8.824

advantage 0.022% -8.418

advantages 0.015% -8.824
adverse 0.015% -8.824
advert 0.015% -8.824

Table 5.2: Right-wing words.

There are a couple of minor tweaks we have to do before trying to classify

a new speech:

1. Remove al l stop words from the texts. Stop words, mentioned in

Chapter 2, are words like the, a, of, etc., words that don’t add any
information but could mess up the statistics. (There are some words
that are sometimes stop words but sometimes aren't, like like. A speech
with many likes in it is very, very easy to classify. As irritating.)
2. There will be words in our new to-be-classified speech that do not
appear in any of our training data. In order for them not to give a log
probability of minus infinity we give them a default log probability of
zero. Although there is indubitably some commodious information in
a speech that is rhetorically portentous.

And then I took the following speech:

happy join today down history greatest demonstration freedom history

nation years great American symbolic shadow stand today signed...

You will have noticed that I have removed a lot of stop words. So it’s
not as immediately recognisable as it might bel
For a bit of fun I took the speech and imagined doing a real-time analysis
of it, in the sort of way you might carefully listen to the report by a CEO
5.6. EXAMPLE: POLITICAL SPEECHES 89

and try to read into his presentation whether the news is good or bad before
quickly putting in your buy or sell order. And the results are shown in Figure
5.1. By 15 important words the speech has already set its tone. (Forty words
of actual speech.)

Probability of...
1

0.8

0.6

0.4

0.2

0 5 10 15 20 25 30 35 40

Numbers of words so far

Figure 5.1: Running probability.

This is supposed to be the probability that the speaker is right wing. Can
you guess who it is?
Well, the person is not exactly famous for being right wing. And he was
not exactly a politician. But he is famous for giving good speech. It is,
of course, the "I Have A Dream” speech by Martin Luther King Jr. What
does this tell us? Maybe MLK was right wing. Maybe this analysis was
nonsense, too little data, too little follow up. Or maybe the classifier has
found something I wasn't looking for, it’s lumped in MLK with Churchill and
Thatcher, and not with Jeremy Corbyn, because of his powers of rhetoric.
There are many ways you can take this sort of analysis, but with much
of ML (that's Machine Learning, not Martin Luther) you have to treat it
with care and don't be too hasty. If this were anything but an illustrative
example I would have used a lot more data, more politicians, more speeches
and done a ful l testing and validation. However I shal l leave it here as I am
simply demonstrating the technique.

Further Reading

For a short and inexpensive overview of the subject of NLP see Intro
duction to Natural Language Processing: Concepts and Fundamentals for
Beginners by Michael Walker, published in 2017.
Chapter 6

Regression Methods

6.1 Executive Summary

Regression methods are supervised-learning techniques. They try to ex

plain a dependent variable in terms of independent variables. The indepen
dent variables are numerical and we fit straight lines, polynomials or other
functions to predict the dependent variables. The method can also be used
for classification, when the dependent variables are usually zeroes and ones.

6.2 What Is It Used For?

Regression methods are used for

• Finding numerical relationships between dependent and independent

variables based on data

• Classifying data based on a set of numerical features

• Example: Find a relationship between the number of tomatoes on a

plant, the ambient temperature and how much they are watered

• Example: Determine the probability of getting cancer given lifestyle

choices (quantity of alcohol consumed, cigarettes smoked, how often
you visit Amsterdam, etc.)

You will no doubt know about regression from fitting straight lines through
a set of data points. For example, you have values and square footage for
many individual houses, is there a simple linear relationship between these
two quantities? Any relationship is unlikely to be perfectly linear so what
is the best straight line to put through the points? Then you can move on

91
92 CHAPTER 6. REGRESSION METHODS

to higher dimensions. Is there a linear relationship between value and both

square footage and number of garage bays?
As an introduction to this supervised-learning technique we shall start
with looking for linear relationships in multiple dimensions, and then move
on to logistic regression which is good for classification problems.

6.3 Linear Regression In Many Dimensions

We saw a brief summary of linear regression in one dimension in Chapter

2. Going over to multiple regression is mathematically simple. We now
have M independent, explanatory, variables, representing the features, and
so we write all xs as vectors. For each of N data points we will have the
independent variable (square footage of a house, number of garage bays,
etc.) and the dependent variable (property value).
We will fit the linear function hg{x) = to the ys, where 6 is the
vector of the as-yet-unknown parameters.
One subtle point is that because we usually want to have a parameter
6q that doesn't multiply any of the independent variables we write x as
(l,xi,... , where T means transpose, so it has dimension M + 1.
The cost function remains the same as in Chapter 2, i.e. the quadratic
function:
JV
1
AO) 2N
n=l
-y
)
As in Chapter 2 we can differentiate this cost function with respect to
each of the 9s, set the result equal to zero, and solve for the 9s. Easy.
However, although there is stil l technically an analytic expression for the
vector 6 it involves matrix inversion so in practice you might as well use
gradient descent.

Finding the parameters numerically using gradient descent

Although numerical methods are not needed when you have linear regres
sion in a single dimension you will need to use some numerical method for
anything more complicated. Batch gradient descent and stochastic gradient
descent have both been mentioned in Chapter 2. To use these methods you
will need
N

dJ _ 1 -y
(n)
de~ N
n=L
6.4. LOGISTIC REGRESSION 93

The Batch Gradient Descent Algorithm

Step 1: Iterate

New 6> = Old 6» -/3 dJ/d6

N
I.e. New 6> = Old 6» -P/NY,n=l

Step 2: Update
Update all 0k simultaneously. Repeat until convergence

Linear regression is such a common technique that I'm going to skip

examples and go straight to something different, regression used for classifi
cation.

6.4 Logistic Regression

Suppose you want to classify an email as to whether or not it is spam.

The independent variables, the xs, might be features such as number of Is
or number of spelling mistakes in each email. And your ys will all be either
0, not spam, or 1, spam. Linear regression is not going to do a good job
of fitting in this case. See Figure 6.1, would you want to fit a straight line
through this data? It would be nonsense.

Data

1 Sigmoidal

Linear(Data)

X
0

1 2 3 4 5 6 7 8

-1

Figure 6.1: Regression for a classification problem.

One way to see why this is nonsense is to add another dot to this figure,
a dot way off to the right, with y = 1. It is clearly not conflicting with the
94 CHAPTER 6. REGRESSION METHODS

data that’s already there, if it had y 0 then that nnight be an issue. But
even though it seems to be correctly classified it would cause a linear fit to
rotate, to level out and this would affect predictions everywhere.
Also sometimes the vertical axis represents a probability, the probability
of being in one class or another. In that case any numbers that you get that
are below zero or above 1, which you will get with a linear fit, will also be
nonsense.

The conclusion is that for classification problems we need something

better than linear. We tend to use a sigmoidal function, such as the logistic
function,
1
ho(x) =
X -(- ^-Bu-Oix ’
or, since we are going to go immediately to multiple regression,
1
/io(x) =
1 + e-e'x’

This function is shown in the figure. We would fit this function to the data
and then given a new data point (email) we would determine whether it was
spam or not according to a threshold for Iiq. We might have a threshold of
0.5, so that anything above that level would go into our spam box. Or we
might use a different threshold. If we are worried that genuine emails are
going into our spam box (we have friends who are very bad at speeling and
use lots of exclamation marks) then the threshold might be 0.8, say.

The cost function

An important difference when we want to do logistic regression instead

of linear regression is in the choice of cost function.
There are some obvious properties for a cost function: It should be pos
itive except when we have a perfect fit, when it should be zero; It should
have a single minimum. Our previous, quadratic, cost function satisfies the
first of these but when hg is the logistic function it does not necessarily have
a single minimum.
However, the following cost function does have these properties.
N
1
(n)
j{e) N
71=1
(2/^”^ In T (1 -y
)ln(l-/r,(x<”))))
(6.1)
So why does this cost function work? Remember that y can only take
values 0 or 1. When y = 1 the function J{0) is smoothly, monotonically
decreasing to a value of zero when hg is also one. And when y = 0 the cost
function is also zero when hg = 0. But perhaps most importantly this cost
6.5. EXAMPLE: POLITICAL SPEECHES AGAIN 95

function has a nice interpretation in terms of maximum likelihood estimation

for classification problems as explained in Chapter 2.
We no longer have an analytic solution for the minimum of the cost
function so we have to solve numerically. But rather beautifully we find that
with the new definition for ho and the new cost function we still have

N
dj
de
n=l

This means that our gradient descent algorithm remains, surprisingly, un-

changed.

6.5 Example: Political speeches again

In the last chapter I used some data from the speeches and writings of
politicians to determine the nature of an unseen speech. I'm not sure that
it quite went according to plan, but was nevertheless a fascinating exercise,
I thought. I'm going to use the same data, the same eight speeches, here
but in a different way, using a regression method.
Again I shall take speeches/writings by = 8 politicians. And I shall
label each politician as either 0 for left wing or 1 for right wing. These
are the for n = 1,..., N. But instead of looking at the frequency of
individual words as before I shall look at the types of words used. Are the
words positive words, negative words, irregular verbs, etc.? For a total of
M features. So the n'^*' politician is represented by x^”), a vector of length
M + 1. The first entry is 1. The second entry would be the fraction of
positive words the rC'' politician uses, the third entry the fraction of negative
words, and so on.
But how do I know whether a word is positive, negative, etc.? For this I
need a special type of dictionary used for such textual analysis.
One such dictionary, easily found online, is the Loughran-McDona ld dic
tionary. This is a list of words classified according various categories. These
categories are: Negative; Positive; Uncertainty; Litigious; Constraining; Su
perfluous; Interesting; Modal; Irregular Verb; Flarvard IV; Syllables. This list
is often used for financial reporting (hence the litigious category). Most of
the category meanings are clear. Modal concerns degree from words such as
"always" (strong modal, 1) through "can" (moderate, 2) to “might" (weak,
3). Harvard IV is a Psychosociological Dictionary.
The results of the analysis of the speeches is shown in Table 6.1. The
fractions of positive, negative, etc. words is very small because the vast
96 CHAPTER 6. REGRESSION METHODS

majority of words in the dictionary 1 used are not positive, negative, etc.

Benn Churchill Corbyn JFK M&Es May Thatcher Trump

Negative 0.501% 1.048% 1.216% 0.562% 2.067% 0.395% 1.626% 1.261%
Positive 0.213% 0.380% 0.517% 0.243% 0.760% 0.137% 1.018% 1.048%
Uncertainty 0.395% 0.441% 0.274% 0.137% 0.289% 0.091% 0.425% 0.289%
Litigious 0.152% 0.061% 0.274% 0.213% 0.441% 0.091% 0,304% 0.228%
Constraining 0.000% 0.030% 0.137% 0.122% 0.304% 0.122% 0.046% 0.122%
Superfluous 0.000% 0.000% 0.000% 0,000% 0.030% 0.000% 0,000% 0.000%
Interesting 0.030% 0.061% 0.000% 0.000% 0.152% 0.030% 0.106% 0.061%
Modal 1,307% 1.975% 1.945% 0,881% 1.276% 0.380% 2.097% 2.112%
Irregular verb 0.228% 0.745% 0.608% 0,334% 0.988% 0.289% 0.942% 0.699%
Harvard IV 1.352% 3.206% 4.513% 1.869% 7.765% 1.596% 4.270% 4.255%
Syllables 1.52 1.44 1.57 1.41 1.68 1.68 1.50 1.49

Table 6.1: Results of the analysis of political speeches.

Because I was only using eight politicians for the training I did not use
all 11 of these categories for the regression fitting. If I did so then I would
probably get a meaningless perfect fit. (Twelve unknowns and eight equa
tions.) So I limited the features to Just positive, negative, irregular verbs and
syllables.
I found that the 9s for positive words, irregular verbs and number of
syllables were all positive, with the 9 for negative words being negative. I
leave the reader to make the obvious interpretation.
However, you should know the disclaimer by now!
Finally I thought I would classify my own writing. So I took a sample
from that other famous political text Paul Wilmott On Quantitative Finance,
second edition. And I found that I was as right wing as Margaret Thatcher.
Take from that whatever you want.
But seriously, if you were to do this for real you would use the above
methodology but improve on its implementation in many ways. A couple of
obvious improvements would be;

• More training data, meaning many, many more speeches/writings from

a larger selection of politicians

• Use a better dictionary, one more suited to classifying politicians in

stead of one for analysing speeches about the growth in sales of widgets

6.6 Other Regression Methods

You can go a lot further with regression methods than the techniques
I've covered here. Just briefly. . .
6.6. OTHER REGRESSION METHODS 97

The usual suspects

There are many (basis) functions in common use for fitting or approxi
mating functions. You will no doubt have used some yourself. For example,
Fourier series, Legendre polynomials, Flermite polynomials, radial basis func
tions, wavelets, etc. All might have uses in regression. Some are particularly
user friendly being orthogonal (the integral of their products over some do
main, perhaps with a weighting function, is zero).

Polynomial regression

Rather obviously fit a polynomial in the independent variables. However,

the higher the degree of the polynomial the greater the danger of overfitting.
You can use OLS again to find the parameters by treating each non-linear
term as if it were another independent variable. So a polynomial fit in one
dimension becomes a linear fit in higher dimensions. That is, instead of

^0 T

think of
^0 T (^\X\ + 02X2-
Because of the obvious correlation between x and x'^ it can be tricky to
interpret exactly what the coefficients in the polynomial mean.

Ridge regression

I mentioned in Chapter 2 how one sometimes adds a regularization term

to OLS:
/ N \
1
■no) = 2N
- y
\n=l

And similarly for other cost functions. Here I am using 0 to mean the vector
with the first entry being zero (i.e. there is no 9q). And for comparison with
below, this is the L^ or Euclidean norm here. This extra penalty term has
the effect of reducing the size of the (other) coefficients.
Why on earth would we want to do this? General ly it is used when there
is quite a strong relationship between several of the factors. Fitting to both
height and age might be a problem because height and age are strongly
related. An optimization without the regularization term might struggle
because there won't, in the extreme case of perfect correlation, be a unique
solution. Regularization avoids this and would balance out the coefficients
of the related features.
98 CHAPTER 6. REGRESSION METHODS

Lasso regression

LASSO stands for Least Absolute Shrinkage and Selection Operator. This
is similar to ridge regularization but the penalty term is now the L^ norm,
the sum of the absolute values rather than the sum of squares.
Not only does Lasso regression shrink the coefficients it also has a ten
dency to set some coefficients to zero, thus simplifying models. To check
this last point, plot \9\ = constant in 0 space (stick to two dimensions!)
and draw comparisons between minimizing the loss function with penalty
term and minimizing the loss function with a constraint (see Chapter 2 and
Lagrange multipliers).

Further Reading

Applied Regression Analysis, 3rd Edition by Norman Draper and Harry

Smith published by Wiley covers everything you'll need for linear regression.
But it ain’t cheap.
After you've mastered linear regression go on to nonlinear with another
Wiley book Nonlinear Regression Analysis and Its Applications by Douglas
Bates and Donald Watts.
Chapter 7

Support Vector Machines

7.1 Executive Summary

Support vector machines are a supervised-learning classification tech

nique. You have classified data represented by vectors of features. This
method divides data according to which side of a hyperplane in feature space
each point lies.

7.2 What Is It Used For?

Support Vector Machines are used for

• Classifying data based on a set of numerical features

• Example: Identify plant types from features of their petals

• Example: Estimate the risk of prostate cancer from MRI images

• Example: Find likely customers from subjects mentioned in their Twit

ter posts

The Support Vector Machine (SVM) is another supervised-learning clas

sification technique. There's a little bit more mathematics to it than the
methods we've covered so far, hence me saving it until now. It's still imple-
mentable in Excel, at least as far as learning the technique is concerned if
you don't have too much data. As with all the methods we discuss if you
want to use them professionally then you are going to have to code them up
properly.

99
100 CHAPTER 7. SUPPORT VECTOR MACHINES

7.3 Hard Margins

In Figure 7.1 are shown sannple vectors divided into two classes (the
circles and the diamonds). There are very clearly two groups here. These
classes can be divided using a straight line, they are linearly separable. This
dividing straight line can then tell us to which class a new, unclassified,
sample belongs, depending on whether it is on the circle or the diamond side
of the line. With such a clear distinction between these groups it should be
easy to find such a line. It is, but it is so easy that there are many lines that
will do the job, as illustrated in the figure. So which is the best one?
One definition is that the best line is the one that has the largest margins
between it and the samples. This is shown as the bold line in the figure.

Figure 7.1: Two classes and three possible borders between them.

These margins are shown in Figure 7.2, where I've also shown the vector
that is orthogonal to the hyperplane dividing the two classes. I say hyper
plane because we will generally be in many dimensions, here of course this
hyperplane is Just the straight line.
Our goal is to find the hyperplane such that the margins are furthest
apart. The cases that define the margins are called the support vectors.
You'll notice that this section is called Hard Margins. This is a reference
to there being a clear boundary between the two classes. If one sample strays
into the wrong region, a diamond over to the circle side, say, then we will
need to do something slightly different from what immediately follows. And
we’ll look at that in the Soft Margins and Kernel Trick sections.
7.3. HARD MARGINS 101

Figure 7.2: Margins and the vector orthogonal to the dividing hyperplane.

Throughout this book I am trying to be as consistent as possible in

my notation. So I am going to use notation that is similar to, but subtly
different from, that in the previous chapter. For example I am going to
use 0 to denote the vector shown in Figure 7.2 that is orthogonal to the
hyperplane. This is not quite the same 0 as in Chapter 6, because it doesn’t
have that first entry, the Oq, I’m going to keep that separate. It therefore
has M dimensions. Similarly is the M-dimensional vector representing
the n"' sample. I shall use as the class label, being +1 for all diamonds
and —1 for all circles. Again this is slightly different to earlier when we had
0 and 1 as being the labels. We now want some symmetry to keep the
mathematics elegant and so we use instead labels ±1.
The dividing hyperplane is all points such that

6»'^x + 6>o = 0,

where we have to find the vector 0 and the scalar 00-

Now you could ask, “Why not just have 0o = T Paul?" After all we
could simply rescale the above equation. And I would answer, “Because I
want the margins to be represented by x=*= where

0'^x’^ + 00 = ±T” (7.1)

And that's where the scaling happened. The + refers to the margin close to
the diamonds, and the — to the margin close to the circles.
From (7.1) we have
0^(x+ -X")= 2. (7.2)
102 CHAPTER 7. SUPPORT VECTOR MACHINES

Also, for the diamonds (labelled = +1) we will have

0"' +9o>l (7.3)

and for the circles (labelled = -1)

+00 < -1. (7.4)
Multiplying these last two equations by the label they can both be
expressed as simply

yin)(0T^(n) + - l> 0. (7.5)

We also know from Equation (7.2) that (you might have to refresh your
memory about the meaning of the inner product of two vectors in terms of
distances and angles)
2
margin width =
i^r
And so maximizing the margin is equivalent to finding 0 and 6q to minimize
|0| or
1

e,9o 2' '

Of course, subject to the constraint (7.5) for all n.
Once we have found 0 and 6q and given a new, unclassified, data point
u, say, we Just plug everything into

0^ u + 00 (7.6)
and depending on whether this is positive or negative we have a diamond or
a circle. This is known as the primal version of the classifier.
We shall take a break from the mathematics here and implement a simple
example in Excel.

7.4 Example: Irises

There is a popular data set often used in machine learning, a dataset of

measurements for different types of iris, the flower. You can get the dataset
here: https://www.kaggle.com/ucinil/iris. I’m not usually keen on
using the same data as everyone else. Everyone perfecting their algorithms
on the same data is obviously dangerous.
There are three types of iris in the data: Setosa; Versicolor; Virginica.
And there are 50 samples of each, with measurements for sepal length and
width, petal length and width. (My daughter had to remind me what a sepal
was.) The first few lines of raw data in the csv file look like this:
7.4. EXAMPLE: IRISES 103

SepalLength,SepalWidth,PetalLength,PetalWidth,Name
5.1,3.5,1.4,0.2,Iris-setosa
4.9,3,1.4,0.2,Iris-setosa
4.7,3.2,1.3,0.2,Iris-setosa
4.6,3.1,1.5,0.2,Iris-setosa
5,3.6,1.4,0.2,Iris-setosa

In Figure 7.3 are shown the 50 samples for each type of iris, the data
being just petal length and sepal width. (I can't plot in four dimensions!)

4.5

4
%

3.5

T3 3

5 2.5
2
OJ

1.5

1 # Iris-setosa
a Iris-versicolor
0.5
A Iris-virginica
0

0 1 2 3 4 5 6 7

Petal length

Figure 7.3: Sepal width versus petal length

For illustrating the SVM method I am going to make this as simple as

possible and Just use the setosa and versicolor varieties since eyeballing them
shows a very clear demarcation. And Just in two dimensions, sepal width and
petal length, so I can draw the pictures easily. Any results you see below are
only in those two dimensions. I would encourage my readers to repeat the
analyses but in the full four feature dimensions of the dataset.
Results are shown in Figure 7.4. Ok, not the most challenging example.
I found the hyperplane for the two-dimensional, two-class, iris problem
to be

0.686 X sepal width 1.257 X petal length -|- 1.057 = 0.

The two margins are given by the same equation but with a plus or minus
one on the right.
104 CHAPTER 7. SUPPORT VECTOR MACHINES

Q. 3
cu
LO
2

0
0 1 2 3 4 5 6

Petal length

Figure 7.4: The samples, the hyperplane and the margins.

Now given a new iris with values for sepal width and petal length we
can determine whether it is setosa or versicolor by plugging the numbers
into Expression (7.6), or equivalently the left-hand side of the above, and
seeing whether the number that comes out is positive or negative. A positive
number for the left-hand side means you have a setosa. If the number has
absolute value less than one it means we have an iris in the middle, the
no-man's land. We wil l still have a classification but we won’t be confident
in it.

As a little experiment see what happens if you move one of the data
points, say a setosa, clearly into the versicolor group. When you try to solve
for the dividing hyperplane you will find that there is no feasible solution.

7.5 Lagrange Multiplier Version

We can take the mathematics further. Because we have a constrained

optimization problem (with inequality constraints) we can recast the problem
using Lagrange multipliers, the as > 0 below, as finding the critical points
of
N

n=l
+
9.) -1) (7,7)

To find the critical points we differentiate with respect to the scalar 9q

and with respect to the vector 6. Differentiation with respect to a vector
just means differentiating with respect to each of its entries, and the end
7.5. LAGRANGE MULTIPLIER VERSION 105

result is pretty much what you would expect. Setting these derivatives to
zero you end up with
N

n) = 0
(7.8)
n=l

and

=0.
n=l

The second of these gives us 9 (albeit in terms of the as):

X
(7.9)
n=l

which means that our vector orthogonal to the hyperplane is just a linear
combination of sample vectors.
Substituting Equation (7.9) into (7.7) and using (7.8) results in

N N N

^=E n=l
an
(7.10)
i=l j=l

We want to maximize L over the as, all greater than or equal to zero,
subject to (7.8). This is known as the dual problem.
Once we have found the as the dual version of the classifier for a new
point u is then Just
N

^a„y(")x(”)'uT0o,
n=l

whether this is greater or less than zero.

The maximization of L will return almost all as as zero. Those that aren't
zero correspond to the support vectors, the cases that define the margins.
Also note that if the problem is not linearly separable then the Lagrange
formulation is not valid. If that is the case then solving the dual problem
might give you some as but if you were to do a sanity check to see whether all
your sample points are correctly classified then you'd find some that aren't.
I.e. the dual problem has given a solution to a problem that doesn't have a
solution.
106 CHAPTER 7. SUPPORT VECTOR MACHINES

4
5

Q. 3
01

0 1 2 3 4 5 6

Petal length

Figure 7.5: The support vectors and the margins.

When \we solve the dual problem for the two-dimensional, two-class, iris
classification we find that a for the versicolor support vector shown as the
hollow diamond in Figure 7.5 is 1.025, for the upper hollow circle is 0.810,
and for the lower hollow circle 0.215. And indeed all other as are zero.
The end result is quite simple in that the decision boundary is defined
using just a small subset of the sample vectors. This makes classification of
new data very quick and easy.

7.6 Soft Margins

If there is clear water between your groups then SVM works very well.
What if our data is not linearly separable? I'm going to explain two
possible ways to address this situation. (In practice you might use both
methods simultaneously.) In the first method we try to do the best we can
while accepting that some data points are in places you'd think they shouldn't
be. It's all about minimizing a loss function.
In the second method we transform our original problem into higher
dimensions, thereby possibly giving us enough room to find genuinely linearly
separable data. So, to the first method.
7.6. SOFT MARGINS 107

Let me introduce the function

N
1
oXn)
N
n=l
max ~y
+0„),0) +A|0|2 (7.11)

This we want to minimize by choosing 6 and 0o. Comments...

• This looks quite a lot like Equation (7.7), but without the different as,
with a maximum function, and a change of sign

• The parameter A is a weighting between the margin size and whether

or not the data lies on the correct side of the margin

• In the language of optimization the first term is the loss function and
the second term is the regularization
• The maximum function in the first term has the effect of totally ig
noring how far into the correct region a point is and only penalizes
distance into the wrong region, measured from the correct margin.
The plot of the maximum function against its argument is supposed
to look like a hinge, giving this loss function the name hinge loss

Gradient descent

Because this function is convex in the 6s we can apply a gradient descent

method to find the minimum.
For example, with stochastic gradient descent pick an n at random and
update according to

New 6o = Old 6q — /3
-y
(n) if 1 - yin) (e'l'^in) + >0

0 otherwise

and

New 0 = Old 0-/3

2X9 -

2X9
if 1 -
otherwise
(^0'x(”) + 6lo) >0

Technically you might want to call this sub-gradient descent because the
derivatives aren't defined at the hinge.
In Figure 7.6 I've taken the earlier iris data and moved one setosa deep
into versicolor territory. The resulting margins have become wider, and are
softer since a few data points are within those margins.
You can also use the soft-margin approach even if you have linearly sep
arable data. It can be used to give wider margins but some points will now
lie within the margins.
108 CHAPTER 7. SUPPORT VECTOR MACHINES

7
/
/

/
6 /
/

/
/
5 /
JZ

■O
4
y

ro ,
Q. 3
O)
CO
2

0
0 1 2 3 4 5 6

Petal length

Figure 7.6: Soft margins.

7.7 Kernel Trick

We might have data that is not linearly separable but which nonethe
less fal ls neatly into two groups, just not groups that can be divided by a
hyperplane. A simple example would be that shown in Figure 7.7.

a
% m

&

1
^4^
•a » t
a •
• a a

Figure 7.7: Two clear groups, but not linearly separable.

7.7. KERNEL TRICK 109

In this problem we could change to polar coordinates in which case the

data becomes trivially linearly separable.
But there's another thing we could do and that is to go into higher
dimensions.

For the data in Figure 7.7 we could go to three dimensions via the trans
formation

(xi,.T2) ^ {xuX2,xl+xl).
This would have the effect of lifting up the outer group of dots higher than
the inner group in the new third dimension, making them linearly separable.
But once we start moving into higher dimensions we find that the training
procedure gets more time consuming. Is there any way to take advantage of
higher dimensions without suffering too much from additional training time?
Yes, there is. And it follows on from two observations.
First observation: Thanks to the dual formulation, in (7.10), we can see
that finding the as depends only on the products
Second observation: If you plug (7.9) into (7.6) you can see that the
classification of new data only depends on the products u.

So there is something special about this product.

If we use (f){x) to denote a transformation of the sample points (into
higher dimensions) then the above observations show us that all we need to
know for training and for classifying in the higher-dimensional space are

and

0(xW)'^^(u),
for our transformed data.
So rather than needing to know the exact data points we actually only
need a lot less information: The products of the vectors. And this leads on
to an idea that exploits these observations to address problems that are not
immediately linearly separable. And this idea is powerful because we can go
to many, perhaps even infinite, dimensions without adding much numerical
complexity or programming time.
The function 0(x) is known as a feature map. And what is clever about
the feature-map transformation is that you don’t need to explicitly know
what the map is in order to exploit it!
The trick in Kernel Trick is to transform the data from our M-dimensional
space into a higher-dimensional space where the data is linearly separable.
Let me show how this idea works with an example. Specifically let's work
with the example of ^(x) transforming a two-dimensional vector {xi,X2)'^'
into a six-dimensional vector

{b, V2ab Xl V2ab X2,ax\,ax\,\f2 ax\X2^

110 CHAPTER 7. SUPPORT VECTOR MACHINES

where a and b are constant parameters. (Obviously this transformation was

not chosen at random, as we shall see.)
So

b'^ + 2ab xix\ + 2ab X2X2 + a'^ xlx[ + X2X2 + 2a'^XiX2x[x2-

This can be written as

{axix\ + ax2X2 + 6)^ =(ax^x'+ 6)^ . (7.12)

Underwhelmed? Then I haven't explained properly! The key point is that

we have gone from two to five dimensions — the constant in the first position
doesn't really count as a dimension — opening up all sorts of possibilities
for finding linearly separable regions but we didn't need to know the details
of the transformation (/>(x). All we needed was that the product in the new
variables is a function of the product in the original variables. That function
is (7.12).
And all we do now with SVM in the higher-dimensional space is to replace
al l instances of
x''x'

with the kernel function

7C(x,x') = (ax”^ x' T &)^ •

Computationally speaking there is little extra work to be done, just an
addition and raising to a power.
Of course, this is not the only kernel for which this trick works. Here are
a few more:

Af(x,x') = [ax'x'+ bY , polynomial kernel,

X —

?C(x,x') = exp , Gaussian or Radial Basis Function kernel,

2a2

|x-x'|
/C(x, x') = exp 2a
, exponential kernel

and so on. Google SVM + kernel polynomial + gaussian + “Inverse

Multiquadric Kernel" and you should hit some lists.
After swapping the kernel function into (7.10) the quantity to maximize
becomes

N N N

^=E Oti (7.13)

n=\ 2=1 j=\
7.7. KERNEL TRICK 111

The observant will look at the above examples of kernel functions and
say "I can see the product in the polynomial kernel, but I don’t see it in the
Gaussian kernel.” Well, it is in there but we need to do a little bit of work
to see it.

First of all, what do we mean by the product of two vectors in the present
context? Let's write 4>{x.) in long hand:
4>(x.) = {4>i(Xi,..., Xn),(f>2(Xl,...,.X„),...,(prnixi X^)).
Notice how we’ve gone from ntom>n dimensions. So the product we are
interested in is
m

ii ■
i=l

The key element in this is that the sum is made up of products of the same
function of the two initial coordinates {xi,...,Xn) and (xj,... ,x^). So our
goal is to write the Gaussian kernel in this form. Here goes

X —
xf / - 2x''x'+ x'\
exp = exp
2(t2 2ff2
V
T T
X^ X 7x^x' x'
= exp exp exp
2cr2 cr2 2a2

The first and last terms in this are of the form that we want. But not; yet,
the middle. That is easily dealt with by expanding in Taylor series. The
middle term is then
OO
^ 1 /x:^x'\ *
2^ i\ V O'2
«=0

And the job is done, because each of these terms is a polynomial kernel. So
indeed the Gaussian kernel does have the required product form, albeit as
an infinite sum and therefore in infinite dimensions.
Note that some kernels take you from a finite-dimensional space to an
other finite-dimensional space with higher dimension. The polynomial kernel
is an example of this. However others take you to an infinite-dimensional
space.
If we go back to the iris problem in which I moved one setosa into the
versicolor region then using a kernel might be able to make this a separable
problem. However in so doing the boundary of the two regions would in
evitably be very complex. We might find that classifying unclassified irises
would not be very robust. It would almost certainly be better to treat the
odd iris as an outlier, a one off. So if you have data that is only linearly
inseparable because of a small number of outliers then use soft margins. If
however the original problem is not linearly separable but there is structure
then use a kernel.
112 CHAPTER 7. SUPPORT VECTOR MACHINES

Further Reading

You can often find inexpensive, often self published, monographs on

machine-learning topics, sometimes they will be a student’s thesis. Sadly
SVM doesn’t seem to be such a topic and most books are expensive. So if
you could only afford one book it might be Learning with Kernels: Support
Vector Machines, Regularization, Optimization, and Beyond by Bernhard
Schdikopf and Francis Bach published by MIT Press. It's quite advanced
though.
And, of course, The Nature of Statistical Learning Theory by one of the
founders of SVM, Vladimir Vapnik.
Chapter 8

Self-Organizing Maps

8.1 Executive Summary

A self-organizing map is an unsupervised-learning technique. We start

with vectors of features for all our sample data points. These are then
grouped together according to how similar their vectors are. The data points
are then mapped to a two-dimensional grid so we can visualize which data
points have similar characteristics.

8.2 What Is It Used For?

Self-organizing maps are used for

• Grouping together data points according to similarities in their numer

ical features and visualizing the results

• Example: Group people according to the contents of their supermarket

baskets

• Example: Group countries together according to demographics and

economic statistics

A self-organizing map (SOM) is a simple unsupervised-learning method

that both finds relationships between individual data points and also gives
a nice visualization of the results. Some people think of this technique as
a version of a neural network. The end result of the method is a typically
two-dimensional picture, consisting of a usually square array of cells as in
Figure 8.1. Each cell represents a vector and individual items will be placed
into one of the cells. Thus each cell forms one group and the closer that two
cells are to one another the more they are related.

113
114 CHAPTER 8. SELF-ORGANIZING MAPS

Figure 8.1; An array of cells.

8.3 The Method

The data

Each item that we want to analyze and group is represented by a vector,

dimension M, of numerical data. There will be N such items, so n = 1
to N.
As is often the case we shall be measuring and comparing distances.
So we need to make sure that the magnitude of distances are similar for
all elements. Just as we've done before we should translate and scale, to
either make mean and standard deviations the same, or the minimum and
maximum, for all features.
Each group/cell/node is represented by a vector also of dimension
M. And we'll have K such groups. For visualization purposes in two dimen
sions it is usual to make K a square number, 25, 64, 100.. . .These vs are
often called weights.
But we are getting ahead of ourselves. Before we go on to talk about
two-dimensional arrays and visualization let's talk about how we determine
in which group we put item n. We have the vector and we want to
decide whether it goes into Group A, B, C, D, ... , this is represented in
Figure 8.2.
8.3. THE METHOD 115

Group A v(i)

Group B V(2)

X(n) Group C y(3)

X, Group D y(4)

Group E y(5)

Figure 8.2: Which group should item n go into?

Remember that all of and the vs are vectors with M entries. All we
do is to measure the distance between item n and each of the K cell vectors
v:

|x(")
rn=l

The k for which the distance is shortest

argrnin
k

is called the Best Matching Unit (BMU) for that data point n.

But how do we know what the vs are?

Finding the BMU is straightforward. But that's assuming we know what

the vs are. We could specify them manually: Classify dogs according to
different characteristics. But, no, instead we are going to train the system
to find them.
We begin by redistributing our v vectors in a square array, as shown in
Figure 8.3. However unless there is some relationship between cells then this
redistribution is without any meaning. More anon.
116 CHAPTER 8. SELF-ORGANIZING MAPS

Qt

Figure 8.3: Redistributing the vectors into an array.

We are now ready to bring everything together... via learning.

8.4 The Learning Algorithm

The Self Organizing Maps Agor [ I

Step 0: Pick the initial weights
We need to start with some initial vs, which during the learning
process will change. As always there are several ways to choose
from, perhaps pick K of the N items, or randomly generate vectors
with the right order of magnitudes for the entries.

Step 1: Pick one of the N data vectors, the xs, at

random

Suppose it is n nt- Find its BMU.

We are going to be picking from the xs at random as we iterate
during the learning process. Keep track of the number of iterations
using the index t. So t starts at 1, then next time around becomes
2, and so on.
8.4. THE LEARNING ALGORITHM 117

Step 2: Adjust the weights

Change all of the weights for fc = 1 to iC slightly to move in

the direction of The closer a cell is to the BMU that we’ve
just found the more the weight is adjusted. This is where it gets
interesting. Let’s see the mathematical details.
What do we mean by closer? This is where the topography of the
two-dimensional array comes in, and the distribution of cells becomes
meaningful. Look at Figure 8.4 which shows the current BMU and
surrounding cells.

Figure 8.4; The BMU and neighbouring cells.

Measure the distance between the BMU cell and other nodes, ac
cording to the obvious

Distance = D = y/{a - aeMu)^ -(6 - &bmu)^- (8.1)

This distance is going to determine by how much we move each

in the direction of our randomly chosen item The as and bs
are integers representing the position of each node in the array.
Note that this is not the same distance we measured when we were
comparing vectors. It is the distance between the cells, as drawn on
the page and nothing to do with the vectors that these cells represent.
And here is the updating rule:

,(fc) -F
/? (x("‘> - for all 1< k<K.
118 CHAPTER 8. SELF-ORGANIZING MAPS

Here P is the learning rate. It is a function of the distance, D, in

(8.1) and the number of iterations so far, t.
I have to confess that the next part is where I personally start to get
a bit disappointed. And that's because, like so much of numerical
analysis, what you do is a bit arbitrary. So, for example, we might
choose the function P{D,t) to be
Z?2
P{D,t)= 77(f)exp ^ 2o-(f)2
where
t
r}{t) = ?7o exp
Tr,
and
t
a(t) ~ (To exp
Ta

This can be interpreted as;

• The adjustment to each weight decreases with time (i.e. iter

ation)
• The adjustment to each weight decreases with distance from
that iteration's BMU

• The effective range over which each weight is adjusted de

creases with iteration

See what I mean by a bit arbitrary?

If entries in the weights are mean zero, range of one, then a suitable
Tjo might be around 0.2. And ao might be around \\fK- There
are typically two phases during the learning process. The first is
a topological ordering where adjacent cells change to have similar
weights and then convergence.

Step 2; Iterate

Go back to Step 1 above and repeat until convergence.

8.5. EXAMPLE: GROUPING SHARES 119

8.5 Example: Grouping shares

Suppose you want to classify constituents of the SiS^PSOO index. You

could try:

1. Grouping according to sector, market capitalization, earnings, gender

of the CEO,. . .

2. Measure expected returns, volatilities and correlations

3. Principal Components Analysis of returns

But that's what everyone does. The second of these is the popular (at
least in the text books) Modern Portfolio Theory (MPT).
MPT is a clever, Nobel-Prize winning, way of deciding which stocks to
buy or sell based on three types of statistical information: Each stock’s ex
pected return; Each stock’s volatility; The correlations between all stocks'
returns. In this method one aims to maximize the expected return of a port
folio of stocks while holding its volatility, or risk, constant. To achieve this
it exploits correlations between stocks. If two stocks both have a good ex
pected growth but those returns are uncorrelated then spreading your money
across a portfolio of the two of them will be better than holding just one.
And so on for portfolios of any number of stocks. The problem with this
elegant theory is that in practice the parameters, especially the correlations,
are very unstable.
Perhaps SOM might be a more stable method? It is also nonlinear gen
erally, so might capture something that MPT doesn’t.
However, having asked that question I’m not going to fully answer it. I
shall use SOM on stock returns data but won't go too far in the direction
of portfolio selection. That is more for a finance research paper than for
illustrating machine learning.
What I am going to do is a slightly unusual example of self-organized
maps. Usually one would have significantly different features for each stock
(Just like we had different crimes for each local authority when we did K
Means). Perhaps we would have the sector, market capitalization, earnings,
gender of the CEO, etc. mentioned above. But I want to tie this in more
closely to the MPT of classical quantitative finance. For that reason I am
going to use annual stock price returns as a feature. So for each stock I will
have a vector of five features, the last five annual returns. I have chosen
annual data because it fits in with the sort of timescale over which portfolio
allocation methods are used, one doesn’t usually reposition one’s portfolio
every day.
I will use annual returns for each of 476 constituents of the S&P index.
(“Why not 500?" you ask? "Because some stocks left the index and
others Joined during this period," I reply. Since some stocks have left the
120 CHAPTER 8. SELF-ORGANIZING MAPS

index there will be a survivorship bias in my results. But if you're worried

about biases then the choosing-fun-examples bias is probably going to be
more important, and it's throughout this book.)
These returns are the x^"^s. So the features of the stock are returns and
the rn^*' entry in each is the return over the to*'’ year.
And I wil l have K = 25.

A note on scaling Because I've chosen features which are very similar,
just returns over different years, scaling is not as crucial as it would be with
features that are significantly different. So in what follows I have left all data
unsealed.
Let's see what the algorithm gives us.
In Figure 8.6 we see how many of the 476 stocks appear in each node,
with the corresponding contour plot in Figure 8.6. Clearly some nodes are
more popular than others. In those nodes there are a lot of stocks that move
closely in sync.

60

50

40

30

20

10

0 1

0 1

Figure 8.5: Number of stocks per node.

8.5. EXAMPLE: GROUPING SHARES 121

mu-zu

Figure 8.6: Number of stocks per node. Contour plot.

The most popular node, the one with the most stocks, is number 20. Its
constituents are shown in Table 8.1.

ADM BAX DVN GRMN KMX Nl PWR WDC

ADS BEN EBAY HAL KSU NOV PXD WMB
AKAM BWA EOG HES LNC NRG QCOM WYNN
AMG CF ETN HOG LUK NSC RRC XEC
APA CHK FCX HP MOS NTAP STX
APC CMI FLS HST MRO OKE UAL
ARNC COP FMC IP MU PKG UNP
AXP DISH FTI KMI NBL PNR URI

Table 8.1: Constituents of Node 20

Inspired by MPT one could easily use the growth and volatility of each
stock together with the map structure to optimize a portfolio in a way that
would be different from classical methods. For example diversification could
be achieved by not buying stocks that are in nodes that are close to each
other. In Figure 8.7 is shown the average in each node of the ratio of the
stocks’ growths to the volatilities. Perhaps these numbers will be useful. I
shall leave these possibilities as an exercise for the reader. (But please don’t
122 CHAPTER 8. SELF-ORGANIZING MAPS

put any of your hard-earned cash on any investments based on my analysis.)

Figure 8.7; Average growth/volatility for each Node.

I want to finish this example by looking at what this solution has to say
about financial sectors. In Figure 8.8 I show the number of stocks from each
sector in each cell.

Node 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Consumer Discretionary 1 4 2 0 4 3 3 2 4 3 3 0 0 1 5 0 2 1 7 6 2 1 14 3 3
Consumer Staples 0 2 7 1 0 1 2 0 1 1 0 2 1 1 2 0 0 0 0 1 0 1 4 4 1
Energy 1 0 0 0 0 0 0 0 3 3 0 0 0 0 2 0 0 0 0 18 1 0 0 1 1
Financials 0 1 0 0 0 1 4 4 1 1 0 1 0 2 1 0 7 2 0 5 25 3 3 1 1
Healthcare 0 0 1 0 4 8 4 0 2 0 2 0 0 1 3 4 1 3 1 1 1 0 8 11 3
Industrials 0 2 0 1 0 3 6 4 6 2 1 0 0 0 1 1 6 2 0 11 10 2 4 1 1

Information Technology 3 0 0 0 3 6 11 2 2 1 1 1 0 0 3 2 5 0 1 8 9 3 0 1 1
Materials 1 0 0 1 1 0 1 1 0 2 0 1 0 0 0 0 0 2 1 6 3 1 1 1 0
Real Estate 6 8 6 1 0 0 0 0 0 0 4 0 0 1 0 1 0 0 0 1 0 0 0 1 2
Telecommunication Services 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0

Utilities 2 6 7 1 0 0 0 0 1 0 0 2 3 2 0 0 0 0 0 2 1 0 0 1 0

Figure 8.8: Number of stocks from each sector in each cell.

Figure 8.9 shows this as a 3D plot. Node numbers from 1 to 25 run left
to right and sectors go into the page.
8.5. EXAMPLE: GROUPING SHARES 123

ss Consumer Discretionary

M Consumer Staples

^ Energy

PPinsncials

SI Health Care
If ^ Industrials

M Information Technology

r Materials

Real Estate

Telecommunication Services

Utilities
Node number

Figure 8.9: Sectors in each node.

It's not easy to appreciate this in black and white but you can get a rough
idea of what is going on, especially if you also look at the contour map in
Figure 8.10.
There are clearly some sectors that are very focused: Financials are con
centrated around Node 21; Real Estate around Nodes 1-3. Energy around
Node 20.
There are some sectors that are very spread out, such as Consumer Dis
cretionary.

Consumer Discretionary
i- Consumer Staples
Energy
1 Financials
[\
f---f Health Care

Industrials

Information Technology
Materials

Real Estate

Telecommunication Services
.1.
— Utilities
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Figure 8.10: The number of stocks from each sector in each cell. The
contour map.

Note that I never told the algorithm anything about sectors. But it does
look as if SOM has found something rather sector-like, but only for some
sectors.
124 CHAPTER 8. SELF-ORGANIZING MAPS

8.6 Example: Voting in the House of Commons

At time of writing the UK is going through some political turmoil. (As

you are reading this in the future, how did it al l turn out?) So I couldn’t
resist doing something with voting in Parliament. I downloaded voting data
from https://www.publicwhip.org.uk.
I took the last 18 months of voting records of Members of Parliament,
who voted, in which direction and who abstained or wasn't present.
There are 650 members of Parliament, the MPs. However, the data
gets a bit confusing. Some MPs don't appear in the data because Sinn
Fein MPs never attend the House of Commons, they don’t fancy swearing
allegiance to the Queen apparently. And quite a few MPs have been kicked
out of the Labour party, and other MPs have become independent. Every
time something like that happens the MP gets a new ID Number which
complicates things a bit. I haven't dealt with the redesignation of such MPs
properly, but it would be interesting to look at their voting before and after
changing allegiance.
Each entry in the vector represents an individual’s position on a specific
vote. I have used +1 to represent a vote in favour, —1 for a vote against, and
zero for abstention or not being present. Now obviously turning such data
into numbers has to be meaningful for this method to work. Fortunately the
ordering no, abstention, aye is quite well represented by —1, 0, 1. If we had
used data for sock colour instead then replacing red, blue, yellow, striped,
spotted,.. .with numbers would have been silly.
The results are shown in Figures 8.11 and 8.12. The first figure shows
that there are two main groupings, surprise surprise.

300

250

200

150 I...
■'--ic.l.'. z-

100
9
<4.
50
■f- 7

5
0
T*'-
0 V;
1 3
2
3
4
5
6
7 ' 1

Figure 8.11: Number of MPs per node.

8.6. EXAMPLE: VOTING IN THE HOUSE OE COMMONS 125

The second figure shows the breakdown of the nodes by political party.
Conservative and Labour are predominantly in completely different areas of
the grid. However Labour are a little bit more spread out. Interestingly,
there are members of the main parties scattered about, not being where
you'd expect them to be. It would be interesting to look at those MPs who
don't seem to be affiliated with the correct party and why. Note that nodes
11, 21, 31, etc. are all close together in the grid.

Conservative

200

180 Labour, ,,
160

140

120

100

60

40

20

0
1 11 21 31 41 51 61 71 81 91
Node number

Figure 8.12: Political affiliation and nodes.

Obviously I have used this data for illustrative purposes. If this were
being done for real I would take a lot more care, such as testing the results,
and also probably break down the data according to things like the subject
of the vote.

Further Reading

You might want to take a look at the Springer book Self-Organizing

Maps: Third Edition by the creator of the subject Teuvo Kohonen.
There is a paper by Claus Huber, "R Tutorial on Machine Learning; How
to Visualize Option-like Hedge Fund Returns for Risk Analysis,” on SOM
applied to hedge funds, including R code, in Wilmott magazine January
2019 pp36-41. It is also available for free on wilmott.com.
Chapter 9

Decision Trees

9.1 Executive Summary

Decision trees are a supervised-learning technique. Really they are just

flowcharts. They can be used for classification and for regression, so you'll
also hear them referred to as Classification and Regression Trees (or CART).
If used for classification then we start with labelled data points with mul
tiple features. These features describe each data point according to their
attributes (tall or short, man or woman, or numerical values). The method
uses a hierarchy of divisions of the data points according to these attributes.
Trees can also be used for regression if the data has associated numerical
values.

9.2 What Is It Used For?

Decision trees are used for

• Classifying data according to attributes that can be categories or nu

merical

• Example: Predict the number of votes in the Eurovision Song Contest

by looking at type of song, beats per minute, number of band members.
etc.

• Example: Forecast IQ based on books read, type, subjects, authors.

etc.

You will no doubt have seen decision trees before, although perhaps not in
the context of machine learning. You will definitely have played 20 Questions

127
128 CHAPTER 9. DECISION TREES

before. “Is the actor male?" Yes. Is he under 50?” No... “Is he bald?'
Yes. "Bruce Willis?” Yes. If the answer to the first question had been “No”
then you would have taken a different route through the tree.
In 20 Questions the trick is to figure out what are the best Yes/No
questions to ask at each stage so as to get to the right answer as quickly as
possible, or at least in 20 or fewer questions. In machine learning the goal
is similar. You will have a training set of data that has been classified, and
this is used to construct the best possible tree structure of questions and
answers so that when you get a new item to classify it can be done quickly
and accurately. So decision trees are another example of supervised learning.
The questions in a decision tree do not have to be binary, and the answers
can be numerical.
I would not recommend using Excel for decisions trees when you have a
real problem, with lots of data. It’s just too messy. That's because you don't
know the tree structure before you start, making it difficult to lay things out
in a spreadsheet.
First some Jargon and conventions. The tree is drawn upside down, the
root(the first question) at the top. Each question is a condition or an internal
node that splits features or attributes. From each node there will be branches
or edges, representing the possible answers. When you get to the end of the
path through the tree so that there are no more questions/decisions then
you are at a leaf. There are also the obvious concepts of parent and child
branches and nodes. See Figure 9.1.

Root node This is a branch or edge.

It corresponds to one ofthe categories/answers
Inner or internal node
Answer 1 a
Root-Feature 1

1 for tftefeature

Answer 1 b

Feature 2 Feature 3

Answer 3 a

Answer 2
a^ Answer J b ^Answe
r 2c

Feature 4
V”
Answer 4 a, Answei 4 b

This is a leaf, a prediction ^-

Figure 9.1: A schematic diagram showing the characteristics of a decision

tree.

You can use decision trees to classify data, such as whether or not a
mushroom is edible given various physical features. Those features could
have binary categories such as with/without gills, or multiple categories.
9.3. EXAMPLE: MAGAZINE SUBSCRIPTION 129

such as colour, or be numerical, height of mushroom, for example. Decision

trees also can be used for regression, when you have numerical data, how
much is a car worth based on make, model, age, etc.
Building or growing a tree is all about choosing the order in which features
of your data are looked at and what the conditions are. There might be some
pruning to be done and you'll need to know when to stop.

9.3 Example: Magazine subscription

I am going to work with member data from my own website now. I shall
be taking data for a small subset of members to figure out which people are
likely to subscribe to our magazine. (And along the way I'll be shamelessly
promoting both the website and the magazine.) I will not be violating any
data-protection laws.
There are three features of wilmott.com members we wil l look at: Em
ployment Status; Highest Degree Level; CQF Alumnus (whether or not they
have the Certificate in Quantitative Finance qualification). And the classifi
cation is whether or not they are magazine subscribers. The goal is to use
information about those features for new members to determine whether or
not they will be subscribers of the magazine. Wilmott

CQF Magazine
ID Employment Status Degree Level Alumnus Subscriber
1 Self Employed Postgraduate No No
2 Self Employed Postgraduate Yes Yes
3 Employed Postgraduate Yes Yes
4 Student/Postdoc. Postgraduate No Yes
5 Student/Postdoc. Undergraduate Yes Yes
6 Student/Postdoc. Undergraduate Yes No
7 Employed Undergraduate Yes Yes
8 Self Employed Postgraduate No No
9 Self Employed Undergraduate No Yes
10 Student/Postdoc. Undergraduate Yes No
11 Self Employed Undergraduate Yes Yes

12 Employed Postgraduate Yes Yes

13 Employed Undergraduate No Yes

14 Student/Postdoc. Postgraduate Yes No

15 Employed Postgraduate No Yes
16 Student/Postdoc. Postgraduate No No

17 Self Employed Postgraduate No No

Figure 9.2: The raw data.

I am only going to work with a small subset of the full magazine dataset,
so just 17 lines. See Figure 9.2. Out of these 17 members there are ten who
130 CHAPTER 9. DECISION TREES

are magazine subscribers.

The obvious use of any results that we find is in helping decide who to
target for subscriptions.
Just looking at this table we cannot immediately see any single question,
whether they are employed etc., that will determine whether or not someone
is a magazine subscriber. It's not as if having a postgraduate degree means
you do subscribe and not having one means you don’t. So we are going
to have to ask several questions, using more of the data in the table, to
pin down people's subscription status. But in what order should we ask the
questions? What is, in some sense to be decided, most efficient?
Now I must confess that I have tweaked the data a bit. That is so that
I can get a single decision tree that shows as many outcomes as possible.
You will shortly see what I mean. (Obviously the real data shows simply that
the better educated a person is then the more likely they are to subscribe.
Hahal)
I'm first going to explain how the final decision tree works, but how/ I
constructed it in an optimal fashion will come a bit later.
Suppose I start by asking members of wilmott.com ("Serving the Quan
titative Finance Community Since 2001") about their employment status and
whether or not they are magazine subscribers. They can answer Employed,
Self Employed or Student and Yes or No. According to the data in Figure
9.2 we would get results that could be represented by Figure 9.3. This is the
root of our decision tree.

Employment Status

Student/Postdoc. Self Em cloyed Employed

nm I I
PURE!

Figure 9.3: Responses to question about employment.

We read this as follows. In the boxes are two numbers. The number
on the left is the number of people who are magazine subscribers and the
number on the right those who aren't. Is this useful?
Yes and no. If they say they are Employed then this is very useful because
five employed people say they are magazine subscribers and none say they
aren’t. That is very useful information because as soon as we know someone
is employed we know that they will be subscribers. And no more questions
need be asked.
If only it were true that all employed people subscribed to the magazine.
Sadly, this data has been massaged as I mentioned. In practice you would
9.3. EXAMPLE: MAGAZINE SUBSCRIPTION 131

have a lot more data, you'd be unlikely to get such a clear-cut answer, but
you’d have a lot more confidence in the results. However even with these
numbers we can start to see some ideas developing. When you get an answer
that gives perfect predictability like this it is called pure. That is the best
possible outcome for a response. The worst possible outcome is if you get
the answer Self Employed because there is no information contained in that
answer, it is 50:50 whether or not a self-employed person is a subscriber.
In fact we seem to have gone backwards, at least before we had asked any
questions we knew that ten out of 17 people were subscribers, now we are
at a coin toss.

If someone is a Student/Postdoc, then two out of six are subscribers. In

terms of information, two out of six and four out of six are equally useful.
We are now going to look at the sub trees. We can ask the Stu
dent/Postdoc. whether they are a CQF Alumnus. Out of the six Stu
dents/Postdocs there are four who are alumni and two who aren't. See
Figure 9.4. The left branch of the CQF Alumnus, the one labelled Yes, has
four people at the end, the No branch has two. Out of the four CQF Alumni
there is only one magazine subscriber. You see those numbers in the box,
a 1 and a 3. And the two non alumni are equally split between subscribing
and not. So a 1 and a 1.

Employment Status

Student/Postdoc. Self Em Dloyed Employed

UE3 nun nun

PURE!
CQF Alumnus

No

nun
THE END

The remaining question

does not help...both have
postgrad, degrees

Figure 9.4: The first sub tree.

We can move further down the tree. We can ask those two non-alumni
Students/Postdocs what their highest degree is. Unfortunately that does not
help here because both have the same level, they have postgraduate degrees.
There's nothing we can do with their answers to the three questions that
will separate these two individuals.
132 CHAPTER 9. DECISION TREES

Employment Status

Student/Postdoc. Self Em )loyed Employed

I 2/4 I I I
PURE!

CQF Alumnus

No

THE END

Degree Level The remaining question

does not help...both have
postgrad, degrees
Undergrad. Postgrad.

THE END PURE!

No more questions left

Figure 9.5: Moving further down the tree.

Moving to the CQF Alumnus Students/Postdocs we now look at the

answers to the question about their highest degree. We see in Figure 9.5
that if they have a postgraduate degree then they will not be a subscriber.
The split is pure (it's actually just a single, non-subscribing sample). If their
highest degree is undergraduate then we can't be so sure, there's a one-in-
three chance of them being a subscriber. But this is the end of the line,
there are no more questions left. We have to accept the probabilistic result.
We can continue like this to fi l l in the rest of the tree, as shown in Figure
9.6. Luckily the rest of the tree results in pure splits.
If we are given a new data point, such as Self Employed with a Post
graduate Degree and who is a CQF Alumnus then we just run through our
tree and we will see that such a person will be, like all the best people, a
magazine subscriber. Although with so little data we would not be confident
in that conclusion.
9.3. EXAMPLE: MAGAZINE SUBSCRIPTION 133

Emptoyment Status

Student/Postdoc. SelfErmloyed Employed

\ni±j QZO ozo

PURE!

CQF Alumnus Degree Level

Yes No Postgrad.
Undergrad.
r~uri
THE END
I I
Degree level The remaining question PURE!

does not help...both have CQF Alumnus

postgrad, degrees
Undergrad. Postgrad. Yes No

mr-\ r~^ I I
THE END PURE! PURE! PURE!

No more questions left

Figure 9.6: The final tree.

Some observations...

• Ideally you will end with a leaf that is a pure set, which classifies
perfectly

• However if you have an impure set you might find that the remaining
questions do not help to classify

• Or you might run out of questions

• You should always keep track of the numbers (in the boxes) because
that will give you a probabilistic classification and a degree of confi
dence

• Notice how the same questions can appear at different places in the
tree, although there will be no path along which you get asked the
same question twice

• Questions don't have to be binary, yes/no

• And there don't have to be just two classifications (here whether or
not they are magazine subscribers)
That's what decision trees look like and how they work. But you should
be asking how did I construct the tree? Or more specifically how did I know
in which order to ask the questions? Why did I start with Employment Status
as the root, and not another attribute? Which attributes best split the data?
There was a clue in my occasional use of the word information above. We
are going to take inspiration from Information Theory.
134 CHAPTER 9. DECISION TREES

9.4 Entropy

In layman's terms you want to find the attribute that gives you the highest
information gain. I reckon that out of Employment Status, Highest Degree
Level and whether or not a person is a CQF Alumnus the attribute that does
the splitting the best, that gives the highest information gain, is Employment
Status. But what makes me think that?
We need some way of looking at which attribute is best for splitting the
data based on the numbers in each class. Referring to Figure 9.3 we need
some numerical measure that gives us how efficient the split is based on
the numbers 2 / 4, 3 / 3 and 5/0. But first we need to see what the
Employment Status attribute is competing with. In Figure 9.7 we see the
splits you get from having a root that is either CQF Alumnus or Highest
Degree Level.

CQF Alumnus Desree Level

Undergrad. Postgrad.

I I rrfT-]

Figure 9.7: Other possible roots.

It looks like both of these splits do a very poor job. Both of them have
one branch that is a coin toss. And in both cases the other branch is not
much better. But we need to quantify this.
The uncertainty in classification is going to be measured by the entropy.
We saw this, and the Justification for it, in Chapter 2. All we need to do to
determine the best attribute to start with, the root of our tree, is to measure
the entropy for each attribute and choose the one with the lowest value.
Usually this is done by measuring the information gain.
To measure the gain we need a starting point to measure gain relative
to. And that would be the raw data before it is split. We calculate the
entropy for the original data, that is ten magazine subscribers and seven non
subscribers:
9.4. ENTROPY 135

-^plog2(p)=
10 7 \
10 + 7
log2
(i^)- ( 10 + 7y
= 0.977.

It could be worse, i.e. closer to 1, but not much.

Now we go to the data in Figures 9.3 and 9.7 and measure the entropy
for each branch of each attribute.
For the Student/Postdoc, branch of Employment Status we have
^ 2 2 4 / 4
log2 log2 = 0.918.
2+4 2+ 4 2+4 V2 + 4

For the Self Employed branch of Employment Status we have a rather obvious
1, since there is an equal number of subscribers and non. And for the
Employed branch the entropy is zero since the split is pure.
Now we measure the average entropy for Employment Status as
6 6 5
X 0.918 + X 1+ X 0 = 0.677.
17 17 17

That's because six of the 17 are Student/Postdoc. , another six of the 17 are
Self Employed and five are Employed.
Thus the information gain thanks to this split is

Information gain for Employment Status = 0.977 — 0.677 = 0.300.

We do exactly the same for the Highest Degree Level and CQF Alumnus
attributes:

Information gain for Highest Degree Level = 0.034,

Information gain for CQE Alumnus = 0.021.

And thus Employment Status having an information gain of 0.300 is the easy
victor, beating Highest Degree Level and CQE Alumnus. And so it becomes
the root of the decision tree.
Having established the root attribute we now repeat this process at each
branch. For example we go to the left-most branch, the one labelled Stu
dent/Postdoc. in Figure 9.3 and consider which of the two remaining at
tributes will give the larger information gain for the seven cases on that
branch. And we find it is CQF Alumnus. And so on.
136 CHAPTER 9. DECISION TREES

The Decision Tree Algorithm

Step 1: Pick an attribute

Take one of the (remaining) attributes and branches and split the
data.

Step 2: Calculate the entropy

For each branch calculate the entropy. Then calculate the average
entropy over both/all branches for that attribute.
Return to Step 1 until all attributes have been examined.

Step 3: Set attribute to minimize entropy

Choose as the node the attribute that minimizes the entropy (or
equivalently maximizes the information gain relative to the unsplit
data).
Move down/across the tree and return to Step 1.

This algorithm is known as IDS, ID standing for Iterative Dichotomiser.

There are other algorithms one can try. For example C4.5 is an algorithm
created by the inventor of IDS. One of the problems addressed by C4.5
concerns how the algorithm deals with how many branches come from each
attribute. In my example I have two attributes that split into two branches,
this is a binary classification, and one attribute that has three branches, there
are three answers to the question. Flow does this affect entropy? It’s easy
to see that the more branches leaving a node the more likely it is to be
favoured, the entropy will be smaller. Just take the above example and give
every line of data the name of the wilmott.com member, Andrea, Bruce,
Charles, David,. . . If we use the name in the classification then each data
point will get its own leaf and the entropy will be a perfect zero. But it
almost certainly won’t be much use when a Zachary comes along.
A simple modification to allow for the number of branches is to divide
the information gain by another entropy measure, the split entropy, defined
by

Z. N \ NJ
Where the sum is over al l the relevant branches, N is the number of samples
in the parent node and Ni is the number of samples at the end of each
branch. So for the data in Figure 9.3 we would divide the information gain,
9.5. OVERFITTING AND STOPPING RULES 137

0.300, by
6 (6 6 5\
log2 - TT log2 = 1.58.
17 Vl7 17 VI7; VI)
This gain ratio allows for different numbers of partitions.
There are also different measures of uncertainty. A common one, besides
entropy, is Gini impurity. Gini impurity measures how often something would
be incorrectly labelled if that labelling was random. If we have probabilities
Pi for being in set i then the probability of mislabelling is 1 — Pi. The Gini
impurity is the average of this over al l elements in the set;

-Pi) = 1 -

9.5 Overfitting And Stopping Rules

If you have many attributes it is possible that by the time you get down
to the leaf level there are very few records. The danger here is that your
results are not representative of other, non-training, data. In other words,
you have overfitted and all that your decision tree is doing is memorizing
the data it has been given. To reduce the chance of this happening it is
common to introduce stopping rules that say that when you get down to a
certain number of records (either an absolute number or as a fraction of the
whole dataset) you stop splitting the data and create a leaf. This way your
results can remain statistically significant. This just means that you end up
with a reliable probabilistic classification rather than an unreliable, possibly
deterministic, one.
This is also the classic problem of fitting training data well, but doing
badly on test data.
A stopping rule can also help when you have a new data point that you
want to classify but there was no data point in the training set with the same
features.

9.6 Pruning

Another, similar, method for avoiding overfitting is to prune your tree.

You might find that part of the tree is unreliable, perhaps because you have
tried classifying data from a validation set that you had kept back. In that
case you Just cut off the most unreliable nodes, replacing them with a leaf.
The leaf would then either be classified according to whichever classification
has the most samples in that leaf, or left as a probabilistic classification.
138 CHAPTER 9. DECISION TREES

9.7 Numerical Features

What can we do if the answers to our questions are not categories but
numerical? Suppose that we have data for people’s heights, and whether or
not they are magazine subscribers, such as that shown in Figure 9.8. The
height data here is entirely made up but to make things interesting I have
naturally assumed that magazine subscribers tend to be taller, and more
attractive.

Wilmott

CQF Magazine
ID Employment Status Degree Level Alumnus Heights Subscriber
1 Self Employed Postgraduate No 174.6 No

2 Self Employed Postgraduate Yes 173.0 Yes

3 Employed Postgraduate Yes 185.2 Yes

4 Student/Postdoc. Postgraduate No 169.5 Yes

5 Student/Postdoc. Undergraduate Yes 179.9 Yes

6 Student/Postdoc. Undergraduate Yes 167.9 No

7 Employed Undergraduate Yes 177.7 Yes

8 Self Employed Postgraduate No 175.7 No

9 Self Employed Undergraduate No 176.5 Yes

10 Student/Postdoc. Undergraduate Yes 162.5 No

11 Self Employed Undergraduate Yes 178.4 Yes

12 Employed Postgraduate Yes 178.2 Yes

13 Employed Undergraduate No 173.0 Yes

14 Student/Postdoc. Postgraduate Yes 174.7 No

15 Employed Postgraduate No 188.0 Yes

16 Student/Postdoc. Postgraduate No 163.2 No

17 Self Employed Postgraduate No 159.5 No

Figure 9.8: Same data as before but with added (fictitious) height informa
tion.

One simple way of tackling this classification problem within a decision

tree is to choose a threshold s for the height so that being above or below
that threshold determines the branch. The level of s can be chosen to
maximize the entropy gain. With this data, and if we used the height as the
root attribute, then the entropy gain is maximized by a threshold of 176cm.
In principle you could have something more complicated than a simple
threshold for determining the branch.
Now you can mix categories and numerical data and all of the above
explanation for building your decision tree carries over.
9.8. REGRESSION 139

9.8 Regression

In the above I have focused on using decision trees for classification

problems. However the method can also be used for regression.
When used for regression our goal is to come up with a numerical value
or prediction based on features that can be either categories or numbers.
How much is the car worth give its model (category), manual or automatic
(category), age (numerical), mileage (numerical), etc.
Having both categories and numerical data for our features will not cause
a problem, we just use the threshold method described above (or something
similar but more complicated) to turn numerical feature data (mileage) into
categories (above/below 40,000 miles, say).
However we do need to use a measure other than entropy to tell us how
informative our different types of data are. This is because our dependent
variables are no longer classes (yes/no to magazine subscription) but are
numerical (£6,250).

ID Age Seat Transmission Mileage Price

1 6 2 Manual 19,042 £ 3,150
2 5.5 3 Automatic 12,851 £ 6,400
3 6 5 Manual 36,282 £ 5,300
4 6 5 Manual 16,621 £ 5,250
5 6 5 Manual 35,685 £ 5,700
6 5 3 Manual 27,314 £ 5,050
7 6 5 Manual 42,143 £ 5,000
8 6 3 Manual 42,780 £ 3,650
9 5 3 Manual 37,413 £ 4,950
10 6 5 Automatic 74,848 £ 4,850
11 4.5 5 Manual 48,572 £ 6,400
12 5 3 Manual 28,602 £ 4,900
13 5 5 Manual 10,663 £ 5,950
14 5 5 Manual 56,775 £ 5,250
15 7 5 Manual 63,000 £ 2,600
16 4 5 Automatic 34,303 £ 7,150
17 5 5 Automatic 25,584 £ 5,900
18 6 5 Manual 9,499 £ 6,250

Figure 9.9: Auction prices for Peugeot Partner Tepee.

Let’s look at the data shown in Figure 9.9. Here we have recent auction
prices for several Peugeot Partner Tepees (no, me neither, but 1 think they
are cars or vans) together with various features, some are categories such
as transmission type and some are numerical, mileage for example. I have
tweaked this data only a tiny amount and simply to illustrate several things
that can happen in building the tree.
140 CHAPTER 9. DECISION TREES

th
The dependent variable will be representing the value of the n
car, say. For each attribute we are going to look at the sum of squared error
across the branches. But, crucially, each branch wil l have a different model
for the forecast.
To see what I mean let's start with perhaps the simplest attribute,
whether the car is manual or automatic transmission.

Transmission

Manual Automatic

£ 3,150 £ 6,400
£ 5,300 £ 4,850
£ 5,250 £ 7,150
£ 5,700 £ 5,900
£ 5,050
£ 5,000
£ 3,650
£ 4,950
£ 6,400
£ 4,900
£ 5,950
£ 5,250
£ 2,600
£ 6,250

Figure 9.10; The transmission attribute and the data.

In Figure 9.10 we see how the car prices depend on the type of trans
mission. If the car is manual then the average price is £4,957, and £6,075
if automatic. If we had no other information (such as mileage, age, etc.)
so that this was the end of the branches then those would be our forecasts
for any other Peugeot Partner Tepee we might come across. I.e. the model
we use is the average at each leaf, but it’s a different average for each leaf.
We’l l see something a bit more sophisticated than the average shortly.
But how good is that attribute at predicting compared with say predicting
price based on number of seats or mileage? We need to know this if we are
build our tree most efficiently. To figure that out we need something to
replace entropy. And the easiest quantity to use is the sum of squared
errors:

E (s“
Manual
anual V
E(
Auto
ijAuto - y
)
9.8. REGRESSION 141

This is to be interpreted as the sum of the squared differences between

each individual price and the average for that category over all the manual-
transmission cars plus the same for automatic transmission. This is a simple
standard-deviation type of measure. In other words

(3150 - 4957)^ +(5300 - 4957)''^ -f • • • +(6250 - 4957)^

-b (6400 - 6075)^ -h • • • -b (5900 - 6075)^ = 18,916,785.
But how good is that, i.e. how low, compared to a similar calculation
for, say, the mileage of the car? Since mileage is a numerical quantity we
introduce a threshold.
Let’s go over to just symbols. We want to calculate
/ \
2
min
s
mm
V
X
(r,)
<s
+ min
V
X
E (»-b“') ^ (9,1)

And now this is interpreted as follows. The threshold for the mileage, say,
is s. And we are looking at prices for cars under and over the mileage of
th
s. Xm^ means the numerical value of the rn*'’ attribute (mileage) of the n
data point (car). The minimization over y Just means each branch has its
own model i.e. value. Rather obviously the minimization over y will result in
the value for each y being the average of values on that branch. And then
finally we want to find the threshold that gives us the best split of the data.
Let’s look at the example.

Mileage

Under 40,000 Over 40,000

£ 6,250 £ 5,000
£ 5,950 £ 3,650
£ 6,400 £ 6,400
£ 5,250 £ 5,250
£ 3,150 £ 2,600
£ 5,900 £ 4,850
£ 5,050
£ 4,900
£ 7,150
£ 5,700
£ 5,300
£ 4,950

Figure 9.11: Mileage attribute, threshold 40,000, and the data.

142 CHAPTER 9. DECISION TREES

In Figure 9,11 we see the data divided according to the mileage attribute
with the threshold of 40,000. The mean value (the y) for under 40,000
miles is £5,496 and £4.625 for above. The sum of squared errors, as in
expression (9.1), is 19,771,041. This is worse than using the transmission
type for regression. But that’s because I just picked 40,000 at random. We
can minimize this error by choosing a threshold of 60,000 when the squared
error becomes 17,872,343. This is now better than using transmission but
not by much.
Anyway, we can continue this process by looking for the best attribute
to be the root of our decision tree. And then work our way down the tree
looking at the remaining attributes exactly as in the classification problem.
To summarize: Choose an attribute, m, and for that attribute find a
threshold s to minimize (9.1). The m gives us which feature to choose and
the s tells us where the split is.
I don't know whether this will help or confuse but there are three mini
mizations of the standard deviation happening here:

1. There is the minimization within each node, finding the best model.
Here this is trivially the mean of the data in each node

2. Then there is the minimization by finding the best split

3. Then we choose which feature gives us the lowest error and this be
comes the one we use to move further down the tree

If we have a classification problem then we don’t need the first of these.

If our features are non numeric then we don’t need to do the second of
these.

Linear regression

We have possibly thrown away quite a lot of information by just using

a prediction that is the mean on that branch, or in the leaf. If a numerical
attribute divides into leaves then we could relate the forecast to the numerical
values of that attribute.
Let’s look at car price versus age now. Our data is just that in Figure
9.12 that hasn’t been greyed out.
9.8. REGRESSION 143

ID Age Seat Trarismiss^on IVIHeage Price

6 2 Manual 19,m2 £ 3,150
5.5 ■i Automatic 12,851 £ 6,400
3 6 5 Manual 36,283 £ 5,300
4 6 5 Manual 1&.&2M £ 5,250
5 6 S Manual 35,685 £ 5,700
6 5 3 Manual :’7,314 £ 5,050
7 6 5 Manual 42,143 £ 5,000
8 6 3 Manual 42,780 £ 3,650
9 5 3 Manm-3 37,4.13 £ 4,950
10 6 a- Automatk: 74,84S £ 4,850
11 4.5 5 Manual 48,572 £ 6,400
12 5 3 Manual 28,602 £ 4,900
J 7 5 S Manual .10,663 £ 5,950
14 5 S Manual 56,775 £ 5,250
In 7 Manual 63,000 £ 2,600
16 4 a Aiiliomatic 34,303 £ 7,150
5 Automatic 2.5,584 £ 5,900
18 6 5 Manual 9,499 £ 6,250

Figure 9.12: We will do a regression on this data.

A simple splitting of the data into two branches using a threshold of 6.2
years gives the lowest sum of squared errors at 15,616,176. (This happens
to be our best error yet, so really ought to be the root.) But we are lumping
together a lot of data points and throwing out any nuance. You’ l l see that
nuance now when I plot value against age.
What if instead of splitting the data at a threshold we fit a straight line
to the data? We’d get the results in Figure 9.13 and an error of 12,457,500.

8000

7000

6000

5000
(U

4000
Q.

3000

2000

1000

0
0 2 4 6 8

Age

Figure 9.13: A straight line fitted to the data.

144 CHAPTER 9. DECISION TREES

And finally, in the spirit of both splitting and linear regression, split at
5.2 years and fit two straight lines, a different one above and below the
split, gives the results shown in Figure 9.14, with an error of just 8,609,273.
Of course this is no more than a more complicated fitting function than a
straight line.

8000

7000

6000

5000
cu

4000
Q.

3000

2000

1000

0
0 2 4 6 8

Age

Figure 9.14: Splitting plus a linear fit.

You should see what I mean now, by fitting with a straight line or lines
I've kept more information from the data and got a better forecast.
You can get more complicated by making decisions based on (linear per
haps) combinations of features. (Age minus some parameter times the num
ber of seats.)

9.9 Looking Ahead

What I have described is a greedy algorithm, one where at each stage we

choose which feature to split on based on a measure (entropy or standard
deviation) at that point in the tree. There is no consideration for what will
follow further down the tree. But this could be a problem, it might lead to
sub-optimal splitting.
We sort of saw this in the car example. Right at the end I found that a
linear fit to the age data gave the lowest standard deviation. But what if I'd
split on, say, mileage higher up the tree? Or Just split on age but using a
threshold and a constant value (the mean) in the node? And then I'd gone
on to split on transmission etc. further down. If I'd done that then I would
9.10. BAGGING AND RANDOM FORESTS 145

have missed out the rather good linear fit to age.

One can prevent this by looking ahead to see what happens to splits
further down the tree i.e. the sub-trees that are produced.

9.10 Bagging And Random Forests

Decision trees can have a tendency to overfit. Think of the rituals that a
footballer might go through before a match based on experiences from the
past when he won or lost. Lucky socks? Check. Leave dressing room left
foot first? Check. No curry the night before? Check. To reduce overfitting
and variance in predictions one can use various methods for aggregating
results over many trees.
Bootstrap aggregation or bagging for short is a method in which one
samples, with replacement, from a full data set and then applies whatever
machine-learning technique you are using to get a forecast, and then repeats
with another random sample. One does this many times and the final forecast
will be either the majority vote in a classification problem or an average for
a regression problem.
Random forests is the same idea with one twist. In deciding which node
to use for a split you don't look at the best feature for splitting over a//
possible features. Instead you consider a random subset of features and
choose the best feature from that subset.

Further Reading

The self-published Decision Trees and Random Forests: A Visual Intro

duction For Beginners by Chris Smith is cheap and cheerful. Very much for
beginners. There are other machine-learning books in the same series.
Another self-published book Tree-based Machine Learning Algorithms:
Decision Trees, Random Forests, and Boosting by Clinton Sheppard is light
on the mathematics but does have plenty of Python code.
■ •i
Chapter 10

Neural Networks

10.1 Executive Summary

A neural network is a type of machine learning that is meant to mimic

what happens in the brain. Signals are received by neurons where they are
mathematically manipulated and then passed on to more neurons. The input
signal might pass through several layers of neurons before being output, as
a forecast regression or classification. It can be used for either supervised or
unsupervised learning.

10.2 What Is It Used For?

Neural networks are used for

• Modelling complex relationships between inputs and outputs

• Example: Recognising images including handwriting

• Example: Enhancing grainy images
• Example: Translating from one language to another
• Example: Advertising. If advertisers can find a way to use neural
networks they will!

10.3 A Very Simple Network

You will have seen pictures like Figure 10.1, representing a typical neu
ral network. This is a good illustration of the structure of a very simple

147
148 CHAPTER 10. NEURAL NETWORKS

feedforward network. Inputs go in the left of this picture and outputs come
out of the right. In between, at each of the nodes, there will be various
mathematical manipulations of the data. It’s called feedforward because the
data and calculations go in one direction, left to right. There is no feedback
or recurrence like we'll be seeing later.

A.
o. ,\
\

Figure 10.1: A typical neural network. Here there is an input layer, an output
layer and one hidden layer in between.

In this particular example we have an input being an array or vector with

three entries, so three numerical quantities. These are manipulated in the
hidden layer of eight nodes before being passed out to an output vector/array
with two entries. This would be an example of trying to predict two quantities
from a three-dimensional input. The outputs will be numerical also but these
can stil l be used for classification.
I’ll explain the mathematical manipulations shortly. But let me now just
say that there doesn’t have to be a single hidden layer. You can have as
many as you want or need, and each with as many nodes as you want or
need. However, the number of input and output nodes will depend on what
data you have and what you want to forecast. The layout of the network is
called the architecture.
10.4. UNIVERSAL APPROXIMATION THEOREM 149

10.4 Universal Approximation Theorem

The Universal Approximation Theorem says that, under quite weak con
ditions, as long as you have enough nodes then you can approximate any
continuous function with a single-layer neural network to any degree of ac
curacy. The neural-network architecture for this simple task is shown in
Figure 10.2. There is a single input, think x, and a single output, think y.
To get a better approximation you will just need more nodes in the hidden
layer.

o.^
o
o

Figure 10.2: A neural network to represent the Universal Approximation

Theorem.

This is one of the most important uses for a neural network, function
approximation. But usually our problems are not as straightforward as this
implies. Instead of having a single independent variable, the x, we will
typically have many, a whole array. And instead of a single output, the y,
there could be many outputs. And most importantly instead of having a
known function that we want to approximate we have a whole load of data,
the inputs and the outputs, and we want our network to figure out what
happens at the nodes to give the best approximation, one that can be used
on new, unseen, data. With such complicated problems we'll need the richer
150 CHAPTER 10. NEURAL NETWORKS

structure that you can get by having more than one hidden layer.
Ok, now it's time to tell you what goes on in the hidden layer, what are
the mathematical manipulations I've mentioned.

10.5 An Even Simpler Network

Figure 10.3 shows just about the simplest network, and one that can still
be used to describe the manipulations going on.

b
Xi ai
VVi

W-2,
^2 ^2
tti = Xi, 02= X2
Z = M/iOi + W202 + b
y = 9^7)
Figure 10.3: The manipulations in the classical neural network.

Flere we have two inputs on the left, x\ and X2, and a forecast/output
on the right, y.
The inputs are passed unchanged into the first nodes:

(ii = X\ and «2 = ^2-

Then each of these node values is multiplied by a weight, the ws. And
then a bias, b, is added:

z = wia\ + W2a2 T h.

That is, just a linear combination.

This result is then acted on by a function g{z) to give the output,

y = y{z)-

This is a very, very simple transformation of the data, here used to give
some function of a two-dimensional input.
10.6. THE MATHEMATICAL MANIPULATIONS IN DETAIL 151

The function g{z) will be chosen, it is called an activation or trans

fer function. In neural networks we specify the activation function but the
weights, the ws, and the bias, b, will be found numerically in order to best
fit our forecast output with our given data.

10.6 The Mathematical Manipulations In Detail

Now let’s do that with a larger network, showing al l the sub- and super
scripts we’ll be needing.
In Figure 10.4 is a neural network with a single hidden layer. For more
layers the following is no different. Just keep an eye out for the superscript
denoting which layer we are looking at.

(n)
X
1
(I
a..':
(71)
^2 V-
X
yi
(71) (1)
Xm

Figure 10.4: The quantities being input, output and in the nodes.

Notation

I show three types of quantities in the figure. The inputs are the xs. The
outputs are the jjs. The values in each of the nodes are the as. Let’s look
at the sub- and superscripts, etc., and what they mean.
As before the superscript (n) refers to the data point. The subscript
m on X is the position in the vector of features, of which there will be M.
So our inputs will be vectors
152 CHAPTER 10. NEURAL NETWORKS

These quantities are then input into the first node. The input layer of
nodes contain values am\ as shown by the direction of the arrows. We put
the input values, the xs, unchanged into the first layer:
(1)
a —

The nodes in the next layers have the values a^ . Here I represents
the number of the layer, out of L layers, and the j represents the specific
node. Note that the number of nodes in each hidden layer can be anything.
However the input layer has the same number of nodes as features, and the
output layer the number of nodes needed for our forecast.
We continue like this until the output. The (n) means the output associ
ated with the riT data point. That will be another vector with usually
a different dimension from that of the input vector. And the hat in this just
means that this is the forecast value, as opposed to the actual value for the
dependent quantity from the training data.
As I showed above, two things happen to the numerical quantities as we
go through the network.

Propagation

In going from one layer to the next the first thing that happens is a linear
combination of the values in the nodes. Figure 10.5 is similar to Figure 10.3
except with more nodes and sub- and superscripts everywhere.

l3yer/-l

Figure 10.5: A detail of the network and a formula.

10.6. THE MATHEMATICAL MANIPULATIONS IN DETAIL 153

I have labelled the two layers as / — 1 and 1. Also notice that the general
node in the left-hand layer is labelled j and one in the right-hand layer, layer
/, is labelled j'.
We want to calculate what value goes into the node of the layer.
First multiply the value in the node of the previous, (/ — 1)*'’,
layer by the parameter wjjj, and then add another parameter . Then we
add up all of these for every node in layer I — 1.
This is just
Jl-l
(0
E ^3,3'^3 + (10.1)
1=1

where Ji means the number of nodes in layer 1. I'l l call this expression .
This is a bit hard to read, such tiny fonts, and anyway it’s much easier
to write and understand as

(10.2)

with the matrix containing all the multiplicative parameters, i.e. the
weights Wjj/, and is the bias. The bias is Just the constant in the linear
transformation. (Sometimes the bias is represented by an extra node at the
top of the layer, containing value 1, with lines coming down to the next
layer. This way of drawing the structure is exactly equivalent to what we
have here.)
This Just means that the first manipulation, to get to the z, is to take a
linear combination of values in the previous layer. But remember, we don't
specify what the weights and biases are, they are to be found during the
training of the network.
If that were it then it wouldn’t be interesting. But we still have to perform
the other simple transformation.

The activation function

The activation function gets its name from a similar process in physical,
i.e. inside the brain, neurons whereby an electrical signal once it reaches a
certain level will fire the neuron so that the signal is passed on. If the signal
is too small then the neuron does not fire.
Applying the same idea here we simply apply a function to expressions
(10.1) or (10.2). Let's call that function It will be the same for all
nodes in a layer but can differ from layer to layer. And we do specify this
function.
Thus we end up with the following expression for the values in the next
154 CHAPTER 10. NEURAL NETWORKS

layer

(i-i) +
a(') =/)(zW)=g(')(W(') a
b(')) (10.3)

The function of a vector just means taking the function of each entry.
So a signal is passed from all the nodes in one layer to the next layer
where it goes through a function that determines how much of the signal to
pass onwards.
And so on, all the way through the network, up to and including the
output layer which also has an associated activation function.
This can be interpreted as a regression on top of a regression on top of
a . ..

You can see that what comes out of the right is just a messy function of
what goes in the left. I use messy in the technical mathematical sense that
if you were to write the scalar ys explicitly in terms of the scalar xs then it
wouldn't look very pretty. It would be a very long expression, with lots of
sub- and superscripts, and summations. But a function it most definitely is.

Classification problems

I’ve mentioned this before, but it's worth repeating, classification is dif
ferent from regression and function fitting.
Suppose you want to classify fruit. You have peaches, pears, plums,
etc. Your raw data for the xs might be numerical quantities representing
dimensions, shape, colour, etc. But what will the dependent variable(s) be?
You could have a single dependent y which takes values 1 (for peach), 2 (for
pear), etc. But that wouldn't make any sense when you come to predicting a
new out-of-sample fruit. Suppose your output prediction was y = 1.5. What
would that mean? Half way between a peach and a pear perhaps? That’s
fine if there is some logical ordering of the fruit so that in some sense an
peach is less than a pear, which is less than a plum, and so on. But that's
not the case.
It makes more sense to output a vector y with as many entries as there
are fruit. The input data would have a peach as being (1,0,...,0)^, a pear
as (0,1,...,0)"^’ and so on. An output of (0.4,0.2,... , 0.1)^ would then be
useful, most likely a peach but with some pear-like features.

10.7 Common Activation Functions

Here are some of the most common activation functions.

10.7. COMMON ACTIVATION FUNCTIONS 155

Linear function

Not interesting would be

g{x) = X.

There'll be problems with this activation function when it comes to finding

the parameters because its gradient is one everywhere and this can cause
problems with gradient descent. And it also rather misses the essential non
linear transformation nature of neural networks.

Step function/Hard limit

The step function behaves like the biological activation function described
above.
, - f 0 a; < 0
= 1 1 ,r>o •
The signal either gets through as a fixed quantity, or it dies. This might
be a little bit too extreme, leaving no room for a probabilistic interpretation
of the signal for example. It also suffers from numerical issues to do with
having zero gradient everywhere except at a point, where it is infinite. This
messes up gradient descent again. Probably best to avoid.

Positive linear/ReLU function

g{x) = max(0,x).
ReLU stands for Rectified Linear Units. It’s one of the most commonly used
activation functions, being sufFiciently non linear to be interesting when there
are many interacting nodes. The signal either passes through untouched or
dies completely. (Everyone outside machine learning calls it a ramp function.)

Saturating linear function

0 X <0
9{x) X 0<X <1 .
1 X > 1

Similar to the step function but not so extreme.

156 CHAPTER 10. NEURAL NETWORKS

Sigmoid or logistic function

1
g{x) =
1 + e~^

This is a gentler version of the step function. And it’s a function we have
found useful previously. It could be a good choice for an activation function
if you have a classification problem.
The tanh function can also be used, but this is just a linear transformation
of the logistic function.

Softmax function

We saw the softmax function in Chapter 2. The softmax function takes an

array of K values (xi,Z2,...,Zk) and maps them onto K numbers between
zero and one, and summing to one. It is thus a function that turns several
numbers into quantities that can be perhaps interpreted as probabilities.

K
Efc=l

It is often used in the final, output, layer of a neural network, especially with
classification problems.

Hidden layer versus output layer

One can be quite flexible in choosing activation functions for hidden layers
but more often than not the activation function in the final, output, layer
will be pretty much determined by your problem.

10.8 The Goal

Our goal is to ultimately fit a function. But I haven't yet said much
about the function we are fitting. Typically it won’t be the sine function
we’l l be seeing in a moment as our first example. It will come from data.
For each input independent variable/data point of features there will
be a corresponding dependent vector This is our training data.
Our neural network on the other hand takes the as input, manipu
lates it a bit, and throws out Our goal is to make the y*^") and y^”^ as
10.9. EXAMPLE: APPROXIMATING A FUNCTION 157

close to each other as possible. And we do that by choosing the parameters

and for each layer.
This is just a fancy regression. And so you would rightly expect some
discussion of cost functions and numerical methods. Soon.

10.9 Example: Approximating a function

Let’s look at the Universal Approximation Theorem in action. Take a

neural network with one hidden layer and a variety of numbers of nodes in
that layer and use it to fit sin(2x). In Figure 10.6 we see how well we can
fit this function using two, four, six and eight nodes.

Approximating sin(2x)

-8

sin(2x) — 2 nodes 4 nodes 6 nodes 8 nodes

Figure 10.6: Approximating a sine wave with a one-hidden-layer network.

I used a sigmoidal activation function in the hidden layer. Obviously the

more nodes we have the better. With eight nodes the fit is excellent.
How did I derive these results? Well, I confess that since the network
layout, the architecture, here is quite simple I used Excel and Solver, to find
the weights and the biases. And then I checked it using the NeuroLab Python
library, see https://pythonliosted.org/neurolab/.
But that is cheating, according to the premise of this book. So what is
going on inside the code that finds the parameters? First we need to decide
on a cost function.
158 CHAPTER 10. NEURAL NETWORKS

10.10 Cost Function

As in simpler forms of regression we need to have a measure of how

good a job our algorithm is doing at fitting the data. Thus we need a cost
function.
A common one, and the one I used in the sine fitting above, is ordinary
least squares. The cost function is then
N K
(n) (n)\^
n—1k=l
) (10.4)

The notation is obvious, I hope, is the dependent data for the data
point, the k representing the H'' node in the output vector, and is similar
but for the forecast output.
(In the above sine wave example we only had one output so K = 1.)

Classification

In classification problems we stil l associate our classes with numbers or

vectors. If we are labelling emails as spam or not we might label non-spam
emails as 0 and spam emails as 1. That would be a binary classification, and
would require a single output.
If we have types of animal such as mammal, reptile, amphibian, etc. then
we use the vector approach, using a vector with dimension the same as the
number of classes. Just as described above for fruit, (1,0,...,0)^ , etc.
The cost function commonly used for such an output is similar to the
one we used in Chapter 6 for logistic regression. And this is, for a binary,
yes/no, classification
N

+(1 - y(n) )(i-in (y(")))).

71=1

But if we have three or more classes then we have to sum over all of the
K outputs, K being the number of classes:

in)
J =-
n=l fc=l
-Vk )(l-ln(r)))^ (10.5)

This is related to cross entropy again.

To these cost functions could be added a regularization term, as discussed
previously, of the form
A
10.11. BACKPROPAGATION 159

So we now have something to minimize, but how do we do that numeri

cally?

10.11 Backpropagation

We want to minimize the cost function, J, with respect to the parameters,

the components of W and b. To do that using gradient descent we are going
to need the sensitivities of J to each of those parameters. That is we want

dJ dJ
and
^3' dbf}-
If we can find those sensitivities then we can use a gradient descent method
for finding the minimum. But this is going to be much harder here than in
any other machine-learning technique we have encountered so far. This is
because of the way that those parameters are embedded within a function of a
function of a. .. To find the sensitivities requires differentiating that function
of a function of a...And that’s going to be messy, involving repeated use
of the chain rule for differentiation, unless we can find an elegant way of
presenting this. Fortunately we can.
And this is made relatively painless by introducing the quantity

dJ

dz^} ■
Remember what z is, it’s the linear transformation of the values in the
previous layer, but before going into the activation function.
This idea is called backpropagation. Backpropagation is rather like cal
culating the error between the y and the y in the output layer and assigning
that error to the hidden layers. So the error in effect propagates back through
the network. This is slightly strange because although there is an obvious
meaning for the error in the output layer (it’s just the difference between
the actual value of y and the forecast value y) there is no correct a in the
hidden layer with which to make a comparison. But that doesn’t matter.
Backpropagation is going to tel l us all we need in order to find our parame
ters.

I’m sorry but we’re going to have another one of those mini networks.
See Figure 10.7. This shows pretty much all we need in the following. Our
first goal is to find the sensitivity of the cost function to
160 CHAPTER 10. NEURAL NETWORKS

Layer/-I Layer /

/
O /

o Node /’

O
Node j

O
Figure 10.7: The network showing the key formulas.

Let’s play around with the chain rule:

(0
0-1) dJ dJ dz):
E
But

(0 (1-1)
j'
3 3

And so
dJ (i)
dz E PA(0
dg('-i)
dz
(10.6)
J f

Let's see what we have achieved so far, and what we haven't. Equation
(10.6) shows us how to find the 5s in a layer if we know the (fs in all layers
to the right. Fantastic.
But what is so great about these 5s7 That's actually the easy bit:

dJ dJ dzf
3^3' ~ dz^) dwf.,

(10.7)
10.11. BACKPROPAGATION 161

And that's done it! The sensitivity of the cost function, ./, to the ws
can be written in terms of the 5s which in turn are backpropagated from the
network layers that are just to the right, one nearer the output.
And sensitivity of the cost function to the bias, bl That’s even easier.

dJ
5f.

In the above we have the derivative of the activation function with respect
to z. This will be a simple function of z depending on which activation
function we use. If it is ReLU then the derivative is either zero or one. If we
use the logistic function then we find that g'{z) = g{\ - g), rather nicely.
The last hidden layer is different because it feeds into the output. If the
cost function is quadratic, for example, then we have instead

(L)
dz
iVj -Vi)-

(To avoid confusion, if there is a single output then you can drop the j
subscripts.)

The Backpropagation Algorithm

Step 0: Initialize weights and biases

Choose the weights and biases, typically at random. The size of the
weights should decrease as the number of nodes increases.

Step 1: Pick one of the data points at random

Input the vector x into the left side of the network, and calculate all
the zs, as, etc. And finally calculate the output y. (This might also
be a vector.)

Step 2: Calculate contribution to the cost function

You don’t actually need to know the actual value of the cost function
to find the weights and biases but you will want to calculate it so you
162 CHAPTER 10. NEURAL NETWORKS

can monitor convergence.

Step 3: Starting at the right, calculate all the

For example, if using the quadratic cost function in one dimension,
then

dz
(y-y)-

Moving to the left

sf-'> dz
i'

Step 4: Update the weights and biases using

(stochastic) gradient descent

dJ
New w^], = Old -p = Old
id'

and
dJ
New bf = Old bf - p = 0\dhf-p5f.
dhf
Return to Step 1.

10.12 Example: Character recognition

Now for a proper example. I am going to use a neural network to recognise

handwritten characters. This is a good, robust test of the technique. It’s
also relatively easy, and hence commonly found in text books, because of the
ease of access to data and there is sample Python code all over the internet.
The inputs, the xs, are just many examples of handwritten numbers from
0 to 9. Each of these is represented by a vector of dimension 784. That
Just means that the digits have been pixelated in a square of 28 by 28. Each
entry is a number between 0 and 255, representing how grey each pixel is.
And where did I get this data from? There is a very famous data
set, the Modified National Institute of Standards and Technology (MNIST)
database of 60,000 training images and 10,000 testing images. These are
10.12. EXAMPLE: CHARACTER RECOGNITION 163

samples handwritten by American Census Bureau employees and American

high school students. Easy-to-read versions of this data can be found at
http://yaim.lecun.com/exdb/mnist/,
Here's one example, in Figure 10.8, this is the first line of the raw MNIST
training data. It is interpreted as follows. The first number in the top left
corner, here 5, is the number represented. The rest of the numbers are the
grey ness of each pixel.

ipython console

[4. Console i/A O m ^ ik

0,8,0^0,0,0,0,8,0,0,8,0,0,0,8,0,8,8^8,0,0,Qj0,0,0,0,0,6,0,0,8,0,0,8,0,8,0,6, 8,e,@,0,8,
8,8,0,9,0,0,8,0,0,8,8,0,0,8,0,8,8,0,8,8,0,8,8,0,8,e,0,8,0,0,0,8,0,0,0,8,8,e,8,0,6,0,8,8,0,0,8,
0,6,0,0,6,0,8,8,0,8,8,8,3,18,18,18,126,136,175,26,165,255,247,127,8,0,0,8,0,8,0,0,8,0,0,8,38,3
6,94,354,178,253,253,253,253,253,225,172,253,242,195,64,0,9,8,0,9,0,0,0,0,0,0,48,233,253,253,2
53,253,253,253,253,253,251,93,82,82,56,39,8,0,0,8,0,8,0,0,0,8,0,0,13,219,253,253,253,253,253,1
93,182,247,241,8,0,0,0,0,8,8,0,8,0,0,0,0,0,8,8,0,9,80,156,107,253,253,205,11,0,43,154,8,0,0,0,
0,8,0,9,0,0,0,8,8,8,8,0,0,,8,0,14,1,154,253,98,0,8,8,8,0,8,8,0,6,0,8,8,8,0,8,8,0,0,8,0,0,0,0,0,
0,139,253,190,2,0,0,0,0,0,0,9,0,0,9,0,0,9,8,0,9,8,0,3,0,0,9,8,0,11,190^253,79,0,0,0,0,0,0,0,0,
0,0,8,0,0,6,0,0,8,0,0,8,0,0,8,0,0,35,241,225,158,108,1,0,0,8,0,0,6,0,0,8,0,0,8,0,0,6,0,0,8,0,0
,0,0,9,31,240,253,253,119,25,0,0,0,0,9,0,0,8,0,0,0,0,0,9,8,0,9,0,0,9,0,0,9,45,186,253,253,159,
27,0,0,6,0,0,6,0,0,8,0,0,6,0,0,6,0,0,8,8,0,8,0,0,16,93,252,253,187,8,0,0,9,8,0,0,8,0,0,9,0,0,6
,8,9,0,0,9,0,0,9,0,0,9,249,253,249,64,9,9,0,0,9,0,8,9,0,8,8,8,0,9,0,6,9,0,0,9,0,46,130,133,253
,253,207,2,0,0,0,8,0,0,8,0,0,8,0,9,0,0,8,0,0,3,8,39,148,229,253,253,253,250,182,0,9,8,8,3,8,0,
0,6,8,9,0,8,8,0,0,0,0,24,114,221,253,253,253,253,201,78,0,8,8,9,0,8,0,0,8,0,0,8,0,9,8,0,8,23,6
5,213,253,253,253,253,198,81,2,0,0,0,0,0,0,9,8,0,9,0,0,9,0,0,9,18,171,219,253’253,253,253,195,
38,9,0,8,0,0,0,0,0,8,0,0,8,0,0,6,0,8,55,172,226,253,253,253,253,244,133,11,8,8,8,6,0,0,0,0,0,6
,0,0,8,0,0,0,0,9,136,253,253,253,212,135,132,16,0,0,9,8,8,9,0,0,0,8,6,9,0,0,9,0,0,9,8,0,9,8,0,

9,e,0,9,0,0,9,0,0,9,0,0,9,8,0,9,0,0,9,0,6,9,0,6,9,0,0,9,0,0

Figure 10.8: The digitized number 5.

And this is what that 5 looks like, Figure 10.9.

10 -

IS

20

25

0 5 10 15 20 2S

Figure 10.9: The sample 5.

The network that I use has 784 inputs, just one hidden layer with a
sigmoidal activation function and 50 nodes, and 10 outputs. Why 10 outputs
164 CHAPTER 10. NEURAL NETWORKS

and not just the one? After all we are trying to predict a number. The reason
is obvious, as a drawing it is not possible to say that, say, a 7 is mid way
between a 6 and an 8. So this is a classification problem rather than a
regression.

10.13 Training And Testing

For the first time in this book I'm actually going to do both the training
and the testing.
The MNIST training set consists of 60,000 handwritten digits. The net
work is trained on this data by running each data point through the network
and updating the weights and biases using the backpropagation algorithm
and stochastic gradient descent. If we run all the data through once that is
called an epoch. It will give us values for the weights and biases. Although
the network has been trained on all of the data the stochastic gradient de
scent method will have only seen each data point once. And because of the
usually small learning rate the network will not have had time to converge.
So what we do is give the network another look at the data. That would
then be two epochs. Gradually the weights and biases move in the right
direction, the network is learning. And so to three, four, and more epochs.
We find that the error decreases as the number of epochs increases. It
will typically reach some limit beyond which there is no more improvement.
This convergence won’t be monotonic because there will be randomness in
the ordering of the samples in the stochastic gradient descent.
In Figure 10.10 is a plot of the error, measured by the fraction of digits
that are misclassified, against the number of epochs.

0.07

0.06

2 0.05
0.04

0.03

0.02

0.01

0
5 10 15 20

Number of epochs

Figure 10.10: Error versus number of epochs, training set.

10.13. TRAINING AND TESTING 165

After 19 epochs we are getting a 97.4% accuracy.

So the network has learned, the error is decreasing. But what has it
learned? Has it learned the right thing?
We want the network to learn to recognise handwritten digits, but we
don't want it to memorize the data we have given it. We don't want to
overfit. That's why we hold back some data for testing purposes. And
obviously the data we hold back will be a random subset of the entire data
set.

Let’s look at how well the trained network copes with the test data. In
Figure 10.11 we see the error versus epoch for the test data as well as for
the training data.
Clearly the network is not doing so well on the test data, with only a
95% accuracy. That's almost double the error on the training data. But it's
still pretty good for such a simple network.

0.07
Training set
0.06 Test set

2 0.05 ^%
^^
0.04

0.03

0.02

0.01

5 10 15 20

Number of epochs

Figure 10.11: Error versus number of epochs, training and testing sets.

In practice one would conduct further experiments, varying the number

of nodes in the hidden layer and also varying the number of hidden layers.
As a rule the more neurons you have then the more epochs you'll need for
training.
Also with more neurons you often find that for the test data the error
gets worse with increasing number of neurons. This is a sign that you have
definitely overfitted the data. And it's really easy to overfit if you have a lot
of neurons. Take a look in Chapter 2 to see the discussion of training, test
and validation datasets.
Once one has an idea of accuracy versus architecture one can then look at
speed of prediction for new samples. The time of training is not necessarily
important but if speed of prediction is important then one will need an
architecture that is fast. It's very easy to estimate the time taken for an
166 CHAPTER 10. NEURAL NETWORKS

architecture. For example suppose one has 784 inputs, 50 nodes in one
hidden layer and ten outputs then the time taken will be proportional to

784 X 50 + 50 X 10 = 39,700.

But if we had, say, 784 inputs, one hidden layer of 30, another of 20, and
ten outputs then the time would be proportional to

784 X 30 + 30 X 20 + 20 X 10 = 24,320.

The same number of nodes, but only two thirds of the time.
(This is assuming that different activation functions all take roughly the
same time to compute.)

My digits
I showed a few of my handwritten digits to the trained network. In Figure
10.12(a) is my number 3 together with the digitised version. It got this one
right. Figure 10.12(b) is my seven, it got this one wrong. Perhaps because
it's European style, with the bar.

1
io i

(a) My number 3. (b) My number 7.

Figure 10.12: My numbers.

The output from the network is a ten-dimensional vector, and by looking

at the entries we can see what the network thinks of my handwriting. This
output suggests that the network sees my number 7 as 84.2% number 2,
6.8% number 7 and 5.7% number 4. You can see where it’s coming from.

Nonsense

Often one sees tests of character-recognition networks using input noise

to see which number is forecast. I thought I'd try something similar but
different.
10.13. TRAINING AND TESTING 167

So as a final experiment I gave the network a couple of famous album

covers and a Wilmott magazine cover, all converted to 28 x 28, just to see
how these are interpreted. These are shown in Figure 10.13.
The White Album was interpreted by the network as a five, Smell The
Glove was a zero, and the Wilmott magazine (featuring Ed Thorp, see the
next chapter) was seen as a three.

Thn SEW l.ES

(a) The White Album, The (b) 28 X 28 version of The

Beatles. White Album.

(c) Smell The Glove, Spinal (d) 28 X 28 version of Smell

Tap. The Glove.

(e) Wilmott magazine, (f) 28 X 28 magazine cover.

January 2017.

Figure 10.13: Covers

168 CHAPTER 10. NEURAL NETWORKS

10.14 More Architectures

Autoencoder

The autoencoder is a very clever idea that has outputs the same as the
inputs. Whaaat? The idea is to pass the inputs through a hidden layer (or
more than one) having significantly low/er dimension than the input layer and
then output a good approximation of the inputs. It’s like a neural network
version of Principal Components Analysis in which the original dimension of
the data is reduced (to the number of nodes in the hidden layer).
Training is done exactly as described above, just that now the ys are the
same as the xs.
You can see the idea illustrated in Figure 10.14.

-0.23 (jo.2-1 )
1.07

-0.16 -0.14

\,

■\ ■\ ./• / -N
-0.98 / i-1.00
.y-'
0.84

\
0.67
-
) -j^67
0.11

0 •1.74
/' .y-'
-0.14 f-0.14 ]

1.83

2.11 '

/-
{d.70 ■0.69

Figure 10.14: Il lustrating the autoencoder.

The data goes in the left-hand side (the figure Just shows one of the sam
ples). It goes through the network, passing through a bottleneck, and then
comes out the other side virtual ly unchanged. You’l l see in the figure that
the output data is slightly different from the input since some information
has been lost. But we have reduced the dimension from ten to three in this
example.
Now if you want to compactly represent a sample you Just run it through
10.14. MORE ARCHITECTURES 169

the autoencoder to the bottleneck and keep the numbers in the bottleneck
layer (here the 0.84, 0.11, -1.74). Or if you want to generate samples then
just put some numbers into the network at the bottleneck layer and run to
the output layer.
There doesn't have to be Just the one hidden layer. As always there can
be any number. But the lowest dimension is the dimension of the smallest
layer.

Radial basis function network

The radial-basis-function network uses typically the Gaussian function

instead of a linear transformation. So in a node in a hidden layer you would
see

Wi e
-bIx-Cjp

instead of, say, a sigmoidal function. And then the output layer would add
up all of these, perhaps with another activation function transformation as
well.
You can probably see overlaps with self-organizing maps, support vector
machines and K nearest neighbours.

Recurrent neural network and long short-term memory

More complicated network architectures have feedback loops. These
recurrent networks (RNN) have connections such that the output from a
neuron feeds back into this neuron as well as being passed on. Suitably con
structed such networks have a memory that means they can cope well with
data that is sequential in nature, where the order matters, such as speech.
Such networks are often represented by something like Figure 10.15.

a(n) vW

Figure 10.15: A schematic of a recurrent neural network.

This can be unpacked as shown in Figure 10.16.

170 CHAPTER 10. NEURAL NETWORKS

^(n) a(n)

^(n+i: a(n+1) 'y(n+l7

xCn-n a(n+2; 'y(n+2f

Figure 10.16: The RNN unpacked.

These networks can be complemented with long short-term memory

(LSTM) building blocks. These units of a network have gates that can
pass information on, reject it, or decide how much to remember.

10.15 Deep Learning

You will often hear the phrase Deep Learning. Different people mean
different things when they use the phrase, there doesn't seem to be a gen
erally recognised definition. But what all the different interpretations have
in common is that neural networks are involved and that they have a large
number of neurons and hidden layers.
As the amount of data that we have access to explodes and the speed
of computation increases so it makes sense to explore the possibilities of
networks that are deeper and deeper. Why stop at L layers? When you get
more data you might as well look at the performance of L + 1 layers and
more sophisticated architecture.

Further Reading

The first version of the Universal Approximation Theorem was by George

Cybenko in 1989, but only for activation functions with sigmoidal-type asymp
totic behaviour. The theorem applies even if the activation function is not
monotonic. See "Approximations by superpositions of sigmoidal functions,"
Mathematics of Control, Signals, and Systems 2(4), 303—314.
10.15. DEEP LEARNING 171

A simple, straightforward book that explains what is going on and has

Python code for character recognition is Make Your Own Neural Network
by Tariq Rashid. The code actually works, helped me produce some of the
results above, and almost made we want to learn Python. Almost. Thank
you, Tariq.
Quite a detailed book describing many architectures is Neural Network
Design by Martin Hagan, Howard Demuth, Mark Beale and Orlando De
Jesus.
Chapter 11

Reinforcement Learning

11.1 Executive Summary

Reinforcement learning is one of the main types of machine learning.

Using a system of rewards and punishments an algorithm learns how to act
or behave. It might learn to play a game or move around an environment.
The key to reinforcement learning is that the game or environment is not
explicitly programmed. There is no rule book to which we can refer. This
means that the algorithm has to learn the consequences of taking actions
from taking those actions. The algorithm learns by trial and error.
Unlike the topics in the previous couple of chapters you can do plenty of
reinforcement learning in a spreadsheet, at least to get you started.

11.2 What Is It Used For?

Reinforcement learning is used for

• Learning how to interact with an environment especially when the

environment or the result of interactions aren’t known in advance

• Example: Learn how to optimally bid at an auction

• Example: Learn which adverts to present to individual consumers

• Example: Learn how to play video games

I have saved what might be the most interesting methodology to last.

It’s interesting if you have children or dogs. While unsupervised learning is
about finding relationships in data, and supervised learning is about deciding
what something is, reinforcement learning is about teaching the machine to

173
174 CHAPTER 11. REINFORCEMENT LEARNING

do something. And it really is inspired by behavioural psychology. Sit, down,

fetch, roll over, are all commands that with the right sort of reinforcement any
child will eventually understand. We want an algorithm to learn what actions
to take so as to maximize some reward (and/or minimize punishment).
Because you want the machine to learn to attain some goal it is very
common to see the method used for playing games. And our examples here
will also be from, or related to, games. But the trick with reinforcement
learning is that you don't necessarily tell the machine explicitly what the
goal of the game is or even what the rules are. It will learn by trial and error
as it interacts with its environment.
You can see this happening if you give a two-year old the TV remote
control. They’ve seen you use the remote control to switch on the TV and
change channels so they know of some link there. But when first given the
remote they will press buttons at random until something happens. There
will be a phase during which they home in on the most important buttons,
and after a while they learn which is the button to switch the TV on, to
change channels, to start a DVD. And thereafter they will use a subset of
buttons reliably. (Mind you, which adult knows what more than a 20%
subset of the buttons do as well?) Funnily though they never seem to learn
how to switch the TV off.
Another way of looking at trial and error is in terms of exploiting and
exploring. On one hand you want the machine to take advantage of/exploit
everything it has learned in order to win the game, say, but on the other hand
it won’t learn anything unless it has done plenty of exploration beforehand.
Getting a balance between exploiting and exploring will be important to our
learning algorithms.

11.3 Going Offroad In Your Lamborghini 400 GT

The structure of this chapter is to alternate motivating examples with

mathematics. The motivating examples are Noughts Crosses, fruit ma
chines, a simple maze and Blackjack.
You will be seeing some rigorous and fun mathematics based around these
simple examples. However, and I want to emphasis this, when it comes to
reinforcement learning proper as applied to real problems we won’t necessarily
have that much in the way of rigorous underpinnings. Our examples all have
elegant formulations that inspire the mathematics. But real-world problems
typically don’t.
Anyway, elegance is not the point of reinforcement learning proper. Rein
forcement learning comes into its own when we have a problem that doesn’t
have any such elegant formulations. The problem might be too messy, too
complicated, too uncertain, etc.
11.4. JARGON 175

To me this is a slightly strange topic. We do lots of mathematics but

in practice that mathematics is going to ultimately be irrelevant. At least
that's how I see it. An analogy comes from motoring.
You design, build and test a beautiful new car. It is designed to look
good parked outside Harrods. It is built carefully by skilled craftsmen. And
it is tested on very straight, very smooth motorways. It is a dream of a car.
It goes into production. But the new owners decide that they want to use
the car for offroading in Wales. What could possibly go wrong?
The designing, building and testing are what we will do on our simple,
elegant problems. That's not exactly pushing reinforcement learning to its
limits. And then we let the algorithm loose in a totally new and very com
plicated terrain. That is reinforcement learning. And quite frankly we don't
know whether or not it’s going to work. That is, either until it beats the
professionals at Go or it gets stuck fording a stream in Wales.
And this is why I keep referring to reinforcement learning proper. The
"proper" is to emphasise that we typically use reinforcement learning when
we have little clue as to what the environment is, or it’s just too complicated
to represent compactly.
I end the chapter, and the main body of the book, with the example of
Blackjack. It is perfect for trying out these techniques. And we are going to
treat the game as if we have no idea as to what the game is all about. That
means that we won't be relying on any of the aforementioned foundations.
Of course, there's the usual disclaimer, do not take any of the results here
into a casino and expect to win. You really mustn't rely on my programming.
Oh, and I’ve also saved this method until last because it's a bit heavier
on the mathematics. And it’s the longest chapter.

11.4 Jargon

I’m going to explain some preliminary, basic, jargon while referring to a

few common games.

• Action: What are the decisions you can make? Which one-armed
bandit do you choose in a casino? Where do you put your cross in the
game of Noughts and Crosses? How many, and which, cards do you
exchange in a game of draw poker? Those are al l example of actions.

• Reward/Punishment: You take an action and you might get re

warded. You press a button on the wall in Doom and the BFG is
revealed. You take your opponent's piece in checkers. It's white choco
late and you eat it. Those are examples of immediate rewards. But
176 CHAPTER 11. REINFORCEMENT LEARNING

there might not be any reward until the end of the game. At the end
of the chess game the winner gets the $1,000 prize.
But there aren't just rewards. To win the prize you have to be the first
to solve the jigsaw puzzle. Every second you take can be thought of
as a punishment.

• State: The state is a representation of how the game is now. Think

of it as a snapshot of the gameboard, for example. It might be the
positions of the Os and Xs in a game of O&iXs. Or the positions of the
stones in a game of konane. The state is an interesting concept. How
much information is needed to represent a state? In O&Xs Donald
Michie needed 304 matchboxes. In Blackjack, which we shall see in
detail later, you need at a minimum to know the count of the cards
you hold, and how many Aces, and what the dealer's upcard is. To
stand a chance of winning in a casino you also need to know something
about what cards have been dealt from the deck(s). But sometimes
the amount of information you must store is prohibitively large. In Go
there are typically 361 points, each of which could be empty, or be
occupied by a white or black stone. Or the state might not even be
represented by discrete quantities. At what angle should you kick the
ball when taking a penalty? And for reinforcement learning proper we
won't even know what information represents the state.

• Markov Decision Process(MDP): Markovian means that what hap

pens going forward only depends on the current state. Chess is a
Markov Decision Process, the state of the board tells you everything
you need to know to make a decision about your next move, it doesn't
th
matter how you got there. If we use ,sy to denote the state at the t
time step then for a Markov process we could write Prob(,St+i s' I
Si, S2, • ■ • ,St) = Probj-st+i = s' I Si). I.e. the probability of being at
some state s' at the next step only depends on the current state. Note
that we can include time or time step as a component in the state.

11.5 A First Look At Blackjack

Blackjack, a game we shall look at in detail later in this chapter, is an

MDP if you keep track of enough information to represent the state. That
state is not represented simply by your cards. You need to keep track of
a lot more information to capture the state. Part of that information is
knowing the dealer's upcard. But even knowing the dealer's upcard as well
as your cards is not going to totally specify the state. The reason is that
what happens next, in the drawing of cards, depends on what cards are left
11.6. THE CLASSICAL MDP APPROACH IN O&XS 177

in the deck (or decks, plural, in casino Blackjack the dealer will start with
five or more decks shuffled together). Or equivalently knowing what cards
have been dealt out already. (Well, not their suits, and 10, Jack, Queen and
King all count as 10s, but that's still a lot of into you need to know.) So
Blackjack is an MDP if you are able to memorize all of this information.
But that's unrealistic for a mere mortal.. .and you aren't allowed to use a
computer in a casino. See Rain Man. At the end of this chapter I’ll briefly
show you how you can approximate the state sufficiently to win at Blackjack.
An exception to this is when there is an infinite number of decks being
dealt from, in an online game. In that case Blackjack is an MDP if the
state is represented by your cards and the dealer’s upcard. That’s because
the probabilities for the next cards to be dealt never changes. But then you
can't win if you play with an infinite deck. Confused? You will be!
Markov refers to there being no memory if your state includes enough
information. So that is part of the trick of learning how to play any game.
Keep track of as many variables as needed, but no more.
You can get a lot more sophisticated with different types of MDP. For ex
ample in a Partially Observable Markov Decision Process(POMDP) you only
get to see some of the current state. And so you might have a probabilistic
model for what you can't see.

11.6 The Classical MDP Approach In O&Xs

The state in O&Xs is represented by the positions of the Os and Xs. The
order in which they were played, i.e. how you got to that state, is irrelevant.
Starting with an empty grid and it is us, with Xs, to go first we could sketch
something like Figure 11.1 showing the three possible moves. There are only
three despite the nine cells because of symmetries.

Figure 11.1; A tree structure shows what could happen next.

This structure continues for all possible plays in the game. From each
178 CHAPTER 11. REINFORCEMENT LEARNING

state branch out other states for the next play. Some finish when one player
gets three in a row. And some end up with all cells filled and no winner. See
Figure 11.2.

X X
End
X
00 0
© y
X )Ljx_XOX
/
X X
0 0 o 0

W'"-
X X
X X
0X
0 0
o 0

\
\ XXX
End
0 0

Figure 11.2: Part way through a game.

In Figure 11.2 I have labelled moves from state S as L for Lose, D for
Draw and W for Win. For example at the very top right you will see that
our opponent, the Os, have got three in a row. That's an L for us.
Now we ought to make some assumptions about how good our opponent
is. That will help us make decisions about what actions to take.

A good opponent If we are playing against a good opponent then we

must try not to get into any state from which an L follows. Referring to the
figure that means that we must not put the X in the middle, that is we must
try not to get to state S. That's why the line going into state S has been
labelled L.
One would usually assign numerical values to winning, drawing and losing.
Let's say win is +1, a draw is 0 and a loss —1. That means that if we are
playing against a good player then State S in the figure has value —1.
Each branch coming out of a state will have a value. And if we are
playing against a good player then we must assume that they will choose
whatever gives us the lowest value.
Classically this is known as a minimax problem: You assume that on their
move your opponent does the most to harm you, and then on your move you
make the optimal move to minimize that harm.
The values wil l trickle back down the tree structure.
11.7. MORE JARGON 179

Opponent playing randomly However if our opponent is putting their 0

in vacant cells at random then State S is rather good. There is one losing
position, —1, one draw, 0, and two (thanks to symmetry) wins, each +1,
giving an average value for State S as 0.25.

And what if we don’t know how good our opponent is? No problem.
Because that's life! And that's what reinforcement learning is all about.
Learning on the job, while experiencing the environment rather than having
the environment programmed in.

All this is a gentle introduction to...

11.7 More Jargon

• Value Function: A value function measures how good or bad a state

or an action is. If we are in some state now and al l future states are
good then the value at our present state will be high. But that value
depends on what actions we take now and in the future. At each state
we wil l have some choices to make. Value comes from accumulation
of rewards and is how good our current position is given what we do
now and in the future. Notice that I mention both state and action
here. When we get to the mathematics you’ll see how I distinguish
between two types of value function, one specific to the state and one
that is associated with both state and action. Given a state and how
we decide what action to take defines our. . .

• Policy: A policy is a set of rules governing what action to take in each

state. That policy might be deterministic. Or it might be random.
Ultimately in reinforcement learning we want the policy to maximize
value at each state. But of course a priori we don't know what the
best policy is. .. that's what we are trying to get the machine to learn.

In O&Xs each state would have a value, a value that percolates down
from later states in the game.
I'l l be talking about the value function in relation to reinforcement learn
ing in some detail and give some ideas about how to set it up.
So that's how you might address the game of O&Xs classically. Other
games and problems can be set up in the expanding tree structure as well.
Let's start to move out of the waffly, descriptive mode, and start at least
hinting at some mathematics. We shall do this by looking at an extremely
popular illustrative example.
180 CHAPTER 11. REINFORCEMENT LEARNING

11.8 Example: The multi-armed bandit

You'l l know of the one-armed bandit. It is a gambling machine found in

casinos and bars. Originally these machines had a lever, the arm, that you
pull, and this causes cylinders, on which there are pictures of various fruit, to
spin. (Hence the name fruit machine.) If they stop on the right combination
of lemons, cherries, etc. then you win a prize.
In a casino you will have many of these in a row. (So, I think multiple
bandits is a much better, and less clunky, name than multi-armed bandit.)
In the multi-armed bandit problem we have several such bandits each with
a different probability of winning a fixed amount, the same amount for each
bandit. The goal of the multi-armed bandit as a problem in reinforcement
learning is to choose among the bandits, pull the lever, see if you win, try
a different bandit, and by looking at your rewards learn which is the bandit
with the best odds.
This problem is quite straightforward. There is first of all no state as
such. But there are actions, which bandit to choose. We want to assign a
value to each action. And then based on these values decide which action
to take.
This is how it goes.. .

• There will be ten bandits. The probabilities of winning for each bandit
are

Bandit 1 10%
Bandit 2 50%
Bandit 3 60%
Bandit 4 80%
Bandit 5 10%
Bandit 6 25%
Bandit 7 60%
Bandit 8 45%
Bandit 9 75%
Bandit 10 65%

Table 11.1: Table of probabilities of winning in the multi-armed bandit ex

ample.

• The value of each action, each bandit, will simply be the average
reward for that bandit. This average is only updated after each pull of
a lever. To be precise, it is only the chosen bandit that has its average
11.8. EXAMPLE: THE MULTI-ARMED BANDIT 181

updated. And the average is calculated as the total actual reward so

far, using randomly generated success/fail at each pull, divided by the
total number of times that bandit has been chosen. To al l intents and
purposes the reinforcement algorithm does not know about the odds,
only experiences them.

• After each pull we have to choose the next bandit to try. This is done
by most of the time choosing the action (the bandit) that has the
highest value at that point in time. But every now and then, let's say
at a random 10% of the time, we simply choose an action (bandit) at
random with all bandits equally likely.

That last bullet point describes what is known as an e-greedy policy. You
choose the best action so far, but you also do occasional exploration in case
you haven’t yet found the best policy. The random policy happens a fraction,
e, of the time.
I'd also like to emphasise that only you, the reader, and I know the prob
abilities of winning for each bandit. We will not be telling the reinforcement
learning algorithm explicitly what these probabilities are. As I said above,
the algorithm only indirectly experiences them.

Bandit 4 Value
Bandit 3

■ n Bandit 9

“ /V I
0.6 I
&
7
1
0.4

0.2

Number of pulls
0

0 10 20 30 40 50 60 70 80 90 100

Q(l) Q(2) Q(3) •Q(4) Q(5)

QI6) Q(7) Q(8) Q(9) 'Q(IO)

Figure 11.3: Value for each bandit versus number of pulls.

In Figure 11.3 is shown the value function for each of the ten bandits as
a function of the total number of pulls so far. I have used Q to denote this
value, we'll see more of Q later. And in Figure 11.4 we see the fraction of
time each bandit has been chosen.
182 CHAPTER 11. REINFORCEMENT LEARNING

Fraction of time each bandit has been chosen

1

<\
\
0.8 Sand it 9
Bandits \

0.6
\
\
Bandit 4
0.4

Number of pulls
0.2

0
0 10 20 30 40 50 60 70 80 90 100

Bandit 1 — Bandit 2 - - -Bandit 3 Bandit 4 Bandit 5

Bandit 6 Bandit 7 Bandit 8 Bandit 9 Bandit 10

Figure 11.4: Fraction of time each bandit has been chosen.

It's hard to see what's going on in black and white with so many curves,
but let me walk you through it. Bandit 3 was chosen at random for the first
pull. The first pull is always random, we have no data for the value yet, and
all Qs are set to zero. Bandit 3 loses. For the next pull the Qs are still all
zero and so each bandit is equally likely. Coincidentally Bandit 3 is chosen
again. This time Bandit 3 wins. Its Q value is now 0.5 and so is currently
the best bandit. This means that it will be chosen preferentially except when
the 10% random choice kicks in. Bandit 4 eventually gets picked thanks to
this random choice. It wins and thus now becomes the preferred choice. It
wins twice, then loses. And so it goes on. Up to 100 pulls it is looking like
Bandit 9 is the best. But then...

Fraction of time each bandit has been chosen

1

0.8

0.6 Bandit 4

0.4
Bandit 9

0.2
Number of pulls
0
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Bandit 1 Bandit 2 Bandit 3 *«w«w«8andit 4 Bandit 5

Bandit 6 Bandit 7 Bandit 8 Bandit 9 Bandit 10

Figure 11.5: Fraction of time each bandit has been chosen, longer run.
11.9. GETTING MORE SOPHISTIGATED 1 183

In Figure 11.5 is shown the fraction of pulls for each bandit up to 10,000
pulls. Clearly Bandit 4 has become the best choice. And if you look at Table
11.1 you’l l see that it does indeed have the highest probability of success.
The correct bandit will eventually be chosen however the evolution of the
Qs and the fraction of time each bandit is chosen will depend for a while on
which bandits are chosen at random. So in Figure 11.6 I show the results
of doing 100 runs of 10,000 pulls each. Having 100 runs has the effect of
averaging out the randomness in the experiments. Note the logarithmic scale
for the horizontal axis.

Fraction of time each bandit has been chosen

0.8 (log scale) - averaged over 100 runs

0.6
Bandit 4

0.4
Bandit 9
A'

0.2

0
1 10 100 1000 10000

Bandit 1 Bandit 2 Bandit 3 Bandit 4 Bandit 5

Bandit 6 Bandit 7 Bandit 8 Bandit 9 Bandit 10

Figure 11.6: Fraction of time each bandit has been chosen (log scale) —
averaged over 100 runs.

11.9 Getting More Sophisticated 1: Known

environment

We’ve done the preparatory work, seen basic ideas of how and where
reinforcement learning can be applied, now let’s start putting some proper
mathematical flesh on these bones.
In the next few sections we are going to do some of the mathematics
behind valuation for a given policy, policy evaluation, and then find the
optimal policy, the control problem. But initially we are going to restrict our
attention to known environments. Known environment in an MDP means
that for any state and action we have a known probabilistic description of
our next state and any reward.
184 CHAPTER 11. REINFORCEMENT LEARNING

In a sense Blackjack is a known environment because in principle we

could write down the probabilities of going from one state to another after
taking a certain action. So maybe have that idea in mind while we next learn
about the subject of Dynamic Programming.
This isn’t really reinforcement learning yet. That comes when we move
to an unknown environment.
We are going to be justifying reinforcement learning by looking at related
topics. Along the way we'l l be introducing things like transition probabilities.
But when we finally get to reinforcement learning proper we won't be seeing
them anymore. And that's simply because reinforcement learning is designed
to work even if you don’t know these probabilities.

Basic notation

We shall need notation to represent states, actions, rewards, and transi

tion probabilities between states. And some other things. Everything that
follows uses the standard notation seen in most of the literature. Under
standing notation is half the battle of learning any new subject.

States Let’s use S to denote the general state and s to mean a specific
state. I'll use s' to mean the next state (after we’ve taken some action).
And sometimes I’l l give s a subscript when I want to emphasise the time or
step, St.

Actions I'll use A to mean the general action and a to denote a specific
action. Again the following action will be a' and sometimes the action will
have a subscript when I want to emphasise the time or step at which the
action is taken, aj. Note that taking a certain action does not guarantee
what state we’l l end up in. For example, in Blackjack taking another card
does not tel l us what our next card count is. But we could have a simple
case, like in the maze that we'l l see later, where the action of moving right
does tel l us our next state.

Rewards Generally the reward can depend on any or all of action a, cur-
rent state s and next state s'. I'll use r to denote the actual reward. Some
times the r will have a subscript if 1 need to make it clear which reward I am
talking about. For example ry+i is the reward that comes between state St
and state S(,+i.
11.9. GETTING MORE SOPHISTICATED 1 185

I will also write

Rss' E[n+i I St = s, at = a,St+i = s'].

In this expected reward we've got the before and after states, as well as the
action. Reward includes punishment, that would just be a negative reward.
For example if we are trying to find the shortest path from start to finish in
a maze then we would have a reward of —1 for every action (step) regardless
which way it is. Then maximising the sum of rewards would be the same as
finding that required path.

Transition probabilities In general we will need to specify the probability

of going from state s to state s' after taking action a. We denote these
probabilities by

P.
ss' Prob (si+i = s' St = s,at = a).
For our maze below these probabilities would be either 1 or 0, because a
given action tells us exactly where we go next. But for Blackjack the action
of taking a card puts us in a random next state.

Policies Given a state S = s then there will generally be several possible

actions. And generally speaking those actions can have probabilistic elements
to them. We write such a stochastic policy as

7r(a|s) = Prob(^ = a \ S = s).

In words, this is the probability of taking action a when in state s. I have
chosen to write this using the|symbol to separate the variables, but you
will often see a simple comma in the literature. I have made this choice to
emphasise two things: That you first get to a state then the action comes
next; In each state the choice of actions could be completely different (buy
an apple or orange in the shop, or walk left or right out of the house).
I shall also use tt as a subscript shortly. Then it just means "for the given
policy." You’ll see.

Goal or return The total reward, called the return or goal, that lies ahead
of us is
OO

G. = n+i + 7rt+2 +'r‘^ri+3 + ■ ■■=

j=0

The parameter 7 G [0,1] is a discount factor that says that the sooner the
reward the more we value it.
186 CHAPTER 11. REINFORCEMENT LEARNING

The discount factor If you come from a finance background then you
will know that use of discount factors, related to interest rates, is extremely
common. We talk of the time value of money and opportunity cost meaning
all things being equal it's better to get money upfront. A bird in the hand is
worth two in the bush. However the discount factor in reinforcement learning
can seem a bit arbitrary. Non-financial reasons for introducing one are:

• It can avoid infinite sums, when adding up an infinite number of re

wards. (There might be loops in a game giving rewards and rather
than seeking a better solution we get caught up in this loop)

• If we aren't totally sure about the environment then there is model

uncertainty in the future so best take those rewards now

• In a real-life scenario the rewards might be perishable!

And the geometric time decay (the increasing powers of 7) is the most
common discounting mechanism since it has nice mathematical properties,
as we shal l see.

11.10 Example: A maze

I've described the concept of Markovian in reference to states and mem

ory. A Markov chain is a sequence of events in which we move from state
to state according to given probabilities.
Our example now is a maze, see Figure 11.7. This is about the smallest
maze that I can use to illustrate the idea of Markov chains.
It's a simple example because first of all the reward will be -1 for every
action, i.e. every step. And initially the policy will also be simple, we will
move to any adjacent cell with equal probability.

A B C

START

FINISH
3

Figure 11.7: A very, very simple maze.

11.10. EXAMPLE: A MAZE 187

We start in Cell Al, can only go right from there, and when we get to
Cell C3 we stop. Our state is represented by which cell we are in, B3 for
example.
In a Markov chain the probability of moving from one state to another is
given. But recall that in our notation above we had a potentially stochastic
action (the policy, tt) and a potentially stochastic transition (the transition
probabilities, P^^,). In the simple Markov chain here these two can be com
bined into one transition probability matrix, and that's because given an
action (albeit random) the following state is deterministic: Move down from
B1 and you always end up in B2. (Another way of thinking of this is that
Pgg, only contains ones and zeroes.)
To represent the (combined) transition probabilities we would need a 6
X 6 matrix (there are six cells). See Figure 11.8. In this example I am going
to make a move from one cell to a neighbouring one equally likely: If there
is only one neighbour then the probability of moving to it is 100%, if two
neighbours then 50% each, etc. At this point there is nothing special about
Cells Al and C3, other than we can only go in one direction from Al and
we stop when we get to C3, there is no attempt to go from Start to Finish.
The transition probabilities tell us where to go with no goal in mind.

To

Al A3 B1 B2 B3 C3
From Al 0 0 1 0 0 0

A3 0 0 0 0 1 0

B1 0.5 0 0 0.5 0 0

B2 0 0 0.5 0 0.5 0

B3 0 0.333 0 0.333 0 0.333

C3 0 0 0 0 0 1

Figure 11.8: Transition probabilities.

Solving the Markov-chain problem

If the next move we make is given by the transition matrix in Figure 11.8
then we can write down a recursive relationship for the expected number of
steps it will take to get from any cell to the finish. Denote the expected
188 CHAPTER 11. REINFORCEMENT LEARNING

number of steps as a function of the state, i.e. the cell we are in, v{st).
We have
r;{Al) = 1 +v(Bl)
because whatever the number of expected steps from Bl, the expected num
ber from A1 is one more. Similarly

v{A3) = 1 + w(B3).

From Cell Bl we can go in two directions, both equally likely, and so

r;(Bl) = l + i(r;(Al)+t;{B2)).
And so on.. .

u(B2) = l + i(,;(Bl)+,;(B3)),
r;(B3) = 1 + ^(w(A3)+ r;(B2) + v{C3))
O

(from B3 there are three actions you can take) and finally
v{C3) = 0 + r;(C3).
You never leave Cel l C3.
These can be written compactly using vector notation. Let’s write v as
a vector v with each entry representing a state, Al, A3, ... , C3. Similarly
we'l l write the reward as a vector r.
In the above we have added 1 for every step in order to calculate the
expected number of steps in each state. But to make this more like a
reinforcement learning problem I am going to change the sign of the reward,
so that we are penalized for every step we take. This ties in with the later
idea of optimizing return. Maximizing the total reward when each reward is
negative amounts to minimizing the number of steps. The final answer for
V will be the negative of the expected number of steps for each state.
So the first five entries in this reward vector, r, will be minus one, and
the final entry will be zero. This just means that in going from state to state
we get a reward of —1, but since we can't leave cell C3 there is a zero for
that entry. And we shall write the transition matrix in Figure 11.8 as P.
Writing the above in vector form to find v all we have to do is solve
V = r + Pv. (11.1)

This tells us the relationship between all the expected values, one for each
state. It is a version of the Bellman equation. We shall be seeing different
forms of the Bellman equation later.
This can be solved by putting the two terms in v onto one side and
inverting a matrix. However since inverting matrices can be numerically
11.10. EXAMPLE: A MAZE 189

time consuming it is often easier, and certainly will be in high dimensions,

to iterate according to
Vfc+i = r + Pvfe.

We've seen iterative methods in gradient descent and we are going to be

seeing more iterative methods when we finally get to reinforcement learning.
We can see the iteration process in action in Figure 11.9. Here we started
with the Vo having all entries zero.

A B c

START 0.00 0.00 k =0

^ p
3 10.00
I
0.00

0.00 0.00 FINISH

-1.00 -1.00 ^=1

I
-1.00

-1.00 -1.00 0.00

-2.00 -2.00 k=2

I
-2.00

-2.00 -1.67 0.00

-3.00 -3.00 Ar = 3

I
-2.83

-2.67 -2.33 0.00

-4.00 -3.92 /c = 4

I
-3.67

-3.33 -2.83 0.00

Figure 11.9: The iterative procedure for finding v.

190 CHAPTER 11. REINFORCEMENT LEARNING

The final result, after nnany iterations, is that shown in Figure 11.10.
The numbers are easy to check.

A B C

START -18 -17

2 -14

3 -10 -9 0 FINISH

Figure 11.10: Solution of the Markov chain.

That looks to me like it's taking an awfully long time to get from A1 to
C3 if you move randomly, 18 steps on average.
Of course, the big difference between Markov chains and MDPs is in the
potential for us to make (optimal) decisions about which actions to take.
Ultimately our goal in the maze is trying to get from start to finish asap. We
want to deduce, rather trivially in this maze, that, for example, the optimal
policy when in cell B3 is to move right. That will come later, after I’ve
introduced some more notation.

11.11 Value Notation

State-value function

Let's forget the maze example for a while and set out a fairly general
problem in which we have a given policy, given rewards and given probabilities
for changing from state to state. Everything can be stochastic and the
rewards and transition probabilities can be functions of ail of s, a and s'.
In the above I used v (or v) to represent the expected (negative of the)
number of steps. More generally in an MDP we can use v to represent a
value for each state. Such a state-value function is defined as

v„{st) = E,[Gt I 5 = sy]. (11.2)

This function adds up all the expected rewards, possibly dis

counted, and this tells us how good a particular state is. The subscript
on the V is just to acknowledge that the value depends on the policy. In our
maze that policy was stochastic. But this will change soon.
Expression (11.2) simply says that we take the expected value of the
11.11. VALUE NOTATION 191

sum of all future rewards. It is crucially important that we are adding up

(expected) rewards that are yet to come. Recall the discussion of the O&Xs
MDP where I mentioned valuing back down the branching tree. This seems
a bit counterintuitive, surely we should add up the rewards we’ve collected?
Well, no, because the rewards that we've collected already aren't going to,
or rather shouldn't, influence our action now and in the future. It's what we
might collect in the future starting from each state that will determine our
behaviour now. It’s no good crying over spilt milk.
Note also that if the rs are positive then our value function will decrease
as we go forward in time.
We shall find it useful to introduce another value function the...

Action-value function

. . . defined as

IS— A — ai].
This means the value when in state St of taking action A = at and afterwards
following the policy n. So that's the immediate reward plus the value that
comes after.
(And that’s the Q we saw in the section on the multi-armed bandit.)
This is the quantity that will later tell us what is the best action to take
next, the one that maximizes Q^(st,at).
The relationship between the two value functions is

(st) ='^-K{a\st)Q^{st,a).

-11.-r

Q(Bo,UP)
irob. = 3.333
A B

START -18 -17 Q(B3,LEFT) Q(B3,RIGHT)/

2 -14 prob. = 0.333 -9
3 10 -9 0 FINISH v(B3)

Figure 11.11: The relationship between the value functions for Cell B3 in
our maze.
192 CHAPTER 11. REINFORCEMENT LEARNING

This relationship between the two types of value function is shown in

Figure 11.11 for Cell B3 of our maze. I've written the -1 reward as -r
just to help emphasise what the numbers mean, that the reward is added
between states, after an action.
Since our policy is stochastic with Left, Up and Right actions all having
probability one third we have

t;(B3) = i(g(B3),LEFT)+ Q(B3), UP)+ Q(B3), Right))

1
= -(-10 - r - 14 - r +0- r) = -9.
O

You can already start to see the benefits of the action-value function Q
from the figure. Out of the three possible actions in state B3 it’s moving to
the right that has the highest Q value. Of course, that's assuming that the
state values in the adjacent cells are correct.
You should anticipate an iterative solution for the Q function, and thus
the fastest route through the maze. And that's sort of where we are heading.
Except that we will get there by going through the maze many times, using
trial and error. Enough of the maze.

11.12 The Bellman Equations

We can write these two value functions recursively: The value function
now is just the expected value of the immediate reward plus the expectation
of the value function where you go to. Let me show you how this works.

?;^(s) = E^[Gt I S't = s] = E, n+i + jrt+2 +'y'^n+s h— \ St = s

= [rt+i + j{n+2 + Vt+s + • • ■) I -S'* = .
= IEtt [^t+1 + lGt+^ I St = s].
We can write this as the recursive relationship

v^{st) = [rt+i -L I St = St]. (11.3)

But since there are potentially two probabilistic elements (the policy and
the transitions) we can make this more explicit using the notation from
Section 11.9:

(s) = J]7r(a|,s)^P“,{R ss' + 7ri^(s')). (11.4)

s'

These final results are the Bellman equation, relating the current state-value
function to the state-value function at the next step.
11.13. OPTIMAL POLICY 193

Note that this recursive relationship between the value function now and
the value function at the next step is only possible because of the simple
geometric discounting of future rewards. That’s what I meant when I said
earlier that geometrical discounting has nice mathematical properties.
The same process also gives us the recursive equation for the action-value
function

Q7r(st,flt) — |St — ,st,At —ot]•

Or more explicitly

Qtt{s,a) — y ^ ss' ss

a'

11.13 Optimal Policy

The whole point of setting up this problem is to find the optimal policy
at each state. So we are seeking to maximize v.^{s) over all policies tt;
v^{s) = maxw^(s).
TT

The * is used to denote optimal.

And similarly
Q,(s,a) = maxQ^(s,a).
For clarity, this expression is the expected return in state s when taking
action a and from then on following the optimal policy.
There are two more Bellman equations now, one for each of u* and Q,,
called the Bellman optimality equations:

v^s)= nrdxJ2 Pss'(P ss'

and

Q4s,a)= (i?. + 7iniM:Q*(s',o')^

'ss'

It’s the second of these that we focus on because it explicitly distinguishes

the next action:

= maxQ*(s,a).
a

And if we can find the next action then in principle the job is done because of
the recursive nature of the Bellman equation. This is the benefit of working
with <5, isolating the immediate next action.

1 if a = argrnax^ Q*{s,a)
7T*(a|s) = 0 otherwise
194 CHAPTER 11. REINFORCEMENT LEARNING

Once we have found the optimal policy then typically the 7r(a|s) becomes
deterministic, with probability of one for the optimal action and zero for other
actions.
For completeness, here is the action-value function Q* for our earlier
maze. That is, the action-value function for the optimal policy. Meaning
the value for taking an action and thereafter finding the optimal policy.

A B c

1 -4 -5

-3
-4

-2

-3

3 -2 -3 -1

Figure 11.12: The Q* function for our maze.

The position of the numbers in the cells just refers to the move. So the
-4 at the top of Cell B2 means the value if you go up and thereafter do
what is optimal.

11.14 The Role Of Probability

It's worth taking a moment out to look at the role of probability here. We
started out by talking about a policy being probabilistic, hence all those earlier
expectations. We've now moved on to talking about a deterministic policy.
That doesn't mean all probabilistic elements have necessarily disappeared.

• There could well be elements of the environment that are always ran
dom. This is the case with the bandits and Blackjack. In both cases
there will be deterministic optimal policies (based on the state in Black
jack) but the consequences still have random components. You may
be on a losing streak, but that shouldn’t change your policy. This is
where we are in our discussion of the theory so far.
11.15. GETTING MORE SOPHISTICATED 2 195

• Even if ultimately we are seeking a deterministic policy, such as in the

bandit example, we might get there via a stochastic policy: We were
trying to determine which bandit to bet on but we did that via a policy
with occasional random choices. Think of the probabilistic element as
just being a numerical method for getting to the right answer. This is
something coming up later. (Keep an eye out for e greedy.)

• There might genuinely be random elements that are part of the optimal
policy. Rock, Paper, Scissors is the obvious example. However we
aren't going to look at such cases here.

11.15 Getting More Sophisticated 2: Model free

In the first Getting More Sophisticated section I showed the mathematics

for MDPs with a known environment, how to find value functions for a known
policy and then how to optimize the policy. In my main example I kept things
simple by focusing on a maze with a deterministic environment. However I
did set up the mathematics for a known stochastic environment as well.
In the following sections we are going to move in the direction of unknown
environments. We won't know a priori the transition probabilities or rewards.
This is where reinforcement learning really starts.
A good example to look at is, of course, Blackjack. Technically the
Blackjack environment is known because we could write down the transition
probabilities: You hold a 12 count with the dealer's upcard being an eight,
if the action is to take a card then what are the probabilities of going to 13,
14,... , bust? We could write down a largish matrix and use the above
techniques but I'd rather use this as an example of an unknown environment,
because then we really are in reinforcement-learning territory. Instead of
going through the process of figuring out the transition probabilities, we
simply play many games.

11.16 Monte Carlo Policy Evaluation

The method we'll look at first is Monte Carlo policy evaluation.

This method Just plays the game, or whatever, many times and calculates
expected, empirical, returns for each state over many games. You don't need
complete knowledge of the environment, like you do for dynamic program
ming. All you need is to have experience of the environment. It's a simple
method and very common.
It does have a drawback and that is that it only really works when you
196 CHAPTER 11. REINFORCEMENT LEARNING

have complete returns, which means that the MDP terminates: An episodic
MDP has a well-defined start and end, it doesn’t go on forever. You start
the game, play, and then finish. That's one episode, and then you start all
over again.
You couldn't easily calculate an expected return if the game never ter
minated.
We will use T to denote the step at which the game finishes. T can be
different for each episode, i.e. each time we play the game. So now
T-t-l
Gt = n+i -h 7n+2 -h 7^n+3 -h ... + 7 TT-

Again our value function is defined as the expected return

A {st) = E,[Gt I 5 = St]

for our chosen policy. But this expected value will (have to) be the empirical
mean return rather than some theoretical mean, since we aren't given the
probabilities.
In doing a simulation there are two important points that we have to
address:

• Can we be sure that we will visit each state enough times to be able
to get a decent estimate of the expectation? The standard deviation
of the estimated expectation for a state decreases in proportion to the
inverse of the square root of the number of episodes for which we've
been in that state.

• What do we do if we visit a state more than once during the same

episode? For example in a game of chess. There are two approaches...

Suppose we have the following three episodes:

■Si ^ S2 ^ Si ^ S3 ^ Si ^ terminates,

S2 ^ S'4 ^ S2 ^ Si ^ terminates,
ru) J-l,
Si S2 terminates.

The first is interpreted as start in state si and move to state S2 picking up

a reward of ri en route etc.

First-visit evaluation We calculate the average for a state by looking at

total return over many episodes and dividing by the number of episodes in
which you went through that state. Do not divide by number of times you
went through the state, you might visit the state several times in one game.
11.16. MONTE CARLO POLICY EVALUATION 197

For the above three episodes, without any discounting, we get:

1

^(si) = 3 ((»'1 + r-2 + r3 + T4 + rs)+(rg)+(no +rn)).

The three in the denominator is because ,si was visited in all three episodes.
And I have separated the rewards from those three episodes using parenthe
ses.

■^(^2) = 3 ((r-2 + r3 -I- T4 -F rs) -F (re + rr -F rs -F rg) + (rn)),

V{S3) j((r4 + r5) + (0) + (0))
and
1
V{S4) -((0) + (r7+r8-Frg)-F(0)).
In practice we'd have a lot more than three episodes.

Every-visit evaluation The alternative is to count every visit to a state

and every reward from that state from that point until termination. This
means that some rewards will count several times. For the above three
episodes we would have the following.
1
v{si) ^ ((ri + T2 + T3 + T4 + rs -F 7-3 + r4 + rs T r5)+
{rg) + (no +?-ii)) •
State Si is visited five times in the three episodes.
1

^^(S2) = ^ ((r2 + rs + r-4 + r5)+

(re + r-7 + rg + rg + rg + rg) + (ru)).
And the remaining two are the same as under first-visit since neither states
S3 nor S4 are visited more than once in any episode.
While we are just valuing a given policy, as opposed to finding the optimal
policy, is doesn't matter if we don't visit all of the states. We only have to
visit those states which can be reached by the given policy.

Updating the expectations

The mean value that we want to calculate is Just the simple arithmetic
average of al l values so far. That takes the form
n(si)
1
(i)
v{st) (11.5)
n{st) i=l
198 CHAPTER 11. REINFORCEMENT LEARNING

(*)
where I've used G to mean the return realised in the i*'’ episode in which
we have experienced state St, and n{st) is the total number of episodes in
which we have experienced state Sf so far. At the end of the next episode
in which we visit state St we'l l get an updated expectation
n(.5,,)+l
1 (*)
Vnew{st) —
n{st) + 1 E cl
i=l

(n(si)+l)
= V
o
ld(«t)+/3(G[ — V
id(-^«)) >
o

1
where /3 n(.‘it)+l ■
This is a simple updating that only requires remembering the current
value and the relevant number of episodes (in which we have reached that
state). But it can be simplified further.
What would it mean if instead of having (3 depending on the number
n(st) we instead made it a small parameter? This would tie in with the sort
of updating we've see in other numerical techniques, gradient descent for
example. Would it be cheating? It would certainly make things a bit easier,
not having to keep track of n{st).
Well, it wouldn’t matter as long as the environment is not changing, and
that j3 did eventually decrease to zero. If the environment is stationary then
either way we would converge to the right average.
But if the environment is changing, i.e. is non stationary, then we should
use an average that takes this into account. And that what's the exponen
tially weighted average does. Of course, then there is the matter of choosing
the parameter j3 that reflects the timescale over which the environment is
changing, the right half life. But that's a modelling issue.
The rule for updating thus becomes

v{st) ^ v{st) + P{Gi- v{st)). (11.6)

The parameter (3 is again the learning rate.

We will see many techniques now that use this idea of a learning rate for
updating our value functions as new information/samples come along.
Notice how neither of (11.5) or (11.6) contain any vs other than the one
we are updating, i.e. only v{st). That is going to be very different when we
come to other methods.

11.17 Temporal Difference Learning

I said earlier that a disadvantage of Monte Carlo policy evaluation is that

you have to wait until termination of an episode before updating our values.
11.17. TEMPORAL DIFFERENCE LEARNING 199

We get in our car, start the motor and a warning sign comes up on
the dashboard. In Monte Carlo policy valuation we don’t let this affect our
expected journey time (our value) until we get to our destination (or not).
But we've experienced this warning light a couple of times before. In
temporal difference (TD) learning our expected journey time is possibly im
mediately changed.
Notice that in both MC and TD I am not saying that we change our
policy (go to the garage for example), because at the moment we are still in
policy-evaluation mode.
TD learning uses what is called online updating. This means that updat
ing is done during the episode, we don’t wait until the end to update values.
In contrast offline updating is when we wait until the end of an episode before
updating values.
We don’t need an episode to terminate before learning from it. In fact
TD learning doesn’t even need the sequences to be episodic. Unlike MC we
can learn from neverending sequences.

The TD target and TD error

Instead of moving v{st) towards Gt as in (11.6), we replace Gt with

n+i +7'!^(st-M)-

This is called the TD target. And here w(sti) is the sample that we have
found so far, not the true expected return. (Which means that this can be
a source of bias.) So the updating rule is

v{st) t- v{st) + p {n+i +'yv{st+i) v{st)).

The Tt+i +7w(st+i) —v{st) is called the TD error.

Whereas MC policy evaluation calculates the average return in the ob
vious way, by adding up all returns for each state over many episodes and
dividing by number of episodes in which that state was experienced — the
rather obvious method — TD learning trickles down from the next state
to the current state while adding the immediate reward. This makes it an
application of the Bellman equation.
This method differs in that the updating rule takes information not just
from v{st) but also from t;(st+i), the next state in our episode.
200 CHAPTER 11. REINFORCEMENT LEARNING

11.18 Pros And Cons: MC v TD

Monte Carlo Temporal Difference

Pros Easy to understand; Does not require termination
Zero bias (uses actual returns) (can use unfinished sequences);
Can be more efficient than MC;
Low variance
Cons Requires termination/episodic; Biased;
High variance (adds up all rewards) Sensitive to initial values;

11.19 Finding The Optimal Policy

We now move on to the control problem, finding the optimal policy. We

are going to bring together the known-environment dynamic-progra mming
ideas and the unknown-environment policy-evaluation ideas all seen above
to address the problem of control in an unknown environment.
Our first observation is that using the state-value function v{s) alone is
not going to work in an unknown environment. That's because we can't
evaluate which is the best action to take since we don't know where the
actions will take us. In Blackjack terms we don't know whether it’s better
to hit a 15 when the dealer shows an eight or to stand, because we don’t
know where hitting 15 will take us, the next state. For this reason we are
going to be working with the action-value function Q{s,a).

Avoiding getting stuck

One issue we will have to address frequently is how to avoid getting stuck
in a policy that is not optimal. If we always choose what seems to be the
best policy then we might be premature. We might not have explored other
actions sufficiently, or perhaps have an inaccurate estimate of values. If we
always choose what we think is optimal it is called greedy.
It is much better to spend some time exploring the environment before
homing in on what you think might be optimal. A fully greedy policy might
be be counterproductive. We might be lucky to win when we first stand on
a 14 when the dealer has an eight. We would then never experience hitting
14 when the dealer has an eight.
A simple way of avoiding getting stuck is to use e-greedy exploration.
This means that with probability 1 — e, for some chosen e, you take the
greedy action but with probability e you choose any one of the possible
actions, all being equally likely. In that way you will explore the suboptimal
solutions, you never know they might eventually turn out to be optimal.
Of course, as our policy improves we will need to decrease e otherwise
we will forever have some randomness in our policy.
11.20. SARSA 201

11.20 Sarsa

We've seen enough updating rules now that I can cut straight to the
chase. The method is called sarsa, which stands for “state action reward
state action."

Update the action-value according to

Q{st,at) ^ Q{st,at) + /?(rt+\ -h ^Q{st+\,at+{) - Q{st,at)). (11.7)

This updating is done at every step of every episode. The actions that are
chosen are from an e-greedy policy.
Let’s go through the algorithm.

The Sarsa Algorithm

Step 0: initialize

First choose starting values for Q{s,a). Although the rate of con
vergence will not be affected by this choice the time to convergence
to within some error might. Perhaps make everything zero.

Step 1: Start an episode

Start a new episode, with the initial state s.

Step 2; Move to the next state

In s choose an action using the e-greedy method and the current

values of Q{s,a). This gives you a. Take the action a and move
to next state, s'. In state s' choose the next action a' based on the
values of Q{s',a') for the possible actions at the new state and using
the e-greedy method.

Step 3: Update Q(s,a)

Update the action-value function at (s,a) according to the next re
ward you get, r and the action-value function at {s', a') with the
chosen a'.
202 CHAPTER 11. REINFORCEMENT LEARNING

Hence (11.7)

Q{s,a)^ Q{s,a)+ /3(r + 7<3(s',a')- Q{s,a))

and
s ^ s' and a ^ a'.

Here I have emphasised the updating to the state and the action as
well. See Figure 11.13.

action o'

Q(s,a) action a

state s collect r

Figure 11.13: The quantities used in the sarsa algorithm.

Repeat Steps 2 and 3 until termination of the episode and then return
to Step 1 for a new episode.

You can see where the name comes from, St, at, rj+i, St+i, at+\.

On policy More jargon. Sarsa is an example of an on-policy method.

This means that we are following a policy while also valuing or optimizing it.

Off policy We'll be looking at an off-policy method next. An off-policy

method is one where you value one policy while following a different one.
Again, Blackjack comes to the rescue... imagine tossing a coin to decide to
hit or stand. We could follow that policy, even though it's probably going to
be rubbish, and it would stil l give us information about whether hit or stand
is better in each state and thus eventually lead to the optimal policy. The
main reason for employing off-policy methods is because they can be used
for exploration and ultimately finding optimality without getting tied down
to anything that prematurely looks optimal.
11.21. Q LEARNING 203

11.21 Q Learning

A special case of an off-policy method is Q learning. In Q learning we

follow one policy, the behaviour policy, which gives us our route through the
states. But we learn from a different policy. We peek ahead to the next state
and look at all the actions we could take, update our Q state-value function
according to the best action we could have taken and then perversely possibly
take a different action. I say perversely but I don't really mean it. It's like
saying that you've got an idea which is the best route to the shops but you
are going to experiment with a different route. Ok, you might take longer
this time, but also you might find a better route. Taking a new route isn't
going to make the best possible time any worse.

The Q Learning Algorithm

Step 0: Initialize

First choose starting values for Q{s,a). Although the rate of con
vergence will not be affected by this choice the time to convergence
to within some error might. Perhaps make everything zero.

Step 1: Start an episode

Start a new episode, with the initial state s.

Step 2: Move to the next state

In s choose an action using the behaviour policy. For example, the

e-greedy method and the current values of Q{s,a). This gives you
a. Take the action a and move to next state, s'.

Step 3: Update Q{s,a)

Update your current Q{s,a) using the best action you could have
taken:

Q{s,a) Q{s,a)+ p(r + ^ma^Q{s',a")- Q(s,a)^

and
s ^ s'.

I've used a" to mean all the actions you could possibly take. The
action taken according to the behaviour policy would be a' which
204 CHAPTER 11. REINFORCEMENT LEARNING

does not appear in the updating.

In Figure 11.14 we see the algorithm in action. The actual action
taken at s', a', is faint because it doesn't appear in the updating.

o'
ibehaviour policy;
action a
Q{s,a)
state s collect r
^tate s',
S
^action a"
^(maximizes Q(s',a"))

Figure 11.14: The quantities used in the Q-learning algorithm.

Repeat Steps 2 and 3 until termination of the episode and then return
to Step 1 for a new episode.

11.22 Example: Blackjack

I’m going to be looking at Blackjack in detail throughout the rest of

this chapter. It's a good example, often found in reinforcement-learning text
books, because it is simple to understand, not trivial, but easy to code up.
First the rules of Blackjack as played in casinos.

1. There is one dealer/banker/house. In a casino the dealer will be stand

ing at the typically kidney-shaped Blackjack table

2. There are one or more players, seated around said table

3. Each player places their bet in front of them before any cards are dealt

4. The dealer deals one card face up to each player and one to themselves.
Another face-up card is then dealt to all players and another card, this
time face down (the hole card), to the dealer

5. The count of a hand is the sum of the values of al l the individual cards;
Aces are one or 11, 2-9 count as the number of pips, 10, Jack, Queen
and King count as ten

6. The goal of each player is to beat the dealer, i.e. get a higher count
than the dealer
11.22. EXAMPLE: BLACKJACK 205

7. Each player in turn gets to take (Hit) another card, an offer that is
repeated, or Stand

8. If a player goes over 21 they have bust and lose their bet immediately
9. If a player has an Ace which they are treating as 11 for the count and
then a hit takes them over 21 then they simply treat that Ace as a
value one going forward

10. A hand without any ace, or a hand in which aces have a value of one,
is called a hard hand. A hand in which there is an ace that is counted
as 11 is called a soft hand

11. If a player is dealt a natural meaning an Ace plus a 10-count card in

their first two cards then, unless the dealer also has a natural, they are
usually paid off at 3:2 immediately

12. A dealer's natural beats a player who has 21 with three or more cards

13. A player can double their bet before taking any cards, on condition
that they then only take one extra card

14. If a player is initially dealt two cards of the same value they can be
split to make two new hands. The player puts up another bet of the
same size and the dealer deals another card to each of the two hands

15. If the dealer's upcard is an Ace then he can offer insurance. This is a
side bet paying 2:1 if his hole card is a ten count

16. Once all players in turn have stood or gone bust it is the dealer's turn

17. The dealer in a casino does not have the flexibility in whether to hit
or stand that a player has. Typically they have to hit a count of 16 or
less and stand on 17 or more

18. If the player beats the dealer they get paid off at evens
19. If player and dealer have the same final count then it is a tie or push

Those are the basic rules. In some casinos these rules are fine tuned:

18. In some casinos a dealer with a count of 17 including an Ace will hit.
So the dealer will hit a soft 17

19. In some casinos the dealer's hole card isn't dealt until after all players
have played

20. In some casinos if a dealer has an upcard that is an Ace (sometimes

also a 10) then he will look at his hole card to see if he has a natural
206 CHAPTER 11. REINFORCEMENT LEARNING

You can see from the rules that the player has a lot of decisions to make:
Hit; Stand; Split; Take out insurance; Count Ace as one or 11; And how
much to bet. This is all to the player's advantage, they can optimize their
actions. The 3:2 payoff for a natural is also to the player's advantage. On
the other hand the dealer is constrained to always play the same way: Hit
16 or less; Stand on 17 or more. Yet despite all this asymmetry in favour of
the player the bank still has a significant edge. And that's because there is
one asymmetry in favour of the house: If the player goes bust they lose their
bet, even if the dealer goes bust later.

States and actions

We are to going to approach Blackjack as a problem in reinforcement

learning.
(But let me be clear, this is (probably) not how you would approach this
problem in practice. There is something special about the game of Blackjack
that is going to be totally ignored in the approach below. And that is how
from a hard card count of X and taking a card you can't go backwards to a
card count of less than or equal to X. A more natural approach is to figure
out what is best to do with a count of X, and once that has been done
look at a count of X - 1. That is, iteratively look at the optimal strategy
working backwards. Rather like how I explained O&iXs. If you only read one
thing from my further readings in this book then make it Ed Thorp's Beat
the Dealer.)
We are going to list the states of play, actions that we can take, and try
and find the optimal policy. Technically we could treat this as a dynamic
programming problem in a known environment because we could, if we had
the inclination, calculate the probabilities of transitions betweens states. But
that is rather messy, and defeats the point of the exercise.
I'm going to simplify things a little bit below. Just so we don't get bogged
down in the details. For example let's forget about doubling, splitting and
insurance. I'm also not going to worry about the player getting a natural
because at that point there is no decision to be made. I shall also have the
dealer hitting soft 17 which is common practice.

States; How complicated do you want to get? To make this a proper

MDP we need to know the following information: Our current count; Whether
we have at least one Ace; The dealer's upcard; How many Aces, 2-9s, and
10-count cards have been dealt from this shoe (and how many decks were
in the shoe to start with). That's a lot of information to keep track of. To
simplify things for now let's assume that the dealer is using an infinite num-
11.22. EXAMPLE: BLACKJACK 207

ber of decks. That means that the probability of dealing any particular card
never changes. The state is thus represented by our current count, whether
we have at least one Ace and the dealer's upcard.
How many states are there? The dealer could have A-10. We could have
12-21 (we would always hit 11 or less). Although we would never hit 21 we
need to know whether we have 21 because we might reach that state from
another state. And we may or may not have an Ace that's valued at one.
That makes a total of 10 x 10 x 2 = 200 states. That's fewer than Noughts
and Crosses.

Actions: Hit or stand. That's it.

Now let's look at the techniques I've described above.

Monte Carlo policy evaluation

Let's start with Monte Carlo policy evaluation.

To evaluate a policy we need a policy to evaluate. This is not about
finding the optimal policy.
An obvious player policy is to do just what the dealer would do: Hit 16
or less and soft 17s, otherwise stand.
To code this up you will need

1. An implementation of the player strategy

2. An implementation of the dealer strategy

3. An implementation of dealing cards

4. An implementation of hard and soft hands

5. Two grids of 10 x 10, one for the state-value function for hard hands
and one for soft hands

6. If you want to take arithmetic averages for finding the state-value

functions you will also need two similar grids to store the number of
episodes that have seen each state

7. Scissors and glue for cutting and pasting (only kidding!)

Let's play one game (an episode) and see what it looks like. We shall
follow the same policy as the dealer.
Suppose the dealer has a 7 and the player is dealt A, 2, 4, 6, 5, then
stands, and beats the dealer who goes bust. We can represent the state by
208 CHAPTER 11. REINFORCEMENT LEARNING

(Player Count, Hard or Soft, Dealer). So this episode is

(13,Soft,7) '■4“ (17, Soft, 7) 4° (13, Hard, 7)
(18, Hard, 7) Terminates with reward of 1.
There are no intermediate rewards and the final reward of 1 then gets
added to the state-value function, r;(Player Count, Hard or Soft, Dealer), for
each of those four states, changing the averages for those four states.
Note that we had to run the episode all the way to termination otherwise
the state-value function would miss out on the important, and here only,
reward, that at the end.
Each of

w(13,Soft,7), v(17,Soft,7), r;(13,Hard,7) and v(18,Hard,7)

will be updated, independently of each other and other states.
This is exactly as described in Section 11.16. There is almost never
any difference between first-visit and every-visit evaluation here because in
Blackjack it's uncommon to revisit a state. (You’d have to have two or more
Aces in your hand and then go over 21.)
In order to calculate the averages for the state-value function I chose to
run al l the simulations first before calculating the averages, rather than using
a learning-rate parameter.

Figure 11.15: The state-value, v{s), function for hard hands for the policy
in which the player fol lows the same rules as the dealer: MC.
 11.22. EXAMPLE: BLACKJACK 209

In Figure 11.15 is shown the state-value function for hard hands after a
few thousand episodes.
Please note that I am not claiming that these are the final results, or that
they are correct. Before going to a casino read the book mentioned at the
end of this chapter. By the way, if you follow the dealer's strategy you are
going to lose quite quickly. (Look at all the negative numbers in 11.15.)
Without knowing the transition probabilities, that is without knowing the
environment (as we are assuming), then we cannot use the information in
Figure 11.15 to tell us how to improve our policy from the one we have
adopted. For that we need the action-value function. That will come soon.

TD learning

Changing the code to implement TD for policy evaluation is quite sim

ple. The big difference between this and MC is that the updating takes
information from the next state in the episode.
The updating rule is

v{st) v{st)+ p {rt+x + jvist+i)- v{st)).

In our problem we have 7 = 1.
Instead of averaging all the rewards that each state gets, independently
of other states, we use the next reward and the next state. The next state is
a substitute for all the rewards that are to come (after the immediate one).
Let's see an example.
We shall follow the same policy as the dealer again. And we'll use the
same example as before: The dealer has a 7 and the player is dealt A, 2, 4,
6, 5 and beats the dealer. This is represented by the episode

(13,Soft,7) (17,Soft,7)’’4° (13, Hard,7)

y Q
—>• (18, Hard,7) Terminates with reward of 1.
Going through the updating slowly we do the following, in order:

w(13. Soft, 7)^ ^;(13, Soft,7)+ /?(0 + v{17, Soft,7)- v{U,Soft, 7)).
(The 'y(17,Soft,7) at this point is whatever it was for the previous episode.)
Then

v{l7, Soft,7)^ v{17, Soft,7)+ /3(0 + r;(13, Hard,7)- v{17,Soft, 7)).

(Now v(17, Soft,7) is being updated, but it's still the old ?;(13, Hard, 7).)
Then

w(13. Hard,7)^ w(13. Hard,7)+ /3(0 + v(18. Hard,7)- w(13. Hard,7)).

210 CHAPTER 11. REINFORCEMENT LEARNING

And finally

v{18, Hard,7)^ Hard,7)+ P {1 t;(18, Hard,7)).

The state-value function for hard hands wil l be similar to that shown in
Figure 11.15.
Now let’s move on to finding the optimal policy for Blackjack.

Sarsa

Finding the optimal policy using sarsa first involves going over to the
action-value function: Q(Player, Hard/Soft, Dealer, Hit/Stand).
The updating rule is now

Q{s,a) <- Q{s,a)+P(r + jQ{s',a') - Q(s,a)),

with 7 = 1 and the reward being zero until termination. And I have used an
e-greedy policy to determine which action to take in each state.
For example, suppose that the play is again

(13,Soft,7)'4° (17,Soft,7)''4’(13, Hard,7)

5-° (18, Hard,7) Terminates with reward of 1.
The updating would involve all of

g((13,Soft,7),Hit), g((13. Soft, 7), Stand), g((17.Soft, 7), Hit),

g((17. Soft, 7),Stand), g((13. Hard,7), Hit),
g((13. Hard,7), Stand), g((18. Hard,7), Hit)
and g((18. Hard,7), Stand).
(The multiple parentheses in the above are to emphasise the difference be
tween states and actions.) The Hits would be compared with the Stands
for each state to decide on the action. And the state-action pairs that were
realised would be updated.
The updating works as follows:

1. We start in state (13,Soft,7)

2. Then the value of g((13.Soft,7), Hit) versus g((13. Soft, 7), Stand)
tells us whether to hit or stand. Although with probability e we toss a
coin to make that decision. We decide to hit

3. We move to state (17,Soft, 7), collecting no reward

11.22. EXAMPLE: BLACKJACK 211

4. Q((17,Soft, 7), Hit) versus Q((17,Soft, 7),Stand) tells us whether to

hit or stand. Although with probability e we toss a coin to make that
decision. We decide to hit

5. Etc.

I have stopped at the etc. because already we have enough information

to start updating:

Q((13,Soft, 7), Hit)^ Q((13,Soft, 7), Hit)+

13 {0 + Q((17,Soft, 7), Hit) - Q((13,Soft, 7), Hit)).
This particular updating is illustrated in Figure 11.16. Each cell would
contain the current estimates for the action-value function Q, as a function
of the state and the action. There'd be a second grid, not shown, for the hard
hands. The dashed lines represent the order in which the game is played.
We go from a Soft 13 and Hit to a Soft 17 and another Hit. The second Hit
takes us onto the Hard grid, that's why I’ve drawn it leaving the soft grid.
The bent line represents the flow of information in the updating. We hit
when we held Soft 13, so Q((13,Soft, 7), Hit) is updated. And it is updated
using the action-value function where we went to, which was Soft 17, and
the action we took there, a Hit, i.e. <5((17, Soft, 7), Hit).

Deafer
T
2 3 4 5 6 7 8 9 10 A

S 12 H j S H ^ S ; H|S H j S : H 5|H ' S i H j S [hJ S ^ H S|H|S

O 13

F 14
QU ,
I
• I
T i 15
I
16
-f __4
17

18 ■f'
•f-
Stf
19
t 1*0.
20 [
21 I

Figure 11.16: One part of the updating for a hand of Blackjack: Sarsa.

And there would be similar updating for each of the other realised state-
action pairs. The final one would be

Q((18, Hard, 7), Stand) ^ Q((18, Hard, 7), Stand) +

/3(l-g((18, Hard, 7), Stand)) .
212 CHAPTER 11. REINFORCEMENT LEARNING

Hit or stand Dealer

2 3 4 5 6 7 8 9 10 A

12
[ H H H H H H H H H H

H 13 H S S s S H H H H H

A 14 S S S S s H H H H H

R 15 s S S S S H H H H H

D 16 S S S S S H H H H H

17 S s s s S s s s s s

18 S s s s s s s s s s

19 S s s s s s s s s s

20 S s s s s s s s s s

21 S s s s s s s s s s

Figure 11.17: Results from sarsa for hard hands. The recognised optimal
strategy for hitting given by the boxes.

Results for hard hands after a few million episodes are shown in Figure
11.17. My results are labelled as FI or S for Hit or Stand, and the Hits are
grey. All I've done here is take the Q function and label the cell according to
which is greater the Q for Hit or the Q for Stand in that state. The results
are homing in on the classical, recognised optimal strategy for hitting given
by the bordered boxes in that figure. I would need to run it for longer to
perfect convergence. You can find the classical results in any decent book
on Blackjack strategy.

Q learning

The updating rule is now

Q{s,a) c- Q{s,a)+ 13(r +j Q{s',a")- Q{s,a)j .

Use an e-greedy policy to determine which action to take in each state.
But note that the updating rule does not explicitly mention the action a'
taken at state s because we are looking over all possible actions in state s'.
Of course a' is in there implicitly because that’s how we got to state s'.
Suppose that the play is yet again

(13,Soft,7)’'4“ (17,Soft,7)4“(13, Hard,7)

’4'(18, Hard,7) Terminates with reward of 1.
11.22. EXAMPLE: BLACKJACK 213

The updating works as follows:

1. We start in state (13,Soft,7)

2. Then the value of Q((13,Soft,7), Hit) versus Q((13,Soft, 7), Stand)

tells us whether to hit or stand. Although with probability e we toss a
coin to make that decision. We decide to hit

3. We move to state (17,Soft, 7), collecting no reward

4. We take the maximum of Q{{17,Soft, 7), Hit) and

Q((17,Soft, 7),Stand), even though we might not take the maximizing
action

5. Etc.

And so:

g((13. Soft, 7), Hit) ^ g((13. Soft,7), Hit)+

/3(0 + max(g((17,Soft, 7), Hit), g((17. Soft, 7),Stand))

g((13,Soft,7),Hit)).

And there would be similar updating for each of the other realised state-
action pairs.
This particular updating is shown in Figure 11.18. This differs from the
equivalent Sarsa diagram, 11.16, in two ways:

• First, we don’t care whether our action in state Soft 17 is Hit or Stand,
hence the question marks in the diagram.

• Second, the information that gets passed to g((13. Soft, 7), Hit) is the
maximum over the possible action-value functions in state Soft 17.
214 CHAPTER 11. REINFORCEMENT LEARNING

Deafer

2 3 4 5 6 7 8 9 10 A

H i S i H S l H i S i H s i H S ' H S S ; H : S|H|S i H , S i H|S

S 12
1
0

F
13

14
ESii::r I
.1 T
T 15 i
16
■f
j t
i r
17 . 1 ake the max.
T
18

19
1
I
20

21
X
tfj
I
i
f

Figure 11.18: One part of the updating for a hand of Blackjack: Q learning.

How to win at Blackjack

I cannot leave you without giving a few clues as to how to actually win
at Blackjack.
We've seen how you can find the optimal policy with sarsa and Q learning.
But still that's not enough.
To win at Blackjack you need, as wel l as the optimum strategy,

• Finite number of decks (the fewer the better)

• Dealer deals sufficiently far down into the deck

• Player must keep track of extra variable(s)

With the infinite deck we've looked at so far the deck has no memory, the
probabilities never change. With a finite number of decks the probabilities
change as cards are drawn from the deck. We can take advantage of this by
keeping track of more than the player's count, soft/hard and dealer's card.
The memory, as in which cards have been dealt and thus which cards are
left in the shoe, is important because it changes the odds of winning. You
can see this easily by considering what happens if the shoe is rich in high-
count cards (because lots of twos, threes, . . . have been dealt). Remember
that the dealer has to hit 12-16. If the deck has many tens then that means
there is a higher chance than normal of him going bust.
So if, as the player, you have kept track of what has been dealt and you
find the deck starting to look favourable then you simply increase your bet
size.
11.23. LARGE STATE SPACES 215

No one expects you to keep track of everything that has been dealt.
So various simple techniques have been devised to help you track the most
important information. One way of doing this is to keep a running count in
your head of the following cards, Aces, twos,..., sixes, tens as follows.
With a fresh shoe the running count is zero. Every time you see a two-six
dealt (to other players, yourself, or to the dealer) you add 1 to that running
count. And for every Ace or ten you see you subtract one. So if a round goes
like this: an Ace, a two, two threes, a six, a nine, an eight, one Jack, one
Queen and a seven, then the running count would go -1,0,1,2,3,3,3,2,1,1,
ending up at plus one. I.e. the deck is now ever so slightly better than before.
And you keep this count going, only ever going back to zero for a new shoe.
If the running count gets large enough relative to the number of cards left
in the shoe then it's time to increase your bet size. (Of course then you get
thrown out of the casino because the dealer is probably also counting cards,
as it’s called, and now he knows you’re some kinda wise guy.)

Optimal betting

I said above that you have to adjust your bet, increasing it when the
deck is favourable and decreasing the bet when it isn't. Then you might be
able to win. There is some mathematics to this, going under the name of
Kelly criterion and Kelly fraction. There's a simple formula, you can derive
it on the back of an envelope. However since this isn’t really a book about
gambling I'm going to stop here, and direct you to the book listed at the
end of this chapter.

11.23 Large State Spaces

My examples have all had relatively small state spaces, and Just a small
choice of actions. More often than not in real problems you will be faced with
very large state spaces and/or actions. For example, you might be working
with a complex game such as chess. Often the state space will have infinite
dimensions, for example if there is a continuum of states. Or there will be a
continuum of actions.
We can approach such a problem in reinforcement learning by approxi
mating our value functions using simple or complicated functional forms with
a set of parameters. These parameters are the quantities that are updated
during the learning process.
We must convert the potentially infinite-dimensional states to finite
dimensional feature vectors. This process is called feature extraction.
216 CHAPTER 11. REINFORCEMENT LEARNING

How you best approximate the value functions is going to be problem

dependent. At the simplest level you might use a linear approximation. If
the problem is more complicated then perhaps approximate with a neural
network.

Further Reading

There is just one book you will need for reinforcement learning, the classic
and recently updated Reinforcement Learning: An Introduction by Richard
Sutton and Andrew G. Barto.
But the one book you must get, although it’s nothing about machine
learning, is Beat The Dealer by Ed Thorp, published by Random House. It's
a classic if you are into card games, or gambling, or finance, or to learn
about other methods for approaching the mathematics of Blackjack. Or just
because it’s a great read.
Datasets

I've used a variety of data sets in this book to demonstrate the various
methods. Some of these data sets are available to the public and some were
private. I only occasionally adjusted the data and then only to identify and
explain some idea as compactly as possible. I am not claiming that whatever
method I employed on a particular problem was the best for that problem.
But equally they were probably not the worst.
Good places to start looking for data are
https://toolbox.google.com/datasetsearch
and https://www.kaggle.com. You'll need to register with the latter.
Here are some datasets you might want to download to play around with.
Some I used, some I didn't.

Titanic

A very popular dataset for machine-learning problems is that for passen

gers on the Titanic, and in particular survivorship based on information such
as gender, class of ticket, etc. Go to
https://www.haggle.com/c/titanic
I decided not to use this data as I thought it a bit tasteless.

Irises

The famous iris dataset that I used can be found everywhere, for example
https://www.haggle.com/uciml/iris

217
218 DATASETS

Heights and weights

Ten thousand heights and weights of adults can be found here

https://www.haggle.com/mustafaali96/weight-height

MNIST

Seventy-thousand handwritten images of digits can be found at

http://yann.lecun.com/exdb/nmist/
Or rather matrix representations of the images written by employees of the
American Census Bureau and American high school students in CSV format.

House of Commons Voting

The site https://www.publicwhip.org.uk collects data for activity in

the UK's Houses of Commons and Lords. I used some of it for my SOM
analysis. Dig down to
https://www.publicwhip.org.uk/project/data.php
and https://www.publicwhip.org.uk/data/ to get at the raw data.

Mushrooms

Fancy predicting whether a mushroom is edible or poisonous? Well, pop

over to

https://www.kaggle.com/uciml/mushroom-classification
This contains data for over 8,000 mushrooms with information about their
geometry, colours, etc. and whether they are edible. Rather you than me.

Financial

Plenty of historical stock, index and exchange rate data can be found at
https://finamce.yahoo.com
For specific stocks go to
https://finance.yahoo.com/quote/[STOCKSYMBOLHERE]/history/
The Bank of England has economic indicators, interest rates etc. at
https://www.bankofengland.co.uk/statistics/
 219

Banknotes

Data for characteristics of banknotes can be found here

http://archive.ics.uci.edu/ml/datasets/banknote+
authentication

Animals

Data for characteristics of various animals can be found here

http://archive.ics.uci.edu/ml/datasets/zoo
Epilogue

I hope you enjoyed that as much as I did. My Job is over, so I can sit
down with that single malt. But for you the work is only Just beginning.
Please don't be a stranger, email me, paulOwilmott.com. Perhaps you’l l
find typos, or maybe even major mistakes. Or maybe you'll want clarification
of some technical point or have ideas for new material. Probably you'll end
up teaching me, and that’s how it should be. You can also sign up at
wilmott.com, it's free! There we can all get together to shoot the breeze,
and plan the fightback for when the machines take over.

P.S. I’ll leave the final words to The Machine, from which (whom?) I
have learned so much.

Final Words

Oh boy! It was a very enjoyable shoot. Thanks for reading! What did
you guys think of the new series? Was this a fun experience or did it take
some time to get started?
What makes a good leader? A dear vision and focus. Leadership is what
really matters — the vision of the future. It matters how you work it.
To truly understand how we're going to get there, we need to understand
how you are going to be able to execute on your vision. Don't let a lack of
leadership distract you from your true purpose. Your first task is to define
where you are in this new world, and then to make those changes.
The new world is the new vision.
It is not the future, but it's the future.

221
Index

Numbers in bold are pages with major entries about the subject.

e greedy, 181, 195, 200, 201, How to win at, 214

203, 210, 212 BMU, see Best Matching Unit
Bootstrap aggregation, 145
Action, 175, 184, 206 Branch, 128
Action-value function, 191, 200 Butterfly Effect, 14
Activation function, 151, 153
Algorithm, 58, 68, 93, 116, 136, CART, see Classification and
161, 201, 203 Regression Trees
AlphaGo, 5 Cellular automata, 12
Architecture of a neural network Chaos, 14
148, 157, 168 Character recognition, 162
Area under the ROC curve, 28 Classification, 18, 56
AUC, see Area under the ROC Classification and Regression
curve Trees, 127
Autoencoder, 168 Cluster, 65, 67
Complementary slackness, 45
Backpropagation, 159, 161, 164 Confusion matrix, 26
Bagging, see Bootstrap Constraints, 44
aggregation Control, 183
Barnsley Fern, 13 Cost function, 28, 92, 94, 157,
Batch gradient descent, 34 158
Bayes Theorem, 51, 84 Cross entropy, 32, 49, 158
Behaviour policy, 203 Curse of Dimensionality, 20, 74
Bellman equation, 188, 192, 193
199 Data points, 17
Bellman optimality equation, 193 Decision Tree, 8, 127
Best Matching Unit, 115 Deep Learning, 170
Bias, 37, 61. 150, 153 DeepMind, 5
Blackjack, 175, 176, 184, 194, Discount factor, 186
195, 200, 204, 206 Distance, 19, 115

223
224 INDEX

Chebyshev, 20 Interest rates, 77, 80

Cosine similarity, 20 Intra-cluster variance, 68
Euclidean, 19
Manhattan, 20 Jensen Interceptor Convertible, 2
Donald Michie, 4 Jump volatility, 76
Drivers, 9
K Means Clustering, 7, 65
Dual problem, 105 K Nearest Neighbours, 7, 55
Dynamic Programming, 184 Karush-Kuhn-Tucker, 45
Kelly criterion, 215
Eager learning, 58
Kelly fraction, 215
Elbow, 71
Kernel smoothing, 63
Entropy, 47, 134 Kernel trick, 108
Episode, 196 Known environment, 183
Epoch, 35, 164
Every-visit evaluation, 197 Lagrange multipliers, 44, 49, 98
104
False negative, 26 Lagrangian, 44
False positive, 26 Lamborghini 400 GT, 174
Feature extraction, 215 Lasso regression, 98
Feature map, 109 Lazy, 58
Feature vector, 17, 215 Leaf, 128
Features, 17 Learning factor, 33
Feedforward, 148 Learning rate, 118
First-visit evaluation, 196 Linear function, 155
Fractals, 13 Linear regression, 92, 142
Local linear regression, 63
Gain ratio, 137 Logistic function, 156
Gaussian kernel, 63 Logistic map, 14
GDP, 80 Logistic regression, 93, 94
Gini Impurity, 137 Long short-term memory, 169
Gradient descent, 33, 92, 93, Looking ahead, 144
107, 155, 159, 198 Loss function, see Cost function
Greedy, 200 Lyapunov Exponent, 14

Hard limit, 155 Margins

Hard margins, 100 Hard, 100
Hidden layer, 148 Soft, 106
Hinge loss, 107 Markov chain, 186
Markov Decision Process, 176,
Industrial mathematics, 1 183

Inflation, 77, 80 Episodic, 196

Information gain, 134 Mathematical models, 1, 9, 11
Information Theory, 47, 133 Maximum Likelihood Estimation,
Input layer, 148 22, 31, 85, 95
 225

Maze, 186 Policy, 179, 185

MDP, see Markov Decision Policy evaluation, 183
Process Polynomial regression, 97
MENACE, 5 Positive linear function, 155
Minimax problem, 178 Primal problem, 102
MLE, see Maximum Likelihood Principal Components Analysis,
Estimation 21, 75, 168
MNIST dataset, 162 Propagation, 152
Modern Portfolio Theory, 119 Pruning, 137
Momentum, 34 Punishment, 175
Monte Carlo policy evaluation, Pure, 131
195, 207
Multi-armed bandit, 180, 191, Q Learning, 203, 212
194

Multiple classes, 45 Radial basis functions, 169

Ramones, 53
Naive Bayes Classifier, 7, 83 Random Forests, 145
Natural Language Processing, 20, Receiver operating characteristic
50, 83 27

Neural Network, 8, 113, 147 Recurrent neural network, 169

NLP, see Natural Language Regression, 8, 18, 29, 55, 62, 91
Processing 139
Node, 128 Regularization, 32, 97, 107, 158
Notation, 17 Reinforcement Learning, 5, 7,
Noughts and Crosses, 4, 176 173
Null error rate, 27 ReLU function, 155
Return, 185
Off policy, 202 Reward, 175, 184
Offline updating, 199 Ridge regression, 97
OLS, see Ordinary least squares ROC (curve), 27
On policy, 202 Root, 128
One versus all, 46
One versus one, 46 S&P500 Index, 119
One-hot encoding, 18, 46 Sarsa, 201, 210
Online updating, 199 Saturating linear function, 155
Optimal betting, 215 Scaling, 18, 58, 59, 68, 79, 114
Optimal policy, 183, 193, 200 Scree plots, 71
Ordinary least squares, 29, 157, Self-Organizing Map, 8, 113
158 Shoes, 15
Output layer, 148 Sigmoid function, 94, 156
Overfitting, 35, 39, 137, 165 Soft margins, 106
Softmax function, 46, 156
PCA, see Principal Components State, 176, 206
Analysis State-value function, 190, 192
226 INDEX

Step function, 155 True positive, 26

Stochastic gradient descent, 34,
164 Underfitting, 39
Supervised Learning, 6, 55, 83, Universal Approximation
91, 99, 127, 147 Theorem, 149, 157
170
Support Vector Machine, 8, 99
Surprisal, 47 Unsupervised Learning, 6, 65,
113, 147
Tanh function, 156
TD error, 199 Validation, 35
TD target, 199 Value function, 179
Temporal Difference Learning, Variables, 9
198, 209 Variance, 37, 61
Testing, 35, 164 Volatility, 75
The Game of Life, 12 Voronoi diagram, 79, 81
Training, 35, 164
Transition probabilities, 77, 185 Weight, 114, 150, 153
True negative, 26 Word2vec, 20, 51
 Made in the USA
Middletown, DE
28 February 2020

85483948R00135
MACHINE LEARNING; AN APPUED MATHEMATICS INTRODUCTION
Machine Learning:An Applied Mathematics Introduction covers the essential mathematics
behind all of the following topics

•/(Nearest Neighbours • Regression Methods • Decision Trees

•KMeans Clustering •Support Vector Machines • Neural Networks
• Naive Bayes Classifier •Self-Organizing Maps • Reinforcement Learning

PauiWilmott brings three decades of experience in mathematics education,

and his inimitable style, to the hottest of subjects.This book is an accessible
introduction for anyone who wants to understand the foundations but also
wants to"get to the meat without having to eat too many vegetables."

> Paul Wilmott has been called "cult derivatives lecturer" by the Financial Times
and "financial mathematics guru"by the BBC.

ISBN
5 1999

781916 081604