Machine Learning

A Probabilistic Approach

David Barber
http://www.idiap.ch/∼barber
c David Barber 2001, 2002,2003,2004,2006

Contents

1 Introduction 2
1.1 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Unsupervised Learning . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Supervised Learning . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Supervised Learning Approaches . . . . . . . . . . . . . . . . . . . 4

I Machine Learning : More Traditional Approaches 8

2 Generalisation 9
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.1 Supervised Learning . . . . . . . . . . . . . . . . . . . . . . 9
2.1.2 Training Error . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.3 Test Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.4 Validation Data . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.5 Dodgy Joe and Lucky Jim . . . . . . . . . . . . . . . . . . . 11
2.1.6 Regularisation . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3 Nearest Neighbour Classification 15

3.1 Nearest Neighbour . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1.1 Problems with Nearest Neighbours . . . . . . . . . . . . . . 17
3.2 K Nearest Neighbours . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3 Handwritten digit Example . . . . . . . . . . . . . . . . . . . . . . 18
3.4 A Probabilistic Intepretation . . . . . . . . . . . . . . . . . . . . . 19

1
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 2

4 Linear Dimension Reduction 21

4.1 Principal Components Analysis . . . . . . . . . . . . . . . . . . . . 21
4.1.1 Example : Reducing the dimension of digits . . . . . . . . . 24
4.1.2 PCA and Nearest Neighbours . . . . . . . . . . . . . . . . . 24
4.1.3 Mega Dimensional Data . . . . . . . . . . . . . . . . . . . . 25
4.1.4 PCA is good because it is a poor compressor! . . . . . . . . 25
4.2 Deriving the Optimal Linear Reconstruction . . . . . . . . . . . . . 26
4.3 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.4 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5 Linear Discriminant Analysis 32

5.1 Unsupervised Dimension Reduction . . . . . . . . . . . . . . . . . . 32
5.1.1 Using PCA for visualisation . . . . . . . . . . . . . . . . . . 32
5.2 Fishers Linear Discriminant . . . . . . . . . . . . . . . . . . . . . . 32
5.2.1 One dimensional projection . . . . . . . . . . . . . . . . . . 33
5.3 Canonical Variates . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.3.1 Using Canonical variates on the Digits Data . . . . . . . . . 35

6 Linear Parameter Models 36

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.1.1 Regression and PCA . . . . . . . . . . . . . . . . . . . . . . 37
6.2 Linear Parameter Models (Generalised Linear Models) . . . . . . . 37
6.2.1 Training LPMs . . . . . . . . . . . . . . . . . . . . . . . . . 38
6.2.2 Regularisation and numerical stability . . . . . . . . . . . . 39
6.2.3 Higher Dimensional Outputs . . . . . . . . . . . . . . . . . 39
6.2.4 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.3 Radial Basis Functions . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.4 The curse of Dimensionality . . . . . . . . . . . . . . . . . . . . . . 40
6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

7 Layered Neural Networks 42

7.1 Sequential Layered Processing . . . . . . . . . . . . . . . . . . . . . 42
7.2 The Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
7.3 Multilayer Perceptrons . . . . . . . . . . . . . . . . . . . . . . . . . 43
7.3.1 Understanding Neural Networks . . . . . . . . . . . . . . . 44
7.4 Training multi-layered perceptrons . . . . . . . . . . . . . . . . . . 46
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 3

7.4.1 Single Hidden Layer . . . . . . . . . . . . . . . . . . . . . . 47

7.4.2 Back Propagation . . . . . . . . . . . . . . . . . . . . . . . 48
7.4.3 Training ensembles of networks . . . . . . . . . . . . . . . . 48
7.5 Adaptive Basis Function Networks . . . . . . . . . . . . . . . . . . 49
7.5.1 Adaptive Basis Functions . . . . . . . . . . . . . . . . . . . 49
7.6 Training Adaptive Basis Functions . . . . . . . . . . . . . . . . . . 50
7.6.1 Non-local Basis Functions . . . . . . . . . . . . . . . . . . . 51
7.7 Committees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.8 Summary and Outlook . . . . . . . . . . . . . . . . . . . . . . . . . 53

8 Autoencoders 54
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
8.1.1 Linear Dimension Reduction (PCA) . . . . . . . . . . . . . 54
8.1.2 Manifolds : The need for non-linearity . . . . . . . . . . . . 54
8.2 Non-linear Dimension Reduction . . . . . . . . . . . . . . . . . . . 55
8.2.1 Training Autoencoders . . . . . . . . . . . . . . . . . . . . . 55
8.3 Uses of Autoencoders . . . . . . . . . . . . . . . . . . . . . . . . . 56
8.3.1 A Visualisation example . . . . . . . . . . . . . . . . . . . . 57

9 Data Visualisation 58
9.1 Classical Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
9.1.1 Finding the optimal points . . . . . . . . . . . . . . . . . . 59
9.1.2 The Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 60
9.2 Sammon Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
9.3 A word of warning . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

II Inference and Learning in Probabilistic Models 63

10 Introducing Graphical Models 64

10.1 Belief Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
10.1.1 Tracey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
10.2 A word on notation . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
10.3 Example : Was it the Burglar? . . . . . . . . . . . . . . . . . . . . 67
10.4 Belief Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
10.4.1 Conditional Independence . . . . . . . . . . . . . . . . . . . 70
10.4.2 Intuition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 4

10.4.3 d-Separation . . . . . . . . . . . . . . . . . . . . . . . . . . 71
10.5 Graphical Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
10.5.1 Markov Random Fields . . . . . . . . . . . . . . . . . . . . 75
10.5.2 Expressiveness of Graphical Models . . . . . . . . . . . . . 76
10.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

11 Inference in Belief Networks 80

11.1 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
11.2 Variable Elimination in a simple chain . . . . . . . . . . . . . . . . 80
11.3 Bucket Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
11.4 Belief Propagtion : Inference in Singly Connected Graphs . . . . . 84
11.4.1 Undirected Belief Propagation . . . . . . . . . . . . . . . . 85
11.4.2 Directed Belief Propagation . . . . . . . . . . . . . . . . . . 86
11.4.3 Example : Directed Belief Propagation . . . . . . . . . . . . 89
11.5 Belief Revision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
11.6 The Generalised Distributive Law . . . . . . . . . . . . . . . . . . . 92
11.7 Inference in Multiply-Connected Graphs . . . . . . . . . . . . . . . 93
11.7.1 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . 94
11.8 Cluster Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
11.9 KL divergence approach to marginalisation on Trees . . . . . . . . 97
11.10Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
11.11Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

12 The Junction Tree Algorithm 107

12.1 Absorption and Marginal Consistency . . . . . . . . . . . . . . . . 107
12.2 Junction Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
12.3 Constructing Junction Trees for Singly-Connected Distributions . . 111
12.4 Junction Trees for Multiply-Connected Distributions . . . . . . . . 113
12.5 Triangulation Algorithms . . . . . . . . . . . . . . . . . . . . . . . 115
12.6 Finding a JT from a Triangulated Graph . . . . . . . . . . . . . . 116
12.7 The Junction Tree Algorithm . . . . . . . . . . . . . . . . . . . . . 116
12.7.1 Finding the Most Likely State . . . . . . . . . . . . . . . . 117
12.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
12.9 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 5

13 Variational Learning and EM 121

13.1 Maximum Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . 121
13.1.1 Missing Data/Variables . . . . . . . . . . . . . . . . . . . . 124
13.2 Variational Learning and Expectation Maximisation . . . . . . . . 125
13.3 Optimising the Likelihood by Gradient methods . . . . . . . . . . 133
13.4 Iterated Proportional Fitting . . . . . . . . . . . . . . . . . . . . . 134
13.5 Bayesian Methods and ML-II . . . . . . . . . . . . . . . . . . . . . 135
13.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
13.7 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

III Probabilistic models in Machine Learning 139

14 Introduction to Bayesian Methods 140

14.1 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
14.2 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

15 Bayesian Regression 152

15.1 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
15.2 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

16 Logistic Regression 156

16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
16.1.1 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
16.1.2 Gradient Ascent . . . . . . . . . . . . . . . . . . . . . . . . 159
16.1.3 Avoiding Overconfident Classification . . . . . . . . . . . . 162
16.1.4 Logistic Regression and PCA ? . . . . . . . . . . . . . . . . 162
16.1.5 An Example : Classifying Handwritten Digits . . . . . . . . 163
16.2 The Kernel Trick . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
16.3 Mixture Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
16.3.1 Mixtures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
16.3.2 Mixture of Experts . . . . . . . . . . . . . . . . . . . . . . . 166
16.3.3 A ‘Bayesian’ approach to setting the regularisation parameter167
16.3.4 Evidence Procedure . . . . . . . . . . . . . . . . . . . . . . 168
16.4 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
16.5 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 6

17 Naive Bayes 173

17.1 Why Naive Bayes? . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
17.2 Understanding Conditional Independence . . . . . . . . . . . . . . 173
17.3 Are they Scottish? . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
17.3.1 Further Issues . . . . . . . . . . . . . . . . . . . . . . . . . . 175
17.3.2 Text Classification . . . . . . . . . . . . . . . . . . . . . . . 176
17.4 Pitfalls with Naive Bayes . . . . . . . . . . . . . . . . . . . . . . . 176
17.5 Estimation using Maximum Likelihood : Bernoulli Process . . . . . 177
17.5.1 Classification Boundary . . . . . . . . . . . . . . . . . . . . 178
17.6 Naive Bayes : The multinomial case . . . . . . . . . . . . . . . . . 178
17.6.1 Dirichlet Prior . . . . . . . . . . . . . . . . . . . . . . . . . 179
17.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
17.8 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

18 Mixture Models : Discrete Hidden Variables 183

18.1 Mixture Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
18.2 Gaussian Mixture Models . . . . . . . . . . . . . . . . . . . . . . . 185
18.3 K Means . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
18.3.1 The algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 190
18.3.2 Uses of K Means . . . . . . . . . . . . . . . . . . . . . . . . 191
18.4 Classification using Mixture Models . . . . . . . . . . . . . . . . . 191
18.5 Mixture of Multi-nomials . . . . . . . . . . . . . . . . . . . . . . . 192
18.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
18.7 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

19 Factor Analysis and PPCA 197

19.1 Linear Subspace Methods . . . . . . . . . . . . . . . . . . . . . . . 197
19.2 A Toy Comparision of FA and PPCA . . . . . . . . . . . . . . . . 202
19.3 Non-linear Subspace Methods . . . . . . . . . . . . . . . . . . . . . 204
19.3.1 Non linear Factor Analysis . . . . . . . . . . . . . . . . . . 204
19.4 Probabilistic PCA . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
19.5 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
19.6 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 7

20 Dynamic Bayesian Networks : Discrete Hidden Variables 210

20.1 The Importance of Time . . . . . . . . . . . . . . . . . . . . . . . . 210
20.1.1 Parallel and Sequential Inference . . . . . . . . . . . . . . . 214
20.1.2 Rauch Tung Striebel and the α − γ recursions . . . . . . . . 217
20.1.3 Viterbi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
20.2 Applications of HMMs . . . . . . . . . . . . . . . . . . . . . . . . . 223
20.3 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
20.4 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

21 Dynamic Continuous Hiddens : Linear Dynamical Systems 229

21.1 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
21.1.1 The Forward Pass : The Kalman Filter . . . . . . . . . . . 230
21.1.2 The Kalman Smoother : The Rauch-Tung-Striebel Smoother 231
21.1.3 The Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . 233
21.2 EM Algorithm for Learning . . . . . . . . . . . . . . . . . . . . . . 233
21.3 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
21.4 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236

22 Switching Linear Dynamical Systems 237

22.1 Expectation Correction . . . . . . . . . . . . . . . . . . . . . . . . 238
22.1.1 Forward Pass (Filtering) . . . . . . . . . . . . . . . . . . . . 238
22.1.2 Collapsing Gaussians . . . . . . . . . . . . . . . . . . . . . . 240
22.1.3 Backward Pass (Smoothing) . . . . . . . . . . . . . . . . . . 241
22.1.4 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
22.1.5 Using Mixtures in the Backward Pass . . . . . . . . . . . . 245
22.2 Relation to other methods . . . . . . . . . . . . . . . . . . . . . . . 246

23 Gaussian Processes 250

23.1 The Bayesian approach to Regression . . . . . . . . . . . . . . . . . 250
23.1.1 Parameterised Models . . . . . . . . . . . . . . . . . . . . . 251
23.1.2 Making Predictions . . . . . . . . . . . . . . . . . . . . . . . 251
23.1.3 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . 251
23.2 Generalised Linear Models . . . . . . . . . . . . . . . . . . . . . . . 251
23.2.1 Understanding the Prior . . . . . . . . . . . . . . . . . . . . 252
23.2.2 Sampling the function space prior . . . . . . . . . . . . . . 252
23.2.3 Understanding the Posterior . . . . . . . . . . . . . . . . . . 253
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 8

23.2.4 Understanding Model Selection Issues . . . . . . . . . . . . 253

23.3 Gaussian Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
23.3.1 Specifying the Prior . . . . . . . . . . . . . . . . . . . . . . 254
23.3.2 Making Predictions . . . . . . . . . . . . . . . . . . . . . . . 255
23.3.3 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . 256
23.3.4 Classification problems . . . . . . . . . . . . . . . . . . . . . 256
23.4 Gaussian Processes for Classification . . . . . . . . . . . . . . . . . 257
23.4.1 Maximizing P (y∗ , y|t) . . . . . . . . . . . . . . . . . . . . . 258
23.4.2 Parameterizing the covariance function . . . . . . . . . . . . 259
23.4.3 Integration over the hyperparameters . . . . . . . . . . . . 259
23.5 Multiple classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
23.5.1 Finding the mode of the distribution . . . . . . . . . . . . . 261
23.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

IV Approximate Inference Methods 264

24 Sampling 265
24.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
24.2 Markov Chain Monte Carlo (MCMC) . . . . . . . . . . . . . . . . 269

A Basic Concepts in Probability 283

A.1 What does random mean? . . . . . . . . . . . . . . . . . . . . . . . 283
A.2 What is Probability? . . . . . . . . . . . . . . . . . . . . . . . . . . 283
A.3 Rules of Probability . . . . . . . . . . . . . . . . . . . . . . . . . . 284

B Graph Terminology: 286

B.1 Graphs : Basic Concepts and Definitions . . . . . . . . . . . . . . . 286
B.2 Basic Concepts and Definitions . . . . . . . . . . . . . . . . . . . . 286
B.2.1 Undirected Graphs . . . . . . . . . . . . . . . . . . . . . . . 287
B.2.2 Directed Graphs . . . . . . . . . . . . . . . . . . . . . . . . 289

C Some Standard Distributions: 291

D Bounds on Convex Functions 292

D.1 Kullback Leibler Divergence KL(q||p) . . . . . . . . . . . . . . . . 292
D.1.1 Jensen vs Variational Jensen . . . . . . . . . . . . . . . . . 293
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 9

E Positive Definite Matrices and Kernel Functions: 294

F Approximating Integrals: 298

G Inference with Gaussian Random Variables 299

G.1 Gaussian Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . 300
G.2 Gaussian Propagation . . . . . . . . . . . . . . . . . . . . . . . . . 300
 1 Introduction

There are many motivations as to why one might want a machine to “learn” from
data. By “learn”, I have in mind applications that would be hard to program in a
traditional manner, such as the task of face recognition. Formally specifying why
you recognise a collection of images as examples of John’s face may be extremely
difficult. An alternative is to give examples of John’s face and let a machine “learn”
– based on the statistics of the data – what it is that differentiates John’s face from
other faces in the database. That is not to say that all information can be learned
solely on the basis of large databases – prior information about the domain is often
crucial to the successful application of machine learning.
The connection between probability and machine learning stems from the idea
that probabilistic models enable us to form a compact description of complex
phenomena underlying the generation of the data. Graphical models are simply
ways of depicting the independence assumptions behind a probabilistic model.
They are useful in modelling since they provide an elegant framework to therefore
graphically express basic independence assumptions about the processes generating
the data. This is useful since the calculus of probability will transfer to graph
theoretic operations and algorithms, many of which have deep roots in computer
science and related areas[1].
This book is intended as a (non-rigorous) introduction to machine learning, prob-
abilistic graphical models and their applications. Formal proofs of theorems are
generally omitted. These notes are formed from the basis of lectures given to both
undergraduate and graduate students at Aston University, Edinburgh University,
and EPF Lausanne.

1.1 Machine Learning

Machine learning is traditionally split into two main areas: supervised, and un-
supervised learning. The difference between the two depends on what kind of
question you wish the data to try to answer (and possibly on the data available).

1.1.1 Unsupervised Learning

A baby processes a mass of initially confusing sensory data. After a while the
baby begins to understand her environment in the sense that novel sensory data
from the same environment is familiar or expected. When a strange face presents
itself, the baby recognises that this is not familiar and may be upset. The baby
has learned a representation of the familiar and can distinguish the expected from
the unexpected, without an explicit supervisor to say what is right or wrong. Un-
Clustering supervised learning just addresses how to model an environment.Clustering is an
example of unsupervised learning, whereby the aim is to partition a set of data
into clusters. For example, given a set of questionnaire responses, find clusters
whereby in each cluster the responses are ‘similar’. This area is also sometimes
Descriptive modelling called descriptive modelling, where we just wish to fit a model which describes
succinctly and accurately the data in the database.

10
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 11

60

50

40

30

20

10

−10
−10 0 10 20 30 40 50

Figure 1.1: In unsupervised learning, we wish to make a compact description of

the data. Here, a natural description would be that the data lies in two clusters.

For example, here are some points:

x1 -2 -6 -1 11 -1 46 33 42 32 45
x2 7 22 1 1 -8 52 40 33 54 39
This is an example of unlabelled data. We can visualise this data by plotting it
in 2 dimensions: By simply eye-balling the data, we can see that there are two
apparent clusters here, one centred around (0,0) and the other around (35,35). A
reasonable model to describe this data might therefore be to describe it as two
clusters, centred at (0,0) and (35,35), each with a variance (spread) of around 1.

1.1.2 Supervised Learning

I’m fond of the following story :

“A father decides to teach his young son what a sports car is. Finding it difficult
to explain in words, he decides to try to explain by examples. They stand on a
motorway bridge and, as each car passes underneath, the father cries out ‘that’s
a sports car!’ when a sports car passes by. After ten minutes, the father asks his
son if he’s got it. The son says, ‘sure, it’s easy’. An old red VW Beetle passes by,
and the son shouts – ‘that’s a sports car!’. Dejected, the father asks – ‘why do you
say that?’. ‘Because all sports cars are red!’, replies the son.”

This story is an example of supervised learning. Here the father is the supervisor,
and his son is the ‘learner’, or ‘machine learner’ or ‘predictor’. The nice point
about this story is that you can’t expect miracles – unless you explicitly give ex-
tra information, learning from examples may not always give you what you might
hope for. On the other hand, if they had been there the whole week, probably the
son would have learned a reasonably good model of a sports car, and helpful hints
by the father would be less important. It’s also indicative of the kinds of problems
typically encountered in machine learning in that it is not really clear anyway what
a sports car is – if we knew that, then we wouldn’t need to go through the process
of learning!

Predictive modelling We typically have a training set of labelled data, for example, here are some data
nationality British Dutch Taiwanese British
height(cm) 175 195 155 165
sex m m f f
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 12

We might have a large database of such entries. A supervised learning problem

might be: given a new, previously unseen (nationality,height) pair, predict the
sex of the person. For example, given that a person is Taiwanese and 167cm tall,
are they going to be male or female? In this case we see the training data as a
collection of (input,output) pairs, where the output or label has been given by a
‘supervisor’. Ultimately, we wish to form a mapping from the inputs to the output
(possibly more than one output) that accurately describes the label/output given
the inputs. Ideally, we would like our model to generalise well (predict accurately)
novel test data not seen previously during the model building process.
Uncertainty This is a good example to motivate our later ideas about probability/uncertainty –
there is clearly not going to be absolute certainty about our predictions in this case
since there are always going to be tall females and shorter males that will make
classifying a novel person an inexact science. However, we may be able to infer
what is the probability that a novel person is male, given our trained model. In
practice, uncertainty often plays a major role in machine learning, and we need to
use a framework that can handle this. Uncertainty is not just an issue in supervised
learning. Also we may be uncertain as to the exact values in an unsupervised set
of data, and we may wish to take this into account in building a model.
Supervised learning problems traditionally come in two flavours, classification and
regression.

Classification Given a set of inputs, predict the class (one of a finite number of discrete labels).
Normally, the class is ordinal (there is no intrinsic information in the class la-
bel). For example, given an image of a handwritten digit, predict whether it is
0,1,2,3,4,5,6,7,8 or 9. This would be a 10-class classification problem. Many prob-
lems involve binary classes (you can always convert a multi-class problem into a
set of binary class problems – though this is not always natural or desirable). For
binary classes, there is usually no information as to whether we say the data are
labelled as class 0 or 1, or alternatively as class 1 or 2. For example, the sports-car
classification problem would have been the same if the father said ‘1’ or ‘0’ when
the car passing by was a sports car or not. A great deal of problems in the ma-
chine learning arena are classification problems. Uncertainty will ultimately play
a key role in any real world application. Can we really say that Mr Smith will
definitely default on his loan? This may seem a very strong statement if there is
little obvious difference between the attributes of Mr Smith and Mr Brown.
Regression Given a set of inputs, predict the output (a continuous value). For example, given
historical stock market data, predict the course of the FTSE for tomorrow.
Reinforcement Learning Reinforcement learning is a kind of supervised learning in which the supervisor
provides rewards for actions which improve a situation and penalties for deleterious
actions.

1.2 Supervised Learning Approaches

Consider trying to make a system that can distinguish between male and female
faces. Each image is represented as a real-valued vector x, formed by concatenating
the pixels of the image into a long vector. We have a database of P images,
xµ , µ = 1, . . . , P ), along with a label cµ ∈ {0, 1} stating if the image is a male
(c = 0) or female (c = 1) face. We wish to make a model of the image,sex
environment, p(x, c).
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 13

f
m f
m f
f
m m
m m f f
f
m f f
f x*

Figure 1.2: Here, each point in this space represents a high dimensional vector
x, which has an associated class label, either Male or Female. The point x∗ is
a new point for which we would like to predict whether this should be male or
female. In the generative approach, a Male model would be produced which would
ideally generate data which would be similar to the ‘m’ points. Similarly, another
model, the Female model should ideally generate points that are similar to the
‘f’ points above. We then use Bayes’ rule to calculate the probability p(male|x∗ )
using the two fitted models, as given in the text. In the discriminative case, we
directly make a model of p(male|x∗ ), which cares less about how the points ‘m’ or
‘f’ are distributed, but more about whether or not there is a simple description of
a boundary which can separate the two classes, as given by the line.

Generative Approach

In a generative approach, we define a model for generating data v belonging to

a particular class c ∈ 1, . . . , C in terms of a distribution p(v|c). Here, v will
correspond, say to the image vector. The class c will be say male or female.
For each class c we train a separate model p(v|c) with associated parameters Θc
by maximising the likelihood of the observed signals for that class. We then use
Bayes’ rule to assign a novel point v ∗ to a certain class c according to

p(v ∗ |c)p(c)
p(c|v ∗ ) = . (1.2.1)
p(v ∗ )

That model c with the highest posterior probability p(c|v ∗ ) is designated the pre-
dicted class.
Advantages : In general, the potential attraction of a generative approach is that prior
information about the structure of the data is often most naturally specified
through the generative model p(v|c).
Disadvantages : A potential disadvantage of the generative approach is that it does not di-
rectly target the central issue which is to make a good classifier. That is, the
goal of generative training is to model the observation data v as accurately
as possible, and not to model the class distribution. If the data v is complex,
or high-dimensional, it may be that finding a suitable generative data model
is a difficult task. Furthermore, since each generative model is separately
trained for each class, there is no competition amongst the models to explain
the data. In particular, if each class model is quite poor, there may be little
confidence in the reliability of the prediction. In other words, training does
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 14

not focus explicitly on the differences between mental tasks, but rather on
accurately modelling the data distributions from each associated class.

Discriminative Approach

In a discriminative probabilistic approach we define a single model p(c|v) common

to all classes. The parameters Θ of this model are trained to maximise the prob-
ability of the class label c. This is in contrast to the generative approach above
which models the data, and not the class. Given novel data v ∗ , we then directly
calculate the probabilities p(c|v ∗ ) for each class c, and assign v ∗ to the class with
the highest probability.
Advantages : A clear potential advantage of this discriminative approach is that it directly
addresses the issue we are interested in solving, namely making a classifier.
We are here therefore modelling the discrimination boundary, as opposed to
the data distribution in the generative approach. Whilst the data from each
class may be distributed in a complex way, it could be that the discrimination
boundary between then is relatively easy to model.
Disadvantages : A potential drawback of the discriminative approach is that they are usually
trained as ‘black-box’ classifiers, with little prior knowledge of how the signal
is formed built into the model.
In principle, one could use a generative description, p(v|c), building in prior
information, and use this to form a joint distribution p(v, c), from which a
discriminative model p(c|v) may be formed, using Bayes rule. Subsequently
the parameters Θc , c = 1, . . . C for this model could be found by maximising
the discriminative class probability. This approach is rarely taken in the
machine learning literature since the resulting functional form of p(c|v) is
often complex and training is difficult.

What are the basic assumptions of machine learning?

Arguably all machine learning approaches are based on some notion of smooth-
ness or regularity underlying the mechanism that generated the observed data.
Roughly speaking : if two datapoints are close neighbours, they are likely to be-
have similarly.
The general procedure will be to postulate some model and then adjust it’s para-
meters to best fit the data. For example in a regression problem, we may think
that the data {(xµ , y µ ), µ = 1, . . . , P }, where x is an input and y an output, is well
modelled by the function y = wx, and our task is to find an appropriate setting
of the parameter w. An obvious way to do this is to see how well the current
model predicts the training data that we have, and then to adjust the parameter
w to minimise the errors that our model makes on predicting the data. This gen-
eral procedure will usually involve therefore optimisation methods, usually in high
dimensional spaces (although the above is a one-dimensional example).
Noise, overfitting and In the case that there is noise on the data (sometimes, the father might be in-
Generalisation consistent in his labelling of sports cars, or there might be essentially random
perturbations on the FTSE index), we don’t want to model this noise. That is,
we have to be careful to make sure that our models only capture the underlying
process that we are truly interested in, and not necessarily the exact details of the
training data. If we have an extremely flexible model, it may overfit noisy training
 c David Barber 2001,2002,2003,2004,2006
Machine Learning : A probabilistic approach : 15

data be a very poor predictor of future novel inputs (that is, it will generalise
poorly). This is very important topic and central to machine learning. We shall
return to this in a later chapter.
I. Machine Learning : More Traditional Approaches

16
 2 Generalisation

2.1 Introduction
One major goal in supervised learning is, on the basis of labelled training data, to
encapsulate the underlying mechanism which generated the data, thereby learning
a model with predictive power. That is, given a novel unlabelled instance, to make
an accurate prediction.

2.1.1 Supervised Learning

Formally, supervised learning consists of the following scenario: A given set of

training data, Dtrain = {(xµ , tµ ), µ = 1, . . . , P } where each xµ is a vector of (in
general real valued) attributes, and tµ is the associated target for the µth pattern.
(In the binary classification tasks we have been considering, tµ ∈ {0, 1}). If tµ can
take on one of only a finite set of values, we call this a classification problem. If
tµ can be any value, we call this a regression problem.
Our basic paradigm is the following : There is some underlying process which
generates “clean” data. The data is then possibly corrupted by noise to give the
actual data that we observe. The aim in supervised learning is to try to recover
the underlying clean data generating mechanism.
Prediction without Consider the following problem. What is the next number in the sequence 1,2,3,4,5,
assumptions is meaningless ?1 Well, there is no “correct” answer. I could predict anything, and who’s to say
that I’m incorrect? It may well be perfectly reasonable to a Martian to say the
next number is 78. In fact, the “answer” that I was looking for was 63. This is
the number of the bus that follows busses 1,2,3,4 and 5 in my home town. “Not
fair!”, you might say, “we didn’t know that you were talking about busses in your
home town”. Well, that’s what learning from data is about – you have to try to
collate as much information about the data domain as you can, and hope that
your assumptions are reasonable. Whatever you do, your predictions are only as
good as your assumptions.
Consider the training data in fig(2.1). This is an example of a regression problem
in which the targets t are real values. In this case, the inputs x are also real values.
Our aim is to fit a function f (x|θ) to this data. What kinds of functions might we
fit? For example, a straight line fit f (x|θ) = θ0 + θ1 x may look reasonable.
P10
Or is it a 10th order polynomial, f (x|θ) = i=0 xi θi , see fig(2.2)? In fact, the
data was generated using the rule t = sin(2.5x) + η, where η is zero mean Gaussian
noise of variance 0.22 .
To find a “good” curve, we need appropriate beliefs/assumptions about the smooth-
ness of the “clean” underlying function and the level of noise. If our assumptions
are not correct, our predictions will be poor.
Classes of Predictors Our general approach to supervised learning is to make some function f (x|θ) so
1 This is a supervised problem since a sequence has a temporal ordering and can be written as
(t, x) pairs : (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, ?).

17
 18

Raw Noisy Data Straight line fit

1.5 1.5

1 1

0.5 0.5

0 0

t
−0.5 −0.5

−1 −1

−1.5 −1.5
−1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1
x x

Figure 2.1: Left: Training Data for a regression problem. We wish to fit a function
f (x|θ) to this data. Right: A straight line fit might look reasonable.
10th order polynomial fit True underlying generating function
1.5 1.5

1 1

0.5
0.5
0
0
t

t
−0.5
−0.5
−1

−1.5 −1

−2 −1.5
−1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1
x x

Figure 2.2: Left: What about a 10t h order polynomial. This has zero training
error. Right: The “correct” clean underlying function which generated the data.

that, given a novel point x our prediction f (x|θ) will be accurate. What do we
mean by accurate? If we had some extra data, Dtest , different from the training
data and generated in the same manner, then we would like that the error made
by our predictions is roughly the same as the error that would be made even if we
knew exactly what the clean underlying data generating process were. Of course,
this is in some sense, an impossible task. However, we can devise procedures that
can help give us some confidence in our predictions.

2.1.2 Training Error

The typical way that we train/learn the adjustable parameters θ of our model is
to optimise some objective function. For example, if our current model outputs
f (xµ |θ) on an input xµ and the training data output for that xµ is tµ , we would like
to adjust the parameters θ such that f (xµ |θ) and tµ are close. We can measure how
close these values are by using a function d(x, y) which measures the discrepancy
between two outputs x and y. To find the best parameter settings for the whole
training set, we use
X
Etrain (θ) = d(f (xµ |θ), tµ ) (2.1.1)
(xµ ,tµ )∈Dtrain

If we adjust the parameters θ to minimise the training error, what does this tell
us about the prediction performance on a novel point x? In principle, nothing!
However, in practice, since the mechanisms which generate data are in some sense
smooth, we hope that our predictions will be accurate. We saw that in the case of
using a perceptron, we can always find a hyperplane that separates data, provided
 19

that the dimension of the data space is larger that the number of training examples.
In this case, the training error is zero. We saw, however, that the error on the
600 test examples was non-zero. Indeed, if the training data is believed to be a
corrupted version of some clean underlying process, we may not wish to have a
zero training error solution since we would be “fitting the noise”. What kind of
error would we expect that our trained model would have on a novel set of test
data?

2.1.3 Test Error

Imagine that we have gone through a procedure to minimise training error. How
can we assess if this will have a good predictive performance – i.e., will generalise
well? If we have an independent set of data Dtest , the test error
X
Etest (θ) = d(f (xµ |θ), tµ ) (2.1.2)
(xµ ,tµ )∈Dtest

is an unbiased estimate of the prediction error of our model f (x|θ).

2.1.4 Validation Data

Consider two competing prediction model classes, f1 (x|θ 1 ) and f2 (x|θ 2 ). We train
each of these by minimising the training error to end up with training error “opti-
mal” parameter settings θ∗1 and θ∗2 . Which is the better model? Is it the one with
the lower training error? No. We can say that model with setting θ∗1 is better
than a model with setting θ∗2 by comparing the test errors, Etest (θ 1 ) < Etest (θ2 ).
Using test data in this way enables us to validate which is the better model.
The standard procedure is to split any training data into three sets. The first is
the training data, Dtrain , used to train any model. The second Dvalidate is used to
assess which model has a better test performance. Once we have chosen our optimal
model on the basis of using validation data, we can get an unbiased estimate of the
expected performance of this model by using a third set of independent data Dtest .
This data should not have been used in any way during the training procedure if
we wish to obtain an unbiased estimate of the expected test performance of the
model.

2.1.5 Dodgy Joe and Lucky Jim

Perhaps the following parody will make the above arguments clearer:
Let me introduce two characters, “Lucky Jim” and “Dodgy Joe”. Lucky Jim in-
vents some new procedure, and initially, finds that it works quite well. With further
experimentation, he finds that it doesn’t always work, and that perhaps it requires
some rather fine tuning to each problem. Undeterred, this charismatic scientist
attracts both funds and attention enough to stimulate a world wide examination of
his method. Working independently of each other, surely enough research groups
from around the world begin to report that they manage to achieve zero test error
on each problem encountered. Eventually, some research group reports that they
have found a procedure, based on Lucky Jim’s method that is able to give zero test
error on every problem that has ever been known to exist. After so many years of
hard work, Lucky Jim happily announces his universal predictor (perhaps a billion
hidden unit neural network with fixed parameters), with the (indeed true) claim
 20

that it gives zero test error on every known problem that ever existed. He markets
this product and hopes to claim a fortune.
Contrast the dogged determination of Lucky Jim now with the downright un-
scrupulous behaviour of Dodgy Joe. Quite frankly, he doesn’t have the patience
of Lucky Jim, and he simply assembles all the known problems that ever existed,
and their corresponding test sets. He then constructs his method such that, when
asked to perform the prediction on problem A with corresponding test set B, he
simply makes the output of his method the output for the test set B (which he of
course knows). That is, his algorithm is nothing more than a lookup table - if the
user says, “this is the test set B” then Dodgy Joe’s algorithm simply reads off the
predictions for test set B which, by definition, will give zero error. He then also
markets his universal predictor package as giving zero test performance on every
known problem (which is indeed true) and also hopes to make a fortune.
If we look at this objectively, both Lucky Jim and Dodgy Joe’s programs are doing
the same thing, even though they arrived at the actual code for each method in
a different way. They are both nothing more than lookup tables. The point is
that we have no confidence whatsoever that either Lucky Jim’s or Dodgy Joe’s
package will help us in our predictions for a novel problem. We can only have
confidence that a method is suitable for our novel problem if we believe that a
particular method was successful on a similar problem to ours in the past, or the
assumptions that resulted in successful prediction on a previous problem might
well be expected to hold for a novel problem – smoothness of the problems for
example.
The above also highlights the issue that it is not enough to assess a method only
on the reported results of a subset of independent research groups. It may be that,
with the same method (eg neural nets with a fixed architecture but undetermined
parameters) one of a hundred groups which decide to tackle a particular problem
is able to find that particular set of parameter values (essentially by chance) that
gives good test performance, whilst the other 99 groups naturally do not report
their poor results. In principal, real comparison of a method on a problem requires
the collation of all results from all sources (attempts).
WowCo.com WowCo.com is a new startup prediction company. After years of failures, they
eventually find a neural network with a trillion hidden units that achieves zero
test error on every learning problem posted on the internet up till January 2002.
Each learning problem included a training and test set. Proud of their achievement,
they market their product aggressively with the claim that it ‘predicts perfectly
on all known problems’. Would you buy this product?
Model Comparison : An Let us reconsider our favourite digit classification problem. There are 1200 ex-
example amples of the digit 1 and 7. Let us split this to form a new training set of 400
examples, and a validation set of 200 examples. We will retain a further 600 ex-
amples to measure the test error. I used PCA to reduce the dimensionality of
the inputs, and then nearest neighbours to perform the classification on the 200
validation examples. Based on the validation results, I selected 19 as the number
of PCA components retained, see fig(2.3). The independent test error on 600 in-
dependent examples using 19 dimensions is 14. Once we have used the validation
data to select the best model, can we use both training and validation data to
retrain the optimal model? In this case, we would have decided that 19 is the op-
timal dimension to use, based on 200 training and 100 validation points. Can we
now, having decided to use 19 components, retrain the model on the 300 training
 21

12

10

number of errors
8

0
0 20 40 60 80 100
number of eigenvalues

Figure 2.3: 400 training examples are used, and the validation error plotted on
200 further examples. Based on the validation error, we see that a dimension of
19 is reasonable.

and validation points? This is a fairly subtle point. In principle, the answer is
no, since the new procedure, strictly speaking, corresponds to a different model.
However, in practice, there is probably little to be lost by doing so, and possibly
something to be gained since we are making use of more training data in setting
many parameters. These issues highlight some of the philosophical complications
that can arise based on a frequentist interpretation of probability. No such diffi-
culties arise in the Bayesian framework, where all the training data can be used in
a clear and consistent manner for model selection.

2.1.6 Regularisation

If the data generation process includes noise (additive), then the true, clean data
generating process will typically be smoother than the observed data would directly
suggest. To try to discover this smoother clean underlying process, we need to
ensure that our model for the clean underlying process does not fit the noise in
the observed data. That is, it is undesirable to have a zero training error, and we
need to encourage our model to be smooth. One way to achieve this is through
regularisation in which an extra “penalty” term is added on the the standard
training error, to form the regularised training error:

Eregtrain (θ, λ) = Etrain (θ) + λEreg (θ) (2.1.3)

The larger λ is, the smoother will be solution which minimises the regularised
training error. If we regularise too much, the solution will be inappropriate and
too smooth. If we don’t regularise at all, the solution will be over complex, and
the solution will be fitting the noise in the data. (Regularisation only really makes
sense in the case of models which are complex enough that overfitting is a potential
problem. There is little point in taming a pussy-cat; taming a lion however, might
be worthwhile!). How do we find the “optimal” value for λ? Training is then done
in two stages:
• For a fixed value of λ, find θ ∗ that optimises Eregtrain . Repeat this for each
value
 ∗ of λ that you wish to consider. This gives rise to a set of models,
θ λi , i = 1, . . . , V .
• For each of these models, on a separate validation set of data (different from
the training data used in the first step), calculate the validation error:
X
Eval (θ∗ ) = d(f (xµ |θ ∗ ), tµ ) (2.1.4)
(xµ ,tµ )∈Dval
 22

The “optimal” model is that which has the lowest validation error.
Regularisation : An example In fig(2.4), we fit the function t = a sin(wx) to data, learning the parameters
a and w. The unregularised solution badly overfits the data, and has a high
validation error. To encourage a smoother solution, I used a regularisation term
Ereg = w2 . I then computed the validation error based on several different values
of the regularisation parameter λ, finding that λ = 0.5 gave a low validation error.

6
1.5

4 1

2 0.5

0 0

−2 −0.5

−4 −1

−6 −1.5
−3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3

Figure 2.4: Left: The unregularised fit (λ = 0) to training given by ×. Whilst the
training data is well fitted, the error on the validation examples, + is high. Right:
the regularised fit (λ = 0.5). Whilst the training error is high, the validation error
(which is all important) is low. The true function which generated this noisy data
is the dashed line, and the function learned from the data is the solid line.

2.2 Problems
Exercise 1 WowCo.com is a new startup prediction company. After years of
failures, they eventually find a neural network with a trillion hidden units that
achieves zero test error on every learning problem posted on the internet up till
January 2002. Each learning problem included a training and test set. Proud of
their achievement, they market their product aggressively with the claim that it
‘predicts perfectly on all known problems’. Would you buy this product? Justify
your answer.

2.3 Solutions
 3 Nearest Neighbour Classification

In classification, we have a training set of data which contains both attributes

x and a class label c. For example, the vector x might represent an image of a
digit, and c labels which digit it is, c ∈ {0, 1, . . . , 9}. A dataset D of P training
datapoints is given by D = {xµ , cµ }, µ = 1, . . . , P . The aim is, given a novel x, to
return the “correct” class c(x). A simple strategy we adopt in this chapter can be
very loosely stated as:

In other words, ‘just say Things x which are similar (in x-space) should have the same class label
whatever your neighbour
says!’
(This is a kind of smoothness assumption. Note that in this chapter, we won’t
explicitly construct a ‘model’ of the data in the sense that we could generate fake
representative data with the model. It is possible, however, to come up with a
model based on the above neighbourhood type idea which does just this. We will
see how to do this when we learn about density estimation in a later chapter.)
What does ‘similar’ mean? The key word in the above strategy is ‘similar’. Given two vectors x and y rep-
resenting two different datapoints, how can we measure similarity? Clearly, this
would seem to be rather subjective – two datapoints that one person thinks are
‘similar’ may be to someone else dissimilar.
The dissimilarity function Usually we define a function d(x, y), symmetric in its arguments (d(x, y) = d (y, x))
d(x, y) that measures the dissimilarity between the datapoints x and y.
It is common practice to adopt a simple measure of dissimilarity based on the
squared euclidean distance d(x, y) = (x − y)T (x − y) (often more conveniently
written (x − y)2 ) between the vector representations of the datapoints. There
can be problems with this but, in general, it’s not an unreasonable assumption.
However, one should bear in mind that more general dissimilarity measures can,
and often are used in practice.

Machine Learning’s “Dirty Secret”

Some say that nearest neighbours methods might be construed as machine learn-
ing’s “dirty secret” – one can often get very good results with such a simple method,
and the more sophisticated methods don’t really provide much more. Well, it’s no
secret that machine learning depends, as we discussed, a sense of smoothness in
the data – this is no dirty secret – it’s the fundamental assumption upon which
most machine learning algorithms are based. Having said that, nearest neighbour
methods are a good starting point in many applications, since they are intuitive
and easy to program. I would recommend this approach as a first starting point
to trying to understand the problem.

3.1 Nearest Neighbour

To classify a new vector x, given a set of training data (xµ , cµ ), µ = 1, . . . , P :

23
 24

1
? 3 3
1
2 3
1 1 3
2
2 3
1 2
1 2
2 2 2

Figure 3.1: In nearest neighbour classification, a new vector with an unknown

label, ?, is assigned the label of the vector in the training set which is nearest. In
this case, the vector will be classified as a 2.

1. Calculate the dissimilarity of the test point x to each of the stored points,
dµ = d (x, xµ ).
∗
2. Find the training point xµ which is ‘closest’ to x by finding that µ∗ such
∗
that dµ < dµ for all µ = 1, . . . , P .
∗
3. Assign the class label c(x) = cµ .
In the case that there are two or more ‘equidistant’ (or equi-dissimilar) points with
different class labels, the most numerous class is chosen. If there is no one single
most numerous class, we can use the K-nearest-neighbours case described in the
next section.

The decision boundary In general, the decision boundary is the boundary in input space such that our
decision as to the class of the input changes as we cross this boundary. In the
nearest neighbour algorithm above based on the squared euclidean distance, the
decision boundary is determined by the lines which are the perpendicular bisectors
Voronoi Tessellation of the closet training points with different training labels, see fig(3.2). This is called
a Voronoi tessellation.

1 1
2

Figure 3.2: The decision boundary for the nearest neighbour classification rule is
piecewise linear with each segment corresponding to the perpendicular bisector
between two datapoints belonging to different classes.
 25

2 2 2
2 2 2 2 2 2 2
1 1
? 1 1 1
1 1 1 1

Figure 3.3: Consider data which lie close to (hyper)planes. The Euclidean distance
would classify ? as belonging to class 2 – an undesirable effect.

3.1.1 Problems with Nearest Neighbours

The nearest neighbours algorithm is extremely simple yet rather powerful, and
used in many applications. There are, however, some potential drawbacks:
How should we measure the distance between points? Typically one uses the
euclidean square distance, as given in the algorithm above. This may not always
be appropriate. Consider a situation such as in fig(3.3), in which the euclidean
Invariance to linear distance leads to an undesirable result. If we use the Eucliean distance, (x −
transformation y)T (x − y) then the distance between the orthogonally transformed vectors M x
and M y (where M T M is the identity matrix) remains the same. (This is not
true for the Mahalanobis distance). Since classification will be invariant to such
transformations, this shows that we do not make a sensible model of how the data
is generated – this is solved by density estimation methods – see later chapter.
Mahalanobis Distance The Mahalanobis distance (x − y)T Ai (x − y) where usually Ai is the inverse co-
variance matrix of the data from class i can overcome some of these problems. I
think it’s better to use density estimation methods.
In the simple version of the algorithm as explained above, we need to store the
whole dataset in order to make a classification. However, it is clear that, in general,
only a subset of the training data will actually determine the decision boundary.
Data Editing This can be addressed by a method called data editing in which datapoints which
do not affect (or only very slightly) the decision boundary are removed from the
training dataset.
Dimension Reduction Each distance calculation could be quite expensive if the datapoints are high dimen-
sional. Principal Components Analysis (see chapter on linear dimension reduction)
is one way to address this, by first replacing each high dimensional datapoing xµ
with it’s low dimensional PCA components vector pµ . The euclidean distance of

2
2
the of two datapoints xa − xb is then approximately given by pa − pb – thus
we need only to calculate distance among the PCA representations of data. This
can often also improve the classification accuracy.
Sensitivity to outliers An outlier is a ‘rogue’ datapoint which has a strange label – this maybe the result
of errors in the database. If every other point that is close to this rogue point
has a consistently different label, we wouldn’t want a new test point to take the
label of the rogue point. K nearest neighbours is a way to more robustly classify
datapoints by looking at more than just the nearest neighbour.

3.2 K Nearest Neighbours

As the name suggests, the idea here is to include more than one neighbour in the
decision about the class of a novel point x. I will here assume that we are using
the Euclidean distance as the simmilarity measure – the generalisation to other
dissimilarity measures is obvious. This is achieved by considering a hypersphere
 26

2
1 ?

Figure 3.4: In K-nearest neighbours, we centre a hypersphere around the point we

wish to classify. The first circle corresponds to the nearest neighbour method, and
we would therefore class ? as class 1. However, using the 3 nearest neighbours, we
find that there are two 2’s and one 1 – and we would therefore class ? as a 2.

centred on the point x with radius r. We increase the radius r until the hypersphere
contains exactly K points. The class label c(x) is then given by the most numerous
class within the hypersphere. This method is useful since classifications will be
robust against “outliers” – datapoints which are somewhat anomalous compared
with other datapoints from the same class. The influence of such outliers would
be outvoted.
How do we choose K? Clearly if K becomes very large, then the classifications will become all the same
– simply classify each x as the most numerous class. We can argue therefore that
there is some sense in making K > 1, but certainly little sense in making K = P
(P is the number of training points). This suggests that there is some “optimal”
Generalisation intermediate setting of K. By optimal we mean that setting of K which gives the
best generalisation performance. One way to do this is to leave aside some data
that can be used to test the performance of a setting of K, such that the predicted
class labels and the correct class labels can be compared. How we define this is
the topic of a later chapter.

3.3 Handwritten digit Example

We will apply the nearest neighbour technique to classify handwritten digits. In
our first experiement, we will first look at a scenario in which there are only two
digit types, zeros, and ones. There are 300 training examples of zeros, and 300
training examples of ones, fig(3.5). We will then use the nearest neighbour method
to predict the label of 600 test digits, where the 600 test digits are distinct from
the training data and contain 300 zeros and 300 ones (although, of course, the test
label is unknown until we assess the performance of our predictions). The nearest
neighbour method, applied to this data, predicts correctly the class label of all 600
test points. The reason for the high success rate is that examples of zeros and ones
are sufficiently different that they can be easily distinguished using such a simple
distance measure.
In a second experiment, a more difficult task is to distinguish between ones and
sevens. We repeated the above experiment, now using 300 training examples of
ones, and 300 training examples of sevens, fig(3.6). Again, 600 new test examples
(containing 300 ones and 300 sevens) were used to assess the performance. This
 27

time, 18 errors are found using nearest neighbour classification – a 3% error rate for
this two class problem. The 18 test points on which the nearest neighbour method
makes errors are plotted in fig(3.7). Certainly this is a more difficult task than
distinguishing between zeros and ones. If we use K = 3 nearest neighbours, the
State of the art classification error reduces to 14 – a slight improvement. Real world handwritten
digit classification is big business. The best methods classify real world digits (over
all 10 classes) to an error of less than 1% – better than human average performance.

Figure 3.5: (left) Some of the 300 training examples of the digit zero and (right)
some of the 300 training examples of the digit one.

Figure 3.6: Some of the 300 training examples of the digit seven.

Figure 3.7: The Nearest Neighbour method makes 18 errors out of the 600 test
examples. The 18 test examples that are incorrectly classified are plotted (above),
along with their nearest neightbour in the training set (below).

3.4 A Probabilistic Intepretation

The Nearest Neighbour method is powerful, and its use widespread. In the context
of probabilistic models, however, it does not at first sight appear to fit well. Here
we’ll show how NN methods indeed can be seen as limiting cases of probabilistic
models. This is useful since this insight opens up the way to generalise upon the
simple NN methods to more complex approaches in a natural way.
Consider the situation where we have (for simplicity) data from two classes – class
0 and class 1. We make the following model for data from class 0:
1 1 X µ 2 2
p(x|c = 0) = e−(x−x ) /(2σ )
P0 (2πσ 2 )N/2
µ∈class0
 28

where N is the dimension of a datapoint x and P0 are the number of training

datapoints of class 0, and σ 2 is the variance. This is simple version of a so-
called Parzen estimator, which simply models the data distribution as a sum of
distributions centered on the training points.
Similarly, for data from class 1:
1 1 X µ 2
) /(2σ2 )
p(x|c = 1) = e−(x−x
P1 (2πσ 2 )N/2
µ∈class1

Then, in order to classify a new datapoint x∗ , we need to calculate

p(x∗ |c = 0)p(c = 0)
p(c = 0|x∗ ) =
p(x∗ |c = 0)p(c = 0) + p(x∗ |c = 1)p(c = 1)

which follows from using Bayes’ rule. One can show (exercise) that the maximum
likelihood setting of p(c = 0) is P0 /(P0 + P1 ), and p(c = 1) = P1 /(P0 + P1 ). A
similar expression holds for p(c = 1|x∗ ). Hence

p(c = 0|x∗ ) p(x∗ |c = 0)p(c = 0)

∗
= (3.4.1)
p(c = 1|x ) p(x∗ |c = 1)p(c = 1)

If σ is very small, the numerator, which is a sum of exponential terms, will be

dominated by that term for which xµ0 in class 0 is closest to the point x∗ . Similarly,
the denominator will be dominated by that point xµ1 in class 1 which is closest to
x∗ . Hence
∗ µ0 2 2 ∗ µ0 2 2
p(c = 0|x∗ ) e−(x −x ) /(2σ ) p(c = 0)/P0 e−(x −x ) /(2σ )
≈ =
p(c = 1|x∗ ) µ 2 2
e−(x −x 1 ) /(2σ ) p(c = 1)/P1
∗ µ 2
e−(x∗ −x 1 ) /(2σ2 )

Taking the limit σ 2 → 0, we will with certainty therefore classify x∗ as class 0 if

x∗ has a point in the class 0 data which is closer than the closest point in the class
1 data, and vice versa. This, of course, is simply the nearest neighbour method.
The motivation of using K nearest neighbours is produce a result that is more
robust to outliers (mislabelled training data points). To ensure a similar kind of
robustness in the probabilistic interpretation, we can simply use a larger value of
σ 2 . This will smooth the extreme probabilities of classification and mean that more
points (not just the nearest) will have an effective contribution to the numerator
and denominator of equation (3.4.1).
The extension to case of more than two classes is straightforward.
This interpretation is nice, since we can see how naturally such a popular algorithm
as NN can be seen as a limiting case of a generative model for data. Indeed, the
extension to something akin to KNN is natural. To go further forward, it would
therefore be natural to relax the assumption about using a Parzen estimator, and
use something more complex. We will examine such cases in some detail in later
chapters.
4 Linear Dimension Reduction

Often in machine learning, the data is very high dimensional. In the case of
the hand-written digits from chapter(3), the data is 784 dimensional. Images are
a good example of high dimensional data, and a good place where some of the
basic motivations and assumptions about machine learning come to light. For
simplicity, consider the case of the handwritten digits in which each pixel is binary
– either 1 or 0. In this case, the total possible number of images that could ever
exist is 2784 ≈ 10236 – this is an extremely large number (very much larger than
the number of atoms in the universe). However, it is clear that only perhaps
at most a hundred or so examples of a digit 7 would be sufficient (to a human)
to understand how to recognise a 7. Indeed, the world of digits must therefore
lie in a highly constrained subspace of the 784 dimensions. It is certainly not
true that each dimension is independent of the other in the case of digits. In
other words, certain directions in the space will be more important than others
A hook for machine learning for describing digits. This is exactly the hope, in general, for machine learning –
that only a relatively small number of directions are relevant for describing the
true process underlying the data generating mechanism. That is, any model of the
Features data will have a relatively low number of effective degrees of freedom. These lower
dimensional independent representations are often called ‘feature’ representations,
since it is these quintessential features which succinctly describe the data.
Linear Dimension Reduction In general, it seems clear that the way dimensions depend on each other is, for a
general machine learning problem (and certainly the digits data) very complex –
certain dimensions being ‘on’ means that others are likely to be ‘off’. This suggests
that non-linear effects will, in general, be important for the efficient description of
data. However, finding non-linear representations of data is numerically difficult.
Here, we concentrate on linear dimension reduction in which a high dimensional
datapoint x is represented by y = Fx where the non-square matrix F has di-
mensions dim(y) × dim(x), dim(y) < dim(x). The matrix F represents a linear
projection from the higher dimensional x space to the lower dimensional y space.
The form of this matrix determines what kind of linear projection is performed
and, classically, there are several popular choices. The two most popular corre-
spond to Principal Components Analysis (PCA) and Linear Discriminants. The
first is an unsupervised and the latter a supervised projection. We concentrate
in this chapter on the more generic PCA, leaving linear discriminants to a later
chapter. Note that, again, these methods do not describe any model from which
we could generate data and, are also non-probabilistic. However, probabilistic data
generating versions do exist which are model based but are beyond the scope of
this course.

4.1 Principal Components Analysis

If data lies in a high dimensional space, we might hope that it lies close to a
hyperplane, as in fig(19.1). We then can approximate each data point by using
the vectors that span the hyperplane alone. I will sometimes refer to this small
set of vectors as the “basis” set. Strictly speaking this is not a basis for the

29
 30

x
x

x
x

Figure 4.1: In linear dimension reduction we hope that data that lies in a high
dimensional space lies close to a hyperplane that can be spanned by a smaller
number of vectors.

whole space, rather is is a ‘basis’ which approximately spans the space where
the data is concentrated. Effectively, we are trying to choose a more appropriate
low dimensional co-ordinate system that will approximately represent the data.
Mathematically, we write
M
X
x≈c+ wi bi (4.1.1)
i=1

The vectors bi , i ∈ 1, . . . M are chosen to be orthonormal. That is (bi )T bj = 0

for i 6= j, and (bi )T bi = 1. There is no loss of generality in this, since any non-
orthonormal basis would simply correspond to a transformation of the coefficients
wiµ .
If the dimension of the data space, dim(x) = N , our hope is that we can describe
the data using only a small number M of vectors. If we can do so, we can reduce
greatly the information needed to accurately describe the data. For example, if the
data lies in a 784 dimensional space, we might hope that we can describe the data
accurately using the above linear prescription with a much smaller dimensional
representation.
One can show (see end of chapter) that the optimal lower dimensional represen-
tation (optimal in the sense of minimal squared reconstruction error) is given by
projecting the data onto the eigenvectors of covariance matrix with the largest M
eigenvalues. Algorithmically, this is :
1. Find the mean and covariance matrix of the data:
P P
1 X µ 1 X µ
m= x , S= (x − m)(xµ − m)T (4.1.2)
P µ=1 P − 1 µ=1

2. Find the eigenvectors e1 , . . . , eM of the covariance matrix S which have the

largest eigenvalues. Form the matrix E = [e1 , . . . , eM ] which has the largest
eigenvectors as its columns.
3. The lower dimensional represention of each data point xµ is given by y µ =
ET (xµ − m).
 31

Figure 4.2: Projection of two dimensional data using one dimensional PCA. Plotted
are the original datapoints (crosses) and their reconstructions using 1 dimensional
PCA (circles). The two lines represent the eigenvectors and their lengths their
corresponding eigenvalues.

4. The approximate reconstruction of the original datapoint xµ is

xµ ≈ m + Ey µ (4.1.3)

5. The totalPsquared error over all the training data made by the approximation
is (P −1) N j=M+1 λj where λj , j = M +1 . . . N are the eigenvalues discarded
in the projection.
One can view the PCA reconstructions (though there is usually little use for these
except to check that they give an adequate representation of the original data) as
orthogonal projections of the data onto the subspace spanned by the M largest
eigenvectors of the covariance matrix, see fig(4.2).
Interpreting the Eigenvectors Do the eigenvectors themselves explicitly have any meaning? No! They only
act together to define the linear subspace onto which we project the data – in
themselves they have no meaning. We can see this since, in principle, any basis
which spans the same subspace as the eigenvectors of the covariance matrix is
equally valid as a representation of the data. For example, any rotation of the
basis vectors within the subspace spanned by the first M eigenvectors would also
have the same reconstruction error. The only case when the subspace is uniquely
defined is when we only use one basis vector – that is, the principal component of
the correlation matrix alone.
The “intrinsic” dimension of How many dimensions should the linear subspace have? As derived (at the end of
data the chapter), the reconstruction error is dominated by the largest eigenvalues of
the covariance matrix. If we plot the eigenvalue spectrum (the set of eigenvalues
ordered by decreasing value), we might hope to see a few large values and many
small values. Indeed, if the data did lie very close to say a M dimensional linear
manifold (hyperplane), we would expect to see M large eigenvalues, and the rest to
be very small. This would give an indication of the number of degrees of freedom
in the data, or the intrinsic dimensionality. The directions corresponding to the
small eigenvalues are then interpreted as “noise”.
Warning! It might well be that a small reconstruction error can be made by using a small
number of dimensions. However, it could be that precisely the information required
to perform a classification task lies in the “noise” dimensions thrown away by the
above procedure (though this will hopefully be rather rare). The purpose of linear
discriminants is to try to deal with this problem.
 32
mean

original original original original

Figure 4.3: (left) Four of the 892 images. (right) The mean of the 892 images
5
x 10
6

eigenvalue
3

0
0 10 20 30 40 50 60 70 80 90 100
eigenvalue number

Figure 4.4: The 100 largest eigevalues

Non-linear Dimension Whilst it is straightforward to perform the above linear dimension reduction, bear
Reduction in mind that we are presupposing that the data lies close to a hyperplane. Is this
really realistic? More generally, we would expect data to lie on low dimensional
curved manifolds. Also, data is often clustered – examples of handwritten ‘4’s look
similar to each other and form a cluster, separate from the ‘8’s cluster. Neverthe-
less, since linear dimension reduction is so straightforward, this is one of the most
powerful and ubiquitous techniques used in dimensionality reduction.

4.1.1 Example : Reducing the dimension of digits

We have 892 examples of handwritten 5’s. Each is a 21*23 pixel image – that is,
each data point is a 483 dimensional vector. We plot 4 of these images in fig(4.3).
The mean of the data is also plotted and is, in a sense, an archetypal 5. The
covariance matrix has eigenvalue spectrum as plotted in fig(4.4), where we plot
only the 100 largest eigenvalues. The reconstructions using different numbers of
eigenvectors (10, 50 and 100) are plotted in fig(4.5). Note how using only a small
number of eigenvectors, the reconstruction more closely resembles the mean image.

4.1.2 PCA and Nearest Neighbours

In the chapter on nearest neighbour methods, we needed to calculate (many times)

distances between vectors. This can be computationally demanding, and it is often
a good idea (when using the euclidean distance) to project the data onto a lower
dimension first. For example, in the case where we wanted to make a classifier to
distinguish between the digit 1 and the digit 7 we first use PCA by ignoring the
classlabel (to make a dataset of 1200 training points). Each of the training points

original 10 e−vec reconstr. 50 e−vec reconstr. 100 e−vec reconstr.

Figure 4.5: The reconstruction using different linear subspace dimensions

 33

xµ is then projected onto its 50 dimensional PCA representation y µ . Subsequently,

any distance calculations (xa − xb )2 are replaced by (y a − y b )2 .
To see that this is reasonable, consider

(xa − xb )T (xa − xb ) ≈ (Ey a − m − Ey b + m)T (Ey a − m − Ey b + m)

= (y a − y b )T ET E(y a − y b )
= (y a − y b )T (y a − y b ) (4.1.4)

where the last equality is due to the orthogonality of eigenvectors : ET E = I.

Using 50 principal components, the error rate using the nearest neighbour rule to
classify ones and sevens gave an error of 13 in 600 examples – better than without
using PCA! How can this be? A possible explanation for this type of phenomenon is
that the new PCA representation of the data is more relevant (effectively, common,
irrelevant directions in the data are ignored), so that distances between these more
relevant representations of the data can produce better results. This is not always
to be expected, however.

4.1.3 Mega Dimensional Data

You might be wondering how it is possible to perform PCA on extremely high

dimensional data. For example, if we have 500 images each of 1000 × 1000 = 106
pixels, the covariance matrix will be 106 × 106 dimensional – well beyond the
storage capacities of many computers.
One approach around this difficulty is to perform the calculations in a lower di-
mensional space. Note that there can only be at most P non-zero eigenvalues.
Using X to denote the (zero mean) data and E the matrix of eigenvectors – this
is non-square since there will be fewer eigenvalues than dimensions. We write the
eigenvalues as a diagonal matrix Λ. The eigenvalue requirement is

XXT E = EΛ (4.1.5)
T T T
X XX E = X EΛ (4.1.6)
XT XẼ = ẼΛ (4.1.7)

where we defined Ẽ = XT E. The last line above represents the eigenvector equation
for XT X. This is a matrix of dimensions P × P – in the above example, a 500 × 500
matrix as opposed to a 106 × 106 matrix previously. We then can calculate the
eigenvectors Ẽ and eigenvalues Λ of this matrix more easily. Once found, we then
use

E = XẼΛ−1 (4.1.8)

4.1.4 PCA is good because it is a poor compressor!

A moments thought throws up the following condundrum: It seems that we wish to

compress high dimensional data to a lower dimensional representation. However,
clearly, the optimal compressed representation retains no structure since, if it did,
further compression would still be possible. The goal of feature extraction is not
consistent with optimal compression, since we wish to remove some redundancy,
yet retain enough structure in the lower dimensional representation such that any
 34

further use of the data – making a machine which can generalise from the lower
dimensional representations for example – has a chance. Hence, perhaps somewhat
perversely, PCA is a reasonable feature extraction method because it is such a poor
compressor!

4.2 Deriving the Optimal Linear Reconstruction

We can assume wlog that the bj are orthonormal
 2
X X
E= xµi − wjµ bji 
µ,i j

differentiate with respect to wkµ gives (using the orthonormality constraint)

X
wkµ = bki xµi
i

Defining [U ]i,j = bji , the term

X µ j X j j µ X
wj bi = bi bk xk = Ui,j Uk,j xµk = [U U T xµ ]i
j k,j j,k

and the orthogonality constraint is U T U = I. The objective E (neglecting the

orthonormality constraint) becomes
X

2
E= I − U U T )xµ
µ

For simplicity, define the symmetric matrix

B̃ = I − U U T

One can show easily that B̃ B̃ = B̃. Then

 
E = trace (xµ )T B̃xµ

using the permutability of trace, this is

!
X
µ µ T
E = trace (x ) (x ) B̃
µ

Hence the objective becomes (neglecting constants)

E = −trace SU U T

where S is the correlation matrix of the data. The constraint can be written (using
a set of Lagrange multipliers)




−trace SU U T + trace M U T U − I

Since the constraint is symmetric, we can assume that M is also symmetric. Dif-
ferentiating with respect to M , we get

SU = U M
 35

Clearly, one solution is given by taking M to be diagonal and then U to be the

matrix whose
columns are the corresponding eigenvectors of S. In this case,
trace SU T U = trace (M ), which is the sum of the eigenvalues corresponding
to the eigenvectors forming U . Since we wish to minimise E, we should then take
the largest eigenvalues to minimize E.

Another possibility exists: Since the constraint is U T U = I, and the matrix Ũ =

RU , where R is an orthogonal matrix, RT R = I also  satisfies the constraint
Ũ T Ũ = I. In this case, the error term is trace Ũ T S Ũ , so that the error function
is invariant with respect to rotations. Hence, any solution which is a rotated
version of the eigenspace is also fine, and indeed has the same value for the error.
In summary, therefore, the solution to the optimal subspace is given by orthogonal
vectors that span the principal subspace of the correlation matrix.

4.3 Problems
Exercise 2 Consider AA−1 = I. By applying a differential operator ∂, show that

∂A−1 = A−1 ∂AA−1

By using the identity

elog A = A

show that

∂ log(A) = A−1 ∂A

Show that

log det A = trace (log A)

Exercise 3 Consider a dataset in two dimensions where the data lies on the cir-
cumference of a circle of unit radius. What would be the effect of using PCA on
this dataset, in which we attempt to reduce the dimensionality to 1? Suggest an
alternative one dimensional representation of the data.
a
Exercise 4 P Consider two vectors
PMx and xb and their corresponding PCA approx-
M
imations c+ i=1 ai e and c+ i=1 bi e , where the eigenvectors ei , i = 1, . . . M are
i i

mutually orthogonal and have unit length. The eigenvector ei has corresponding
eigenvalue λi .
Approximate (xa − xb )2 by using the PCA representations of the data, and show
that this is equal to (a − b)2 .

Exercise 5 The Gaussian distribution in one dimension is defined as

1 1 2
p(x|c) = √ e− 2σ2 (x−µ)
2πσ 2
We decide to fit a Gaussian to each class from a dataset of one-dimensional
Pn1 data.
Show that the Maximum Likelihood estimator of µ1 is µ̂1 = n11 x∈class1 x and
Pn1
that the ML estimate of σ12 is σ̂12 = n11 x∈class1 (x − µ̂1 )2
 36

Exercise 6 Let S be the covariance matrix of the data. The Mahalanobis distance
between xa and xb is defined as

T

xa − xb S −1 xa − xb .

Explain how to approximate this distance using the M -dimensional PCA approxi-
mations, as described above.

Exercise 7 You are hired as the researcher in a startup company specializing in

image processing. You are hired to provide the theory, algorithms and eventually to
build a hand-written digit classifier to be used for Postcode recognition by the Post
Office, giving a value from 0 to 9 for the class of a novel testpoint. The training
data is of the form xµ , µ = 1, . . . , P , where each image xµ has a corresponding
class label, cµ ∈ {0, 1, . . . , 9}.
Your line manager is very worried. She argues that for the case of binary valued
images, with 100×100 = 10000 pixels, there are in total 210000 such binary images.
She argues that this number is astronomically larger than the number of atoms in
the universe and that therefore the measly 25000 training examples they have will
be woefully inadequate for training. Explain how you might persuade your line
manager that the situation may not be so hopeless, giving compelling reasons for
your optimism.
Your line manager is so impressed with your argument that she decides to discard
all the training data currently in the database, and busies herself with making 1000
training examples of her own fair hand. Is this a reasonable thing to do? Justify
your answer.
As a first step, you decide to use the K nearest neighbour method (KNN) to classify
a novel test point x∗ .
Describe fully how the KNN algorithm works, including how to determine the op-
timal number of neighbours K to use.
Your line manager is pleased with your algorithm but is concerned that it performs
as well as you say it does. How can you persuade her that it really performs
according to your claims?
One morning your line manager is delighted to tell you that she now has more
training data than ever before, indeed the training data consists of P = 100, 000
real valued images. You estimate that your current KNN method is going to be too
slow to do real time classification of digits and you decide to use PCA to increase
classification speed.
Describe fully and mathematically how to use PCA to replace an N dimensional
vector x with an M dimensional vector y where M < N .
Derive a formula for approximating the distance (xa − xb )2 between two vectors xa
and xb using their corresponding PCA representations y a and y b .
Your line manager is pleased with your faster algorithm, which you claim provides
95% classification accuracy. She tells you that it is important to make sure that
99% are classified correctly in the end, even if this means that 10% of test images
need to be classified by a human. She asks you to adapt your algorithm accord-
ingly. Suggest an amendment to your algorithm and explain how you would decide
whether or not to leave a novel test point to be classified by a human.
 37

Exercise 8 In a recent radio lecture, the following phrase was uttered by a famous
Professor of Experimental Psychology:

“In a recent data survey, 90% of people claim to have above average intelligence,
which is clearly nonsense!” [Audience Laughs]. Discuss the truth or falsity of this
statement, and justify your answer.

Exercise 9 (PCA with external inputs) In some applications, one may sus-
pect that certain variables have a strong influence on how the data x is distributed.
For example, we could have a set of variables vkµ ,k = 1, . . . K for each observation
xµ , µ = 1, . . . P , which we think will heavily influence each xµ . It may therefore
be sensible to include these variables, and assume an approximation
X µ X µ
xµ ≈ wj bj + vk ck (4.3.1)
j k

where the coefficients wiµ , i = 1, . . . N , µ = 1, . . . P and basis vectors bj , j = 1, . . . J

and ck , k = 1, . . . K are to be determined.
The sum squared error loss between the xµ and their linear reconstruction equation
(4.3.1) is
 2
X X X
E= xµi − wjµ bji − vkµ cki  (4.3.2)
µ,i j k

Find the parameters that minimise E.

4.4 Solutions
6 Using the approximations, we have
X X X X
(xa − xb )T S −1 (xa − xb ) ≈ ( ai e i − bi ei )T S −1 ( aj e j − bj ej )
i i j i
P
Due to the orthonormality of the eigenvectors, this is i a2i /λi −2ai bi /λi +b2i /λi =
(a − b)T D−1 (a − b) where D is a diagonal matrix containing the eigenvalues.
7 Even though 25000 is a very small number compared to 210000 , the point is that
digits are not simply random point in a 10000 dimensional space. There is a great
deal of regularity and constraint on the form that each digit can take, so that digits
will occupy only a very small fraction of the space of all possible images. Indeed,
humans are capable of learning digits based on only a small number of training
examples, and there is therefore every reason to be optimistic that a machine could
do the same.
If we wish to make a classifier that works well on a wide variety of peoples hand-
writing, we need training data that is representative of a wide variety of styles.
Otherwise, the trained classifier may be appropriate for recognizing the handwrit-
ing of the line manager, but not necessarily anyone else.
The classification of the KNN method is based on finding the K nearest neighbours.
If none of the neighbours is very close, this will result is potentially inaccurate
classification. A simple method is therefore to use an independent testset, and
 38

set a threshold value. Measure the distance to the nearest neighbour for each
testpoint to be classified, and discard this point if it is greater than the threshold.
For the remaining undiscarded points, determine the classification. If this is not
99%, increase the threshold and repeat the procedure until a just sufficient value
of the threshold has been found.
8 Clearly false. A canny student will be able to give an example to demonstrate
this, which is surely the result of a highly non-symmetric distribution with many
(slightly) above average values and a few extremely low values. A simple demon-
stration is to assume that the average IQ is 100, and the minimum 0. If there are
only two possible scores, the above average score, and the below average score, then
one can easily show that 90 percent of people can indeed have an above average
IQ if the above average IQ score is less than 111.111.
9 To optimise equation (4.3.2), it is straightforward
P µ to P
show that we should first
µ
transform the data to be zero mean : µ x = 0 and µ vk = 0, k = 1, . . . , K.
We may assume, without loss of generality, that the bj are orthonormal (since we
could rescale the wjµ if not). However, we cannot assume that the ck , k = 1, . . . , K
are orthonormal, since we cannot rescale the v. Similarly, we assume nothing, a
priori, regarding the relationship between the vectors bj and ck . Differentiating
equation (4.3.2) with respect to wµ gives (using the orthonormality constraint on
the bi )
!
X X µ
wµ = (bi )T xµ − vl cl
i l

The residual vector (difference between xµ and the linear reconstruction) is then
!
X X µ X µ
µ µ i T µ
r =x − (b ) x − l
vl c bi − vj cj
i l j
P
By defining B̃ ≡ I − i bi (bi )T ≡ I − U U T , (using the notation of the previous
section), the residual is
 
X µ
rµ = B̃ xµ − vj cj 
j
P
Differentiating E = µ (rµ )T rµ with respect to ci , we get
X µ XX µ µ
vi B̃xµ = vj vi B̃cj
µ j µ

Define
X X
[Ṽ ]ij = viµ vjµ , [X̃]ij = viµ xµj , C = [c1 , . . . , cK ]
µ µ

Then the above has solution

C = X̃ Ṽ −1
Hence, the objective involves the matrix
X T
S̃ = (xµ − dµ ) (xµ − dµ )
µ
 39

where
X
dµ = vjµ cj
j

Hence, the optimal solution is given by taking the principal e-vecs of S̃, with C
set as above.
I believe this is a special case on Constrained Principal Components Analysis
(CPCA)[2].
 5 Linear Discriminant Analysis

We will be interested here in how we can exploit class label information to improve
the projection. That is to make supervised projections of the data. We begin with
a discussion of a simpler method in which the projection occurs in an unsupervised
way without using class information.

5.1 Unsupervised Dimension Reduction

Dimension reduction has been shown to be useful, not just in terms of reducing the
number of parameters that we ultimately might use in our model, but can indeed
be positively beneficial in removing irrelevant dimensions in the input space. PCA
was one of our main methods for dimension reduction and, more generally, we
could use autoencoders for non-linear dimension reduction. These methods are
all unsupervised in the sense that only the inputs xµ , µ = 1, . . . , P are used to
determine the reduction. The targets y µ , µ = 1, . . . , P are only used later, after
the dimension reduction technique. One potentially useful aspect of dimension
reduction is to be able to “see” the data. If we reduce the dimension to only two
or three, we can visualise the data by plotting the corresponding points y µ , µ =
1, . . . , P in the two or three dimensional space. This is potentially very useful since
the human eye is very good at interpreting spacial relationships in data, and may
give us some intuition about an appropriate model for the data.

5.1.1 Using PCA for visualisation

By finding say the largest three principal components of a dataset (independent

of any class labels they may have), we can reduce each high dimensional vector
xµ to its low three dimensional representation y µ . Then, we can plot each point
y µ and colour it according to its class label. Code to do this for our well known
threes and fives digit data is given below.
The result of this procedure is plotted in fig(5.1). We can see that the classes
are not particularly well separated spatially by this projection. Indeed, we did
not make any use of the class labels in determining the projection, so we cannot
necessarily expect that the projected data should appear well separated. Increasing
the complexity to use a non-linear projection will not necessarily improve the
situation either.

5.2 Fishers Linear Discriminant

How can we use class information to produce a projection that improves the sepa-
rability of the classes? Linear Discrimant Analysis is designed to solve this probken
by projecting the data linearly onto a subspace in such a way that two classes have
maximum separability.

40
 41

1500

1000

500

−500

−1000

−1500
−1500 −1000 −500 0 500 1000 1500

Figure 5.1: Projection of data onto two dimensions, formed by the two principal
components of the data. The fives are plotted with an ’x’, and the threes as an
’o’. We can see that the classes are partially separated in this two dimensional
projection, but there is a good deal of class overlap.

Figure 5.2: Two linear projections of two data classes. The dotted lines represent
the distributions for the projection that maximises the difference between the pro-
jected means. The full curves are Fisher’s projection. Fisher’s projection clearly
provides a better one-dimensional measure by which we can separate the classes.

5.2.1 One dimensional projection

We restrict attention here to two classes of data – the generalisation to more classes
is given in the subsequent section (although only the algorithm is described). Also,
for simplicty, we will project the data down to one dimension. The algorithm in a
later section deals with the higher dimensional multiple class case, and is known
as canonical variates.

Gaussian Assumption

Assume that, for each class, we can model the data with a Gaussian. That is

p(x1 ) = N (µ1 ; S1 ) , p(x2 ) = N (µ2 ; S2 ) (5.2.1)

We now wish to project these Gaussians down onto one dimension

y1µ = wT xµ1 , y2µ = wT xµ2 (5.2.2)

 42

2.5

1.5

1
1

0.5 0.5

0
0

−0.5
−0.5

−1
−1 3
−1.5 2
1
−1.5
0
−2 −3
−2 −1
−1
0 −2
1
−2.5 2 −3
−2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5 3

(a) Two dimensional projection (b) Three dimensional projection

Figure 5.3: Projection of the Leptograpsus Crab data onto the canonical variate
directions. (a) Two dimensional projection. Note that the data is almost perfectly
separated by this simple projection method. (b) Increasing the dimension of the
projection to three does not significantly improve the separability of the projected
classes. This can be expected since the Eigenvalues are 7.6, 3.2 and 0.15. That is,
the third direction contributes very little to separating the classes.

where y1µ is the projection of an input xµ that is in class one. Similarly, y2µ is
the projection of a datapoint that is in class 2. We want to find w such that,
in some sense, there is maximal separability of the classes in the one dimensional
projections. The aim of this is that a classifier can then be made based on where,
along this one dimension, a novel input lies. Because the projection is linear, the
projected distributions onto the one dimension are also Gaussian,



p(y1 ) = N m1 ; σ12 , p(y2 ) = N m2 ; σ22 (5.2.3)

m1 = wT µ1 , σ12 = wT S1 w m2 = wT µ2 , σ22 = wT S2 w. We could search for

a projection w such that the means of the Gaussians in the one dimension are
maximally separated (subject to the constraint that |w| = 1. However, if the
variances in the one dimension are large, there could be a large overlap still in the
classes. Fisher proposed therefore to use the objective function
2
(m1 − m2 )
(5.2.4)
π1 σ12 + π2 σ22

where πi represents the fraction of the dataset in class i. What this does is tries to
maximize the separation of the projected means whilst at the same time penalising
projections of large variance, see fig(5.2). Note that this objective function is
invariant to linear rescaling of w, so that there is no need to include a restriction
that wT w = 1. The optimal w is then given by

w ∝ Sw −1 (m2 − m1 ) (5.2.5)

where Sw = π1 S1 + π2 S2 .

5.3 Canonical Variates

Canonical variates generalises linear discriminants to more than two classes and
more than one projected dimension. We only state the algorithm here – the inter-
ested reader may consult Bishop’s book.
 43

100

80

60

40

20

−20

−40

−60
100
−80 50
0
−100
−50
−300 −200 −100 0 100 200 −100
300 400

Figure 5.4: The projection of the two classes using canonical variates. Here we
project onto three dimensions. Note how the data is well separated in the projec-
tions, and indeed, is almost linearly separated using this projection.

1. For each class form a covariance matrix Sk and mean mk . Define

c
X
S= Nk Sk (5.3.1)
k=1

where Nk is the number of datapoints in class k, and c is the total number

of classes.
2. Find m the mean of the whole dataset and mk , the mean of the each class
k. Form
c
X T
SB = Nk (mk − m) (mk − m) (5.3.2)
k=1

To project onto an m dimensional space, the optimal projection matrix W

corresponds to the first m eigenvectors of S−1 SB . The projections are then
given by y = W′ ∗ x.

5.3.1 Using Canonical variates on the Digits Data

We can apply the method of canonical variates as described above to project the
digit data onto a small number of dimensions (in the code below we project onto
three dimensions). We use here 600 examples of a three and 600 examples of a
five. Thus, overall , there are 1200 examples which lie in a 784 (28 × 28 pixels)
dimensional space. Since there are more datapoints than dimensions in the space,
the points cannot, a priori, be trivially separated by a linear decision boundary.
Note how the projection onto three dimensions enables the data to be separated
almost perfectly, see fig(5.4) Canonical variates is a useful method for dimension
reduction for labelled data, preserving much more class relevant information in the
projection than PCA. We can use the lower dimensional representations to help
visualise the data and also for use in further processing such as building a classifier.
6 Linear Parameter Models

6.1 Introduction
Consider the data in fig(6.1), in which we plot the number of chirps per second
for crickets, versus the temperature in degrees Fahrenheit. A biologist believes
that there is a simple relation between the number of chirps and the temperature.
Modelling such a relation is a regression problem. The biologist decides to make
a straight line fit :

c = a + bt (6.1.1)

where she needs to determine the parameters a and b. How can she determine these
parameters based on the training data (cµ , tµ ), µ = 1, . . . , 15 ? For consistency with
our previous notations, let us use y rather than c, and x in place of t, so that our
model is y = a + bx. The sum squared training error is
P
X
E(a, b) = (y µ − a − bxµ )2 (6.1.2)
µ=1

Differentiating with respect to a, we find

X
(y µ − a − bxµ ) = 0 (6.1.3)
µ

Differentiating with respect to b, we find

X
(y µ − a − bxµ )xµ = 0 (6.1.4)
µ

Dividing by P , we thus have two simultaneous linear equations

hyi − a − b hxi = 0 (6.1.5)


hyxi − a hxi − b x2 = 0 (6.1.6)
(6.1.7)
PP
where we used the notation h·i to denote P1 µ=1 · .We can easily solve these linear
equations to determine a and b. The important thing to note about this regression
model is that the parameters only appear in a linear fashion. We could also, more
conveniently write our model as

y = wT φ (6.1.8)

where w = (a, b)T and φ = (1, x)T . The training error then is
P
X
E(w) = (y µ − wT φµ )2 (6.1.9)
µ=1

44
 45

26

24

22

20

chirps per sec

18

16

14

12

10

8
70 75 80 85 90
temperature (F)

Figure 6.1: Data from crickets – the number of chirps per second, versus the
temperature in Fahrenheit.

where φµ = (1, xµ )T . We now wish to determine the parameter vector w. Writing

out the error in terms of the components,
P
X X X
E(w) = (y µ − wi φµi )(y µ − wj φµj ) (6.1.10)
µ=1 i j

Differentiating with respect to wk , this gives

X X X µ µ
y µ φµk = wi φi φk (6.1.11)
µ i µ

or, in matrix notation,

X X
y µ φµ = φµ (φµ )T w (6.1.12)
µ µ

Hence, the solution is

!−1
X X
µ µ T
w= φ (φ ) y µ φµ (6.1.13)
µ µ

Putting in the actual data, we get a = −0.3091, b = 0.2119. The fit is plotted in
fig(6.2). Although the solution is written in terms of the inverse matrix, we never
actually compute the inverse numerically; we use instead Gaussian elimination –
see the MATLAB code.

6.1.1 Regression and PCA

In an earlier chapter, we discussed using PCA to reduce the dimensionality of

data, based on the idea that data may lie close to a low dimensional hyperplane.
Since a line is a low dimensional hyperplane, one may wonder what the difference
is between using PCA to fit a line and the above regression approach. The answer
is that the objective functions are different. Regression finds a line that minimizes
the vertical distance between a datapoint and the line; PCA finds a line that
minimizes the distance between a datapoint and the line – see fig(6.2).

6.2 Linear Parameter Models (Generalised Linear Models)

A linear parameter model is defined as
y(x) = wT φ(x) (6.2.1)
 46

30 26

24
25 22

20

chirps per sec

chirps per sec
20
18

15 16

14

10 12

10
5
8
65 70 75 80 85 90 95 100 70 75 80 85 90
temperature (F) temperature (F)

Figure 6.2: Left: Straight line regression fit to the cricket data. Right: PCA fit to
the data. In regression we minimize the residuals – the fit represents the shortest
vertical distances. In PCA the fit minimizes the orthogonal projections to the line.

30

25

chirps per sec

20

15

10

65 70 75 80 85 90 95 100
temperature (F)

Figure 6.3: Cubic polynomial fit to the cricket data.

As we saw above, straight line regression fits to data are examples of this. If
we choose the coefficients of the vector φ to be non-linear functions of x, then
the mapping x → y will be non-linear. The phrase “linear” model here refers
to the fact that the model depends on its parameters in linear way. This is an
extremely important point. Unfortunately, the terminology is a little confused in
places. These models are often referred to as “generalised linear models”. However,
sometimes people use this same phrase to refer to something completely different
– beware!

6.2.1 Training LPMs

In the derivation above, there was nothing specific about the form of φ. Hence,
the solution in equation (6.2.5) holds in general. That is, you simply put in a
different φ vector if you wish to find a new solution. For example, consider the
case of fitting a cubic function y = w1 + w2 x + w3 x2 + w4 x3 to the data. In this
case, we would choose

T
φ = 1, x, x2 , x3 (6.2.2)

The solution has the same form, except w is now a 4 dimensional vector
The above MATLAB code implements LPM in general. All that needs to be
changed in the above code for a different model is the function phi_fn. Note that,
rather than using the inv function in MATLAB to solve the linear equations, it
is much better to use the slash function \ – this implements Gaussian elimination
 47

to solve linear systems. This is both much faster and more accurate. As a rule we
never invert matrices unless you need to – and you never need to if you only want
to solve the linear system.
Choosing between Different How would we decide if a straight line fit is preferable to a cubic polynomial fit?
Models We saw in the previous chapter that a general way to address this problem is to
use some validation data to test how accurate each model predicts the validation
data. The more accurate model on the validation data would then be preferred.

6.2.2 Regularisation and numerical stability

It should be fairly clear from the above that all polynomial regression fits are
simply special cases of LPMs. Also, the more terms there are in polynomial, the
more curved can be the fit to the data. One way to penalise too complex models
is to use a penalty term

Ereg (w) = wT w (6.2.3)

The regularised training error is then

P
X
Eregtrain (w, λ) = (y µ − wT φµ )2 + λwT w (6.2.4)
µ=1

If we differentiate the regularised training error to find the optimal w for a given
λ, we find: Hence, the solution is
!−1
X X
µ µ T
w= φ (φ ) + λI y µ φµ (6.2.5)
µ µ

where I is the n × n identity matrix and n = dim(w). Another beneficial aspect

of using a quadratic penalty term is that the solution is more numerically stable
– this can be a problem in cases where there is limited training data. We can
determine λ by using a validation set.

6.2.3 Higher Dimensional Outputs

It is straightforward to generalise the above framework to cases where there is

more than one output variable – rather there is an output vector y:

yi (x) = wiT φ(x) (6.2.6)

The mathematics follows similarly to before, and this is left as an exercise for the
interested reader.

6.2.4 Classification

One way to adapt the LPM model to classification is to use p(c = 1|x) = σ(wT φ(x)).
The logistic regression model simply used a special case in which the vector
φ(x) = x. However, there is nothing to stop us using this more general method.
The nice thing is that the decision boundary is then a non-linear function of x.
Clearly, instead of using the euclidean square distance as the error measure, we
now use the log-likelihood, exactly as in the chapter on logistic regression. Again,
 48
1 1.5

0.9

1
0.8

0.7
0.5
0.6

t
0.5

0.4
−0.5
0.3

0.2 −1

0.1
−1.5
0 0.2 0.4 0.6 0.8 1
0
−1 −0.5 0 0.5 1 1.5 2 x

Figure 6.4: Left: A set of radial basis functions, α = 5, with m =

−1, −0.8, −0.6, . . . , 2. Right: Data to be fitted. The × are the training points,
and the + are the validation points.

however, the training to find w will not be so straightforward, since the objective
function is not quadratic. However, the surface remains well behaved so that find-
ing a solution is not numerically difficult. We leave it as an exercise for the reader
to work out the details.

6.3 Radial Basis Functions

A popular choice for the vector φ(x) is the radial basis function :
2
φi (x) = e− 2α2 (x−m )
1 i
(6.3.1)

where the vectors mi , i = 1, . . . , M define M centres in the input space. The

parameter α determines the width of each basis function. These basis functions
are bump shaped, with the position of the bump being given by m and the width
by α. An example is given in fig(7.8)(Left) in which several RBFs are plotted. In
regression, we can then use a linear combination of these “bumps” to fit the data.
For example consider fitting the data in fig(7.8)(Right).
Setting α We use the validation data to set α. Throughout these experiments, I set the
regularisation parameter λ = 0.0001. In principle one could use the validation set
to optimise over both α and λ. In fig(6.5) we plot the validation error as a function
of α. Based on this graph, we can find the best value of α; that which minimises
the validation error. The predictions are also given in fig(6.5).

6.4 The curse of Dimensionality

We saw that using radial basis functions we can get good predictions, provided
that we choose appropriate basis functions (set the widths correctly). It seems
intuitively clear that if the data has non-trivial behaviour over some region in
x, then we need to cover the region of x space fairly densely with “bump” type
functions. In the above case, we used 16 basis functions for this one dimensional
space. In 2 dimensions, we can also use bump type functions. However, we now
need to cover a 2 dimensional space. If we wish to cover it to the same discretisation
level, we would need 162 = 256 basis functions. In an n dimensional input space,
we would need 16n functions. This is an extremely rapidly growing function of n
so that in 10 dimensions, we would need 1610 = 1012 basis functions. This means
we would have to solve linear systems in 1012 variables! This cannot be easily
 49

8
1.5
7

1
6

validation error
5 0.5

4
0
3

−0.5
2

1 −1

0
0 0.2 0.4 0.6 0.8 1 −1.5
alpha 0 0.2 0.4 0.6 0.8 1

Figure 6.5: Left: The validation error as a function of the basis function width.
Right: The predictions. The solid line is the correct underlying function sin(10x);
the dashed line is the best predictor based on the validation set. The dotted line
is the worst predictor based on the validation set.

done. This explosion in the apparent number of basis functions required is the
famous “curse of dimensionality”.
A possible solution is to make the basis functions very broad to cover more of the
high dimensional space. However, this will mean a lack of flexibility of the fitted
function. Another approach is to place basis functions centred on the training
input points that we have, and add some more basis functions randomly placed
close to the training inputs. The rational behind this is that when we come to do
prediction, we will most likely see novel x that are close to the training points –
we do not need to make “accurate” predictions over all the space.
A further approach is to make the positions of the basis functions adaptive, allow-
ing them to be moved around in the space to minimise the error. This approach
motivates the neural network models.
The criticism of the curse of dimensionality is, in my humble opinion, rather weak,
and good results are often obtained by using basis functions which are dense around
the training inputs.

6.5 Summary
• Linear Parameter models are regression models that are linear in the para-
meters.
• They are very easy to train (no local minima).
• Criticised in high dimensional input spaces due to the curse of dimensionality.
• Judicious placement of the basis functions on close to the training inputs is
a workaround for the curse of dimensionality. Otherwise we need to optimise
the placement of the basis functions – that is, use neural networks.
• Easily adapted to classification (though the training is now more difficult
and needs to be solved using optimisation).
7 Layered Neural Networks

7.1 Sequential Layered Processing

In natural systems, information processing is often found to occur in stages. For
example, in human vision, the light falling on the retina is transformed first in a
non-linear logarithmic fashion. Local parts of the image are then “recognised” by
neurons specialised to respond to particular local image patterns. The information
from these “feature” extraction neurons is then fed into subsequent layers which
correspond to higher cognitive functions. Artificial layered networks mimic such
sequential processing.
In this chapter, we shall consider that each “neuron” or processing unit computes
a deterministic function of its input. In this sense :Neural Networks are graphical
representations of functions.

7.2 The Perceptron

The perceptron is essentially just a single neuron like unit that computes a non-
linear function y of its inputs x,
 
X

y =g wj xj + µ = g wT x + µ (7.2.1)
j

where the weights w encode the mapping that this neuron performs. Graphically,
this is represented in fig(7.1). We can consider the case of several outputs as
follows:
 
X
yi = g  wij xj + µi 
j

and can be used to model an input-output mapping x → y, see fig(7.1)(right).

Coupled with an algorithm for finding suitable weights, we can use a perceptron
for regression. Of course, the possible mappings the perceptron encodes is rather
restricted, so we cannot hope to model all kinds of complex input-output mappings

x1 x2 x3 x4 x5 x1 x2 x3 x4 x5

y y1 y2

Figure 7.1: (Left) A simple perceptron. We use square boxes to emphasise the
deterministic nature of the network. (Right) We can use two perceptrons with
weights w1 and w2 to model a mapping (x1 , x2 , x3 , x4 , x5 ) → (y1 , y2 )

50
 51

(a) A linearly separable (b) A non-linearly separable

problem problem

Figure 7.2: Linear separability: The data in (a) can be classified correctly using a
hyperplane classifier such as the simple perceptron, and the data is termed linearly
separable. This is not the case in (b) so that a simple perceptron cannot correctly
learn to classify this data without error.

x1 xn

l1 lr

h1 hk

m1 ms

y1 ym

Figure 7.3: A multilayer perceptron (MLP) with multiple hidden layers, modeling
the input output mapping x → y. This is a more powerful model than the single
hidden layer, simple perceptron. We used here boxes to denote the fact that the
nodes compute a deterministic function of their inputs.

successfully. For example, consider the case in which g (x) = Θ (x) – that is, the
output is a binary valued function (Θ(x) = 1 if x ≥ 0, Θ(x) = 0 if x < 0) . In this
case, we can use the perceptron for binary classification. With a single output we
can then classify an input x as belonging to one of two possible classes. Looking
at the perceptron,
P equation (7.2.1), we see that we will classifyP the input as being
in class 1 if w x
j j j + µ ≥ 0, and as in the other class if j wj xj + µ < 0.
Mathematically speaking, the decision boundary then forms a hyperplane in the x
space, and which class we associate with a datapoint x depends on which side of
the hyperplane this datapoint lies, see fig(7.2).

7.3 Multilayer Perceptrons

If the data that we are modeling is not linearly separable, we have a problem since
we certainly cannot model this mapping using the simple perceptron. Similarly,
for the case of regression, the class of function mappings that our perceptron forms
is rather limited, and only the simplest regression input-output mappings will be
able to be modelled correctly with a simple perceptron. These observations were
pointed out in 1969 by Minsky and Papert and depressed research in this area for
several years. A solution to this perceived problem was eventually found which
included “hidden” layers in the perceptron, thus increasing the complexity of the
 52

0.8

0.6

0.4

0.2

−0.2

−0.4 sigma(x)
tanh(x)
−0.6 exp(−0.5 x2)

−0.8

−1
−10 −5 0 5 10
x

Figure 7.4: Common types of transfer functions for neural networks.

x1 x2

h1 h2

Figure 7.5: A MLP with one hidden layer.

Transfer Functions mapping. Each hidden node computes a non-linear function of a weighted linear
sum of its inputs. The specific non-linearity used is called the transfer function. In
principle, this can be any function, and different for each node. However, it is most
common to use an S-shaped (sigmoidal) function of the form σ(x) = 1/(1 + e−x ).
This particular choice is mathematically convenient since it has the nice derivative
property dσ(x)/dx = σ(x)(1 − σ(x)). Another popular choice is the sigmoidal
1 2
function tanh(x). Less “biological” transfer functions include the Gaussian, e− 2 x ,
see fig(7.4). For example, in fig(7.3), we plot a simple single hidden layer function,



h1 = σ w1T x + b1 , h2 = σ w2T x + b2 , y = r(v T h + b3 ) (7.3.1)

where the adaptable parameters are θ = {w1 , w2 , v, b1 , b2 , b3 }. Note that the out-
put function r(·) in the final layer is usually taken as the idendity function r(x) = x
in the case of regression – for classification models, we use a sigmoidal function.
The biases, b1 , b2 are important in shifting the position of the “bend” in the sigmoid
function, and b3 shifts the bias in the output.
Generally, the more layers that there are in this process, the more complex becomes
the class of functions that such MLPs can model. One such example is given in
fig(7.3), in which the inputs are mapped by a non-linear function into the first layer
outputs. In turn, these are then fed into subsequent layers, effectively forming new
inputs for the layers below. However, it can be shown that, provided that there
are sufficiently many units, a single hidden layer MLP can model an arbitrarily
complex input-output regression function. This may not necessarily give rise to
the most efficient way to represent a function, but motivates why we concentrate
mainly on single hidden layer networks here.
 53

7.3.1 Understanding Neural Networks

There are a great number of software packages that automatically set up and train
the networks on provided data. However, following our general belief that our
predictions are only as good as our assumptions, if we really want to have some
faith in our model, we need to have some insight into what kinds of functions
neural networks are.
The central idea of neural networks is that each neuron computes some function
of a linear combination of its inputs:
h(x) = g(wT x + b) (7.3.2)
where g is the transfer function, usually taken to be some non-linear function.
Alternatively, we can write
h(x) = g(a(x)) (7.3.3)
where we define the activation a(x) = wT x + b. The parameters for the neuron are
the weight vector w and bias b. Each neuron in the network has its own weight
and bias, and in principle, its own transfer function. Consider a vector w⊥ defined
to be orthogonal to w, that is, wT w⊥ = 0. Then

T
a(x + w⊥ ) = x + w⊥ w + b (7.3.4)

T
= xT w + b + w⊥ w (7.3.5)
| {z }
0
= a(x) (7.3.6)
Since the output of the neuron is only a function of the activation a, this means
that any neuron has the same output along directions x which are orthogonal to
w. Such an effect is given in fig(7.6), where we see that the output of the neuron
does not change along directions perpendicular to w. This kind of effect is general,
and for any transfer function, we will always see a ridge type effect. This is why
a single neuron cannot achieve much on its own – essentially, there is only one
direction in which the function changes (I mean that unless you go in a direction
which has a contribution in the w direction, the function remains the same). If
the input is very high dimensional, we only see variation in one direction.
Combining Neurons In fig(7.7) we plot the output of a network of two neurons in a single hidden layer.
The ridges intersect to produce more complex functions than single neurons alone
can produce. Since we have now two neurons, the function will not change if we go
in a direction which is simultaneously orthogonal to both w1 and w2 . In this case,
x is only two dimensional, so there is no direction we can go along that will be
orthogonal to both neuron weights. However, if x where higher dimensional, this
would be possible. Hence, we now have variation along essentially two directions.

In general, if we had K neurons interacting in a single hidden layer in this way, we

would essentially have a function which can vary along K independent directions
in the input space.

7.4 Training multi-layered perceptrons

To a statistician, neural networks are a class of non-linear (adaptive basis function)
models. Let us consider, for convenience, only a single output variable y. Given
 54

1
1

0.8
0.5

0.6

0 0.4
1
0.2

0.5 0
1

0 0.5

0
1
−0.5 1
0.8 0.5
0.6
0.4 −0.5
0.2 0
x(2) 0
−0.2
−0.4 −0.5
−1 −0.8
−0.6 x(2) −1
−1 −1
x(1) x(1)

Figure 7.6: The output for a single neuron, w = (−2.5, 5)T , b = 0. Left: The
network output using the transfer function exp(−0.5x2 ). Right: using the transfer
function σ(x). Note how the network output is the same along the direction
perpendicular (orthogonal) to w, namely w⊥ = λ(2, 1)T .

1.5 2

1 1.5

0.5
1

0
0.5
1

0
0.5 1

0.5
0

0
−0.5
1
−0.5 0.5
x(2) 0.8 1 0
−1 0.2 0.4 0.6
−0.4 −0.2 0 x(2) −0.5
−1 −0.8 −0.6 −1
−1
x(1) x(1)

Figure 7.7: The combined output for two neurons, w1 = (−5, 10)T , b2 = 0, w2 =
(7, 5)T , b2 = 0.5. The final output is linear, with weights v = (1, 1)T and zero bias.
Left: The network output using the transfer functions exp(−0.5x2 ). Right: using
the transfer function σ(x) – this is exactly the function in equation (7.3.1) with r
the identity function.
 55

a set of input-output pairs, D = {(xµ , y µ ), µ = 1, . . . , P }, how can we find appro-

priate “weights” θ that minimise the error that the network makes in fitting this
function? In neural-network terminology, we would define an “energy” function
that measures the errors that the network makes, and then try to minimise this
function with respect to θ.
Regression For example, a suitable choice of energy or error function for regression might be
X 2
Etrain (θ) = (y µ − f (xµ , θ)) (7.4.1)
µ

where f (xµ , θ) is the output of the network for input xµ , given that the parameters
describing the network are θ. We can train this network by any standard (non-
linear) optimisation algorithm, such as conjugate gradient descent.
Classification A suitable choice of energy or error function to minimise for classification is the
negative log likelihood (if y µ ∈ {0, 1})
X
Etrain (θ) = − (y µ log f µ + (1 − y µ ) log(1 − f µ )) (7.4.2)
µ

where f µ = f (xµ , θ). In this case, we would need that the final output r(x) is
bounded between 0 and 1 in order that it represents a probability. The case of
more than two classes is handled in a similar way using the so-called soft-max
function (see Bishops book for references).
Regularisation In principle, the problem of training neural networks is equivalent to the general
statistical problem of fitting models to data. One of the main problems when
fitting complex non-linear models to data is how to prevent “overfitting”, or, more
generally, how to select the model that not only fits the data, but also generalises
well to new data. We have already discussed this issue in some generality, and
found that one approach is to use a penalty term which encourages smoother
functions. In the case of MLPs, smoother functions can be encouraged if we
penalise large weight values. The reason for this is that the larger the weights wi
are, the more rapidly the function can change as x changes (since we could flip
from close to one near saturated region of the sigmoid to the other with only a
small change in x).
A term which penalises large weights,

Eregtrain (θ) = Etrain (θ) + λθT θ (7.4.3)

We can set λ as usual by using a validation set.

7.4.1 Single Hidden Layer

A MLP with a single hidden layer is

K
!
X
f (x, θ) = r vi g(wi ·x + bi ) + b (7.4.4)
i=1

an example of which is given in fig(7.3).

Regression In the case of regression, we would use an output function r to be the identity,
 56

and the squared output to form the error1:

P
X K
X
E(θ) = (f (xµ , θ) − y µ )2 + λ (wk )T wk (7.4.5)
µ=1 k=1

To use the conjugate gradients algorithm to optimise this objective function, we

need to know the derivatives with respect to all the parameters ∂E/∂θi .

∂E XP
∂f (xµ , θ) X dim(w
Xk) ∂wj,k
=2 (f (xµ , θ) − y µ ) +2 wj,k (7.4.6)
∂θi µ=1
∂θi j=1
∂θi
k

The final term is zero unless we are differentiating with respect to a parameter
that is included in the regularisation term. If θi is included in the regularisation
term, then the final term simply is 2θi . All that is required then is to calculate the
derivatives of f with respect to the parameters. This is a straightforward exercise
in calculus, and we leave it to the reader to show that, for example,
∂f (xµ , θ)
= g(w1T xµ + b1 ) (7.4.7)
∂v1
and
∂f (xµ , θ)
= v2 g ′ (w2T xµ + b2 )xµ1 (7.4.8)
∂w1,2

where g ′ (x) is the derivative of g(x). Example code for regression using a single
hidden layer is given below. It is straightforward to adapt this for classification.
This code is not fully vectorised for clarity, and also uses the scg.m function, part
of the NETLAB (see http://www.ncrg.aston.ac.uk) package which implements
many of the methods in these chapters.

7.4.2 Back Propagation

In computing the gradient of the error function, naively it appears that we need of
the order of P W 2 operations (if W is the number of parameters in the model and
P is the number of training patterns), since computing the output of the network
involves roughly W summations for each of the P patterns, and the gradient is
a W -dimensional vector. The essence of the backpropagation procedure is that
the gradient can instead by computed in order P W operations. If the training
set is very large, standard computation of the gradient over all training patterns
is both time-consuming and sensitive to round-off errors. In that case, “on-line
learning”, with weight updates based on the gradient for individual patterns, offers
an alternative. Back propagation is most useful in cases where there are more than
one hidden layer in the network. In this case, the gradient can be computed more
efficiently, and time saved therefore to find the optimal parameters.

7.4.3 Training ensembles of networks

A problem with neural networks is that they are difficult to train. This is because
the surface of the error function E(θ) is very complicated and typically riddled
with local minima. No algorithm can guarantee to find the global optimum of
1 There is no need to penalise the biases, since they only really affect a translation of the
functions, and don’t affect how bumpy the functions are.
 57

the error surface. Indeed, depending on the initial conditions that we use, the
parameters found by the optimisation routine will in general be different. How
are we to interpret these different solutions? Perhaps the simplest thing to do
is to see which of the solutions has the best error on an independent validation
set. Many algorithms have been proposed on how to combine the results of the
separate networks into a single answer and for computing error bars that indicate
the reliability of this answer. Imagine that we have used optimisation several
times, and found the different solutions θ i , i = 1, . . . , M . One simple approach
(for regression) is to combine the outputs of each of the trained models,
M
1 X
f¯(x) = f (x, θ i ) (7.4.9)
M i=1

This is also useful since we can make an estimate of the variance in the predictions
at a given point,
M
1 X
2
var(f (x) = f (x, θi ) − f¯ (7.4.10)
M i=1
p
This can then be used to form error bars f¯(x) ± var(f (x)).

7.5 Adaptive Basis Function Networks

Linear weighted inputs In neural networks, typically the output of each node (or neuron) in the network is
some non-linear function of a linear combination of the nodes entering the network
(the parents). That is,
 
X
yi = gi  wij xj + µi  (7.5.1)
j

As previously discussed, because the output of the node only depends on a linear
combination of the inputs to the network node/neuron, essentially there is only
variability in one direction in the input space (where by input I mean the inputs
to the node). We can make a bump, but only a one dimensional bump, albeit in
a high dimensional space. To get variability in more than one direction, we need
to combine neurons together. Since it is quite reasonable to assume that we want
variability in many dimensions in the input space, particularly in regions close to
the training data, we typically want to make bumps near the data.

7.5.1 Adaptive Basis Functions

In the case of linear parametric models, we saw how we can approximate a func-
tion using a linear combination of fixed basis functions. Localised Radial Basis
Functions(exp(−(x − m)2 )) are a reasonable choice for the “bump” function type
approach. The output of this function depends on the distance between x and the
centre of the RBF m. Hence, in general, the value of the basis function will change
as x moves in any direction, apart from those that leave x the same distance from
m, see fig(7.8). Previously, we suggested that a good strategy for placing centres of
basis functions is to put one on each training point input vector. However, if there
are a great number of training patterns, this may not be feasible. Also, we may
wish to use the model for compression, and placing a basis function on each train-
ing point may not give a particularly high compression. Instead we could adapt
 58

0.5 1.5

1
1
0
1
0.5
0.8 0.5
0.6 1
0.4 0
0.2 0.5 0
0 1
−0.2 0 0.5 −0.5
−0.4 0
−0.6 −0.5 −0.5 x(1)
−0.8 −1
x(2) −1 −1 x(1) x(2)
−1

2
Figure 7.8: Left: The output of an RBF function exp(− 21 x − m1 /α2 ). Here
m1 = (0, 0.3)T and α = 0.25. Right: The combined output for two RBFs, m2 =
(0.5, −0.5)T .
1.5

0.5

−0.5

−1
−3 −2 −1 0 1 2 3

Figure 7.9: A RBF function using five basis functions. Note how the positions of
the basis function centres, given by the circles, are not uniform.

the positions of the centres of the basis functions, treating these also as adaptable
parameters. In general, an adaptive basis function network is of the form
X
y(x, θ) = wi φi (x, bi ) (7.5.2)
i

where now each basis function φi (x, bi ) has potentially its own parameters that
can be adjusted. θ represents the set of all adjustable parameters. If the basis
functions are non-linear, then the overall model is a non-linear function of the
parameters.

7.6 Training Adaptive Basis Functions

Let us consider, for convenience, only a single output variable y. Given a set
of input-output pairs, D = {(xµ , y µ ), µ = 1, . . . , P }, how can we find appropriate
paramters θ that minimise the error that the network makes in fitting this function?
Regression A suitable choice of energy or error function for regression is
X
Etrain (θ) = (y µ − f (xµ , θ))2 (7.6.1)
µ
 59

We can train this network by any standard (non-linear) optimisation algorithm,

such as conjugate gradient descent.

However, one should always bear in mind that, in general, the training of complex
non-linear models with many parameters is extremely difficult.
Classification A suitable choice of energy or error function to minimise for classification is the
negative log likelihood (if y µ ∈ {0, 1})
X
Etrain (θ) = − (y µ log f µ + (1 − y µ ) log(1 − f µ )) (7.6.2)
µ

where f µ = f (xµ , θ).

Regularisation The smootheness of the RBF mapping is mainly determined by the width of the
basis functions. The easiest approach is to use a validation set to determine α and
not to regularise any of the other parameters.
Initialisation The quality of the solutions is critically dependent on the initial parameter settings,
in particular where we initially speculatively place the basis function centres.
One reasonable initialisation strategy is to place the centres on a randomly chosen
subset of the data, and then solve for the hidden to output weights w easily (this
is just a linearised parameter model if we consider the basis functions fixed).
Another approach is to use K-means clustering (see later chapter) to set the centres
of the basis functions. Given the initial centres of the basis functions, we can solve
for the weights easily. This gives an initial setting for the basis function and weight
values.
Optimisation Strategies Perhaps the most obvious thing to do is to treat both the weights w and basis
function parameters bi together as one parameter set, θ, and optimise the objective
function with respect to θ. Example code for regression using a this approach is
given below. It is straightforward to adapt this for classification. This code is not
fully vectorised for clarity, and also uses the scg.m function, part of the NETLAB
(see http://www.ncrg.aston.ac.uk) package.
An example is given in fig(7.9) where we see that the optimal solution (as found by
the optimisation algorithm) produces a non-uniform placing of the basis function
centres. However, there is another strategy which, in practice, may be preferable:
1. For fixed basis function parameters bi , find the best weights w (this is easy
to do since this is just corresponds to solving a linear system).
2. For fixed weights w, find the best basis function parameters. (This is the
difficult step since there will typically be many basis function parameters,
and the objective function depends in a highly non-linear way on the basis
function parameters).
We can iterate these two stages to improve the solution. The slight practical
advantage of this is that the parameter space in which we search for a solution to
a non-linear optimisation problem is slightly reduced since we only optimise with
respect to the bi parameters.
 60

7.6.1 Non-local Basis Functions

If we use basis functions that decay rapidly from a ‘centre’, as in the case exp(−(x−
m)2 ), the basis function value will always decay to zero once we are far away from
the training data. In the case of binary classification and a logistic sigmoid for
the class output, this may be reasonable since we would then predict any new
datapoint far away from the training data with a complete lack of confidence, and
any assignment would be essentially random. However, in regression, using say
a linear combination of basis function outputs would always give zero far from
the training data. This may give the erroneous impression that we are therefore
extremely confident that we should predict an output of zero far away from the
training data whilst, in realilty, this is simply an artefact of our model. For this
reason, it is sometimes preferable to use basis functions that are non-local – that
is, they have appreciable value over all space, for example, (x − m)2 log((x − m)2 ).
Whilst any single output will tend to infinity away from the training data, this
serves to remind the user that, far from the training data, we should be wary of
our predictions.

7.7 Committees
Drawbacks of the non-linear approaches we have looked at – neural networks and
their cousins adaptive basis functions – are
1. Highly complex energy/error surfaces give rise to multiple solutions since
global optima are impossible to find.
2. We have no sense of the confidence in the predictions we make (particularly
in regression).
Whilst there are alternative (and in my view more attractive) approaches around
these problems, we can exploit the variability in the solutions found to produce
a measure of confidence in our predictions. The idea is to form a committee
of networks from the solutions found. For example, for regression, we could train
(say) M networks on the data and get M different parameter solutions θ1 , . . . , θM .
The average network function would then be
M
¯ 1 X
f (x) = f (x, θ i ). (7.7.1)
M i=1

A measure of the variability in the predictions is given by the variance :

M
1 X
var(f )(x) = (f (x, θ i ) − f¯(x))2 . (7.7.2)
M i=1

A useful plot of confidence in our predictions is then to use one standard deviation
error bars :
p
f¯(x) ± var(f )(x) (7.7.3)

In fig(7.10) we give an example using a committee of six adaptive basis functions.

The committee idea is quite general and applicable to any model. Whilst this
approach is rather heuristic and leaves some questions unanswered (why did we
choose uniform weighting of the solutions for example) it is nevertheless an intuitive
 61
1.5 1.5

1 1

0.5 0.5

0 0

−0.5 −0.5

−1 −1

−1.5 −1.5
−3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3

Figure 7.10: Left: A single solution using Adaptive Basis Functions to fitting the
training data (crosses). The centres of the five basis functions are given by the
circles. Right: A committee prediction from six individual solutions of the form
given on the left. The central line is the average prediction – note how this still
decays to zero away from the training data. The lines around the central line a
one standard deviation confidence intervals.

an reasonably robust way of gaining confidence in model predictions (these issues

can be solved by a Bayesian treatment beyond the scope of this course). Note
that the committee approach does not necessarily solve the issue of over confident
regression predictions away from training data. As seen in fig(7.10) both the mean
and confidence will collapse around zero as we move far from the training data.
This is a good reason to use non-local basis functions in this case since typically,
the variability will become unbounded as we move away from the training data.

7.8 Summary and Outlook

The field of neural networks has contributions from and makes contributions to
many different areas. Although, ultimately, the motivation for much research has
been biological, there are many areas in which artificial neural networks can and
have been used successfully. More specifically, these include areas where the un-
derlying process behind data generation is highly non-linear, and there now exists
techniques that are able to give some confidence and insight into the performance
of such models.

Two features of artificial neural networks stand out as being of particular impor-
tance – their non-linearity, and stochasticity (although this latter aspect is not
always exploited in many applications). These properties can be used to define
local computation units which, when coupled together suitably, can combine to
produce extremely rich patterns of behaviour, whether these be dynamic, or static
input-output relationships. One of the most import consequences of neural net-
work research has been to bring the techniques and knowledge of artificial intelli-
gence and statistics much closer together. Typically it was the case that problems
in artificial intelligence were tackled from a formal specification aspect. On the
other hand, statistics makes very loose formal specifications, and lets the data try
to complete the model. Neural networks can be seen as a statistical approach
to addressing problems in artificial intelligence, obfuscating the need for formal
specifications of how the program works – just learn how to do it from looking
at examples. For example, rather then formally specifying what constitutes the
 62

figure “2”, a neural network can learn the (statistical) structure of “2”s by being
asked to learn (find appropriate weights for) how to differentiate between “2”s
and non-”2”s. This idea is especially powerful in the many human computer in-
teraction applications where formally specifying, for example, what constitutes an
individuals facial characteristics that differentiate them from others, is extremely
difficult.
 8 Autoencoders

8.1 Introduction
The general idea of autoencoders is that they are simply approximations of the
identity mapping. We do not really need to invoke the concepts of neural networks
to talk about these. However, many applications use neural networks to implement
autoencoders.
Dimension Reduction The major use of autoencoders is in dimension reduction, that is to replace a high
N -dimensional vector x with a lower M -dimensional vector y. Clearly, this only
makes sense when we have a set of data, xµ , µ = 1, . . . , P .

8.1.1 Linear Dimension Reduction (PCA)

In linear dimension reduction, y = Wx + b, where W is a M × N matrix and b

is an M dimensional vector. We have already encountered one method of linear
dimension reduction, PCA,

y = ET (x − m) (8.1.1)

where E is the M × N matrix whose P columns are the largest eigenvectors

P of the
T
covariance matrix of the data, C = P1 µ (xµ − m) (xµ − m) , m = P1 µ xµ .
The reconstruction of a higher dimensional vector, using the lower dimensional
PCA representation y is

x̃ = m + Ey (8.1.2)

PCA was shown to be optimal in terms of squared reconstruction error, E =

P µ µ 2
µ (x̃ − x ) . We can represent the process of the mapping x → y → x̃ using
a single hidden layer neural network, as in fig(8.1), in which the transfer function
of each node (neuron) is the identity, g(s) = s. Such a network is known as an
autoencoder, since the aim is to reproduce at the output layer, the same vector as
at the input. That is, the network should try to encode as accurately as possible
the identity mapping x → x. The reason that this is non-trivial is that there is
a reduced number of dimensions in the hidden layer, creating a lower dimensional
“bottleneck” through which information must be squeezed. It is the activations
of the units in this lower dimensional layer that we can then use to represent the
higher dimensional data.

8.1.2 Manifolds : The need for non-linearity

Consider the situation in fig(8.2), where a “piece of paper” has been wrinkled to
form a three dimensional object. However, to describe exactly the position of any
point on the surface of the paper, we only need two co-ordinates, namely how
far to go along y1 and how far to go along y2 . Of course, the actual position
of the surface point is a three dimensional vector, but it is only a function of y1
and y2 . Clearly, in this case, x is a non-linear function of y. A manifold is a

63
 64

higher dimensional generalisation of the idea of a “piece of paper” – that is, we

can describe points in a higher dimensional space using a lower dimensional co-
ordinate system. (More correctly, a manifold is described by potentially a set of
overlapping pieces of paper). In general, we might hope that data lies on such
lower dimensional manifold. To discover this, or at least to approximate it, we
need to make a non-linear model.
Non-linear transfer functions? A first attempt to make a non-linear manifold would be to use the same autoen-
coder network as for PCA, however, now replacing the identity transfer function
in the hidden layer with a non-linear function. It is a well known result (a little
beyond the scope of these lectures) that this will not result in a better reconstruc-
tion error. That is : for single hidden layer autoencoders, the optimal minimum
squared reconstruction error solution is always PCA, regardless of the transfer
function used.

8.2 Non-linear Dimension Reduction

Non-linear Autoencoder To obtain a more powerful model than PCA, we need both a non-linearity in the
hidden layer transfer functions, and in the output layer transfer functions. To see
this, consider a simple two dimensional manifold (see fig(8.3)):
T
x = (y1 , sin(y1 + y2 ), cos(y1 )) (8.2.1)

In this case the optimal hidden layer activations would be y1 = x1 , y2 = sin−1 (x2 )−
x1 . Clearly, there are other possibilities available. Given y1 and y2 , to make our
reconstruction, we use
T
x̃ = (y1 , sin(y2 ), cos(y1 )) (8.2.2)

If we use a neural network (by which we mean that the outputs of the hidden
units are non-linear functions of a weighted linear combination of the units inputs),
both the hidden unit transfer functions and output transfer functions need to (in
general) be non-linear. Note that the above would need more than one hidden layer
to be represented by an autoencoder. Graphically, we can represent a multiple
hidden layer neural network autoencoder as in fig(8.4). In principle, no restriction
on the form of the mappings from layer to layer need be made. However, it is
common to use non-linear perceptron like mappings from layer to layer, so that
the output of each node is a non-linear function of its linearly weighted inputs.

8.2.1 Training Autoencoders

The standard approach to training autoencoders is to use the sum squared recon-
struction error. If θ are the parameters of the autoecoder, then the autoencoder

x1 xn

y1 ym

x̃1 x̃n

Figure 8.1: Autoencoder with a single hidden layer. If minimal reconstruction

error is used, the optimal solution is given by PCA.
 65

x3
y1 y2

x2

x1

Figure 8.2: A two dimensional manifold embedded in a three dimensional space.

expresses a mapping f (x, θ). Since we want the output to resemble the input as
closely as possible, we form the error:
P
X 2
E(θ) = (xµ − f (x, θ)) (8.2.3)
µ=1

We then minimise this error with respect to the parameters θ. Normally, we

will use an optimisation routine like scaled conjugate gradients to find the best
parameters. This requires us to calculate the derivatives ∂E/∂θi . To calculate
these derivatives efficiently, we use the back-propagation routine. Essentially, back-
propagation just makes efficient use of the fact that information is passed between
layers. We will not go into the details of back-propagation in this course, but this
is an important issue in the practical training of networks, since it is much more
efficient than calculating the derivatives in a naive way. There are many software
packages that implement neural networks, and all use back-propagation.

8.3 Uses of Autoencoders

Preprocessing Preprocessing of inputs x can have a major impact on the quality of the mapping
that we wish to learn from inputs to targets. One of the most common pre-
processing steps is dimension reduction, such as PCA. We can use the hidden unit

0.8

0.6

0.4

0.2

−0.2

−0.4

−0.6

−0.8

−1
1 5
0.5
0 0
−0.5
−1 −5

Figure 8.3: A two dimensional manifold embedded in a three dimensional space.

 66

x1 xn

l1 lr

h1 hk

m1 ms

x1 xn

Figure 8.4: Autoencoder with multiple hidden layers. This is a more powerful
autoencoder than the single hidden layer case, only provided that the hidden to
output layers encode a non-linear function.
30 30 30 30
1 4 11
22 29
1 29
14 28 28
27 27
0.8
26 26
25 25 25 25
5 24 24
0.6
17 23 23
21
25 22 22
0.4 21 21
3 30 20 20
20 20
719
10
6 19
19
0.2 18 18
17 17
16 16
0 12 15 15 15 15
14 14
9 13 13
−0.2 12 12
27 11 11
29 10 10 10 10
−0.4 9 9
23
20 8 8
7 7
−0.6 2 6 6
5 5 5 5
24 4 4
−0.8 8 3 3
16 2 2
28
13 1 1
−1 18 26 15 0 0
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1

Figure 8.5: A two dimensional data set (left) represented by a one dimensional
PCA (middle) and one dimensional autoencoder (right). Note that in the middle
and right plots, the y axis is irrelevant and simply used to aid visual separation of
the data.

activation values in an autoencoder to form a lower dimensional representation y

of the input x, and then use this as our new input for any subsequent processing.
Hence, autoencoders can be used for non-linear dimension reduction.
Visualisation Visualisation of data can be very useful, and is a topic to which we shall return
in a later chapter. If we squeeze the data through an autoencoder in which the
bottleneck is only of two or three dimensions, then we can plot the resulting two
or three dimensional points, and get a feeling for how the low dimensional (and by
implication also the high-dimensional) representations of the data are distributed.

8.3.1 A Visualisation example

In fig(8.5)(left) we plot the original two dimensional data. In fig(8.5)(middle) we

plot the one dimensional data PCA representation (the y axis is irrelevant and
simply used to make the plot more readable). In fig(8.5)(right) we plot the one
dimensional data autoencoder representation using a 2 − 4 − 1 − 4 − 2 autoencoder
architecture.
9 Data Visualisation

In data visualisation we attempt to gain intuition about the structure of a dataset.

Typically, this method is unsupervised (does not make use of any target values),
although supervised visualisation is also possible. We have seen how to use PCA
to reduce the dimensionality of data in a linear fashion to such a degree that we
can plot the reduced dimension dataset in two or three dimensions. Canonical
Variates (see elsewhere) also performs linear dimension reduction exploiting class
information – if we reduce the dimension to only two or three, we can also visualise
the data. Non-linear dimension reduction, such as autoencoders can also be used in
the same way for visualisation. In autoencoders, we constructed the error function
to be the squared error loss between the input and the output. However, there was
no explicit requirement that the low dimensional representation of the data should,
in some sense, be a good visualisation of the high dimensional data. This issue is
addressed here by considering methods that try to preserve (at least locally) the
topology of the high dimensional data.
Multidimensional Scaling In multidimensional scaling (MDS) we are given distances drs between every pair
of observations (that is, we may not have direct access to any high-dimensional
data, but we do have access to a measure of the “distances” between every two
points). The idea is to try to reconstruct what the original data was, based solely
on these distances. For example, given only the distances between towns, can we
construct a map for the coordinates of the towns themselves? A practical area for
such methods is in the visualisation of proteins and other macromolecules based
on measures of similarity between the molecules.

9.1 Classical Scaling

Consider a set of datapoints {xµ , µ = 1, . . . , P }, where the dimension of each dat-
apoint, dim(xµ ) = n. From this data we can form the distances between all the
datapoints:
x
Tab = (xa − xb )2 (9.1.1)

to form a P ×P distance matrix Tx between the x datapoints. The idea in Classical

Scaling is to find a set of vectors y µ , µ = 1, . . . , P such that the distance matrix Ty
between the y points matches as closely as possible the distance matrix Tx . The
dimension of the datapoints, dim(y) is typically chosen to be small, either two or
three so that we can visualise the data.
In other words : Given a distance matrix T only, how can we find a set of points
y µ that has this distance matrix?
The interesting thing about Classical Scaling is that the solution to this problem,
for the case of using Euclidean squared distance, is analytic, as described below.

67
 68

9.1.1 Finding the optimal points

Here we briefly describe the mathematics behind the solution of the Classical
Scaling. If we consider a single element of the distance matrix, we have

Tab = xa · xa − 2xa · xb + xb · xb (9.1.2)

For convenience, let us define a matrix

Xij = xij (9.1.3)

Furthermore, define the matrix

E = XX T (9.1.4)

Then we can express one element of T as

Tab = Eaa − 2Eab + Ebb (9.1.5)

If it were not for the terms Eaa and Ebb , life would be easy since, in that case,
we would have a known matrix, T expressed as the outer product of an unknown
matrix, X which would be easy to solve. What we need to do therefore is to express
the unknown matrix elements Eaa and Ebb in terms of the known matrix T . In
order
P ato do this, we make the following extra assumption – the data has zero mean,
a xi = 0. Clearly, this does not
Paffect thePsolution since it is only defined up to
an arbitrary shift. In that case, a Eab = ai xai xbi = 0. Hence,
X X X
Tab = Eaa − 2 Eab + P Ebb (9.1.6)
a a a
X
= Eaa + P Ebb (9.1.7)
a
P
This means that we could express Ebb in terms of T , if only we knew what a Eaa
is. But this can also be obtained by now summing over b:
X X X
Tab = P Eaa + P Ebb (9.1.8)
ab a b
X
= 2P Eaa (9.1.9)
a

This means
X X
P Ebb = Tab − Eaa (9.1.10)
a a
X 1 X
= Tab − Tab (9.1.11)
a
2P
ab

so that
1 X 1 X 1 X
Tab = Tab − 2 Tab + Tab − 2Eab (9.1.12)
P a P P
ab b

In other words, we can write

!
 T
 1 1 X 1 X 1 X
XX ab
=− Tab − Tab + 2 Tab − Tab (9.1.13)
2 P a P P
ab b
 69

Mexico City
Buenos Aires
Honolulu
Los Angeles
Caracas San Francisco

Rio de Janeiro
Chicago
Washington DC
New York
Montreal
Sydney

Lisbon
London Tokyo
Paris
Berlin Stockholm
Rome Warsaw
Moscow
Cape Town Istanbul Shanghai
Manila
Cairo Hong Kong

Calcutta

Figure 9.1: Classical scaling solution to representing 28 world cities on a two

dimensional map, given only their intercity distances.

The right hand side are elements of a now known matrix, T ′ , for which we can
find an eigen-decomposition

T ′ = V ΛV T (9.1.14)

where V is an orthogonal matrix and Λ is diagonal matrix containing the eigen-

values. Since each column of T ′ sums to zero, this matrix has at most rank P − 1.
1
A solution for the data position is given by taking the first r columns of V Λ 2 ,
′ µ
where r is the rank of T . This means that if we have P vectors x , µ = 1, . . . , P
based only on the Euclidean square distances between them, we can reconstruct a
set of P objects in a P − 1 dimensional space that has exactly the same distance
structure.
If we wish to look for lower dimensional approximate reconstructions (in the sense
that the distance matrix in the lower dimensional space will not exactly match the
1
given distance matrix T) , we can simply take those columns of V Λ 2 corresponding
to the largest eigenvalues of T ′ .

9.1.2 The Algorithm

Given a P × P distance matrix T :

1. Calculate the P × P matrix M with elements
!
1 1 X 1 X 1 X
Mab =− Tab − Tab + 2 Tab − Tab (9.1.15)
2 P a P P
ab b

2. Calculate the m largest eigenvalues λi , i = 1, . . . , m of M , and their corre-

sponding eigenvectors ei .
3. The points y j , j√= 1, . . . P in the m dimensional space are then gives by the
positions yij = λi eij .
Example : intercity We are given the intercity distances of 28 major cities in the world. This is therefore
 70

2 11
2 3 3 3 1 1
3 3 111 1
5 1
3 35 3 2 1 9
8
2
3 85 8 86 5
0 5 3
6
26 6 2 9 99
2 8
0 56 0 6
6 92 97
2
0 0 6 65 5 74 77
7
0 4 8 7 4 77
0 0 65 8 8 85 7
0 74
9 4
2 9
0 4
4 94
9
4
4

Figure 9.2: Classical scaling solution to representing digits in two dimensions.

Note how some digits are more closely clustered together than others.

a 28 × 28 dimensional matrix T. The above algorithm is coded below in Matlab

to form a 3 dimensional representation of the cities.
The result is given in fig(9.1) where we have plotted only two of the three dimen-
sions. Note how the representation is roughly correct from our experience of where
cities are in the world.
Example : Digit data We can also use classical scaling to reduce the dimension. I took 10 examples for
each of the ten classes of digit – 100 datapoints therefore in total. Each digit is
represented as a 784 dimensional vector. I then formed the 100 × 100 dimensional
distance matrix T, and used classical scaling to plot the resulting 3 dimensional
reconstructions. The results are given in fig(9.2).
 71

9.2 Sammon Mapping

The Sammon mapping is a technique more general (and more widely used) than
classical scaling. The idea is very simple. Given a P × P dissimilarity matrix dij ,
and a function d(y i , y j ) that measures the dissimilarlity of two vectors y i and y j ,
we look to place objects in a space such that their dissimilarities are close to the
given dissimilarities dij . An objective function to achieve this is

2
1X dij − d(y i , y j )
E= P (9.2.1)
ij dij i<j dij

We minimise E with respect to the positions y i , i = 1, . . . p. The division by dij

is included in order to encourage the solution to deal with small dissimilarities
accurately. (We do not divide by d2ij since then small and large dissimilarities
would be treated roughly equally). In order to train such a model, standard (non-
linear) optimisation techniques can be employed.
For example, we could define dissimilarities as

4
d(y i , y j ) = y i − y j . (9.2.2)

Then, given a set of target dissimilarities dij we then need to arrange the vectors y i
to minimize the (weighted) difference between the given dissimilarities and those
measured above. The parameters of the optimization are therefore the vectors y i
themselves.
Strictly speaking, the Sammon “Mapping” is not a mapping, since it does not yield
a function that describes how general points in one space are mapped to another
(it only describes how a limited set of points is related).
Making a Mapping Given points xi in a n-dimensional space (possibly very high dimensional) to rep-
resent them by points y i in a m-dimensional space (possibly very low dimensional,
say 2) in such a way that the separation between the points in the two spaces is
roughly similar. One way to obtain this mapping is to parameterize the positions
of the objects in the lower dimensional space

y = f (x; W) (9.2.3)

The distance then between two mapped points is a function of the parameters of
the mapping W. The optimal parameters can then be found by optimization. The
method Neuroscale is one such procedure.

9.3 A word of warning

It can be that very high dimensional datasets appear to lie on ring when plotted
using visualisation methods. It may well be that the data really does have this
kind of structure. However, in high dimensions, the distance matrix (between every
pair of points) will be dominated by those points that are furthest apart. This
will give the impression that most points are a long way from each other, and a
ring or circular type two dimensional representation will likely be the visualisation
solution. One should therefore bear in mind that global topological constraints on
the data are unlikely to be accurately represented by these visualisation procedures,
and one should be wary of reading too much into the precise structure of the
visualisation.
II. Inference and Learning in Probabilistic Models

72
 10 Introducing Graphical Models

10.1 Belief Networks

Belief Networks (also called Bayes’ Networks or Bayesian Belief Networks) are
the most straightforward kind of graphical model to introduce[7, 8]. Their use is
widespread and they have tremendous applicability, ranging from troubleshooting
facilities in Microsoft operating systems, expert reasoning under uncertainty to
machine learning in general. We will here consider a simple example of a discrete
variable BN.

10.1.1 Tracey

Tracey lives in sunny Birmingham. One morning she leaves her house and realizes
that her grass is wet. Is it due to rain or has she forgotten to turn off the sprinkler
?
Next she notices that the grass of her neighbour, Jack, is also wet. She concludes
therefore that it has probably been raining, and that “explains away” to some extent
the possibility that her sprinkler was left on.

Making a model

We can model the above situation using probability by following a general mod-
elling approach.
First we define what variables we wish to include in our model. In the above
situation, the natural variables are
R ∈ {0, 1} (R = 1 means that it has been raining, and 0 otherwise).

S ∈ {0, 1} (S = 1 means that it she has forgotten to turn off the sprinkler, and 0
otherwise).

J ∈ {0, 1} (J = 1 means that Jack’s grass is wet, and 0 otherwise).

T ∈ {0, 1} (T = 1 means that Tracey’s Grass is wet, and 0 otherwise).

A model of Tracey’s world then corresponds to a probability distribution on the

joint set of the variables of interest p(T, J, R, S) (note that the order of the variables
is irrelevant).
Since each of the variables in this example can take one of two states, it would
appear that we naively have to specify the values for each of the 24 = 16 states,
e.g. p(T = 1, J = 0, R = 1, S = 1) = 0.7 etc. However, since there are normali-
sation conditions for probabilities, surely, we do not need to specify all the state
probabilities, since some will be determined by normalisation.

73
 74

To see how many states need to be specified in general, consider the following
decomposition. Without loss of generality (WLOG) and repeatedly using Bayes’
rule, we may write1 :
p(T, J, R, S) = p(T |J, R, S)p(J, R, S) = p(T |J, R, S)p(J|R, S)p(R, S)
= p(T |J, R, S)p(J|R, S)p(R|S)p(S)
That is, we may write the joint distribution as a product of conditional distri-
butions. The first term p(T |J, R, S) requires us to specify 23 = 8 values, say
for p(T = 1|J, R, S) given the 8 possible states jointly of J, R, S. The other value
p(T = 0|J, R, S) is given by normalisation : p(T = 0|J, R, S) = 1−p(T = 1|J, R, S).
Similarly, we need 4 + 2 + 1 values for the other factors, making a total of 15 values
in all. In general, for a set of n binary variables, we need to specify 2n − 1 values
in the range [0, 1]. The important point here is that the number of values that
need to be specified in general scales exponentially with the number of variables
in the model – this is extremely bad news, and motivates simplifications.

Conditional Independence

The modeler often knows that certain simplifications often occur. Indeed, it is
arguably the central role of modelling to make the simplest model that fits with
the modelers beliefs about an environment. For example, in the scenario above,
Tracey’s grass is wet dependent only directly on whether or not is has been raining
and or whether or not her sprinkler was on. That is, we make the conditional
independence assumption for this model that
p(T |J, R, S) = p(T |R, S)
Similarly, since whether or not Jack’s grass is wet is influenced only directly by
whether or not is has been raining, we write
p(J|R, S) = p(J|R)
and since the rain is not directly influenced by the sprinkler!
p(R|S) = p(R)
which means that our model now becomes :
p(T, J, R, S) = p(T |R, S)p(J|R)p(R)p(S)
We can represent these conditional independencies graphically, as in fig(10.1).
This reduces the number of values that we need to specify to 4 + 2 + 1 + 1 = 8, a big
saving over the previous 15 values in the case where no conditional independencies
had been assumed.
The heart of modelling is in judging which variables are dependent on each other.
Specifying the values To complete the model, we need to numerically specify the values of the conditional
1 Note that a probability distribution simply assigns a value between 0 and 1 for each of the states
jointly of the variables. For this reason, p(T, J, R, S) is considered equivalent to p(J, S, R, T )
(or any such reordering of the variables), since in each case the joint setting of the variables
is simply an index to the same probability. This situation is more clear in the set theoretic
notation p(J ∩ S ∩ T ∩ R). We abbreviate this set theoretic notation by using the commas
– however, one should be careful not to confuse the use of this indexing type notation with
functions f (x, y) which are in general dependent on the variable order. Whilst the variables
to the left of the conditioning bar may be written in any order, and equally those to the right
of the conditioning bar may be written in any order, moving variables across the bar is not
allowed, so that p(x1 |x2 ) 6= p(x2 |x1 ).
 75

R S

J T

Figure 10.1: Belief network structure for the “wet grass” example. Each node in
the graph represents a variable in the joint distribution, and the variables which
feed in (the parents) to another variable represent which variables are to the right
of the conditioning bar.

probabilty tables (CPTs). Let the prior probabilities for R and S be p(R) =
(0.2, 0.8) (that is, p(rain = yes) = 0.2 and p(rain = no) = 0.8) and p(S) =
(0.1, 0.9). Note, for clarity I use here for example p(R = y) instead of p(R = 1) –
of course, the labels we use for the states are irrelevant. Let’s set the remaining
probabilities to
p(J = y|R = y) = 1, p(J = y|R = n) = 0.2 (sometimes Jack leaves his own
sprinkler on too).

p(T = y|R = y, S) = 1,p(T = y|R = n, S = y) = 0.9 (there’s a small chance

that even though the sprinkler was left on, it didn’t wet the grass noticeably),
p(T = y|R = n, S = n) = 0
The prior belief that the sprinkler is responsible is p(S = y) = 0.1.

Inference

Now that we’ve made a model of an environment, we can perform inference. Let’s
calculate the probability that the sprinkler was on overnight, given that Tracey’s
grass is wet: p(S = y|T = y).
To do this, we use Bayes rule:

p(S = y, T = y)
p(S = y|T = y) =
p(T = y)
P
J,R p(T = y, J, R, S = y)
= P
J,R,S p(T = y, J, R, S)
P
J,R p(J|R)p(T = y|R, S = y)p(R)p(S = y)
= P
J,R,S p(J|R)p(T = y|R, S)p(R)p(S)
P
RP p(T = y|R, S = y)p(R)p(S = y)
=
R,S p(T = y|R, S)p(R)p(S)
0.9 ∗ 0.8 ∗ 0.1 + 1 ∗ 0.2 ∗ 0.1
=
0.9 ∗ 0.8 ∗ 0.1 + 1 ∗ 0.2 ∗ 0.1 + 0 ∗ 0.8 ∗ 0.9 + 1 ∗ 0.2 ∗ 0.9
0.092
= = 0.3382
0.272
so that the belief that the sprinkler is on increases above the prior probability 0.1,
due to the fact that the grass is wet.
 76

Let us now calculate the probability that Tracey’s sprinkler was on overnight, given
that her grass is wet and that Jack’s grass is also wet, p(S = y|T = y, J = y). We
use Bayes rule again:

p(S = y, T = y, J = y)
p(S = y|T = y, J = y) =
p(T = y, J = y)
P
R p(T = y, J = y, R, S = y)
= P
R,S p(T = y, J = y, R, S)
P
p(J = y|R)p(T = y|R, S = y)p(R)p(S = y)
= RP
R,S p(J = y|R)p(T = y|R, S)p(R)p(S)

substituting in the numbers, as before, we get

0.0344
= = 0.1604
0.2144

What this shows is that the probability that the sprinkler is on, given the extra
evidence that Jack’s grass it wet, is lower than the probability that the grass is
wet given only that Tracey’s grass is wet. That is, that the grass is wet due to
the sprinkler is (partly) explained away by the fact that Jack’s grass is also wet
– this increases the chance that the rain has played a factor in making Tracey’s
grass wet.

10.2 A word on notation

Graphs are widely used, but differ markedly in what they represent. Here we try
to cover two common misconceptions.
State Transition Diagrams Such graphical representations are common in Markov Chains and Finite
State Automata. A set of states is written as set of nodes(vertices) of a
graph, and a directed edge between node i and node j (with an associated
weight pij ) represents that a transition from state i to state j can occur with
probability pij . From the graphical models perspective we would simply
write down a directed graph x(t) → x(t + 1) to represent this Markov Chain.
The state-transition diagram simply provides a graphical description of the
conditional probability table p(x(t + 1)|x(t)).
Neural Networks Neural networks also have vertices and edges. In general, however, neural
networks are graphical representations of functions, whereas as graphical
models are representations of distributions (a much more powerful generalisa-
tion). Neural networks (or any other parametric description) may be used to
represent the conditional probability tables, as in sigmoid belief networks[9].

10.3 Example : Was it the Burglar?

Here’s another example using binary variables. Sally comes home to find that the
burglar alarm is sounding (A = 1). Has she been burgled (B = 1), or was the
alarm triggered by an earthquake (E = 1)? She turns the car radio on for news of
earthquakes (R = 1).
 77

p(B) B E p(E)

p(A|B,E) A R p(R|E)

Figure 10.2: Belief Network for the Burglar model. Here, for pedagological pur-
poses only, we have explicitly written down which terms in the distribution each
node in the graph represents.

Using Bayes’ rule, we can write, without loss of generality,

p(B, E, A, R) = p(A|B, E, R)p(B, E, R)

We can repeat this for p(B, E, R), and continue

p(B, E, A, R) = p(A|B, E, R)p(R|B, E)p(E|B)p(B)

However, the alarm is surely not directly influenced by any report on the Radio
– that is, p(A|B, E, R) = p(A|B, E). Similarly, we can make other conditional
independence assumptions such that

p(B, E, A, R) = p(A|B, E)p(R|E)p(E)p(B)

Specifying Conditional Each node has an associated conditional probability distribution:

Probability Tables
Alarm = 1 Burglar Earthquake
0.9999 1 1 Radio = 1 Earthquake
0.99 1 0 1 1
0.99 0 1 0 0
0.0001 0 0

The remaining tables are p(B = 1) = 0.01 and p(E = 1) = 0.000001. The tables
and graphical structure fully specify the distribution.
Explaining Away Now consider what happens as we observe evidence.
Initial Evidence: The Alarm is sounding:
P
p(B=1,E,A=1,R)
p(B = 1|A = 1) = PE,R p(B,E,A=1,R)
B,E,R
P
PE,R p(A=1|B=1,E)p(B=1)p(E)p(R|E)
=
B,E,R p(A=1|B,E)p(B)p(E)p(R|E) ≈ 0.99

Additional Evidence: The Radio broadcasts an Earthquake warning:

A similar calculation gives p(B = 1|A = 1, R = 1) ≈ 0.01
Thus, initially, because the Alarm sounds, Sally thinks that she’s been burgled.
However, this probability drops dramatically when she hears that there has been
an Earthquake. That is, the Earthquake “explains away” to a large extent the fact
that the Alarm is ringing.
General Calculations Of course, we don’t wish to carry out such inference calculations by hand all
the time. General purpose algorithms exist for this, such as the Junction Tree
Algorithm, and we shall introduce these later.
 78

x1 x2 x3 x4

x3 x4 x1 x2

Figure 10.3: Two Belief networks for a 4 variable distribution. In this case, both
graphs are representations of the same distribution p(x1 , x2 , x3 , x4 ). The extension
of this ‘cascade’ to many variables is obvious, and always results in an acyclic graph.

10.4 Belief Networks

The reader may have been aware that in the above two examples, we had a choice
as to how we used Bayes’ rule. For example, in a general 4 variable case, we could
choose the factorisation,

p(x1 , x2 , x3 , x4 ) = p(x1 |x2 , x3 , x4 )p(x2 |x3 , x4 )p(x3 |x4 )p(x4 )

equally valid is (see fig(10.3))

p(x1 , x2 , x3 , x4 ) = p(x3 |x4 , x1 , x2 )p(x4 |x1 , x2 )p(x1 |x2 )p(x2 ).

Of course, if one wishes to make independence assumptions, then the initial choice
becomes significant. However, one should bear in mind that, in general, two dif-
ferent graphs may represent the same distribution.
Indeed, the observation that any distribution may be written in the cascade form
fig(10.3) gives an algorithm for constructing a belief network on variables x1 , . . . , xn
: write down the n−variable cascade graph; assign any ordering of the variables
to the nodes; you may then delete any of the directed connections.
Variable Order To ensure maximum sparsity, add “root causes” first, then the variables they
influence, and so on, until the leaves are reached. Leaves have no direct causal2
influence over the other variables.
Conditional Probability Once the graphical structure is defined, the actual values of the tables p(xi |pa (xi ))
Tables (CPTs) need to be defined. That is, for every possible state of the parental variables pa (xi ),
a value for each of the states (except one, since this is determined by normalisation)
needs to be specified. For a large number of parents, writing out a table of values
is intractable, and the tables are usually parameterised in some simple way. More
on this later.
2 ‘Causal’ is a tricky word since here there is no temporal ‘before’ and ’after’, merely correlations
or dependencies. For a distribution p(a, b), we could write this as either p(a|b)p(b) or p(b|a)p(a).
In the first, we might think that b ‘causes’ a, and in the second case, a ‘causes’ b. Clearly,
this is not very meaningful since they both represent exactly the same distribution, and any
apparent causation is merely spurious. Nevertheless, in constructing belief networks, it can be
helpful to think about dependencies in terms of causation since our intuitive understanding is
that often one variable ‘influences’ another. This is discussed much more deeply in [10], where
a true calculus of causality is developed.
 79

10.4.1 Conditional Independence

Consider the three variable distribution p(x1 , x2 , x3 ). We may write this is any of
the 6 ways p(xi1 |xi2 , xi3 )p(xi2 |xi3 )p(xi3 ) where (i1 , i2 , i3 ) is any of the 6 permu-
tations of (1, 2, 3). Hence, whilst all graphically different, they all represent the
same distribution which does not make any conditional independence statements.
To make an independence statement, we need to drop one of the links. This gives
rise in general to 4 graphs in fig(10.4). Are any of these graphs equivalent, in the

x1 x2 x1 x2 x1 x2 x1 x2

x3 x3 x3 x3
(a) (b) (c) (d)

Figure 10.4: By dropping say the connection between variables x1 and x2 , we

reduce the 6 possible graphs amongst three variables to 4.

sense that they represent the same distribution? A simple application of Bayes’
rule gives :

p(x2 |x3 )p(x3 |x1 )p(x1 ) = p(x2 , x3 )p(x3 , x1 )/p(x3 ) = p(x1 |x3 )p(x2 , x3 )
| {z }
graph(c)

= p(x1 |x3 )p(x3 |x2 )p(x2 ) = p(x1 |x3 )p(x2 |x3 )p(x3 )
| {z } | {z }
graph(d) graph(b)

and hence graphs (b),(c) and (d) represent the same distribution. However, graph
(a) represents something fundamentally different: there is no way to transform the
distribution p(x3 |x1 , x2 )p(x1 )p(x2 ) into any of the others.
Graphs (b),(c) and (d) all represent the same conditional independence assumption
that, given the state of variable x3 , variables x1 and x2 are independent. We write
this as I(x1 , x2 |x3 ).
Graph (a) represents something different, namely marginal independence : p(x1 , x2 ) =
p(x1 )p(x2 ). Here we have marginalised over the variable x3 .

10.4.2 Intuition

In a general Belief Network, with many nodes, how could we check if two variables x
and y are independent, once conditioned on another variable z? In fig(10.5)(a,b),
it is clear that x and y are independent when conditioned on z. It is clear in
collider fig(10.5)(c) that they are dependent. In this situation, variable z is called a collider
– the arrows of its neighbours are pointing towards it. What about fig(10.5)(d)?
In (d), when we condition on z, then, in general, x and y will be dependent, since
X
p(z|w)p(w|x, y)p(x)p(y) 6= p(x|z)p(y|z)
w

– intuitively, variable w becomes dependent on the value of z, and since x and y

are conditionally dependent on w, they are also conditionally dependent on z.
Roughly speaking, if there is a non-collider z which is conditioned on along the
path between x and y (as in fig(10.5)(a,b)), then this path does not make x and y
 80

x y

x y x y x y w

z z z z
(a) (b) (c) (d)

Figure 10.5: In graphs (a) and (b), variable z is not a collider. (c) Variable z is
a collider. Graphs (a) and (b) represent conditional independence I(x, y|z). In
graphs (c) and (d), x and y are conditionally dependent given variable z.

b c

Figure 10.6: The variable d is a collider along the path a − b − d − c, but not along
the path a − b − d − e.

dependent. Similarly, if there is a path between x and y which contains a collider,

provided that this collider is not in the conditioning set (and neither are any of its
children) then this path does not make x and y dependent.
Note that a collider is defined relative to a path. In fig(10.6), the variable d is a
collider along the path a − b − d − c, but not along the path a − b − d − e (since,
relative to this path, the two arrows to not point inwards to d).
These intuitions lead us to a more formal statement about conditional indepen-
dence:

10.4.3 d-Separation

If two variables are d-separated relative to a set of variables Z in a directed graph,

then they are independent conditional on Z in all probability distributions such a
graph can represent. Roughly, two variables X and Y are independent conditional
on Z if knowledge about X gives you no extra information about Y once you have
knowledge of Z. In other words, once you know Z, X adds nothing to what you
know about Y.
Formally, to define d-separation, we need to define d-connection:
d-connection If G is a directed graph in which X, Y and Z are disjoint sets of vertices, then X
and Y are d-connected by Z in G if and only if there exists an undirected path U
between some vertex in X and some vertex in Y such that for every collider C on
U, either C or a descendent of C is in Z, and no non-collider on U is in Z.
X and Y are d-separated by Z in G if and only if they are not d-connected by Z
in G.
 81

See http://www.andrew.cmu.edu/user/scheines/tutor/d-sep.html from where

this definition was taken for more details, and also some nice demos.
Bayes Ball The Bayes Ball algorithm, “Bayes-Ball: The Rational Pastime” R.D. Shachter
(UAI 98) provides a linear time complexity algorithm which given a set of nodes
X and Z determines the set of nodes Y such that I(X, Y |Z). Y is called the set
of irrelevant nodes for X given Z
Example (1) Consider the simple graphs in fig(10.7).

A C E A C E

B D B D

Figure 10.7: Examples for d-separation – Is I(a, e|b)? Left: If we sum out variable
d, then we see that a and e are independent given b, since the variable e will ap-
pear as an isolated factor independent of all other variables, hence indeed I(a, e|b).
Whilst b is a collider which is in the conditioning set, we need all colliders on
the path to be in the conditioning set (or their descendents) for d-connectedness.
Right: Here, if we sum out variable d, then variables c and e becomes intrinsi-
cally linked, and the distribution p(a, b, c, e) will not factorise into a function of a
multiplied by a function of e – hence they are dependent.

Example (2) Consider the simple graph in fig(10.8).

B G F

T S

Figure 10.8: Example for d-separation.

Are the variables T and F unconditionally independent, i.e. I(T, F |∅)? Remember
that the key point are the colliders along the path between the two variables. Here
there are two colliders, namely G and S – however, these are not in the condi-
tioning set (which is empty), and hence they are d-separated, and unconditionally
independent.
What about I(T, F |G)? Well, now there is a collider on the path between T and
F which is in the conditioning set. Hence T and F are d-connected conditioned
on G, and therefore T and F are not independent conditioned on G.
Note that this may seem a bit strange – initially, when there was no conditioning,
T and F were independent. However, conditioning on G makes them dependent.
An even simpler example would be the graph A → B ← C. Here A and B are
unconditionally independent. However, conditioning of B makes them dependent.
Intuitively, whilst we believe the root causes are independent, given the value of
the observation, this tells us something about the state of both the causes, coupling
them and making them dependent.
What about I(B, F |S)? Since there is a collider on the path between T and F
which is in the conditioning set, namely S, B and F are conditionally dependent
given S.
 82

Deterministic Dependencies

Sometimes the concept of independence is perhaps a little strange. Consider the

following : variable x and y are both binary, being either in state 0 or 1. We define
the distribution such that x and y are always in opposite states : p(x = 0|y =
0) = 0, p(x = 0|y = 1) = 1, p(x = 1|y = 0) = 1, p(x = 1|y = 1) = 0. To complete
the specification of p(x, y) = p(x|y)p(y), we define p(y = 0) = p(y = 1) = 0.5.
Hence, x and y are in opposite states and they are each in state 0 with probability
0.5. Now, are x and y dependent? Indeed, a quick calculation shows that they
are. However, if we changed the prior probability, so that y is always in state 1,
p(y = 0) = 0 and p(y = 1) = 1, are x and y dependent? If you carry out the
maths, you’ll find that they are independent! This may seem very strange – we
know for sure that x will always be in state 0 and y will always be in state 1 – they
are in opposite states, yet they are independent. This apparent strangeness results
from the deterministic nature of the distribution – the distribution is trivially
concentrated in a single joint state. In that case p(x|y) = p(x) since x can only
ever be in a single state, whatever the value of y is (indeed y can also only ever
be in a single state). Perhaps the best way to think about dependence is to ask
whether or not knowing the state of variable y tells you something more than you
knew before about variable x, where ‘knew before’ means working with the joint
distribution of p(x, y) to figure out (without conditioning on y) what we can know
about x, namely p(x).

10.5 Graphical Models

We’ve so far considered Belief Networks (also called Bayes or Bayesian Networks)
as distributions defined as directed graphs.
Graphical models are a more general marriage of graph and probability theory.
Their usefulness stems from the ease of interpretation of a graphical representation
of a distribution. In particular, they provide a framework for unifying a wide class
of probabilistic models and associated algorithms.
There are two main types, directed and undirected, based on whether or not the
underlying graph is directed or undirected. Chain graphical models contain both
directed and undirected links.

Directed Graphical Models

More formally, Belief networks are directed acyclic graphs (DAGs), in which the
nodes in the graph represent random variables in a probability distribution. To
each variable A with parents B1 . . . Bn , there is an associated probability table
p (A|B1 . . . Bn )
If A has no parents then the table reduces to unconditional probabilities p(A).
Chain Rule Let BN be a Bayesian network over
U = {A1 , . . . An }
Then the joint probability distribution p(U ) is the product of all conditional prob-
abilities specified by the BN:
Y
p(U ) = P (Ai |pa (Ai ))
i
 83

where pa (Ai ) is the parent set of Ai .

Note : it is the acyclic form which enables us to easily write down a joint distrib-
ution which is consistent with the given conditional distributions. (Contrast this
with the more general case of Markov Random Fields below).

Undirected Graphical Models

It is clear that Belief Networks correspond to a special kind of factorisation of the

joint probability distribution in which each of the factors is itself a distribution.
However, we may consider other distributions, for example
1
p(a, b, c) = φ(a, b)φ(b, c)
Z
where φ(a, b) and φ(b, c) are simply two (non-negative) potential functions, and Z
is a constant which ensures normalisation of the distribution. In the above case,
since we could equally write, for example φ̃(b, a) ≡ φ(a, b) (that is, the variables
simply index the value of the function, and are otherwise interchangeable) we can
represent the distribution as an undirected graph: In this sense, a potential is

a b c

associated with each link on the graph.

It is clear that directed distributions can be represented as undirected distribu-
tions since one can associate each (normalised) factor in a directed distribution
with a potential. For example, the distribution p(a|b)p(b|c)p(c) can be factored as
φ(a, b)φ(b, c), where φ(a, b) = p(a|b) and φ(b, c) = p(b|c)p(c), with Z = 1.
Undirected models will turn out to be extremely useful as part of general algorithms
for computations with graphs. They are also useful models in themselves, and have
a long history in different branches of science, especially statistical mechanics on
lattices and as models in visual processing.

An undirected graph

Consider a model in which our desire is that states of the binary valued vari-
ables x1 , . . . , x9 , arranged on a lattice (as below) should prefer their neighbouring
variables to be in the same state

x1 x2 x3

x4 x5 x6

x7 x8 x9

1 Y
p(x1 , . . . x9 ) = φij (xi , xj )
Z <ij>
 84

where < ij > denotes the set of indices where i and j are neighbours in the
undirected graph. Then a set of potentials that would encourage neighbouring
variables to have the same state would be
1 2
φij (xi , xj ) = e− T (xi −xj )

This model actually corresponds to a well-known model of the physics of magnetic

systems, called the Ising Model. For high ‘temperatures’ T , the variables take on
essentially random states, with no constraints. For low values of T , there is a strong
constraint that neighbouring variables become aligned. Remarkably, one can show
that, in a very large two-dimensional lattice, below the Curie temperature, Tc ,
the system admits a phase change in that all the variables become aligned. That
this phase change happens for non-zero temperature is a fascinating result and
has driven over 50 years of research in this area[11]. Essentially, global coherence
spontaneously appears from weak local constraints.
Similar local constraints are popular in image restoration algorithms to clean up
noise, under the assumption that noise will not show any local spatial coherence,
whilst ‘signal’ will.

Chain Graphical Models

Chain graphical models contain both directed and undirected links.

10.5.1 Markov Random Fields

Just as a vector field describes a vector v which can be dependent on (say) a

spatial location, v(x), a MRF defines a probability distribution for each ‘location’,
where the location is simply given by an index. That is, a MRF is defined by a
set of distributions p(xi |pa(xi )) > 0 (the positivity constraint is required), where
i ∈ {1, . . . , n} indexes the distributions (usually, but not necessarily, on a lattice),
and pa(xi ) are the “parents” of variable xi , namely that subset of the variables
x1 , . . . , xn that the distribution of variable xi depends on. The term Markov
indicates that this is a proper subset of the variables. From this definition it
is not obvious if there exists a distribution p∗ (x1 , . . . , xn ) which has conditional
marginals consistent with the set of given marginals. The Hammersley-Clifford
theorem specifies what the functional form of the any such joint distribution must
be.

Hammersely Clifford Theorem

Imagine that we define a set of local distributions p(xi |pa(xi )) > 0. When indeed
will this define a consistent joint distribution p(x1 , . . . , xn )? The Hammersley
Clifford Theorem states that the MRF defines a consistent joint distribution if
and only if p(x1 , . . . , xn ) is a so-called Gibbs distribution
!
1 X
p(x1 , . . . , xn ) = exp − Vc (xc )
Z c

where the sum is over all cliques (maximal complete subgraphs), c and Vc (xc ) is a
real function defined over the variables in the clique c. The graph over which the
cliques are defined is an undirected graph with a link between all parents pa(xi )
and a link between xi and each parent pa(xi ), repeated over all the variables xi .
 85

A A

B C B C

D D

Figure 10.9: Left: An undirected model. Middle: Every DAG with the same
structure as the undirected model must have a situation where two arrows will
point to a node, such as node d. Summing over the states of variable in this DAG
will leave a DAG on the variables A, B, C with no link between B and C – which
cannot represent (in general) the undirected model since when one marginals over
D in that, this adds a link between B and C.

Besag has originally a nice proof of this, which requires the positivity constraint[12],
and a counter example shows that this is necessary[13].
Note : It’s easy to go from the Gibbs distribution to find the local conditional
distributions. The other way round is not necessarily so easy, since we would have
to know the so-called partition function Z. This is reminiscent of Gibbs sampling
(see the appendix on sampling) : effectively, one can easily define a sampler (based
on a Gibbs distribution), but it does not mean that we know the joint distribution,
ie the partition function (normalisation constant) Z.

10.5.2 Expressiveness of Graphical Models

It’s clear that every Belief Network can be represented as an undirected graphical
model, by simple identification of the factors in the distributions.
Can every undirected model be represented by a Belief Network with the same link
structure? Consider the example in fig(10.9) (from Zoubin Ghahramani)
As a final note, of course, every probability distribution can be represented by
some Belief Network, though it may not necessarily have any obvious structure
and be simply a “fully connected” cascade style graph.

Discussion
Graphical models have become a popular framework for probabilistic models in
artificial intelligence and statistics. One of the reasons for this is that the graphical
depiction of the model contains no information about the content of the conditional
probability tables. This is advantageous in that algorithms can be formulated for
a graphical structure, independent of the details of the parameterisation of the
local tables in the model. However, despite the elegance of such an approach, the
issue of tractability can be heavily dependent on the form of the local probability
tables. For example, for Gaussian tables all marginals are tractable although, in
general, marginalising high dimensional distributions is highly non-trivial.

10.6 Problems
(Thanks to Chris Williams for some of these questions)

Exercise 10 (From Tipping, §2.1.3) . Box 1 contains 8 apples and 4 oranges.

Box 2 contains 10 apples and 2 oranges. Boxes are chosen with equal probability.
 86

What is the probability of choosing an apple? (3/4). If an apple is chosen, what

is the probability that it came from box 1? (4/9)

Exercise 11 (R & N, Ex 14.5) Prove from first principles the conditionalized

version of the general product rule

P (X, Y |Z) = P (X|Z)P (Y |X, Z).

Also prove the conditionalized version of Bayes’ rule

P (Y |X, Z)P (X|Z)

P (X|Y, Z) = .
P (Y |Z)

Exercise 12 (Elvis’ twin) Approximately 1/125 of all births are fraternal twins,
and 1/300 births are identical twins. Elvis Presley had a twin brother (who died
at birth). What is the probability that Elvis was an identical twin? You may
approximate the probability of a boy or girl birth as 1/2. (Biological information:
identical twins must be either both boys or both girls, as they are derived from one
egg.)

Exercise 13 (Prosecutor’s fallacy) This question concerns “DNA fingerprint-

ing” evidence. The probability that there is a DNA match given that a person is
innocent is estimated as 1/100, 000. Assume that the probability that there is a
match given that a person is guilty is 1. Suppose that the defendant in a trial lives
in a city where there are 10, 000 people who could have committed the crime, and
that there is a DNA match to the defendant. Calculate P (guilty| DN A match).
How does this vary as the size of the population changes?

Exercise 14 (The Three Prisoners problem) (From Pearl, 1988) Three pris-
oners A, B and C are being tried for murder, and their verdicts will be read and
their sentences executed tomorrow. They know only that one of them will be de-
clared guilty and will be hanged while the other two will go free; the identity of the
condemned prisoner is revealed to a reliable prison guard, but not to the prisoners.
In the middle of the night Prisoner A makes the following request. “Please give
this letter to one of my friends – to one who is to be released. You and I know
that at least one of them will be released.”. The guard carries out this request.
Later prisoner A calls the guard and asks him to whom he gave the letter. The
guard tells him that he gave the letter to prisoner B. What is the probability that
prisoner A will be released?

Exercise 15 (The Monte Hall problem) I have three boxes. In one I put a
prize, and two are empty. I then mix up the boxes. You want to pick the box with
the prize in it. You choose one box. I then open another one of the boxes and show
that it is empty. I then give you the chance to change your choice of boxes—should
you do so? How is this puzzle related to the Three Prisoners problem?

Exercise 16 Consider the following fictitious scientific information: The doctors

find that people with Kreuzfeld-Jacob disease (KJ) almost invariably ate lots of
hamburgers, thus p(HamburgerEater|KJ) = 0.9. The probability of an individual
having KJ is currently rather low, about one in 100,000. The instance of eating
lots of hamburgers is rather widespread, so p(HamburgerEater) = 0.5.
 87

1. What is the probability that a regular hamburger eater will have Kreuzfeld-
Jacob disease?
2. If the case had been that the number of people eating hamburgers was rather
small, say p(HamburgerEater) = 0.001, what is the probability that a reg-
ular hamburger eater will have Kreuzfeld-Jacob disease? Comment on the
difference with the result in the previous part of the question.

Exercise 17 There is a synergistic relationship between Asbestos (A) exposure,

Smoking (S) and Cancer (C). A model describing this relationship is given by

p(A, S, C) = p(C|A, S)p(A)p(S)

1. What kinds of independence assumptions have been made in this model?

2. How could you adjust the above model to account for the fact that people who
work in the building industry have a higher likelihood to also be smokers and
also a higher likelihood to asbestos exposure?

Exercise 18 Explain why any distribution on a set of variables Q x = (x1 , . . . , xn )

can be written as a belief network : p(x1 , x2 , . . . , xn ) = ni=1 p(xi |pa(xi )), where
pa(xi ) are the parental variables of xi .

Exercise 19 Inspector Clouseau arrives at the scene of a crime. The victim lies
dead in the room, and the inspector quickly finds the murder weapon, a knife. The
Butler (B) and Maid (M) are his main suspects. The inspector has a prior be-
lief of 0.8 that the Butler is the murderer, and a prior belief of 0.2 that the Maid
is the murderer. These probabilities are independent in the sense that p(B, M ) =
p(B)p(M ). (It is possible that both the Butler and the Maid could be the murderer).

The inspector’s extensive criminal knowledge can be formulated mathematically as

follows:

p(B = murderer) = 0.8, p(M = murderer) = 0.2

p(knife used|B = not murderer, M = not murderer) = 0.3

p(knife used|B = not murderer, M = murderer) = 0.2
p(knife used|B = murderer, M = not murderer) = 0.6
p(knife used|B = murderer, M = murderer) = 0.1

What is the the probability that the Butler is the murderer?

Exercise 20 The belief network shown below is the famous “Asia” example of
Lauritzen and Speigelhalter (1988). It concerns the diagnosis of lung disease (tu-
berculosis, lung cancer, or both, or neither). In this model a visit to Asia is assumed
 88

to increase the probability of tuberculosis.

visit to
Asia? smoking?

lung
tuberculosis? bronchitis?
cancer?

tuberculosis
or lung
cancer?

positive shortness
X-ray? of breath?

State if the following conditional independence relationships are true or false

I(tuberculosis?, smoking?|shortness of breath?),
I(lung cancer?, bronchitis?|smoking?),
I(visit to Asia?, smoking?|lung cancer?)
I(visit to Asia?, smoking?|lung cancer?, shortness of breath?).

Exercise 21 Consider the three variable distribution

p1 (a, b, c) = p(a|b)p(b|c)p(c)

where all variables are binary. How many parameters are needed to specify distri-
butions of this form?
Now consider an undirected distribution on the same set of variables,

p2 (a, b, c) = φ(a, b)φ(b, c)/Z

where Z is a normalisation constant. How many parameters are needed to specify

the parameters of the all the potential functions here? How many degrees of freedom
do you think there are in this representation?
Discuss whether or not any setting of the tables in p1 can be represented by some
setting of the potential functions in p2 , and whether or not the opposite is true.
If you believe that p1 and p2 are different representations of the same family of
distributions (ie any setting of the distribution in one representation can always be
transformed expressed using some setting in the other representation), then discuss
whether or not the potential function representation is an over-parameterisation
of distributions from that class or not.
(For future discussion : explain the relationship between this question and Condi-
tional Random Fields and whether or not CRF on a chain are an overparameter-
isation of an equivalent Markov Chain).
11 Inference in Belief Networks

11.1 Inference
Calculating conditional marginals, as in the Wet Grass example seen previously,
is a form of inference. Although in simple graphs, such as the Wet Grass DAG, it
is straightforward to carry out the calculations to calculate marginals by hand, in
general, this problem can be computationally non-trivial. Fortunately, for singly-
connected graphs (poly-trees) there exist efficient algorithms for inference, and it
is instructive to understand how these algorithms work.
In this chapter we will consider two main algorithms, based on simple ideas. The
first, variable elimination works on general mutliply-connected distributions (al-
beit not-necessarily efficiently) and is particularly appropriate for answering single
queries.
The second algorithm we consider is Pearl’s Belief Propagation[5], which works
only for singly-connected graphs, yet has the advantage that it can answer multiple
queries efficiently.
These two classes of algorithms are useful as a precursor to developing algorithms
that run, essentially as efficiently as can be reasonably made, on any graphical
model (see the Junction Tree Algorithm chapter) and which are efficient in dealing
with answering multiple queries.

11.2 Variable Elimination in a simple chain

Consider the distribution

p(a, b, c, d) = p(a|b)p(b|c)p(c|d)p(d)

A B C D

and imagine that our inference task is to calculate the marginal distribution p(a).
Also, for simplicity, let’s assume that each of the variables can take one of two
states ∈ {0, 1}. Then
X
p(a = 0) = p(a = 0, b, c, d) (11.2.1)
b∈{0,1},c∈{0,1},d∈{0,1}
X
= p(a = 0|b)p(b|c)p(c|d)p(d) (11.2.2)
b∈{0,1},c∈{0,1},d∈{0,1}

It’s clear that we could carry out this computation by simply enumerating each of
the probabilities for the 2 ∗ 2 ∗ 2 = 8 states of the variables b,c and d. However,
in a more general chain of length T , this would imply that we would need a large

89
 90

number of summations ∝ eT . The question is how can we exploit the structure of

the graph to reduce the number of summations required?.
In the above chain, it is clear that we may push the summation over d as far to
the right as possible:
X X
p(a = 0) = p(a = 0|b)p(b|c) p(c|d)p(d)
b∈{0,1},c∈{0,1} d∈{0,1}
| {z }
fd (c)

where fd (c) is a (two state) function. Similarly, we can distribute the summation
over c as far to the right as possible:
X X
p(a = 0) = p(a = 0|b) p(b|c)fd (c)
b∈{0,1} c∈{0,1}
| {z }
fc (b)

Then, finally,
X
p(a = 0) = p(a = 0|b)fc (b)
b∈{0,1}

Hence, in this case, we have made 2 + 2 + 2 = 6 summations. Whilst this saving

may not appear much, the important point is that it is clear that the number of
computations for a chain of length T would simply scale linearly with T , since each
variable i that we sum over simply requires two additions to define the correspond-
ing fi (i − 1). This procedure is naturally enough called variable elimination, since
each time we sum over the states of a variable, we eliminate it from the remaining
distribution. It is clear that we can always perform variable elimination in a chain
efficiently, since there is a natural way to distribute the summations “to the right”
(or “left” as needs be).
Note that this is related to the associativity of matrix multiplication. We can
define matrices

[MAB ]i,j = p(a = i|b = j), [MBC ]i,j = p(b = i|c = j),
[MCD ]i,j = p(c = i|d = j), [MD ]i = p(d = i), [MA ]i = p(a = i)

Then the marginal MA can be written

MA = MAB MBC MCD MD = MAB (MBC (MCD MD ))

since matrix multiplication is associative. This matrix formulation of calculating

marginals is called the transfer matrix method, and is particularly popular in the
physics literature.

If we had somehow been rather myopic and not realised that the distribution was
a chain and, instead, had placed the summations non-optimally, we may still have
ended up with an exponentially large amount of computation in a long chain – as
in the case where we do not push variables at all, which as we saw above, may
result in extreme inefficiency.
 91

A B

C D E

F G

Figure 11.1: A simple Polytree. Inference is the problem of calculating the conse-
quences of (possible) evidence injection on the individual nodes.

E p(e)p(g|d, e)

C p(c|a)

B p(b)p(d|a, b) p(b)p(d|a, b)

G γE (d, g) γE (d, g)

A p(a) p(a) p(a)γB (d, a) p(a)γB (d, a)

D p(f |d) p(f |d) p(f |d) p(f |d)γG (d) p(f |d)γG (d) γA (d)

F γD (f )

Figure 11.2: The bucket elimination algorithm applied to the graph fig(11.1). At each stage, at least one
node is eliminated from the graph.

11.3 Bucket Elimination

We shall here consider a general variable eliminiation method for calculating
marginals that works for any directed distribution (including multiply connected
graphs). In itself, this is not a particularly popular nor useful algorithm (since
it does not efficiently deal with multiple queries). Nevertheless, it does provide a
simple way to calculate marginals, and is easy to remember.
It is helpful to consider a concrete example, such as that shown in fig(11.1). This
polytree represents the distribution

p(a, b, c, d, e, f, g) = p(f |d)p(g|d, e)p(c|a)p(d|a, b)p(a)p(b)p(e).

For simplicity, we will consider calculating only marginals, since the generalisation
to conditional marginals is straightforward. Consider the problem of calculating
the marginal p(f ).
X X
p(f ) = p(a, b, c, d, e, f, g) = p(f |d)p(g|d, e)p(c|a)p(d|a, b)p(a)p(b)p(e)
a,b,c,d,e,g a,b,c,d,e,g

We can distribute the summation over the various terms as follows: e,b and c are
 92

end nodes, so that we can sum over their values:

! ! !
X X X X
p(f ) = p(f |d)p(a) p(d|a, b)p(b) p(c|a) p(g|d, e)p(e)
a,d,g b c e
P
For
P convenience, lets write the terms in P the brackets as b p(d|a, b)p(b) ≡ γB (a, d),
e p(g|d, e)p(e) ≡ γE (d, g). The term c p(c|a) is equal to unity, and we therefore
eliminate this node directly. Rearranging terms, we can write
X
p(f ) = p(f |d)p(a)γB (a, d) γE (d, g)
a,d,g

If we think of this graphically, the effect of summing over b, c, e is effectively to

remove or “eliminate” those nodes. We can now carry on summing over a and g
since these are end points of the new graph:
! !
X X X
p(f ) = p(f |d) p(a)γB (a, d) γE (d, g)
d a g

Again, this defines new functions γA (d), γG (d), so that the final answer can be
found from
X
p(f ) = p(f |d)γA (d) γG (d)
d

We illustrate this graphically in fig(11.2). Initially, we define an ordering of the

variables, beginning with the one that we wish to find the marginal for – a suitable
ordering is therefore, f, d, a, g, b, c, e. Then starting with the highest node, e, we
put all the functions that mention e in the e bucket. Continuing with the next
highest bucket, c, we put all the remaining functions that mention c in this c
bucket, etc. The result of this initialisation procedure is that terms (conditional
distributions) in the DAG are distributed over the buckets, as shown in the left
most column of fig(11.2). Eliminating then the highest bucket e, we pass a message
to node g. Immediately, we can also eliminate bucket c since this sums to unity.
In the next column, we have now two less buckets, and we eliminate the highest
remaining bucket, this time b, passing a message to bucket a[14].
There are some important observations we can make about this procedure:
• For simplicity, assume that each variable can take two states. Then the total
number of summations that the above procedure makes in calculating p(f ) is
equal to two times the number of functions γ that we defined, plus two extra
summations to eliminate the final node d. That is, the number of summation
operations is (n−1)∗s where n is the number of variables in the graph, and s
is the number of states of each of the variables. Hence, for this simple graph,
the marginal p(f ) has been computed in a time which scales linearly with
the number of variables in the graph. If we had attempted to compute p(f )
by brute force summation over the other nodes, this would have contained
2n terms – that is, the brute force summation is exponentially costly. The
independencies in this graph have enabled us to find an efficient procedure
for computing p(f ).
• We could also have carried out the same kind of node elimination procedure
to calculate the marginals p(a), p(b), p(c), p(d), p(e) and p(g) in a similar
way. Here, however, each query (calculation of a marginal in this case) would
require re-running the algorithm.
 93

A B A B

C D C D

E F E F
(a) Undirected (b) Directed

Figure 11.3: (a) An undirected graph. This graph is singly-connected – there is a

unique path between any two nodes. (b) An acyclic directed graph. This graph is
multiply connected, or loopy – we can link A and E through ACE or ADE.

• In general, bucket elimination constructs multi-variable messages – that is,

not dependent on a single node, but potentially several nodes. For general
polytrees, therefore, the complexity of this problem is exponential in the
maximal family size, both in time and storage. It is also clear that if we
repeated the above procedure for computing the marginals of other variables,
we would end up redefining some of the same messages. It would be more
efficient to simply reuse these messages, rather than recalculating them each
time. There is an algorithm that can achieve this, and we describe this in
the following section.
• It is clear that in trees (and in general, singly connected graphs) that we
can always choose an ordering to render the amount of computation to scale
linearly with the number of variables (nodes) in the graph. Such an ordering
is called perfect, and indeed it can be shown that a perfect ordering can
always easily be found for singly-connected graphs (see [15]). However, it is
also clear that orderings exist for which Bucket Elimination will be extremely
inefficient.
• The Bucket Elimination procedure will work also on undirected graphs, with
the proviso that marginals will be computed upto a missing normalisation
constant.

11.4 Belief Propagtion : Inference in Singly Connected Graphs

Graphical models come in two main flavours – directed and undirected, see fig(11.3).
An undirected graph specifies a partial factorisation of the functional form of the
joint probability distribution, up to an overall normalisation constant. The graph
in fig(11.3a) represents distributions of the form
1
p (a, b, c, d, e, f) = Ψ (a, d)Ψ (b, d)Ψ (c, e)Ψ (d, e)Ψ (d, f ) (11.4.1)
Z
where Z is a constant which ensures correct normalisation of the distribution. The
potential functions Ψ are non-negative functions, and describe the strength of links
between neighbouring nodes. The potential functions do not correspond directly
to probability distributions on their arguments, since the potentials themselves are
not necessarily normalised.
Directed models correspond to a partial factorisation of the joint distribution into
a set of conditional distributions, one for each node in the graph. The graph in
 94

fig(11.3b) represents distributions

p (a, b, c, d, e, f) = p(d|a, b)p(a)p(b)p(c|a)p(e|c, d)p(f |d).
The parents of a node are those nodes that feed into a node. For example, the
parents of d, denoted pa (d), are a and b. Similarly, children are nodes which a
node feeds into, so that the children of node d, denoted ch (d), are e and f . We
will only consider directed graphs that are acyclic. That is, there is no path such
that, starting at any node, and following the directed links to other nodes, we
eventually return to the start node.
An important issue in the tractability of graphical models, directed or undirected,
is whether or not the graph is “non-loopy” (singly-connected) or “loopy” (multiply-
connected). Neglecting the directions of any links on the graph, a graph is non-
loopy provided that for every pair of nodes, there is only a single path connecting
them. Thus fig(11.3b) is loopy, since ADECA forms a loop.
Roughly speaking, inference (finding marginals) in non-loopy graphs is straightfor-
ward, since such graphs are topologically equivalent to a tree. Belief Propagation
(BP) is a procedure which exploits this non-loopy structure, and is an exact al-
gorithm for inference on non-loopy graphs. Two versions of BP exist – one for
directed graphs (DBP), and one for undirected graphs (UBP). Recent interest has
been the (strictly erroneous) application of BP to loopy graphs as an approximate
scheme for inference in otherwise intractable situations. Results suggest that, un-
der certain conditions, such as long shortest-loop-length (meaning the number of
links in the path that take the loop back to the starting node), the application of
BP to loopy graphs can provide an accurate approximation. Indeed, a loop may
be considered long if BP converges in a number of iterations less than the loop
length. One of the difficulties in applying directed belief propagation is that, un-
like the undirected version, its complexity is exponential in the number of parental
connections of a node.
For completeness, we briefly present an intuitive derivation of the belief propa-
gation algorithms. The central idea is to form a self-consistent message passing
scheme, based on only locally available information to a node, and from the re-
sulting iterated messages, calculate marginals of the graph.

11.4.1 Undirected Belief Propagation

In undirected graphs, the situation

P is a relatively straightforward. Consider calcu-
lating the marginal p(d) = a,b,c,e,f p(a, b, c, d, e, f ) for the pairwise Markov net-
work in fig(11.3a). The partial functional factorisation of the distribution equation
(11.4.1), enables us to distribute the summation. To keep the notationP simple, we
denote both a node and its state by the same symbol, so that b Ψ(d, b) denotes
summation over the states of the variable b. This results in a message λb→d (d)
which contains information passing from node b to node d and is a function of the
state of node d. This works as follows:
1 X X X X X
p(d) = Ψ (b, d) Ψ (a, d) Ψ (d, f ) Ψ (d, e) Ψ (c, e)
Z a e c
b f
| {z } | {z }| {z } | {z }
λb→d (d) λa→d (d) λf →d (d) λc→e (e)
| {z }
λe→d (d)

where we have defined messaged λn1 →n2 (n2 ) sending information from node n1 to
node n2 as a function of the state of node n2 . It is intuitively clear that we can in
 95

general define messages as

X Y
λa→b (b) = Ψ (a, b) λc→a (a)
a c∈N (a)\b

where N (a) is the set of neighbouring nodes to a. Iterating these equations results

D C
λd,a
λc,a

A λa,b
E λe,a
B

Figure
P 11.4: Undirected BP : calculating a message λa→b (b) =
a Ψ (a, b)λc→a (a) λd→a (a) λe→a (a).

in a convergent
Q scheme for non-loopy graphs. The marginal is then found from
p(d) ∝ c∈N (d) λc→d (d), the prefactor being determined from normalisation. In
contrast to directed belief propagation (described in the following section) whose
complexity scales exponentially with the number of parents of a node, the complex-
ity of calculating a message in undirected belief propagation scales only linearly
with the number of neighbours of the node.

11.4.2 Directed Belief Propagation

Intuition into the derivation of the directed belief propagation algorithm can also
be gained by considering marginalisation on a simple graph, such as depicted in
fig(11.5).

A B C

E F G

Figure 11.5: Intuition for DBP can be gleaned from consideration of simple graphs.
The marginal p(d) can be calculated from information passing from its parents
a, b, c and children e, f, g.

This represents distributions of the form

p (a, b, c, d, e, f, g) = p(d|a, b, c)p(a)p(b)p(c)p(e|d)p(f |d)p(g|d).

Consider calculating the marginal p (d). This involves summing the joint distrib-
ution over the remaining variables a, b, c, e, f, g. Due to the partial factorisation
 96

that the directed structure encodes, we can distribute this summation as follows
X X X X
p (d) = p (d|a, b, c) p(a) p(b) p(c) p (e|d) p (f |d) p (g|d) .
|{z} |{z} |{z} e g
abc f
ρa→d (a) ρb→d (b) ρc→d (c) | {z }| {z }| {z }
λe→d (d) λf →d (d) λg→d (d)

(11.4.2)

We have defined here two types of messages for node d, λ messages that contain
information passing up from children, and ρ messages that contain information
passing down from parents.
If the children of node d are not fixed in any particular state (there is no “evi-
dence”), then the λ messages are trivially 1. If however, there is evidence so that,
for example, node e is fixed in state 1, then λe→d (d) = p (e = 1|d).
It is clear that the marginal for any node can be calculated from the local messages
incoming to that node. The issue that we now address is how to find a recursion
for calculating such messages.
Consider the case where the graph of fig(11.5) has some extra connections, as in
fig(11.5).

I A B C

J D O

E F G N

Figure 11.6: To calculate p(d), we only need to adjust the messages passing from
node a to node d and node g to node d. Information from all the other nodes is
as in fig(11.5)

The only messages that need to be adjusted to find the marginal p(d) are those
from a to d, namely ρa→d (a) and from g to d, namely λg→d (d). The marginal for
d will have exactly the same form as equation (11.4.2), except with the following
adjustments to the messages
X X
ρa→d (a) = p (a|h, i) p(h) p(i) p(j|a)
|{z} |{z}
h,i j
ρh→a (h) ρi→a (i) | {z }
λj→a (a)
X X X
λg→d (d) = p (g|o, d) p(o) p(m|g) p(n|g) .
g,o
|{z} m n
ρo→g (o) | {z }| {z }
λm→g (g) λn→g (g)

The structure of the above equations is that to pass a message from a node n1 to
a child node n2 , we need to take into account information from all the parents of
n1 and all the children of n1 , except n2 . Similarly, to pass a message from node n2
 97

to a parent node n1 , we need to gather information from all the children of node
n2 and all the parents of n2 , except n1 .
Essentially, to formulate the messages for a node, it is as if the parents were dis-
connected from the rest of the graph, with the effect of this disconnection being a
modified prior for each parental node. Similarly, it is as if the children are discon-
nected from the rest of the graph, and the effect of the child nodes is represented
by a modified function on the link (e.g., instead of p(e|d) we have λe,d (d)).
From these intuitions, we can readily generalise the situation to a formal algorithm.

The Directed Belief Propagation Algorithm

A general node d has messages coming in from parents and from children, and we
can collect all the messages from parents that will then be sent through d to any
subsequent children as[15]
X Y
ρd (d) = p (d|pa (d)) ρi,d (i) .
pa(d) i∈pa(d)

Similarly, we can collect all the information coming from the children of node d
that can subsequently be passed to any parents of d
Y
λd (d) = λi,d (d) .
i∈ch(d)

The messages are defined as

X X Y
λc,a (a) = λc (c) p (c|pa (c)) ρi,c (i)
c i∈pa(c)\a i∈pa(c)\a

Y
ρb,d (b) = ρb (b) λi,b (b)
i∈ch(b)\d

The initialisation is as follows: For all evidential nodes i set

• ρi (i) = 1 for node i in the evidential state, 0 otherwise.
• λi (i) = 1 for node i in the evidential state, 0 otherwise.
For all non-evidential nodes i with no parents, set ρi (i) = p(i). For all non-
evidential nodes i with no children, set λi (i) = 1.
For every non-evidential node i we then iterate:
a) : If i has received the ρ-messages from all its parents, calculate ρi (i).
b) : If i has received λ-messages from all its children, calculate λi (i).
c) : If ρi (i) has been calculated, then for every child j of i such that i has received
the λ-messages from all of its other children, calculate and send the message
ρi,j (i).
d) : If λi (i) has been calculated, then for every parent j of i such that i has
received the ρ-messages from all of its other parents, calculate and send the
message λi,j (j).
 98

A B

C∗ D E∗

F G

Figure 11.7: A simple singly connected distribution. Here variables C and E are
clamped into evidential states, and we wish to infer the marginals p(x|c, e) for the
remaining unclamped variables x ∈ {a, b, d, f, g}

Repeat the above (a,b,c,d) until all the λ and ρ messages between any two adjacent
nodes have been calculated. For all non-evidential nodes i compute ρi (i) λi (i).
The marginal p(i|evidence) is then found by normalising this value.
For binary valued nodes, both the λ and ρ messages for each node are binary
vectors, expressing the messages as a function of the two states that the node can
exist in. It is only the relative value of the messages in their two states which is
important. For this reason, we are free to normalise both λ and ρ messages, which
is useful in avoiding overflow and underflow problems.
The complexity of belief propagation is time exponential in the maximum family
size and linear in space.
One of the main benefits of BP is that we can define a completely local algorithm.
In the BP algorithm above, nodes had to wait for certain messages before they
could pass other messages. However, if we start with some randomly chosen mes-
sages used for the nodes which are not in the initialisation procedure, then we
claim that we can calculate the messages in any order. Provided that a sweep is
made through all the nodes in the graph, then the messages will, after at most n
iterations, converge to the correct messages given by BP. We will not prove this
here, although the intuition why this is true is clear: Imagine a simple graph which
is a chain. If we start somewhere in the middle of the chain, and calculate the
messages, then these will be incorrect. However, provided that we sweep through
all the nodes, eventually, we will hit the end points of the chain. Since the ini-
tialisation of the end nodes is correct, the messages from these end nodes will be
calculated correctly. Similarly, when we repeat a sweep through all the nodes in
the graph, we will pass through nodes which are adjacent to the end nodes. Since
the messages from the end nodes are correct, the messages from the nodes adjacent
to the end nodes will also be correct. We see therefore that the correct messages
are filled in from the ends, one by one, and that eventually, after at most n sweeps
though all the nodes, all the messages will be calculated correctly. This intuition
holds also for the more general case of singly-connected graphs.

11.4.3 Example : Directed Belief Propagation

Let’s perform inference for the distribution fig(11.7), in which c and e are eviden-
tial. To denote that c and e are clamped into some particular state, we use the
notation p∗ (c|a) and p∗ (e) which sets these tables so that the states of c and e
which do not correspond to the clamped states, have zero probability.
 99

λc→a (a) = p∗ (c|a)

X
λf →d (d) = p(f |d) = 1
f

ρb→d (b) = p(b)

ρe→g (e) = p∗ (e)
X X
λg→d (d) = p(g|d, e)ρe→g (e) = ρe→g (e) = 1
g,e e
X
ρa→d (a) = λc→a (a) p(a)
c
X
λd→b (b) = λg→d (d) λf →d (d) p(d|a, b)ρa→d (a)
d
X
λd→a (a) = λg→d (d) λf →d (d) p(d|a, b)ρb→d (b)
d
X
ρd→f (d) = λg→d (d) p(d|a, b)ρa→d (a) ρb→d (b)
a,b
X
ρd→g (d) = λg→d (d) p(d|a, b)ρa→d (a) ρb→d (b)
a,b

Multiple Queries

Now that the messages have been calculated, we can easily find all the marginals
using final values for the messages.
For example,
X
p(d|c, e) ∝ λf →d (d) λg→d (d) p(d|a, b)ρa→d (a) ρb→d (b)
a,b

Party Animal example

The party animal corresponds to the network in fig(11.8). Then, given that we
observe that the Boss is Angry, and that the worker has a Headache, we wish to
find the probability hat the worker has been to a party.
To complete the specifications, the probabilities are given as follows:

p(u = T |p = T, d = T ) = 0.999
p(u = T |p = F, d = T ) = 0.9
p(u = T |p = T, d = F ) = 0.9
p(u = T |p = F, d = F ) = 0.01
p(d = T ) = 0.05, p(h = T |p = T ) = 0.9, p(h = T |p = F ) = 0.1
p(a = T |u = T ) = 0.99, p(a = T |u = F ) = 0.2

Let’s perform inference for the distribution fig(11.7), in which c and e are eviden-
tial. To denote that c and e are clamped into some particular state, we use the
 100

P D

H∗ U

A∗

Figure 11.8: All variables are binary. When set to 1 the statements are true:
P = Been to Party, H = Got a Headache, D = Demotivated at work, U =
Underperform at work, A =Boss Angry. The stars denote that the variables are
observed in the true state.

notation p∗ (c|a) and p∗ (e) which sets these tables so that the states of c and e
which do not correspond to the clamped states, have zero probability.

 
0.9 true state, T
λh→p (p) = p(h∗ |p) =
0.1 false state, F
 
0.99
λa→u (u) = p(a∗ |u) =
0.2
 
0.05
ρd→u (d) = p(d) =
0.95
 
X X 0.9150
λu→p (p) = λa→u (u) p(u|p, d)ρd→u (d) =
0.2431
u d
     
0.1 0.9 0.9150
p(p|h, a) ∝ p(p)λh→p (p) λu→p (p) = · ·
0.9 0.1 0.2431
 
0.7901
p(p|h, a) =
0.2099

11.5 Belief Revision

Belief propagation finds the marginal distribution of nodes in the graph. However,
we may be interested in what is the most likely state of the nodes in the graph.
That is
n
Y
argmax p (x) = argmax p (xi |pa (xi )) (11.5.1)
x x
i=1

We can exploit the independency structure of the graph just as we did in belief
propagation. That is, we can distribute the maximization operator over the net-
work, so that only local computations are required. In fact, the only difference
between the belief revision and the belief propagation algorithms is that wherever
there was a summation in belief propagation, it is replaced with a maximization
operation in belief revision.
To see more clearly why this is the case, consider a simple function which can be
represented as an undirected chain,

f (x1 , x2 , x2 , x4 ) = φ(x1 , x2 )φ(x2 , x3 )φ(x3 , x4 )

 101

and that we wish to find the joint state x∗ which maximises f . Firstly, let’s calcu-
late the value of the state that maximises f (the corresponding state is straight-
forward to find by backtracking) :

max f (x) = max φ(x1 , x2 )φ(x2 , x3 )φ(x3 , x4 )

x x1 ,x2 ,x3 ,x4

= max φ(x1 , x2 )φ(x2 , x3 ) max φ(x3 , x4 )

x1 ,x2 ,x3 x4
| {z }
γ(x3 )

= max φ(x1 , x2 ) max φ(x2 , x3 )γ(x3 )

x1 ,x2 x3
| {z }
γ(x2 )

= max φ(x1 , x2 )γ(x2 )

x1 ,x2

It is clear that the chain structure of the function, coupled with the maximisation
operation, which may be distributed over the function, that the maximal value
(and its state) can be computed in time which scales linearly with the number of
factors in the function.
Note that there is no requirement here that the function f corresponds to a prob-
ability distribution. This hints that there is a more general class of functions and
operations on them that permit computational simplifications, as described in the
next section.

11.6 The Generalised Distributive Law

The GDL generalises the idea of how operations may be distributed over functions
described in terms of products[16], and is based on the algebra of commutative
semirings.

Commutative Semiring

A commutative semiring is a set K, together with two binary operations called

“+” and “·”, which satisfy the following three axioms:
S1 The operation “+” is associative and commutative, and there is an additive
identity element called “0” such that k + 0 = k for all k ∈ K. (This axiom
makes (K, +) a commutative monoid.)
S2 The operation “·” is also associative and commutative, and there is a multi-
plicative identity element called “1” such that k · 1 = k for all k ∈ K. (Thus
(K, ·) is also a commutative monoid.)
S3 The distributive law holds,

(a · b) + (a · c) = a · (b + c)

i.e., for all triples (a, b, c) from K.

Then consider that we wish to “marginalize a product function” (MPF), as in
the case where “+” indeed corresponds to addition and · to multiplication in
the case of finding marginals in probability distributions written as products of
factors (undirected representation). The standard computation of marginals in
undirected graphical models corresponds in the GDL case to the so-called sum-
product algorithm, since this is the form that the semiring takes in this case.
 102

A B

C D E

F G

Figure 11.9: A multiply connected graph.

B C

Figure 11.10: A part of a network which contains a loop.

The most-probable value of an undirected graphical model corresponds to MPF

by setting “ + ” to the max operation, 0 to 0, · to multiplication and 1 to 1. This
is called the max-product realisation of the GDL.
The properties of the semiring enable us to distribute the operations to compute
the MPF (remember that what the MPF is corresponds to the particular realisation
of the semiring). The idea is that one can then define local messages within the
semiring structure (as we saw above for the case of sum-product and max-product
definitions of γ messages on a chain). These messages can be generalised to graphs
which are multiply connected. This is most easily achieved using the Junction
Tree structure, as described in the next chapter.
An important point here is that there is no requirement that the algebra cor-
responds to a probability distribution. Indeed, this is why the area of dynamic
programming[17] is not thought of as directly related to computations in graphical
models.

11.7 Inference in Multiply-Connected Graphs

When we consider graphs which contain a loop (there is more than one path
between two nodes in the graph when the directions are removed), we run into
some difficulty with the belief propagation idea.
To offer an intuitive explanation why cycles cause difficulties, consider the appli-
cation of belief propagation to the graph in fig(11.10). Since there is a loop, the
initialisation procedure cannot be carried out such that we will be in a position to
calculate messages. In order to begin the procedure, therefore, we need to make
an assumption for at least one of the messages. Assume that we have assigned a
value to ρa,b (a). Since B has received its ρ messages from all its parents, and λ
messages from all its other children (there are none), we can calculate the message
 103

ρb,d (b):
X
ρb,d (b) = p(b|a)ρa,b (a)
a

Now we can find the message

X
λd,c (c) = p(d|b, c)ρb,d (b) λD (d)
b,d

X
λc,a (a) = p(c|a)λd,c (c)
c

Finally, we are in a position to calculate the message

ρa,b (a) = ρ(a)λc,a (a)

We now have a consistency requirement:

ρa,b (a) = ρ(a)λc,a (a)

X
= ρ(a) p(c|a)λd,c (c)
c
X X
= ρ(a) p(c|a) p(d|b, e)ρb,d (b) λD (d)
c b,d
X X X
= ρ(a) p(c|a) p(d|b, c) p(b|a)ρa,b (a) λD (d)
c b,d a

Our initial assignment for ρa,b (a) gets updated after we have cycled once around
the loop. If there are other loops in the graph, the messages from this loop will
get passed to other loops, which will feed back messages to this loop in return.
There is no guarantee that the messages will ultimately converge to something
consistent.

11.7.1 Conditioning

One way to solve the difficulties of multiply connected (loopy) graphs is to identify
nodes that, if were not present, would mean that the reduced graph was singly-
connected[5].
Consider the example if fig(11.11). Imagine that we wish to calculate a marginal,
say p(D). Then
X X
p(d) = p(c|a)p(a) p(d|a, b)p(b) p(f |c, d) p(g|d, e)
c
| {z } | {z }
a,b,e,f,g
p∗ (a) p∗ (f |d)

where the p∗ definitions are not necessarily distributions. It is clear that, for
each state of c, the form of the products of factors remaining as a function of
a, b, e, f, g is singly-connected, and that therefore, standard propagation (sum-
product) methods can be used to perform inference. We will need to do this for as
many states as there are in variable c, each state defining a new singly-connected
graph (with the same structure) but modified potentials.
More generally, we can define a set of variables C, called the loop-cut set. So,
for the price of a factor exponential in the loop-cut size, we can calculate the
 104

A B A B

C D E C D E

F G F G
(a) (b)

Figure 11.11: (a) A multiply connected graph reduced to a singly connected graph
(b) by conditioning on the variable C.

A B C D A,B B B,C C C,D

(a)
(b)

Figure 11.12: (a) A belief network. (b) A cluster graph representation of the
network. The cluster potentials are defined on the round/oval nodes, and the
separator potentials are defined on the square nodes, which share common variables
with their neighbours.

marginals for any (loopy) DAG. The computationally difficult part of conditioning
is in determining a small cut set, and there is no guarantee that this will anyway
be small for a given graph.
Whilst this method is able to handle loops in a general manner, it is not particularly
elegant, and no more efficient than the Junction Tree approach, as described in
the next chapter.

11.8 Cluster Graphs

Consider a directed chain
p(U ) = p (a|b) p (b|c) p (c|d) p(d) (11.8.1)
where U , the “universe” represents all the variables in the graph. A cluster graph
is defined as a set of potentials ψCi , i = 1, . . . C, where Ci is a cluster of variables,
and ψCi is a function on this set of variables. Each cluster potential is represented
by a node in the cluster graph. A separator is defined as a potential on the
variables from the intersection of Ci and Cj . The separator is drawn as a square
node between the two clusters it separates. The cluster graph distribution is then
defined as the product of all the cluster potentials, divided by the product of the
separator potentials.
In fig(11.12), for the cluster graph to represent the BN we need
Ψ (a, b) Ψ (b, c) Ψ (c, d)
p(U ) = p (a|b) p (b|c) p (c|d) p(d) =
Ψ (b) Ψ (c)
One such assignment of the cluster and separator potentials to satisfy this would be
Ψ (a, b) = p (a|b), Ψ (b, c) = p (b|c), Ψ (c, d) = p (c|d) p(d), and Ψ (b) = 1, Ψ (c) = 1.
Note that here we have defined the potentials to be functions of the (cluster) node
of variables, in contrast to the example of an undirected graph given in a previ-
ous chapter, where the potentials where functions of the links and not of the nodes.
 105

For every cluster representation, we claim that there exists another cluster rep-
resentation for which the clusters contain the marginals of the distribution. For
example, from the definition of conditional probability, we can rewrite equation
(11.8.1) as
p (a, b) p (b, c) p (c, d) Ψ∗ (a, b) Ψ∗ (b, c) Ψ∗ (c, d)
p(U ) = =
p(b)p(c) Ψ∗ (b) Ψ∗ (c)
Where the cluster and separator potentials are set to Ψ∗ (a, b) = p (a, b), Ψ∗ (b, c) =
p (b, c), Ψ (c, d) = p (c, d), and Ψ∗ (b) = p(b), Ψ∗ (c) = p(c). It turns out that every
singly-connected graph can always be represented as a product of the the clique
marginals divided by the product of the separator marginals – this is a useful and
widely applied result in the development of exact and approximate inference algo-
rithms.

Directed graphs which are trees will always have a cluster representation of this
form, since each factor in the distribution can be written using Bayes’ rule, which
will add a clique marginal and corresponding separator.

Cluster representation of Undirected Graphs

In the previous section, we saw an example of a directed graphical model that has
a cluster representation. What about undirected graphical models – do they also
have a cluster graph representation? In an undirected graph, each link contains a

A B C D
(a)

Figure 11.13: (a) An undirected graph.

potential. Hence, the distribution in fig(11.13) is

p(a, b, c, d) = φ(a, b)φ(b, c)φ(c, d)
Let’s try to transform this into a different representation based on marginals of
this distribution :
X X
p(a, b) = p(a, b, c, d) = φ(a, b) φ(b, c)φ(c, d)
c,d c,d
X X X X
p(b, c) = p(a, b, c, d) = φ(b, c) φ(a, b)φ(c, d) = φ(b, c) φ(a, b) φ(c, d)
a,d a,d a d
X X
p(c, d) = p(a, b, c, d) = φ(c, d) φ(a, b)φ(b, c)
a,b a,b

Using these, we can rewrite the distribution as

p(a, b) p(b, c) p(c, d)
p(a, b, c, d) = P P P P
c,d φ(b, c)φ(c, d) a φ(a, b) d φ(c, d) a,b φ(a, b)φ(b, c)

Let’s examine the denominator. This can be rewritten as

X X
φ(a, b)φ(b, c)φ(c, d) φ(a, b)φ(b, c)φ(c, d)
a,c,d a,b,d
| {z } | {z }
p(b) p(c)
 106

which, gives simply:

p(a, b)p(b, c)p(c, d)

p(a, b, c, d) =
p(b)p(c)

and hence has the same cluster potential representation as fig(11.12b).

Indeed, one can show that any singly-connected undirected graph can be repre-
sented in the same way, as a product of clique potentials divided by separator
potentials, and that there exists a setting for these values for which the cliques
contain the marginals (on the variables in each clique) of the distribution.

Variable Elimination on Singly-Connected Graphs

The previous sections discussed how we can always represent a singly-connected

graphical model (whether directed or undirected) in the form of a cluster repre-
sentation1 .
However, we didn’t specify explicitly an algorithm. It is clear that we can calculate
any marginal by simple variable elimination, in which we start at the leaves of the
tree, and eliminate the variable there, and then work inwards, nibbling off each
time a leaf of the remaining tree. This is guaranteed to enable us to calculate any
marginal p(xi ) in roughly the number of variables in the graph multiplied by the
largest summation.
A key point for a tree (singly-connected graph) is that provided we perform elimi-
nation from the leaves inwards, then the structure of the remaining graph is simply
a subtree of the original tree, albeit with modified potentials.
The dissatisfying thing about this is that we would need to rerun variable elimi-
nation each time afresh to calculate a new marginal.

11.9 KL divergence approach to marginalisation on Trees

Since we know that the distribution has the form of the cluster graph representa-
tion, we could possibly minimise the KL divergence between the cluster representa-
tion and the original distribution (see appendix). Consider again the distribution

p = φ1 (a, b)φ2 (b, c)φ3 (c, d)

We know that this has a representation of the form

q(a, b)q(b, c)(c, d)

q=
q(b)q(c)

In order to find the settings of q that make a perfect match between the distribu-
tions q and p, we can minimise the Kullback-Leibler divergence between the two
1 In the directed case, we need to carry out a moralisation step– see later section
 107

distributions (see appendix). Thanks to the structured form of q and p, this is

X X X
KL(q||p) = q(a, b) log q(a, b) + q(b, c) log q(b, c) + q(c, d) log q(c, d)
a,b b,c c,d
X X
− q(b) log q(b) − q(c) log q(c)−
b c
X X X
q(a, b) log φ1 (a, b) − q(b, c) log φ2 (b, c) − q(c, d) log φ3 (c, d)−
a,b b,c c,d
(11.9.1)

However,
P we have bear
P in mind that P there are consistency
P constraints, namely
a q(a, b) = q(b) = c q(b, c) and b q(b, c) = q(c) = d q(c, d). In addition,
there are constraints that each probability must sum to 1. (I’ll ignore these since
they do no interact with other terms, and just simply will constitute a rescaling of
the tables).
We can enforce these constraints by adding Lagrange multipliers (the reason for
the labeling of the Lagrange multipliers will become clearer later):
!
X X
γ21 (b) q(a, b) − q(b)
b a
!
X X
γ12 (b) q(b, c) − q(b)
b c
!
X X
γ32 (c) q(b, c) − q(c)
c b
!
X X
γ23 (c) q(c, d) − q(c)
c d

Let’s now differentiate wrt q(b, c). This gives, at the extreme point

log q(b, c) − log φ2 (b, c) − γ12 (b) − γ32 (c) = 0

Or

q(b, c) ∝ φ2 (b, c)λ12 (b)λ32 (c)

where λ(x) = exp(γ(x)). Similarly,

q(a, b) ∝ φ1 (a, b)λ21 (b)

and

q(c, d) ∝ φ3 (c, d)λ23 (c)

Similarly, we can differentiate wrt q(b) :

log q(b) − γ21 (b) − γ12 (b)

Or

q(b) ∝ λ21 (b)λ12 (b)

 108

Similarly,

q(c) ∝ λ32 (c)λ23 (c)

P
The marginalisation constraint a q(a, b) = q(b) gives
X
φ1 (a, b)λ21 (b) ∝ λ21 (b)λ12 (b)
a

or
X
φ1 (a, b) ∝ λ12 (b)
a
P
Similarly, c q(b, c) = q(b) gives
X
φ2 (b, c)λ12 (b)λ32 (c) ∝ λ21 (b)λ12 (b)
c

or
X
φ2 (b, c)λ32 (c) ∝ λ21 (b)
c
P
The constraint, b q(b, c) = q(c) gives
X
φ2 (b, c)λ12 (b)λ32 (c) ∝ λ32 (c)λ23 (c)
b

or
X
φ2 (b, c)λ12 (b) ∝ λ23 (c)
b
P
And finally, d q(c, d) = q(c) gives
X
φ3 (c, d)λ23 (c) ∝ λ32 (c)λ23 (c)
d

or
X
φ3 (c, d) ∝ λ32 (c)
d

Note that the normalisation constants in the λ messages can be dropped since we
do not need to know these – they are only used at the end to calculate the nor-
malisation of the marginals. Hence, the proportions may be written as equalities.
This gives
X
λ12 (b) = φ1 (a, b) (11.9.2)
a
X
λ23 (c) = φ2 (b, c)λ12 (b) (11.9.3)
b
X
λ32 (c) = φ3 (c, d) (11.9.4)
d
X
λ21 (b) = φ2 (b, c)λ32 (c) (11.9.5)
c
 109

These equations have a definite starting point (at each end), and can then be
solved deterministically. This means that, for trees (and singly-connected struc-
tures) there is only one fixed point, and this is indeed the minimum zero KL
divergence solution. These equations define the Belief Propagation Algorithm and
demonstrates why (which was based on the fact that a tree has this representa-
tion) the solution found by iterating these equations is unique, and is the global
minimum. Note that when the equations are used in the order defined above, then
we always have enough information in the preceding equations in order to define
the next equation. This is always the case for trees, and any ordering which starts
from the leaves inwards will be fine. Note that the final marginals are determined
directly from the λ messages, as given in the previous equations – just a simple
normalisation is required.
There is an interesting property of these Belief Propagation equations, namely
that they can be parallelised, since the forward and backward equations for the
messages do not interfere. Once these two sets of messages have been calculated,
they can be locally combined to produce marginals. This is in contrast to the
Hugin scheme below which cannot be parallelised in the same way – however, in
the case of trees with many branches, the Hugin approach may be slightly more
efficient since it does not need to calculate products of messages, which is the case
for the Belief Propagation style approach.

An Alternative : Hugin

We can rewrite the above equations in a manner that will be more efficient in some
cases, particularly when there are many branches in the cluster graph.
Recall that for the running example we have been considering, the cluster graph
is of the form :
φ(a, b)φ(b, c)φ(c, d)
q(a, b, c, d) =
φ(b)φ(c)
where we initially set φ(b) and φ(c) to be the identity functions, and potentials in
the numerator to those that make a match with the distribution p.
The forward equations for λ can be run independently of the backward equations
since the two sets of equations do not interfere. We consider here an alterna-
tive, in which we run first the forward equations, and subsequently the backward
equations.
Let’s start with λ12 (b). We shall call this a new potential:
X
φ∗ (b) = φ(a, b)
a

If we wish this new potential to replace the old φ(b), but q to remain unchanged
(this is desired if the initialisation corresponds to the correct distribution p), then
we need to make the change:
∗
φ(a, b)φ(b, c) φφ(b)
(b)
φ(c, d)
q=
φ∗ (b)φ(c)
Let’s therefore define
φ∗ (b)
φ∗ (b, c) = φ(b, c)
φ(b)
 110

so that q is simply
φ(a, b)φ∗ (b, c)φ(c, d)
q=
φ∗ (b)φ(c)
Now consider λ23 (c). We shall call this a new potential φ∗ (c). A simple substitution
reveals
X
φ∗ (c) = φ∗ (b, c)
b

Again, in order to leave the distribution unchanged, we make define

φ∗ (c)
φ∗ (c, d) = φ(c, d)
φ(c)
Now, according to our equation q(c, d) ∝ φ(c, d)λ2,3 (c), and since φ(c) = 1, then
φ∗ (c, d) ∝ p(c, d)
Now, rather than simply using the backward λ equations, we will use the marginal
equations
q(b, c) ∝ φ(b, c)λ1,2 (b) λ3,2 (c)
| {z }
φ∗ (b,c)

Using q(c) ∝ λ3,2 (c)λ2,3 (c), we have

φ∗ (b, c)q(c)
q(b, c) ∝
φ∗ (c)
But, since q(c) ∝ φ∗∗ (c), we may define
φ∗∗ (c)
φ∗∗ (b, c) = φ∗ (b, c)
φ∗ (c)
and this will be proportional to the marginal p(b, c). Similarly, a final step shows
that
X φ∗∗ (b)
φ∗∗ (b) = φ∗∗ (b, c), and φ∗∗ (a, b) = φ(a, b) ∗
c
φ (b)

is proportional to p(a, b).

These equations have the pleasant form that, in general one defines a new separator:
X
φ∗ (s) = φ(v)
v\s

and potential
φ∗ (s)
φ∗ (w) = φ(w)
φ(s)
Once one has then done a full forward sweep and backward sweep of these equa-
tions, the final potentials will be proportional to the marginal. Similarly, one can
show that the final separators will be also proportional to the marginal of the
separator variable). This is a reformulation of the Belief Propagation equations
based on defining separators and new cluster potentials. The benefit of doing this
in lieu of the BP approach is not immediately obvious. Hopefully it will become
more apparent in the general case below.
 111

The General Case on a Singly-Connected graph

Let vi be a set of variables on a cluster φ(vi ). Our task is, from the general
distribution
Y
p(v) = φ(vi )
i

To find a representation
Q
q(vi )
q(v) = Q i
<ij> q(vij )

here vij = vi ∩ vj , < ij > means the set of pairs i, j for P which cluster i is a
neighbour of cluster j. v is the union of all the vi . Below vi f (vi ) means the
sum of the clique function f (vi ) over all the joint states of the set of variables vi .
Then KL(q||p) may be written as
XX XX XX
q(vi ) log q(vi ) − q(vij ) log q(vij ) − q(vi ) log ψ(vi )
i vi <ij> vij i vi
 
XX X
− γji (vij )  q(vi ) − q(vij ) (11.9.6)
<ij> vij vi \vij

(as before, the normalisation constraints have been omitted since they play only a
trivial non-interacting role).
Differentiating wrt to q(vi ) and q(vij ) we arrive at
Y
q(vi ) ∝ ψ(vi ) λji (vij )
j∈n(i)

where n(i) is the set of neighbours of cluster i, and

q(vij ) ∝ λij (vij )λji (vij )

Using the marginalisation constraint

X
q(vi ) = q(vij )
vi \vij

we have
X Y
ψ(vi ) λki (vik ) = λij (vij )λji (vij )
vi \vij k∈n(i)

The term λji (vji ) cancels to give

X Y
λij (vij ) = ψ(vi ) λki (vik )
vi \vij k∈n(i),k6=j

which are the usual Belief Propagation equations. This is also called the Shenoy-
Shafer updating scheme. However, if we didn’t cancel the λ terms, we could rewrite
X
q old (vi ) = q new (vij ) (11.9.7)
vi \vij
 112

Hence, we can form updates for the separator potentials in this fashion. What
about the potentials in the numerator? In order to do this, we need to invoke an-
other piece of information. One approach is that if we made an initial assignment
that the q was equal to p (say by making a simple assignment of the numerators,
with the separators set to unity), then we need q to be invariant under the trans-
formation equation (11.9.7). The separator q(vij ) always occurs with a numerator
term q(vj ), and hence the requirement that the distribution q remains equal to p
reduces to the requirement

q new (vj ) q old (vj )

new
= old
q (vij ) q (vij )
or
q new (vij )
q new (vj ) = q old (vj ) (11.9.8)
q old (vij )

This form of updates is called Hugin propagation, and together the equations equa-
tion (11.9.8) and equation (11.9.7) form a procedure called absorption.

11.10 Problems
Exercise 22 Consider the distribution p(a, b, c) = p(c|a, b)p(a)p(b). If we clamp
b into an evidential state, what effect will this have on a? Explain your answer
intuitively.

Exercise 23 Consider the belief network given below, which concerns the proba-
bility of a car starting.

P(b=bad) = 0.02 P(f=empty) = 0.05

Battery Fuel

Gauge
P(g=empty|b=good, f=not empty) = 0.04
P(g=empty| b=good, f=empty) = 0.97
P(g=empty| b=bad, f=not empty) = 0.10
P(g=empty|b=bad, f=empty) = 0.99
Turn Over

P(t=no|b=good) = 0.03
P(t=no|b=bad) = 0.98 Start

P(s=no|t=yes, f=not empty) = 0.01

P(s=no|t=yes, f=empty) = 0.92
Heckerman (1995) P(s=no| t = no, f=not empty) = 1.0
P(s=no| t = no, f = empty) = 1.0

Calculate P (f = empty|s = no), the probability of the fuel tank being empty con-
ditioned on the observation that the car does not start. Do this calculation “by
hand”, i.e. do not use or create a computer program to do this.
 113

Exercise 24 Consider the Asia Bayesian Network represented below. Calculate

by hand the values for p(D), p(D|S = yes), p(D|S = no).

A S

T L B

D
X

The table values are:

p(a = yes) = 0.01

p(s = yes) = 0.5
p(t = yes|a = yes) = 0.05
p(t = yes|a = no) = 0.01
p(l = yes|s = yes) = 0.1
p(l = yes|s = no) = 0.01
p(b = yes|s = yes) = 0.6
p(b = yes|s = no) = 0.3
p(e = yes|t, l) = 0 only if both T and L are“no”, 1 otherwise.
p(x = yes|e = yes) = 0.98
p(x = yes|e = no) = 0.05
p(d = yes|e = yes, b = yes) = 0.9
p(d = yes|e = yes, b = no) = 0.3
p(d = yes|e = no, b = yes) = 0.2
p(d = yes|e = no, b = no) = 0.1
 114

11.11 Solutions
24 (Thanks to Peter Mattsson for typesetting this) The marginal p(d) can be cal-
culated as
X
p(d) = p(a, s, t, l, b, e, x, d)
a,s,t,l,b,e,x
X
= p(a)p(s)p(t|a)p(l|s)p(b|s)p(e|t, l)p(x|e)p(d|b, e)
a,s,t,l,b,e,x
!
X X
= p(a)p(s)p(t|a)p(l|s)p(b|s)p(e|t, l)p(d|b, e) p(x|e)
a,s,t,l,b,e x
!
X X
= p(s)p(l|s)p(b|s)p(e|t, l)p(d|b, e) p(t|a)p(a) ,
s,t,l,b,e a
P
where we have noted that x p(x|e) = 1. The final term on the RHS is just the
marginal p(t), which is

p(t = yes) = p(t = yes|a = yes) × p(a = yes) + p(t = yes|a = no) × p(a = no)
= 0.05 × 0.01 + 0.01 × 0.99 = 0.01 × 1.04
= 0.0104.

Armed with this, we can further simplify the expression for p(d) to
!
X X
p(d) = p(s)p(l|s)p(b|s)p(d|b, e) p(e|t, l)p(t) .
s,l,b,e t

The last term on the RHS is now p(e|l), which is

p(e = yes|l = yes) =p(e = yes|l = yes, t = yes) × p(t = yes)

+ p(e = yes|l = yes, t = no) × p(t = no)
=1 × 0.0104 + 1 × 0.9896 = 1,
p(e = yes|l = no) =p(e = yes|l = no, t = yes) × p(t = yes)
+ p(e = yes|l = no, t = no) × p(t = no)
=1 × 0.0104 + 0 × 0.9896 = 0.0104.

The marginal p(d) is now

X X
p(d) = p(s)p(b|s)p(d|b, e) p(e|l)p(l|s),
s,b,e l

which we can further simplify by calculating p(e|s), which is

p(e = yes|s = yes) =p(e = yes|l = yes) × p(l = yes|s = yes)

+ p(e = yes|l = no) × p(l = no|s = yes)
=1 × 0.1 + 0.0104 × 0.9 = 0.10936,
p(e = yes|s = no) =p(e = yes|l = yes) × p(l = yes|s = no)
+ p(e = yes|l = no) × p(l = no|s = no)
=1 × 0.01 + 0.0104 × 0.99 = 0.020296.
 115

This now gives us

X X
p(d) = p(b|s)p(s) p(d|b, e)p(e|s),
s,b e

leading us to calculate p(d|b, s), which is

p(d = yes|b = yes, s = yes) =p(d = yes|b = yes, e = yes) × p(e = yes|s = yes)
+ p(d = yes|b = yes, e = no) × p(e = no|s = yes)
=0.9 × 0.10936 + 0.2 × 0.89064 = 0.276552,
p(d = yes|b = yes, s = no) =p(d = yes|b = yes, e = yes) × p(e = yes|s = no)
+ p(d = yes|b = yes, e = no) × p(e = no|s = no)
=0.9 × 0.020296 + 0.2 × 0.979704 = 0.2142072,
p(d = yes|b = no, s = yes) =p(d = yes|b = no, e = yes) × p(e = yes|s = yes)
+ p(d = yes|b = no, e = no) × p(e = no|s = yes)
=0.3 × 0.10936 + 0.1 × 0.89064 = 0.121872,
p(d = yes|b = no, s = no) =p(d = yes|b = no, e = yes) × p(e = yes|s = no)
+ p(d = yes|b = no, e = no) × p(e = no|s = no)
=0.3 × 0.020296 + 0.1 × 0.979704 = 0.1040592.

We now have
X X
p(d) = p(s) p(d|b, s)p(b|s),
s b

so we calculate p(d|s), which is

p(d = yes|s = yes) =p(d = yes|b = yes, s = yes) × p(b = yes|s = yes)
+ p(d = yes|b = no, s = yes) × p(b = no|s = yes)
=0.276552 × 0.6 + 0.121872 × 0.4 = 0.21468,
p(d = yes|s = no) =p(d = yes|b = yes, s = no) × p(b = yes|s = no)
+ p(d = yes|b = no, s = no) × p(b = no|s = no)
=0.2142072 × 0.3 + 0.1040592 × 0.7 = 0.1371036.
P
Now, at last, we can calculate p(d) = s p(d|s)p(s), which is

p(d = yes) =p(d = yes|s = yes) × p(s = yes) + p(d = yes|s = no) × p(s = no)
=0.21468 × 0.5 + 0.1371036 × 0.5 = 0.1758918.

Thus we have p(d = yes) = 0.1758918, p(d = yes|s = yes) = 0.21468 and
p(d = yes|s = no) = 0.1371036.
12 The Junction Tree Algorithm

Parts of this chapter are based on Expert Systems and Probabilistic Network Model
by E. Castillo, J. Gutierrez, and A. Hadi (Springer, 1997), and also An introduction
to bayesian networks by F.V.Jensen (Springer 1996). Both are excellent introduc-
tions to the field.
One may have had the nagging suspicion during the discussion of the algorithms
relating to DAGs, that what we are really exploiting is the underlying graphical
structure, whether this be directed or not. Does it really matter if the graph
is directed? For example, in bucket elimination, we are essentially removing the
directedness of the graph and defining (undirected) functions in their place. What
this suggests is that the complexity of the calculations on directed graphs can be
transformed into an undirected graph, possibly of greater connectivity than the
directed graph from which it was derived.
Indeed, there is an algorithm that does this. A graph, directed or undirected, is
transformed into an undirected graph on which the relevant computations can be
performed. This is called the junction tree algorithm.
I’ll derive the algorithm in the context of graphical models (probability distribu-
tions). The reader should bear in mind that a more general approach shows how
the Junction Tree algorithm is also appropriate for forming a structure for which
the Generalised Distributive Law of the previous chapter is guaranteed to work for
a graph of any structure, singly or multiply-connected.

12.1 Absorption and Marginal Consistency

As we saw in chapter(11), any singly connected graph can be represented in terms
of a product of clique marginal distributions, divided by the product of the sepa-
rator distributions. This can be proved by induction. Furthermore, we can begin
with a cluster potential assignment, and modify the cluster potentials so that, at
the end of the calculation, the modified cluster potentials contain the marginals
distributions, and similarly the separators are marginal distributions.
An alternative exposition comes from the following consistency argument, whose
generaliation will lead to the Junction Tree Algorithm (JTA).
Consistent link Consider a cluster representation with neighbouring clusters v and w, sharing the
variables s in common. If our aim is that the JTA modifies the potentials such that
the marginal of the distribution p(w) is given by the (modified) potential Ψ (w),
then1
X
p(s) = Ψ (w)
w\s

1 Note that, in the beginning, the assignment of the cluster potentials does not satisfy the
consistency requirement. The aim is to find an algorithm that modifies them so that ultimately
consistency is achieved.

116
 117

Ψ (V) Ψ (S) Ψ (W)

Figure 12.1: In absorption, we consider two neighbouring cliques in the cluster

graph, which contain the potentials Ψ(V ) and Ψ(W ), with intersection variables
S ≡ V ∩ W.

Ψ (V) Ψ*(S) Ψ*(W) W absorbs from V

Similarly,
X
p(s) = Ψ (v)
v\s

This then requires

X X
Ψ (w) = p(s) = Ψ (v) ,
w\s v\s

We identify the (modified) separator potential Ψ (s) = p(s).

Imagine that by some process, we have managed to achieve consistency, but that
now some new evidence changes Ψ (v) to Ψ∗ (v) (this is achieved by clamping one of
the variables in v to a particular state). In order that the link remains consistent,
we need to change Ψ (w) and Ψ (s) in order to satisfy
X X
Ψ∗ (w) = Ψ∗ (s) = Ψ∗ (v) .
w\s v\s

Absorption Let v and w be neighbours in a cluster tree, let s be their separator, and let
Ψ∗ (v), Ψ (w) and Ψ (s) be their potentials. Absorption replaces the tables Ψ∗ (s)
and Ψ∗ (w) with
X
Ψ∗ (s) = Ψ (v)
v\s

Ψ∗ (s)
Ψ∗ (w) = Ψ (w)
Ψ (s)

The idea behind this definition is that, under the update of the table for v, the
table for the separator s and neighbour w are updated such that the link remains
consistent. To see this consider

X X Ψ∗ (s) Ψ∗ (s) X Ψ∗ (s) X

Ψ∗ (w) = Ψ (w) = ∗ Ψ (w) = Ψ (s) = Ψ∗ (s) = Ψ∗ (v)
Ψ (s) Ψ (s) Ψ (s)
w\s w\s w\s v\s

Absorption passes a “message” from one node to another:

 118

Note that if Ψ (s) can be zero, then we need also Ψ (w) to be zero when Ψ (s) is
zero for this procedure to be well defined. In this case, the potential takes the value
unity at that point. (This requirement is on Ψ (w) and not Ψ∗ (s) since we are
considering whether or not it is possible to transmit the information through the
current state of the link). We say that a link is supportive if it allows absorption
in both directions (that is Ψ (v) and Ψ (w) will both be zero when Ψ (s) is zero).
Note that supportiveness is preserved under absorption.
Invariance of Cluster Tree Let T be a supportive cluster tree. Then the product of all cluster potentials
under Absorption divided by the product of all separator potentials is invariant under absorption.
Proof: When w absorbs v though the separator s only the potentials of w and s are
changed. It is enough therefore to show that the fraction of the w and s tables is
unchanged. We have
∗

Ψ∗ (w) Ψ (w) ΨΨ(s)

(s)
Ψ (w)
= =
Ψ∗ (s) Ψ∗ (s) Ψ (s)

So that if we start with a BN over U , construct a corresponding cluster tree T ,

and then perform a series of absorptions, then T remains a representation of p(U )
and is given by the product of all cluster potentials divided by the product of all
separator potentials.
Note how this consistency argument has reproduced the Hugin form of Belief
Propagation from chapter(11).

Propagation on Cluster Trees

Having defined the local message propagation approach, we need to define an

update ordering for absorption. In general, a node V can send exactly one message
to a neighbour W , and it may only be sent when V has received a message from
each of its other neighbours. We continue this sequence of absorptions until a
message has been passed in both directions along every link. See, for example
fig(12.1). Note that the message passing scheme is generally not unique.

A 3 4 B

C 1 6 7
D
2 9 8 5
10 E F

Figure 12.2: An example of the updating procedure for Hugin message passing.

The Hugin scheme is slightly advantageous over the standard Belief Propagation
scheme since one doesn’t need to take the products of all the messages at junctions.
However, it is not as readily parallisable as the BP scheme, since the messages are
not independent.
What we would like to do for a general distribution is to define a potential rep-
resentation of the graph such that, coupled with a suitable algorithm to modify
these potentials, the effect will be, as above, that the marginals of individual or
 119

groups (in fact cliques) can be read off directly from the modified potentials. This
is the idea of the junction tree algorithm.
Comment : When the original distribution p is singly-connected, one can show
that there is a cluster potential representation that is also singly-connected (actu-
ally, it will also be guaranteed to satisfy the running-intersection property so that
locally consistent links will propagate to ensure global consistency). In that case,
Hugin Belief Propagation may be performed. However, in the more general case of
multiply-connected graphs, it may not be clear that an appropriate cluster repre-
sentation exists (apart from trivial ones which essentially put all the variables into
one cluster). How to construct efficient and suitable cluster-trees is at the heart
of the JTA.

12.2 Junction Trees

Let T be a cluster tree over U and let a be a variable in U and suppose that
a
Pis an elementPof the nodes v and w. If T is consistent, we would expect that
v\a Ψ (v) = w\a Ψ (w). Certainly this is true it v and w are neighbours, but
otherwise there is no guarantee.
Global Consistency We say that a consistent cluster tree is globally consistent if for any nodes v and
w with intersection I we have
X X
Ψ (v) = Ψ (w)
v\I w\I

Junction Tree A cluster tree is a junction tree if, for each pair of nodes, v and w, all nodes on
Running Intersection the path between v and w contain the intersection v ∩ w. This is also called the
running intersection property.
From this definition, it is clear that, in a consistent junction tree, the local consis-
Local = Global consistency tency will be passed on to any neighbours. That is, a consistent junction tree is
globally consistent.
Marginals Let T be a consistent junction tree over U , and let Ψ (U ) be the product of all
potentials divided by the product of all separator potentials. Let v be a node with
potential Ψ (v). Then
X
Ψ (v) = Ψ (U ) = p(v)
U\v

To gain some intuition about the meaning of this theorem, consider the junction
tree in fig(12.3). After a full round of message passing on this tree, each link
is consistent, and the product of the potentials divided by the product of the
separator potentials is just the original distribution itself. Imagine that we are
interested in calculating the marginal for the node ABC. That requires summing
over all the other variables, D, E, F, G, H. If we consider summing over H then,
because the link is consistent,
X
Ψ (eh) = Ψ (e)
h
P
so that the ratio h Ψ(eh)
Ψ(e) is unity, so that the effect of summing over node H is
that the link between EH and DCE can be removed, along with the separator.
 120

ABC

C C

DCE CF

E E

EG EH

Figure 12.3: A junction tree. This satisfies the running intersection property that
for any two nodes which contain a variable a, the path linking the two nodes also
contains the variable a.

The same happens for the link between node EG and DCE, and also for CF to
ABC. The only nodes remaining are now DCE and ABC and their separator C,
which have so far been unaffected by the summations. We still need to sum out
over D and E. Again, because the link is consistent,
X
Ψ (d, c, e) = Ψ (c)
de
P
so that the ratio de Ψ(d,c,e)
Ψ(c) = 1. The result of the summation of all variables
not in ABC therefore produces unity for the cliques and their separators, and the
summed potential representation reduces simply to the potential Ψ (a, b, c) which
is the marginal p(a, b, c). It is clear that a similar effect will happen for other
nodes. Formally, one can prove this using induction.
We can then obtain the marginals for individual variables by simple brute force
summation over the other variables in that potential. In the case that the number
of variables in each node is small, this will not give rise to any computational
difficulties. However, since the complexity is exponential in the clique size of the
Junction Tree, it is prudent to construct the Junction Tree to have minimal clique
sizes. Although, for a general graph, this is itself computationally difficult, there
exist efficient heuristics for this task.

12.3 Constructing Junction Trees for Singly-Connected Distributions

For directed distributions, an initial step is required, which is not required in the
case of undirected graphs.
Moral Graph When we construct a cluster tree corresponding to a DAG, then for all variables a
there must be a cluster v containing pa (a)∪a. We can illustrate this on a graph by
having a link between any pair of variables which must appear in the same cluster.
This means that we take the DAG, add a link between any pair of variables with
a common child, and drop the direction of the original links. The resulting graph
is the moral graph. From the moral graph you can find the clusters, namely the
cliques in the graph.
Then for both directed and undirected graphs, we may continue.
 121

A B A B

D C D C

E F E F

G H G H
(a) DAG (b) Moral Graph

ABC ABC

C C C C

DCE C CF DCE CF

E E E E

EG E EH EG EH
(c) Junction Graph (d) Junction tree

Figure 12.4: Construction of a junction tree for a singly connected DAG.

ABE B BCF C CDG

A B C D

E F G E F F G

H I J K EH FI F FJ GK
(a) DAG (b) Junction Graph

Figure 12.5: (a) A singly connected graph and (b) its junction graph. By removing
any of the links in (b) with separator F you get a junction tree.
 122

A B A B A,B,C

C D C D B,C,D

Figure 12.6: If we were to form a clique graph from the graph on the left, this would
not satisfy the running intersection property, namely that if a node appears in two
cliques, it appears everywhere on the path between the cliques. By introducing the
extra link (middle picture), this forms larger cliques, of size three. The resulting
clique graph (right) does satisfy the running intersection property (separator set
not shown). Hence it is clear that loops of length four or more certainly require
the addition of such chordal links to ensure the running intersection property in
the resulting clique graph. It turns out that adding a chord in for all loops of
length four or more is sufficient to ensure the running intersection property for
any resulting clique graph.

Junction Graph and Tree Then, between any two clusters with a non-empty intersection add a link with the
intersection as the separator. The resulting graph is called a junction graph. All
separators consist of a single variable, and if the junction graph contains loops,
then all separators on the loop contain the same variable. Therefore any of the
links can be removed to break the loop, and by removing links until you have a
tree, you get a junction tree.
Consider the graph in fig(12.5a). Following the above procedure, we get the junc-
tion graph fig(12.5b). By breaking the loop BCF − −F − −F I − −F − −F J −
−F − −BCF anywhere we obtain a junction tree.
The previous section showed how to construct a JT for a singly-connected graph.
If we attempt to do this for a multiply connected (loopy) graph, we find that the
above procedure generally does not work since the resulting graph will not neces-
sarily satisfy the running intersection property. The idea is to grow larger clusters,
such that these the resulting graph does satisfy the running intersection property.
Clearly, a trivial solution would be to include all the variables in the graph in one
cluster, and this will complete our requirements. However, of course, this does not
help in finding an efficient algorithm for computing marginals. What we need is
a sufficient approach that will guarantee that we can always form a junction tree
from the resulting junction graph. This operation is called triangulation, and it
generally increases the minimum clique size, sometimes substantially.

12.4 Junction Trees for Multiply-Connected Distributions

When there are loops, a variable elimination approach will, in general, change the
structure of the remaining graph. To see this, consider the following distribution
shown in fig(12.8),
p(a, b, c, d) = φ(a, b)φ(b, c)φ(c, d)φ(d, a)
Let’s try to make a cluster style representation as before. We clearly here have a
choice about which variable first to marginalise over. Let’s choose d:
X
p(a, b, c) = φ(a, b)φ(b, c) φ(c, d)φ(d, a)
d
 123

A B A B A B A B

D C C D C D C
(a) (b) (c) (d)

Figure 12.7: (a) An undirected graph with a loop. (b) Eliminating node D adds
a link between A and C in the subgraph. (c) The induced representation for the
graph in (a). (d) An alternative equivalent induced representation.

The remaining subgraph therefore has an extra connection between a and b.

p(a, b, c)
p(a, b, c, d) = P φ(c, d)φ(d, a)
d φ(c, d)φ(d, a)

Let’s try to replace the numerator terms with probabilties. We can do this by
considering
X
p(a, c, d) = φ(c, d)φ(d, a) φ(a, b)φ(b, c)
b

Plugging this into the above equation, we have

p(a, b, c)p(a, c, d)
p(a, b, c, d) = P P
d φ(c, d)φ(d, a) b φ(a, b)φ(b, c)

We recognise that the denominator is simply p(a, c), hence

p(a, b, c)p(a, c, d)
p(a, b, c, d) = .
p(a, c)

Hence, if we were to form a cluster graph based on products of cliques divided

by products of separators, we could use an induced representation fig(12.7c). If
we take the induced representation, and write down the cliques ABC and ACD,
divided by the separator AC, this will form a correct cluster graph.
Alternatively, we could have marginalised over variables a and c, and ended up
with the equivalant representation fig(12.7d).
Let’s look at a slightly more complex loopy undirected distribution depicted in
fig(12.8),

p = φ(a, b)φ(b, c)φ(c, d)φ(d, e)φ(e, f )φ(a, f )

As above, there are many different representations depending on which variables

A B C A B C

F E D F E D
(a) (b)

Figure 12.8: (a) An undirected graph with a loop. (b) An induced representation.
 124

1 2 5 7 10

3 4 8 9

6 11

(a) (b)

1 2 5 7 10

3 4 8 9

6 11

(c) (d)

Figure 12.9: (a) An undirected graph which is not triangulated. (b) We start the
algorithm, labeling nodes until we reach node 11. This has neighbours 6 and 8
that are not adjacent. (c) We can correct this then by adding a link between 6 and
8, and restarting the algorithm. (d) The reader may check that this is a correct
triangulation.

we decide to eliminate. However, the reader may convince herself that one such
induced representation is given by fig(12.8b).
Generally, the result from variable elimination and re-representation in terms of
the induced graph is that a link between any two variables on a loop (of length 4
or more) which do not have a chord is added. This is called triangulation. Any
triangulated graph can be written in terms of the product of marginals divided by
the product of separators.
Armed with this new induced representation, we can carry out a message propa-
gation scheme as before.

12.5 Triangulation Algorithms

Formally, we need define the triangulation operations as follows:
Chord This is a link joining two non-consecutive vertices of a loop.
Triangulation An undirected graph is triangulated if every loop of length 4 or more has a chord.
The importance of this definition derives from the following theorem.
Triangulated= ∃ Junction An undirected graph is triangulated if and only if its junction graph has a junction
Tree tree.

Alternative algorithm : Maximum Cardinality Checking

The following is an algorithm that terminates with success if and only if the graph
is triangulated[18]. It processes each node and the time to process a node is
quadratic in the number of adjacent nodes
(see http://www.cs.wisc.edu/∼dpage/cs731/).
 125

Choose any node in the graph and label it 1

For i = 2 to n
• Choose the node with the most labeled neighbours and label it i.
• If any two labeled neighbours of i are not adjacent to each other, FAIL.
end
SUCCEED
Where there is more than one node with the most labeled neighbours, the tie may
be broken arbitrarily.
This algorithm gives us a way to make a triangulation – we simply add a link
between the two neighbours that caused the algorithm to FAIL, and then restart
the algorithm.
Comment : This is one algorithm for performing triangulation. Since the complex-
ity of inference will scale exponentially (for discrete variables) with the size of the
cliques in the resulting triangulated graph, clearly it is of some interest to find a
triangulated graph with small clique sizes. However, finding the smallest possible
triangulated graph, in this sense, is an NP-hard problem[]. The above algorithm
is one of a number which are believed to be generically reasonable, although there
may be cases where an alternative algorithm may be considerably more efficient.

12.6 Finding a JT from a Triangulated Graph

Above, we claimed that, provided that we have a carried out triangulation, then
any Junction Tree consistent with the triangulation is sufficient to ensure that
propagation on the JT will produce the correct marginals. One missing piece in
this jigsaw is how to form a JT from a triangulated graph. This can be achieved
from the following Theorem

Theorem 12.6.1 Any maximal weight spanning tree is a junction tree

The weight of a tree is defined to be the sum of all the separator weights of the tree,
where the separator weight is defined as the number of variables in the separator.
A simple algorithm to find the spanning tree with maximal weight is as follows.
Start by picking the edge with the largest weight, and add this to the edge set.
Then pick the next candidate edge which has the largest weight and add this to
the edge set – if this results in an edge set with cycles, then reject the candidate
edge, and find the next largest edge weight.
Note that there may be many maximal weight spanning trees. This algorithm
provides one.

12.7 The Junction Tree Algorithm

We now have all the material we need for inference in multiply connected graphs.
We need to do the following steps :
Moralisation This is required only for directed distributions.
Triangulation From the undirected (moralised) representation, ensure that every loop of
length 4 or more has a chord.
 126

A A

B D B D

C C

E F G E F G

H I H I
(a) Original Graph (b) Moralised and Tri-
angulated

Figure 12.10: Example of the JTA. In (a) is the original loopy graph. (b) The
moralisation links are between nodes E and F and between nodes F and G. The
other additional links come from triangulation. The clique size of the resulting
clique tree (not shown) is four.

Form the Junction Tree Form a Junction Tree by forming a cluster representation from cliques of the
triangulated graphs, removing any unnecessary links in a loop on the cluster
graph. Algorithmically, this can be achieved by finding a tree with maximal
spanning weight.
Potential Assignment Assign the potentials to the cliques on the Junction Tree and assign the
separator potentials on the JT to unity.
Message Propagation Then carry out the absorption procedure until updates have been passed
along both directions of every link on the JT.
Then the clique marginals can be read off from the JT. An example is given in
fig(12.10).

Comments on the JTA

There are some interesting points about the JTA. It provides an upper bound
on the computation required to calculate marginals in the graph. This means
that there may indeed exist more efficient algorithms in particular cases, although
generally it is believed that there cannot be much more efficient approaches than
the JTA since every other approach must perform a triangulation[19, 20].
However, there are, in general, many different ways to carry out the triangulation
step. Ideally, we would like to find a triangularised graph which has minimal clique
size. However, it can be shown to be a hard-computation problem (N P -hard) to
find the most efficient triangulation. In practice, the triangulation algorithms used
are somewhat heuristic, and chosen to provide reasonable, but clearly not optimal,
performance.

12.7.1 Finding the Most Likely State

Previously, we have mainly concentrated on finding marginal probabilities, p(xi ).

However, another natural question to ask is what is the most likely state of the
 127
−3
x 10 0.018
3.5
0.016
3
0.014

2.5
0.012

2 0.01

0.008
1.5
0.006
1
0.004

0.5
0.002

0 0
−5 0 5 −5 0 5

(a) p(x) (b) p∗ (x)

Figure 12.11: p∗ (x) ∝ p(x)10 . In both figures the vertical dashed line indicates
(on the x-axis the mean value for x. Note how p∗ becomes much more peaked
around its most probable valuem, and how the mean value in p∗ shifts to be close
to the most likely value. In the limit p∗ (x) ∝ (p(x))β ,β → ∞, then the mean of
the distribution p∗ tends to the most-likely value.

distribution? There is a simple trick which will enable us to convert the JTA to
enable us to answer this2 .
In general, a probability distribution may be written as
1 Y
p= φ(xc )
Z c

where φ(xc ) is the potential for cluster c. Consider a modified distribution in which
we wish to re-weight the states, making the higher probability states exponentially
more likely than lower probability states. This can be achieved by defining
1 Y β
p∗ = φ (xc )
Zβ x
c

where β is a very large positive quantity. This makes the distribution p∗ very
peaked around the most-likely value of p, see fig(12.11).
In the JTA, we need to carry out summations over states. However, in the limit
β → ∞ it is clear that only the most-likely state will contribute, and hence that the
summation operation can be replaced by a maximisation operation in the definition
of absorption. The algorithm thus proceeds as normal, replacing the summations
with maximisations, until the final stage, whereby from the table one reads off
argmax φ(xc ) for the variables in the modified final potential on cluster c to find
xc
the most likely state.

A simple example of the JTA

Consider running the JTA on the simple graph

p(a, b, c) = p(a|b)p(b|c)p(c)

2 As with the comments at the beginning of the chapter, the reader should bear in mind that
the Generalised Distributive Law can be extended to the loopy case by using the updating
equations on the Junction Tree. In this sense, any operations within the semiring algebra are
admissible.
 128

a b c a,b b b,c
(a) (b)

Figure 12.12: (a) A belief network. (b) JTA for the network.

There are three questions we are interested in (i) What is p(b)? (ii)? What is
p(b|a = 1, c = 1) (iii) What is the likelihood of the evidence p(a = 1, c = 1)?
For this simple graph, the moralisation and triangulation steps are trivial, and the
JTA is given immediately by fig(12.12b). A valid assignment is Ψ (a, b) = p(a|b),
Ψ (b) = 1, Ψ (b, c) = p(b|c)p(c).
First let’s absorb from (a, b) through the separator b to (b, c):

Finding a marginal p(b)

First we just run the JTA as usual.

P P
• The new separator is given by Ψ∗ (b) = a Ψ (a, b) = a p(a|b) = 1.
∗
• The new potential on (b, c) is given by Ψ∗ (b, c) = Ψ(b,c)Ψ
Ψ(b)
(b)
= p(b|c)p(b)×1
1 .
P P
• The new separator is given by Ψ∗∗ (b) = c Ψ∗ (b, c) = c p(b|c)p(c).
∗∗
P
p(a|b) c p(b|c)p(c)
• The new potential on (a, b) is given by Ψ∗ (a, b) = Ψ(a,b)Ψ
Ψ ∗ (b)
(b)
= 1 .
P
This is therefore indeed equal to the marginal since c p(a, b, c) = p(a, b).
P
Also, the new separator Ψ∗∗ (b) contains the marginal p(b) since Ψ∗∗ (b) = c p(b|c)p(c) =
P
c p(b, c) = p(b).

Finding a conditional marginal p(b|a = 1, c = 1)

First we clamp the evidential variables in their states. Then we claim that the
effect of running the JTA is to produce on the cliques, the joint marginals p(a =
1, b, c = 1), p(a = 1, b, c = 1) and p(a = 1, b, c = 1) for the final potentials on the
two cliques and their separator. We demonstrate this below:
P P
• In general, the new separator is given by Ψ∗ (b) = a Ψ (a, b) = a p(a|b) =
1. However, since a is clamped in state a = 1, then the summation is not
carried out over a, and we have instead Ψ∗ (b) = p(a = 1|b).
Ψ(b,c)Ψ∗ (b)
• The new potential on the (b, c) clique is given by Ψ∗ (b, c) = Ψ(b) =
p(b|c=1)p(c=1)p(a=1|b)
1 .
P P
• The new separator is normally given by Ψ∗∗ (b) = c Ψ∗ (b, c) = c p(b|c)p(c).
However, since c is clamped in state 1, we have instead Ψ∗∗ (b) = p(b|c =
1)p(c = 1)p(a = 1|b)
Ψ(a,b)Ψ∗∗ (b) p(a=1|b)p(b|c=1)p(c=1)p(a=1|b)
• The new potential on (a, b) is given by Ψ∗ (a, b) = Ψ∗ (b) = p(a=1|b) =
p(a = 1|b)p(b|c = 1)p(c = 1).
Hence, here in this special case, all the cliques contain the joint distribution
p(a = 1, b, c = 1).
 129

In general, the effect of clamping a set of variables V in their evidential states, and
running the JTA is that, for a clique i which contains the set of non-evidential vari-
ables H i , the potentials after the end of the JTA contains the marginal p(H i , V ).
Then calculating the conditional marginal p(b|a = 1, c = 1) is a simple matter since
p(b|a = 1, c = 1) ∝ p(a = 1, b, c = 1), where the proportionality is determined by
the normalisation constraint.

Finding the likelihood p(a = 1, c = 1)

By the above procedure, the effect of clamping the variables in their evidential
states and running the JTA produces the joint marginals, such as Ψ∗ (a, b) =
p(a = 1, b, c = 1). Then calculating the likelihood is easy since we just sum out
over
P the non-evidential
P variables of any converged potential : p(a = 1, c = 1) =
∗
b Ψ (a, b) = b p(a = 1, b, c = 1).
Whilst we have demonstrated these results only on such a simple graph, the same
story holds in the general case. Hence calculating conditional marginals and like-
lihoods can be obtained in exactly the same way. The main thing to remember is
that clamping the variables in evidential states means that the joint distribution
on the non-evidential variables in a clique with all the evidential variables clamped
in their evidential states is what is found a the end of the JTA. From this condi-
tionals and the likelihood are straightforward to calculate.

12.8 Problems
Exercise 25 Consider the following undirected graphical model:

p(x1 , x2 , x3 , x4 ) = φ(x1 , x2 )φ(x2 , x3 )φ(x3 , x4 )

1. Draw a clique graph that represents this distribution, and indicate the sepa-
rators on the graph.
2. Write down an alternative formula for the distribution p(x1 , x2 , x3 , x4 ) in
terms of the marginal probabilities p(x1 , x2 ), p(x2 , x3 ), p(x3 , x4 ), p(x2 ), p(x3 )

Exercise 26 Consider the distribution

p(x1 , x2 , x3 , x4 ) = φ(x1 , x2 )φ(x2 , x3 )φ(x3 , x4 )φ(x4 , x1 )

1. Write down a Junction Tree for the above graph.

2. Carry out the absorption procedure and demonstrate that this gives the correct
value for the marginal p(x1 ).

12.9 Solutions
 13 Variational Learning and EM

13.1 Maximum Likelihood

Imagine that we have some training data, X = x1 , . . . , xP . Our model p(x|Θ) has
some parameters Θ. These could result from a parameterisation of the conditional
probability tables in a Belief Network, or indeed could be the table entries them-
selves. In general, they are simply some parameters of a probability distribution.
How can we use the training data to determine the best setting of the parameters
Θ?
Bayes rule provides an elegant and immedidate response to this question. In
Bayesian Learning, we would consider the posterior distribution,
p(Θ|V ) ∝ p(V |Θ)p(Θ).
What this says is that, in general, there is no such ‘optimal’ single setting of the
parameters, rather that there is a distribution of parameter values, where each
value of Θ has a weight according to the above distribution. That is, each value
for Θ is assessed by how likely it is to generate the data, but also is moderated by
any prior beliefs p(Θ). Note that, in a purely probabilistic sense, the specification
of prior beliefs p(Θ) is unavoidable, and there is no sense in ‘assumption free’
determination of the parameters.
However, dealing with such posterior distributions is often computationally too
ambitious, and the simpler Maximum A Posteriori (MAP) solution is preferred,
ΘMAP = arg max p(X|Θ)p(Θ)
Θ

in which a single ‘best’ estimate for Θ is chosen. If the user does not feel able
to specify any prior preference for Θ (a so-called “flat” prior p(Θ) = const), the
parameters are given by Maximum Likelihood
ΘML = arg max p(V |Θ)
Θ

which simply says that we set the parameter Θ to that value for which the observed
data was most likely to have been generated.

Belief Nets

Previously, we have emphasised two aspects of graphical models – the independence

assumptions, and the conditional probability tables. Here we will assume that we
are happy about the independence assumptions, but that we do not know the
values of the conditional probability tables. We address the issue of how to learn
values for the CPTs given a set of data. This brings the subject much closer to
the ethos of machine learning.
Consider the following model of the relationship between exposure to Asbestos
(A), being a smoker (S) and the incidence of lung cancer (C)
p(A, S, C) = p(C|A, S)p(A)p(S)

130
 131

A S

Figure 13.1: A simple model for the relationship between lung Cancer, Asbestos
exposure and Smoking.

which is depicted in fig(13.1). Here we have made the assumption that Cancer is
dependent on both exposure to Asbestos and being a Smoker, but that there is no
direct relationship between Smoking and exposure to Asbestos. This is the kind
of assumption that we may be able to elicit from experts such as doctors who have
good intuition/understanding of the relationship between variables.
Furthermore, we assume that we have a list of individuals characteristics in the
population, where each row represents a training example. This is perhaps taken
from hospital records or a general survey of the population.

A S C
1 1 1
1 0 0
0 1 1
0 1 0
1 1 1
0 0 0
1 0 1

Figure 13.2: A database containing information about the Asbestos exposure (1

signifies exposure), being a Smoker (1 signifies the individual is a smoker), and
lung Cancer (1 signifies the individual has lung Cancer). Each row contains the
information for an individual, so that there are 7 individuals in the database.

Intuitive Table Settings Looking at instances where A = 0, S = 0, we find always C = 0, and hence
p(C = 1|A = 0, S = 0) = 0. Similarly, we can count other cases to form a
CPT table. Counting the instances of A = 1, we find p(A = 1) = 4/7, and
similarly, p(S = 1) = 4/7. These three CPTs then complete the full distribution
specification.
A S p(C = 1|A, S)
0 0 0
0 1 0.5
1 0 0.5
1 1 1

Maximum Likelihood Learning

Actually, what we did intuitively, by counting the relative number of occurrences,

corresponds mathematically to maximum likelihood learning.
For a set of P observations (training data), X = x1 , . . . , xP , and independently
 132

gathered observations, the log likelihood of X is

P
X
log p(X) = log p(xµ )
µ=1

For DAGs, the factorisation enables us to separate the CPT terms:

P X
X n
log p(X) = log p (xµi |pa (xµi ))
µ=1 i=1

We want to learn the entries of the CPTs. For convenience, let pa (x1 ) = {x2 , x3 },
and say we want to find the CPT entry p(x1 = 1|x2 = 1, x3 = 0).
Counting the Occurrences Naively, the number of times p(x1 = 1|x2 = 1, x3 = 0) occurs in the log likelihood
is equal to c(1, 1, 0), the number of such occurrences in the training set. However,
since (by the normalisation constraint) p(x1 = 0|x2 = 1, x3 = 0) = 1 − p(x1 =
1|x2 = 1, x3 = 0), the total contribution of p(x1 = 1|x2 = 1, x3 = 0) to the log
likelihood is

c(1, 1, 0) log p(x1 = 1|x2 = 1, x3 = 0)+c(0, 1, 0) log (1 − p(x1 = 1|x2 = 1, x3 = 0))

Differentiating wrt p(x1 = 1|x2 = 1, x3 = 0) gives

c(x1 = 1, x2 = 1, x3 = 0)
p(x1 = 1|x2 = 1, x3 = 0) =
c(x1 = 1, x2 = 1, x3 = 0) + c(x1 = 0, x2 = 1, x3 = 0)

Corresponding to the previous intuitive counting procedure. Alernatively, we may

write
P
µ:xµ µ µ 1
1 =1,x2 =1,x3 =0
p(x1 = 1|x2 = 1, x3 = 0) = P P
µ:xµ =1,xµ =1,xµ =0 1 +
1 2 µ:xµ =0,xµ =1,xµ =0 1
3 1 2 3

From the above example, it is clear that we can set values for the all the table
entries. However, consider a smaller dataset:
A S C
1 1 1
1 0 0
0 1 1
0 1 0
According to the ML principal above, we will not be be able to determine entry
p(c|a = 0, s = 0), since there are no entries in the database which jointly contain
the setting a = 0 and s = 0. In this case, we either need additional information,
or assumptions about how to set the missing table entries. One approach that
may lead to a fuller specification is to require that not only all the jointly observed
training data should be maximally likely, but also that any marginal observations
should also be maximally likely – that is, we restrict attention to a subset of
the variabes, say here C alone, and require that the model is maximally likely
to generate the observed statistics for the C variable alone.P Since calculating the
marginal likelihood p(c) involves summing over all the states s,a p(c|s, a)p(s)p(a),
we obtain an objective function that contains at least the parameters p(c|s = 0, a =
0). How to choose such marginals, and how to weight this requirement with the
 133

x1 x2 xn

joint ML requirement is an open question, but could potentially be related to

standard generalisation issues. An interesting question is, even if we have enough
data to specify all the table entries by ML, whether or not the learned model is
consistent with all the marginals in the training data? It is left as an exercise for
the reader to show that, in general, it is not the case that all the marginals from
the learned model are consistent with the sample marginals. In general these can
only be made consistent if there are no restrictions on the form of the probability
distribution fitted – that is, no independence assumptions are made.

Conditional Probability Functions

Imagine that we have a node with n parents, in state x = (x1 , . . . , xn ). For binary
variables, there are therefore 2n entries in the CPT to specify for that node. This
rapidly becomes infeasible, and we need to use a functional specification of the
table. For example, a sigmoidal function

p(Y = 1|x) = 1/(1 + exp(w · x))

where we only need to specify the n-dimensional parameter w.

So, rather than use Maximum Likelihood to learn the entries of the CPTs directly,
we instead learn the value of the parameter w. Since the number of parameters
in w is small (n), then we have some hope that with a small number of training
examples (say of the order of n), we can learn a reliable value for w.

13.1.1 Missing Data/Variables

Previously, we assumed that all the variables are ‘visible’, or ‘evidential’. That is,
each training data point specified that values of all the variables in the distribution.
In many cases, we simply will not be able to directly measure certain variables,
although we know that they are important in the modelling process. Such variables
are ‘hidden’. In cases where we have a training set, and sometimes the variables
are specified, and in some cases not, then we have a ‘missing data’ problem. Both
of these cases can be dealt with by using Maximum Likelihood – however, we
calculate the likelihood for each training example on only those visible variables.
For example, consider two training examples, x1 and x2 , in which x1 is fully
observed, but x2 has an unobserved first component, i.e. x2 = (?, x22 , x23 , . . . x2n ).
The log likelihood of the data is then

log p(x11 , x12 , . . . , x1n ) + log p(x22 , x23 , . . . x2n )

Using the general BN structure, this would be

n
X X



Y


log p x1i |pa x1i + log p(x1 |pa x21 ) p x2i |pa x2i
i=1 x21 i>1
 134

The structure of the second term is a summation over the missing values for the
variable missing for that example. If there are many missing datapoints, calculat-
ing the summations may be difficult. However, one can see that the log-likelihood
remains a function of the tables, and one can optimise this as usual. However,
this direct approach is rarely taken in practice. An alternative, general and more
elegant approach to this problem, is given by the EM algorithm, as described
below.

13.2 Variational Learning and Expectation Maximisation

For simplicity here, we will deal only with the case that certain variables are con-
sistently unobserved in the training data set. The generalisation to the case that
for some examples a certain variable is observed and in others not, is straight-
forward. We use the notation v for those components of the variables x that are
‘visible’ (i.e for which we have observed values), and h for those components of
the variables x that are ‘hidden’ (i.e for which we do not have observed values).
In ML learning, our aim is to maximise the likelihood of the observed/visible
data, V = {v µ , µ = 1, . . . , P } with respect to any parameters Θ (CPTs or their
parameters) of the distribution.
Calculating p(V |Θ) Assuming that the data instances are gathered independently,
P
X
log p(V |Θ) = log p(v µ |Θ)
µ=1

In the above, we could use marginalisation to calculate the likelihood of the visible
data,
X
p(v µ |Θ) = p(v µ , hµ |Θ)
hµ

However, there are reasons why this may not be such a good idea. For example,
there are so many hidden units that we cannot carry out the summation (Junction
Tree Cliques are too large). Or the resulting log likelihood is difficult to optimise
using standard approaches – the objective function is extremely complicated.
There exists a useful general procedure for learning with hidden units. Special
cases of this approach include the Expectation Maximisation (EM) algorithm,
Generalised EM (GEM) algorithms and (probably all) the other EM variants. In
the machine learning literature, Neal and Hinton[21] made the connection between
the traditional EM algorithm[], and the more general variational treatment. See
[22] for a more standard exposition.
The variational EM has several potential positive aspects
• Can (often but not always) help deal with intractability
• Provides a rigorous lower bound on the likelihood
• May make larger parameter updates than gradient based approaches.
Before we can talk about the Variational Learning algorithm, which is a special
variational technique based on the Kullback-Leibler divergence between two dis-
tributions, we need a digression into Information Theory.
 135

x-1 > log(x) x-1

log(x)
(1,0)
p(x)
q(x)

(a) (b)

Figure 13.3: (a) The probability density functions for two different distributions
p(x) and q(x). We would like to numerically characterise the difference between
these distributions. (b) A simple linear bound on the logarithm enables us to
define a useful distance measure between distributions (see text).

The Kullback Leibler Divergence KL(q, p)

The KL diveregence KL(q, p) measures the “difference” between distributions q

and p, fig(D.1a). In many ways, this is a natural measure to use, and is well
motivated from Information theoretic arguments.
For two distributions q(x) and p(x), it is defined as

KL(q, p) ≡ hlog q(x) − log p(x)iq(x)

where the notation hf (x)ir(x) denotes average of the function f (x) with respect
to the distribution r(x). For a continuous variable, this would be hf (x)ir(x) =
R P
f (x)r(x)dx, and for a discrete variable, we would have hf (x)ir(x) = x f (x)r(x).
The advantage of this notation is that much of the following holds independent of
whether the variables are discrete or continuous.
KL(q, p) ≥ 0 The KL divergence is always ≥ 0. To see this, consider the following simple linear
bound on the function log(x) (see fig(D.1b)):

log(x) ≤ x − 1

Replacing x by p(x)/q(x) in the above bound

p(x) p(x)
− 1 ≥ log ⇒ p(x) − q(x) ≥ q(x) log p(x) − q(x) log q(x)
q(x) q(x)
R
Now
R integrate (or sum in the case of discrete variables) both sides. Using p(x)dx =
1, q(x)dx = 1, and rearranging gives
Z
{q(x) log q(x) − q(x) log p(x)} dx ≡ hlog q(x) − log p(x)iq(x) ≥ 0

Furthermore, one can show that the KL divergence is zero if and only if the two
distributions are exactly the same.

Bounding the Likelihood

The KL divergence enables us immediately to find a lower bound on the marginal

likelihood for a single training example, p(v). Consider

KL (q(h|v), p(h|v)) ≡ hlog q(h|v) − log p(h|v)iq(h|v) ≥ 0

 136

L(θ )

θ1 θ2 θ3

Figure 13.4: The idea in variational learning is to bound a possibly complex like-
lihood L(Θ) by a simpler function, for which the maximum is easy to find, here
say finding Θ1 . Subsequently, a new lower bound on the likelihood is fitted, using
the previous best value for the optimal Θ, and a new optimal bound value Θ2 is
found. This is repeated until convergence, each time pushing up the lower bound,
and therefore hopefully improving our estimate of where the maximum of the like-
lihood is. This iterative procedure can often lead to the rapid finding of (locally)
optimal values for the parameters.

Hence, using Bayes rule,

hlog q(h|v)iq(h|v) − hlog p(h, v)iq(h|v) + log p(v) ≥ 0

Rearranging, we therefore have the bound1

log p(v) ≥ − hlog q(h|v)iq(h|v) + hlog p(h, v)iq(h|v)

| {z } | {z }
Entropy Energy

Summing over the training data, we get the bound on the marginal likelihood
P
X P
X
log p(V |Θ) ≥ − hlog q µ (h|v)iqµ (h|v) + hlog p(hµ , v µ |Θ)iqµ (h|v)
µ=1 µ=1
| {z } | {z }
Entropy Energy

This bound is exact (that is, it is equal to the log marginal likelihood) when we
set q µ (h|v) = p(hµ |v µ ). Recalling that our aim is to find an algorithm that will
adjust any parameters of p to maximize the likelihood, a reasonable thing to do
would be a relaxed version, namely to maximize a lower bound on the likelihood.
That is, to iteratively adjust the parameters Θ to push up the lower bound on
the (marginal) likelihood, and in so doing hopefully push up the true (marginal)
likelihood.

Variational Learning

Since the parameter Θ only occurs in the Energy term, this suggests that we can
iteratively firstly set the optimal parameters Θ by optimising the Energy term
1 This is analogous to the Mean Field bound on the partition function in statistical physics, and
motivates the terminology ‘energy’ and ‘entropy’.
 137

(for fixed q µ (h|v)). And then, we can optimise (push up the lower bound) by
finding a better set of fixed q µ (h|v), by optimising with respect to the variational
distributions q µ (h|v):
1. Expectation (E) step : Choose a set of distributions

q µ (h|v), µ = 1 . . . P

from a chosen class of distributions, for which each q µ (h|v) minimises the
KL divergence KL(q µ (h|v), p(hµ |v µ )).
2. Maximisation (M) step : Set
P
X
Θ ← arg max hlog p(hµ , v µ |Θ)iqµ (h|v)
Θ
µ=1

Iterate (1,2) until parameter convergence. Steps (1) and (2) are guaranteed to
increase the lower bound on the likelihood.

The EM algorithm cannot decrease the likelihood

Whilst, by definition, the EM algorithm cannot decrease the bound on the likeli-
hood, an important question is whether or not the iterations cannot decrease the
likelihood itself.
Another way to rephrase our bound on the likelihood log p(v|θ ′ ) ≥ LB(θ′ |θ) is as

log p(v|θ ′ ) = LB(θ′ |θ) + KL(p(h|v, θ)|p(h|v, θ ′ ))

That is, the KL divergence is simply the difference between the lower bound and
the true likelihood. Similarly, we may write

log p(v|θ) = LB(θ|θ) + KL(p(h|v, θ)|p(h|v, θ))

| {z }
zero

Hence

log p(v|θ ′ ) − log p(v|θ) = LB(θ ′ |θ) − LB(θ|θ) + KL(p(h|v, θ)|p(h|v, θ′ ))

| {z } | {z }
≥0 ≥0

The first assertion is true since, by definition, we search for a θ′ which has a higher
value for the bound than our starting value θ. The second assertion is trivially true
by the property of the KL divergence. Hence we reach the important conclusion
that the EM (or GEM/variational implementation), not only essentially increases
the lower bound on the likelihood, but also increases the likelihood itself (or, at
least, the EM cannot decrease these quantities).
EM Algorithm Clearly, if we do not restrict the class of distributions that the q can take, the
optimal choice is

q µ (h|v) = p(hµ |v µ )

Using these q ′ s, corresponds to the standard “EM” algorithm.

Intractable Energy The algorithm assumes that we can calculate hlog p(hµ , v µ |Θ)iqµ (h|v) . However, in
 138

θ
B(θ ,q)

Figure 13.5: The EM approach is an axis aligned way to find a maximum of the
lower bound B(θ, q). This proceeds by, for fixed q, finding the best parameters θ
(the M -step), and then for fixed θ, finding the best distributions q (the E-step).
Of course, any other optimisation procedure is valid, and indeed may result in
faster convergence than this simple axis aligned approach. However, an advantage
of the EM style is that is leads to a simple-to-implement-and-interpret algorithm.

general, it may be that we can only carry out the average over q for a very
Q restricted
class of distributions. For example, factorised distributions q µ (h|v) = j q µ (hj |v).
Hence, in practice, we Q often choose a simpler class of distributions, Q, e.g Q =
factorised q µ (h|v) = i q µ (hi |v), which may make the averaging required for the
energy simpler.
Determining the best Imagine we parameterise our distribution class Q using a parameter θQ . We can
distribution in the class find the best distribution in class Q by minimising the KL divergence between
q µ (h|v, θQ ) and p(hµ |v µ , Θ) numerically using a non-linear optimisation routine.
Alternatively, one can assume a certain structured form for the q distribution, and
learn the optimal factors of the distribution by free form functional calculus.
Using a class of simpler q distributions like this corresponds to a Generalised EM
algorithm (GEM).

Application to Belief Networks

The previous variational learning theory is very general. To make things more
concrete, we apply the previous theory to learning the CPTs in a BN in which
certain variables are hidden. We first apply it to a very simple network.

p(a, c, s) = p(c|a, s)p(a)p(s)

Imagine, as in table fig(13.1), we have a set of data, but that we do not know the
states of variable a. That is,
S C
1 1
0 0
1 1
1 0
1 1
0 0
0 1
 139

Firstly, let’s assume that we have chosen some values for the distributions q µ (a|c, s),
e.g. q 1 (a = 1|c = 1, s = 1) = 0.6, q 2 (a = 1|c = 0, s = 0) = 0.3, q 3 (a = 1|c3 = 1, s =
1) = 0.7, q 4 (a = 1|c = 0, s = 1) = 0.1 . . .. Now we write down the Energy term:
7
X
E= hlog p(cµ |aµ , sµ ) + log p(aµ ) + log p(sµ )iqµ (a|c,s)
µ=1

7 n
X o
E= hlog p(cµ |aµ , sµ )iqµ (a|c,s) + hlog p(aµ )iqµ (a|c,s) + log p(sµ )
µ=1

Remember that our goal is to learn the CPTs p(c|a, s) and p(a) and p(s).
Pleasingly, the final term is simply the log likelihood of the variable s, and p(s)
appears explicitly only in this term. Hence, the usual maximum likelihood rule
applies, and p(s = 1) is simply given by the relative number of times that s = 1
occurs in the database (hence p(s = 1) = 4/7, p(s = 0) = 3/7).
The parameter p(a = 1) occurs in the terms
X
{q µ (a = 0|c, s) log p(a = 0) + q µ (a = 1|c, s) log p(a = 1)}
µ

which, using the normalisation constraint is

X X
log p(a = 0) q µ (a = 0|c, s) + log(1 − p(a = 0)) q µ (a = 1|c, s)
µ µ

Differentiating with respect to p(a = 0) we get

P µ
µ q (a = 0|c, s)
p(a = 0) = P µ
P µ
µ=1 q (a = 0|c, s) + µ q (a = 1|c, s)

That is, whereas in the standard ML estimate, we would have the real counts of
the data in the above formula, here they have been replaced with our guessed
values q µ (a = 0|c, s) and q µ (a = 1|c, s).
A similar story holds for the more complex case of say p(c = 1|a = 0, s = 1). The
contribution of this term to the Energy is
X
q µ (a = 0|c = 1, s = 1) log p(c = 1|a = 0, s = 1)
µ:cµ =1,sµ =1
X
+ q µ (a = 0|c = 0, s = 1)) log(1 − p(c = 1|a = 0, s = 1))
µ:cµ =0,sµ =1

which is
X
log p(c = 1|a = 0, s = 1) q µ (a = 0|c = 1, s = 1)
µ:cµ =1,sµ =1
X
+ log(1 − p(c = 1|a = 0, s = 1)) q µ (a = 0|c = 1, s = 1)
µ:cµ =0,sµ =1

Optimising with respect to p(c = 1|a = 0, s = 1) gives

p(c = 1|a = 0, s = 1) =
P
q µ (a = 0|c = 1, s = 1)
µ:cµ =1,sµ =1
P µ
P µ
µ:cµ =1,sµ =1 q (a = 0|c = 1, s = 1) + µ:cµ =0,sµ =1 q (a = 0|c = 0, s = 1)
 140

Again, this has an intuitive relationship to ML for the complete data case, in which
the missing data has been filled in by the assumed distributions q.
What about the parameters q µ (a|c, s)? If we use the standard EM algorithm, we
should set these to
q µ (a|c, s) = p(aµ |cµ , sµ ) ∝ p(aµ , cµ , sµ )
q µ (a|c, s) ∝ p(cµ |aµ , sµ )p(aµ )p(sµ )
where the current set of values for the p’s have been assumed from the previous
calculation. These two stages are then iterated : in the next step, we use these
new values of q µ (a|c, s) to calculate the next p′ s etc. These equations will converge
to a local optimum of the bound.

More general Belief Networks

The form of the energy term for belief networks is

X XX
hlog p(hµ , v µ )iqµ (h|v) = hlog p(xµi |pa (xµi ))iqµ (h|v)
µ µ i

where each xi is either clamped into a visible state, or is a hidden unit. Note that
p(xµi |pa (xµi )) is only a function of the variables xi ∪ pa (xi ), the family of node xi
and that, in general, some of these may be hidden. The hidden nodes of this family
are giµ ≡ xi ∪ pa (xi ) \v µ . Since the term p(xµi |pa (xµi )) therefore only depends on
giµ , we only require to average with respect to q µ (giµ |v). If we use the optimal
choice (EM setting), q µ (giµ |v) = p(giµ |v µ ), it is clear that this marginal is easy to
calculate for any (poly)tree, since the marginal can be calculated by the JTA, and
that therefore this term can be computed efficiently.
To be more specific, consider a simple five variable distribution with discrete
variabl-es,
p(x1 , x2 , x3 , x4 , x5 ) = p(x1 |x2 )p(x2 |x3 )p(x3 |x4 )p(x4 |x5 )p(x5 ),
in which the variables x2 and x4 are consistently hidden in the training data, and
training data for x1 , x3 , x5 are always present. In this case, the contributions to
the energy have the form
X
hlog p(xµ1 |x2 )p(x2 |xµ3 )p(xµ3 |x4 )p(x4 |xµ5 )p(xµ5 iqµ (x2 ,x4 |x1 ,x3 ,x5 )
µ

Which may be written as

X
hlog p(xµ1 |x2 )iqµ (x2 ,x4 |x1 ,x3 ,x5 )
µ
X
+ hlog p(x2 |xµ3 )iqµ (x2 ,x4 |x1 ,x3 ,x5 )
µ
X
+ hlog p(xµ3 |x4 )iqµ (x2 ,x4 |x1 ,x3 ,x5 )
µ
X
+ hlog p(x4 |xµ5 )iqµ (x2 ,x4 |x1 ,x3 ,x5 )
µ
X
+ log p(xµ5 ) (13.2.1)
µ
 141

A useful property can now be exploited, namely that each term depends on only
those hidden variables in the family that that term represents. Thus we may write
X
hlog p(xµ1 |x2 )iqµ (x2 |x1 ,x3 ,x5 )
µ
X
+ hlog p(x2 |xµ3 )iqµ (x2 |x1 ,x3 ,x5 )
µ
X
+ hlog p(xµ3 |x4 )iqµ (x4 |x1 ,x3 ,x5 )
µ
X
+ hlog p(x4 |xµ5 )iqµ (x4 |x1 ,x3 ,x5 )
µ
X
+ log p(xµ5 ) (13.2.2)
µ

It is clear that the final term causes us no difficulties, and this table can be set
using the standard ML framework. Let us consider therefore a more difficult table,
namely p(x1 |x2 ). When will the table entry p(x1 = i|x2 = j) occur in the energy?
This happens whenever xµ1 is in state i. Since there is a summation over all the
states of variables x2 (due to the average), there is also a single time when variable
x2 is in state j. Hence the contribution to the energy from terms of the form
p(x1 = i|x2 = j) is
X
I[xµ1 = i]q µ (x2 = j|x1 , x3 , x5 ) log p(x1 = i|x2 = j)
µ

where the indicator function I[xµ1 = i] equals 1 if xµ1 is in state i and is zero
otherwise. To ensure normalisation of the table, we add a Lagrange term:
( )
X µ
X
µ
I[x1 = i]q (x2 = j|x1 , x3 , x5 ) log p(x1 = i|x2 = j)+λ 1 − p(x1 = k|x2 = j)
µ k

Differentiating with respect to p(x1 = i|x2 = j) we get

X q µ (x2 = j|x1 , x3 , x5 )
I[xµ1 = i] =λ
µ
p(x1 = i|x2 = j)
or
X
p(x1 = i|x2 = j) ∝ I[xµ1 = i]q µ (x2 = j|x1 , x3 , x5 ).
µ

Hence
P
µ I[xµ1 = i]q µ (x2 = j|x1 , x3 , x5 )
p(x1 = i|x2 = j) = P
µ,k I[xµ1 = k]q µ (x2 = j|x1 , x3 , x5 )

Using the EM algorithm, we would use q µ (x2 = j|x1 , x3 , x5 ) = p(x2 = j|xµ1 , xµ3 , xµ5 ).
Note that this optimal distribution is easy to find for any polytree since this just
corresponds to the marginal on the family, given some nodes in the graph are
clamped in their evidential state. Hence, for EM, an update for the table would
be
P µ
new µ I[x1 = i]p
old
(x2 = j|xµ1 , xµ3 , xµ5 )
p (x1 = i|x2 = j) = P µ old (x = j|xµ , xµ , xµ )
(13.2.3)
µ,k I[x1 = k]p 2 1 3 5
 142

Similar expressions can be derived for the other tables. The important thing to
note is that we only ever need local marginals for the variables in a family. These
are always easy to obtain in polytrees (assuming that the number of states in a
family is not too large), since this corresponds to inference in a tree conditioned
on some evidence. Hence all updates in the EM algorithm are computable.
What about the table p(x2 = i|x3 = j)?
To ensure normalisation of the table, we add a Lagrange term:
( )
X µ
X
µ
I[x3 = j]q (x2 = i|x1 , x3 , x5 ) log p(x2 = i|x3 = j)+λ 1 − p(x2 = k|x3 = j)
µ k

As before, differentiating, and using the EM settings, we have

P µ
new µ I[x3 = j]p
old
(x2 = i|xµ1 , xµ3 , xµ5 )
p (x2 = i|x3 = j) = P µ old (x = k|xµ , xµ , xµ )
(13.2.4)
µ,k I[x3 = j]p 2 1 3 5

There is a simple intuitive pattern to equation (13.2.3) and equation (13.2.4) :

If there were no hidden data, equation (13.2.3) would read
X
pnew (x1 = i|x2 = j) ∝ I[xµ1 = i]I[xµ2 = j]
µ

and equation (13.2.4) would be

X
pnew (x2 = i|x3 = j) ∝ I[xµ3 = j]I[xµ2 = i]
µ

All that we do, therefore, in the general EM case, is to replace those determin-
istic functions such as I[xµ2 = i] by their missing variable equivalents pold (x2 =
i|xµ1 , xµ3 , xµ5 ).

13.3 Optimising the Likelihood by Gradient methods

For latent variable models, sometimes the EM algorithm is very slow to converge.
According to [23], when the missing information is small compared to the complete
information, EM exhibits approximate Newton behavior and enjoys fast, typically
superlinear convergence in the neighborhood of the optimum point. If the fraction
of missing information approaches unity, EM will exhibit extremely slow conver-
gence.
An alternative is to compute the gradient of the likelihood directly.

L(θ) = log p(v|θ)

Then
1
∂θ L(θ) = ∂θ p(v|θ)
p(v|θ)
Z
1
∂θ L(θ) = ∂θ p(v, h|θ)
p(v|θ) h
 143

At this point, it may seem that computing the derivative is difficult. However, we
may observe
R
p(v, h|θ)
∂θ L(θ) = h ∂θ log p(v, h|θ)
p(v|θ)
Z
∂θ L(θ) = p(h|v, θ)∂θ log p(v, h|θ)
h
The rhs is just the average of the derivative of the complete likelihood. This is
closely related to the EM algorithm, though note that the average is performed
with respect the current distribution parameters θ and not θold as in the EM case.
Used in this way, computing the derivatives of latent variable models is relatively
straightforward. These derivatives may then be used as part of a standard opti-
misation routine such as conjugate gradients[23].

13.4 Iterated Proportional Fitting

An interesting questions is how to fit undirected models efficiently using Maximum
Likelihood. Conditional Random Fields are a good example. (Wiegerinck-Heskes
paper[24]. Also the work by Tony Jebara[25] on the reversed Jensen for Exponen-
tial Family models.)
The basic idea is as follows:
Consider an undirected distribution
1 Y
p(v) = φc (v)
Z c
P Q
where Z = v c φc (v) ensures normalisation.
Given a set of data, v µ , µ = 1, . . . , P , how do we learn the ML parameters of the
φc ? Assuming iid data, the log likelihood is
XX
L= log φc (v µ ) − P log Z
µ c

What makes this problem awkward is that the parameters also occur in Z, and
hence the objective function does not split into a set of isolated parameter terms.

An upper bound on Z

Consider the bound, for positive x.

log x ≤ x − 1 ⇒ − log x ≥ 1 − x

Hence
Z Z Z
− log ′
≥ 1 − ′ ⇒ − log Z ≥ − log Z ′ + 1 − ′
Z Z Z
Let’s call the parameters θ. Then we can write the bound (for a single datapoint,
P = 1) as
Z(θ)
L(θ) ≥ E(θ) − log Z(θold ) + 1 −
Z(θold )
| {z }
LB(θ,θ old )
 144

θ′

θ θ

v v
(a) (b)

Figure 13.6: (a) Standard ML learning. (b) ML-II learning.

P µ
where E(θ) represents c log φc (v ) Hence
old old
L(θ) − L(θ ) ≥ LB(θ, θ ) − L(θold )
Using, the property that L(θold ) = LB(θold , θold )), we have
L(θ) − L(θold ) ≥ LB(θ, θold ) − LB(θold , θold )
Hence, provided we can find a θ that increases the lower bound on the likelihood,
we are guaranteed to increase the likelihood itself. This is similar to the guarantees
provided by the EM algorithm. The generalisation to multiple datapoints P > 1
just follows by summing the above over the datapoints.
The IPF procedure then follows iteratively maximising wrt θ. The potential ad-
vantage of this method over gradient based procedures is apparent if the optimum
of LB(θ, θold with respect to θ can be achieved in closed form. Otherwise, there
may be little advantage[26].

13.5 Bayesian Methods and ML-II

Consider a parameterised distribution p(v|θ), and that we wish to set the para-
meters θ given some data. The model p(v|θ) is depicted in fig(13.6a), where the
diamond indicates that no distribution is specified for that variable. For a sin-
gle observed datapoint v, then setting θ by ML would correspond to finding the
parameter θ that maximises p(v|θ).
In some cases, though, we may have an idea about which parameters θ are more
appropriate. We can express this prior preference using a distribution p(θ). If
the prior were fully specified, then there is nothing to ‘learn’ since p(θ|v) is now
fully known. However, in many cases in practice, we are unsure of the exact
parameter settings of the prior, and hence specify a parametersised prior using
hyperparameters θ′ , using a distribution p(θ|θ′ ). This is depicted in fig(13.6b).
The learning R corresponds to finding the optimal θ′ that maximises the likelihood
′ ′
p(v|θ ) = θ p(v|θ)p(θ|θ ). This is known as an ML-II procedure since it corre-
sponds to maximum likelihood, but at the higher, hyperparameter level. See [27]
and [28]. This is a form of approximate Bayesian analysis since, although θ′ is set
using maximum likelihood, after training, we have a distribution over parameters,
p(θ|v, θ′ ).

13.6 Problems
Exercise 27 (Printer Nightmare) Cheapco is, quite honestly, a pain in the
neck. Not only did they buy a dodgy old laser printer from StopPress and use
 145

it mercilessly, but try to get away with using substandard components and mater-
ial. Unfortunately for StopPress, they have a contract to maintain Cheapco’s old
warhorse, and end up frequently sending the mechanic out to repair the printer.
After the 10th visit, they decide to make a model of Cheapco’s printer, so that
they will have a reasonable idea of the fault based only on the information that
Cheapco’s secretary tells them on the phone. In that way, StopPress hopes to be
able to send out to Cheapco only a junior repair mechanic.
Based on the manufacturer’s information, StopPress has a good idea of the depen-
dencies in the printer, and what is likely to directly affect other printer components.
However, the way that Cheapco abuse their printer is a mystery, so that the ex-
act probabilistic relationships between the faults and problems is idiosyncratic to
Cheapco. However, StopPress has the following table of faults for each of the 10
visits. (Each column represents a visit, the transpose of the normal format).
fuse assembly malfunction 0 0 0 1 0 0 0 0 0 0
drum unit 0 0 0 0 1 0 0 1 0 0
toner out 1 1 0 0 0 1 0 1 0 0
poor paper quality 1 0 1 0 1 0 1 0 1 1
worn roller 0 0 0 0 0 0 1 0 0 0
burning smell 0 0 0 1 0 0 0 0 0 0
poor print quality 1 1 1 0 1 1 0 1 0 0
wrinkled pages 0 0 1 0 0 0 0 0 1 0
multiple pages fed 0 0 1 0 0 0 1 0 1 0
paper jam 0 0 1 1 0 0 1 1 1 1

Based on their knowledge of the printer, they build a graphical model.

1. Load printer.bbnet
Using the above training data, complete the Conditional Probability Tables
for the model. The CPT for “poor print quality” has already been completed.
All other tables have been set initially to uniform values, and need to be
altered to match the above table of training data.
You will find that at times, there is insufficient data to complete a CPT –
use your imagination to create a reasonable value for the missing CPT entry.
2. Experiment with this network by clamping the bottom “problem” nodes.
3. How could the junior engineer use the network to help detect faults?

Exercise 28 Consider the graphical model X1 → X2 . Let θk|j = P (X2 = sk |X1 =

sj ) denote the probability that X2 is in state k given that X1 is in state j. Clearly
P
k θk|j = 1 for all j. We are given n observations of the states of X1 and X2 .
Let the number of transitions from X1 = sj to X2 = sk be denoted nkj . Write
down the likelihood
P of the data, and show that the maximum likelihood estimator
θ̂k|j = nkj / l nlj .
Consider the case where X0 is also a parent of X2 (and not of X1 ). Write down the
ML estimator for θk|ij = P (X2 = sk |X0 = si , X1 = sj ) given count data {nkij }.

Exercise 29 You have a machine that measures property x, the “orangeness” of

liquids. You wish to discriminate between C1 = “IrnBru” and C2 = “Orangina”.
It is known that

10 1.0 ≤ x ≤ 1.1
p(x|C1 ) =
0 otherwise
 146


200(x − 1) 1.0 ≤ x ≤ 1.1
p(x|C2 ) =
0 otherwise

The prior probabilities P (C1 ) = 0.6 and P (C2 ) = 0.4 are also known from experi-
ence. Calculate the optimal Bayes’ classifier and P (error).

Exercise 30 If n observations y0 , . . . , yn−1 were noisy iid measurements of an

underlying variable x, then the graphical model (for n = 3) would be
x

y0 y1 y2

The notation √ P (x) ∼ N (µ, σ 2 ) is shorthand for the Gaussian distribution P (x) =
2 2
e−(x−µ) /2σ / 2πσ 2 . Assume that P (x) ∼ N (0, σ02 ) and P (yi |x) ∼ N (x, σ 2 ) for
i = 0, . . . , n − 1. Show that P (x|y0 , . . . , yn−1 ) is Gaussian with mean

nσ02
µ= y
nσ02 + σ 2

where y = (y0 + y1 + . . . + yn−1 )/n and variance σn2 such that

1 n 1
= 2 + 2.
σn2 σ σ0

Exercise 31 (Bayesian analysis) . Consider the beta distribution p(θ) = c(α, β)θα−1 (1−
θ)β−1 , where c(α, β) is a normalizing constant. The mean of this distribution is
E[θ] = α/(α + β). For α, β > 1 the distribution is unimodal (i.e. it has only one
maximum). Find the value θ∗ where this maximum is attained, and compare it to
the mean. For what values of α and β to the mean and θ∗ coincide?

Exercise 32 Consider the multivariate Gaussian distribution p(x) ∼ N (µ, Σ) on

the vector x with components x1 , . . . , xn :
1
e− 2 (x−µ) Σ (x−µ)
1 T −1
p(x) = p
|2πΣ|

Calculate p(xi |x1 , . . . , xi−1 , xi+1 , . . . , xn ).

Exercise 33 Suppose that instead of using the Bayes’ decision rule to choose class
k if P (Ck |x) > P (Cj |x) for all j 6= k, we use a randomized decision rule, choosing
class j with probability Q(Cj |x). Calculate the error for this decision rule, and
show that the error is minimized by using Bayes’ decision rule.

Exercise 34 (Bayesian analysis) . Consider the problem of the Bayesian analy-

sis of θ, the probability that a certain coin will come up heads. You specify the prior
distribution as Beta(αh , αt ), and then repeatedly toss the coin. After each toss you
update the posterior distribution for θ. Write a short matlab program that does the
following:
• Takes the true value of θ
 147

• Takes values for αh and αt .

• Generates a sequence of (pseudo) random draws from the coin [Hint: use
rand and test to which side of the true θ this value falls. rand generates
pseudo random numbers in (0, 1).]
• Plots the posterior distribution for θ sequentially, i.e. as each new draw comes
in. [Hint: for drawing the distribution you do not need to calculate the Γ
function normalization terms.]
As the number of observations becomes large, explain the behaviour you observe.

Exercise 35 The Gaussian distribution in one dimension is defined as

1 1 2
p(x) = √ e− 2σ2 (x−µ)
2πσ 2
R∞
and satisfies −∞
p(x)dx = 1.

R∞
Show that −∞
xp(x)dx = µ.

R∞
Show that −∞ (x − µ)2 p(x)dx = σ 2 .

Exercise 36 Consider data xiP , i = 1, . . . , P . Show that the Maximum Likeli-

hood estimator of µ is µ̂ = P1 i=1 xi and that the ML estimate of σ 2 is σˆ2 =
P

1
P P i 2
P i=1 (x − µ)

Exercise 37 A training set consists of one dimensional examples from two classes.
The training examples from class 1 are

0.5, 0.1, 0.2, 0.4, 0.3, 0.2, 0.2, 0.1, 0.35, 0.25

and from class 2 are

0.9, 0.8, 0.75, 1.0

Fit a (one dimensional) Gaussian using Maximum Likelihood to each of these two
classes. Also estimate the class probabilities p1 and p2 using Maximum Likelihood.
What is the probability that the test point x = 0.6 belongs to class 1?

Exercise 38 Given the distributions p(x|class1) = N (µ1 , σ12 ) and p(x|class2) =

N (µ2 , σ22 ), with corresponding prior occurrence of classes p1 and p2 (p1 + p2 = 1),
calculate the decision boundary explicitly as a function of µ1 , µ2 , σ12 , σ22 , p1 , p2 . How
many solutions are there to the decision boundary, and are they all reasonable?

13.7 Solutions
III. Probabilistic models in Machine Learning

148
 14 Introduction to Bayesian Methods

Introduction
Regarding the general problem of fitting models to data, we are rarely certain
about either our data measurements (they may be inherently ‘noisy’) or model
beliefs. It is natural to use probabilities to account for these uncertainties. How
can we combine our data observations with these modelling uncertainties in a
consistent and meaningful manner? The Bayesian approach provides a consistent
framework for formulating a response to these difficulties, and is noteworthy for
its conceptual elegance[29, 27, 30, 31]. Indeed, throughout these chapters, we have
been using the Bayesian framework, since this is simply inherent in the correct
use of probabilities in graphical models and, in particular, inference in graphical
models. However, here we investigate a little more Bayes’ rule and it’s applications
in some very simple graphical models.
As a reminder to the reader, the fundamental probabilistic relationship required
for inference is the celebrated Bayes’ rule which, for general events A,B,C is
p(B|A, C)p(A|C)
p(A|B, C) = (14.0.1)
p(B|C)
In modelling data, it is convenient to think of different levels of uncertainty in
formulating a model. At the lowest level, we may assume that we have the correct
model, but are uncertain as to the parameter settings θ for this model. This
assumption details how observed data is generated, p (data|θ, model). The task
of inference at this level is to calculate the posterior distribution of the model
parameter. Using Bayes’ rule, this is
p (data|θ, model) p (θ|model)
p(θ|data, model) = (14.0.2)
p (data|model)
Thus, if we wish to infer model parameters from data we need two assumptions:
(1) How the observed data is generated under the assumed model, the likelihood
p (data|θ, model) and (2) Beliefs about which parameter values are appropriate,
before the data has been observed, the prior p(θ|model). (The denominator in
equation (14.0.2) is the normalising constant for the posterior and plays a role in
uncertainty at the higher, model level). That these two assumptions are required
is an inescapable consequence of Bayes’ rule, and forces the Bayesian to lay bare
all necessary assumptions underlying the model.

Coin Tossing Example

Let θ be the probability that a coin will land up heads. An experiment yields the
data, D = {h, h, t, h, t, h, . . .}, which contains H heads and T tails in H + T flips
of the coin. What can we infer about θ from this data? Assuming that each coin
is flipped independently, the likelihood of the observed data is
T
p (D|θ, model) = θH (1 − θ) (14.0.3)

149
 150

(a)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

(b)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

(c)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

prob of head

Figure 14.1: Coin Tossing: (a) The prior: this indicates our belief that the coin
is heavily biased. (b) The likelihood after 13 Tails and 12 Heads are recorded,
θML = 0.48. (c) The posterior: the data has moderated the strong prior beliefs
resulting in a posterior less certain that the coin is biased. θMAP = 0.25, θ̄ = 0.39

A standard approach in the statistical sciences is to estimate θ by maximising

the likelihood, θML = arg maxθ p(D|θ, model). Strictly speaking, this approach
is non-Bayesian since it does not require the specification of a prior and, con-
sequently, theories which deal with uncertainty in ML estimators are primarily
concerned with the data likelihood, and not directly posterior parameter uncer-
tainty. In the Bayesian approach, however, we need to be explicit about our prior
beliefs p(θ|model). These are updated by the observed data to yield the posterior
distribution
T
p (θ|D, model) ∝ θH (1 − θ) p (θ|model) (14.0.4)

The Bayesian approach is more flexible than maximum likelihood since it allows
(indeed, instructs) the user to calculate the effect that the data has in modifying
prior assumptions about which parameter values are appropriate. For example,
if we believe that the coin is heavily biased, we may express this using the prior
distribution in fig(14.1a). The likelihood as a function of θ is plotted in fig(14.1b)
for data containing 13 Tails and 12 Heads. The resulting posterior fig(14.1c) is
bi-modal, but less extreme than the prior. It is often convenient to summarise
Rthe posterior by either the maximum a posteriori (MAP) value, or the mean, θ̄ =
θp(θ|D)dθ. Such a summary is not strictly required by the Bayesian framework,
and the best choice of how to summarise the posterior depends on other loss
criteria[27].

Model Comparison and Hierarchical Models

The above showed how we can use the Bayesian framework to assess which para-
meters of a model are a posteriori appropriate, given the data at hand. We can
carry out a similar procedure at a higher, model level to asses which models are
more appropriate fits to the data. In general, the model posterior is given by

p(M |D) = p(D|M ) p(M ) /p(D) (14.0.5)

| {z } | {z }
Model Likelihood Model Prior
 151

If the model is parameterised by some unknown variable θ, we need to integrate

this out to calculate the model likelihood,
Z
p(D|M ) = p(D|θ, M )p(θ|M )dθ (14.0.6)

Comparing two competing model hypotheses M1 and M2 is straightforward

p(M1 |D) p(D|M1 ) p(M1 )

= (14.0.7)
p(M2 |D) p(D|M2 ) p(M2 )
| {z }
Bayes Factor

In the coin example, we can use this to compare the biased coin hypothesis (model
M1 with prior given in fig(14.1a)) with a less unbiased hypothesis formed by using
a Gaussian prior p(θ|M2 ) with mean 0.5 and variance 0.12 (model M2 ). This gives
a Bayes factor p(D|M1 )/p(D|M2 ) ≈ 0.00018. If we have no prior preference for
either model M1 or M2 , the data more strongly favours model M2 , as intuition
would suggest. If we desired, we could continue in this way, forming a hierarchy
of models, each less constrained than the submodels it contains.

Bayes automatically penalises overcomplex models

For simplicity, consider first two models M 1 and M 2 whose parameter spaces
are of the sameR dimension, and for which the prior is flat. The model likelihood
p(D|M 1) = θ p(D|θ, M 1)p(θ|M 1) is therefore essentially dominated by only the
high likelihood region. If model M 2 has roughly the same likelihood values where
it fits well, but has a higher volume of them, then the likelihood for model M 2 is
higher than for model M 1, see fig(14.2). This also explains why a model which
has a higher dimension parameter space will usually be rejected in favour of a
model which fits equally well with a smaller parameter space, since the prior in
the latter case is a unit mass spread out in a smaller number of dimensions, which
will therefore have a higher weight.

Non Bayesian and Minimal Empirical Risk criteria

The philosophy in the Bayesian approach is that parameter (or model) uncertainty
is reduced in the presence of observed data. However, except in pathological cases
(such as an infinite amount of training data) there still remains uncertainty in the
parameters. Whilst the Bayesian principal is well-established, not everyone is con-
vinced. A popular, non-bayesian method for model or parameter determiniation
runs somewhat as follows. We split the available training data into two sets, a
training set and a validation set. Model M 1 and model M 2 are both trained on
the training data, giving a single ‘optimal’ parameter for each model. Then each
model with its optimal parameter is then tested on the validation set. That model
which has the better performance on the validation set is then preferred. In this
sense, the uncertainty is not in the parameter space (since only a single optimal
parameter is retained). Rather, the uncertainty is in the predictive performance
of each model.
Need to talk about classical hypothesis testing....... The predictive performance
is (often, but not necessarily) assumed to be Gaussian. Then we would perhaps
 152

M1 θ M2 φ
High Likelihood
Region

Figure 14.2: The points represent data for which we wish to find a function that
goes through the datapoints well. We consider two model classes M 1 and M 2 –
each with their own parameter spaces. For example, M 1 might represent poly-
nomicals of order 20, and M 2 polynomials of order 10. In M 1, there is only a
small region of parameter space for which the function fits well, for example, the
solid red curve. If we move slightly away from this region, and use the red-dashed
function, due to the complex nature of the model, by ‘definition’ this means that
it will fit many other kinds of data, and hence be sensitive in this way. On the
otherhand, for model M 2, there is a large R area of the parameter space for which
the functions fit well. Since p(D|M ) = θ p(D|θ, M )p(θ|M ), this means that the
‘evidence’ for how well the model fits is roughly the volume of the space for which
the likelihood is very high. For two models for which the likelihood is equally high,
since the prior p(θ|M ) is a unit mass spread out over the parameter space, then
the model for which the likelihood covers a higher amount of the space will be
preferred.
 153

observe a certain performance, and judge whether or not this is more typical of
model 1 or 2....blah...
Need to explain the differences in these approaches, (see also the Laplace to su-
pernova paper).

A detailed example : Bayesian Error Analysis

We consider here how to assess if two classifiers, based on a set of test error re-
sults, are performing equally well. This question is often considered in the realm
of sampling theory, based on classical hypothesis testing. Here we present a simple
Bayesian treatment that is quite general, and also is able to deal with the (practi-
cally common) case where the errors that two classifiers make are dependent. This
is an introduction to Bayesian statistics applied to the analysis of experimental
results. In particular, the situation that we are interested in is how to tell if two
machine learning classifiers are performing the same1 . This is a standard problem
in assessing the results of a machine learning experiment2 . Let’s say classifier A
makes 20 errors, and 35 correct classifications, whereas classifier B makes 23 errors
and 32 correct classifications – is classifier A really better than classifier B? Of
course, intuitively, the uncertainty stems from the small amount of results. If we
had rather that classifier A makes 20000 errors and 35000 correct classifications,
classifier B makes 23000 errors and 32000 correct classifications, intuitively, we
would be much more confident than in the previous scenario that classifier A is
better than classifier B.
Initially, we will try to answer here the above question by appealing to a basic
Bayesian analysis. In doing this, we assume that the errors/labels that two clas-
sifiers make do not depend on each other. Later we’ll relax this to show how one
can make a test to see if they are dependent.
This small note is essentially a classic exercise in Bayesian statistics. The interested
reader may consult [32] (chapter 37 is most closely related to this area, although
the whole of part IV is relevant) and references therein for an introduction to the
general area.

Error Analysis

Consider a situation where two classifiers A and B have been tested on some data,
so that we have, for each example in the test set, an error pair

(ea (µ), eb (µ)) , µ = 1, . . . P

where P is the number of test data points, and ea ∈ {1, . . . Q} (and similarly for
eb ). That is, there are Q possible types of error that can occur. This is useful
in text classification, where TruePositive, FalseNegative, TrueNegative and False-
Positive might form four kinds of ‘errors’. For notational simplicity we also call
a TruePositive an ‘error’. It might be more appropriate to use a term such as
‘outcome label’, although this should also not be confused with the class label of
1 The theory is readily extensible to multiple classifiers, and is left as an exercise for the interested
reader.
2 Ideally, a true Bayesian will use a Bayesian Classifier, for which there will always, in principle,

be a direct way to estimate the suitability of the model in explaining the experimental data.
We consider here the less fortunate situation where two non-Bayesian classifiers have been
used, and only their test performances are available for evaluating the classifiers.
 154

the classifier – it is their evaluation against the truth that we are interested in.

Let’s call ea = {ea (µ), µ = 1, . . . , P }, the sample set A, and similarly for eb .

We are here interested in the question :

‘How much evidence is there supporting that the two classifiers are performing dif-
ferently?’

Mathematically, our major assumption here is that this is the same question as :

‘How much evidence is there in favour of two sample sets being from different
multinomial distributions?’
The main question that we address here is to test whether or not two classifiers
are essentially performing the same. To do this, we have two hypotheses :
1. Hindep : The sample sets are from different distribution.
2. Hsame : The sample sets are from the same distribution.
We need then to formally mathematically state what these two hypotheses mean.
In both cases, however, we will make the independence of trials assumption
Y
p(ea , eb ) = p(ea (µ), eb (µ)).
µ

Hindep : Independence of classifiers

For Hindep , we assume

p(ea (µ), eb (µ)|Hindep ) = p(ea (µ)|Hindep )p(eb (µ)|Hindep )

Note that we don’t need to make this independence-of-errors assumption, since it

is often quite reasonable to assume that both classifiers will tend to perform well
on the same ‘easy’ example, and perhaps poorly on the same ‘difficult’ example,
see fig(14.3). We’ll consider how to implement this case in a later section.

Since each classifier can make one of Q types of errors, we need to specify what
the probability of making such an error could be. For classifier A, we write
X
α = (α1 , . . . , αQ ) , αq = 1
q

and similarly for β. (These are the values of the probability tables for generating
errors).

Hence, under the independence assumption a probability for generating an error

on the µth example,
p(ea (µ)|α, Hindep )
and likewise for classifier B.
 155

The data likelihood

Since the data is assumed generated by a multinomial distribution, the likelihood

of generating the training data is
Q
Y ca
p(ea |α) = αqq
q=1

where caq is the number of times that classifier A makes error q 3 . A similar expres-
sion holds for classifier B.

Dirichlet Prior

Since we are dealing with multinomial distributions, it is convenient to use the

Dirichlet prior, which is conjugate to the multinomial distribution:
1 Y uq −1
p(α) = α
Z(u) q q

where
QQ
Γ(uq )
Z(u) = q=1
P 
Q
Γ q=1 u q

The prior parameter u controls how strongly the mass of the distribution is pushed
to the corners of the simplex. Setting uq = 1 for all q corresponds to a uniform
prior. The uniform prior assumption is reasonable, although there may be situa-
tions where it would preferable to use non-uniform priors4.

Posterior

With a Dirichlet prior and a multinomial likelihood term, the posterior is another
Dirichlet distribution (dropping the a index, since this result is general),
Q
1 Y
p(α|e) = αcq +uq −1 (14.0.8)
Z(u + c) q=1 q

where c are the counts of the errors.

3 The kinds of errors are assumed mutually exclusive and exhaustive. Clearly, the exhaustive
condition means that there are dependencies in the errors produced at one time – this is
taken care of by the constraint that the probabilities sum to one. Also, a potential source of
confusion is whether or not we view the dataset errors as a sample from a distribution with
a fixed total number of errors, say 50 TruePositve, 76 FalseNegative, etc, which would add
combinatorial prefactors to the data likelihoods. Clearly, also in this case, if we know the total
number of datapoints, then the errors are also not independent. Essentially, in the context
here, using such a system is incorrect, since we only have a single dataset, and our interest is
in the likelihood of generating this single dataset, and not in the likelihood of generating error
counts.
4 We could also use p(θ) to specify a distribution of priors p(u|θ), over which one can then

integrate.
 156

α β P α β
α

ea eb ea eb ea , eb ea eb
(a) (b) (d)
(c)

Figure 14.3: (a) Hindep : Corresponds to the errors for the two classifiers being
independently generated. (b) Hsame : both errors are generated from the same
distribution. (c) Hdep : the errors are dependent (‘correlated’). (d) Hrelated : In
this case the distributions α and β which generate ea and eb are related in some
way – for example they may be constrained to be similar through the variable r.
This case is not considered in the text.

Hindep : Model Likelihood

In the Bayesian framework, we want to find how likely it is that a model/hypothesis

is responsible for generating the data:

p(Hindep |ea , eb ) = p(ea , eb |Hindep )p(Hindep )/p(ea , eb )

where p(Hindep ) is our prior belief that Hindep is the correct hypothesis. Note that
the normalising constant p(ea , eb ) does not depend on the hypothesis. Then

p(ea , eb )p(Hindep |ea , eb ) (14.0.9)

Z
= p(ea , eb |α, β, Hindep )p(α, β|Hindep )p(Hindep )dαdβ
Z Z
= p(Hindep ) p(ea |α, Hindep )p(α|Hindep )dα p(eb |β, Hindep )p(β|Hindep )dβ

where we assumed (pretty innocuously) p(α, β|Hindep ) = p(α|Hindep )p(β|Hindep ).

Let’s calculate
Z
1 Y caq +uq −1 Z(u + ca )
p(ea |α, Hindep )p(α|Hindep )dα = αq =
Z(u) q Z(u)

Hence
Z(u + ca ) Z(u + cb )
p(Hindep |ea , eb ) = p(Hindep )
Z(u) Z(u)

Hsame : Model Likelihood

In Hsame , the hypothesis is that the errors for the two classifiers are generated
from the same multinomial distribution. Hence

p(ea , eb )p(Hsame |ea , eb ) (14.0.10)

Z
= p(Hsame ) p(ea |α, Hsame )p(eb |α, Hsame )p(α|Hsame )dα

Z(u + ca + cb )
= p(Hsame )
Z(u)
 157

Bayes Factor

If we assume that we have no prior preference for either hypothesis (p(Hindep ) =

p(Hsame )), then

p(Hindep |ea , eb ) Z(u + ca )Z(u + cb )

=
p(Hsame |ea , eb ) Z(u)Z(u + ca + cb )

This is the evidence to suggest that the data were generated by two different
multinomial distributions. In other words, this is the evidence in favour of the two
classifiers being different.

Examples

In the experiments that I demonstrate here and elsewhere, I’ll assume that there
are three kinds of ‘errors’, Q = 3.
• We have the two error counts ca = [39, 26, 35] and cb = [63, 12, 25]
Then, the above Bayes factor is 20.7 – strong evidence in favour of the two
classifiers being different. (This is consistent with the model I used to gen-
erate the data – they were indeed from different multinomial distributions).
• Alternatively, if we have the two error counts ca = [52, 20, 28] and cb =
[44, 14, 42]
Then, the above Bayes factor is 0.38 – weak evidence against the two clas-
sifiers being different. (This is consistent with the model I used to generate
the data – they were indeed from the same multinomial distributions)
• As a final example, consider counts ca = [459, 191, 350] and cb = [465, 206, 329].
This gives a Bayes factor of 0.008 – strong evidence that the two classifiers
are statistically the same (Indeed, the errors were in this case generated by
the same multinomial).
These results show that the Bayesian analysis performs in a way that is consistent
with the intuition that the more test data we have, the more confident we are in
our statements about which is the better model.

Dependent Error Analysis

Here we consider the (perhaps more common) case that errors are dependent. For
example, it is often the case that if classifier A works well, then classifier B will
also work well. Similarly, if one classifier performs poorly, then often the other
will too. Here, we assume that dependencies exist, but we make no preferences for
one to another (of course, such preferences would be straightforward to include if
desired). (There may be some interest in situations where if classifier A performs
poorly, then classifier B is likely to perform well ). Thus we want to consider the
Hypothesis

Hdep : the errors that the two classifiers make are dependent.

For convenience, let’s write

e = (ea , eb )
 158

Mathematically, we need to specify a distribution:

p(ea (µ), eb (µ)|Hdep ) = p(ea (µ), eb (µ)|P, Hdep )

here P is a Q × Q matrix of probabilities:

[P ]ij = p(ea = i, eb = j)

namely that the ij element of P is the probability that A makes error i, and B
makes error j.

Then, as before
1
p(e)p(Hdep |e) = p(e|Hdep )
p(Hdep )
Z
= p(e, P |Hdep )dP
Z
= p(e|P, Hdep )p(P |Hdep )dP

Assuming a Dirichlet prior on P , with parameters U , we have

Z(vec (U + C))
p(e)p(Hdep |e) = p(Hdep )
Z(vec(U ))

where vec(D) simply forms a vector by concatenating the rows of the matrix D.
Here C is the count matrix, with [C]ij equal to the number of times that joint
error (ea = i, eb = j) occurred in the P datapoints. As before, we can then use
this in a Bayes factor calculation. For the uniform prior, [U ]ij = 1, ∀i, j.

Testing for dependencies in the classifiers

Imagine that we wish to test whether or not the errors of the classifiers are depen-
dent Hdep , against the hypothesis that they are independent Hindep .

p(Hindep ) Z(u + ca ) Z(u + cb ) Z(vec(U ))

=
p(Hdep ) Z(u) Z(u) Z(vec (U + C))

Examples

• Consider the error count matrix C

 
98 7 93
 168 13 163 
245 12 201

so that ca = [511, 32, 457], and cb = [198, 344, 458]. Then

p(Hindep )
= 3020
p(Hdep )

– very strong evidence that the classifiers perform independently (indeed,

this is consistent with the way I generated the data).
 159

• Consider the error count matrix C

 
82 120 83
 107 162 4 
170 203 70

so that ca = [359, 485, 156], and cb = [284, 273, 443]. Then

p(Hindep )
= 2 × 10−18
p(Hdep )

– extremely strong evidence that the classifiers perform dependently (again,

this is consistent with the way I generated the data).

Dependent vs the Same

Perhaps the most useful test that can be done practically is between the Hdep versus
Hsame . This is because, in practice, it is reasonable to believe that dependencies
are quite likely in the errors that classifiers make (both classifiers will do well on
‘easy’ test examples, and badly on ‘difficult’ examples). In this sense, it is natural
to believe that dependencies will most likely exist in practice. The relevant question
is : are these dependencies strong enough to make us believe in fact that the errors
are coming from the same process? In this sense, we want to test

p(Hsame ) Z(u + ca + cb ) Z(vec(U ))

=
p(Hdep ) Z(u) Z(vec (U + C))

• Consider an experiment which gives the test error count matrix C

 
105 42 172
 42 45 29 
192 203 170

so that ca = [339, 290, 371], and cb = [319, 116, 565]. Then

p(Hsame )
= 4.5 × 10−38
p(Hdep )

– extremely strong evidence that the classifiers are performing differently

(this is consistent with the way I generated this data set).
• Consider an experiment which gives the test error count matrix C
 
15 8 10
 5 4 8 
13 12 25

so that ca = [33, 24, 43], and cb = [33, 17, 50]. Then

p(Hsame )
= 42
p(Hdep )

– strong evidence that the classifiers are performing the same (this is consis-
tent with the way I generated this data set).
 160

Hdep :Looking at the Posterior

Personally, I think that in machine learning, it is most natural to consider Hdep ,

due to the typical situation that when classifier A works well on example µ, then
so will classifier B. Conversely, when one works badly on an example, then so will
the other. In this case, we may simply want to work with Hdep , and try to assess
this posterior directly.
Consider the following simple example where Q = 2. In this case P is a 2 × 2
matrix, with elements Pij representing the probability that classifier A generates
an error of type i, and classifier B makes an error on type j on the same example.
...to be continued. I want to write down the posterior mean, and standard error
bars around the posterior Pij . This involves marginals of Dirichlet. It would also
be good to try to answer if A makes more errors than B – this involves integration
over a restricted space.....

14.1 Problems
Exercise 39 blah

14.2 Solutions
39
15 Bayesian Regression

Bayesian Regression
Regression refers to inferring an unknown input-output mapping on the basis of
observed data D = {(xµ , tµ ), µ = 1, . . . P }, where (xµ , tµ ) represents an input-
output pair. For example, fit a function to the crosses in fig(15.1a). Since there
is the possibility that each observed output tµ has been corrupted by noise, we
would like to recover the underlying clean input-output function. We assume that
each (clean) output is generated from the model f (x; w) where the parameters w
of the function f are unknown and that the observed outputs tµ are generated by
the addition of noise η to the clean model output,
t = f (x; w) + η (15.0.1)
If the noise is Gaussian distributed, η ∼ N (0, σ 2 ), the model M generates an
output t for input x with probability
  √
1
p(t|w, x, M ) = exp − 2 (t − f (x; w))2 / 2πσ 2 (15.0.2)
2σ
If we assume that each data input-output pair is generated identically and inde-
pendently from the others, the data likelihood is
P
Y
p(D|w, M ) = p(tµ |w, xµ , M ) (15.0.3)
µ=1

(Strictly speaking, we should write p(t1 , . . . , tP |w, x1 , . . . , xP , M ) on the left hand

side of the above equation. However, since we assume that the training inputs are
fixed and non-noisy, it is convenient and conventional to write p(D|w, M )). The
posterior distribution p(w|D, M ) ∝ p(D|w, M )p(w|M ) is
βX µ 2 P
log p(w|D, M ) = − (t − f (xµ ; w)) +log p (w|M )+ log β+const. (15.0.4)
2 µ 2

where β = 1/σ 2 . Note the similarity between equation (15.0.4) and the sum square
regularised training error used in standard approaches to fitting functions to data,
for example using neural networks [33]. In the Bayesian framework, we can mo-
tivate the choice of a sum square error measure as equivalent to the assumption
of additive Gaussian noise. Typically, we wish to encourage smoother functions
so that the phenomenon of overfitting is avoided. One approach to solving this
problem is to use a regulariser penalty term to the training error. In the Bayesian
framework, we use a prior to achieve a similar effect. In principle, however, the
Bayesian should make use of the full posterior distribution, and not just a single
weight value. In standard neural network training, it is good practice to use com-
mittees of networks, rather than relying on the prediction of a single network[33].
In the Bayesian framework, the posterior automatically specifies a committee (in-
deed, a distribution) of networks, and the importance attached to each committee
members prediction is simply the posterior probability of that network weight.

161
 162

(a) (b) (c)

Figure 15.1: Along the horizontal axis we plot the input x and along the vertical
axis the output t. (a) The raw input-output training data. (b) Prediction using
regularised training and fixed hyperparameters. (c) Prediction with error bars,
using ML-II optimised hyperparameters.

RBFs and Generalised Linear Models

Generalised linear models have the form
X
f (x; w) = wi φi (x) ≡ wT Φ(x) (15.0.5)
i

Such models have a linear parameter dependence, but nevertheless represent a

non-linear input-output mapping if the basis functions φ(x), i = 1, . . . , k are non-
linear. Radial basis functions are an example of such a network[33]. A popular

2
choice is to use Gaussian basis functions φi (x) = exp(− x − µi /(2λ2 )). In this
discussion, we will assume that the centres µi are fixed, but that the width of the
basis functions λ is a hyperparameter that can be adapted. Since the output is
linearly dependent on w, we can discourage extreme output values by penalising
large weight values. A sensible weight prior is thus
α k
log p(w|α) = − wT w + log α + const. (15.0.6)
2 2
Under the Gaussian noise assumption, the posterior distribution is
P
βX µ α
log p(w|Γ, D) = − (t − wT Φ(x))2 − wT w + const. (15.0.7)
2 µ=1 2

where Γ represents the hyperparameter set {α, β, λ}. (We drop the fixed model
dependency wherever convenient). The weight posterior is therefore a Gaussian,
p(w|Γ, D) = N (w̄, S) where
P
!−1 P
X X
S = αI + β Φ (xµ ) ΦT (xµ ) w̄ = βΣ tµ Φ (xµ ) (15.0.8)
µ=1 µ=1

The mean predictor is straightforward to calculate,

Z
f¯(x) ≡ f (x; w)p(w|D, Γ)dw = w̄T Φ (x) .

Similarly, error bars are straightforward,

p var(f (x)) = Φ(x)T SΦ(x) (predictive stan-
2
dard errors are given by var(f ) + σ ). In fig(15.1b), we show the mean prediction
on the data in fig(15.1a) using 15 Gaussian basis functions with width λ = 0.03
spread out evenly over the input space. We set the other hyperparameters to be
β = 100 and α = 1. The prediction severely overfits the data, a result of a poor
choice of hyperparameters.
 163

Determining Hyperparameters: ML-II

How would the mean predictor be calculated if we were to include the hyperpara-
meters Γ as part of a hierarchical model? Formally, this becomes
Z Z Z 
¯
f (x) = f (x; w)p(w, Γ|D)dwdΓ = f (x; w)p(w|Γ, D)dw p(Γ|D)dΓ

(15.0.9)
The term in curly brackets is the mean predictor for fixed hyperparameters. We
therefore weight each mean predictor by the posterior probability of the hyperpa-
rameter p(Γ|D). Equation (15.0.9) shows how to combine different models in an
ensemble – each model prediction is weighted by the posterior probability of the
model. There are other non-Bayesian approaches to model combination in which
the determination of the combination coefficients is motivated heuristically.
Provided the hyperparameters are well determined by the data, we may instead
approximate the above hyperparameter integral by finding the MAP hyperpara-
meters Γ∗ = arg maxΓ p(Γ|D). Since p(Γ|D) = p(D|Γ)p(Γ)/p(D), if the prior belief
about the hyperparameters is weak (p(Γ) ≈ const.), we can estimate the optimal
hyperparameters by optimising the hyperparameter likelihood
Z
p(D|Γ) = p(D|Γ, w)p(w|Γ)dw (15.0.10)

This approach to setting hyperparameters is called ‘ML-II’ [33, 27] and assumes
that we can calculate the integral in equation (15.0.10). In the case of GLMs, this
involves only Gaussian integration, giving
P
X 2
2 log p(D|Γ) = −β (tµ ) +dT S−1 d−log |S|+k log α+P log β +const. (15.0.11)
µ=1
P
where d = β µ Φ(xµ )tµ . Using the hyperparameters α, β, λ that optimise the
above expression gives the results in fig(15.1c) where we plot both the mean pre-
dictions and standard predictive error bars. This solution is more acceptable than
the previous one in which the hyperparameters were not optimised, and demon-
strates that overfitting is avoided automatically. A non-Bayesian approach to
model fitting based on minimsing a regularised training error would typically use
a procedure such as cross validation to determine the regularisation parameters
(hyperparameters). Such approaches require the use of validation data[33]. An
advantage of the Bayesian approach is that hyperparameters can be set without
the need for validation data, and thus all the data can be used directly for training.

The Kernel Trick

(See also the section on logistic regression). We can write the solution to w in the
form
X
w= αµ φ(xµ )
µ

And hence the scalar product wT φ(x) is of the form

X
αµ φT (x)φ(xµ )
µ
| {z }
K(x,xµ )
 164

This means we can just use the kernels K throughout, and use the αµ as the
parameters. It’s an analogous treatment as for classification.... One point to bear
in mind though is that the predictions usually will decay to zero away from the
data (this depends on the choice of the kernel, but is usually the case). This
means that we will predict very confidently that the regression should be zero, far
from the training data1 . This is not really what we want – we want to be highly
uncertain away from the training data. This isn’t a problem if we use finite basis
functions φ which are non-local, for example they grow to infinity at infinity. To
be continued...relationships to Gaussian Processes.

Relation to Gaussian Processes

The use of GLMs can be difficult in cases where the input dimension is high since
the number of basis functions required to cover the input space fairly well grows
exponentially with the input dimension – the so called ‘curse of dimensionality’[33].
If we specify n points of interest xi , i ∈ 1, . . . n in the input space, the GLM
specifies an n-dimensional
Gaussian distribution on the function values f1 , . . . , fn
with mean f¯i = w̄T Φ xi and covariance matrix with elements cij = c(xi , xj ) =

T

Φ xi ΣΦ xj . The idea behind a GP is that we can free ourselves from the
restriction to choosing a covariance function c(xi , xj ) of the form provided by
the GLM prior – any valid covariance function can be used instead. Similarly,
we are free to choose the mean function f¯i = m(x
i ). A common choice for the
covariance function is c(xi , xj ) = exp −|xi − xj |2 . The motivation is that the
function space distribution will have the property that for inputs xi and xj which
are close together, the outputs f (xi ) and f (xj ) will be highly correlated, ensuring
smoothness. This is one way of obviating the curse of dimensionality since the
matrix dimensions depend on the number of training points, and not on the number
of basis functions used. However, for problems with a large number of training
points, computational difficulties can arise, and approximations again need to be
considered.

15.1 Problems
Exercise 40 The question relates to Bayesian regression.
• Show that for

f = wT x

and p(w) ∼ N (0, Σ), that p(f |x) is Gaussian distributed. Furthermore, find
the mean and covariance of this Gaussian.
• Consider a target point t which is related to the function f by additive noise
σ 2 . What is p(f |t, x)? Hint : use p(f |t, x) ∝ p(t|f, x)p(f |x).

15.2 Solutions
40

1 For classification, this isn’t a problem since the argument of the sigmoid function goes to zero,
which means that there is complete uncertainty in the class prediction.
 16 Logistic Regression

16.1 Introduction
We’ve talked about using Generative Models to do classification. Now we look at
a discriminative approach.

A common application of machine learning is to classify a novel instance x as

belonging to a particular class. Here we concentrate on only two class problems.
Explicitly, we are given some training data, D = {(xµ , tµ ), µ = 1 . . . P }, where the
targets c ∈ {0, 1}. An example is given is given in fig(16.1) in which the training
inputs x are two dimensional real values, and the associated target values are
plotted.
We need to make an assignment for a novel point x to one of the two classes. More
generally, we can assign the probability that a novel input x belongs to class 1

p(c = 1|x) = f (x; w) (16.1.1)

where f is some function parameterised by w. Since the function f (x) represents

a probability, f (x) must be bounded between 0 and 1.
In previous chapters we have used class conditional density estimation and Bayes
rule to form a classifier p(c|x) ∝ p(x|c)p(c). Here, we take the direct approach and
postulate a model explicitly for p(c|x). There are advantages and disadvantages in
both of these approaches – my personal favourite is to try the indirect approach
more often than the direct approach.
Logistic Sigmoid Function
One of the simplest choices of function is the sigmoid function, f (x) = 1/(1 +
exp(x)), which is plotted in fig(16.2). What about the argument of the function
f ? Logistic regression corresponds to the choice

p(c = 1|x) = σ(b + xT w) (16.1.2)

where b is a constant scalar, and w is a constant vector. When the argument of

the sigmoid function b + xT w is above zero, the probability that the input point
x belongs to class 1 is above 0.5. The greater the argument value is, the higher is
the probability that x is in class 1 (according to our logistic regresssion model).
Similarly, the more negative is the argument, the more likely it is that x belongs
to class 0.

Linear (Hyperplane) Decision The hyperplane b + xT w = 0 forms the decision boundary (where p(c = 1|x) = 0.5)
Boundary – on the one side, examples are classified as 1’s, and on the other, 0’s. The
“bias” parameter b simply shifts the decision boundary by a constant amount. The
orientation of the decision boundary is determined by w – indeed, w represents the

165
 166

0.9

0
0.8

0.7
1
0
0.6

1 0 1 1
0.5

0.4 0
0
0.3
1

0.2

0.1

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Figure 16.1:
1

0.9

0.8

0.7

0.6

sigma(x)
0.5

0.4

0.3

0.2

0.1

0
−10 −8 −6 −4 −2 0 2 4 6 8 10
x

Figure 16.2: The logistic sigmoid function σ(x) = 1/(1 + e−x ).

normal to the hyperplane. To understand this, consider a new point x∗ = x + w⊥ ,

where w⊥ is a vector perpendicular to w (wT w⊥ = 0). Then


b + wT x∗ = b + wT x + w⊥ = b + wT x + wT w⊥ = b + wT x = 0 (16.1.3)

Thus if x is on the decision boundary, so is x plus any vector perpendicular to w.

In n dimensions, the space of vectors that are perpendicular to w occupy an n − 1
dimensional linear subspace, in otherwords an n − 1 dimensional hyperplane. For
example, if the data is two dimensional, the decision boundary is a one dimensional
hyperplane, a line. This situation is depicted in fig(16.3). If all the training data
for class 1 lie on one side of the line, and for class 0 on the other, the data is said
Classification confidence to be linearly separable. We plot σ(b + xT w) for different values of w in fig(16.4)
and fig(16.5). The decision boundary is at σ(x) = 0.5. Note how the classification
becomes more confident as the size of the weight vector components increases –
that is, as we move only a short distance away from the decision boundary, we
predict very confidently the class of x if the weights are large.
The Perceptron As we have defined it sofar, x is assigned to class 1 with some probability. It is not
certainly in class 1 unless p(c = 1|x) = 1, which cannot happen unless the weights
tend to infinity.
The perceptron is a historical simpler model in which x is assigned to class 1 with
complete certainty if b + wT x ≥ 0, and to class 0 otherwise. Alternatively, we can
define a new rule :

p(c = 1|x) = θ(b + xT w) (16.1.4)

 167

0 0
0 0
0
0 w 1 1

1 1
1
1
1 1

Figure 16.3: The decision boundary p(c = 1|x) = 0.5 (solid line). For two dimen-
sional data, the decision boundary is a line. If all the training data for class 1 lie
on one side of the line, and for class 0 on the other, the data is said to be linearly
separable.
w(1)=7, w(2)=−3.5, b=0

0.9

0.8

0.7

0.6

0.5

0.4

0.3
−1
0.2
−0.5
0.1

0 0
−1
−0.5 0.5
0
0.5 1
1 x(1)
x(2)

Figure 16.4: The logistic sigmoid function σ(x) = 1/(1 + e−x ), with x = wT x + b.

where the “theta” function is defined as θ(x) = 1 if x ≥ 0, and θ(x) = 0 if

x < 0. Since the perceptron is just a special case (the deterministic limit) of
logistic regression, we develop here training algorithms for the more general case.

16.1.1 Training

Given a data set D, how can we adjust/“learn” the weights to obtain a good
classification? Probabilistically, if we assume that each data point has been drawn
independently from the same distribution that generates the data (the standard
 168

w(1)=14, w(2)=−7, b=0

0.8

0.6

0.4

−1
0.2
−0.5

0 0
−1
−0.5 0.5
0
0.5
1
1 x(1)
x(2)

Figure 16.5: The logistic sigmoid function σ(x) = 1/(1 + e−x ), with x = wT x + b .
1000 iterations
1.2

0.8

0
1
0.6

x(2)
0 0 1 1

0.4 0
1

0.2 1

−0.2
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
x(1)

Figure 16.6: The decision boundary p(c = 1|x) = 0.5 (solid line) and confidence
boundaries p(c = 1|x) = 0.9 and p(c = 1|x) = 0.1.

i.i.d assumption), the likelihood of the observed data is1

P
Y P
Y µ µ
p(D) = p(cµ |xµ ) = (p(c = 1|xµ ))c (1 − p(c = 1|xµ ))1−c (16.1.5)
µ=1 µ=1

Thus the log likelihood is

P
X
L= cµ log p(c = 1|xµ ) + (1 − cµ ) log (1 − p(c = 1|xµ )) (16.1.6)
µ=1

Using our assumed logistic regression model, this becomes

P
X

L(w, b) = cµ log σ(b + wT xµ ) + (1 − cµ ) log 1 − σ(b + wT xµ ) (16.1.7)
µ=1

16.1.2 Gradient Ascent

We wish to maximise the likelihood of the observed data. To do this, we can make
use of gradient information of the likelihood, and then ascend the likelihood.
1 Note that this is not quite the same strategy that we used in density estimation. There we
made, for each class, a model of how x is distributed. That is, given the class c, make a model
of x, p(x|c). We saw that, using Bayes rule, we can use p(x|c) to make class predictions p(c|x).
Here, however, we assume that, given x, we wish to make a model of the class probability,
p(c|x) directly. This does not require us to use Bayes rule to make a class prediction. Which
approach is best depends on the problem, but my personal feeling is that density estimation
p(x|c) is worth considering first.
 169

The gradient is given by (using σ ′ (x) = σ(x)(1 − σ(x)))

P
X
∇w L = (cµ − σ(xµ ; w))xµ (16.1.8)
µ=1

and the derivative with respect to the biases is

P
dL X µ
= (c − σ(xµ ; w)) (16.1.9)
db µ=1

Gradient ascent would then give

wnew = w + η∇w L (16.1.10)

bnew = b + ηdL/db (16.1.11)

where η, the learning rate is a small scalar chosen small enough to ensure conver-
gence of the method (a reasonable guess is to use η = 0.1). The application of the
above rule will lead to a gradual increase in the log likelihood.
Batch version Writing the above result out in full gives explicitly
P
X
wnew = w + η (cµ − σ(xµ ; w))xµ (16.1.12)
µ=1

P
X
new
b =b+η (cµ − σ(xµ ; w)) (16.1.13)
µ=1

This is called a “batch” update since the parameters w and b are updated only
after passing through the whole (batch) of training data – see the MATLAB code
below which implements the batch version (note that this is not written optimally
to improve readability). We use a stopping criterion so that if the gradient of the
objective function (the log likelihood) becomes quite small, we are close to the
optimum (where the gradient will be zero), and we stop updating the weights.
Online version An alternative that is often preferred to Batch updating, is to update the parame-
ters after each training example has been considered:
η µ
wnew = w + (c − σ(xµ ; w))xµ (16.1.14)
P
η µ
bnew = b + (c − σ(xµ ; w)) (16.1.15)
P
These rules introduce a natural source of stochastic (random) type behaviour in
the updates, and can be useful in avoiding local minima. However, as we shall see
below, the error surface for logistic regression is bowl shaped, and hence there are
no local minima. However, it is useful to bear in mind the online procedure for
other optimisation problems with local minima.
 170

% Learning Logistic Linear Regression Using Gradient Ascent (BATCH VERSION)

n0 = 16; x0 = randn(2,n0) + repmat([1 -1]’,1,n0); % training data for class 0

n1 = 11; x1 = randn(2,n1) + repmat([-1 1]’,1,n1); % training data for class 1

eta = 0.1; % learning rate

w = [0 0]’; b = 0; % initial guess about the parameters
it = 0; itmax = 1000; % maximum number of iterations
gb = 1; gw = zeros(size(w)); % set gradients initally to ensure at least update

while sum(abs(gw)) + abs(gb) > 0.1 % continue whilst gradient is large

it = it + 1; % increment the number of updates carried out

gb = 0; gw = 0*gw; % reset gradients to zero

for d = 1:n1 % cycle through the class 1 data

c = 1 - 1/(1+exp(-(b+w’*x1(:,d))));
gb = gb + c;
gw = gw + c*x1(:,d);
end

for d = 1:n0 % cycle through the class 0 data

c = 0 - 1/(1+exp(-(b+w’*x0(:,d))));
gb = gb + c;
gw = gw + c*x0(:,d);
end

w = w + eta*gw; % update the weight vector

b = b + eta*gb; % update the bias scalar

if it > itmax; break; end

end

% calculate the probabilities p(c=1|x) for the training data :

disp(’p(c=1|x) for class 1 training data : ’);
1./(1+exp(-(repmat(b,1,n1)+w’*x1)))

disp(’p(c=1|x) for class 0 training data : ’);

1./(1+exp(-(repmat(b,1,n0)+w’*x0)))

One important point about the training is that, provided the data is linearly
separable, the weights will continue to increase, and the classifications will become
extreme. This may be an undesirable situation in case some of the training data
has been mislabelled, or a test point needs to be classified – it is rare that we
could be absolutely sure that a test point belongs to a particular class. For non-
linearly separable data, the predictions will be less certain, as reflected in a broad
confidence interval – see fig(16.7).
The error surface is
bowl-shaped
The Hessian of the log likelihood is

∂2H X µ µ
Hij ≡ =− xi xj σ µ (1 − σ µ ) (16.1.16)
∂wi wj µ
 171
10000 iterations
0.8

0.7

0.6
0 0
1
0.5

x(2)
0 1

1
0.4 0

0.3

1
0.2 1

0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
x(1)

Figure 16.7: The decision boundary p(c = 1|x) = 0.5 (solid line) and confidence
boundaries p(c = 1|x) = 0.9 and p(c = 1|x) = 0.1 for non-linearly separable data.
Note how the confidence interval remains broad.

This is negative definite since

 2
X X X
wi Hij wj = − wi xµi wj xµj σ µ (1−σ µ ) = −  wi xµi  σ µ (1−σ µ ) (16.1.17)
ij i,j,µ i,µ

This means that the error surface has a bowl shape, and gradient ascent is guar-
anteed to find the best solution, provided that the learning rate η is small enough.
Perceptron Convergence One can show that, provided that the data is linearly separable, the above pro-
Theorem cedure used in an online fashion for the perceptron (replacing σ(x) with θ(x))
converges in a finite number of steps. The details of this proof are not important
for this course, but the interested reader may consult Neural Networks for Pattern
Recognition, by Chris Bishop. Note that the online version will not converge if the
data is not linearly separable. The batch version will converge (provided that the
learning rate η is small) since the error surface is bowl shaped.

16.1.3 Avoiding Overconfident Classification

We saw that in the case that data is linearly separable, the weights will tend to
increase indefinitely (unless we use some stopping criterion). One way to avoid this
is to penalise weights that get too large. This can be done by adding a penalty
term to the objective function L(θ) where θ is a vector of all the parameters,
θ = (w, b),
L′ (θ) = L(θ) − αθ T θ. (16.1.18)
The scalar constant α > 0 encourages smaller values of θ (remember that we wish
to maximise the log likelihood). How do we choose an appropriate value for α?
We shall return to this issue in a later chapter on generalisation.

16.1.4 Logistic Regression and PCA ?

In previous chapters, we have looked at first using PCA to reduce the dimension
of the data, so that a high dimensional datapoint x is represented by a lower
dimensional vector y.
If e1 , . . . , em are the eigenvectors with largest eigenvalues of the covariance matrix
of the high-dimensional data, then the PCA representation is
yi = (ei )T (x − c) = (ei )T x + ai (16.1.19)
 172

1 0

0 1

Figure 16.8: The XOR problem. This is not linearly separable.

where c is the mean of the data, and ai is a constant for each datapoint. Using
vector notation, we can write

y = ET x + a (16.1.20)

where E is the matrix who’s ith column is the eigenvector ei . If we were to use
logistic regression on the y, the argument of the sigmoid σ(h) would be
h = wT y + b = wT (E T x + a) + b (16.1.21)
T T T
= (Ew) x + b + w a = w̃ x + b̃ (16.1.22)

Hence, there is nothing to be gained by first using PCA to reduce the dimension of
the data. Mathematically, PCA is a linear projection of the data. The argument
of the logistic function is also a linear function of the data, and a linear function
combined with another is simply another linear function.
However, there is a subtle point here. If we use PCA first, then use logistic regres-
sion afterwards, although overall, this is still representable as a logistic regression
problem, the problem is constrained since we have forced linear regression to work
in the subspace spanned by the PCA vectors. Consider 100 training vectors ran-
domly positioned in a 1000 dimensional space each with a random class 0 or 1.
With very high probability, these 100 vectors will be linearly separable. Now
project these vectors onto a 10 dimensional space: with very high probability, 100
vectors plotted in a 10 dimensional space will not be linearly separable. Hence, ar-
guably, we should not use PCA first since we could potentially transform a linearly
separable problem into a non-linearly separable problem.
The XOR problem
Consider the following four training points and class labels
{([0, 0], 0), ([0, 1], 1), ([1, 0], 1), ([1, 1], 0)}.
This data represents a basic logic function, the XOR function, and is plotted in
fig(16.8). This function is clearly not representable by a linear decision boundary,
an observation much used in the 1960’s to discredit work using perceptrons. To
overcome this, we clearly need to look at methods with more complex, non-linear
decision boundaries – indeed, we encountered a quadratic decision boundary in a
previous chapter. Historically, another approach was used to increase the complex-
ity of the decision boundary, and this helped spawn the area of neural networks,
to which we will return in a later chapter.

16.1.5 An Example : Classifying Handwritten Digits

If we apply logistic regression to our often used handwritten digits example, in

which there are 300 ones, and 300 sevens in the training data, and the same
 173

number in the test data, the training data is found to be linearly separable. This
may surprise you, but consider that there are 784 dimensions, and only 600 training
points. The stopping criterion used was the same as in the example MATLAB code
in this chapter. Using the linear decision boundary, the number of errors made on
the 600 test points is 12.

16.2 The Kernel Trick

The simple logistic method of doing discriminative classification is very popular.
In particular, one of the major benefits is the speed and simplicity of the training
algorithm. However, one major drawback is the simplicity of the decision surface
– a hyperplane. One way to improve the method is to consider mapping the inputs
x in a non-linear way to ψ(x):


p(c = 1|x) = σ wT ψ(x)

(adding a bias is a trivial extra modification). Note that ψ(x) does not have
to be of the same dimension as w. For example, the one-dimensional input x
could get mapped to a two dimensional vector (x2 , sin(x)). (Lower dimensional
mappings are also possible, but less popular since this can make it more difficult
to find a simple classifier). The usual motivation for this is that mapping into
a high dimensional space makes it easier to find a separating hyperplane in the
high dimensional space (remember that any set of points that are independent
can be linearly separated provided we have as many dimensions as datapoints –
this motivates the use of non-linear mappings since the related high-dimensional
datapoints will then usually be independent).
If we wish to use the ML criterion, we can use exactly the same algorithm as in
the standard case, except wherever there was a x before, this gets replaced with
ψ(x).

Is there another way to do training? Informally, if we assume that we begin with

the zero vector, and make updates according to the gradient ascent rule, then the
updates are of the form
X
wnew = w + γ(w)ψ(xµ )
µ

where γ is some scalar function. The point is that, by iterating the above equation,
any solution will therefore be of the form of a linear combination of the points ψ µ ,
where for simplicity, we write ψ µ ≡ ψ(xµ ). Hence, we may assume a solution
X
w= αµ ψ µ
µ

and try to find a solution in terms of the vector of parameters αµ . This is poten-
tially advantageous since there may be less training points than dimensions of ψ.
The classifier depends only on scalar products
X
wT ψ(x) = αµ ψ(xµ )T ψ(x)
µ

Hence, the only role that ψ plays is in the form of a scalar product:

K(x, x′ ) = ψ(x)T ψ(x′ )

 174

Since the right is a scalar product, it defines a positive definite (kernel) function
(see section (E)). These are symmetric functions for which, roughly speaking, the
corresponding matrix defined on a set of points xi , i = 1, . . . , N is positive define.
Indeed, Mercer’s Theorem states that a function defines a positive definite kernel
function if and only if it has such an inner product representation. What this
means is that we are then free to define a function which is positive definite kernel
function, since this is the only thing that the classifier depends on. (This is well
established in classical statistics, and forms the basis of Gaussian Processes – see
later chapter). Hence, we can define
!
X
µ
p(c = 1|x) = σ αµ K(x, x )
µ

For convenience, we can write the above as

p(c = 1|x) = σ αT k(x)
where the P dimensional vector k(x) has elements [k(x)]µ = K(x, xµ ). Then the
above is of exactly the same form as the original specification of logistic regression,
namely as a function of a linear combination of vectors. Hence the same training
algorithm to maximise the likelihood can be employed.
For example
′ 2
K(x, x′ ) = e−λ(x−x )
defines a positive definite kernel (see problems).
It should be clear from that above that essentially any method which depends on a
scalar product wT ψ(x), or indeed, possibly a set of such scalars, can be kernelised.
Recently, a small industry producing kernelised methods has been in action, based
on the same basic idea.

Support Vector Machines

The realisation that the higher the dimension of the space is, the easier it is
to find a hyperplane that linearly separates the data, forms the basis for the
Support Vector Machine method. The main idea (contrary to PCA) is to map each
vector in a much higher dimensional space, where the data can then be linearly
separated. Training points which do not affect the decision boundary can then
be discarded. We will not go into the details of how to do this in this course,
but the interested reader can consult http://www.support-vector.net. Related
methods currently produce the best performance for classifying handwritten digits
– better than average human performance. Essentially, however, the distinguishing
feature of the SVM approach is not in the idea of a high-dimensional projection, but
rather in the manner of finding the hyperplane. The idea is to find the hyperplane
such that the distance between the hyperplane and (only) those points which
determine the placement of the plane, should be maximal. This is a quadratic
programming problem. In this case, usually only a small set of the training data
effects the decision boundary. However, this method is not probabilistic, and no
satisfactory manner of formulating the SVM directly as a probabilistic model has
been achieved (although there have been numerous approaches, all of which contain
a fudge somewhere). More later......
A similar method which retains the benefits of a probabilistic analysis is the Rel-
evance Vector Machine.
 175

Are Kernels really necessary?

In the above, we saw how to define classifier

!
X
µ
p(c = 1|x) = σ αµ K(x, x )
µ

In the case that K is a Kernel, we can interpret this as essentially fitting a hy-
perplane through a set of points, where the points are the data training points
projected into a (usually) higher dimensional space. However, if one does not
require this condition, we can define a more general classifier
!
X
p(c = 1|x) = σ αi Ki (x)
i

where the Ki (x), i = 1, . . . , F are a fixed set of functions mapping the vector x
to a scalar. For example, if we set Ki (x) = tanh(xT wi ), and treat also wi as
a parameter, the solution will not be representable in a Kernel way. In these
more general settings, training is more complex, since the error surface cannot
be guaranteed to be convex, and simple gradient ascent methods (indeed, any
optimisation method) will potentially get trapped in a local optimum. In this
sense, Kernels are useful since they mean we can avoid training difficulties.
As an aside, consider if we set Ki (x) = tanh(xT wi ), for fixed wi , and treat only
the αi as adjustable parameters, then the solution is representable as a Kernel
(since the argument of the sigmoid is representable as a scalar product between a
parameter vector and a fixed vector function of x).

16.3 Mixture Models

The big advantage of the ‘kernelised’ verions of logistic regression is that it is
probabilistic. This means that we can do a Bayesian style analysis for training
(more later). Also, we can do things such as mixtures, and mixtures of experts.

16.3.1 Mixtures

How can we increase the power of the above methods? One way to do this is to
write
H
X H
X
p(c = 1|x) = p(c = 1, h|x) = p(c = 1|h, x)p(h|x)
h=1 h=1

Usually, the hidden variable h is taken to be discrete. Here, then p(c = 1|h, x) is
one of a set of H classifiers.
In a standard mixture model, we assume independence, p(h|x) = p(h).

16.3.2 Mixture of Experts

In a mixture of experts (cite Jordan) model, we assume that p(h|x) has some
parametric form, for example using a softmax function
h T
e(w ) x
p(h|x) = P
(w h′ )T x
h′ e
 176

In both cases, the natural way to train them is to use the variational EM approach,
since the h is a hidden variable.
Note that these methods are completely general, and not specific to logistic regres-
sion. To do... example of mixture of experts applied to handwritten digits.

16.3.3 A ‘Bayesian’ approach to setting the regularisation parameter

In a previous section, we mentioned a problem with the ML criterion (remember

that this is just a heuristic). How might we correct for this problem? Well, let’s
go back to basics, and see what we can do. From Bayes’ rule, we know that we are
ultimately interested in p(w|D), and that this is proportional to p(D|w)p(w). It’s
clear that the assumed flat prior on p(w) (which is related to the ML heuristic)
is letting us down, since very large values of w will give rise to over-confident
classifications. From the previous discussions, we know that we can kernelise the
logistic method to consider in more generality,
 
XP
′
p(c = 1|x) = σ  wµ′ K(x, xµ ) + b
µ′ =1

(In the following, I’ll set b to zero, just for notational clarity).

Note: a potential confusion here with notation. I’m now using w where previously
I used α. Now α refers to the precision.

How can we prevent the classifications becoming too severe? If the Kernel val-
ues themselves are bounded (for the squared exponential kernel, this is clearly
the case), then putting a soft constraint on the size of the components wµ will
discourage overly confident classifications.
A convenient prior p(w) one is to use a Gaussian constraint:

αP/2 T
p(w|α) = P/2
e−αw w/2
(2π)
where α is the inverse variance (also called the precision) of the Gaussian distrib-
ution. (Remember that here the dimension of w is equal to the number of training
points).
More formally, we could put another distribution on p(α), say a Gamma distribu-
tion (see section (C)), as part of an hierarchical prior.
Z Z
α T
p(w) = p(w|α)p(α) = e− 2 w w αγ−1 e−α/β dα
α

It’s clear that the RHS is another Gamma distribution (the Gaussian and Gamma
distributions are conjugate). Indeed (exercise) the reader can easily show that the
distribution of w is a t-distribution.
Here we’ll generally keep life simple, and assume the above ‘flat’ prior on α. We
have therefore a GM of the form
P
1 Y
p(w, α|D) = p(w|α)p(α) p(cµ |xµ , w)
Z µ=1
 177

w x

Figure 16.9: Graphical representation of logistic regression.

where the constant Z ensures normalisation,

Z P
Y
Z= p(w|α)p(α) p(cµ |xµ , w).
α,w µ=1

The above suggests the following strategies:

Full Bayesian Ultimately, we’ll be interested in using the classifier in novel situations. In
this sense, we would like to compute
Z
p(c = 1|x, D) = p(c = 1|x, w)p(w, α|D)
w,α

However, the P dimensional integrals over w cannot be analytically calcu-

lated. This means that approximations need to be considered. One approach
would be to draw samples from p(w, α|D) (see section (24)). Whilst this is
relatively straightforward, due to the relative benign nature of the posterior
in this particular case, we may as well exploit the relative simplicity of the
posterior to make an analytic based approximation.
Variational Method We could fit a variational distribution to the posterior. Fairly straightfor-
ward. Blah.... Not really necessary in this case since the posterior is simple.
Laplace Approximation If we have enough training datapoints, we might expect the posterior dis-
tribution to become reasonably well peaked around a most-probably value
of w. This is even more the case for α, since this is just a one dimensional
variable – almost certainly, given a reasonable amount of training data, the
posterior distribution p(α|D) will be very sharply peaked.
In the following, we’ll take the path of least resistance, and use the simplest ap-
proximation, the Laplace method. This is justifiable in this case since the posterior
p(w|α, D) is unimodal. Historically, this corresponds to one of the earliest appli-
cations of approximate Bayesian methods in machine learning[32].

16.3.4 Evidence Procedure

What about the setting of p(α)? If we assume a flat prior on α, this will effectively
mean that we favour smaller values of the variance, and hence small values of
the weights. In this case, finding the α that maximises p(α|D) is called ML-II
estimation (we don’t use ML at the first level to determine w, but rather use ML
at the second, hyperparameter level).
 178

Alternatively, we notice that we can integrate out analytically (usually) over the
one-dimensional α to obtain
P
1 Y
p(w|D) = p(w) p(cµ |xµ , w)
Z µ=1
R
where p(w) = p(w|α)p(α)dα and
Z P
Y
Z= p(w) p(cµ |xµ , w)
w µ=1

The main difficulty in both approaches above is that we cannot analytically inte-
grate over the w since the distribution, and we are forced to make an approxima-
tion. The best way to make an approximation has been a topic of some intense
debate. Should we integrate out the hyperparameter distribution first, and then at-
tempt to approximate the posterior distribution p(w|D), or approximate the joint
distribution p(w, α|D)? Since we have a good idea that p(α|D) will be sharply
peaked, and p(w|α, D) is unimodal, the argument goes that it makes sense to
make the simple unimodal Laplace approximation on the simple p(w|α, D), rather
than the more complex p(w|D).
Z Y

p(α|D) ∝ p(α) p(w|α) σ (2cµ − 1)(wT k µ )
w µ

µ µ i
where [k ]i ≡ K(x , x ). A simple Laplace approximation section (F) gives
1 P
log p(α|D) ≈ log p(α) − E(w∗ ) − log det 2πH + log α + const.
2 2
and
P
α T X

E(w) = w w− log σ wT hµ
2 µ=1

hµ = (2cµ − 1)k µ . The Laplace approximation states that we need to find the
minimum of E(w). Differentiating, we get
X
∇E = αw − (1 − σ µ )hµ
µ


where σ µ ≡ σ wT hµ . We could then use a simple gradient descent algorithm.
However, since the surface is convex, and the Hessian is simple to calculate,
P
X T
H(w) = αI + σ µ (1 − σ µ )hµ (hµ )
µ=1
| {z }
J

we may as well use a Newton update:

wnew = w − H −1 (∇E) (16.3.1)
Once this iteration has converged to a value w∗ , we are in a position to approximate
the likelihood. Since, ultimately, we want to calculate
dL ∂L ∂L ∂w∗
= + ∗
dα ∂α |∂w
{z } ∂α
=0
 179

to optimise L with respect to α, we only need consider the terms with an explicit
α dependence,
α 1 P
L(α) ≈ − (w∗ )T w∗ − log det(αI + J) + log α + const.
2 2 2


Differentiating wrt α, using ∂ log det(M ) = trace M −1 ∂M , and setting to zero,
we can make a fixedpoint iteration
P
αnew =   (16.3.2)
−1
(w∗ )T w∗ + trace (αI + J)

We have then the celebrated ‘evidence procedure’[32]:

1. Initialise w and α to (sensible!) values.
2. Find w∗ . This is achieved here by iterating equation (16.3.1) to convergence.
3. Update α according to equation (16.3.2). (Sometimes slightly different fixed
point updates are used.)
4. Iterate steps 2 and 3 until convergence of α.
To make predictions on novel inputs x, we can do the following
Z Z
p(c = 1|x, D) = p(c = 1|x, w)p(w|D)dw = σ (h) p(h|x, D)dh

where p(h|x, D) is the distribution of the quantity xT w. Under the Laplace ap-
proximation, w is Gaussian,

p(w∗ |D) = N (µ, Σ)

where µ = w∗ , and Σ = (H(w∗ ))−1 . This means that h (which is linearly related
to w) is also Gaussian distributed

p(h|x, D) = N (xT w∗ , xT Σx)

Then, to make predictions, we can do the following

Z
p(c = 1|x, D) = σ (h) p(h|x, D)dh

The simple one-dimensional integral over h is carried out numerically.

Need to extend the evidence stuff to determining the hyperparameters λ. This is
all related to GP classification.

Relevance Vector Machine

One might wonder if all datapoints are equally important in determining the so-
lution of a classification problem. Intuitively, it would be reasonable to think that
many points don’t really affect the solution, and could be effectively removed, or
pruned from the dataset. We can achieve this by repeating the previous evidence
framework analysis, but this time, rather than having a global α for the whole
weight vector, rather we have a penalty
PP 2
p(w|α) ∝ e− ı=1 αi wi /2
 180

6 0.4
0.5 0.3
0.2

0.
4
4

0.3
0.1

0.2
0.5

0.3
0.4
0.

0.1
2 6
0.2

0.6
0.7 0.
0.5 3
0.9 0.4

0.
8
0.7 0.5
0 0.4

0.8

0.8

0.7
0.2 0.3
0. 0.9

0.6
0.570.
−2 0.4 6

0.
0.

1
0.
2

0.1
0.
−4

2
3

0.
0.

0.3

0.
5
0.4
−6
−6 −4 −2 0 2 4 6

Figure 16.10: An example using Logistic Regression with the squared exponential
′ 2
kernel, e−(x−x ) . The green points are training data from class 1, and the red
points are training data from class 0. The contours represent the probability of
being in class 1. The optimal value of α found by the evidence procedure in this
case is 0.45.

The reader may verify that the only alterations in the previous evidence procedure
are simply
X
[∇E]i = αi wi − (1 − σ µ )hµi
µ

H(w) = diag(α) + J
These are used in the Newton update formula as before. The implicit equation for
the α’s is given by
1
αi =
wi2 + Σii

where Σ = (H(w))−1 . Running this procedure, one typically finds that many of
the α’s tend to infinity, and may be effectively pruned from the dataset. Those
remaining tend to be rather in the centres of mass of a bunch of datapoints of the
same class. Contrast this with the situation in SVMs, where the retained data-
points tend to be on the decision boundaries. In that sense, the RVM and SVM
have very different characteristics. The number of training points retained by the
RVM tends to be very small – smaller indeed that the number retained in the
SVM framework. However, the RVM is a little more computationally expensive
than SVMs, but otherwise retains the advantages inherited from a probabilistic
framework[34]. Naturally enough, one can extend this idea of sparseness to many
other probabilistic models, and is a special case of the automatic relevance deter-
mination (ARD) method introduced by MacKay and Neal[32]. Finding such sparse
representations has obvious applications in compression. A hot research issue to
speed up training of RVMs.
 181
8 8
0.4 0.4

0.2 0.3 0.2

6 0.3 6 0.1
0.1

0.

0.
4

4
0.
3

0.3
0.4

0.4
4 4

0.1
0.5

0.2
0.5 0. 0.
4 4

0. 0.1
0.
0. 0

1
2 .1

0.3
2
2 2 0.6 0.30.2

3
0

0.
0.7 .3

0.6
8 0.9 0.8 0.9 0.7
0.
0.

0.5

0.5
6
0 0.5 0.4 0 0.4 0.5 0.4
0.4
0.2 3

0
0..67 0.3
3 0. 0.2 1

0.7 4 .3
0.

0.7

0.7
0.1

0. 0
0.
−2 0.9 .8 −2 0.9

0.3
0 0.8

0.4 0.2
0.3
0.6

0.1

0.1
0.6
0. 0.

0.2
5 5
−4 −4

0.1
0.1

0.

0.4

0.
0.

5
2
0.2

0.
4
−6 0.3 −6 0.3
0.4
0.4

−8 −8
−8 −6 −4 −2 0 2 4 6 8 −8 −6 −4 −2 0 2 4 6 8

Figure 16.11: An example using RVM classification with the squared exponential
′ 2
kernel, e−(x−x ) . The green points are training data from class 1, and the red
points are training data from class 0. The contours represent the probability of
being in class 1. On the left are plotted the training points. On the right we plot
the training points weighted by their relevance value 1/αµ . Nearly all the points
have a value so small that they effectively vanish.

16.4 Problems
Exercise 41 Show that
′ 2
K(x, x′ ) = e−λ(x−x )

defines a positive definite kernel.

Hint: one simple way to show this is to consider expanding the exponent, and
then to consider the properties of the power series expansion of the exponential
function).

16.5 Solutions
41
17 Naive Bayes

17.1 Why Naive Bayes?

Naive Bayes is one of the simplest density estimation methods from which we can
form one of the standard classification methods in machine learning. Its fame is
partly due to the following properties:
• Very easy to program and intuitive
• Fast to train and to use as a classifier
• Very easy to deal with missing attributes
• Very popular in fields such as computational linguistics/NLP
Despite the simplicity of Naive Bayes, there are some pitfalls that need to be
avoided, as we will describe. The pitfalls usually made are due to a poor un-
derstanding of the central assumption behind Naive Bayes, namely conditional
independence.

17.2 Understanding Conditional Independence

EasySell.com considers that its customers conveniently fall into two groups – the
‘young’ or ‘old’. Based on only this information, they build general customer
profiles for product preferences. Easysell.com assumes that, given the knowledge
that a customer is either ‘young’ or ‘old’, this is sufficient to determine whether
or not a customer will like a product, independent of their likes or dislikes for any
other products. Thus, given that a customer is ‘young’, she has a 95% chance
to like Radio1, a 5% chance to like Radio2, a 2% chance to like Radio3 and a
20% chance to like Radio4. Similarly, they model that an ‘old’ customer has a 3%
chance to like Radio1, an 82% chance to like Radio2, a 34% chance to like Radio3
and a 92% chance to like Radio4. Mathematically, we would write

p(R1, R2, R3, R4|age) = p(R1|age)p(R2|age)p(R3|age)p(R4|age)

where each of the variables R1, R2, R3, R4 can take the values either ‘like’ or
‘dislike’, and the ‘age’ variable can take the value either ‘young’ or ‘old’. Thus the
information about the age of the customer is so powerful that this determines the
individual product preferences without needing to know anything else. This kind
of assumption is indeed rather ‘naive’, but can lead to surprisingly good results.
In this chapter, we will take the conditioning variable to represent the class of the
datapoint x. Coupled then with a suitable choice for the conditional distribution
p(xi |c), we can then use Bayes rule to form a classifier. We can generalise the
situation of two variables to a conditional independence assumption for a set of
variables x1 , . . . , xN , conditional on another variable c:
N
Y
p(x|c) = p(xi |c) (17.2.1)
i=1

182
 183

x1 x2 x3 x4 x5

See fig(17.2) for the graphical model. In this chapter, we will consider two cases
of different conditional distributions, one appropriate for discrete data and the
other for continuous data. Furthermore, we will demonstrate how to learn any
free parameters of these models.

17.3 Are they Scottish?

Consider the following vector of attributes:

(likes shortbread, likes lager, drinks whiskey,eats porridge,

watched England play football)T (17.3.1)

A vector x = (1, 0, 1, 1, 0)T would describe that a person likes shortbread, does
not like lager, drinks whiskey, eats porridge, and has not watched England play
football. Together with each vector xµ , there is a class label describing the na-
tionality of the person: Scottish, or English. We wish to classify a new vector
x = (1, 0, 1, 1, 0)T as either Scottish(S) or English(E). We can use Bayes rule to
calculate the probability that x is Scottish or English:

p(x|S)p(S)
p(S|x) =
p(x)

p(x|E)p(E)
p(E|x) =
p(x)
Since we must have p(S|x) + p(E|x) = 1, we could also write

p(x|S)p(S)
p(S|x) =
p(x|S)p(S) + p(x|E)p(E)

It is straightforward to show that the “prior” class probability p(S) is simply given
by the fraction of people in the database that are Scottish, and similarly p(E) is
given as the fraction of people in the database that are English. What about
p(x|S)? This is where our density model for x comes in. In the previous chapter,
we looked at a using a Gaussian distribution. Here we will make a different, very
strong conditional independence assumption:

p(x|S) = p(x1 |S)p(x2 |S) . . . p(x5 |S)

What this assumption means is that knowing whether or not someone is Scottish,
we don’t need to know anything else to calculate the probability of their likes and
dislikes.
Matlab code to implement Naive Bayes on a small dataset is written below, where
each row of the datasets represents a (row) vector of attributes of the form equation
(17.3.1).
 184

% Naive Bayes using Bernoulli Distribution

xE=[0 1 1 1 0 0; % english
0 0 1 1 1 0;
1 1 0 0 0 0;
1 1 0 0 0 1;
1 0 1 0 1 0];

xS=[1 1 1 1 1 1 1; % scottish
0 1 1 1 1 0 0;
0 0 1 0 0 1 1;
1 0 1 1 1 1 0;
1 1 0 0 1 0 0];

pE = size(xE,2)/(size(xE,2) + size(xS,2)); pS =1-pE; % ML class priors pE = p(c=E), pS=p(c=S)

mE = mean(xE’)’; % ML estimates of p(x=1|c=E)

mS = mean(xS’)’; % ML estimates of p(x=1|c=S)

x=[1 0 1 1 0]’; % test point

npE = pE*prod(mE.^x.*(1-mE).^(1-x)); % p(x,c=E)

npS = pS*prod(mS.^x.*(1-mS).^(1-x)); % p(x,c=S)

pxE = npE/(npE+npS) % probability that x is english

Based on the training data in the code above, we have the following : p(x1 = 1|E) =
1/2,p(x2 = 1|E) = 1/2,p(x3 = 1|E) = 1/3,p(x4 = 1|E) = 1/2,p(x5 = 1|E) = 1/2,
p(x1 = 1|S) = 1,p(x2 = 1|S) = 4/7,p(x3 = 1|S) = 3/7,p(x4 = 1|S) = 5/7,p(x5 =
1|S) = 3/7 and the prior probabilities are p(S) = 7/13 and p(E) = 6/13.
For x∗ = (1, 0, 1, 1, 0)T , we get
1 × 73 × 37 × 75 × 47 × 13
7
p(S|x∗) = 3 3 (17.3.2)
1× 7 × 7 × × 7 × 13 + 2 × 2 × 31 ×
5
7
4 7 1 1 1
2 × 1
2 × 6
13

which is 0.8076. Since this is greater than 0.5, we would classify this person as
being Scottish.

17.3.1 Further Issues

Consider trying to classify the vector x = (0, 1, 1, 1, 1)T . In the training data, all
Scottish people say they like shortbread. This means that p(x, S) = 0, and hence
that p(S|x) = 0. This demonstrates a difficulty with sparse data – very extreme
class probabilities can be made. One way to ameliorate this situation is to smooth
the probabilities in some way, for example by adding a certain small number M to
the frequency counts of each class. This ensures that there are no zero probabilities
in the model:

number of times xi = 1 for class c + M

p(xi = 1|c) = (17.3.3)
number of times xi = 1 for class c + M + number of times xi = 0 for class c + M
 185

Continuous Data

Fitting continuous data is also straightforward using Naive Bayes. For example, if
we were to model each attributes distribution as a Gaussian, p(xi |c) = N (µi , σi ),
this would be exactly equivalent to using a conditional Gaussian density estimator
with a diagonal covariance matrix.

17.3.2 Text Classification

Naive Bayes has been often applied to classify documents in classes. We will outline
here how this is done. Refer to a computational linguistics course for details of
how exactly to do this.
Consider a set of documents about politics, and a set about sport. We search
Bag of words through all documents to find the, say 100 most commonly occuring words. Each
document is then represented by a 100 dimensional vector representing the number
of times that each of the words occurs in that document – the so called ‘bag of
words’ representation (this is clearly a very crude assumption since it does not take
into account the order of the words). We then fit a Naive Bayes model by fitting
a distribution of the number of occurrences of each word for all the documents of,
first sport, and then politics.
The reason Naive Bayes may be able to classify documents reasonably well in this
way is that the conditional independence assumption is not so silly : if we know
people are talking about politics, this perhaps is almost sufficient information to
specify what kinds of other words they will be using – we don’t need to know
anything else. (Of course, if you want ultimately a more powerful text classifier,
you need to relax this assumption).

17.4 Pitfalls with Naive Bayes

So far we have described how to implement Naive Bayes for the case of binary
attributes and also for the case of Gaussian continuous attributes. However, very
often, the software that people seem to commonly use requires that the data is in
the form of binary attributes. It is in the transformation of non-binary data to a
binary form that a common mistake occurs.
Consider the following attribute : age. In a survey, a person’s age is marked down
using the variable a ∈ 1, 2, 3. a = 1 means the person is between 0 and 10 years
old, a = 2 means the person is between 10 and 20 years old, a = 3 means the
person is older than 20. Perhaps there would be other attributes for the data, so
that each data entry is a vector of two variables (a, b)T .
1-of-M encoding One way to transform the variable a into a binary representation would be to
use three binary variables (a1 , a2 , a3 ). Thus, (1, 0, 0) represents a = 1, (0, 1, 0)
represents a = 2 and (0, 0, 1) represents a = 3. This is called 1 − of − M coding
since only 1 of the binary variables is active in encoding the M states. The problem
here is that this encoding, by construction, means that the variables a1 , a2 , a3 are
dependent – for example, if we know that a1 = 1, we know that a2 = 0 and
a3 = 0. Regardless of any possible conditioning, these variables will always remain
completely dependent, contrary to the assumption of Naive Bayes. This mistake,
however, is widespread – please help preserve a little of my sanity by not making
the same error. The correct approach is to simply use variables with many states
 186

– the multinomial rather than binomial distribution. This is straightforward and

left as an exercise for the interested reader.

17.5 Estimation using Maximum Likelihood : Bernoulli Process

Here we formally derive how to learn the parameters in a Naive Bayes model from
data. The results are intuitive, and indeed, we have already made use of them in
the previous sections. Additionally, some light can be cast on the nature of the
decision boundary (at least for the case of binary attributes).
Consider a dataset X = {xµ , µ = 1, . . . , P } of binary attributes. That is xµi ∈
{0, 1}. Each datapoint xµ has an associated class label cµ . Based upon the
class label, we can split the inputs into those that belong to each class : X c =
{x|x is in class c}. We will consider here only the case of two classes (this is called
a Bernoulli process – the case of more classes is also straightforward and called the
multinomial process). Let the number of datapoints from class c = 0 be n0 and
the number from class c = 1 be n1 .

For each class of the two classes, we then need to estimate the values p(xi = 1|c) ≡
θic . (The other probability, p(xi = 0|c) is simply given from the normalisation
requirement, p(xi = 0|c) = 1 − p(xi = 1|c) = 1 − θic ). Using the standard assump-
tion that the data is generated identically and independently, the likelihood of the
model generating the dataset X c (the data X belonging to class c) is
Y
p(X c ) = p(xµ |c) (17.5.1)
µ from class c

Using our conditional independence assumption

Y Y
p(x|c) = p(xi |c) = (θic )xi (1 − θic )1−xi (17.5.2)
i i

(remember that in each term in the above expression, xi is either 0 or 1 and

hence, for each i term in the product, only one of the two factors will contribute,
contributing a factor θic if xi = 1 and 1 − θic if xi = 0). Putting this all together,
we can find the log likelihood
X µ
L(θc ) = xi log θic + (1 − xµi ) log(1 − θic ) (17.5.3)
i,µ

Optimising with respect to θic ≡ p(xi = 1|c) (differentiate with respect to θic and
equate to zero) gives

number of times xi = 1 for class c

p(xi = 1|c) = (17.5.4)
(number of times xi = 1 for class c) + (number of times xi = 0 for class c)

A similar Maximum Likelihood argument gives the intuitive result:

number of times class c occurs
p(c) = (17.5.5)
total number of data points
 187

17.5.1 Classification Boundary

If we just wish to find the most likely class for a new point x, we can compare the
log probabilities, classifying x∗ as class 1 if
log p(c = 1|x∗ ) > log p(c = 0|x∗ ) (17.5.6)
Using the definition of the classifier, this is equivalent to (since the normalisation
constant − log p(x∗ ) can be dropped from both sides)
X X
log p(x∗i |c = 1) + log p(c = 1) > log p(x∗i |c = 0) + log p(c = 0)
i i

Using the binary encoding xi ∈ {0, 1}, we classify x∗ as class 1 if

X X
x∗i log θi1 + (1 − x∗i ) log(1 − θi1 ) + log p(c = 1) > x∗i log θi0 + (1 − x∗i ) log(1 − θi0 ) + log p(c = 0)
i i

P
This decision rule can be expressed in the form: classify x∗ as class 1 if i wi x∗i +
a > 0 for some suitable choice of weights wi and constant a (the reader is invited
to find the explicit values of these weights). The interpretation of this is that w
specifies a hyperplane in the x space and x∗ is classified as a 1 if it lies on one side
of the hyperplane. We shall talk about other such “linear” classifiers in a later
chapter.

17.6 Naive Bayes : The multinomial case

Consider a (visible) variable xi that can be in a discrete state s ∈ {1, . . . S} (the
generalisation to having a different number of states for a different i is straight-
forward). Consider the fitting a model to the data from class c. Under the naive
Bayes assumption, a discrete valued vector x will have probability
Y
p(x|c) = p(xi |c)
i

subject to the normalisation constraint

X
p(xi = s|c) = 1
s

For a set of data vectors xµ ,µ = 1, . . . P , belonging to class c assuming iid, the

likelihood of the data from class c is
P
Y P Y
Y N Y
S Y
C
µ µ
µ µ
p(x |c ) = p(xi = s|c)I[xi =s]I[c =c]

µ=1 µ=1 i=1 s=1 c=1

or
P X
X N X
S X
C
L= I[xµi = s]I[cµ = c] log p(xi = s|c)
µ=1 i=1 s=1 c=1

The parameters are p(xi = s|c). If we optimize this with respect to these para-
meters, using a lagrange multiplier to ensure normalisation (one for each of the
outputs i):
P X
X N X
S X
C C X
X N S
X
L= I[xµi = s]I[cµ = c] log p(xi = s|c) + λci p(xi = s|c)
µ=1 i=1 s=1 c=1 c=1 i=1 s=1
 188

Differentiating this with respect to p(xi = s|c), we get

P
X I[xµi = s]I[cµ = c]
= λci
µ=1
p(xi = s|c)

Hence, by normalisation,
P
µ I[xµi = s]I[cµ = c]
p(xi = s|c) = P P
I[xµi = s′ ]I[cµ′ = c]
′
s′ µ′

In words, this means simply that the optimal ML setting for the parameter p(xi =
s|c) is simply (for the class c data), the relative number of times that attribute i
is in state s. (Analogous to the binary example before).

17.6.1 Dirichlet Prior

The previous ML derivation suffers if there are no cases in which a variable is in

state s in the training data. In that case, the probabilities become certainties, and
classification can be overconfident. A simple approach around this is to consider
putting priors on the probabilities p(xi = s|c). A natural prior to use is a Dirichlet
distribution (see appendix). First, though, let’s see how we would use, in general,
a prior distribution in the classification of a novel point x∗ .
let D denote the training data (xµ , cµ ), µ = 1, . . . , P . We will have, for each i, a
distribution p(xi |c). Since each xi can take one of S states (the derivation below
is general enough to include the case that each i can have a different number of
states), we need to specify a probability for each of these states. This is described
by the S dimensional vector αi (c), so that p(xi = s|c) ≡ αis (c).
We would like to calculate
Z YZ
∗ p(x∗ , c, D) ∗
p(c|x , D) = ∝ p(x , c, D, α) ∝ p(x∗i |αi (c))p(α(c)|D)
p(x∗ , D) α i αi (c)

Let’s look at the posterior distribution

p(α(c)|D) ∝p(c)p(D|α(c))p(α(c)) (17.6.1)

! !
Y Y
= p(xµi |αi (cµ )) p(αi (c)) (17.6.2)
i µ

It’s clear therefore that the posterior factorises

Y
p(α(c)|D) = p(αi (c)|D)
i

where
Y
p(αi (c)|D) ∝ p(αi (c)) p(xµi |αi (c))
µ:cµ =c

When the prior is a Dirichlet distribution,

p(αi (c)) = Dirichlet(αi (c)|ui (c))

 189

the posterior is also a Dirichlet distribution (since the Dirichlet distribution is

conjugate to the multinomial distribution),

p(αi (c)|D) = Dirichlet(αi (c)|ûi (c))

where the vector ûi (c) has components

X
[ûi (c)]s = uis (c) + I[xµi = s]
µ:cµ =c

Here the parameter ui (c) describes the form of the prior. If we take this to be the
unit vector, the distribution on the (infinite set of) distributions p(αi (c)) is flat.
This is not an unreasonable assumption to make. This then becomes
Y Z(u∗i (c))
p(c|x∗ , D) ∝ p(c)
i
Z(ûi (c))

where

u∗i i ∗
s (c) = ûs (c) + I[xi = s]

and Z(u) is the normalisation constant of the distribution Dirichlet(α|u) Repeating

the previous analysis on the ‘Are they Scottish?’ data, the probability under a
uniform Dirichlet prior for all the tables, gives a value of 0.236 for the probability
that (1, 0, 1, 1, 0) is Scottish, compared with a value of 0.192 under the standard
Naive Bayes assumption.
An advantage of this Dirichlet prior framework is that it works also when there
are zero counts in the data.
CHECK: is there a simpler way to write the above ratio using the relationships
of the Gamma(x + 1) = xGamma(x)? Maybe this gives an equivalent form to
smoothing the counts?

17.7 Problems
Exercise 42 A local supermarket specializing in breakfast cereals decides to ana-
lyze the buying patterns of its customers.
They make a small survey asking 6 randomly chosen people which of the breakfast
cereals (Cornflakes, Frosties, Sugar Puffs, Branflakes) they like, and also asking
for their age (older or younger than 60 years). Each respondent provides a vector
with entries 1 or 0 corresponding to whether they like or dislike the cereal. Thus
a respondent with (1101) would like Cornflakes, Frosties and Branflakes, but not
Sugar Puffs.
The older than 60 years respondents provide the following data :

(1000), (1001), (1111), (0001)

For the younger than 60 years old respondents, the data is

(0110), (1110)

A novel customer comes into the supermarket and says she only likes Frosties and
Sugar Puffs. Using Naive Bayes trained with maximum likelihood, what is the
probability that she is younger than 60?
 190

Exercise 43 A psychologist does a small survey on ‘happiness’.

Each respondent provides a vector with entries 1 or 0 corresponding to whether
answer ‘yes’ to a question or ‘no’, respectively. The question vector has attributes

x = (rich, married, healthy)

In addition, each respondent gives a value c = 1 if they are content with their
lifestyle, and c = 0 if they are not.
Thus, a response (1, 0, 1) would indicate that the respondent was ‘rich’, ‘unmar-
ried’, ‘healthy’.

The following responses were obtained from people who claimed also to be ‘content’
:

(1, 1, 1), (0, 0, 1), (1, 1, 0), (1, 0, 1)

For the ‘not content’ respondents, the data is

(0, 0, 0), (1, 0, 0), (0, 0, 1), (0, 1, 0), (0, 0, 0)

Using Naive Bayes on this data, what is the probability that a person who is ‘not
rich’, ‘married’ and ‘healthy’ is ‘content’?
What is the probability that a person who is ‘not rich’ and ‘married’ is ‘content’?
(That is, we do not know whether or not they are ‘healthy’).
Consider the following vector of attributes :

x1 = 1 if customer is younger than 20 ; x1 = 0 otherwise

x2 = 1 if customer is between 20 and 30 years old ; x2 = 0 otherwise

x3 = 1 if customer is older than 30 ; x3 = 0 otherwise
x4 = 1 if customer walks to work ; x4 = 0 otherwise

Each vector of attributes has an associated class label “rich” or “poor”.

Point out any potential difficulties with using your previously described approach
to training using Naive Bayes. Hence describe how to extend your previous Naive
Bayes method to deal with this dataset. Describe in detail how maximum likelihood
could be used to train this model.

17.8 Solutions
42 Looking at the data, the estimates using maximum likelihood are

p(C = 1|Y oung) = 0.5, p(F = 1|Y oung) = 1, p(SP = 1|Y oung) = 1, p(B = 1|Y oung) = 0

and

p(C = 1|Old) = 0.75, p(F = 1|Old) = 0.25, p(Sp = 1|Old) = 0.25, p(B = 1|Old) = 0.75
 191

and p(Y oung) = 2/6 and p(Old) = 4/6. Plugging this into Bayes formula, we get

p(Y oung|C = 0, F = 1, SP = 1, B = 0) ∝ 0.5 ∗ 1 ∗ 1 ∗ 1/6

p(Old|C = 0, F = 1, SP = 1, B = 0) ∝ 0.25 ∗ 0.25 ∗ 0.25 ∗ 0.25 ∗ 4/6

Using the fact that these probabilities sum to 1, this gives p(Y oung|C = 0, F =
1, SP = 1, B = 0) = 64/65
18 Mixture Models : Discrete Hidden Variables

18.1 Mixture Models

Figure 18.1: A mixture model has a trivial graphical representation as a DAG with
a single hidden node, which can be in one of H states, i = 1 . . . H.

A common assumption is that data lies in ‘clusters’. For example, if we have

examples of handwritten digits, then the training vectors which represent the digit
‘1’ will typically be much closer to each other (in the Euclidean sense) than those
vectors which represent the digit ‘8’. This suggests that, in this case, it is natural
to expect the presence of two main clusters in the total training data – that is,
when all the data, regardless of whether it is classed as a ‘1’ or an ‘8’ is examined
together.
In a probabilistic sense, a cluster equates to a mixture component, so that the data
is represented by a summation (mixture) of models, each model being responsible
for an individual ‘cluster’ :
H
X
p(x|Θ) = p (x|θi , i) p (i) (18.1.1)
i=1

Where i represents mixture component i and Θ = {θ1 . . . θh } are the parameters

that determine the mixture distributions. The general form of a mixture model is
given in fig(18.1).
Whilst there are some cases, such as the Gaussian Mixture Model discussed below,
where there is a clear visual interpretation of the meaning of ‘cluster’, the reader
should bear in mind that the intuitive meaning of ‘cluster’ is based on datapoints
being ‘close’ in some sense to each other. The tremendous advantage of the reali-
sation that mixture models generalise the idea of modelling clusters, is that we are
freed from the conceptual constraint of distances in some Euclidean type sense.
Instead, two datapoints become ‘close’ if they are both likely with respect to the
model for that ‘cluster’. Hence, we can immediately start to ‘cluster’ all kinds
of data – music, shopping purchases etc – things which do not necessarily have a
natural ‘distance’ measure.

Training Mixture Models

Although it is perfectly feasible to find appropriate MAP parameters by standard

optimization techniques, the variational learning approach is usually far superior.
According to the general theory, we need to consider the energy term:

192
 193

p(x|i=1) p(x|i=2) p(x|i=3)

Figure 18.2: It is clear that the black dots, which represent the one dimensional
data values are naturally clustered into three groups. Hence, a reasonable
P3 model
of this data would be p(x) = p(x|1)p(1)+p(x|2)p(2)+p(x|3)p(3)
P = i=1 p(x|i)p(i)
where p(x|i) is the model for the data in cluster i, and i p(i) = 1.

Figure 18.3: Gaussian Mixture Models place blobs of probability mass in the space.
Here we have 4 mixture components in a 2 dimensional space. Each mixture has
a different covariance matrix and mean.

X
q µ (i) log (p (xµ |θi , i) p (i)) (18.1.2)
i,µ

and maximise this with respect to the parameters θi , i = 1, . . . , H, and p(i), i =

1, . . . , H.
Considering the dependence of the above function on p(i), and including a nor-
malisation Lagrangian, we obtain
!
X X
µ
q (i) log p (i) + λ 1 − p(i)
i,µ i

Differentiating, and ensuring normalisation gives

X
pnew (i) ∝ q µ (i) (18.1.3)
µ

The parameters θi , i = 1, . . . , H are determined by

X
argmax q µ (i) log p (xµ |θi , i) (18.1.4)
θi ,i=1,...,H i,µ

Provided that the parameters are not shared by the mixture components, we have
simply, for each θi
X
θinew = argmax q µ (i) log p (xµ |θi , i)
θi µ

The choice for the variational distributions is user dependent. The optimal EM
setting is
q µ (i) = p(i|xµ , θi ) ∝ p (xµ |θi , i) p (i) (18.1.5)
 194

Equations (18.1.3,18.1.4,18.1.5) are repeated until convergence. The initialisation

of the parameters θi and mixture probabilities can severely affect the quality of the
solution found. If random initialisations are used, it is recommended to record the
value of the likelihood itself, to see which converged parameters have the higher
likelihood. In the case that the likelihood is difficult to compute exactly, and the
variational method is used to form a bound,
( )
X X X
µ µ µ µ
B(Θ) = − q (i) log q (i) + q (i) log (p(x |θi , i)p(i))
µ i i

then the bound serves as a way to assess which converged parameters are to be
preferred.

18.2 Gaussian Mixture Models

One commonly used form for the mixture components is to use Gaussian (normal)
distributions.
An n dimensional Gaussian distribution is specified as
 
1 1 T
p (x|m, S) = √ exp − (x − m) S −1 (x − m) (18.2.1)
det 2πS 2

where m is the mean and S is the covariance matrix. If we have a separate

Gaussian for each component, with no shared parameters, then θi = {mi , Si }. It
is customary to choose a number of mixture components that is smaller than the
number of datapoints.
Infinite Troubles A difficulty arises here with the application of ML to training Gaussian mixture
models. By taking the limit of an infinitely narrow covariance matrix, the con-
tribution to the likelihood from that datapoint becomes infinite. This says that,
according to the pure ML criterion, optimally, we should place infinitely narrow
Gaussians on some datapoints, and not worry about the rest. This is clearly an
undesirable solution to the problem, and arises because in this case, the ML solu-
tion does not constrain the parameters in a sensible way. This is a classic example
of when ML can lead to poor results. Exercise for the reader : why does this
problem not occur when we only fit one Gaussian (to more than one datapoint).

All computational methods which aim to fit mixtures of Gaussians using ML there-
fore either succeed by getting trapped in serendipitous local maxima, or by the ad
hoc addition of “extra constraints” on the width of the Gaussians. A more reason-
able approach is to incorporate such necessary assumptions on the widths of the
Gaussians in the prior beliefs. This has the advantage of transparency and clarity
in the line of thought. In this sense, therefore, we can use the MAP approach in
preference to the ML solution. In practice, however, it is more commonplace to
use a simple criterion which prevents the eigenvalues of the covariance matrices
from becoming too small. A Bayesian solution to this problem is possible and re-
quires a prior on covariance matrices. The natural prior in this case is the so-called
Wishart Distribution, which we shall discuss later.
 195

Finding the optimal mi

In the case of using Gaussian mixture components,

1 T 1
log p (xµ |θi , i) = − (xµ − mi ) Si−1 (xµ − mi ) − log det 2πSi (18.2.2)
2 2
We can then readily optimize the associated energy term
 
X
µ 1 µ T −1 µ 1
q (i) − (x − mi ) Si (x − mi ) − log det 2πSi (18.2.3)
µ
2 2

with respect to mi to obtain

P µ µ
µ q (i)x
mi = P µ (18.2.4)
µ q (i)

Finding the optimal Si

Optimising equation (18.2.3) with respect to Si is slightly more difficult.

Using trace (log A) = log det A, and ∂ log A = A−1 ∂A, where ∂ represents

a dif-
ferentiation operator, one can show that ∂ log det A = trace A−1 ∂A . Also, since
A−1 A = I, this gives ∂A−1 = A−1 ∂AA−1 . Using these results we get, for the
derivative wrt Si ,
 
X
µ 1 µ T −1 −1 µ 1 −1


q (i) (∆x ) Si ∂Si Si ∆x − trace Si ∂Si (18.2.5)
µ
2 2


where ∆xµ ≡ xµ − mi . Using aT Aa ≡ trace AaaT , we obtain
!
X  
T
trace Si−1 ∂Si Si−1 q µ (i) ∆xµ (∆xµ ) − Si (18.2.6)
µ

This derivative is clearly zero if

P µ µ µ T
µ q (i) (x − mi ) (x − mi )
Si = P µ (18.2.7)
µ q (i)

To get the mixing coefficients, use a Lagrange multiplier to give

P µ
µ q (i) 1 X µ
p(i) = P µ
= q (i) (18.2.8)
i,µ q (i) P µ

where P is the number of training examples. The EM choice for the variational
distributions is
µ
1
−mi )T Si−1 (xµ −mi )
q µ (i) ∝ p(i)e− 2 (x (18.2.9)

The above equations (18.2.4,18.2.7,18.2.8,18.2.9) are iterated until convergence. A

useful intialisation strategy is to set the covariances to be diagonal, with large
variance – this gives the components a chance to ‘sense’ where data lies. An
illustration of the performance of the algorithm is given in fig(18.4).
 196

(a) 1 iteration (b) 50 iterations

(c) 125 iterations (d) 150 iterations

Figure 18.4: Training a mixture of 10 Gaussians (a) If we start with large variances
for the Gaussians, even after one iteration, the Gaussians are centred close to the
mean of the data. (b) The Gaussians begin to separate (c) One by one, the
Gaussians move towards appropriate parts of the data (d) The final converged
solution. Here the Gaussians were constrained so that the variances could not go
below 0.01.

Symmetry Breaking An interesting observation about the performance of EM applied to mixture models
is that, initially, there appears as if little is happening, as each model jostles
with the others to try to explain the data. Eventually, some almost seemingly
random effect causes one model to break away from the jostling and explain data
close to that model. The origin of this jostling is an inherent symmetry in the
solution to this problem: it makes no difference to the likelihood if we relabel
what the components are called. This permutation symmetry causes the initial
confusion amongst the models as to who is going to explain which parts of the
data. Eventually, this symmetry is broken, and a local solution is found. This
can severely handicap the performance of EM when there are a large number of
models in the mixture. An heuristic is to therefore begin with a small number of
models, say two, for which symmetry breaking is less problematic. Once a local
broken solution has been found, then more models are included into the mixture,
initialised close to the currently found broken solutions. In this way, a hierarchical
breaking scheme is envisaged. Clearly, there is potentially some bias introduced in
this scheme as to the kinds of solutions found – however, this may be a small price
to pay in the light of waiting for a very long time as the models jostle unnecessarily.
Another popular method for initialisation is to center the means to those found
by the K-means algorithm – however, this itself requires a heuristic initialisation.

Parzen Estimator

The Parzen estimator is one of the simplest density estimators. The idea is simply
to put a Gaussian at each datapoint xk , k = 1 . . . P . Usually, this Gaussian is
chosen to be isotropic – that is, with covariance matrix Σ = σI, where σ is some
 197

pre-chosen value. In general, therefore, the density estimator is

P
1 X 1
e− 2 (x−x )Σ (x−x )
1 µ −1 µ
p(x) = √ (18.2.10)
P µ=1 det 2πΣ

Whilst an intuitively reasonable thing to do, if one is working with large datasets in
high dimensional spaces, one needs to store all the datapoints in order to calculate
the density, which can be prohibitive.
Unless the widths of the Gaussians are chosen to be broad, only a small region of
the space is covered by each Gaussian bump. In high dimensional spaces therefore,
the Parzen estimator will only have appreciable density very close to the data or,
if the Gaussians are broad, the density will be underestimated close to the data.
 198

% demo for fitting mixture of isotropic Gaussians

%make an annulus of data :

l = 0.2; r1 = 0.5; for r = 1:50
rad = r1 + rand*l; theta = rand*2*pi; X(1,r) = rad*cos(theta); X(2,r) = rad*sin(theta);
end

h = 5; % number of mixtures
d = size(X,1); % dimension of the space
n = size(X,2); % number of training patterns
Smin = 0.001; % minimum variance of Gaussians

r = randperm(n); M = X(:,r(1:h)); % initialise the centres to random datapoints

S = 100*ones(1,h); % initialise the variances to be large
P = ones(1,h)./h; % intialise the component probilities to be uniform

for its = 1:150 % number of iterations

for i = 1:h
for k = 1:n
v = X(:,k) - M(:,i);
Q(k,i) = exp(-0.5*(v’*v)/S(i)).*P(i)./sqrt((S(i))^d);
end
end
su = sum(Q,2);
for k =1:n
Q(k,:) = Q(k,:)./su(k); % responsibilities p(i|x^n)
end

for i = 1:h % now get the new parameters for each component
N(i) = sum(Q(:,i));
Mnew(:,i) = X*Q(:,i)./N(i);
Snew(i) = (1/d)*sum( (X - repmat(Mnew(:,i),1,n)).^2 )*Q(:,i)./N(i);
if Snew(i) < Smin % don’t decrease the variance below Smin
Snew(i) = Smin;
end
end

Pnew = N; Pnew = Pnew./sum(Pnew);

S = Snew; M = Mnew; P = Pnew; % update the parameters
end
 199

18.3 K Means
A non-probabilistic limit of fitting Gaussian mixtures to data is given by the K
means algorithm, in which we simply represent an original set of P datapoints by
K points.

18.3.1 The algorithm

1. Initialise the centres µi to K randomly chosen datapoints.

2. For each cluster mean, j, find all the x for which cluster j is the nearest
cluster. Call this set of points Sj . Let Nj be the number of datapoints in
set Sj .
3. Assign
1 X
µj = x (18.3.1)
Nj
x∈Sj

We then iterate steps 2 and 3 above until some convergence criterion.

The code below implements this algorithm.

% demo for K Means

x = [randn(2,50) 5+randn(2,50) (repmat([-4 4]’,1,50)+randn(2,50))]; % 150 2-dim datapoints

K = 3; % number of clusters
r = randperm(size(x,2));
m(:,1:K) = x(:,r(1:K)); % initialise the clusters to K randomly chosen datapoints
mold =m;

for its = 1: 100 % maximum number of iterations

for p = 1:size(x,2) % calculate the distances (this could be vectorised)

for k = 1:K
v = x(:,p) - m(:,k); d(k,p) = v’*v;
end
end

[a,b]=min(d); % find the nearest centres

for k = 1:K
if length(find(b==k))>0
m(:,k) = mean(x(:,find(b==k))’)’;
end
end

if mean(sum( (m-mold).^2)) < 0.001; break; end; mold =m; % termination criterion
end

cla; plot(x(1,:),x(2,:),’.’); hold on;

plot(m(1,:),m(2,:),’rx’,’markersize’,15);

An example is given in fig(18.5) in which we represented 150 datapoints using 3

clusters.
 200

10

−2

−4
−6 −4 −2 0 2 4 6 8

Figure 18.5: Result of fitting K = 3 means to 150 two dimensional datapoints.

The means are plotted as crosses.

Note that the Kmeans algorithm can be dervived as the limit σ → 0 for fitting
isotropic Gaussian mixture components.

18.3.2 Uses of K Means

The K means algorithm, despite its simplicity is very useful. Firstly, it converges
extremely quickly and often gives a reasonable clustering of the data, provided that
the centres are initialised reasonably (using the above procedure for example). We
can use the centres we found as positions in which to place basis function centres
in the linear parametric models chapter.

18.4 Classification using Mixture Models

One popular use of mixture models is in classification. Consider the case in which
we have two classes, 1 and 2. We can fit a Gaussian Mixture model to each class.
That is, we could fit a mixture model to the data from class 1 :
K
X
p(x|c = 1) = p(x|k, c = 1)p(k|c = 1) (18.4.1)
k=1

and a mixture model to the data from class 2. (One could use a different number
of mixture components for the different classes, although in practice, one might
need to avoid overfitting one class more than the other. Using the same number
of mixture components for both classes avoids this problem.)
K
X
p(x|c = 2) = p(x|k, c = 2)p(k|c = 2) (18.4.2)
k=1

So that each class has its own set of mixture model parameters. We can then form
a classifier by using Bayes rule :

p(x|c = i)p(c = i)
p(c = i|x) = (18.4.3)
p(x)
 201

v1 v2 v3

Figure 18.6: A mixture model has a trivial graphical representation as a DAG with
a single hidden node, which can be in and one of H states, i = 1 . . . H.

Only the numerator is important in determining the classification since the de-
nominator is the same for the case of p(c = 2|x). This is a more powerful approach
than our original approach in which we fitted a single Gaussian to each digit class.
Using more Gaussians enables us to get a better model for how the data in each
class is distributed and this will usually result in a better classifier.

18.5 Mixture of Multi-nomials

A company sends out a questionnaire, which contains a set of v1 , . . . , vn ‘yes/no’
questions. P customers send back their questionnaires, v 1 , . . . , v P , and the com-
pany wishes to perform an analysis to find what kinds of customers it has. Let us
assume that the company has good reason to suspect that there are H essential
type of customer and, given that we know which type we are dealing with, the
Qnwell defined in the sense that p(v1 , . . . , vn |h)
profile of their responses will be quite
may be assumed to be factorised, i=1 p(vi |h). (This is the same assumption as
the Naive Bayes classifier – here the difference ultimately is that we will not have
any training labels for the class of the data).
A model that captures the above situation would be a mixture of Binomials
H
X n
Y
p(v1 , . . . , vn ) = p(h) p(vi |h)
h=1 i=1

where each term p(vi |h) is a Binomial distribution – that is, there are two states
vi ∈ {0, 1}. The generalisation to many states is straightforward.

EM training

In order to train the above model, we can use the EM algorithm, since we have a
hidden variable. Formally, we can figure out the algorithm by, as usual, writing
down the energy:
X XX X
hlog p(v1µ , v2µ , v3µ , h)iqµ (h) = hlog p(viµ |h)iqµ (h) + hlog p(h)iqµ (h)
µ µ i µ

and then performing the maximisation over the table entries. However, from our
general intuition, we may immediately jump to the results:
X µ
p(vi = 1|h = j) ∝ I[vi = 1]q µ (h = j)
µ
X
p(h = j) ∝ q µ (h = j)
µ
Y
q µ (h = j) ∝ p(v µ |h = j)p(h = j) ∝ p(h = j) p(viµ |h = j)
i
 202

10

10 20 30 40 50 60

Figure 18.7: Data from questionnaire responses. There are 10 questions, and
60 people responded. White denotes a ‘yes’ and black denotes ‘no’. Gray denotes
that the absence of a response (missing data). This training data was generated by
three component Binomial mixture. Missing data was then simulated by randomly
removing values from the dataset.

These equations are iterated until convergence. Code that implements the above
method is provided later in this chapter.
One of the pleasing aspects of this model is that if one of the attribute values
is missing in the data set, the only modification to the algorithm is to drop the
corresponding factor p(viµ |h) from the algorithm. The verification that this is is a
valid thing to do is left to the reader.

Example : Questionnaire

Data from a questionnaire is presented below in fig(18.7). The data has a great
number of missing values. We have reason to believe that there are three kinds of
respondents.
Running the EM algorithm on this data, with random initial values for the tables,
gives an evolution for the lower bound on the likelihood as presented in fig(18.8a).
The EM algorithm finds a good solution. The 3 hidden states probabilities learned
are p(h = 1) ≈ 1/3,p(h = 2) ≈ 1/3,p(h = 3) ≈ 1/3, which is in rough agreement
with the data generating process. The solution is permuted, but otherwise fine,
and the three basic kinds of respondents have been well identified. Note how
difficult this problem is to solve by visual inspection of the data fig(18.7).
Code that implements this problem is given at the end of this chapter.

Mixtures of HMM
A useful way to cluster temporal sequences is to use a mixture of HMMs. This
can be trained in the usual EM way, and good for clustering temporal sequences :
BioInformatics, Music etc.
 203
−100
1 1

−110 2 2

3 3
−120
4 4

−130 5 5

6 6
−140

7 7

−150
8 8

−160 9 9

10 10
−170
0 5 10 15 20 25 30 35 40 45 50 0.5 1 1.5 2 2.5 3 3.5 0.5 1 1.5 2 2.5 3 3.5

(a) (b) “true” p(vi = 1|h) values. (c) learned p(vi = 1|h) values

Figure 18.8: (a) The evolution of the lower bound on the likelihood as we go
through eventually 50 iterations of EM. The different regimes initially in the evo-
lution are signatures of the symmetry breaking. (b) The ‘true’ value for the pa-
rameters p(vi = 1|h). Black corresponds to the value 0.95 and white to the value
0.1. (c) The solution p(vi = 1|h) found by the converged EM approach. (b) and
(c) are closely related, except for a trivial permutation of the hidden label.

The key intuition in this chapter is that clustering corresponds to using a mixture
model. These may be trained with EM, although some care is required with
initialisation and symmetry breaking.
ANSWER: It’s true that the likelihood will be higher for the more complex model.
What we need to do is to introduce a prior over the parameters of the model, and
then integrate (as opposed to finding the set of parameters that maximises the
likelihood). This will provide the effective Occam’s factor that will penalise the
overly complex model. Need to do this. Note that this phenomenon only kicks in
when we integrate over the unknown model parameters. Essentially, it’s similar
to the fact that a more complex model will always have a lower training error –
however, what we need to measure is something more like the effective volume
of parameter space that has a low training error – this is given by the Bayesian
solution.
 204

function [pv,ph]=fitmixbern(v,nh,num_em_loops)

% MIXTURE OF BERNOULLI’s trained using EM

% missing data coded with -1

n=size(v,1); P=size(v,2);
pv = rand(n,nh); % random initialisation for the probs
ph = rand(1,nh); ph = ph./sum(ph);

for em = 1:num_em_loops

for mu = 1:P
for i = 1:nh
p(i,mu)=bern_prob(v(:,mu),pv(:,i))*ph(i); % p(i|vmu)*const.
end
p = p ./repmat(sum(p,1),nh,1); % p(i|vmu)
end

% update hidden probs

sp = sum(p,2);
for i = 1:nh
phnew(i) = sp(i);
end
phnew = phnew./sum(phnew); % hidden probabilities

% now update the tables p(v|i):

pv1 = zeros(n,nh); pv0 = zeros(n,nh);

for i = 1:nh
for datadim = 1:n
for mu = 1:P
pv1(datadim,i) = pv1(datadim,i) + (v(datadim,mu)==1).*p(i,mu);
pv0(datadim,i) = pv0(datadim,i) + (v(datadim,mu)==0).*p(i,mu);
end
end
end

for i = 1:nh
for datadim = 1:n
pvnew(datadim,i) = pv1(datadim,i)./(pv0(datadim,i)+pv1(datadim,i));
end
end

entropy = -sum(sum(p.*log(0.0000001+p))); energy=0;

for mu=1:P
for datadim=1:n
energy = energy + (v(datadim,mu)==1)*sum(log(0.0000001+pvnew(datadim,:)).*(p(:,mu)’));
energy = energy + (v(datadim,mu)==0)*sum(log(0.0000001 + 1-pvnew(datadim,:)).*(p(:,mu)’));
end
end

energy=energy+sum((log(0.0000001+phnew))*p);

bound(em) = entropy+energy; plot(bound); drawnow pv=pvnew;

ph=phnew;

end

function p = bern_prob(c,p)

p = prod(p(find(c==1)))*prod((1-p(find(c==0))));
 205

18.6 Problems
Exercise 44 If a and b are d × 1 column vectors and M is a d × d symmetric
matrix, show that aT M b = bT M a.

Exercise 45 Write the quadratic forms x21 −4x1 x2 +7x22 and (x1 +x2 )2 +(x3 +x4 )2
in the form xT Cx where C is a symmetric matrix.

Exercise 46 Consider data points generated from two different classes. Class 1
has the distribution P (x|C1 ) ∼ N (µ1 , σ 2 ) and class 2 has the distribution P (x|C2 ) ∼
N (µ2 , σ 2 ). The prior probabilities of each class are P (C1 ) = P (C2 ) = 1/2. Show
that the posterior probability P (C1 |x) is of the form
1
P (C1 |x) =
1 + exp −(ax + b)

and determine a and b in terms of µ1 , µ2 and σ 2 . The function f (z) = 1/(1 + e−z )
is known as the logistic function, and is a commonly-used transfer function in
artificial neural networks.

Exercise 47 The Poisson distribution is a discrete distribution on the non-negative

integers, with

e−λ λx
P (x) = x = 0, 1, 2, . . .
x!
You are given a sample of n observations x1 , . . . , xn drawn from this distribution.
Determine the maximum likelihood estimator of the Poisson parameter λ.

Exercise 48 Generate a sample of 100 points from a bivariate Gaussian distribu-

tion. You should use the matlab functions randn and chol (the Cholesky decompo-
sition) to help you. Hint: the Cholesky decomposition U of a matrix Σ is such that
U T U = Σ. If x is a random vector drawn from N (0, I), then y = U T x is a random
vector that has mean 0 and covariance E[yy T ] = E[U T xxT U ] = U T U = Σ.

18.7 Solutions
19 Factor Analysis and PPCA

Introduction
The notion of a continuous mixture is somewhat less clear than in the discrete case,
where each discrete mixture component has the intuitive meaning of representing
a “cluster”. Continuous mixture models generally do not have the same cluster
type intuition, since the hidden space will usually be connected. What happens in
the continuous case is that preferences for how the hidden variables are distributed
are expressed.
Such models have many extremely useful properties, and are widely applied. They
correspond to our belief that there is some continuous hidden process p(h), from
which (usually continuous) visible variables are observed, p(v|h). The literature in
this area is vast, and in this chapter we will consider only some of the most well
known examples, beginning with some relatively numerically tractable modelling
of subspaces.

19.1 Linear Subspace Methods

If data lies in a high dimensional space, we might hope that it lies close to a
hyperplane, as in fig(19.1). We then can approximate each data point by using
the vectors that span the hyperplane alone. I will sometimes refer to this small
set of vectors as the “basis” set. Strictly speaking this is not a basis for the
whole space, rather is is a ‘basis’ which approximately spans the space where
the data is concentrated. Effectively, we are trying to choose a more appropriate
low dimensional co-ordinate system that will approximately represent the data.
Mathematically, we write If the dimension of the data space, dim(x) = N , our
hope is that we can describe the data using only a small number M of vectors. If
we can do so, we can reduce greatly the information needed to accurately describe
the data.
In general, datapoints will not lie exactly on the hyperplane, and we may wish to
model such non-linearity as noise. Two well known models, Factor Analysis and
Principal Components Analysis, differ only in how this noise is dealt with.
Although not a particularly flexible density estimator, the clarity of the linear
relationship between the variables may prove insightful, and is a useful starting
point for more complex approaches.
The assumption is that the process generating the data is linear, dependent on a set
of latent or hidden variables, h. For consistency with notation in other chapters, we
will use h to represent the hidden or latent variables, and v to represent the visible
or observable variables. Then the assumption of a linear subspace corresponds to

v = Wh + b + ǫ (19.1.1)

where the noise ǫ is Gaussian distributed, ǫ ∼ N (0; Ψ), and the matrix W para-
meterises the linear mapping. The constant bias b essentially sets the origin of the

206
 207

x
x

x
x

Figure 19.1: In linear modelling of a subspace, we hope that data in the high di-
mensional space lies close to a hyperplane that can be spanned by a smaller number
of vectors. Here, each three-dimensional datapoint can be roughly described by
using only two components.

coordinate system. The essential difference between PCA and Factor Analysis is
in the choice of Ψ.

Factor Analysis
In factor analysis, one assumes that the covariance for the noise is diagonal Ψ =
diag (ψ1 , . . . , ψn ). This is a reasonable assumption if we believe that each com-
ponent of the data, xi has Gaussian measurement error, independent of the other
components. We see therefore that, given h, the data is assumed to be Gaussian
distributed with mean W h + b and covariance Ψ

p (v|h) ∝ e− 2 (v−W h−b) Ψ (x−W h−b)

1 T −1
(19.1.2)

To complete the model, we need to specify the hidden distribution p(h). Since
tractability is always a concern for continuous distributions, an expedient choice
is a Gaussian
T
p (h) ∝ e−h h/2
(19.1.3)

This therefore means that the coordinates h will be limited, and will most likely
be concentrated around values close to 0. If we were to sample from such a p(h)
and then draw a value for v using p(v|h), we would see that the v vectors that we
sample would be look like a saucer in the v space.

Indeed, in this case we can easily calculate the exact form of p(v):
Z
p (v) = p (v|h) p (h) dh (19.1.4)

Since v = W h + b + η and we know that p(h) is a zero mean Gaussian with unit
covariance, and η is zero mean with Covariance Ψ, v will be Gaussian distributed
with mean b and covariance matrix W W T + Ψ.

The form of the covariance matrix is interesting and tells us some thing about the
solution: Since the matrix W only appears in the final model p(v) in the form
 208

h1 h2 h3

v1 v2 v3 v4 v5

Figure 19.2: Graphical representation of factor analysis for a model with 3 hidden
or latent variables, which generate the visible or output variable v = (v1 , . . . , v5 )T .

W W T + Ψ, an equivalent model is W R(W R)T + Ψ, where R is any orthogonal

matrix RRT = I. Hence, the solution space for W is not unique.

Warning! Since the so-called factor loadings W are equally likely as any rotated version of
them, one should be very careful about interpreting the coefficients of the W –
in particular, about attributing meaning to each of the values. Such practice is
commonplace in the social sciences and, in general, is very poor science.

Training FA using EM

We assume that we are given a set of data v µ , µ = 1, . . . , P , and wish to adjust

the parameters W , b and Ψ to maximise the likelihood of the observed data.

A natural way to train Factor Analysis, is to use our standard variational learning
framework. Of course, in one could also attempt to maximise the likelihood di-
rectly (and the likelihood is relatively simple to calculate here). However, as usual,
the variational procedure tends to converge rather more quickly, and is the one we
shall describe here.

As usual, we need to consider the energy which, neglecting constants is

X 1 T

P
E=− (v µ − W h − b) Ψ−1 (v µ − W h − b) − log det Ψ
µ
2 µ
q (h) 2

It is left as an exercise for the interested reader to show that the following con-
ditions hold at the maximum of the the energy. Maximising E with respect to b
gives
1 X µ 1 X
b= v −W hhiqµ (h)
P µ P µ

Maximising E with respect to W gives

W = AH −1

where
1 X µ T
A= c hhiqµ (h)
P µ

cµ = v µ − b
1 X
T
H= hh qµ (h)
P µ
 209

Finally
1 X D E
T
Ψ= diag (cµ − W h) (cµ − W h)
P µ qµ (h)
( )
1 X µ µ T T T
= diag c (c ) − 2W A + W HW
P µ


The above recursions depend on the statistics hhiqµ (h) and hhT qµ (h) . Using the
EM optimal choice

q µ (h) ∝ p(v µ |h)p(h)

which is a Gaussian with covariance

−1
Σ = I + W T Ψ−1 W

and mean

−1
mµ = hhiqµ (h) = I + W T Ψ−1 W W T Ψ−1 cµ
From which
1 X µ µ T
H =Σ+ m (m )
P µ

The above equations then define recursions in the usual EM manner. Unfortu-
nately, the lack of a closed form solution to these equations means that FA is less
widely used than the simpler PCA (and its probabilistic variant).
A nice feature of FA is that one can perform the calculations on very high dimen-
sional data without difficulty. (In the standard PCA this is an issue, although
these problems can be avoided – see the text).
Also, unlike in PCA, the matrix W that is learned need not be orthogonal.

What about a correlated p(h)?

A seemingly obvious extension of FA analysis is to consider dependent or, equiva-

lently, in the Gaussian case, a correlated hidden distribution

p(h) = N 0, ΣH )
Does this really improve the representative power of the model? For notational
simplicity, let’s consider

v = Wh + ǫ

where ǫ ∼ N (0, σ 2 I). Then it is clear that

v ∼ N (0, W ΣH W T + σ 2 I)

which, may be written in the reparameterised form

v ∼ N (0, W ′ W ′T + σ 2 I)
1
where W ′ ≡ W ΣH 2
. Hence, there is nothing to be gained from using a correlated
Gaussian prior p(h).
 210

Figure 19.3: A comparison of factor analysis and PCA. The underlying data gen-
erating process is y = x + ǫ, where ǫ is Gaussian noise of standard deviation σ. In
the plots from left to right, σ takes the values 0.5, 1.2, 2, 3, 4. The FA solution is
given by the solid arrow, and the PCA solution by the dashed arrow. The correct
direction is given by the solid line. Note how the PCA solution “rotates” upwards
as the noise level increases, whereas the FA solution remains a better estimate of
the underlying correct direction.

Probabilistic Principal Components Analysis

(See also the section on Linear Dimension reduction for more details). PPCA is
a special case of factor analysis in which the noise term Ψ is isotropic,Ψ = σ 2 I.
In this case, we can calculate the ML solution exactly. The optimal matrix W
(so-called factor loadings), for a H dimensional hidden space and a V dimensional
visible space, is given by

1
W = UH λH − σ 2 I 2 R (19.1.5)

where the H column vectors in UH are the first H eigenvectors of the sample
covariance matrix S,
P
1 X µ T
S= (v − m) (v µ − m) (19.1.6)
N µ=1
P
where m is the sample mean µ v µ /P . λH is a diagonal matrix containing the
corresponding eigenvalues of S. R is an arbitrary orthogonal matrix (representing
an arbitrary rotation). For this choice of W, the optimal ML noise is given by
V
1 X
σ2 = λj (19.1.7)
V −H
j=H+1

where λj is the jth eigenvalues of S. This has the interpretation as the variance
lost in the projection, averaged over the lost dimensions.
 211

This means that we can rapidly find a ML linear subspace fit based on the eigen-
decomposition of the sample covariance matrix and sample mean.
An advantage of a proper probabilistic approach to PCA is that one can then,
in a principled manner, for example, contemplate discrete mixtures of Principal
Component Analysers, or indeed, a mixture of different kinds of models. Without
a probabilistic framework, it is difficult to justify how a set models should be
combined.

Standard Principal Component Analysis

There are several ways to understand PCA. However, in the current context, PCA
is defined as the limit of PPCA in which σ → 0 and R = I. That is, the mapping
from the latent space to the data space is deterministic. In this case, the columns
of W are given by simply the eigenvectors of the sample covariance matrix, scaled
by the square root of their corresponding eigenvalues.

Very High Dimensional Data

You might be wondering how it is possible to perform PCA on extremely high

dimensional data. For example, if we have 500 images each of 1000 × 1000 = 106
pixels, the covariance matrix will be 106 × 106 dimensional – well beyond the
storage capacities of many computers.
One approach around this difficulty is to perform the calculations in a lower di-
mensional space. Note that there can only be at most P non-zero eigenvalues.
Using X to denote the (zero mean) data and E the matrix of eigenvectors – this
is non-square since there will be fewer eigenvalues than dimensions. We write the
eigenvalues as a diagonal matrix Λ. The eigenvalue requirement is

XXT E = EΛ (19.1.8)
XT XXT E = XT EΛ (19.1.9)
T
X XẼ = ẼΛ (19.1.10)

where we defined Ẽ = XT E. The last line above represents the eigenvector equation
for XT X. This is a matrix of dimensions P × P – in the above example, a 500 × 500
matrix as opposed to a 106 × 106 matrix previously. We then can calculate the
eigenvectors Ẽ and eigenvalues Λ of this matrix more easily. Once found, we then
use

E = XẼΛ−1 (19.1.11)

19.2 A Toy Comparision of FA and PPCA

We trained both PPCA and FA to try to model handwritten digits of the number 7.
From a database of 100 such images, we fitted both PPCA and FA (100 iterations
of EM) using 5 hidden units. The learned values for these models are in fig(19.4).
To get a feeling for how well each of these models the data, we drew 25 samples
from each model, as given in fig(??) and fig(19.5). In FA, clearly, the individual
noise on each visible variable enables a cleaner representation of the regions of zero
variance, compared to the PPCA approach. However, on the whole, despite the
FA model being in principal a more powerful model, it does not here constitute a
 212

dramatic improvement over the PCA model.

Certainly one advantage of these probabilistic approaches is that they may now be
used in discrete mixture models in a principled way, and this can indeed improve
performance considerably.
FA bias FA FA FA FA FA

PCA mean PCA PCA PCA PCA PCA

Figure 19.4: For a 5 hidden unit model, here are plotted the results of training
PPCA and FA on 100 examples of the handwritten digit seven. Along with the
PPCA mean and FA bias, the 5 columns of W are plotted for FA, and the 5 largest
eigenvectors from PPCA are plotted.

function [W,Psi,b]=fa(v,H,num_em_loops)

% Factor Analysis Training using EM

P = length(v); V = size(v{1},1);

b=randn(V,1); W = rand(V,H); Psi = rand(V,1);

for emloop =1:num_em_loops

Sigma = inv(eye(H)+W’*diag(1./Psi)*W);
mtot = zeros(H,H);

(a) Factor Analysis (b) PCA

Figure 19.5: (a) 25 samples from the learned FA model. (b) 25 samples from the
learned PPCA model.
 213

diagcont = zeros(V,1);
A =zeros(V,H);
btot = zeros(V,1);
for mu=1:P
c{mu} = v{mu}-b;
diagcont = diagcont + c{mu}.^2;
m{mu} = Sigma*W’*diag(1./Psi)*c{mu};
mtot = mtot + m{mu}*m{mu}’;
A = A + c{mu}*m{mu}’;
btot = btot + v{mu}-W*m{mu};
end
Hmat = Sigma + mtot./P;
A = A./P;
diagcont = diagcont./P;
diagWA = diag(W*A’);
Psi = diagcont -2*diagWA+diag(W*Hmat*W’);
b = btot./P;
W = A/Hmat;
end

19.3 Non-linear Subspace Methods

19.3.1 Non linear Factor Analysis

The idea is the same as in FA, except that the transformation from the latent
space to the data space is non-linear. That is
 
1 −1
p (x|h) ∝ exp − (x − u) Ψ (x − u) (19.3.1)
2

where u = φ (h) where φ(t) is, in general, a non-linear function of t. If we take the
same Gaussian prior as before, in general, we cannot calculate the integral over
the latent space analytically anymore.
This can be approximated by
L
1X

p (x) = p x|hl (19.3.2)
L
l=1

where we have sampled L latent points from the density p (h). This is straightfor-
ward to do in the case of using a Gaussian prior on h.
What this means is that the density model is therefore a mixture of Gaussians,
constrained somewhat through the non-linear function.
One approach is to parameterise the non-linearity as
X
φ(h) = wi φi (h) (19.3.3)
i

where the φi are fixed functions and the weights wi form the parameters of the
mapping. These parameters, along with the other parameters can then be found
using variational learning (EM).
 214

GTM

The Generative Topographic Mapping (GTM) is a special case of the non-linear

factor analysis. It is a density estimator, but is most appropriate for very low
(latent) dimensional representations of data, typically two or three. For this reason,
it is mostly used to visualise data, and we shall describe this in some detail later.

6
2.5

2
4
1.5
2 1

0.5
0
0
−2 −0.5

−1
−4
−1.5
−6 −2
3 3
2 2
3 3
1 2 1 2
0 1 0 1
−1 0 −1 0
−1 −1
−2 −2 −2 −2
−3 −3 −3 −3

(a) Latent Space (b) Data Space (c) Data Space

Figure 19.6: (a) The latent space usually corresponds to a low dimensional space,
here 2 dimensional, so that a point h represented as the black dot in this space
is specified by coordinates (h1 , h2 ). Associated with this latent space is a prior
belief about where the latent parameters are. Here this is a Gaussian distribution.
(b) Each point in the latent space is mapped to some point in a typically higher
dimensional space, here 3 dimensional. The mapping here is linear so that the
object in the higher dimensional space is simply a plane – that is, a point in
the lower dimensional space gets mapped to corresponding point (black dot) in
the plane. Similarly, there will be an associated density function in this higher
dimensional space, inherited from the density function in latent space. (c) Here
the mapping from latent space to data space is non-linear, and produces a two
dimensional manifold embedded in the three dimensional space.

19.4 Probabilistic PCA

A probabilistic version of PCA would be advantageous, since all the usual benefits
inherited from the probabilistic viewpoint follow – automatic ways to do model
selection, error bars, etc[35].
Since PCA fits a H dimensional hyperplane in a V dimensional space, it is natural
to consider the model
p(v|h) = N (W h, σ 2 I)
where V is a V × H matrix. Equivalently, we may write
v = Wh + ǫ
where ǫ ∼ N (0, σ 2 I). Here we use an isotropic noise,σ 2 I in the original version of
PCA, no preference was given for the error in any direction off the hyperplane. To
complete the model, we need to specify a prior p(h). To make things tractable, we
specify this to be a Gaussian. The reader may satisfy herself that, without loss of
generality, we may specify
p(h) = N (0, I)
What is the distribution
R p(v)? Formally, one can find this by calculating the
integral p(v) = h p(v|h)p(h). Doing this, we find that p(v) is Gaussian. This is an
exercise that most people go though before they realise that there is a shortcut.
 215

h1 h2 h3

v1 v2 v3 v4 v5

Figure 19.7: Graphical representation of PPCA for a model with 3 hidden or latent
variables, which generate the visible or output variable v = (v1 , . . . , v5 )T .

Since p(v) is going to be Gaussian, all we need to do is find its mean and covariance.
Since the noise is zero mean, then v will be zero mean. The covariance is given by

T D T
E
vv = (W h + ǫ) (W h + ǫ)

where the angled brackets denote an average with respect to all sources of fluctua-
tions, namely the noise and the hidden distribution. Since these noise sources are
uncorrelated, we have

T
vv = W W T + σ 2 I

Hence

p(v) = N (0, Σ = W W T + σ 2 I)

Now consider a dataset, v µ , µ = 1, . . . , P . Under the usual assumption that the

data are independently and identically distributed,
P
Y
1 P
p(v , . . . , v ) = p(v µ ) (19.4.1)
µ=1
P
Y 1
e−(v ) Σ v /2
µ T −1 µ
= √ (19.4.2)
µ=1 det 2πΣ
 −1 PP 
1 −trace Σ µ µ T
µ=1 v (v ) /2
= P/2
e (19.4.3)
(det 2πΣ)
 −1 
1 −trace Σ S/2
= e (19.4.4)
(det 2πΣ)P/2
PP
where S ≡ µ=1 v µ (v µ )T is the sample correlation matrix of the data. To find the
maximum likelihood setting of W , it’s a little easier to consider the log likelihood:
P P

L=− log det Σ − trace Σ−1 S
2 2
where we dropped irrelevant constants. Now, differentiating with respect to W ,
and equating to zero, we obtain



0 = −trace Σ−1 ∂W Σ + trace Σ−1 (∂W Σ)Σ−1 S

Using ∂W (W W T ) = W (∂W W T ) + (∂W W )W T , and using the symmetries, a sta-

tionary point is certainly given when

Σ−1 W = Σ−1 SΣ−1 W

 216

Or, assuming invertibility of Σ,

W = SΣ−1 W

This is reminiscent of an eigen-problem. If we represent W using its singular value

decomposition, W = U LV T , where U is a V × H dimensional matrix, U T U = I, L
is a diagonal, matrix containing the singular values, and V is a H × H orthogonal
matrix, we have


SU L = U σ 2 I + L2 L

which gives the eigen-equation

SU = U σ 2 I + L2

Hence U are the eigenvectors of the correlation matrix S and λi = σ 2 + li2 are the
eigenvalues. This constraint requires therefore li = (λi − σ 2 )1/2 , which means that
the solutions are of the form

1/2
W = E Λ − σ2 I R

where R is an arbitrary orthogonal matrix. The reader may verify, by plugging this
solution back into the log-likelihood expression, that the eigenvalues and associated
eigenvectors which maximise the likelihood correspond to the H largest eigenvalues
of S. The standard, non-probabilistic variant PCA is given as the limiting case
σ 2 → 0 of PPCA.
Let’s order the eigenvalues so that λ1 ≥ λ2 , ... ≥ λV . The value for the log
likelihood is then (see exercises)
H V
!
P X 1 X
L=− V log(2π) + log λi + 2 λi + (V − H) log σ 2 + H
2 i=1
σ
i=H+1

The reader may then verify that the optimal ML setting for σ 2 is
V
2 1 X
σ = λj
V −H
j=H+1

Of course, we could have trained the above method using the standard EM algo-
rithm. What’s convenient about PPCA is that the solution is analytic, and boils
down to a simple eigenproblem.
Note: in the above, we clearly need λi ≥ σ 2 for the retained eigenvectors. However,
for the ML solution this is guaranteed, since σ 2 is set to the average of the discarded
eigenvalues, which must therefore be smaller that any of the retained eignvalues.

Mixtures of PPCA

One of the benefits of a probabilistic model is that we can form a mixture[36].

We’ll talk more about this in the chapter on discrete mixture models.
 217

19.5 Problems
Exercise 49 In one dimension, dim(x) = 1, the Gaussian distribution is defined
as
1 1 2
p(x) = √ e− 2σ2 (x−µ)
2πσ 2
You decide to fit a Gaussian to each class and use the ML estimates of the means
µ̂1 and µ̂2 . From the data, you find that the ML estimates of σ12 and σ22 are
equal, that is, σ̂12 = σ̂22 . Write down the explicit x value that defines the decision
boundary.
Point out any potential numerical difficulties in directly comparing the values p(c =
1|x) and p(c = 2|x) and explain how you might overcome this.
In more than one dimension, the multi-variate Gaussian is defined as
1
e− 2 (x−µ) S (x−µ)
1 T −1
p(x) = √
det 2πS

Given a dataset of iid samples, x1 , . . . , xP , derive an expression for the Maximum

Likelihood estimator µ̂.
Explain with the aid of a diagram, the nature of the decision boundary in the case
that σ̂12 6= σ̂22

Exercise 50 In factor analysis there is a prior distribution P (z) ∼ N (0, Im ) over

the m-dimensional latent variables, and a likelihood term P (x|z) ∼ N (W z, Ψ) of
a data point x given a value z of the latent variable vector.

The posterior distribution for z is given by P (z|x) ∝ P (z)P (x|z) (we don’t need to
worry too much about the normalization of the posterior distribution in this ques-
tion).

Show that the posterior distribution is Gaussian with mean (Im +W T Ψ−1 W )−1 W T Ψ−1 x,
and state the covariance matrix of the posterior distribution.

Hint: If P (y) ∼ N (m, V ) then P (y) ∝ exp{− 21 (y − m)T V −1 (y − m)}.

Exercise 51 Factor analysis and scaling. Assume that a m-factor model holds
for x. Now consider the the transformation y = Cx, where C is a non-singular
diagonal matrix. Show that factor analysis is scale invariant, i.e. that the m-factor
model also holds for y, with the factor loadings appropriately scaled. How must the
specific factors be scaled?

Exercise 52 Consider data points generated from two different classes. Class 1
has the distribution P (x|C1 ) ∼ N (µ1 , σ 2 ) and class 2 has the distribution P (x|C2 ) ∼
N (µ2 , σ 2 ). The prior probabilities of each class are P (C1 ) = P (C2 ) = 1/2. Show
that the posterior probability P (C1 |x) is of the form
1
P (C1 |x) =
1 + exp −(ax + b)
 218

and determine a and b in terms of µ1 , µ2 and σ 2 . The function f (z) = 1/(1 + e−z )
is known as the logistic function, and is a commonly-used transfer function in
artificial neural networks.

Exercise 53 Consider a one-dimensional probabilistic PCA model, so that P (x|z) ∼

N (wz, σ 2 I) and P (z) ∼ N (0, 1). We are given data vectors xi drawn from the
model, but do not have the corresponding latent variables zi , so this is a missing-
data problem.

Calculate the EM update for the factor loadings w.

Generalize this to the case where the latent variable is a vector.

Exercise 54 Consider a V ×H dimensional matrix E = [e1 , . . . , eH ] where (ea )T eb =

δab for all a, b ≤ H, and a H × H dimensional diagonal matrix D. Show that
H
Y
det(EDE T + σ 2 I) = (σ 2 )(V −H) (Dii + σ 2 )
i=1

HINT : use the fact that the determinant of a matrix is the product of its eigen-
values.
Using Sea = λ̃a ea , a = 1, . . . , V calculate explicitly the value of the expression


trace (EDE T + σ 2 I)−1 S

in terms of the λ̃a and σ 2 . HINT: use the fact that the trace of a matrix is the
sum of its eigenvalues.

Exercise 55 Explain why there is no closed form solution to the extension of

PPCA to the case where the noise is diagonal, but non-isotropic. That is, hǫi ǫj i =
σi2 δij .

19.6 Solutions
20 Dynamic Bayesian Networks : Discrete Hidden Variables

20.1 The Importance of Time

In many physical applications, time plays a crucial role. Indeed, many models of
the physical world are based on differential equations, such as Newton’s First Law
dx
=v
dt
meaning that things keep moving in a straight line when not acted upon by an
external force. A computational approximate implementation of this differential
equation is given by a difference equation :
x(t + δ) − x(t)
= v ⇒ x(t + δ) = x(t) + vδ
δ
For convenience, we can rescale time so that a unit time step represents δ seconds
:

x(t + 1) = x(t) + c, c = const.

In an imperfect world, however, physical processes are often perturbed slightly by

noise, (perhaps slight air currents are disturbing the otherwise perfect motion, for
example). These effects could be modelled by

x(t + 1) = x(t) + c + ǫ(t)

where ǫ(t) is a random variable sampled from some distribution. For example, if
the noise is Gaussian with zero mean and variance σ 2 , then
1 1 2
p(x(t + 1)|x(t)) = √ e− 2σ2 (x(t+1)−x(t)−c)
2πσ 2

Markov Process This is an example of a Markov chain. Processes are Markov if the future state
x(t + 1) only depends on the current state x(t). This is called a first order Markov
process which refers to the dependency on only the first immediately preceding
state in time. More formally,

p(x(t + 1)|x(t), x(t − 1), . . . , x(1)) = p(x(t + 1)|x(t))

Therefore the joint distribution admits the factorisation

TY
−1
p(x(T ), . . . , x(1)) = p(x(1)) p(x(t + 1|x(t))
t=1

Other common physical systems depend on second order differential equations.

For example, Newton’s 2nd law states that, under a force F ,

d2 x
= k1 , k1 = const
dt2

219
 220

filtering
111111111111
000000000000
000000000000
111111111111
000000000000
111111111111
000000000000
111111111111
t
smoothing
111111111111111111111
000000000000000000000
000000000000000000000
111111111111111111111
000000000000000000000
111111111111111111111
000000000000000000000
111111111111111111111

prediction t
11111111
00000000
00000000
11111111
00000000
11111111
11111111111111111111111
00000000000000000000000
00000000
11111111
t
1111
0000
0000
1111
0000
1111
0000
1111 denotes the extent of data
available

Figure 20.1:

Again, if we were to write down a discrete time difference approximation, we would

have an equation of the form (including a possibly stochastic noise term):

x(t + 1) − 2x(t) + x(t − 1) = k2 + ǫ(t)

Or

x(t + 1) = 2x(t) − x(t − 1) + k2 + ǫ(t)

Second Order Here the state of the future world depends on the present and the immediate past1 .
In general, then we would have
TY
−2
p(x(T ), . . . , x(1)) = p(x(1), x(2)) p(x(t + 2|x(t + 1), x(t))
t=1

The above is an example of a second order Markov process. The generalisation to

a k th order process is obvious. We also call these models Markov Models. In the
above examples it would be natural to consider the variables x(t) to be continuous.

Inference Problems

• p(ht |v1 , . . . , vt ) filtering

• p(ht |v1 , . . . , vs ) t > s, prediction
• p(ht |v1 , . . . , vu ) t < u, smoothing
• p(v1 , . . . , vT ) likelihood calculation
• Find sequence h∗1 , . . . , h∗T that maximizes p(h1 , . . . , hT |v1 , . . . , vT ) [Viterbi
alignment]

Transition Diagrams and Finite State Automata

Models with discrete variables are common, and have significant application in
many fields, ranging from sequence modelling in Bioinformatics to text and speech
processing. Here is a simple example:
1 It is a deep (and at least to me, somewhat mysterious) property that all laws of physics are
only maximally second order differential equations, and hence can be well approximated by
second order stochastic differential equations.
 221

Hilde is an interesting chimp. She has been trained to press the buttons 1, 2 and
3 always in sequence, although the starting state doesn’t matter. For example,
2,3,1,2,3,1,2,3,1,2,3. Hilde is quite good at this, but sometimes makes a mistake
and presses a button out of sequence. The probability that she makes a transition
from state j to state i, p(i|j) is given by the matrix elements below:
1 2 3
1 0.1 0.1 0.8
2 0.8 0.1 0.1
3 0.1 0.8 0.1
which can be represented as a matrix pij ≡ p(i|j). Alternatively, a state transition
diagram can be used, as in fig(20.2) below. To make this more informative, one
sometimes shows also the values of the transitions on the links. This is a general-

3 2

Figure 20.2: A state transition diagram, for a three state markov chain. Note that
a state transition diagram is not a graphical model – it simply graphically displays
the non-zero entries of the transition matrix p(i|j)

isation of Finite State Automata. In FSAs, the transitions are deterministic, the
corresponding table entries being either 0 or 1.

Hidden Markov Models

Vernon is another, slightly less reliable chimp. He has been trained such that
whenever he sees that Hilde has pressed either button 1 or 2, he grunts A and
whenever he sees a 3 he grunts B. However, he also makes mistakes, as characterised
by the probability state transition matrix below,
1 2 3
A 0.7 0.6 0.25
B 0.3 0.4 0.75
which can be represented by pij ≡ p(v(t) = i|h(t) = j).

Flippa is a super clever dolphin. She has sent to her a sequence of grunts from
Vernon, e.g. B,A,A,B,A,B and has been trained to figure out from the sequence
Vernon grunts, what is is that Hilde typed. Of course, this is not strictly solvable
in an exact sense. Flippa reports back to her trainer the most likely sequence of
buttons pressed by Hilde.

A model of the above situation is given in fig(20.3), a so called Hidden Markov

Model (HMM)[37].
The idea here is that there is some internal (hidden) dynamics (modelled by Hilde
in our example above), and an observation or visible variable for each of the hidden
variables (modelled by Vernon). In our example, each hidden variable would have
Stationarity 3 states, and each visible variable would have 2 states. We assume stationarity
 222

h1 h2 h3 h4

v1 v2 v3 v4

Figure 20.3: A Hidden Markov Model.

– that is, the model for each time step to time step holds for all times. The
distribution is therefore fully described by the
Transition matrix

Ai′ ,i = p(ht+1 = i′ |ht = i)

an Emission matrix

Bj,i = p(vt = i|ht = j)

and finally a Prior

πi = p(h1 = i).

Structured Transition Matrices

In some applications (often in Automatic Speech Recognition), it is useful to con-

strain the state sequence to go ‘forwards’ through time. For example, if the HMM
models a certain word, where the utterance may be broken into a subsequence of
phonemes, then we expect the word to follow a set of phonemes/states, and not
to go back to them. Sometimes, we might stay in one state longer than another,
or even maybe skip some states. However, the important thing is that there may
be a sense of structure therefore to the transition matrices. For those that go
forwards in this sense, the structure of the transition matrices is upper triangular
(or lower, depending on your tastes), or even a banded triangular matrix. Such
forward constraints describe a so-called left-to-right transition matrix.

JTA for HMMs

How can Flippa solve her problem? That is find argmax p(h1 , . . . , hT |v1 , . . . , vT )?
h1 ,...,hT
As we saw previously, such most probable state calculations can be carried out
by a slight modification of the JTA. The first step then is to find a Junction Tree
for the HMM. The HMM is already moralised and triangularised. A suitable JT,
along with a valid assignment of the potentials is given in fig(20.4).

JTA and the Forward-Backward Algorithm for HMMs

A valid sequence of absorptions would consist of a forward sweep, and then a

backward sweep.
Forward sweep For t = 1, . . . T − 2 :
Absorb from Ψ (vt , ht ) to Ψ (ht , ht+1 ). Then from Ψ (ht , ht+1 ) to Ψ (ht+1 , ht+2 ).
Repeat.
Absorb from Ψ (vT −1 , hT −1 ) to Ψ (hT −1 , hT ). Absorb from Ψ (hT −1 , hT ) to
Ψ (vT , hT ).
 223

h1 , h2 h2 , h3 h3 , h4

v1 , h1 v2 , h2 v3 , h3 v4 , h4

Figure 20.4: A first order Hidden Markov Model. A suitable assignment

of the potentials is: Ψ (h1 , h2 ) = p(h1 )p(h2 |h1 ), Ψ (v1 , h1 ) = p(v1 |h1 ),
Ψ (h2 , h3 ) = p(h3 |h2 ), Ψ (v2 , h2 ) = p(v2 |h2 ), Ψ (h3 , h4 ) = p(h4 |h3 ), Ψ (v3 , h3 ) =
p(v3 |h3 ),Ψ (v4 , h4 ) = p(v4 |h4 ). All separator potentials (not shown) are initially
set to unity.

Backward sweep Absorb from Ψ (vT , hT ) to Ψ (hT −1 , hT )

For t = T − 1, . . . 2 :
Absorb from Ψ (ht , ht+1 ) to Ψ (vt , ht ). Then from Ψ (ht , ht+1 ) to Ψ (ht−1 , ht ).
Repeat.
Absorb from Ψ (h1 , h2 ) to Ψ (v1 , h1 )
This formulation of the JTA is equivalent to the classic Forward-Backward al-
gorithm for HMMs. Flippa can solve her problem by using the max version of
absorption. The complexity of hidden variable inference in a HMM scales linearly
in T , V (the number of visible states) and quadratically in H, giving a time com-
plexity of O(T H(V + H)). The important point is that it scales linearly with
time so that, even with a lengthy observed sequence, we can still perform inference
quickly.

20.1.1 Parallel and Sequential Inference

We are interested in the so-called ‘smoothed’ posterior p(ht |v1:T ). There are two
main approaches to computing this.

Parallel Method

p(ht |v1:T ) ∝ p(ht , v1:T )

∝ p(ht , vt , vt+1:T )
∝ p(ht , v1:t , vt+1:T )
∝ p(vt+1:T |ht ,  )p(ht |v1:t )
v1:T
(20.1.1)

Hence, in this method, we need to compute the so-called ‘filtered’ posteriors

p(ht |v1:t ), and combine them with the conditional terms p(vt+1:T |ht ). As we will
see below, the filtered posteriors are easy to get using a Forward recursion. The
terms p(vt+1:T |ht ) are also easy to obtain using a Backward recursion. The Forward
and Backward recursions may be called in parallel, with their results combined to
obtain the smoothed posterior.
 224

Sequential Method

X
p(ht |v1:T ) ∝ p(ht , ht+1 , v1:T )
ht+1
X
∝ p(ht |ht+1 , v1:t ,  )p(ht+1 |v1:T )
vt+1:T
ht+1

(20.1.2)

This then gives a Backwards recursion for p(ht |v1:T ). As we will see below, the
term p(ht |ht+1 , v1:t may be computed based on the filtered results p(ht |v1:t .

The classical α − β recursions

As we mentioned in the chapter on inference, the difference between Belief Propa-

gation and the Junction Tree algorithm on singly-connected structures, is just in
the parameterisation of the messages.
For readers less familiar with the probabilistic approach, we’ll briefly describe
here an alternative derivation of Belief Propagation inference on simple chain
distributions[7, 16]. The presentation here follows that presented in[38]. First,
let’s simplify the notation, and write the distribution as
Y
p= φ (xt−1 , vt−1 , xt , vt )
t

where xt ≡ ht , and φ (xt−1 , vt−1 , xt , vt ) = p(xt |xt−1 )p(vt |xt ). Our aim is to define
‘messages’ ρ, λ (these correspond to the α and β messages in the Hidden Markov
Model framework[37, 39]) which contain information from past observations and
future observations respectively. Explicitly, we define ρt (xt ) ∝ p(xt |v1:t ) to repre-
sent knowledge about xt given all information from time 1 to t. Similarly, λt (xt )
represents knowledge about state xt given all information from the future obser-
vations from time T to time t + 1. In the sequel, we drop the time suffix for
notational clarity. An important point is that λ(xt ) is not a distribution in xt , but
rather implicitly defined through the requirement that the marginal inference is
then given by

p(xt |v1:T ) ∝ ρ (xt ) λ (xt ) (20.1.3)

Similarly, the pairwise marginal is given by

p(xt−1 , xt |v1:T ) ∝ ρ (xt−1 ) φ (xt−1 , vt−1 , xt , vt ) λ (xt ) (20.1.4)

Taking the above equation as a starting point, we can calculate the marginal from
this
X
p(xt |v1:T ) ∝ ρ (xt−1 ) φ (xt−1 , vt−1 , xt , vt ) λ (xt ) (20.1.5)
xt−1

Consistency with equation (22.2.1) requires (neglecting irrelevant scalings)

X
ρ (xt−1 ) λ (xt ) ∝ ρ (xt ) φ (xt−1 , vt−1 , xt , vt ) λ (xt ) (20.1.6)
xt−1
 225

Similarly, we can integrate equation (22.2.2) over xt to get the marginal at time
xt−1 which by consistent should be proportional to ρ (xt−1 ) λ (xt−1 ). From such
considerations we arrive at
P
xt−1 ρ (xt−1 ) φ (xt−1 , xt ) λ (xt )
ρ (xt ) ∝ , (20.1.7)
λ (xt )
P
ρ (xt−1 ) φ (xt−1 , xt ) λ (xt )
λ (xt−1 ) ∝ xt (20.1.8)
ρ (xt−1 )

where the divisions can be interpreted as preventing overcounting of messages.

The common factors in the numerator and denominator exactly cancel to give
X
Forward Recursion: ρ (xt ) ∝ ρ (xt−1 ) φ (xt−1 , vt−1 , xt , vt ) (20.1.9)
xt−1

X
Backward Recursion: λ (xt−1 ) ∝ φ (xt−1 , vt−1 , xt , vt ) λ (xt ) (20.1.10)
xt

which are the usual definitions of the messages defined as a set of independent
recursions. In engineering, the ρ message is called the α message, and the λ
message is called the β message. This method of performing inference is called a
parallel method since the α and β recursions are independent of each other and
can therefore be implemented in parallel. After computation, they may then be
combined to compute the smoother posterior.
The extension to more general singly connected structures is straightforward and
results in partially independent recursions which communicate only at branches of
the tree [7].
From equation (22.2.1) it is straightforward to see that λ (xt ) ∝ p(vt+1:T |xt , v1:t ) =
p(vt+1:T |xt ). By definition ρ(xt ) ∝ p(xt |v1:t ) is the filtered estimate.

Logs or Normalise?

The repeated application of the recursions equation (20.1.9) and equation (20.1.9)
may lead to numerical under/over flow. There are two strategies for dealing with
this. One is to work in log space, so that only the log of the messages are defined.
The other (which is more common in the machine learning literature) is to nor-
malise the messages ρ and λ at each stage of the iteration, so that the messages
sum to unity. Normalisation is valid since both the filtered p(xt |v1:T ) ∝ ρ (xt )
and smoothed inferences p(xt |v1:T ) ∝ ρ (xt ) λ (xt ) are simply proportional to the
messages. The missing proportionality constants can be worked out easily since
we know that distributions must sum to one.
 226

20.1.2 Rauch Tung Striebel and the α − γ recursions

In the above, we found a forward (α or ρ) recursion for the filtered inference

p(ht |v1:t ). Explicitly, for the HMM, the forward pass is
X
p(ht |v1:t ) ∝ p(ht , ht−1 , v1:t−1 , vt ) (20.1.11)
ht−1
X
= p(vt |v1:t−1 , h1:t , ht , ht−1 )p(ht |v1:t−1 , ht−1 )p(v1:t−1 , ht−1 )
ht−1
(20.1.12)
X
∝ p(vt |ht )p(ht |ht−1 )p(ht−1 |v1:t−1 ) (20.1.13)
ht−1

Here we derive an alternative way to compute the smoothed inference p(ht |v1:T )
by correcting these filtered results. We start with the recursion
X
γ(ht ) ≡ p(ht |v1:T ) = p(ht , ht+1 |v1:T ) (20.1.14)
ht+1
X
= p(ht |ht+1 , v1:t )p(ht+1 |v1:T ) (20.1.15)
ht+1

Hence, we can form a backwards recursion for the smoothed inference. p(ht , ht+1 |v1:T )
is given by the above before summing over ht+1 . We therefore need

p(ht |ht+1 , v1:t ) ∝ p(ht+1 , ht |v1:t ) ∝ p(ht+1 |ht )p(ht |v1:t )

The denominator is just found by normalisation. In the above, we see that the
smoothed recursion makes explicit use of the filtered results. In contrast to the α−β
independent recursion, the above procedure is called a sequential procedure since
we need to first complete the α recursions, after which the γ recursion may begin.
Formally, the α − β and α − γ recursions are related through γ(ht ) ∝ α(ht )β(ht ).

The Likelihood p(v1:T )

The likelihood is found from recursing

Y
p(v1:t ) = p(vt |v1:t−1 )
t

Each factor
X
p(vt |v1:t−1 ) = p(vt , ht |v1:t−1 ) (20.1.16)
ht
X
= p(vt |ht )p(ht |v1:t−1 ) (20.1.17)
ht
X X
= p(vt |ht ) p(ht |ht−1 )p(ht−1 |v1:t−1 ) (20.1.18)
ht ht−1

where the final term p(ht−1 |v1:t−1 ) are just the filtered inferences. Note, therefore,
that the likelihood of a output sequence requires only a forward computation.
 227

20.1.3 Viterbi

Consider the general HMM problem: Find the most likely state of

p(h1:T |y1:T )

This is the same as the most likely state of

Y
p(h1:T , y1:T ) = p(yt |ht )p(ht |ht−1 )
t

This is easy to find, by using the max version of the JTA/BP algorithms. To make
this explicit though, we write down exactly how this would proceed:
To make the notation a little easier, let’s define the potential functions :

φ(ht−1 , ht ) = p(yt |ht )p(ht |ht−1 )

where for the first time step we just define φ(h1 ) = p(y1 |h1 )p(h1 ). The finding
the most likely hidden state sequence is equivalent to finding the state h1:T that
maximises the funcion
T
Y
φ = φ(h1 ) φ(ht−1 , ht )
t=2

The dependency on h1 appears only in the first two terms φ(h1 ) and φ(h1 , h2 ).
Hence when we perform the max over h1 , we can write

YT
max φ = max max φ(h1 )φ(h1 , h2 ) φ(ht−1 , ht )
h1:T h2:T h1
| {z } t=3
f (h2 )

At the next stage, we can perform the max over h2 :

YT
max φ = max max f (h2 )φ(h2 , h3 ) φ(ht−1 , ht )
h1:T h3:T h2
| {z } t=4
f (h3 )

We can continue this procedure, at each stage defining the new potentials

f (ht ) = max f (ht−1 )φ(ht−1 , ht )

ht−1

until we reach the end of the chain, and we have defined f (h2 ), . . . , f (hT ). Then,
to find which states actually correspond to the maxima, we need to backtrack: We
have at the end of the chain f (hT ). Hence, the most likely state is given by

h∗T = arg max f (hT )

hT

With this most likely state, we can write

max φ = f (hT −1 )φ(hT −1 , h∗T )

h1:T −1

So that we can find the optimal state hT −1 by computing

h∗T −1 = arg max f (hT −1 )φ(hT −1 , h∗T )

hT −1
 228

and similarly,

h∗t−1 = arg max f (ht−1 )φ(ht−1 , h∗t )

ht−1

for t = T − 1, . . . , 1, where we define f (h1 ) ≡ p(h1 ).

For the HMM this special case of the max-product algorithm is called the Viterbi
algorithm, a terminology from speech research.

Second Order HMM

We can look at more complex time dependencies in the hidden variables by in-
creasing the range of the temporal dependencies. For example, a second order
HMM is given in fig(20.5). The inference can again be carried out using the JTA,

h1 h2 h3 h4 h5

v1 v2 v3 v4 v5

Figure 20.5: A 2nd order Hidden Markov Model.

and a suitable JT is given in fig(20.6).

h1 , h2 , h3 h2 , h3 , h4 h3 , h4 , h5

v1 , h1 v2 , h2 v3 , h3 v4 , h4 v5 , h5

Figure 20.6: A 2nd order Hidden Markov Model. A suitable assignment of

the potentials is: Ψ (h1 , h2 , h3 ) = p(h1 )p(h2 |h1 )p(h3 |h1 , h2 ), Ψ (h2 , h3 , h4 ) =
p(h4 |h2 , h3 ),Ψ (h3 , h4 , h5 ) = p(h5 |h3 , h4 ), Ψ (v1 , h1 ) = p(v1 |h1 ), Ψ (v2 , h2 ) =
p(v2 |h2 ), Ψ (v3 , h3 ) = p(v3 |h3 ),Ψ (v4 , h4 ) = p(v4 |h4 ),Ψ (v5 , h5 ) = p(v5 |h5 ). Again,
separator potentials are not shown. They may all be set initially to unity.

Now the complexity is still linear in time, but O(T H(V + H 2 ). In general, the
complexity will be exponential in the order of the interactions.

Learning HMMs

Previously, we concentrated on inference in HMMs. We also have a general frame-

work for learning in graphical models. Here, as an example application of our
previous general framework, we show how it applies to HMMs. Historically, the
procedure of using the EM algorithm was called the Baum-Welch algorithm. Per-
sonally, I think that you should really remember the general approach, and be
delighted to see that it produces BW algorithm as a special case.

Baum-Welch Algorithm

A HMM is trained by treating the output nodes as evidence nodes and the state
nodes as hidden nodes. This is clearly tractable since the moralization and trian-
gulation steps do not add any extra links. The cliques are of size N 2 where N is
 229

the dimension of the state nodes. Inference therefore scales as O(N 2 T ) where T
is the length of the times series.
To find the parameters of the model, A, B, π, a variational type (EM) procedure
can be used, which can be constructed using our previous EM framework.
To make the notation reasonably simple, we write v = (v1 , v2 , . . . , vT ), and simi-
larly, h = (h1 , h2 , . . . , hT ).

Let’s look at the energy function :

XX
hlog p(v1µ , v2µ , . . . , vTµ , hµ1 , hµ2 , . . . hµT iqµ (h|v)
µ t

Using the form of the HMM, we obtain

( T −1 T
)
X X X µ
hlog p(h1 )iqµ (h1 |v) + hlog p(ht+1 |ht )iqµ (ht ,ht+1 |vµ ) + hlog p(vt |ht )iqµ (ht |vµ )
µ t=1 t=1

To avoid potential confusion, we write pnew (h1 = i) to denote the (new) table
entry for probability that the intial hidden variable is in state i. The prior term,
by the previously derived EM approach then gives
X
πinew ≡ pnew (h1 = i) ∝ pold (h1 = i|v µ ) (20.1.19)
µ

which is the average number of times that the first hidden variable is in state i.
Similarly,

X TX
−1
Anew
i′ ,i ≡p new ′
(ht+1 = i |ht = i) ∝ pold (ht = i, ht+1 = i′ |v µ ) (20.1.20)
µ t=1

which is the number of times that a transition from hidden state i to hidden state
i′ occurs, averaged over all times (since we assumed stationarity) and training
sequences. Finally,
T
XX
new
Bj,i ≡ pnew (vt = j|ht = i) ∝ I[vtµ = j]pold (ht = i|v µ ) (20.1.21)
µ t=1

which is the expected number of times that, for the observation being in state j, we
are in hidden state i. The proportionalities are trivially determined by the normal-
isation constraint. Together, the above three equations define the new prior, tran-
sition and emission probabilities. Using these values for the HMM CPTs, at the
next step we can calculate the quantities pold (h1 = i|v µ ), pold (ht = i, ht+1 = i′ |v µ )
and pold (ht = i|v µ ) using the JTA (or the so-called ‘Forward-Backward’ algo-
rithm, which is equivalent). The equations (20.1.19,20.1.20,20.1.21) are repeated
until convergence.

Parameter Initialisation

The above EM approach is guaranteed to converge to a local maxima of the likeli-

hood (one can show explicitly that the re-estimation formulae correspond to fixed
point equations representing the point where the gradient of the likelihood is zero).
 230

(Of course, if we were to use a restricted class of q µ functions, we would only con-
verge to a local maximum of the lower bound on the likelihood). There is no
guarantee that the algorithm will find the global maximum, and indeed, the value
of the local maximum found is often critically dependent on the initial settings of
the parameters. How best to initialise the parameters is a thorny issue. According
to Rabiner :

“Experience has shown that either random (subject to the stochastic and the
nonzero value constraints) or uniform initial estimates of the π and A parameters
is adequate for giving useful re-estimates of these parameters in almost all cases.
However, for the B parameters, experience has shown that good initial estimates
are helpful in the discrete case, and are essential in the continuous distribution case
(see later). Such initial estimates can be obtained in a number of ways, including
manual segmentation of the observation sequence(s) into states with averaging of
observations within states, maximum likelihood segmentation of observations with
averaging, and segmental k-means segmentation with clustering.”

Continuous Output, Discrete Hiddens

In many cases, the observation sequence is continuous. In fig(20.8) we saw an

approach using vector quantisation to transform continuous outputs into discrete
variables. Here we will consider a different approach which retains continuous
outputs. In later sections we will also deal with the case of continuous hidden
variables. However, for the moment, it is simpler to deal with discrete hidden
dynamics.
What is required is a specification
p(v|h)
where v is continuous vector variable and h is discrete. Using a continuous output
will not significantly affect the previous update equations provided we assume
that all the observations are indeed visible. In this case, the contributions to the
energy are as before, but simply the factors p(v(t)|h(t)) will have a numerical value
determined by the new density.
A natural candidate for the above are Mixture Models
X
p(v|h) = p(k|h)p(v|k, h)
k

where k is a discrete summation variable. And a common choice is that each

distribution p(v|k, h) is a Gaussian
p(v|k, h) = N (µk,h , Σk,h )
This Gaussian mixture model therefore requires the learning of KH V -dimensional
mean vectors (where V is the dimension of the output, K is the number of mixture
components and H is the number of hidden states), and also KH covariance
matrices (having V (V + 1)/2 parameters each). The update equations in learning
for π and A remain the same. However, the emission probabilities now require us
to find those optimal mean, covariances and p(k|h) parameters that maximise
T
XX
hlog p(vt |ht )iqµ (ht |V µ )
µ t=1
 231

xt−1 xt xt+1

ht−1 ht ht+1

yt−1 yt yt+1

Figure 20.7: Graphical model of the IOHMM. Nodes represent the random vari-
ables and arrows indicate direct dependence between variables. In our case the
output variable yt is discrete and represents the class label, while the input variable
xt is the continuous (feature extracted from the) EEG observation. The yellow
(shaded) nodes indicate that these variables are given, so that no associated dis-
tributions need be defined for x1:T .

where V µ is the total sequence of observation vectors. We may consider the k as

hidden variables, and then use EM algorithm as before to obtain update equations
for the parameters of the Gaussian Mixture Model (or indeed the parameters of
any Mixture Model). We leave it to the reader to complete this straightforward
derivation. GMMs for the emissions in this way are used in state of the art speech
recognition software.

Related Models
Input-Output HMM

The IOHMM is just a HMM augmented with outputs (visible variables) y1:T and
hidden states h1:T . However, we now consider that we are given for each time step
an input xt . This input can be continuous or discrete and affects the transitions
as
Y
p(y1:T , h1:T |x1:T ) = p(yt |ht , xt )p(ht |ht−1 , xt )
t

This is just another HMM, and extending inference and learning to this case is
straightforward. IOHMM is usually used as a conditional classifier, where the out-
puts yt represent a class label at time t. (There are other ways to train this model,
say by specifying a label only at the end of the sequence). In the case of continuous
inputs, the tables p(yt |ht , xt ) and p(ht |ht−1 , xt ) are usually parameterised using a
non-linear function, eg.
T
p(yt = y|ht = h, xt = x) ∝ ewh,y x

Inference then follows the same line as for the standard HMM:
X
p(ht |x, y) = p(ht , ht+1 |x, y) (20.1.22)
ht+1
X
= p(ht |ht+1 , x1:t+1 , y1:t )p(ht+1 |x, y) (20.1.23)
ht+1

(20.1.24)
 232

Hence, we can form a backwards recursion. p(ht , ht+1 |x1:T , y1:T ) is given by the
above before summing over ht+1 .
We therefore need
p(ht+1 , ht |x1:t+1 , y1:t ) p(ht+1 |ht , xt+1 )p(ht |x1:t , y1:t )
p(ht |ht+1 , x1:t , y1:t ) = =
p(ht+1 |x1:t+1 , y1:t ) p(ht+1 |x1:t+1 , y1:t )

The denominator is just found by normalisation. To find the rest, we use a forward
pass
X
p(ht |x1:t , y1:t ) ∝ p(ht , ht−1 , x1:t , y1:t−1 , yt ) (20.1.25)
ht−1
X
= p(yt |y1:t−1 , x1:t , ht , ht−1 )p(ht |y1:t−1 , x1:t , ht−1 )p(y1:t−1 , x1:t , ht−1 )
ht−1
(20.1.26)
X
∝ p(yt |xt , ht )p(ht |xt , ht−1 )p(ht−1 |x1:t−1 , y1:t−1 ) (20.1.27)
ht−1

Q
The likelihood is found from recusing p(y|x) = p(yt |y1:t−1 , x)

p(yt |y1:t−1 , x) = p(yt |y1:t−1 , x1:t )

Direction Bias

The IOHMM and related conditionally trained models ‘suffer’ from the fact that
any prediction p(vt |h1:T ) in fact depends only on the past p(vt |h1:t ). This is not
true, of course, of the most likely output sequence. Such ‘direction bias’ is identified
in some sections of the literature (particularly in natural language modelling) as
problematic, and motivates the use of undirected models, such as the Conditional
Random Field.

20.2 Applications of HMMs

Speech Processing

This is arguably one of the most successful application areas of HMMs.

This section is based on Mike Alder’s excellent book on An Introduction to Pattern
Recognition, http://heavenforbooks.com.
While by no means meant to be an attempt to explain the state of the art in speech
recognition, we here give a flavour of one of the common approached to solving
this problem. For simplicity, we here think of the case of recognizing only single
words.
If you say a word, perhaps the name of a digit, for example ‘one’, into a microphone,
then it is straightforward to sample and digitise the resulting signal, and feed it
into a computer as a longish sequence of numbers measuring the voltage generated
by the microphone and your voice. Typically, a word may take one third to half
a second to enunciate, and the signal is sampled perhaps twenty thousand times
a second, giving around seven thousand numbers. Each number will be quantised
to perhaps 12 or 16 bit precision. Thus we may be looking at a data rate of
around 30 to 40 kilobytes per second. This present paragraph, would, if spoken
 233

k k
f f
g j g j
h h
e e
i i

d d
c c
a b a b

(a) (b)

Figure 20.8: A simple way to transform continuous signals into discrete signals is
to use vector quantisation. (a) After preprocessing, a section of speech is repre-
sented by a trajectory through a high dimensional space (here depicted as three
dimensions). For example, we represent one trajectory by the dotted line. Many
different utterances of the same word will hopefully produce similar trajectories to
the mean trajectory (here shown as the solid curve). Codebook vectors are rep-
resented by circles. Points in this space on the mean trajectory that are equally
separated in time are represented by a small dot. (b) A novel trajectory (the tri-
angles) is compared to the codebook vectors so that it can be transformed into a
string, here abcdefhhhjk. Note, however, that this string does not take the time
aspect into account. To map this into a string which represents the state of the
system at equally spaced times, this would be aabcddeffhhhjjkk.

at a reasonable reading rate, occupy over two megabytes of disk space. If printed,
it would occupy around a kilobyte. There is therefore a considerable amount of
compression involved in Automatic Speech Recognition (ASR).
There are various methods of proceeding from this point, but the most fundamental
and conceptually simplest is to take a Discrete Fourier Transform (DFT) of a short
chunk of the signal, referred to as the part of the signal inside a window. Imagine
that we have a sound and something like a harp, the strings of which can resonate
to particular frequencies. For any sound whatever, each string of the harp will
resonate to some extent, as it absorbs energy at the resonant frequency from the
input sound. So we can represent the input sound by giving the amount of energy
in each frequency which the harp extracts, the so-called energy spectrum.
We take, then, some time interval, compute the fast Fourier transform (FFT) and
then obtain the power spectrum of the wave form of the speech signal in that time
interval of, perhaps, 32 msec. Then we slide the time interval, the window, down
the signal, leaving some overlap in general, and repeat. We do this for the entire
length of the signal, thus getting a sequence of perhaps ninety vectors, each vector
in dimension perhaps 256, each of the 256 components being an estimate of the
energy in some frequency interval between, say, 80 Hertz and ten KHz.
Practical problems arise from trying to sample a signal having one frequency with a
sampling rate at another; this is called ‘aliasing’ in the trade, and is most commonly
detected when the waggon wheels on the Deadwood Stage go backwards, or a news
 234

program cameraman points his camera at somebody’s computer terminal and gets
that infuriating black band drifting across the screen and the flickering that makes
the thing unwatchable. There is a risk that high frequencies in the speech signal
will be sampled at a lower frequency and will manifest themselves as a sort of
flicker. So it is usual to kill off all frequencies not being explicitly looked for,
by passing the signal through a filter which will not pass very high or very low
frequencies. Very high usually means more than half the sampling frequency, and
very low means little more than the mains frequency.
The 256 numbers may usefully be ‘binned’ into some smaller number of frequency
bands, perhaps sixteen of them, also covering the acoustic frequency range.
This approach turns the utterance into a longish sequence of vectors in represent-
ing the time development of the utterance, or more productively as a trajectory.
Many repetitions of the same word by the same speaker might reasonably be ex-
pected to be described as trajectories which are fairly close together. If we have a
family of trajectories corresponding to one person saying ‘yes’ and another family
corresponding to the same person saying ‘no’, then if we have an utterance of one
of those words by the same speaker and wish to know which it is, then some com-
parison between the new trajectory and the two families we already have, should
allow us to make some sort of decision as to which of the two words we think most
likely to have been uttered.
Put in this form, we have opened up a variant of traditional pattern recognition
which consists of distinguishing not between different categories of point in a space,
but different categories of trajectory in the space. Everything has become time
dependent; we deal with changing states.
An example of such a procedure is given in fig(20.8). There we see that a particular
speech signal has been transformed into a string of states. What we would like
to do is then, given a set of different utterances of the same word, with their
corresponding strings, to learn a representation for the transitions between the
states of these strings. This is where the HMM comes in. For each state (one of
the symbols in the string) there is a probability (that we need to learn) of either
staying in the same state (holding probability) or switching to one of the other
states. We can use standard HMM algorithms to learn such transitions.
Given then two models, say one of “yes” and the other of “no”, how do we use these
models to classify a novel utterance? The way that we do this is to find for which
of the two models was the sequence more likely. For example, imagine that we have
an utterance, “yeh”. Then we wish to find under which model is this utterance the
most likely. That is, we compare p(“yeh′′ |model“yes′′ ) with p(“yeh′′ |model‘′ no′′ ).
To calculate these likelihoods, we can use the standard marginalisation techniques
for graphical models.
A book by one of the leading speech recognition experts is available online at
http://labrosa.ee.columbia.edu/doc/HTKBook21/HTKBook.html.

BioInformatics

Biological sequences are often successfully modelled by HMMs, and have many
interesting and powerful applications in BioInformatics, for example for multiple
sequence alignment.
 235

The technical report Hidden Markov Models in Computational Biology : Appli-

cations to Protein Modelling UCSC-CRL-93-92 by Krogh et. al. is a nice intro-
duction to this area. The following book is also very good, Biological Sequence
Analysis : Probabilistic Models of Proteins and Nucleic Acids (Cambridge Uni-
versity Press, 1999) by Richard Durbin, Sean R. Eddy, Anders Krogh, Graeme
Mitchison.

Linguistic Sequences e.g. part-of-speech tagging.

See http://www.comp.lancs.ac.uk/ucrel/annotation.html#POS
for a statement of the problem and some probabilistic solutions. For example, we
have a sentence, as below, in which each word has been linguistically tagged (eg
NN is the singular common noun tag, ATI is the article tag etc.).
hospitality_NN is_BEZ an_AT excellent_JJ virtue_NN ,_,
but_CC not_XNOT when_WRB the_ATI guests_NNS have_HV
to_TO sleep_VB in_IN rows_NNS in_IN the_ATI cellar_NN !_!

One can attempt a solution to these tagging problems by using a HMM to model
the way that tag to tag transitions tend to occur from a corpus of tagged word
sequences. This forms the hidden space dynamics. An emission probability to go
from a tag to an observed word is also used, so that then for a novel sequence of
words, the most likely tag (hidden) sequence can be inferred.

Multi-electrode spike-train analysis

There are many applications of HMMs to NeuroInformatics. One area assumes

that there is a common underlying mechanism generating the observed data. This
might be say an epileptic event (the hidden variables) which is recorded on the sur-
face of the scalp by multiple electrodes (the observations). Based on the physics of
how signals get dispersed by the skull (the emission probabilty), and some smooth-
ness constraints on the underlying hidden transitions, one can make reasonable
models of neurologically significant events such that a future epileptic event can
be detected automatically.

Tracking objects through time

One of the original, and still very common applications of HMMs is in tracking.
They have been particularly successful in tracking moving objects, whereby an
understanding of newtonian dynamics in the hidden space, coupled with an un-
derstanding of how an object with a known position and momentum would appear
on the screen/radar image, enables one to infer the position and momentum of an
object based only on radar. This has obvious military applications and is one of
the reasons that some of the algorithms associated with HMMs and related mod-
els were classified until recently (although doing the inference was probably well
understood anyway!).
 236

20.3 Problems
Exercise 56 Consider a HMM with 3 states (M = 3) and 2 output symbols, with
a left-to-right state transition matrix
 
0.5 0.0 0.0
A =  0.3 0.6 0.0 
0.2 0.4 1.0
where Aij ≡ p(h(t + 1) = i|h(t) = j), an output probabilities matrix Bij ≡ p(v(t) =
i|h(t) = j)
 
0.7 0.4 0.8
B=
0.3 0.6 0.2

and an initial state probabilities vector π = (0.9 0.1 0.0)T . Given that the observed
symbol sequence is 011, compute
(i) P (v1:T )
(ii) P (h1 |v1:T ). [As there are 3 observations the HMM will have three time
slices—you are asked to compute the posterior distribution of the state vari-
able in the second time slice, numbering the times 0, 1, 2.] You can check
this calculation by setting up the HMM in JavaBayes.
(iii) Find the best hidden state sequence given a sequence of observations, and
apply it to the model (Viterbi algorithm)

Exercise 57 Suppose the matrix A above had its columns all equal to the initial
probabilities vector π. In this case the HMM reduces to a simpler model—what is
it?

Exercise 58 Show that if a transition probability aij in a HMM is set to zero

initially, then it will remain at zero throughout training.

Exercise 59 Consider the problem : Find the most likely joint output sequence
v1:T for a HMM. That is,
arg max p(v1:T )
v1:T

where
Y
p(h1:T , v1:T ) = p(vt |ht )p(ht |ht−1 )
t

(i) Explain how the above problem can be formulated as a mixed maxproduct/sumproduct
criterion.
(ii) Explain why a local message passing algorithm cannot, in general, be found
for this problem to guarantee to find the optimal solution.
(iii) Explain how to adapt the Expectation-Maximisation algorithm to form a re-
cursive algorithm, with local message passing, to guarantee at each stage of
the EM algorithm an improved joint output state.

Exercise 60 Explain how to train a HMM using EM, but with a constrained tran-
sition matrix. In particular, explain how to learn a transition matrix with a trian-
gular structure.
 237

20.4 Solutions
21 Dynamic Continuous Hiddens : Linear Dynamical Systems

Linear Gaussian State Space Models

Consider a dynamical system

ht+1 = Aht

where h is a vector, and A is a transition matrix. For example, if h is a two

dimensional vector, and A is a rotation matrix through θ degrees, then h will trace
out points on a circle through time. If we were to define a related variable, say

v(t) = [ht ]1

namely the projection of the hidden variable dynamics, then v would describe a
sinusoid through time. More generally, we could consider a model

v(t) = Bh(t)

which linearly related the visible variable v(t) to the hidden dynamics at time t.
This is therefore a linear dynamical system.
A drawback to the above models is that they are all deterministic. To account for
possible stochastic behaviour, we generalise the above to

ht = Aht−1 + ηth

vt = Bht + ηtv
where ηth and ηtv are noise vectors. As a graphical model, we write
T
Y
p(h1:T , v1:T ) = p(h1 )p(v1 |h1 ) p(ht |ht−1 )p(vt |ht )
t=2

It is computationally particularly convenient to consider Gaussian noise models,

which defines a Linear Dynamical System, also known as a Linear Gaussian State
Space model (LGSSM). In Engineering, these models are also called Kalman Fil-
ters. I prefer not to use this terminology here, since the word ‘filter’ refers to a
specific kind of inference within the LGSSM. That is, we will reserve the word
Kalman Filter for filtered inference, as we will see later.
A LGSSM is the Gaussian equivalent of a HMM. In this case, the hidden transition
and emission probabilities are assumed Gaussian. Now each hidden variable is a
multidimensional Gaussian distributed vector ht , as is the vector output vt .
 
1 1 T −1
p(ht |ht−1 ) = p exp − (ht − Aht−1 ) ΣH (ht − Aht−1 )
|2πΣH | 2

which states that ht+1 has a mean equal to Aht and has Gaussian fluctuations
described by the covariance matrix ΣH .

238
 239

h1 h2 h3 h4

v1 v2 v3 v4

Figure 21.1: A LGSSM. Both hidden and visible variables are Gaussian Distrib-
uted.

Similarly,
 
1 1 T −1
p(vt |ht ) = p exp − (vt − Bht ) ΣV (vt − Bht )
|2πΣV | 2

describes an output vt with mean Bht and covariance ΣV . At time step 1,

p(h1 ) ∼ N (µ, Σ)

The above defines a stationary LGSSM since the parameters of the model are fixed
through time. The non-stationary case allows for different parameters at each time
step, for example ΣV (t). The above definitions are for the first order LGSSM, since
the hidden state depends on only the first previous hidden state – the extension
to higher order variants, where the hidden state depends on several past hidden
states,is straightforward. We could also consider having an external known input
at each time, which will change the mean of the hidden variable. The generalisa-
tion to this case is straightforward, and left as an exercise for the interested reader.

LGSSMs are extremely popular models in temporal sequence analysis. In partic-

ular, such models have a very natural application to tracking.

21.1 Inference
Consider an observation sequence v1:T . How can we infer the marginals of the
hiddens p(ht |v1:T )?
We cannot in this case directly use the JTA, since we cannot in general pass the
table entries for a continuous distribution – there are effectively an infinite number
of table entries. However, since Gaussians are fully specified by a small set of
parameters (the sufficient statistics), namely their mean and covariance matrix,
we can alternatively pass parameters during the absorption procedure to ensure
consistency. The reader is invited to carry out this procedure (or the alternative
Belief Propagation method). Whilst this scheme is valid, the resulting recursions
are numerically complex, and may be unstable. One approach to avoid this is
to make use of the Matrix Inversion Lemma to reduce the recursions to avoid
unnecessary matrix inversions. An alternative is to use the RTS-smoothing style
scheme that we introduced in the HMM chapter.

21.1.1 The Forward Pass : The Kalman Filter

The forward pass is a Gaussian with mean ft and covariance Ft ,

p(ht |v1:t ) ∼ N (ft , Ft )

 240

We can find the joint distribution p(ht , vt |v1:t−1 ), and then condition on vt to easily
find the distribution p(ht |v1:t ). The term p(ht , vt |v1:t−1 ) is a Gaussian and can be
found easily using the relations

vt = Bht + η v , ht = Aht−1 + η h

Using the above, we readily find


 T v
∆vt ∆vtT |v1:t−1 = B ∆ht ∆hT
t |v1:t−1 B + Σ


 T h
∆ht ∆hT T
t |v1:t−1 = A ∆ht−1 ∆ht−1 |v1:t−1 A (st ) + Σ



∆vt ∆hT T
t |v1:t−1 = B(st ) ∆ht ∆ht |v1:t−1

hvt |v1:t−1 i = BA hht−1 |v1:t−1 i , hht |v1:t−1 i = A hht−1 |v1:t−1 i

In the above, using our moment representation of the forward messages


hht−1 |v1:t−1 i ≡ ft−1 , ∆ht−1 ∆hT t−1 |v1:t−1 ≡ Ft−1

Then, using conditioning1 p(ht |vt , v1:t−1 ) will have mean


−1
ft ≡ hht |v1:t−1 i + ∆ht ∆vtT |v1:t−1 ∆vt ∆vtT |v1:t−1 (vt − hvt |v1:t−1 i)

and covariance



−1

Ft ≡ ∆ht ∆hT T
t |v1:t−1 − ∆ht ∆vt |v1:t−1 ∆vt ∆vtT |v1:t−1 ∆vt ∆hT
t |v1:t−1

A nice thing about the above approach is that we work always in the moment
representation, and the iteration is expected to be numerically stable when the
noise covariances are small. This procedure is called the Forward Pass in the
LGSSM inference algorithm (albeit with a change to the standard notation in the
literature for representing the filtered posterior).

21.1.2 The Kalman Smoother : The Rauch-Tung-Striebel Smoother

In principle, we can apply the Belief Propagation method to form a backpass to find
p(ht |v1:T ) (see barberieee). However,we would like to avoid defining λ messages
here since it is awkward to extend BP to the SKF case. Here we show how for
the simple case of the Kalman Filter, how a smoothing backpass can be formed
without defining λ messages. Instead we form directly a recursion for the smoothed
distribution p(ht |v1:T ).
Imagine that we have completed a forward pass, so that we have, for the KF,
the filtered distributions p(ht |v1:t ). We can form a recursion for the smoothed
posteriors p(ht |v1:T ), directly without using λ recursions as follows:
X
p(ht |v1:T ) ∝ p(ht |v1:T , ht+1 )p(ht+1 |v1:T ) (21.1.1)
ht+1
X
∝ p(ht |v1:t , ht+1 )p(ht+1 |v1:T ) (21.1.2)
ht+1

The term p(ht |v1:t , ht+1 ) can be found by conditioning the joint distribution p(ht , ht+1 |v1:t ) =
p(ht+1 |ht )p(ht |v1:t ). We can work out this joint distribution in the usual manner
1 p(x|y) is a Gaussian with mean µx + Σxy Σ−1 −1
yy (y − µy ) and covariance Σxx − Σxy Σyy Σyx .
 241

by finding its mean and covariance. The term p(ht |v1:t ) is a known Gaussian from
the Forward Pass with mean ft and covariance Ft . Hence the joint distribution
p(ht , ht+1 |v1:t ) has means

hht |v1:t i = ft , hht+1 |v1:t i = Aft

and covariance elements



∆ht ∆hTt |v1:t = Ft , ∆ht ∆hT
t+1 |v1:t = Ft A
T
(21.1.3)

 h
∆ht+1 ∆hT T
t+1 |v1:t = AFt A + Σ (21.1.4)

To find the conditional distribution p(ht |v1:t , ht+1 ), we use the conditioned Gaussian
results which says that the conditional mean will be


−1
hht |v1:t i + ∆ht ∆hT
t+1 |v1:t ∆ht+1 ∆hT t+1 |v1:t (ht+1 − hht+1 |v1:t i)

and conditional covariance will be

←−


−1
Σ t ≡ ∆ht ∆hT T
t |v1:t − ∆ht ∆ht+1 |v1:t ∆ht+1 ∆hT
t+1 |v1:t hht+1 |v1:t i

From this we can easily write

←
−
p(ht |v1:t , ht+1 ) ≡ ht = A t ht+1 + ←
− +←
m t
−
ηt

where
←−

−1
A t ≡ ∆ht ∆hT
t+1 |v1:t ∆ht+1 ∆hT
t+1 |v1:t

←
− ≡ hh |v i −
∆h ∆hT |v 
∆h ∆hT |v −1 hh |v i
m t t 1:t t t+1 1:t t+1 t+1 1:t t+1 1:t
←
−
and ←
−
η t ∼ N (0, Σ t ). Then p(ht |v1:T ) is a Gaussian distribution with mean
←
− − ≡←−
gt ≡ hht |v1:T i = A t hht+1 |v1:T i + ←
m t A t gt+1 + ←
−
m t

and covariance

 ←−
←
−T ← − ←
− ←−T ← −
Gt ≡ ∆ht ∆hT T
t |v1:T = A t ∆ht+1 ∆ht+1 |v1:T A t + Σ t ≡ A t Gt+1 A t + Σ t

In this way, we directly find the smoothed posterior without defining the problem-
atic λ messages. This procedure is equivalent to the Rauch-Tung-Striebel Kalman
smoother[40]. A key trick was dynamics reversal. This is sometimes called a ‘cor-
rection’ method since it takes the filtered estimate p(ht |v1:t ) and ‘corrects’ it to
form a smoothed estimate p(ht |v1:T ).
This procedure is called the Backward Pass in the LGSSM inference algorithm
(albeit with a change to the standard notation in the literature for representing
the smoothed posterior).

The cross moment

An advantage of the probabilistic interpretation given above is that the cross mo-
ment, which is required for learning is given by

 ←−−− T T
 T
ht−1 hT
t p(h ,h |v )
= A t−1 Pt + ĥ t ĥTt
t−1 t 1:T

This is far simpler than common expressions found in the literature.

 242

21.1.3 The Likelihood

The likelihood p(v1:T ) is often required. To compute this, the simplest way is to
use the recursion:

p(v1:t+1 ) = p(vt+1 |v1:t )p(v1:t )

Clearly, p(vt+1 |v1:t ) will be a Gaussian in vt+1 . It is straightforward to show that

this has mean and covariance (for t > 1)


µt ≡ BAft , Σt ≡ B AFt AT + Σh B T + Σv

Then the log likelihood is given by

T
X 1 T 1
− (vt − µt ) Σ−1
t (vt − µt ) − log det (2πΣt )
t=1
2 2

where, at time 1,

µ1 ≡ Bµ, Σ1 ≡ BΣB T + Σv

MAP vs Marginal

In general, we have seen that there is a difference between the most probable joint
posterior state, and the joint posterior mean. However, in the case of Gaussians,
there is no difference between these two. The interested reader is invited to show
formally that the most likely state of a Gaussian is its mean. Hence, in order to
infer the most likely hidden state, this is equivalent to finding the marginal – that
is, the mean of the hidden variables.

21.2 EM Algorithm for Learning

A straightforward application of the EM algorithm yields the following updates:
1 X T
 T
Σnew
V = vt vt − vt hht iT B T − B hht i vtT + B ht hT
t B
T t
1 X


 T
 T

Σnew
H = ht+1 hT T T
t+1 − A ht ht+1 − ht+1 ht A + A ht ht A
T
T −1 t
µnew = hh1 i


Σnew
π = h1 hT 1 − µµ
T

T −1 T −1
!−1
X
 X

Anew = ht+1 hT
t ht hT
t
t=1 t=1
T T
!−1
X T
X

new
B = vt hht i ht hT
t
t=1 t=1

If B is updated according to the above, the reader may show that the first equation
can be simplified to
1 X T T

Σnew
V = vt vt − vt hht i B T
T t
 243

Algorithm 1 LGSSM: Forward and Backward Recursive Updates. The filtered

posterior p(ht |v1:t ) is returned with means ĥtt and covariances Ptt . The smoothed
posterior p(ht |v1:T ) means and covariances are ĥTt and PtT .
procedure Forward
ĥ01 ← µ
K ← ΣB T (BΣB T + ΣV )−1
P11 ← (I − KB)Σ
ĥ11 ← ĥ01 + K(vt − B ĥ01 )
for t ← 2, T do
Ptt−1 ← APt−1 t−1 T
A + ΣH
t−1 t−1
ĥt ← Aĥt−1
K ← P B T (BPtt−1 B T + ΣV )−1
Ptt ← (I − KB)Ptt−1
ĥtt ← ĥt−1
t + K(vt − B ĥt−1
t )
end for
end procedure
procedure Backward
for t ← T − 1, 1 do
←−
At ← Ptt AT (Pt+1 t
)−1
T t ←− T t ←
−
Pt ← Pt + At (Pt+1 − Pt+1 )At T
←−
ĥTt ← ĥtt + At (ĥT t
t+1 − Aĥt )
end for
end procedure

Similarly, if A is updated according to EM algorithm, then the second equation

can be simplified to
1 X


Σnew
H = ht+1 hT T
t+1 − A ht ht+1
T −1 t

Dealing with restricted forms of the matrices is also easy to deal with. For example,
it may be that one wishes to search for independent generating processes, in which
case A will have a block diagonal structure. This restriction is easy to deal with
and left as an exercise for the reader.
The last two equations are solved by Gaussian Elimination. The averages in the
above equations are the posterior averages conditioned on the visible variables –
these are given by the Kalman Smoother routine.
The extension of learning to multiple time series is straightforward and left as an
exercise for the reader.

An example : Simple Trajectory Analysis

A toy rocket is launched in the air. The rocket has unknown mass and initial
velocity. In addition, the constant accelerations from rocket’s propulsion system
are unknown. What is known is that the Newton’s laws apply. An instrument
can measure the vertical height and horizontal distance of the rocket at each time
x(t), y(t) from the origin. Based on noisy measurements of x(t) and y(t), our task
is to infer the position of the rocket at each time – a trajectory analysis.
 244

Newton’s law states that

d2 fx d2 fy
x= , y=
dt2 m dt2 m
where m is the mass of the object, and f is the (constant) vertical force applied
(gravity in this case) Hence
dx fx dy fy
= t + a, =t +b
dt m dt m
fx fy
x = t2 + at + c, y = t2 + bt + d
2m 2m
As they stand, these equations are not in a form directly usable in the LGSSM
framework. There are several ways to rewrite them to make them suitable – here
we choose a very naive approach. First, we reparameterise time to use the variable
t̃ such that t ≡ t̃∆

x((t̃ + 1)∆) = x(t̃∆) + ∆x′ (t̃∆)

y((t̃ + 1)∆) = y(t̃∆) + ∆y ′ (t̃∆)

dy
where y ′ (t) ≡ dt . We can write an update equation for the x′ and y ′ as

x′ ((t̃ + 1)∆) = x′ (t̃∆) + fx ∆/m, y ′ ((t̃ + 1)∆) = y ′ (t̃∆) + fy ∆/m

These equations are then discrete time difference equations indexed by t̃. However,
the instrument which measures x(t) and y(t) is not completely accurate. What is
actually measured is x̂(t) and ŷ(t), which are noisy versions of x(t) and y(t). For
simplicity, we relabel ax (t) = fx (t)/m(t), ay (t) = fy (t)/m(t) – these accelerations
will be assumed to be roughly constant, but unknown :

ax ((t̃ + 1)∆) = ax (t̃∆) + ηx

where ηx is a very small noise term. The prior for ax is chosen to be vague – a
zero mean Gaussian with large variance. A similar equation holds for the ay . (Of
course, another approach would be to assume strictly constant accelerations and
learn them).

One way to describe the above approach is to consider x(t), y(t), x′ (t), y ′ (t), ax (t)
and ay (t) as hidden variables. We can put a large variance prior on their initial
values, and attempt to infer the unknown trajectory. A simple demonstration for
this is given in fig(24.3), for which the code is given in the text. It is pleasing
how well the Kalman Filter infers the object trajectory despite the large amount
of measurement noise.

21.3 Problems
Exercise 61 A scaler Rth order Autoregressive Model (AR) model is defined as
R
X
vt+1 = ai vt−i + ηt+1
i=1

where vt is a scalar, and η is Gaussian noise. Explain how to formulate an Rth

order AR model as a first order LGSSM.
 245

3000

2500

2000

1500

y
1000

500

−500

−1000
−2000 0 2000 4000 6000 8000 10000
x

Figure 21.2: Kalman Smoother estimate of a trajectory based on noisy observa-

tions. The small points are the noisy observations (which have also a time label).
The “x” points are the true positions of the object, and the crosses are the esti-
mated positions of the object plotted every several time steps. The estimates are
plotted with standard errors around their mean values.

Exercise 62 • Explain how to model a sinusoid, rotating with angular velocity

ω using a two-dimensional LGSSM.
• Explain how to model a sinuoid using an AR model.
• Explain the relationship between the second order differential equation ẍ =
−λx, which describes a Harmonic Oscillator, and the second order difference
equation which approximates this differential equation. Is it possible to find
a difference equation which exactly matches the solution of the differential
equation at chosen points?

21.4 Solutions
39
22 Switching Linear Dynamical Systems

The Linear Dynamical System (LDS), chapter(21) is a key temporal model in

which a latent linear process generates the observed series. For more complex
time-series which are not well described globally by a single LDS, we may break
the time-series into segments, each modelled by a potentially different LDS. This
is the basis for the Switching LDS (SLDS) where, for each time t, a switch variable
st ∈ 1, . . . , S describes which of the LDSs is to be used1 . The observation (or
‘visible’) vt ∈ RV is linearly related to the hidden state ht ∈ RH by

vt = B(st )ht + η v (st ), η v (st ) ∼ N (v̄(st ), Σv (st )) (22.0.1)

where N (µ, Σ) denotes a Gaussian distribution with mean µ and covariance Σ.

The transition dynamics of the continuous hidden state ht is linear,


ht = A(st )ht−1 + η h (st ), η h (st ) ∼ N h̄(st ), Σh (st ) (22.0.2)

The switch variable st itself is Markovian, with transition p(st |st−1 ).

Here we consider the more general ‘augmented’ model in which the switch st is
dependent on both the previous st−1 and ht−1 . An equivalent probabilistic model
is2 (see fig(22.1))
T
Y
p(v1:T , h1:T , s1:T ) = p(vt |ht , st )p(ht |ht−1 , st )p(st |ht−1 , st−1 )
t=1

with


p(vt |ht , st ) = N (v̄(st ) + B(st )ht , Σv (st )) , p(ht |ht−1 , st ) = N h̄(st ) + A(st )ht , Σh (st )

At time t = 1, p(s1 |h0 , s0 ) simply denotes the prior p(s1 ), and p(h1 |h0 , s1 ) denotes
p(h1 |s1 ).
The SLDS is used in many disciplines, from econometrics to machine learning
[41, 42, 43, 44, 45, 46]. The aSLDS has been used, for example, in state-duration
modelling in acoustics [47] and econometrics [48]. See [49] and [50] for recent
reviews of work.
The SLDS can be thought of as a marriage between a Hidden Markov Model and
a Linear Dynamical system. Each of these two models are tractable. However, the
SLDS is computationally intractable, and requires specialised approximations.
1 These systems also go under the names Jump Markov model/process, switching Kalman Filter,
Switching Linear Gaussian State Space models, Conditional Linear Gaussian Models.
2 The notation x
1:T is shorthand for x1 , . . . , xT .

246
 247

s1 s2 s3 s4

h1 h2 h3 h4

v1 v2 v3 v4

Figure 22.1: The independence structure of the aSLDS. Square nodes denote dis-
crete variables, round nodes continuous variables. In the SLDS links from h to s
are not normally considered.

Inference

We consider here the filtered estimate p(ht , st |v1:t ) and the smoothed estimate
p(ht , st |v1:T ), for any 1 ≤ t ≤ T . Both filtered and smoothed inference in the
SLDS is intractable, scaling exponentially with time [49]. To see this informally,
consider the filtered posterior, which may be recursively computed using
XZ
p(st , ht |v1:t ) = p(st , ht |st−1 , ht−1 , vt )p(st−1 , ht−1 |v1:t−1 ) (22.0.3)
st−1 ht−1

At timestep 1, p(s1 , h1 |v1 ) = p(h1 |s1 , v1 )p(s1 |v1 ) is an indexed set of Gaussians.
At timestep 2, due to the summation over the states s1 , p(s2 , h2 |v1:2 ) will be an
indexed set of S Gaussians; similarly at timestep 3, it will be S 2 and, in general,
gives rise to S t Gaussians.

22.1 Expectation Correction

EC mirrors the Rauch-Tung-Striebel ‘correction’ smoother for the LDS [40, 41]
presented in chapter(21). The correction approach consists of a single forward pass
to recursively find the filtered posterior p(ht , st |v1:t ), followed by a single backward
pass to correct this into a smoothed posterior p(ht , st |v1:T ). The forward pass is
equivalent to standard Assumed Density Filtering [51].

22.1.1 Forward Pass (Filtering)

Readers familiar with Assumed Density Filtering may wish to continue directly
to section (22.1.3). Our aim is to form a recursion for p(st , ht |v1:t ), based on a
Gaussian mixture approximation3 of p(ht |st , v1:t ). Without loss of generality, we
may decompose the filtered posterior as

p(ht , st |v1:t ) = p(ht |st , v1:t )p(st |v1:t ) (22.1.1)

The exact representation of p(ht |st , v1:t ) is a mixture with a O(S t ) components.
We therefore approximate this with a smaller I-component mixture
I
X
p(ht |st , v1:t ) ≈ p(ht |it , st , v1:t )p(it |st , v1:t )
it =1

3 This derivation holds also for the aSLDS, unlike that presented in [52].
 248

where p(ht |it , st , v1:t ) is a Gaussian parameterised with mean4 f (it , st ) and covari-
ance F (it , st ). To find a recursion for these parameters, consider
X
p(ht+1 |st+1 , v1:t+1 ) = p(ht+1 , st , it |st+1 , v1:t+1 )
st ,it
X
= p(ht+1 |st , it , st+1 , v1:t+1 )p(st , it |st+1 , v1:t+1 )
st ,it
(22.1.2)

Evaluating p(ht+1 |st , it , st+1 , v1:t+1 )

We find p(ht+1 |st , it , st+1 , v1:t+1 ) from the joint distribution p(ht+1 , vt+1 |st , it , st+1 , v1:t ),
which is a Gaussian with covariance and mean elements5
Σhh = A(st+1 )F (it , st )AT (st+1 ) + Σh (st+1 ),
Σvv = B(st+1 )Σhh B T (st+1 ) + Σv (st+1 )
Σvh = B(st+1 )F (it , st )
µv = B(st+1 )A(st+1 )f (it , st )
µh = A(st+1 )f (it , st ) (22.1.3)
These results are obtained from integrating the forward dynamics, Equations
(22.0.1,22.0.2) over ht , using the results in Appendix (G.2). To find p(ht+1 |st , it , st+1 , v1:t+1 )
we may then condition p(ht+1 , vt+1 |st , it , st+1 , v1:t ) on vt+1 using the results in Ap-
pendix (G.1).

Evaluating p(st , it |st+1 , v1:t+1 )

Up to a trivial normalisation constant the mixture weight in equation (22.1.2) can

be found from the decomposition

p(st , it |st+1 , v1:t+1 ) ∝ p(vt+1 |it , st , st+1 , v1:t )p(st+1 |it , st , v1:t )p(it |st , v1:t )p(st |v1:t )
(22.1.4)
The first factor in equation (22.1.4), p(vt+1 |it , st , st+1 , v1:t ) is given as a Gaussian
with mean µv and covariance Σvv , as given in equation (22.1.3). The last two
factors p(it |st , v1:t ) and p(st |v1:t ) are given from the previous iteration. Finally,
p(st+1 |it , st , v1:t ) is found from
p(st+1 |it , st , v1:t ) = hp(st+1 |ht , st )ip(ht |it ,st ,v1:t ) (22.1.5)

where h·ip denotes expectation with respect to p. In the standard SLDS, equation
(22.1.5) is replaced by the Markov transition p(st+1 |st ). In the aSLDS, however,
equation (22.1.5) will generally need to be computed numerically. A simple ap-
proximation is to evaluate equation (22.1.5) at the mean value of the distribution
p(ht |it , st , v1:t ). To take covariance information into account an alternative would
be to draw samples from the Gaussian p(ht |it , st , v1:t ) and thus approximate the
average of p(st+1 |ht , st ) by sampling6 .
4 Strictly speaking, we should use the notation ft (it , st ) since, for each time t, we have a set of
means indexed by it , st . This mild abuse of notation is used elsewhere in the paper.
5 We derive this for h̄
t+1 , v̄t+1 ≡ 0, to ease notation.
6 Whilst we suggest sampling as part of the aSLDS update procedure, this does not equate this

with a sequential sampling procedure, such as Particle Filtering. The sampling here is a form of
exact sampling, for which no convergence issues arise, being used only to numerically compute
equation (22.1.5).
 249

Closing the recursion

We are now in a position to calculate equation (22.1.2). For each setting of the
variable st+1 , we have a mixture of I × S Gaussians which we numerically collapse
back to I Gaussians to form
I
X
p(ht+1 |st+1 , v1:t+1 ) ≈ p(ht+1 |it+1 , st+1 , v1:t+1 )p(it+1 |st+1 , v1:t+1 )
it+1 =1

Any method of choice may be supplied to collapse a mixture to a smaller mixture.

A straightforward approach that we use in our code is based on repeatedly merging
low-weight components, as explained in section (22.1.2). In this way the new
mixture coefficients p(it+1 |st+1 , v1:t+1 ), it+1 ∈ 1, . . . , I are defined.
The above completes the description of how to form a recursion for p(ht+1 |st+1 , v1:t+1 )
in equation (22.1.1). A recursion for the switch variable is given by
X
p(st+1 |v1:t+1 ) ∝ p(st+1 , it , st , vt+1 , v1:t )
it ,st

The r.h.s. of the above equation is proportional to

X
p(vt+1 |st+1 , it , st , v1:t )p(st+1 |it , st , v1:t )p(it |st , v1:t )p(st |v1:t )
st ,it

where all terms have been computed during the recursion for p(ht+1 |st+1 , v1:t+1 ).

The Likelihood p(v1:T )

The likelihood p(v1:T ) may be found by recursing p(v1:t+1 ) = p(vt+1 |v1:t )p(v1:t ),
where
X
p(vt+1 |vt ) = p(vt+1 |it , st , st+1 , v1:t )p(st+1 |it , st , v1:t )p(it |st , v1:t )p(st |v1:t )
it ,st ,st+1

In the above expression, all terms have been computed in forming the recursion
for the filtered posterior p(ht+1 , st+1 |v1:t+1 ).

The procedure for computing the filtered posterior is presented in 2.??

22.1.2 Collapsing Gaussians

The user may provide any algorithm of their choice for collapsing a set of Gaussians
to a smaller set of Gaussians [53]. Here, to be explicit, we present a simple one
which is fast, but has the disadvantage that no spatial information about the
mixture is used.
First, we describe how to collapse
Pa mixture to a single Gaussian: We may collapse
aPmixture of Gaussians P p(x) = i pi N (x|µ
i , ΣiT) to a single Gaussian with mean
T
i pi µi and covariance i pi Σi + µi µi − µµ .

To collapse a mixture to a K-component mixture we retain the K − 1 Gaussians

with the largest mixture weights – the remaining N − K Gaussians are simply
merged to a single Gaussian using the above method. The alternative of recur-
sively merging the two Gaussians with the lowest mixture weights gave similar
experimental performance.
 250

Algorithm 2 aSLDS Forward P Pass. Approximate the filtered posterior

p(st |v1:t ) ≡ ρt , p(ht |st , v1:t ) ≡ it wt (it , st )N (ft (it , st ), Ft (it , st )). Also we return
the approximate log-likelihood log p(v1:T ). We require I1 = 1, I2 ≤ S, It ≤ S×It−1 .
θ(s) = A(s), B(s), Σh (s), Σv (s), h̄(s), v̄(s).
for s1 ← 1 to S do
{f1 (1, s1 ), F1 (1, s1 ), p̂} = LDSFORWARD(0, 0, v1 ; θ(s1 ))
ρ1 ← p(s1 )p̂
end for
for t ← 2 to T do
for st ← 1 to S do
for i ← 1 to It−1 , and s ← 1 to S do
{µx|y (i, s), Σx|y (i, s), p̂} = LDSFORWARD(ft−1 (i, s), Ft−1 (i, s), vt ; θ(st ))
p∗ (st |i, s) ≡ hp(st |ht−1 , st−1 = s)ip(ht−1 |it−1 =i,st−1 =s,v1:t−1 )
p′ (st , i, s) ← wt−1 (i, s)p∗ (st |i, s)ρt−1 (s)p̂
end for
Collapse the It−1 × S mixture of Gaussians defined by µx|y ,Σx|y ,
and weights p(i, s|st ) ∝ p′ (st , i, s) to a Gaussian with It compo-
PI
nents, p(ht |st , v1:t ) ≈ itt=1 p(it |st , v1:t )p(ht |st , it , v1:t ). This defines
the new means ft (it , st ), covariances Ft (it , st ) and mixture weights
wt (it , st ) ≡ p(itP
|st , v1:t ).
Compute ρt (st ) ∝ i,s p′ (st , i, s)
end for
normalise ρt P
L ← L + log st ,i,s p′ (st , i, s)
end for

More sophisticated methods which retain some spatial information would clearly
be potentially useful. The method presented in [43] is a suitable approach which
considers removing Gaussians which are spatially similar (and not just low-weight
components), thereby retaining a sense of diversity over the possible solutions.

22.1.3 Backward Pass (Smoothing)

The main difficulty is to find a suitable way to ‘correct’ the filtered posterior
p(st , ht |v1:t ) obtained from the forward pass into a smoothed posterior p(st , ht |v1:T ).
We initially derive this for the case of a single Gaussian representation. The ex-
tension to the mixture case is straightforward and is given in section (22.1.5). Our
derivation holds for both the SLDS and aSLDS. We approximate the smoothed
posterior p(ht |st , v1:T ) by a Gaussian with mean g(st ) and covariance G(st ), and
our aim is to find a recursion for these parameters. A useful starting point for a
recursion is:
X
p(ht , st |v1:T ) = p(st+1 |v1:T )p(ht |st , st+1 , v1:T )p(st |st+1 , v1:T )
st+1
 251

The term p(ht |st , st+1 , v1:T ) may be computed as

Z
p(ht |st , st+1 , v1:T ) = p(ht , ht+1 |st , st+1 , v1:T )
ht+1
Z
= p(ht |ht+1 , st , st+1 , v1:T )p(ht+1 |st , st+1 , v1:T )
ht+1
Z
= p(ht |ht+1 , st , st+1 , v1:t )p(ht+1 |st , st+1 , v1:T )
ht+1
(22.1.6)

The recursion therefore requires p(ht+1 |st , st+1 , v1:T ), which we can write as

p(ht+1 |st , st+1 , v1:T ) ∝ p(ht+1 |st+1 , v1:T )p(st |st+1 , ht+1 , v1:t ) (22.1.7)

The difficulty here is that the functional form of p(st |st+1 , ht+1 , v1:t ) is not squared
exponential in ht+1 , so that p(ht+1 |st , st+1 , v1:T ) will not be Gaussian. One
possibility would be to approximate the non-Gaussian p(ht+1 |st , st+1 , v1:T ) by a
Gaussian (or mixture thereof) by minimising the Kullback-Leilbler divergence be-
tween the two, or performing moment matching in the case of a single Gaussian. A
simpler alternative is to make the assumption p(ht+1 |st , st+1 , v1:T ) ≈ p(ht+1 |st+1 , v1:T ),
see fig(22.2). This makes life easy since p(ht+1 |st+1 , v1:T ) is already known from
the previous backward recursion. Under this assumption, the recursion becomes
X
p(ht , st |v1:T ) ≈ p(st+1 |v1:T )p(st |st+1 , v1:T ) hp(ht |ht+1 , st , st+1 , v1:t )ip(ht+1 |st+1 ,v1:T )
st+1

(22.1.8)

The procedure resulting from the conditional independence assumption is called

‘standard’ EC. Equation (22.1.8) forms the basis of the standard EC backward
pass. How to implement the recursion for the continuous and discrete factors is
detailed below7 .

Evaluating hp(ht |ht+1 , st , st+1 , v1:t )ip(ht+1 |st+1 ,v1:T )

hp(ht |ht+1 , st , st+1 , v1:t )ip(ht+1 |st+1 ,v1:T ) is a Gaussian in ht , whose statistics we will
now compute. First we find p(ht |ht+1 , st , st+1 , v1:t ) which may be obtained from
the joint distribution

p(ht , ht+1 |st , st+1 , v1:t ) = p(ht+1 |ht , st+1 )p(ht |st , v1:t ) (22.1.9)

which itself can be found from a forward dynamics from the filtered estimate
p(ht |st , v1:t ). The statistics for the marginal p(ht |st , st+1 , v1:t ) are simply those of
p(ht |st , v1:t ), since st+1 carries no extra information about ht 8 . The only remaining
7 Equation (22.1.8) has the pleasing form of an RTS backpass for the continuous part (analogous
to LDS case), and a discrete smoother (analogous to a smoother recursion for the HMM). In
the standard Forward-Backward algorithm for the HMM [37], the posterior γt ≡ p(st |v1:T )
is formed from the product of αt ≡ p(st |v1:t ) and βt ≡ p(vt+1:T |st ). This approach is also
analogous to EP [38]. In the correction approach, a direct recursion for γt in terms of γt+1
and αt is formed, without explicitly defining βt . The two approaches to inference are known
as α − β and α − γ recursions.
8 Integrating over h
t+1 means that the information from st+1 passing through ht+1 via the term
p(ht+1 |st+1 , ht ) vanishes. Also, since st is known, no information from st+1 passes through st
to ht .
 252

st−1 st st+1 st+2

ht−1 ht ht+1 ht+2

vt−1 vt vt+1 vt+2

Figure 22.2: The EC backpass approximates p(ht+1 |st+1 , st , v1:T ) by

p(ht+1 |st+1 , v1:T ). Motivation for this is that st only influences ht+1 through ht .
However, ht will most likely be heavily influenced by v1:t , so that not knowing the
state of st is likely to be of secondary importance. The green (darker) node is the
variable we wish to find the posterior state of. The yellow (lighter shaded) nodes
are variables in known states, and the hashed node a variable whose state is indeed
known but assumed unknown for the approximation.

uncomputed statistics are the mean of ht+1 , the covariance of ht+1 and cross-
variance between ht and ht+1 , which are given by

hht+1 i = A(st+1 )ft (st )

Σt+1,t+1 = A(st+1 )Ft (st )AT (st+1 ) + Σh (st+1 ), Σt+1,t = A(st+1 )Ft (st )

Given the statistics of equation (22.1.9), we may now condition on ht+1 to find
p(ht |ht+1 , st , st+1 , v1:t ). Doing so effectively constitutes a reversal of the dynamics,
←
−
ht = A (st , st+1 )ht+1 + ←
−
η (st , st+1 )
←
− − , s ), ← −
where A and ← −
η (st , st+1 ) ∼ N (←m(s t t+1 Σ (st , st+1 )) are easily found using the
conditioned Gaussian results in Appendix (G.1). Averaging the above reversed dy-
namics over p(ht+1 |st+1 , v1:T ), we find that hp(ht |ht+1 , st , st+1 , v1:t )ip(ht+1 |st+1 ,v1:T )
is a Gaussian with statistics
←
− − , s ), Σ = ←− ←− ←−
µt = A (st , st+1 )g(st+1 )+←
m(s t t+1 t,t A (st , st+1 )G(st+1 ) A T (st , st+1 )+ Σ (st , st+1 )

These equations directly mirror the standard RTS backward pass.

Evaluating p(st |st+1 , v1:T )

The main departure of EC from related methods is in treating the term

p(st |st+1 , v1:T ) = hp(st |ht+1 , st+1 , v1:t )ip(ht+1 |st+1 ,v1:T ) (22.1.10)

The term p(st |ht+1 , st+1 , v1:t ) is given by

p(ht+1 |st+1 , st , v1:t )p(st , st+1 |v1:t )

p(st |ht+1 , st+1 , v1:t ) = P ′ ′ (22.1.11)
s′ p(ht+1 |st+1 , st , v1:t )p(st , st+1 |v1:t )
t

Here p(st , st+1 |v1:t ) = p(st+1 |st , v1:t )p(st |v1:t ), where p(st+1 |st , v1:t ) occurs in the
forward pass, equation (22.1.5). In equation (22.1.11), p(ht+1 |st+1 , st , v1:t ) is found
by marginalising equation (22.1.9).
Computing the average of equation (22.1.11) with respect to p(ht+1 |st+1 , v1:T ) may
be achieved by any numerical integration method desired. The simplest approxi-
 253

mation is to evaluate the integrand at the mean value of the averaging distribution9
p(ht+1 |st+1 , v1:T ). Otherwise, sampling from the Gaussian p(ht+1 |st+1 , v1:T ), has
the advantage that covariance information is used10 .

Closing the Recursion

We have now computed both the continuous and discrete factors in equation
(22.1.8), which we wish to use to write the smoothed estimate in the form p(ht , st |v1:T ) =
p(st |v1:T )p(ht |st , v1:T ). The distribution p(ht |st , v1:T ) is readily obtained from the
joint equation (22.1.8) by conditioning on st to form the mixture
X
p(ht |st , v1:T ) = p(st+1 |st , v1:T )p(ht |st , st+1 , v1:T )
st+1

which may be collapsed to a single Gaussian (or mixture if desired). The smoothed
posterior p(st |v1:T ) is given by
X
p(st |v1:T ) = p(st+1 |v1:T )p(st |st+1 , v1:T )
st+1
X
= p(st+1 |v1:T ) hp(st |ht+1 , st+1 , v1:t )ip(ht+1 |st+1 ,v1:T ) . (22.1.12)
st+1

Numerical Stability

Numerical stability is a concern even in the LDS, and the same is to be expected for
the aSLDS. Since the standard LDS recursions LDSFORWARD and LDSBACKWARD
are embedded within the EC algorithm, we may immediately take advantage of
the large body of work on stabilizing the LDS recursions, such as the Joseph or
square root forms [54].

22.1.4 Remarks

The standard-EC Backpass procedure is closely related to Kim’s method [55, 45].
In both standard-EC and Kim’s method, the approximation
p(ht+1 |st , st+1 , v1:T ) ≈ p(ht+1 |st+1 , v1:T ), is used to form a numerically simple
backward pass. The other ‘approximation’ in EC is to numerically compute the
average in equation (22.1.12). In Kim’s method, however, an update for the dis-
crete variables is formed by replacing the required term in equation (22.1.12) by

hp(st |ht+1 , st+1 , v1:t )ip(ht+1 |st+1 ,v1:T ) ≈ p(st |st+1 , v1:t ) (22.1.13)

This approximation11 decouples the discrete backward pass in Kim’s method from
the continuous dynamics, since p(st |st+1 , v1:t ) ∝ p(st+1 |st )p(st |v1:t )/p(st+1 |v1:t )
9 Replacing ht+1 by its mean gives the simple approximation
1 T −1
1 e− 2 zt+1 (st ,st+1 )Σ (st ,st+1 |v1:t )zt+1 (st ,st+1 )
hp(st |ht+1 , st+1 , v1:t )ip(ht+1 |st+1 ,v1:T ) ≈ p p(st |st+1 , v1:t )
Z det Σ(st , st+1 |v1:t )
where zt+1 (st , st+1 ) ≡ hht+1 |st+1 , v1:T i − hht+1 |st , st+1 , v1:t i and Z ensures normalisation
over st . Σ(st , st+1 |v1:t ) is the filtered covariance of ht+1 given st , st+1 and the observations
v1:t , which may be taken from Σhh in equation (22.1.3).
10 This is a form of exact sampling since drawing samples from a Gaussian is easy. This should

not be confused with meaning that this use of sampling renders EC a sequential Monte-Carlo
sampling scheme.
11 In the HMM, this is exact, but in the SLDS the future observations carry information about

st .
 254

Algorithm 3 aSLDS: EC Backward Pass. Approximates p(st |v1:T ) and

PJt
p(ht |st , v1:T ) ≡ jt =1 u t (jt , s t )N (g t (jt , s t ), Gt (jt , s t )) using a mixture of Gaus-
sians. JT = IT , Jt ≤ S × It × Jt+1 . This routine needs the results from 2.??
GT ← FT , gT ← fT , uT ← wT
for t ← T − 1 to 1 do
for s ← 1 to S, s′ ← 1 to S, i ← 1 to It , j ′ ← 1 to Jt+1 do
(µ, Σ)(i, s, j ′ , s′ ) = LDSBACKWARD(gt+1 (j ′ , s′ ), Gt+1 (j ′ , s′ ), ft (i, s), Ft (i, s), θ(s′ ))
p(i, s|j ′ , s′ ) = hp(st = s, it = i|ht+1 , st+1 = s′ , jt+1 = j ′ , v1:t )ip(ht+1 |st+1 =s′ ,jt+1 =j ′ ,v1:T )
p(i, s, j ′ , s′ |v1:T ) ← p(st+1 = s′ |v1:T )ut+1 (j ′ , s′ )p(i, s|j ′ , s′ )
end for
for st ← 1 to S do
Collapse the mixture defined by weights p(it = i, st+1 = s′ , jt+1 =
j ′ |st , v1T ) ∝ p(i, st , j ′ , s′ |v1:T ), means µ(it , st , jt+1 , st+1 ) and covari-
ances Σ(it , st , jt+1 , st+1 ) to a mixture with Jt components. This
defines the new means gt (jt , st ), covariances Gt (jt , st ) and mixture
weights utP (jt , st ).
p(st |v1:T ) ← it ,j ′ ,s′ p(it , st , j ′ , s′ |v1:T )
end for
end for

can be computed simply from the filtered results alone. The fundamental differ-
ence therefore between EC and Kim’s method is that the approximation, equation
(22.1.13), is not required by EC. The EC backward pass therefore makes fuller
use of the future information, resulting in a recursion which intimately couples the
continuous and discrete variables. Unlike [55] and [43], where gt , Gt ≡ ft , Ft and
only the backward pass mixture weights are updated from the forward pass, EC
actually changes the Gaussian parameters gt , Gt in a non-trivial way. The result-
ing effect on the quality of the approximation can be profound, as we will see in
the experiments.
The Expectation Propagation algorithm, discussed in more detail in section (22.2),
makes the central assumption, as in EC, of collapsing the posteriors to a Gaussian
family [50]. However, in EP, collapsing to a mixture of Gaussians is difficult –
indeed, even working with a single Gaussian may be numerically unstable. In con-
trast, EC works largely with moment parameterisations of Gaussians, for which
relatively few numerical difficulties arise. As explained in the derivation of equa-
tion (22.1.8), the conditional independence assumption p(ht+1 |st , st+1 , v1:T ) ≈
p(ht+1 |st+1 , v1:T ) is not strictly necessary in EC. We motivate it by computa-
tional simplicity, since finding an appropriate moment matching approximation
of p(ht+1 |st , st+1 , v1:T ) in equation (22.1.7) requires a relatively expensive non-
Gaussian integration. The important point here is that, if we did treat p(ht+1 |st , st+1 , v1:T )
more correctly, the only assumption in EC would be a collapse to a mixture of
Gaussians, as in EP. As a point of interest, as in EC, the exact computation re-
quires only a single forward and backward pass, whilst EP is an ‘open’ procedure
requiring iteration to convergence.
 255

22.1.5 Using Mixtures in the Backward Pass

The extension to the mixture case is straightforward, based on the representation

J
X
p(ht |st , v1:T ) ≈ p(jt |st , v1:T )p(ht |jt , v1:T ).
jt =1

Analogously to the case with a single component,

X
p(ht , st |v1:T ) = p(st+1 |v1:T )p(jt+1 |st+1 , v1:T )p(ht |jt+1 , st+1 , it , st , v1:T )
it ,jt+1 ,st+1

· hp(it , st |ht+1 , jt+1 , st+1 , v1:t )ip(ht+1 |jt+1 ,st+1 ,v1:T )

The average in the last line of the above equation can be tackled using the same
techniques as outlined in the single Gaussian case. To approximate p(ht |jt+1 , st+1 , it , st , v1:T )
we consider this as the marginal of the joint distribution

p(ht , ht+1 |it , st , jt+1 , st+1 , v1:T ) = p(ht |ht+1 , it , st , jt+1 , st+1 , v1:t )p(ht+1 |it , st , jt+1 , st+1 , v1:T )

As in the case of a single mixture, the problematic term is p(ht+1 |it , st , jt+1 , st+1 , v1:T ).
Analogously to before, we may make the assumption

p(ht+1 |it , st , jt+1 , st+1 , v1:T ) ≈ p(ht+1 |jt+1 , st+1 , v1:T )

meaning that information about the current switch state st , it is ignored. As in

the single component case, in principle, this assumption may be relaxed and a
moment matching approximation be performed instead. We can then form
X
p(ht |st , v1:T ) = p(it , jt+1 , st+1 |st , v1:T )p(ht |it , st , jt+1 , st+1 , v1:T )
it ,jt+1 ,st+1

This mixture can then be collapsed to smaller mixture using any method of choice,
to give
X
p(ht |st , v1:T ) ≈ p(jt |st , v1:T )p(ht |jt , v1:T )
jt

The resulting algorithm is presented in 3.?? which includes using mixtures in both
forward and backward passes.

22.2 Relation to other methods

Approximate inference in the SLDS has been a long-standing research topic, gen-
erating an extensive literature, to which it is difficult to serve justice. See [49] and
[50] for good reviews of previous work. A brief summary of some of the major
existing approaches follows.
Assumed Density Filtering : Since the exact filtered estimate p(ht |st , v1:t ) is an (exponentially large) mix-
ture of Gaussians a useful remedy is to project at each stage of the recursion
equation (22.0.3) back to a limited set of K Gaussians. This is a Gaussian
Sum Approximation [52], and is a form of Assumed Density Filtering (ADF)
[51]. Similarly, Generalised Pseudo Bayes2 (GPB2) [41, 56] also performs
filtering by collapsing to a mixture of Gaussians. This approach to filtering
 256

is also taken in [43] which performs the collapse by removing spatially similar
Gaussians, thereby retaining diversity.
Several smoothing approaches directly use the results from ADF. The most
popular is Kim’s method, which updates the filtered posterior weights to
form the smoother. As discussed in section (22.1.4), Kim’s smoother cor-
responds to a potentially severe loss of future information and, in general,
cannot be expected to improve much on the filtered results from ADF. The
more recent work of [43] is similar in spirit to Kim’s method, whereby the
contribution from the continuous variables is ignored in forming an approx-
imate recursion for the smoothed p(st |v1:T ). The main difference is that for
the discrete variables, Kim’s method is based on a correction smoother, [40],
whereas Lerner’s method uses a Belief Propagation style backward pass [6].
Neither method correctly integrates information from the continuous vari-
ables. How to form a recursion for a mixture approximation, which does
not ignore information coming through the continuous hidden variables is a
central contribution of our work.
[44] used a two-filter method in which the dynamics of the chain are reversed.
Essentially, this corresponds to a Belief Propagation method which defines a
Gaussian sum approximation for p(vt+1:T |ht , st ). However, since this is not
a density in ht , st , but rather a conditional likelihood, formally one cannot
treat this using density propagation methods. In [44], the singularities re-
sulting from incorrectly treating p(vt+1:T |ht , st ) as a density are heuristically
finessed.
Expectation Propagation : EP [51] corresponds to an approximate implementation of Belief Propaga-
tion12 [6, 38]. Whilst EP may be applied to multiply-connected graphs,
it does not fully exploit the numerical advantages present in the singly-
connected aSLDS structure. Nevertheless, EP is the most sophisticated rival
to Kim’s method and EC, since it makes the least assumptions. For this rea-
son, we’ll explain briefly how EPQworks. First, let’s simplify the notation,
and write the distribution as p = t φ (xt−1 , vt−1 , xt , vt ), where xt ≡ ht ⊗ st ,
and φ (xt−1 , vt−1 , xt , vt ) ≡ p(xt |xt−1 )p(vt |xt ). EP defines ‘messages’ ρ, λ13
which contain information from past and future observations respectively14 .
Explicitly, we define ρt (xt ) ∝ p(xt |v1:t ) to represent knowledge about xt
given all information from time 1 to t. Similarly, λt (xt ) represents knowl-
edge about state xt given all observations from time T to time t + 1. In
the sequel, we drop the time suffix for notational clarity. We define λ(xt )
implicitly through the requirement that the marginal smoothed inference is
given by
p(xt |v1:T ) ∝ ρ (xt ) λ (xt ) (22.2.1)
Hence λ (xt ) ∝ p(vt+1:T |xt , v1:t ) = p(vt+1:T |xt ) and represents all future
knowledge about p(xt |v1:T ). From this
p(xt−1 , xt |v1:T ) ∝ ρ (xt−1 ) φ (xt−1 , vt−1 , xt , vt ) λ (xt ) (22.2.2)
12 Non-parametric belief propagation [57], which performs approximate inference in general con-
tinuous distributions, is also related to EP applied to the aSLDS, in the sense that the messages
cannot be represented easily, and are approximated by mixtures of Gaussians.
13 These correspond to the α and β messages in the Hidden Markov Model framework [37].
14 In this Belief Propagation/EP viewpoint, the backward messages, traditionally labeled as β,

correspond to conditional likelihoods, and not distributions. In contrast, in the EC approach,

which is effectively a so-called α−γ recursion, the backward γ messages correspond to posterior
distributions.
 257

Taking the above equation as a starting point, we have

Z
p(xt |v1:T ) ∝ ρ (xt−1 ) φ (xt−1 , vt−1 , xt , vt ) λ (xt )
xt−1

Consistency with equation (22.2.1) requires (neglecting irrelevant scalings)

Z
ρ (xt ) λ (xt ) ∝ ρ (xt−1 ) φ (xt−1 , vt−1 , xt , vt ) λ (xt )
xt−1

Similarly, we can integrate equation (22.2.2) over xt to get the marginal at

time xt−1 which, by consistency, should be proportional to ρ (xt−1 ) λ (xt−1 ).
Hence
R R
xt−1 ρ (xt−1 ) φ (xt−1 , xt ) λ (xt ) ρ (xt−1 ) φ (xt−1 , xt ) λ (xt )
ρ (xt ) ∝ , λ (xt−1 ) ∝ xt
λ (xt ) ρ (xt−1 )
(22.2.3)

where the divisions can be interpreted as preventing overcounting of mes-

sages. In an exact implementation, the common factors in the numerator and
denominator cancel. EP addresses the fact that λ(xt ) is not a distribution by
using equationR (22.2.3) to form the projection (or R‘collapse’). In the numera-
tor, the terms xt−1 ρ (xt−1 ) φ (xt−1 , xt ) λ (xt ) and xt ρ (xt−1 ) φ (xt−1 , xt ) λ (xt )
represent p(xt |v1:T ) and p(xt−1 |v1:T ). Since these are distributions (an in-
dexed mixture of Gaussians in the SLDS), they may be projected/collapsed
to a single indexed Gaussian. The update for the ρ message is then found
from division by the λ potential, and vice versa15 . To perform this divi-
sion, the potentials in the numerator and denominator are converted to their
canonical representations. To form the ρ update, the result of the division is
then reconverted back to a moment representation. The collapse is nominally
made to a single Gaussian since then explicit division is well defined. The
resulting recursions, due to the approximation, are no longer independent
and [38] show that using more than a single forward sweep and backward
sweep often improves on the quality of the approximation. This coupling is
a departure from the exact recursions, which should remain independent, as
in our EC approach.
Applied to the SLDS, EP suffers from severe numerical instabilities [38] and
finding a way to minimize the corresponding EP free energy in an efficient,
robust and guaranteed way remains an open problem. Damping the parame-
ter updates is one suggested approach to heuristically improve convergence.
Variational Methods : [42] used a variational method which approximates the joint distribution
p(h1:T , s1:T |v1:T ) rather than the marginal inference p(ht , st |v1:T ). This is
a disadvantage when compared to other methods that directly approximate
the marginal. The variational methods are nevertheless potentially attractive
since they are able to exploit structural properties of the distribution, such
as a factored discrete state-transition.
15 In EP the explicit division of potentials only makes sense for members of the exponential
family. More complex methods could be envisaged in which, rather than an explicit division,
the new
R messages are defined by minimising some measure of divergence between ρ(xt )λ(xt )
and x ρ (xt−1 ) φ (xt−1 , xt ) λ (xt ), such as the Kullback-Leibler divergence. Whilst this is
t−1
certainly feasible, it is somewhat unattractive computationally since this would require for each
timestep an expensive minimization.
 258

Sequential Monte Carlo (Particle Filtering) : These methods form an approximate implementation of equation
(22.0.3), using a sum of delta functions to represent the posterior (see, for
example, [58]). Whilst potentially powerful, these non-analytic methods typ-
ically suffer in high-dimensional hidden spaces since they are often based on
naive importance sampling, which restricts their practical use. ADF is gen-
erally preferential to Particle Filtering since in ADF the approximation is
a mixture of non-trivial distributions, which is better at capturing the vari-
ability of the posterior. In addition, for applications where an accurate com-
putation of the likelihood of the observations is required (see, for example
[59]), the inherent stochastic nature of sampling methods is undesirable.
23 Gaussian Processes

VERY DRAFTY!!!

23.1 The Bayesian approach to Regression

We write our model of the data generating process as

y = f (x) (23.1.1)

and we aim to choose a function f that fits the data well. We assume that the
data output we observe, t has been corrupted with additive Gaussian noise1

t = f (x) + η (23.1.2)

where η ∼ normal(0, σ 2 ). This means that the likelihood of a datapoint t under

this model is
1 2
p(t|f ) = normal(f (x), σ 2 ) ∝ e− 2σ2 (t−f (x)) (23.1.3)

Note that it is not necessary to explicitly write normalising constants for distributions
since they are uniquely given by the normalisation condition for probabilities. In this

1/2
case the normalising constant is 1/ 2πσ 2 .

Assuming that individual datapoints are independently and identically distributed,

the likelihood for a vector of observed data outputs t = (t1 . . . tP ) is2
P
Y 1 2
p(t|f ) = p(ti |f ) ∝ e− 2σ2 (t−f ) (23.1.4)
i=1

where f = (f (x1 ) . . . f (xP )). The term (t − f )2 in the exponent is referred to in

neural networks as the training error, so that maximum likelihood fitting in this
context is equivalent to finding that function f which minimizes the training error.
However, we may have some extra beliefs about the model f which we can express
through a prior distribution p(f ). For example, certain f may not be smooth
enough, and we would believe them to be unlikely. We may prefer to express these
beliefs by assigning a prior on the parameters θ.

Together, the likelihood and prior on the function f complete the specification of
the model. We can then use Bayes rule to find the posterior distribution of f in
light of the observed data D.
p(t|f )p(f )
P (f |t) = (23.1.5)
p(t)
1 Although not necessary, this assumption is convenient so that the following theory is analyti-
cally tractable. P 2
2 In equation (23.1.4) we use the notation v2 to mean
i vi for an arbitrary vector v.

259
 260

23.1.1 Parameterised Models

It may be that our function f is parameterised in a certain way,

y = f (x|θ) (23.1.6)
Where θ represents all the parameters that we could adjust to make the function
f fit the data well. For example, f (x|θ) = θ1 x + θ2 would parameterize straight
line models. In this case, it is more natural to think of the likelihood p(t|θ) and
the prior p(θ) as functions of these parameters.

23.1.2 Making Predictions

In contrast to other formalisms, the Bayesian approach automatically gives rise

to an ensemble of models, namely the posterior distribution p(f |t). Assuming
a squared error model, the best estimate of the model is given by the posterior
average of the functions3
Z
hf (x)i ≡ f (x)p(f |t)df (23.1.7)

Similarly, we can calculate error bars for the predictions,

Z
2
var(f (x)) ≡ [f (x) − hf (x)i] p(f |t)df (23.1.8)

which gives a measure of the confidence in the prediction hf (x)i. Note that if we
wish to make predictive error bars, we need to include possible noise corrupting
processes. For example, in the case of additive Gaussian noise, we believe that
the actual data points we observe are modelled by the process f (x) + η, where
η ∼ normal(0, σ 2 ). Given that the posterior distribution is independent of the
noise process, this means that predicitve error bars are given by simply adding σ 2
to equation (23.1.8).

In principle, in contrast to non-Bayesian approaches, there is no need to set aside

a portion of the data to test the fidelity of the model. Nevertheless, one may wish
to monitor a quantity such as the test error to check if f is capable of modeling
the data well enough to produce the kind of test error performance we might hope
for.

23.1.3 Model Selection

In the Bayesian framework, the quantity that we need in order to assess the fi-
delity of a model M is it’s likelihood, p(t|M ). This is also sometimes called the
“evidence”. It will be simpler to explain how to use such quantities in the context
of a specific model, and we defer this discussion till section (23.2.4)

23.2 Generalised Linear Models

Generalised linear models have the form
k
X
yl = wi φi (xl ) (23.2.1)
i=1
3




This is readily shown by considering (f − t)2 = (f − hf i + hf i − t)2 = (f − hf i)2 + (hf i −
2
t) , which has a minimum at f = hf i.
 261

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
X

Figure 23.1: A set of 5 Gaussian basis functions. The model output for a particular
x is given by a linear combination of the basis function values at that x, here given
by the intersection of the basis curves with the line x = 0.8.

That is, the model is linear in the parameters w, although the output y depends
non-linearly on the input x. If we have several points that we wish to make a
prediction at, x1 . . . xl , then we write the prediction vector (y 1 . . . y l )T = y = Φw
where we have defined the design matrix Φji = φi (xj ). Note that an upper index
refers to the datapoint number and the lower to a basis component. In fig(23.1)
we plot 5 Gaussian basis functions. The models value for say x = 0.8 is then a
linear combination of the basis function values at that input (given here by the
intersection of the vertical line with x = 0.8).
We shall assume that, in addition to w, there may be some tunable (hyper)parameters,
such as the width of the basis functions and our belief about the noise level.

23.2.1 Understanding the Prior

 
2
The Gaussian basis functions have the form φ(x) = exp −0.5 (x − c) /(2λ2 )
which are local in the sense that they decay (quickly) to zero as we move away
from the centre c.

The Log basis

 functions havea non-local form,  extending to infinity
 at infinity:
2 2 2 2
φ(x) = exp −0.5 (x − c) /λ + (x − c) log (x − c) + 0.0001 . However, they
retain some “bump” Gaussian shape close to the centre c. Such non-local forms
can be useful if our beliefs about the function only apply in a limited region. Non-
local basis functions (as we shall see) typically give rise to large error bars away
from the training data, and do not necessarily affect the quality of predictions close
to the training data.

One needs to cover the input region of interest sufficiently well so that the func-
tional form of the output distribution p(f ) is expressive enough to capture the
kinds of models that we are interested in finding.

23.2.2 Sampling the function space prior

We assume that our prior belief about the function f can be expressed by a belief
about the distribution of the parameters w. Here we assume that this takes the
 262

form of a Gaussian distribution, with zero mean, and a user specified variance:
 α 
p(w) = normal(0, α−1 I) ∝ exp − w2 (23.2.2)
2
Using this distribution, we can draw (say 6) random weight vectors w1 . . . w6 and
plot the corresponding functions w1 ·φ . . . w6 ·φ, where φ = {φ1 (x) . . . φk (x)} is a
vector of basis function values.

23.2.3 Understanding the Posterior

The posterior for the weights of the GLM is

1 
p(w|t) ∝ p(t|w)p(w) ∝ exp − β (Φw − t)2 + αw2 (23.2.3)
2
For convenience we define β = 1/σ 2 , the reciprocal of our noise belief. It is not
difficult to see that p(w|t) is a Gaussian distribution for the weights (since the
exponent is quadratic in the weights). In fact it is the distribution

p(w|t) = normal(βC −1 ΦT t, C −1 ) (23.2.4)

where C = αI + βΦT Φ. That is, p(w|t) is a multivariate

Gaussian distribution

with mean hwi = βC −1 ΦT t and covariance matrix (w − hwi)(w − hwi)T = C −1 .
For the linear model, the mean predictions over the posterior are simply given by
T
hf (x)i = hwi φ(x) (23.2.5)

Similarly, the variance is given by

var(f (x)) = φ(x)T C −1 φ(x) (23.2.6)

The predictions becoming more confident towards the edges in the Gaussian basis
function case is simply a consequence of the form of the basis functions. This is
an important point - you only get out from the method answers consistent with
your model assumptions.
What is the posterior distribution p(y(x1 ) . . . y(xl )|t) for a set of chosen x1 . . . xl ?,
induced by the Gaussian weight posterior equation (23.2.4)?

23.2.4 Understanding Model Selection Issues

As we have seen, the distribution of the weights w of the GLM are determined
automatically through the Bayesian procedure. The only parameters that are left
to the user to control are the width of the basis functions, the noise belief, the scale
α and the number and type of the basis functions. Let’s denote such parameters
by Γ. It may be that we would like to carry out a Bayesian analysis for these
parameters too, so that we can assess the relevance of different model parameter
settings in light of the observed data.
In principle, this can be viewed as just another level in a hierarchy of models. The
determined Bayesian would assign a (hyper)prior to these parameters p(Γ) and
perform model averaging over them (just as we did in the weights w case),
Z Z Z 
hf (x)i = f (x|w)p(w, Γ|t)dwdΓ = f (x|w)p(w|Γ, t)dw p(Γ|t)dΓ (23.2.7)
 263

Where p(Γ|t) = p(t|Γ)p(Γ)/p(t) and p(Γ) is our prior belief about the (hyper)parameters.
R
The “evidence” p(t|Γ) is obtained by integrating over the weights, p(t|Γ) = p(t|w)p(w|Γ)dw.
Typically, the integrations in equation (23.2.7) are extremely difficult to carry out
(even if p(Γ|t) is tractable) and one needs to resort to techniques such as Monte
Carlo.
A simpler alternative is to consider using those Γ that correspond to a maximum
of the model posterior p(Γ|t). Provided that the posterior p(Γ|t) is sharply peaked
around it’s optimum value, this may still give a faithful value for the average in
equation (23.2.7). Assuming a flat prior on Γ, this corresponds to using the Γ that
maximize the likelihood p(t|Γ).

In the linear case here, and with the Gaussian noise model assumption, calculating
the model likelihood involves only Gaussian integrals, giving

β 1 1 k P
log p(t|Γ) = − t2 + β 2 tT ΦT C −1 Φt − log det(C) + log α − log(2π/β)
2 2 2 2 2
(23.2.8)

GLMs can be very flexible regression models and one advantage from the Bayesian
point of view is that the model likelihood p(t|Γ) can be calculated exactly. This
makes combining models which have say different numbers of basis functions easy
to do – we just use equation (23.2.7).

23.3 Gaussian Processes

Gaussian Processes are (in a certain sense) formally identical to GLMs, but differ
in their computational implementation. The main idea is to go back to the general
Bayesian framework section (23.1), using the Gaussian noise model, but now to
specify the form of the prior on the functions directly.

23.3.1 Specifying the Prior

From equation (23.1.4) we see that, since we already have a (Gaussian) definition
for the likelihood p(f |t), all we need to do is specify a prior distribution p(f ) on
the function space f to complete the model specification. The most natural choice
is to specify a Gaussian distribution here, since that will mean that the posterior
is also Gaussian.
Imagine that we are given a set of inputs x1 . . . xl . Consider a particular xi and it’s
corresponding possible function value y i . If we have a space of possible functions,
then they will pass through different y i for the same xi (see say x1 in fig(23.2)).
Indeed, we can construct the prior on functions so that the distribution of these
values should be Gaussian, centered around some mean value (we will take this to
be zero for simplicity) with a certain variance.
Consider now two inputs, xi and xj and their separation, |xi − xj |. Note that y i
and y j fluctuate as different functions are sampled from some function space prior.
How can we incorporate ideas of smoothness? If |xi − xj | is small, we may expect
that a set of values at y i and a set at y j should be highly correlated (as in fig(23.2)
for x1 and x2 ) . This means that we might well think that the output values y i and
y j should be highly correlated if |xi − xj | is small. Conversely, if |xi − xj | is large,
we (probably) do not expect that y i and y j will be at all correlated (as for values at
 264

30

20

10

y
−10

−20

−30
−1 −0.5 0 0.5 1 1.5 2
x1 x2 x x3

Figure 23.2: Samples functions from a Gaussian Process Prior. The correlation
between y i and y j decreases with the distance |xi − xj |.

x1 and x3 in fig(23.2)). We can express these beliefs about the correlations of the
components of a vector of values y = (y 1 . . . y l ) at inputs x1 . . . xl as a multivariate
Gaussian distribution
 
1
p(y) ∝ exp − y T k−1 y (23.3.1)
2
where K is the covariance matrix of the outputs y. The elements of k are specified
by the covariance function c(xi , xj ). As we argued above, we might expect that
the correlation between yi and yj decreases the further apart xi and xj are.
 
2
In the Gaussian case, the covariance function is c(xi , xj ) = α exp −0.5λ (xi − xj ) .
Note that the shape of this function is smooth at zero.

In the Ornstein Uhlenbeck case, the covariance function is c(xi , xj ) = α exp (−0.5λ|xi − xj |).

Note how the Ornstein Uhlenbeck process gives rise to much less smooth functions
than those formed with the Gaussian covariance function.

 What is the relationship between the derivative of the covariance function at

the origin and the smoothness of the function space?

Changing the length scale of the covariance function affects the range over which
the functions are correlated. See how changing α alters the scale of the outputs.

23.3.2 Making Predictions

Imagine that we have some new inputs x∗ and we wish to make some predictions
for their outputs y ∗ . According to the Bayesian philosophy, we need to specify a
likelihood and a prior. We already specified a prior in section (23.3.1),
p(y∗ , y) = normal(0, K) (23.3.2)
 265

T
where K can be partitioned into matrices k, Kx∗ x∗ , Kxx∗ , Kxx ∗ . k has elements

c(xi , xj ), and Kxx∗ elements c(xi , x∗ ) etc.. The likelihood is

p(t|y∗ , y) = p(t|y) = normal(y, σ 2 I) (23.3.3)

Since the prior and likelihood are Gaussian, it is clear that the posterior p(y∗ , y|t) ∝
p(t|y∗ , y)p(y∗ , y) is also Gaussian in y∗ , y. The marginal distribution p(y∗ |t) is
therefore also Gaussian. You might like to convince yourselves in your own time
that it takes the form

−1

2 −1
p(y∗ |t) = normal(Kxx∗ k + σ 2 T
t, Kx∗ x∗ − Kxx ∗ k + σ I Kxx∗ ) (23.3.4)

First we see predictions for one training point. The red curve is the mean prediction
and the green curve are the error bars (one standard deviation). The blue crosses are
the training data points.

We can now try to understand the posterior as in the case of GLMs. In the same
way, we alter the noise belief and actual noise and see what happens to the pre-
dictions.

Note how the error bars collapse onto the data for a single datapoint. See how
this also happens for two datapoints as well.

Can you observe any differences between the GP predictions and the GLM predic-
tions?
What do you think could be the connection between GPs and GLMs?

23.3.3 Model Selection

It is straightforward to show that the likelihood for a GP model is

1 1 1
log p(t|Γ) = log(det(k + σ 2 )) − tT (k + σ 2 I)t − P log(2π) (23.3.5)
2 2 2

23.3.4 Classification problems

There are two basic methods for making predictions in classification problems (see,
e.g Ripley, 1996); (i) the sampling paradigm, where a class-conditional density
p(x|k) and a prior are created for each class k, and Bayes’ theorem is used to
determine p(k|x) given a new input x, or (ii) the diagnostic paradigm, where the
aim is to predict p(k|x) directly via some function of the input. As p(k|x) must
lie in [0, 1], this condition is usually achieved by using an output (or transfer)
function which enforces the constraint. For the two class problem a common
choice is the logistic function σ(y) = 1/(1 + e−y ). For a k > 2 class problem a
simple generalization of the logistic function, the softmax function, is frequently
used.
We will follow the diagnostic paradigm and use the logistic function, an approach
also used widely in the neural networks literature. In the simplest method of this
kind, logistic regression, the input to the sigmoid function y is simply computed as
a linear combination of the inputs, plus a bias, i.e. y = wT x + b. Neural networks
and other flexible methods allow y to be a non-linear function of the inputs.
 266

1
3
y
1
π
σ
⇒ ⇒
0
−5 0 5

−3
0

Figure 23.3: π(x) is obtained from y(x) by “squashing” it through the sigmoid
function σ.

23.4 Gaussian Processes for Classification

By using the logistic transfer function σ to produce an output which can be in-
terpreted as π(x), the probability of the input x belonging to class 1, the job of
specifying a prior over functions π can be transformed into that of specifying a
prior over the input to the transfer function, which we shall call the activation, and
denote by y, with π(x) = σ(y(x)) (see figure 23.3). For input xi , we will denote
the corresponding probability and activation by πi and yi respectively.
Given that the GP contains adjustable hyperparameters, how should they be
adapted given the data ? Maximum likelihood or (generalized) cross-validation
methods are often used, but we will prefer a Bayesian solution. A prior distribu-
tion over the hyperparameters P (θ) is modified using the training data to obtain
the posterior distribution P (θ|t) ∝ P (t|θ)P (θ).
To make predictions we then integrate over the posterior; for example, the mean
value π(x∗ ) for test input x∗ is given by
Z Z 
π(x∗ ) = π(x∗ )P (π(x∗ )|t, θ)dπ P (θ|t)dθ. (23.4.1)

We show in section 23.4.3 how to perform the integral in 23.4.1 over the hyper-
parameters P (θ|t). Here we consider the hyperparameters to be fixed, and are
interested in the posterior distribution P (π∗ |t) = P (π(x∗ )|t) for a new input x∗ .
This can be calculated by finding the distribution P (y∗ |t) (y∗ is the activation of
π∗ ) and then using the appropriate Jacobian to transform the distribution. For-
mally the equations for obtaining P (y∗ |t) are identical to equation ??. However,
even if weQuse a GP prior so that P (y∗ , y) is Gaussian, the usual expression for
P (t|y) = i πiti (1 − πi )1−ti for classification data (where the t’s take on values
of 0 or 1), means that the average over π in equation 23.4.1 is no longer exactly
analytically tractable.
After transforming equation 23.4.1 to an integral over activations, we will employ
Laplace’s approximation, i.e. we shall approximate the integrand P (y∗ , y|t) by a
Gaussian distribution centred at a maximum of this function with respect to y∗ , y
with an inverse covariance matrix given by −∇∇ log P (y∗ , y|t). The necessary
integrations (marginalization) can then be carried out analytically (see, e.g. Green
and Silverman (1994) §5.3) and, we provide a derivation in the following section.
The averages over the hyperparameters will be carried out using Monte Carlo
techniques, which we describe in section 23.4.3.
 267

23.4.1 Maximizing P (y∗ , y|t)

Let y+ denote (y∗ , y), the complete set of activations. By Bayes’ theorem log P (y+ |t) =
log P (t|y) + log P (y+ ) − log P (t), and let Ψ+ = log P (t|y) + log P (y+ ). As P (t)
does not depend on y+ (it is just a normalizing factor), the maximum of P (y+ |t)
is found by maximizing Ψ+ with respect to y+ . We define Ψ similarly in relation
to P (y|t). Using log P (ti |yi ) = ti yi − log(1 + eyi ), we obtain
n
X 1 T −1 1 n+1
Ψ+ = tT y − log(1 + eyi ) − y+ K+ y+ − log |K+ | − log(23.4.2)
2π
i=1
2 2 2
n
X 1 1 n
Ψ = tT y − log(1 + eyi ) − y T K −1 y − log |K| − log 2π (23.4.3)
i=1
2 2 2
where K+ is the covariance matrix of the GP evaluated at x1 , . . . xn , x∗ . K+ can
be partitioned in terms of an n × n matrix K, a n × 1 vector k and a scalar k∗ ,
viz.
 
K k
K+ = (23.4.4)
kT k∗

As y∗ only enters into equation 23.4.2 in the quadratic prior term and has no data
point associated with it, maximizing Ψ+ with respect to y+ can be achieved by first
maximizing Ψ with respect to y and then doing the further quadratic optimization
to determine y∗ . To find a maximum of Ψ we use the Newton-Raphson (or Fisher
scoring) iteration y new = y − (∇∇Ψ)−1 ∇Ψ. Differentiating equation 23.4.3 with
respect to y we find
∇Ψ = (t − π) − K −1 y (23.4.5)
−1
∇∇Ψ = −K −N (23.4.6)
where the ‘noise’ matrix is given by N = diag(π1 (1 − π1 ), .., πn (1 − πn )). This
results in the iterative equation,

y new = (K −1 + N )−1 N (y + N −1 (t − π)) (23.4.7)

To avoid unnecessary inversions, it is usually more convenient to rewrite this in

the form

y new = K(I + KN )−1 (N y + (t − π)) (23.4.8)

Given a converged solution ỹ for y, y∗ can easily be found using y∗ = kT K −1 ỹ =

kT (t − π̃). var(y∗ ) is given by (K+ −1
+ N+ )−1
(n+1)(n+1) , where N+ is the N matrix
with a zero appended in the (n + 1)thRdiagonal position. Given the mean and vari-
ance of y∗ it is then easy to find π̂∗ = π∗ P (π∗ |t)dπ∗ , the mean of the distribution
of P (π∗ |t)(see equation 23.4.1). In order to calculate the Gaussian integral over
the logistic sigmoid function, we employ an approximation based on the expansion
of the sigmoid function in terms of the error function. As the Gaussian integral of
an error function is another error function, this approximation is fast to compute.
Specifically, we use a basis set of 5 scaled error functions to interpolate the logistic
sigmoid at chosen points. This gives an accurate approximation (to 10−4 ) to the
desired integral with a small computational cost.
The justification of Laplace’s approximation in our case is somewhat different from
the argument usually put forward, e.g. for asymptotic normality of the maximum
 268

likelihood estimator for a model with a finite number of parameters. This is because
the dimension of the problem grows with the number of data points. However, if
we consider the “infill asymptotics”, where the number of data points in a bounded
region increases, then a local average of the training data at any point x will
provide a tightly localized estimate for π(x) and hence y(x), so we would expect
the distribution P (y) to become more Gaussian with increasing data.

23.4.2 Parameterizing the covariance function

There are many reasonable choices for the covariance function. Formally, we are
required to specify functions which will generate a non-negative definite covariance
matrix for any set of points (x1 , . . . , xk ). From a modelling point of view we wish
to specify covariances so that points with nearby inputs will give rise to similar
predictions. We find that the following covariance function works well:
d
1X
C(x, x′ ) = v0 exp{− wl (xl − x′l )2 } (23.4.9)
2
l=1

where xl is the lth component of x and θ = log(v0 , w1 , . . . , wd ) plays the role of

hyperparameters4. We define the hyperparameters to be the log of the variables
in equation (23.4.9) since these are positive scale-parameters. This covariance
function can be obtained from a network of Gaussian radial basis functions in the
limit of an infinite number of hidden units (Williams, 1996).
The wl parameters in equation 23.4.9 allow a different length scale on each input
dimension. For irrelevant inputs, the corresponding wl will become small, and the
model will ignore that input. This is closely related to the Automatic Relevance
Determination (ARD) idea of MacKay and Neal (Neal, 1995). The v0 variable
gives the overall scale of the prior; in the classification case, this specifies if the π
values will typically be pushed to 0 or 1, or will hover around 0.5.

23.4.3 Integration over the hyperparameters

To make predictions we integrate the predicted probabilities over the posterior

P (θ|t) ∝ P (t|θ)P (θ), as given by equation 23.4.1. R For the regression problem
P (t|θ) can be calculated exactly using P (t|θ) = P (t|y)P (y|θ)dy, but this in-
tegral is not analytically tractable for the classification problem. Again we use
Laplace’s approximation and obtain
1 n
log P (t|θ) ≃ Ψ(ỹ) + |K −1 + N | + log 2π (23.4.10)
2 2
where ỹ is the converged iterate of equation 23.5.1, and we denote the right-hand
side of this equation by log Pa (t|θ) (where a stands for approximate).
The integration over θ-space also cannot be done analytically, and we employ
a Markov Chain Monte Carlo method. We have used the Hybrid Monte Carlo
(HMC) method of Duane et al (1987), with broad Gaussian hyperpriors on the
parameters.
HMC works by creating a fictitious dynamical system in which the hyperpara-
meters are regarded as position variables, and augmenting these with momentum
4 We call θ the hyperparameters rather than parameters as they correspond closely to hyperpa-
rameters in neural networks; in effect the weight parameters have been integrated out exactly.
 269

variables p. The purpose of the dynamical system is to give the hyperparameters

“inertia” so that random-walk behaviour in θ-space can be avoided. The total
energy, H, of the system is the sum of the kinetic energy, K = pT p/2 and the po-
tential energy, E. The potential energy is defined such that p(θ|D) ∝ exp(−E), i.e.
E = − log P (t|θ) − log P (θ). In practice log Pa (t|θ) is used instead of log P (t|θ).
We sample from the joint distribution for θ and p given by P (θ, p) ∝ exp(−E−K);
the marginal of this distribution for θ is the required posterior. A sample of hy-
perparameters from the posterior can therefore be obtained by simply ignoring the
momenta.
Sampling from the joint distribution is achieved by two steps: (i) finding new points
in phase space with near-identical energies H by simulating the dynamical system
using a discretised approximation to Hamiltonian dynamics, and (ii) changing the
energy H by Gibbs sampling the momentum variables.
Hamilton’s first order differential equations for H are approximated using the
leapfrog method which requires the derivatives of E with respect to θ. Given a
Gaussian prior on θ, log P (θ) is straightforward to differentiate. The derivative of
log Pa (θ) is also straightforward, although implicit dependencies of ỹ (and hence
π̃) on θ must be taken into account by using equation 23.4.5 at the maximum point
to obtain, ỹ ′ = I + KN −1 K ′ (t − π). The calculation of the energy can be quite
expensive as for each new θ, we need to perform the maximization required for
Laplace’s approximation, equation 23.4.10. The Newton-Raphson was initialized
each time with π = 0.5, and iterated until the mean relative difference of the
elements of N between consecutive iterations was less than 10−4 .
The same step size ε is used for all hyperparameters, and should be as large as
possible while keeping the rejection rate low. We have used a trajectory made
up of L = 20 leapfrog steps, which gave a low correlation between successive
states5 . This proposed state is then accepted or rejected using the Metropolis rule
depending on the final energy H ∗ (which is not necessarily equal to the initial
energy H because of the discretization).
The priors over hyperparameters were set to be Gaussian with a mean of −3 and a
standard deviation of 3. In all our simulations a step size ε = 0.1 produced a very
low rejection rate (< 5%). The hyperparameters corresponding to the wl ’s were
initialized to −2 and that for v0 to 0. The sampling procedure was run for 200
iterations, and the first third of the run was discarded; this “burn-in” is intended
to give the hyperparameters time to come close to their equilibrium distribution.

23.5 Multiple classes

The extension of the preceding framework to multiple classes is essentially straight-
forward, although notationally more complex.
We shall throughout employ a one-of-m class coding scheme6 , and use the multi-
class analogue of the logistic function - the softmax function - to describe the
class probabilities. The probability that an instance labelled by n is in class m is
n
denoted by πm , so that an upper index to denotes the example number, and a lower
index the class label. Similarly, the activations associated with the probabilities
5 In our experiments θ is only 7 or 8 dimensional, so the trajectory length needed is much shorter
than that for neural network HMC implementations.
6 That is, the class is represented by a vector of length m with zero entries everywhere except

for the mth component which contains 1

 270

n
are denoted by ym . Formally, the softmax link function relates the activations and
probabilities through
n
n exp ym
πm = n
Σm′ exp ym ′

P n
which automatically enforces the constraint m πm = 1. The targets are similarly
n
represented by tm , which are specified using a one-of-m coding.
P
The log likelihood takes the form L = n,m tnm ln πm n
, which for the softmax link
function gives
!
X X
n n n
L= tm ym − ln exp πm′ (23.5.1)
n,m m′

As for the two class case, we shall assume that the GP prior operates in activation
n
space; that is we specify the correlations between the activations ym .
One important assumption we make is that our prior knowledge is restricted to cor-
relations between the activations of a particular class. Whilst there is no difficulty
in extending the framework to include inter-class correlations, we have not yet
encountered a situation where we felt able to specify such correlations. Formally,
the activation correlations take the form,
′ ′
n n n,n
hym ym′ i = δm,m′ Km (23.5.2)
′
n,n
where Km is the n, n′ element of the covariance matrix for the mth class. Each
individual correlation matrix Ki has the form given by equation 23.4.9 for the two
class case). We shall make use of the same intraclass correlation structure as that
given in equation 23.4.9 with a separate set of hyperparameters for each class.
For simplicity, we introduce the augmented vector notation,


y+ = y11 , ....y1n , y1∗ , y21 , ....y2n , y2∗ , ....ym
1 n
, ....ym ∗
, ym

where, as in the two class case, yi∗ denotes the target activation for class i; this
notation is also used to define t+ and π + . In a similar manner, we define y, t and
π by excluding the corresponding target values, denoted by a ‘*’ index.
With this definition of the augmented vectors, the GP prior takes the form,
 
1 T +
P (y+ ) ∝ exp − y+ K y+ (23.5.3)
2

where, from equation 23.5.2, the covariance matrix K + is block diagonal in the
matrices, K1+ , ..., Km
+
. Each individual matrix Ki+ expresses the correlations of
activations within class i, with covariance function given by equation 23.4.9, as for
the two class case.

23.5.1 Finding the mode of the distribution

The GP prior and likelihood, defined by equations 23.5.3, 23.5.1 respectively, define
the posterior distribution of activations, P (y+ |t). Again, as in section 23.4.1 we
are interested in a Laplace approximation to this posterior, and therefore need to
find the mode with respect to y+ . Dropping unnecessary constants, the multi-class
 271

analogue of equation 23.4.2 for terms involving y+ in the exponent of the posterior
are:
1 T −1 X X
Ψ+ = − y+ K+ y+ + tT y − ln n
exp ym
2 n n

By the same principle as in section 23.4.1, we define Ψ by analogy with equation

23.4.3, and first optimize Ψ with respect to y, afterwards performing the quadratic
optimization of Ψ+ with respect to y∗ .
In order to optimize Ψ with respect to y+ , we make use of the Hessian given by,

∇∇Ψ = −K −1 − N

Although this is in the same form as for the two class case, equation eq:deldelpsi,
there is a slight change in the definition of the ‘noise’ matrix, N . A convenient
way to define N is by introducing the matrix Π which is an (m ∗ n+ ) × (n+ ) matrix
n n
of the form Π = (diag(π11 ..π1 + ), .., diag(πm
1
..πm+ )). Using this notation, we can
write the noise matrix in the form of a diagonal matrix and an outer product,
n
N = −diag(π11 ..π1 + , .., πm
1 n+
..πm ) + ΠΠT (23.5.4)

The update equation for iterative optimization of Ψ with respect to the activation
y then follow the same form as that given by equation . The advantage of the
representation of the noise matrix in equation 23.5.4 is that we can then invert
matrices and find their determinants using the identities,

−1
(A + HH T )−1 = A−1 − A−1 H I + H T A−1 H H T A−1 (23.5.5)

and

det(A + HH T ) = det(A) det(I + H T A−1 H) (23.5.6)

n
where A = K + diag(π11 ..π1 + ). Thus, rather than requiring determinants and
inverses of (m∗n+ )×(m∗n+ ) matrices, we only need to carry out expensive matrix
computations on (n+ ) × (n+ ) matrices. Some care must be taken in manipulating
the noise matrixPN as this is singular due to the linear constraint imposed upon
n
its elements by m πm = 1. The resulting update equations for y are then of
the same form as given in equation 23.5.1, where the noise matrix and covariance
matrices are now in their multiple class form.
Essentially, these are all the results needed to generalise to the multiple class
method. Although, as we mentioned above, the time complexity of the problem
does not scale with the m3 , but rather m (due to the identities (23.5.5,23.5.6)),
calculating the function and its gradient is still rather expensive. We experimented
with several methods of mode finding for the Laplace approximation. The advan-
tage of the Newton iteration method is its fast quadratic convergence. An integral
part of each Newton step is the calculation of the inverse of a matrix M acting
upon a vector, i.e.,M −1 b . In order to speed up this particular step, we used a
conjugate gradient method to solve the corresponding linear system M x = b.
As for the 2 class case, after approximating the posterior by a Gaussian, we average
the softmax (sigmoid for 2 classes) output over the Gaussian approximation to the
posterior. At present, we simply sample this integral using 1000 draws from a
Gaussian random vector generator.
 272

23.6 Discussion
One should always bear in mind that all models are wrong! (If we knew the cor-
rect model, we wouldn’t need to bother with this whole business). Also, there is
no such thing as assumption free predictions, or a “universal” method that will
always predict well, regardless of the problem. In particular, there is no way that
one can simply look at data and determine what is signal and what is noise. The
separation of a signal into such components is done on the basis of belief about the
noise/signal process.

As far as the Bayesian is concerned, the best thing to do is to incorporate as much

of the knowledge we have about possible solutions, quantifying our beliefs about
which models are more or less likely. These beliefs are then updated in light of the
observed data, giving rise to a principled method of model selection. Also, con-
cepts such as model averaging are intrinsically inherent in this framework. That
predictions are based on such subjective prior belief is in no sense a drawback.
Indeed, the insistence of this framework in making the user specify his/her model
assumptions explicitly greatly enhances different scientists ability to evaluate each
others work.

Note that our stated aim in this practical was to find a good regression model and
not to try to interpret the data. This is an important difference and should be kept
in mind. It may well be that using a non-linear model, we can (also) fit the data
well using far fewer adjustable parameters. In that case, we may be able to place
more emphasis on interpreting such lower dimensional representations and perform
feature extraction (as potentially in neural networks). However, linear models are
generally easier to work with and are a useful starting place in our search for
a good regression model. Coupled with the Gaussian noise assumption, using a
Gaussian prior on the weights of a linear model defines a Gaussian Process in the
output space. In this sense, generalised linear models are Gaussian Process with a
particular covariance function. Once this is realised, one is free to directly specify
the form of the covariance function, as we did in the latter half of the practical,
and this obviates the need for a weight space. This is in some cases convenient
since it therefore also deals with the problem of the curse of dimensionality. As far
as the Bayesian is concerned in this regression context, without any explicit belief
about the data generating process, the only requirements/prior belief one has are
typically expressed in terms of the smoothness of the function itself. That is, the
question of parameter complexity is irrelevant - the Bayesian is perfectly happy
to use a model with a billion parameters or one with 10 parameters. Whichever
model most aptly captures his/her belief about the data generating function is the
preferred choice.
IV. Approximate Inference Methods

273
 24 Sampling

Sampling is a big topic. An excellent introduction to this area is Neal, R. M.

(1993) Probabilistic Inference Using Markov Chain Monte Carlo Methods, Tech-
nical Report CRG-TR-93-1, Dept. of Computer Science, University of Toronto.

See also related articles at

http://www.cs.toronto.edu/~radford/publications.html

Readers are also invited to read the chapter on sampling methods methods in
David MacKay’s book.

24.1 Introduction
Consider the distribution p(x). Sampling is the process of generating a vector
x from the distribution p(x), with probability given by p(x). One way to view
this is that if we have a procedure S(p) from which we can generate a set of P
samples x1 , . . . , xP , then, in the limit of P → ∞, the relative frequency that the
sample value x occurs tends to p(x). (In the continuous distribution case, R this can
be defined as the limiting case of the relative frequency x ∈ ∆ tending to x∈∆ p(x).

In both cases, sampling simply means drawing examples from the distribution with
the correct frequency.

One dimensional Discrete distribution

In the sequel, we assume that a random number generator exists which is able to
produce a value uniformly at random from the unit interval [0, 1]. We will make
use of this uniform random number generator to draw samples from non-uniform
distributions.

As an example of the above, consider the one dimensional discrete distribution

p(x) where x can be in any of the states 1 to K. To be specific, consider the three
state distribution p(x = 1) = 0.6, p(x = 2) = 0.1, p(x = 3) = 0.3.

1 × 2 3

This represents a partitioning of the unit interval [0, 1] in which the interval [0, 0.6]
has been labelled as state 1, [0.6, 0.7] as state 2, and [0.7, 1.0] as state 3. If we
were to drop a point × anywhere at random, uniformly in the interval [0, 1], the
chance that × would land in interval 1 is 0.6, and the chance that it would be in
interval 2 is 0.2 and similarly, for interval 3, 0.3. This therefore defines for us a
valid sampling procedure for discrete one-dimensional distributions:

274
 275

Cumulant Let pi , i = P 1, K label the K state probabilities. Calculate the so-called cumu-
lant, ci = j≤i pj , and set c0 = 0. (In the above, we have (c0 , c1 , c2 , c3 ) =
(0, 0.6, 0.7, 1)). Draw a value u uniformly at random from the unit interval [0, 1].
Find that i for which ci−1 ≤ u ≤ ci . The sampled state is then i. In our example,
we may have sampled u = 0.66. Then the sampled x state would be state x = 2,
since this is in the interval [c1 , c2 ].

Continuous Case

Intuitively, the generalisation of the discrete case to the continuous case is clear.
First we calculate the cumulant density function
Z y
C(y) = p(x)dx
−∞

Then we generate a random u uniformly from [0, 1], and then obtain the corre-
sponding sample value x by solving C(x) = u. For some special distributions, such
as Gaussians, very efficient equivalent ways to achieve this are usually employed.

Multi-variate discrete distributions

One way to generalise the one dimensional case to a higher dimensional case
p(x1 , . . . , xn ) would be to translate the higher dimensional case into an equiva-
lent one-dimensional distribution. We can enumerate all the possible joint states
(x1 , . . . , xn ), giving each a unique integer y from 1 to the total number of states
accessible. This then transforms the multi-dimensional distribution into an equiv-
alent one-dimensional distribution, and sampling can be achieved as before. Of
course, in high dimensional distributions, we would have, in general, exponentially
many states if x, and an explicit enumeration would be impractical.

An alternative exact approach would be to capitalise on the relation

p(x1 , x2 ) = p(x2 |x1 )p(x1 )
This suggests that we can sample from the joint distribution p(x1 , x2 ) by first sam-
pling a value for x1 from the one-dimensional p(x1 ), and then, with x1 clamped to
this value, sampling a value for x2 from the one-dimensional p(x2 |x1 ).

It is clear how to generalise this to say three or more variables by using

p(x1 , x2 , x3 ) = p(x3 |x2 , x1 )p(x2 |x1 )p(x1 )
In order to calculate p(x1 ) we need to marginalise the joint distribution p(x1 , . . . , xn )
over all the other variables (and similar calculations are required for the other con-
ditional distributions). Such marginals, in general, will require the summation over
an exponential number of states, and will, except for small n, generally also be
impractical.

Belief Networks

Here we specify the joint probability by factors of lower dimensional conditional

distributions.
Y
p(x) = p(xi |pa (xi ))
i
 276

For example

p(x1 , . . . , x6 ) = p(x1 )p(x2 )p(x3 |x1 , x2 )p(x4 |x3 )p(x5 |x3 )p(x6 |x4 , x6 )

as shown below.
By making a so-called ancestral ordering (in which parents always come before

x1 x2

x3

x4 x5

x6

Figure 24.1: A Belief Network without any clamped variables.

children), as in the equation above, one can sample first from those nodes that
do not have any parents (here, x1 and x2 ). Given these values, one can then
sample x3 , and then x4 ,x5 and finally x6 . Hence, despite the presence of loops in
the graph, such a forward sampling procedure is straightforward. Any quantity of
interest, for example, a marginal p(x5 ), is approximated by counting the relative
number of times that x5 is in a certain state in the samples.

How can we sample from a distribution in which certain variables are clamped
in evidential values? One approach would be to proceed as above with forward
sampling, and then discard any samples which do not match the evidential states.
This can be extremely inefficient, and is not recommended.

Gibbs Sampling

One of the simplest ways to more effectively account for evidence is to employ a
recursive procedure. One way to motivate the procedure is to assume that someone
has presented you with an sample x1 from the distribution p(x). (For the moment,
we leave aside the issue of evidence). We then consider a particular variable, xi .
We may write

p(x) = p(xi |x1 , . . . , xi−1 , xi+1 , . . . , xn )p(x1 , . . . , xi−1 , xi+1 , . . . , xn )

(One may view this decomposition as xi given all its parents, mutliplied by the
probability of the parents). Since we assume that someone has already provided us
with a sample x1 , from which we can readoff the ‘parental’ state x11 , . . . , x1i−1 , x1i+1 , . . . , x1n ,
we can then draw a sample from

p(xi |x11 , . . . , x1i−1 , x1i+1 , . . . , x1n )

This distribution is usually easy to sample from since it is one-dimensional (this

holds for both discrete and continuous distribtions). We call this new sample (in
which only xi has been updated) x2 . One then selects another variable xj to sam-
ple and, by this procedure, one generate a set x1 , . . . , xP of ‘samples’.
 277

There are a couple of important remarks about this procedure. Clearly, if the
initial sample x1 is not representative – that is, it is fact a part of the state space
that is relatively extremely unlikely, then we should not really expect that the
samples we draw will initially be very representative either. This motivates the
so-called ‘burn in’ stage in which, perhaps 1/3 of the samples are discarded. An-
other remark is that it is clear there will be a high degree of correlation in any two
successive samples, since only one variable is updated. What we would really like
is that each sample x is simply drawn ‘at random’ from p(x) – clearly, in general,
such random samples will not possess the same degree of correlation as those from
Gibbs sampling. This motivates so-called subsampling, in which, say, every 10th ,
sample xK , xK+10 , xK+20 , . . ., is taken, and the rest discarded.

Gibbs sampling is reminiscent of MinConflicts type procedures in Computer Sci-

ence, in which a single variable is updated to see if this is a better or worse solution
to a minimisation problem. Essentially, Gibbs sampling is the generalisation of this
to the stochastic case.

Evidence

Evidence is easy to deal with in the Gibbs sampling procedure. One simply clamps
for all time those variables that are evidential into their evidential states. There
is also no need to sample for these variables, since their states are known.

Despite its simplicity, Gibbs sampling is one of the most useful and popular sam-
pling methods, especially in discrete cases. However, one should bear in mind that
convergence is a major issue – that is, answering questions such as ‘how many
samples are needed to be reasonably sure that my sample estimate p(x5 ) is accu-
rate?’, is, to a large extent, an unknown. Despite many mathematical results in
this area, little is really known about these issues, and general rules of thumb, and
sensible awareness on behalf of the user are rather required. (Indeed, if one were
able to answer such questions, one would understand the distribution well enough
that usually some exact technique would be preferable).

Caution

As with most sampling schemes, a word of caution is required. Whilst there are
some formal results that show that Gibbs sampling (under certain restrictions) is
a correct sampling procedure, one can easily construct cases where it will fail. In
fig(24.2), we show such a case in which the two dimensional continuous distribution
has mass only in the lower left and upper right regions. In that case, if we start in
the lower left region, we will always remain there, and never explore the upper right
region. This problem occurs essentially because there are two regions which are not
connected by a path which is reachable by Gibbs sampling. Such multi-modality
is the scourge of sampling in general, and is very difficult to address.

Importance Sampling
The aim here is to replace sampling with respect to the intractable distribution
p(x), and instead sample from a tractable, simpler distribution q(x). We need to
in someway adjust/reweight the samples from q(x) such that, in the limit of a large
 278
x2

x1

Figure 24.2: A two dimensional distribution for which Gibbs sampling fails. The
upper right region is never explored. This is a case where the sampler is non-
ergodic. For an ergodic sampler there is a non-zero chance any region of the space
will be visited.

number of samples, the correct results will be obtained. Consider the average
Z R
f (x)p∗ (x)
f (x)p(x) = R ∗ dx (24.1.1)
p (x)
R ∗
f (x) pq(x)
(x)
q(x)dx
= R p∗ (x) (24.1.2)
q(x) q(x)dx

Let x1 , . . . xP be samples from q(x), then we can approximate the above by

PP ∗ µ
Z µ p (x ) P
µ=1 f (x ) q(xµ ) X
f (x)p(x) ≈ PP p∗ (xµ ) = f (xµ )rµ
µ=1 q(xµ ) µ=1

where
p∗ (xµ )
µ q(xµ )
r = PP p∗ (xµ )
µ=1 q(xµ )

Hence, in principle, this reweighting of the samples from q will give the correct
result. In high dimensional spaces x, however, the rµ will tend to have only one
dominant value close to 1, and the rest will be zero, particularly if the sampling
distribution q is not well matched to p, since then the ratio q/p will not be close to
unity. However, in a moderate number of dimensions, perhaps less than 10 or so,
this method can produce reasonable results. Indeed, it forms the basis for a simple
class of algorithms called particle filters, which are essentially importance sampling
for temporal Belief Networks (eg non-linear Kalman Filters), in which one forward
samples from a proposal distribution q, and one can exploit the simplified Markov
structure to recursively define reweighting factors.
See http://www-sigproc.eng.cam.ac.uk/smc/index.html for references.

24.2 Markov Chain Monte Carlo (MCMC)

Let’s restate the problem of sampling from a distribution p(x). Let’s write
1 ∗
p(x) = p (x)
Z
where Z is the normalisation constant ofR the distribution. Usually, we will be able
to evaluate p∗ (x), but not Z, since Z = x p∗ (x) is an intractable high dimensional
summation/integration.
 279

Gibbs sampling is a special case of a more general procedure, in which, given a

current sample xµ , we wish to generate a new sample xµ+1 such that together, the
sequence x1 , . . . , xP represents a representative set of samples of p(x).

Understanding MCMC

Consider the conditional distribution p(xt+1 |xt ). If we are given an initial sample
x1 , then we can recursively generate samples x1 , x2 , . . . , xt . After a long time
t >> 1, we can plot the samples xt . Are the samples xt samples from some
distribution and, if so, which distribution? The answer to this is (generally), yes,
they are samples from the stationary distribution p∞ (x) which is defined as
Z
p∞ (x′ ) = p(x′ |x)p∞ (x)
x

This equation defines the stationary distribution, from which we see that the sta-
tionary distribution is equal to the eigenfunction with unit eigenvalue of the tran-
sition kernel. Under some mild properties (STATE!! ergodicity usually required),
every transition distribution has a stationary distribution. This is also unique
(STATE conditions required).
The idea in MCMC is to reverse this process. If we are given the distribution p(x),
can we find a transition p(x′ |x) which has p(x) as its stationary distribution? If we
can, then we can draw samples from the Markov Chain, and use these as samples
from p(x).
Note that whilst (usually) every Markov transition p(x′ |x) has a unique stationary
distribution, every distribution p(x) has a great many different transitions p(x′ |x)
with p(x) as their equilibrium distributions. (This is why there are very many
different MCMC sampling methods for the same distribution).

Detailed Balance

How do we construct transitions p(x′ |x) with given p(x) as their stationary distrib-
utions? One convenient trick is to assume detailed balance. This is the assumption

p(x′ |x)p(x) = p(x|x′ )p(x′ )

which is required to hold for all x and x′ .

If p(x′ |x) and p(x|x′ ) satisfy detailed balance, then
Z Z
′
p(x |x)p(x) = p(x|x′ )p(x′ ) = p(x′ )
x x

That is, detailed balance is a sufficient condition for stationarity. It is not, however,
a necessary condition.
For example, consider drawing samples from the uniform distribution U [0, 1]. One
(rather silly!) way to do this would be to draw x′ as follows. Draw a random
number y from a small interval uniform distribution y ∼ U [0, ǫ] where, say ǫ = 0.5.
Then take x′ to be the value x + y with wrapped boundary conditions. That is, a
point x′ = 1 + δ gets mapped to δ. Clearly, under this scheme, we will eventually
sample correctly from the uniform distribution U [0, 1] however, in a left to right
manner. This clear irreversibility of the chain shows that detailed balance is not
a necessary criterion for correct MCMC sampling.
 280

Metropolis/Hastings Sampling

The detailed balance criterion can be written

p(x′ |x) p(x′ )
= , ∀x, x′
p(x|x′ ) p(x)
This can make the process of constructing a suitable Markov Chain easier since
only the relative value of p(x′ ) to p(x) is required, and not the absolute value of
p(x) or p(x′ ), in the criterion.
For example, in the continuous domain, we might postulate that1
 Z 
p(x′ |x) = q(x′ |x)f (x′ , x) + δ(x′ |x) 1 − q(x′′ |x)f (x′′ , x)
x′′

would be a candidate transition. The reader may verify that this is indeed a
distribution since
Z Z Z
p(x′ |x) = q(x′ |x)f (x′ , x) + 1 − q(x′′ |x)f (x′′ , x) = 1
x′ x′ x′′

The above transition clearly splits into two cases, namely when x′ = x and x′ 6= x.
When x′ = x, then clearly detailed balance trivially holds. In the case x′ 6= x,
then

p(x′ |x) = q(x′ |x)f (x′ , x)

Then we require (for x′ 6= x)

q(x′ |x)f (x′ , x) p(x′ )

′ ′
=
q(x|x )f (x, x ) p(x)
or
f (x′ , x) q(x|x′ )p(x′ )
′
=
f (x, x ) q(x′ |x)p(x)
Then consider the function
q(x|x′ )p(x′ )
f (x′ , x) = min(1, )
q(x′ |x)p(x)

Then if q(x|x′ )p(x′ ) > q(x′ |x)p(x) f (x′ , x) = 1, and f (x, x′ ) = q(x′ |x)p(x)/q(x|x′ )p(x′ ),
and hence
f (x′ , x) 1 q(x|x′ )p(x′ )
′
= ′ ′ ′
=
f (x, x ) q(x |x)p(x)/q(x|x )p(x ) q(x′ |x)p(x)

The reader may show that, conversely, if q(x|x′ )p(x′ ) ≤ q(x′ |x)p(x), we also get
f (x′ ,x) q(x|x′ )p(x′ ) ′
f (x,x′ ) = q(x′ |x)p(x) . Hence the function f (x , x) as defined above is a suitable
′
function to ensure that p(x |x) satisfies detailed balance. This function is called
the Metropolis-Hastings acceptance function. Other acceptance functions may also
be derived.
1 One could contemplate, for example, a normalisation by division style method. However, it is
not necessarily easy to sample from this transition distribution. The beauty of the Metropolis
method is that this subtractive normalisation results in distribution that is easy to sample
from, as we will see.
 281

How do we then sample from p(x′ |x)? Imagine we draw a candidate sample x′
from q(x′ |x). If q(x|x′ )p(x′ ) > q(x′ |x)p(x), then f (x′ , x) = 1, and we must have
x′ 6= x (since, otherwise, p(x) > p(x), which cannot be true) and we simply have
p(x′ |x) = q(x′ |x) – namely we accept the sample x′ . Conversely, if q(x|x′ )p(x′ ) ≤
′
)p(x′ )
q(x′ |x)p(x), then f (x′ , x) = q(x|x ′
q(x′ |x)p(x) , and we cannot rule out that x = x. Hence
 Z 
p(x′ |x) = q(x′ |x)f (x′ , x) + δ(x′ |x) 1 − q(x′′ |x)f (x′′ , x)
x′′

This can be interpreted as a mixture distribution with two distributions

R q(x′ |x) and
δ(x |x), and associated mixture coefficients f (x , x) and 1 − x′′ q(x |x)f (x′′ , x).
′ ′ ′′

To sample from this mixture, we sample from the mixture weight f (x′ , x). We
therefore with probability f (x′ , x) draw a sample from q(x′ |x) (that is, we accept
the candidate) and otherwise take the sample x′ = x. A common mistake in
MCMC is, when we reject the candidate x′ , simply to restart the procedure. The
correct approach is that, if the candidate x′ is rejected, we take the original x as
a new sample. Hence, another copy of x is included in the sample set – ie each
iteration of the algorithm produces a sample – either a copy of the current sample,
or the candidate sample.
The reader may show that Gibbs sampling can be put in the this framework for a
suitable proposal q(x′ |x).
Whilst all of this is quite cool, the reader should bear in mind a couple of points.
Firstly, having a ‘correct’ transition does not guarantee that the samples will indeed
be from p(x). The proposal distribution q(x′ |x) may not explore all regions, in
which case we have not shown detailed balance holds for all points x and x′ . This
is what can happen in the example of the Gibbs sampler, which is not ergodic –
we locally satisfy detailed balance in a region, but not over all space, hence the
samples are not samples from p(x).
Another related point is that, even if we can guarantee ergodicity of the chain, we
have no clue as to how long we need to wait until we have drawn a representative
sample from p(x). The reason for this is essentially that, if the chain is ergodic, then
indeed, eventually, we will be drawing samples from the stationary distribution –
but when?. Assessing convergence is a major headache in MCMC, and in some
sense, just as difficult as sampling from p(x) itself, since we need to have some
global idea of the distribution in order to know how long before we are likely to
have reached a representative point of p(x). In practice, there are some heuristics....

Gaussian Proposal distribution

If we use
1 ′ 2
q(x′ |x) ∝ e− 2σ2 (x −x)

Then q(x′ |x) = q(x|x′ ), and the acceptance criterion is simply

p∗ (x′ )
a=
p∗ (x)
Hence, if the unnormalised probability of the candidate state is higher than the
current state, we accept the candidate. Otherwise, if the unnormalised probability
of the candidate state is lower than the current state, we accept the candidate only
with probability p∗ (x′ )/p∗ (x). Otherwise, the candidate is rejected, and the new
 282

Figure 24.3: One thousand samples from a non-Gaussian distribution. The con-
tours plotted are iso-probability contours. Here, Metropolis sampling was used
with a standard deviation of 1 in each dimension.

sample is taken to be x.

The Metropolis algorithm with isotropic Gaussians above is intuitive, and simple
to implement. However, it is not necessarily very efficient. Intuitively, we will
certainly accept the candidate if the unnormalised probability is higher than the
probability at the current state. We attempt to find a higher point on the distribu-
tion essentially by making a small jump in a random direction. In high dimensions,
it is unlikely that a random direction will result in a value of the probability which
is higher than the current value. Because of this, only very small jumps (which
will typically result in a < 1) will be accepted. However, if only very small jumps
are accepted, the speed with which we explore the space x is extremely slow, and
a tremendous number of samples would be required.

Assessing convergence

.... a bunch of heuristics that I’ll write down sometime....

Auxilliary Variable Methods

General idea is that we wish to sample from p(x). We introduce a distribution
p(y|x), and hence have a joint distribution p(x, y) = p(y|x)p(x). We then draw
samples (xµ , y µ ) from this joint distribution. A valid set of samples from p(x) are
then given by the xµ . In order for this to be useful, we do not always simply draw
a sample from p(x), and then a sample for y from p(y|x). Repeating this would be
senseless. However, if we, for example, where to perform Gibbs sampling, sampling
alternately from p(y|x) and p(x|y) (which is found from p(x|y) ∝ p(y|x)p(x)), the
auxiliary variable may enable us to find an easy sampling distribution p(x|y) and
consequently, we may be able to mode hop.
 283

Hybrid Monte Carlo

This is a method for continuous systems that aims to make non-trivial jumps in
the samples and, in so doing, to jump potentially from one mode to another.
It is customary (though not necessary) to derive Hybrid MCMC in terms of Hamil-
tonians. We will follow this approach here as well.
Let’ define the difficult distribution from which we wish to sample as2
1 Hx (x)
p(x) = e
Zx
for some given ‘Hamiltonian’ H(x). We then define another, ‘easy’ distribution
1 Hy (y)
p(y) = e
Zy

so that we can define a joint distribution

1 Hx (x)+Hy (y)
p(x, y) = p(x)p(y) = e
Z
The algorithm alternates between a Gibbs and a so-called dynamic step. In the
Gibbs step, we simply draw a new value for y from p(y). In the dynamic step, we
draw a sample of p(x) by drawing a sample from p(x, y) and discarding then the
sampled y.

P(x,y) = P(x)P(y)

(x’,y’)

(x,y’)

y
(x,y) x

Figure 24.4: Hybrid Monte Carlo. Starting from the point x, y, we first draw a
new value for y from the Gaussian p(y). Then we use Hamiltonian dynamics to
traverse the distribution at roughly constant energy H(x, y) to reach a point x′ , y ′ .
We accept this point if H(x′ , y ′ ) > H(x, y ′ ). Otherwise this candidate is accepted
with probability exp(H(x′ , y ′ ) − H(x, y ′ )).

In the standard form of the algorithm, we choose a multi-dimensional Gaussian

auxiliary distribution, usually with dim(y) = dim(x):
1
Hy (y) = − y T y
2
The Gibbs step is the trivial since we just sample from the Gaussian p(y).
2 In physics, the convention is to write p(x) = Z1 e−Hx (x) since probable states correspond to
x
low energy (Hamiltonian) states. De gustibus non est disputandum!
 284

The dynamic step is a Metropolis step, with a very special kind of proposal dis-
tribution. In the dynamic step, the main idea is to go from one point of the space
x, y to a new point x′ , y ′ that is a non-trivial distance from x, y and which will
be accepted with a high probability. In the basic formulation using a symmetric
proposal distribution, we will accept the candidate x′ , y ′ if the values H(x′ , y ′ ) is
higher than or close to the value H(x, y ′ ). How can such a non-trivial distance
between x, y and x′ , y ′ be accomplished? One way to do this is to use Hamiltonian
dynamics.

Hamiltonian Dynamics

Imagine that we have a Hamiltonian H(x, y). We wish to make an update x′ =

x + ∆x, y ′ = y + ∆y for small ∆x and ∆y. Hamiltonian dynamics is defined by
the requirement of energy preservation. That is,

H(x′ , y ′ ) = H(x, y)

We can satisfy this (up to first order) by considering the Taylor expansion

H(x′ , y ′ ) = H(x + ∆x, y + ∆y)

≈ H(x) + ∆xT ∇x H(x, y) + H(y) + ∆y T ∇y H(x, y) + O(|∆x|2 ) + O(|∆y|2 )
(24.2.1)

The energy preservation requirement is then that

∆xT ∇x H(x, y) + ∆y T ∇y H(x, y) = 0

This is a single scalar requirement, and there are therefore many different solutions
for ∆x and ∆y that satisfy this single condition. In physics, it is customary to
assume isotropy, which limits dramatically the number of possible solutions to
essentially just the following:

∆x = ǫ∇y H(x, y) ∆y = −ǫ∇x H(x, y)

where ǫ is a small value to ensure that the Taylor expansion is accurate. Hence

x(t + 1) = x(t) + ǫ∇y H(x, y) y(t + 1) = y(t) − ǫ∇x H(x, y)

Defines Hamiltonian dynamics – a locally energy preserving dynamical system.

Discarding the isotropy requirement opens up the possibility for other dynamics,
for example, relativistic dynamics. In practice, however, the simple Hamiltonian
dynamics is usually considered adequate.
For the Hybrid MC method, H(x, y) = H(x)+H(y), so that ∇x H(x, y) = ∇x H(x)
and ∇y H(x, y) = ∇y H(y) For the Gaussian case, ∇y H(x, y) = −y.

x(t + 1) = x(t) − ǫy y(t + 1) = y(t) − ǫ∇x H(x)

There are specific ways to implement the dynamic equations above (called Leapfrog
discretization) that are more accurate – see the Radford Neal reference. (Special
case of simplectic discretization I think).
We can then follow the Hamiltonian dynamics for many time steps (usually of the
order of several hundred) to reach a candidate point x′ , y ′ . If the Hamiltonian
dynamics was well behaved, H(x′ , y ′ ) will have roughly the same value as H(x, y).
 285

We then do a Metropolis step, and accept the point x′ , y ′ if H(x′ , y ′ ) > H(x, y)
and otherwise accept it with probability exp(H(x′ , y ′ ) − H(x, y)). If rejected, we
take the initial point x, y as the sample.
In order to make a symmetric proposal distribution, at the start of the dynamic
step, we choose ǫ = +ǫ0 or ǫ = −ǫ0 uniformly. This means that there is the same
chance that we go back to the point x, y starting from x′ , y ′ , as vice versa.
Combined with the Gibbs step, we then have the general procedure.
1. Start from x, y. Draw a new sample y ′ from p(y).
2. Then, starting from x, y ′ , choose a random (forwards or backwards) and
then perform Hamiltonian dynamics for some time steps until we reach a
candidate x′ , y ′ . Accept x′ , y ′ if H(x′ , y ′ ) > H(x, y), otherwise accept it with
probability exp(H(x′ , y ′ ) − H(x, y)). If rejected, we take the sample as x, y ′ .
3. The above steps are repeated.

One obvious feature of HMC is that we now use, not just the potential H(x) to find
define candidate samples, but the gradient of H(x) as well. An intuitive reason
for the success of the algorithm is that it is less myopic than straightforward
Metropolis, since the use of the gradient enables the algorithm to feel it’s way
to other regions of high probability, by following at all times likely paths in the
augmented space. One can also view the auxiliary variables as momentum variables
– it is as if the sample has now a momentum. Provided this momentum is high
enough, we can escape local minima......more later.

Slice Sampling

Blah

Swendson-Wang

This is a classic algorithm used for discrete variables. The main motivation here
is to introduce p(y|x) in such a way that the distribution p(x|y) is easy to sample
from.
Originally, the SW method was introduced to alleviate the problems encountered
in sampling from Ising Models close to their critical temperature, in which Gibbs
sampling completely breaks down.
In it’s simplest form, the Ising model with no external interactions on a set of
variables x1 , . . . , xn , xi ∈ {0, 1} takes the form
1 Y βI[xi =xj ]
p(x) = e
Z i∼j

which means that this is a pairwise Markov network with a potential contribution
eβ if neighbouring nodes i and j are in the same state, and a contribution 1
otherwise. We assume that β > 0 which encourages neighbours to be in the same
state. The lattice based neighbourhood structure makes this difficult to sample
from, and especially when the inverse temperature encourages large scale islands
to form. In that case, the probability of an individual variable being flipped by
Gibbs sampling is negligible.
 286

We wish to introduce a distribution p(y|x) so that p(x|y) is easy to sample from.

The easiest kind of distributions to sample from are factorised. Let’s see if we can
make p(x|y) factorised.
Y
p(x|y) ∝ p(y|x)p(x) ∝ p(y|x) eβI[xi =xj ]
i∼j

It’s clear that we need to employ p(y|x) to cancel the terms eβI[xi =xj ] . We can do
this by making
Y Y 1
p(y|x) = p(yij |xi , xj ) = I[0 < yij < eβI[xi =xj ] ]
i∼j i∼j
z ij

where I[0 < yij < eβI[xi =xj ] ] denotes a uniform distribution between 0 and
eβI[xi =xj ] . zij is the normalisation constant zij = eβI[xi =xj ] . Hence

p(x|y) ∝ p(y|x)p(x) (24.2.2)

Y 1
∝ βI[xi =xj ]
I[0 < yij < eβI[xi =xj ] ]eβI[xi =xj ] (24.2.3)
i∼j
e
Y
∝ I[0 < yij < eβI[xi =xj ] ] (24.2.4)
i∼j

Let’s assume that we have a sample yij . If yij > 1 then to draw a sample from
p(x|y), we must have 1 < eβI[xi =xj ] , which means that xi and xj are in the same
state. Otherwise, if yij < 1, then what constraint does this place on what the x
can be? None! Hence, wherever yij > 1, we bond xi and xj to be in the same
state. The probability
Z ∞
1 eβ − 1
yij > 1 = I[0 < yij < eβI[xi =xj ] ] β = 1 − e−β
yij =1 zij e

Hence, if xi = xj , we bind xi and xj to be in the same state with probability

1 − e−β . After doing this for all the xi and xj pairs, we will end up with a graph
in which we have clusters of like-state bonded variables. Which state each cluster
is equal. Hence the algorithm simply chooses a random state for each cluster.
The algorithm then does the following:
1. If xi = xj , we bond variables xi and xj with probability 1 − e−β . Repeat
this for all variables.
2. For each cluster formed from the above, set the state of the cluster uniformly
at random.
3. Repeat the above steps.
Pictures from Higdon. Matlab code.

Temporal Distributions
Many applications involve temporal distributions of the generic form
T
Y
p(v1:T , h1:T ) = p(v1 |h1 )p(h1 ) p(vt |ht )p(ht |ht−1 )
t=2
 287

We encountered a few already, namely the Kalman Filter, Hidden Markov Model
and Switching Kalman Filter. Our interest here will be in the calculation of
p(ht |v1:T ).
In the mentioned models, we have used either exact inference methods, or devel-
oped (in the SKF case) approximate inference methods. However, there are cases
where the transitions are such that it may not be clear how to form an appropriate
analytic approximation procedure (although, in my experience, such situations are
rare), and more general numerical approximations are sought.
It should be born in mind that tayloring the approximation method to the model
at hand is usually vital for reasonable performance. Nevertheless, we’ll discuss
below some fairly general sampling procedures that may be brought to bear, and
have proved popular, mainly due to their implementational simplicity.

h1 h2 h3 h4

v1 v2 v3 v4

Figure 24.5: A Switching Kalman Filter. The variables h and v are Gaussian dis-
tributed. The Switch variables s are discrete, and control the means and variances
of the Gaussian transitions.

Particle Filters

Despite our interest in p(ht |v1:T ), PFs make the assumption that the so-called
‘filtered estimate’ p(ht |v1:t ) would be a reasonable approximation or, at least, a
quantity of interest.
The traditional viewpoint of a Particle Filter is as a recursive importance sampler.
Here, we show how it can be viewed as a (somewhat severe) approximation of the
Forward Pass in Belief Propagation.
Z
ρ(ht ) ∝ p(vt |ht ) p(ht |ht−1 )ρ(ht−1 ) (24.2.5)
ht−1

where the ρ message has the interpretation

ρ(ht ) ∝ p(ht |v1:t )
A PF can be viewed as an approximation of equation (24.2.5) in which the message
ρ(ht−1 ) has been approximated by a sum of δ-peaks:
L
X
ρ(ht−1 ) = wt−1 (i)δ (ht−1 − ht−1 (i)) (24.2.6)
i=1
PL
where wt−1 (i) are mixture components i=1 wt−1 (i) = 1, and the points ht−1 (i)
are given values. In other words, the ρ message is represented as a weighted mixture
of delta-spikes where the weight and position of the spikes are the parameters of
the distribution.
Using equation (24.2.6) in equation (24.2.5), we have
L
X
ρ(ht ) = kp(vt |ht ) p(ht |ht−1 (i))wt−1 (i) (24.2.7)
i=1
 288

The constant k is used to make ρ(ht ) a distribution. Although ρ(ht−1 ) was a simple
sum of delta peaks, in general ρ(ht ) will not be – the peaks get ‘broadened’ by the
hidden-to-hidden and hidden-to-observation factors. One can think of many ways
to approximate ρ(ht ).
In PFs, we make another approximation of ρ(ht ) in the form of a weighted sum of
delta-peaks. There are many ways to do this, but the simplest is to just sample
a set of points from the (unnormalised) distribution equation (24.2.7). There are
many ways we could carry out this sampling. One simple way is to that equation
(24.2.7) represents a mixture distribution.

Sampling from an unnormalised mixture using Importance Sampling

Consider
I
1 X
p(h) = wi φi (h)
Z i=1

How can we sample from this distribution? Clearly, there are many approaches.
A simple idea is to use importance sampling.

Z
1 X φi (h)
hf (h)i = wi qi (h)f (h) (24.2.8)
Z i h qi (h)
1 X X φi (hµi )
≈ wi f (hµi ) (24.2.9)
Z i µ
qi (hµi )
P P µ
µ φi (hi )
i wi µ f (hi ) qi (hµi )
≈ P P φi (hµi ) (24.2.10)
i wi µ qi (hµi )
P µ µ
i,µ ri f (hi )
≈ P µ (24.2.11)
i,µ ri

φ (hµ )
where riµ ≡ wi qii(hµi ) If, say for each mixture component i, we generate a set of P
i
samples hµi , µ = 1, . . . , P , then we will have I × P weights riµ . We then need to
select from this set, a smaller set (usually of size I again) points riµ . This can either
be done by discarding small riµ , or sampling from the unnormalised distribution riµ .
One done, we have a set of retained riµ , from which a new set of mixture weights wi∗
can be found by normalising the selected weights. Heuristics are usually required
since, as is nearly always the case with naive IS, only a few of the weights will
have significant value – exponential dominance problem. In practice, repopulation
heuristics are usually employed to get around this. And other hacks.....

A better sampling approach

In my humble opinion, there is little advantage in using the (very poor) importance
sampler. Rather, it is better to look again at the equation
Z
ρ(ht ) ∝ p(vt |ht ) p(ht |ht−1 )ρ(ht−1 ) (24.2.12)
ht−1

This can be interpreted as the marginal of the two time potential

ρ(ht , ht−1 ) ∝ p(vt |ht )p(ht |ht−1 )ρ(ht−1 )
 289

Assuming that we have a sample hµt−1 from ρ(ht−1 ), we can then draw a sample
ht from ρ(ht , ht−1 ) by Gibbs sampling, ie by sampling from the unnormalised
conditional ρ(ht , ht−1 = hµt−1 ). For each sample, we can then proceed to the next
time step. This will then generate a single sample path hµ1 , hµ2 , . . . hµT . We repeat
this procedure to get a set of sample paths (this can, of course, be also done in
parallel, so that we generate at each time step, a set of sample hµt , µ = 1, . . . P .
The advantage of this approach is that any of the more powerful sampling methods
developed over the last 50 years can be used, and one is not hampered by the
miserable performance of importance sampling.
 290

% DEMO for sampling from a 2D distribution

close all; hold on

x(:,1)=randn(2,1); % intial sample

s=1; % width of Metropolis candidate distribution

yj=0; xi=0; xxx=-6:0.1:6; yyy=-6:0.1:6; for yy=yyy

yj=yj+1; xi=0;
for xx=xxx
xi=xi+1;
z(xi,yj)=exp(logp([xx yy]));
end
end
contour(xxx,yyy,z,20); % iso-probability contours

for mu=2:1000
x(:,mu)=metropolis(x(:,mu-1),s,’logp’);
plot(x(1,mu),x(2,mu),’.’);
if mod(mu,100)==1
drawnow
end
end

function xnew=metropolis(x,s,logp)

% Get a new sample xnew from distribution exp(logp) using

% metropolis sampling with standard deviation s, and current sample x

xcand=x+s*randn(size(x));
loga=feval(logp,xcand)-feval(logp,x);
if loga>0
xnew=xcand;
else
r=rand;
if log(r)<loga
xnew=xcand;
else
xnew=x;
end
end

function l=logp(x)

% make a non-Gaussian distribution.

% larger f makes the distribution more bimodal.

l1 = exp(- (x(1)^2+x(2)^2+sin(x(1)+x(2))^2));
f=3;
l2 = exp(- ((x(1)-f)^2+(x(2)-f)^2+sin(x(1)+x(2)-2*f)^2));
l=log(l1+l2);
 291

Rao-Blackwellisation
Explain why this is very often a red-herring since it assumes that one has a good
sampler (which indeed is the whole problem of sampling!). Give a picture where it’s
easy to sample if high dimensions, but more multi-modal in the lower projected
dimension, compounding the difficulty of sampling. (Rao-Blackwellisation says
that the variance of the sampler will always be higher in the higher space – but
that is in fact a good thing in many practical cases.) My feeling is that RB is just
another inappropriate piece of theory that misses the point.
Appendix A Basic Concepts in Probability

A.1 What does random mean?

Arguably, it’s ultimately a question of physics as to whether or not the world is
inherently random. That is, are phenomena that appear random just examples of
highly complex processes, or does, indeed, ‘God play dice’[60]? The concepts of
randomness and uncertainty are intrinsically linked, and it is our desire to deal
with uncertain processes that motivates us here.
A viewpoint that we will often use here is that of a generating process. This
viewpoint stems from the idea that we only have a limited ability to observe the
world, and that we may not be able to observe all associated conditions relevant
to the generating process. For example, Sally tosses a coin – this is the generating
process. We can certainly observe the outcome of this process – whether the
coin is heads or tails – these states are observable or visible. However, we may
not be able to observe all the relevant conditions which may have been involved
in the generating process, such as the state of Sally’s finger, whether or not she
subconsciously has a preference for holding the coin in her hand a certain way –
the states of such conditions as assumed hidden.
One pragmatic definition of randomness would be that, given the current (limited)
information about the process which generates the data, the outcome of the exper-
iment (generation) cannot be predicted which certainty. However, we would like
to be able to make statements about the outcomes of a generating process, even
though it may be predicted without certainty. That is, we wish to form a calculus
for uncertainty.

A.2 What is Probability?

Probability is a contentious topic, stretching over several hundred years. It is not
necessarily the axioms of probability that are contentious, rather what, if any,
physical reality do they correspond to? A summary of the history of these ideas
and interpretations is given in [61]. Personally, I very much like the phrase[62]
Probability theory is nothing but common sense reduced to computation (Laplace).
One approach to thinking of probability is that it enables us to make compact
descriptions of phenomena.
Science is about finding compact descriptions of phenomena, and the concepts of
probability/randomness help us to achieve this.
For example, Sally has a coin, and does some experiments to investigate its prop-
erties. She tosses the coin, and find the outcomes

H, T, H, H, T, T, H, H, H, T, H, T, H

Her professor asks her to make an analysis of the results of her experiments. Ini-
tially, Sally is tempted to say that she does not wish to summarise the experiments,

292
 293

since this would constitute a loss of information. However, she realises that there is
little to be gained from simply reporting the outcomes of the experiment, without
any summarisation of the results.
Independence → compact She therefore make an assumption (which she states in her report), namely that
description the outcome of tossing a coin at one time, did not influence the outcome at any
other time (so, if the coin came up heads now, this will not influence whether
or not the coin comes up heads or tails the next throw). This is a common
assumption and called independence of trials. Under this assumption, the ordering
of the data has no relevance, and the only quantity which is therefore invariant
under this assumption is the total number of heads, and the total number of tails
observed in the experiment. See how this assumption has enabled us to make
a compact description of the data. Indeed, it is such independence assumptions,
their characterisation and exploitation that is the subject of graphical models.
Random → compact model Sally repeats the experiment many times, and notices that the ratio of the number
description of heads observed to the total number of throws tends to roughly 0.5. She therefore
summarises the results of her experiments by saying that, in the long run, on
average, she believes half the time the coin will end up heads, and half the time
tails. Key words in the above sentence are believes and long run. We say believes
since Sally cannot repeat the experiment an infinite number of times, and it is
therefore her belief that if she were to toss the coin an infinite number of times, the
number of heads occurring would be half that of the total. Hence, she invokes the
concept of randomness/probability to describe a model that she believes accurately
reflects the kind of experimental results she has found.
In a sense, Sally (and her environment) operates like a random number generator.
(If we knew all the possible things that could influence Sally, and the coin, and how
she might toss it, we might conclude that we could predict the outcome of the coin
toss – there is nothing ‘random’ about it. However, to do so would not render itself
to a compact description – hence the usefulness of the concept of randomness).

A.3 Rules of Probability

Events and the Rules

Events are possible outcomes of a generating process. For example ‘the coin is
heads’ is and event, as is ‘the coin is tails’. In this case, these two events are
mutually exclusive, since they cannot both simultaneously occur.
• We’ll take the pragmatic viewpoint that the probability of an event occurring
is simply a number p(event occurs) between 0 and 1.
• p(event occurs) = 0 means that it is certain that the event cannot occur.
Similarly, p(event occurs) = 1 means that it is certain that the event occurs.
• We need a rule for how events interact :

p(x or y) = p(x) + p(y) − p(x and y)

I’ll use p(x, y) to denote the joint event p(x and y).
 294

Conditional Probability / Bayes Rule

A useful definition is that of conditional probability. The probability of event x

conditional on knowing that event y has occurred (or more shortly, the probability
of x given y) is defined as

p(x, y)
p(x|y) ≡
p(y)

Here’s one way to think about conditional probability : Imagine we have a dart
board, split into 20 sections. We may think of a drunk dart thrower, and describe
our lack of knowledge about the throwing by saying that the probability that a
dart occurs in any one of the 20 regions is p(region i) = 1/20. Imagine then, that
our friend, Randy, the random dart thrower is blindfolded, and throws a dart.
With pint in hand, a friend of Randy tells him that he hasn’t hit the 20 region.
What is the probability that Randy has hit the 5 region?
Well, if Randy hasn’t hit the 20 region, then only the regions 1 to 19 remain and,
since there is no preference for Randy for any of these regions, then the probability
is 1/19. To see how we would calculate this with the rules of probability :

p(region 5, not region 20)

p(region 5| not region 20) =
p( not region 20)
p(region 5) 1/20 1
= = =
p(not region 20) 19/20 19

giving the intuitive result.

Degree of Belief

The above dart board example is easy to think about in terms of probability –
it’s straightforward to imagine many repetitions of the experiment, and one could
think about the ‘long’ run way of defining probability.
Here’s another problem :
What’s the probability that I will like the present my grandmother has bought
me for Christmas? In a purely technical sense, if we were to define probability
as a limiting case of infinite repetitions of the same experiment, this wouldn’t
make much sense in this case. We can’t repeat the experiment. However, simply
the predictability or degree of belief interpretation sidesteps this issue – it’s just
a consistent framework for manipulating real values consistent with our intuition
about probability.
TODO: Confidence intervals, Central limit theorem, Asymptotic Equipartition
Theorem.
Appendix B Graph Terminology:

B.1 Graphs : Basic Concepts and Definitions

Parts of this section are taken from Expert Systems and Probabilistic Network
Models by E. Castillo, J. Gutierrez, and A. Hadi (Springer, 1997), and also An in-
troduction to bayesian networks by F.V.Jensen (Springer 1996). Both are excellent
introductions to the field.

B.2 Basic Concepts and Definitions

Graph A graph G = (X, L) is defined by two sets X and L, where X is a finite set of
nodes X = {X1 , X2 , . . . Xn } and L is a set of links (also called connections), that
is, a subset of all possible ordered pairs of distinct nodes.
For example in fig(B.1a), we have

X = {A, B, C, D, E, F } , L = {LAD , LDB , LBC , LF D , LEF , LDE }

Directed and Undirected Let G = (X, L) be a graph. When Lij ∈ L and Lij ∈ L, the link Lji 6∈ L is called
Links a directed link. Otherwise, the link is called undirected.
Directed and Undirected A graph in which all the links are directed is called a directed graph and a graph in
Graphs which all the links are undirected is called an undirected graph. A graph containing
both directed and undirected links is called a chain graph.
Adjacency Set Given a graph G = (X, L) and a node Xi , the adjacency set of Xi is the set of
nodes directly attainable from Xi , that is Adj (Xi ) = {Xj |Lij ∈ L}. That is, in a
directed graph, the adjacency set of a node G is the set of nodes that G points to.
In an undirected graph, it is the set of nodes that G is directly connected to.
The directed graph in fig(B.1a) has adjacency sets

Adj {A} = {D} , Adj {B} = {C} , Adj {C} = ∅,

Adj {D} = {B, E} , Adj {E} = {F } , Adj {F } = {D} ,

A A

B D F B D F

C E C E
(a) (b)

Figure B.1: (a) A directed graph (b) An undirected graph

295
 296

The undirected graph in fig(B.1b) has adjacency sets

Adj {A} = {B, E} , Adj {B} = {C, D} , Adj {C} = {B, F } ,

Adj {D} = {A, B, E, F } , Adj {E} = {D, F } , Adj {F } = {D, E, C} ,

B.2.1 Undirected Graphs

Complete set A subset of nodes S of a graph G is said to be complete if there are links between
every pair of nodes in S. Correspondingly, a graph is complete if there is a link
between every pair of nodes in the graph.
For example, fig(B.2.1) is a complete graph of 5 nodes.

B C

D E

Figure B.2: A complete graph with 5 nodes

A F A F

E E

B G B G

D D

C H C H
(a) (b)

Figure B.3: Two different graphs and their associated cliques. (a) Here there are
two cliques of size 3. (b) Here there are several cliques of size three and one clique
of size 4. Links belonging to cliques of size two have not been coloured.

Clique A complete set of nodes C is called a clique if C is not a subset of another complete
set. In other words, a clique cannot be expanded since the expanded set would
not be complete.
For example, in fig(B.3a), the cliques are

{E, F, G} {D, G, E} {A, B} {B, C} {C, D} {D, H} {A, E}

DEF G is not a clique since there is no connection between D and F .

In fig(B.3b), the cliques are

{A, B, D, E} , {B, C, D} , {D, H} , {D, E, G} , {E, F, G}

ABE is not a clique since this is a complete subset of a larger complete subset,
namely ABED.
 297

A B A B

C D D

E F E F

G H G H

I J I J
(a) (b)

E F

G H

I J
(c)

Figure B.4: (a) A disconnected graph. (b) A Tree (c) A loopy graph

Loop A loop is a closed path (a series of nodes with intersecting adjacency sets) in an
undirected graph.
For example, B − D − F − C − D in fig(B.1b) is a loop.
Neighbours of a node The set of nodes adjacent to a node Xi in an undirected graph is referred to as
the neighbours of Xi , N br (Xi ) = {Xj |Xj ∈ Adj (Xi )}
So, in an undirected graph, the neighbours of a node are identical to the adjacency
set of that node.
Connected Undirected An undirected graph is connected if there is at least one path between every pair
Graphs on nodes. Otherwise, the graph is disconnected.
For example, fig(B.4a) is an disconnected graph.
Tree A connected undirected graph is a tree if for every pair of nodes there exists a
unique path.
For example fig(B.4b) is a tree.
Multiply-connected or Loopy A connected undirected graph is multiply-connected (or loopy) if it contains at
Graphs least one pair of nodes that are joined by more than one path, or equivalently, if
it contains at least one loop.
 298

A B A B

C D C D

E F E F

G H G H

I J I J
(a) (b)

Figure B.5: (a) Parents (A, B) and children (E, F ) of node D (b) C → E → G →
I → C is a cycle.

A A B

D C D

E F E F

G H G H

I J I J
(a) (b)

Figure B.6: (a) A Simple Tree (b) A polytree

For example fig(B.4c) is a loopy graph.

B.2.2 Directed Graphs

Parents and Children When there is a directed link from Xi to Xj , then Xi is said to be a parent of Xj ,
and Xj is a child of Xi .
For example, in fig(B.5a) the parents of node E are C and D. Node E has only
one child, node G.
Cycle A cycle is a closed directed path in a directed graph. If a graph contains no cycles,
it is called acyclic.
For example C, in fig(B.5b) C → E → G → I → C is a cycle. The nodes along
the path connected D, F, H, J, G, E, D do not form a cycle since the directions of
the links are not consistent.
Family The set consisting of a node and its parents is called the family of the node.
Simple Trees and Polytrees A directed tree is called a simple tree if every node has at most one parent. Oth-
 299

erwise it is called a polytree.

For example, in fig(B.6a), the graph is a simple tree, whereas fig(B.6b) is a polytree.
Markov Blanket The Markov blanket of a node A consists of the parents of node A, the children of
node A, and the parents of the children of node A. For example, in fig(B.6b), the
Markov Blanket of node D is {A, B, C, E, F }.
Appendix C Some Standard Distributions:

Gamma Distribution

 γ−1
1 x − x0
e− β (x−x ) ,
1 0
p(x) = x ≥ x0 , β > 0 (C.0.1)
βΓ(γ) β
γ is called the shape parameter, x0 the location parameter, β is the scale parameter
and
Z ∞
Γ(a) = ta−1 e−t dt
0

The parameters are related to the mean and variance as follows

 µ 2 s2
γ= , β=
s µ
where µ is the mean of the distribution and s is the standard deviation. In practice
we often encounter a zero location distribution
1 γ−1 −x/β
p(x) = x e , x ≥ 0, β > 0
Z
Need some plots.... Basically a unimodal distribution with a bump which can be
placed close to the location x0 .

Dirichlet Distribution

The Dirichlet distribution is a distribution on probability distributions:

Q
! Q
1 X Y
p(α) = δ αi − 1 αuq q −1
Z(u) i=1 q=1

where
QQ
Γ(uq )
Z(u) = q=1
P 
Q
Γ q=1 u q

It is conventional to denote the distribution as

Dirichlet(α|u)

The parameter u controls how strongly the mass of the distribution is pushed to
the corners of the simplex. Setting uq = 1 for all q corresponds to a uniform
distribution.
In the binary case Q = 2, this is also called a Beta distribution.

300
Appendix D Bounds on Convex Functions

D.1 Kullback Leibler Divergence KL(q||p)

The KL diveregence KL(q||p) measures the “difference” between distributions q
and p, fig(D.1a). In many ways, this is a natural measure to use, and is well
motivated from Information theoretic arguments.

x-1 > log(x) x-1

log(x)
(1,0)
p(x)
q(x)

(a) (b)

Figure D.1: (a) The probability density functions for two different distributions
p(x) and q(x). We would like to numerically characterise the difference between
these distributions. (b) A simple linear bound on the logarithm enables us to
define a useful distance measure between distributions (see text).

For two distributions q(x) and p(x), it is defined as

KL(q||p) ≡ hlog q(x) − log p(x)iq(x)

where the notation hf (x)ir(x) denotes average of the function f (x) with respect
to the distribution r(x). For a continuous variable, this would be hf (x)ir(x) =
R P
f (x)r(x)dx, and for a discrete variable, we would have hf (x)ir(x) = x f (x)r(x).
The advantage of this notation is that much of the following holds independent of
whether the variables are discrete or continuous.
KL(q||p) ≥ 0 The KL divergence is always ≥ 0. To see this, consider the following simple linear
bound on the function log(x) (see fig(D.1b)):
log(x) ≤ x − 1
Replacing x by p(x)/q(x) in the above bound
p(x) p(x)
− 1 ≥ log ⇒ p(x) − q(x) ≥ q(x) log p(x) − q(x) log q(x)
q(x) q(x)
R
Now
R integrate (or sum in the case of discrete variables) both sides. Using p(x)dx =
1, q(x)dx = 1, and rearranging gives
Z
{q(x) log q(x) − q(x) log p(x)} dx ≡ hlog q(x) − log p(x)iq(x) ≥ 0

Furthermore, one can show that the KL divergence is zero if and only if the two
distributions are exactly the same.

301
 302

D.1.1 Jensen vs Variational Jensen

Imagine we wish to bound the quantity

Z
J = log p(x)f (x)
x

where f (x) ≥ 0. Then Jensen’s bound is

Z
J ≥ p(x) log f (x)
x

By considering a distribution r(x) = p(x)f (x)/Z, and variational distribution q(x),

we have
Z
J ≥ (−q(x) log q(x) + q(x) log p(x) + q(x) log f (x))
x

Interestingly, one can write this as

Z
J ≥ −KL(q(x)||p(x)) + q(x) log f (x)
x

Furthermore, one can recognise the second term as a KL divergence too:

J ≥ −KL(q(x)||p(x)) − KL(q(x)||f (x)) − H(q(x))

Clearly these two bounds are equal for the setting q(x) = p(x) (this reminds me of
the difference in sampling routines, between sampling from the prior and sampling
from a distribution which is more optimal for calculating the average). The first
term encourages q to be close to p. The second encourages q to be close to f , and
the third encourages q to be sharply peaked.
Appendix E Positive Definite Matrices and Kernel Functions:

Outer Product Representation of Matrices

If we have a symmetric matric A = AT , then (a fundamental theorem) then two

eigenvectors ei and ej of A are orthorgonal (ei )T ej = 0 if their eigenvalues λi and
λj are different. This is easy to show by considering:

Aei = λi ei

Hence

(ej )T Aei = λi (ej )T ei

But, since A is symmetric, the LHS is equivalent to

((ej )T A)ei = (Aej )T ei = λj (ej )T ei

Hence we must have λi (ej )T ei = λj (ej )T ei . If λi 6= λj , the only way this condition
can be satisfied is if (ej )T ei = 0 – ie, the eigenvectors are orthogonal.
This means also that we can represent a symmetric matrix as
X
A= λi ei (ei )T
i

Hermitian Matrices

A square matrix is called Hermitian if

A = AT ∗

where ∗ denotes the conjugate operator. The reader can easily show that for
Hermitian matrices, the eigenvectors form an orthogonal set, and furthermore
that the eigenvalues are real.

Positive Definite Matrix

A matrix A is positive definite if and only if

y T Ay > 0

for any real y 6= 0.

Using the above result, this means that
X X
y T Ay = λi y T ei (ei )T y = λi (y T ei )2
i i

This is clearly greater than zero if and only if all the eigenvalues are positive. This
is therefore an equivalent condition for positive definiteness.

303
 304

Kernel

Kernels are closely related to the idea of Gaussian Processes. See [63] for an
excellent review. Consider a collection of points, x1 , . . . , xP , and a symmetric
function K(xi , xj ) = K(xj , xi ). We can use this function to define a symmetric
matrix K with elements
[K]ij = K(xi , xj )
The function K is called a Kernel if the corresponding matrix (for all P ) is positive
definite.

Eigenfunctions

Z
K(x′ , x)φa (x) = λa φa (x′ )
x
By an analogous argument that proves the theorem of linear algebra above, the
eigenfunctions are orthogonal:
Z
φa (x)φ∗b (x) = δab
x

where φ∗ (x) is the complex conjugate of φ(x)1 . From the previous results, we know
that a symmetric real matrix K must have a decomposition in terms eigenvectors
with positive, real eigenvalues. Since this is to be true for any dimension of matrix,
it suggests that we need the (real symmetric) kernel function itself to have a
decomposition (provided the eigenvalues are countable)
X
K(xi , xj ) = λµ φµ (xi )φ∗µ (xj )
µ

since then
X X X X X
yi K(xi , xj )yj = λµ yi φµ (xi )φ∗µ (xj )yj = λµ ( yi φµ (xi )) ( yi φ∗µ (xi ))
i,j i,j,µ µ i i
| {z }| {z }
zi zi∗

which is greater than zero if the eigenvalues are all positive (since for complex z,
zz ∗ ≥ 0).
If the eigenvalues are uncountable (which happens when the domain of the kernel
is unbounded), the appropriate decomposition is
Z
K(x , x ) = λ(s)φ(xi , s)φ∗ (xj , s)ds
i j

How can we check if K is a Kernel?

It is not always straightforward to check if K is a kernel. If we can show that K

can be expressed in the form
X
K(x, x′ ) = ψi (x)ψi (x′ )
i
1 This definition of the inner product is useful, and natural particularly in the context of trans-
lation invariant kernels. We are free to define the inner product, but this conjugate form is
often the most useful.
 305

for real functions ψi (x), then K is a Kernel. Note that we do not require the
functions to be orthogonal. However, we know that if K is a Kernel, then an
alternative expansion in terms of orthogonal functions exists. More generally, K
is a Kernel if it has a representation:
Z
K(x, x ) = ψ(x, s)ψ(x′ , s)ds
′

for real functions ψ(x, s).

A second approach would be to find the eigenfunctions ea (x) of K,
Z
k(x, x′ )ea (x′ )dx′ = λa ea (x)

and show that the eigenvalues λa are all positive.

Translation Invariance

In the case that k(x, x′ ) = k(x − x′ ) (note that we do not impose symmetry here
of the function k(x, x′ ), so that this section holds for more general functions than
kernels), the function is called stationary. The eigenproblem becomes
Z
k(x − x′ )e(x′ )dx′ = λe(x)

In this case, the LHS is in the form of a convolution. It makes sense therefore to
take the Fourier Transform:

k̃ẽ = λẽ

This means that ẽ is a delta function, and that therefore the eigenfunctions are
eisx .
When k is Hermitian, it has an expansion in terms of eigenfunctions. Note that the
form of the conjugate expansion automatically ensures that the Kernel is transla-
′
tion invariant, since φ(x+ a)φ∗ (x′ + a) = eis(x+a−x −a) = φ(x)φ∗ (x′ ). (Indeed, this
shows generally why Fourier representations are central to systems which possess
translation invariance). Exercise: what happens if we take the Laplace transform
of a translation invariant operator?

Application to the Squared Exponential Kernel

For the squared exponential kernel, the Fourier Transform of the kernel is a
2
Gaussian, and hence λ(s) = e−s . Hence, we have a representation of the ker-
nel as
Z Z
−(x−x′ )2 −s2 isx −isx′ 2 ′
e = e e e ds = e−s +is(x−x ) ds (E.0.1)

The reader may verify that this is indeed an identity by considering that this is
just a rewriting of the Fourier Transform of a Gaussian:
Z
2 2
e−w = e−x eiwx dx

The form of the representation equation (E.0.1) of the kernel verifies that it is
indeed a kernel, since the eigenvalues are all positive.
 306

Blochner’s Theorem

For a stationary process (the Kernel is translation invariant), we can define the
so-called correlation function which, for K(0) > 0, is

K(x)
ρ(x) =
K(0)

Blochner’s theorem states that ρ(x) is positive semidefinite, if and only if it is the
characteristic function of a variable ω,
Z
ρ(x) = eixω f (ω)dω

for a probability distribution f (ω). This means that we can prove positive semi-
definiteness of K by checking that its Fourier Transform is non-negative.
For stationary Kernels which are also isotropic,(K(x) = K(|x|)), one can show
that ρ must be representable as a Hankel transform[63].

Mercer’s Theorem

(Not really sure why this is useful!!) For x, x′ being in a bounded domain of RN ,
then any positive (semi) definite Kernel has an expansion of the form
X
K(x, x′ ) = φi (x)φi (x′ )
i

Note that this may possibly be an infinite series. Also, note that there is no
requirement that the functions be orthogonal.
Aside : it is interesting to think about the unbounded case since, for example, the
conditions of Mercer’s theorem do not apply to the squared exponential Kernel
′ 2
e−(x−x ) . From Blochner’s Theorem, or using simple Fourier Analysis, we can
indeed form an expansion of this Kernel in terms of an integral representation of
complex orthogonal functions. What is interesting about the squared exponential
case is that we can indeed also find a Mercer representation (exercise!), even though
the conditions of the theorem do not hold.
Appendix F Approximating Integrals:

The Laplace Approximation

Consider a distribution
1 −E(w)
p(w) = e
Z
In many cases (due to some form of the asymptotics), distributions will often
tend to be come rather sharply peaked. The Laplace distribution aims to make a
Gaussian approximation of p(w). The Laplace approach is a simple perturbation
expansion, which assumes that the Gaussian is sharply peaked around some value
w∗ . If we find

w∗ = arg min E(w)

w

then a natural approximation is to make a Taylor expansion up to second order:

1 T
E(w) ≈ E(w∗ ) + (w − w∗ ) ∇E|w∗ + (w − w∗ ) H (w − w∗ )
2
where H ≡ ∇∇E(w)|w∗ . At the optimum, ∇E|w∗ = 0, and an approximation of
the distribution is given by the Gaussian
1 − 1 (w−w∗ )T H(w−w∗)
p∗ (w) = e 2
Z∗
which has mean w∗ and covariance H −1 . This means that the normalisation
constant is Z ∗ = det 2πH −1 . Similarly, we can use the above expansion to estimate
the integral
Z Z
∗ 1 ∗ T ∗
e−E(w) ≈ e−E(w )− 2 (w−w ) H(w−w )
w w

Hence
Z
∗ √
e−E(w) ≈ e−E(w )
det 2πH −1
w

Note that although the Laplace approximation essentially fits a Gaussian to a

distribution, it is not necessarily the best Gaussian approximation. Other criteria,
such as a variational fit, are sometimes more powerful ways of approximating the
integral. However, arguably the major benefit of Laplace’s method is its speed and
simplicity.

307
Appendix G Inference with Gaussian Random Variables

We have two variables X and Y . Imagine the X models the position of an object
in the world (in one dimension) and Y is an observation, say in a camera, of the
position of the object in the camera. A camera calibration procedure tells us the
relationship between X and Y ; in our case we assume

Y = 2X + 8 + Ny

where Ny is some Gaussian measurement noise with zero mean and variance 1.
Thus our model for P (y|x) is
1 1
P (y|x) = √ exp{− (y − 2x − 8)2 }.
2π 2
Also we assume that x ∼ N (0, 1/α) so that
r
α αx2
P (x) = exp − .
2π 2

Given this, we want to infer the distribution of X given that Y = y. To do this

we compute the mean and covariance of (X, Y )T , and then condition on Y = y.
The mean vector is easily calculated as
   
µx 0
µ= = .
µy 8

For the covariance matrix, we have that var(X) = 1/α. For var(Y ) we find
4
var(Y ) = E[(Y − µy )2 ] = E[(2X + Ny )2 ] = + 1,
α
and for covar(XY ) we find
2
covarXY = E[(X − µx )(Y − µy )] = E[X(2X + Ny )] =
α
and thus

 
1/α 2/α
Σ= .
2/α 4/α + 1

Given a vector of random variables split into two parts X1 and X2 with
 
µ1
µ=
µ2

and
 
Σ11 Σ12
Σ=
Σ21 Σ22

308
 309

the general expression for obtaining the conditional distribution of X1 given X2 ,

is

µc1|2 = µ1 + Σ12 Σ−1

22 (x2 − µ2 ),

Σc1|2 = Σ11 − Σ12 Σ−1

22 Σ21 .

Applying this to the case above, we obtain

2 α 2
µx|y = 0 + . (y − 8) = (y − 8)
α 4+α 4+α
and
1 2 α 2 1
var(x|y) = − . . = .
α α 4+α α 4+α

The obvious estimator of X is (y − 8)/2, which is obtained from inverting the

dependence between y and x on the assumption that the noise is zero. We see
that this is obtained in the limit α → 0, which corresponds to an improper prior
on X with infinite variance. For non-zero α, the effect is to “shrink” µx|y towards
zero, which corresponds to the information in the prior on X that zero is its most
likely value. Note that if α → ∞, which corresponds to being certain at the outset
that X = 0, then this information overwhelms the information coming from the
observation, and in this limit µx|y = 0. Notice also that the posterior variance
1/(4 + α) is smaller than the prior variance 1/α.

G.1 Gaussian Conditioning

For a joint Gaussian distribution over the vectors x and y with means µx , µy and
covariance elements Σxx ,Σxy ,Σyy , the conditional p(x|y) is a Gaussian with mean
µx + Σxy Σ−1 −1
yy (y − µy ) and covariance Σxx − Σxy Σyy Σyx .

G.2 Gaussian Propagation

Let y be linearly related to xR through y = M x + η, where η ∼ N (µ, Σ), and
x ∼ N (µx , Σx ). Then p(y) = x p(y|x)p(x) is a Gaussian with mean M µx + µ and
covariance M Σx M T + Σ.
Bibliography

[1] D. L. Alspach and H. W. Sorenson. Nonlinear Bayesian Estimation Using

Gaussian Sum Approximations. IEEE Transactions on Automatic Control,
17(4):439–448, 1972.
[2] Y. Bar-Shalom and T. E. Fortmann. Tracking and Data Association. Acad-
emic Press, 1988.
[3] Y. Bar-Shalom and Xiao-Rong Li. Estimation and Tracking : Principles,
Techniques and Software. Artech House, Norwood, MA, 1998.
[4] R. E. Bellman. Dynamic Programming. Dover Publications, Incorporated,
2003.
[5] J. O. Berger. Statistical Decision Theory and Bayesian Analysis. Springer,
Second edition, 1985.
[6] J. Besag. Spatial Interactions and the Statistical Analysis of Lattice Systems.
Journal of the Royal Statistical Society, Series B, 36(2):192–236, 1974.
[7] K. Binder. Spin Glasses: Experimental Facts, Theoretical Concepts and Open
Questions . Rev. Mod. Phys, 58:901, 1986.
[8] C. M. Bishop. Neural Networks for Pattern Recognition. Oxford University
Press, 1995.
[9] G.E.P. Box and G.C. Tiao. Bayesian Inference in Statistical Analysis.
Addison–Wesley, Reading, MA, 1973.
[10] A. T. Cemgil, B. Kappen, and D. Barber. A Generative Model for Music
Transcription. IEEE Transactions on Audio, Speech and Language Processing,
14(2):679 – 694, 2006.
[11] S. Chib and M. Dueker. Non-markovian regime switching with endogenous
states and time-varying state strengths. Econometric Society 2004 North
American Summer Meetings 600, 2004.
[12] R. Dechter. Bucket Elimination: A unifying framework for probabilistic in-
ference algorithms. Uncertainty in Artificial Intelligence, 1996.
[13] B. Deylon. Remarks on Linear and Nonlinear Filtering. IEEE Transactions
on Information Theory, 41(1):317–322, 1995.
[14] A. Doucet, N. de Freitas, and N. Gordon. Sequential Monte Carlo Methods
in Practice. Springer, 2001.
[15] J. M. Gutierrez E. Castillo and A. S. Hadi. Expert Systems and Probabilistic
Network Models. Springer Verlag, 1997.
[16] Z. Ghahramani and G.E. Hinton. Variational learning for switching state-
space models. Neural Computation, 12(4):963–996, 1998.
[17] A. Gibbons. Algorithmic Graph Theory. Cambridge University Press, 1991.

310
 311

[18] S. Hawking. Does god play dice? http://www.hawking.org.uk/lectures/dice.html,

2003.
[19] T. Heskes and O. Zoeter. Expectation Propagation for approximate inference
in dynamic Bayesian networks. In A. Darwiche and N. Friedman, editors,
Uncertainty in Artificial Intelligence, pages 216–223, 2002.
[20] E.T. Jaynes. Probability Theory : The Logic of Science. Cambridge University
Press, 2003.
[21] T. Jebara and A. Pentland. On reversing Jensen’s inequality. Advances in
Neural Information Processing Systems (NIPS 13), 2000.
[22] F. V. Jensen. Bayesian Networks and Decision Graphs. Springer Verlag, 2001.
[23] F.V. Jensen and F. Jensen. Optimal Junction Trees. Uncertainty in Artificial
Intelligence, 1994.
[24] M. I. Jordan. Learning in Graphical Models. MIT Press, 1998.
[25] C-J. Kim. Dynamic linear models with Markov-switching. Journal of Econo-
metrics, 60:1–22, 1994.
[26] C-J. Kim and C. R. Nelson. State-Space models with regime switching. MIT
Press, 1999.
[27] G. Kitagawa. The Two-Filter Formula for Smoothing and an implementation
of the Gaussian-sum smoother. Annals of the Institute of Statistical Mathe-
matics, 46(4):605–623, 1994.
[28] S. L. Lauritzen. Graphical Models. Oxford University Press, 1996.
[29] S. L. Lauritzen and D. J. Spiegelhalter. Local computations with probabilities
on graphical structures and their application to expert systems. Journal of
Royal Statistical Society, Series B, 50(2):157–224, 1988.
[30] V. Lepar and P.P. Shenoy. A Comparison of Lauritzen-Spiegelhalter, Hugin,
and Shenoy-Shafer Architectures for Computing Marginals of Probability Dis-
tributions. Uncertainty in Artificial Intelligence, 1998.
[31] U. Lerner, R. Parr, D. Koller, and G. Biswas. Bayesian Fault Detection and
Diagnosis in Dynamic Systems. In Proceedings of the Seventeenth National
Conference on Artificial Intelligence (AIII-00), pages 531–537, 2000.
[32] U. N. Lerner. Hybrid Bayesian Networks for Reasoning about Complex Sys-
tems. PhD thesis, Stanford University, 2002.
[33] T. J. Loredo. From Laplace To Supernova Sn 1987A: Bayesian Inference
In Astrophysics. In P.F. Fougere, editor, Maximum Entropy and Bayesian
Methods, pages 81–142. Kluwer, 1990.
[34] D. J. C. MacKay. Bayesian interpolation. Neural Computation, 4(3):415–447,
1992.
[35] D. J. C. MacKay. Information Theory, Inference and Learning Algorithms.
Cambridge University Press, 2003.
[36] D.J.C. MacKay. Information Theory, Inference and Learning Algorithms.
Cambridge University Press, 2003.
 312

[37] G. McLachlan and T. Krishnan. The EM Algorithm and Extensions. John

Wiley and Sons, 1997.
[38] B. Mesot and D. Barber. Switching linear dynamical systems for noise robust
speech recognition. IDIAP-RR 08, 2006.
[39] T. Minka. A family of algorithms for approximate Bayesian inference. PhD
thesis, MIT Media Lab, 2001.
[40] T. Minka. A comparison of numerical optimizers for logistic regression. Tech-
nical report, Microsoft Research, 2003.
[41] J. Moussouris. Gibbs and Markov Random Systems with Constraints. Journal
of Statistical Physics, 10:11–33, 1974.
[42] R. M. Neal. Connectionist Learning of Belief Networks. Artificial Intelligence,
56:71–113, 1992.
[43] R. M. Neal and G. E. Hinton. A View of the EM Algorithm That Justifies
Incremental, Sparse, and Other Variants. In M.J. Jordan, editor, Learning in
Graphical Models, chapter 1. MIT Press, 1998.
[44] V. Pavlovic, J.M Rehg, and J. MacCormick. Learning switching linear models
of human motion. In Advances in Neural Information Processing systems
(NIPS 13), pages 981–987, 2001.
[45] J. Pearl. Probabilistic Reasoning in Intelligent Systems : Networks of Plausible
Inference. Morgan Kaufmann, 1988.
[46] J. Pearl. Causality: Models, Reasoning and Inference. Cambridge University
Press, 2000.
[47] L. R. Rabiner. A tutorial on hidden Markov models and selected applications
in speech recognition. Proc. of the IEEE, 77(2), 1989.
[48] H. E. Rauch, G. Tung, and C. T. Striebel. Maximum Likelihood estimates of
linear dynamic systems. American Institute of Aeronautics and Astronautics
Journal (AIAAJ), 3(8):1445–1450, 1965.
[49] R. Salakhutdinov, S. Roweis, and Z. Ghahramani. Optimization with em
and expectation-conjugate-gradient. Intl. Conf. on Machine Learning ICML,
2003.
[50] M. Seeger. Gaussian Processes for Machine Learning. International Journal
of Neural Systems, 14(2):69–106, 2004.
[51] G. Shafer. What is probability? http://www.glennshafer.com/assets/downloads/article4
[52] M. A. Srinvas and R. J. McEliece. The Generalised Distributive Law. IEEE
Transactions of Information Theory, 46(2):325–343, 2000.
[53] E. B. Sudderth, A. T. Ihler, and A. S. Freeman, W. T.and Willsky. Non-
parametric belief propagation. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, volume 1, pages 605–612, 2003.
[54] Y. Takane and M.A. Hunter. Constrained Principal Component Analysis : A
Comprehensive Theory. Applicable Algebra in Engineering, Communication
and Computing, 12(5):391–419, 2001.
 313

[55] R. E. Tarjan and M. Yannakakis. Simple linear-time algorithms to test

chordality of graphs, test acyclicity of hypergraphs, and selectively reduce
acyclic hypergraphs. SIAM Journal on Computing, 13(3):566–579, 1984.
[56] M. Tipping and C.M. Bishop. Mixtures of probabilistic principal component
analysers. Neural Computation, 11(2):443482, 1999.
[57] M. E. Tipping. Sparse bayesian learning and the relevance vector machine.
Journal of Machine Learning Research, 1:211:244, 2001.
[58] M. E. Tipping and C.M. Bishop. Probabilistic principal component analysis.
Journal of the Royal Statistical Society, Series B, 61(3):611–622, 1999.
[59] D. M. Titterington, A. F. M. Smith, and U. E. Makov. Statistical analysis of
finite mixture distributions. Wiley, 1985.
[60] M. Verhaegen and P. Van Dooren. Numerical Aspects of Different Kalman Fil-
ter Implementations. IEEE Transactions of Automatic Control, 31(10):907–
917, 1986.
[61] J. Whittaker. Graphical Models in Applied Multivariate Statistics. Wiley:
Chichester, 1990.
[62] W. Wiegerinck and T. Heskes. IPF for discrete chain factor graphs. Uncer-
tainty in Artificial Intelligence (UAI 2002), 2002.
[63] O. Zoeter. Monitoring non-linear and switching dynamical systems. PhD
thesis, Radboud University Nijmegen, 2005.