Introduction to Bayesian Learning

Aaron Hertzmann
University of Toronto
SIGGRAPH 2004 Tutorial

Evaluations: www.siggraph.org/courses_evaluation
CG is maturing …
… but it’s still hard to create
… it’s hard to create in real-time
Data-driven computer graphics

What if we can get models from the

real world?
Data-driven computer graphics

Three key problems:

• Capture data (from video,
cameras, mocap, archives, …)
• Build a higher-level model
• Generate new data

Ideally, it should be automatic,

flexible
Example: Motion capture

mocap.cs.cmu.edu
Example: character posing
Example: shape modeling

[Blanz and Vetter 1999]

Example: shape modeling

[Allen et al. 2003]

 Key problems
• How do you fit a model to data?
– How do you choose weights and
thresholds?
– How do you incorporate prior
knowledge?
– How do you merge multiple sources
of information?
– How do you model uncertainty?

Bayesian reasoning provides solutions

 Bayesian reasoning is …
Probability, statistics, data-fitting
 Bayesian reasoning is …
A theory of mind
 Bayesian reasoning is …
A theory of artificial intelligence

[Thrun et al.]
 Bayesian reasoning is …
A standard tool of computer vision
 and …
Applications in:
• Data mining
• Robotics
• Signal processing
• Bioinformatics
• Text analysis (inc. spam filters)
• and (increasingly) graphics!
 Outline for this course
3:45-4pm: Introduction
4pm-4:45: Fundamentals
- From axioms to probability theory
- Prediction and parameter estimation
4:45-5:15: Statistical shape models
- Gaussian models and PCA
- Applications: facial modeling, mocap
5:15-5:30: Summary and questions
More about the course
• Prerequisites
– Linear algebra, multivariate
calculus, graphics, optimization
• Unique features
– Start from first principles
– Emphasis on graphics problems
– Bayesian prediction
– Take-home “principles”
 Bayesian vs. Frequentist
• Frequentist statistics
– a.k.a. “orthodox statistics”
– Probability = frequency of
occurrences in infinite # of trials
– Arose from sciences with
populations
– p-values, t-tests, ANOVA, etc.
• Bayesian vs. frequentist debates
have been long and acrimonious
 Bayesian vs. Frequentist
“In academia, the Bayesian
revolution is on the verge of
becoming the majority viewpoint,
which would have been
unthinkable 10 years ago.”
- Bradley P. Carlin, professor of
public health, University of
Minnesota
New York Times, Jan 20, 2004
 Bayesian vs. Frequentist
If necessary, please leave these
assumptions behind (for today):

• “A probability is a frequency”
• “Probability theory only applies
to large populations”
• “Probability theory is arcane and
boring”
Fundamentals
 What is reasoning?
• How do we infer properties of the
world?
• How should computers do it?
 Aristotelian logic
• If A is true, then B is true
• A is true
• Therefore, B is true

A: My car was stolen

B: My car isn’t where I left it
 Real-world is uncertain
Problems with pure logic:
• Don’t have perfect information
• Don’t really know the model
• Model is non-deterministic

So let’s build a logic of uncertainty!

 Beliefs
Let B(A) = “belief A is true”
B(¬A) = “belief A is false”

e.g., A = “my car was stolen”

B(A) = “belief my car was stolen”
 Reasoning with beliefs
Cox Axioms [Cox 1946]
1. Ordering exists
– e.g., B(A) > B(B) > B(C)
2. Negation function exists
– B(¬A) = f(B(A))
3. Product function exists
– B(A ∧ Y) = g(B(A|Y),B(Y))
This is all we need!
The Cox Axioms uniquely define
a complete system of reasoning:
This is probability theory!
Principle #1:

“Probability theory is nothing more

than common sense reduced to
calculation.”
- Pierre-Simon Laplace, 1814
 Definitions

P(A) = “probability A is true”

= B(A) = “belief A is true”
P(A) 2 [0…1]
P(A) = 1 iff “A is true”
P(A) = 0 iff “A is false”

P(A|B) = “prob. of A if we knew B”

P(A, B) = “prob. A and B”
 Examples
A: “my car was stolen”
B: “I can’t find my car”

P(A) = .1
P(A) = .5
P(B | A) = .99
P(A | B) = .3
 Basic rules
Sum rule:

P(A) + P(¬A) = 1

Example:
A: “it will rain today”

p(A) = .9 p(¬A) = .1
 Basic rules
Sum rule:

∑i P(Ai) = 1
when exactly one of Ai must be true
 Basic rules
Product rule:

P(A,B) = P(A|B) P(B)

= P(B|A) P(A)
 Basic rules
Conditioning
Product Rule
P(A,B) = P(A|B) P(B)
P(A,B|C) = P(A|B,C) P(B|C)

Sum Rule
∑i P(Ai) = 1 ∑i P(Ai|B) = 1
 Summary
Product rule P(A,B) = P(A|B) P(B)
Sum rule ∑i P(Ai) = 1

All derivable from Cox axioms;

must obey rules of common sense
Now we can derive new rules
 Example
A = you eat a good meal tonight
B = you go to a highly-recommended
restaurant
¬B = you go to an unknown restaurant

Model: P(B) = .7, P(A|B) = .8, P(A|¬B) = .5

What is P(A)?
 Example, continued
Model: P(B) = .7, P(A|B) = .8, P(A|¬B) = .5

1 = P(B) + P(¬B) Sum rule

1 = P(B|A) + P(¬B|A) Conditioning

P(A) = P(B|A)P(A) + P(¬B|A)P(A)

= P(A,B) + P(A,¬B) Product rule

= P(A|B)P(B) + P(A|¬B)P(¬B) Product rule

= .8 .7 + .5 (1-.7) = .71
 Basic rules
Marginalizing

P(A) = ∑i P(A, Bi)

for mutually-exclusive Bi

e.g., p(A) = p(A,B) + p(A, ¬B)

Principle #2:

Given a complete model, we can

derive any other probability
 Inference
Model: P(B) = .7, P(A|B) = .8, P(A|¬B) = .5
If we know A, what is P(B|A)?
(“Inference”)

P(A,B) = P(A|B) P(B) = P(B|A) P(A)

P(A|B) P(B)
P(B|A) = = .8 .7 / .71 ˜ .79
P(A)
Bayes’ Rule
 Inference
Bayes Rule Likelihood
Prior

P(D|M) P(M)
P(M|D) =
P(D)
Posterior
Principle #3:
Describe your model of the
world, and then compute the
probabilities of the unknowns
given the observations
Principle #3a:

Use Bayes’ Rule to infer unknown

model variables from observed data
Likelihood
Prior

P(D|M) P(M)
P(M|D) =
Posterior
P(D)
 Discrete variables
Probabilities over discrete
variables

C 2 { Heads, Tails }

P(C=Heads) = .5

P(C=Heads) + P(C=Tails) = 1
 Continuous variables
Let x 2 RN
How do we describe beliefs over x?
e.g., x is a face, joint angles, …
 Continuous variables
Probability Distribution Function (PDF)
a.k.a. “marginal probability”

p(x) P(a · x · b) = sab p(x) dx

Notation: P(x) is prob

p(x) is PDF
x
 Continuous variables
Probability Distribution Function (PDF)
Let x 2 R
p(x) can be any function s.t.
s-11 p(x) dx = 1
p(x) ¸ 0

Define P(a · x · b) = s ab p(x) dx

 Uniform distribution
x » U(x0, x1)
p(x) = 1/(x0 – x1) if x0 · x · x1
=0 otherwise

p(x)

x0 x1
 Gaussian distributions
x » N(µ, σ2)
p(x|µ,σ2) = exp(-(x-µ)2/2σ2) / p 2πσ2

µ
 Why use Gaussians?
• Convenient analytic properties
• Central Limit Theorem
• Works well
• Not for everything, but a good
building block
• For more reasons, see
[Bishop 1995, Jaynes 2003]
 Rules for continuous PDFs

Same intuitions and rules apply

“Sum rule”: s-11 p(x) dx = 1

Product rule: p(x,y) = p(x|y)p(x)
Marginalizing: p(x) = s p(x,y)dy

… Bayes’ Rule, conditioning, etc.

 Multivariate distributions

Uniform: x » U(dom) Gaussian: x » N(µ, Σ)

 Inference
How do we reason about the world from
observations?

Three important sets of variables:

• observations
• unknowns
• auxiliary (“nuisance”) variables

Given the observations, what are the

probabilities of the unknowns?
 Inference
Example: coin-flipping
P(C = heads|θ) = θ
p(θ) = U(0,1)

Suppose we flip the coin 1000 times and

get 750 heads. What is θ?

Intuitive answer: 750/1000 = 75%

 What is θ?
p(θ) = Uniform(0,1)
P(Ci = h|θ) = θ, P(Ci = t|θ) = 1-θ
P(C1:N | θ) = ∏i P(Ci = h | θ)
p(θ | C1:N) = P(C1:N| θ) p(θ)
Bayes’ Rule
P(C1:N)
= ∏i P(Ci | θ) P(θ) / P(C1:N)
/ θH (1-θ)T
H = 750, T = 250
 What is θ?
p(θ | C1, … CN) / θ750 (1-θ)250

θ
“Posterior distribution:” new beliefs about θ
 Bayesian prediction
What is the probability of another
head?
P(C=h|C1:N) = s P(C=h,θ|C1:N) dθ
= s P(C=h|θ, C1:N) P(θ | C1:N) dθ
= (H+1)/(N+2)
= 751 / 1002 = 74.95 %
Note: we never computed θ
 Parameter estimation
• What if we want an estimate of θ?
• Maximum A Posteriori (MAP):
θ* = arg maxθ p(θ | C1, …, CN)
=H/N
= 750 / 1000 = 75%
 A problem
Suppose we flip the coin once
What is P(C2 = h | C1 = h)?

MAP estimate: θ* = H/N = 1

This is absurd!
Bayesian prediction:
P(C2 = h | C1 = h) = (H+1)/(N+2) = 2/3
What went wrong?

p(θ | C1:N) p(θ | C1)

 Over-fitting
• A model that fits the data well
but does not generalize
• Occurs when an estimate is
obtained from a “spread-out”
posterior
• Important to ask the right
question: estimate CN+1, not θ
Principle #4:
Parameter estimation is not
Bayesian. It leads to errors,
such as over-fitting.
Advantages of estimation
Bayesian prediction is usually
difficult and/or expensive
p(x|D) = s p(x, θ | D) dθ
 Q: When is estimation safe?
A: When the posterior is “peaked”
• The posterior “looks like” a spike
• Generally, this means a lot more data
than parameters
• But this is not a guarantee (e.g., fit a
line to 100 identical data points)
• Practical answer: use error bars
(posterior variance)
Principle #4a:
Parameter estimation is easier
than prediction. It works well
when the posterior is “peaked.”
Learning a Gaussian

{x2} µ, σ2
 Learning a Gaussian
p(x|µ,σ2) = exp(-(x-µ)2/2σ2) / p 2πσ2
p(x1:K|µ, σ2) = ∏ p(xi | µ, σ2)
Want: max p(x1:K|µ, σ2)
= min –ln p(x1:K|µ, σ2)
= ∑i (x-µ)2/2σ2 + K/2 ln 2π σ2
Closed-form solution:
µ = ∑i xi / N
σ2 = ∑i (x - µ)2/N
 Stereology
[Jagnow et al. 2004 (this morning)]
Model:

PDF over solids

p(θ) p(S | θ) p(I|S)

Problem: What is the PDF over solids?

Can’t estimate individual solid shapes:
arg max p(θ, S | I) is underconstrained)
 Stereology

PDF over solids

p(θ) p(S | θ) p(I|S)

Marginalize out S:
p(θ | I) = s p(θ, S | I) dS
can be maximized
Principle #4b:
When estimating variables,
marginalize out as many
unknowns as possible.

Algorithms for this:

•Expectation-Maximization (EM)
•Variational learning
Regression
 Regression
Curve fitting

?
 Linear regression
Model:
ε » N(0, σ2I)
y y=ax+b+ε

Or:
p(y|x,a,b,σ2) =
x x N(ax + b, σ2I)
 Linear regression
p(y|x, a, b,σ2) = N(ax + b, σ2I)
2
p(y1:K | x1:K, a, b, σ ) = ∏i p(yi | xi, a, b, σ2)
Maximum likelihood:
a*,b*,σ2* = arg max ∏i p(yi|xi,a,b,σ2)
= arg min –ln ∏i p(yi|x, a, b, σ2)
Minimize:
∑i (yi-(axi+b))2/(2σ2) + K/2 ln 2 π σ2

Sum-of-squared differences: “Least-squares”

 Linear regression
Same idea in higher dimensions
y = Ax + µ + ε

y
x y

x
 Nonlinear regression
Model:
ε » N(0, σ2I)
y y = f(x;w) + ε
Curve parameters
Or:
p(y|x,w,σ2) =
x N(f(x;w), σ2I)
 Typical curve models
Line
f(x;w) = w0 x + w1
B-spline, Radial Basis Functions
f(x;w) = ∑i wi Bi(x)
Artificial neural network
f(x;w) = ∑i wi tanh(∑j wj x + w0)+w1
 Nonlinear regression
p(y|x, w, σ2) = N(f(x;w), σ2I)
2
p(y1:K | x1:K, w, σ ) = ∏i p(yi | xi, a, b, σ2)
Maximum likelihood:
w*,σ2* = arg max ∏i p(yi|xi,a,b,σ2)
= arg min –ln ∏i p(yi|x, a, b, σ2)
Minimize:
∑i (yi-f(xi;w))2/(2σ2) + K/2 ln 2 π σ2

Sum-of-squared differences: “Least-squares”

Principle #5:
Least-squares estimation is a
special case of maximum
likelihood.
Principle #5a:
Because it is maximum
likelihood, least-squares suffers
from overfitting.
Overfitting
 Smoothness priors
Assumption: true curve is smooth

Bending energy:
p(w|λ) ~ exp( -s kr fk2 / 2 λ2)
Weight decay:
p(w|λ) ~ exp( -kwk2 / 2 λ2)
 Smoothness priors
MAP estimation:
arg max p(w|y) = p(y | w) p(w)/p(y)=
arg min –ln p(y|w) p(w) =
∑i (yi – f(xi; w))2/(2σ2) + kwk 2/2λ2 + K ln σ

Sum-of-squares differences Smoothness

Underfitting
Underfitting
Principle #5b:
MAP estimation with smoothness
priors leads to under-fitting.
Applications in graphics
Two examples:
Shape interpolation Approximate physics

[Grzeszczuk et al. 1998]

[Rose III et al. 2001]

 Choices in fitting
• Smoothness, noise parameters
• Choice of basis functions
• Number of basis functions

Bayesian methods can make these

choices automatically and
effectively
 Learning smoothness
Given “good” data, solve
λ* , σ2* = arg max p(λ, σ2 | w, x1:K, y1:K)
Closed-form solution
Shape reconstruction
in vision [Szeliski 1989]
 Learning without shape
Q: Can we learn smoothness/noise
without knowing the curve?
A: Yes.
 Learning without shape
λ* , σ2* = arg max p(λ, σ2 | x1:K, y1:K)
(2 unknowns, K measurements)

p(λ, σ2 | x1:K, y1:K) = s p(λ, σ2, w | x1:K,y1:K) dw

/ s p(x1:K,y1:K|w,σ2,λ)p(w|λ,σ2)dw
 Bayesian regression
don’t fit a single curve, but keep
the uncertainty in the curve:
p(x | x1:N, y1:N)

x
 Bayesian regression

MAP/Least-squares Gaussian Process regression

(hand-tuned λ, σ2, (learned parameters λ, σ 2)
basis functions)
Bayesian regression
Principle #6:
Bayes’ rule provide principle for
learning (or marginalizing out)
all parameters.
 Prediction variances

More info: D. MacKay’s Introduction to Gaussian Processes

NIPS 2003 Feature Selection
Challenge
• Competition between classification
algorithm, including SVMs, nearest
neighbors, GPs, etc.
• Winners: R. Neal and J. Zhang
• Most powerful model they could
compute with (1000’s of parameters)
and Bayesian prediction
• Very expensive computations
 Summary of “Principles”
1. Probability theory is common sense
reduced to calculation.

2. Given a model, we can derive any

probability

3. Describe a model of the world, and

then compute the probabilities of the
unknowns with Bayes’ Rule
 Summary of “Principles”
4. Parameter estimation leads to over-fitting
when the posterior isn’t “peaked.” However,
it is easier than Bayesian prediction.

5. Least-squares estimation is a special case of

MAP, and can suffer from over- and under-
fitting

6. You can learn (or marginalize out) all

parameters.
 Statistical shape and
appearance models with PCA
 Key vision problems
• Is there a face
in this image?
• Who is it?
• What is the 3D
shape and
texture?

Turk and Pentland 1991

 Key vision problems
• Is there a person in this picture?
• Who?
• What is their 3D pose?
 Key graphics problems
• How can we easily create new
bodies, shapes, and appearances?
• How can we edit images and
videos?
 The difficulty
• Ill-posed problems
– Need prior assumptions
– Lots of work for an artist
 Outline
• Face modeling problem
– Linear shape spaces
– PCA
– Probabilistic PCA
• Applications
– face and body modeling
 Background: 2D models
• Eigenfaces
– Sirovich and Kirby 1987, Turk and
Pentland 1991
• Active Appearance
Models/Morphable models
– Beier and Neely 1990
– Cootes and Taylor 1998
 Face representation
• 70,000 vertices with (x, y, z, r, g, b)
• Correspondence precomputed

[Blanz and Vetter 1999]

 Data representation
yi = [x1, y1, z1, …, x70,000, y70,000, z70,000]T
Linear blends:
y1 y2 y3
.5 . + .5. =

ynew = (y1 + y2) / 2

a.k.a. blendshapes, morphing

 Linear subspace model

y = ∑i wi yi (s.t., ∑i wi = 1)
= ∑i xi ai + µ
=Ax+µ

x y

Problem: can we learn this linear space?

 Principal Components Analysis
(PCA)

Same model as linear regression

Unknown x

y
x y

x
 Conventional PCA
(Bayesian formulation)
x, A, µ » Uniform, AT A = I
ε » N(0, σ2 I)
y=Ax+µ+ε
Given training y1:K, what are A, x, µ, σ2?

Maximum likelihood reduces to:

∑i k yi – (A xi + µ) k 2 / 2σ2 + K/2 ln 2 π σ2

Closed-form solution exists

 PCA with missing data

ML point

x y

Linear constraint

Problems:
•Estimated point far from data if data is noisy
•High-dimensional y is a uniform distribution
•Low-dimensional x is overconstrained
Why? Because x » U
 Probabilistic PCA
x » N(0,I)
y = Ax + b + ε

x y

[Roweis 1998, Tipping and Bishop 1998]

 Fitting a Gaussian
y » N(µ, Σ)
easy to learn, and nice properties
… but Σ is a 70,0002 matrix

y
 PPCA vs. Gaussians
However…
PPCA: p(y) = s p(x,y) dx
= N(b, A AT + σ2 I)
This is a special case of a Gaussian!
PCA is a degenerate case (σ2=0)
 Face estimation in an image

p(y) = N(µ, Σ)
y
p(Image | y) = N(Is(y), σ2 I)

Image
[Blanz and Vetter 1999]

-ln p(S,T | Image) = kImage – Is(y)k 2 /2σ 2 + (y-µ)TΣ-1(y-µ)/2

Image fitting term Face likelihood

Use PCA coordinates for efficiency

Efficient editing in PCA space
 Comparison
PCA: unconstrained latent space –
not good for missing data
Gaussians: general model, but
impractical for large data
PPCA: constrained Gaussian – best
of both worlds
Estimating a face from video

[Blanz et al. 2003]

The space of all body shapes

[Allen et al. 2003]

The space of all body shapes

[Allen et al. 2004]

 Non-rigid 3D modeling from video

What if we don’t have training data?

[Torresani and Hertzmann 2004]

Non-rigid 3D modeling from
video
• Approach: learn all parameters
– shape and motion
– shape PDF
– noise and outliers
• Lots of missing data (depths)
– PPCA is essential
• Same basic framework, more
unknowns
 Results

Lucas-Kanade tracking

Tracking result

3D reconstruction

Reference frame
Results

Robust algorithm

3D reconstruction

[Almodovar 2002]
 Inverse kinematics

DOFs (y)

Constraints

[Grochow et al. 2004 (tomorrow)]

Problems with Gaussians/PCA

Space of poses may is nonlinear,

non-Gaussian

x y
 Non-linear dimension reduction

y = f(x;w) + ε
Like non-linear regression w/o x

f(x;w)

x y

NLDR for BRDFs: [Matusik et al. 2003]

Problem with Gaussians/PPCA
 Style-based IK

Walk cycle:

f(x;w)

Details: [Grochow 2004 (tomorrow)]

Discussion and frontiers
Designing learning algorithms
for graphics
Write a generative model
p(data | model)
Use Bayes’ rule to learn the model
from data
Generate new data from the model
and constraints
(numerical methods may be
required)
 What model do we use?
• Intuition, experience,
experimentation, rules-of-thumb
• Put as much domain knowledge in as
possible
– model 3D shapes rather than pixels
– joint angles instead of 3D positions
• Gaussians for simple cases; nonlinear
models for complex cases (active
research area)
Q: Are there any limits to
the power of Bayes' Rule?
http://yudkowsky.net/bayes/bayes.html:

A: According to legend, one who

fully grasped Bayes' Rule would
gain the ability to create and
physically enter an alternate
universe using only off-the-shelf
equipment. One who fully grasps
Bayes' Rule, yet remains in our
universe to aid others, is known
as a Bayesattva.
 Problems with Bayesian methods

1. The best solution is usually

intractable
• often requires expensive numerical
computation
• it’s still better to understand the real
problem, and the approximations
• need to choose approximations
carefully
 Problems with Bayesian methods

2. Some complicated math to do

• Models are simple, algorithms
complicated
• May still be worth it
• Bayesian toolboxes on the way
(e.g., VIBES, Intel OpenPNL)
 Problems with Bayesian methods

3. Complex models sometimes

impede creativity
• Sometimes it’s easier to tune
• Hack first, be principled later
• Probabilistic models give insight
that helps with hacking
 Benefits of the Bayesian
approach
1. Principled modeling of noise
and uncertainty
2. Unified model for learning and
synthesis
3. Learn all parameters
4. Good results from simple models
5. Lots of good research and
algorithms
Course notes, slides, links:
http://www.dgp.toronto.edu/~hertzman/ibl2004

Course evaluation
http://www.siggraph.org/courses_evaluation

Thank you!