Black or White Video

• Face Detection
• Face Localization
Lecture 3: Face Detection • Segmentation
• Face Tracking
• Facial features localization
Reading: Eigenfaces – online paper • Facial features tracking
FP pgs. 505-512 • Morphing

Handouts: Course Description

PS1 Assigned
www.youtube.com/watch?v=ZI9OYMRwN1Q

Why is Face Detection

• “…If I look at your face I immediately recognize that I have
seen it before. …Yet there is no machine which, with that Difficult?
speed, can take a picture of a face and say even that it is a
man; and much less that it is the same man that you • Severe illumination change
showed it before—unless it is exactly the same picture. If
the face is changed; if I am closer to the face; if I am
further from the face; if the light changes—I recognize it
anyway. Now, this little computer I carry in my head is
easily able to do that. The computers that we build are not
able to do that. …”

Richard P. Feynman, Dec. 29, 1959

There's Plenty of Room at the Bottom
An Invitation to Enter a New Field of Physics

Automated Face Detection Face Detection

Why is it Difficult?
• Varying viewpoint, illumination, etc.

1
Coincidental appearance of faces Coincidental appearance of face
in rock? profile in rock?

http://bensguide.gpo.gov/3-5/symbols/print/mountrushmore.html http://www.cs.dartmouth.edu/whites/old_man.html

Face Detection

Nearest Neighbor Clasiffier

• Euclidean distance:

∑ ( y1c − y2 c )
2 N
= y1 − y 2
2
d =
c =1 Linear Models
• Given an input image y (also called a probe), the NN
classifier will assign to y the label associated with the
closest image in the training set. So if, it happens to be
closest to another face it will be assigned L=1 (face),
otherwise it will be assigned L=0 (nonface)

2
 Images Image Representation
..
............... .
i ∈ ℜ kl ×1

pixel kl
pixel value axis representing pixel 1

......

255
l2
pi
xe  i1  1  0  0 
25
5
 i1 i2 ... il         
  i2  0  1   
 il+1 .         
I ∈ ℜ k ×l pixel 1 I = . .  i = M

 = i1  M  + i 2  0  L + i kl  M 

 . .       
 .     M 0 
0
kl×1
255 
• An image is a point in ℜ dimensional space  il(k−1)+1
 ikl  
i


     
 kl   0   0  1 
     

© 2002 by M. Alex O. Vasilescu

Representation
Image Representation

 i1 
 i1  1 0 L 0 
 i1 i2 ... il    • Find a new basis matrix that results in a
i2 
   i2 
  0 1   compact representation
 il+1 .   
=   M 
I = . .  i = M
  M O  
 . . 
 .      
 il(k−1)+1
 i 
kl 
   0 1   
i  i 
 kl   kl 
Basis Matrix, B

i = Bc vector of coefficients, c

© 2002 by M. Alex O. Vasilescu

Toy Example - Representation Toy Example - Representation

Heuristic Heuristic
• Consider a set of images of N people under the same viewpoint and lighting • Consider a set of images of N people under the same viewpoint and lighting
• Each image is made up of 3 pixels and pixel 1 has the same value as pixel 3 • Each image is made up of 3 pixels and pixel 1 has the same value as pixel 3
for all images for all images
i1 n  i1 n 
   
i n = i2 n  s.t . i1 n = i3 n and 1 ≤ n ≤ N i n = i2 n  s.t . i1 n = i3 n and 1 ≤ n ≤ N
   
pixel 2

pixel 2

i3 n  i3 n 
..
................
l3 l3 1  0  0 
  1 0 0  i1n 
. pi
xe
pi
xe    
i n = i1n 0  + i2 n 1  + i3 n 0  = 0 1 0 i 

......          2n 
0  0  1  0 0 1 i3n 
i2 i3 i2 i3
Basis Matrix, B
i1 pixel 1 i1 pixel 1

© 2002 by M. Alex O. Vasilescu © 2002 by M. Alex O. Vasilescu

3
 Toy Example - Representation
Toy Example-Recognition
Heuristic
• Consider a set of images of N people under the same viewpoint and lighting Solve for and store the coefficient matrix C: ↑ ↑ ↑  1 0 ↑ ↑ ↑ 
• Each image is made up of 3 pixels and pixel 1 has the same value as pixel 3     
for all images i1 i 2 L i N  = 0 1 c1 c 2 L c N 
    

pixel 2
i1 n  ↓ ↓ ↓  1 0 ↓ ↓ ↓ 
 
i n = i2 n  s.t . i1 n = i3 n and 1 ≤ n ≤ N
..
................
D, data matrix C, coefficient matrix
l3
  xe
pixel 2

pi
−1
i3 n 
C=B D
.. .....
................ l3 1  0  0 
pi
xe       1 0 0  i1n  Given a new image, inew :
i n = i1n 0  + i2 n 1  + i3 n 0  = 0 1 0 i 

.....          2n  .5 0 .5  i1new 

0  0  1  0 0 1 i3n 
c new = B −1i new =   i2 new 
i2
pixel 1
 0 1 0  i3new 
1  0  1 0
     i  • Next, compare c new a reduced dimensionality
1n representation of inew against all coefficient
= i1n 0  + i2 n 1  = 0 1   = Bcn
pixel 1       i2n  vectors c n 1 ≤ n ≤ N
new basis 1  0  1 0
© 2002 by M. Alex O. Vasilescu New Basis Matrix, B •One possible classifier: nearest-neighbor classifier

Principal Component Analysis: Variables

Eigenfaces
• Response Variables – are directly measurable,
they measure the outcome of a study.
– Pixels are response variables that are directly
• Employs second order statistics to compute measurable from an image.
in a principled way a new basis matrix
• Explanatory Variables, Factors – explain or
cause changes in the response variable.
– Pixel values change with scene geometry, illumination
location, camera location which are known as the
explanatory variables

Response vs. Explanatory Variables The Principle Behind

Principal Component Analysis1
• Pixels (response variables, directly measurable from data)
change with changes in view and illumination, the
explanatory variables (not directly measurable but of actual
interest). • Also called: - Hotteling Transform2 or the
- Karhunen-Loeve Method 3.

• Find an orthogonal coordinate system such that

data is approximated best and the correlation
between different axis is minimized.

1 I.T.Jolliffe; Principle Component Analysis; 1986

2 R.C.Gonzalas, P.A.Wintz; Digital Image Processing; 1987
3 K.Karhunen; Uber Lineare Methoden in der Wahrscheinlichkeits Rechnug; 1946
M.M.Loeve; Probability Theory; 1955

4
 PCA: Theory PCA-Dimensionality Reduction
x2

e1
• Consider a set of images, & each image is made up of 3 pixels and pixel 1 has the same value
xy2 as pixel 3 for all images
i n = [i1n i2 n i3n ]
T
s.t. i1n = i3 n and 1 ≤ n ≤ N

e2
x
x
x • PCA chooses axis in the direction of highest variability of the data, maximum scatter
x x
x x
x
x
x x
PCA x
x x
| | |  | | | 

pixel 2
x
x
1st axis
   
i1 i 2 L i N  = B c1 c 2 L c N 
x1 yx11
..
.................
xe
l3    
| | |  | | | 
pi

...... 2
• Define a new origin as the mean of the data set
data matrix, D
• Find the direction of maximum variance in the samples (e1) and align it with nd axis • Each image i n is now represented by a vector of
the first axis (y1),
coefficients c n in a reduced dimensionality space.
• Continue this process with orthogonal directions of decreasing variance,
aligning each with the next axis pixel 1 D = USV T (svd of D) set B = U
• Thus, we have a rotation which minimizes the covariance. • B minimize the following function

E = BT ST B such that BT B = Identity

PCA: Some Properties of the

The Covariance Matrix
Covariance/Scatter Matrix
• Define the covariance (scatter) matrix of the input samples:
• The matrix ST is symmetric
N
ST = ∑ (in − µ)(in − µ)T • The diagonal contains the variance of each parameter
n =1
(where µ is the sample mean) (i.e. element ST,ii is the variance in the i’th direction).

ST = (D − M)(D − M)T M = [µ L µ]
• Each element ST,ij is the co-variance between the two
where directions i and j, represents the level of correlation
(i.e. a value of zero indicates that the two dimensions are
← i1 − µ → uncorrelated).
 ↑ ↑ ↑  
  ← i 2 − µ →
S T = i1 − µ i2 − µ L iN − µ
 M 
 ↓ ↓ ↓   
 ← i − µ →
 N 

PCA: Goal Revisited

SVD of a Matrix
• Look for: - B
Scatter of matrix: ST = (D − M)(D − M)T • Such that:

– [c … c ] = BT [i … i ]
1 N 1 N

– correlation is mininmized Cov(C) is diagonal

(D − M) = UΣV T
by svd of (D - M) set B = U
Note that Cov(C) can be expressed via Cov(D) and B :

(D − M)(D − M) = UΣ U (svd of ST )T 2 T
set B = U
CCT = BT (D − M)(D − M) T B
= BT ST B

5
 Selecting the Optimal B Data Reduction: Theory

How do we find such B ?

• Each eigenvalue represents the the total variance in
its dimension.
( D − µ )( D − µ) T b i = λi b i
• Throwing away the least significant eigenvectors in
S T B = ΛB
Bopt means throwing away the least significant
Bopt contains the eigenvectors of the covariance of D
variance information

Bopt = [b1|…|bd]

PCA for Recognition Data and Eigenfaces

i n = [i1n i2 n i3n ] • Data is composed of 28 faces photographed under same
T
• Consider the set of images s.t. i1n = i3n and 1 ≤ n ≤ N
lighting and viewing conditions
• PCA chooses axis in the direction of highest variability of the data

• Given a new image, , compute the vector of coefficients

i
with the new basis, B new c new associated

c new = BT i new B −1 = BT
pixel 2

1st axis • Next, compare c new a reduced dimensionality

.... ..
representation of inew against all coefficient • Each image below is a column vector in the basis matrix B
l3

................
xe
pi vectors c n 1 ≤ n ≤ N

. .. 2 nd axis
•One possible classifier: nearest-neighbor
classifier

pixel 1 © 2002 by M. Alex O. Vasilescu © 2002 by M. Alex O. Vasilescu

Eigenimages Linear Representation:

• Principal components (eigenvectors) of image ensemble

.. .......... ..
pixel kl

.
pixel kl

+ c3 + c9
.
+ c28
.....
c1
255

......
c3 c2
255

2
x el c1
pi e l2
5 pix
25 5
25

pixel 1
pixel 1
0 255
0 255
di = Uc i
Running Sum: 1 term 3 terms 9 terms 28 terms

• Eigenvectors are typically computed using the Singular

Value Decomposition (SVD) algorithm

6
 The Covariance Matrix PIE Database (Weizmann)
• Define the covariance (scatter) matrix of the input samples:
N
ST = ∑ (in − µ)(in − µ)T
(where µ is the sample mean)

n =1

+ +

EigenImages-Basis Vectors PCA Classifier

• Distance to Face Subspace:

2
d f (y ) = y − U f U f y
T

• Each image bellow is a column vector in the basis matrix B

• PCA encodes encodes the variability across
• Likelihood ratio (LR) test to classify a probe y as face or
images without distinguishing between variability in people, nonface. Intuitively, we expect dn (y) > df (y) to suggest that y
viewpoints and illumination
is a face.
• The LR for PCA is defined as:
L =1
d n (y ) >
∆d = η
d f (y ) <
L =0
© 2002 by M. Alex O. Vasilescu