L E C T U R E 07 Eigenfaces and

Fisherfaces
Face Recognition

• Face recognition algorithms

• Comparison
• Short summary

2
Face Recognition Algorithms

•We will introduce

• Eigenfaces
• Fisherfaces

3
Eigenfaces

4
Eigenfaces

• Developed in 1991 by M.Turk

• Based on Principal Component Analysis (PCA)
• Relatively simple
• Fast
• Robust

5
 Eigenfaces
• PCA seeks directions that are efficient for representing the data

not efficient efficient

Class A Class A

Class B Class B

6
 Eigenfaces
• PCA maximizes the total scatter

scatter

Class A

Class B

7
Eigenfaces

•PCA reduces the dimension of the data

•Speeds up the computational time

8
9
Eigenfaces, the algorithm

• Assumptions
• Square images with Width = Height = N
• M is the number of images in the database
• P is the number of persons in the database

10
Eigenfaces, the algorithm

• The database

a1 b1 c1 d1
a2 b2 c2 d2
= = = =

a b c d
N 2 N 2
N2 N2

e1 f1 g1 h1
e2 f2 g2 h2
= = = =

e f gN h
2 N2 2
N2
N

11
Eigenfaces, the algorithm

• We compute the average face

a1 + b1 + + h1
1 a2 + b2 + + h2
m= , where M = 8
M
a N 2 + bN 2 + + h N 2

12
 Eigenfaces, the algorithm

• Then subtract it from the training faces

a1 m1 b1 m1 c1 m1 d1 m1
a 2 – m2 b2 m2 c 2 – m2 d 2 – m2
am = , bm = , cm = , dm = ,

a N 2 – mN 2 b N 2 – mN 2 c N 2 – mN 2 d N 2 – mN 2

e1 m1 f1 m1 g1 m1 h1 m1
e 2 – m2 f 2 – m2 g 2 – m2 h 2 – m2
em = , fm = , gm = , hm =

e N 2 – mN 2 f N 2 – mN 2 g N 2 – mN 2 h N 2 – mN 2

13
Eigenfaces, the algorithm

• Now we build the matrix which is N2 by M

A = a m bm cm d m em f m g m hm

• The covariance matrix which is N2 by N2

T
Cov = AA

14
 Eigenfaces, the algorithm
• Find eigenvalues of the covariance matrix
• The matrix is very large
• The computational effort is very big

• We are interested in at most M eigenvalues

• We can reduce the dimension of the matrix

15
Eigenfaces, the algorithm
• Compute another matrix which is M by M

L=A A T

• Find the M eigenvalues and eigenvectors

• Eigenvectors of Cov and L are equivalent

• Build matrix V from the eigenvectors of L

16
Eigenfaces, the algorithm
• Eigenvectors of Cov are linear combination of image space with the
eigenvectors of L

U = AV V is Matrix of
eigenvectors

A = a m bm cm d m em f m g m hm

• Eigenvectors represent the variation in the faces

17
Eigenfaces, the algorithm

A: collection of the U: Face Space /

training faces Eigen Space

18
 Eigenfaces
• Eigenface of original faces

19
 Eigenfaces, the algorithm
l Compute for each face its projection onto
the face space
1 = U T (am ) , 2 ( )
= U T bm , 3 = U T (cm ) , 4 ( )
= U T dm ,

5 = U T (em ) , 6 =U T (f ),
m 7 = U T (g m ) , 8 = U (h )
T
m

l Compute the threshold

1
= max
2
{ i j }for i, j = 1.. M
20
 Eigenfaces: Recognition Procedure

l To recognize a face
r1
r2
=

rN 2

l Subtract the average face from it

r1 m1
r 2 – m2
rm =

r N – mN
2 2
21
 Eigenfaces, the algorithm
l Compute its projection onto the face space U

=U T
(rm )
lCompute the distance in the face space
between the face and all known faces

2 2
i = i for i = 1.. M
 Eigenfaces, the algorithm
l Reconstruct the face from eigenfaces

s =U
l Compute the distance between the face
and its reconstruction

2 2
= rm – s
 Eigenfaces, the algorithm
l Distinguish between
lIf then it’s not a face; the distance
between the face and its reconstruction is
larger than threshold
lIf < and min {i }< then it’s a new face
lIf < and i , (i = 1.. M ) then it’s a known
face because the distance in the face space
between the face and all known faces is larger
than threshold
Eigenfaces, the algorithm

• Problems with eigenfaces

• Different illumination

25
Eigenfaces, the algorithm
•Problems with eigenfaces
• Different head pose
• Different alignment
• Different facial expression

26
Fisherfaces

27
Fisherfaces
• Developed in 1997 by P.Belhumeur et al.
• Based on Fisher’s Linear Discriminant Analysis (LDA)
• Faster than eigenfaces, in some cases
• Has lower error rates
• Works well even if different illumination
• Works well even if different facial express.

28
Fisherfaces
• LDA seeks directions that are efficient for
discrimination between the data

Class A

Class B

29
Fisherfaces

• LDA maximizes the between-class scatter

• LDA minimizes the within-class scatter

Class A
Class B

30
Fisherfaces, the algorithm

•Assumptions
• Square images with Width=Height=N
• M is the number of images in the database
• P is the number of persons in the database

31
Fisherfaces, the algorithm

• The database

a1 b1 c1 d1
a2 b2 c2 d2
= = = =

a b c d
N 2 N2 N2 N2

e1 f1 g1 h1
e2 f2 g2 h2
= = = =

e f gN h
2 N2 2
N2
N

32
Fisherfaces, the algorithm

• We compute the average of all faces

a1 + b1 + + h1
1 a2 + b2 + + h2
m= , where M = 8
M
a N 2 + bN 2 + + h N 2

33
 Fisherfaces, the algorithm
• Compute the average face of each person

a1 + b1 c1 + d1
1 a2 + b2 1 c2 + d 2
x= , y= ,
2 2
a N 2 + bN 2 cN 2 + d N 2
e1 + f1 g1 + h1
1 e2 + f 2 1 g 2 + h2
z= , w=
2 2
e N2 + fN2 g N 2 + hN 2

34
Fisherfaces, the algorithm

• And subtract them from the training faces

a1 x1 b1 x1 c1 y1 d1 y1
a2 – x2 b2 – x2 c2 – y2 d2 y2
am = , bm = , cm = , dm = ,

a N2 – xN 2 b N2 – xN 2 c N2 – yN 2 d N2
– yN 2

e1 z1 f1 z1 g1 w1 h1 w1
e2 z2 f2 z2 g 2 – w2 h2 – w2
em = , fm = , gm = , hm =

e N2 – zN 2 f N2
– zN 2 g N2 – wN 2 h N2 – wN 2

35
Fisherfaces, the algorithm

• We build scatter matrices S1, S2, S3, S4

( ) (
S1 = am amT + bm bmT , S 2 = cm cmT + d m d mT , )
S3 = (e e
T
m m + f m f mT ), S = (g
4 g
m m
T
+h h )
T
m m

• And the within-class scatter matrix SW

SW = S1 + S 2 + S3 + S 4

36
 Fisherfaces, the algorithm

• The between-class scatter matrix

T T T T
SB = 2 (x m)( x m) + 2 ( y m )( y m )+ 2 (z m )( z m ) + 2 (w m )(w
m)

• We are seeking the matrix W maximizing

W T S BW
J (W ) =
W T SW W

37
 Fisherfaces, the algorithm
If SW is nonsingular ( M N 2 ):
l Columns of W are eigenvectors of SW 1SB
l We have to compute the inverse of SW

l We have to multiply the matrices

l We have to compute the eigenvectors
 Fisherfaces, the algorithm
If SW is nonsingular ( M N 2 ):
l Simpler:

l Columns of W are eigenvectors satisfying

S B wi = i SW wi
l The eigenvalues are roots of
SB i SW = 0
l Get eigenvectors by solving
(SB i SW ) wi = 0
39
 Fisherfaces, the algorithm
2
If SW is singular ( M < N ):
l Apply PCA first
l Will reduce the dimension of faces from N2 to M
l There are M M-dimensional vectors

l Apply LDA as described

 Fisherfaces, the algorithm
l Project faces onto the LDA-space
xLDA = W T x, y = W T y,
LDA

z LDA = W T z , w = W Tw
LDA

l To classify the face

l Project it onto the LDA-space
l Run a nearest-neighbor classifier
Fisherfaces, the algorithm
•Problems
•Small databases
•The face to classify must be in the DB

42
PCA & Fisher’s Linear Discriminant

43
PCA & Fisher’s Linear Discriminant

44
Comparison
• FERET database

best ID rate: eigenfaces 80.0%, fisherfaces 93.2%

45
Comparison
• Eigenfaces
• project faces onto a lower dimensional sub-space
• no distinction between inter- and intra-class variabilities
Ø optimal for representation but not for discrimination

46
Comparison

•Fisherfaces
• find a sub-space which maximizes the ratio of
inter-class and intra-class variability
• same intra-class variability for all classes

47
Local Feature Analysis
-- Elastic Bunch-Graph Matching

For Further Study

48
 Face Features

• Facial recognition utilizes distinctive features of the face

– including: distinct micro elements like:
• Mouth, Nose, Eye, Cheekbones, Chin, Lips, Forehead, Ears
• Upper outlines of the eye sockets, the areas surrounding
the cheekbones, the sides of the mouth, and the
location of the nose and eyes.
• The distance between the eyes, the length of the nose,
and the angle of the jaw.

49
Face Features

• Some technologies do not utilize areas of the face located

near the hairline, so they are somewhat resistant to
moderate changes in hairstyle.
• When used in identification mode, facial recognition
technology generally returns candidate lists of close matches
as opposed to returning a single definitive match as does
fingerprint and iris-scan.
• The file containing facial micro features is called a
"template."
• Using templates, the software then compares that image
with another image and produces a score that measures
how similar the images are to each other.
50
 Face Features
• Typical sources of images for use in facial recognition include video
camera signals and pre-existing photos such as those in driver's license
databases.
including:
• Distance between the micro elements
• A reference feature
• Size of the micro element
• Amount of head radiated from the face (unseen by human eye). Heat can be
measured using an infrared camera.

51
A face recognition based on local feature analysis
• A face is represented as a graph, whose nodes,
positioned in correspondence to the facial fiducial
points.
• A fiducial point is a point or line on a scale used for
reference or comparison purposes.
• A face recognition system uses an automatic approach
to localize the facial fiducial points.
• It then determines the head pose and compares the
face with the gallery images.
• This approach is invariant to rotation, light and scale.

52
A template for the 34 fiducial points on a face image: 53
 EBGM

• Elastic Bunch-Graph Matching (EBGM) algorithm locates

landmarks on an image, such as the eyes, nose, and mouth.
• Gabor jets are extracted from each landmark and are used to
form a face graph for each image. A face graph serves the same
function as the projected vectors in the PCA or LDA algorithm;
they represent the image in a low dimensional space.
• After a face graph has been created for each test image, the
algorithm measures the similarity of the face graphs.

54
Summary

•Three algorithms have been introduced

•Eigenfaces
• Reduce the dimension of the data from N2 to M
• Verify if the image is a face at all
• Allow online training
• Fast recognition of faces
• Problems with illumination, head pose etc

55
Summary
•Fisherfaces
• Reduce dimension of the data from N2 to P-1
• Can outperform eigenfaces on a representative DB
• Works also with various illuminations
• Can only classify a face which is “known” to DB
•Elastic Bunch-Graph Matching
• Reduce the dimension of the data from N2 to M
• Recognize face with different poses
• Recognize face with different expressions

56