Pattern Recognition Letters 25 (2004) 429–436

www.elsevier.com/locate/patrec

An improved face recognition technique based

on modular PCA approach
Rajkiran Gottumukkal, Vijayan K. Asari *

VLSI Systems Laboratory, Department of Electrical and Computer Engineering, Old Dominion University,
231 Kaufman Hall, Norfolk, VA 23529-0246, USA
Received 13 September 2002; received in revised form 3 November 2003

Abstract

A face recognition algorithm based on modular PCA approach is presented in this paper. The proposed algorithm
when compared with conventional PCA algorithm has an improved recognition rate for face images with large vari-
ations in lighting direction and facial expression. In the proposed technique, the face images are divided into smaller
sub-images and the PCA approach is applied to each of these sub-images. Since some of the local facial features of an
individual do not vary even when the pose, lighting direction and facial expression vary, we expect the proposed method
to be able to cope with these variations. The accuracy of the conventional PCA method and modular PCA method are
evaluated under the conditions of varying expression, illumination and pose using standard face databases.

2003 Elsevier B.V. All rights reserved.

Keywords: PCA; Face recognition; Modular PCA; Pose invariance; Illumination invariance

1. Introduction is as independent as possible of these image vari-

ations.
Face recognition is a difficult problem because Automatic recognition of faces is considered
of the generally similar shape of faces combined as one of the fundamental problems in computer
with the numerous variations between images of vision and pattern analysis, and many scientists
the same face. The image of a face changes with from different areas have addressed it. Chellappa
facial expression, age, viewpoint, illumination et al. (1995) presented a survey on several statis-
conditions, noise etc. The task of a face recogni- tical-based, neural network-based and feature-
tion system is to recognize a face in a manner that based methods for face recognition. Currently, one
of the methods that yields promising results on
frontal face recognition is the principal component
analysis (PCA), which is a statistical approach
*
Corresponding author. Tel.: +1-757-683-3752; fax: +1-757-
where face images are expressed as a subset of their
683-3220. eigenvectors, and hence called eigenfaces (Sirovich
E-mail address: [email protected] (V.K. Asari). and Kirby, 1987; Turk and Pentland, 1991;

0167-8655/$ - see front matter
2003 Elsevier B.V. All rights reserved.
doi:10.1016/j.patrec.2003.11.005
430 R. Gottumukkal, V.K. Asari / Pattern Recognition Letters 25 (2004) 429–436

Moghaddam and Pentland, 1997; Martinez, 2000; 2. Review of the PCA method
Graham and Allinson, 1998). PCA has also been
used for handprint recognition (Murase et al., The PCA method has been extensively applied
1981), human-made object recognition (Murase for the task of face recognition. Approximate
and Nayar, 1995), industrial robotics (Nayar et al., reconstruction of faces in the ensemble was per-
1996), and mobile robotics (Weng, 1996). But re- formed using a weighted combination of eigen-
sults show that the recognition rate is not satis- vectors (eigenpictures), obtained from that
factory for pose variations exceeding 30 and ensemble (Sirovich and Kirby, 1987). The weights
extreme changes in illumination. that characterize the expansion of the given image
The main objective of this research is to im- in terms of eigenpictures are seen as global facial
prove the accuracy of face recognition subjected features. In an extension of that work, Kirby and
to varying facial expression, illumination and Sirovich (1990) included the inherent symmetry of
head pose. As stated before, PCA method has faces in the eigenpictures.
been a popular technique in facial image recog- All the face images in the face database are
nition. But this technique is not highly accurate represented as very long vectors, instead of the
when the illumination and pose of the facial usual matrix representation. This makes up the
images vary considerably. In this research work entire image space where each image is a point.
an attempt is made to improve the accuracy of Since the faces have a similar structure (eye, nose
this technique under the conditions of varying and mouth, position, etc.), the vectors representing
facial expression, illumination and pose. We them will be correlated. We will see that faces of
propose the modular PCA method, which is an the same class will group at a certain location in
extension of the conventional PCA method. In the image space. Hence the face images are rep-
the modular PCA method the face images are resented by a set of eigenvectors developed from a
divided into smaller images and the PCA method covariance matrix formed by the training of face
is applied on each of them. Whereas in the tra- images. The idea behind eigenimages (in our case
ditional PCA method the entire face image is eigenfaces) is to find a lower dimensional space in
considered, hence large variation in pose or illu- which shorter vectors will describe face images.
mination will affect the recognition rate pro- Fig. 1 illustrates this idea graphically.
foundly. Since in the case of modular PCA
method the original face image is divided into 2.1. Computing eigenfaces
sub-images the variations in pose or illumination
in the image will affect only some of the sub- Consider the face images in the face database to
images, hence we expect this method to have be of size L by L. These images can be represented
better recognition rate than the conventional as a vector of dimension L2 , or a point in L2 -
PCA method. A similar method called modular dimensional space. A set of images therefore cor-
eigenspaces was proposed by Pentland et al. responds to a set of points in this high dimensional
(1994). In this method PCA is performed on the
eyes and nose of the face image.
Image space coordinate 3
This paper is organized as follows: Section 2 Face images of the same
describes the conventional PCA method. Section class with local coordinates

3 explains the modular PCA method. Section 4

Face images not
describes the face databases used for testing the belonging to any class
face recognition methods. Section 5 presents
simulation results obtained by applying the PCA Image space coordinate 2
method and the proposed modular PCA method
to the face image sets with large light and pose
Image space coordinate 1
variations. Finally, a conclusion is drawn in
Section 6. Fig. 1. The image-space and face-space coordinate system.
 R. Gottumukkal, V.K. Asari / Pattern Recognition Letters 25 (2004) 429–436 431

space. Since facial images are similar in structure, 3. Modular PCA method
these points will not be randomly distributed, and
therefore can be described by a lower dimensional The PCA based face recognition method is not
subspace. PCA gives the basis vectors for this very effective under the conditions of varying pose
subspace (which is called the ‘‘face space’’). Each and illumination, since it considers the global
basis vector is of length L2 , and is the eigenvector information of each face image and represents
of the covariance matrix corresponding to the them with a set of weights. Under these conditions
original face images. the weight vectors will vary considerably from the
Let I1 ; I2 ; . . . ; IM be the training set of face weight vectors of the images with normal pose and
images. The average face is defined by illumination, hence it is difficult to identify them
correctly. On the other hand if the face images
1 XM
were divided into smaller regions and the weight
A¼ Ii ð1Þ vectors are computed for each of these regions,
M i¼1
then the weights will be more representative of the
Each face differs from the average face by the local information of the face. When there is a
vector Yi ¼ Ii  A. The covariance matrix C is ob- variation in the pose or illumination, only some of
tained as the face regions will vary and rest of the regions
will remain the same as the face regions of a nor-
1 XM mal image. Hence weights of the face regions not
C¼ Yi  YiT ð2Þ affected by varying pose and illumination will
M i¼1
closely match with the weights of the same indi-
The eigenvectors of the covariance matrix are vidualÕs face regions under normal conditions.
computed and the M 0 significant eigenvectors are Therefore it is expected that improved recognition
chosen as those with the largest corresponding rates can be obtained by following the modular
eigenvalues. From these eigenvectors, the weights PCA approach. We expect that if the face images
for each image in the training set are computed as are divided into very small regions the global
information of the face may be lost and the
WiK ¼ EKT  ðIi  AÞ 8i; K ð3Þ
accuracy of this method may deteriorate.
where EK Õs are the eigenvectors corresponding to In this method, each image in the training set is
the M 0 largest eigenvalues of C and K varies from 1 divided into N smaller images. Hence the size of
to M 0 . each sub-image will be L2 =N . These sub-images
can be represented mathematically as
2.2. Classification  
L L
Iij ðm;nÞ ¼ Ii pffiffiffiffi ðj  1Þ þ m; pffiffiffiffi ðj  1Þ þ n 8i;j
A test image Itest is projected into face space by N N
the following operation: ð5Þ
Wtest K ¼ EKT  ðItest  AÞ 8K ð4Þ where i varies from 1 to M, M being the number of
The weights WiK form a vector TpT¼ ½w1 ; images in the training set, j varies from 1 to N , N
w2 ; . . . ; wM 0 , which describes the contribution of being the numberpffiffiffiffiof sub-images and m and n vary
each eigenface in representing the input face from 1 to L= N . Fig. 2 shows the result of
image. This vector can then be used to fit the test dividing a face image into four smaller images
image to a predefined face class. A simple tech- using Eq. (5) for N ¼ 4.
nique is to compute distance of Wtest K from Tp , The average image of all the training sub-ima-
where Tp is the mean weight vector of the pth class. ges is computed as
The test image can be classified to be in class p
1 X M X N
when minðDp Þ < hi , where Dp ¼ kWtest  Tp k and hi A¼ Iij ð6Þ
is the threshold. M  N i¼1 j¼1
432 R. Gottumukkal, V.K. Asari / Pattern Recognition Letters 25 (2004) 429–436

minðDp Þ < hi for a particular value of p, the cor-

responding face class in the training set is the
closest one to the test image. Hence the test image
is recognized as belonging to the pth face class.
Fig. 2. A face image divided into N smaller images, where
N ¼ 4.
4. Image databases

The next step is to normalize each training sub- The performance of the conventional PCA
image by subtracting it from the mean as based algorithm and the modular PCA based
algorithm were evaluated with two image data-
Yij ¼ Iij  A 8i; j ð7Þ bases, UMIST and Yale. The UMIST database
From the normalized sub-images the covariance consists of images with varying pose and the Yale
matrix is computed as database consists of images with varying illumi-
nation and expressions. All the images in both the
1 X M X N
databases were normalized and cropped to a size
C¼ Yij  YijT ð8Þ
M  N i¼1 j¼1 of 64 · 64 pixels.
Next we find the eigenvectors of C that are asso- 4.1. UMIST database-pose variant
ciated with the M 0 largest eigenvalues. We repre-
sent the eigenvectors as E1 ; E2 ; . . . ; EM 0 . The For our tests we took a partial set of face
weights are computed from the eigenvectors as images consisting of 10 images each of 20 different
shown below: individuals from the UMIST face database. Each
WpnjK ¼ EKT  ðIpnj  AÞ 8p; n; j; K ð9Þ image of a person is taken at a different pose, with
a normal expression. Out of the ten images of a
where K takes the values 1; 2; . . . ; M 0 , n varies from
person, only eight were used for training and the
1 to C, C being the number of images per indi-
remaining two were used to test the recognition
vidual, and p varies from 1 to P , P being the
rate. Fig. 3a and b show the set of images of a
number of individuals in the training set. Weights
person used for training and testing respectively.
are also computed for the test sub-images using the
The choice of the training and testing images was
eigenvectors as shown in the next equation:
 
Wtest jK ¼ EKT  Itest j  A 8j; K ð10Þ
Mean weight set of each class in the training set is
computed from the weight sets of the class as
shown below:
M0 X
1 X C
TpjK ¼ WpnjK 8p; j ð11Þ
C K¼1 n¼1

Next the minimum distance is computed as shown

below:
M0 
1 X 
Wtest jK  TpjK 
Dpj ¼ 0 ð12Þ
M K¼1

1 XN
Dp ¼ Dpj ð13Þ Fig. 3. Images of an individual used for (a) training and (b)
N j¼1
testing.
 R. Gottumukkal, V.K. Asari / Pattern Recognition Letters 25 (2004) 429–436 433

made to test both the algorithms with head pose during training and testing the recognition rate
angles that lie outside the head pose angles they with the images left out. This was repeated 11
were trained with. The PCA and modular PCA times by leaving out a different image each time.
methods may perform poorly with this selection of This kind of testing is referred to as leave out one
training and testing images, but our aim is to testing in the remainder of the paper.
compare their performance for test images whose
head pose angles lie outside the head pose angles
of the training images. 5. Test results

4.2. Yale database-expression and illumination We tested the performance of PCA and modu-
variant lar PCA algorithms for varying number of eigen-
vectors. Considering more eigenvectors results in
The Yale database has 165 images of 15 adults, increased recognition rates, however the increase
11 images per person. The face images vary with in computational cost is linear with the number
respect to facial expression and illumination. The of eigenvectors. Fig. 5 shows the recognition rates
images have normal, sad, happy, sleepy, surprised, of PCA and modular PCA for varying number of
and winking expressions. There are also images eigenvectors. The results shown in Fig. 5 were
where the position of the light source is at the obtained using the Yale face database by leaving
center, left and right. In addition to these there are out one testing. Threshold was not used for this
images with and without glasses. Out of the 11 testing; hence there are no rejections, only correct
images of a person, only eight were used for recognition or false recognition. It can also be
training and the remaining three were used to test observed from Fig. 5 that the recognition rate is
the recognition rates. Fig. 4a and b show the set of increasing in both PCA and modular PCA meth-
images of a person used for training and testing ods as we increase the value of M 0 , and there is not
respectively. The choice of the training and test much improvement for M 0 > 30. Similar results
images was made to facilitate comparison of per- have been observed for values N ¼ 4, 16, 64, 256
formance of both the methods for test images with and 1024. The modular PCA results have also been
uneven illumination and partial occlusion. compared with the results of modular eigenspaces
We also conducted experiments by leaving out described by Pentland et al. (1994). It has been
one image from each individualÕs set of 11 images observed that the modular PCA algorithm pro-
vides better recognition rate with the added
advantage that it does not require the detection of
specific features like eye, nose and mouth.

1
0.9
0.8
Recognition Rate

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1 5 10 20 30 40 50
M' (Number of eigenvectors)
PCA MPCA N=16

Fig. 4. Images of an individual used for (a) training and (b) Fig. 5. Recognition rates of PCA and modular PCA with
testing. varying M 0 .
434 R. Gottumukkal, V.K. Asari / Pattern Recognition Letters 25 (2004) 429–436

Tests in the remainder of this paper were con-

ducted at M 0 ¼ 20, i.e. eigenvectors corresponding
to the 20 maximum eigenvalues of the covariance
matrix. The aim of our tests was to compare the
two algorithms with varying pose and illumination
face images and varying M 0 would have the same
effect on both the algorithms as shown in Fig. 5, Fig. 7. (a) Reconstructed image using PCA method for a test
image from the Yale database, (b) original image from the Yale
hence only the first 20 eigenvectors were consid- database, and (c) reconstructed image using modular PCA
ered for the tests. method at N ¼ 4 for a test image from the Yale database.
When the size of the sub-images is less than or
equal to 4 · 4 (N P 256), the number of eigenvec-
tors that can be obtained from the covariance tate visual comparison with the reconstructed
matrix will be less than 20, since the size of the image obtained for PCA method.
covariance matrix is less than or equal to 16 · 16.
For the case where L2 =N < 20, L2 =N eigenvectors 5.1. Results for pose variation
are considered. For example when N ¼ 256, 16
eigenvectors are considered. For the case where In this experiment we compared the recognition
N ¼ 4096, the algorithm reduces to comparing the rate, false recognition rate and false rejection rate
pixel values of test image and training images, of the two methods for large pose variations using
pixel by pixel. the images in the UMIST database. The training
We applied the PCA method and the modular and test images were chosen as described in Sec-
PCA method to reconstruct the test images. In the tion 4.1. Furthermore we vary N from 4 to 4096 to
case of the PCA method the image is reconstructed observe the effect of N on face recognition. Since
as the size of all the images in the database is 64 · 64
pixels, the maximum value N can take is 4096, i.e.
Itest ¼ A þ EKT  Wtest K ð14Þ a sub-image is a single pixel. Fig. 8 shows the
The test image is reconstructed in a similar manner recognition rate, false recognition rate and false
for the modular PCA method and is given as rejection rate for the modular PCA method with
varying N . In the case of PCA the recognition rate
Itest j ¼ A þ EKT  Wtest jK ð15Þ was 0.3, false recognition rate was 0.625 and false
Figs. 6 and 7 show the reconstructed images of a rejection rate was 0.075.
face image from the test set of the UMIST and
1.2
Yale databases using both the methods. In the
figures, the reconstructed images obtained for the 1
modular PCA method are concatenated to facili-
0.8
Rate

0.6

0.4

0.2

0
4 16 64 256 1024 4096
N

Recognition False Recognition

Fig. 6. (a) Reconstructed image using PCA method for a test False Rejection
image from the UMIST database, (b) original image from the
UMIST database, and (c) reconstructed image using modular Fig. 8. Recognition, false recognition and false rejection rates
PCA method at N ¼ 4 for a test image from the UMIST of the modular PCA method for different values of N . For PCA
database. the respective rates were 0.3, 0.625 and 0.075.
 R. Gottumukkal, V.K. Asari / Pattern Recognition Letters 25 (2004) 429–436 435

From the results we note that the modular PCA 0.9

method has a slightly better recognition rate and 0.8

0.7
false recognition rate at N ¼ 4 and 16, but the
0.6
conventional PCA method has a slightly lesser
0.5

Rate
false rejection rate. Hence the proposed method 0.4
does not have significant improvement over the 0.3
PCA method under the condition of varying pose. 0.2
0.1

5.2. Results for expression and illumination varia- 0

4 16 64 256 1024 4096
tion
N

In this experiment we compared the recognition Recognition False Recognition

rate, false recognition rate and false rejection rate False Rejection
of the two methods for large expression and illu-
mination variations using the images in the Yale Fig. 10. Recognition, false recognition and false rejection rates
of modular PCA method for different values of N . For PCA the
database. The training and test images were cho- respective rates were 0.48, 0.36 and 0.16.
sen as described in Section 4.2. As before we vary
the value of N from 4 to 4096 to observe the effect We can observe from the results that the mod-
it has on face recognition. Fig. 9 shows the rec- ular PCA method completely outperforms the
ognition rate, false recognition rate and false PCA method in all aspects for N at 4, 16 and 64.
rejection rate for the modular PCA method with However, best results were obtained for N at 16.
varying N . In the case of PCA the recognition rate Reconstruction of one of the test images was
was 0.44, false recognition rate was 0.31 and false performed using PCA and modular PCA for N at
rejection rate was 0.24. 16. The results of the reconstruction are shown in
A second set of experiments were performed by Fig. 11, the first image is the reconstructed image
leaving out one testing. The results obtained for obtained using PCA method, the second image is
modular PCA are shown in Fig. 10. For PCA, the original image and the third image is the con-
recognition rate was 0.48, false recognition rate catenation of the reconstructed images obtained
was 0.36 and false rejection rate was 0.16. using the modular PCA method for N ¼ 16.
The PCA based method was not very effective
under the conditions of varying illumination, since
it considers the global information of each face
0.9
image and represents them with a set of weights.
0.8
0.7
Under this condition the weight vectors of the test
0.6
image will vary considerably from the weight
0.5 vectors of the training images with normal illu-
Rate

0.4 mination, hence it is difficult to identify them

0.3 correctly. The huge improvement in the case of
0.2 modular PCA was observed since the face images
0.1 were divided into smaller regions and the weight
0
4 16 64 256 1024 4096
N
Recognition False Recognition
False Rejection

Fig. 9. Recognition, false recognition and false rejection rates

of modular PCA method for different values of N . For PCA the Fig. 11. Reconstructed images for a test image with varying
respective rates were 0.44, 0.31 and 0.24. illumination using the PCA and modular PCA method.
436 R. Gottumukkal, V.K. Asari / Pattern Recognition Letters 25 (2004) 429–436

vectors were computed for each of these regions, Graham, D.B., Allinson, N.M., 1998. Characterizing virtual
hence weight vectors will be more representative of eigensignatures for general purpose face recognition. In:
Face Recognition: From Theory to Applications, NATO
the local information of the face. Therefore for ASI Series F, Computer and Systems Sciences, vol. 163, pp.
variations in illumination, the weights of the face 446–456.
regions not affected by varying illumination closely Kirby, M., Sirovich, L., 1990. Application of the Karhunen–
match with the weights of the same individualÕs Loeve procedure for the characterization of human faces.
face regions under normal conditions. This leads IEEE Trans. Pattern Anal. Machine Intell. 12 (1), 103–
108.
to better recognition results using modular PCA as Martinez, A.M., 2000. Recognition of partially occluded and/or
observed in the experimental results. imprecisely localized faces using a probabilistic approach.
In: Proc. of Computer Vision and Pattern Recognition, vol.
1, pp. 712–717.
Moghaddam, B., Pentland, A., 1997. Probabilistic visual
6. Conclusion learning for object representation. IEEE Trans. Pattern
Anal. Machine Intell. PAMI-19 (7), 696–710.
A modular PCA method, which is an extension Murase, H., Kimura, F., Yoshimura, M., Miyake, Y., 1981. An
of the PCA method for face recognition has been improvement of the auto-correlation matrix in pattern
proposed. The modular PCA method performs matching method and its applications to handprinted
ÔHIRAGANAÕ. Transactions on IECE J64-D(3).
better than the PCA method under the conditions Murase, H., Nayar, S., 1995. Visual learning and recognition of
of large variations in expression and illumination. 3-D objects from appearance. Int. J. Computer Vision 14,
For large variations in pose there is no significant 5–24.
improvement in the performance of modular PCA. Nayar, S.K., Nene, N.A., Murase, H., 1996. Subspace methods
For face recognition, the modular PCA method for Robot vision. IEEE Trans. Robot. Automat. RA-12 (5),
750–758.
can be used as an alternative to the PCA method. Pentland, A., Moghaddam, B., Starner, T., 1994. View-based
In particular, the modular PCA method will be and modular eigenspaces for face recognition. IEEE Conf.
useful for identification systems subjected to large on Computer Vision and Pattern Recognition.
variations in illumination and facial expression. Sirovich, L., Kirby, M., 1987. A low-dimensional procedure for
the characterization of human faces. J. Opt. Soc. Amer. A 4
(3), 519–524.
Turk, M., Pentland, A., 1991. Eigenfaces for recognition. J.
References Cognitive Neurosci. 3 (1).
Weng, J.J., 1996. Crescepton and SHOSLIF: towards compre-
Chellappa, R., Wilson, C.L., Sirohey, S., 1995. Human and hensive visual learning. In: Nayar, S.K., Poggio, T. (Eds.),
machine recognition of faces: A survey. Proc. IEEE 83 (5), Early Visual Learning. Oxford University Press, pp. 183–
705–740. 214.