2015 International Conference on Advances in Computer Engineering and Applications (ICACEA)

IMS Engineering College, Ghaziabad, India

A Comparative Study on different approaches of Real

Time Human Emotion Recognition based on Facial
Expression Detection
Anurag De Ashim Saha
Computer Science and Engineering Dept. Computer Science and Engineering Dept.
NIT Agartala NIT Agartala
Agartala, India Agartala, India
[email protected] [email protected]

Abstract—Facial Expression Recognition lies in one of the Human emotion recognition can have various applications
crucial areas of research for human-computer interaction and and implementations in numerous contexts. While the most
human emotion identification. For a system to recognize a facial encouraging one of them is mainly the man-machine
expression it needs to come across multiple variability of human interaction, patient monitoring, analyzing a suspect for anti-
face like color, texture, posture, expression, orientation and so on. social motives etc. might be other useful areas for emotion
The first step to recognize a facial expression of a person with recognition. With emotion recognition system the center can
various facial movements of the muscles beneath the eyes, nose analyze customer’s reaction on seeing certain product or
and lips are to be detected and further classifying those features advertisement or upon receiving a particular piece of
by comparing them with a set of trained data values using a good information or message. Based on the response whether they
classifier for recognizing the emotion. In this paper a are happy or sad or disgusted, etc. the service center can
comparative study of the different approaches initiated for
modify their approach [12].
automatic real-time facial expression recognition is undertaken
along with their benefits and flaws which will further help in
developing and improving the system. In a hypothesized form of a facial expression recognition
system, an input sensing device such as a webcam or a basic
Keywords—Facial Expression; Human-Computer Interaction; camera obtained the input image from a subject and then it
Real-time; Emotion Recognition communicates with the computer. After detection of the
representative features from the face region, the emotionally
expressive facial image are obtained, it is then preprocessed
I. INTRODUCTION
and a classifier is used to classify them into one of the emotion
A facial expression is one or more motions or positions of classes such as anger, fear, surprise, happy, neutral etc. There
the muscles lying beneath the skin of the human face. These are several detection method as well as classifier algorithms
muscle movements are used to convey the emotional state of that can be used in the detection and classification [12]. The
an individual to various observers. Facial expressions are a rest of the paper is organized as follows. Section II describes
type of nonverbal communication. If Facial expression the importance and role of facial expressions. In section III the
Recognition or Human facial emotion recognition software is background study and literature review has been discussed.
carefully equipped in an analysis center, it can very much Section IV describes the emotion taxonomy. In section V we
produce valuable outcomes. Expression Recognition or describe the related work on emotion recognition and Section
emotion recognition is one of the fresh concepts which is VI compares the different algorithms of real-time facial
gaining momentum in the vast field of research on intelligent expression recognition followed by conclusion and future
systems and Human Computer Interaction. Though facial scope in section VII.
expressions obviously does not necessarily convey emotions,
in the human-computer interaction community, the term
“facial expression recognition” often refers to the II. THE IMPORTANCE OF FACIAL EXPRESSION
classification of facial features into one of the six so called Eye is considered an important part of the face for facial
basic emotions: happiness, sadness, fear, disgust, surprise and expression recognition. In addition to that muscular
anger, as introduced by Ekman [2]. This concept of movements of the mouth and lip regions plays a very
interpretation is based on the assumption that the appearance significant and important role in conveying the emotional state
of emotions is universal across peoples as well as human of a human being. Fasel and Luttin define facial expressions as
ethnics, races and cultures. temporally deformed facial features such as eye lids, eye
brows, nose, lips and skin texture generated by contractions of
facial muscles [3]. They observed typical changes of muscular
978-1-4673-6911-4/15/$31.00©2015 IEEE
483
 2015 International Conference on Advances in Computer Engineering and Applications (ICACEA)
IMS Engineering College, Ghaziabad, India

activities to be brief, “lasting for a few seconds, but rarely

more than five seconds or less than 250ms” [3]. They also
pointed out the important fact that felt emotions are only one
source of facial expressions besides others like verbal and
non-verbal communication or physiological activities.

People generally use number of nonverbal signs such as

facial expressions, vocal nuances, hand and body gestures, and
body posture to communicate their emotions. It acts as our
most natural and significant way of communicating and
understanding other humans' emotional state and intentions.
This is aided by the experiments conducted in [1] which show
in everyday interaction, 7% of the communication happens
through language, 38% via paralanguage whereas facial
expressions contribute to the 55% of the communication. It is
indeed, that the study of human facial expressions and
emotions has various aspects, starting from computer
simulation and analysis to understanding its number of roles in
art, nonverbal communication, and the emotional process.

III. BACKGROUND AND LITERATURE REVIEW

Human emotion detection through facial expression Fig. 1. The Circumflex Model of Russell.
recognition is gaining high popularity for its increasing scope
of applications in human-computer interaction based systems. facial expressions. The universality of these basic emotions,
Though several method of emotion recognition, including having its roots in the universality thesis proposed by Charles
facial expression, voice, gesture and posture has been studied Darwin, was further supported by the cross-cultural studies in
in the literature, emotion recognition mainly comprises two [9]. This categorical description has gain popularity and
basic steps involving feature extraction and classification [12]. possesses an advantage from the fact that facial expressions
Feature extraction refers to determining a set of features or connected to basic emotions are easily recognized and
attributes, preferably they are independent, which together described by humans. This model of emotion subspace has
represents a given emotional facial expression. While become the most common model for measuring emotion, and
classification does mapping of emotional features into one of the facial expressions associated with these basic emotions
several emotion classes such as happy, anger, surprise, sad, have dictated the studies related to facial expression
disgust, etc. The set of features that are considered for recognition over the last four decades. An alternative
extraction and the classifier that is used for the task of description model of human emotion was proposed by Russel
classification are both equally important to determine the [11] where emotional states are represented by circle as in two
performance of a facial expression recognition system. For a dimensional bipolar space (Pleasantness unpleasantness,
poorly selected set of features, sometimes, even a good arousal-sleep) rather than specific discrete categories. For
classification algorithm cannot give a optimum result. Thus, example anger might be perceived as conveyance of extreme
selecting better features has always remained a pre-requisite displeasure and moderately high arousal. Fig. 1, shows the
for high classification accuracy and good result. Several Circumflex Model of Russell [11].
methods of emotion recognition from facial expression have
been developed and implemented over the last three decades.
V. RELATED WORK ON EMOTION RECOGNITION
Although emotion recognition has a broad literature, it is
IV. EMOTION TAXONOMY still now considered a complicated problem for research for
Emotion theorists and psychologists have defined several the following reasons like the nature and surroundings of
models for emotion classification ranging from universally people are not the same all over the planet. People sometimes
displayed basic emotions to culturally specific complex ones. tend to express a feeling that is in his mind but cannot always
Out of the various models in emotion research, there are two express through physical expressions in the same way as
that have dominated facial expression research: Ekman's basic others do which in result gives rise to mixed and complex
set of emotions [2], and Russell's circumplex model of affect emotional states which makes it difficult for an expression
[11]. Ekman and Freisen in 1971 [2] proposed six prototypical recognition system to recognize. Among alluring works on
basic emotions such as anger, disgust, fear, joy, sadness and facial expression recognition, the work by Ekman and Friesen
surprise which are universally displayed and recognized from [8] needs to be kept in mind. They proposed a scheme for
recognition of facial expression from different regions of face,

484
 2015 International Conference on Advances in Computer Engineering and Applications (ICACEA)
IMS Engineering College, Ghaziabad, India

VI. COMPARISON BETWEEN DIFFERENT ALGORITHMS OF REAL

TIME EMOTION RECOGNITION BASED ON FACIAL EXPRESSION
DETECTION
A. Real time facial expression recognition using a novel
method, 2012- by Saumil Srivastava

This research work proposes a method of real time facial

expression recognition from a live video input by a webcam
facing the challenges like Real Time Expression Recognition
results, person independent design, possession of Light
Invariant and Pose Invariant properties [4].

Fig. 2. An Example of Real Time Facial Expression Recognition using a live 1) Proposed Algorithm: The steps of the proposed
video feed for expression Happiness. algorithm are as follows.
a) Frame Extraction: It is primarily used for sequencing
the input video from the webcam into separate frames.
b) Face Detection: In this step, Haar Classifiers and
interleaved Haar Classifiers are used to process each frame
first. Then Skin Detection Algorithm is being used to detect
the skin part of the face for further processing.
c) Facial Feature Point Extraction: The feature points
on the face is being located in this phase.
d) Emotion Classification: SVM is used to for the
classification of emotions into classes like Neutral, Happiness,
Anger and Excitement.
Fig. 3. An Example of Real Time Facial Expression Recognition using a live
video feed for expression Surprise. 2) Benefit(s): This model of real time facial expression
recognition generated result directly proportional to intensity
e. g. cheek, chin, and wrinkles. It reports a direct correlation of of the amount of training provided. The result is robust to pose
facial expression with the eyes, the eye-brows, and the mouth. variations, light intensity changes, possess person independent
Saumil Srivastava [4] proposed a method of real-time facial property and also robust to background changes.
expression recognition from a live video input by a webcam
facing some of the various challenges like Real Time 3) Flaw(s): The overall facial expression recognition
Expression Recognition results, person independent design,
process conducted by this model depends upon the distance
possession of Light Invariant and Pose Invariant properties.
PENG Zhao-yi, ZHU Yan-hui and ZHOU Yu [5] modeled a between the webcam and the user and the effect of the
real-time facial expression recognition system applying an distance is to be minimized and also training database is not
adaptive Canny operator edge detection combined with AAM much extensive for optimum classification results.
(Active Appearance Model) algorithm for extraction features
and least squares method for classifying the expressions. P.M. B. Real-time Facial Expression Recognition Based on
Chavan et al. [6] described the framework for implementation
Adaptive Canny Operator Edge Detection, 2010- by PENG
of a real time facial feature extraction and facial expression
recognition application in which the frontal face of the subject Zhao-yi, ZHU Yan-hui and ZHOU Yu
is automatically detected from the input face image and
classified with respect to seven emotions in real time which The Real-time Facial Expression Recognition model
are neutral, anger, disgust, fear, joy, sadness and surprise. proposed by them uses an adaptive Canny operator edge
Gwen Littlewort et al. [7] designed a software tool named detection combined with AAM (Active Appearance Model)
CERT for fully automatic real-time facial expression algorithm for feature extraction phase and classifies and
recognition. He Li et al. [10] illustrated a real-time user recognizes the facial expressions using least squares method
independent facial expression recognition system that [5].
recognizes various facial expressions depending upon
illumination corrected image sequences. Fig. 2 and Fig. 3 1) Proposed Algorithm: The steps of the proposed
displays examples of real-time facial expression recognition algorithm are as follows.
for expression happiness and surprise respectively [13]. a) Video Capture: It is required for sequencing the input
video into individual image sequences.

485
 2015 International Conference on Advances in Computer Engineering and Applications (ICACEA)
IMS Engineering College, Ghaziabad, India

b) Image Preprocessing. x The system recognizes emotions well for faces with
c) Face Detection. different shapes, complexions as well as skin tones
and to an extent successful in sensing basic six
d) Facial Feature Extraction. emotional facial expressions.
e) Expression Feature Extraction. x The system also handles face rotations across x-axis
and successfully obtains accurate results for
f) Classification and Recognition: The output of step 4 horizontal rotations.
and step 5 is supplied as input to the final phase of Expression
x The system is accurate enough in detecting the
classification and recognition. emotions for a single face in the input image.

2) Benefit(s): The Real Time Facial Expression 3) Flaw(s): The disadvantages of the system are stated as
Recognition System proposed here is capable of suppressing follows.
the noise produced to an extent and the Time Complexity of
the algorithm is less than the traditional canny operator with x The system cannot handle face rotations along y-axis
improvement in facial feature detection capability. and thus fails to obtain results for vertical rotations of
image.
3) Flaw(s): The system is not much efficient in real time x It is also unable to recognize compound or mixed
multi-pose facial expression recognition which increases the emotions.
difficulty of extracting facial features and results in decreased x The emotion recognition performance of the system
recognition accuracy. reduces drastically for multiple faces in the input
image, thus failing to detect multiple faces.

C. Real Time Emotion Recognition through Facial

Expressions for Desktop Devices, 2013- by P.M. Chavan D. The Computer Expression Recognition Toolbox (CERT),
et al. 2011- by Gwen Littlewort et al.

This paper illustrates the framework for implementation of It is a software tool designed for fully automatic real-time
a real time facial feature extraction and facial expression facial expression recognition. It can automatically code the
recognition application in which the frontal face of the subject intensity of nineteen different facial actions and six different
is automatically detected from the input image and classified prototypical facial expressions. It can also estimate the
with respect to seven emotions in real time which are neutral, locations of ten facial features together with the 3-D
anger, disgust, fear, joy, sadness and surprise [6]. orientation (yaw, pitch and roll) of the head [7].

1) Proposed Algorithm: The steps of the proposed 1) Proposed Algorithm: The steps of the proposed
algorithm are as follows. algorithm are as follows.
a) Live Streaming: Live streaming of the input video a) Face Detection.
until input image frame is captured. b) Facial Feature Detection.
b) Input image frame is obtained. c) Face Registration.
c) Skin color segmentation of the input image frame. d) Gabor Feature Extraction.
d) Face Detection: Face part is detected from the skin e) SVM Classification for estimating the Action Unit
color segmented image. Intensities.
e) Eye Detection: Eye region is detected from the face f) Expression Recognition.
part.
2) Benefit(s): The CERT facial expression recognition
f) Lip Detection: Lip region is detected from the face
system is capable of implementing itself in applications like
part.
Automated Detection of driver fatigue and Automated
g) The detected eye and lip region is given input to the
Bezier Curve Algorithm. Teaching Systems and is a completely automatic real-time
h) Longest Binary Pattern is obtained from the face facial expression recognition system.
part.
3) Flaw(s): The disadvantages of the system are stated as
i) Emotion Recognition: The Bezier Curves along with
follows.
the Longest Binary Pattern is compared with the training
database to recognize the specified emotion. x Facial Expression Recognition from non-frontal head
j) Output Display: The emotion of the input image is poses is not efficient.
displayed as output. x Recognition performance across a wide range of
ethnicities is not satisfactory.
2) Benefit(s): The advantages of the system are stated as
follows.

486
 2015 International Conference on Advances in Computer Engineering and Applications (ICACEA)
IMS Engineering College, Ghaziabad, India

x Learning algorithms supporting unlabeled and weakly illumination corrected image sequences, etc. which will be
labeled datasets not yet developed. helpful for further research and development of the system.
E. Real-Time Facial Expression Recognition with
Illumination –corrected Image Sequences, 2008- by He Li ACKNOWLEDGMENT
et. al. This research work has been carried out in National
Institute of Technology, Agartala, India sponsored by All
This research work models a real-time user independent India Council for Technical Education (AICTE), Govt. of
facial expression recognition system that recognizes various India under Research Promotional Scheme (RPS)(Ref No:
facial expressions at 30fps [10]. 8023/RID/RPS-23/(NER)/2011-12).

1) Proposed Algorithm: The steps of the proposed

algorithm are as follows. REFERENCES
a) Face Alignment: The Face is tracked in this stage. [1] Mehrabian, Communication without words, Psychology Today, vol.2,
no. 4, pp.53-56, 1968.
b) Face Modelling: This stage applies changes in
[2] P. Ekman and Friesen, “Constants across cultures in the face and
illumination with a user-independent appearance-based model. emotion,” Journal of Personality and Social Psychology, vol. 17, no. 2,
c) Facial Expression Recognition: This is the last phase pp.124–129, 1971.
and it comprises of sub steps Dimensionality Reduction and [3] B. Fasel and J. Luettin, “Automatic facial expression analysis: a survey,”
Sequential Classification. Pattern Recognition, vol.36, no.1, pp. 259-275, 2003.
[4] Saumil Srivastava, “Real time facial expression recognition using a
2) Benefit(s): The advantages of the system are stated as novel method,” The International Journal of Multimedia & Its
Applications (IJMA), vol. 4, no. 2, April 2012, doi:
follows. 10.5121/ijma.2012.4204
[5] PENG Zhao-yi, ZHU Yan-hui, ZHOU Yu, “Real time facial expression
x It uses user independent models both in the tracking recognition based on canny operator edge detection,” 2010 Second
and classification phase. International Conference on MultiMedia and Information Technology,
ISBN: 978-0-7695-4008-5/10 © 2010 IEEE.
x A change in the training dataset can enable the
system to identify other facial expressions. [6] P. M. Chavan, Manan C .Jadhav, Jinal B. Mashruwala, Aditi K. Nehete,
Pooja A. Panjari, “Real time emotion recognition through facial
x Successful in achieving an 86% expression expressions for desktop devices,” International Journal of Emerging
recognition success on a set of 322 image sequences Science and Engineering (IJESE), ISSN: 2319–6378, vol. 1, no. 7, May
from the Cohn-Kanade database. 2013.
x Presents a complete system which can detect, track [7] Gwen Littlewort et al., “The computer expression recognition toolbox
and recognize a human facial expression in real-time. (CERT),” 2011 IEEE International Confernce on Automatic Face &
Gesture Recognition and Workshops (FG 2011), ISBN: 978-1-4244-
9140-7, doi: 10.1109/FG.2011.5771414.
3) Flaw(s): The disadvantages of the system are stated as [8] P. Ekman and W. V. Friesen, “Unmasking the face: a guide to
follows. recognizing emotions from facial clues,” Englewood Cliffs, NJ:
Prentice-Hall, 1975.
x The performance of the user and illumination [9] P. Ekman, “ Strong evidence for universals in facial expressions: a reply
to Russell's mistaken critique,” Psychology Bulletin, vol. 115, no. 2,
independence deteriorates for a non-average face pp.268-287, 1994.
appearing user. [10] He Li, Jose M. Buenaposada, Luis Baumela, “Real time facial
x Overall recognition performance of the model expression recognition with illumination-corrected image sequences,”
worsens for untrained users. 8th IEEE International Confernce on Automatic Face & Gesture
Recognition, 2008. FG ’08, ISBN: 978-1-4244-2154-1/08/$25.00 ©
2008 IEEE.
VII. CONCLUSION [11] J.A. Russell, “A circumflex model of affect,” Journal of Personality and
Social Psychology, vol. 39, no. 6, Dec 1980, 1161-1178.
Emotion recognition through facial expression detection is [12] Ashim Saha, Anurag De, Dr. M .C. Pal and Nirmalya Kar, “Different
a challenging task in the area of image processing and human- techniques of automatic facial expression recognition: a survey,” Proc.
computer interaction. Extensive research have already been of the Second Intl. Conf. on Advances In Computing, Communication
conducted in this field for around past two decades and last and Information Technology- CCIT 2014. Copyright © Institute of
few years it received a great amount of attention due of its Research Engineers and Doctors, USA, ISBN: 978-1-63248-005-7, doi:
10.15224/ 978-1-63248-005-7-25.
various applications and implementations in many domains. In
[13] http://www.visual-recognition.nl/Demo.html
this paper we have presented a comparative study on various
approaches of real-time emotion recognition through detection
of facial expression from a live video feed by a webcam using
approaches such as adaptive Canny operator edge detection,
modelling a software tool for fully automatic real-time facial
expression recognition, facial expression recognition using

487