See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/361108119

FACIAL EMOTION DETECTION AND RECOGNITION

Article · May 2022

DOI: 10.33564/IJEAST.2022.v07i01.027

CITATIONS READS

15 12,214

3 authors, including:

Radhey Shyam
Babu Banarasi Das University
40 PUBLICATIONS 231 CITATIONS

SEE PROFILE

All content following this page was uploaded by Radhey Shyam on 06 June 2022.

The user has requested enhancement of the downloaded file.

 International Journal of Engineering Applied Sciences and Technology, 2022
Vol. 7, Issue 1, ISSN No. 2455-2143, Pages 176-179
Published Online May 2022 in IJEAST (http://www.ijeast.com)

FACIAL EMOTION DETECTION AND

RECOGNITION
Amit Pandey, Aman Gupta, Radhey Shyam
Computer Science Department
SRMCEM, AKTU
Lucknow, India

Abstract– Facial emotional expression is a part of face various facial extraction features, emotional databases,
recognition, it has always been an easy task for humans, classifier algorithms, and so on.[4] The classical FER
but achieving the same with a computer algorithm is consists of two main steps: feature extraction and emotion
challenging. With the recent and continuous recognition. In addition, image pre-processing, including
advancements in computer vision and machine learning, face detection, cropping, and resizing. Face detection crops
it is possible to detect emotions in images, videos, etc. A the facial region after removing the backdrop and non-face
face expression recognition method based on the Deep areas. Finally, the retrieved characteristics are used to
Neural Networks especially the convolutional neural classify emotions, which is commonly done with the help of
network (CNN) and an image edge detection is proposed. neural networks (NN) and other machine learning
The edge of each layer of the image is retrieved in the approaches. The challenge of facial emotion recognition is
convolution process after the facial expression image is to automatically recognize facial emotion states with high
normalized. To maintain the texture picture's edge accuracy. Therefore, it is challenging to find the similarity
structure information, the retrieved edge information is of the same emotional state between different people since
placed on each feature image. In this research, several they may express the same emotional state in various ways.
datasets are investigated and explored for training As an example, the expression may vary in different
expression recognition models. The purpose of this situations such as the individual’s mood, skin colour, age,
paper is to make a study on face emotion detection and and the environment surrounding. Generally, FER is
recognition via Machine learning algorithms and deep separated into three major stages as shown in Figure 1: (i)
learning. This research work will present deeper insights Face Detection, (ii)Feature Extraction, and (iii) Emotion
into Face emotion detection and Recognition. It will also Classification.
highlight the variables that have an impact on its
efficacy.

Keywords - Convolutional neural network, machine

learning, deep learning, computer vision, emotion
recognition.
Fig 1: Classification Stages
I. INTRODUCTION
Human-computer interaction technology refers to a kind of In the first, stage, which is a pre-processing stage, an image
technology that takes computer equipment as the medium, of a face is detected and facial components of the face will
so as to realize the interaction between humans and be detected from the region.
computers. Face recognition system (FRS) is a mechanism
that allows cameras to automatically identify people. In the second stage, an informative feature will be extracted
Because of the importance of correct and effective FRS, it from different parts of the face. In the last stage, a classifier
drives the activeness of biometric research in the race to the needs to be trained before been use a classifier needs to be
digital world.[8] In recent years, with the rapid development trained before being used to generate labels for the
of pattern recognition and artificial intelligence, more and Emotions using the training data.[4] Facial actions are
more research has been conducted in the field of human- classified into different Action Units (AUs) and emotions
computer interaction technology. [1] Facial Emotion are categorized using collections of AU’s. Deep learning is a
Recognition (FER) is a flourishing study topic in which part of machine learning approaches that can be adapted to
many breakthroughs are being made in industries, such as emotion recognition and facial expression analysis.
automatic translation systems and machine-to-human However, deep learning depends on data size which may
contact. In contrast, the paper focus to survey and reviewing affect its performance.

176
 International Journal of Engineering Applied Sciences and Technology, 2022
Vol. 7, Issue 1, ISSN No. 2455-2143, Pages 176-179
Published Online May 2022 in IJEAST (http://www.ijeast.com)

II. LITERATURE REVIEW surveys state that developments in emotion recognition

Facial expression is the common signal for all humans to make complex systems simpler. Emotion Recognition is a
convey a mood. There are many attempts to make an difficult process since emotions can differ depending on the
automatic facial expression analysis tool as it has environment, appearance, culture, and facial reaction,
applications in many fields such as robotics, medicine, resulting in unclear data.[6]
driving assist systems, and lie detector.[5] Since the
twentieth century, Ekman et al. defined seven basic C. Deep Learning
emotions, irrespective of culture in which a human grows Deep Learning is a machine learning technique that models
with the seven expressions (anger, fear, happiness, sad, the data that are designed to do a particular task. Deep
contempt, disgust, and surprise). Discusses an extensive learning in neural networks has wide applications in the area
study on face emotion identification, including the dataset's of image recognition, classification, decision making,
features and the facial emotion recognition study classifier. pattern recognition, etc. [3].
Visual features of images are examined and some of the
classifier techniques are discussed in [6,7] which is helpful IV. PROPOSED METHODOLOGY
in the further inspection of the methods of emotion The proposed technique, the emotion database used for the
recognition. This paper examined the prediction of future study, and the Inception model are all explained in this
reactions from images based on the recognition of emotions, section. This paper uses a Haar classifier for human
using different classes of classifiers. Some of the detection. The Haar classifier is trained by Haar-like small
classification algorithms like Support vector machines, and features and also the Haar-like may be a commonly used
Neural Networks such as Convolution Neural networks. texture descriptor, and its main features are linear, edge,
There are many issues like excessive makeup pose and canter, and diagonal. The Haar-like feature can reflect the
expression which are solved using convolutional networks. grey level change of image, so it's very effective to explain
The development of computer vision and machine learning the face because many features of external body parts have
has made emotion recognition much more accurate and obvious contrast change characteristics. However, the
accessible to the general public. As a result, facial calculation of eigenvalues is extremely time-consuming. so
expression detection as a sub-field of image processing is as to enhance the calculation speed, this paper uses the
quickly expanding. Some of the possible applications are integral graph method to calculate the Haar-like values.
human-computer interaction, psychiatric observations,
drunk driver recognition, and the most important is a lie 1. Face Detection
detector. Face detection could be a pre-processing phase to
acknowledge the facial expressions of humans. a picture is
III. BACKGROUND INFORMATION segmented into two parts which have faces and other non-
A. Emotion Recognition face regions [5]. There are numerous methods used for face
Facial Recognition is a branch of computer science that detection.
deals with methods and strategies for detecting emotions in
facial expressions. It is expected that expressions can be the A. Haar Classifier
next communication medium with computers the majority Haar features are often measured by expanding or reducing
of this field's research focuses on recognizing human the dimensions of the pixel group. It used Haar-like features
emotions from movies or auditory data. The majority of the to detect a picture. Objects of varying sizes will be
research has focused on recognizing and matching faces, but discovered using this method. Haar classifier will identify a
no convolutional neural networks have been utilized to group of features that are most contributing to the face
infuse emotions into photos. Emotion Recognition is the detection problem in the training phase itself. Therefore, it's
study of identifying emotions, as well as the strategies and suitable for face detection in the training phase because it
procedures utilized to do so. Emotions can be detected may indicate high detection accuracy since the computation
through facial expressions, verbal signals, and other complexity is little.
indicators. Machine learning, neural networks, artificial
intelligence, and emotional intelligence are just a few of the 2. Feature Extraction
methods that have been used to infer emotions. Emotion Feature extraction transforms pixel data from the face region
Recognition is gaining traction in the study, which is critical into a higher-level representation of the face or its
to solving a variety of challenges. components shape, colour, texture, and spatial configuration
.Feature extraction will reduce the dimension of the input
B. Facial Emotion Recognition space while keeping the important information. Feature
Facial Emotion Recognition is a research area that tries to extraction is vital in formulating a stronger emotion
identify the emotion from the human facial expression. The categorization because the extracted facial feature gives

177
 International Journal of Engineering Applied Sciences and Technology, 2022
Vol. 7, Issue 1, ISSN No. 2455-2143, Pages 176-179
Published Online May 2022 in IJEAST (http://www.ijeast.com)

inputs to the classification module which finally categorizes A. Support Vector Machine (SVM): SVM is one of the
different emotions. famous statistical techniques employed in machine learning
Feature extraction will be divided into two categories which to analyse data used for classification and multivariate
are; (i) feature base and (ii) appearance base. analysis. SVM used different kernel functions to map data
in input space into high-dimensional feature spaces.
A. Convolutional Neural Network (CNN) Currently, CNN is
one of the foremost mainstream approaches to deep learning B. Neural Network (NN): NN executes a nonlinear
techniques. It uses a variation of multilayer perceptron reduction of the input dimensionality. It makes a statistical
designed to want minimal pre-processing. It gets its name determination regarding the category of the observed
from the type of hidden layers it has. Convolutional layers, expression. Every output unit will estimate the probability
pooling layers, fully connected layers, and normalising of the examined expression belonging to the associated
layers are common components of a CNN's hidden layers. category [5].
[2]
1. Inception-V1 toV3
The Inception network was a significant step forward in the
evolution of CNN classifiers. It is a 22-layer design with a
total of 5M parameters. It employed numerous techniques to
improve performance, both in terms of speed and precision.
This is frequently utilised in machine learning applications
[20, 21]. Inception V2, It is the 24M parameter successor to
Inception-V1. Inception-v3 is a popular image recognition
model that has been shown to achieve more than 78.1
Fig 2: Emotion Detection Process percent accuracy on the Image Net dataset. However, it is
not widely utilised. [1]

V. DATASET
To perform an experiment on FER, a regular database is
required. The information will be perceived as primary or
secondary. A primary dataset consumes an extended period
to be completed with dataset collection. For study in FER, a
Fig 3: Image Classification spread of datasets available currently There are few datasets
available for the emotion recognition problem; among those,
Karolinska Directed Emotional Faces (KDEF)and Japanese
Female facial features (JAFFE) datasets are well-known and
regarded during this study. The dataset’s images are divided
into seven main emotion categories [3]. The KDEF dataset
(also refer as KDEF for simplicity, henceforth) was
developed by Karolinska Institute, Sweden. Specifically, the
aim of the dataset was to use for perception memory
emotional attention, and backward masking experiment. The
Fig 4: Convolution Filter Operation dataset contains 4900 photos of 70 people, each of whom is
depicted in seven different emotional states.
3. Expression Classification
This stage is performed by a classifier. There are various VI. RESULT AND DISCUSSION
classifications methods accustomed extract expressions. To analyse the performance of the algorithm, the FER-2013
expression dataset was used initially. Dataset had only 7178
Supervised Learning- with 412 posers, causing accuracy to reach up to 55%
Supervised learning is a way of training a system using maximum. To overcome the problem of low efficiency,
labelled data. The tagged data serves as a manager. The multiple datasets were downloaded from the Internet, and
model is given both inputs and outputs to learn from. also author’s own pictures of different expressions were
Following that, the model would forecast for a new data included. As the number of images in the dataset increases,
point. Classification and regression are the two types of the accuracy also increased. We kept 70% of 11K dataset
supervised learning. [3] images as training and 30% of dataset images as testing
images. The number of layers and the number of filters, for

178
 International Journal of Engineering Applied Sciences and Technology, 2022
Vol. 7, Issue 1, ISSN No. 2455-2143, Pages 176-179
Published Online May 2022 in IJEAST (http://www.ijeast.com)

background removal CNN (first-part CNN) as well as face VII. CONCLUSION

feature extraction CNN (the second-part CNN), are the We propose a face expression identification approach based
same. The number of layers in this experiment ranged from on a CNN model that effectively extracts facial features in
one to eight. We discovered that the greatest accuracy was this research. The suggested method uses training sample
around 4. We assumed that the number of layers is directly image data to directly input the picture pixel value. The
proportional to accuracy and inversely proportional to ability to accurately determine emotions was greatly
execution time, which was counterintuitive. Hence due to enhanced by the removal of the background. Emotion
the maximum accuracy obtained with 4 layers, we selected expression is important in communication, hence improving
the number of layers to be 4. The execution time increased the quality of interaction between humans. Furthermore, in
as the number of layers increased, however, it did not bring the near future, the study of facial expression detection may
much value to our research. Based on the achieved test set provide improved feedback to society as well as the
accuracies, the new method outperformed the existing ones. interaction between Human-Robot interfaces (HRI).
Emotion detection mostly involves the geometric part of the
face (e.g.; eyes, eyebrows, and mouth). The review takes
into consideration of experiments which been conducted in
a controlled environment, in real-time, and in wild images.
The recent research, particularly the performance with
profile views, will be applicable to a greater range of real-
world commercial applications, such as patient monitoring
in a hospital or surveillance security. Furthermore, the
concept of facial emotion recognition could be expanded to
Table 1: Accuracy Table
include emotion recognition from speech or body motions in
order to address emerging industrial applications.
It's worth noting that the proposed method only
misclassified a few photographs with perplexing
VIII. REFERENCES
perspectives, and overall identification accuracy remains
impressive. As a result, the method suggested in this work [1] K. F. Azizan Illiana, "Facial Emotion Recognition: A
holds promise in a real-world environment where non- Brief Review," in International Conference on
frontal or angularly captured photos are the norm. Sustainable Engineering, Technology and
Management 2018 (ICSETM-2018), 2020.
[2] R. Shyam, "Convolutional Neural Network and its
Architectures.," Journal of Computer Technology &
Applications, vol. 12, no. 2, pp. 6-14, 2021.
[3] R. Shyam, "Machine Learning and Its Dominant
Paradigms," Journal of Advancements in Robotics,
vol. 8, no. 2, pp. 1-10, 2021.
[4] R. Shyam, "Automatic Face Recognition in Digital
Fig 5. Happy Emotion Fig 6. Angry Emotion
World," Advances in Computer Science and
Information Technology (ACSIT), vol. 2, no. 1, pp.
64-70, 2015.
[5] S. R. N. S. M. A. H. Akhand, "Facial Emotion
Recognition Using Transfer Learning in the Deep
CNN," MDPI, vol. 10, no. 9, 2021.
Fig 7. Neutral Emotion Fig 8. Sad Emotion [6] N. Mehendale, "Facial emotion recognition using
convolutional neural networks (FERC)," SN Applied
The output images of different images are show from figure Sciences, vol. 2, no. 3, 2020.
number 5 to 8. When numerous faces were present in the [7] N. R. S, "Emotion Recognition from Facial
same image and were at the same distance from the camera, Expression using deep learning," International
the algorithm failed. It was discovered that when the Journal of Engineering and Advanced Technology
number of photons increases, accuracy decreases due to (IJEAT), vol. 8, no. 6S, 2019.
over-fitting. Also, when the number of training photos is [8] R. Shyam, "Enhanced Object Detection with Deep
reduced, accuracy remains low. The ideal number of images Convolutional Neural Networks," International
was found to be in the range of 2000–11,000 for FER to Journal of All Research Education and Scientific
work properly. Methods (IJARESM), vol. 9, no. 7, pp. 27-36, 2021.

179

View publication stats