See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/340279880

Real-Time Feeling Detection through Facial Expression Recognition: A

Machine Learning Approach

Conference Paper · December 2019

CITATIONS READS

0 4,668

2 authors, including:

Mohamed Nizzad Ahamed Rameez

British College of Applied Studies
9 PUBLICATIONS 14 CITATIONS

SEE PROFILE

All content following this page was uploaded by Mohamed Nizzad Ahamed Rameez on 30 March 2020.

The user has requested enhancement of the downloaded file.

 Real-Time Feeling Detection through Facial Expression Recognition:
A Machine Learning Approach

Kunaratnam Kunarakulan1, Ahamed Rameez Mohamed Nizzad2

1Undergraduate, Department of Computing, BCAS, Kalmunai, Sri Lanka,
2Senior Lecturer, Department of Computing, BCAS, Kalmunai, Sri Lanka

Abstract
Facial expression is a task that humans perform routinely and effectively in their day to day life. It is clear
that understanding the emotional state of people can be useful for a range of applications from developing
a better understanding of human psychology, to investigating behaviour for improved user experiences
and etc. Humans are precise in reading the facial expressions of fellow humans. They usually predict
someone’s facial expression via observing the features of the face with the facial parts moment-to-moment
basis in order to glimpse into their minds. Humans naturally learn the model of facial expressions from the
environment and surroundings. They logically have facial expression models in their brain, and with the
comparison of models, they are able to predict the facial expressions of others. However, when computer
artificial intelligence (AI) technology is concerned, it is a hot topic in the field of computer vision and pattern
recognition. It plays a significant role in various application domains such as surveillance, human-robot
interaction, computer games’ user experiences, stress-monitoring, and other kinds of artificial intelligence-
based software programs. This research as an AI based facial expression recognition system with the help
of computer vision and computational algorithms attempted to make it possible to mimic the face reading
skills of humans to computers. This research aimed at designing a robust Facial Expression Recognition
system by combining various techniques from computer vision and machine learning. This research also
falls into the broad category of the psychological fundamentals of human facial expressions. Because,
before the development process of this research as computer-based artificial intelligence system, it is very
essential to study the human psychological factors about the facial expression in a more accurate manner.
Therefore, for the training and demonstration purpose some human volunteers were involved into this
research to proceed their facial expressions for different sorts of emotions. Depending on the data of
volunteers’ facial expressions and other internet resources’ facial expression images, the system had been
trained. This study presents a novel facial expression recognition system with modified Local Binary
Patterns (LBP) for feature extraction and the Convolutional Neural Network (CNN) for feature optimization
and edge detection. This research included many different kinds of artificial intelligence-based
technologies, tools that were essential to build this facial expression recognition system. The ultimate aim
of this research was to give expression detection ability to computers to find out the facial expression of a
human being. In order to accomplish this task in a precise manner, the research involved into many
different types of sub-topics and concepts in artificial intelligence such as computer vision, machine
learning, face detection, face features extraction and some other kinds of pattern recognition mechanisms.
The outcome of the research suggests that further research into this topic would lead to commercially
viable solutions in future and the researchers have intended to continue the research in future too.

Author Keywords
Machine learning, Artificial intelligence, Computer vision, Pattern recognition, Facial expression,
Computational Algorithms, Surveillance, Human-robot interaction, Local Binary Patterns (LBP),
Convolutional Neural Network (CNN)

Introduction
In this modern world, computer plays major role in all aspects. As such, Artificial Intelligence (AI) is one of
the most advanced technologies in computing and revolutionizing all industries. Computer vision is a sub set
of AI. There are many useful applications have been developed with the use of computer vision technologies
such as object detection, face recognition and etc. The AI technology is about mimicking human abilities to
computers. At present, computers are able to see real world objects, human faces, and they can even
understand human voices and perform some actions according to instructions.
In computer vision, there are many technologies available such as face detection and face recognition,
pattern recognition, image processing and etc. Therefore, with the help of those computer vision
technologies, this research explores suitable method to create a facial expression recognition system to track
human emotions. Therefore, it is essential to study the psychological background of human facial
expressions and underlying concepts.
A facial expression is one or more motions or positions of the muscles beneath the skin of the face.
According to one set of controversial theories, these movements convey the emotional state of an individual
to observers. Facial expressions are a form of nonverbal communication and Humans have the ability to
recognize facial expression naturally. There are some minor differences on facial expressions due to the
cultural changes.
Alert Mehrabian studied and found that, nonverbal part is the most informative channel in social
communication. This means Verbal part contributes about 7% of the message, vocal is 34% and facial
expression about 55%. (Merhabian, 1971).

Figure 1: Communication Theory

The Paul Ekman found that, there are 6 basic universally accepted types of facial expressions. Those are
happiness, sadness, fear, surprise, anger, and disgust (Ekman, 2009).

Figure 2: Basic six types of facial expressions

There are 7 types of basic face shapes are available such as round, oval, square, oblong, heart, triangle
and diamond. Face contains 46 action units which are responsible for muscles movements.
 Figure 3: Facial action coding system (FACS)

With the help of these psychological background of facial expression related information, This research tries
to give computers the ability to recognize facial expressions with the integration of machine learning
techniques and algorithms such as face detection and alignment, image normalization, feature extraction,
and classification.

Experimental
In order to find an effective approach to solve this problem, the psychological background of facial
expression has been thoroughly studied. Because, without having deeper level of understanding about facial
expression, it is impossible to give these humanly skills to computers.
The data collected related to psychological aspects are of Qualitative nature. However, this research also
dealt with quantitative data such as statistics, logical, binary, and mathematical tensor data in terms of
Machine Learning aspect. Therefore, this research approach can be categorized as mixed one.
At the beginning, data related to facial expressions such as muscle details, basic types of emotions, face
shapes, feeling of humans, deeper level understanding of facial expressions were collected.
 Figure 04: Sample Dataset of Faces with different expressions

Then studied the technological part of the research such as algorithms, dataset, Support Vector Machine
(SVM), Convolutional Neural Network (CNN), Local Binary Pattern (LBP) and etc

Figure 05: Architecture

In conclusion, this research bases problems and objectives in view of Qualitative data which is the
psychological background of facial expression. Then with the help of those data, it tried to find out solutions
to solve those problems and objectives using machine learning algorithms, methods and techniques. The
machine learning involved in finding solutions for those problems and objectives via Quantitative approaches
of Machine Learning.
This research outlines that there are six types of basic facial expressions: happiness, sadness, surprise, fear,
anger, and disgust as found by Paul Ekman in 1972 and face consists of 46 action units which are related to
contraction of a specific set of facial muscles. Further, according to psychological research conducted by
Alert Mehrabian, nonverbal part is the most informative channel in social communication and verbal part
contributes about 7% of the message, vocal is 34% and facial expression about 55%. Also there are seven
types of basic facial shapes are available such as round, oval, square, oblong, heart, triangle and diamond.
Further, according to the research conducted by the Northeastern University of America, the professor Lisa
Barrett found that the cultural differences will make slightly minor different changes in Facial Expressions. In
terms of Machine Learning, tensorflow is a machine learning framework for developing Artificial Intelligence
 based applications and OpenCV is another machine learning library which is used to develop computer
vision applications. Further, Haarcascade_frontalface.xml is a pre-trained dataset which is used for locating
faces in a video frame. In addition to that, CK+ is a Facial Expression Dataset which includes basic six types
of emotions such as happiness, sadness, surprise, fear, anger, and disgust.
In Conclusion, this research problem can be addressed with the help of computer vision and pattern
recognition utilizing suitable machine learning algorithms. Further, In order to give facial expression reading
skills to computers, it is suggested to Viola-Jones face detection algorithm for locating human faces in a
video frame via using Haarcascade Frontalface pre-trained XML dataset. And Convolutional Neural Network
(CNN) and Local Binary Pattern (LBP) are the right suitable machine learning algorithm for feature extraction
and classification. Preprocessing must be carried out before the feature extraction and classification stages
because, the preprocessing is a process which can be used to improve the performance of the FER system
via removing noises from images. Further, the tensorflow grap model must me trained more than 60000
steps in order to obtain better results and the input image must be converted into grayscale and should be a
single channel.

References

Ansari, s., 2017. Pattern Recognition. [Online]

Available at: https://www.geeksforgeeks.org/pattern-recognition-introduction/

Jones, P. V. a. M., 2001. Rapid Object Detection using a Boosted Cascade of Simple Features.

Mehrabian, A. and Wiener, M. (1967). Decoding of inconsistent communications, Journal of Personality and
Social Psychology, 6, 109-114

Mehrabian, A., and Ferris, S.R. (1967), Inference of Attitudes from Nonverbal Communication in Two
Channels, Journal of Consulting Psychology, 31, 3, 48-258

Mehrabian, A. (1971). Silent messages, Wadsworth, California: Belmont

Mehrabian, A. (1972). Nonverbal communication. Aldine-Atherton, Illinois: Chicago

T. Dalgleish and M. Power (Eds.). Handbook of Cognition and Emotion. Sussex, U.K.: John Wiley & Sons,
Ltd., 1999

View publication stats