04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards

Towards Data Science

To make Medium work, we log user data.

Get started Open in app
By using Medium, you agree to our
Privacy Policy, including cookie policy.

Follow 594K Followers

You have 2 free member-only stories left this month. Sign up for Medium and get an extra one

The Ultimate Guide to Emotion Recognition

from Facial Expressions using Python
Build a Face Emotion Recognition (FER) Algorithm that works on both, Images and
Videos

Rahulraj Singh Jul 26 · 10 min read

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 1/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

To make Medium work, we log user data.

Get started Open in app
By using Medium, you agree to our
Privacy Policy, including cookie policy.

A Wave of Human Emotions | Photo by Andrea Piacquadio from Pexels

Emotion is one of the very few words in the English language that do not have a
concrete definition and it is understandable. It is abstract. Yet, almost every decision we
have ever made in our lives is driven by emotion. Marketing research has proven that
predicting sentiments correctly can be a huge source of growth for businesses and that’s
what we will be working on today — Reading Emotions. In the world of data and
machine learning, this concept falls under the umbrella of cognitive systems. Let us try
to decode the science behind Emotion Recognition Algorithms, and build one for
ourselves.

What exactly is a cognitive emotion detection algorithm trying to accomplish? The idea
is to replicate the human thought process based on training data (in the form of images

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 2/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

and videos of humans) and try to segment the emotions present in this data. To
To make Medium work, we log user data.
Get started Open in app
perform our analysis in this
By chapter we will
using Medium, be concentrating
you agree to our on pre-recorded images
Privacy Policy, including cookie policy.
and videos that showcase an emotion, but the same can also be implemented on a live
stream of the video feed for real-time analytics.

Visual Understanding of the World

Computer Vision is a portion of Artificial Intelligence that deals with visual data. With
the advent use of machine learning and deep learning models, computer systems today
can work with digital images and videos to understand and emotionally identify the
characteristics of the video’s contents.

Computer Vision as a computational concept was

originally seen in the 1950s where some neural
networks were used to detect edges of objects which
later advanced to handwritten text, speech, and
languages.
Several convoluted use cases justify the use of computer vision in today’s industry.
Some very basic uses are, in an examination conducted online. A web camera can read
the expressions of the user to interpret their state of mind. This is also useful in testing
emotional strength for pilots and race car drivers before they move to the cockpit for
the final drive. Many robots today, including voice assistants like Alexa and Siri, are
successfully able to mimic human behavior and speak back sentimentally. This also
traces to cognitive therapy, which deals with stress and anxiety disorders among war
veterans and equity traders, who are constantly under emotional strain.

Cognitive Science and Sentiment Analysis

Artificial Intelligence today has reached heights and lengths that were not imagined a
few years back. Programs and computer systems can now mimic human behavior,
reactions, and responses to a high level of accuracy.

Sentiment Analysis
The analysis of human sentiments, which is also referred to as mining of opinions or
Emotion AI in circumstances, is the study of different states of the human brain. Factors
that are responsible for making sentiment analysis possible are natural language
processing, computational linguistics, text mining, and analysis of biometrics.

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 3/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

The basic task of any sentiment analysis program is to isolate the polarity of the input
To make Medium work, we log user data.
Get started Open in app
(text, speech, facial expression, etc.)
By using to understand
Medium, whether the primary sentiment
you agree to our
Privacy Policy, including cookie policy.
presented is positive, negative, or neutral. Based on this initial analysis, programs then
often dig deeper to identify emotions like enjoyment, happiness, disgust, anger, fear, and
surprise.

There are two precursors to this analysis. One is the

quantifying of the input data for algorithms to read
and process, secondly, it’s the psychological research
that helps in identifying which expression stands for
what emotion.

Programmatic Analysis of Sentiments | Image by Author

Cognitive Science
In terms of computing systems, cognitive science is the study of scientific processes that
occur in the human brain. It is responsible for examining the functions of cognition,
namely, perception of thoughts, languages, memory of the brain, reasoning, and
processing received information. At a broader level, it is the study of intelligence and
behavior.

The goal of Cognitive Science is to study the human brain and understand its principles of
intelligence. This is done with the hope that by building computer systems from the
knowledge of human intelligence, machines will be able to mimic learning and develop
intelligent behavior patterns like humans.

Cognitive Science operates in three distinct levels of analysis:

1. The computational theory: At this level, the goals of the analysis are specified and
fed to the computer system. This could be an imitation of speech or an
understanding of emotions.

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 4/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

2. Representation and algorithms: In general Machine Learning terms, this is the

To make Medium work, we log user data.
Get started Open in app
training stage. Here the
By ideal input and
using Medium, output
you agree scenarios are presented to the
to our
Privacy Policy, including cookie policy.
machine and algorithms are put in place that will ultimately be responsible for
transforming input to output.

3. The hardware implementation: This is the final cognition phase. It is the enactment
of the algorithm in the real world and analysis of its working trajectory against a
human brain.

Human Cognition and its potential applications in AI | Image by Author

Face Emotion Recognition from Images

It is often exclaimed that our feelings at heart are

reflected on the face.
Expressions on the face are a vital mode of communication in humans as well as
animals. Human behavior, psychological traits, are all easily studied using facial
expressions. It is also widely used in medical treatments and therapies. In this section,
we will be working on images of facial expressions and portraits of faces to decode the
sentiment presented in the image. In a later section, we will work on doing the same
steps with a video-based input.

Face Emotion Recognizer

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 5/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

The Face Emotion Recognizer (generally knowns as the FER) is an open-source Python
To make Medium work, we log user data.
Get started Open in app
library built and maintained by Justin
By using Shenk
Medium, and
you agree is used for sentiment analysis of
to our
Privacy Policy, including cookie policy.
images and videos. The project is built on a version that uses a convolution neural
network with weights mentioned in the HDF5 data file present in the source code (The
execution of FER can be found here) of this system’s creation model. This can be
overridden by using the FER constructor when the model is called and initiated.

1. MTCNN (multi cascade convolutional network) is a parameter of the constructor. It

is a technique to detect faces. When it is set to ‘True’ the MTCNN model is used to
detect faces, and when it is set to ‘False’ the function uses the default OpenCV
Haarcascade classifier.

2. detect_emotions(): This function is used to classify the detection of emotion and it

registers the output into six categories, namely, ‘fear’, ‘neutral’, ‘happy’, ’sad’, ‘anger’,
and ‘disgust’. Every emotion is calculated, and the output is put on a scale of 0 to 1.

The flow of Logic: The program starts by taking into input the image or video that
needs analysis. The FER() constructor is initialized by giving it a face detection
classifier (either Open CV Haarcascade or MTCNN). We then call this constructor’s
detect emotions function by passing the input object (image or video) to it. The result
achieved is an array of emotions with a value mentioned against each. Finally, the
‘top_emotion’ function can seclude the highest valued emotion of the object and return
it.

Dependencies for installing FER are OpenCV version 3.2 or greater, TensorFlow version 1.7
or greater, and Python 3.6. Let us now see the implementation of this algorithm for images.

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 6/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

To make Medium work, we log user data.

Get started Open in app
By using Medium, you agree to our
Privacy Policy, including cookie policy.

Output
The code individually takes images as input and details out various emotions and their
separate intensity levels in the output. Using top_emotion() we then extract the most
dominant sentiment of the image.

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 7/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

To make Medium work, we log user data.

Get started Open in app
By using Medium, you agree to our
Privacy Policy, including cookie policy.

The output of the Image Analysis Code | Image by Author

We have now observed how images can be analyzed to retrieve the expressions and
emotional states of people present in those images. In the next part, we will perform
the same analysis using videos.

Complete Code Repository (Image Input)

ai-with-python-series/06. Emotion Recognition using Facial

Images
Python for AI. Tutorial — Emotion Recognition using Facial
Expressions from Images.
github.com

Although the code snippet above implements the core logic of the program, I would
recommend going through the complete Colab Notebook present in the linked
repository above for a detailed understanding of the entire working code.

Recognizing Facial Emotions from Live Videos

Similar to our processing of images for sentiment extraction, in this section, we will
work on videos.

Theoretically, a video is a combination of continuous

image frames put together in motion.
So essentially, the working of any algorithm for both videos and images is the same.
The only additional step in processing videos is splitting the video into all of its
individual frames and then applying the image processing algorithms to it.

The flow of Logic: Although the underlying algorithm is similar for both images and
videos, there are a few key changes that we will follow for videos.

1. Video_analyze(): This function is responsible for extracting the individual image

frames from a video and then analyze those independently.

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 8/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

2. Each frame analyzed by this function is stored as a separate image by the algorithm
To make Medium work, we log user data.
Get started Open in app
in the root directory folder where
By using the
Medium, youcode
agreeis
to running.
our Also, this function later
creates a replica of thePrivacy Policy, including cookie policy.
original video by placing a box around the face and showing
live emotions within the video.

3. We then create a Pandas DataFrame from these analyzed values and plot this
dataframe using matplotlib. In this plot, we can see every emotion plotted against
time.

4. We can further analyze this dataframe by taking individual emotion values that
were recognized by the model and finding which sentiment was dominant across
the entire video.

This way, we can work on videos by extracting individual image frames and analyzing
them. This process is displayed in the diagram below that shows how an additional step
gets added for processing videos. We will be seeing this implementation in the section
below.

Integrating an additional step into the Image Processing Algorithm and extending it to work for Videos | Image by
Author

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 9/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

To make Medium work, we log user data.

Get started Open in app
By using Medium, you agree to our
Privacy Policy, including cookie policy.

Output

The sequence of Emotions throughout the video’s length is plotted on a graph | Part of Code Output, Image
by Author

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 10/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

To make Medium work, we log user data.

Get started Open in app
By using Medium, you agree to our
Privacy Policy, including cookie policy.

Final Resultant Emotion Intensities | Image by Author

Live Output generated by the Video Processing code | Image by Author

With this, we conclude the analysis of both images and videos to perform Emotion
Recognition. We were able to successfully work with human faces and understand
sentiments shown in facial expressions.

Complete Code Repository (Video Input)

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 11/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

To make Medium work, we log user data.

Get started Open in app
ai-with-python-series/07.By
Emotion Recognition
using Medium, you agreeusing
to our Live Video
Privacy Policy, including cookie policy.
Python for AI — Tutorial — Emotion Recognition from Live Video
Input
github.com

Similar to the example for working with images, the code snippet above shows a
working logic. Yet, I would encourage you to go through the complete working code
present in the Colab Notebook with the repository linked above.

Conclusion

Facial Emotions | Photo by Tengyart on Unsplash

Sentiment Analysis and Face Detection, individually have numerous use-cases in

today’s world. We see object detection algorithms in public parking lots, traffic
monitoring systems, etc. that take images of people driving vehicles to keep records.
Sentiment Analysis is furthermore used in therapy where physical meetings of the
therapist and their patient are not possible. The study of human cognition has also
evolved medicines. On the technological front, virtual assistants, profile evaluation

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 12/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

assistants, and automationTobots are built to mimic the actions of humans and replace
make Medium work, we log user data.
Get started Open in app
them with the hope of increasing accuracy
By using Medium, and decreasing
you agree to our errors. It is therefore a very
Privacy Policy, including cookie policy.
important part of the Artificial Intelligence inspired world we live in today. A more
engrossing and complicated approach to computer vision is by using cloud-based
algorithms like Azure Cognitive Services or Deep Learning mechanisms, which we have
not covered in this story, but could come in handy for complex scenarios. Through this
story, we have learned the below:

Cognitive Science is the study of human thought processes and aims at delivering
human reactions and emotions to machines through algorithms.

Computer Vision is the branch of Artificial Intelligence that focuses on

implementing Cognitive Science in the real world by working with human data in
the form of images.

Image Processing is a part of all Computer Vision algorithms that helps algorithms
understand images, process them, work with them as numeric vectors and perform
necessary operations.

We used the power of Artificial Intelligence to work on Cognitive Science and dealt
with human faces, this space is generally referred to as Computer Vision. We were able
to extract emotions out of photos and videos of human faces.

Read Extensive Step-by-Step Tutorials on Machine Learning

Implementing an End-to-End Machine Learning Workflow with

Azure Data Factory
Build a complete MLOps Lifecycle on One Screen using Microsoft’s
Azure Data Factory (ADF)
towardsdatascience.com

A Step-by-Step Guide to Speech Recognition and Audio Signal

Processing in Python
The Science of Teaching Human Vocabulary to Machines
towardsdatascience.com

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 13/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

To make Medium work, we log user data.

Get started Open in app
By using Medium, you agree to our
Privacy Policy, including cookie policy.

Logic Programming and the Design of Humanistic AI using Python

Build Logic, and Facilitate Artificial General Intelligence in Machines
towardsdatascience.com

About Me
I am Rahul, currently researching Artificial Intelligence and implementing Big Data
Analytics on Xbox Games. I work with Microsoft. Apart from professional work, I am
also trying to work out a program that deals with understanding how economic
situations can be improved across developing nations in the world by using AI.

I am at Columbia University in New York, and you are free to connect with me on
LinkedIn on Twitter.

References
1. https://en.wikipedia.org/wiki/Digital_image_processing

2. https://sisu.ut.ee/imageprocessing/book/1

3. https://en.wikipedia.org/wiki/Video_processing

4. https://www.sciencedirect.com/topics/computer-science/video-processing

5. https://cognitiveclass.ai/courses/python-for-data-science

6. https://www.geeksforgeeks.org/python-process-images-of-a-video-using-opencv/

7. https://towardsdatascience.com/face-detection-recognition-and-emotion-
detection-in-8-lines-of-code-b2ce32d4d5de

8. https://www.frontiersin.org/articles/10.3389/fnhum.2021.621493/full

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 14/15
04-11-2021 01:41 The Ultimate Guide to Emotion Recognition from Facial Expressions using Python | by Rahulraj Singh | Towards Data Science

9. https://analyticsindiamag.com/face-emotion-recognizer-in-6-lines-of-code/
To make Medium work, we log user data.
Get started Open in app
By using Medium, you agree to our
Privacy Policy, including cookie policy.

Sign up for The Variable

By Towards Data Science

Every Thursday, the Variable delivers the very best of Towards Data Science: from hands-on tutorials
and cutting-edge research to original features you don't want to miss. Take a look.

Get this newsletter

Data Science Technology Programming Artificial Intelligence Machine Learning

About Write Help Legal

Get the Medium app

https://towardsdatascience.com/the-ultimate-guide-to-emotion-recognition-from-facial-expressions-using-python-64e58d4324ff 15/15