University Institute of Technology Rajiv Gandhi

ProudyogikiVishwavidyalaya BHOPAL (M.P.)

MAJOR PROJECT REPORT

ON

“Animate”: Create your own Emoji

DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

University Institute of Technology– RGPV

(State Technological University of Madhya Pradesh)

Airport Road, Bhopal-462033, MP, India.

Submitted by: Under the guidance of:

Annanya Jain (0101CS171014) Prof. Raju Baraskar

KomalKabir (0101CS171052) Prof. Priyanka Sharma

Mahak Joshi (0101CS171056)

MinaksheeWanode (0101CS171060)
 UNIVERSITY INSTITUTE OF TECHNOLOGY
RAJIV GANDHI PROUDYOGIKI VISHWAVIDYALAYA

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

JULY 2020-JANUARY 2021

CERTIFICATE
This is to certify that Annanya Jain, KomalKabir, Mahak Joshi,
MinaksheeWanodeof B.E 4th Year, Computer Science & Engineering have
completed his/her major project entitled “Animate: Create your own emoji”
during academic year 2020-21 under our guidance and supervision.
We approve the report for the submission for the partial fulfillment of the
degree from UIT-RGPV, Bhopal (M.P.) is a record of bona fide work carried out
by them undersupervision.

Prof. Raju Baraskar(DoCSE) Dr. Sanjay Silakari

(Guide) Professor and HoD

Prof. Priyanka Sharma (DoCSE) DoCSE, UIT-RGPV

(Guide)

2|Page
 DECLARATION

I hereby declare that the work which is being presented in this Major Project
entitled. “Animate: Create your own emoji” Submitted to University Institute
of Technology, RGPV in the partial fulfillment of the requirements for the
award of the degree of BACHELOR OF ENGINEERING in COMPUTER SCIENCE
AND ENGINEERING.
The work has carried out at University Institute of Technology, RGPV Bhopal is
an authentic record of our work under the supervision of Prof. Raju
Baraskarand Prof. Priyanka Sharma Department of Computer Science and
Engineering, UIT-RGPV , Bhopal.
The matter of this project has not been submitted by us for the award of any
other degree or Professional diploma.

Annanya Jain (0101CS171014)

KomalKabir (0101CS171052)

Mahak Joshi (0101CS171056)

MinaksheeWanode (0101CS171060)

3|Page
 ACKNOWLEDGEMENT

We take the opportunity to express our cordial gratitude and sense of

indebtedness to our guides Prof. Raju Baraskarand Prof. Priyanka Sharma, for
the valuable guidance and inspiration throughout the duration. We are
thankful to them for their innovative ideas, which led to successful completion
of this reportwork. They have always welcomed our problem and helped us to
clear our doubt. We will always be grateful to them for providing us moral
support and sufficient time.

At the same time, we would also like to thank Dr. Sanjay Silakari (HOD,
Department of Computer Science And Engineering) for his valuable
cooperation.

We would also like to thank our instructors, faculty members and staff without
whom this project would have been a distant reality. We also extend our
heartful thanks to our family and friends.

4|Page
 Table of contents
CHAPTER-1
1. Introduction………………………………………………………………………..7-9
1.1 What is Deep learning……………………………………………………..7-7
1.2 AL vs ML vs DL………………………………………………………………….8-9
1.3 Objective………………………………………………………………………….9-9

CHAPTER-2
2. Literature Review…………………………………………………………………10-11
2.1 Google Colab…………………………………………………………………….10-10
2.2 Kaggle……………………………………………………………………………….10-10
2.3 OpenCV…………………………………………………………………………….11-11

CHAPTER-3
3. Problem Statement……………………………………………………………….12-12

3.1 Problem Statement………………………………………………………………..12-12

CHAPTER-4

4. Proposed Work/Methodology……………………………………..……..13-17
4.1 Overview……………………………………………………………………….…13-13
4.2 Use Case diagram………………………………………………………….…14-17

CHAPTER-5

5|Page
 5. Design and Development………………………………….………………18-21
5.1 Hardware Requirement………………………………………..………18-18
5.2 Software Requirement………………………………………………….18-19
5.3 Technologies Specification…………………………………………….20-20
5.4 Hardware Specification………………………………………………….21-21

CHAPTER-6

6. Implementation…………………………………………………………………22-33
6.1 Codes……………………………………………………………………………..22-30
6.2 Results……………………………………………………………………………31-33

CHAPTER-7

7. Testing………………………………………………………………………………..34-37
7.1 Goal of DL testing…………………………………………………………….34-34
7.2 Model Evaluation…………………………………………………………….34-35
7.3 Evaluation using metrices………………………………………………..36-37
7.4 Conclusion……………………………………………………………………….37-37

CHAPTER-8

8. Conclusion and Future…………………………………………………………38-38

8.1 Conclusion……………………………………………………………………….38-38
8.2 Future………………………………………………………………………………38-38

CHAPTER-9

9. Reference /Bibliography………………………………………………………39-39
CHAPTER-10
10. Plagiarism Report………………………………………………………………..40-44

6|Page
 Chapter 1: Introduction

Emoji characters are becoming more and more popular for communicating
emotions along with written messages. Emoji characters are restricted to
predefined character. These characters are not original and personalized. So in
order to make emoji characters more personalized and creative, we use
computer visions and deep learning to detect human emotions from images
and create original emojis from them. In this deep learning project we will
classify human facial expressions to filter and map corresponding emojis or
avatars.

1. What is Deep Learning?

Deep Learning is a subset of Machine Learning, which on the other hand is a
subset of Artificial Intelligence. Artificial Intelligence is a general term that refers
to techniques that enable computers to mimic human behavior. Machine
Learning represents a set of algorithms trained on data that make all of this
possible.

7|Page
2. AI. vs ML. vs DL.

Deep Learning, on the other hand, is just a type of Machine Learning, inspired by
the structure of a human brain. Deep learning algorithms attempt to draw
similar conclusions as humans would by continually analyzing data with a given
logical structure. To achieve this, deep learning uses a multi-layered structure of
algorithms called neural networks.

A typical Neural Network

The design of the neural network is based on the structure of the human brain.
Just as we use our brains to identify patterns and classify different types of
information, neural networks can be taught to perform the same tasks on data.

The individual layers of neural networks can also be thought of as a sort of filter
that works from gross to subtle, increasing the likelihood of detecting and
outputting a correct result.
The human brain works similarly. Whenever we receive new information, the
brain tries to compare it with known objects. The same concept is also used by
deep neural networks.
Neural networks enable us to perform many tasks, such as clustering,
classification or regression. With neural networks, we can group or sort

8|Page
unlabeled data according to similarities among the samples in this data. Or in the
case of classification, we can train the network on a labeled dataset in order to
classify the samples in this dataset into different categories.

Artificial neural networks have unique capabilities that enable deep learning
models to solve tasks that machine learning models can never solve.
We can even go so far as to say that today a new industrial revolution is taking
place, driven by artificial neural networks and deep learning.
At the end of the day, deep learning is the best and most obvious approach to
real machine intelligence we’ve had so far.

Objective

Today, most common way of communication among people is virtual platforms,

whether using internet or phones. The current generation uses online
applications and platforms to communicate and exchange conversations.
However, communicating emotions is difficult. As a result, small and simple
pictures, otherwise known as emoji characters, are used to supplement
emotions when written language. Emoji characters are becoming more and
more popular therefore the variation of these characters has increased. This
project aims to use facial recognition and deep learning algorithm to extract
face from the picture and smooth it to create its avatar.It aims to create an
emoji of a specific, user-defined object within a picture.

9|Page
 Chapter 2: Literature Review

Google colab:

To be precise, Colab is a free Jupyter notebook environment that runs entirely

in the cloud. Most importantly, it does not require a setup and the notebooks
that you create can be simultaneously edited by your team members - just the
way you edit documents in Google Docs. Colab supports many popular
machine learning libraries which can be easily loaded in your notebook.

What Colab Offers You?

As a programmer, you can perform the following using Google Colab.

• Write and execute code in Python
• Document your code that supports mathematical equations
• Create/Upload/Share notebooks
• Import/Save notebooks from/to Google Drive
• Import/Publish notebooks from GitHub
• Import external datasets e.g. from Kaggle
• Integrate PyTorch, TensorFlow, Keras, OpenCV
• Free Cloud service with free GPU

Kaggle:

Kaggle, a subsidiary of Google LLC, is an online community of data

scientists and machine learning practitioners. Kaggle allows users to find and
publish data sets, explore and build models in a web-based data-science
environment, work with other data scientists and machine learning engineers,
and enter competitions to solve data science challenges.

10 | P a g e
OpenCV:

Python is an interpreted, object-oriented, high-level programming language

with dynamic semantics. Its high-level built in data structures, combined with
dynamic typing and dynamic binding, make it very attractive for Rapid
Application Development, as well as for use as a scripting or glue language to
connect existing components together. Python's simple, easy to learn syntax
emphasizes readability and therefore reduces the cost of program
maintenance. Python supports modules and packages, which encourages
program modularity and code reuse. The Python interpreter and the extensive
standard library are available in source or binary form without charge for all
major platforms, and can be freely distributed.

11 | P a g e
 Chapter 3: Problem Statement

Today, most common way of communication among people is virtual platforms,

whether using internet or phones. The current generation uses online
applications and platforms to communicate and exchange conversations.
However, communicating emotions is difficult. As a result, small and simple
pictures, otherwise known as emoji characters, are used to supplement
emotions when written language. Emoji characters are becoming more and
more popular therefore the variation of these characters has increased. This
project aims to use facial recognition and deep learning algorithm to extract
face from the picture and smooth it to create its avatar.It aims to create an
emoji of a specific, user-defined object within a picture.

12 | P a g e
 Chapter 4: Proposed Work/

Methodology

Overview:
In this deep learning project, we will classify human facial expressions to filter
and map corresponding emojis or avatars.With advancements in computer
vision and deep learning, it is now possible to detect human emotions from
images. This emojification will be done using ML and Keras. In this project it is
assumed that the background of the facial image is not complex and there is
only a single face on it. For training the machine we will be using FER2013
dataset (facial expression recognition) which consists of 48*48 pixel grayscale
face images. This dataset consist of facial emotions of following categories:

• 0:angry
• 1:disgust
• 2:fear
• 3:happy
• 4:sad
• 5:surprise
• 6:natural

Further each image is assumed in a same size, the image quality and resolution
is assumed to be sufficient enough, the illumination is uniformed and the input
images are colour images.

We will build a deep learning model to classify facial expressions from the
images. Then we will map the classified emotion to an emoji or an avatar.

13 | P a g e
 -Facial expressions recognition general CNN architecture

14 | P a g e
 Complete Block Diagram

15 | P a g e
 Use Case Diagram

16 | P a g e
 Block Diagram for Emotion Detection

17 | P a g e
Chapter 5:Design And Development

HARDWARE REQUIREMENTS:

As a user/developer all you need is a stable internet connection.

SOFTWARE REQUIREMENTS:

Google Colab:

To be precise, Colab is a free Jupyter notebook environment that runs entirely

in the cloud. Most importantly, it does not require a setup and the notebooks
that you create can be simultaneously edited by your team members - just the
way you edit documents in Google Docs. Colab supports many popular
machine learning libraries which can be easily loaded in your notebook.

What Colab Offers You?

As a programmer, you can perform the following using Google Colab.

• Write and execute code in Python
• Document your code that supports mathematical equations
• Create/Upload/Share notebooks
• Import/Save notebooks from/to Google Drive
• Import/Publish notebooks from GitHub
• Import external datasets e.g. from Kaggle
• Integrate PyTorch, TensorFlow, Keras, OpenCV
• Free Cloud service with free GPU

18 | P a g e
Colab is used extensively in the machine learning community with applications
including:

- Getting started with TensorFlow

- Developing and training neural networks

- Experimenting with TPUs

- Disseminating AI research

- Creating tutorials

19 | P a g e
Technologies Specifications:

Kaggle

Kaggle, a subsidiary of Google LLC, is an online community of data

scientists and machine learning practitioners. Kaggle allows users to find and
publish data sets, explore and build models in a web-based data-science
environment, work with other data scientists and machine learning engineers,
and enter competitions to solve data science challenges.

Python

Python is an interpreted, object-oriented, high-level programming language

with dynamic semantics. Its high-level built in data structures, combined with
dynamic typing and dynamic binding, make it very attractive for Rapid
Application Development, as well as for use as a scripting or glue language to
connect existing components together. The Python interpreter and the
extensive standard library are available in source or binary form without charge
for all major platforms, and can be freely distributed.

OpenCV

Python is an interpreted, object-oriented, high-level programming language

with dynamic semantics. Its high-level built in data structures, combined with
dynamic typing and dynamic binding, make it very attractive for Rapid
Application Development, as well as for use as a scripting or glue language to
connect existing components together.

20 | P a g e
Hardware Specifications:

Computer:

Most deep learning researchers and practitioners have access to a computer

with a fair amount of memory, computation, some form of an accelerator such
as a GPU, or multiples thereof

Memory

At its most basic memory is used to store data that needs to be readily
accessible. At present CPU RAM is typically of the [DDR4] variety, offering 20--
25 GB/s bandwidth per module. Each module has a 64-bit-wide bus. Typically
pairs of memory modules are used to allow for multiple channels. CPUs have
between 2 and 4 memory channels, i.e., they have between 4 0GB/s and 100
GB/s peak memory bandwidth. Often there are two banks per channel. For
instance AMD's Zen 3 Threadripper has 8 slots.

Cloud Storage

Cloud storage provides a configurable range of performance. That is, the

assignment of storage to virtual machines is dynamic, both in terms of quantity
and in terms of speed, as chosen by users. We recommend that users increase
the provisioned number of IOPs whenever latency is too high, e.g., during
training with many small records.

21 | P a g e
 Chapter 6:Implementation

CODE:
# -*- coding: utf-8 -*-
"""Copy of Animate-final-draft1

Automatically generated by Colaboratory.

Original file is located at

https://colab.research.google.com/drive/1EDNEBzzExmwKuOjogYguvdycPxZTLw8J
"""

from google.colabimport drive

drive.mount('/content/drive')

""" **Checked if we have the required libraries**"""

"""## ***MODEL DEPENDENCIES***"""

pip install numpy

pip install keras

pip install opencv-python

"""***HARCASCADE DEPENDENCIES AND FUNCTIONS***

"""

# using haarcascade face detection

!wget --no-check-certificate \
https://raw.githubusercontent.com/computationalcore/introduction-to-
opencv/master/assets/haarcascade_frontalface_default.xml \
-O haarcascade_frontalface_default.xml

# important dependencies
from IPython.displayimport display, Javascript, Image
from google.colab.outputimport eval_js
from google.colab.patchesimport cv2_imshow
from base64 import b64decode, b64encode
import cv2
import numpyas np
import io
import html
import time
import matplotlib.pyplotas plt
from PIL import Image
from pathlibimport Path

22 | P a g e
"""**OPEN CV FUNCTIONS**

**FUNCTION 1- Convert JAVASCRIPT OBJECT INTO OPENCV IMAGE**

"""

# function to convert javascript object into Opencv image

defjs_to_image(js_reply):
"""
Params:
js_reply: JavaScript object containing image from webcam
Returns:
img: OpenCV BGR image
"""
# decode base64 image
image_bytes = b64decode(js_reply.split(',')[1])
# convert bytes to numpy array
jpg_as_np = np.frombuffer(image_bytes, dtype=np.uint8)
# decode numpy array into OpenCV BGR image
img = cv2.imdecode(jpg_as_np, flags=1)

return img

"""**FUNCTION -2 UTILIZE COMPUTER WEBCAM**"""

# snippet from camera capture to utilize computer's webcam

deftake_photo(filename='photo.jpg', quality=0.8):
js = Javascript('''
async function takePhoto(quality) {
const div = document.createElement('div');
const capture = document.createElement('button');
capture.textContent = 'Capture';
div.appendChild(capture);

const video = document.createElement('video');

video.style.display = 'block';
const stream = await navigator.mediaDevices.getUserMedia({video: true});

document.body.appendChild(div);
div.appendChild(video);
video.srcObject = stream;
await video.play();

// Resize the output to fit the video element.

google.colab.output.setIframeHeight(document.documentElement.scrollHeight, true);

// Wait for Capture to be clicked.

await new Promise((resolve) =>capture.onclick = resolve);

const canvas = document.createElement('canvas');

canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
canvas.getContext('2d').drawImage(video, 0, 0);
stream.getVideoTracks()[0].stop();
div.remove();
return canvas.toDataURL('image/jpeg', quality);
}
''')
display(js)

23 | P a g e
 data = eval_js('takePhoto({})'.format(quality))
binary = b64decode(data.split(',')[1])
with open(filename, 'wb') as f:
f.write(binary)
return filename

"""**FUNCTION 3 - TO CREATE LIVE VIDEO USING WEB CAM**

"""

# javascript to create live video using webcam video as input

defvideo_stream():
js = Javascript('''
var video;
var div = null;
var stream;
varcaptureCanvas;
varimgElement;
varlabelElement;

varpendingResolve = null;
var shutdown = false;

function removeDom() {
stream.getVideoTracks()[0].stop();
video.remove();
div.remove();
video = null;
div = null;
stream = null;
imgElement = null;
captureCanvas = null;
labelElement = null;
}

function onAnimationFrame() {
if (!shutdown) {
window.requestAnimationFrame(onAnimationFrame);
}
if (pendingResolve) {
var result = "";
if (!shutdown) {
captureCanvas.getContext('2d').drawImage(video, 0, 0, 640, 480);
result = captureCanvas.toDataURL('image/jpeg', 0.8)
}
varlp = pendingResolve;
pendingResolve = null;
lp(result);
}
}

async function createDom() {

if (div !== null) {
return stream;
}

div = document.createElement('div');
div.style.border = '2px solid black';

24 | P a g e
div.style.padding = '3px';
div.style.width = '100%';
div.style.maxWidth = '600px';
document.body.appendChild(div);

constmodelOut = document.createElement('div');
modelOut.innerHTML = "<span>Status:</span>";
labelElement = document.createElement('span');
labelElement.innerText = 'No data';
labelElement.style.fontWeight = 'bold';
modelOut.appendChild(labelElement);
div.appendChild(modelOut);

video = document.createElement('video');
video.style.display = 'block';
video.width = div.clientWidth - 6;
video.setAttribute('playsinline', '');
video.onclick = () => { shutdown = true; };
stream = await navigator.mediaDevices.getUserMedia(
{video: { facingMode: "environment"}});
div.appendChild(video);

imgElement = document.createElement('img');
imgElement.style.position = 'absolute';
imgElement.style.zIndex = 1;
imgElement.onclick = () => { shutdown = true; };
div.appendChild(imgElement);

const instruction = document.createElement('div');

instruction.innerHTML =
'<span style="color: red; font-weight: bold;">' +
'When finished, click here or on the video to stop this demo</span>';
div.appendChild(instruction);
instruction.onclick = () => { shutdown = true; };

video.srcObject = stream;
await video.play();

captureCanvas = document.createElement('canvas');
captureCanvas.width = 640; //video.videoWidth;
captureCanvas.height = 480; //video.videoHeight;
window.requestAnimationFrame(onAnimationFrame);

return stream;
}
async function stream_frame(label, imgData) {
if (shutdown) {
removeDom();
shutdown = false;
return '';
}

varpreCreate = Date.now();
stream = await createDom();

varpreShow = Date.now();
if (label != "") {
labelElement.innerHTML = label;
}

25 | P a g e
 if (imgData != "") {
varvideoRect = video.getClientRects()[0];
imgElement.style.top = videoRect.top + "px";
imgElement.style.left = videoRect.left + "px";
imgElement.style.width = videoRect.width + "px";
imgElement.style.height = videoRect.height + "px";
imgElement.src = imgData;
}

varpreCapture = Date.now();
var result = await new Promise(function(resolve, reject) {
pendingResolve = resolve;
});
shutdown = false;

return {'create': preShow - preCreate,

'show': preCapture - preShow,
'capture': Date.now() - preCapture,
'img': result};
}
''')

display(js)

defvideo_frame(label, bbox):
data = eval_js('stream_frame("{}", "{}")'.format(label, bbox))
return data

"""**Kaggle dataset**

INSTALLING KAGGEL LIBRARY

"""

from google.colabimport files

# Install Kaggle library

!pip install -q kaggle

"""UPLOADING KAGGLE API KEY FILE"""

# Upload kaggle API key file

uploaded = files.upload()

!pip install kaggle -q # At first, I suspect the kaggle API lose effect so it
doesn't have .kaggle folder. (not working)
!rm -rf /root/.kaggle# when I created the folder, it says the file or dir already
exits
!mkdir /root/.kaggle# successful
!mv kaggle.json /root/.kaggle/kaggle.json# not sure if I have to use full
destination path, I previously only used /root/.kaggle and it failed. Don't have
time to validate this thought.
!ls /root/.kaggle/kaggle.json

"""**DOWNLOADING KAGGLE DATASET**"""

!kaggle datasets download -d msambare/fer2013

!unzip \*.zip

26 | P a g e
"""TO CHECK GPU ALLOCATION"""

# Commented out IPython magic to ensure Python compatibility.

# %tensorflow_version 2.x
import tensorflowas tf
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))

"""**DEPENDENCIES FOR CNN MODEL**"""

import numpyas np
import cv2
from keras.modelsimport Sequential
from keras.layersimport Dense, Dropout, Flatten
from keras.layersimport Conv2D
from keras.optimizersimport Adam
from keras.layersimport MaxPooling2D
from keras.preprocessing.imageimport ImageDataGenerator

"""***PARTITIONING DATA INTO 2 SUB DATASETS ***

> 1. TRAIN DATA SET

>2. VALIDATION DATA SET

"""

train_dir = '/content/train'
val_dir = '/content/test'
train_datagen = ImageDataGenerator(rescale=1./255)
val_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
train_dir,
target_size=(48,48),
batch_size=64,
color_mode="grayscale",
class_mode='categorical')
validation_generator = val_datagen.flow_from_directory(
val_dir,
target_size=(48,48),
batch_size=64,
color_mode="grayscale",
class_mode='categorical')

"""**CNN MODEL**"""

emotion_model = Sequential()
emotion_model.add(Conv2D(32, kernel_size=(3, 3), activation='relu',
input_shape=(48,48,1)))
emotion_model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
emotion_model.add(Dropout(0.25))

27 | P a g e
emotion_model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
emotion_model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
emotion_model.add(Dropout(0.25))
emotion_model.add(Flatten())
emotion_model.add(Dense(1024, activation='relu'))
emotion_model.add(Dropout(0.5))
emotion_model.add(Dense(7, activation='softmax'))

"""**COMPILING AND TRAINING THE MODEL**"""

emotion_model.compile(loss='categorical_crossentropy',optimizer=Adam(lr=0.0001,
decay=1e-6),metrics=['accuracy'])
emotion_model_info = emotion_model.fit_generator(
train_generator,
steps_per_epoch=28709 // 64,
epochs=50,
validation_data=validation_generator,
validation_steps=7178 // 64)

"""**SAVING THE WEIGHTS IN .h5 FILE**"""

emotion_model.save_weights('model.h5')

"""***HTML CODE FOR FINAL OUTPUT***"""

import IPython
emotion_dict = {0: "Angry", 1: "Disgusted", 2: "Fearful", 3: "Happy", 4:
"Neutral", 5: "Sad", 6: "Surprised"}

angry = \
'''
<h1 >Angry</h1>
<imgsrc ='https://imgur.com/jTYPnQ8.jpg' width="400" height="300"</img>
'''
happy = \
'''
<h1>HAPPY</h1>
<imgsrc ='https://imgur.com/GWKTdeB.jpg' width="400" height="300"</img>
'''
sad = \
'''
<h1>SAD</h1>
<imgsrc ='https://imgur.com/eUBW24K.jpg' width="400" height="300"</img>
'''
disgusted = \
'''
<h1>DISGUSTED</h1>
<imgsrc ='https://imgur.com/dlWMeTk.jpg' width="400" height="300"</img>
'''
fearful = \
'''
<h1>FEARFUL</h1>
<imgsrc ='https://imgur.com/bqu0aXB.jpg' width="400" height="300"</img>
'''
neutral = \
'''
<h1>NEUTRAL</h1>

28 | P a g e
<imgsrc ='https://imgur.com/gA4fHh7.jpg' width="400" height="300"</img>
'''
surprised = \
'''
<h1>SURPRISED</h1>
<imgsrc ='https://imgur.com/AQnE6NS.jpg' width="400" height="300"</img>
'''

emotion_dic_pics =
{0:angry,1:happy,2:sad,3:disgusted,4:fearful,5:neutral,6:surprised}

"""**TAKING INPUT FROM USER AND SVAING IT AS .img**

"""

cv2.ocl.setUseOpenCL(False)

emotion_dict = {0: "Angry", 1: "Disgusted", 2: "Fearful", 3: "Happy", 4:

"Neutral", 5: "Sad", 6: "Surprised"}

#cap = cv2.VideoCapture(0)

image_file=take_photo()
img=cv2.imread(image_file)
bbox_array = np.zeros([480,640,4], dtype=np.uint8)
bounding_box = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
gray_frame = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
num_faces = bounding_box.detectMultiScale(gray_frame,scaleFactor=1.3,
minNeighbors=5)

#print(num_faces)
for (x, y, w, h) in num_faces:
bbox_array=cv2.rectangle(img, (x, y-50), (x+w, y+h+10), (255, 0, 0), 2)
roi_gray_frame = gray_frame[y:y + h, x:x + w]
cropped_img = np.expand_dims(np.expand_dims(cv2.resize(roi_gray_frame, (48, 48)),
-1), 0)
#emotion_prediction = emotion_model.predict(cropped_img)
#maxindex = int(np.argmax(emotion_prediction))
#cv2.putText(frame, emotion_dict[maxindex], (x+20, y-60),
cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2, cv2.LINE_AA)

cv2_imshow(img)
cv2.imwrite('/content/sample_data/faces/pic.png',img)
if cv2.waitKey(1) &0xFF == ord('q'):
cap.release()
cv2.destroyAllWindows()
break

"""**GIVING THIS SAVED IMAGE AS INPUT TO EMOTION PREDICTION MODEL**"""

emotion_dict = {0: "Angry", 1: "Disgusted", 2: "Fearful", 3: "Happy", 4:

"Neutral", 5: "Sad", 6: "Surprised"}
emotion_model.load_weights('model.h5')
emotion_prediction = emotion_model1.predict(cropped_img)
maxindex = int(np.argmax(emotion_prediction))
#emotion_dict[maxindex]

defread_file(filename):
img = cv2.imread(filename)

29 | P a g e
 cv2_imshow(img)
return img

defcolor_quantization(img, k):
# Transform the image
data = np.float32(img).reshape((-1, 3))

# Determine criteria
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 20, 0.001)

# Implementing K-Means
ret, label, center = cv2.kmeans(data, k, None, criteria, 10,
cv2.KMEANS_RANDOM_CENTERS)
center = np.uint8(center)
result = center[label.flatten()]
result = result.reshape(img.shape)
return result

defedge_mask(img, line_size, blur_value):

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
gray_blur = cv2.medianBlur(gray, blur_value)
edges = cv2.adaptiveThreshold(gray_blur, 255, cv2.ADAPTIVE_THRESH_MEAN_C,
cv2.THRESH_BINARY, line_size, blur_value)
return edges

img = read_file('/content/photo.jpg')
#img = cropped_img
#line_size = 7
#blur_value = 7

line_size = 9
blur_value = 5

#edges = edge_mask(img, line_size, blur_value)

edges = edge_mask(img, line_size, blur_value)
#cv2_imshow(edges)
#total_color = 9
total_color = 11
img = color_quantization(img, total_color)
#cv2_imshow(img)
blurred = cv2.bilateralFilter(img, d=7, sigmaColor=200,sigmaSpace=200)
#cv2_imshow(blurred)
cartoon = cv2.bitwise_and(blurred, blurred, mask=edges)
#cv2_imshow(cartoon)

cv2_imshow(cartoon)
print("Emotion depicted: ")
emotion_dict[maxindex]
display(IPython.display.HTML(emotion_dic_pics[maxindex]))

30 | P a g e
RESULT IMAGES:

31 | P a g e
32 | P a g e
33 | P a g e
Chapter 7: Testing

Goal of DL Testing:
What is the goal of DL testing?

First of all, what are we trying to achieve when performing ML testing, as well as
any software testing whatsoever?

Quality assurance is required to make sure that the software system works
according to the requirements. Were all the features implemented as agreed?
Does the program behave as expected? All the parameters that you test the
program against should be stated in the technical specification document.

Moreover, software testing has the power to point out all the defects and flaws
during development. You don’t want your clients to encounter bugs after the
software is released and come to you waving their fists. Different kinds of
testing allow us to catch bugs that are visible only during runtime.

However, in machine learning, a programmer usually inputs the data and the
desired behavior, and the logic is elaborated by the machine. This is especially
true for deep learning. Therefore, the purpose of machine learning testing is,
first of all, to ensure that this learned logic will remain consistent, no matter
how many times we call the program.

Model Evaluation:
Usually, software testing includes:

Unit tests. The program is broken down into blocks, and each element (unit) is
tested separately.

34 | P a g e
Regression tests. They cover already tested software to see if it doesn’t
suddenly break.

Integration tests. This type of testing observes how multiple components of the
program work together.

Moreover, there are certain rules that people follow: don’t merge the code
before it passes all the tests, always test newly introduced blocks of code, when
fixing bugs, write a test that captures the bug.

Machine learning adds up more actions to your to-do list. You still need to
follow DL’s best practices. Moreover, every DL model needs not only to be
tested but evaluated. Your model should generalize well. This is not what we
usually understand by testing, but evaluation is needed to make sure that the
performance is satisfactory

First of all, you split the database into three non-overlapping sets. You use a
training set to train the model. Then, to evaluate the performance of the model,
you use two sets of data:

Validation set. Having only a training set and a testing set is not enough if you
do many rounds of hyperparameter-tuning (which is always). And that can
result in overfitting. To avoid that, you can select a small validation data set to
evaluate a model. Only after you get maximum accuracy on the validation set,
you make the testing set come into the game.

Test set (or holdout set). Your model might fit the training dataset perfectly
well. But where are the guarantees that it will do equally well in real-life? In
order to assure that, you select samples for a testing set from your training set
— examples that the machine hasn’t seen before. It is important to remain
unbiased during selection and draw samples at random. Also, you should not
use the same set many times to avoid training on your test data. Your test set
should be large enough to provide statistically meaningful results and be
representative of the data set as a whole.

35 | P a g e
Evaluation using metrices:
Evaluating the performance of the model using different metrics is integral to
every data science project. Here is what you have to keep an eye on:

Accuracy

Accuracy is a metric for how much of the predictions the model makes are true.
The higher the accuracy is, the better. However, it is not the only important
metric when you estimate the performance.

Loss

Loss describes the percentage of bad predictions. If the model’s prediction is

perfect, the loss is zero; otherwise, the loss is greater.

Precision

The precision metric marks how often the model is correct when identifying
positive results. For example, how often the model diagnoses cancer to patients
who really have cancer.

Recall

This metric measures the number of correct predictions, divided by the number
of results that should have been predicted correctly. It refers to the percentage
of total relevant results correctly classified by your algorithm.

Confusion matrix

A confusion matrix is an N\times NN×N square table, where NN is the number

of classes that the model needs to classify. Usually, this method is applied to
classification where each column represents a label. For example, if you need to
categorize fruits into three categories: oranges, apples, and bananas, you draw
a 3\times33×3 table. One axis will be the actual label, and the other will be the
predicted one.

36 | P a g e
Conclusion:
Performing DL tests is necessary if you care about the quality of the model. ML
testing has a couple of peculiarities: it demands that you test the quality of
data, not just the model, and go through a couple of iterations adjusting the
hyperparameters to get the best results. However, if you perform all the
necessary procedures, you can be sure of its performance.

37 | P a g e
 Chapter8: Conclusion and Future work

Conclusion
Deep learning is not only the future but it also the present of solvingcomplex
computing. In this project we will build a deep learning model to classify facial
expressions from the images. Then we will map the classified emotion to an
emoji or an avatar in the real time. The main objective of this project is to
understand deep learning and start our journey towards Artificial Intelligence.
We will build a convolution neural network to recognize facial emotions. We will
train our model on the FER2013 dataset. Then we will map these emotions with
the corresponding emojis or avatars. This project will also give us opportunity to
use OpenCV’shaar cascade xml.

Future Work
The emoji is here to stay and will become even more powerful. Apple, for
instance, has put more emphasis on them by making them bigger in iOS 10. The
Unicode Consortium recently introduced the Unicode 9 standard, which adds
72 emoji characters to a total of over 128,172.

• Emoji can also be applied in the area of password security.

• Due to their visual and emotional attributes, emoji can be used in
marketing activities.
• In the field of linguistics, research focuses on the pragmatic functions of
emoji and the possibility that they could become an independent
language.

38 | P a g e
 Chapter 8: Reference /Bibliography

www.towardsdatascience.com

http://www.freecodecamp.org/

https://machinelearningmastery.com/what-is-deep-learning/

https://towardsdatascience.com/what-is-deep-learning-and-how-does-it-work-
2ce44bb692ac

https://www.kdnuggets.com/

39 | P a g e
 Plagiarism Report
By check-plagiarism.com
Report Title: pl_report

Report Link: (Use this link to send report to anyone) https://www.check-

plagiarism.com/plag-report/3402960217a74402de5754b41a859f6 5b27c21625148658

Report Generated Date: 01 July, 2021

Total Words: 2006

Total Characters: 12123

Keywords/Total Words Ratio: 0%

Excluded URL: No

Unique: 89%

Matched: 11%

Sentence wise detail:

Chapter 1: Introduction Emoji characters are becoming more and more popular in
communicating with emotions and text messages. Emoji characters are limited to predefined
characters. These characters are not real and customized. So to make emoji characters
personal and creative, we use computer ideas and in-depth learning to find peoples feelings
in pictures and create real emojis from them. In this in-depth learning project we will
separate the look of a person's face to filter and count emoji or related avatars. Deep
Learning Deep learning is a piece of learning equipment, on the other hand the essence of
Artificial Intelligence. Artificial Intelligence is a common term referring to computer-enabled
techniques that mimic human behavior. Learning Machines represent a set of data-trained
algorithms that make all of this possible. 2. AI. vs ML. vs DL. Deep learning, on the other
hand, is merely a form of machine learning, promoted by the human brain. In deep learning
algorithms attempt to draw conclusions similar to those of humans by constantly analyzing
data with a logical structure provided. To achieve this, in-depth learning uses the structure of
multi-sectoral algorithms called neural networks. Standard Neural Network The design of the
neural network is based on the structure of the human brain. Just as we use our brain to
identify patterns and separate different types of information, neural networks can be taught
to perform similar functions in data. Individual layers of neural networks can also be thought
of as a type of filter that operates from the worst to the most hidden, increasing the chances
of finding and producing the right result. (0) The human mind works in the same way.
Whenever we receive new information, the brain tries to compare it with what is known. The
same concept is used for deep neural networks. Neural networks enable us to perform many
tasks, such as merging, dividing, or retrieving. With neural networks, we can collect or
organize non-labeled data according to the similarities between samples in this data. Or in
the case of fragmentation, we can train the network in a labeled database to divide samples

40 | P a g e
of this database into different categories. Neural networks made with different capabilities
enable deeper learning models to solve tasks that machine learning tools cannot solve. We
can also say that today a new industrial revolution is taking place, driven by artificial neural
networks and in-depth learning. At the end of the day, in-depth study is the best and clearest
way of real-world intelligence that we have had so far. Purpose Today, the most common
form of communication between people is through virtual platforms, whether they use the
Internet or telephones. The current generation uses online applications and platforms to
communicate and exchange conversations. (1) However, emotional communication is
difficult. As a result, smaller and simpler images, otherwise known as emoji characters, are
used to add emotion to language writing. Emoji characters are becoming more and more
popular so the diversity of these characters has increased.This project aims to use face
recognition and in-depth learning algorithm to remove a face from an image and smooth it to
create its own image. It aims to create an emoji of something, defined by the user within the
image. Chapter 2: Literature Review Google colab: To be clear, Colab is Jupyters free
writing platform that works well in the cloud. Most importantly, it doesn't need to be set up
and the textbooks you edit can be edited simultaneously by your team members - the way
you edit documents in Google Docs. Colab supports many popular libraries that can be
easily downloaded from your notebook. Why is Colab good for us? As an engineer, you can
do the following using Google Colab. • Write and use code with Python • Write your code
that Technical Information: Kaggle Kaggle, a subsidiary of Google LLC, is an online
community of data scientists and (2) machine-learning scientists. Kaggle allows users to
discover and publish data sets, test and build models in a data-based environment, work
with other data scientists and machine learning engineers, and enter contests to solve data
science challenges. (3) Python Python is a translated, object-oriented, high-quality language
with dynamic semantics. Advanced built-in data architecture, combined with powerful typing
and dynamic binding, makes it extremely attractive in Quick Application Development, as
well as being used as a scripting or paste to connect existing components together. (4) The
Python interpreter and the general library are available in the source or in the form of a free
binary on all major platforms, and can be distributed freely. OpenCV OpenCV is a large
library of open source computer software, machine learning, and image processing and now
plays a major role in real-time performance that is critical to today's programs. By using it,
one can process photos and videos to identify objects, faces, or handwriting. (5) When
combined with various libraries, such as Numpy, python is able to process OpenCV analysis
structure. To identify a picture pattern and its various features we use a vector space and
perform mathematical functions on these elements. (5) Hardware details: Computer: Most
advanced researchers have access to a computer with the right amount of memory,
calculations, a specific type of accelerator such as a GPU, or a duplicate of it. (7) Memory Its
basic memory is used to store data that needs to be easily accessible. Currently CPU RAM
is usually different from [DDR4], which provides 20-25 GB / s bandwidth per module. Each
module has a 64-bit-wide bus. (8) Memory modules are often used to allow multiple
channels. CPUs have between 2 and 4 memory channels, that is, they have between 4 0GB
/ s and 100 GB / s memory bandwidth. There are usually two banks per station. For example
the Zen 3 Threadripper for AMD has 8 spaces. Cloud Storage Cloud storage provides a
flexible range of performance. That is, the provision of storage on virtual machines is
powerful, both in quantity and speed, as selected by users. We recommend that users
increase the provided number of IOPs whenever latency is very high, e. g. , during training
with multiple small records. Chapter 7: Testing DL Test Policy: What is the goal of DL

41 | P a g e
testing? First, what are we trying to achieve when we do ML tests, as well as other software
tests? Quality assurance is required to ensure that the software system operates according
to requirements. Are all the features done as agreed? Does the system behave as
expected? (9) All parameters you are testing the system should be specified in the specified
technical document. In addition, software testing has the ability to identify all defects and
errors during development. You do not want your clients to come in contact with the bugs
after the software has been released and come to you punching. (10) A variety of tests allow
us to catch bugs that are only visible during operation. However, in machine learning, the
designer himself enters the required data and behavior, and the concept is defined bythe
machine. This is especially true for in-depth reading. Therefore, the purpose of machine
learning tests, first of all, is to ensure that the learning concept will remain consistent, no
matter how often we call the system. Model Testing: Typically, software tests include: Unit
test. The system is divided into blocks, and each item (unit) is tested separately. Recurrence
Test. They install software that has already been tested to see if it does not break suddenly.
Integration testing. (11) This type of test looks at how many parts of the system work
together. In addition, there are certain rules that people follow: do not compile code before
passing all tests, always test newly introduced code blocks, when repairing bugs, write an
insect repellent test. Machine learning adds other actions to your to-do list. You still need to
follow good DL practices. In addition, every DL model requires not only testing but also
testing. Your model should fit well. This is not what we usually understand by testing, but
testing is necessary to ensure that performance is satisfactory First, you divide the database
into three non-overlapping sets. (12) He uses a training set to train the model. After that, to
test the performance of the model, using two sets of data: Authentication is set. Having only
a training set and a test set is not enough if you do multiple parameters to redeem the
(permanent) parameters. And that can lead to overeating. (13) To avoid that, you can
choose a small data set to test the model. After gaining the highest accuracy on the
verification set, he performs a test set in the game. Test set (or hold set). Your model may fit
well with a good training database. But where are the guarantees that it will do equally well
in real life? (14) To confirm that, you select test set samples from your training set -
examples the machine has never seen before. It is important to remain neutral during the
selection and drawing of random samples. Also, you should not use the same set many
times to avoid training in your test data. Your test set should be large enough to provide
statistical results and represent complete data setup. Check using metrics: Model
performance testing using different metrics is integrated into the entire data science project.
Heres what you need to remember: Accuracy Accuracy is a metaphor for how much a model
makes predictions. The higher the accuracy, the better. (15) However, it is not the only
important metrics when measuring performance. Loss Loss means a percentage of bad
predictions. If the model prediction is complete, the loss is zero; otherwise, the loss is huge.
Accuracy Matric accuracy indicates how often the model is correct when positive results are
identified. For example, the model often detects cancer in really cancer patients. Remember
This metric measures the number of relevant predictions, divided by the number of results
that should have been performed earlier The matrix of confusion The confusion matrix is a
square table of N times NN × N times, where NN is the number of classes the models need
to divide. (15) Typically, this method is used in the section where each column represents
the label. For example, if you need to divide fruit into three categories: oranges, apples, and
bananas, you draw a table 3 \ times33 × 3. One axis will be a real label, and the other will be
what is predicted. Conclusion: Performing a DL test is necessary if you care about the

42 | P a g e
quality of the model. ML testing has a few features: it requires you to test the quality of the
data, not just the model, and then go a few times to adjust the hyperparameters to get the
best results. However, if you perform all the necessary procedures, you can be sure of its
effectiveness.Chapter 8: The End and the Work of the Future The conclusion In-depth
learning is not only the future but also the time to solve a complex computer. In this project
we will create an in-depth learning model to distinguish facial expressions in pictures. After
that we will map the emotions separated by emoji or avatar in real time. The main purpose of
this project is to understand in-depth learning and begin our journey towards Artificial
Intelligence. We will build a neural convolution network to get face to face. We will train our
model in the FER2013 database. After that we will map these feelings with the same emoji
or avatars. This project will also give us the opportunity to use OpenCVshaar cascade xml.
Work for the Future The emoji is here to stay and will be even more powerful. Apple, for
example, has prioritized upgrading them to iOS 10. The Unicode Consortium recently
introduced the Unicode 9 standard, adding another 72 characters to 128,172. • Emoji can
also be used in the password protection area. • Because of their visual and emotional
properties, emoji can be used in marketing activities. • In the field of languages, research
focuses on the ongoing activities of emoji and the possibility of becoming an independent
language.

Match Urls:

0: https://www.merriam-webster.com/dictionary/result

1: http://pages.cs.wisc.edu/~dyer/cs534/hw/hw5/Project-Reports2015/farrokhebrah-HW5-
Report.pdf

2: https://en.wikipedia.org/wiki/Kaggle

3: https://www.merriam-webster.com/thesaurus/challenges

4: https://www.merriam-webster.com/dictionary/together

5: https://irjmets.com/rootaccess/forms/uploads/IRJMETS265099.pdf

6: https://www.merriam-webster.com/dictionary/it

7: https://d2l.ai/chapter_computational-performance/hardware.html

8: https://resources.sei.cmu.edu/asset_files/TechnicalNote/2010_004_001_15194.pdf

9: https://twitter.com/comeoutpunching/status/1331282322692468737

10: https://www.testingxperts.com/blog/what-is-integration-testing

11: https://byjus.com/maths/sets/

12: https://www.mdanderson.org/publications/focused-on-health/What-happens-when-you-
overeat.h23Z1592202.html

13: https://betterprogramming.pub/testing-machine-learning-models-434e973aa14d

43 | P a g e
14: https://www.statology.org/regression-vs-classification/

15: https://www.merriam-webster.com/dictionary/divide

44 | P a g e