SMART MUSIC PLAYER INTEGRATING FACIAL EMOTION

RECOGNITION AND MUSIC MOOD RECOMMENDATION

CHAPTER 1

1.1 INTRODUCTION

In today's digital age, personalized recommendation systems have become

increasingly popular, catering to the diverse preferences and interests of
users. Emotion-based music recommendation systems represent a unique
approach to personalized recommendations, leveraging the emotional
impact of music to enhance user experience. Our project, the Emotion-
Based Music Recommender, aims to provide users with personalized song
recommendations tailored to their current emotional state.

Fig 1.1 Model Flow Chart

1
1.1.1 PROBLEM STATEMENT

Traditional music recommendation systems often rely on user listening

history, genre preferences, or collaborative filtering techniques. However,
these approaches may not capture the nuanced emotional responses that
music can evoke. Users may seek music that aligns with their current mood
or emotional needs, but existing recommendation systems may struggle to
provide relevant suggestions based on emotional context alone. Therefore,
there is a need for an emotion-based music recommendation system that can
accurately detect and respond to users' emotional states in real-time,
enhancing the overall listening experience.

1.1.2 OBJECTIVE

(1) Develop a real-time emotion detection system using facial landmarks and
hand gestures captured through a webcam feed.

(2) Implement a deep learning-based model for emotion recognition, trained

on labelled facial expression datasets.

(3) Integrate the emotion detection system with a web application using the
Stream lit framework, allowing users to interact with the system through a
user-friendly interface.

(4) Provide personalized song recommendations based on the user's detected

emotion, preferred language, and singer inputs.

(5) Evaluate the effectiveness and user satisfaction of the Emotion-Based

Music Recommender through user testing and feedback collection.

2
1.1.3 BACKGROUND AND MOTIVATION FOR THE PROJECT.

Traditional music recommendation systems often overlook the emotional

context of music consumption, focusing primarily on user preferences and
listening history. However, music has a profound impact on our emotions
and mood, influencing how we feel and experience the world around us.
Recognizing this, the Emotion-Based Music Recommender project aims to
address this gap by incorporating real-time emotion detection into the
recommendation process. By capturing users' emotional states through
facial expressions and hand gestures using a webcam feed, our system offers
personalized song recommendations that align with users' current emotional
needs and preferences. This innovative approach not only enhances the user
experience by providing more relevant and engaging recommendations but
also showcases the potential of technology to deepen our emotional
connection with music.

In summary, the Emotion-Based Music Recommender project is driven by

the recognition of the importance of emotional engagement in music
listening and the desire to create a more personalized and enriching music
discovery experience for users. Through our innovative approach, we aim to
revolutionize the way users interact with music, fostering deeper emotional
connections and enhancing overall satisfaction with music recommendation
systems.

3
Fig1.2 Facial Emotions

4
 CHAPTER 2

LITERATURE SURVEY

"A REVIEW ON MUSIC EMOTION RECOGNITION

TECHNIQUES" BY ABHISHEK KUMAR AND R. K. SHARMA:

This paper provides a comprehensive review of various techniques and

methodologies for music emotion recognition. It covers approaches such as
audio feature extraction, machine learning algorithms, and deep learning
models used for analysing and recognizing emotional content in music.

"DEEP FACIAL EXPRESSION RECOGNITION: A SURVEY" BY

ZHIWEI DENG, JIANI HU, AND JUN GUO:

This survey paper discusses the state-of-the-art techniques and

advancements in facial expression recognition using deep learning. It
explores various architectures, datasets, and evaluation metrics employed
in facial emotion recognition systems, which can inform the development
of the emotion detection module in our project.

"REAL-TIME EMOTION DETECTION WITH PYTHON" BY

DIVYANSHU SHEKHAR:

This blog post provides practical insights and code examples for
implementing real-time emotion detection using python, Open CV, and
deep learning models. It offers a step-by-step guide for capturing facial
expressions from webcam feeds and processing them to recognize
emotions in real-time, which is relevant to our project's emotion detection
module.

5
"BUILDING A REAL-TIME EMOTION RECOGNITION APP
WITH STREAM LIT AND TENSORFLOW.JS" BY MADHURIMA
DAS:

This tutorial demonstrates how to build a real-time emotion recognition

application using stream lit and tensorflow.js. It covers the process of
creating a web-based interface for capturing and analysing facial
expressions in real-time, which aligns with our project's goal of integrating
emotion detection with a user-friendly web application.

"DEEP LEARNING" BY IAN GOODFELLOW, YOSHUA BENGIO,

AND AARON COURVILLE:

This textbook offers a comprehensive overview of deep learning

techniques, including convolutional neural networks (CNNs) used in facial
emotion recognition. It covers topics such as image classification, object
detection, and natural language processing, providing foundational
knowledge relevant to our project's implementation of deep learning
models.

"HANDS-ON MACHINE LEARNING WITH SCIKIT-LEARN,

KERA’S, AND TENSORFLOW" BY AURÉLIEN GÉRON:

This book provides practical guidance and examples for building machine
learning models using popular libraries such as Scikit-Learn, Keras, and
Tensor Flow. It covers topics such as data preprocessing, model training,
and evaluation, which are essential for implementing the emotion
recognition model and recommendation system in our project.

6
"CONVOLUTIONAL NEURAL NETWORKS" BY ANDREW NG
(COURSERA COURSE):

This online course offers in-depth coverage of convolutional neural

networks (CNNs), which are widely used in image recognition tasks such
as facial expression recognition. It provides theoretical insights and
practical exercises for understanding CNN architectures, training
techniques, and applications in computer vision.

Fig 2.1 Flow Chart from face to emotion

7
 CHAPTER 3

METHODOLOGY

3.1 DESCRIPTION OF THE SYSTEM ARCHITECTURE AND

COMPONENTS:

 The emotion-based music recommender system architecture follows a

modular design, comprising three main components:

 Emotion detection module: this component is responsible for real-

time emotion detection from webcam feeds. It utilizes the mediapipe
library to detect facial landmarks and hand gestures, which are then
processed to infer the user's emotional state.

 Recommendation engine: the recommendation engine generates

personalized song recommendations based on the detected emotion,
preferred language, and singer inputs provided by the user. It queries
online platforms such as youtube to retrieve relevant music content.

 User interface: developed using streamlit, the user interface provides

an intuitive web-based platform for users to interact with the system.
It displays the webcam feed with overlays indicating detected facial
landmarks and hand gestures, along with options to input preferences
and trigger song recommendations.

8
 Fig 3.1 Flow Chart from face to emotion

3.2 OVERVIEW OF THE TECHNOLOGIES AND LIBRARIES USED:

 Streamlit: Streamlit is a python library used for building interactive

web applications with minimal code. It simplifies the development of
user interfaces and data visualization directly from python scripts.

 Mediapipe: developed by google, mediapipe is an open-source library

for building machine learning pipelines for perception tasks such as
facial recognition, hand tracking, and pose estimation. It provides pre-
trained models and tools for real-time inference on various platforms.

 Keras: keras is a high-level deep learning api that simplifies the

development and deployment of deep neural networks. In this project,
keras is used to load a pre-trained deep learning model for emotion
recognition from facial landmarks.

9
  Opencv: opencv (open source computer vision library) is a popular
open-source computer vision and machine learning software library. It
is used for image and video processing tasks such as webcam capture,
image manipulation, and feature extraction.

Fig 3.2 Landmarking Example

3.3 EXPLANATION OF THE EMOTION DETECTION AND

RECOGNITION ALGORITHMS EMPLOYED:

 Facial landmark detection: the mediapipe library is employed to detect

key facial landmarks, including points on the face such as eyes, nose,
and mouth. These landmarks serve as input features for analyzing
facial expressions and inferring the user's emotional state.

 Hand gesture recognition: hand gestures are also detected using the
mediapipe library, providing additional cues for emotion inference.
The positions and movements of the hands are analyzed to further
refine the estimation of the user's emotional state.

 Emotion recognition model: a pre-trained deep learning model, loaded

using keras, is employed for emotion recognition from facial
landmarks. The model is trained on labeled datasets of facial
expressions to predict the user's emotion based on the extracted

10
 features. It outputs a classification label corresponding to the predicted
emotion, which is then used to generate personalized song
recommendations.

Fig3.3 Emotion mapping

3.4 SYSTEM REQUIREMENTS:

Hardware requirements:

 Webcam: the system requires a webcam or integrated camera for real-

time video capture.

 Adequate processing power: the system may require sufficient cpu and
gpu resources, especially for real-time video processing and deep
learning inference tasks.

 Internet connection: an active internet connection is necessary to

retrieve song recommendations from online platforms such as youtube.

11
 Software requirements:

 Operating system: the program is compatible with various operating

systems, including windows, macos, and linux.

 Python environment: python 3.x should be installed on the system to

run the program and its dependencies.

 Python libraries: the following python libraries are required:

 Streamlit: for building the user interface.

 Mediapipe: for real-time emotion detection from facial landmarks and

hand gestures.

 Opencv: for webcam video capture and processing.

 Keras: for loading pre-trained deep learning models for emotion

recognition.

 Webbrowser: for opening web browser windows to display song

recommendations.

 Other dependencies as specified in the program code.

Additional requirements:

 Pre-trained models: the system may require pre-trained deep learning

models for emotion recognition, which should be available in the
specified file formats (e.g., .h5 for keras models).

12
 Access to online platforms: to retrieve song recommendations, the
system needs access to online platforms such as youtube. Users should
ensure that their internet connection allows access to these platforms.

Fig3.4 Software Used

13
 CHAPTER 4

IMPLEMENTATION

4.1 DETAILED EXPLANATION OF THE IMPLEMENTATION

PROCESS:

I. The implementation process involves several key steps:

 Setting up the python environment and installing required

libraries.

 Downloading pre-trained model and data files.

 Writing the code for the emotion-based music recommender

program.

 Updating file paths in the code to point to the locations of

model and data files.

 Running the program and interacting with the user interface to

provide input preferences and trigger song recommendations.

4.2 CODE SNIPPETS, ALGORITHMS, OR MODELS USED:

I. Code snippets from the implementation include:

 Webcam video capture and processing using opencv.

 Real-time emotion detection from facial landmarks and hand

gestures using mediapipe.

 Loading pre-trained deep learning model for emotion

recognition using keras.

 User interface creation using streamlit.

14
II. Algorithms used in the project include facial landmark
detection, hand gesture recognition, and emotion recognition
from neural networks.

Fig 4.2 Hand detection

4.3 CHALLENGES FACED DURING IMPLEMENTATION AND

SOLUTIONS:

I. Challenge: integrating real-time emotion detection with the

recommendation engine.

 Solution: utilized the mediapipe library for efficient real-time

processing of webcam feeds and extracting facial landmarks
and hand gestures for emotion inference.

II. Challenge: ensuring robustness and accuracy of emotion

recognition.

 Solution: trained and fine-tuned a deep learning model using

labeled datasets of facial expressions to improve the accuracy
of emotion prediction. Implemented error handling and
fallback mechanisms for cases where emotion detection fails.
15
 III. Challenge: handling dependencies and compatibility issues.

 Solution: thoroughly tested the program with different

versions of python and required libraries to ensure
compatibility across various environments. Used virtual
environments to manage dependencies and isolate the project
environment.

4.4 HANDLING ERRORS AND EDGE CASES:

 Challenge: anticipating and handling errors, edge cases, and

unexpected user inputs that may occur during program
execution.

 Solution: implement robust error handling mechanisms, such

as try-except blocks, to gracefully handle exceptions and
provide informative error messages to users. Incorporate input
validation to prevent invalid inputs and guide users towards
correct usage.

4.5 DEPLOYMENT AND PLATFORM COMPATIBILITY:

 Challenge: ensuring compatibility and smooth deployment of the

emotion-based music recommender across different platforms and
environments.

 Solution: test the program on various operating systems (e.g.,

windows, macos, linux) and web browsers to identify and address
platform-specific issues. Consider containerization techniques (e.g.,
docker) for packaging the application and ensuring consistent
behavior across environments.

16
4.6 MODEL INTEGRATION AND COMPATIBILITY:

 Challenge: Integrating pre-trained deep learning models (e.g.,

for emotion recognition) into the project and ensuring
compatibility with the existing codebase can be complex.

 Solution: Use established deep learning frameworks like

Keras or TensorFlow for model loading and inference. Ensure
that the model architecture, input preprocessing, and output
format are compatible with the requirements of the project.

17
 CHAPTER 5

CONCLUSION

The Emotion-Based Music Recommender represents a significant

advancement in the field of personalized music recommendation systems,
leveraging real-time emotion detection to enhance the relevance and
engagement of song suggestions. Through the integration of computer
vision techniques, deep learning algorithms, and web application
development tools, our system offers users a more intuitive and empathetic
music discovery experience.

By capturing users' emotional states through facial expressions and hand

gestures, our system goes beyond traditional recommendation approaches
to provide personalized song recommendations that resonate with users'
current emotional needs and preferences. The seamless integration of real-
time video processing and web browser interaction enables users to
interact with the system in a user-friendly and intuitive manner, fostering
deeper emotional connections with the music they love.

Through our project, we have demonstrated the potential of technology to

deepen our emotional connection with music and create more meaningful
and satisfying music listening experiences. By leveraging cutting-edge
technologies and user-centric design principles, our Emotion-Based Music
Recommender opens up new avenues for exploring and discovering music
that resonates on an emotional level.

Moving forward, further research and development efforts can focus on

refining the emotion detection algorithms, expanding the music
recommendation capabilities, and exploring additional features to enhance
user engagement and satisfaction. Additionally, user feedback and iterative
refinement will be essential in ensuring the continued relevance and

18
effectiveness of the system in meeting the evolving needs and preferences
of users.

In summary, the Emotion-Based Music Recommender project represents a

step towards revolutionizing the way users interact with music, offering a
personalized and immersive music discovery experience that enriches our
emotional well-being and enhances our enjoyment of music in the digital
age.

19
 CHAPTER 6

REFERENCES

1. BOOKS:

I- "DEEP LEARNING" BY IAN GOODFELLOW, YOSHUA BENGIO, AND

AARON COURVILLE: PROVIDES COMPREHENSIVE COVERAGE OF DEEP
LEARNING TECHNIQUES, INCLUDING CONVOLUTIONAL NEURAL
NETWORKS (CNNS) USED IN FACIAL EMOTION RECOGNITION.

II- "HANDS-ON MACHINE LEARNING WITH SCIKIT-LEARN, KERAS, AND

TENSORFLOW" BY AURÉLIEN GÉRON: OFFERS PRACTICAL INSIGHTS INTO
BUILDING MACHINE LEARNING MODELS WITH KERAS, WHICH IS USED IN
THE EMOTION RECOGNITION MODEL.

2. RESEARCH PAPERS:

I - "A REVIEW ON MUSIC EMOTION RECOGNITION TECHNIQUES" BY

ABHISHEK KUMAR AND R. K. SHARMA: PROVIDES AN OVERVIEW OF
VARIOUS TECHNIQUES FOR MUSIC EMOTION RECOGNITION, WHICH CAN
INFORM THE RECOMMENDATION ASPECT OF THE PROJECT.

II- "DEEP FACIAL EXPRESSION RECOGNITION: A SURVEY" BY ZHIWEI

DENG, JIANI HU, AND JUN GUO: DISCUSSES STATE-OF-THE-ART
APPROACHES FOR FACIAL EXPRESSION RECOGNITION USING DEEP
LEARNING, RELEVANT FOR EMOTION DETECTION FROM FACIAL
LANDMARKS.

3. BLOGS AND ARTICLES:

I- "REAL-TIME EMOTION DETECTION WITH PYTHON" BY DIVYANSHU

SHEKHAR: OFFERS INSIGHTS INTO IMPLEMENTING REAL-TIME EMOTION
DETECTION USING OPENCV AND DEEP LEARNING MODELS.

II- "BUILDING A REAL-TIME EMOTION RECOGNITION APP WITH

STREAMLIT AND TENSORFLOW.JS" BY MADHURIMA DAS: PROVIDES A
TUTORIAL ON BUILDING A REAL-TIME EMOTION RECOGNITION

20
 APPLICATION USING STREAMLIT AND TENSORFLOW.JS, RELEVANT FOR
THE PROJECT'S WEB APPLICATION ASPECT.

4. ONLINE COURSES AND TUTORIALS:

I - COURSERA: "CONVOLUTIONAL NEURAL NETWORKS" BY ANDREW NG:

COVERS THE FUNDAMENTALS OF CNNS, WHICH ARE USED IN FACIAL
EMOTION RECOGNITION.

II- UDEMY: "DEEP LEARNING A-Z™: HANDS-ON ARTIFICIAL NEURAL

NETWORKS" BY KIRILL EREMENKO AND HADELIN DE PONTEVES: OFFERS
A COMPREHENSIVE COURSE ON DEEP LEARNING TECHNIQUES,
INCLUDING EMOTION RECOGNITION.

5. RESEARCH WEBSITES:

I - ARXIV.ORG: A PREPRINT REPOSITORY WHERE YOU CAN FIND

RESEARCH PAPERS ON TOPICS RELATED TO DEEP LEARNING, COMPUTER
VISION, AND EMOTION RECOGNITION.

II- IEEE XPLORE: PROVIDES ACCESS TO ACADEMIC JOURNALS AND

CONFERENCE PROCEEDINGS IN THE FIELD OF ENGINEERING AND
TECHNOLOGY, INCLUDING RESEARCH ON MUSIC RECOMMENDATION
SYSTEMS AND EMOTION RECOGNITION.

THESE REFERENCES THAT SERVE AS VALUABLE SOURCES OF

INFORMATION AND GUIDANCE FOR UNDERSTANDING THE UNDERLYING
CONCEPTS, TECHNIQUES, AND METHODOLOGIES RELEVANT TO THE
EMOTION-BASED MUSIC RECOMMENDER PROJECT.

21