Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL1016

Recognition of Facial Expression with the Help of

IoT, AI and Robotics
1
Alka Mishra; 2Akash Mishra; 3 Vandna Pathak
1,2
M Tech Scholar, Industrial Automation and Robotics , 3Head of Department, Industrial Automation and Robotics
1
Ambalika Institute of Management and Technology, Dr A.P.J Abdul Kalam Technical University, Lucknow,
2
Ambalika Institute of Management and Technology, Dr A.P.J Abdul Kalam Technical University, Lucknow
3
Ambalika Institute of Management and Technology, Dr A.P.J Abdul Kalam Technical University, Lucknow

Abstract:- The emerging field of "Smart Face The main objective of this project is to create and
Recognition" utilizes IoT and machine learning to implement a novel, automated system for identifying
accurately identify individuals based on their facial emotions and recognizing faces using Artificial Intelligence
characteristics. Various industries such as security, retail, (AI) and the Internet of Things (IoT). Important concepts
and healthcare are leveraging this technology to enhance involved are Face Recognition, Emotion Detection, Artificial
customer satisfaction and increase productivity. By Intelligence, and Internet of Things.
combining IoT and machine learning, large amounts of
data can be collected from multiple sources, such as  Problem Definition:-
cameras and sensors, and used to train algorithms for Our goal is to create a system that makes use of machine
real-time, precise identification of individuals. This learning and the Internet of Things (IoT) to effectively and
technology is gaining popularity due to its accuracy, intelligently detect faces. Traditional face recognition
speed, and scalability, making it essential for applications systems rely on manual input, which can be time-consuming,
like security and access control. Recognizing human facial error-prone, and have low accuracy. By capitalizing on the
emotions is a key focus in today's technological landscape, growing popularity of IoT devices and machine learning
with robotic applications across various sectors methods, we can develop a system that can accurately identify
highlighting the importance of emotion recognition for individuals without the need for user input, by learning from
effective human-robot interaction. This project aims to facial feature patterns. This system can be applied in various
develop and implement a new automated system for areas such as attendance tracking, security systems, and
emotion detection and facial recognition using Artificial personalized marketing. However, the main challenge is to
Intelligence (AI) and the Internet of Things (IoT). build a system that is both reliable and secure, while also
protecting the privacy of users.
Keywords:- Face Recognition, Emotion Detection, Artificial
Intelligence, and Internet of Things.  Objectives
The objective of integrating intelligent facial
I. INTRODUCTION recognition technology through IoT and machine learning is
to improve the security and productivity of various
Artificial intelligence, simply put, involves the enterprises and public spaces. By utilizing cameras and
development of artificial humans or intelligent machines sensors, this technology captures images and videos of
capable of processing information in a manner similar to individuals, which are then analyzed by machine learning
humans. Illustrations of AI models encompass the Siri Virtual algorithms to identify faces and compare them against a
Assistant found on Apple iPhones and autonomous vehicles.t database of known individuals.
human intervention. The implementation of AI has
revolutionized the way work is done today, with many  Hardware and Software Tools Used
companies replacing workers with AI-powered robots or Integrating smart facial recognition can help
automation technologies. Tasks that used to take days to organizations improve security measures by monitoring
complete can now be done in a matter of minutes thanks to employee attendance, detecting unauthorized access, and
AI. From surveillance to data mining to warfare, AI-based identifying potential risks promptly. This technology also
robots are being used in various fields to reduce human enables quick and easy access control to restricted areas such
casualties. In fact, US-based companies are developing as banks, airports, and government facilities.
different types of robotic armies for warfare.
Furthermore, the use of IoT-enabled smart facial
 Overview recognition has the potential to improve the consumer
The identification of human facial emotions is a crucial experience across different industries such as hospitality,
aim in the contemporary technological sphere. Robotic retail, and healthcare. By analyzing clients' faces and
applications are now prevalent in nearly all sectors, preferences, services and recommendations can be
emphasizing the significance of emotion recognition for personalized accordingly. The use of IoT and machine
successful human-robot interaction. Version 1: learning in developing smart facial recognition aims to boost

IJISRT24JUL1016 www.ijisrt.com 783

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL1016

security, efficiency, and customer satisfaction across different instances and functions assessed on these instances.
commercial sectors and public spaces. Generalization refers to a machine learning system's ability to
accurately perform on new, unseen data instances after being
II. LITERATURE SURVEY trained on a set of learning data instances. The training
examples are sourced from a probability distribution that is
A. Recent Advances in Face Recognition using Convolutional generally unknown, and the learner must construct a general
Neural Networks: model of this space to make sufficiently accurate predictions
The application of convolutional neural networks on new cases. The system's ability to generalize is typically
(CNNs) has greatly improved the accuracy of face assessed based on its capacity to replicate known knowledge
recognition systems. In a study by Taigman et al. (2014), a from newer examples. While there are various types of
deep CNN was trained on a large dataset of face images, machine learning, the primary ones are supervised learning
leading to outstanding results in face identification and and unsupervised learning. Supervised learning refers to the
verification tasks. The research emphasizes that the deep process of training a machine learning model to infer a
neural network effectively extracted highly distinctive function based on provided training data. This training data
features from facial pictures, consequently enhancing the consists of examples that include both input subjects and their
overall performance. corresponding desired output. By analyzing this training data,
a supervised learning algorithm generates a function, also
B. IoT-based Facial Recognition Systems: known as a classifier or regression function. The ultimate goal
The Internet of Things (IoT) has played a significant role of this function is to accurately predict the correct output
in enhancing the capabilities of facial recognition systems. A value for any given input object. To achieve this, the learning
research conducted by Fan et al. (2018) showcased the algorithm must be able to generalize from the provided
utilization of Raspberry Pi to develop an IoT-based facial training data to handle unseen situations effectively.
recognition system. This system effectively captured and
processed facial photographs, enabling swift and precise A. Humanoid Robot and Robot Structure
identification of individuals in real-time. Humanoids are a combination of artificial intelligence
and robotics. They usually possess a physical form
C. Facial Recognition using GANs: resembling that of humans, frequently featuring skin and
The application of generative adversarial networks eyes, and are outfitted with sensors and cameras for detecting
(GANs) has resulted in a recent enhancement in the accuracy of human faces, reacting to voice instructions, and participating
facial recognition systems. In 2018, Zhang and colleagues in dialogues.
developed a GAN-driven method to produce high-quality facial
images, which were subsequently utilized for the training of a The robotic arm is made up of three key parts: the
facial recognition system. The study demonstrated the efficacy shoulder, elbow, and wrist, which together create the main
of GANs in improving the precision of facial recognition structure of the arm. Located at the bottom of the arm, the
assignments. shoulder is usually linked to the controller and allows for
movement in different directions such as forward, backward,
D. Hybrid Face Recognition Systems: and rotational motion.
Furthermore, there have been advancements in hybrid
facial recognition systems that incorporate traditional B. Existing System
computer vision techniques as well as machine learning  The current system has selected five unique algorithms
algorithms. Zhou et al. (2018) conducted a study where they based on the most crucial criteria. These algorithms
merged Local Binary Patterns (LBP) with Support Vector include Wavelet, Linear Discriminant Analysis (LDA),
Machines (SVMs) to establish a hybrid face recognition Principal Component Analysis (PCA), and Artificial
system. This system demonstrated exceptional accuracy Neural Networks (ANN). Additional criteria, such as type
when employed in facial recognition tasks. and size specifications, have been used to evaluate these
algorithms.
III. MACHINE LEARNING  A sophisticated facial recognition camera system captures
an individual's face as they approach. The system then
Machine learning is a subfield of artificial intelligence processes the image to extract facial features like the
that concentrates on the development and examination of positions of the mouth, nose, and eyes. Subsequently, this
systems capable of learning from data. Instead of relying on data is transmitted to the cloud server for further analysis.
a fixed equation, machine learning algorithms use  The cloud server compares the extracted facial features
computational techniques to extract information directly from with the images stored in the database. If a match is found,
data. These algorithms continuously improve their the system identifies the individual and provides any
performance as they are exposed to more data samples. additional information stored in the database. If no match
According to Tom M. Mitchell, a computer program learns is found, the system may request more information from
from experience E in relation to a specific set of tasks T and the user or simply deny the request.
performance measure P if its performance in tasks within T,
as measured by P, improves with experience E. The essence
of machine learning lies in representation and generalization.
All machine learning systems involve representing data

IJISRT24JUL1016 www.ijisrt.com 784

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL1016

C. Limitations: -  Face Alignment:

Once the face has been detected, the subsequent task
 Dependence on High-Quality Data: An issue with current involves aligning it accurately. The process of face alignment
IoT-based and machine learning-based smart facial holds immense significance as it aids in eliminating
recognition systems is their reliance on high-quality data discrepancies in face positions and scales. Numerous face
for optimal performance. If the input data is of low quality alignment techniques exist, such as MTCNN, 3D Morphable
or poorly structured, the algorithm may struggle to Model, and more.
accurately identify faces, leading to incorrect matches or
rejections.  Feature Extraction:
 Sensitivity to Environmental Factors and Lighting: The The subsequent stage includes the extraction of
effectiveness of smart facial recognition systems is characteristics from the facial image. Different algorithms for
significantly influenced by environmental factors, feature extraction, such as Local Binary Pattern (LBP), Scale-
particularly lighting conditions. lements like shadows, Invariant Feature Transform (SIFT), Histogram of Oriented
glare, and variations in illumination can significantly Gradients (HOG), and Convolutional Neural Networks
impact the system's precision, resulting in less than ideal (CNNs), are employed for this task.
performance in specific environments or lighting
situations.  Facial Recognition:
 Limited Adaptability to Face Variations: While machine After extracting the characteristics, the facial recognition
learning techniques can improve the precision of facial algorithm compares them with the stored database of
recognition systems, their capacity to identify faces with recognized faces in order to determine the individual's
significant variations is restricted. Factors such as facial identity. There are several methods that can be utilized for the
hair, changes in hairstyle, and facial expressions can recognition process, including Eigenfaces, Fisherfaces, Local
present difficulties for the system, resulting in recognition Binary Patterns Histograms (LBPH), Deep Neural Networks,
inaccuracies. and more.
 Privacy Concerns: A key drawback of intelligent facial
recognition systems is the potential for privacy violations.  Decision Making:
These technologies may gather facial data without the The last step involves reaching a conclusion. After the
individual's consent, raising concerns about data misuse. completion of the face recognition process, the system checks
Additionally, the system's security vulnerabilities could if the face matches any of the faces saved in the database. If
lead to unauthorized access and misuse of facial data. a match is found, the system can reveal the person's identity;
 Ethical and Legal Considerations: The utilization of smart however, if there is no match, the face is classified as
facial recognition systems raises ethical and legal unknown.
challenges, particularly concerning bias, discrimination,
and surveillance. Users may encounter unjust treatment To sum up, this flowchart gives an outline of the
based on their ethnicity, gender, or other attributes. standard process of a face recognition system, starting from
Moreover, the technology's application for surveillance gathering data and ending with the decision-making phase.
purposes may violate individuals' right to privacy, leading Depending on the specific setup, there might be variations in
to adverse legal and ethical consequences. of Current the algorithms and methods utilized.
System.  Innovative algorithm
 Advanced Learning Techniques
D. Proposed System  CNNs

 Data Collection: IV. METHODOLOGY

The first step in the face recognition system is the
collection of data. The system must collect images of faces  Dataset
from various angles and in different lighting situations. This The challenging and captivating subject of face
can be done by using a camera or a video camera to gather the identification has attracted researchers from various
necessary data. disciplines such as psychology, pattern recognition, neural
networks, computer vision, and graphics.
 Face Detection: ● The following face recognition methods are utilized:
In the second phase, the system recognizes the existence ● Utilizing oriented gradient histograms (HOG) to detect all
of a face in the taken image. Different face detection faces
algorithms, such as the Viola-Jones algorithm, deep learning- ● Engaging in a demonstration and showcasing Face
based algorithms like CNN, YOLO, and SSD, utilize a variety Landmark Estimation
of methods to precisely identify faces in an image. ● Encoding the faces
● Extracting the individuals' names from the encoding.

IJISRT24JUL1016 www.ijisrt.com 785

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL1016

Fig.1 Schematic Diagram of Face Recognition

Utilize an FTDI programmer to connect the ESP32-  A camera with a resolution of at least 2 MP or higher can
CAM board to your computer. A jumper that lets you choose be utilized
between 3.3V and 5V is found on several FTDI programmers.
Make sure the jumper is in the appropriate position to choose  ESP-32:
5V. The ESP32 microcontroller, a System-on-a-chip (SoC),
is an affordable and energy-efficient device suitable for
V. DESCRIPTION various applications, such as face recognition. Its integrated
WiFi and Bluetooth capabilities make it a popular choice for
A. Hardware Requirements: - IoT and smart home projects.
 To ensure smooth processing, it is necessary to have a
Random Access Memory (RAM) of 4 GB or higher. The ESP32 can be employed for facial recognition by
 It is essential to have a hard drive with a minimum capacity capturing images using a camera module and analyzing them
of 10 GB to accommodate all the data. with a deep learning algorithm. The algorithm can be trained
 A Dual Core Processor is a prerequisite, or alternatively, for real-time facial recognition and identification.
the latest and most advanced model should be utilized. Subsequently, the ESP32 can perform additional face
 The operating system can be either the most recent version recognition tasks or initiate actions based on the identified
of MacOS or Windows (7/8/10). individual's profile

Fig.2 ESP-32

 FTDI Programmer:
FTDI (Future Technology Devices International) is a well-known manufacturer of USB-to-serial converter chips that are
commonly used to provide a USB interface for microcontrollers like the ESP32. In the case of the ESP32 development boards, the
USB-to-serial converter chip integrated into them is referred to as FTDI. This chip enables the ESP32 to establish a connection with
a computer via USB, allowing for firmware updates and interaction through a terminal program. Due to the absence of a USB port
on the ESP32- CAM, we require an FTDI programmer to upload code using the U0R and U0T pins (serial pins).

IJISRT24JUL1016 www.ijisrt.com 786

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL1016

Fig.3 FTDI Programmer

B. Software Requirements: Description: projects involving face recognition, it cannot directly perform
face recognition itself. Instead, it can serve as a control hub
 Arduino IDE: to process signals from cameras and other sensors and
An Arduino is a platform for open-source electronics interface with them. To implement facial recognition, you
that utilizes user-friendly hardware and software. Its purpose would need to connect a camera module, such as the OV7670,
is to simplify the development of interactive electronic to the Arduino board. Additionally, you would need to utilize
projects for individuals. Arduino boards microcontrollers can a software library like OpenCV to process the camera photos
interface with a wide range of sensors, motors, and other and execute the facial recognition algorithms. Once the face
hardware components, and they can be programmed using recognition algorithm produces results, the Arduino can take
various programming languages, including C++. While the appropriate actions based on those findings.
Arduino microcontroller platform can be used to build

Fig.4 Arduino IDE

 Language Used: Utilizing machine learning techniques, face recognition

Python is a widely used programming language that systems can still perform face identification and classification
enables the development of intelligent face recognition by leveraging pre-defined features. For instance, a support
systems through the integration of IoT and machine learning. vector machine (SVM) classifier can be employed to match
The application of face recognition extends to various facial traits with known individuals stored in a database. The
domains such as marketing, access control, and security, IoT component of these systems may incorporate cameras or
where it serves the purpose of identifying or verifying an other sensors to capture facial photos or data, which can then
individual's identity based on their facial characteristics. be processed using Python scripts.

IJISRT24JUL1016 www.ijisrt.com 787

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL1016

 Editor Used: pinpoint the exact whereabouts of a criminal or enabling the

PyCharm is an integrated development environment transmission of alerts to the nearest crime investigation
(IDE) that supports the Python programming language. It department for apprehending suspects can be integrated upon
offers a robust editor with features such as code completion, approval and successful validation.
debugging, and intelligent code analysis. PyCharm can be
utilized for various tasks like data preparation, face REFERENCES
recognition using machine learning models, and other
software development activities. Moreover, PyCharm is [1]. [TP91] M. Turk and A. Pentland. Eigenfaces for
instrumental in creating software components that interact recognition. Journal of Cognitive Neuroscience, 3(1):71-
with IoT devices. It also provides support for numerous 86, 1991.
Python libraries and frameworks commonly employed in [2]. [ZKC+98] W. Zhao, A. Krishnaswamy, R. Chellappa,
machine learning and computer vision, such as OpenCV and D. Swets and J. Weng.Discriminant analysis of principal
Tensor Flow. components for face recognition, pages 73-85. Springer
Verlag Berlin, 1998.
 Library Used:- [3]. [GHW12] M. Günther, D. Haufe and R.P. Würtz. Face
CMake is a build system tool utilized for generating recognition with disparity corrected Gabor phase
build files for C++ programming, rather than being a library. differences. In Artificial neural networks and machine
When developing projects related to image recognition, learning, volume 7552 of Lecture Notes in Computer
particularly intelligent face recognition, it is common to Science, pages 411-418. 9/2012.
combine CMake with libraries such as OpenCV, TensorFlow, [4]. Khan, M. H. Javed, E. Ahmed, S. A. A. Shah and S. U.
and other machine learning libraries. Additionally, CMake Ali, "Facial Recognition usingConvolutional Neural
generates native workspaces and make files that are Networks and Implementation on Smart Glasses," 2019
compatible with various compiler environments. Kitware International Conference on Information Science and
developed the CMake tool set to offer a reliable, cross- Communication Technology (ICISCT), 2019, pp. 1-6,
platform build environment for open-source programs like doi: 10.1109/CISCT.2019.8777442.
ITK and VTK. [5]. Mehedi Masud, Ghulam Muhammad, Hesham
Alhumyani, Sultan S Alshamrani, Omar Cheikhrouhou,
 Dlib: Saleh Ibrahim, M. Shamim Hossain, Deep learning-
The Dlib general-purpose cross-platform software based intelligent face recognition in IoT-cloud
library in the programming language C++ was developed environment, Computer Communications, Volume 152,
with inspiration from component-based software engineering 2020, Pages 215-222, ISSN 0140- 3664.
and design by contract. As a result, it consists mainly of [6]. Bhatti, K., Mughal, L., Khuhawar, F., &Memon, S.
standalone software components. It is classified as open- (2018). Smart attendance management system using face
source software. recognition. EAI Endorsed Transactions on Creative
Technologies, 5(17).
VI. CONCLUSION [7]. Kumar, P. M., Gandhi, U., Varatharajan, R., Manogaran,
G., & Vadivel, T. (2019). Intelligent face recognition and
It has been proven that face recognition-based systems navigation system using neural learning for smart
for human or criminal detection are reliable and secure. security in the Internet of Things. Cluster Computing,
Through the use of specific methods and setups involving a 22(4), 7733-7744.
variety of hardware and software like OpenCV, face detection [8]. Agarwal, L., Mukim, M., Sharma, H., Bhandari, A., &
and identification can be accomplished effectively. Mishra, A. (2021, March). Face recognition based smart
Furthermore, the integration of Arduino ide, ESP-32, and and robust attendance monitoring using deep CNN. In
FTDI has shown to be successful in capturing images from 2021 8th International Conference on Computing for
the surroundings and matching them with a stored database, Sustainable Global Development (INDIACom) (pp.
resulting in a higher recognition rate and lower false rate. This 699-704). IEEE.
system can function as a security surveillance system, and by [9]. Kumar, T. A., Rajmohan, R., Pavithra, M., Ajagbe, S. A.,
adding a Raspberry Pi infrared camera module, the Hodhod, R., & Gaber, T. (2022). Automatic face mask
identification rate can be further improved. detection system in public transportation in smart cities
using IoT and deep learning. Electronics, 11(6), 904.
FUTURE SCOPE [10]. Atik, M. E., & Duran, Z. (2020, October). Deep
learning-based 3d face recognition using derived
If there is an increase in the demand for face recognition features from point cloud. In The Proceedings of the
technology, it will prove beneficial for all in the upcoming Third International Conference on Smart City
years. The suggested technology will not only be limited to Applications (pp. 797-808). Springer, Cham.
criminal identification but will also be beneficial for various [11]. BoserB ,Guyon I.G,Vapnik V., "A Training Algorithm
purposes such as monitoring attendance, enhancing home for Optimal Margin Classifiers", Proc. Fifth Ann.
security, improving business operations, optimizing retail Workshop Computational Learning Theory,pp. 144-
experiences, and managing parking facilities, among other 152, 1992.
applications. Furthermore, supplementary functionalities like
incorporating a location-based IP address tracking system to

IJISRT24JUL1016 www.ijisrt.com 788

Volume 9, Issue 7, July – 2024 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 https://doi.org/10.38124/ijisrt/IJISRT24JUL1016

[12]. Mitchell, T. (1997). Machine Learning, McGraw Hill.

ISBN 0-07-042807-7., McGraw-Hill, Inc. New York,
NY, USA. Published on March 1, 1997
[13]. Alex C, Boston A. (2016).Artificial Intelligence, Deep
Learning, and Neural Networks, Explained (16:n37)
[14]. Varun G., Lily P., Mark C., “Development and validation
of a deep learning Algorithm for Detection of Diabetic
Retinopathy”, December 2016.
[15]. Tiago T.G. “Machine Learning on the Diabetic
Retinopathy Debrecen Dataset”, knowledge- Based
System60, 20-27. Published on June 25, 2016.
[16]. Yau JW, Rogers SL, Kawasaki R, Lamoureux EL,
Kowalski JW, Bek T, et al. Global prevalence and major
risk factors of diabetic retinopathy. Diabetes Care.
2012;35:556–64
[17]. Boser B. E, Guyon I. M. and Vapnik V. N. (1992). “A
training algorithm for optimal margin
classiers”.Proceedings of the 5th Annual Workshop on
Computational Learning Theory COLT'92, 152
Pittsburgh, PA, USA. ACM Press, July 1992. On
Page(s): 144-152

IJISRT24JUL1016 www.ijisrt.com 789