111Equation Chapter 1 Section 1

UTTARA UNIVERSITYT

Department of Electrical & Electronics Engineering

Face Recognition and RFID Based Door Lock

xx
11
xx
22
xx
33
xx
44

In Partial Fulfillment of the Requirements for the Degree of BSc. in Electrical &
Electronics Engineering

October 2023
 Declaration

We hereby declare that this thesis is based on the results found by ourselves. Materials of
work found by other researcher are mentioned by reference. This thesis, neither in whole nor
in part, has been previously submitted for any degree.

Signature of Supervisor Signature of Authors

----------------------------- ----------------------------

----------------------------

----------------------------
 Acknowledgement

Special thanks to our honorable supervisor ______ for his help and support to proceed our thesis on
Face Recognition and RFID Based Door Lock to our co-supervisor ______for his/her co-operation.
 Abstract

The RFID and Face-Recognized Door Lock system presented in this project integrates two advanced
technologies, Radio-Frequency Identification (RFID) and facial recognition, to enhance the security
and convenience of door access control. This system is implemented using the ESP32-CAM and
NodeMCU microcontrollers, which offer a powerful and flexible platform for IoT applications.

The system's core functionality includes two primary methods of user authentication: RFID
Authentication: RFID technology is utilized to enable access through the door lock. Users are
provided with RFID cards or key fobs. When a user presents their RFID card to the RFID reader, the
system authenticates the user's identity based on the unique RFID tag information. If a valid RFID tag
is detected, the door lock mechanism is triggered to unlock the door. Facial Recognition
Authentication: The system employs the ESP32-CAM module, which is equipped with a camera and
the necessary processing capabilities for facial recognition. Users need to stand in front of the camera
for the facial recognition system to capture their face. The captured image is processed using facial
recognition algorithms to identify and authenticate the user. If the facial recognition process confirms
the user's identity, the door lock is unlocked.

This RFID and Face-Recognized Door Lock system offers a robust and versatile solution for access
control, combining the reliability of RFID technology with the security of facial recognition. Its
implementation on the ESP32-CAM and NodeMCU ensures a cost-effective and scalable solution
suitable for residential and commercial applications, providing enhanced security and user
convenience.
Table of contents

Declaration...............................................................................................................................................ii

Acknowledgement...................................................................................................................................iii

Abstract....................................................................................................................................................iv

Table of contents......................................................................................................................................v

List of Figure..........................................................................................................................................vii

List of Table..........................................................................................................................................viii

1 Introduction......................................................................................................................................1

1.1 Introduction..............................................................................................................................1

1.2 Objective...................................................................................................................................2

1.3 Problem Statement....................................................................................................................3

1.4 Background...............................................................................................................................5

1.5 Literature Review.....................................................................................................................5

2 Theory...............................................................................................................................................7

2.1 Theory.......................................................................................................................................7

2.2 Methodology.............................................................................................................................9

2.3 Required Component..............................................................................................................10

2.3.1 NodeMCU (ESP8266)....................................................................................................10

2.3.2 ESP32-CAM...................................................................................................................11

2.3.3 Servo Motor....................................................................................................................13

2.3.4 RFID Module..................................................................................................................13

2.3.5 Power Supply..................................................................................................................14

2.3.6 LCD Display...................................................................................................................15

2.3.7 Web Server Interface......................................................................................................15

2.3.8 Arduino IDE...................................................................................................................16

3 Process and results..........................................................................................................................18

3.1 RFID System Result...............................................................................................................18

3.2 Face Reorganization Result....................................................................................................20

4 Discussion.......................................................................................................................................21

5 Conclusions....................................................................................................................................23

5.1 Conclusion..............................................................................................................................23

5.2 Advantages.............................................................................................................................23

5.3 Future Work............................................................................................................................24

References..............................................................................................................................................25

Appendix A............................................................................................................................................A1
List of Figure

Figure 2.1 Block Diagram of the System...............................................................................................16

Figure 2.2 Circuit Diagram of the System..............................................................................................16
Figure 2.3 Flowchart...............................................................................................................................17
Figure. 2.4. NodeMCU...........................................................................................................................17
Figure. 2.5. ESP 32 Camera...................................................................................................................19
Figure. 2.6. Servo Motor........................................................................................................................20
Figure 2.7 RFID Module / Receiver.......................................................................................................21
Figure 2.8 Power Supply........................................................................................................................22
Figure 2.9 LCD Display.........................................................................................................................22
Figure 2.10 Blynk Server and App.........................................................................................................23
Figure 2.11 Arduino IDE........................................................................................................................24
Figure. 3.1 System Initialize...................................................................................................................25
Figure. 3.2 Unauthorized Access............................................................................................................25
Figure. 3.3 Access by Student 1.............................................................................................................26
Figure. 3.4 Access by Student 2.............................................................................................................26
Figure 3.5 Apps Authorization...............................................................................................................26
List of Table

Table 2.1: Pins of NodeMCU.................................................................................................................11

Table 2.2: Pins of ESP 32 Cam..............................................................................................................12
Table 2.3: Pins of Servo Motor..............................................................................................................13
Table 2.4: Pins of RFID Module............................................................................................................14
1 Introduction

1.1 Introduction

In an era of increasing technological innovation and the Internet of Things (IoT), securing our physical
spaces has become a paramount concern. Traditional door locks are evolving into smart and
multifunctional access control systems. The integration of Radio-Frequency Identification (RFID)
technology and facial recognition capabilities into a door lock system has ushered in a new era of
security and convenience.

This project introduces a cutting-edge solution for access control, the "RFID and Face-Recognized
Door Lock," which leverages the capabilities of the ESP32-CAM and NodeMCU microcontrollers.
These two components are powerful and versatile, making them ideal choices for building IoT-based
security systems.

RFID technology has long been used for access control, and its reliability and ease of use make it a
popular choice for many applications. RFID cards or key fobs are provided to authorized users, who
simply need to present their RFID tags to the door lock for entry. This project seamlessly integrates
RFID technology to enable secure and efficient access control.

Facial recognition, on the other hand, represents a revolutionary advancement in the field of biometric
security. With the ESP32-CAM module, equipped with a camera and processing capabilities, users can
gain access to a secured space through facial recognition. This technology identifies individuals based
on their unique facial features, adding an additional layer of security and eliminating the need for
physical credentials.

The RFID and Face-Recognized Door Lock project is designed to offer more than just access control.
It provides a user-friendly web-based interface, accessible through the NodeMCU, allowing easy
management of authorized users. This web interface facilitates the addition and removal of users,
making administration a straightforward task. Real-time notifications and event logging also enhance
security and monitoring capabilities, providing a comprehensive record of access events and user
activity.

This system is highly adaptable and can be integrated with IoT platforms, allowing for remote access
control and monitoring, further enhancing the overall functionality and security of the door lock
system.
In summary, the RFID and Face-Recognized Door Lock using ESP32-CAM and NodeMCU
introduces a secure, versatile, and technologically advanced solution for access control. By merging
the reliability of RFID technology with the security of facial recognition, this system offers both
convenience and robust security, making it suitable for residential, commercial, and industrial
applications in our increasingly interconnected world.

1.2 Objective

The primary objectives of implementing an RFID and Face-Recognized Door Lock system using
ESP32-CAM and NodeMCU are to enhance security, convenience, and control over access to a
secured area. The project aims to achieve the following key goals:

1. Multi-Factor Authentication: To provide a high level of security, the system combines two distinct
methods of authentication – RFID and facial recognition. This multi-factor approach ensures that only
authorized individuals can access the secured area, making it significantly more secure than traditional
locks.

2. Access Control: The system's core objective is to control access to a specific area. It allows
authorized users to enter the area while preventing unauthorized access. This enhances the overall
security of the premises.

3. User Management: The project facilitates easy user management through a web-based interface
hosted on the NodeMCU. Administrators can add, modify, or remove users as needed, streamlining
the management of access privileges.

4. Facial Recognition Technology: The integration of facial recognition technology aims to provide a
secure and contactless means of access. This is especially important in situations where hygiene and
touchless access are essential.

5. RFID Technology: RFID technology offers a reliable and efficient means of access control. Users
can gain entry by presenting their RFID cards or key fobs, which are compared to a list of authorized
users.
6. Real-Time Monitoring: The system aims to provide real-time monitoring and logging of access
events. This allows administrators to track who enters and exits the secured area, which is crucial for
security and audit purposes.

7. Remote Access Control: Optionally, the system can be integrated with IoT platforms to enable
remote access control. This allows administrators to manage access even when they are not physically
present at the site.

8. Customizability: The project is designed to be flexible and customizable. Users can define which
individuals are authorized to enter the premises and adapt the system to different access control
scenarios.

9. Convenience: By combining RFID and facial recognition technologies, the system provides a
convenient and efficient means of access. Authorized users can gain entry with minimal effort while
maintaining a high level of security.

10. Enhanced Security: Overall, the primary objective is to enhance the security of the secured area.
This is achieved by implementing a multi-factor authentication system, real-time monitoring, and
secure user management.

11. Scalability: The system should be scalable and adaptable to different environments, such as
residential, commercial, or industrial settings.

In summary, the objective of the RFID and Face-Recognized Door Lock system using ESP32-CAM
and NodeMCU is to create a technologically advanced, secure, and convenient access control solution
that leverages the strengths of RFID and facial recognition technologies. This project seeks to provide
a versatile and robust solution for access control in various real-world scenarios.

1.3 Problem Statement

Problem Statement for a Door Lock Security System:

In the rapidly evolving landscape of security and access control, traditional lock and key mechanisms
are often inadequate in ensuring the safety and convenience of individuals and properties. The
problem statement for a Door Lock Security System addresses several critical challenges:
1. Inadequate Security: Traditional locks can be easily picked or tampered with, rendering them
susceptible to unauthorized access, break-ins, and theft. There is a growing need for more secure
access control methods to protect homes, businesses, and sensitive areas.

2. Key Management Issues: Traditional keys can be lost, duplicated, or shared without control. This
poses a significant challenge for property owners who need a reliable method for granting and
revoking access permissions.

3. Security Vulnerabilities: In a world increasingly connected through the Internet of Things (IoT),
physical locks are often isolated from digital security systems, creating potential vulnerabilities.
Addressing these vulnerabilities is a priority.

4. Limited Access Control: Traditional locks offer limited options for access control, such as granting
temporary access to guests or service providers. There is a demand for flexible and dynamic access
management.

5. Physical Contact Concerns: In a post-pandemic world, touchless and contactless access solutions
have become essential. Traditional keys and locks involve physical contact and hygiene concerns.

6. User Convenience: People are seeking convenient access methods that do not require carrying
physical keys or memorizing complex access codes. The need for quick and efficient access is
paramount.

7. Remote Access Management: Property owners, administrators, or homeowners often require the
ability to manage access remotely, particularly in situations where on-site access may not be feasible.

8. Monitoring and Accountability: There is a growing need for monitoring access events and
maintaining a record of who enters and exits secured areas. This is crucial for security and audit
purposes.

9. Scalability: Different environments, such as residential, commercial, or industrial, may have

varying access control requirements. Finding a scalable solution to cater to these diverse needs is a
challenge.

10. Integration with IoT: Integrating access control systems with IoT platforms or home automation
systems poses integration challenges. Ensuring seamless connectivity and compatibility is a key
problem to address.
11. Security Policy Compliance: Compliance with security regulations and policies is essential for
businesses and organizations. Ensuring that access control systems meet these standards can be a
complex problem.

12. Cost-Effective Solutions: Security enhancements often come with cost considerations. Finding
cost-effective solutions that don't compromise security is a key challenge for businesses and
homeowners.

In response to these challenges, the development and implementation of advanced door lock security
systems are essential. These systems should aim to provide enhanced security, convenience, and
flexibility while addressing the evolving needs of users in an increasingly digital and interconnected
world.

1.4 Background

Face recognition is a biometric technology that identifies and verifies individuals based on their
unique facial features. It has gained popularity in recent years due to advancements in computer vision
and artificial intelligence. How it works: Face recognition systems capture an image of a person's face
using cameras and then analyze the unique facial characteristics, such as the distances between eyes,
nose, and mouth, to create a facial template. This template is then compared to a database of known
faces to determine if there is a match.
RFID is a technology that uses radio waves to identify and track objects, including access control
cards or key fobs equipped with RFID chips. RFID door locks have been used for many years in
various industries. How it works: RFID door locks contain a reader that emits radio waves. When an
RFID card or key fob is brought into proximity with the reader, it activates and sends a signal to
unlock the door. The RFID chip contains a unique identifier, which is used to grant or deny access.

1.5 Literature Review

Since past few years most of the industries are working in the fields of machine learning, artificial
intelligence, big data analytics, IoT based project the major moto of all these is to make things easy
and smart. These became need for digitalizing with lot security tools by these our daily life locks
become smart and also introduced the locks movable with stepper motor and need to digital number
pad to get input from user, and it need to add infrared or any Bluetooth module to operate all these
devices. A major difference in face recognition door lock is that no need to use stepper motor and the
application detects the face with stored images in the application program in our application. We have
eliminated unwanted components of stepper motor and drivers which are in existing models. We have
added newer and unparalleled features of facial detection as an access point to open or close the door.
Where it is the combination of relay module and solenoid lock for opening the door and it is unique
and user friendly. Here have been used USB attachable with HD WebCam to do efficient and
dependable facial detection and it can be stored by using cloud computing for future need. This facial
recognition door model allowing people to get more interested in because of its features and
advantages and also due to its functionality. There are some of drawbacks in that based on that the
industries working in this field and improving the security models day by day. Currently, there are lots
of fraud things and thefts are going on, became significant issues for all. Even if we have locks to the
door, others can enter inside by getting key but if we have password lock also they can get and access
it by these modern technology. By these the facial recognition doors become more secure in these
field, here we can use biometric for face recognition, so others cannot enter inside. In this new era face
recognition plays important role for security and privacy purpose. These face recognition identify the
people, who the person is, if the face is matched with stored database then it will automatically open
the door otherwise it will be sent an alert message to the owner. RELATED WORK The most
commonly used system for locking and unlocking the door is a lock and a physical key. The entire
process is a mechanical one. If the key is lost, misplaced or stolen, then the entire locking mechanism
has to be replaced. This problem with the physical keys intensifies when it comes to big companies
where employees are needed to carry several keys for different doors. Apart from the extra burden, all
the keys add to become vulnerable to getting lost. An alternative used for physical keys currently is
RFID (Radio-Frequency Identification). There are RFID cards being used as pass keys. The RFID card
reader unit is installed near the door. When the card is brought near the reader, it identifies the radio
frequency of the card and thus verifies the key. Multiple cards can be paired with the device. But again
they are vulnerable to theft or getting lost. It also does not solve the purpose of not carrying a key. Just
like traditional keys, you can forget your keycard. It is possible the system can be hacked or bypassed
by someone who is tech-savvy, so they aren't foolproof. One major problem is that electric RFID
systems may malfunction during power outages.
2 Theory

2.1 Theory

Presentation of theories used in the work.

Implementing an RFID and Face-Recognized Door Lock using ESP32-CAM and NodeMCU
combines various technologies to create a secure and efficient access control system. Here's a detailed
theory of how this system works: 1. Components and Hardware Setup: The system consists of several
key components, including the ESP32-CAM, NodeMCU, RFID reader, servo motor, camera module,
and a web server interface. The RFID reader is connected to the NodeMCU to read RFID card
information, while the servo motor controls the physical locking mechanism. The ESP32-CAM,
equipped with a camera module, captures facial images for recognition. The NodeMCU hosts a web-
based interface for user management and system configuration. 2. RFID Authentication: Authorized
users are provided with RFID cards or key fobs, each containing a unique RFID tag. When a user
presents their RFID card to the RFID reader, the NodeMCU reads the RFID tag information. The
system checks the RFID tag against a list of authorized users stored in its memory. If a valid RFID tag
is detected, the servo motor is triggered to unlock the door. 3. Facial Recognition Authentication: To
use facial recognition, a person stands in front of the ESP32-CAM, which captures their facial image
using the onboard camera. The captured image is processed by facial recognition software or
algorithms running on the ESP32-CAM. The software analyzes the facial features, matches them to
pre-registered users, and confirms the user's identity. If the facial recognition process identifies the
user as authorized, the servo motor is activated to unlock the door. 4. User Management: The
NodeMCU hosts a web-based interface accessible to administrators. Through this interface,
administrators can add, modify, or remove authorized users. User data, including RFID tag
information and facial recognition templates, is stored and managed through this interface. 5. Access
Control: The system coordinates both RFID and facial recognition methods for user authentication. If
either method (RFID or facial recognition) confirms the user's identity, the servo motor is activated to
unlock the door. The system logs access events in real-time, allowing for monitoring and
accountability. 6. Remote Access Control: Optionally, the system can be integrated with IoT platforms
to enable remote access control and monitoring. Administrators can manage access permissions and
monitor the system from a remote location using a web interface. 7. Security Considerations: Security
is a primary concern, and the system aims to prevent unauthorized access. Encryption and secure
storage of user data are critical to safeguarding information. Facial recognition software should have
anti-spoofing measures to prevent unauthorized access using photos or videos. 8. Scalability: The
system is designed to be scalable and adaptable to different environments, whether for residential,
commercial, or industrial use. Customization options allow users to define specific access control
scenarios. 9. Convenience: The combination of RFID and facial recognition offers a convenient,
touchless, and efficient means of access. Authorized users can enter the secured area without the need
for physical keys or access codes. 10. Maintenance and Updates: Regular maintenance and software
updates are essential to address security vulnerabilities and improve functionality. In summary, the
RFID and Face-Recognized Door Lock system using ESP32-CAM and NodeMCU provides a robust
and technologically advanced solution for access control. It combines the reliability of RFID
technology with the security and convenience of facial recognition, offering a versatile and secure
access control system suitable for various real-world scenarios. The implementation of an RFID and
Face-Recognized Door Lock system using ESP32-CAM and NodeMCU may involve various
equations and algorithms, particularly in the facial recognition and access control processes. Here are
some equations and concepts relevant to such a system: Facial Recognition Algorithm: The specific
facial recognition algorithm used in the system may involve complex mathematical equations for
feature extraction, face matching, and similarity scoring. These equations vary based on the algorithm
selected (e.g., eigenfaces, Fisherfaces, LBPH, deep learning-based approaches like Convolutional
Neural Networks). Image Processing: Image processing equations may be employed for tasks such as
face detection, normalization, and preprocessing. Common techniques include histogram equalization,
Gaussian blur, and resizing, which involve mathematical transformations on image pixel values.
Feature Extraction: Feature extraction equations are used to capture distinctive facial features for
recognition. These equations can involve techniques like Principal Component Analysis (PCA) or
Eigenfaces for dimensionality reduction and feature extraction. Similarity Score Calculation: In facial
recognition, a similarity score is computed between the captured face and reference faces in the
database. Common equations for similarity scores include Euclidean distance or cosine similarity.
Thresholding: A decision threshold equation is used to determine whether the similarity score exceeds
a predefined threshold, indicating a positive match. Access Control Logic: The system's access control
logic may involve mathematical equations for decision-making, such as determining whether to
unlock the door based on successful authentication. RFID Authentication Matching: In the case of
RFID authentication, an equation is used to compare the RFID tag information read by the reader with
the stored authorized user data. Time Constraints: Equations related to time constraints may be used
for actions like locking the door after a specified duration, to enhance security. Power Management:
Equations for calculating power requirements and usage may be necessary to ensure efficient
operation of the system. Data Encryption: Equations for cryptographic algorithms, such as AES or
RSA, may be used to encrypt and decrypt sensitive data, ensuring secure storage and transmission of
user information. Remote Access Control: Equations related to network protocols and communication
may be needed for remote access and control, such as HTTP requests for interfacing with the
NodeMCU. Cost Estimation: Equations for cost estimation can be used to calculate the expenses
associated with hardware, software, development, and maintenance. These equations and algorithms
are specific to the various components and functions within the RFID and Face-Recognized Door
Lock system. They play a critical role in the authentication, decision-making, and operation of the
system to provide secure and efficient access control. The actual equations used will depend on the
specific technologies and algorithms chosen for implementation.

2.2 Methodology

Figure 2.1 Block Diagram of the System

Figure 2.2 Circuit Diagram of the System

 Figure 2.3 Flowchart

2.3 Required Component

To build an RFID and Face-Recognized Door Lock we need the following components:

2.3.1 NodeMCU (ESP8266)

NodeMCU will serve as the central controller for the system. It will handle RFID authentication, user
management, and control of the servo motor. NodeMCU is a low-cost open source IoT platform. It
initially included firmware which runs on the ESP8266 Wi-Fi SoC from Espressif Systems, and
hardware which was based on the ESP-12 module. Later, support for the ESP32 32-bit MCU was
added.

Figure. 2.4. NodeMCU

 Pin Category Name Description

Power Micro-USB, Micro-USB: NodeMCU can be powered through the USB port
3.3V, GND, 3.3V: Regulated 3.3V can be supplied to this pin to power the board
Vin GND: Ground pins
Vin: External Power Supply

Control Pins EN, RST The pin and the button resets the microcontroller

Analog Pin A0 Used to measure analog voltage in the range of 0-3.3V

GPIO Pins GPIO1 to NodeMCU has 16 general purpose input-output pins on its board
GPIO16

SPI Pins SD1, CMD, NodeMCU has four pins available for SPI communication.
SD0, CLK

UART Pins TXD0, NodeMCU has two UART interfaces, UART0 (RXD0 & TXD0) and
RXD0, UART1 (RXD1 & TXD1). UART1 is used to upload the
TXD2, firmware/program.
RXD2

I2C Pins NodeMCU has I2C functionality support but due to the internal
functionality of these pins, you have to find which pin is I2C.

Table 2.1: Pins of NodeMCU

2.3.2 ESP32-CAM

ESP32-CAM is equipped with a camera module and will capture facial images for recognition. It also
performs facial recognition processing. ESP32-CAM comes with an onboard camera module. Ensure
it's compatible with your ESP32-CAM board. The ESP32-CAM is a small-size, low-power camera
module based on ESP32. It comes with an OV2640 camera and provides an onboard TF card slot. This
board has 4MB PSRAM which is used for buffering images from the camera into video streaming or
other tasks and allows you to use higher quality in your pictures without crashing the ESP32. It also
comes with an onboard LED for flash and several GPIOs to connect peripherals.
 Figure. 2.5. ESP 32 Camera

Pin No. Name Type Function

1 5V POWER 5V Supply

2 GND POWER Ground pin

3 IO12 I/O GPIO 12/MicroSD DATA2

4 IO13 I/O GPIO 13/MicroSD DATA3

5 IO15 I/O GPIO 15/MicroSD CMD

6 IO14 I/O GPIO 14/MicroSD CLK

7 IO2 I/O GPIO 2/MicroSD DATA0

8 IO4 I/O GPIO 12/MicroSD DATA1/ Flash

9 GND POWER Ground pin

10 UOT/IO1 I/O UART TX /GPIO 1

11 UOR/IO3 I/O UART RX /GPIO 3

12 VCC POWER 5V/3.3V

13 3V3 POWER 3.3V Supply

Table 2.2: Pins of ESP 32 Cam

2.3.3 Servo Motor

A servo motor is used to control the physical locking and unlocking of the door. Ensure it has enough
torque to move the locking mechanism. A servo motor is a self-contained electrical device, that rotate
parts of a machine with high efficiency and with great precision. The output shaft of this motor can be
moved to a particular angle, position and velocity that a regular motor does not have.

Figure. 2.6. Servo Motor

Pi Name Function
n

1 Signal pin (Orange The PWM signal which states the axis position is given
pin) through this pin.

2 VCC (Red pin) Positive power supply for servo motor is given to this pin.

3 Ground(Brown pin) This pin is connected to ground of circuit or power supply.

Table 2.3: Pins of Servo Motor

2.3.4 RFID Module

Radio frequency identification, or RFID, has been a popular data transmission tool for many years. An
RFID reader is used to identify uniquely tagged assets, aiding and streamlining workflows in
industries like government, healthcare, supply chain, and enterprise.
The RC522 is a 13.56MHz RFID module that is based on the MFRC522 controller from NXP
semiconductors. The module can supports I2C, SPI and UART and normally is shipped with a RFID
card and key fob. It is commonly used in attendance systems and other person/object identification
applications.
 Figure 2.7 RFID Module / Receiver

Pin Number Pin Name Description

1 Vcc Used to Power the module, typically 3.3V is used

2 RST Reset pin – used to reset or power down the module

3 Ground Connected to Ground of system

4 IRQ Interrupt pin – used to wake up the module when a device

comes into range

5 MISO/SCL/Tx MISO pin when used for SPI communication, acts as SCL for
I2c and Tx for UART.

6 MOSI Master out slave in pin for SPI communication

7 SCK Serial Clock pin – used to provide clock source

8 SS/SDA/Rx Acts as Serial input (SS) for SPI communication, SDA for IIC
and Rx during UART

Table 2.4: Pins of RFID Module

2.3.5 Power Supply

To power the components, we used a stable power supply. Depending on the components' power
requirements, this is a 5V power source.

Figure 2.8 Power Supply

2.3.6 LCD Display

A 16x2 LCD means it can display 16 characters per line and there are 2 such lines. In this LCD each
character is displayed in 5x7 pixel matrix. This LCD has two registers, namely, Command and Data.

Figure 2.9 LCD Display

2.3.7 Web Server Interface

Develop a web-based interface that runs on the NodeMCU for user management and configuration.
This may require web development skills and additional hardware, such as an SD card for storage.
Blynk is a platform that enables developers to build Internet of Things (IoT) applications for various
devices and microcontrollers. It consists of two main components: the Blynk Server and the Blynk
App. The Blynk Server is the backend infrastructure that facilitates communication between IoT
devices and the Blynk App. It acts as a bridge, relaying data and commands between the devices and
the mobile app. The Blynk Server is hosted in the cloud by Blynk Inc., and it handles the secure and
reliable communication required for IoT projects. Blynk offers a cloud-hosted version of the Blynk
Server, but you can also set up your own private Blynk server if you need more control over your
infrastructure or additional features.

Figure 2.10 Blynk Server and App

2.3.8 Arduino IDE

The Arduino IDE (Integrated Development Environment) is an open-source software application that
provides a platform for programming and developing applications for Arduino microcontrollers. It is a
user-friendly and versatile environment that simplifies the process of writing, compiling, and
uploading code to Arduino boards.

Key features and functions of the Arduino IDE include:

1. Code Editor: The IDE offers a code editor with features like syntax highlighting, auto-indentation,
and code suggestions to help developers write code more efficiently.

2. Compiling and Uploading: It allows users to compile their code and upload it to an Arduino board
with just a few clicks. The IDE handles the process of converting human-readable code into machine
code that the microcontroller can execute.

3. Library Management: The IDE provides a library manager that simplifies the process of adding and
managing libraries for various sensors, modules, and components. This makes it easier to use pre-
written code for different hardware.
4. Serial Monitor: The Serial Monitor tool allows developers to communicate with the Arduino board
and monitor the data being sent and received over the serial port. This is crucial for debugging and
testing.

5. Hardware Support: Arduino IDE supports a wide range of Arduino boards, including the popular
Arduino Uno, Arduino Mega, Arduino Nano, and many others. It can also be extended to work with
various third-party boards.

6. Cross-Platform Compatibility: The Arduino IDE is available for Windows, macOS, and Linux,
making it accessible to users on different operating systems.

7. Community and Documentation: The Arduino ecosystem has a vast and active community that
provides extensive documentation, tutorials, and example code. This support network is valuable for
both beginners and experienced developers.

8. Open Source: The Arduino IDE is open source, allowing users to modify and customize it to suit
their needs. Additionally, the underlying Arduino programming language is based on C and C++.

The Arduino IDE is particularly well-suited for prototyping and developing projects that involve
microcontrollers and sensors. It abstracts much of the complexity involved in programming
microcontrollers, making it accessible to individuals with a wide range of technical backgrounds. This
simplicity and the large Arduino community have contributed to its widespread use in education,
hobbyist projects, and professional applications.
Figure 2.11 Arduino IDE
3 Process and results

The theory is applied on the subject. Results are presented.

The final result of implementing face recognition and RFID door lock systems will depend on various
factors, including the specific use case, the quality and integration of the chosen technology, and the
effectiveness of the security measures. Here's a summary of what you can expect from each system:
Here are some snaps of door lock unlock using RFID system

3.1 RFID System Result

Quick and convenient access: Users can quickly unlock doors by presenting their RFID cards or key
fobs. Reliability: RFID technology is known for its reliability, and it can work in various
environmental conditions. Tracking and control: RFID systems offer the ability to track access and
control who can enter specific areas. Cost-effective: RFID technology can be cost-effective, especially
for large installations.

Figure. 3.1 System Initialize

Figure. 3.2 Unauthorized Access

Figure. 3.3 Access by Student 1

Figure. 3.4 Access by Student 2

Figure 3.5 Apps Authorization

3.2 Face Reorganization Result

High accuracy: Modern face recognition systems can achieve high accuracy in identifying individuals.
Convenience: Users don't need physical cards or keys; their face serves as the key. Non-intrusive: It's a
non-contact method of access control, making it hygienic and convenient. Potential for additional
features: Face recognition systems can be integrated with other security and analytics tools.
4 Discussion

The final result will also depend on the quality of the chosen components, the installation, and the
policies and procedures in place to govern access control. In any case, a well-implemented
combination of face recognition and RFID door lock systems can provide a robust and flexible access
control solution for various applications, from corporate offices to hotels and secure facilities.
Combining face recognition and RFID door lock systems can offer a comprehensive and robust
approach to access control. This integration leverages the unique advantages of both technologies and
addresses some of the limitations each system might have individually. Here is a more in-depth
discussion of the two technologies when used together:

1. Enhanced Security:
The integration provides an additional layer of security, making it more difficult for unauthorized
individuals to gain access.
Face recognition helps ensure that the person presenting the RFID card is the authorized user,
minimizing the risk of card theft or misuse.

2. Convenience and User Experience:

Users have the option to choose between using their face or presenting an RFID card for access,
accommodating personal preferences.
This flexibility caters to the needs of a diverse user base, ensuring a smooth and convenient access
control experience for everyone.

3. Redundancy and Fail-Safe Measures:

The integration serves as a redundancy measure, ensuring that access can still be granted if one of the
systems fails or malfunctions.
If the face recognition system encounters issues, users can rely on their RFID cards to gain access, and
vice versa.

4. Auditability and Tracking:

The combined system enables a more comprehensive audit trail, allowing administrators to track who
accessed which areas at specific times.
This data can be invaluable for security investigations, ensuring accountability, and improving overall
security protocols.
5. Cost-Effectiveness and Scalability:
Integrating the two systems can be cost-effective in the long run, as it maximizes the benefits of both
technologies without overly burdening the budget.
The integrated solution can be scaled and customized to fit the specific needs of different
environments, whether it's a small office or a large-scale facility.

6. User Privacy and Data Security:

It is essential to implement stringent data protection measures to safeguard the privacy of users,
especially concerning the storage and use of facial recognition data.
Secure encryption and access controls should be implemented to protect the sensitive information
stored within the system.

7. Maintenance and Management:

Proper maintenance and regular updates are crucial to ensure that both the face recognition and RFID
systems are functioning optimally and are secure against potential vulnerabilities.
Effective management practices, such as user access control and permissions management, should be
implemented to ensure that the system remains secure and efficient.

In conclusion, the integration of face recognition and RFID door lock systems represents a
sophisticated approach to access control, offering a balanced solution that maximizes security,
convenience, and user experience. However, careful consideration should be given to privacy
concerns, data security, and effective maintenance practices to ensure the seamless and secure
operation of the integrated system.
5 Conclusions

5.1 Conclusion

In conclusion, the integration of face recognition and RFID door lock systems provides a multifaceted
and highly effective approach to access control and security. By combining these two technologies,
users benefit from enhanced security measures, convenience, and flexibility. The use of face
recognition ensures that individuals are who they claim to be, reducing the risk of unauthorized access,
while RFID cards offer a reliable and quick means of entry. The redundancy and fail-safe capabilities
of this integrated system provide an additional layer of protection, ensuring access even in the event of
system malfunctions.

Furthermore, the ability to maintain a comprehensive audit trail facilitates tracking and accountability,
which is invaluable for security and management purposes. The integrated approach is cost-effective,
scalable, and adaptable to various environments, making it suitable for both small-scale and large-
scale access control needs.

However, it's essential to address privacy and data security concerns associated with facial recognition
data, implementing robust measures to protect user information. Maintenance and management
practices play a crucial role in ensuring the system's ongoing effectiveness and security. In essence,
the integration of face recognition and RFID door lock systems represents a balanced and powerful
solution that combines the strengths of both technologies to deliver a robust, user-friendly, and secure
access control system.

5.2 Advantages

 Non-intrusive and convenient.

 Difficult to impersonate or spoof.

 High accuracy with modern AI algorithms.

 Quick and convenient access.

 Ability to control and track access electronically.

 Durable and reliable technology.

5.3 Future Work

 In future we will add fingerprints for more accuracy and one step ahead security.

 In future we will use our own server for best internet security.
References

[1] E. Shirisha, K.M.V Madan Kumar, G. Swarnalatha, IOT BASED HOME SECURITY AND
AUTOMATION USING GOOGLE ASSISTANT, Turkish Journal of Computer and
Mathematics Education Vol.12 No.6(2021); 117-122.
[2] J Gal Bally, S Marcel, J Ferez. Image quality assessment for face recognition. IEEE transactions
on image processing. 2014; 23: 710-724.
[3] MA Turk, AP Pentland. Facial recognition using eigen-faces in Computer Vision and Pattern
Recognition. CVPR'91. IEEE Computer Society Conference on. 1991; 586-591.
[4] GJ Edwards, TF Cootes, CJ Taylor. Facial recognition by use of active appearance models in
European conference on computer vision. 1998; 581-595.
[5] G Guo, SZ Li, K Chan. Face detection by support vector machines in automatic facial and gesture
recognition. Fourth Proceedings of IEEE International Conference on 2000; 196-201.
[6] B Moghaddam, T Jabara, A Pentland. Bayesian face recognition, Pattern Recognition. 2000; 33:
1771- 1782.
[7] OM Parkhi, A Veraldi, A Zisserman. Deep Face Recognition in BMVC. 2015
[8] M. N Kamat, Shinde D. Smart Door Lock Control System Using Raspberry Pi. International
Journal of Innovations & Advancemen in Computer Science. 2017 November
[9] Frank Thornton, B. H. (2006). RFID Security. (J. Kleinschmidt, Ed.) Rockland: Syngress
Publishing, Inc.
[10] Keziah Andrew, M.Savitha Devi, A Proportional Learning on Smart Card Authentication,
International Journal ofEngineerings & Research Technology, January 2015.
[11] John Boxall, "Arduino Workshop", 2013.5. Juels, A. (2006). RFID Security and Privacy: A
Research Survey. IEEE Journal On Selected Areas In Communications ,381-394.
[12] Melanie R. Rieback, B. C. (2005, July). RFID Guardian: A Battery-Powered Mobile Device for
RFID PrivacyManagement. Australasian Conference on Information Security and
Privacy(ACISP). , 62-69
[13] Khaing Myat Nwe , Kaythi Wut Mhone Khin , Zin May Win and Zarni Sann, “IOT based Smart
Home Security System using RFID and Bluet ooth” International Journal of Scientific Research
and Engineering Development - – Volume 2 Issue4, July – Aug 2019, pp: 508-514
[14] Keziah Andrew and M.Savitha Devi, “A Proportional Learning On Smart Card Authentication”
International Journal of Engineering Sciences & Research Technology, 4(1): January, 2015
Appendix A

A1. Programming Code for RFID Security

#define BLYNK_PRINT Serial

#include <Servo.h>
#include <ESP8266WiFi.h>
#include <BlynkSimpleEsp8266.h>
#include <SPI.h>
#include <MFRC522.h>
#include <LiquidCrystal_I2C.h>
#include <Wire.h>
#define SS_PIN D3 // sda pin D2
#define RST_PIN D4 // RST (flash) pin D2

Servo myservo;
LiquidCrystal_I2C lcd(0x27,16,2);

char auth[] = "d8SQQEtQDxubNjAPCLHc28aCnXi7yOED";

char ssid[] = "iot";
char pass[] = "iot12345";

MFRC522 mfrc522(SS_PIN, RST_PIN); // Create MFRC522 instance.

SimpleTimer timer;
int button1 = 0;
int button2 = 0;
int b = D8;
int button3 = 0;
WidgetTerminal terminal(V2);

void setup()
{
Serial.begin(115200);
Blynk.begin(auth, ssid, pass, "blynk.cloud", 80);
Wire.begin();

A1
 lcd.begin();
lcd.backlight();
myservo.attach(D0);
myservo.write(180);
pinMode(b, INPUT);
SPI.begin(); // Init SPI bus
mfrc522.PCD_Init(); // Init MFRC522 card
timer.setInterval(1000L, iot_rfid);
lcd.setCursor(0,0);
lcd.print("IOT Door");
lcd.setCursor(0,1);
lcd.print("Security By UU");
delay(5000);
lcd.clear();
}

void loop() {
timer.run(); // Initiates SimpleTimer
Blynk.run();
button3=digitalRead(b);
if (button3 == 1)
{
Serial.println("Access Granted to S3");
Blynk.virtualWrite(V2, "Access Granted to S3" );
lcd.setCursor(0,0);
lcd.print("Access Done Face");
delay(500);
myservo.write(0);
delay(5000);
myservo.write(180);
}
}
void iot_rfid()
{

// Prepare key - all keys are set to FFFFFFFFFFFFh at chip delivery from the factory.
MFRC522::MIFARE_Key key;

A2
 for (byte i = 0; i < 6; i++) {
key.keyByte[i] = 0xFF;
}
// Look for new cards
if ( ! mfrc522.PICC_IsNewCardPresent()) {
return;
}

// Select one of the cards

if ( ! mfrc522.PICC_ReadCardSerial()) {
return;
}
// Now a card is selected. The UID and SAK is in mfrc522.uid.

// Dump UID
Serial.print("Card UID:");
for (byte i = 0; i < mfrc522.uid.size; i++) {
Serial.print(mfrc522.uid.uidByte[i] < 0x10 ? " 0" : " ");
Serial.print(mfrc522.uid.uidByte[i], DEC);
}
Serial.println();

// Dump PICC type

byte piccType = mfrc522.PICC_GetType(mfrc522.uid.sak);
// Serial.print("PICC type: ");
//Serial.println(mfrc522.PICC_GetTypeName(piccType));
if ( piccType != MFRC522::PICC_TYPE_MIFARE_MINI
&& piccType != MFRC522::PICC_TYPE_MIFARE_1K
&& piccType != MFRC522::PICC_TYPE_MIFARE_4K) {
//Serial.println("This sample only works with MIFARE Classic cards.");
return;
}
// defining Cards here

if ( ((mfrc522.uid.uidByte[0] == 163) && (mfrc522.uid.uidByte[1] == 17) &&

(mfrc522.uid.uidByte[2] == 220) && (mfrc522.uid.uidByte[3] == 50)) && (button1 == 1) )
{

A3
 Serial.println("Access Granted to Student 1");
Blynk.virtualWrite(V2, "Access Granted to Student 1" );
lcd.setCursor(0,0);
lcd.print("Access Done S1 ");
delay(500);
myservo.write(0);
delay(5000);
myservo.write(180);
}
else if (( (mfrc522.uid.uidByte[0] == 163) && (mfrc522.uid.uidByte[1] == 49) &&
(mfrc522.uid.uidByte[2] == 113) && (mfrc522.uid.uidByte[3] == 189)) && (button2 == 1) )
{
Serial.println("Access Granted to Student 2");
Blynk.virtualWrite(V2, "Access Granted to Student 2" );
lcd.setCursor(0,0);
lcd.print("Access Done S2 ");
delay(500);
myservo.write(0);
delay(5000);
myservo.write(180);
}

else {
Serial.println("Unregistered user");
Blynk.virtualWrite(V2, "Unregistered user Trying to Access your Door Lock " );
Serial.println("Access denied");
Blynk.virtualWrite(V2, "Access denied");
lcd.setCursor(0,0);
lcd.print("Access Denied ");
delay(500);
}
}
// in Blynk app writes values to the Virtual Pin 3
BLYNK_WRITE(V3)
{
button1 = param.asInt(); // assigning incoming value from pin V3 to a variable

A4
}

// in Blynk app writes values to the Virtual Pin 4

BLYNK_WRITE(V4)
{
button2 = param.asInt(); // assigning incoming value from pin V4 to a variable

BLYNK_WRITE(V5)
{
button3 = param.asInt(); // assigning incoming value from pin V5 to a variable

A5
A2. Programming Code for Face Recognized Door Lock

#include <ArduinoWebsockets.h>
#include "esp_http_server.h"
#include "esp_timer.h"
#include "esp_camera.h"
#include "camera_index.h"
#include "Arduino.h"
#include "fd_forward.h"
#include "fr_forward.h"
#include "fr_flash.h"

const char* ssid = "iot";

const char* password = "iot12345";

#define ENROLL_CONFIRM_TIMES 5
#define FACE_ID_SAVE_NUMBER 7

// Select camera model

//#define CAMERA_MODEL_WROVER_KIT
//#define CAMERA_MODEL_ESP_EYE
//#define CAMERA_MODEL_M5STACK_PSRAM
//#define CAMERA_MODEL_M5STACK_WIDE
#define CAMERA_MODEL_AI_THINKER
#include "camera_pins.h"

using namespace websockets;

WebsocketsServer socket_server;

camera_fb_t * fb = NULL;

long current_millis;
long last_detected_millis = 0;

#define relay_pin 2 // pin 12 can also be used

unsigned long door_opened_millis = 0;
long interval = 5000; // open lock for ... milliseconds

A6
bool face_recognised = false;

void app_facenet_main();
void app_httpserver_init();

typedef struct
{
uint8_t *image;
box_array_t *net_boxes;
dl_matrix3d_t *face_id;
} http_img_process_result;

static inline mtmn_config_t app_mtmn_config()

{
mtmn_config_t mtmn_config = {0};
mtmn_config.type = FAST;
mtmn_config.min_face = 80;
mtmn_config.pyramid = 0.707;
mtmn_config.pyramid_times = 4;
mtmn_config.p_threshold.score = 0.6;
mtmn_config.p_threshold.nms = 0.7;
mtmn_config.p_threshold.candidate_number = 20;
mtmn_config.r_threshold.score = 0.7;
mtmn_config.r_threshold.nms = 0.7;
mtmn_config.r_threshold.candidate_number = 10;
mtmn_config.o_threshold.score = 0.7;
mtmn_config.o_threshold.nms = 0.7;
mtmn_config.o_threshold.candidate_number = 1;
return mtmn_config;
}
mtmn_config_t mtmn_config = app_mtmn_config();

face_id_name_list st_face_list;
static dl_matrix3du_t *aligned_face = NULL;

httpd_handle_t camera_httpd = NULL;

A7
typedef enum
{
START_STREAM,
START_DETECT,
SHOW_FACES,
START_RECOGNITION,
START_ENROLL,
ENROLL_COMPLETE,
DELETE_ALL,
} en_fsm_state;
en_fsm_state g_state;

typedef struct
{
char enroll_name[ENROLL_NAME_LEN];
} httpd_resp_value;

httpd_resp_value st_name;

void setup() {
Serial.begin(115200);
Serial.setDebugOutput(true);
Serial.println();

digitalWrite(relay_pin, LOW);
pinMode(relay_pin, OUTPUT);

camera_config_t config;
config.ledc_channel = LEDC_CHANNEL_0;
config.ledc_timer = LEDC_TIMER_0;
config.pin_d0 = Y2_GPIO_NUM;
config.pin_d1 = Y3_GPIO_NUM;
config.pin_d2 = Y4_GPIO_NUM;
config.pin_d3 = Y5_GPIO_NUM;
config.pin_d4 = Y6_GPIO_NUM;
config.pin_d5 = Y7_GPIO_NUM;

A8
 config.pin_d6 = Y8_GPIO_NUM;
config.pin_d7 = Y9_GPIO_NUM;
config.pin_xclk = XCLK_GPIO_NUM;
config.pin_pclk = PCLK_GPIO_NUM;
config.pin_vsync = VSYNC_GPIO_NUM;
config.pin_href = HREF_GPIO_NUM;
config.pin_sscb_sda = SIOD_GPIO_NUM;
config.pin_sscb_scl = SIOC_GPIO_NUM;
config.pin_pwdn = PWDN_GPIO_NUM;
config.pin_reset = RESET_GPIO_NUM;
config.xclk_freq_hz = 20000000;
config.pixel_format = PIXFORMAT_JPEG;
//init with high specs to pre-allocate larger buffers
if (psramFound()) {
config.frame_size = FRAMESIZE_UXGA;
config.jpeg_quality = 10;
config.fb_count = 2;
} else {
config.frame_size = FRAMESIZE_SVGA;
config.jpeg_quality = 12;
config.fb_count = 1;
}

#if defined(CAMERA_MODEL_ESP_EYE)
pinMode(13, INPUT_PULLUP);
pinMode(14, INPUT_PULLUP);
#endif

// camera init
esp_err_t err = esp_camera_init(&config);
if (err != ESP_OK) {
Serial.printf("Camera init failed with error 0x%x", err);
return;
}

sensor_t * s = esp_camera_sensor_get();
s->set_framesize(s, FRAMESIZE_QVGA);

A9
#if defined(CAMERA_MODEL_M5STACK_WIDE)
s->set_vflip(s, 1);
s->set_hmirror(s, 1);
#endif

WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");

app_httpserver_init();
app_facenet_main();
socket_server.listen(82);

Serial.print("Camera Ready! Use 'http://");

Serial.print(WiFi.localIP());
Serial.println("' to connect");
}

static esp_err_t index_handler(httpd_req_t *req) {

httpd_resp_set_type(req, "text/html");
httpd_resp_set_hdr(req, "Content-Encoding", "gzip");
return httpd_resp_send(req, (const char *)index_ov2640_html_gz, index_ov2640_html_gz_len);
}

httpd_uri_t index_uri = {
.uri = "/",
.method = HTTP_GET,
.handler = index_handler,
.user_ctx = NULL
};

void app_httpserver_init ()

A10
{
httpd_config_t config = HTTPD_DEFAULT_CONFIG();
if (httpd_start(&camera_httpd, &config) == ESP_OK)
Serial.println("httpd_start");
{
httpd_register_uri_handler(camera_httpd, &index_uri);
}
}

void app_facenet_main()
{
face_id_name_init(&st_face_list, FACE_ID_SAVE_NUMBER, ENROLL_CONFIRM_TIMES);
aligned_face = dl_matrix3du_alloc(1, FACE_WIDTH, FACE_HEIGHT, 3);
read_face_id_from_flash_with_name(&st_face_list);
}

static inline int do_enrollment(face_id_name_list *face_list, dl_matrix3d_t *new_id)

{
ESP_LOGD(TAG, "START ENROLLING");
int left_sample_face = enroll_face_id_to_flash_with_name(face_list, new_id, st_name.enroll_name);
ESP_LOGD(TAG, "Face ID %s Enrollment: Sample %d",
st_name.enroll_name,
ENROLL_CONFIRM_TIMES - left_sample_face);
return left_sample_face;
}

static esp_err_t send_face_list(WebsocketsClient &client)

{
client.send("delete_faces"); // tell browser to delete all faces
face_id_node *head = st_face_list.head;
char add_face[64];
for (int i = 0; i < st_face_list.count; i++) // loop current faces
{
sprintf(add_face, "listface:%s", head->id_name);
client.send(add_face); //send face to browser
head = head->next;
}

A11
}

static esp_err_t delete_all_faces(WebsocketsClient &client)

{
delete_face_all_in_flash_with_name(&st_face_list);
client.send("delete_faces");
}

void handle_message(WebsocketsClient &client, WebsocketsMessage msg)

{
if (msg.data() == "stream") {
g_state = START_STREAM;
client.send("STREAMING");
}
if (msg.data() == "detect") {
g_state = START_DETECT;
client.send("DETECTING");
}
if (msg.data().substring(0, 8) == "capture:") {
g_state = START_ENROLL;
char person[FACE_ID_SAVE_NUMBER * ENROLL_NAME_LEN] = {0,};
msg.data().substring(8).toCharArray(person, sizeof(person));
memcpy(st_name.enroll_name, person, strlen(person) + 1);
client.send("CAPTURING");
}
if (msg.data() == "recognise") {
g_state = START_RECOGNITION;
client.send("RECOGNISING");
}
if (msg.data().substring(0, 7) == "remove:") {
char person[ENROLL_NAME_LEN * FACE_ID_SAVE_NUMBER];
msg.data().substring(7).toCharArray(person, sizeof(person));
delete_face_id_in_flash_with_name(&st_face_list, person);
send_face_list(client); // reset faces in the browser
}
if (msg.data() == "delete_all") {
delete_all_faces(client);

A12
 }
}

void open_door(WebsocketsClient &client) {

if (digitalRead(relay_pin) == LOW) {
digitalWrite(relay_pin, HIGH); //close (energise) relay so door unlocks
Serial.println("Door Unlocked");
client.send("door_open");
door_opened_millis = millis(); // time relay closed and door opened
}
}

void loop() {
auto client = socket_server.accept();
client.onMessage(handle_message);
dl_matrix3du_t *image_matrix = dl_matrix3du_alloc(1, 320, 240, 3);
http_img_process_result out_res = {0};
out_res.image = image_matrix->item;

send_face_list(client);
client.send("STREAMING");

while (client.available()) {
client.poll();

if (millis() - interval > door_opened_millis) { // current time - face recognised time > 5 secs
digitalWrite(relay_pin, LOW); //open relay
}

fb = esp_camera_fb_get();

if (g_state == START_DETECT || g_state == START_ENROLL || g_state ==

START_RECOGNITION)
{
out_res.net_boxes = NULL;
out_res.face_id = NULL;

A13
 fmt2rgb888(fb->buf, fb->len, fb->format, out_res.image);

out_res.net_boxes = face_detect(image_matrix, &mtmn_config);

if (out_res.net_boxes)
{
if (align_face(out_res.net_boxes, image_matrix, aligned_face) == ESP_OK)
{

out_res.face_id = get_face_id(aligned_face);
last_detected_millis = millis();
if (g_state == START_DETECT) {
client.send("FACE DETECTED");
}

if (g_state == START_ENROLL)
{
int left_sample_face = do_enrollment(&st_face_list, out_res.face_id);
char enrolling_message[64];
sprintf(enrolling_message, "SAMPLE NUMBER %d FOR %s",
ENROLL_CONFIRM_TIMES - left_sample_face, st_name.enroll_name);
client.send(enrolling_message);
if (left_sample_face == 0)
{
ESP_LOGI(TAG, "Enrolled Face ID: %s", st_face_list.tail->id_name);
g_state = START_STREAM;
char captured_message[64];
sprintf(captured_message, "FACE CAPTURED FOR %s", st_face_list.tail->id_name);
client.send(captured_message);
send_face_list(client);

}
}

if (g_state == START_RECOGNITION && (st_face_list.count > 0))

{
face_id_node *f = recognize_face_with_name(&st_face_list, out_res.face_id);

A14
 if (f)
{
char recognised_message[64];
sprintf(recognised_message, "DOOR OPEN FOR %s", f->id_name);
open_door(client);
client.send(recognised_message);
}
else
{
client.send("FACE NOT RECOGNISED");
}
}
dl_matrix3d_free(out_res.face_id);
}

}
else
{
if (g_state != START_DETECT) {
client.send("NO FACE DETECTED");
}
}

if (g_state == START_DETECT && millis() - last_detected_millis > 500) { // Detecting but no

face detected
client.send("DETECTING");
}

client.sendBinary((const char *)fb->buf, fb->len);

esp_camera_fb_return(fb);
fb = NULL;
}
}

A15