See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/372955685

Arduino Based Real-Time Face Recognition And Tracking System

Article in International Journal of Advanced Trends in Computer Science and Engineering · August 2023
DOI: 10.30534/ijatcse/2023/011242023

CITATIONS READS

3 2,798

2 authors:

Muneera Altayeb Amani Al-Ghraibah

Al-Ahliyya Amman University Al-Ahliyya Amman University
13 PUBLICATIONS 36 CITATIONS 21 PUBLICATIONS 181 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Muneera Altayeb on 07 August 2023.

The user has requested enhancement of the downloaded file.

 ISSN 2278-3091
Volume 12, No.4, July - August 2023
Muneera Altayeb et al., International Journal of Advanced Trends in Computer Science and Engineering, 12(4), July - August 2023, 144 - 150
International Journal of Advanced Trends in Computer Science and Engineering
Available Online at http://www.warse.org/IJATCSE/static/pdf/file/ijatcse011242023.pdf
https://doi.org/10.30534/ijatcse/2023/011242023

Arduino Based Real-Time Face Recognition And

Tracking System
Muneera Altayeb1, Amani Al-Ghraibah2
1
Al-Ahliyya Amman University, Amman, Jordan, [email protected]
2
Al-Ahliyya Amman University, Amman, Jordan, [email protected]

Received Date June 11, 2023 Accepted Date: July 22, 2023 Published Date: August 06, 2023

Biometrics is used in a facial recognition system to map

facial features from a photograph or video. Facial
ABSTRACT recognition is a technology-based method for human face
recognition and matching identification in which the system
People must remember more passwords and carry more compares information to a database of known faces and can
cards these days, as security becomes more important every help verify a person's identity [1],[2]. Near Field
day. However, such implementations are becoming less safe Communication (NFC) technology and a mobile
and feasible, resulting in a surge in interest in biometric application are used in the attendance system as presented
systems techniques. Biometrics systems are those that by Bhise, Abhilash, et al. Each student is given an NFC tag
record the physical characteristics of persons in an with a unique ID during their enrolment into the
electronic environment and use that information to college. The lecturer's cell phone will then be used to track
recognize them when needed. It works on the idea of attendance for each lesson by clicking on or re-positioning
identifying a person's physical traits that he cannot change these tags. The phone's built-in camera will then capture the
student's face, allowing all data to be sent to the server for
and are distinguishable from others. The designed system
confirmation and verification. The advantages of this
aimed to successfully achieve face recognition through real-
technology are the ease of use of NFC as it creates a fast
time detection, comparing the result with pre-recorded face connection between colleges and significantly reduces the
images and tracking the human’s face using a webcam- time it takes for students to attend. The professor also found
based system. The system accuracy has been tested in the convenience of a system that uses the lecturer's phone
different cases; based on the detection distance, person’s as an NFC reader, inconvenient. [3].
appearance variation, multi-angle posing and multi-people.
The system successfully implements face recognition, Priambodo, et. al introduced Face Tracking for Flying
detection, and tracking with a satisfactory accuracy of 93%. Robot which is created to follow a face object. The system
is based on 2 stages: the face recognition stage and the
Key words: Artificial Intelligence, Image processing, Face position control system. Haar cascade approach is used as a
recognition, OpenCV. face detection algorithm which is implemented using
Python and OpenCV computer languages. A camera is used
1. INTRODUCTION and installed on DJI Tello mini drone. and A Laptop is
connected via Wi-Fi; the camera transmits the captured
The majority of us use smartphones, which frequently use image of the detected person’s face to the laptop to be
face recognition technology to unlock the device. This processed using a computer vision algorithm. There are two
technology provides a powerful approach to protect basic network streams; It is a two-way duplex
personal data and ensure that sensitive data stays communication for sending and receiving requests between
unavailable to the criminal even if the phone is stolen. Face the laptop and the drone, and a one-way, half-duplex
recognition technology is used in many areas, such as connection for streaming video from the drone to the laptop
safety, security, and payments. Face recognition refers to connection. The face detection system is written in Python
the difficulty of recognizing or validating a person in a and uses the Open-CV and djitellopy libraries. The location
digital image or video frame based on the facial biometric of the drone is then compared to the fixed point in the center
pattern. To authenticate a person, the system acquires a set of the picture to determine inaccuracies, allowing control
of unique biometric data points related to facial expressions. signals for up/down, forward/backward, and right/left
face recognition is a type of personal identification system motions to be determined [4].
that uses a person's personal traits to identify them. The
process of human face recognition is divided into two Bhaganagare, Bhakti, and Avinash D. Harale suggested in
phases: face detection, which occurs relatively quickly in their research a security system that uses iris recognition to
humans unless in situations when the item is positioned at a identify the person. Iris recognition is a biometric access
short distance away, and the second phase recognizes a face control system that leverages the most unique feature of the
as an individual (Identifying the individual). human body, and it is used in automated border crossings,
national ID systems, and other applications. The suggested
technology displays information about recognized people

144
Muneera Altayeb et al., International Journal of Advanced Trends in Computer Science and Engineering, 12(4), July - August 2023, 144 - 150

on an LCD, allowing users to enter the restricted area while successfully by detecting a human face in real-time,
remaining anonymous. The iris recognition data of the comparing the result with pre-recorded face images and
people are stored on personal computers and sent to the tracking the human’s face using a webcam-based system.
microcontroller through serial interface RS232, and used This system consists of Arduino Nano as a controller, a
for serial communication between the PC and the webcam, two servo motors to move the webcam, and a PC.
microcontroller. The DC power supply required for the The PC is used to create a database for facial images. This
system and the Liquid Crystal Display (LCD) is used to database is implemented using Python and PyCharm
display the person’s status [5]. Integrated Development Environment (IDE).

Setyawan, et. al proposes a system to make facial tracking 2. HARDWARE REQUIREMENTS

resistant to occlusion and background characteristics. This
face-tracking system is based on an enhanced Mean Shift In this project, the Arduino is the microcontroller of the
(MS) algorithm which is used to monitor the face as a target system that performs all the decisions such as determining
and enhance the toughness of occlusion. Face detection and the direction of movement of the camera. The system is
tracking are part of this algorithm which is Viola-Jones. The provided with two servo motors connected to the Arduino,
MS tracking technique is used to decrease occlusion, while one of which is responsible for horizontal movement and
corrected background-weighted histogram (CBWH) is the other for vertical movement. Figure 1 describes the
utilized to reduce background interference; CBWH is a system block diagram while Figure 2 shows the servo
motors connection. The basic elements required for this
method used to increase MS tracking capability in object
project are listed below
tracking location mode due to background noise [6].
Zarkasi, Ahmad, et al presented in their paper a technique  Arduino Nano
for face recognition based on auto-extracted facial marks.  USB Webcam
It’s based on extracting the face landmarks and a triplet of  Servo motor
regions and their related geometric invariance. The points  Computer
on the outer corners of the eyes and nose will subsequently  System chassis
be joined with lines to form a triangle then, the triangle's
area is calculated. The gathered data is then trained to detect
faces using the Support Vector Machine technique. This
technology is implanted into a mobile robot using
Raspberry Pi. The robot camera is mounted to a Pan Tilt
servo, which serves as the robot's neck. If the camera detects
a face, the robot calculates the area of the detected face
rectangle and the center point of the rectangle to determine
where it will go. The faces captured on camera are used as
input, then, the picture is processed for face detection using
the Haar Cascade approach. Then, the feature extraction
using the Facial landmarks detection technique is Figure 1: The system block diagram
performed, and the feature values acquired are compared to The servo motors need 5 Volt to operate. Due to this, 220
a feature database previously trained using the Support Volts (AC) adapter is used which converts the AC power
Vector Machine approach. The recognized face is the result from the source and into DC power by decreasing the 220
of the software development [7]. Volts level and provides 12 Volts DC. Which then stepped
Lian, Zhichao, et al introduced in their work a real time face down to 5 Volts by a Buck converter (LM2596) to provide
tracking system based on multiple information fusion, the Arduino and the servo motors with the needed voltage.
which uses a multiple object tracking. Face detection is The buck converter is a DC-DC converter that provides an
handled using a multi-task convolutional neural network output voltage that is lower than the input voltage; it
(MTCNN). The appearance, motion, and shape features are converts the uncontrolled DC supply to a lower output
used for tracking to deal with tracking failures that caused voltage that is stabilized. The power supply circuit shown
by object occlusion or fast object movement. The weights in Figure 3.[9]-[11]
of different characteristics for feature fusion are adjusted
depending on the tracking condition The approach of
modifying feature weights based on scenes may handle
problems like continuous tracking, interruptible tracking,
and object interaction [8].
The novelty of this work lies in the mechanism of building
and operating the real-time face recognition and tracking
system that is guided by a new combination of Android
application, Arduino, and digital image processing, The
designed system aimed to achieve face recognition Figure 2: Schematic of the servo motors connection

145
Muneera Altayeb et al., International Journal of Advanced Trends in Computer Science and Engineering, 12(4), July - August 2023, 144 - 150

The LM2596 buck converter is used in the project to 3.1 System flow chart
stabilize the output voltage to 5V, it has excellent load and
line regulation. Which has a high-precision potentiometer, The flow chart in Figure 5. describes the function of the
and is capable of efficiently driving a load of up to 3A as system. When the Arduino is started, the system will receive
shown in Figure 3. . commands from the computer via a USB connection, and
then the camera will start capturing clips as the system
detects faces in the captured images, then the system applies
an algorithm to control the movement of camera to track
faces according to the images stored in the database in the
system.

Figure 3: The power supply implemented circuit

3. SOFTWARE FRAMEWORK
In this part, the system is configured so that the Arduino is
connected to the computer by an integrated development
environment (IDE). Figure 4 shows the Arduino
connections, so that the commands are sent from the
computer to the Arduino, the steps for configuration the
Arduino Nano are listed as follows[12]-[18]:

 Step 1: Installing Python on the Computer

 Step 2: Install PySerial
PySerial is a Python API module which is used to
read and write serial data to Arduino or any other
Microcontroller.
 Step 3: Python Code
First up, to get the Python sending data over the
serial port, a simple program is needed.
 Step 4: Arduino Code
To initiate a connection with the Arduino from
Python, user should first figure out which COM Port
the Arduino is on.

Figure 5: System general working flowchart.

The flowchart of controlling the servo motors by the

Arduino is shown in Figure 6 and it’s described as follow:

- If the Arduino receives “+” command, then it will pan the

“pan- servo motor” 1 degree to the right.

- If the Arduino receives “-” command, then it will pan the

“pan- servo motor” 1 degree to the left.

- If the Arduino receives “U” command, then it will tilt the

“tilt- servo motor” up with 1 degree.

- If the Arduino receives “D” command, then it will tilt the

Figure 4: Arduino connection “tilt-servo motor” down with 1 degree.

146
Muneera Altayeb et al., International Journal of Advanced Trends in Computer Science and Engineering, 12(4), July - August 2023, 144 - 150

- If the Arduino receives “C” command, then it will return

both the servos to the origin.

Figure 6: Servo motors control Flowchart

3.2 Image Processing Figure 7: Face detection and labeling

Image processing is accomplished via program PyCharm on

the PC using Python. PyCharm is used in this project as an
IDE to create Python-based database. This is the last part of
implementing the project. It starts with downloading
PyCharm program on the PC [19]-[22]. Then the images’
database has been created, To create the facial images
database, we should first create a folder that would include
the images, then this folder’s images will be encoded into
the system[23] –[25]. Feed the frame to the system where
the system runs an algorithm to detect the location of the
face, to obtain the coordinates of the face in the frame and
then the system applies the recognition steps as shown in
Figure 8-b/c/d. After that a rectangle appears around the
face and the name of the detected person as shown in figure
7-(e) or or “unknown” If the detected person is unknown,
then the system checks if the image location is larger than Figure 8: System prototype.
the center, it would move to the right. otherwise, it would
move to the left by sending serial commands (up, down, left,
or right). 4.1 Face recognition accuracy based on detection
distance
4. RESULT AND DISCUSSION
The aim of the system is to match a name to a specific face.
This system is really special as computer vision and First, the system has been tested 10 times to determine the
artificial intelligence are used to detect and track faces. The algorithm accuracy at recognizing one person from a data
accuracy of the system has been tested with a number of
set that contains multiple images of many people based on
conditions such as distance, appearance variation, and
the detection distance. The result of each trial is expressed
disguise. The final System prototype of this project is
shown in Figure 8. in Table 1. The face recognition accuracy can be calculated
as the percentage of the recognized faces per the total
number of tested faces of the same person/persons. In the

147
Muneera Altayeb et al., International Journal of Advanced Trends in Computer Science and Engineering, 12(4), July - August 2023, 144 - 150

face recognition test based on the detection distance, the Table 2: Accuracy obtained based on person’s appearance
face recognition accuracy can be measured using eq. 1.: variation
𝑛𝑜.𝑜𝑓 𝑐𝑜𝑟𝑟𝑒𝑐𝑡𝑙𝑦 𝑟𝑒𝑐𝑜𝑔𝑛𝑖𝑧𝑒𝑑 𝑓𝑎𝑐𝑒 Appearance variation Result Face Recognition
Accuracy = *100% (1) Accuracy
𝑡𝑜𝑡𝑎𝑙 𝑛𝑜.𝑜𝑓 𝑓𝑎𝑐𝑒 𝑡𝑒𝑠𝑡𝑒𝑑

Without any additions 10/10 100%

Table 1: Accuracy obtained based on detection distance
With Eyeglasses & cap 9/10 90%
Trial # Detection Distance Result
With Eyeglasses only 10/10 100%
1 0.5 meter succeeded in
recognizing With cap only 10/10 100%

2 1.5 meters succeeded in With sunglasses only 9/10 90%

recognizing
Accuracy average 90%
3 2.5 meters succeeded in
recognizing

Face recognition success result 10/10

When a person is in close proximity to the Cam (0.5 , 1.5,

2.5 ) meter away, the system succeeds in detecting his face
and recognizing it as shown in Figure 9 .

Figure10: (a)With Eyeglasses & cap, (b) With Eyeglasses only,

(c) With cap only (d) With sunglasses only
.
4.3 Face Recognition Accuracy Based on Multi-Angle
Pose
The system has been tested based on multi-angle pose as
shown in Figure 11, each pose has been tested 10 times to
determine the algorithm accuracy at recognizing one person
Figure 9: (a) Person is 0.5 meter away ,(b) Person is 1.5 meter in the image. The result of this test is expressed in Table 3.
away, (c) Person is 2.5 meter away
Table 3: Accuracy obtained based on multi-angle pose
Angle Pose Result Face Recognition
4.2 Face recognition accuracy based on a person's Accuracy
appearance variation
90 ̊ 10/10 100%
The system has been tested based on a person’s appearance
variation, each change has been tested 10 times to determine 135 ̊ 10/10 100%
the algorithm accuracy at recognizing one person in the 180 ̊ 6/10 60%
image. The result of this test is expressed in Table 2. Figure
10 shows different cases tested according to a person’s Accuracy average 86.67%
appearance variation.

148
Muneera Altayeb et al., International Journal of Advanced Trends in Computer Science and Engineering, 12(4), July - August 2023, 144 - 150

5. CONCLUSION
This work presented the process of designing Real Time
Face Recognition and Tracking System. As this system
works on comparing the result with pre-recorded face
images and tracking the human’s face using a webcam-
based system. The novelty of this work lies in the idea of
building and operating a different and unique system for
real-time face recognition and tracking that is guided by a
new combination of Android applications, Arduino and
digital image processing, as well as system is fully
functional and built from readily available components. It is
worth noting that this system can be used in various
applications such as surveillance systems at airports and the
classroom attendance system, as well as many security and
military uses. The system has been implemented and tested
in different cases with a number of conditions such as
distance, appearance variation and disguise. It proved that it
has high accuracy in detecting individuals’ faces recognize
and tracking them; the performance was about 93%.

6. ACKNOWLEDGMENT
Special thanks to our students Mohammad Abulhasan
Figure 11:Face recognition test when the posing (a) Angle ,Jassem Ibrahim Alsairafi and Ali Taleb Alblushi for
90̊ (b) Angle is 135̊ ,(c) angle 180̊. contributing to this work by performing experiments to
validate the accuracy of the system.
4.4 Face Recognition Accuracy of Multi-People in One
Image REFERENCES
In this case, the system has been tested when multi people 1. Bhise, A., Khichi, R., Korde, A., & Lokare, P. D.
are in one image, each case has been tested 10 times to (2015). Attendance system using NFC
determine the algorithm accuracy at recognizing faces in
technology with embedded camera on mobile
multi-people images. Figure 12 shows the system can
successfully recognize the faces of multiple people in one device. International Journal of Advanced
image. The two people are known; they are included in the Research in Computer and Communication
facial image dataset. The results of this test are expressed. Engineering, 4(2), 350-353.
Table 4: Accuracy obtained based on in case of multi-people 2. Minaee, S., Luo, P., Lin, Z., & Bowyer, K. (2021).
Going deeper into face detection: A
Number of persons Result Face Recognition
Accuracy
survey. arXiv preprint arXiv:2103.14983.
3. Selvi, K. Senthamil, P. Chitrakala, and A. Antony
2 known persons 10/10 100% Jenitha. "Face recognition based attendance
1 known person with 1 10/10 100% marking system." International Journal of
unknown person Computer Science and Mobile Computing 3.2
(2014): 337-342
3 persons: 2 known, 1 8/10 80%
unknown 4. Abioye, David & Omotehinwa, Temidayo
Oluwatosin. (2018). Design and Implementation
Accuracy average 93.3%
of a Fingerprint Based Identity Management
System. 7. 75-80.
5. Devi, N. S., & Hemachandran, K. (2017). Content
based feature combination method for face
image retrieval using neural network and SVM
classifier for face recognition. Indian Journal of
Science and Technology, 10(24), 1-11.
6. Priambodo, A. S., Arifin, F., Nasuha, A., &
Winursito, A. (2021, November). Face Tracking
for Flying Robot Quadcopter based on Haar
Cascade Classifier and PID Controller.
In Journal of Physics: Conference Series (Vol.
Figure 12: 2 people known, 1 unknown person 2111, No. 1, p. 012046). IOP Publishing.

149
Muneera Altayeb et al., International Journal of Advanced Trends in Computer Science and Engineering, 12(4), July - August 2023, 144 - 150

7. Ghoshal, A. M., Aspat, A., & Lemos, E. (2021). 18. Ravishankar, V., Vinod, V., Kumar, T., & Bhalla,
OpenCV Image Processing for AI Pet K. (2022). Sensor integration and facial
Robot. International Journal of Applied Sciences recognition deployment in a smart home
and Smart Technologies, 3(1), 65-82. system. In Proceedings of the 2nd International
8. Zarkasi, A., Abdau, F., Anda, A. J., Nurmaini, S., Conference on Recent Trends in Machine
Stiawan, D., Suprapto, B. Y., ... & Kurniati, R. Learning, IoT, Smart Cities and Applications:
(2022). Implementation of Facial Landmarks ICMISC 2021 (pp. 759-771). Springer Singapore.
Detection Method for Face Follower Mobile 19. Reitz, K., & Schlusser, T. (2016). The
Robot. Generic, 14(1), 19-24. Hitchhiker's guide to Python: best practices for
9. Altayeb, M., & Al-Ghraibah, A. (2022). Voice development. " O'Reilly Media, Inc.".
controlled Camera Assisted Pick and Place 20. El Mrabet, A., Tber, A., Benaly, M., Hlou, L., &
Robot Using Raspberry Pi. Indonesian Journal of El Gouri, R. (2023). Unlocking Doors: A
Electrical Engineering and Informatics TinyML-based Approach for real-time Face
(IJEEI), 10(1), 51-59. Mask Detection in Door Lock
10. Renuka, B., Sivaranjani, B., Lakshmi, A. M., & Systems. Indonesian Journal of Electrical
Muthukumaran, D. N. (2018). Automatic Enemy Engineering and Informatics (IJEEI), 11(2), 515-
Detecting Defense Robot by using Face 527.
Detection Technique'. Asian Journal of Applied 21. ben Abdel Ouahab, I., Elaachak, L., Bouhorma,
Science and Technology, 2(2), 495-501. M., & Alluhaidan, Y. A. (2021, June). Real-time
11. Reddy, M. R., & SreelekhaAssistant, M. IOT facemask detector using deep learning and
BASED FACE RECOGNITION AND DOOR raspberry pi. In 2021 International Conference on
LOCK SYSTEM USING ARDUINO. Digital Age & Technological Advances for
12. Hughes, J. M. (2016). Arduino: a technical Sustainable Development (ICDATA) (pp. 23-30).
reference: a handbook for technicians, IEEE.
engineers, and makers. " O'Reilly Media, Inc.". 22. Ismael, K. D., & Irina, S. (2020). Face recognition
13. Rathnala, P., Patnaik, M. P. K., Sura, S. R., Prasad, using Viola-Jones depending on
B., Siva, N., Rao, M., & Rayan, D. N. Design of Python. Indonesian Journal of Electrical
an Efficient Face Recognition system using Engineering and Computer Science, 20(3), 1513-
Deep Learning Technique. 1521.
14. Almufti, S. M., Marqas, R. B., Nayef, Z. A., & 23. Su, X., Gao, M., Ren, J., Li, Y., Dong, M., & Liu,
Mohamed, T. S. (2021). Real time face-mask X. (2022). Face mask detection and
detection with arduino to prevent covid-19 classification via deep transfer
Spreading. Qubahan Academic Journal, 1(2), 39- learning. Multimedia Tools and Applications, 1-
46. 20.
15. Joshi, P. (2015). OpenCV with Python by 24. Zheng, G., & Xu, Y. (2021). Efficient face
example. Packt Publishing Ltd. detection and tracking in video sequences based
16. Hasan, R. T., & Sallow, A. B. (2021). Face on deep learning. Information Sciences, 568, 265-
Detection and Recognition Using 285.
OpenCV. Journal of Soft Computing and Data 25. Mahbub, U., Sarkar, S., & Chellappa, R. (2019).
Mining, 2(2), 86-97. Partial face detection in the mobile
17. Dinalankara, L. (2017). Face detection & face domain. Image and Vision Computing, 82, 1-17.
recognition using open computer vision
classifies. ResearchGate.

150
View publication stats