Drowsiness Detection using ESP32CAM

Kartikay Chiranjeev Gupta Gaganjot Kaur Ankit Chaudhary

dept. of Mechatronics Engineering dept. of Mechatronics Engineering dept. of Mechatronics Engineering
Chandigarh University Chandigarh University Chandigarh University
Punjab, India Punjab, India Punjab, India
[email protected] [email protected] [email protected]

Abstract—According to the National Highway Traffic Safety I. INTRODUCTION

Administration (NHTSA), vehicular crashes result in an estimated
1550 deaths, 71000 injuries, and 12.5 billion dollars in monetary Every age group, gender, color, and degree of literacy
losses each year. Developing an affordable, convenient-to-use, easy- has drowsy drivers. There are several categories at a higher
to-install, and non-wearable system to monitor drowsiness and fatigue risk of drowsy driving, including younger drivers, persons
may provide employers a way to address drowsy driving and with untreated sleep problems, and shift workers. While
workplace fatigue, this will not only prevent fatigue-related accidents certain groups are more frequently drowsy drivers than
but positively affect road safety. To address this issue, the system others, once sleep-deprived they all have similar accident
works by monitoring the driver’s eye, closing the eyes for too long
risks. Personnel in public safety, such as paramedics and
triggers an alarm for a few seconds, if the alarm gets triggered 3 times
in a row, it will continuously go on until the device is either rebooted emergency medical technicians, is one area of special
or switched off. The system consists of ESP32CAM for video concern.
transmission over Wi-fi, and a computer for detection, classification, The safety, health, and overall quality of life of people are
and alarming process.
all negatively impacted by drowsiness. Negative effects may
Keywords—Drowsiness-detection, Eye-detection, Internet of result from lack of sleep brought on by a newborn infant
Things, Public safety personnel, non-wearable technology, Machine waking every couple of hours, a late or lengthy shift at
learning. work, staying up late with friends, or a long and continuous
journey for the holidays. These include diminished cognitive
ability and performance, traffic accidents, employment
mishaps, and health effects.[1]
In an NHTSA-sponsored poll of 4,010 drivers, 4.2% of
those who were 18 years of age or older reported falling
asleep behind the wheel in the previous month, and 11% in
the previous year. [2] A Gallup poll conducted in 2002 on
behalf of the NHTSA found that 95% of respondents viewed
tiredness as a danger to their safety. According to the AAA's
2014 Traffic Safety Culture Index, people think driving is
inappropriate when feeling drowsy or having trouble
keeping their eyes open. Despite the results of all these
polls, over 29.4% of drivers said that, within the previous 30
days, they had to drive a car even if it was difficult to keep
their eyes open. A fifth of those surveyed (almost 19.8%)
claimed to have done this more than once, and 2.4% claimed
to have done it frequently or even routinely. [3]
To identify sleepiness and exhaustion, several methods
based on various approaches are available. While some of
these technologies are external devices that require
additional study to make them effective, accurate, and
simple to use, others are integrated inside many expensive
automobiles, making them more expensive and more
difficult to acquire.
II. RELATED WORK
Some technologies used to prevent accidents in
automobiles caused by drowsiness:
 Electronic stability control (ESC): Electronic stability
control systems use automated computer-assisted
braking separately for each wheel to help the driver in
maintaining control of the car in abnormal driving
situations. [4]
  Blind-spot monitoring: Blind-spot monitoring (BSM) III. SYSTEM DESIGN
helps to keep an eye on the space around the rear As machine vision continues to grow, we can automate
quarter areas of your vehicle. almost everything today, instead of writing complex
 Mercedes: Attention Assist: Works by capturing the algorithms and deploying a large number of sensors
driving pattern of the driver in the first few minutes of everywhere, we decided to use machine learning and
driving the algorithm of Attention Assist creates an address the problem of drowsiness and fatigue detection
individual profile for every driver’s driving style using ESP32CAM and Convolution Neural network.
which is then constantly compared to the latest
To make the system easy and convenient to deploy (in
feedback it receives from the sensors.
cars, industries, etc.), the Wi-fi and Bluetooth capabilities of
 Nissan: Driver Attention Alert: When the car is
ESP32CAM are used to transmit the video frames wirelessly
moving above the speed of 60 km/h, the system starts
to the image processing end as shown in Fig. 3.1. Since the
to monitor steering behavior to detect abnormal
activities and movements using a statistical analysis system monitors the eyes, it is sensitive to dark
of steering correction errors. The system creates a environments. To tackle this issue, a light-dependent resistor
profile for every different driver. module is used to monitor the light intensity of the
 Vigo: This Bluetooth headset tracks eye blink patterns environment and trigger an LED to maintain proper light
to determine a user's level of alertness. To quantify intensity for eye detection. To adjust the threshold of the
differences in your blinks, Vigo tracks over 20 light-dependent resistor a potentiometer is provided, this
factors, including blink rates, blink durations, and potentiometer is accessible to the user to change the light
drooping eyelids. sensitivity of the device. The system aims to provide a
 Although Vigo’s solution may sound very similar to convenient-to-use, easy-to-install, and non-wearable
this project, this product is not very convenient to use solution for preventing drowsiness and fatigue-related
and only works if the user is wearing the headset. accidents. The system design can be divided into two parts:
Hence making a non-wearable system with advanced A. Workflow at Video Streaming end:
classification techniques is needed. When the system is started it creates a WIFI server
object and connects to the LAN network via network
Concerning the drowsiness detection system, several name and SSID, after the dynamic IP address is assigned
researchers have proposed their methodologies, some of to the system, it advertises it using Bluetooth for 10
which are mentioned below: seconds (indicated by the built-in LED of ESP32CAM at
 This method, proposed by Alireza, et al [5] GPIO 33), within these 10 seconds the client will scan for
demonstrated the use of algorithms to effectively the available Bluetooth devices and the IP address (which
classify drowsiness using eye blink frequency. The will be used to stream video frames) will be visible as
Drowsiness of a driver is directly proportional to the host’s Bluetooth-name. Once the Bluetooth advertisement
eye blink frequency. is closed after 10 seconds, the system will keep listening
 This method was proposed by S. Mehta, et al [6] and to requests for video frames and monitoring the light
employs extracting facial features from video frames intensity of the environment. We have used Python’s
and feeding them to a machine learning algorithm. built-in socket library for creating a connection between
The system can be deployed on an Android the client and host.
application, making it less practical to use. B. Workflow at Video processing end:
Drowsiness and fatigue are serious concerns, whether The processing-side first scans for all the available
while driving or at the workplace. Although there exist Bluetooth devices, among these devices the device whose
many solutions to address this issue, none of them are name starts from “K—G<Dynamic IP address>” is
affordable for everyone and lack accuracy, resulting in fatal considered and the “K—G” substring is removed, this
accidents and billions of dollars of monetary losses every gives the processing end the IP address and this address
year. will be used to get video frames from ESP32CAM server.
Once the connection between the processing end and
server is made, the processing end starts receiving video
frames every one-tenth of a second (to avoid too much
frame buffer). The received video frames are then converted
to grayscale using the open-cv library to make the image 2-
dimensional (to reduce computation requirement for fast
image processing), the images are then fed to the eye
detection CNN model (also available in open-cv), which
returns a list containing coordinates of all possible eyes
detected in the image. The eye portion from the image is
extracted and gets resized to 24x24. This resized eye image
is then fed to the second CNN classifier [7] which returns if
the eye is open or closed, if the eye is classified as CLOSE
the score gets incremented by 1, else gets decremented by 1.
If the drowsy score is more than the set threshold value,
an alarm is triggered and goes off after a few seconds, but if
the alarm gets triggered 3 times in a row, the alarm cannot
be turned off until the system is rebooted or switched off.
Since the system works in proper light intensity and light
direction, a 5mm white LED is provided in the system
interfaced with an LDR sensor module (refer Fig. 3.2 (a),
3.2 (b) for structure and dimension of the system). If the
light intensity drops below 20 lx (this threshold is adjustable
via the potentiometer situated on top of the system) the LDR
outputs a HIGH signal, the flashlight is set HIGH, LOW
otherwise. The overall flowchart and system construction
are shown in Fig.3.3 and Fig. 3.4 respectively.

Fig.3.1 System workflow

Fig. 3.3 Flowchart of the System

Fig. 3.2 (a) Structure and Dimension (in cm)

Fig. 3.4 System components

IV. RESULTS
After testing the system in different environments and
times (Table-1), it can be concluded that the whole system
works properly under an environmental constraint that is
Fig. 3.2 (b) Exploded view of the structure
sufficient light intensity (more than 20 lx) and incident
direction. The assembled view of the system in normal and
low light intensity is shown in Fig. 4.1(a) and Fig. 4.1(b).
As tested, the ESP32CAM captures proper images in a
sufficient light-intensity environment. It is recommended to
use the system with light intensity 80-100 lx incident on the
face. The ESP32CAM consumes nearly 200mA when used
with the camera and WIFI. Since we are using 4 AA
batteries, we can get roughly 3600mAh (900mAh x 4)
 3600 mAh
battery life= preventing drowsiness-related accidents and improving
200 mA safety in both driving and occupational settings.
¿ 18 hours of battery life
This time is theoretical, in practical use (including losses
due to resistance) you can expect nearly 12 hours of battery VI. FUTURE SCOPE
life from the device. We have used 4 AA batteries for Following are a few suggestions that can improve the
powering ESP32CAM, these batteries are connected in system performance based on the deviation noted:
series and contribute a total of 6 Volts, since this is supplied  Use of Infrared Light: Using a 5mm White Flash
to the 5V pin of ESP32CAM, there is a voltage regulator on- LED is inconvenient as it makes it difficult to see
board which step-down this voltage to 5 Volts. Due to this properly when the LED flashes on the user’s face
step-down operation, heat is generated: directly. To address this issue, Infrared light can be
used. But to implement this solution we need to
Power=voltage difference x current = (6V – 5V) x change the camera OVA2640 (which is by default
200mA = 0.2 W
used by ESP32CAM) because it has a built-in
This power is dissipated as heat. We have used a heatsink on infrared filter, making it incompatible with infrared
the voltage regulator to dissipate this heat properly. light.
 Adjustable camera position: Since the camera is fixed
at an angle of 78 degrees, this limits the system and
user position relative to each other. To address this
issue, a screw-based adjustable hinge can be used
which provides a wider range of angles for the camera
it can be set to, making it more versatile to use in
every environment.
 Application-based dataset: Current datasets consist of
images collected during the day or in well-lit
environments which may not accurately reflect the
Fig. 4.1 (a) at normal light intensity (b) at low light intensity real-world scenarios of driving at night. Including
images captured in low-light conditions or with IR
TABLE – I PERFORMACE OF THE SYSTEM cameras can improve the system's accuracy to work at
Trial no. Time Accuracy night.
1 Morning 93.4 % REFERENCES
2 Afternoon 96.6 % [1] Timothy Stewart, ‘Overview of Motor Vehicle Crashes in 2020’
(Report No. DOT HS 813 266), National Highway Traffic Safety
3 Evening 82.5 % Administration, 2022.
[2] Anne G. Wheaton, et al., ‘Drowsy Driving — 19 States and the
4 Night 72.9 % District of Columbia, 2009–2010’ (Report No. Vol. 61 / Nos. 51 & 52),
MMWR, January 4, 2013.
[3] National Highway Traffic Safety Administration, ‘Asleep at the Wheel
V. CONCLUSION - A National compendium of efforts to eliminate Drowsy driving’
In conclusion, the presented drowsiness detection system (Report No. DOT HS 812 352), National Highway Traffic Safety
Administration, 2017.
using ESP32CAM and convolutional neural networks
[4] Chris Hardesty, ‘Electronic Stability Control: Everything You Need to
(CNN) offers a promising solution to address the critical Know’, Kelley Blue Book. https://www.kbb.com/car-advice/electronic-
issue of drowsy driving and workplace fatigue. The system, stability-control/ (accessed Mar. 02, 2023).
by monitoring the driver's eyes can detect drowsiness based [5] S. Mehta, S. Dadhich, S. Gumber, and A. J. Bhatt, ‘Real-Time Driver
on eye closure duration. If the alarm is triggered repeatedly, Drowsiness Detection System Using Eye Aspect Ratio and Eye
Closure Ratio’, SSRN Electronic Journal, 2019.
indicating persistent drowsiness, it continues until the device [6] M. Poursadeghiyan, A. Mazloumi, G. Saraji, mohammad mehdi
is rebooted or switched off. The performance evaluation of Baneshi, A. Khammar, and M. Ebrahimi, ‘Using Image Processing in
the system showcased promising results in environments the Proposed Drowsiness Detection System Design’, Iran J Public
Health, vol. 47, pp. 1370–1377, Apr. 2018.
with sufficient light intensity and incident direction,
[7] Dustin Stewart, ‘Preventing Drowsy-Driving Accidents Using
emphasizing the importance of affordable and reliable Convolutional Neural Networks,’ Towards Data Science, Apr 13,
solutions in enhancing road safety and overall work safety 2021. [Online]. Available: https://towardsdatascience.com/drowsiness-
well-being. detection-using-convolutional-neural-networks-face-recognition-and-
tensorflow-56cdfc8315ad
This research paper provides foundations for developing
an end-consumer product that can significantly contribute to