DRIVER AWAKENING SYSTEM USING IOT

A PROJECT REPORT

Submitted by

N. S. SALMAN FARDEEN 812419104055

S. SHAMEER AHAMED 812419104058

U. SIVARAMAN 812419104065

in partial fulfillment for the award of the

degree of

BACHELOR OF ENGINEERING

in

COMPUTER SCIENCE AND ENGINEERING

M.I.E.T. ENGINEERING COLLEGE, TRICHY – 620007

ANNA UNIVERSITY :: CHENNAI – 600 025

JUNE 2022
DRIVER AWAKENING SYSTEM USING IOT

A PROJECT REPORT

Submitted by

N. S. SALMAN FARDEEN 812419104055

S. SHAMEER AHAMED 812419104058

U. SIVARAMAN 812419104065

in partial fulfillment for the award of the

degree of

BACHELOR OF ENGINEERING

in

COMPUTER SCIENCE AND ENGINEERING

M.I.E.T. ENGINEERING COLLEGE, TRICHY – 620007

ANNA UNIVERSITY :: CHENNAI – 600 025

JUNE 2022

i
 ANNA UNIVERSITY : CHENNAI 600 025

BONAFIDE CERTIFICATE

Certified that this project report “DRIVER AWAKENING SYSTEM USING IOT” is

the Bonafide of “N.S.SALMAN FARDEEN” (812419104055), “S.SHAMEER

AHAMED” (812419104058), “U.SIVARAMAN” (812419104065) who carried out the

project under the supervision.

SIGNATURE SIGNATURE

Mrs.G.NALINA KEERTHANA M.E., Mrs.G.NALINA KEERTHANA M.E.,

HEAD OF THE DEPARTMENT SUPERVISOR

ASSISTANT PROFESSOR

Department of Computer Science and Department of Computer Science and

Engineering Engineering

M.I.E.T. Engineering College, M.I.E.T. Engineering College,

Trichy – 620007 Trichy – 620007

Submitted for the viva-voce held on ___________________

INTERNAL EXAMINER EXTERNAL EXAMINER

ii
 ACKNOWLEDGMENT

First of all we thank God for this shower of blessing and his divine help which
enables us to complete the project successfully.

We extend our sincere thanks to Alhaj.Janab.Er.A.MOHAMED YUNUS,

B.E., M.Sc., (Engg.) Founder & Chairman of M.I.E.T Engineering College,
Trichy for offering the means of attaining our most cherished Goal
Environment.

We extend our deepest gratitude to Principal Dr.A.NAVEEN SAIT,M.E;

Ph.D. M.I.E.T. Engineering College, Trichy, for giving us permission to do
the project work successfully.

It gives immense pleasure to extend my sincere and heartfelt gratitude to our

project Guide Mrs.G.NALINA KEERTHANA M.E, Assistant Professor &
HOD Department of CSE for her untiring valuable and timely suggestions in
dispensable situation during the period of study.

We are extremely thankful to our parents for enlightening us by providing

Professional education and for their prayerful support that makes us to
complete.

Also heartfelt thanks to our friends, Teaching and Non-teaching staff

members who helped us to finish the project successfully.

iii
 ABSTRACT

Driver drowsiness has currently been a severe issue in threatening road

safety. Hence it is vital to develop an effective drowsiness recognition

algorithm to avoid road accidents. However, recognizing drowsiness is still

very challenging, due to the large intra-class variations in facial expression,

head pose and illumination condition. A new deep learning framework based

on the hybrid Artificial intelligence (AI) which refers to the simulation of

human intelligence in machines that are programmed to think like humans and

mimic their actions. A real-time driver drowsiness detection and traffic

analysis system is used for driving safety. Based on computer vision

techniques, the driver’s face is located from a color video captured in a car.

Then, face detection is employed to locate the regions of the driver’s eyes,

which are used as the templates for eye tracking in subsequent frames. Finally,

the tracked eye’s images are used for drowsiness detection in order to generate

warning alarms. This system also contains an IOT device, which is used to

wake the driver up if he/she has drowsed. This concept of taking necessary

steps immediately will reduce the accident levels in a higher ratio.

iv
 TABLE OF CONTENTS

TITLE
CHAPTER NO PAGE NO.
ABSTRACT iv
LIST OF FIGURES ix
1 INTRODUCTION 1
2 LITERATURE SURVEY 6
2.1 IMPACT OF THE FACE ANGLE TO
TRAVELING TRAJECTORY DURING 6
THE RIDING STANDING-TYPE
PERSONAL MOBILITY DEVICE
2.2 USING REAL-LIFE ALERT-BASED 7
DATA TO ANALYSE DROWSINESS
AND DISTRACTION OF
COMMERCIAL DRIVERS
2.3 SMARTPHONES AS AN 8
INTEGRATED PLATFORM FOR
MONITORING DRIVER BEHAVIOR
2.4 DRIVER-IN-THE-LOOP FORWARD
COLLISION WARNING USING
MULTISAMPLE REINFORCEMENT 9
LEARNING

v
 2.5 A NOVEL MODEL-BASED DRIVING 10
BEHAVIOR RECOGNITION SYSTEM
USING MOTION SENSORS.
2.6 DRIVER DROWSINESS
MONITORING SYSTEM USING
VISUAL BEHAVIOR AND MACHINE 11
LEARNING
2.7 ESTIMATION OF THE DRIVING
STYLE BASED ON THE USERS’
ACTIVITY AND ENVIRONMENT 13
INFLUENCE
3 SYSTEM ANALYSIS 15
3.1 EXISTING SYSTEM 15
3.1.1 Disadvantages 15
3.2 PROPOSED SYSTEM 16
3.2.1 Advantages 17
4 SYSTEM REQUIREMENTS 18
4.1 HARDWARE REQUIREMENTS 18
4.2 SOFTWARE REQUIREMENTS 18
5 SYSTEM DESIGN 19
5.1 SYSTEM ARCHITECTURE 19
5.2 DATA FLOW DIAGRAM 20
5.3 USE CASE DIAGRAM 22

vi
 5.3.1 Class Diagram 24
5.3.2 Sequence Diagram 24
6 MODULE DESCRIPTION 26
6.1 LIST OF MODULES 26
6.1.1 Data Collection 26
6.1.2 Receive the Image Input 28
6.1.3 Detect face create ROI 29
6.1.4 Detect eyes from ROI and 31
create classifier
6.1.5 Categorize the state of the eye 33
7 CONCLUSION AND FUTURE 35
ENHANCEMENT
7.1 CONCLUSION 35
7.2 FUTURE ENHANCEMENT 36
APPENDICES 38
1. SAMPLE CODE 38
2. SCREENSHOT 47
REFERENCES 51

vii
 LIST OF FIGURES

FIGURE NO FIGURE NAME PAGE NO

5.1 System Architecture 20

5.2 Data Flow Diagram(Level 0) 21

5.3 Data Flow Diagram(Level 1) 22

5.4 Use Case Diagram 23

5.5 Class Diagram 24

5.6 Sequence Diagram 25

6.1 Dataset of open eyes 27

6.2 Dataset of closed eyes 27

6.3 Sample of cascade algorithm output 32

A.2.1 Open State 47

A.2.2 Close State 48

A.2.3 Water Spray 49

A.2.4 Arduino kit 50

viii
 CHAPTER 1

INTRODUCTION

Most of the road accidents are caused because of drowsiness and drunk

driving and also working environments, reduced sleep and time factor. Driver

drowsiness and fatigue drunk driving reduces the driver decision making

capability and perception level. These two situations affect the ability to

control the vehicle. There are some techniques which are used to detect

drowsiness in drivers like by sensing of driver operation or physiological

characteristics of driver like or vehicle movement etc. A new approach

towards automobile safety and security with autonomous region primarily

based automatic automotive system is projected during this conception. We

have a tendency to propose 3 distinct however closely connected ideas viz.

Drowsy Driver Detection system and a traffic detection system with external

vehicle intrusion dodging primarily based conception. In recent time's

automobile fatigue connected crashes have very enlarged. so as to attenuate

these problems, we've incorporated driver alert system by watching each the

2
driver's eyes still as sensing still because the driver state of affairs based

primarily based native setting recognition based AI system is projected.

Over the world, a lot of individuals wish to purchase automobiles. It is

important to recognise that as the number of vehicles using the roads rises, so

too does the threat of traffic accidents. Road accidents are very common in

nations with densely populated streets and roads. According to a poll by the

National Crime Records Bureau (NCRB), 0.13 million lives were lost in road

accidents in India alone in the year 2020. The leading cause of death

worldwide is this. In a recent study, the World Health Organization (WHO)

identified several risk factors for accidents, including speeding, drunk driving,

distracted driving, etc. The majority of traffic accidents are caused by drivers

who are irresponsible and who disregard traffic laws and safety precautions,

according to almost all of these criteria. Lack of sleep, night time driving, or

even both can cause drowsiness, which wears out the driver and takes their

attention away from the road. It is fairly typical for bus and truck drivers in

the transportation sector to nod off while operating the vehicle, especially in

the early morning hours, as a result of tiredness. The aforementioned

3
circumstances necessitate that people are made aware of them in order to

prevent them and spare countless lives in the past. Technology is developing

quickly, and automation is making busy people's lives easier while also

delivering services perfectly, quickly and safely.

Figure 1.1 : Sample Drowsiness captured images

Hence, a system that automatically detects tiredness and predicts the

mood of the driver must be created for real-time applications. This will

improve public safety and help decrease traffic accidents. In the field of

accident prevention or accident-avoidance systems, the development of the

technologies necessary to put the driver drowsiness tools into practice

4
becomes a laborious process. The industry has created numerous strategies

based on various aspects because of how serious the issue is. The lack of

sleep, carelessness, or other factors that could divert the driver's focus from

driving could be to blame for their inattention. Alkinanietal used deep learning

techniques and challenges to conduct a thorough study of how people drive.

Apart from detecting the drowsiness in drivers, this system also contains the

application of initiating alarm system and water spray for waking up the

driver. Implementing these applications will help to improve the performance

by reacting as soon as the driver sleeps. Detecting the drowsiness alone is not

going to reduce the accidents occurred due to drowsiness, but reducing the

number of drivers sleeping does. Hence, this system will be very useful and

efficient in day-to-day life as it reduces the accidents and especially the threat

of driving at night time.

5
 CHAPTER 2

LITERATURE SURVEY

2.1 IMPACT OF THE FACE ANGLE TO TRAVELING TRAJECTORY

DURING THE RIDING STANDING-TYPE PERSONAL

MOBILITY DEVICE

AUTHOR: Jeyeon Kim, Kenta Sato, Naohisa Hashimoto (2019)

DESCRIPTION:

We investigate the impact of face direction during traveling by

Standing-Type Personal Mobility Device (PMD). The use of PMD devices

has been a popular choice for recreational activities in the developed countries

such as in the USA and the countries in Europe. These devices are not

completely risk free and various accidents have been reported. Since that, the

risk factors leading to accidents have to be investigated. Unfortunately, the

research studies on the risk factors on riding PMD devices have not been

matured as much as the studies on driving cars. We evaluate the impacts of

face angle on travelling trajectory during travelling in a PMD. We showed by

6
experiments that, the face direction is an important factor in risk assessment

for traveling by a PMD. In addition, it showed that the deviation of the

running locus occurred in the direction opposite to the direction the subject

was closely watching. These results showed how distracted driving is

dangerous and causes the risks of undeliberate trajectory changing.

2.2 USING REAL-LIFE ALERT-BASED DATA TO ANALYZE

DROWSINESS AND DISTRACTION OF COMMERCIAL

DRIVERS

AUTHOR: Sara Ferreira, Zafeiris Kokkinogenis, António Couto (2019)

DESCRIPTION:

Professional drivers are particularly exposed to drowsiness and

distraction inasmuch as they drive for long periods of time and as a daily

routine. Therefore, several studies have been conducted to investigate drivers´

behavior, supported by controlled experiments (e.g. naturalistic and driving

simulator studies). However, due to emerging technologies, new study

methods can be developed to complement existing studies. In this study,

retrospective data gathered from a driver monitoring system (DMS), which

7
monitored 70 professional drivers from different companies, was used to

investigate the effect of journey characteristics on the number of alerts due to

either distraction or drowsiness. Two separate negative binomial models were

developed, including explanatory variables describing the continuous driving

time (sub-journey time), the journey time (a set of sub-journeys), the number

of breaks and the breaking duration time. Dummy variables were also

included. Interesting results were observed such as increasing continuous

driving time, the number of distraction and drowsiness alerts increase too.

2.3 SMARTPHONES AS AN INTEGRATED PLATFORM FOR

MONITORING DRIVER BEHAVIOR: THE ROLE OF SENSOR

FUSION AND CONNECTIVITY

AUTHOR: Stratis Kanarachosa, Stavros-Richard G. Christopoulosa,

Alexander Chroneosa (2020)

DESCRIPTION:

Nowadays, more than half of the world’s web traffic comes from mobile

phones, and by 2020 approximately 70 percent of the world’s population will

be using smartphones. The unprecedented market penetration of smartphones

8
combined with the connectivity and embedded sensing capability of

smartphones is an enabler for the large-scale deployment of Intelligent

Transportation Systems (ITS). On the downside, smartphones have inherent

limitations such as relatively limited energy capacity, processing power, and

accuracy. These shortcomings may potentially limit their role as an integrated

platform for monitoring driver behaviour in the context of ITS. This study

examines this hypothesis by reviewing recent scientific contributions. The

Cybernetics theoretical framework was employed to allow a systematic

comparison. First, only a few studies consider the smartphone as an integrated

platform. Second, a lack of consistency between the approaches and metrics

used in the literature is noted.

2.4 DRIVER-IN-THE-LOOP FORWARD COLLISION WARNING

USING MULTISAMPLE REINFORCEMENT LEARNING

AUTHOR: Salma Elmalaki, Huey-Ru Tsai (2020)

DESCRIPTION:

On one side, a common thread in ADAS applications is to focus entirely

on the context of the vehicle and its surrounding vehicles leaving the human

9
(driver) context out of consideration. On the other side, and due to the

increasing sensing capabilities in mobile phones and wearable technologies,

monitoring complex human context becomes feasible which paves the way to

develop driver-in-the-loop context-aware ADAS that provide personalized

driving experience. In this paper, we propose Sentio1; a Reinforcement

Learning based algorithm to enhance the Forward Collision Warning (FCW)

system leading to Driver-in-the-Loop FCW system. Since the human driving

preference is unknown a priori, varies between different drivers, and

moreover, varies across time for the same driver, the proposed Sentio

algorithm needs to take into account all these variabilities which are not

handled by the standard reinforcement learning algorithms. We verified the

proposed algorithm against several human drivers.

2.5 A NOVEL MODEL-BASED DRIVING BEHAVIOR

RECOGNITION SYSTEM USING MOTION SENSORS.

AUTHOR: Wu M, Zhang S, Dong Y. (2019)

DESCRIPTION:

A novel driving behavior recognition system based on a specific

physical model and motion sensory data is developed to promote traffic

10
safety. Based on the theory of rigid body kinematics, we build a specific

physical model to reveal the data change rule during the vehicle moving

process. In this work, we adopt a nine-axis motion sensor including a three-

axis accelerometer, a three-axis gyroscope and a three-axis magnetometer, and

apply a Kalman filter for noise elimination and an adaptive time window for

data extraction. Based on the feature extraction guided by the built physical

model, various classifiers are accomplished to recognize different driving

behaviors. Leveraging the system, normal driving behaviors (such as

accelerating, braking, and lane changing and turning with caution) and

aggressive driving behaviors (such as accelerating, braking, lane changing and

turning with a sudden) can be classified with a high accuracy of 93.25%.

Compared with traditional driving behavior recognition methods using

machine learning only, the proposed system possesses a solid theoretical

basis, performs better and has good prospects.

11
2.6 DRIVER DROWSINESS MONITORING SYSTEM USING VISUAL

BEHAVIOR AND MACHINE LEARNING

AUTHOR: Ashish Kumar, Rusha Patra (2019)

DESCRIPTION:

Drowsy driving is one of the major causes of road accidents and death.

Hence, detection of driver’s fatigue and its indication is an active research

area. Most of the conventional methods are either vehicle based, or behavioral

based or physiological based. Few methods are intrusive and distract the

driver, some require expensive sensors and data handling. Therefore, in this

study, a low cost, real time driver’s drowsiness detection system is developed

with acceptable accuracy. In the developed system, a webcam records the

video and driver’s face is detected in each frame employing image processing

techniques. Facial landmarks on the detected face are pointed and

subsequently the eye aspect ratio, mouth opening ratio and nose length ratio

are computed and depending on their values, drowsiness is detected based on

developed adaptive thresholding. Machine learning algorithms have been

implemented as well in an offline manner. A sensitivity of 95.58% and

12
specificity of 100% has been achieved in Support Vector Machine based

classification.

2.7 ESTIMATION OF THE DRIVING STYLE BASED ON THE

USERS’ ACTIVITY AND ENVIRONMENT INFLUENCE

AUTHOR: Mikhail Sysoev, Andrej Kos, Jože Guna, Matevž Pogačnik (2020)

DESCRIPTION:

New models and methods have been designed to predict the influence of

the user’s environment and activity information to the driving style in

standard automotive environments. For these purposes, an experiment was

conducted providing two types of analysis: (i) the evaluation of a self-

assessment of the driving style; (ii) the prediction of aggressive driving style

based on drivers’ activity and environment parameters. Sixty seven h of

driving data from 10 drivers were collected for analysis in this study. The new

parameters used in the experiment are the car door opening and closing

manner, which were applied to improve the prediction accuracy. An Android

application called sensoric was developed to collect low-level smartphone

data about the users’ activity. The driving style was predicted from the user’s

environment and activity data collected before driving. The prediction was

13
tested against the actual driving style, calculated from objective driving data.

The prediction has shown encouraging results, with precision values ranging

from 0.727 up to 0.909 for aggressive driving recognition rate.

14
 CHAPTER 3

SYSTEM ANALYSIS

3.1. EXISTING SYSTEM

In the existing system, the cascaded objection algorithm has been

utilized in order to extract the face and eyes in the given input. Eye joints

are used to find the state of eye. Based on joints Drowsiness is defined.

Tracking Eye State for Fatigue Detection has implemented. It focuses on

eye states tracking. Images are captured using a camera and used for

tracking as input of the proposed method.

3.1.1. DISADVANTAGES

• Need to improve the performance of the overall system.

• Accuracy of the models is need to be improved.

• The operation time is more.

15
3.2 PROPOSED SYSTEM
In the proposed system, eye state is detected by cascade object detection

method. The detected features are processed through machine learning

models. Once train the model, model will be stored for model deployment

purpose. The accuracy of the model is calculated by using sklearn python

package.

Figure 3.1 : Block Diagram

16
3.2.1 ADVANTAGES

• Detection time is low

• More accuracy.
• Python is environment friendly, hence easy to implememented in any
device.
• Multiple work executed at same time
• Better services to passengers.

17
 CHAPTER 4

SOFTWARE REQUIREMENT

HARDWARE REQUIREMENTS:

• processor - Pentium – IV

• RAM - 4 GB (min)

• Hard Disk - 20 GB

SOFTWARE REQUIREMENTS:

• Operating System : Windows 10

• Software : Python IDLE

18
 CHAPTER 5

SYSTEM ANALYSIS

5.1 SYSTEM ARCHITECTURE

A system architecture is the conceptual model that defines the structure,

behaviour, and more views of a system. An architecture description is a

formal description and representation of a system, organized in a way that

supports reasoning about the structures and behaviours of the system. System

architecture can comprise system components, the externally visible

properties of those components, the relationships (e.g. the behaviour) between

them.

The input is given as a video, which is then converted into frames. These

frames are now forwarded for face detection. If face is not detected, the loop

continues. If face is detected, the region of interest is detected, from which eye

aspect ratio is calculated and converted to grayscale. Finally, this frame is

used for classification into respective states.

19
 Figure 5.1 : System Architecture

5.2 DATA FLOW DIAGRAM

1. The DFD is also called as bubble chart. It is a simple graphical

formalism that can be used to represent a system in terms of input data

to the system, various processing carried out on this data, and the output

data is generated by this system.

2. The data flow diagram (DFD) is one of the most important modeling

tools. It is used to model the system components. These components are

20
 the system process, the data used by the process, an external entity that

interacts with the system and the information flows in the system.

3. DFD shows how the information moves through the system and how it

is modified by a series of transformations. It is a graphical technique that

depicts information flow and the transformations that are applied as data

moves from input to output.

4. DFD is also known as bubble chart. A DFD may be used to represent a

system at any level of abstraction. DFD may be partitioned into levels

that represent increasing information flow and functional detail

DFD DIAGRAM

Level 0

Driver Start Video

camera capture

Figure 5.2: Data Flow Diagram(Level 0)

21
Level 1

Real Time Process Yolo

Face Capture Training

Eye State
detect

Figure 5.3 : Data Flow Diagram(Level 1)

5.3 USE CASE DIAGRAM

A use case diagram in the Unified Modeling Language (UML) is a type

of behavioral diagram defined by and created from a Use-case analysis. Its

purpose is to present a graphical overview of the functionality provided by a

system in terms of actors, their goals (represented as use cases), and any

dependencies between those use cases. The main purpose of a use case

diagram is to show what system functions are performed for which actor.

Roles of the actors in the system can be depicted.

22
Figure 5.4 : Use Case Diagram

23
5.3.1 Class Diagram

In software engineering, a class diagram in the Unified Modeling

Language (UML) is a type of static structure diagram that describes the

structure of a system by showing the system's classes, their attributes,

operations (or methods), and the relationships among the classes. It explains

which class contains information.

Figure 5.5: Class Diagram

5.4.2 Sequence Diagram

A sequence diagram in Unified Modeling Language (UML) is a kind of

interaction diagram that shows how processes operate with one another and in

what order. It is a construct of a Message Sequence Chart. Sequence diagrams

are sometimes called event diagrams, event scenarios, and timing diagrams.

24
Figure 5.6: Sequence Diagram

25
 CHAPTER 6
MODULE DESCRIPTION
6.1 LIST OF MODULES

1. Dataset collection.

2. Receive the image input.

3. Detect face and create ROI.

4. Detect eyes from ROI and create classifier.

5. Categorize the state of the eye.

6.1.1 Dataset collection

The dataset used for this model is created using multiple photos of the

human eye with open and close states. To create the dataset, a script that

captures eyes from a camera and stores them in our local disk is written. They

are separated into their respective labels ‘Open’ or ‘Closed’. This process is

called label encoding.

The data was manually cleaned by removing the unwanted images

which were not necessary for building the model. In this case, this process is

26
done by using a cropping tool, in which the eyes alone are cropped and as the

next process, both the left eye and right eye are cropped independently.

Figure 6.1 : Dataset of open eyes

Figure 6.2 : Dataset of Closed eyes

27
 The data comprises around 7000 images of people’s eyes under different

lighting conditions. All the images undergoes Normalization process in which

the pixel value ranges are taken into control using : img = img/255. After

training the model on our dataset, we have attached the final weights and

model architecture file “models/cnnCat2.h5”

6.1.2 Receive the image input

The input for our project is taken using the webcam. Webcam is

accessed, which records as videos and converts it into frames. The camera

actually runs in an infinite loop. Opencv generally has 24 frames per second

of a video. But the number of frames per second in a video is completely

dependent on the camera the user uses. If the camera has the number of

frames per second higher than 24, then opencv will use 24fps. If the number

of frames is less than 24, then the camera’s frames per second is used.

Opencv uses cv2.VideoCapture(0) method to access the camera. The

parameter (0) indicates that the programmer must use the in-built or integrated

camera fitted in the system (if any). If the system must use the secondary or

external camera, the parameter for cv2.VideoCapture must be 1. When

28
VideoCapture method starts recording, read() method is used to convert it into

frames. These frames are stored as images and sent to the next process as

input.

6.1.3 Detect face and create ROI

This module receives the image as input. The objective of this module is

to detect face in the given image. If the face is not detected, this module skips

the further steps and processes the next frame or image. Before processing the

image for face detection, the image has to be converted into grayscale. Image

processing techniques are not capable of processing colored images. Hence,

each frame is converted into grayscale before detecting the face. This

conversion is done by the cv2.cvtcolor() method, which is a part of opencv.

This method requires two parameters, the image or frame which has to be

converted and an integer value which specifies the color scheme to which the

frame has to be converted.

The method, cv2.cvtcolor() returns the grayscale image which has to be

stored for further processing. The next part of the process is the most

important part of image processing, classification. Classification is the process

29
of determining whether the required property is present in the image and also

to focus only on that property. In this case, the function of the classifier is to

detect the face and focus the face alone for the further process.

ROI represents the region of interest, which means the part of the image

whose properties are used for classification. HAAR cascade classifier is used

for classification of images into their states, i.e., as opened or closed. This

state is used for initiating alarm systems and water spray. A cascade XML file

is used for classification. This xml file contains the required properties to be

fulfilled in the image. The XML file, haarcascade_frontalface_alt.xml file,

which is present inside the cascade folder of the project is used for this

purpose.

Opencv contains the Classification function named

cv2.CascadeClassifier which is used to set the classifier. In this case we use

this function for setting a classifier for face. This function accepts a

haarcascade_frontalface_alt.xml file as argument. Then with this classifier, we

detect the region of interest(in this case, face). The classifier which has been

set into a variable face. This classifier variable contains a function,

detectMultiScale which accepts the grayscale image as argument and returns

30
an array of detections with x,y coordinates, height and the width of the

boundary box of the object.

The boundary box represents the part of the image which satisfies the

properties mentioned in the cascade xml file used for setting the classifier. At

this phase of the project, each frame which has faces in it are filtered and

boundary boxes are drawn for each face.

6.1.4 Detect eyes from ROI and create classifier

The next phase of this project is to classify the left and right eye from

the faces that are classified in the previous phase. Similar process that is used

for the classification of face from a frame is used in this phase to classify the

eyes from the face. The same procedure to detect faces is used to detect eyes.

Separate classifier xml files are used for left and right eyes. Separate

classifiers are set for left eye and right eye namely, Leye and Reye.

Similar to the previous phase, a variable Leye is declared and assigned

values that are returned by cv2.cascadeClassifier, which accepts the cascade

xml file, haarcascade_lefteye_2splits.xml as argument. This Leye variable is a

classifier for the left eye. Similarly, a variable Reye is declared and assigned

to the values that are returned by cv2.cascadeClassifier, which accepts the

31
cascade xml file, haarcascade_righteye_2split.xml as argument. This Reye

variable is a classifier for right eye.

Figure 6.3 : Sample of cascade algorithm output

Then with this classifier, we detect the region of interest(in this case,

face). The classifier which has been set into a variable lefteye and righteye.

This classifier variable contains a function, detectMultiScale which accepts

the grayscale image as argument and returns an array of detections with x,y

coordinates, height and the width of the boundary box of the object. The

boundary box represents the part of the image which satisfies the properties

mentioned in the cascade xml file used for setting the classifier. At this phase

32
of the project, the classifier function filters the left eye and right eye and saves

it in lefteye and righteye respectively.

6.1.5 Categorize the state of the eye

This project uses HAAR cascade classifier to classify whether the eyes

in the given image are open or closed. A Video is processed into several steps

including, converting into frames, convert them into grayscale, detect the

region of interest, detect left eye and right eye respectively etc., then finally

the frame with boundary boxes for the eyes are received as input for this

phase. In the first phase the machine has been trained using images which

undergoes resizing into 24*24 pixels. Thus, the image which is given as input

for classification will also be resized into same 24*24 pixels in order to

increase the accuracy of the model.

In real time, the images that are taken as input from a vehicle may

contain glare or unwanted lighting in the image, which can be removed by

using a process called normalization. The frames are normalized in order to

increase convergence. Normalization is done using reye = reye/255 and leye =

leye/255. This process will be arranged between 0 to 1. Then finally

33
prediction begins. If the output of the prediction is 0, then the eye is open. If

the output of the prediction is 1, then the eye is closed. These open and closed

states are taken into action by accounting both the right and left eyes.

Based on the states of the eyes, the alarm system and water spray are

triggered. As a product, the camera, alarm system and water spray will be

integrated with raspberry pi, which acts as a mini computer. Due to its wired

connection, the reaction time will be fast. The cut-off time has been set to 5

seconds, which means, if the driver closes his/her eyes for more than 5

seconds, then the alarm system and water spray are initiated. This value is

hardcoded and can be changed if required.

34
 CHAPTER 7

CONCLUSION AND FUTURE ENHANCEMENT

7.1 CONCLUSION

This system proposed a real-time system using image processing and

deep learning techniques to detect the drivers’ drowsiness with low

complexity and high accuracy. According to the report 2020 based on the road

accidents in India presented by Ministry of Road Transport & Highway,

disclose that 4, 77,044 accidents took place in states as well as in Union

Territories.

The proposed method predict Drowsiness based on machine learning

and cascade objection detection. The Proposed method, achieves higher

accuracy along with low operation time. For further research, we will create

our own drowsiness recognition dataset and validate the proposed method. In

addition, how to learn illumination invariant descriptor for drowsiness

recognition is still an important topic in our future work

35
7.2 FUTURE ENHANCEMENT

Implement deep learning concepts to determine the new trends

automatically in day-to-day lifestyle. This would help to learn new trends like

few people yarn while they feel sleepy, few people even sleep as eyes

open(mentally absent) even this would cause accidents as inattention in roads.

Trends like this will change from person to person and cannot be trained in

machines. Thus, implementing deep learning would help the system to learns

these trends and make this project more efficient and accurate.

The project can be connected to the central system of cars to

automatically park in a corner or roadside if the driver sleeps more often.

Nowadays, many of the vehicles contain auto-pilot, auto-parking facilities.

Integrating our project with this kind of ongoing technologies would make it

easier to safely park the vehicle roadside and make the driver rest for few

hours, until which the vehicle cannot be moved.

An admin portal can be created which monitors the behaviour of each

driver in an organisation. This portal allows the administrator to check how

his/her drivers work, how often he/she sleeps, risk level etc., through which

36
the can determine the driver’s work and take necessary actions. A monitoring

system will make the drivers to be active at it risks their life as well as their

job.

37
 APPENDIX:1

SAMPLE CODE

driver_drowsiness.py:

import cv2

import os

from keras.models import load_model

import tensorflow as tf

import numpy as np

from pygame import mixer

import time

from sending import led_on_off

path = os.getcwd()

mixer.init()

sound = mixer.Sound('audio.mp3')

## cascating files

face = cv2.CascadeClassifier('cascade\haarcascade_frontalface_alt.xml')

leye = cv2.CascadeClassifier('cascade\haarcascade_lefteye_2splits.xml')

reye = cv2.CascadeClassifier('cascade\haarcascade_righteye_2splits.xml')

38
## number of class

lbl=['Close','Open']

## load model

model = tf.keras.models.load_model('models/drowsiness_model.h5')

## to get web cam capture

cap = cv2.VideoCapture(0)

font = cv2.FONT_HERSHEY_COMPLEX_SMALL

## initialize the some parameter

count=0

score=0

thicc=2

rpred=[99]

lpred=[99]

while(True):

ret, frame = cap.read()

height,width = frame.shape[:2]

gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

39
 faces =
face.detectMultiScale(gray,minNeighbors=5,scaleFactor=1.1,minSize=(25,25)
)

left_eye = leye.detectMultiScale(gray)

right_eye = reye.detectMultiScale(gray)

cv2.rectangle(frame, (0,height-50) , (200,height) , (0,0,0) ,

thickness=cv2.FILLED )

for (x,y,w,h) in faces:

cv2.rectangle(frame, (x,y) , (x+w,y+h) , (100,100,100) , 1 )

for (x,y,w,h) in right_eye:

r_eye=frame[y:y+h,x:x+w]

count=count+1

r_eye = cv2.cvtColor(r_eye,cv2.COLOR_BGR2GRAY)

r_eye = cv2.resize(r_eye,(24,24))

r_eye= r_eye/255

r_eye= r_eye.reshape(24,24,-1)

r_eye = np.expand_dims(r_eye,axis=0)

rpred_ = model.predict(r_eye)

lpred = np.argmax(rpred_,axis=1)

40
 #print(lpred)

if(rpred[0]==1):

lbl='Open'

if(rpred[0]==0):

lbl='Closed'

break

for (x,y,w,h) in left_eye:

l_eye=frame[y:y+h,x:x+w]

count=count+1

l_eye = cv2.cvtColor(l_eye,cv2.COLOR_BGR2GRAY)

l_eye = cv2.resize(l_eye,(24,24))

l_eye= l_eye/255

l_eye=l_eye.reshape(24,24,-1)

l_eye = np.expand_dims(l_eye,axis=0)

lpred_ = model.predict(l_eye)

lpred = np.argmax(lpred_,axis=1)

if(lpred[0]==1):

lbl='Open'

41
 if(lpred[0]==0):

lbl='Closed'

break

if(rpred[0]==0 or lpred[0]==0):

print('sleep')

# send()

score=score+1

cv2.putText(frame,"Closed",(10,height-20), font,
1,(255,255,255),1,cv2.LINE_AA)

if(rpred[0]==1 or lpred[0]==1):

#score=score-3

score=0

cv2.putText(frame,"Open",(10,height-20), font,
1,(255,255,255),1,cv2.LINE_AA)

print('No sleep')

#ser.write(b'0')

time.sleep(2)

print('-------------')

#print('0 Send')

42
 print('-------------')

if(score<0):

score=0

cv2.putText(frame,'Score:'+str(score),(100,height-20), font,
1,(255,255,255),1,cv2.LINE_AA)

if(score>6):

#person is feeling sleepy so we beep the alarm

cv2.imwrite(os.path.join(path,'image.jpg'),frame)

cv2.imwrite('image.jpg', frame)

#report_send_mail('image.jpg')

#ser.write(b'1')

time.sleep(2)

print('-------------')

#print('1 Send')

print('-------------')

try:

sound.play()

led_on_off()

43
 except:

pass

if(thicc<16):

thicc= thicc+2

else:

thicc=thicc-2

if(thicc<2):

thicc=2

cv2.rectangle(frame,(0,0),(width,height),(0,0,255),thicc)

# to plot on screen

cv2.imshow('frame',frame)

if cv2.waitKey(1) & 0xFF == ord('q'):

break

## release image

cap.release()

cv2.destroyAllWindows()

44
Send_data.py:

import serial

import time

# Define the serial port and baud rate.

ser = serial.Serial('COM6', 9600)

def led_on_off():

time.sleep(2) # wait for the serial connection to initialize

time.sleep(0.1)

ser.write(b'H')

Main.ino:

#include <Servo.h>
int angle=0;
int servoPin = 9;
Servo servo;
void setup() {
Serial.begin(115200);
pinMode(LED_BUILTIN, OUTPUT);
servo.attach(servoPin);
}

45
void loop() {
if(Serial.available() > 0){
String msg=Serial.readString();
// while(true){
digitalWrite(LED_BUILTIN, HIGH); // turn the LED on (HIGH is the
voltage level)
delay(1000); // wait for a second
digitalWrite(LED_BUILTIN, LOW); // turn the LED off by making the
voltage LOW
delay(1000);
for(angle = 0; angle < 180; angle++) {
servo.write(angle);
delay(15);
}
for(angle = 180; angle > 0; angle--) {
servo.write(angle);
delay(15);
}
}
}

46
 APPENDIX:2

SCREENSHOTS

Figure A.2.1 : Open State

47
Figure A.2.2 : Close State

48
Figure A.2.3 : Water Spray

49
Figure A.2.4 : Arduino kit

50
REFERENCES:

1. Z. Chen, J. Yu, Y. Zhu, Y. Chen, and M. Li, “D3: Abnormal driving

behaviors detection and identification using smartphone sensors,” in
Proc. 12th Annu. IEEE Int. Conf. Sens., Commun., Netw. (SECON),
Jun. 2015, pp. 524–532.
2. A.Fedotov, I. Lashkov, and A. Kashevnik, “Web-service for drive safely
system user analysis: Architecture and implementation,” in Proc. 22nd
Conf. Open Innov. Assoc. (FRUCT), May 2018, pp. 40–47.
3. D. Geronimo, A. M. Lopez, A. D. Sappa, and T. Graf, “Survey of
pedestrian detection for advanced driver assistance systems,” IEEE
Trans. Pattern Anal. Mach. Intell., vol. 32, no. 7, pp. 1239–1258, Jul.
2010.
4. J. Jo, S. J. Lee, J. Kim, H. G. Jung, and K. R. Park, “Vision-based
method for detecting driver drowsiness and distraction in driver
monitoring system,” Proc. SPIE, vol. 50, no. 12, Dec. 2011, Art. no.
127202.
5. A.Kashevnik et al., “Context-based cyclist intelligent support: An
approach to e-bike control based on smartphone sensors.” in Proc. Int.
Conf. Next Gener. Wired/Wireless Netw. Cham, Switzerland: Springer,
Aug. 2018, pp. 16–22.

51
6. J. Kim et al., “Impact of the face angle to traveling trajectory during the
riding standing-type personal mobility device,” in Proc. Web Conf.
(MATEC), vol. 161, Apr. 2018, Art. no. 003001.
7. Lashkov, A. Smirnov, A. Kashevnik, and V. Parfenov, “Ontologybased
approach and implementation of ADAS system for mobile device use
while driving,” in Proc. 6th Int. Conf. Knowl. Eng. Semantic Web, Sep.
2015, pp. 117–131.
8. L. Moreira-Matias and H. Farah, “On developing a driver identification
methodology using in-vehicle data recorders,” IEEE Trans. Intell.
Transp. Syst., vol. 18, no. 9, pp. 2387–2396, Sep. 2017.
9. S. Sivaraman and M. M. Trivedi, “Looking at vehicles on the road: A
survey of vision-based vehicle detection, tracking, and behavior
analysis,” IEEE Trans. Intell. Transp. Syst., vol. 14, no. 4, pp. 1773–
1795, Dec. 2013.
10. A. Smirnov, A. Kashevnik, I. Lashkov, N. Hashimoto, and A. Boyali,
“Smartphone-based two-wheeled self-balancing vehicles rider assistant,”
in Proc. 17th IEEE Conf. Open Innov. Assoc. (FRUCT), Apr. 2015, pp.
201–209.

52