Project Report

on

DRIVER DROWSINESS DETECTION SYSTEM

(A dissertation submitted in partial fulfilment of the requirements of Bachelor of Technology in Computer

Science and Engineering of the Maulana Abul Kalam Azad University of Technology, West Bengal)

Submitted by
❖ Md Sukun Ul Qalb – 11200217034
❖ Md Sadiqueuddin – 11200217035
❖ Naveen Kumar Singh – 11200217033
❖ Richa Pandey – 11200218006
❖ Aditya Kr Paswan – 11200217048

Under the guidance of :-

Shri Falguni Sinhababu

Assistant Professor, Dept. of Computer Science and Engineering

Government College of Engineering and Leather Technology

(Affiliated to MAKAUT, West Bengal)
Kolakta - 700106, WB

2020-2021
 Certificate of Approval

This is to certify that the project report on “Driver Drowsiness Detection System” is a record of
bonafide work, carried out by Md Sukun Ul Qalb , Md Sadiqueudin , Naveen Kumar Singh, Richa Pandey and
Aditya Kr Paswan under my guidance and supervision.

In my opinion, the report in its present form is in conformity as specified by Government College of
Engineering and Leather Technology and as per regulations of the Maulana Abul Kalam Azad University of
Technology, West Bengal. To the best of my knowledge the results presented here are original in nature and
worthy of incorporation in project report for the B.Tech. Program in Computer Science and Engineering.

Signature of Signature of
Supervisor/ Guide Head, Dept. of CSE
 ACKNOWLEDGEMENT

With great pleasure, I would like to express my profound gratitude and indebtedness to Prof. Falguni
Sinhababu , Dept. of Computer Science and Engineering, Government College of Engineering and Leather
Technology, W.B. for his continuous guidance, valuable advice and constant encouragement throughout the
project work. His valuable and constructive suggestions at many difficult situations are immensely acknowledged.
I am in short of words to express his contribution to this thesis through criticism, suggestions and discussions.

I would like to take this opportunity to thank Dr. Surjadeep Sarkar, Project Coordinator and Prof. Santanu Halder,
HOD, Department of Computer Science & Engineering and Information Technology, Government College of
Engineering and Leather Technology.

1. MD SUKUN UL QALB – 11200217034

2. MD SADIQUEUDDIN - 11200217035 –

3. NAVEEN KUMAR SINGH – 11200217033 –

4. RICHA PANDEY – 11200218006 -

5. ADITYA KR PASWAN - 11200217048 -

 ABSTRACT
Driver’s fatigue is one of the major causes of traffic accidents, particularly for drivers of large vehicles (such as
buses and heavy trucks) due to prolonged driving periods and boredom in working conditions. As we have all
known the fact India is among the most top countries where the accidents due to these reasons happens every
seconds at night. So the idea behind making of this project This document is a review report on the research
conducted and the project made in the field of computer engineering to develop a system for driver drowsiness
detection to prevent accidents from happening because of driver fatigue and sleepiness. The report proposed
the results and solutions on the limited implementation of the various techniques that are introduced in
the project. Whereas the implementation of the project give the real world idea of how the system works and
what changes can be done in order to improve the utility of the overall system In this , propose a vision-based
fatigue detection system for driver monitoring, which is easy and flexible for deployment in buses and large
vehicles. The system consists of modules dlib facial landmark detector, 68 landmark, eye region, eye region of
interest, eye aspect ratio (EAR). A robust measure of eye aspect ratio (EAR) on the continuous level of eye
openness is defined, and the driver states are classified on it. In experiments, systematic evaluations and analysis
of proposed algorithms, as well as comparison with ground truth on EAR measurements, are performed. The
experimental results show the advantages of the system on accuracy and robustness for the challenging situations
when a camera of an oblique viewing angle to the driver’s face is used for driving state monitoring.

Keywords—Driver drowsiness; eye detection; Dlib Face Detector; Landmark Prediction.

 CONTENTS
CHAPTER 1 : INTRODUCTION
1.1 Motivation 2
1.2 Background 3
1.4 Summary of present work 4
1.5 Organization of the thesis 5
1.6 Hardware/Software 6

CHAPTER 2 : THE DETECTION MODEL

2.1 The Overview of Model
2.1.1 Basic details
2.2.2 Detection details
2.2 Libraries required
2.2.1
2.2.2
2.2.3
2.2.4
2.3 Methods
2.3.1
 INTRODUCTION
Drowsiness is a state of near sleep, where the person has a strong desire for sleep. It has two distinct meanings,
referring both to the usual state preceding falling asleep and the chronic condition referring to being in that state
independent of a daily rhythm. Sleepiness can be dangerous when performing tasks that require constant
concentration, such as driving a vehicle. When a person is sufficiently fatigue while driving, they will
experience drowsiness and this leads to increase the factor of road accident.

Fig 1.1 Road Accidents and deaths of year 2015,2016,2017.

The aim of this project is to develop a simulation of drowsiness detection system. The focus will be placed on
designing a system that will accurately monitor the open or closed state of the driver’s eyes . By monitoring the
eyes, it is believed that the symptoms of driver's drowsiness can be detected in sufficiently early stage, to avoid
a car accident.

1. Motivation
Due to highly increase in accidents nowadays in our country, we have come up with this basic detection model
for drowsiness detection which can be easily installed in cars and heavy vehicles with proper technology. Uses
of technologies are growing so why not make a new application which will detect drowsiness and save the
driver from any rash driving by sounding alarm beforehand. It is a basic prevention technique model which we
have created here and we can add various other detection with it according to the need of the vehicle and budget.
By implementing these techniques we will be able to see a significant decrease in the accidents which happens
at night majorly with heavy vehicles drivers.

2.. Background
The analysis of face images is a popular research area with applications such as face recognition, and human
identification and tracking for security systems. This project is focused on the localization of the eyes, which
involves looking at the entire image of the face, and determining the position of the eyes, by applying the
existing methods in image processing algorithm. Once the position of the eyes is located, the system is designed
to determine whether the eyes are opened or closed, and detect fatigue and drowsiness.

The model created here is a python language based model where we have used various libraries and detected the
face of the driver which will result in detecting the drowsiness through the eyes on the face.

In this project, it is required to detect the face of the driver and then detect the driver’s eyes to check whether he
is blinking. We have used image processing to synthesize and analyze the real time captured images of the
driver from the camera to detect the face and use them for probable fatigue detection. For these detections we
have used 68 Facial landmark detection as shown in fig 1.2.
 Fig. 1.2 Opened eyes and closed eyes detection

In this the main techniques used for eye blink detection is Eye Aspect Ratio (EAR) method. The Ear method is
done by calculating a quantity named EAR. In normal condition the value of EAR is almost constant. If the
driver is in fatigue the EAR value will be approximately will be below (0.25) . But with experimentally with
practical observation with our camera strength we have changed the required value of EAR which should be
below (0.24) here in our case. Thus by implementing above method we can detect whether the driver is in
fatigue state or not and accomplish our aim moto of the project.

3. Summary
The aim of this project is to develop a prototype drowsiness detection system. This system will primarily
measure and record the real time features of the driver or the driving pattern and continuously evaluate them on
the basis of the levels predetermined to indicate fatigue. The driver may show signs of fatigue in numerous
ways. The project surveys different methods of eye detection and checking whether the detected eyes are open
or closed. On the basis of the number of times the eyes are found to be closed, it can be determined whether the
person is drowsy or not another method of fatigue detection is the evaluation of the features of the mouth. If the
driver yawns, showing signs of fatigue, it can be used to trigger the alarm systems. Expressions indicating
extreme anger, disappointment, shock, excitement, etc show that the driver is not in the best state to drive and
may falter at it. This should be used to remind the driver in the form of an appropriate alarm to either park the
car or consciously regain the composure to drive safely.

There have been many varied ways to detect the human face that have been tested positively till now in various
parts of the world. However, the aim of this project is to have fast and efficient detection of face and features of
the face and hence real time face detection is necessary to detect the eyes and then find if the value of EAR (
Eye Aspect ratio) and declare the drowsiness accordingly.

4. Organisation of the thesis

5. Hardware/ Software used
Hardware required :

➢ Laptop with basic Hardware and with any windows Operating System.
➢ Fully functional Webcam with clear view of face
Software required :

➢ Python 3
• Pycharm or any Pyhton updated IDE
➢ Libraries
• Numpy
• Scipy
• Dlib
• Opencv
• Pygame
 Chapter 2 : THE DETECTION MODEL
1. OVERVIEW
1.1 Basic Details
The detection model we have made here is to detect a drowsiness, and we achieve that by detecting the
closeness of eye for more than 4 seconds in real time. It can detect the closeness of eyes with the straight face
and also for the face looking sideways or tilted face with closed eyes with our without spectacles on. The
algorithm processes the images captured in grey-scale method; where the colour from the images is then
transformed into black and white. For all the above process mentioned we require OpenCv libraries which
consists of various methods which will help in acquiring the desired result. Working with black and white
images is easier because only two parameters have to be measured. Then we perform the edge detection to
detect the edges of eyes so that the value of eyelid area can be calculated. The problem occurring with this
method is that the size area of eye might vary from one person to another. Someone may have small eyes and
looks like it is sleepy but some are not. The images that being captured must be in certain range from the camera
because when the distance is far from the camera, the image gets blurred.

1.2 Detection details

In this project we are detecting if a person is feeling drowsy or not with the help of python modules we have
created a fully functional code which will help in achieving our goal. Following are the process to detect the
same :-

➢ We gather the data set of eye and the face of general person which is stored in one or more directories
indexed by text file.
➢ Then we detect the edges of face of a person at real time with the help of a webcam and convert the
coloured face which captured as a frame to a black and white image.
➢ Then from that frame we capture the edges of both the eye and store their edges coordinate value in an
array.
➢ Then we determine the EAR (Eye aspect ratio) of each eye which is equal to Vertical distance divided
by Horizontal distance between the edges.
➢ If EAR < (0.24) of eye in that frame , then we say that the eye is closed and if its above this value then
we say that the eye is opened.
➢ So for drowsiness detection we need continuous frames which are having their EAR < (0.24) otherwise
if it is single frame then it will be counted as an eye blink.
➢ Thus after detecting the drowsiness by the above mentioned processes we have to alert the drivers by
sounding one alarm to prevent him from further driving and risking his/her life and the people who
care and depend on them.

Fig 2.1 (a) Feeling drowsy hence detected (b) Eyes opened hence nothing detected
 2. MODULES REQUIRED
The code for the detection model is in python and ran on Pycharm Compiler. Following are the modules
required to accomplish the same :-

2.1 OpenCv Module :-

OpenCV [OpenCV] is an open source computer vision library available from
http://SourceForge.net/projects/opencvlibrary. OpenCV was designed for computational efficiency and having a
high focus on real-time image detection. OpenCV is coded with optimized C and can take work with multicore
processors.

OpenCVs goals is to provide a simple-to-use computer vision infrastructure which helps people to build highly
sophisticated vision applications fast. The OpenCV library, containing over 500 functions, spans many areas in
vision. Because computer vision and machine learning oft en goes hand-in-hand, OpenCV also has a complete,
general-purpose, Machine Learning Library (MLL).

Following are its implementation :-

➢ Image processing
➢ Video capture
➢ Face detection and Object detection.

We capture the live camera frame by frame of our image through this and convert it into Gray scale images to
simplify further operation on the image.

2.1.1 OpenCv Structure and Content

OpenCV can be broadly structured into five primary components, four of which are shown in the figure
below. The CV component contains mainly the basic image processing and higher-level computer vision
algorithms; MLL the machine learning library includes many statistical classifiers as well as clustering tools.
High GUI component contains I/O routines with functions for storing, loading video & images, while CX Core
contains all the basic data structures and content.

2.2 Dlib Module

Dlib is a toolkit for making real world machine learning and data analysis applications in C++. While
the library is originally written in C++, it has good, easy to use Python bindings. It is used dlib for face detection
and facial landmark detection. The frontal face detector in dlib works really well. It is simple and just works out
of the box. This detector is based on histogram of oriented gradients (HOG) and linear SVM. While the
HOG+SVM based face detector has been around for a while and has gathered a good number of users. The
CNN (Convolutional Neural Network) based face detector available in dlib. The HOG based face detector in
dlib, is a good “frontal” face detector and it is, indeed. It detects faces even when they are not perfectly frontal
to a good extend.
 It contains software components for dealing with networking, threads, graphical user interfaces, data
structures, linear algebra, machine learning, image processing, data mining, XML and text parsing, numerical
optimization, Bayesian networks, and many other tasks. In recent years, much of the development has been
focused on creating a broad set of statistical machine learning tools.

2.3 Scipy Module

SciPy contains modules for optimization, linear algebra, integration, interpolation, special
functions, FFT, signal and image processing, ODE solvers and other tasks common in science and engineering.
SciPy is also a family of conferences for users and developers of these tools: SciPy (in the United States),
EuroSciPy (in Europe) and SciPy.in (in India).[5] Enthought originated the SciPy conference in the United States
and continues to sponsor many of the international conferences as well as host the SciPy website.
The SciPy library is currently distributed under the BSD license, and its development is sponsored and
supported by an open community of developers. It is also supported by Num FOCUS, a community foundation
for supporting reproducible and accessible science.
Here the Scipy module is used for determination of the EAR ( Eye Aspect Ratio) which will help in detection of
closing or opening of an eye.

2.4 Pygame Module

Pygame is a cross-platform set of Python modules designed for writing video games. It includes
computer graphics and sound libraries designed to be used with the Python programming language. Pygame
uses the Simple Direct Media Layer (SDL) library, with the intention of allowing real-time computer game
development without the low-level mechanics of the C programming language and its derivatives. This is based
on the assumption that the most expensive functions inside games can be abstracted from the game logic,
making it possible to use a high-level programming language, such as Python, to structure the game.

Other features that SDL doesn't have include vector math, collision detection, 2D sprite scene graph
management, MIDI support, camera, pixel-array manipulation, transformations, filtering, advanced freetype font
support, and drawing. Applications using Pygame can run on Android phones and tablets with the use of
Pygame Subset for Android. Sound, vibration, keyboard, and accelerometer are supported on Android.

Here it has been used for sounding the alarm whenever our model detects any drowsiness hence a mixer method
of this module is called which consist of one alert tone which will get ringed.

3. METHODS

3.1 Dlib Facial landmark detector

We use Dlib and OpenCV to detect facial landmarks in an image. Facial landmarks are used to localize
and represent salient regions of the face, such as:
➢ Eyes
➢ Eyebrows
➢ Nose
➢ Mouth
➢ Jawline

Facial landmarks have been successfully applied to face alignment, head pose estimation, face swapping, blink
detection and much more.
3.2 Detection and Extraction of landmarks

The pre-trained facial landmark detector inside the dlib library is used to estimate the location of 68
(x,y) coordinates that map to facial structures on the These annotations are part of the shape predictor 68.dat file
which the dlib facial landmark predictor was trained on.
Regardless of which dataset is used, the same dlib framework can be leveraged to train shape predictor on the
input training data.
Detecting facial landmarks with dlib, OpenCV, and Python The first utility function is , short for The dlib face
landmark detector will return a shape object containing the 68 (x, y)-coordinates of the facial landmark regions.
Using the shape_to_np function, we can convert this object to a NumPy array. We used these two helper
functions, to detect facial landmarks in images.

3.3 Eye Aspect Ratio & Alert Alarm –

In this technique, we are using different landmarks to detect the opening and closing of eye. This landmark
detector that capture most of the characteristic points on a human face image. The eye blink is a fast closing and
reopening of a human eye. Each individual person has a little bit different pattern of blinks. The pattern differs
in the speed of closing and opening of the eye, a degree of squeezing the eye and in a blink duration. The eye
blink lasts approximately 100-400ms.
From the landmarks detected in the image, we derive the eye aspect ratio (EAR) that is used as an estimate of
the eye opening state. For every video frame, the eye landmarks are detected. The eye aspect ratio between
height and width of the eye is computed. From the fig. 2 P1,P2,…,P6 are the landmarks on the eye.It is used to
compute the ratio of distances between the vertical eye landmarks and the distances between the horizontal eye
landmarks. The return value of the eye aspect ratio will be approximately constant when the eye is open. The
value will then rapidly decrease towards zero during a blink.

3.4. Extracting eye geometrical coordinates

Extraction of eye takes place to localize eye coordinates. Eye geometrical coordinates labelled as a
p1,p2,p3,p4 ,p5 and p6.
Measuring EAR From the landmarks detected in the image, we derive the eye aspect ratio (EAR) that is used as
an estimate of the eye opening state.For every video frame, the eye landmarks are detected.

The eye aspect ratio between height and width of the eye is computed

Fig 3.1 The edges of the Eyes (p1 – p6)

From the fig 3.1 P1,P2,…,P6 are the landmarks on the eye. It is used to compute the ratio of distances between
the vertical eye landmarks and the distances between the horizontal eye landmarks.
The first step in building a blink detector is to perform facial landmark detection to the eyes localize in a given
frame from a video stream.

Once we have the facial landmarks for both eyes, we compute the eye aspect ratio for each eye, which gives us a
singular value, relating the distances between the vertical eye landmark points to the distances between the
horizontal landmark points. Once we have the eye aspect ratio, we can threshold it to determine if a person is
blinking — the eye aspect ratio will remain approximately constant when the eyes are open and then will rapidly
approach zero during a blink, then increase again as the eye opens.
3.5 Monitoring of EAR for blinks detection:-
In this, different landmarks are used to detect the opening and closing of eye. This landmark detector
that capture most of the characteristic points on a human face image. The eye blink is a fast closing and
reopening of a human eye. Each individual person has a little bit different pattern of blinks. The pattern differs
in the speed of closing and opening of the eye, a degree of squeezing the eye and in a blink duration. The eye
blink lasts approximately 100-400ms.

So, the monitoring of EAR takes place to monitor a blink of eye whether a blink of eye takes place or not. It
plays an important role for blink detection of eye.
Estimation of fatigue periods between blinking Estimation of fatigue periods takes place during a blink detection
of eye.

3.6 Audio Visual warning on fatigue detection :-

In this step audio visual warning is run when the fatigue, drowsiness of a driver is detected. It shows a
visual message after detecting drowsiness with EAR threshold value the alert alarm runs and shows the
message as “Drowsy ? Are You Sleepy ?”.
CHAPTER 3 : THE ALGORITHM