See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/322250457

Real Time Driver Drowsiness Detection Based on Driver’s Face Image Behavior
Using a System of Human Computer Interaction Implemented in a
Smartphone

Chapter  in  Advances in Intelligent Systems and Computing · January 2018

DOI: 10.1007/978-3-319-73450-7_53

CITATIONS READS

47 27,739

5 authors, including:

Erick Galarza Franklin Silva

Universidad Peruana de Ciencias Aplicadas (UPC) Universidad de las Fuerzas Armadas-ESPE
6 PUBLICATIONS   54 CITATIONS    18 PUBLICATIONS   97 CITATIONS   

SEE PROFILE SEE PROFILE

Eddie Galarza
Universidad de las Fuerzas Armadas-ESPE
11 PUBLICATIONS   58 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Diseño e implementacion de un calefon a induccion electromagnética para proporcionar agua caliente a las viviendas unifamiliares. View project

All content following this page was uploaded by Eddie Galarza on 15 April 2018.

The user has requested enhancement of the downloaded file.

 Real Time Driver Drowsiness Detection Based on
Driver’s Face Image Behavior Using a System of Human
Computer Interaction Implemented in a Smartphone.

Eddie E. Galarza1,1, Fabricio D. Egas1, Franklin M. Silva1, Paola M. Velasco1,

Eddie D. Galarza1,
1
Universidad de las Fuerzas Armadas - ESPE,
Sangolquí, Ecuador
{eegalarza, fmsilva, pmvelasco, gqeddie}@espe.edu.ec, [email protected]

Abstract. The main reason for motor vehicular accidents is the driver
drowsiness. This work shows a surveillance system developed to detect and
alert the vehicle driver about the presence of drowsiness. It is used a
smartphone like small computer with a mobile application using Android
operating system to implement the Human Computer Interaction System. For
the detection of drowsiness, the most relevant visual indicators that reflect the
driver's condition are the behavior of the eyes, the lateral and frontal assent of
the head and the yawn. The system works adequately under natural lighting
conditions and no matter the use of driver accessories like glasses, hearing aids
or a cap. Due to a large number of traffic accidents when driver has fallen
asleep this proposal was developed in order to prevent them by providing a
non-invasive system, easy to use and without the necessity of purchasing
specialized devices. The method gets 93.37% of drowsiness detections.

Keywords: Drowsiness Detection, Artificial Vision, Mobile App, Perclos, Face

Detection.

1 Introduction

Sleeping is one of the basic needs of the human being, sleep lack causes the body
to react inefficiently, reducing both reaction time and wakefulness, also produce low
alertness and lose of concentration which reduces the ability to perform activities
based on care that is necessary in the case of driving a car.
According to many researches drowsiness is related to thousands of traffic
accidents each year, the accidents produces approximately 50% of death or serious
injuries [1], as they tend to be impacts at high speed because the driver who has fallen
asleep cannot brake or deviate to avoid or reduce impact. To mitigate these accidents,

1 This work was supported by the Universidad de las Fuerzas Armadas, Sangolquí – Ecuador.
Eddie E. Galarza (IEEE member), Franklin M. Silva, Paola M. Velasco and Eddie D. Galarza
work at the University of the Fuerzas Armadas at the campus in Latacunga City. Fabricio D.
Egas is an Electronic Engineer graduated in that university.
manufacturers have developed drowsiness detection systems that recognize signs of
possible drowsiness, alerting the driver to their condition [2].
In the research: "A smartphone-based driver safety monitoring system using data
fusion. Sensors", Lee and Chung [3] propose a method to monitor driver safety
levels using a data fusion approach such as: eye characteristics, variation of biological
signals, temperature inside the vehicle and vehicle speed. This system is developed as
an application for an Android-based smartphone, where measuring security-related
data that does not require additional costs or additional equipment. The system has an
efficiency of 96% to detect that the driver is awake and 97% to detect that he is
asleep. This information allows knowing the signs that shows a sleepy driver.
In work “Detection of fatigue using Smartphone aims to use a smartphone (with
Android operating system or IOS) to detect fatigue in the driver” [4] Roberson and
others uses the front camera of the smartphone to capture images of the driver and
then uses advanced algorithms of computer vision to detect his face and eyes.
Rotation and tilting of the head and blinking of the eyes are detected as indicators of
fatigue. The smartphones is used to assist driver using front and rear camera [5], for
drowsy driving detection system [6], for the wavelet analysis of heart rate variability
and a support vector machine classifier [7], and for identification of dangerous driving
situations [8].
The PERCLOS (Percent of the time Eyelids are CLOSed) metrics is used to
measure drowsiness in the work “Eye tracking based driver fatigue monitoring and
warning system” [9]. The system estimates with a non - parametric methods for
detecting drowsiness, the vehicle steering wheel variability is considered to determine
the amount of drowsiness because drivers makes variability greater as driver become
more drowsy. The PERCLOS metrics for alerting driver is used in [10] to detect
drowsiness in heavy vehicles, to monitor and alert the driver [11], for line departure
warnings [12] and to detect drowsiness conditions in drivers [13].
The HCI systems allows to interrelate the human being with an electronic device
(computer) which is capable of giving solutions to a great number of problems that
can affect him. The development and use of HCI has been very important, so it must
be implemented with adequate usability criteria [14] and satisfy users' needs
efficiently [15]. A relevant aspect is that not only sought a simple interaction also
sought to assist humans with special skills to satisfy their needs even overcoming
their limitations [16-19] and can be implemented using low cost systems [20-25]. The
smart phones being mass-use are actually a low-cost computer, if are used in an HCI
would allow to massify its use and therefore offer greater solutions to improve the
quality of life of any person satisfying their needs even if the person presents some
limitation in one or more of their senses.
The objective of this work was to implement a surveillance system to the vehicular
driver based on artificial vision techniques and implemented in a smartphone in order
to detect and alert when the driver have drowsiness signs. To achieve this objective it
was analyzed other works related with detecting drowsiness in drivers, the drowsiness
symptoms in vehicle drivers; we identify the technical parameters and algorithms that
allow to process signals of the state of drowsiness. In this work we present a
developed drowsiness detection algorithm, the interface in which the state of
drowsiness is displayed and the necessary adjust to get the correct functioning of the
implemented system.
 Some of the aspects that are included in this work have been considered in the
investigations referred to, but it differs essentially in the use of the new systems for
digital image processing in smartphones.

1.1 Drowsiness Characteristics

Drowsiness is a physiological state with a tendency to fall asleep. Technically,

drowsiness is different from the fatigue that is the lack of willingness to continue
performing the same activity. Fatigue occurs by performing tasks that are always
performed in the same way using the same muscle groups, their repetition rate is high
and are usually performed by adopting forced postures such as monitoring a screen
[26]. A person may be fatigued without being drowsy, but conditions that produce
fatigue such as driving cars over great distances unmask the presence of physiological
drowsiness, but do not cause fatigue.
Among the effects of being sleepy we have a lowered wakefulness, reaction time,
psychomotor coordination and decreased information processing. For the driver the
main effect is the progressive withdrawal of attention in demands of road, traffic and
signaling, which causes a low driving performance producing accidents [27]. People
who are drowsy have signs like frequent blinking, rubbing eyes, repeated yawning,
head tilt, and distractions are the most important among which it can mention.

1.2 Drowsiness detection methods

Detection methods are divided into two main groups: methods based on driver
performance and methods based on driver status [28]. The methods centered on driver
status are divided into two subgroups: methods that use physiological signals and
methods that use artificial vision techniques. Figure 1 shows classification of
drowsiness detection methods.

Fig. 1. Drowsiness detection methods.

 The drowsiness detection using patterns analysis are generated based on
measurable variables that are obtained experimentally. These variables can be speed,
acceleration, braking, gear shifting, hand pressure on the steering wheel and the car's
path in the road lane. This method has the disadvantage that its modeling depends on
the characteristics of each car and the way of driving that is specific to each driver.
By the use of the image processing, driver states can be determined. From the
image of the face it can be detected if the driver is awake or asleep. The drowsiness of
the driver can be determined because the driver is trying to close his eyes [29]. This
method has the advantage of not being intrusive and can be used techniques like the
template pairing technique where a driver templates is defined. The technique of the
behavior of the eyes, calculates the blinking frequency and the time interval of eyes
closing in order to determine the rate of drowsiness.
One of the most used indexes to calculate the level of sleepiness is PERCLOS
(Percent of the time Eyelids are CLOSed), which measures the percentage of time a
person's eyes are closed at 80% to 100% in a period. According to a study by Walter
Wierwille and colleagues [30], PERCLOS is among the most important real-time alert
measures for vehicle drowsiness detection systems.
Perclose = (Closed eyes time /(closed eyes time + open eyes time))*100 (1)
The Yawning technique is based on the driver's yawn frequency [29]. The opening
of the driver mouth is greater when yawning than when speaking normally. The
mouth is compared with a reference point experimentally obtained by the programmer
and the number of times the driver has yawned is calculated to generate a drowsiness
index. The analysis based on changing physiological measures use sensors that
measure physiological variables of the human body to analyze states of drowsiness.
These variables are the heart rate, brain activity, heart rate variability, respiration,
peripheral skin temperature, and blood pressure [31].

1.3 Mobile applications in Smartphones

Smartphones are electronics devices that combines the functionalities of a mobile

phone and the functionalities of a computer. The term intelligent is used commercially
to refer to the ability to use as a handheld computer, sometimes leading to a
replacement of a personal computer.
Smartphones are built on a mobile platform similar to the operating system of a
computer, which makes it equally vulnerable to viruses. Among the basic
characteristics of a smartphone we have that includes an operating system, it can send
and receive calls and text messages, it has multimedia services, includes basic
applications such as clock, alarm, calendar, calculator, games, digital agenda, it has
internet connection, includes front and rear video cameras, it can read and edit
documents, it has sensors like gyroscope, accelerometer, barometer, thermometer, etc.
Google provides a set of open source visual programming interfaces (APIs) for
mobile devices [32], the APIs can make face detection and tracking, barcode scanner,
text recognition. The API also offers information about the state of facial features like
open eyes and smiles [33].
2 Materials and Method

This section describes different aspects of the system considered in its

implementation; they include the functional requirements as well as the tools used and
devices selected for system testing in different study cases.
The used algorithm processes the color information present in the image,
converting it to grayscale. To determine the face in the image, the image is divided in
sub regions determining whether the subregion is a face or not. The use of this
algorithm means a time saving and only the subregions that contains a face are
processed.
The gesture detection is done from the residual error that is modeled considering a
linear combination of facial movement models. A similar model is considered to
detect the position and inclination of the face. It includes a system that allows
detecting facial gestures in the presence of head movement.
Figure 2 shows the flowchart of the system. The code used to implement the
algorithms was created taking into account the limitations that have mobile devices
like the limited content or features in the interface, slow or limited hardware and use
situations. Their success in functionality is based on the way they are designed and
optimized by company that owns Android.

Fig. 2. Drowsiness detection methods.

A system general scheme is seen in Figure 3 in which it is shown that the driver
must take into account the alerts presented by the system, while the smartphone is in
charge of processing the information acquired from the driver's face in real time. The
main use case diagram is shown in Figure 4.
 Fig. 3. General scheme of the drowsiness detection system.

Fig. 4. Main use case diagram.

Fig. 5. System user interface

The user interface of the system is presented in Figure 5, in which two buttons are
presented that allow to show information regarding the situation of the driver, which
is necessary for research aspects. In addition, there are indicators that shows the
presence or absence of the driver’s face at the interface, open eyes, left or right face
tilt, left or right distraction, and yawn presence are included. Technical information is
also presented with respect to the operating characteristics of the system that the user
does not necessarily need to know.
Subjects: To perform the system tests, 20 drivers, 10 men and 10 women of
different ages were included, each of whom was accompanied by a "co-pilot" who
was in charge of managing the controlled events of sleepiness only when the external
conditions in the road and close to the vehicle were safe.
The events to validate the system were the following: yawning detection, front and
lateral assent of the head, left and right distraction and blinking. In the process of
validating the system, the co-pilot will direct them, making the drivers repeat 10 times
the test. The number of correctly detected events, false positives, false negatives, and
the efficiency of the implemented system were registered.

3 Results

This section presents the results on the detection of visual indicators of drowsiness.
Collecting the data set to properly evaluate the system is a challenge, this is because
dangerous drowsiness events are not guaranteed to occur during daily driving for
application testing.

Table 1. Detection levels for drowsiness parameter under normal conditions.

Test Number of Number of Percentage

observations hits of hits
Yawn detection 170 143 84.11 %
Front nodding 200 184 92.0 %
Assent of the head to the right 200 190 95.0 %
Assent of the head to the left 200 191 95,5 %
Distraction to the right 200 184 92.0 %
Distraction to the left 200 193 96.5 %
Blink detection 200 197 98.5 %

Table 1 presents the results of the detection of somnolence considering the normal
operation of the system in which the responses are obtained from each of the drivers
that were submitted to the addresses issued by the co-pilot who recorded the results.
The level of total hits on detection represents an average percentage of 93.37%.

Table 2. Detection levels for different drowsiness parameter under special conditions.

Test Number of Number of Percentage of

observations hits hits
Driver with a cab 1400 1295 92.5 %
Driver with glasses 1400 1183 85.5 %
 Table 2 presents the results obtained by placing additional objects on the clothes of
the drivers, which in this case were the caps and glasses. The average percentage of
hits is 88.5%.

Table 3. Detection levels for different drowsiness parameter considering the hair covering
face.

Test Number of Number Percentage

observations of hits of hits
Hair covering driver’s face 70 65 92.8 %
Hair not covering driver´s face 70 49 70.0 %
Table 3 presents the results obtained by considering the hair covering driver’s face.
The drivers were women and the average percentage of hits was 81.4 %.
Los resultados presentados indican la eficiencia del sistema que es de alto nivel y
son comparable e inclusive superiores a otros sistemas a los que se hace referencia en
el presente trabajo. Los menores niveles de acierto se presentan cuando los usuarios
incluyen elementos que no permiten ser identificados los gestos de la cara de manera
correcta, pero que a pesar de aquello sus niveles son satisfactorios.
The presented results indicate the efficiency of the system that is of higher level
and even better than other systems referred to in this work. The lower levels of
accuracy occur when users include elements that do not allow to correctly identify the
face gestures, but despite that, their levels are satisfactory.

5 Conclusions

The study has shown promising results in applying the vehicular driver
surveillance based on artificial vision techniques and implemented in a smartphone.
The implemented system allows an efficient detection of the indicators that appear in
drowsiness, as long as the measurements are carried out under the established
conditions. The correct functioning of the system depends on these conditions.
The increase in the processing characteristics in smartphones made possible to
develop an application of artificial vision, capable of detecting the face and visual
indicators present in a person who suffers from drowsiness such as: yawning, head
movements and the state of the eyes.
The symptoms that people present during the transition between awake and asleep
are appearing as the intensity of drowsiness increases. The greater intensity of
drowsiness means a higher loss of concentration and a lower ability of driver reaction.
In development this work, the implementation of 3 levels of sleepiness allows the
system to alert the driver about their condition, not necessarily at a critical level where
it may have serious repercussions, rather at early levels where drowsiness is just
emerging.
An HCI could be implemented using smartphones like shown in this work, which
would allow massify their use and therefore provides greater solutions improving the
quality of life of the people even if has specials skills..
References

1. World Health Organization. Road Safety: Basic Facts. ; 2016

(http://www.who.int/violence_injury_prevention/publications/road_traffic/Road_safety_me
dia_brief_full_document.pdf).
2. Ji Q., Yang X.: Real-time eye, gaze, and face pose tracking for monitoring driver vigilance.
Real-Time Imaging. 8(5):357-77 (2002)
3. Lee B. G., Chung W. Y.: A smartphone-based driver safety monitoring system using data
fusion. Sensors. 12(12):17536-52 (2012)
4. He J, Roberson S, Fields B, Peng J, Cielocha S, Coltea J. Fatigue detection using
smartphones. Journal of Ergonomics. 3(03):1-7 (2013)
5. Chang K, Oh BH, Hong KS. An implementation of smartphone-based driver assistance
system using front and rear camera. In Consumer Electronics (ICCE), 2014 IEEE
International Conference (pp. 280-281). IEEE (2014)
6. Xu L, Li S, Bian K, Zhao T, Yan W. Sober-Drive: A smartphone-assisted drowsy driving
detection system. InComputing, Networking and Communications (ICNC), 2014
International Conference. pp. 398-402. IEEE (2014)
7. Li G, Chung WY.: Detection of driver drowsiness using wavelet analysis of heart rate
variability and a support vector machine classifier. Sensors. 13(12):16494-511 (2013)
8. Smirnov AV, Kashevnik A, Lashkov I, Baraniuc O, Parfenov V. Smartphone-based
identification of dangerous driving situations: Algorithms and implementation. In FRUCT
pp. 306-313 (2016)
9. Singh H, Bhatia JS, Kaur J.: Eye tracking based driver fatigue monitoring and warning
system. InPower Electronics (IICPE), 2010 India International Conference. pp. 1-6. IEEE
(2011)
10. Grace R, Byrne VE, Bierman DM, Legrand JM, Gricourt D, Davis BK, Staszewski JJ,
Carnahan B.: A drowsy driver detection system for heavy vehicles. In Digital Avionics
Systems Conference, 1998. Proceedings., 17th DASC. The AIAA/IEEE/SAE 1998 Nov
(Vol. 2, pp. I36-1). IEEE.
11. Grace R, Steward S.: Drowsy driver monitor and warning system. In International driving
symposium on human factors in driver assessment, training and vehicle design. Vol. 8, pp.
201-208 (2001).
12. Kozak K, Pohl J, Birk W, Greenberg J, Artz B, Blommer M, Cathey L, Curry R.:
Evaluation of lane departure warnings for drowsy drivers. In Proceedings of the human
factors and ergonomics society annual meeting (Vol. 50, No. 22, pp. 2400-2404). Sage CA:
Los Angeles, CA: Sage Publications (2006)
13. Garcia I, Bronte S, Bergasa LM, Almazán J, Yebes J.: Vision-based drowsiness detector for
real driving conditions. In Intelligent Vehicles Symposium (IV), IEEE. (pp. 618-623). IEEE
(2012).
14. Harshul, G., Tanupriya, C., & Praveen, K. (2017). Comparison between significance of
usability and security in HCI. 2017 3rd International Conference on Computational
Intelligence & Communication Technology (CICT) (págs. 1 - 4). Ghaziabad, India: IEEE.
15. Zhiqiang, X., Xuejun , Q., & Jialiang , H. (2017). A novel multimedia human-computer
interaction (HCI) system based on Kinect and depth image understanding. International
Conference on Inventive Computation Technologies (ICICT) (págs. 1 - 6). Coimbatore,
India: IEEE.
16. Fernandez Montenegro, J., & Argyriou, V. (2016). Gaze estimation using EEG signals for
HCI in augmented and virtual reality headsets. 2016 23rd International Conference on
Pattern Recognition (ICPR). Cancun, Mexico: IEEE.
17. El-Shazly, E., Abdelwahab, M., & Shimada, A. (2016). Real time algorithm for efficient
HCI employing features obtained from MYO sensor. 2016 IEEE 59th International
 Midwest Symposium on Circuits and Systems (MWSCAS) (págs. 1 - 4). Abu Dhabi,
United Arab Emirates: IEEE.
18. Itkarkar, R. R., & Nandi, A. V. (2016). A survey of 2D and 3D imaging used in hand
gesture recognition for human-computer interaction (HCI). 2016 IEEE International WIE
Conference on Electrical and Computer Engineering (WIECON-ECE) (págs. 188 - 193).
Pune, India: IEEE.
19. Zuo, H. (2016). Implementation of HCI software interface based on image identification
and segmentation algorithms. 2016 Online International Conference on Green Engineering
and Technologies (IC-GET) (págs. 1 - 6). Coimbatore, India: IEEE.
20. Gabrielli, L., aBussolotto, M., & Squartini, S. (2014). Reducing the latency in live music
transmission with the BeagleBoard xM through resampling. 2014 6th European Embedded
Design in Education and Research Conference (EDERC) (págs. 302 - 306). Milano, Italy:
IEEE.
21. Leboeuf-Pasquier, J., Villa, A. G., Burgos, K. H., & Carr-Finch, D. (2014). Implementation
of an embedded system on a TS7800 board for robot control. 2014 International
Conference on Electronics, Communications and Computers (CONIELECOMP) (págs. 135
- 141). Cholula, Mexico: IEEE.
22. Toshniwal, K., & Conrad, J. M. (2010). A Web-based Sensor Monitoring System on a
Linux-based Single Board Computer Platform. Proceedings of the IEEE SoutheastCon
2010 (SoutheastCon) (págs. 371 - 374). Concord, NC, USA: IEEE.
23. Jaziri, I., Chaarabi, L., & Jelassi, K. (2015). A remote DC motor control using Embedded
Linux and FPGA. 2015 7th International Conference on Modelling, Identification and
Control (ICMIC) (págs. 1 - 5). Sousse, Tunisia: IEEE.
24. Kang, P., Wei, Y., & Wei, Z. (2017). Control System for Granary Ventilation Based on
Embedded Networking and Qt Technology. 2017 29th Chinese Control And Decision
Conference (CCDC) (págs. 2275 - 2280). Chongqing, China: IEEE.
25. Degada, A., & Savani, V. (2015). Design and Implementation of Low Cost,
Portable Telemedicine System: An Embedded Technology and ICT Approach.
2015 5th Nirma University International Conference on Engineering (NUICONE)
(págs. 1 - 6). Ahmedabad, India: IEEE.
26. Stutts JC, Wilkins JW, Vaugh BV.: Why do people have drowsy driving crashes? Input
from drivers who just did. AAA Foundation for Traffic Safety, Washington DC.
27. Verwey WB, Zaidel DM.: Preventing drowsiness accidents by an alertness maintenance
device. Accident Analysis & Prevention;31(3):199-211 (1999)
28. Flores MJ, De la Escalera A.: Sistema avanzado de asistencia a la conducción para la
detección de la somnolencia. Revista Iberoamericana de Automática e Informática
Industrial RIAI. 30;8(3):216-28 (2011)
29. Fuletra JD, Bosamiya D. A: Survey on Driver’s Drowsiness Detection Techniques.
International Journal on Recent and Innovation Trends in Computing and Communication.
1(11):816-9 (2013).
30. Dinges DF, Grace R.: PERCLOS: A valid psychophysiological measure of alertness as
assessed by psychomotor vigilance. US Department of Transportation, Federal Highway
Administration, Publication Number FHWA-MCRT-98-006. (1998)
31. Jo J, Lee SJ, Jung HG, Park KR, Kim J.: Vision-based method for detecting driver
drowsiness and distraction in driver monitoring system. Optical Engineering.
1;50(12):127202 (2011)
32. Joyce G, Lilley M, Barker T, Jefferies A.: Mobile Application Tutorials: Perception of
Usefulness from an HCI Expert Perspective. In International Conference on Human-
Computer Interaction (pp. 302-308). Springer International Publishing. (2016)
33. https://www.sitepoint.com/mobile/app-development/ available at 10 sept 2017.

View publication stats