Santa Clara University

Scholar Commons
Electrical Engineering Senior Theses Engineering Senior Theses

6-10-2017

CMAG: IoT Baby Monitor

Xiaoting Liu
Santa Clara University, [email protected]

Kyle Takeuchi
Santa Clara University, [email protected]

Follow this and additional works at: https://scholarcommons.scu.edu/elec_senior

Part of the Electrical and Computer Engineering Commons

Recommended Citation
Liu, Xiaoting and Takeuchi, Kyle, "CMAG: IoT Baby Monitor" (2017). Electrical Engineering Senior Theses. 34.
https://scholarcommons.scu.edu/elec_senior/34

This Thesis is brought to you for free and open access by the Engineering Senior Theses at Scholar Commons. It has been accepted for inclusion in
Electrical Engineering Senior Theses by an authorized administrator of Scholar Commons. For more information, please contact [email protected].
 CMAG: IoT Baby Monitor

By

Xiaoting Liu
and
Kyle Takeuchi

Advisors:
Professor Tokunbo Ogunfunmi
and
Mr. Shivakaumar Mathapathi

SENIOR DESIGN PROJECT REPORT

Submitted to
the Department of Electrical Engineering

of

SANTA CLARA UNIVERSITY

in Partial Fulfillment of the Requirements

for the degree of
Bachelor of Science in Electrical Engineering

Santa Clara, California

June 10, 2017

i
 Acknowledgements

The CMAG team would like to thank the following people for helping with the development of
the project over the year:
 Professor Ogunfunmi for his guidance as faculty advisor
 Mr. Shivakaumar Mathapathi for providing the Artik and technical support
 Nick Mikstas for providing much assistance with the MATLAB code
 Jitendra Kumar for providing the Twilio code foundation
 Santa Clara School of Engineering for funding the project
 All friends and family who helped or supported the project along the way

ii
 Abstract

Sudden Infant Death Syndrome, also known as SIDS, is the leading cause of mortality in infants
from one month to one year of age. We propose a video-based baby monitoring system with
Internet of Things (IoT) capabilities to help shorten the response time of SIDS cases. Using a
video amplification technique developed at MIT dubbed “Eulerian Magnification” to amplify
subtle movements we can compare pixel color differences in frames for breathing detection in a
recorded video of a baby. In the event that abnormal movement is detected from the baby an
alarm will be generated to notify the parents or guardians.

iii
 Table of Contents

Abstract...........................................................................................................................................iii
Table of Contents............................................................................................................................iv
List of Figures…………………………………………………………………………………....vii
List of Tables……………………………………………………………………………………viii

Chapter 1: Introduction
1.1 SIDS Background................................................................................................................1
1.2 Project Need…………….…………………………………………………………….…...1
1.3 Project Objective..................................................................................................................1
1.4 About the Internet of Things……………………………………………………………....2
1.5 Eulerian Magnification Background……………………………………………………....2

Chapter 2: Design and Implementation

2.1 About the Artik 10 Board…………………………………………………………………3
2.2 System Diagram…………………………………………………………………………...3
2.3 Requirements….…………………………………………………………………………..4
2.4 Constraints………………………………………………………………………………...5
2.5 Initial Implementation Method……………………………………………………………5
2.6 Implementation Alternatives……………………………………………………………...6
2.7 Technical Familiarity………………………………………………………………….…..7
2.8 Technologies Used…………………………………………………………………….…..7
2.8.1 Night Vision Camera……………………………………………………………...7
2.8.2 PuTTY……………………………………………………………………………..8
2.8.3 MATLAB………………………………………………………………………….8
2.8.4 Twilio……………………………………………………………………...............9
2.9 Cost...……………………………………………………………………………………...9
2.10 Risk Analysis…….………………………………………………………………………..9
2.11 Development Timeline......................................................................................................10

iv
Chapter 3: Roadblocks
3.1 Internet Connectivity ........................................................................................................13
3.2 Streaming...........................................................................................................................13
3.3 Running Eulerian on Artik.................................................................................................14
3.4 Using Twilio for SMS........................................................................................................15

Chapter 4: Breathe Detection Algorithm Testing and Results

4.1 Capturing Video.…………………………………………………………………...…….16
4.2 Counting Pixels………………………………………………………………...…….......16
4.3 Representing Motion Over Time………………………………………………..……….17
4.4 Detecting Abnormalities………………………………………………………………....19
4.5 Using Twilio for Alarm Generation…………………………………………………...…23

Chapter 5: Professional Issues and Constraints

5.1 Ethics……………………………………………………………………..........................24
5.2 Ethical Justification………………………………………………………….…………...24
5.3 Sustainability…………………………………………………………………….……….24
5.3.1 Environmental Sustainability…………………………………………….………25
5.3.2 Social Sustainability…………………………………………………..………….25
5.3.3 Economic Sustainability…………………………………………………………26
5.4 Health and Safety………………………………………………………………………...26
5.5 Manufacturability and Usability…………………………………………………………26
5.6 Civic Engagement…...27

Chapter 6: Future Work and Conclusion

6.1 Acceptance Criteria………………………………………………………………………28
6.2 Future Work.......................................................................................................................28
6.3 Lessons Learned.................................................................................................................29

References……………………………………………………………………..............................30

v
Appendices
MATLAB.........................................................................................................................................I
Twilio…………………………………………………………………………………….………IV

vi
 LIST OF FIGURES

Figure 2.1: The Samsung Artik 10 Board.…………………………………………………….3

Figure 2.2: Final Design Pictorial Diagram…………………………………………………...4
Figure 2.2: Initial Implementation Block Diagram…………………………………………....6
Figure 2.4: Night Vision Camera……………………………………………………………...8

Figure 4.1: Pixel Representation of a Frame…………………………………….…………...16

Figure 4.2: Chest Movement Generated from Original Video……………………………….17
Figure 4.3: Chest Movement Generated from Eulerian Video……………………………….18
Figure 4.4: Chest Movement Plot from 13 Different Videos………………………………...18
Figure 4.5: Normalized Chest Movement Plot from 13 Different Videos…………………...19
Figure 4.6: Normalized Eulerian Video……………………………………………………...19
Figure 4.7: Frequency Domain Plot of Normalized Video…………………………………..20
Figure 4.8: The Parks-McClellan Linear-Phase FIR Low Pass Filter………………………..21
Figure 4.9: Frequency Domain Plot After Low Pass Filter. …………………………………21
Figure 4.10: Large Magnitude Breathing Frequency Plot with Threshold Level……………..22
Figure 4.11: Small Breathing Frequency Plot with Threshold Level………………………….22
Figure 4.12: No Breathing Frequency Plot with Threshold Indicated………………………...22
Figure 4.13: Text Messages Indicating Normal and Abnormal Breathing………………………23

vii
 LIST OF TABLES

Table 2.1: Functional and Nonfunctional Requirements……………………….…………….4

Table 2.2: Cost of Materials ………………………………………………...………………..9
Table 2.3: Risk Analysis…………………………………………………………………….10
Table 2.4: Gantt Chart for Fall Quarter……………………………………………………...11
Table 2.5: Gantt Chart for Winter Quarter…………………………………………………..12
Table 2.6: Gantt Chart for Spring Quarter…………………………………………………..12

viii
Chapter 1

Introduction

1.1 SIDS Background

Sudden Infant Death Syndrome, abbreviated as SIDS and also known as crib death, is one
of the leading causes of mortality among infants between one month and one year of age. While
the number of SIDS cases has decreased to an all-time low thanks to awareness raised by various
institutes, there are still around 1600 cases in the United States alone [1]. The exact cause of
SIDS is not known, but it is believed to be linked to abnormalities in the arcuate nucleus, which
is the portion of the brain that controls breathing and arousal from sleep. Sudden Infant Death
Syndrome is a single category under the classification of Sudden Infant Unexpected Death
Syndrome, or SIUDS [2]. The numbers alone shown the prevalence and seriousness related to
SIDS and SIUDS. Throughout the rest of the paper both cases will be referred to as SIDS as the
project is more targeted towards tracking the breathing rate of the subject.

1.2 Project Need

There is a need in the world for a baby monitor that can more accurately keep track of a
baby’s condition. Many current baby monitors consist of either a video camera or a microphone
or a combination of the two. Naturally there are some limitations to having such a simple system
with a main drawback being that a parent may not know if the baby is experiencing an episode of
SIDS. Cameras and microphones by themselves and in tandem are not able to detect when SIDS
is happening, which causes a greater response time from the parent, which in turn could lead to
the loss of life. Being able to keep track of a baby’s state should give parents a better peace of
mind which results in better sleep at night, something that every parent or guardian could use.

1.3 Project Objectives

The goal of the project is to create a noninvasive video based baby monitoring system
with Internet of Things, or IoT, capabilities. By detecting if the baby is experiencing an
abnormal breathing rate the alarm will be generated to notify the parents or guardian. Through
the generation of an alarm the parents or guardian will have a quicker response time in the event
of an episode of SIDS, resulting in a saved life. The project was inspired by a motion

1
magnification technique in video developed by MIT called Eulerian magnification, discussed
further below. The IoT is utilized to communicate through SMS, or short message services, to
message the parent or guardian as soon as an abnormality is detected.

1.4 About the Internet of Things

The Internet of Things, abbreviated as IoT, simply refers to a network of objects that are
connected to the internet. Since the Internet of Things is such a broad category it encompasses
many devices and is growing at a rapid rate. In 2015 there were approximately 15.4 billion IoT
devices. IHS Markit, a financial resource company, expects there to be 30.7 billion IoT devices
by 2020 while Intel, a technology company, expects there to be 200 billion by 2020 [3].
Currently of the 15.4 billion devices about 28.3 million are wearable, but that number is expected
to increase to around 80 million by 2020. The total global spending on the IoT in 2016 was 737
billion dollars, and was projected to reach 1.29 trillion dollars in 2020. As one can see based off
of the numbers alone, the IoT is a prominent field that is only getting bigger. Perhaps one of the
main reasons why the IoT is so large is that it aims to make life more convenient, and people are
more likely to invest in things that make their lives easier. The IoT is integrated into our baby
monitoring system for a quick response time and to provide a greater sense of security for
parents during the night.

1.5 Eulerian Magnification Background

In 2013 a group of nine MIT students released a video processing technique called
Eulerian Magnification. The goal of Eulerian Magnification was to magnify movements and
motions that cannot be seen with the naked eye. Eulerian Magnification operates by taking in
any standard video sequence as an input and applies spatial decomposition and temporal filtering
to the frames [4]. The end result is an output video with greatly amplified movement even if
very little visible change could be detected in the original video. Previous attempts made to
exaggerate subtle motions in video involved the Lagrangian perspective, which is based off the
theory of tracking trajectory over time to do motion estimation. However, these methods have
proved to be computationally expensive and prone to unwanted behavior compared to more
recently developed Eulerian magnification.

2
Chapter 2

Design and Implementation

2.1 About the Artik 10 Board

The team was given an Artik 10 board to be used in the implementation of the project.
The Artik 10 is an IoT board released by Samsung with a octa-core architecture and a quad
processor. It also has a 32 bit ARM CPU that makes it ideal for high computation activities such
as image processing, 3D graphics, and drones. The board has DRAM and flash memory
capabilities while also supporting Wi-Fi, Bluetooth, and Zigbee. In addition the Artik has USB
and MIPI camera compatibility and a Mali T628 MP6 graphic processing unit that can show HD
videos at 120 frames per second. Finally, the physical board comes at a relatively small size of
29x39x1.3 mm.

Figure 2.1: The Samsung Artik 10 board

2.2 System Diagram

Figure 2.2 is a pictorial diagram that represents the final operation of our system. The
system can be broken down into three major subcategories: the video acquisition portion, the
breath detection algorithm portion, and the alarm portion. The first system subcategory, the
video acquisition portion, is represented by the first two sequential images in the diagram, which
are a video camera and the Artik board. A night vision video camera is connected to the Artik
board and used to capture a video of the baby. The second system subcategory, the motion
detection algorithm portion, is represented by the third and fourth images in the sequence. The
third image represents the process of Eulerian magnification and the fourth image represents the
algorithm for motion detection. The third and final system subcategory, the alarm portion, is

3
represented by the fifth and sixth sequential images, which are Twilio and a cell phone. Twilio
is used to generate an alarm to a parent’s phone in the event that an emergency situation is
detected from the algorithm.

Figure 2.2: Final Design Pictorial Diagram

2.3 Requirements
The requirements for the baby monitoring system can be broken down into two
categories: functional and nonfunctional requirements. Functional requirements regard what the
system does while nonfunctional requirements outlines the manner in which the functional
requirements will be accomplished. Both types of requirements are tabulated below:

Table 2.1: Functional and Nonfunctional requirements

Functional Requirements Nonfunctional Requirements

(CMAG will…) (CMAG will be…)

Accurately keep track of baby’s state Easy to use

Run in real time Maintainable

Run for long periods of time Reliable

Generate alarm in emergency situation Safe

4
2.4 Constraints
One of the main constraints dealt with in the project are the limitations of Eulerian
magnification. Since Eulerian magnification was designed by an outside team making specific
changes without understanding the entire overview could risk potentially breaking the algorithm.
Many of the files in Eulerian magnification are connected to one another, so one small change
could trickle down and cause unforeseen problems down the line. The team avoided modifying
the Eulerian code as much as possible to avoid these potential mistakes.
Eulerian magnification also works best on a stable video, which means that it is
imperative to have the camera stabilized while filming is taking place. To ensure camera
stability a bendable stand with a clip on the end was used to hold the camera in place. The stand
was bendable enough to be shaped to the required position but sturdy enough to hold the camera
still when it was attached. If the camera is moved during filming Eulerian magnification may
produce an unstable result and therefore the data from the outputted video may be unable to be
accurately analyzed.
The runtime of MATLAB was one of the final major constraints that needed to be taken
into consideration. While the algorithms the team developed ran rather quickly in contrast
running Eulerian magnification took a rather substantial amount of time. Processing a 30 second
video clip took around a minute and thirty seconds in MATLAB, which is a rather lengthy
amount of time. The runtime of Eulerian magnification was one of the constraints that was not
able to be successfully worked around, but is something that can hopefully be reduced in the
future.

2.5 Initial Implementation Method

Figure 2.2 is a block diagram of outlining the team’s initial implementation method.
Originally the plan was to capture and store a video of a baby on the Artik board in real time.
After that Eulerian magnification would be run on the video on the board, creating a new video
file that contained amplified movement. The chest movements of the baby from the amplified
video would then be converted into a sinusoidal wave for analysis. The wave, representing the
baby’s chest movements, would then be uploaded into a cloud, which could then be accessed
later by a parent or medical provider. The wave would then be analyzed in the cloud for any
abnormalities. If any abnormality is detected than an alarm would be generated to the user’s

5
phone. In addition to the streamed and amplified video would be available to view at any point,
most likely through the user’s phone.

Figure 2.3: Initial Implementation Block Diagram

2.6 Implementation Alternatives

One of the main implementation alternatives considered was to use a raspberry pi. A
raspberry pi would have been cheaper and more compact, which are two pretty significant
features over the Artik board. However, as the team received the Artik 10 board for free there
was an obligation to see what could be done with the technology. If working with the Artik
proved to be too complicated or if there were too many unsupported features then a raspberry pi
could be considered.
There were also a couple of viable different implementation methods that could be used
for alarm generation. One possible idea that was suggested was creating a phone application
with a bell. The bell would shake in the event of an emergency. Perhaps the main issue
associated with an application would be the risk of the user having a dead phone or
unintentionally muting their phone before bed. Another possible implementation method for the
alarm system could be a wireless receiver attached to any sort of physical noisemaker that would
generate noise in the event of an emergency.

6
 In the end the team decided to work with the Artik board as all of the tests that involved
capturing short videos through the Artik were successful. There was also the prospect of running
Eulerian magnification through the Artik board due to its high processing capabilities. The final
implementation method used for generating an alarm was to send a text message to the user’s
phone. Even though there was still risk of having a user with a muted or dead phone using SMS
was the simplest yet effective way to let someone know that something is wrong.

2.7 Technical Familiarity

Before beginning the project the team had some experience coding in MATLAB and C.
Experience with MATLAB came from taking a variety of electrical engineering courses, such as
Electric Circuits, Linear Systems, and Digital Signal Processing. Experience with C came from
taking introductory computer engineering courses such as Introduction to Programming,
Advanced Programming, and Data Structures. The Linear Systems and Digital Signal
Processing course provided invaluable framework for the project.
Working with the Artik was a challenge for the team since it was a relatively new product
and had limited documentation. Going through various tutorials was helpful to get a better
understanding of the functionality of the board. Even though the team was familiar with
MATLAB, more functions had to be learned in order to advance the project. Eulerian
magnification had many previously unheard of functions that needed to be researched for better
understanding. In addition some Java needed to be learned in order to generate a text message
with Twilio.

2.8 Technologies Used

In addition to the Artik board, additional other hardware and software were required for
use in the project. The only other piece of hardware used in the project was a night vision
camera. Access to a computer and cellular phone was also required to make use of software such
as PuTTY, MATLAB, and Twilio. The hardware and software use is detailed below:

2.8.1 Night Vision Camera

Since the intent of the project was to record footage of a baby sleeping during the night a
night vision camera was necessary. The team has selected to use a 5 MP CMOS UVC USB

7
Camera module with a 2.1 mm lens and 24 infrared LEDs. The camera also has a high frame
rate of 30 fps and a 1280x720 resolution in addition to coming attached to a 1m cable.
Originally the team looked for MIPI cameras but chose a USB camera in the end due to a wider
variety of night vision options and compatibility. The camera chosen is a relatively high quality
piece of equipment as the team was initially not sure what the optimum resolution and fps would
be to work at.

Figure 2.3: Night Vision Camera

2.8.2 PuTTY
PuTTY is a free open source SSH, or secure shell, program written in C and was the
primary method of issuing commands and communicating with the Artik board. By connecting
the Artik board to a computer through USB and launching PuTTY one can log in to the board.
On the first run through PuTTY has to be initialized by inputting the serial communication port
number that is read through the device manager. The speed of the serial link in bits per second
should be set to 115200 with connection type set to serial. After these parameters are saved
PuTTY does not need to be reinitialized on subsequent launches.

2.8.3 MATLAB
MATLAB, an abbreviation of Matrix Laboratory, is a numerical computing environment
that is used by millions of scientists and engineers across the world. MATLAB is a very
powerful tool that comes with a large variety of applications and functions that can be used for a
wide range of purposes, such as graph creation and system simulations. Eulerian magnification
was constructed in MATLAB so access to the program was necessary for the project. MATLAB

8
was also the primary choice for breath detection algorithm development due to how powerful the
numerical analysis is and the existing compatibility with MIT’s code.

2.8.4 Twilio
Twilio is a cloud communication platform as a service that allows developers to send and
receive text messages using a web service application programming interface, or API. Using the
Eclipse IDE, or integrated development, for Java one can use Twilio to generate text messages to
a user’s phone. After creating an account one receives a unique security identifier, or SID, and
authentication token that needs to be used in order to send and receive text messages. A Twilio
trial account also provides one free phone number to send messages from with the user’s desired
input.

2.9 Cost
The total cost for the project is detailed in Table 2.2. The total cost for all materials sums
to $200. As the Artik board was loaned to us for free the effective cost to the team was $50. The
school provides access to MATLAB for free, which reduced the cost of the project as basic
access to MATLAB costs $50 for students and $150 for home users. A free Twilio account was
also used although it could be potentially upgraded for further functionality, such as access to
multiple phone numbers.

Table 2.2: Cost of Materials

Item Cost

Artik 10 Board $150

Night vision camera $50

MATLAB Access Free with school

Twilio Access Free trial account

Total: $200

2.10 Risk Analysis

During the design process there were certain risks that needed to be addressed. The

9
following table, table 2.2, lays out five of the biggest risks, the probability of occurrence, the
severity of the issue, the impact in the event of occurrence, and a mitigation strategy. The impact
is calculated by multiplying the probability by the severity.

Table 2.3: Risk Analysis

Risk Probability Severity Impact Mitigation

Strategy

Artik does not .6 7 4.2 -Try to utilize

have necessary known features
feature(s) -Use different
hardware

Running out of .5 8 4 -Try to stick to

development schedule
time -Have weekly
meetings to stay
updated

Lack of technical .5 7 3.5 -Do more

ability research
-Get outside help

Requirements .4 4 1.6 -Be flexible to

change new
requirements
-Ask why did
old requirements
change

Components .3 5 1.5 -Keep

break components in
box
-Keep spare
components

2.11 Development Timeline

The Gantt chart for each quarter as predicted is as detailed below, and there are three
quarters. Green highlight indicates that initial plan was followed to completion while orange
highlight indicates that the initial plan was not followed.

10
Table 2.4: Gantt Chart for Fall Quarter

11
Table 2.5: Gantt Chart for Winter Quarter

Table 2.6: Gantt Chart for Spring Quarter

12
Chapter 3

Roadblocks

3.1 Internet Connectivity

One of the first issues that arose when working with the Artik board was connecting to
the internet. Since the team initially wanted to store data in a cloud the board would need to be
able to connect to the internet. After connecting the Wi-Fi antenna to the Artik board and
launching PuTTY the wpa_supplicant.conf file can be modified to allow access to the internet.
In PuTTY a Wi-Fi scan command can be entered to see the available networks. Initially when
the scan command was run no networks appeared. The no network problem was remedied by
updating the Artik Image.
By running the cat /etc/artik-release command the current version of the board was found
and it was discovered that an update was necessary. Through use of a microSD card and reader
and the program Etcher the image on the board was able to be updated and some of the issues
were resolved. Checking to see if the latest image was installed on the board should have been
done earlier but it is sometimes easy to overlook factors like that.
After updating the Artik image networks appeared when running the scan command.
From then the wpa_passphrase command was run to save the school’s network information to
the board. Using the vi command the wpa_supplicant was able to be verified by matching the
correct contents of the file. Despite making sure that the supplicant file had what was required,
the board still did not have connectivity when trying to do an additional update. The lack of
connectivity may have had to deal with stricter security measures on the network created by the
school. To circumvent the Wi-Fi connectivity issue an ethernet cable was plugged directly into
the board to finish the updates.

3.2 Streaming
Video streaming would have ideally been used in order to continuously keep track of the
baby. An ffserver.conf file could be modified in order to stream using the Artik, using Artik as a
RTSP, or real time streaming protocol, server. Using a video player program on a computer such
as mplayer it would then be possible to view the video captured from the board. Despite creating
a ffserver.conf file that was exactly the same as the recommended one, launching the ffserver

13
yielded no response in PuTTY.
Since running the ffserver was unsuccessful, python could also be used to implement
streaming in combination with OpenCV and Flask. Flask is a micro web framework that uses a
technique where the server provides a response to a request in chunks. The next chunk in line
replaces the previous chunk to produce an animated effect in a browser. Unfortunately
attempting to stream with python code also yielded no response. Debugging errors with the
Artik and PuTTY is a rather difficult task since the Artik is a relatively new product, shipped in
May of 2016. Due to the shorter lifespan of the product there is much less documentation
available, and if the official documentation does not run as expected then there are not many
other resources to turn to. It is still not known why both attempts to stream using the Artik have
failed.
Even if the streaming had worked as intended there would still be the issue of running
Eulerian magnification on the stream in real time. To work around the issue of not being able to
stream and running Eulerian magnification on a stream a method was devised utilizing a known
working feature of the Artik board. The method devised involved capturing short video
segments through Artik and then running Eulerian magnification on the shorter clips for later
analysis, which was the process used in the final design.

3.3 Running Eulerian on Artik

The MATLAB Coder application can be used to convert MATLAB code into C code.
The Coder was intended to be used in order to run Eulerian magnification on the Artik board.
When running the Coder on MATLAB code a compatibility measurement is created on a scale of
one to five. Eulerian magnification is composed of 95 separate MATLAB files and there are
frequent occurrences of calling MATLAB unique functions that had poor compatibility with the
Coder. Poor compatibility can be fixed by altering and reformatting the code contents, but the
team did not want to risk changing files without understanding the bigger effects. After realizing
that too many of the files had poor compatibility conversion ratings and realizing that too much
of the file contents had to be changed the idea of attempting to run Eulerian magnification on the
board was abandoned. To carry on with the project and to test the legitimacy of the idea
Eulerian magnification was run directly through the computer instead.

14
3.4 Using Twilio for SMS
There were a two big initial issues when using Twilio in an attempt to send a text
message using Java and Eclipse. The first issue that arose was that the code failed to generate
any message at all. Since the code was derived based off of given sample code from the Twilio
website the fault probably did not lie in the code itself. The failure to generate a SMS was fixed
by authenticating a phone number into the Twilio console user settings. Ideally other phone
numbers could receive text messages by changing the user’s number within the Java code,
however when testing with different numbers the phones did not receive any sort of message.
The reason the other phones were unable to receive messages is likely due to the limitations of
using a Twilio trial account and having only one phone number assigned to the account.

15
Chapter 4

Breath Detection Algorithm Testing and Results

4.1 Capturing Video

To capture video clips through Artik the ffmpeg package needs to be installed. Ffmpeg is
commonly used package used to encode and decode video. Once the package has been installed
a video can be captured using the ffmpeg -f alsa -ac 2 -i hw:0 -f v4l2 -s 640x480 -i /dev/video0 -t
20 video.mpg command. Despite having a higher available resolution on the camera, the film
was captured at a lower resolution of 640x480. The lower resolution speeds up the process of
capturing clips and a higher resolution was not necessary for the purposes of the breath detection
algorithm.

4.2 Counting Pixels

After the video clip is captured and moved to a computer Eulerian magnification can be
run through MATLAB. Once the movement in the clip has been amplified the movement of the
baby’s chest can then be more easily tracked over time. Figure 4.1 is a 640x480 matrix that
contains a number corresponding to a pixel color value in a single frame. The frame rate of the
video, 30 fps, and the total number of frames in the video are stored as variables for further use.

Figure 4.1: Pixel Representation of a Frame

After Eulerian magnification is run the clip is converted to grayscale using the rgb2gray

16
function to reduce the numerical range of the pixel values, which also reduces the variance of
different recorded scenarios. After converting the video to grayscale and going through multiple
frames 200 was chosen as a threshold value to represent significant change, and anything below
that value could be considered as noise. Using MATLAB a basic filter was designed to ignore
the pixel values below the cutoff value and to calculate the sum of the total count of pixels with
value greater than 200 in each frame.

4.3 Representing Motion Over Time

Using MATLAB to plot the sum of the pixel count greater than the threshold value per
frame a sinusoidal wave is able to be generated that represents the baby’s chest movements over
the course of the video clip. The contrast between figure 4.2 and figure 4.3 shows the necessity
for Eulerian magnification. Figure 4.2 is a pixel summation over time from a video not
amplified by Eulerian magnification while figure 4.3 is the pixel summation over time from the
same video amplified by Eulerian magnification. The large initial spike is due to a rapid
movement. A much clearer sinusoidal wave can be observed in figure 4.3 compared to figure
4.2 due to the amplified chest movements, which results in a greater color change in each frame.

Figure 4.2: Chest Movement Plot Generated from Original Video

17
 Figure 4.3: Chest Movement Plot Generated from Eulerian Video

To simulate different scenarios the team created 13 different sample videos for analysis.
Some of the videos contained breathing and others were videos containing no movement at all.
The thirteen different videos after Eulerian magnification is plotted in figure 4.4. Even though
many of the videos displayed similar sinusoidal behavior, the magnitudes of each plot vastly
differed, partially due to different lighting conditions. The vast differences in magnitudes makes
it harder to generate an algorithm that will work in multiple scenarios as some of the videos with
no movement, or “noise”, had larger magnitudes than videos with breathing.

Figure 4.4: Chest Movement Plot from 13 Different Videos

18
 To better analyze the plots the magnitude of each plot was “normalized” by calculating
the pixel differences between each frame over time. Using this strategy the magnitude of all of
the different videos with breathing were able to be fit within the range of -3000 to 3000, making
them easier to compare to one another. In addition to that, the plots of the videos containing
noise were reduced to a significantly smaller magnitude than the plots of the videos that
contained breathing. This method of normalization allows for videos of different scenarios to be
easier analyzed.

Figure 4.5: Normalized Chest Movement Plot from 13 Different Videos

Figure 4.6: Normalized Eulerian Video Plot

19
4.4 Detecting Abnormalities
It would be easier to detect an abnormal breathing pattern and generate an alarm working
with data in the frequency domain. After obtaining a normalized pixel count plot in time
domain, a fast Fourier transform, X1 = abs(fftshift(fft(ab, fft_len))), was used in MATLAB to
convert the time domain plot into a frequency plot. The resulting plot from the original video is
shown in figure 4.7.

Figure 4.7: Frequency Domain Plot of Normalized Video

A newborn baby’s maximum breathing rate is 60 breaths per minute, which corresponds
to 1 Hz. Therefore we do not care about frequency values that are much larger than 1 Hz, and so
1.6 Hz was chosen as a cutoff. To eliminate these unnecessary frequencies and to clean up the
plot a low pass filter was added using the B = firpm(100, [0 .09 .1 1], [1 1 0 0]) command in
MATLAB, which is a linear-phase FIR filter using the Parks-McClellan algorithm. Figure 4.8 is
a plot of the low pass filter used and it was designed with small band edges, dropping from 1 at
0.09 to 0 at 0.1. The low pass filter was than applied to the frequency domain plot using the
command filt_out = filter(B, 1, ab). The result is shown in figure 4.9.

20
 Figure 4.8: The 100th Order Parks-McClellan Linear-Phase FIR Low Pass Filter

Figure 4.9: Frequency Domain Plot After Low Pass Filter

A threshold level was determined by analyzing the magnitudes of the frequency plots
from the thirteen different videos. If the magnitude of the frequency plot exceeded the threshold
level that indicates the respiration rate in the video is normal and the system will keep on
working. If the magnitude falls below the threshold level, that indicates that the respiration rate
from the video is not normal and an alarm should be generated. Based on looking at frequency
plots from videos with breathing and videos of just noise a magnitude threshold of 8000 was
chosen. Figure 4.10 and 4.11 are both plots from videos with breathing, with magnitude
differences due to different light conditions in the videos. Figure 4.12 is significantly smaller

21
and is a plot from a video with no breathing and falls under the chosen threshold level.

Figure 4.10: Large Magnitude Breathing Frequency Plot with Threshold Level

Figure 4.11: Low Magnitude Breathing Frequency Plot with Threshold Level

Figure 4.12: No Breathing Frequency Plot with Threshold Level

22
4.5 Using Twilio for Alarm Generation
By utilizing the threshold level a text file is able to be generated from the video indicating
the state of the baby. If the baby is breathing okay then the text file would contain a “1” and if
an emergency is detected then the text file would contain a “0”. With some outside help the
team was able to come up with Java code that would read the contents of the file and utilize
Twilio in order to send a message to a desired phone number. The code was run in the program
Eclipse, which is the most commonly used Java Integrated Development Environment. Upon
running the code a text message would be received to the desired phone number indicating if
normal or abnormal breathing was detected, as shown in figure 4.13. Sending messages when
normal breathing is detected would not be necessary, but was done during testing to see if there
was proper functionality.

Figure 4.13: Text Messages Indicating Normal and Abnormal Breathing

23
Chapter 5

Professional Issues and Constraints

5.1 Ethics
Ethics is ultimately one of the main driving forces behind engineering. The goal of
engineering is to use science and math principles in a way that will improve the lives of others.
However, not all engineers will use their mental capabilities in a manner that is good for the race,
which brings upon the importance of having ethical consideration and discussion. Stopping for a
moment to consider the ethical impacts of a product may actually end up changing the way the
product is designed. If one is not sure about the ethical implications of a product there is also the
possibility to have an ethical discussion with fellow engineers. Engineering is not just a one
person field and there are many people around willing to have these types of discussions. Ethical
consideration is important because there is possibility that even the best product can be misused
in the wrong hands.

5.2 Ethical Justification

The subject of SIDS is sometimes a difficult matter to discuss as it can appear to happen
out of the blue and is often unexplainable, causing much grievance among the family. While the
project cannot guarantee SIDS prevention, it can shorten parent’s response time in the
occurrence of an episode, perhaps leading to a saved life. In that light the project is more of a
reactive measure than a guaranteed solution to the problem. People should be educated to
understand that a complete solution to SIDS may not even be possible, but also understand that
taking these preventative measures is never a waste of effort.

5.3 Sustainability
Successful businesses and practices are distinguished from unsuccessful ones by their
entrepreneurial skills and the quality of their product. To understand some of the defining
aspects of entrepreneurship and quality and how to ensure them it helps to look at a seemingly
unrelated topic, sustainability. There are three different aspects of sustainability: environmental,
social, and economic, and all three are equally important links between entrepreneurship, quality,
and success. If a product was noticeably environmentally unsustainable, it would likely be

24
boycotted in the market. If a product was socially unsustainable, then consumers would have no
desire to purchase the product as it does not improve upon their daily lives. Likewise, if a
product is not economically sustainable people would not purchase it due to its unaffordability or
the company producing it would go out of business. Sustainability factors are very crucial to the
success of any product or project.

5.3.1 Environmental Sustainability

As an engineer it is also irresponsible to release a product that is environmentally
unsustainable as it is part of our duty as human beings to take care of the planet for ourselves and
for future generations. A video based baby monitoring system has good environmental
sustainability as it is not a detriment to the environment. The Artik board needs to be connected
to a wall socket to be powered and connected to a computer for communication with the camera
receiving power from the board. There are no additional power requirements necessary to run
the project, excluding having access to a powered phone.
As long as none of the major components are severely damaged then the project will
likely have a long lifespan compared to the time the project is actually in use. Due to the
nonviolent nature of the usability of the project frequent disposability should not be an issue.
Since the Artik 10 is a relatively newly released product it is hard to estimate its lifespan but is
likely similar to that of the raspberry pi, which is two to three years. For the times when the
product does need to be disposed of, many of the components can be turned into a plant that
handles and recycles circuit boards.

5.3.2 Social Sustainability

Social sustainability is an equally important aspect in any project or product as it is the
driving force behind people’s willingness to use or purchase the project or product. Hopefully
people will understand that the project is for the social purpose of saving babies lives while also
bringing parents a better sense of security at night. These are two factors that are hard to put a
price on. SIDS cases happen around the world and is a truly terrible occurrence. If our project
truly works as intended then there would be no social substitute for it since it is an attempt to
reduce the number of lives lost and the heartbreak of the families that follows.

25
5.3.3 Economic Sustainability
The project is perhaps weakest in the economic sustainability department. The Artik 10
board is new and not very cheap, coming in at $150. The purchased camera was around $50, so
the hardware total is around $200. The hardware price is slightly problematic since many
commercial baby monitors can fall anywhere in the range of $15 to $180, so the project comes in
on the more expensive side. The price of hardware also does not include the price of necessary
software subscriptions either, so that is a hidden added cost. With time the cost of the Artik
board will hopefully decrease significantly, bringing down the total cost of the project. Cheaper
alternative hardware components could also be used if cost really becomes that big of an issue.
While the price is quite steep, people will hopefully find it to be worth the cost.

5.4 Health and Safety

The health and safety of the baby is the project’s top priority, and therefore causing injury
to the baby in any way would mean failure. Since the hardware would be operated out of the
baby’s reach there should be little choking or strangulation risk involved. Infrared LEDs on the
camera for night vision may not be the healthiest for use on young babies, especially when
considering how long they will be in use for. Hopefully holding the camera several feet above
the baby will help to mitigate this danger, but unfortunately there are few alternatives as video
will need to be captured at night. Another major potential issue is generating an alarm when
there is no emergency or failing to generate an alarm when there is an emergency. Unfortunately
both of these cases are not easy to plan for, so further testing will need to be done to mitigate
such scenarios.

5.5 Manufacturability and Usability

The project is relatively easy to manufacture since there are not too many components
involved. The setup for the hardware components involves attaching a camera into the Artik
board, which is a one step process that any user could do. A more complicated aspect involves
stabilizing the camera. Since the team has not created a dedicated stand this is an issue that
could be further improved on, but having a stabilized camera is imperative to the success of the
project. In an ideal scenario the Artik board and camera would be setup on the top of a baby’s
crib, with the camera several feet above the baby’s chest.

26
 The software does not have to be manufactured more once it is completed since it can just
be copied multiple times. However, using the software components is a little more tricky since it
requires basic knowledge of Eclipse and MATLAB. Some of the breathing detection MATLAB
algorithm is designed to loop through the number of captured video clips in an attempt to make
the process easier for the user. The alarm written in Eclipse is also designed to run simply
through the use of the run feature after the breathing algorithm has been run. Ideally in the end
the whole process will be able to be operated off a single unit, but the team was not able to bring
the project that far yet.

5.6 Civic Engagement

The project pertains to the community in the sense that it was designed for the safety and
security of families. Modern society works because at the end of the day people have the
opportunity to go home to their families where they can feel safe and secure. When the right to
familial time is taken away or, in the case of SIDS, threatened, that can have serious repercussion
to the functionality of society. By providing this sense of safety and security to families the team
hopes that parents will keep their willingness to continue with their civic duties.

27
Chapter 6

Conclusion and Future Work

6.1 Acceptance Criteria

Unfortunately not all of the requirements initially set by the team could be met. The
main requirement that could not be met was having the baby monitoring system run in real time.
The system is not ready for use in a real environment since it does not run in real time. As the
real time portion of the project was not completed the long duration run time of the system could
not accurately be tested or measured either. A work around of capturing short clips was used in
order to test the functionality of the rest of the project.
Despite the incompletion of the having an automated system run in real time, the goal of
detecting breathing in a video was successfully accomplished. Being able to track and monitor a
person’s breathing rate with a video camera has potentially huge applications within the medical
field. In addition to monitoring the subject’s breathing rate for abnormal patterns the team was
also able to generate a text message in the event of an emergency. A breath detector would not
be very useful in its own right if it constantly had to be monitored, so utilizing SMS smoothens
the process

6.2 Future Work

As stated above the main feature that needs to be worked on is the automation and real
time performance of the system. In addition to the real time and automation another issue that
needs to be dealt with is testing the system on babies. Getting access to and testing on babies is
an issue that was avoided by testing the system on the team members. The principle concept
remains the same, but some threshold values for breath detection will likely need to be changed
due to anatomical differences in newborns.
A major limitation holding back the real time process is the lengthy run time of Eulerian
magnification. It takes approximately a minute and thirty seconds to process a thirty second
video with MATLAB. The long run time hurts the viability of the project, but can hopefully be
improved on in time. Improving the runtime of Eulerian magnification is certainly not an easy
task and will likely require a great amount of research into the entire process and probably a
dedicated team.

28
 Additional features could also be added to the baby monitor in order to improve accuracy
and functionality while utilizing more of the features of the Artik board. The main example is
the addition of a microphone to be added in tandem with the video processing to act as additional
monitor on the baby’s condition. A separate alarm could be generated if the baby is making loud
noise or making noise for a long period of time. Another feature that could be worked on is
having an amplified video of the baby streaming in real time, as the team did not prioritize
getting this feature to work.
Finally, the entire system would ideally operate on one contained unit. If it were possible
for the board to run Eulerian magnification, the algorithm for breath detection, and generate an
alarm to the user’s phone then the project could be considered truly complete. Although the
team did not complete all of the originally intended features they have hopefully provided a
strong framework to see the rest of the project to completion.

6.3 Lessons Learned

The team learned invaluable lessons working on a project with such a large scope. As the
senior design project was the first project both members have worked on that took place over an
entire year it would be an exaggeration to say that the entire process went smoothly. One lesson
learned from the process involved the importance of setting goals and appropriately changing
said goals if they cannot realistically be met. The initial scope of the project was found to not be
feasible within the team’s technical ability so the scope had to be changed, which ended up being
in the project’s best interest.
In addition the team learned about the importance of giving precise presentations. On
different occasions through the quarter the team had to give two intermediary presentations
lasting fifteen minutes each and one final presentation lasting twenty minutes. There is a vast
amount of technical details and information that could be given, but a truly great presentation is
able to convey the most important points in a concise manner. The team also learned about the
importance of practice, as it is certainly a rarity in life to get something right on the first run
through.

29
References
[1] "Sudden Unexpected Infant Death and Sudden Infant Death Syndrome." Centers for
Disease Control and Prevention. Centers for Disease Control and Prevention, 17 Apr.
2017. Web. 20 May 2017.

[2] "Fast Facts About SIDS." Eunice Kennedy Shriver National Institute of Child Health and
Human Development. U.S. Department of Health and Human Services, n.d. Web. 20 May
2017.

[3] 28, 2017 Kelvin Claveria | April. "13 Stunning Stats on the Internet of Things." Vision
Critical. N.p., 28 Apr. 2017. Web. 31 May 2017.

[4] Wu, Hao-Yu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, and William
Freeman. "Eulerian Video Magnification for Revealing Subtle Changes in the
World."ACM Transactions on Graphics 31.4 (2012): 1-8. Web.

30
Appendix A
MATLAB

clearvars;
clc;

myDir = pwd; %gets directory

myFiles = dir(fullfile(myDir,'vid*.avi')); %gets all txt files in struct

for vid_num=1:length(myFiles)

v = VideoReader(eval(sprintf('''vid%d.avi''', vid_num)));
%FFT length.
fft_len = 512;

%Video rate is 30 frames per second.

vid_rate = 30;
%Get number of frames in the video file.
vid_len = v.NumberOfFrames;

%Generate arrays of zeros.

aa = zeros(1,fft_len+1);
ab = zeros(1,fft_len);

%Loop through every frame and calculate a sum for each frame.
for ii = 1:fft_len+1
video1 = read(v,ii);
video1bw = rgb2gray(video1);
%frame_val = sum(sum(video1bw));
frame_val = sum(video1bw(:) > 200);
aa(ii) = frame_val;
end
%Loop through and calculate the difference between frames.
for ii = 1:fft_len
video1 = aa(ii);
video2 = aa(ii+1);
ab(ii) = video1 - video2;
end

I
%Calculate x-axis values -15Hz to 15Hz.
dW = vid_rate/fft_len;
W = 1 : 1 : fft_len;
W = W - .5*fft_len;
W = W*dW;
%FFT of difference calculation.
X1 = abs(fftshift(fft(ab, fft_len)));
%Create low pass filter with cutoff of 1.5Hz.
B = firpm(100, [0 .09 .1 1], [1 1 0 0]);
[H,WD]=freqz(B,1,1024,'whole');

%Filter difference data.

filt_out = filter(B, 1, ab);
X2 = abs(fftshift(fft(filt_out, fft_len)));

figure(1);
plot(aa);
xlabel('num of frames');
ylabel('count of pixels value > 200');
grid on;
hold on;

figure(2);
plot(ab);
xlabel('Num of frames');
ylabel('Normalized pixels count');
grid on;
hold on;

%{
figure();
plot(W, X1);
ylim([0 max(X1)]);
xlabel('Frequncy/Hz');
grid on;
grid minor;
set(gca,'xtick',[-15:1:15]);

II
 %ylim([0 80000])

figure();
plot((WD-pi)/pi, abs(fftshift(H)));
title('Low Pass Filter w/ 100th order')
%}

figure(vid_num);
plot(W, X2);
ylim([0 max(X1)]);
xlabel('Frequncy/Hz');
grid on;
grid minor;
set(gca,'xtick',[-15:1:15]);
ylim([0 80000]);

trigger = max(X2);

if(trigger) > 8000;

disp(' Breathing Detected.');

fp = fopen(sprintf('breath%d.txt', vid_num),'w');
fprintf(fp, '1');
fclose(fp);
else
disp('No Breathing Detected!');
fp = fopen(sprintf('breath%d.txt', vid_num),'w');
fprintf(fp, '0');
fclose(fp);
end

end

III
Appendix B
Twilio
package santaC;

import com.twilio.Twilio;
import com.twilio.rest.api.v2010.account.Message;
import com.twilio.type.PhoneNumber;

import java.io.IOException;
import java.io.FileReader;
import java.io.BufferedReader;

public class sendSMS {

public static final String ACCOUNT_SID ="ACca0c11ecdf5922a8be6f79f08ce7080b";

public static final String AUTH_TOKEN = "867cf67acecbd34a6d3463af5214f628";

public static void main(String[] args) {

readFromFile();
System.out.println("Done.");

static int readFromFile(){

try{
for (int incr = 1; incr < 10; incr ++){
String path =
"C:\\Users\\Kyle\\Desktop\\santaC\\src\\santaC\\breath";
FileReader fr = new FileReader(path+incr+".txt");
BufferedReader readerFile = new BufferedReader(fr);
String str = readerFile.readLine();
int val = Integer.parseInt(str);
System.out.println(str);
if(val == 0) sendAlertSMS(); //send SMS alert if file value is 0.
if(val == 1) noAlertSMS();
}

IV
 }catch (Exception e){}
return 0;
}

static void sendAlertSMS(){

Twilio.init(ACCOUNT_SID, AUTH_TOKEN);

Message message = Message.creator(new PhoneNumber("+18083893235"),

new PhoneNumber("+18086383611"),

"Alert! Abnormal breathing rate detected!").create();

System.out.println(message.getSid());
}
static void noAlertSMS(){
Twilio.init(ACCOUNT_SID, AUTH_TOKEN);

Message message = Message.creator(new PhoneNumber("+18083893235"),

new PhoneNumber("+18086383611"),

"Normal breathing rate detected").create();

System.out.println(message.getSid());
}
}