Activity Recognition using Cell Phone Accelerometers

Jennifer R. Kwapisz, Gary M. Weiss, Samuel A. Moore

Department of Computer and Information Science

Fordham University
441 East Fordham Road
Bronx, NY 10458
{kwapisz, gweiss, asammoore}@cis.fordham.edu

ABSTRACT
Mobile devices are becoming increasingly sophisticated and the
1. INTRODUCTION
Mobile devices, such as cellular phones and music players, have
latest generation of smart cell phones now incorporates many
recently begun to incorporate diverse and powerful sensors. These
diverse and powerful sensors. These sensors include GPS sensors,
sensors include GPS sensors, audio sensors (i.e., microphones),
vision sensors (i.e., cameras), audio sensors (i.e., microphones),
image sensors (i.e., cameras), light sensors, temperature sensors,
light sensors, temperature sensors, direction sensors (i.e., mag-
direction sensors (i.e., compasses) and acceleration sensors (i.e.,
netic compasses), and acceleration sensors (i.e., accelerometers).
accelerometers). Because of the small size of these “smart” mo-
The availability of these sensors in mass-marketed communica-
bile devices, their substantial computing power, their ability to
tion devices creates exciting new opportunities for data mining
send and receive data, and their nearly ubiquitous use in our soci-
and data mining applications. In this paper we describe and evalu-
ety, these devices open up exciting new areas for data mining
ate a system that uses phone-based accelerometers to perform
research and data mining applications. The goal of our WISDM
activity recognition, a task which involves identifying the physi-
(Wireless Sensor Data Mining) project [19] is to explore the re-
cal activity a user is performing. To implement our system we
search issues related to mining sensor data from these powerful
collected labeled accelerometer data from twenty-nine users as
mobile devices and to build useful applications. In this paper we
they performed daily activities such as walking, jogging, climbing
explore the use of one of these sensors, the accelerometer, in or-
stairs, sitting, and standing, and then aggregated this time series
der to identify the activity that a user is performing—a task we
data into examples that summarize the user activity over 10-
refer to as activity recognition.
second intervals. We then used the resulting training data to in-
duce a predictive model for activity recognition. This work is We have chosen Android-based cell phones as the platform for
significant because the activity recognition model permits us to our WISDM project because the Android operating system is free,
gain useful knowledge about the habits of millions of users pas- open-source, easy to program, and expected to become a domi-
sively—just by having them carry cell phones in their pockets. nant entry in the cell phone marketplace (this is clearly happen-
Our work has a wide range of applications, including automatic ing). Our project currently employs several types of Android
customization of the mobile device’s behavior based upon a phones, including the Nexus One, HTC Hero, and Motorola Back-
user’s activity (e.g., sending calls directly to voicemail if a user is flip. These phones utilize different cellular carriers, although this
jogging) and generating a daily/weekly activity profile to deter- is irrelevant for our purposes since all of the phones can send data
mine if a user (perhaps an obese child) is performing a healthy over the Internet to our server using a standard interface. How-
amount of exercise. ever, much of the data in this work was collected directly from
files stored on the phones via a USB connection, but we expect
Categories and Subject Descriptors this mode of data collection to become much less common in
I.2.6 [Artificial Intelligence]: Learning-induction future work.
All of these Android phones, as well as virtually all new smart
General Terms phones and smart music players, including the iPhone and iPod
Algorithms, Design, Experimentation, Human Factors Touch [2], contain tri-axial accelerometers that measure accelera-
tion in all three spatial dimensions. These accelerometers are also
Keywords capable of detecting the orientation of the device (helped by the
Sensor mining, activity recognition, induction, cell phone, accel- fact that they can detect the direction of Earth’s gravity), which
erometer, sensors can provide useful information for activity recognition. Acceler-
ometers were initially included in these devices to support ad-
Permission to make digital or hard copies of all or part of this work for vanced game play and to enable automatic screen rotation but
personal or classroom use is granted without fee provided that copies are they clearly have many other applications. In fact, there are many
not made or distributed for profit or commercial advantage and that useful applications that can be built if accelerometers can be used
copies bear this notice and the full citation on the first page. To copy to recognize a user’s activity. For example, we can automatically
otherwise, or republish, to post on servers or to redistribute to lists, re-
monitor a user’s activity level and generate daily, weekly, and
quires prior specific permission and/or a fee.
SensorKDD ’10, July 25, 2010, Washington, DC, USA.
monthly activity reports, which could be automatically emailed to
Copyright 2010 ACM 978-1-4503-0224-1…$10.00. the user. These reports would indicate an overall activity level,

SIGKDD Explorations Volume 12, Issue 2 Page 74

 which could be used to gauge if the user is getting an adequate including data collection, data preprocessing, and data transfor-
amount of exercise and estimate the number of daily calories mation. Section 3 describes our experiments and results. Related
expended. These reports could be used to encourage healthy prac- work is described in Section 4 and Section 5 summarizes our
tices and might alert some users to how sedentary they or their conclusions and discusses areas for future research.
children actually are. The activity information can also be used to
automatically customize the behavior of the mobile phone. For
2. THE ACTIVITY RECOGNITION TASK
example, music could automatically be selected to match the ac-
In this section we describe the activity recognition task and the
tivity (e.g., “upbeat” music when the user is running) or send calls
process for performing this task. In Section 2.1 we describe our
directly to voicemail when the user is exercising. There are un-
protocol for collecting the raw accelerometer data, in Section 2.2
doubtedly numerous other instances where it would be helpful to
we describe how we preprocess and transform the raw data into
modify the behavior of the phone based on the user activity and
examples, and in Section 2.3 we describe the activities that will be
we expect that many such applications will become available over
predicted/identified.
the next decade.
In order to address the activity recognition task using supervised 2.1 Data Collection
learning, we first collected accelerometer data from twenty-nine In order to collect data for our supervised learning task, it was
users as they performed activities such as walking, jogging, as- necessary to have a large number of users carry an Android-based
cending stairs, descending stairs, sitting, and standing. We then smart phone while performing certain everyday activities. Before
aggregated this raw time series accelerometer data into examples, collecting this data, we obtained approval from the Fordham Uni-
as described in Section 2.2, where each example is labeled with versity IRB (Institutional Review Board) since the study involved
the activity that occurred while that data was being collected. We “experimenting” on human subjects and there was some risk of
then built predictive models for activity recognition using three harm (e.g., the subject could trip while jogging or climbing
classification algorithms. stairs). We then enlisted the help of twenty-nine volunteer sub-
The topic of accelerometer-based activity recognition is not new. jects to carry a smart phone while performing a specific set of
Bao & Intille [3] developed an activity recognition system to activities. These subjects carried the Android phone in their front
identify twenty activities using bi-axial accelerometers placed in pants leg pocket and were asked to walk, jog, ascend stairs, de-
five locations on the user’s body. Additional studies have simi- scend stairs, sit, and stand for specific periods of time.
larly focused on how one can use a variety of accelerometer- The data collection was controlled by an application we created
based devices to identify a range of user activities [4-7, 9-16, 21]. that executed on the phone. This application, through a simple
Other work has focused on the applications that can be built based graphical user interface, permitted us to record the user’s name,
on accelerometer-based activity recognition. This work includes start and stop the data collection, and label the activity being per-
identifying a user’s activity level and predicting their energy con- formed. The application permitted us to control what sensor data
sumption [8], detecting a fall and the movements of user after the (e.g., GPS, accelerometer) was collected and how frequently it
fall [12], and monitoring user activity levels in order to promote was collected. In all cases we collected the accelerometer data
health and fitness [1]. Our work differs from most prior work in every 50ms, so we had 20 samples per second. The data collection
that we use a commercial mass-marketed device rather than a was supervised by one of the WISDM team members to ensure
research-only device, we use a single device conveniently kept in the quality of the data.
the user’s pocket rather than multiple devices distributed across
the body, and we require no additional actions by the user. Also,
we have generated and tested our models using more users 2.2 Feature Generation & Data Transformation
(twenty-nine) than most previous studies and expect this number Standard classification algorithms cannot be directly applied to
to grow substantially since we are continuing to collect data. The raw time-series accelerometer data. Instead, we first must trans-
few studies that have involved commercial devices such as smart form the raw time series data into examples [18]. To accomplish
phones have focused either on a very small set of users [21] or this we divided the data into 10-second segments and then gener-
have trained models for particular users [4] rather than creating a ated features that were based on the 200 readings contained
universal model that can be applied to any user. within each 10-second segment. We refer to the duration of each
segment as the example duration (ED). We chose a 10-second ED
Our work makes several contributions. One contribution is the
data that we have collected and continue to collect, which we plan because we felt that it provided sufficient time to capture several
to make public in the future. This data can serve as a resource to repetitions of the (repetitive) motions involved in some of the six
other researchers, since we were unable to find such publically activities. Although we have not performed experiments to deter-
available data ourselves. We also demonstrate how raw time se- mine the optimal example duration value, we did compare the
ries accelerometer data can be transformed into examples that can results for a 10-second and 20-second ED and the 10-second ED
be used by conventional classification algorithms. We demon- yielded slightly better results (as well as twice as many training
strate that it is possible to perform activity recognition with com- examples).
monly available (nearly ubiquitous) equipment and yet achieve
Next we generated informative features based on the 200 raw
highly accurate results. Finally, we believe that our work will help
accelerometer readings, where each reading contained an x, y, and
bring attention to the opportunities available for mining wireless
sensor data and will stimulate additional work in this area. z value corresponding to the three axes/dimensions (see Figure 1).
We generated a total of forty-three summary features, although
The remainder of this paper is structured as follows. Section 2 these are all variants of just six basic features. The features are
describes the process for addressing the activity recognition task,

SIGKDD Explorations Volume 12, Issue 2 Page 75

 described below, with the number of features generated for each
feature-type noted in brackets:

• Average[3]: Average acceleration (for each axis)

• Standard Deviation[3]: Standard deviation (for each axis)
• Average Absolute Difference[3]: Average absolute
difference between the value of each of the 200 readings
within the ED and the mean value over those 200 values
(for each axis)
• Average Resultant Acceleration[1]: Average of the square
roots of the sum of the values of each axis squared
√(xi2 + yi2 + zi2) over the ED
• Time Between Peaks[3]: Time in milliseconds between
peaks in the sinusoidal waves associated with most
activities (for each axis)
• Binned Distribution[30]: We determine the range of values
for each axis (maximum – minimum), divide this range into Figure 1: Axes of Motion Relative to User
10 equal sized bins, and then record what fraction of the
200 values fell within each of the bins. Figure 2 plots the accelerometer data for a typical user, for all
three axes and for each of the six activities. It is clear that sitting
The “time between peaks” feature requires further explanation. and standing (Figure 2e,f) do not exhibit periodic behavior but do
The repetitive activities, like walking, tend to generate repeating have distinctive patterns, based on the relative magnitudes of the
waves for each axis and this feature tries to measure the time x, y, and z, values, while the four other activities (Figure 2a-d),
between successive peaks. To estimate this value, for each which involve repetitive motions, do exhibit periodic behavior.
example we first identify all of the peaks in the wave using a Note that for most activities the y values have the largest accel-
heuristic method and then identify the highest peak for each axis. erations. This is a consequence of Earth’s gravitational pull,
We then set a threshold based on a percentage of this value and which causes the accelerometer to measure a value of 9.8 m/s2 in
find the other peaks that met or exceed this threshold; if no peaks the direction of the Earth’s center. For all activities except sitting
meet this criterion then the threshold is lowered until we find at this direction corresponds to the y axis (see Figure 1).
least three peaks. We then measure the time between successive
The periodic patterns for walking, jogging, ascending stairs, and
peaks and calculate the average. For samples where at least three
descending stairs (Figure 2a-d) can be described in terms of the
peaks could not be found, the time between peaks is marked as
time between peaks and by the relative magnitudes of the
unknown. This method was able to accurately find the time
acceleration values. The plot for walking, shown in Figure 2a,
between peaks for the activities that had a clear repetitive pattern,
demonstrates a series of high peaks for the y-axis, spaced out at
like walking and jogging. Certainly more sophisticated schemes
approximately ½ second intervals. The peaks for the z-axis
will be tried in the future.
acceleration data echo these peaks but with a lower magnitude.
The number of examples generated per user for each activity var- The distance between the peaks of the z-axis and y-axis data
ies. These differences are due to the time limitations that some represent the time of one stride. The x-axis values (side to side)
users may have or physical limitations that impact the time they have an even lower magnitude but nonetheless mimic the peaks
spend on each activity. Our data set is summarized in Section 3.1. associated with the other axes. For jogging, similar trends are
seen for the z-axis and y-axis data, but the time between peaks is
less (~¼ second), as one would expect. As one might expect, the
2.3 The Activities range of y-axis acceleration values for jogging is greater than for
In this study we consider six activities: walking, jogging, ascend- walking, although the shift is more noticeable in the negative
ing stairs, descending stairs, sitting, and standing. We selected direction.
these activities because they are performed regularly by many
people in their daily routines. The activities also involve motions For descending stairs, one observes a series of small peaks for y-
that often occur for substantial time periods, thus making them axis acceleration that take place every ~½ second. Each small
easier to recognize. Furthermore, most of these activities involve peak represents movement down a single stair. The z-axis values
repetitive motions and we believe this should also make the ac- show a similar trend with negative acceleration, reflecting the
tivities easier to recognize. When we record data for each of these regular movement down each stair. The x-axis data shows a series
activities, we record acceleration in three axes. For our purposes, of semi-regular small peaks, with acceleration vacillating again
the z-axis captures the forward movement of the leg and the y- between positive and negative values. For ascending stairs, there
axis captures the upward and downward motion. The x-axis cap- are a series of regular peaks for the z-axis data and y-axis data as
tures horizontal movement of the user’s leg. Figure 1 demon- well; these are spaced approximately ~¾ seconds apart, reflecting
strates these axes relative to a user. the longer time it takes to climb up stairs.

SIGKDD Explorations Volume 12, Issue 2 Page 76

 20 20
Y Axis
Y Axis
15 15

10 10

Acceleration
Acceleration

5 5

0 0

-5 X Axis -5
Z Axis
X Axis Z Axis
-10 -10
0 0.5 1 1.5 2 2.5 0 0.5 1 1.5 2 2.5
Time (s) Time (s)
(a) Walking (b) Jogging

20 20
Y Axis
15 Y Axis
15 Z Axis

10
Acceleration
10
Acceleration

5 Z Axis 5

0 0

-5 -5
X Axis X Axis

-10 -10
0 0.5 1 1.5 2 2.5 0 0.5 1 1.5 2 2.5
Time (s) Time (s)
(c) Ascending Stairs (d) Descending Stairs

10 10

Y Axis
X Axis
Z Axis
Acceleration

5
Acceleration

Z Axis

0 Y Axis 0

X Axis
-5 -5
0 0.5 1 1.5 2 2.5 0 0.5 1 1.5 2 2.5
Time (s) Time (s)
(e) Sitting (f) Standing

Figure 2: Acceleration Plots for the Six Activities (a-f)

SIGKDD Explorations Volume 12, Issue 2 Page 77

 As one would expect, sitting and standing do not exhibit any Note that certain activities contain fewer examples than others,
regular periodic behavior and all of the acceleration values are mainly because the users were not asked to perform strenuous
relatively constant. As mentioned earlier, the primary differences activities (e.g., jogging, climbing stairs) for very long and because
between these activities is the relative magnitudes of values for we thought that the patterns in other activities (e.g., standing)
each axis, due to the different orientations of the device with would become apparent quickly so that there would be no need to
respect to the Earth when the user is sitting and standing Thus it waste the users time literally “standing around.” Furthermore,
appears easy to differentiate between sitting and standing, even certain activities, like standing and sitting, were only added after
though neither involves much movement. Note that because the the study began, so we have no data for these activities for some
accelerometers are themselves able to determine orientation with users.
respect to the Earth’s gravitational field, it would be relatively
Once the data set was prepared, we used three classification tech-
straightforward to compensate/correct for any changes in the cell
niques from the WEKA data mining suite [20] to induce models
phone’s orientation due to the phone shifting position in a user’s
for predicting the user activities: decision trees (J48), logistic
pocket. We plan to implement this correction in future work.
regression and multilayer neural networks. In each case we used
3. EXPERIMENTS the default settings. We used ten-fold cross validation for all ex-
In this section we describe our experiments and then present and periments and all results are based on these ten runs.
discuss our results for the activity recognition task.
3.2 Results
3.1 Description of Experiments The summary results for our activity recognition experiments are
Our experiments first require us to collect the labeled raw accel- presented in Table 2. This table specifies the predictive accuracy
erometer data and then transform that data into examples. This associated with each of the activities, for each of the three learn-
process was described in Section 2. The resulting examples con- ing algorithms and for a simple “straw man” strategy. The straw
tain 43 features and cover twenty-nine users. This forms the data man strategy always predicts the specified activity (i.e., walking
set, described in Table 1, which is subsequently used for training for the first row in Table 2 and jogging for the second row of
and testing. The last row in Table 1 shows the percentage of the Table 2) or, when assessing the overall performance of the classi-
total examples associated with each activity. fier (i.e., the last row of Table 2), always predicts the most fre-
quently occurring activity, which happens to be walking. The
Table 1: Number of Examples per User and Activity baseline straw man strategy allows us to consider the degree of
class imbalance when evaluating the performance of the activity
ID Walk Jog Up Down Sit Stand Total
recognition system.
1 74 15 13 25 17 7 151
2 48 15 30 20 0 0 113
3 62 58 25 23 13 9 190 Table 2: Accuracies of Activity Recognition
4 65 57 25 22 6 8 183 % of Records Correctly Predicted
5 65 54 25 25 77 27 273
J48 Logistic Multilayer Straw
6 62 54 16 19 11 8 170
Regression Perceptron Man
7 61 55 13 11 9 4 153
8 57 54 12 13 0 0 136 Walking 89.9 93.6 91.7 37.2
9 31 59 27 23 13 10 163 Jogging 96.5 98.0 98.3 29.2
10 62 52 20 12 16 9 171 Upstairs 59.3 27.5 61.5 12.2
11 64 55 13 12 8 9 161 Downstairs 55.5 12.3 44.3 10.0
12 36 63 0 0 8 6 113
Sitting 95.7 92.2 95.0 6.4
13 60 62 24 15 0 0 161
14 62 0 7 8 15 10 102 Standing 93.3 87.0 91.9 5.0
15 61 32 18 18 9 8 146 Overall 85.1 78.1 91.7 37.2
16 65 61 24 20 0 8 178
17 70 0 15 15 7 7 114
Table 2 demonstrates that in most cases we can achieve high lev-
18 66 59 20 20 0 0 165
els of accuracy. For the two most common activities, walking and
19 69 66 41 15 0 0 191
jogging, we generally achieve accuracies above 90%. Jogging
20 31 62 16 15 4 3 131
appears easier to identify than walking, which seems to make
21 54 62 15 16 12 9 168
sense, since jogging involves more extreme changes in accelera-
22 33 61 25 10 0 0 129 tion. It appears much more difficult to identify the two stair
23 30 5 8 10 7 0 60 climbing activities, but as we shall see shortly, that is because
24 62 0 23 21 8 15 129 those two similar activities are often confused with one another.
25 67 64 21 16 8 7 183 Note that although there are very few examples of sitting and
26 85 52 0 0 14 17 168 standing, we can still identify these activities quite well, because,
27 84 70 24 21 11 13 223 as noted earlier, the two activities cause the device to change
28 32 19 26 22 8 15 122 orientation and this is easily detected from the accelerometer data.
29 65 55 19 18 8 14 179 Our results indicate that none of the three learning algorithms
Sum 1683 1321 545 465 289 223 4526 consistently performs best, but the multilayer perceptron does
% 37.2 29.2 12.0 10.2 6.4 5.0 100 perform best overall. More detailed results are presented in Tables

SIGKDD Explorations Volume 12, Issue 2 Page 78

 3-5, which show the confusion matrices associated with each of Table 6: Confusion Matrix for J48 Model (Stairs Combined)
the three learning algorithms.
Predicted Class Accur.
Walk Jog Stairs Sit Stand (%)
Table 3: Confusion Matrix for J48
Walk 1524 7 148 2 2 90.6
Predicted Class

Actual Class
Walk Jog Up Down Sit Stand Jog 10 1280 31 0 0 96.9
Walk 1513 14 72 82 2 0 Stairs 185 33 784 4 4 77.6
Actual Class

Jog 16 1275 16 12 1 1 Sit 4 0 2 272 4 96.5

Up 88 23 323 107 2 2 Stand 3 1 10 0 209 93.7
Down 99 13 92 258 1 2
Sit 4 0 2 3 270 3
Stand 4 1 2 7 1 208
4. RELATED WORK
Activity recognition has recently gained attention as a research
Table 4: Confusion Matrix for Logistic Regression topic because of the increasing availability of accelerometers in
Predicted Class consumer products, like cell phones, and because of the many
Walk Jog Up Down Sit Stand potential applications. Some of the earliest work in accelerometer-
Walk 1575 14 53 36 2 3 based activity recognition focused on the use of multiple acceler-
ometers placed on several parts of the user’s body. In one of the
Actual Class

Jog 15 1294 6 6 0 0
Up 277 36 150 77 1 4 earliest studies of this topic, Bao & Intille [3] used five biaxial
accelerometers worn on the user’s right hip, dominant wrist, non-
Down 259 6 136 57 3 4
dominant upper arm, dominant ankle, and non-dominant thigh in
Sit 1 0 4 11 260 6
order to collect data from 20 users. Using decision tables, in-
Stand 3 1 7 3 15 194
stance-based learning, C4.5 and Naïve Bayes classifiers, they
created models to recognize twenty daily activities. Their results
Table 5: Confusion Matrix for Multilayer Perceptron indicated that the accelerometer placed on the thigh was most
Predicted Class powerful for distinguishing between activities. This finding sup-
Walk Jog Up Down Sit Stand ports our decision to have our test subjects carry the phone in the
most convenient location—their pants pocket.
Walk 1543 5 73 60 1 1
Actual Class

Jog 3 1299 16 3 0 0 Other researchers have, like Bao & Intille, used multiple acceler-
Up 84 24 335 98 2 2 ometers for activity recognition. Krishnan et. al. [9] collected data
Down 108 10 136 206 2 3 from three users using two accelerometers to recognize five ac-
Sit 0 2 4 1 268 7 tivities—walking, sitting, standing, running, and lying down. This
Stand 1 0 5 4 8 205 paper claimed that data from a thigh accelerometer was insuffi-
cient for classifying activities such as sitting, lying down, walk-
ing, and running, and thus multiple accelerometers were neces-
The most important activities to analyze are the climbing-up and sary (a claim our research contradicts). In another paper, Krishnan
climbing-down stair activities, since these were the only activities et. al. [10] examined seven lower body activities using data col-
that that were difficult to recognize. The confusion matrices indi- lected from ten subjects wearing three accelerometers. This
cate that many of the prediction errors are due to confusion be- method was tested in supervised and semi-naturalistic settings.
tween these two activities. If we focus on the results for the J48 Tapia et. al. [16] collected data from five accelerometers placed
decision tree model in Table 3, we see that when we are climbing on various body locations for twenty-one users and used this data
up stairs the most common incorrect classification occurs when to implement a real-time system to recognize thirty gymnasium
we predict “downstairs,” which occurs 107 times and accounts for activities. A slight increase in performance was made by incorpo-
a decrease in accuracy of 19.6% (107 errors out of 545). When rating data from a heart monitor in addition to the accelerometer
the actual activity is climbing downstairs, walking slightly out- data. Mannini and Sabitini [23] used five tri-axial accelerometers
paces “upstairs” in terms of the total number of errors (99 vs. 92), attached to the hip, wrist, arm, ankle, and thigh in order to recog-
but this is only because walking occurs more than three times as nize twenty activities from thirteen users. Various learning meth-
often as climbing upstairs in our dataset. If we look at Figures 2a, ods were used to recognize three “postures” (lying, sitting, and
2c, and 2d, we see that the patterns in acceleration data between standing) and five “movements” (walking, stair climbing, run-
“walking”, “ascending stairs” and “descending stairs” are some- ning, and cycling). Foerster and Fahrenberg [28] used data from
what similar. To limit the confusion between the ascending and five accelerometers in one set of experiments and from two of
descending stair activities, we ran another set of experiments those accelerometers in another for activity recognition. Thirty-
where we combine ascending stairs and descending stairs into one one male subjects participated in the study and a hierarchical
activity. The resulting confusion matrix for the J48 algorithm is classification model was built in order to distinguish between
shown in Table 6 (in the interest of space we do not show them postures such as sitting and lying at specific angles, and motions
for the other two algorithms). We see that the results are substan- such as walking and climbing stairs at different speeds.
tially improved, although stair climbing is still the hardest activity
to recognize. Researchers have used a combination of accelerometers and other
sensors to achieve activity recognition. Parkka et. al. [27] created

SIGKDD Explorations Volume 12, Issue 2 Page 79

 a system using twenty different types of sensors (including an Several researchers have considered the use of widely-available
accelerometer worn on the chest and one worn on the wrist) in mobile devices such as cell phones to address the activity recogni-
order to recognize activities such as lying, standing, walking, tion problem. However the earlier approaches did not take advan-
running, football, swinging, croquet, playing ball, and using the tage of the sensors incorporated into the mobile devices them-
toilet in specific locations. Lee and Mase [25] created a system to selves. For example, Gyorbiro et. al. [7] used “MotionBands”
recognize a user’s location and activities, including sitting, stand- attached to the dominant wrist, hip, and ankle of each subject to
ing, walking on level ground, walking upstairs, and walking distinguished between six different motion patterns. Each Mo-
downstairs using a sensor module that consisted of a biaxial ac- tionBand contained a tri-axial accelerometer, magnetometer, and
celerometer and an angular velocity sensor worn in the pocket gyroscope. As the MotionBand collected data, the data was then
combined with a digital compass worn at the user’s waist. transmitted to a smart phone carried by the user to be stored. Ravi
Subramayana et. al. [26] addressed similar activities by building a et. al. [15] collected data from two users wearing a single acceler-
model using data from a tri-axial accelerometer, two micro- ometer-based device and then transmitted this data to the HP
phones, phototransistors, temperature and barometric pressure iPAQ mobile device carried by the user. Using this data for activ-
sensors, and GPS to distinguish between a stationary state, walk- ity recognition, researchers compared the performance of eighteen
ing, jogging, driving a vehicle, and climbing up and down stairs. different classifiers. Lester et. al. [11] used accelerometer data,
along with audio and barometric sensor data, to recognize eight
While these systems using multiple accelerometers or a combina- daily activities from a small set of users. While these studies
tion of accelerometers and other sensors were capable of identify- could have used a cell phone to generate the accelerometer data,
ing a wide range of activities, they are not very practical because they did not do this. Instead, the data was generated using distinct
they involve the user wearing multiple sensors distributed across accelerometer-based devices worn by the user and then sent to a
their body. This could work for some short term, small scale, cell phone for storage.
highly specialized applications (e.g., in a hospital setting) but
would certainly not work for the applications that we envision. A few studies, like ours, did use an actual commercial mobile
device to collect data for activity recognition. Such systems offer
Some studies have also focused on combining multiple types of an advantage over other accelerometer-based systems because
sensors in addition to accelerometers for activity recognition. they are unobtrusive and do not require any additional equipment
Maurer et al. [13] used “eWatch” devices placed on the belt, shirt for data collection and accurate recognition. Miluzzo et. al. [14]
pocket, trouser pocket, backpack, and neck to recognize the same explored the use of various sensors (such as a microphone, accel-
six activities that we consider in our study. Each “eWatch” con- erometer, GPS, and camera) available on commercial smart
sisted of a biaxial accelerometer and a light sensor. Decision phones for activity recognition and mobile social networking
trees, k-Nearest Neighbor, Naïve Bayes, and Bayes Net classifiers applications. In order to address the activity recognition task,
with five-fold cross validation were used for learning. Choudhury they collected accelerometer data from ten users to build an activ-
et. al [6] used a multimodal sensor device consisting of seven ity recognition model for walking, running, sitting, and standing
different types of sensors (tri-axial accelerometer, microphone, using J48. This model had particular difficulty distinguishing
visible light phototransitor, barometer, visible+IR light sensor, between the sitting and standing activities, a task that our models
humidity/temperature reader, and compass) to recognize activities easily achieve. Yang [21] developed an activity recognition sys-
such as walking, sitting, standing, ascending stairs, descending tem using the Nokia N95 phone to distinguish between sitting,
stairs, elevator moving up and down, and brushing one’s teeth. standing, walking, running, driving, and bicycling. This work also
Cho et. al. [5] used a single tri-axial accelerometer, along with an explored the use of an activity recognition model to construct
embedded image sensor worn at the user’s waist, to identify nine physical activity diaries for the users. Although the study
activities. Although these multi-sensor approaches do indicate the achieved relatively high accuracies of prediction, stair climbing
great potential of mobile sensor data as more types of sensors are was not considered and the system was trained and tested using
being incorporated into devices, our approach shows that only one data from only four users. Brezmes et. al. [4] also used the Nokia
type of sensor—an accelerometer—is needed to recognize most N95 phone to develop a real-time system for recognizing six user
daily activities. Thus our method offers a straightforward and activities. In their system, an activity recognition model is trained
easily-implementable approach to accomplish this task. for each user, meaning that there is no universal model that can be
Other studies, like our own, have focused on the use of a single applied to new users, for whom no training data exists. Our mod-
accelerometer for activity recognition. Long, Yin, and Aarts [22] els do not have this limitation.
collected accelerometer data from twenty-four users using a tri-
axial accelerometer worn without regard for orientation at the 5. CONCLUSIONS AND FUTURE WORK
user’s waist. Data was collected naturalistically, and decision In this paper we described how a smart phone can be used to per-
trees as well as a Bayes classifier combined with a Parzen win- form activity recognition, simply by keeping it in ones pocket.
dow estimator were used to recognize walking, jogging, running, We further showed that activity recognition can be highly accu-
cycling, and sports. Lee et. al. [24] used a single accelerometer rate, with most activities being recognized correctly over 90% of
attached to the left waists of five users. Standing, sitting, walking, the time. In addition, these activities can be recognized quickly,
lying, and running were all recognized with high accuracies using since each example is generated from only 10 seconds worth of
fuzzy c-means classification. However unlike these studies, which data. We have several interesting applications in mind for activity
use devices specifically made for research purposes, our method recognition and plan to implement some of these applications in
utilizes commercial devices that are widely-available without any the near future.
additional specialized equipment. This approach enables make
Our work would not have been possible without establishing our
practical real-world applications for our models.
WISDM Android-based data collection platform, and we view

SIGKDD Explorations Volume 12, Issue 2 Page 80

 this software and hardware architecture, where data is transmitted [5] Cho, Y., Nam, Y., Choi, Y-J., and Cho, W-D. 2008. Smart-
by the phone to our Internet-based server, as a key resource pro- Buckle: human activity recognition using a 3-axis acceler-
duced as a consequence of this work. By having this in place we ometer and a wearable camera. In HealthNet.
will be able to mine other mobile sensor data much more quickly. [6] Choudhury, T., Consolvo, S., Harrison, B., LaMarca, A.,
This platform, as well as the data that we collected, will ulti- LeGrand, L., Rahimi, A., Rea, A., Borriello, G., Hemingway,
mately be made public. B., Klasnja, P., Koscher, K., Landay, J., Lester, J., Wyatt, D.,
We plan to improve our activity recognition in several ways. The and Haehnel, D. 2008. The mobile sensing platform: An
straightforward improvements involve: 1) learning to recognize embedded activity recognition system. In IEEE Pervasive
additional activities, such as bicycling and car-riding, 2) obtaining Computing, 7(2), 32-41.
training data from more users with the expectation that this will [7] Gyorbiro, N., Fabian, A., and Homanyi, G. 2008. An activity
improve our results, 3) generating additional and more sophisti- recognition system for mobile phones. In Mobile Networks
cated features when aggregating the raw time-series data, and 4) and Applications, 14(1), 82-91.
evaluating the impact of carrying the cell phone in different loca-
[8] Inooka, H., Ohtaki, Y. Hayasaka, H. Suzuki, A., and Na-
tions, such as on a belt loop. In addition, in the near future we
gatomi, R. 2006. Development of advanced portable device
plan to significantly enhance our WISDM platform so that we can
for daily physical assessment. In SICE-ICASE, International
generate results in real-time, whereas currently our results are
Joint Conference, 5878-5881.
generated off-line and are not reported back to the mobile phone
and the user. We plan to provide real-time results in two ways. [9] Krishnan, N., Colbry, D., Juillard, C., and Panchanathan, S.
The first way minimizes the intelligence required on the phone by 2008. Real time human activity recognition using tri-Axial
having the phone transmit the data to the Internet-based sever accelerometers. In Sensors, Signals and Information Proc-
over the cellular connection, as usual, with the server applying the essing Workshop.
activity recognition model and transmitting the results back to the [10] Krishnan, N. and Panchanathan, S. 2008. Analysis of Low
phone. In one variant, the phone will send the raw accelerometer Resolution Accelerometer Data for Continuous Human Ac-
data and in a second variant the phone will perform the data trans- tivity Recognition. In IEEE International Conference on
formation step and only transmit the data when an example is Acoustics, Speech and Signal Processing, (ICASSP 2008).
generated. The second method involves implementing the activity Pages 3337-3340.
recognition model directly on the cell phone. Given the computa-
tional power of these devices, this is certainly a feasible option. [11] Lester, J., Choudhury, T. and Borriello, G. 2006. A practical
One key advantage of this method is that it removes the need for a approach to recognizing physical activities. Lecture Notes in
server, which makes the solution perfectly scalable, and ensures Computer Science: Pervasive Computing, 1–16.
the user’s privacy, since the sensor data is kept locally on the [12] Mathie, M., Celler B., Lovell N., and Coster A. 2004. Classi-
device. fication of basic daily movements using a triaxial acceler-
ometer. In Medical & Biological Engineering and Comput-
The work described in this paper is part of a larger effort to mine
ing, 42.
sensor data from wireless devices. We plan to continue our
WISDM project, applying the accelerometer data to other tasks [13] Maurer, U., Smailagic, A., Siewiorek, D., & Deisher, M.
besides activity recognition and collecting and mining other sen- 2006. Activity recognition and monitoring using multiple
sor data, especially GPS data. We believe that mobile sensor data sensors on different body positions. In IEEE Proceedings on
provides tremendous opportunities for data mining and we intend the International Workshop on Wearable and Implantable
to leverage our Android-based data collection/data mining plat- Sensor Networks, 3(5).
form to the fullest extent possible. [14] Miluzzo, E., Lane, N., Fodor, K., Peterson, R., Lu, H., Mu-
solesi, M., Eisenman, S., Zheng, X. and Campbell, A. 2008.
6. REFERENCES Sensing meets mobile social networks: The design, imple-
[1] Anderson, I., Maitland, J., Sherwood, S., Barkhuus, L., mentation and evaluation of the CenceMe application. In The
Chalmers, M., Hall, M., Brown, B., and Muller, H. 2007. 6th ACM Conference on Embedded Networked Sensor Sys-
Shakra: Tracking and sharing daily activity levels with un- tems, 337-350.
augmented mobile phones. In Mobile Networks and Applica- [15] Ravi, N., Dandekar, N. 2005. Activity recognition from ac-
tions. 12(2-3). celerometer data. In Proceedings of the Seventeenth Confer-
ence on Innovative Applications of Artificial Intelligence.
[2] Apple iPhone and Apple iPod Touch. 2009. Apple Inc.
www.apple.com. [16] Tapia, E.M., Intille, S.S. et al. 2007. Real-Time recognition
of physical activities and their intensities using wireless ac-
[3] Bao, L. and Intille, S. 2004. Activity Recognition from User-
celerometers and a heart rate monitor. In Proceedings of the
Annotated Acceleration Data. Lecture Notes Computer Sci-
2007 11th IEEE International Symposium on Wearable
ence 3001, 1-17.
Computers, 1-4.
[4] Brezmes, T., Gorricho, J.L., and Cotrina, J. 2009. Activity [17] Unwired View.com. 2009. Google wants to make your
Recognition from accelerometer data on mobile phones. Android phone much smarter with accelerometer and other
In IWANN '09: Proceedings of the 10th International Work- sensors. Stasys Bielinis.http://www.unwiredview.com/2009/
Conference on Artificial Neural Networks, 796-799. 05/21/google-wants-to-make-your-android-phone-much-
smarter-with-accelerometer-and-other-sensors/

SIGKDD Explorations Volume 12, Issue 2 Page 81

 [18] Weiss, G. M., and Hirsh, H. 1998. Learning to predict rare [24] Lee, M., Kim, J., Kim, K., Lee, I., Jee, S.H., and Yoo, S.K.
events in event sequences, In Proceedings of the Fourth In- 2009. Physical activity recognition using a single tri-axis ac-
ternational Conference on Knowledge Discovery and Data celerometer. In Proceedings of the World Congress on Engi-
Mining, AAAI Press, Menlo Park, CA, 359-363. neering and Computer Science 2009, 1.
[19] WISDM (Wireless Sensor Data Mining) Project. Fordham [25] Lee, S.-W. and Mase, K. 2002. Activity and location recog-
University, Department of Computer and Information Sci- nition using wearable sensors. In IEEE Pervasive Comput-
ence, http://storm.cis.fordham.edu/~gweiss/ wisdm/ ing, 1(3):24–32.
[20] Witten, I. H. and Frank, E. Data Mining: Practical Machine [26] Subramanya, A., Raj, A., Bilmes, J., and Fox, D. 2006. Rec-
Learning Tools and Techniques, 2nd ed. Morgan Kaufmann, ognizing activities and spatial context using wearable sen-
June 2005. sors. In Proceedings of the 22nd Conference on Uncertainty
[21] Yang, J. 2009. Toward physical activity diary: Motion rec- in Artificial Intelligence.
ognition using simple acceleration features with mobile [27] Parkka, J., Ermes, M., Korpipaa, P., Mantyjarvi, J., Peltola,
phones, In First International Workshop on Interactive Mul- J., and Korhonen, I. 2006. Activity classification using real-
timedia for Consumer Electronics at ACM Multimedia. istic data from wearable sensors. In IEEE Transactions on
[22] Long, X., Yin, B., and Aarts, R.M. 2009. Single accelerome- Information Technology in Biomedicine, 10(1), 119-128.
ter-based daily physical activity classification. In 31st Annual [28] Foerster F. and Fahrenberg J. 2000. Motion pattern and pos-
International Conference of the IEEE EMBS, 6107-6110. ture: correctly assessed by calibrated accelerometers. In Be-
[23] Mannini, A. and Sabatini A.M. 2010. Machine learning havior Research Methods, Instruments, & Computers, 32(3),
methods for classifying human physical activity from on- 450–7.
body accelerometers. In Sensors 2010, 10, 1154-1175.

SIGKDD Explorations Volume 12, Issue 2 Page 82