Neural Computing and Applications (2022) 34:12749–12763

https://doi.org/10.1007/s00521-022-07144-1(0123456789().,-volV)(0123456789().
,- volV)

ORIGINAL ARTICLE

ApplianceNet: a neural network based framework to recognize daily

life activities and behavior in smart home using smart plugs
Muhammad Fahim1 • S. M. Ahsan Kazmi2 • Asad Masood Khattak3

Received: 9 July 2021 / Accepted: 24 February 2022 / Published online: 21 March 2022
Ó Crown 2022

Abstract
A smart plug can transform the typical electrical appliance into a smart multi-functional device, which can communicate
over the Internet. It has the ability to report the energy consumption pattern of the attached appliance which offer the
further analysis. Inside the home, smart plugs can be utilized to recognize daily life activities and behavior. These are the
key elements to provide human-centered applications including healthcare services, power consumption footprints, and
household appliance identification. In this research, we propose a novel framework ApplianceNet that is based on energy
consumption patterns of home appliances attached to smart plugs. Our framework can process the collected univariate
time-series data intelligently and classifies them using a multi-layer, feed-forward neural network. The performance of this
approach is evaluated on publicly available real homes collected dataset. The experimental results have shown the
ApplianceNet as an effective and practical solution for recognizing daily life activities and behavior. We measure the
performance in terms of precision, recall, and F1-score, and the obtained score is 87%, 88%, 88%, respectively, which is
11% higher than the existing method in terms of F1-score. Furthermore, our scheme is simple and easy to adopt in the
existing home infrastructure.

Keywords Daily activities  Home appliances  Healthcare applications  Intelligent data processing  Time-series analysis

1 Introduction grids and do spatial load forecasting. From a consumer

perspective, electricity usage information has useful
Electricity is an integral part of our communities and its applications [1, 2]. These include (i) to reduce the utility
demand is increasing globally with the electrification of the bills and contribute at an individual level to reduce global
automobile, growing digital connected devices, and Heat- warming [3] (ii) monitoring of daily living activities and
ing, Ventilation, and Air Conditioning (HVAC) of the behavior [4] (iii) non-intrusive monitoring of the aging
building. There is a clear need for a detailed understanding population to provide remote healthcare services [5].
of the usage of electricity to efficiently manage the smart The activities of daily living and behavior can be rec-
ognized intrusively using video camera [6] and embedded
sensors over the home appliances [7, 8]. However, several
& Muhammad Fahim
[email protected] problems are associated with these approaches; for
instance, video cameras are not practical due to privacy
S. M. Ahsan Kazmi
[email protected] reasons. Similarly, home appliances are not designed to
share their interaction patterns and usage. To make them a
Asad Masood Khattak
[email protected] smart appliance, sensors are placed over the appliances,
which put an extra layer of tiny sensors that may compli-
1
School of Electronics, Electrical Engineering and Computer cate interaction with them. An alternative solution is Non-
Science, Queens University Belfast, Belfast, UK Intrusive Appliance Load Monitoring (NIALM) that is
2
Faculty of Computer Science and Creative Technologies, based on a smart meter to disaggregate the load to recog-
University of West of England, Bristol, UK nize the individual appliances [9]. However, energy dis-
3
College of Technological Innovation, Zayed University, aggregation is an extremely challenging problem [5] when
Dubai, UAE

123
12750 Neural Computing and Applications (2022) 34:12749–12763

various appliances are working simultaneously. Similarly, appliances by a feed-forward neural network. Our contri-
such a solution has adoption limitations because instalment butions in this research are as follows.
of the smart meter is related to service providers not the
– The proposed model is based on a data-driven approach
residents of the house.
that does not require domain expert knowledge.
In this paper, we consider a unique perspective to rec-
– The proposed solution is integrated easily with existing
ognize the daily living activities by considering a smart
home infrastructure without disturbing residents’ pri-
plug as an Individual Appliance Monitor (IAM) to collect
vacy and additional sensors that may affect home
the energy consumption data of home appliances [10]. It is
aesthetics or complicate the resident interaction.
considered as Intrusive Load Monitoring (ILM) because
– The home appliances usage differ from one another in
smart plug is attached to each device [11]. The ILM is the
terms of duration. For instance, the duration of
preferred approach for the situations when the consumer is
watching television is longer than the toaster or
interested to monitor daily life activities without adding an
dishwasher. In such scenarios, small usage of home
extra layer of sensors inside the home or involvement of
appliances produces few data samples. Consequently, it
smart meter. The trivial solution for recognizing the elec-
degrades the performance of the machine learning
trical appliance is to statically map the smart plugs for
model. This problem is known as an imbalance dataset.
specific appliances. The potential problem with this
We solve this issue by data augmentation and validate
approach can reduce the applicability of provided services.
performance empirically.
For instance, a smart plug is specified for the oven and if
– The usage pattern of home appliances can provide
households use it for TV or any other device then the
information on behavioral changes, which can be
service model will lead to false-positive results. The oven
captured by our framework and may assist the family
smart plug used for TV can point to some emergency at
to understand the occupant’s behavior.
home in a scenario of 3 hours usage. It can consider stove
is ON for 3 hours while in home setting smart plug is The rest of the research paper is organized as follows. We
attached with TV. The goal of our research is to make the briefly describe related research work and their limitations
smart plug intelligent which can tell which device is con- in Sect. 2. In Sect. 3, we introduce ApplianceNet and its
nected to it. Consequently, it will allow the consumer to implementation for recognizing home appliances. In
use the smart plug freely with any appliance. Furthermore, Sect. 4, experimental setting and hyper-parameter details.
it provides the freedom to residents by specifying which In Sect. 5, we present the results and comparison with the
devices are allowed to be monitored and accessed by smart state-of-the-art techniques. Finally, we conclude our paper
hand-held devices to control its states. Intelligent data in Sect. 6.
processing of smart plugs’ may helps to avoid unseen
problems in daily activities as well as emergency situations
at home. For example, excessive watching of the television 2 Related work
leads to sedentary activity, which may not suitable for the
resident health conditions. Similarly, keeping the micro- Smart homes are designed to monitor vital signs, daily life
wave ‘ON’ after finishing the cooking activity may lead to activities, and possibly avoid emergencies [12, 13]. In this
an emergency. section, we briefly explain different kinds of sensor tech-
In our research, IAM-based solution is considered as an nologies and developed models used for activity recogni-
effective and seamless way to monitor the activities of tion in the home environment. The advancement of neural
daily living and behavior. With the help of smart plugs, we network-based approaches has made significant improve-
monitor basic household appliances, such as toaster, ments in recognition tasks [14]. Mehr et al. [15] proposed a
microwave, fridge, washing machine, etc. To distinguish Convolutional Neural Network (CNN) to recognize human
the type of meal (i.e., breakfast, lunch, or dinner), the time action using multiple cameras inside the home environ-
information is available in collected data streams. The data ment. The experiments were performed on a publicly
stream of smart plugs contains two columns; one is a available dataset, collected using three static cameras to
timestamp and the second is electric power consumption in recognize the ambulatory activities including walking,
watts. The task of recognizing home appliances is very table cleaning, drinking, etc. They achieved 82% of accu-
challenging because most energy consumption patterns of racy for twelve actions available in the dataset. Chen et al.
appliances overlap with other appliances. It makes an [16] also proposed a novel CNN architecture to recognize
unnoticeable difference in readings. The proposed Appli- the fridge, dishwasher, microwave, washing machine, and
anceNet framework (see block diagram in Fig. 1) processes kettle. Their model confirms the applicability of neural
time-series data by extracting features to a repre- network architecture for non intrusive load monitoring.
sentable form and recognizes non-linearly separable home Similarly, Junfeng et al. [17] proposed temporal and

123
Neural Computing and Applications (2022) 34:12749–12763 12751

Fig. 1 Block diagram of the proposed ApplianceNet framework

spectral features learning with two-stream CNN. They health care systems. They performed experiments on a
fused both temporal and spectral feature to recognize the large-scale publicly available dataset collected for five
appliances. They obtained more than 90% accurate results houses in southern England. They learned the activities
based on the fusion of temporal and spectral features. patterns from appliance usage inside the home by mapping
Another possible way to recognize daily life activities is back to energy consumption data from the smart meter.
embedded sensors for household objects. Karvigh et al. [7] This technology adoption in a setting where legacy meters
proposed a framework for real-time activity recognition for are used is a challenge to keep the track of energy usage.
energy efficiency in building. Their approach is capable to Similarly, Delvin et al. [5] classify the activities of daily
detect action, activity recognition, and waste estimation. living using smart meter data. Their approach is based on
Particularly, for activity recognition tap sensors we used neural networks to identify home appliances. They con-
over the objects to sense the ON/OFF states. Their activity sider seven common home appliances and achieved
recognition algorithm is based on ontology and data-driven acceptable (i.e., 77% F1-score) results.
solutions. A number of sophisticated algorithms were France et al. [21] developed an intrusive load monitor-
developed by the research community to monitor home ing approach over the layer architecture and Internet of
activities [8, 18]. These algorithms are based on data-dri- things (IoT) for load monitoring and activity recognition in
ven to knowledge-driven approaches. Most of the devel- smart homes. They consider 5 activities (i.e., dishwasher,
oped models were mature enough to achieve more than hairdryer, iron, oven, and washing machine). The approach
90% accuracy and successfully deployed in different is based on hand-crafted features and neural networks as a
application scenarios [19]. classifier. They reported high accuracy (i.e., 91% F1-score)
Smart plugs were utilized to monitor the energy usage in for considered activities. Similarly, Paganelli et al. [22]
home environments and the developed techniques found in also propose a neural network-based approach for appli-
[20]. Gajowniczek et al. [3] considered the appliances’ ance recognition from smart plugs. The designed approach
status either ON/OFF to detect usage patterns using hier- is developed for home energy management systems in
archical and c-means clustering. The goal of their study Italy. Their approach is also based on hand-crafted fea-
was to contribute in energy consumption awareness and tures. In future work, they also emphasized the need for
energy-saving recommendations. Yan et al. [20] used smart complex event processing techniques.
meter data to extract the characteristics of occupant The above-discussed approaches did not consider the
behavior and energy consumption of home appliances. appliances level usage patterns as well as automatic feature
Seven home appliances, such as refrigerator, electric coo- extraction for daily life activities which can be monitored
ker, air conditioner, television, laptop computer, washing through smart plugs. Our framework is based on appliance
machine, and water dispenser were analyzed in Beijing, level monitoring and automatic feature extraction
China. They developed a Bayes classification based scheme to recognize the daily life activities and behavior
approach to recognize the appliances. inside the home environment depending on the intended
Yassine et al. [9] introduced a mining algorithm to appliance’s usage.
recognize the daily life activities using smart meter data for

123
12752 Neural Computing and Applications (2022) 34:12749–12763

3 Proposed model Fig. 3 Current cost smart plug

[23]

In our proposed model, we consider a set of smart plugs

denoted by P attached with each home appliance, where
each smart plug p 2 P sends univariate time-series mea-
surement vp at each time instant t. Note that this mea-
surement is continuously repeated after an interval of
length Mt. Then, the time series sample for a smart plug p is
represented as a sequence of length N as follows:
V p ¼ ðvp ðt1 Þ; vp ðt2 Þ; . . .; vp ðtN ÞÞT ; 8p 2 P; ð1Þ

where tiþ1 ¼ ti þ Mt and V p ¼ ðÞT represents the transpose

of pth smart plug sample. In this work, it is assumed that
each smart plug p is identically independent with other, the server database for storage and further processing. This
then, we can analyze for a single smart plug p and apply the server can reside inside the home or a cloud service plat-
analysis for the whole system. Thus, we simplify by stating form. The loud computing could provide a cost-effective
vp ¼ v and sequence length t1 ¼ t, where t 2 f1; 2; . . .; Ng. solution as well as zero-level management at the user end.

V ¼ fvt gNt¼1 ð2Þ 3.2 Communication layer

The design of our proposed framework is based on the
The communication layer in smart home connects the
standard layered architecture used in a smart home. The
physical layer object to low-powered wireless networks or
inspired architecture of ApplianceNet alongside the stan-
ZigBee. The ZigBee provides connectivity using radio
dard architecture of a smart home is illustrated in Fig. 2.
signals for seamless communication in home settings as
The architecture consists of four layers and these are
well as it does not interfere with Wi-Fi, Bluetooth, or
described in the following subsections.
phone networks. Smart plugs are connected over the Wi-Fi
access point, while few smart plugs have ZigBee to com-
3.1 Physical layer
municate with the gateway. It establishes communication
to send the energy consumption data stream continuously
Home appliances are plugged into smart plugs for moni-
to the server for storage and further processing. The secure
toring daily energy usage. Smart plugs fits between the
communication is essential for home appliances [24]. The
socket and the home appliance. It seamlessly collects time-
security and network protocol is not the part of this
series data about their energy consumption and track how
research. We assume that it is available from smart plug
much attached appliance consumes electricity. For
providers for secure communication over the Internet.
instance, a microwave oven or any other home appliance
can be monitored for energy usage connected to the smart
3.3 Data processing layer
plug. Figure 3 presents a smart plug ‘‘current cost’’ as an
Individual Appliance Monitor (IAM) to accurately measure
This layer is a central part of our proposed framework. It is
power consumption in watts.
responsible to process the collected data stream from each
The smart plug has the characteristics to connect over
appliance. It consists of three major components, (1) Data
the home Internet and send energy consumption reading to
preprocessing, (2) Feature extraction and selection, and
(3) Classifier. The detailed description and working of
these sub-components are as follows.

3.3.1 Data preprocessing

The energy consumption data is sent to the server from

smart plugs periodically. In the case of the current cost
smart plug, it sends data after every 6  8 seconds. Fig-
ure 4 presents one-week data stream for ‘Microwave’,
‘Toaster’, and ‘Kettle’.

Fig. 2 The layered architecture of ApplianceNet

123
Neural Computing and Applications (2022) 34:12749–12763 12753

methods with the support of domain knowledge or auto-

matic feature extraction over the given time-series. The
automatic feature extraction methods relax the domain
expert constrain to extract the features. In the case of
automatically extracted features, the number of obtained
features is huge as compared to handcrafted features, which
is further refined by the feature selection mechanism. The
required vt can be extracted by applying time series char-
acterization methods fj : vt ! vt;j .
Then, we can obtained the required feature vector vt :
vt ¼ ðf1 ðvt ÞÞ; f2 ðvt ÞÞ; . . .; fM ðvt ÞÞ; 8t 2 N ð4Þ

We used TSFRESH (Time Series FeatuRe Extraction on

basis of Scalable Hypothesis tests) library [25] to extract a
Fig. 4 One week data stream of three home appliances
large number of time series characteristics (i.e., 63 char-
acteristics resulting in M ¼ 794 features) as features. The
In Fig. 4, it is clear that home appliances’ working
extracted temporal features describe the basic and complex
patterns vary from one appliance to another appliance. The
characteristics of given time series, which are listed in
x-axis shows the days of the week while the y-axis presents
Table 1.
the energy consumption. For example, in the case of a
Table 1 represent the subset of extracted features which
toaster, its usage is during 4th and 7th day of the week. The are used in feature selection to reduce the noise, redun-
appliance data becomes more complicated when the dancies or irrelevant information. The feature selection is
number of individual appliance monitor increase in a home also helpful in reducing the computational cost of the
environment. In order to recognize the device at a unit proposed scheme. The feature selection is based on auto-
time, the collected data requires further processing in the matically configuration of feature significance hypothesis
form of feature extraction. tests [26]. A feature vector V/ is defined as meaningful
features for the classification of home appliances. A
3.3.2 Feature extraction and selection
hypothesis test is applied to extract the relevant features:
The time-series measurement vt of smart plug p contains vt  v / ð5Þ
just one attribute of energy consumption. The temporal
In a hypothesis test, each feature vector vt is evaluated
features are extracted over the window of t minutes, peri-
individually and independently with respect to its signifi-
odically. We mapped time series data vt to a well-defined
cance for the classification of home appliances. The result
feature space with a dimension of M and the feature vector
of these tests is a vector of p-values, computing the
is defined as:
implications of each feature for predicting the home
vt ¼ ðvt;1 ; vt;2 ; . . .; vt;M Þ ð3Þ appliance. In order to decide which features to keep, this
vector is evaluated on basis of the Benjamini-Yeku-
The feature extraction is a process to transform the data
tieli(BY) procedure [27]. The BY is a statistical procedure
into a meaningful representation, where the machine
that facilitates to avoid false-positive rate. The procedure
learning model can distinguish between different home
pseudo code is presented in the following algorithm 1
appliances. These features can be obtained by handcrafted

123
12754 Neural Computing and Applications (2022) 34:12749–12763

Table 1 A subset of extracted

Characteristics Measures
primary features
Energy Absolute energy, energy ratio, FFT coefficients
Statistical Auto-correlation, entropy, mean absolute change
Linear trend intercept Standard deviation, max, min, mean and variance
Wavelet Continuous wavelet coefficients and peaks
Spectral Spectral density estimation, signal symmetry
Miscellaneous Number of crossings, number of peaks

We extract the selected feature vector v/ for each smart to every neuron in first layer li and every subsequent layer
plug p and supplied it to the classifier for recognizing home is fully connected to liþ1 with i 2 ½1; L. A weighted sum is
appliances. calculated over each neuron and non-linearity is applied to
squash the strengths.
3.3.3 Classifier " #
Xm
y ¼ / fð wk f k þ bÞ ð6Þ
A non-linear classifier is designed to recognize home k¼0
appliances. The classification network is based on a multi-
where / is the activation function, b is the bias and m is the
layered feed-forward neural network using a back-propa-
number of neurons in each li and considered as hyper-
gation learning mechanism. The designed and implemented
parameters. The model is trained to minimize the cross-
architecture is presented in Fig. 5.
entropy loss (i.e., presented in ‘‘(7)’’) over the extracted
In Fig. 5, architecture contains input and output layer
feature vectors to recognize home appliances.
with five hidden layers. The feature vector v/ is connected

Fig. 5 The architectural

designed of feed-forward neural
network

123
Neural Computing and Applications (2022) 34:12749–12763 12755

X
c Table 3 The number of com-
No. Appliances
LðHÞ ¼  yi logðy^i Þ ð7Þ mon home appliances across the
i¼1 five houses 1. Microwave
where c is the class labels, yi is the original label, and y^ is 2. Kettle
the predicted label for home appliances. The details on the 3. Toaster
training procedure and optimal parameters are discussed in 4. Fridge
Sect. 4. 5. Dishwasher
6. Washing machine
3.4 Interface layer 7. Television
8. Desktop PC
At this layer of smart home, the interface connects the
residents, family members, and caregivers in case of
healthcare consumer service. Based on the service, it can period of 2 years. The dataset consists of whole house
facilitate the user to reduce the utility bills, identify power- aggregate loads logged by smart meter as well as nine
hungry devices, power consumption patterns, and aware- appliances inside the house which are connected with the
ness to reduce the CO2 as a social responsibility for con- ‘‘current cost’’ smart plug. We processed only one-month
tributing towards the reduction of global warming. In the data from five houses to get a sufficient footprint of
case of caregivers, it should integrate with the individual attached home appliances with these smart plugs. The one-
physician to check the performed activities, frequency, or month appliance data signature is sufficient to train the
skipping of daily activities. In the case of residents to send ApplianceNet and learn the pattern to recognize the
reminders for the completion of tasks in case of dementia. appliance. The detail about the houses description is pre-
In case of cardiovascular complications, alerts can be sented in Table 2.
generated to perform some tasks and discontinue sedentary In Table 2, the house numbers are kept the same as
activities like excessive watching of television. Further- appeared in the dataset for compatibility reasons. The
more, information about the duration of the activities can house 2 construction year is missing in the dataset while
be determined and shared with personal doctors via secure the number of occupants, appliances, house type, and size
communication. information is provided. The selection of the houses
depends on the number of maximum common electrical
appliances that were attached with smart plugs to report the
4 Experimental setup and details energy values. The considered common home appliances
are listed in Table 3. In Table 3, all the home appliances
This section explains the dataset, experimental setup, and are essential for daily life activities including meal prepa-
hyper-parameter setting with neural network convergence. ration (breakfast, lunch, dinner), sedentary behavior while
watching television, and doing laundry which is considered
4.1 Dataset as an ambulatory activity. The relationship between the
recognized home appliances and daily life activities is
In our experiments, we used a publicly available REFIT presented in Fig. 6.
dataset [28]. It contains information on the individual
appliance usage by utilizing the smart plug ‘‘current cost’’.
By default, all of the installed load monitoring devices had 4.2 Data augmentation
the voltage pre-set to 240 V, suitable for the UK where the
main voltage is rated at 230 V ?10 to -6% [28]. The The classifier requires a sufficient amount of data to train
dataset was collected continuously from 20 houses over a the feed-forward neural network. In the case of appliance

Table 2 Detailed description of

House Occupancy Construction year Appliances Type Size
the houses including occupancy,
construction year, appliances 2 4 – 15 Semi-detached 3 bed
owned, type and size in terms of
bedrooms 3 2 1988 27 Detached 3 bed
4 2 1850–1899 33 Detached 4 bed
5 4 1878 44 Mid-terrace 4 bed
6 2 2005 49 Detached 4 bed

123
12756 Neural Computing and Applications (2022) 34:12749–12763

Fig. 6 The relationship between

the home appliances and daily
life activities

Fig. 7 The number of training samples before and after data augmentation

recognition, some appliance generates an insufficient In Fig. 7, it is visible that appliance ‘Toaster’, ‘Kettle’
amount of data due to less resident interaction or daily and ‘Dishwasher’, and ‘Washing Machine’ are the minor
usage. In this case, a data augmentation strategy is applied activities which are dominated by ‘Desktop PC’, ‘Micro-
to up-sample the minor classes using scikit learn library wave’ and ‘Television’ activity during the learning phase
[29]. The number of synthetic examples is added to the of the model. By augmentation, we up-sampled the minor
minor classes with the objective to increase the recognition classes with synthetics samples to make the class balance
of electrical appliances during the training phase of the as shown in Fig. 7 (right). It contributes to improve the
model. It is based on bootstrapping procedure to re-sample performance of the model.
the minor classes. The bootstrap is a statistical method to
perform data augmentation. It resamples the data instances 4.3 Training parameters
to create synthetic instances with similar properties to the
original instances. This property is quantified by mean and Our proposed model has nine nobs (i.e., hyperparameters)
histogram. Furthermore, it calculates the standard error, to tune for obtaining the best performance. The number of
confidence interval, and performs hypothesis testing to hidden layers plays an important role to understand the
include the generated sample into the minor classes. Con- feature spaces of the provided data, while a number of
sequently, it increases the performance of the proposed neurons in each layer accommodate the information in
model. Some of the appliance usages are for short time like terms of weights. One of the best practices is to reduce the
‘Toaster’ while ‘Desktop PC’ and ‘Television’ usage is number of neurons in each layer when moving from input
high as compared to other appliances. Figure 7 (left) to output. The loss function for ApplianceNet is cross-en-
presents the number of instances for each appliance usage tropy which aligns well in the case of a classification
and Fig. 7 (right) presents the data samples after problem. The role of the loss function is to guide the net-
augmentation. work to optimize the weights. We also included the batch
processing which stochastic gradient descent (SGD)

123
Neural Computing and Applications (2022) 34:12749–12763 12757

Fig. 8 The network

convergence over the training
and validation set (with data
augmentation)

Fig. 9 The network

convergence over the training
and validation set (without data
augmentation)

optimizer to update the weights of connection between the hyperparameters setting and experimental arrangements
neurons after each batch. Consequently, it helps reduce the can be analyzed by plotting the network convergence
memory utilization because few examples were used in the graph. This graph explains how the learning of the network
batch to update the weights. All the training is going on with respect to reduce the loss. The curve

123
12758 Neural Computing and Applications (2022) 34:12749–12763

Table 4 The optimal hyper-parameter to train the classifier Table 6 The obtained results of ApplianceNet without data
augmentation
Parameters Values/settings
Appliance Precision (p) Recall (r) F1-score Accuracy
Hidden layers 5
Each layer neurons [512, 256, 128, 64, 32] Television 0.80 0.86 0.83 0.85
Loss function Cross-entropy Microwave 0.93 0.96 0.94 0.96
Batch size 100 Kettle 0.79 0.73 0.76 0.73
Batch normalization All 5 layers Washing machine 0.86 0.76 0.81 0.76
Optimizer Stochastic Gradient Descent (SGD) Toaster 0.00 0.00 0.00 0.00
Learning rate 0.001 Dish washer 0.91 0.75 0.83 0.75
Epochs 250 Desktop PC 0.91 0.92 0.91 0.92
Fridge 0.88 0.83 0.86 0.83

smoothness also explains the right learning rate and batch

setting as proceeded by the number of epochs. The fol- TP
PrecisionðpÞ ¼ ð8Þ
lowing convergence graphs with data augmentation and ðTP þ FPÞ
without data augmentation is presented in Figs. 8 and 9. TP
These graphs explain that the choice of our hyperparame- RecallðrÞ ¼ ð9Þ
ðTP þ FNÞ
ters is correct.
In Fig. 8, after two hundred epochs, ApplianceNet 2pr
F1  Score ¼ ð10Þ
converges and loss reached to an optimal position. The ðp þ rÞ
network is saved to recognize the unknown data stream where TP is true positive, FP is false positive and FN is
during the test time. Fig. 9 presents the convergence of the false negative.
model around one hundred and sixty epochs without data
augmentation. After this point, we stop the training to cease
the over-fitting. The model parameters are fined tuned until 5 Results and discussion
we obtain the above convergence graph. The optimal
parameters are presented in Table 4. These parameters are The experiments are performed in open-source python
optimal parameter which is selected using a trial and error environment using SciKit learn [29] and PyTorch [30]
method. libraries. Each appliance precision, recall, F1-score and
Accuracy are presented in Table 5.
4.4 Performance measures In Table 5, the appliance ‘Microwave’, ‘Toaster’, and
‘Desktop PC’ usage is recognized more than 90%. The less
We have evaluated the proposed model performance using precise appliance recognition is ‘Kettle’ as compared to the
standard metrics, such as precision, recall, and F1-score other appliances. However, it is recognized 81% correctly
and accuracy. They are calculated using values of the as shown in Table 5. The signature of appliances was
confusion matrix and computed as: complex and even difficult to learn the minor classes (i.e.,
kettle, toaster, dish washer, and washing machine). The
Table 5 The obtained results of ApplianceNet data augmentation helps to learn the signatures and the
reported model presents the better result as compared to the
Appliance Precision (p) Recall (r) F1-score Accuracy
baseline model (i.e., without data augmentation). Similarly,
Television 0.84 0.85 0.85 0.85 Table 6 presents the results of our model without data
Microwave 0.94 0.95 0.94 0.95 augmentation. It can be seen that the accuracy of recog-
Kettle 0.75 0.81 0.78 0.81 nized appliances decreased in the case of minor activities.
Washing machine 0.82 0.86 0.84 0.86 For instance, in the case of ‘Toaster’, model did not learn
Toaster 0.93 0.95 0.94 0.95 any pattern and no instance is recognized during the model
Dish washer 0.91 0.83 0.87 0.83 evaluation. While in the case of data augmentation
Desktop PC 0.92 0.93 0.93 0.93 ‘Toaster’ is recognized 95% correctly.
Fridge 0.89 0.83 0.86 0.82

123
Neural Computing and Applications (2022) 34:12749–12763 12759

Fig. 10 The obtained results

using data augmentation and
without data augmentation

Fig. 11 One week behavior

pattern of watching television
usage

Fig. 12 One week behavior

pattern of microwave usage

123
12760 Neural Computing and Applications (2022) 34:12749–12763

Fig. 13 One week behavior

pattern of toaster usage

Fig. 14 One week behavior

pattern of washing machine
usage

5.1 Sensitivity analysis Table 7 The comparison with existing work

Comparision Precision (p) Recall (r) F1-score
A sensitivity analysis is performed to analyze the impact of
the data augmentation. The performance metrics (i.e., Devlin et al. [5] 0.76 0.85 0.77
precision, recall, and F1-score) are reported in Table 6. ApplianceNet (Our model) 0.87 0.88 0.88
Furthermore, Fig. 10 presents the accuracy of the pro-
posed model with and without data augmentation.
Figure 10 presents the obtained individual appliance
accuracy. It confirms data augmentation produces accurate to see the sedentary pattern of the resident. The one-week
results for minor classes. In the case of ‘Toaster’, it does behavior has two peeks more than four houses to watch the
not recognize any usage, while in the case of data aug- television, while Friday has the least time to watch it. In
mentation, it is fully recognized. The augmentation assists Fig. 12, we can observe on Friday there is no usage of the
the model to learn the signature of the appliance. microwave. This pattern is the same for Saturday as well. It
may possible to dine out or skip the meal. To extend this
5.2 Behavioural recognition work, we can confirm with the user about such situations.
However, ‘Toaster’ is recognized on Friday as shown in
We also presented the weekly usage of home appliances, Fig. 13. This home appliance can provide the eating pat-
which can be related to resident behavior. In Fig. 11, the tern of the resident that may help the individual or prac-
number of hours is presented to watch the television. This titioner to analyze the daily routines of the resident. In
pattern can be extended to months. It may have an indicator Fig. 14, the usage pattern is found on Friday, Saturday,

123
Neural Computing and Applications (2022) 34:12749–12763 12761

Table 8 The overall ApplianceNet accuracy as compared to the state- desktop PC, and Fridge. The average of performance
of-the-art techniques measures are reported in Table 7.
Model Precision (p) Recall (r) F1- Accuracy In Table 7, the ApplianceNet has better performance as
score compared to the existing method. The common appliances
in both models are ‘Microwave’, ‘Kettle’ and ‘Toaster’,
ApplianceNet 0.87 0.88 0.88 0.89
where our framework is superior by 14%, 7% and 11%
k-nearest neighbor 0.78 0.73 0.73 0.83
respectively in terms of F1-score. Furthermore, we also
Decision tree 0.68 0.62 0.62 0.80
compare the performance of our classifier in ApplianceNet
Naive bayes 0.55 0.48 0.48 0.53
with three state-of-the-art machine learning algorithms.
These methods are decision tree, Naive Bayes classifier,
and k-nearest neighbours with default parameters. The
selection of these algorithms is based on the most fre-
Sunday, and Wednesday. This washing machine usage quently used models for the classification task. The Naive
pattern is frequent and may help to diagnose complications Bayes classifier is probability based model, which is con-
by practitioners or healthcare givers. Such information can sider as gold standard for making comparison, decision tree
also be useful for energy providers as well as residents to is well known method based on information theory to
use the washing machine in an optimal way. Consequently, calculate the information gain, and k nearest neighbour is
it can help to reduce energy consumption. the non-parametric approach that is ranked in top 10
methods in data mining algorithm. Table 8 presents the
5.3 Comparative analysis improvement obtained by ApplianceNet.
Table 8 shows that our model provides better results as
The direct comparison of presented ApplianceNet with compared to the state-of-the-art methods. Furthermore, we
existing models is difficult, since the different datasets have also presented the individual accuracy of each appliance to
been used with different set of appliances. However, a analyze the improvement.
recent study Devlin et al. [5] classify the activities of daily In Fig. 15, it is clear that the proposed ApplianceNet
living using residential smart meter data were identified performs better in most of the cases. Our model obtained
that is similar to our work in terms of recognizing the the best results in the case of ‘Microwave’, ‘Kettle’,
appliances. To recognize daily living activities they con- ‘Toaster’, and ‘Diswasher’ where state-of-the-art models
sider seven appliances—hair dryer, vacuum cleaner, iron, have more confusion to recognize these appliances.
microwave, kettle, toaster, oven. In comparison to our
framework, we recognized eight appliances—television,
microwave, kettle, washing machine, toaster, dish washer,

Fig. 15 The ApplianceNet

individual appliance recognition
accuracy as compared to the
state-of-the-art techniques

123
12762 Neural Computing and Applications (2022) 34:12749–12763

6 Conclusion 5. Devlin MA, Hayes BP (2019) Non-intrusive load monitoring and

classification of activities of daily living using residential smart
meter data. IEEE Trans Consum Electron 65(3):339–348
The advancement in technology has enabled the research 6. Prati A, Shan C, Wang KI-K (2019) Sensors, vision and net-
community to recognize daily life activities in the home works: from video surveillance to activity recognition and health
environment which can be used in multiple services. In this monitoring. J Amb Intell Smart Environ 11(1):5–22
7. Ahmadi-Karvigh S, Ghahramani A, Becerik-Gerber B, Soibel-
research, we present ApplianceNet to identify the home man L (2018) Real-time activity recognition for energy efficiency
appliances using smart plugs to recognize daily life activ- in buildings. Appl Energy 211:146–160
ities in the home environment. Our model processed the 8. Hamad RA, Hidalgo AS, Bouguelia M-R, Estevez ME, Quero JM
temporal energy consumption data to extract basic and (2019) Efficient activity recognition in smart homes using
delayed fuzzy temporal windows on binary sensors. IEEE J
complex characteristics in the form of features. These Biomed Health Inform 24(2):387–395
features are further processed by a multi-layer, feed-for- 9. Yassine A, Singh S, Alamri A (2017) Mining human activity
ward neural network to classify the home appliances which patterns from smart home big data for health care applications.
are non-linear separable after the feature extraction tech- IEEE Access 5:13131–13141
10. Murray DM, Liao J, Stankovic L, Stankovic V (2016) Under-
nique. This approach has the potential to be adopted in standing usage patterns of electric kettle and energy saving
existing homes as a low-cost and sustainable solution with potential. Appl Energy 171:231–242
superior recognition as compared to the existing solutions. 11. Liu Q, Kamoto KM, Liu X, Sun M, Linge N (2019) Low-com-
In future work, the research community can develop plexity non-intrusive load monitoring using unsupervised learn-
ing and generalized appliance models. IEEE Trans Consum
consumer applications based on the daily living activities Electron 65(1):28–37
for diverse applications. 12. Bennett J, Rokas O, Chen L (2017) Healthcare in the smart home:
a study of past, present and future. Sustainability 9(5):840
13. Amiribesheli M, Benmansour A, Bouchachia A (2015) A review
of smart homes in healthcare. J Ambient Intell Humaniz Comput
Declarations 6(4):495–517
14. Tan T-H, Gochoo M, Huang S-C, Liu Y-H, Liu S-H, Huang Y-F
(2018) Multi-resident activity recognition in a smart home using
Conflicts of interest The authors declare that they have no conflict of
rgb activity image and dcnn. IEEE Sens J 18(23):9718–9727
interest.
15. Mehr HD, Polat H (2019) Human activity recognition in smart
home with deep learning approach. In: IEEE 7th international
Open Access This article is licensed under a Creative Commons
Istanbul Smart Grids and Cities Congress and Fair (ICSG),
Attribution 4.0 International License, which permits use, sharing,
pp 149–153
adaptation, distribution and reproduction in any medium or format, as
16. Chen K, Zhang Yu, Wang Q, Hu J, Fan H, He J (2019) Scale-and
long as you give appropriate credit to the original author(s) and the
context-aware convolutional non-intrusive load monitoring. IEEE
source, provide a link to the Creative Commons licence, and indicate
Trans Power Syst 35(3):2362–2373
if changes were made. The images or other third party material in this
17. Chen J, Wang X, Zhang X, Zhang W (2021) Temporal and
article are included in the article’s Creative Commons licence, unless
spectral feature learning with two-stream convolutional neural
indicated otherwise in a credit line to the material. If material is not
networks for appliance recognition in NILM. IEEE Trans Smart
included in the article’s Creative Commons licence and your intended
Grid
use is not permitted by statutory regulation or exceeds the permitted
18. Fahim M, Fatima I, Lee S, Lee Y-K (2013) Eem: evolutionary
use, you will need to obtain permission directly from the copyright
ensembles model for activity recognition in smart homes. Appl
holder. To view a copy of this licence, visit http://creativecommons.
Intell 38(1):88–98
org/licenses/by/4.0/.
19. Chen L, Nugent CD, Wang H (2011) A knowledge-driven
approach to activity recognition in smart homes. IEEE Trans
Knowl Data Eng 24(6):961–974
References 20. Yan D, Jin Y, Sun H, Dong B, Ye Z, Li Z, Yuan Y (2019)
Household appliance recognition through a Bayes classification
1. Carrie Armel K, Gupta A, Shrimali G, Albert A (2013) Is dis- model. Sustain Cities Soc 46:101393
aggregation the holy grail of energy efficiency? The case of 21. Franco P, Martı́nez JM, Kim Y-C, Ahmed MA (2021) IoT based
electricity. Energy Policy Jan 52:213–234 approach for load monitoring and activity recognition in smart
2. Hayes B, Hernando-Gil I, Collin A, Harrison G, Djokić S (2014) homes. IEEE Access 9:45325–45339
Optimal power flow for maximizing network benefits from 22. Paradiso F, Paganelli F, Luchetta A, Giuli D, Castrogiovanni P
demand-side management. IEEE Trans Power Syst (2013) ANN-based appliance recognition from low-frequency
29(4):1739–1747 energy monitoring data. In: Proceedings of the 2013 IEEE 14th
3. Gajowniczek K, Zabkowski T (2015) Data mining techniques for international symposium on’’ a world of wireless, mobile and
detecting household characteristics based on smart meter data. multimedia networks’’ (WoWMoM), pp 1–6. IEEE
Energies 8(7):7407–7427 23. Smart plug (current cost) (accessed date: 7 july 2020). http://
4. Yang D, Gao X, Kong L, Pang Y, Zhou B (2020) An event-driven www.currentcost.com/product-iams.html
convolutional neural architecture for non-intrusive load moni- 24. Yamauchi M, Ohsita Y, Murata M, Ueda K, Kato Y (2020)
toring of residential appliance. IEEE Trans Consum Electron Anomaly detection in smart home operation from user behaviors
66(2):173–182 and home conditions. IEEE Trans Consum Electron
66(2):183–192

123
Neural Computing and Applications (2022) 34:12749–12763 12763

25. Christ M, Braun N, Neuffer J, Kempa-Liehr AW (2018) Time 29. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B,
series feature extraction on basis of scalable hypothesis tests Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al
(tsfresh-a python package). Neurocomputing 307:72–77 (2011) Scikit-learn: machine learning in python. J Mach Learn
26. Christ M, Kempa-Liehr AW, Feindt M (2016) Distributed and Res 12:2825–2830
parallel time series feature extraction for industrial big data 30. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G,
applications. arXiv preprint arXiv:1610.07717 Killeen T, Lin Z, Gimelshein N, Antiga L (2019) et al. Pytorch:
27. Benjamini Y, Yekutieli D (2001) The control of the false dis- an imperative style, high-performance deep learning library. In:
covery rate in multiple testing under dependency. Ann Stat Advances in neural information processing systems,
1165–1188 pp 8026–8037
28. Murray D, Stankovic L, Stankovic V (2017) An electrical load
measurements dataset of UK households from a two-year longi- Publisher’s Note Springer Nature remains neutral with regard to
tudinal study. Sci Data 4(1):1–12 jurisdictional claims in published maps and institutional affiliations.

123