Name:

RU ID:

Towards Environment Independent Device Free Human Activity Recognition

1. Introduction
Human activity recognition is a pivotal technique in many real-world applications, such as fitness
tracking, elder care and well-being monitoring, smart home. As a traditional way to recognize activities,
wearable device-based approaches (e.g., wristband, mobile phone, body attach sensor) derive the human
motion and infer the activity through attaching motion sensors or mobile devices to human body.
However, the device attachment would bring inconvenience and discomfort to the user, especially for
elderly people with age-related memory loss. To release the users from device attachment, researchers
have been actively searching for alternative solutions, and the most promising one is device-free activity
recognition technique.

Device-free activity recognition explores wireless signals transmitted between a pair of wireless devices
(e.g., smartphone, laptop, WiFi access point) to infer the activities of a person located between or around
the sender and the receiver. The insight is that the person’s activity would introduce variations of
wireless signals reflecting activity characteristics. To date, various device-free activity recognition
approaches and systems have been developed. However, a key challenge remains unsolved, that is, the
wireless signals arriving at the receiving devices usually carry substantial information that is specific to
the environment where the activities are recorded and the human subject who conducts the activities.
Due to this reason, an activity recognition model that is trained in a specific subject in a specific
environment will not typically work well when being applied to recognize another user in another
environment.

To address this challenge, the authors of this paper proposed EI [1], a deep-learning activity recognition
framework that can remove the environment specific information entangled in the wireless signals. The
key idea is to train a domain discriminator along with a CNN based activity recognition model, which
has to capability to remove the environment specific characteristics and derive environment-
independent features. The trained CNN model can then be applied to new environment for activity
recognition. To reduce the burden of data collection, beside labeled data, the authors also propose to
use unlabeled data to train the domain discriminator. Extensive evaluations are conducted with four
different device free activity recognition testbeds, including WiFi, ultrasound, 60 GHz mmWave, and
visible light.

2. Approach and Methodology

The objective of the proposed system is to predict
activity with unlabeled data collected in new
environment. To achieve this goal, the authors explore
a series of data processing pipeline and deep learning
methods to achieve environment-independent activity
recognition as show in Figure 1.
Figure 1: System overview.
2.1 Data Collection and Data Preprocessing
To construct the deep learning model, the system first collects wireless signals (e.g., WiFi and acoustic)
of human activities from a few environments. For some environments, part of the collected data are
manually labeled, and for others, the label information is not provided. The objective of the system is
to predict human activities on unlabeled data. Both of the labeled and unlabeled data are converted into
a more suitable form for training through data normalization and segmentation.

2.2 Feature Extractor and Activity Recognizer

The authors utilize convolutional neural
network (CNN) to extract features as shown
in Figure 2, which are popular in image
classification tasks. A three-layer
architecture is developed. Each layer consists
a convolutional layer with 2D filters, a batch
normalization layer. The activation functions
are all rectified linear unit (ReLU)
Figure 2: Deep learning model.

Based on the extracted feature, a fully collected layer is used to learn feature representations and a
SoftMax layer is used to calculate probability of over predefined activities. For the labeled training data,
the authors use cross-entropy between the prediction and the ground truth as the loss function.
Minimizing the cross-entropy during training process increases the prediction performance of the deep
learning model. In addition, for the unlabeled data, the author exploit entropy of the predictions as loss,
which can increase the model’s confidence on predicting unlabeled data.

2.3 Domain Discriminator

In order to achieve environment-independent activity recognition, this paper exploits unsupervised
domain adaptation [2] to remove the environment-specific information. The key idea is to force the
feature extractor to extract environment-independent features, by using a domain discriminator as
shown in Figure 2.

The domain discriminator is a neural network taking the extracted feature as input and predict the
domain (i.e., environment) of input data. This seems to contradict with the goal of learning domain-
independent features of activity. But by using an adversarial loss, the feature extractor can be trained in
a way to cheat the domain discriminator (i.e., minimizing the domain prediction accuracy). Through
this minimax game, the feature extractor can extract common features in different environment. The
adversarial loss is defined as:
𝐿 = 𝐿𝛼 + 𝛼𝐿𝑢 − 𝛽𝐿𝑑 ,
where 𝐿𝛼 is the loss of labeled data and 𝐿𝑢 is the loss of unlabeled data. 𝐿𝑑 is the domain prediction
loss of the domain discriminator. By applying a negative factor to the domain loss (i.e., −𝛽), the feature
extractor will learn to cheat the domain discriminator, by minimizing the domain prediction
performance, and thus remove the domain-specific information.
3. Performance Evaluation
The authors evaluate their system on four
different device free activity recognition
testbeds, i.e., WiFi, ultrasound, 60 GHz
mmWave and visible light, to evaluate the
performance of the proposed system. To
save space we only show the performance
Figure 3: Setup for Figure 4: Setup for evaluation
of WiFi and ultrasound, with the evaluation with WiFi. with Ultrasound.
experiment setup shown in Figure 3 and
Figure 4. The authors involve 12 volunteers as subjects to conduct 6 activities, including wiping the
whiteboard, walking, moving a suitcase, rotating the chair, sitting, as well as standing up and sitting
down. The activities are conducted in 6 different rooms. They compare the proposed system with two
state-of-the-art domain adaptation methods CAT [3] and VADA [4].

WiFi. For WiFi testbed, the authors use channel state information
(CSI) measurement from 30 subcarrier groups as system input. The
CSI measurements are normalized to have zero mean and unit
standardization and denoised with Hampel filter. The processed CSI
is then segmented and fed to a deep learning model for training. The
activity recognition results under different domains (i.e.,
environment and user pair) are shown in Figure 5. The proposed deep Figure 5: Accuracy of the
learning model (EI) shows the best performance. system on CSI data.

Ultrasound. The authors also test the proposed system on ultrasound

signals. An iPad with an ultrasound generator app is used as the
transmitter and a smartphone is used as the receiver. The activity
recognition results are shown in Figure 6. The proposed deep learning
still achieves the highest activity recognition performance, with over
70% accuracies for most cases. In addition, as the number of the
domain increased, the deep learning model has higher accuracy. Figure 6: Accuracy of the
system on ultrasound data.
4. Related Work
Existing work has explored various types of wireless signals (e.g., acoustic signals, WiFi siganls,
mmWave signals) to perform activity recognition. These approaches could have good performance on
activity recognition, but they all face the challenge brought by environmental variation. Recently, deep
learning based-domain adaptation attracts the attention of researchers, due to its capability on
addressing various of domains (e.g., environment). This paper also aligns with this line of research.
Different from existing work, which mainly relies on supervised adaptation, the system proposed in this
paper explores both labeled and unlabeled data for training (i.e., unsupervised adaptation).

5. Conclusion
This paper aims to address a very challenging problem, mitigating the impacts of environmental
variations in device-free activity recognition. While device-free sensing is promising, as it does not
require sensor attachment, the wireless signals are inherently susceptible to environmental dynamics
(domain variations) due to their omni-directional propagation property. Training models robust under
such dynamics is indeed an interesting and practical research problem. To achieve this goal, a domain
discriminator is used to remove environment-specific characteristics reside in wireless signals. The
underlying idea is to formulate a two-player game between the classifier (trained with labeled data from
a source domain) and the domain discriminator (trained to differentiate unlabeled data from the source
and a target domain). By applying an adversarial loss on the feature extractor, the features are optimized
to confuse the domain discriminator, making the features domain(environment)-independent.
Comprehensive evaluation on four different device-free activity recognition testbeds show the
capability of the system.

The idea of leveraging domain discriminator to remove environment-specific factor is interesting.

Compared to the expensive labeled data, which require human efforts for labeling, the cost of unlabeled
data collection is low. The designed unsupervised learning method thus can maintain model’s
performance under environmental dynamics with low costs. However, there are still some concerns for
this paper. First, the activity recognition accuracy is not high enough for practical use, i.e., less than 80%
for all four testbeds, even training with data of a large number of domains (e.g., 6 domains). For practical
use, the accuracy should reach at least 90%. The reason could be the domain discriminator removes a
part of activity characteristics along with environment-dependent characteristics, leading to the
decreased performance of the deep learning model after domain adaptation. Second, the authors do not
study applying the domain adaptation to unseen environments. The method still requires unlabeled data
for training. This means that the system needs to take time to collect unlabeled data in a new
environment before deployment, which reduces the system’s feasibility in practice. A more practical
scenario could be the device-free activity recognition system can be directly deployed in a new
environment without needing any new training data from it. Third, the proposed system solely relies on
learning methods to achieve environment-independent, which could make the system susceptible to
other impacting factors (e.g., weather, air condition).

Instead of solely relying on deep learning, the system performance can be enhanced by incorporating
advanced signal processing techniques. The authors have the potential to craft state-of-the-art filtering
methods that effectively exclude signals echoed from the ambient environment, particularly from
surfaces like walls. A potential method could involve estimating the distance of a signal's propagation
path. With such an approach, the system could detect signals that reflected from a human subject and
those from walls or other static entities, and then filter them out. In addition, Sophisticated features,
grounded in advanced signal processing techniques, can be derived to provide richer information. For
example, time-frequency analysis, spectral analysis of signals, and wavelet transformation, can be
utilized to capture human motion patterns while reducing the impacts of signal reflection from static
objects. Furthermore, integrating multi-dimensional sensing measurements for activity recognition
could also potentially improve the activity recognition performance. For example, the authors may
integrate sensing measurements of multiple antennas and relationships among multiple subcarriers to
improve the activity recognition performance.
Reference
[1] Wenjun Jiang, Chenglin Miao, Fenglong Ma, Shuochao Yao, Yaqing Wang, Ye Yuan, Hongfei Xue
et al. "Towards environment independent device free human activity recognition." In Proceedings of
the 24th annual international conference on mobile computing and networking, pp. 289-304. 2018.
[2] Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation.
In International Conference on Machine Learning. 1180–1189.
[3] Mingmin Zhao, Shichao Yue, Dina Katabi, Tommi S Jaakkola, and Matt T Bianchi. 2017. Learning
sleep stages from radio signals: A conditional adversarial architecture. In International Conference on
Machine Learning. 4100–4109
[4] Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. 2018. A DIRT-T Approach to
Unsupervised Domain Adaptation. In International Conference on Learning Representations.