See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/312056813

EdgeIoT: Mobile Edge Computing for the Internet of Things

Article in IEEE Communications Magazine · December 2016

DOI: 10.1109/MCOM.2016.1600492CM

CITATION
S READS

673 9,288

2 authors:

Xiang Sun
Nirwan Ansari
University of New Mexico
New Jersey Institute of Technology
75 PUBLICATIONS 2,189
CITATIONS 796 PUBLICATIONS 27,457 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Xiang Sun on 13 October 2017.

The user has requested enhancement of the downloaded file.

 INTERNET OF THINGS

EdgeIoT: Mobile Edge Computing for the

Internet of Things
Xiang Sun and Nirwan Ansari

In order to overcome
the scalability problem ABSTRACT there will be 50 to 100 billion smart devices
con- nected to the Internet by 2020 [1]). Also,
of the traditional Internet
In order to overcome the scalability problem Cisco has predicted that the devices connected
of Things architecture of the traditional Internet of Things architecture to the Internet will generate 507.5 ZB/year by
(i.e., data streams (i.e., data streams generated from distributed 2019 [2]. Meanwhile, due to the flexible and
generated from the IoT devices are transmitted to the remote cloud efficient resource provisioning in the cloud [3],
distributed IoT devices via the Internet for further analysis), this arti- the big IoT data generated from the distributed
are transmitted to the cle proposes a novel approach to mobile edge IoT devic- es are transmitted to the remote
remote cloud via the computing for the IoT architecture, edgeIoT, to cloud, a smart “brain” for processing big data,
Internet for further handle the data streams at the mobile edge. via the Internet in the traditional IoT architecture
analysis), the authors Spe- cifically, each BS is connected to a fog [4, 5], as shown in Fig. 1. However, the Internet
node, which provides computing resources is not scalable and efficient enough to handle
propose edgeIoT, a locally. On the top of the fog nodes, the SDN- IoT big data. Mean- while, transferring the big
novel approach to based cellular core is designed to facilitate data is expensive, con- suming a huge amount
mobile edge computing packet forwarding among fog nodes. of bandwidth, energy, and time. Second, since
for the IoT architecture, Meanwhile, we propose a hierarchical fog the IoT big data streams are transmitted to the
intended to handle the computing architecture in each fog node to cloud in high volume and at fast velocity, it is
data streams at the provide flexible IoT services while maintaining necessary to design an efficient data
mobile edge. user privacy: each user’s IoT devices are processing architecture to explore the valu- able
associated with a proxy VM (located in a fog information in real time. Third, user privacy
node), which collects, classifies, and analyzes remains a challenging unsolved issue; that is, in
the devices’ raw data streams, converts them order to obtain services and benefits, users
into metadata, and transmits the metadata to should share their sensed data with IoT service
the cor- responding application VMs (which are providers, and these sensed data may contain
owned by IoT service providers). Each users’ person- al information. Thus, it is critical
application VM receives the corresponding to design a data sharing framework so that
metadata from differ- ent proxy VMs and users can acquire IoT services while their
provides its service to users. In addition, a privacy is guaranteed. In this article, we
novel proxy VM migration scheme is proposed propose an efficient and flexible IoT
to minimize the traffic in the SDN- based core. architecture, edgeIoT, by leveraging fog
comput- ing and software defined networking
INTRODUCTION (SDN) to collect, classify, and analyze the IoT
Today, a tremendous number of smart devic- data streams at the mobile edge. The article
es and objects are embedded with sensors, makes the follow- ing contributions:
enabling them to sense real-time information • We propose edgeIoT by bringing the com-
from the environment. This phenomenon has puting resources close to IoT devices so
culminated in the intriguing concept of the Inter- that the traffic in the core network can be
net of Things (IoT) in which all smart things, allevi- ated and the end-to-end (E2E) delay
such as smart cars, wearable devices, laptops, between computing resources and IoT
sensors, and industrial and utility components, devices is mini- mized.
are connected via a network of networks and • We design a hierarchical fog computing
empowered with data analytics that are forever architecture to provide flexible and
changing the way we work, live, and play. In the scalable computing resource provisioning
past few years, many startups have embraced for each user as well as each IoT service
and actualized the concept of IoT in areas provider.
including smart homes/buildings, smart cities, • We propose and evaluate a novel proxy
intelligent healthcare, smart traffic, smart envi- vir- tual machine (VM) migration scheme to
ronments, and so on. Although IoT can poten- min- imize the traffic in the core network.
tially benefit all of society, many technical The rest of the article is structured as fol-
issues remain to be addressed. lows. We introduce a new mobile edge com-
First, the data streams generated by the IoT puting for IoT architecture (i.e., edgeIoT), and
Digital Object Identifier: devices are high in volume and at fast velocity explain its efficiency and flexibility; we unveil
(the European Commission has predicted that the challenges in designing the edgeIoT archi-
tecture and propose some possible solutions;
we conclude the article.

2 0163-6804/16/$25.00 © 2016 IEEE IEEE Communications Magazine •

2 December 2016
10.1109/MCOM.2016.1600492CM The authors are with the New Jersey Institute of Technology.

IEEE Communications Magazine • 23

December 2016
 Access
Laptop Smart car points Cloud

... Internet
Smartphone Smart home

Wearable Surveillance
device device

••••••
Sensor

FIGURE 1. The traditional IoT architecture. band IoT communications. Thus, a multi-interface BS can be considered as
a wireless gateway to

MOBIlE EDGE COMpUTING FOR IOTS

Fog computing [6], which is defined as a distrib-
uted computing infrastructure containing a bunch
of high-performance physical machines (PMs)
that are well connected with each other, is an
emerg- ing computing paradigm bringing the
computing capabilities close to distributed IoT
devices. Thus, deploying a number of fog nodes
in the network can locally collect, classify, and
analyze the raw IoT data streams, rather than
transmitting them to the cloud; this can
significantly alleviate the traffic in the core
network and potentially speed up the IoT big
data process. However, where to deploy the fog
nodes to facilitate the communi- cations
between IoT devices and fog nodes is still an
open issue. The optimal fog computing deploy-
ment ensures that each IoT device has access
to computing capabilities everywhere with low
E2E delay and without significantly increasing
the traf- fic of the core network. It is difficult to
optimize the deployment of fog nodes due to
the mobility and heterogeneity features of the IoT
devices. For example, wearable devices and
mobile phones move over time, and different
IoT devices have different data transmission
requirements, that is, some energy-insensitive
devices (e.g., mobile phones and surveillance
devices) need high-speed data rate, and some
energy-sensitive devices (e.g., sensor nodes)
require low-speed and low-energy data
transmission. The heterogeneous data trans-
mission requirements among IoT devices result
in different devices adopting different wireless
access technologies.
MUlTI-INTERFACE BASE STATIONS IN
CEllUlAR NETwORk
A huge number of base stations (BSs), which
have already been deployed in the mobile
network, provide high radio coverage. Thus,
distributed BSs have the potential to connect all
IoT devices whether they are moving or static.
In order to sup- port different data transmission
requirements of IoT devices, each BS may be
equipped with multi- ple wireless interfaces, as
shown in Fig. 2, to facili- tate emerging IoT-based
wireless communications technologies such as
Zigbee, device-to-device (D2D)
communications with relay, Bluetooth low
energy, millimeter-wave and massive multiple-in-
put multiple-output (MIMO) communications,
low-power wide area technologies, and narrow-

2 0163-6804/16/$25.00 © 2016 IEEE IEEE Communications Magazine •

2 December 2016
 can
network management operators so that
different network functionalities, such as offload their computing
mobility management, user authentication, workloads to the cloud
aggregate all the autho- rization and accounting, network
raw data streams at the expense of con-
visualization, and QoS control, can be added,
from local IoT removed, and modified flexibly. suming more network
devices. Therefore, Note that each fog node has the ability to resources and higher
a potential access the cloud through the Internet to
deployment is to communications latency.
provision computation availability and flexible
connect each BS to application service deployment. That is, when Furthermore, IoT appli-
a fog node to the fog nodes do not have enough computing cations can be
process the resources to process their local data streams, deployed in the local
aggregated raw data they can offload their computing workloads to
streams. the cloud at the expense of consuming more fog nodes or in the
network resources and higher communications remote cloud to offer
THE EDGEIOT latency. Furthermore, IoT applications can be services to users.
ARCHITECTURE deployed in the local fog nodes or in the remote
Figure 3 shows the cloud to offer services to users. The flexible
proposed edgeIoT. application service deployment is detailed later.
The loca- tions of
the fog nodes are HIERARCHICAl FOG COMpUTING ARCHITECTURE
flexible: a fog node Most of the data generated by users’ devices
can be directly contain personal information, such as photos/
connected to a BS videos taken by mobile phones and smart cars,
via high-speed GPS information, health information sensed by
fibers to transmit the wearable devices, and smart home status
local data streams sensed
with the minimum
E2E delay, or can
be deployed at the
edge of the cellular
core network so that
different BSs can
share the same fog
node to process their
local data streams.
Instead of applying
the tradi- tional
cellular core
network, which
leads to ineffi- cient,
inflexible and
unscalable packet
forwarding and
quality of service
(QoS) management,
the SDN-based
cellular core was
introduced [7, 8].
OpenFlow switches
are adopted in the
SDN cellu- lar core
to separate out all
control functions
from the data
forwarding function.
All the switches as
well as BSs are
controlled by the
OpenFlow con-
troller via the
OpenFlow protocol
[9]. The Open- Flow
controller manages
the forwarding plane
of BSs and
OpenFlow switches,
monitors the traffic
at the data plane,
and establishes user
sessions. Also, it
provides application
programming inter-
faces (APIs) to

IEEE Communications Magazine • 25

December 2016
 Proxy VM decompo-
Zigbee:
D2D communications with relay: Support devices with short transmission
Support devices with different range, low data rate, and long battery
data rate requirements. life requirements.
sition refers to the Multi-
interface BS

deconsolidation
...
of the WiFi(802.11ah):
Bluetooth low energy: Support devices with low data rate,
original proxy VM into Support devices with high data long transmission range, small and
rate and short transmission range. infrequent data transmission, and non-
two separate proxy critical delay.
VMs, each of which
serves a subset of the
registered IoT devices
from the original proxy
VM; conversely, proxy Millimeter-wave and massive Low-power wide area network Narrowband IoT:
VM composition refers MIMO technologies: (LPWAN): Support devices Support devices with long transmission
Support devices with high data rate with low mobility, low power, range, low data rate, and long battery
to the consolidation of transmission. and low cost in a wide area life requirements.
network.

FIGURE 2. An illustration of a multi-interface BS.

VM.
by the sensors deployed in a smart home. Ana-
lyzing such humongous data can benefit not only
the user her/himself but also all of society. For
instance, analyzing the photos/videos taken by
devices can identify and track a terrorist. Specif-
ically, the application provider sends a photo of
the terrorist to each fog node, and each fog node
locally performs face matching to compare the
terrorist’s photo with the photos/videos taken by
local devices. If matched, the fog node will upload
the corresponding photos/videos to the cloud for
further processing. Thus, it seems that users have
to share their personal data in order to provision
such services. The main challenge is to maintain
user privacy in provisioning such services.
To tackle this challenge, we propose a hier-
archical fog computing architecture. As shown in
Fig. 4, each user1 is associated with a proxy VM,
which is considered as the user’s private VM
(located in a nearby fog node) that pro- vides
flexible computing and storage resources. IoT
devices belonging to the user are registered to
the user’s proxy VM, which collects the raw data
streams generated from its registered devic- es
via a multi-interface BS, classifies them into
different groups based on the type of data (i.e.,
structurize the raw data steams), generates the
metadata by analyzing the corresponding data
streams, and sends the metadata to the corre-
sponding application VM. Note that the metadata
contains valuable information generated from the
raw data streams without violating user privacy.
For instance, in the terrorist detection application,
only the locations and timestamps of the matched
photos/videos, rather than the original photos/
1 A user can be a person videos, are uploaded to the application VM. The
who owns various private
IoT devices, an
application VM, which is owned by the IoT ser-
entity/company that vice provider, offers the semantic model for gen-
deploys a set of IoT erating the metadata by each proxy VM (e.g., the
devices in the area, such face matching algorithm in the terrorist detection
as the surveillance
cameras, or a group of
application), receives the metadata from differ-
users who trust each other ent proxy VMs, and provides services to users.
and share the same proxy

24 IEEE Communications Magazine •

December 2016
For instance, all the conversely, proxy VM composition refers to the
terrorists will be consolidation of two proxy VMs (which belong
identified, tracked, to the same user) into one proxy VM, which
and arrested by serves all the registered IoT devices from the
analyzing the original two proxy VMs. In addition, proxy VM
metadata from migration involves moving the whole proxy VM
different proxy VMs, (containing profiles, semantic models, and
thus safeguarding recent sensed data of the registered IoT
our society. devices) from a source PM to a destination
The locations of PM. The proxy VM composition/ decomposition
proxy VMs can be process always invokes the proxy VM migration
dynamic: if the process.
registered devices The locations of application VMs are also
are statically dynamic and flexible: each application VM can
deployed (e.g., the be deployed in the local mode, remote mode,
sensors in the smart or add-on mode.
home), the proxy
VM can also remain
static in the nearby
fog node; if some of
the registered
devices are mobile
(e.g., a user’s
mobile phone and
wearable devices
move from home to
workplace), as
shown in Fig. 5, the
user’s proxy VM can
be decomposed into
two proxy VMs: one
proxy VM continues
to serve the static
IoT devices (in the
home), and the
other proxy VM
migrates to the
other fog nodes as
the mobile IoT
devices roam away.
The purpose of
proxy VM migration
is to minimize the
traffic (i.e.,
uploading the raw
data streams from
mobile devices to a
proxy VM in the fog
node) of the cellular
core network as well
as the E2E delay
between a user’s
mobile IoT devices
and its proxy VM.
Proxy VM
decomposition
refers to the
decon-
solidation of the
original proxy VM
into two sep- arate
proxy VMs, each of
which serves a
subset of the
registered IoT
devices from the
original proxy VM
(i.e., each proxy VM
contains profiles
and semantic
models of its served
IoT devices);

IEEE Communications Magazine • 25

December 2016
 Each mobile IoT
Multi-interface
Fog node base station
Cloud
Internet OpenFlow OpenFlow core
access switch switch
Control link Data link
device’s proxy VM may
vary over time owing
SDN based cellular Network management
core (data plane) operator to the decomposition/
API ••••••
composition process-
OpenFlow QoS Mobility
controller controller management es. Thus, the proxy
•••••• VM needs to inform
... User AAA its registered mobile
database server
SDN based cellular IoT devices when the
core (control plane) decomposition/compo-
sition processes

FIGURE 3. The edgeIoT architecture. triggered.

Local application VM deployment refers to the matching algorithm in order to compare recent
deployment of an application VM in the fog photos/videos captured by the proxy VMs’ reg-
node to analyze the metadata generated by the istered devices to the photos of lost children/
local proxy VMs;2 for instance, in the ParkNet terrorists, and return the locations and time
applica- tion [10], which helps users find stamps of the photos/videos if found.
available parking spots in the urban area, each
local proxy VM col- lects the sensed data HOw TO IMplEMENT EDGEIOT ApplICATIONS
streams from its smart cars (note that each If a user is interested in one IoT application
smart car is equipped with a GPS receiver and (e.g., the ParkNet application), they can
a passenger-side-facing ultrasonic range finder download and install this app in their smart
to generate the location and park- ing spot car/mobile phone. Accordingly, the user’s proxy
occupancy information) and generates the VM will install the semantic model (which
metadata, which identify the available parking calculates the available parking spots based on
spots, to the application VM. The application the sensed data) provid- ed by the ParkNet
VM will inform and assign the available parking application, and the semantic model in the
spots to the local smart cars. user’s proxy VM would have permis- sion to
Remote application VM deployment refers access the sensed data generated by the GPS
to the deployment of an application VM in the receiver and the passenger-side-facing ultra-
remote cloud to analyze the metadata generat- sonic range finder equipped in the user’s smart
ed by the proxy VMs from different fog nodes. car. As a reward, the user can request to find
This deployment is necessary if an application and reserve an available parking spot via the
VM needs information from a large area, such ParkNet application.
as traffic rerouting applications. Specifically, the
goal of the application is to detect the traffic CHAllENGES IN IMplEMENTING EDGEIOT
hotspots and select the best routing (i.e., the In this section, we point out some challenges in
shortest time to reach the destination) for users. implementing the proposed edgeIoT architecture
In order to detect the traffic hotspots, each and the corresponding solutions.
smart car is equipped with sensors to measure
the location and speed of the car. The sensed IDENTIFICATIONS BETwEEN
data streams are transmitted to the proxy VMs, IOT DEvICES AND THEIR pROxy vMS
which locally analyze the data streams and Initially, each user’s IoT devices should be iden-
gen- erate the metadata indicating the traffic tified/registered by its proxy VM. The proxy VM
conges- tion degree of the location. The central should know the IDs3 of all the user’s devices
server in the remote cloud receives the and their corresponding characteristics (i.e.,
metadata from the proxy VMs and selects the static or mobile devices, smart sensors sensing
best route for each user. data or actuators responding with actions, the
Add-on application VM deployment (i.e., types of sensed data, etc.). On the other hand,
event-triggered application VM deployment) the user’s IoT devices should also be informed
implies that an application VM can be locally of the ID of the proxy VM so that sensor devices
created by some events, such as the terrorist can transmit the private information to the
detection application and the find-missing-chil- correct proxy VM or actuator devices can
dren application [11]. The events, like lost receive commands from the correct proxy VM.
children and terrorist activities detection, are Each mobile IoT device’s proxy VM may
reported in a specific area, and the applications vary over time due to the
need to identify and track the lost children/ decomposition/compo- sition processes. Thus,
terrorists. Then the applications will be creat- the proxy VM needs to inform its registered
ed in each fog node in the area and request mobile IoT devices when the
each proxy VM in the fog node to run the face decomposition/composition processes are

26 IEEE Communications Magazine •

December 2016
2Local proxy VMs refer to the proxy VMs and applica- tion
VMs located in the same fog node.
3 Recently, many methods have been proposed to identify
IoT devices, such as electronic product codes, ubiquitous
codes, and the IPv6 addressing method.

IEEE Communications Magazine • 27

December 2016
 Proxy VM
Transmitter Fog node
Add-on Add-on
Metadata Metadata Metadata application VM application VM Application layer

Semantic Semantic Semantic

model 1 model 2 model n Metadata Metadata Metadata Metadata

Classified data
streams Proxy Proxy Proxy User data layer
VM VM VM
Classifier

Raw data

Raw data

Raw data
streams

streams

streams
Raw data
streams
Device profile Registered device list
database ...

FIGURE4. The hierarchical fog computing architecture.

aggregate the

Proxy VM 2
SDN-based
cellular core Semantic Semantic
model model
Registered device list
Proxy VM 1 ...
Semantic Semantic Semantic Semantic
model model model model BS 2
Registered device list
...

BS 1

User

FIGURE 5. The illustration of the proxy VM decomposition and migration process.

PROXy VM MOBILITy MANAGEMENT

When a mobile IoT device roams from one BS
to another BS, it should report its new location
(i.e., the mobile IoT device is within the BS’s
coverage) to the mobility management entity
(MME), which is a network management
operator in the Open- Flow control layer,
through the location update procedure. Mobility
management is critical in edgeIoT because
proxy VM decomposition/ composition
processes and proxy VM migration processes
are determined by the locations of the proxy
VMs’ registered IoT devices. The proxy VM
should be aware of the locations of its
registered IoT devices so that it can
communicate with the corresponding IoT
devices via the IoT device’s associated BS.
Adopting the existing mobility management
in the existing LTE network is one solution;
however, it requires each mobile IoT device to
be equipped with a SIM card for identification
and to support the location update protocol
involved in the LTE network, which is not scal-
able and economical. Since most IoT devices
are attached to their users, one alternative is to
establish a local cluster network (e.g., a body
area network) consisting of mobile IoT devic-
es. The user’s mobile phone or other wearable
device acts as a cluster head, which can be
con- sidered as a gateway to report the
locations of the IoT devices in the network,

28 IEEE Communications Magazine •

December 2016
IoT devices’ sensed data streams, and upload the data streams to the
corresponding proxy VM. Note that the cluster head should have the
localization capability to identify its geograph- ic location or its
associated BS’s ID by apply- ing existing wireless localization
technologies, such as WiFi-based localization, LTE mobility
management, and Bluetooth Low Energy (BLE) beacon-based
localization. The location of the cluster head represents the locations of
all the members in the local cluster network.
IOT DEVICES MIGRATION MANAGEMENT
As mentioned earlier, the IoT device’s proxy VM can be decomposed
and migrated among the fog nodes in order to minimize the latency for
uploading the sensed data streams from the IoT devices as well as
reduce the traffic load of the SDN-based cellular core. It is not necessary
to migrate the IoT device’s proxy VM whenever the IoT device roams
into a new BS’s coverage area, that is, some proxy VM migrations
cannot reduce the latency but increase the traffic load of the core
network. For instance, as shown in Fig. 5, a user’s mobile IoT devices
roam from BS 1 to BS 2, and thus their proxy VM (denoted as proxy VM
2) is decomposed from the original proxy VM (denoted as proxy VM 1)
and migrates to fog node 2. If migrating proxy VM 2 from fog node 1 to
fog node 2 takes T units of time (note that before the migration process
is completed, the mobile IoT devices still need to upload their raw data
streams to proxy VM 1 via the SDN- based cellular core) and the mobile
IoT devices move out of the coverage area of BS 2 before the migration
process is completed, such migration is obviously inappropriate because
it increases the traffic load of the SDN-based cellular core (i.e., all the
raw data streams generated from the user’s mobile IoT devices should
still traverse the SDN-based cellular core; in addition, extra traffic is
introduced by migration) without improving the E2E delay between a
user’s mobile IoT devices and their proxy VM.
It is thus necessary to estimate the profit for
migrating the proxy VM among the fog nodes whenever the user’s
mobile IoT devices roam to a new BS. The migration profit, denoted as
p, is

IEEE Communications Magazine • 29

December 2016
 1.0
4000
Static
Dynamic(M = 2 0.9
3500 GB) Dynamic(M =
Total traffic in the SDN-based cellular core (TB)
4 GB) Dynamic(M 0.8
= 8 GB)
3000 0.7

Percentage of UE
2500 0.6

0.5
2000
0.4
1500
0.3

1000 0.2
Cumulative occurancy probability >
500 0.1 90% Cumulative occurancy probability
> 95% Cumulative occurancy
0.0 probability > 99%
1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
rdata (Mb/s) Number of locations
(a)
(b)

FIGURE 6. Simulation results: a) total traffic in the SDN-based cellular core vs. the average data rate of mobile IoT devices (given  = 500
kb); b) the statistical results of the user mobility trace.

defined as the total SDN-based core network traffic

reduction with and without migrating the proxy thus the value of TBS is predictable. Meanwhile,
VM whenever the user’s mobile IoT devices the values of rmig and rdata can also be estimat-
roam into a new BS: p = Lstatic – Lmig, where ed based on their historical traces. Therefore,
Lmig and Lstatic are the total traffic amounts the value of
generated in the SDN- based core network for T BSrdata  
migrating and nt migrating, respectively. Lmig .
comprises two parts: the migra- tion traffic and rmig  rdata
the total data streams transmitted between the can be reliably estimated. In order to evaluate
proxy VM and its registered IoT devic- es during the migration according to Eq. 1, the migration
the migration process:4 Lmig = Tmig (rmig time Tmig should also be predicted.
+ rdata), where Tmig is the total migration time, Normally, the proxy VM migration process
rmig is the average bandwidth provisioning for comprises many iterations. In the first iteration,
migra- tion, and rdata is the average data rate for all the memory of the source proxy VM is
transmit- ting the data streams between the migrat- ed to the destination. Since the source
user’s mobile IoT devices and their proxy VM. proxy VM is still serving the user’s IoT devices,
Meanwhile, Lstatic contributes to the total data the con- tent of the memory may change during
streams transmitted between the proxy VM and the first iteration. Thus, in the second iteration,
its registered mobile IoT devices when the the dirty memory pages, which are generated in
mobile IoT devices remain in the new BS, that is, the first iteration, will be transmitted to the
destination. The iteration is repeated until the
Lstatic = TBSrdata, where TBS is the retention time
dirty memo- ry pages, which are generated in
of the mobile IoT devices remaining in the new
the previous iteration, are less than the
BS.
predefined threshold, denoted as . Then the
Apparently, an appropriate proxy VM migra-
tion implies that the estimated migration profit is source proxy VM stops serving its IoT devices
larger than a predefined value , that is, Lstatic – and transmits the rest of the dirty memory
pages to the destination; final- ly, the
Lmig > , where   0. Thus, we can derive
destination proxy VM resumes to serve its IoT
devices. Thus, the migration time should be a
T BSrdata  T mig rmig  rdata    function of the average data rate for doing the
migration rmig, the average dirty memory pages
 T mig T BSrdata   (1)
 generation rate rdir, the initial proxy VM memory
rmig  rdata size M, and the threshold value , that is, Tmig =
Equation 1 indicates that the migration can predict- ed based on mathematical models. f(rmig, rdir, M, ).
benefit the network only if the migration time Mobile IoT devices are usually attached to their Based on the model
Tmig is less than users, and proposed by [13], the
migration time can be
T BSrdata   reliably esti- mated
.
rmig  rdata given the average
Due to the fact that about 10 to 30 percent of all transmission data rate
human movements are attributed to their social for doing migration.
relationships, and 50 to 70 percent to periodic In order to
behaviors [12], we believe that the dynamics of investigate how the
future human movements can be reliably proxy VM migration

30 IEEE Communications Magazine •

December 2016
affects the total traffic in the core net- work, we
evaluate the total traffic in the cellular core
network during the day by applying the dynamic
proxy VM migration compared to the static
proxy VM deployment (i.e., each proxy VM
4 After
does not migrate among fog nodes after its migration is complet- ed, the proxy VM is placed in
initial deployment). In order to emulate each the fog node, with a connect- ed BS that serves the mobile
IoT devices, and hence the data streams generated by the
user’s behavior, we have obtained data trac- mobile IoT devices no longer traverse the SDN- based core
network to reach the proxy VM.

IEEE Communications Magazine • 31

December 2016
 es of more than 13,000 users and extracted providers [14].
The proposed edgeIoT their mobility in one day in Heilongjiang Prov-
ince, China. The whole area contains 5962 BSs
architecture can sub- (each BS is connected with a fog node), and
stantially reduce the each user’s location (i.e., a user within the BS’s
traffic load in the core coverage area) is monitored for every minute
during the day. Meanwhile, the SDN-based cel-
network and the E2E lular core network can guarantee the average
delay between IoT transmission rate for doing migration to be 20
devices and computing Mb/s (i.e., rmig = 20 Mb/s). Each user’s mobile
IoT devices is attached to its own user and
resources as compared
generates the data streams over time with the
to the traditional IoT same average data rate rdata. Figure 6a shows
architecture, and thus the total traffic in the cellular core network
facilitate the IoT services during the day by varying the average data rate
for transmitting the data streams between the
provisioning. user’s mobile IoT devices and their proxy VM.
Clearly, applying dynamic proxy VM migration
can reduce more traffic in the SDN-based cel-
lular core compared to the static proxy VM
deployment when rdata increases. However, as
the total amount of memory of each proxy VM
(i.e., M) increases, the total traffic in the core
network significantly increases accordingly.
This is because as the value of M increases,
the migration time becomes longer (i.e., more
traf- fic would be generated for migration), and
thus it is preferable for more proxy VMs to stay
in their original fog nodes in order to avoid a
huge volume of migrating traffic.
One solution to alleviate the traffic load of
the core network (when the value of M is large)
is to pre-allocate replicas of the users’ proxy
VMs in the fog nodes. Specifically, the major
part of the memory is the semantic models and
device profiles (which are not dynamically
changed after initial installation) in the proxy
VM. Thus, the replicas of the mobile IoT’s
semantic mod- els can be pre-allocated to the
corresponding fog nodes, the connected BSs of
which are com- monly visited by the user (e.g.,
the user’s home and workplace). Note that we
further analyze the mentioned user’s mobility
trace and find out that each user mainly settles
in some areas cov- ered by a few BSs; as
shown in Fig. 6b, 92.22, 86.93, and 75.65
percent of the users spend 90, 95, and 99
percent of the time during the day (viz., 21.6,
22.8, and 23.76 h) at only four locations,
respectively. This observation helps us
determine the proper number and locations of
replicas for each user’s IoT devices. Thus, if a
proxy VM tries to migrate to another fog node
(which contains one of the proxy VM’s replicas),
rather than transmit the whole memory of the
proxy VM, only the differences (between the
proxy VM migration and its replicas) need to be
transferred, thus dramatically reducing the
migra- tion time as well as the migration traffic.
ENERGY CONSUMPTION CONSIDERATION
Deploying fog nodes at the network edge may
increase the operational cost for processing the
IoT data streams compared to processing them
in the centralized cloud (which provisions effi-
cient and flexible resource and power manage-
ment to minimize the energy consumption of
the cloud). However, introducing green energy
in the proposed edgeIoT architecture can
substantially reduce the operational cost (i.e.,
reduce on-grid energy consumption) for edgeIoT

32 IEEE Communications Magazine •

December 2016
 Specifically, each fog node can be powered by both green energy and
on-grid energy. The fog node would first consume green energy and
then on-grid energy if green energy is not enough to satisfy the
energy demands of the hosting proxy VMs in the fog node. Some fog
nodes, which have less energy demand and more green energy
generated, would have excessive green energy, while some, which
have more energy demands and less green energy generated, would
consume on-grid energy. Thus, proxy VMs can be migrated from the
fog nodes (which consume on-grid ener- gy) to the fog nodes (which
have excessive green energy) in order to further reduce on-grid
energy consumption.
CONCLUSION
This article proposes a new architecture, edgeIoT, in order to efficiently
handle the raw data streams generated from the massive distributed
IoT devic- es at the mobile edge. The proposed edgeIoT architecture
can substantially reduce the traf- fic load in the core network and the
E2E delay between IoT devices and computing resources compared
to the traditional IoT architecture, and thus facilitate IoT services
provisioning. Moreover, this article has raised three challenges in
imple- menting the proposed edgeIoT architecture and has provided
potential solutions.
REFERENCES
[1] H. Sundmaeker et al., “Vision and Challenges for Realising the Internet of Things,” EC
Info. Soc. and Media, tech. rep., Mar. 2010.
[2] Networking, Cisco Visual, “Cisco Global Cloud Index: Fore- cast and Methodology,
2014-2019,” white paper, 2015.
[3] X. Sun and N. Ansari, “Optimizing Resource Utilization of a Data Center,” IEEE
Commun. Surveys & Tutorials, DOI: 10.1109/COMST.2016.25582032016, Early
Access, 2016.
[4] H. L. Truong and S. Dustdar, “Principles for Engineering IoT Cloud Systems,” IEEE
Cloud Computing, vol. 2, no. 2, Mar.-- Apr. 2015, pp. 68–76.
[5] L. Wang and R. Ranjan, “Processing Distributed Internet of Things Data in Clouds,”
IEEE Cloud Computing, vol. 2, no. 1, Jan.--Feb. 2015, pp. 76--80.
[6] F. Bonomi et al., “Fog Computing and Its Role in the Internet of Things,” Proc. 1st Ed.
MCC Wksp. Mobile Cloud Comput- ing, Aug. 13–17, 2012, Helsinki, Finland, pp.
13--16.
[7] X. Jin et al., “Softcell: Scalable and Flexible Cellular Core Network Architecture,”
Proc. 9th ACM Conf. Emerging Net- working Experiments and Technologies,
Santa Barbara, CA, Dec. 09--12, 2013, pp. 163--74.
[8] X. Sun, N. Ansari, and Q. Fan, “Green Energy Aware Avatar Migration Strategy in
Green Cloudlet Networks,” 2015 IEEE 7th Int’l. Conf. Cloud Computing
Technology and Science, Vancouver, BC, Canada, Nov. 30--Dec. 3, 2015, pp. 139–
46.
[9] A. Lara, A. Kolasani, and B. Ramamurthy, “Network Innova- tion using OpenFlow: A
Survey,” IEEE Commun. Surveys & Tutorials, vol. 16, no. 1, 2014, pp. 483--512.
[10]S. Mathur et al., “Parknet: Drive-By Sensing of Road-Side Parking Statistics,” Proc.
8th Int’l. Conf. Mobile Systems, Applications, and Services, San Francisco, CA,
June 15–18, 2010, pp. 123–36.
[11]M. A. Khan et al., “Moitree: A Middleware for Cloud-As- sisted Mobile Distributed
Apps,” 4th IEEE Int’l. Conf. Mobile Cloud Computing, Services, and
Engineering, Oxford, U.K., Mar. 29–Apr.1, 2016, pp. 21–30.
[12]E. Cho, S. A. Myers, and J. Leskovec, “Friendship and Mobility: User Movement in
Location-Based Social Net- works,” Proc. 17th ACM SIGKDD Int’l. Conf.
Knowledge Discovery and Data Mining, San Diego, CA, Aug. 21--24, 2011, pp.
1082--90.
[13]X. Sun and N. Ansari, “PRIMAL: PRofIt Maximization Avatar pLacement for Mobile Edge
Computing,” Proc. IEEE ICC, Kuala Lumpur, Malaysia, May 23–27, 2016.
[14]X. Sun and N. Ansari, “Green Cloudlet Network: A Dis- tributed Green Mobile Cloud
Network,” IEEE Network, to appear; also available in Computing Research
Repository (CoRR), arXiv:1605.07512, 2016.

IEEE Communications Magazine • 33

December 2016
 BIOGRAPHIES a number of Special Issues covering various emerging topics
Xiang Sun [S’13] received his B.E. degree in electronic and in communications and networking. His current research
information engineering and M.E. degree in technology of com- focuses on green communications and networking, cloud
puter applications from Hebei University of Engineering, China. computing, and various aspects of broadband networks. He
He is currently working toward his Ph.D. degree in electrical has served on the Editorial Boards and Advisory Boards of
engineering at the New Jersey Institute of Technology (NJIT), over 10 journals, including as a Senior Technical Editor of
Newark. His research interests include mobile edge computing, IEEE Communications Magazine (2006--2009). He was
big data networking, green edge computing and communica- elected to serve on the IEEE Communications Society
tions, cloud computing, and the Internet of Things. (ComSoc) Board of Governors as a Member-at-Large (2013–
2015). He has chaired ComSoc Tech- nical Committees, and
nirwan anSari [S’78, M’83, SM’94, F’09] (nirwan.ansari@njit. has actively organized numerous IEEE international
edu) received his B.S.E.E. (summa cum laude with a perfect conferences/symposia/workshops, assuming vari- ous
GPA) from NJIT, his M.S.E.E. from the University of leadership roles. He frequently delivers keynote addresses,
Michigan, Ann Arbor, and his Ph.D. from Purdue University, distinguished lectures, tutorials, and invited talks. Some of his
West Lafay- ette, Indiana. He is Distinguished Professor of recognitions include some Best Paper awards, several Excellence
Electrical and Computer Engineering at NJIT, which he joined in Teaching Awards, the Thomas Alva Edison Patent Award
in 1988. He has also assumed various administrative (2010), New Jersey’s Inventors Hall of Fame Inventor of the
positions at NJIT and has been Visiting (Chair) Professor at Year Award (2012), NCE Excellence in Research Award
(2014), Purdue University Outstanding Electrical and
several universities. He has also (co-)authored more than 500
Computer Engi- neer Award (2015), and designation as an
technical papers, over 200 in widely cited refereed
IEEE Communications Society Distinguished Lecturer (2006–
journals/magazines. He has guest edited
2009). He has also been granted over 30 U.S. patents.

34 IEEE Communications Magazine •

December 2016