EDGE COMPUTING FOR THE INTERNET OF THINGS

Consolidate IoT Edge Computing with Lightweight Virtualization

Roberto Morabito, Vittorio Cozzolino, Aaron Yi Ding, Nicklas Beijar, and Jörg Ott

Abstract Clearly, the highly fragmented and heterogeneous

IoT landscape needs to encompass novel and
Lightweight virtualization (LV) technologies have reactive approaches for dealing with these chal-
refashioned the world of software development by lenges.
introducing flexibility and new ways of managing One emerging paradigm, edge computing,
and distributing software. Edge computing com- represents a new trend to improve the overall
plements today’s powerful centralized data centers infrastructure efficiency by delivering low-latency,
with a large number of distributed nodes that pro- bandwidth-efficient, and resilient services to IoT
vide virtualization close to the data source and end users. Although this new approach is not intend-
users. This emerging paradigm offers ubiquitous ed to replace the cloud-based infrastructure, it
processing capabilities on a wide range of hetero- expands the cloud by increasing the computing
geneous hardware characterized by different pro- and storage resources available at the network
cessing power and energy availability. The scope of edge. One typical example is IoT edge offloading
this article is to present an in-depth analysis on the [2], which revisits the conventional cloud-based
requirements of edge computing from the perspec- computation offloading where mobile devices
tive of three selected use cases that are particularly resort to resourceful servers to handle heavy com-
interesting for harnessing the power of the Internet putation [3]. To cater for the demands of new IoT
of Things. We discuss and compare the applicability services, the computation is reversely dispatched
of two LV technologies, containers and unikernels, by the servers to constrained devices deployed at
as platforms for enabling the scalability, security, and the network edge, close to users and data gener-
manageability required by such pervasive applica- ators.
tions that soon may be part of our everyday lives. To By harnessing the power of distributed edge
inspire further research, we identify open problems resources, the IoT edge computing model can
and highlight future directions to serve as a road support novel service scenarios such as autono-
map for both industry and academia. mous vehicles/drones, smart city infrastructure,
and augmented reality (AR). As highlighted in Fig.
Introduction 1, these three representative domains intersect.
Over the last decade, the development of the Edge computing is the link that helps spawn and
Internet of Things (IoT) has been upheld by the promote appealing joint services.
cloud-based infrastructures that aim to cope with Edge computing aims to satisfy key require-
the increasing number of IoT services provided ments such as scalability, multi-tenancy, securi-
by various connected devices. From the initial ty, privacy, and flexibility. In this respect, the fast
design, IoT was conceived as extending the Inter- evolving lightweight virtualization technologies
net with a new class of devices and use cases [1]. seek to fulfill such demands, given their matching
This has obviously generated an intrinsic associ- features. Meanwhile, we still lack comprehensive
ation between IoT and cloud, where the cloud- guidelines to illustrate how we can exploit the full
based network infrastructures are optimized to potential of lightweight virtualization to enhance
support a multitude of IoT-centric operations such edge computing, especially for IoT use cases.
as service management, computation offloading, As a solid step toward realizing the IoT edge
data storage, and offline analysis of data. computing vision, we aim to answer through this
This notion of cloud-connected IoT deploy- article a major question: Can lightweight virtual-
ment assumes that most IoT networks need to ization (LV), in its different flavors, be exploited for
connect to the cloud (e.g., through some gate- empowering edge architectures and be suitable in
way and tunnel approach). However, the increas- a wide range of IoT pervasive environments? Our
ing stringent performance requirements of IoT use case study, comparison analysis, and prospec-
services, especially in terms of latency and band- tive outlook further address the following ques-
width, challenge this deployment. Specifically, the tions:
existing model is not suitable when: • Which LV features can match the increas-
1. IoT networks create data that needs to be ingly strict requirements of IoT services in
accessed and processed locally. constrained environments?
2. Piping everything to the cloud and back is • How can LV and IoT edge scenarios be effi-
not acceptable under delay constraints. ciently utilized together?
3. The amount of data is too large to transfer • Which challenges must be tackled to effec-
to the cloud (in real time) without causing tively exploit the benefits introduced by LV
congestion on the backhaul. in this context?
Digital Object Identifier:
10.1109/MNET.2018.1700175 Roberto Morabito and Nicklas Beijar are with Ericsson; Vittorio Cozzolino, Aaron Yi Ding, and Jörg Ott are with Technical University of Munich.

102 0890-8044/18/$25.00 © 2018 IEEE IEEE Network • January/February 2018

Authorized licensed use limited to: CZECH TECHNICAL UNIVERSITY. Downloaded on November 02,2023 at 09:37:43 UTC from IEEE Xplore. Restrictions apply.
 The remainder of this article is organized as
Smart lamppost
follows. Motivations of the proposed work are infrastructure
presented in the following section. Then we intro-
duce first the requirements that different edge for
IoT cases entail, and the suitability of LV for mit- Autonomous Smart city
igating and satisfying them. We next introduce vehicles
LV technologies and illustrate three specific use
cases. Finally, we unveil the open issues and chal-
lenges before concluding the article.

Motivation Edge
computing
In the context of IoT, edge computing intro-
duces an intermediate layer in the convention-
al IoT-cloud computing model. The envisioned 564619012046291284
564619012046291284
564619012046291284

edge-driven IoT environment consists of three

components: IoT devices, edge layer, and cloud
back-end. Being a central part of the ecosystem,
the edge layer plays the crucial role of bridging Augmented Augmented
windshield smart house
and interfacing the central cloud with IoT. Essen-
tially, an edge element in this layer can be char-
acterized by a small to medium-size computing Augmented
entity that aims to provide extra computing, stor- reality
age, and networking resources to the applications
deployed across IoT devices, edge, and cloud.
FIGURE 1. A subset of use cases and services enabled by IoT edge computing.
Depending on the specific scenario, its functional-
ities can be executed in cellular base stations, IoT
gateways, or, more generally, low-power nodes support, edge elements must adapt to different
and small data centers. These may be owned and types of traffic and to the application needs. The
operated by the user, a cloud provider, or a tele- key is to ensure a virtuous trade-off between
com operator (in mobile edge computing). design requirements, specific performance tar-
The placement of a “middle layer” between gets, and applications manageability spanning the
the end devices and cloud is an architectural con- entire three-tier IoT edge computing architecture.
cept that is widely utilized in common network
infrastructures. Conventional middle layer func- Empowering IoT Edge Computing with LV
tionalities mainly target connectivity, routing, and To fully attain the potential of edge comput-
network-oriented operations. For example, net- ing for IoT, we need to address four concerns:
work functions virtualization (NFV) [4, 5] virtual- abstraction, programmability, interoperability, and
izes typical network elements, such as firewalls, elasticity. In particular, for the three-tier IoT edge
network address translators, switches, and core computing architecture, it is crucial to provide
network components. simple but efficient configuration and instantiation
For IoT ecosystems, edge computing aims to methods that are independent of the technolo-
meet IoT service providers’ demand of owning gies used by different IoT and cloud providers.
a dedicated infrastructure that is independent The tools embedded in the edge layer should
of a given technology or use case. In addition, share common functionalities and exploit com-
it seeks to satisfy the demanding IoT services’ mon application programming interfaces (APIs)
performance requirements. More importantly, for orchestrating interconnections with different
in contrary to the plain middle layer solutions, networking technologies.
the IoT-centric edge computing must entail pro- To help us acquire a synoptic view, we high-
grammability and flexibility to deliver ubiquitous light the dominant requirements of representative
processing capabilities across a wide range of use cases in Table 1, which encompasses scalabil-
heterogeneous hardware. For instance, besides ity, multi-tenancy, privacy and security, latency,
managing an IoT home network, the edge layer and extensibility.
can simultaneously provide image processing for Compared to alternative virtualization solutions
home camera and data pre-processing operations. such as hypervisors, we envision a trend toward
Obviously, the heterogeneous characteristics using LV technologies in IoT edge computing.
of various instances and applications deployed These emerging software solutions can provide
on top of the edge layer will generate unique the needed support in terms of hardware abstrac-
challenges that need to be addressed. From the tion, programmability, interoperability, and elastic-
architectural perspective, this implies that the ity. A direct benefit that emerges from employing
edge layer has to efficiently and mutually cooper- LV in the IoT edge domain is avoiding the strict
ate with both cloud-based services and IoT devic- dependency on any given technology or use case.
es, by acting as a bridge between elements that Within an LV instance, either a container or a
require a distinct way of interacting. unikernel, we can efficiently deploy applications
In this context, it is crucial to equip the edge designed to manage and use extremely different
layer with tools that allow a flexible, well perform- technologies. In addition, equipping edge ele-
ing, and automated way of efficient services pro- ments with newer services will be made easier
visioning. Hence, edge elements have to embed since we only need to configure and instantiate
service provisioning methods that are independent standalone virtualized applications. This feature
of the managed applications and communication avoids complex reprogramming and updating
patterns. Furthermore, by means of cross-layer operations that are part of the software life cycle

IEEE Network • January/February 2018 103

Authorized licensed use limited to: CZECH TECHNICAL UNIVERSITY. Downloaded on November 02,2023 at 09:37:43 UTC from IEEE Xplore. Restrictions apply.
 Requirements
Scenarios
Scalability Multi-tenancy Privacy and security Latency Extensibility (open API)

Critical. Autonomous vehicles possess Non-critical. Each car

sensitive information about the user. manufacturer will probably
Autonomus Critical. Cars have strict
Non-critical Non-critical Moreover, the constant need of sensors run exclusively their own
vehicles real-time requirements
data for navigation make cars a primary software to ensure security
target for malicious users. and reliability.

Critical. AR applications
Critical. Open API are import-
Augmented Critical when processing sensitive require real-time information
Critical Critical ant in this case to enable new
reality multimedia streams. feed to ensure a smooth and
services and features.
acceptable experience.

Critical to enable the creation

Depends on the specific Smart context Depends. For example, in
of an “IoT Marketplace” where
Smart sensors Critical due to the number (for smart health it is critical but not for the case of Machine Type
Critical developers can offer new
networks of potential users. smart environment). Strict control over Communications (MTC) it’s
and innovative application
which data can be public is required. critical.
exploiting collected data.

Critical especially for messag-

Non-critical. Data and Non-critical. The infrastruc- Critical. Disclosure and analysis of es as Phasor Measurement
Smart grid messages are exchanged at a ture is usually controlled energy consumption information can Unit (PMU) or Advanced Non-critical
fixed, predefined rate. by a single provider. lead to user profiling and tracking. Metering Infrastructure
(AMI).

Critical as multiple
Critical. IoT healthcare Depends. It is critical in
healthcare organizations Critical. IoT-edge medical devices deal Critical to support new
networks must be able to use-cases such as remote
and/or heterogeneous with personal heath data, which need application able to offer a
E-health meet the growing demand of surgery. Nevertheless,
IoT medical devices could to be securely stored. Integrity, privacy, more accurate patients health
services from both individuals response time can be accept-
share the same network and confidentiality must be kept. condition monitoring.
and health organizations. able in other scenarios.
infrastructure.

Critical. Several control units

Non-critical. A single Critical to promptly identify
Distributed are needed in order to grant Critical considering the sensitive Non-critical. Same as Autono-
provider usually controls suspects or recognize on-go-
surveillance the system of better usability information handled. mous vehicles.
the infrastructure. ing crimes.
and robustness.

Critical. Users share large amount of

Critical. A big data analytics personal data and sensitive content
system must be able to Critical. A single big data through their personal devices towards
support very large datasets. system has to be able to applications (e.g., social networks)
Big data Critical to improve and deploy
All the components must be co-locate different use and public clouds. Equipping big data Non-critical
analytics different algorithms and tools.
scalable to accommodate the cases, applications, or systems of secure frameworks capable
constantly growing amount data sets. to store and manage user data with
of data to be handled. high sensitiveness represents a critical
aspect.

Critical. NFV needs to

Critical. Resources are Critical. The use of additional software
leverage real-time delivery
Network Critical. Demand of new ser- shared among customers. (e.g., hypervisors, containers or
services. NFV introduces
functions virtual- vices is high and constantly A large number of multi- unikernels) extends the chain of trust. Non-critical
additional sources of latency
ization (NFV) growing. tenant networks run over a Resource pooling and multi-tenancy
through the virtualization
physical network. bring further security/privacy threats.
layer.

TABLE 1. Example of edge-IoT scenarios requirements.

management. Through LV, such complexity is cir- to “speak the same language.” As suggested in
cumvented because updating a particular service [2], using the same LV instance will enable us
requires changes only within a specific virtualized to efficiently run them both at the edge and in
instance. the cloud, hence achieving a decentralized IoT
To foster integration with the cloud, LV can edge service provisioning architecture. This con-
also enable cross-platform deployment, allowing sequently meets the strict performance require-
a common execution environment across cloud, ments of demanding IoT scenarios, and further
edge elements, and even constrained IoT devices. ensures the crucial requirement of multi-tenancy.
The cross-platform deployment benefit introduced We also note that there are scenarios where
by LV further allows both cloud and edge, regard- virtualization technology is not a suitable option
less of their computational hardware capability, for manifold reasons. In general, virtualization

104 IEEE Network • January/February 2018

Authorized licensed use limited to: CZECH TECHNICAL UNIVERSITY. Downloaded on November 02,2023 at 09:37:43 UTC from IEEE Xplore. Restrictions apply.
 LV technique Property

Instantiation time Image size Memory footprint Programming language Hardware portability Live migration
dependency support
Virtual machine ~5/10 secs ~1000 MBs ~100 MBs No High Yes
• KVM
• QEMU
Container ~800/100 msecs ~50 MBs ~5 MBs No High No
• Docker (http://www.docker.com/)
• rkt (https://coreos.com/rkt)
• Open VZ https://openvz.org/)
• LXC https://linuxcontainers.org/
Unikernel ~< 50 msecs ~< 5 MBs (bundle) ~ 8 MBs Yes (i.e., MirageOS High No. Requires
• MirageOS (https://mirage.io/) unikernels can only be manual
• HaLVM (http://galois.com/project/) written in OCaml) implementation
• IncludeOS (www.includeos.org)
• ClickOS (http://cnp.neclab.eu/clickos/)
• OSv (osv.io)
(a)

App 1 App 2
••• App n App 1 App 2
••• App n App 1 App 2
••• App n

LIBS LIBS LIBS LIBS LIBS LIBS LIBS

BIN LIBS | BIN | RUNTIME BIN BIN BIN & & &
RUNTIME RUNTIME RUNTIME ••• RUNTIME RUNTIME RUNTIME ••• RUNTIME
Guest OS Guest OS Container Container Container Library OS Library OS Library OS

Hypervisor Container engine Hypervisor

Host OS Host OS Host OS

Hardware infrastructure Hardware infrastructure Hardware infrastructure

Virtual machines Containers Unikernels

(b)

FIGURE 2. LV techniques comparison: a) quantitative analysis; b) core architectural differences.

entails additional delay and resource utilization, tual device drivers — thus supporting standalone
which can be challenging for certain real-time or virtual machines (VMs) that are independent and
mission-critical tasks that demand low and pre- isolated from the underlying host system. In each
dictable latency. Moreover, there are fundamental VM instance, a full operating system (OS) is typi-
hardware requirements to run a virtualized envi- cally installed on top of the virtualized hardware,
ronment (e.g., a CPU with specific architectural thus generating large VM images. Furthermore,
features) that are not easily found on low-end IoT the emulation of virtual hardware devices and
and edge devices. related drivers produces non-negligible perfor-
mance overhead.
Overview of Lightweight Virtualization Differently, containers implement process iso-
System virtualization has drastically evolved in lation at the OS level, thus avoiding the virtual-
the last few years, offering system architects and ization of hardware and drivers [6]. Specifically,
developers a plethora of tools to exploit. There- containers share the same OS kernel with the
fore, understanding how and when to utilize a underlying host machine, meanwhile making it
specific technology based on the hardware con- possible to isolate standalone applications that
straints and applicative requirements is a crucial own independent characteristics: independent
step of the system design phase. Shifting our virtual network interfaces, independent process
focus to edge computing and IoT, we identify two space, and separate file systems. This shared ker-
main candidates that could address the challenges nel feature allows containers to achieve a high-
unique to this domain: containers and unikernels. er density of virtualized instances on a single
Figure 2 presents both quantitative metrics and machine thanks to the reduced image volume.
architectural differences between the aforemen- Containers have achieved much more rele-
tioned technologies, highlighting their main char- vance and practical use recently with the advent of
acteristics. Docker, a high-level platform that has made con-
tainers very popular in a short timeframe. Docker
Container -Based Virtualization : Docker introduces an underlying container engine, togeth-
Container-based virtualization provides a differ- er with a practical and versatile API, which allows
ent level of abstraction in terms of virtualization easily building, running, managing, and removing
and isolation compared to other virtualization containerized applications. A Docker container,
solutions. In particular, containers can be con- which is a runnable instance of a Docker image,
sidered as one of the lightweight alternatives to uses a base image stored in specific private or pub-
hypervisor-based virtualization. Conventional lic registries. Docker uses an overlay file system
hypervisor-based virtualization has been the de (UnionFS) to add a read-write layer on top of the
facto technology used during the last decade for image. UnionFS allows Docker images to be stored
implementing server virtualization and isolation. as a series of layers, consequently saving disk
Hypervisors operate at the hardware level — that space. In fact, the different image layers can be
is, building customizable virtual hardware and vir- cached in the disk, allowing the building process to

IEEE Network • January/February 2018 105

Authorized licensed use limited to: CZECH TECHNICAL UNIVERSITY. Downloaded on November 02,2023 at 09:37:43 UTC from IEEE Xplore. Restrictions apply.
 V2E communication A car issues the activation
of a “data mining” task to
another vehicle. This
LV application instantiation could speed up the
recognizing of a fugitive
Cloud computing Launch of in a certain geographic
services Confirmation of instance virtualized area. Before sending
activation instance data to the cloud,
Execution of vehicles can do some
n virtualized initial filtering and
C tio correlation of images.
V2 nica Instance response instance
m u
m
co (b)

V2E V2E communication The car manufacturer exploits an

com edge base-station for issuing an
mu important OBU firmware update
nic LV instance scheduled on a to an approaching car.
V2E atio
com n remote vehicle through LV orchestration tools execute the
mun container orchestrator
V2V icati management operations of
on LV instance activation (still, containerized instances.
communication through container orchestrator)
Launch of
Confirmation of instance virtualized
activation instance
Execution of
virtualized
Instance response instance
(a)
(c)

Despite being characterized by limited

processing resource capabilities, the OBU
encompasses the allocation of lightweight
virtualized instances.
Container-based OBU platform Applications can be tagged by different priority
levels:
• High (e.g., important OBU firmware update)
High Medium Low • Medium (e.g., driver assistance)
Sensor A • Low (e.g., multimedia contents)
Firmware Cameras Multimedia
The CS schedules the execution of virtualized
Sensor B Service 1 applications by following specific allocation
policies. This task is supported by the RM,
which constantly monitors the resources
Container scheduler (CS) employed by all the running software
instances.
Resource monitoring (RM) The CS can efficiently deal with the limited
computation capabilities of the OBU by
opportunistically scheduling the running
applications according to the OBU available
In-car OBUs container-based orchestration hardware resources and the priority of the
incoming scheduled application.
OBU x OBU y
In-vehicle networks include several OBUs. The
Sensor C flexibility introduced by containers can help in
A B C C improving the resource allocation
Service 3 management of in-vehicle networks. For
example, the use of a container orchestrator
D E F G F engine such as Docker Swarm can be the
Service 2 enabling tool for allocating each container to
the most suitable OBU.
CS RM CS RM

(d)

FIGURE 3. a) Vehicular edge computing scenario in its entirety, vehicular-to-edge (V2E) interactions examples; b) car-to-car V2E commu-
nication; c) base station-to-car V2E communication; d) container-based virtualization is used for easier OBU customization. Further-
more, within the same vehicle orchestration tools are exploited for task offloading among different OBUs.

be sped up, and reusing the same cached layer for emerged from the observation that most applica-
building different images. tions running in the cloud do not require many of
The lightweight features embedded in con- the services to come with common operating sys-
tainers ease the integration of such technology tems. Additionally, unikernels provide increased
in various networking fields. Specific to IoT edge security through a reduced attack surface and
computing, containers can enable us to efficiently better performance by dropping unnecessary
run containerized applications even in devices components from the applications.
characterized by lower processing capabilities, Unikernels were designed initially with the
such as single-board computers [7]. cloud in mind, but their small footprint and flexi-
bility make them also fit well with the upcoming
Library Operating Systems : Unikernels IoT edge ecosystem as illustrated through differ-
Unikernels are single-purpose appliances that ent research attempts [2–9]. The main differences
are specialized at compile time into standalone among existing unikernel implementations sprout
kernels [8] and sealed against modification after from the underlying programming language in
deployment. The concept of unikernels has use. MirageOS [8] and HaLVM are unikernels

106 IEEE Network • January/February 2018

Authorized licensed use limited to: CZECH TECHNICAL UNIVERSITY. Downloaded on November 02,2023 at 09:37:43 UTC from IEEE Xplore. Restrictions apply.
 based on functional languages with pervasive
type-safety in the running code. Other solutions There is no uniform operating layer in a unikernel, and everything is directly compiled into the
like IncludeOS and ClickOS are C++ unikernels; application layer. Therefore, each unikernel may have a different set of vulnerabilities, which implies
the former offering a C++ platform to which to
bind generic applications, while the latter is highly that an exploit that can penetrate one may not threaten the others.
specialized in offering dynamic network process-
ing (based on the Click modular router). OSv is
based on Java and therefore heavier than the oth- offers high flexibility in the platform’s software
ers, but more flexible. management, and allows overcoming the com-
Security and unikernels are tightly coupled. plex software updating procedures required by
The attack surface of a unikernel is strictly con- OBUs [11]. Through a conventional VM, car
fined to the application embedded within. There manufacturers can access all controller area net-
is no uniform operating layer in a unikernel, and work (CAN) bus sensors through OBU and dash-
everything is directly compiled into the applica- cam. However, given that OBUs are embedded
tion layer. Therefore, each unikernel may have a systems with limited computational resources,
different set of vulnerabilities, which implies that LV’s lightweight features avoid the performance
an exploit that can penetrate one may not be overhead and allow scaling up/down the run-
threatening to others. Unikernels are principal- ning applications according to specific priorities.
ly designed to be stateless. Therefore, they are Furthermore, by taking into consideration that
perfect to embed general algorithms (e.g., com- several OBUs can be distributed within a car,
pression, encryption, data aggregation functions) virtualization orchestration tools can be used for
or NFV. OBUs’ task outsourcing, still following specif-
ic OBU resource management policies. More
Use Case Scenarios detail on the usage of LV for in-car platforms can
In this section, we present three use cases match- be found in Fig. 3d.
ing the scenarios presented in Fig. 1. Additionally,
we illustrate the reasons for adopting a specific LV Edge Computing for Smart City
technology for each case. In the context of smart city, the measurement of
environmental data has become an important
Toward Vehicular Edge Computing issue, especially for highly crowded urban areas.
The importance of virtualization in vehicular Currently, air pollution monitoring is achieved
scenarios has been widely acknowledged in with sparse deployment of stationary, expensive
the past. Vehicular cloud computing (VCC) measurement units embedding both sensors1 and
represents an efficient architectural model in computing units. Air pollution is predicted based
supporting the Internet of Vehicles (IoV) [10]. on the measured data in combination with com-
However, we envision the need to establish plex mathematical models [12]. Since the cost of
a vehicular edge computing (VEC) paradigm, deploying and maintaining such pollution stations
which will play a crucial role in future devel- is often prohibitive, we envision crowdsensing as
opment of more efficient vehicle-to-everything a tangible solution that combines LV and edge
(V2X) systems. VEC can cope with the increas- computing.
ingly strict requirements of V2X applications, Edge computing offers resources close to the
and will rely on the growing processing capa- crowdsensing entities, which can offload their
bilities that the different actors in IoV encom- collected data through direct connection with-
pass, including cars’ onboard units (OBUs), out using a mobile connection. LV allows part
edge elements (EEs), and cloud services. In VEC of the required mathematical computation to be
environments, various units can play the role of offloaded onto, and distributed and executed by,
EE. Base stations, IoT gateways, and other vehi- EEs without worrying about compatibility issues.
cles themselves can operate as EE by executing For instance, multiple LV images can be created
specific tasks, including lightweight data mining on demand, each containing only the code nec-
operations, generic offloading processing, dash- essary to process the data of a single sensor. The
cam image filtering, and so on. In such con- partial results will then be subsequently uploaded
text, LV can enable the VEC paradigm and be to a more powerful edge device (e.g., edge data
exploited in multiple scenarios, spanning from center) to be merged. Figure 4 provides more
an efficient and flexible customization of cars’ detail regarding how unikernels can both sup-
OBUs to vehicle-to-edge (V2E) interactions. port the execution of specific algorithms related
Figure 3 depicts the VEC scenario in its entirety to air pollution control and provide pre-process-
(Fig. 3a), together with practical examples of the ing of input data for simulations running in the
way in which LV can be employed in V2E inter- cloud.
actions (Figs. 3b and 3c) and distributed in-car The described approach can reduce the load
platforms (Fig. 3d). on the core network, end-to-end latency, and also
V2E Interactions: Different from already well the cloud (and air pollution stations) provision-
established vehicle-to-vehicle (V2V) communica- ing costs. Regarding specific LV technology, we
tion, V2E aims to encompass computation offload- consider unikernels a promising candidate. The 1 Usually gas detection
ing, task outsourcing, and software management algorithms used to assess air pollution levels are sensors (NO, NOx, O3, CO,
operations. In practice, an LV-enabled OBU can generally static and stateless. In other words, they CO2, and particulate matter)
execute a specific task issued by another vehicle can be considered as blackboxes with a defined plus humidity, rain detection,
and wind speed/direction.
or any other EEs, and vice versa, as shown in the range of inputs/outputs. In case of necessity, the
two examples shown in Figs. 3b and 3c. algorithm can be simply changed by replacing it 2 Unikernels are, by design,
In-Car Platforms: Container-based virtual- with a new unikernel instance without incurring a much smaller than other vir-
ization can be used for OBU customization. It long network transfer time.2 tualization techniques.

IEEE Network • January/February 2018 107

Authorized licensed use limited to: CZECH TECHNICAL UNIVERSITY. Downloaded on November 02,2023 at 09:37:43 UTC from IEEE Xplore. Restrictions apply.
 Air pollution server
The air pollution server (data center) selects
B Code A Code C and offloads tasks toward the edge stations.
EXTRACTION Part of the complete application logic running
A Conf A Conf C in the cloud is selected, extracted, and
recompiled in the shape of a unikernel. Each
C MirageOS MirageOS functionality is embedded in a different
unikernel (MirageOS).

Unikernal
Data
Each MirageOS unikernel runs inside the Xen
Hypervisor as a paravirtualized machine
(PVM). In this case, it embeds the logic to
MirageOS MirageOS MirageOS calculate the AQI (or part of it). Additionally,
there is an orchestration layer that monitors
MirageOS MirageOS MirageOS the execution of the unikernels and
communicates with the cloud.
MirageOS MirageOS MirageOS
Xen Xen Xen

Orchestrator Orchestrator Orchestrator Edge devices are connected to specific subsets

of sensor nodes, geographically distributed.
Therefore, from each edge device we can
gather insights about the conditions of a
specific area of the city.

Air pollution stations are sensor arrays

collecting specific information about pollution
Control

Control

Control
(e.g., different pollutant concentrations).
Data

Data

Data
Additionally, we consider the presence of
mobile units able to collect additional data.
Moreover, weather stations could provide
Air pollution station Air pollution station Air pollution station specific information (e.g., humidity of rainfall
Mobile unit Mobile unit Mobile unit amount in mullimeters) to help calculate a
Weather station Weather station Weather station much better estimation of the air pollution in
a specific area.
NO NOx CO CO2 RAIN NOx O3 HUM PM NOx SO2 CO2

FIGURE 4. Air pollution scenario. An offloadable task is air quality index (AQI) calculation, a number used by
government agencies to communicate to the public how polluted the air currently is or how polluted it
is forecast to become. Calculation of the AQI can be executed locally by edge nodes enhancing real-
time monitoring.

brain can identify faces in a dark scene and the

The core features of a wearable/mobile device are light weight, comfort, design, and battery life. requirements of a virtual reality application to
CPU speed, memory, and system capabilities are only secondary, contrary to what is required by the achieve perpetual stability [13, 14]. Longer delays
in such highly interactive and multimedia-based
PC market. Therefore, it is not surprising that overall, wearable/mobile devices are not designed to run applications will lower the end users’ experience.
computationally intensive tasks. A use case where there is strong interplay
between local computational resources and AR
(or, broadly speaking, computer vision) is aug-
mented windshields for autonomous vehicles. The
Augmented Reality driver, at this point passive, might shift his/her
Wearable devices are typically resource-con- attention completely on the windscreen instead of
strained compared to the computer hardware checking the console to search for speed informa-
of the same vintage PC. The core features of a tion. Additionally, the windshield will also provide
wearable/mobile device are light weight, comfort, traffic condition information, a personal agenda,
design, and battery life. CPU speed, memory, and a news feed, gaming interfaces, social networks,
system capabilities are only secondary, contrary and so forth. In order to craft and manage such a
to what is required by the PC market. Therefore, visually rich experience, an edge board mounted
it is not surprising that overall, wearable/mobile on the car is considered necessary.
devices are not designed to run computationally Therefore, with the support of edge computing
intensive tasks. and LV, we have the possibility to offload expen-
A common approach to solve the problem is sive image processing tasks to EEs in proximity
offloading AR tasks to cloud services in order to instead of resorting to cloud back-ends. There-
reduce the power consumption on the device fore, we can limit the latency impact, assuming
and eventually cope with insufficient mobile pro- that the computation time is device-invariant. The
cessing. The drawback is that using cloud service use of virtualization in such a context is addition-
will introduce additional latency, which is crucial ally motivated by the following factors: multi-ten-
for real-time applications. This is especially import- ancy (i.e., multiple users executing multiple tasks)
ant for AR applications, where responsiveness and task isolation for privacy.
and user immersion are paramount. Humans are For this specific use case, a combination of
extremely sensitive to delays affecting real-time Docker and Unikernel represents a potential
interactions (e.g., a phone call). Different stud- approach, as shown in Fig. 5. A Docker image
ies have revealed the speed at which the human containing multiple unikernels can be composed

108 IEEE Network • January/February 2018

Authorized licensed use limited to: CZECH TECHNICAL UNIVERSITY. Downloaded on November 02,2023 at 09:37:43 UTC from IEEE Xplore. Restrictions apply.
 Service
provider

Part of the service provider application logic is offloaded

as a wrapper composed of a container embedding a
collection of unikernels. In the figure, a unikernel is
represented by the “f” box.

Each unikernel represents a single-purpose task. In the

case of augmented reality, such tasks could be image
Container processing, template matching, object recognition, and
so on. In our execution model, we require such tasks to
Matching phase Pipelining phase Execution phase be non-cross-correlated in order to build a loop-free
execution pipeline.
f f f f
f f f
f f f f 1. Matching phase. This is the phase during which the
orchestrator identifies a group of unikernels (each
f f f f f f f representing a specific function) to be pipelined.
2. Pipelining phase. In this phase, the selected
f f f f unikernels are opportunely pipelined based on their
f f f function.
f f f f 3. Execution phase. The pipelline is started and, after
termination, the result is ready to be sent to devices in
proiximity and to the cloud.
Orchestration and control

The orchestration and control layer takes care of

monitoring and organizing execution and pipelining of
the unikernels.

Each edge device hosts the same environment, but the

collection of unikernels is different based on the
capabilities (I/O interfaces, CPU, RAM) and available
data possessed by the hosting device.

(a) (b) (c)

FIGURE 5. a) A biker receiving personalized advertisements rendered in augmented reality on his/her smart
glasses; b) a smart car populating its augmented windshield with contextualized, live feed information;
c) an augmented smart home, where we control IoT devices in proximity through virtual interfaces.

and shipped, each one representing a different

AR stage/task. Therefore, the Docker image can Controlling the network traffic requires the cloud and edge to be orchestrated, an increasingly challeng-
offer the orchestration and control API to external ing task with manifold EEs deployed. Hence, it is crucial to deploy lightweight orchestration modules that
applications, while under the hood, unikernels
would take care of running the required compu- do not overburden the EE, and to seek a fair balance between synchronization and network load.
tations.

Open Issues and Challenges move in relation to the edge processing device
In this section, we discuss the technical challeng- providing the service. Therefore, the service may
es for integrating LV into IoT edge computing need to be redeployed multiple times at different
and further identify open directions for future locations to transparently serve mobile users. In
research. particular, if the service is specific to an individ-
ual user, the number of transfers may be high.
O rchestration and Monitoring Destroying and redeploying is preferred instead
Orchestration of edge elements (EEs) and cloud of moving the service together with its running
architectures brings several challenges. Edge-IoT state. For cloud-native service the general rec-
scenarios require specific tools to deal with the ommendation is to avoid storing state locally or
different processor architectures and storage only to use disposable state. For services requiring
capacity of EEs and cloud services. Controlling local state, the service must store the current state
the network traffic requires the cloud and edge to at an external stable location before exiting and
be orchestrated, an increasingly challenging task load it again on restart. Particular attention must
with manifold EEs deployed. Hence, it becomes be paid to ensure that the new edge node has
crucial to deploy lightweight orchestration mod- available resources for serving the new device,
ules that do not overburden the EE, and to seek and the platform may provide alternative nodes
a fair balance between synchronization and net- or prioritization among services in the case of
work load. Other key aspects concern the defi- overallocation.
nition of optimized policies for efficient vertical Regarding monitoring solutions, both tech-
scaling, in which applications are automatically nologies need high-performing, lightweight, and
prioritized and scaled up/down between EE and scalable monitoring frameworks. This require-
cloud, according to specific QoS requirements ment is strictly related to the fact that these tools
or computing resource saturation of EEs. Mobil- may need to run on EEs characterized by lower
ity is also a relevant aspect. User devices might resource computation capabilities. Another key

IEEE Network • January/February 2018 109

Authorized licensed use limited to: CZECH TECHNICAL UNIVERSITY. Downloaded on November 02,2023 at 09:37:43 UTC from IEEE Xplore. Restrictions apply.
 implementations do not provide full support for
The presence of multiple industry partners and researchers working in this field has given birth to different live migration. Specific frameworks that support
ramifications and interpretations of the same paradigm. Without standards and regulations, merging differ- proactive service migrations for stateless applica-
tions have already been proposed [15]. However,
ent approaches will be a nontrivial task exacerbated by the heterogeneity of the involved technologies. support for stateful applications’ migration need
to be integrated soon for fully exploiting LV bene-
fits in these scenarios.
requirement for monitoring engines is the possi-
bility to track, in real time, the individual resourc- Management Frameworks and Application Portability
es of each virtualized instance. Implementation Employment of container technologies have had
of such frameworks becomes, in parallel with a disruptive rise in the last years, and the enor-
orchestration mechanisms, crucial in resource mous effort that open source communities have
optimization and in developing efficient edge- provided on continually improving full-featured
cloud instance-placement algorithms and policies. management frameworks has paid off. Unikernels
seem to still not be mature enough to be included
Security and Privacy in production-ready environments, and greater
In the analyzed domain, one challenge is the effort is required for featuring the same portabil-
certification of virtualized applications. We need ity of containers. Packaging applications through
to guarantee their authenticity and validity, by unikernels may require an implementation effort
including a signing and validation infrastructure to that somehow slows down, and in some cases
discriminate legit from tampered instances. With- limits, the adaptability toward existing software
out such mechanisms, there is the concrete risk and hardware platforms. This difference comes
of executing malicious code and infringing on the from the different way in which the two technolo-
security requirements. It is crucial to encourage gies are built. Containers are application-agnostic,
the development of lightweight security mech- while unikernels are limited by the programming
anisms that take into account the strict require- language and libraries exposed by the underlying
ments of IoT applications/scenarios and do not minimalistic OS.
impair the lightweight features of the analyzed vir-
tualization technologies, preserving their capacity Data Storage
to not generate performance overhead. From the Containers and, in particular, unikernels are
privacy perspective, EEs may be shared between not suitable for storing persistent data, such as
multiple tenants. It is crucial to be able to isolate data collected from IoT sensors. Moreover, stor-
tenants’ data, but also control the use of tenants’ ing important data on edge nodes can be risky
dedicated resources — for example, CPU and because of both the volatile nature of edge nodes
memory. Finally, sharing data between tenants at and the security risks related to easier physical
the EE level, without going through the cloud-in- exposure of the nodes. Therefore, data typical-
frastructure, requires the definition of EE policies ly need to be stored in centralized nodes and
and specific access control mechanisms. retrieved on demand. This may reduce the fea-
sibility of LV -based edge computation in very
Standards and Regulations data-intensive applications. Moreover, some appli-
IoT and EC are developing faster than standards cations requiring nodes to access data of all other
and regulations. The presence of multiple industry nodes’ data (e.g., for distributed analytics) may be
partners and researchers working in this field has infeasible to distribute. Automatically optimizing
given birth to different ramifications and interpre- the data storage location of distributed applica-
tations of the same paradigm. Without standards tions is a topic requiring further research. On the
and regulations, merging different approaches will other hand, many IoT applications use volatile
be a nontrivial task exacerbated by the heteroge- data locally, while persistent data can be mini-
neity of the involved technologies. For LV technol- mized and stored centrally.
ogies, lately there has been a growing effort to lay
down some guidelines and describe the challeng- Telco Industry Readiness and Perspectives
es in the process of building NFV platforms [5]. The telecommunications sector is currently in a
Nevertheless, this only partially covers the type major paradigm shift moving in the direction of
of functionalities we advocate to offload to EE softwarization of formerly hardware-based net-
nodes. Therefore, we consider an additional stan- work elements — a concept called NFV [4]. As a
dardization effort to be necessary that seeks to lay first step, the current network functions are direct-
down precise guidelines toward the employment ly mapped to corresponding virtualized versions
of LV in a wider range of IoT use case scenarios. implemented as VMs. The fifth generation (5G)
will move toward a more cloud native approach,
Elasticity in Service Provisioning where different network functions are divided
This feature is strictly dependent on the LV into smaller components that can be individual-
engines’ capacity of quickly allocating/deallo- ly deployed and scaled, and communicate with
cating virtualized instances. Data reported in the each other using a message bus. Using MEC as
table of Fig. 2a clearly show how both contain- a platform, virtualized network functions (VNFs)
er and unikernel can promptly scale up/down. can be placed at the edge of the network, and
Furthermore, LV APIs also allow the freezing of decomposition further encourages the use of LV
the execution of an instance and quickly restor- technologies and the allocation of individual ser-
ing it through checkpoint/restore mechanisms. vice components to the edge. From the opera-
However, there is still a lack of research to eval- tor perspective, edge typically means the base
uate the interactions among multiple EEs, with- station, but virtualization on customer premises
out neglecting the fact that current LV engines equipment (CPE), such as residential gateways,

110 IEEE Network • January/February 2018

Authorized licensed use limited to: CZECH TECHNICAL UNIVERSITY. Downloaded on November 02,2023 at 09:37:43 UTC from IEEE Xplore. Restrictions apply.
 may extend the edge further. NFV is the main
driver for edge computing in mobile networks, The adoption of LV technologies in telco networks requires a change of mindset in the industry, but
and a necessity for opening up the operator net- technology questions remain for ensuring the reliability and security required for
work for third-party applications.
While operators may have difficulties compet- telecommunication networks.
ing with established players in the cloud market,
their presence close to the user makes them more [12] M. Doering, “High-Resolution Large-Scale Air Pollution
competitive for edge-dominated computation. Monitoring: Approaches and Challenges,” Proc. 3rd ACM
The adoption of LV technologies in telco networks Int’l. Wksp. MobiArch, ser. HotPlanet ’11, 2011, pp. 5–10.
requires a change of mindset in the industry, but [13] S. R. Ellis et al., “Generalizeability of Latency Detection in
a Variety of Virtual Environments,” Proc. Human Factors and
technology questions remain for ensuring the reli- Ergonomics Society Annual Meeting, vol. 48, no. 23. SAGE
ability and security required for telecommunica- Publications, 2004, pp. 2632–36.
tion networks. As unikernels can be deployed on [14] M. B. Lewis and A. J. Edmonds, “Face Detection: Mapping
the same hypervisors as VMs with minor impact Human Performance,” Perception, vol. 32, no. 8, 2003, pp.
903–20.
on orchestration infrastructure, they are more like- [15] I. Farris et al., “Providing Ultra-Short Latency to User-Centric
ly than containers to be replacements for VMs. 5G Applications at the Mobile Network Edge,” Trans. Emerg-
ing Telecommun. Technologies, 2017.
Summary Biographies
In this article, we examine the challenging prob- Roberto Morabito has worked for Ericsson Research Finland
lem of integrating LV with IoT edge networks. We since May 2014 and has been involved in the FP7 ITN METRICS
first discuss the current issues involving EC and project. He has also been a Ph.D. student since September
IoT network architectures. Therefore, we present 2014 at Aalto University in the Department of Communications
and Networking. His research interests include multi-access
three different IoT use cases, in which LV solu- edge computing, the Internet of Things, and virtualization tech-
tions can bring a set of benefits and desirable nologies. In 2013, he received his Master’s degree in computer
design flexibility. Our analysis provides a clear and telecommunications systems engineering from Mediterra-
holistic vision of such integration, which promotes nea University of Reggio Calabria.
innovative network designs to fully exploit the Vittorio Cozzolino is a Ph.D. researcher at the Technical Uni-
advantages of LV and IoT resources. Finally, we versity of Munich, Germany, where is working at the Chair of
also discuss key technical challenges and identify Connected Mobility. His research focuses on novel OS light-
open questions for future research in this area. weight virtualization techniques, the Internet of Things, compu-
tation offloading, and edge networks. His recent research covers
the development of a system for fine-grained edge offloading
References based on lightweight virtualization. In 2014, he obtained his
[1] A. Zanella et al., “Internet of Things for Smart Cities,” IEEE computer science engineering Master’s degree from the Univer-
Internet of Things J., vol. 1, no. 1, 2014, pp. 22–32. sity Federico II of Naples, Italy.
[2] V. Cozzolino, A. Y. Ding, and J. Ott, “Fades: Fine-Grained
Edge Offloading with Unikernels,” Proc. ACM Wksp. Hot Aaron Yi Ding received his M.Sc. (with distinction) and Ph.D.
Topics in Container Networking and Networked Systems, ser. (with distinction) degrees from the University of Helsinki, Fin-
HotConNet ’17, 2017, pp. 36–41. land. He is a post-doctoral associate and a project leader with
[3] K. et al., “A Survey of Computation Offloading for Mobile the Technical University of Munich. He was a visiting scholar at
Systems,” Mobile Networks and Applications, vol. 18, no. 1, Columbia University in 2014 and the University of Cambridge
2013, pp. 129–40. in 2013 under the supervision of Prof. H. Schulzrinne and Prof.
[4] L. M. Vaquero and L. Rodero-Merino, “Finding Your Way J. Crowcroft, respectively. His research interests include mobile
in the Fog: Towards a Comprehensive Definition of Fog edge computing, IoT security, and system networking. He was
Computing,” SIGCOMM Comp. Commun. Rev., vol. 44, no. a recipient of the ACM SIGCOMM Best of CCR and the Nokia
5, Oct. 2014, pp. 27–32. Foundation Scholarships.
[5] S. Natarajan et al., “An Analysis of Lightweight Virtualization
Technologies for NFV,” 2017; https://tools.ietf.org/html/ Nicklas Beijar received his D.Sc. in networking technology from
draft-natarajan-nfvrg-containers-for-nfv-03. Aalto University, Finland, in 2010 and his M.Sc. from Helsinki
[6] W. Felter et al., “An Updated Performance Comparison of University of Technology in 2002. He joined Ericsson Research
Virtual Machines and Linux Containers,” Proc. 2015 IEEE in 2013 to work on Internet of Things and cloud technologies,
Int’l. Symp. Performance Analysis of Systems and Software, in particular on distributed computing at the network edge. He
Mar. 2015, pp. 171–72. is currently focusing on network slicing and cloud security. Prior
[7] R. Morabito, “Virtualization on Internet of Things Edge to this, he worked at Aalto University as a research scientist
Devices with Container Technologies: A Performance Eval- and postdoctoral researcher on topics related to IP telephony,
uation,” IEEE Access, vol. 5, 2017, pp. 8835–50. routing protocols for ad hoc network, peer-to-peer systems, and
[8] A. Madhavapeddy et al., “Unikernels: Library Operating Sys- distributed search algorithms.
tems for the Cloud,” ACM SIGPLAN Notices, vol. 48, no. 4.
2013, pp. 461–72. Jörg Ott holds the Chair of Connected Mobility in the Depart-
[9] A. Madhavapeddy et al., “Jitsu: Just-in-Time Summoning of ment of Informatics at the Technical University of Munich. He
Unikernels,” NSDI, 2015, pp. 559–73. is also an adjunct professor for networking technology at Aalto
[10] M. Gerla et al., “Internet of Vehicles: From Intelligent Grid University. He received his diploma and doctoral (Dr.-Ing.)
to Autonomous Cars and Vehicular Clouds,” Proc. 2014 IEEE degree in computer science from TU Berlin in 1991 and 1997,
World Forum on Internet of Things, 2014, pp. 241–46. respectively, and his diploma in industrial engineering from TFH
[11] R. Morabito et al., “Lightweight Virtualization as Enabling Berlin in 1995. His research interests are in network architec-
Technology for Future Smart Cars,” Proc. Int’l. Symp. Inte- ture, (Internet) protocol design, and decentralized networked
grated Network Management, 2017. systems.

IEEE Network • January/February 2018 111

Authorized licensed use limited to: CZECH TECHNICAL UNIVERSITY. Downloaded on November 02,2023 at 09:37:43 UTC from IEEE Xplore. Restrictions apply.