Fog and IoT an Overview of Research Opportunities
Fog and IoT an Overview of Research Opportunities
6, DECEMBER 2016
Abstract—Fog is an emergent architecture for computing, Many more smart clients and edge devices, such as drones,
storage, control, and networking that distributes these services industrial and consumer robots, information-transmitting light-
closer to end users along the cloud-to-things continuum. It cov- bulbs, computers on a stick, and button-sized radio frequency
ers both mobile and wireline scenarios, traverses across hardware
and software, resides on network edge but also over access net- tuners, are following right behind.
works and among end users, and includes both data plane and It has therefore become feasible and interesting to ask:
control plane. As an architecture, it supports a growing variety “What can be done closer to the end users?” Can your car
of applications, including those in the Internet of Things (IoT), become your primary data store? Can a single appliance
fifth-generation (5G) wireless systems, and embedded artificial in your house integrate the different services and applica-
intelligence (AI). This survey paper summarizes the opportuni-
ties and challenges of fog, focusing primarily in the networking tions that have been provided by separate systems such as
context of IoT. TV set-boxes, home media centers, Internet access routers,
and smart energy control boxes? What if smartphones them-
Index Terms—Edge computing, edge networking, edge storage,
fog, fog computing, fog control, fog networking, fog storage, selves can collectively perform radio network control functions
Internet of Things (IoT). that are performed by gateways in the LTE core networks
today? What can a crowd of nearby smart endpoints and
network edge devices collectively accomplish through a dis-
tributed and immersive network on the edge? Can smart
I. I NTRODUCTION edge devices collectively enable ultralow or even determin-
OG IS an architecture that distributes computation, com- istic latency to support delay-sensitive applications, such as
F munication, control and storage closer to the end users
along the cloud-to-things continuum. Sometimes the term
real-time data analytics on the edge, mining of streaming data,
and industrial control functions?
“fog” is used interchangeably with the term “edge,” although What these questions point to is a pendulum swinging now
fog is broader than the typical notion of edge. The relevance back from “click” toward “brick,” from “more centralization”
of fog/edge is rooted in both the inadequacy of the traditional to “more immersive distribution,” from “bigger and farther
cloud and the emergence of new opportunities for the Internet away” clouds to not just smaller clouds but computation and
of Things, 5G and embedded artificial intelligence. control closer to sensors, actuators, and users. The pendu-
Over the past decade, moving computing, control, and lum between centralization and distribution is decades-old,
data storage into the cloud has been an important trend. with two distinct flavors of “distribution”: first is the end-to-
In particular, computing, storage, and network management end principle as exemplified by TCP congestion control and
functions are shifted to centralized data centers, backbone IP perhaps peer-to-peer (P2P) multicast overlay, and second is
networks, and cellular core networks. Today, however, cloud leveraging local proximity as in Ethernet and sensor networks.
computing is encountering growing challenges in meeting Fog embodies and further accelerates this click-to-brick swing-
many new requirements in the Internet of Things (IoT). back from both angles, and for not only the data plane but also
At the same time, there has also been a surging num- the control plane.
ber and variety of powerful end-user, network edge, and This paper starts with the range of new challenges in
access devices: smartphones, tablets, smart home appliances, the emerging IoT and the difficulty to address these chal-
small cellular base stations, edge routers, traffic control cab- lenges with today’s computing and networking models. It then
inets along the roadside, connected vehicles, smart meters, discusses why we will need a new architecture—fog for com-
and energy controllers in a smart power grid, smart build- puting, storage, networking, and control—and how it can fill
ing controllers, industrial control systems, just to name a few. the technology gaps and create new business opportunities.
Architecture is about functionality allocation [1]: deciding
Manuscript received April 29, 2016; revised May 30, 2016; accepted who does what and how to “glue” them back together. Unlike
June 2, 2016. Date of publication June 23, 2016; date of current version
January 10, 2017. the more mature technology fields such as serial computa-
M. Chiang is with the Department of Electrical Engineering, Princeton tion, digital communication, and the Internet, where strong
University, Princeton, NJ 08544 USA (e-mail: [email protected]). and solid architectural foundation has been laid, we are still
T. Zhang is with Cisco Systems, San Jose, CA 95134 USA (e-mail:
[email protected]). searching for architectural principles for many emerging sys-
Digital Object Identifier 10.1109/JIOT.2016.2584538 tems and applications such as IoT, cyber-physical systems, and
2327-4662 c 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: Thapar Institute of Engineering & Technology. Downloaded on June 24,2025 at 11:02:46 UTC from IEEE Xplore. Restrictions apply.
CHIANG AND ZHANG: FOG AND IoT: OVERVIEW OF RESEARCH OPPORTUNITIES 855
embedded artificial intelligence (AI). We need to make funda- Many resource-constrained devices will not be able to rely
mental decisions ranging from where to compute and where solely on their own limited resources to fulfill all their comput-
to store data along the “cloud-to-things” continuum to how ing needs. Requiring all of them to interact directly with the
to map computation tasks into a substrate of heterogeneously cloud will be unrealistic and cost prohibitive as well, because
capable and variably available nodes. Fog provides a direction such interactions often require resource-intensive processing
for us to explore such an architecture; and this paper pays and complex protocols. For example, the multitude of micro-
particular attention to IoT as a large application domain over computers on a modern vehicle need firmware updates, but
the fog architectural foundation. requiring each of these resource-constrained devices to per-
form the heavy cryptographic operations and sophisticated
II. N EW C HALLENGES IN I OT R EQUIRES procedures required to obtain firmware updates from cloud
N EW A RCHITECTURE services will be impractical.
The emerging IoT introduces many new challenges that can-
not be adequately addressed by today’s cloud and host comput- D. Cyber-Physical Systems
ing models alone. Here, we discuss several such fundamental As more cyber-physical systems are connected to the IoT,
challenges. the pendulum between the brick versus the click is starting
to swing back toward the brick again, where interactions,
A. Stringent Latency Requirements and often times close integrations, between cyber systems
Many industrial control systems, such as manufactur- and physical systems are becoming increasingly important and
ing systems, smart grids, oil and gas systems, and goods bring new business priorities and operational requirements.
packaging systems, often demand that end-to-end latencies Examples of cyber-physical systems include industrial con-
between the sensor and the control node stay within a few trol systems, smart cities, and connected cars and trains. In
milliseconds [11]. Many other IoT applications, such as such systems, uninterrupted and safe operation is often the top
vehicle-to-vehicle communications, vehicle-to-roadside com- priority. Taking a system offline for any reason can cause sig-
munications, drone flight control applications, virtual reality nificant business loss or intolerable customer inconvenience,
applications, gaming applications, and real-time financial trad- and therefore, must be planned days, weeks, and even months
ing applications, may require latencies below a few tens in advance in some cases [18].
of milliseconds. These requirements fall far outside what 1) Requiring cars to be brought to repair shops just to
mainstream cloud services can achieve. install software update packages can cause intolera-
ble inconvenience and result in heavy cost to both car
B. Network Bandwidth Constraints owners and carmakers.
The vast and rapidly growing number of connected things 2) A nuclear reactor typically runs on 18-month cycles
is creating data at an exponential rate [12]. A connected car, and any downtime can cause tens of thousands of
for example, can create tens of megabytes of data per second. dollars [16].
This will include data about: 1) the car’s mobility such as its 3) Many other industrial control or manufacturing systems,
routes and speeds; 2) the car’s operating conditions such as such as car assembly plants and electrical power gen-
the wear and tear on its components; 3) the car’s surround- erators in the energy grids, have similar requirements
ing environment such as road and weather conditions; and for uninterrupted safe operations and require weeks to
4) videos recorded by the car’s safety cameras. An autonomous months lead times to plan for system down times.
vehicle will generate even more data, which was estimated As a result, unlike the routers, switches, personal com-
to be about one gigabyte per second [13]. The U.S. smart puters, and smartphones in today’s Internet, the timings and
grid is expected to generate 1000 petabytes of data each opportunities for updating the hardware and software in such
year. By comparison, the U.S. Library of Congress gener- cyber-physical systems can be severely limited. Many time-
ated about 2.4 petabytes of data a month, Google trafficked critical control applications, which need to be updated over
about one petabyte a month, and AT&T’s network consumed time, cannot be moved to the cloud due to delay, bandwidth, or
200 petabytes a year in 2010 [14]. other constraints. Therefore, a new computing and networking
Sending all the data to the cloud will require prohibitively architecture will be needed to reduce the needs for the hard-
high network bandwidth. It is often unnecessary or sometimes ware and software in mission-critical systems to be updated
prohibited due to regulations and data privacy concerns. ABI over time.
Research estimates that 90% of the data generated by the end-
points will be stored and processed locally rather than in the E. Uninterrupted Services With Intermittent
cloud [12]. Connectivity to the Cloud
Cloud services will have difficulty providing uninterrupted
C. Resource-Constrained Devices services to devices and systems that have intermittent net-
Many IoT devices will have severely limited resources. work connectivity to the cloud. Such devices include vehicles,
Examples include sensors, data collectors, actuators, con- drones, and oil rigs. For example, an oil rig in the ocean and
trollers, surveillance cameras, cars, trains, drones, and medical far away from shore may have only satellite communication
devices embedded in patients. channels to connect to the cloud. These satellite channels can
Authorized licensed use limited to: Thapar Institute of Engineering & Technology. Downloaded on June 24,2025 at 11:02:46 UTC from IEEE Xplore. Restrictions apply.
856 IEEE INTERNET OF THINGS JOURNAL, VOL. 3, NO. 6, DECEMBER 2016
suffer widely fluctuating quality and intermittent availability. protect a very large number of resource-constrained devices
However, applications such as data collection, data analytics, from security attacks?
and controls for the oil rig have to be available even when 3) Assessing the Security Status of Large Distributed
the rig does not have network connectivity with the cloud. Systems in Trustworthy Manner: IoT will support many large
As another example, when a car traverses an area where it distributed systems. A connected transportation system, for
loses Internet connectivity, many services and applications for example, may have thousands of devices deployed through-
the devices and people in the car must continue to be avail- out a city to control traffic signals and communicate with
able. When a car breaks down in such an area and needs to vehicles. A large carmaker will need to ensure the security
have one of its electronic control unit (ECU) replaced before of tens of millions of cars on the road in a large coun-
it can run again, the new ECU should be authenticated to pre- try such as the USA. An oil and gas company may need
vent any unauthorized and potentially malware-infected ECUs to interconnect hundreds of remote sites such as oil rigs,
from being installed on the vehicle. However, cloud-based exploration sites, refineries, and pipelines. A smart grid will
authentication services will not be available in this scenario. consist of networked subsystems for metering, data collection,
data aggregation, energy distribution, and demand response in
F. New Security Challenges multiple geographical areas.
Existing cyber security solutions for today’s Internet, Therefore, the ability to tell, in a trustworthy manner,
designed primarily for protecting enterprise networks, whether a large number of distributed devices and systems
data centers, and consumer electronics, have focused on pro- are operating securely, will be essential. However, conven-
viding perimeter-based protections. In particular, a system or tional approaches have difficulty meeting both the scalability
an individual device under protection is placed behind firewalls and the trustworthy monitoring requirements at the same time.
that work with intrusion detection and prevention systems to Today’s security health monitoring systems rely on collect-
prevent security threats from breaking through the protected ing security status messages and log data from devices. These
perimeters. Some resource-intensive security functions are systems, however, can often generate untrustworthy results
also being moved to the cloud. Existing cloud-based security when applied in some IoT systems.
services continue to focus on providing perimeter-based pro- 1) Many devices operating in physically unprotected envi-
tection, such as redirecting email and Web traffic to the clouds ronments can be compromised and used to send false
for threat detection, and redirecting access control requests information [22]–[24]. Adversaries can also easily use
to the clouds for authentication and authorization process- these compromised devices to form a local majority in
ing. Should threats penetrate these protections, the common many IoT scenarios. For example, they may compromise
responses have been for human operators to take the system the majority of the smart meters in a house, a building,
offline, clean up or replace compromised files and devices, and or even an entire region. As a result, existing mecha-
then put the system back online. nisms for detecting false information, which typically
This existing security paradigm will no longer be adequate rely on the majority of the data sources to be honest
for addressing many new security challenges in the emerging (i.e., uncompromised and not malfunctioning), will no
IoT. Here, we discuss several such challenges. longer be adequate.
1) Keeping Security Credentials and Software up to Date 2) Attackers can compromise a cyber-physical system
on Large Number of Devices: As the number and variety of and damage the physical equipment while keeping
the connected devices increase, a growing challenge will be the messages to and from the system appear normal.
how to manage the security credentials on these devices and A prime example is the Stuxnet attack on the Iranian
how to keep the security credentials and security software on nuclear facility—the Stuxnet worm masqueraded the
the devices up to date. Requiring every device to connect to attack by sending normal status messages to the sys-
the cloud to update its security credentials and software will tem administers while spinning the nuclear reactor out
be impractical. of control [19]–[21].
2) Protecting Resource-Constrained Devices: Many To increase the trustworthiness of security status mon-
resource-constrained devices in the IoT will not have suf- itoring, remote attestation mechanisms allow a device to
ficient resources to protect themselves adequately. These cryptographically prove its trustworthiness to a remote
devices may have very long lifespans, and the hardware verifier [25], [26]. A device makes a claim about certain prop-
and software on them can be impractical to upgrade. Yet, erties of its hardware, software, or runtime environment to the
these devices will need to remain secure over their long verifier and uses its security credentials (e.g., a hardware-based
lifespans. For example, replacing any hardware on cars, which root of trust and public key certificates) to vouch for these
have already been sold to consumers, can create significant properties. The verifier then cryptographically verifies these
inconvenience to vehicle owners and result in heavy costs claims.
and reputation damages to carmakers. However, over a car’s However, existing remote attestation methods have focused
long lifespan that averages about 11.4 years [17], security on enabling an individual device to attest to its own trust-
threats will become significantly more advanced, many new worthiness. Many resource-constrained devices in the IoT will
threats will appear, and the mechanisms required to combat not be able to support processing-intensive remote attesta-
the growing threats will need to be enhanced and upgraded tion. Even when they can, forcing a large number of devices
accordingly. Therefore, a fundamental question arises: How to to perform remote attestation can result in prohibitively
Authorized licensed use limited to: Thapar Institute of Engineering & Technology. Downloaded on June 24,2025 at 11:02:46 UTC from IEEE Xplore. Restrictions apply.
CHIANG AND ZHANG: FOG AND IoT: OVERVIEW OF RESEARCH OPPORTUNITIES 857
high cost and management complexity. Furthermore, exist- Therefore, today’s highly disruptive incident response
ing remote attestation technology alone cannot handle the paradigm will no longer be adequate for securing the many
case where a device itself is not compromised but its sensory mission-critical systems in the emerging IoT.
input is.
4) Responding to Security Compromises Without Causing
Intolerable Disruptions: Today’s incident response solutions III. A N E MERGING E RA OF F OG
rely predominately on brute-force mechanisms such as shut- Filling the technology gaps in supporting IoT will require
ting down a potentially compromised system, reinstalling a new architecture—fog—that distributes computing, control,
and rebooting its software, or replacing its components and storage, and networking functions closer to end user devices.
subsystems. Such highly disruptive responses, which largely Complementing the centralized cloud, fog stands out along
disregard how severe the compromises actually are, can cause the following three dimensions:
intolerable disruptions to mission-critical systems. However, 1) Carry out a substantial amount of data storage at or near
maintaining uninterrupted and safe operation, even when the the end user (rather than storing data only in remote
system is compromised, is often the highest priority for data centers).
mission-critical systems such as industrial control systems, 2) Carry out a substantial amount of computing and control
manufacturing plants, connected vehicles, drones, and smart functions at or near the end user (rather than performing
grids. all these functions in remote data centers and cellular
1) An electric power generator may be infected by a mal- core networks). Such computing and control functions
ware that merely seeks to steal power for unauthorized can include the following.
use. Shutting down the power generator could cause a) Applications for end users and their devices.
severe disruptions to the smart grid and excessive power b) Functions for controlling and operating end-user
outages. systems such as manufacturing systems, vehicles,
2) Industrial control systems often have little tolerance and smart grids.
for down time. Manufacturing operations can also have c) Services for managing end-user as well as end-to-
critical safety implications. As a result, manufacturers end networks, systems, and applications.
usually value uninterrupted operation and safety over d) Services for supporting cloud-based applications,
system integrity. This means that hardware and soft- such as collecting and preprocessing data to be sent
ware updates can only be installed during a system’s to the cloud.
scheduled down times, which have to be short and far 3) Carry out a substantial amount of communication and
between, rather than every time any security compromise networking at or near the end user (rather than routing
is detected. all network traffic through the backbone networks). This
3) A connected car can be infected by malware that can can include, for example, ways to improve the perfor-
become active while the car is in motion. While the mal- mance and scalability of local D2D networks, intelligent
ware can do a range of damages to the vehicle and can control of radio access networks (RANs), organize and
put the driver and passengers in harm’s way, abruptly manage local mobile ad-hoc networks, and integrate
shutting down the engine each time any malware is local ad-hoc networks with the infrastructure networks.
detected could be an even quicker and surer way to cause Fog and cloud complement each other to form a service
deadly traffic accidents. continuum between the cloud and the endpoints by provid-
4) If a drone flying midair is abruptly turned off just ing mutually beneficial and interdependent services to make
because a security compromise is detected, it can crash computing, storage, control, and communication possible any-
from the sky onto people, houses, and other proper- where along the continuum.
ties to cause serious damages. Instead, safe landing 1) Fog Enables a Service Continuum: Fog fills the gap
or safe return-home mechanisms will be essential for between the cloud and the things to enable a service con-
responding to such security threats that can compromise tinuum. For example, to the wearable devices, a mobile
a drone’s flight. phone may become the fog to provide local control and
5) A server in a data center may be infected by a spyware analytics applications to the wearable devices. When the
that seeks to steal commercial secrets. While allowing user is inside her vehicle, the vehicle can become the
such a compromised server to continue to operate could fog for her mobile phone to allow many smartphone
give the attacker access to some sensitive data, it may functions, such as display, user interface, audio, phone
not directly impact the data-center’s mission-critical ser- book, to be moved to the vehicle. Roadside traffic con-
vices. If we shut down the server, or halt the execution trol equipment can in turn serve as the fog for the vehicle
of the malware-infected files to wait for the malware to to provide traffic information to the vehicle.
be removed, the system downtime could cause signifi- 2) Fog and Cloud Are Interdependent: For example, cloud
cantly more damage, including causing vast economic services may be used to manage the fog. Fog can act as
losses to the data center operator, business disruptions the proxy of the cloud to deliver cloud services to end-
to those who count on the data centers to operate their points, and act as the proxy of the endpoints to interact
businesses, and inconvenience to other users of the data with the cloud. Furthermore, fog can be the beachheads
center. for collecting and aggregating data for the cloud.
Authorized licensed use limited to: Thapar Institute of Engineering & Technology. Downloaded on June 24,2025 at 11:02:46 UTC from IEEE Xplore. Restrictions apply.
858 IEEE INTERNET OF THINGS JOURNAL, VOL. 3, NO. 6, DECEMBER 2016
TABLE I
3) Fog and Cloud Are Mutually Beneficial: Some functions M AIN C HARACTERISTICS OF F OG AND H OW IT C OMPLEMENTS C LOUD
are naturally more advantageous to be carried out in
the fog while others in the cloud. Determining which
functions should be carried out in the fog and how the
fog should interact with the cloud will be key aspects
of fog research and development.
Traditionally, services and applications are provided with
large, centralized, expensive, and hard-to-innovate “boxes”
such as the service gateways and packet data network gate-
ways in the LTE core, large servers in a data center, and the
core gateways and routers in a wide-area-network backbone.
The traditional view is that the edge uses the core networks
and data centers. The fog view is that the edge is part of the
core network and a data center.
Table I outlines the main characteristics of fog and how it
complements cloud.
Authorized licensed use limited to: Thapar Institute of Engineering & Technology. Downloaded on June 24,2025 at 11:02:46 UTC from IEEE Xplore. Restrictions apply.
CHIANG AND ZHANG: FOG AND IoT: OVERVIEW OF RESEARCH OPPORTUNITIES 859
TABLE II
3) Agility: Rapid innovation and affordable scaling. It is F OG P ROVIDES E FFECTIVE WAYS TO A DDRESS I OT C HALLENGES
usually much faster and cheaper to experiment with
client and edge devices. Rather than waiting for ven-
dors of large network and cloud boxes to initiate or
adopt an innovation. Fog will make it easier to create
an open market place for individuals and small teams to
use open application programming interfaces, open soft-
ware development kits (SDKs), and the proliferation of
mobile devices to innovate, develop, deploy, and operate
new services.
4) Latency: Real-time processing and cyber-physical sys-
tem control. Fog enables data analytics at the network
edge and can support time-sensitive functions for local
cyber-physical systems. This is essential for not only sta-
ble control systems but also for the tactile Internet vision
to enable embedded AI applications with millisecond
reaction times as elaborated in the next subsection.
These advantages in turn enable new services and busi-
ness models, and may help broaden revenues, reduce cost, or
accelerate product rollouts as elaborated in the next subsection.
Authorized licensed use limited to: Thapar Institute of Engineering & Technology. Downloaded on June 24,2025 at 11:02:46 UTC from IEEE Xplore. Restrictions apply.
860 IEEE INTERNET OF THINGS JOURNAL, VOL. 3, NO. 6, DECEMBER 2016
Amazon Cloud
SDK
Fig. 1. Data plane and control plane of fog enable different applications.
SDK
mandate, may further push network innovation to the SDK
edge in the U.S. A new regulatory environment does 4G Core Cloud
not mean networks cannot be engineered and managed SDK
K
anymore, but we may need different vantage points of
control: not from inside the network but from around Mobile client devices
the end users. For example, today network operators can
pick which lane (WiFi, Macro-cellular, and Femtocell) Fig. 2. SDK sitting inside clients can enable network inference and configu-
a user device should be in. Since different lanes have ration. Crowd-sensing and byte-counting happen in fog, coordinated through
different speeds and different payment system/amount, a controller sitting in the Amazon cloud, bypassing the traditional reliance on
network policy and configuration in the cellular core cloud.
such practice may not be allowed any more in the
U.S. Instead, we need new systems where each user
device must choose which lane to be in for itself. As long
as the government does not prohibit end-user choices, such as the inference, control, configuration, and management
then we can run fog-based networking from the edge, of networks. We will also see that fog operates across a con-
through client/home-driven control/configuration. tinuum spanning device, access, edge and more, and observe
the collaboration between fog and cloud.
Case 1 [Crowd-Sensing LTE States (in Commercial
IV. F OG U SE C ASE S TUDIES Deployment)]: Through a combination of passive
Architectural research and development asks the question of measurement (e.g., RSRQ), active probing (e.g., packet
“who does what, at what timescale, and how to put the mod- train), application throughput correlation and historical
ules back together?” As an architecture, fog supports a variety data mining, a collection of client devices may be able to, in
of applications, including those typically associated with IoT real-time and useful accuracy, infer the states of an eNB such
and those often viewed as part of fifth-generation (5G) or as the number of resource blocks used [4].
data analytics and data management. Fog is an architecture Case 2 [OTT Network Provisioning and Smart Data Pricing
for computing, storage, as well as for networking. In par- (in Commercial Deployment)]: [27] Fog directly leverages the
ticular, fog network architecture consists of both data plane “things” and phones instead, and removes the dependence
and control plane, each with a rapidly growing number of on boxes-in-the-network altogether. With SDKs sitting behind
examples across protocol layers from the physical layer to the apps on client devices, through tasks such as byte-counting,
application layer. content tagging, location tracking, behavior monitoring, net-
1) Examples of Data Plane of Fog: work services can be innovated much faster. In this case,
a) pooling of clients idle computing/storage/ the client SDKs collectively work through a controller (in
bandwidth resources and local content; the cloud as hosted say by Amazon) but bypass most of the
b) content caching at the edge and bandwidth man- cellular core network (a second type of cloud).
agement at home; Case 3 [Client-Based HetNets Control (in 3GPP
c) client-driven distributed beam-forming; Standards)]: Coexistence of heterogeneous networks
d) client-to-client direct communications (e.g., (e.g., LTE, femto, and WiFi) coexistence is a key feature in
FlashLinQ, LTE direct, WiFi direct, and Air cellular networks today. Rather than through network operator
Drop); control, each client can observe its local conditions and make
e) cloudlets and micro data-centers. decision on which network to join. Through randomization
2) Examples of Control Plane of Fog: and hysteresis, such local actions may emerge globally to
a) over the top (OTT) content management; converge to a desirable configuration [5]. In the case of
b) fog-RAN: Fog driven RAN; hybrid control of HetNets, the fog–cloud interface allows
c) client-based HetNets control; real-time network configuration be carried out by the clients
d) client-controlled cloud storage; themselves, while over longer timescale parameters like RAT
e) session management and signaling load at the edge; stability attribute or hysteresis values can pass from the cloud
f) crowd-sensing inference of network states; (wireless core network) to the clients.
g) edge analytics and real-time stream-mining. Case 4 [“Shred and Spread” Client-Controlled Cloud
Data-plane of fog has been more extensively studied, Storage (in Beta Trial)]: By decoupling massive cheap stor-
e.g., [3]. In this section, we highlight a few particular cases age (in the cloud) from client side control of privacy (in the
that illustrate the potential and challenges of fog control plane, fog), we can achieve the best of both worlds. For example, by
Authorized licensed use limited to: Thapar Institute of Engineering & Technology. Downloaded on June 24,2025 at 11:02:46 UTC from IEEE Xplore. Restrictions apply.
CHIANG AND ZHANG: FOG AND IoT: OVERVIEW OF RESEARCH OPPORTUNITIES 861
Authorized licensed use limited to: Thapar Institute of Engineering & Technology. Downloaded on June 24,2025 at 11:02:46 UTC from IEEE Xplore. Restrictions apply.
862 IEEE INTERNET OF THINGS JOURNAL, VOL. 3, NO. 6, DECEMBER 2016
Authorized licensed use limited to: Thapar Institute of Engineering & Technology. Downloaded on June 24,2025 at 11:02:46 UTC from IEEE Xplore. Restrictions apply.
CHIANG AND ZHANG: FOG AND IoT: OVERVIEW OF RESEARCH OPPORTUNITIES 863
using D4D for pooling idle edge resources as discussed in the To address the above challenges, we need both of the
previous sections, new protocol stacks for end-user devices to following:
support fog-enabled edge networking may be needed. 1) fundamental research, across networking, device hard-
ware/OS, pricing, human-computer interface, and
C. Security data science;
2) industry-academia interactions, as exemplified in the
Fog presents new security challenges. Distributed systems
Open Fog Consortium (OpenFog), a global, nonprofit
are in general more vulnerable to attacks than centralized
consortium launched in November 2015 with founding
systems. While cloud operates in heavily protected facilities
members from ARM, Cisco, Dell, Intel, Microsoft and
selected and controlled by cloud operators, fog often needs to
Princeton University Edge Laboratory.
operate in more vulnerable environments—where they can best
meet customer requirements and often wherever users want
them to be. Many fog systems will be significantly smaller VI. C ONCLUDING R EMARKS
than clouds (e.g., a fog node on a vehicle, in a manufactur- Fog is starting to reshape the future landscape of multiple
ing plant, or on a oil rig), and hence, may not have as much industries, driving innovation across the entire industry food
resources as the clouds to protect themselves. Furthermore, chain, including the following.
each fog system may not have the global intelligence necessary 1) End user experience providers (e.g., GE, Toyota, Sony,
for detecting threats. Walmart, etc.).
At the same time, however, fog’s proximity to end users 2) Network operators (e.g., AT&T, Verizon, Comcast, etc.).
and locality on the edge enable it to help address certain 3) Network equipment vendors (e.g., Cisco, Nokia,
new IoT security challenges as discussed in the previous sec- Ericsson, Huawei, etc.).
tions. Fog can, for example, act as the first nodes for access 4) Cloud service providers (e.g., VMWare, Amazon, etc.).
control and traffic encryption, provide contextual integrity 5) System integrators (e.g., IBM, HP, etc.).
and isolation, serve as the aggregation and control points for 6) Edge device manufacturers (e.g., Linksys, Samsung,
privacy-sensitive data before the data leaves the edge, and etc.).
act as the proxies of resource-constrained devices to carry 7) Client and IoT device manufacturers (e.g., Dell,
out selected security functions for these resource-constrained Microsoft, Apple, Google, etc.).
devices. 8) Computer chip suppliers (e.g., Intel, ARM, Qualcomm,
Broadcom, etc.).
D. Incentivization of Device Participation The past 15 years have seen the pendulum swinging toward
“click.” Now it has started to swing back closer to the
In some IoT use cases, it is not too many un-trustworthy
“brick,” pointing to a co-existence of fog and cloud. Cloud
clients that create concerns but too few clients willing to par-
has the advantages for massive storage, heavy duty compu-
ticipate. This can be the case when, for example, clients are
tation, global coordination and wide-area connectivity, while
expected to voluntarily contribute their computing or storage
fog will be useful for real time processing, rapid innova-
resources or to collaborate with each other to support applica-
tion, user-centric service and edge resource pooling. 2016 is
tions. Market systems and incentive mechanisms will become
an interesting year to start systematically exploring what fog
useful.
might look like and the differences it will bring to the world
of networking and computing in the next 15 years.
E. Convergence and Consistency
Local interactions could lead to divergence, oscillation, ACKNOWLEDGMENT
and inconsistency of global system states, which are typi- The authors are grateful for the inspiring conversations
cal issues in distributed systems and can become more acute with colleagues at Princeton Edge Lab and Cisco, as well as
in a massive, under-organized, possibly mobile fog system with many colleagues in industry and academia, especially,
with diverse capabilities, and potentially virtualized pool of Flavio Bonomi, Russell Hsing, Bharath Balasubramanian,
resources shared unpredictably. Use cases in edge analyt- Sangtae Ha, Junshan Zhang, Raj Savoor, John Smee,
ics and stream mining provide additional challenges on this Chonggang Wang, and representatives of many member com-
recurrent challenge in distributed systems. panies and universities in the Open Fog Consortium.
Authorized licensed use limited to: Thapar Institute of Engineering & Technology. Downloaded on June 24,2025 at 11:02:46 UTC from IEEE Xplore. Restrictions apply.
864 IEEE INTERNET OF THINGS JOURNAL, VOL. 3, NO. 6, DECEMBER 2016
[4] A. Chakraborty, V. Navda, V. N. Padmanabhan, and R. Ramjee, [25] R. Chen, L. Wei, H. Zou, and M. Zhai, “A TCM-based remote anony-
“Coordinating cellular background transfers using loadsense,” in Proc. mous attestation protocol for power information system,” in Proc. Int.
Mobicom, Miami, FL, USA, 2013, pp. 63–74. Power Electron. Mater. Eng. Conf. (IPEMEC), Dalian, China, May 2015.
[5] E. Aryafar, A. Keshavarz-Haddard, M. Wang, and M. Chiang, “RAT [26] A. Francillon, Q. Nguyen, K. B. Rasmussen, and G. Tsudik,
selection games in HetNets,” in Proc. IEEE INFOCOM, Turin, Italy, “A minimalist approach to remote attestation,” in Proc. Conf. Design
2013, pp. 998–1006. Autom. Test Europe (DATE), Dresden, Germany, 2014, pp. 1–6.
[6] J. Y. Chong, C. Joe-Wong, S. Ha, and M. Chiang, “CYRUS: Towards [27] S. Ha, S. Sen, C. Joe-Wong, Y. Im, and M. Chiang, “TUBE: Time-
client-defined cloud storage,” in Proc. EuroSys, Bordeaux, France, 2015, dependent pricing for mobile data,” in Proc. ACM SIGCOMM, Helsinki,
Art. no. 17. Finland, Aug. 2012, pp. 247–258.
[7] L. Canzian and M. van der Schaar, “Realtime stream mining: Online
knowledge extraction using classifier networks,” IEEE Netw., vol. 29,
no. 5, pp. 10–16, Sep./Oct. 2015.
[8] X. Chen, B. Proulx, X. Gong, and J. Zhang, “Social trust and social
reciprocity based cooperative D2D communications,” in Proc. ACM Int.
Symp. Mobile Ad Hoc Netw. Comput. (MOBIHOC), Bengaluru, India,
Jul. 29–Aug. 1, 2013, pp. 187–196.
[9] F. M. F. Wong, S. Ha, C. Joe-Wong, Z. Liu, and M. Chiang, “Mind
your own bandwidth: Adaptive traffic management on network edge,”
in Proc. IEEE IWQoS, Portland, OR, USA, 2015. Mung Chiang (S’00–M’03–SM’08–F’12) is the
[10] Y. Du, E. Aryafar, J. Camp, and M. Chiang, “iBeam: Intelligent client- Arthur LeGrand Doty Professor of Electrical
side multi-user beamforming in wireless networks,” in Proc. IEEE Engineering with Princeton University, Princeton,
INFOCOM, Toronto, ON, CA, 2014, pp. 817–825. NJ, USA. His book Networks: Friends, Money
[11] M. Weiner, M. Jorgovanovic, A. Sahai, and B. Nikolié, “Design of and Bytes and online course reached 250 000 stu-
a low-latency, high-reliability wireless communication system for con- dents since 2012. He founded the Princeton Edge
trol applications,” in Proc. IEEE Int. Conf. Commun. (ICC), Sydney, Laboratory in 2009, which bridges the theory-
NSW, Australia, 2014, pp. 3829–3835. practice gap in edge networking research by span-
[12] R. Kelly. Internet of Things Data to Top 1.6 Zettabytes ning from proofs to prototypes. He co-founded a few
by 2022. Accessed on Apr. 7, 2016. [Online]. Available: startups in mobile pricing, IoT, big data areas,
https://campustechnology.com/articles/2015/04/15/internet-of-things- and the Open Fog Consortium. He is the Director
data-to-top-1-6-zettabytes-by-2020.aspx of Keller Center for Innovations in Engineering Education in Princeton
[13] L. Mearian. Self-Driving Cars Could Create 1GB of Data University and the inaugural Chairman of the Princeton Entrepreneurship
a Second. Accessed on Apr. 7, 2016. [Online]. Available: Council.
http://www.computerworld.com/article/2484219/emerging-technology/ Prof. Chiang was the recipient of the 2013 Alan T. Waterman Award, the
self-driving-cars-could-create-1gb-of-data-a-second.html highest honor to U.S. young scientists and engineers.
[14] N. Cochrane. (Mar. 23, 2010). US Smart Grid to Generate
1000 Petabytes of Data a Year. Accessed on Apr. 7, 2016. [Online].
Available: http://www.itnews.com.au/news/us-smart-grid-to-generate-
1000-petabytes-of-data-a-year-170290#ixzz458VaITi6
[15] K. Stouffer, J. Falco, and K. Scarfone, “Guide to industrial control sys-
tems (ICS) security,” Nat. Inst. Standards Technol. (NIST), U.S. Dept.
Commerce, Washington, DC, USA, Special Pub. 800-82, Jun. 2011.
[16] W. Ashford. (Oct. 15, 2014). Industrial Control Systems: What are the
Security Challenges? Accessed on Jan. 28, 2016. [Online]. Available:
http://www.computerweekly.com/news/2240232680/Industrial-control-
systems-What-are-the-security-challenges Tao Zhang (F’00) received the B.S. and M.S.
[17] Bureau of Transportation Statistics. U.S. Dept. Transp., Washington, degrees in electrical engineering from Northern
DC, USA. Accessed on Mar. 2, 2016. [Online]. Available: Jiaotong University, Beijing, China, and the Ph.D.
http://www.rita.dot.gov/bts/sites/rita.dot.gov.bts/files/publications/ degree in electrical and computer engineering
national_transportation_statistics/html/table_01_26.html_mfd from the University of Massachusetts, Amherst,
[18] G. Gan, Z. Lu, and J. Jiang, “Internet of Things security analysis,” MA, USA.
in Proc. Int. Conf. Internet Technol. Appl. (iTAP), Wuhan, China, He joined Cisco Systems, San Jose, CA, USA, as
Aug. 2011, pp. 1–4. the Chief Scientist for Smart Connected Vehicles, in
[19] N. Falliere, L. O Murchu, and E. Chien, “W32.stuxnet Dossier,” 2012. Since then, he has also been leading initiatives
Symantec Security Response, Ver. 1.4, Mountain View, CA, USA, to develop strategies, architectures, technology, and
Feb. 2011. eco-systems for the Internet of Things and fog com-
[20] K. Zetter, Countdown to Zero Day: Stuxnet and the Launch of the puting. From 1995 to 2012, he was with Telcordia Technologies (formerly
World’s First Digital Weapon. New York, NY, USA: Crown, 2014. Bell Communications Research or Bellcore), Piscataway, NJ, USA, where he
[21] Stuxnet. Accessed on Mar. 2, 2016. [Online]. Available: was a Chief Scientist and the Director of Mobile and Vehicular Networking.
https://en.wikipedia.org/wiki/Stuxnet For over 25 years, he has been in various technical and executive positions,
[22] L. Delgrossi and T. Zhang, Vehicle Safety Communications: Protocols, directing research and product development. He holds over 50 U.S. patents and
Security, and Privacy. Hoboken, NJ, USA: Wiley, 2012. co-authored two books Vehicle Safety Communications: Protocols, Security,
[23] T. Zhang, H. Antunes, and S. Aggarwal, “Defending connected vehicles and Privacy (Wiley, 2012) and IP-Based Next Generation Wireless Networks
against malware: Challenges and a solution framework,” IEEE Internet (Wiley, 2004).
Things J., vol. 1, no. 1, pp. 10–21, Feb. 2014. Dr. Zhang is a Co-Founder and a Board Director of the Open Fog
[24] T. Zhang, H. Antunes, and S. Aggarwal, “Securing connected vehicles Consortium and the CIO and a Board Governor of the IEEE Communications
end to end,” in Proc. SAE World Congr. Exhibit., Detroit, MI, USA, Society. He was the Founding Board Director of the Connected Vehicle Trade
Apr. 2014. Association.
Authorized licensed use limited to: Thapar Institute of Engineering & Technology. Downloaded on June 24,2025 at 11:02:46 UTC from IEEE Xplore. Restrictions apply.