CONTRIBUTED

P A P E R

Software-Defined Networking:
A Comprehensive Survey
This paper offers a comprehensive survey of software-defined networking
covering its context, rationale, main concepts, distinctive features,
and future challenges.
By Diego Kreutz, Member IEEE , Fernando M. V. Ramos, Member IEEE ,
Paulo Esteves Verı́ssimo, Fellow IEEE , Christian Esteve Rothenberg, Member IEEE ,
Siamak Azodolmolky, Senior Member IEEE , and Steve Uhlig, Member IEEE

ABSTRACT | The Internet has led to the creation of a digital network evolution. In this paper, we present a comprehensive
society, where (almost) everything is connected and is acces- survey on SDN. We start by introducing the motivation for SDN,
sible from anywhere. However, despite their widespread adop- explain its main concepts and how it differs from traditional
tion, traditional IP networks are complex and very hard to networking, its roots, and the standardization activities regard-
manage. It is both difficult to configure the network according ing this novel paradigm. Next, we present the key building
to predefined policies, and to reconfigure it to respond to blocks of an SDN infrastructure using a bottom-up, layered
faults, load, and changes. To make matters even more difficult, approach. We provide an in-depth analysis of the hardware
current networks are also vertically integrated: the control and infrastructure, southbound and northbound application prog-
data planes are bundled together. Software-defined network- ramming interfaces (APIs), network virtualization layers,
ing (SDN) is an emerging paradigm that promises to change this network operating systems (SDN controllers), network prog-
state of affairs, by breaking vertical integration, separating the ramming languages, and network applications. We also look at
network’s control logic from the underlying routers and cross-layer problems such as debugging and troubleshooting.
switches, promoting (logical) centralization of network control, In an effort to anticipate the future evolution of this new pa-
and introducing the ability to program the network. The radigm, we discuss the main ongoing research efforts and
separation of concerns, introduced between the definition of challenges of SDN. In particular, we address the design of
network policies, their implementation in switching hardware, switches and control platformsVwith a focus on aspects
and the forwarding of traffic, is key to the desired flexibility: by such as resiliency, scalability, performance, security, and
breaking the network control problem into tractable pieces, dependabilityVas well as new opportunities for carrier trans-
SDN makes it easier to create and introduce new abstractions in port networks and cloud providers. Last but not least, we ana-
networking, simplifying network management and facilitating lyze the position of SDN as a key enabler of a software-defined
environment.

KEYWORDS | Carrier-grade networks; dependability; flow-

based networking; network hypervisor; network operating sys-
Manuscript received June 15, 2014; revised October 6, 2014; accepted
November 10, 2014. Date of current version December 18, 2014.
tems (NOSs); network virtualization; OpenFlow; programmable
D. Kreutz and F. M. V. Ramos are with the Department of Informatics of Faculty of networks; programming languages; scalability; software-
Sciences, University of Lisbon, 1749-016 Lisbon, Portugal (e-mail: [email protected];
[email protected]).
defined environments; software-defined networking (SDN)
P. E. Verı́ssimo is with the Interdisciplinary Centre for Security, Reliability and
Trust (SnT), University of Luxembourg, L-2721 Walferdange, Luxembourg
(e-mail: [email protected]).
C. E. Rothenberg is with the School of Electrical and Computer
I . INTRODUCTION
Engineering (FEEC), University of Campinas, Campinas 13083-970, Brazil
(e-mail: [email protected]).
The distributed control and transport network protocols
S. Azodolmolky is with the Gesellschaft für Wissenschaftliche Datenverarbeitung mbH running inside the routers and switches are the key tech-
Göttingen (GWDG), 37077 Göttigen, Germany (e-mail: [email protected]).
S. Uhlig is with Queen Mary University of London, London E1 4NS, U.K.
nologies that allow information, in the form of digital
(e-mail: [email protected]). packets, to travel around the world. Despite their wide-
Digital Object Identifier: 10.1109/JPROC.2014.2371999 spread adoption, traditional IP networks are complex and
0018-9219  2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
14 Proceedings of the IEEE | Vol. 103, No. 1, January 2015
 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

hard to manage [1]. To express the desired high-level net-

work policies, network operators need to configure each
individual network device separately using low-level and
often vendor-specific commands. In addition to the config-
uration complexity, network environments have to endure
the dynamics of faults and adapt to load changes. Automa-
tic reconfiguration and response mechanisms are virtually
nonexistent in current IP networks. Enforcing the required
policies in such a dynamic environment is therefore highly
challenging.
To make it even more complicated, current networks
are also vertically integrated. The control plane (that de-
cides how to handle network traffic) and the data plane
(that forwards traffic according to the decisions made by
the control plane) are bundled inside the networking de-
vices, reducing flexibility and hindering innovation and
evolution of the networking infrastructure. The transition Fig. 1. Simplified view of an SDN architecture.
from IPv4 to IPv6, started more than a decade ago and still
largely incomplete, bears witness to this challenge, while
in fact IPv6 represented merely a protocol update. Due to
the inertia of current IP networks, a new routing protocol handling rules (flow table). Each rule matches a subset of
can take five to ten years to be fully designed, evaluated, the traffic and performs certain actions (dropping, for-
and deployed. Likewise, a clean-slate approach to change warding, modifying, etc.) on the traffic. Depending on the
the Internet architecture (e.g., replacing IP) is regarded as rules installed by a controller application, an OpenFlow
a daunting taskVsimply not feasible in practice [2], [3]. switch canVinstructed by the controllerVbehave like a
Ultimately, this situation has inflated the capital and ope- router, switch, firewall, or perform other roles (e.g., load
rational expenses of running an IP network. balancer, traffic shaper, and in general those of a
Software-defined networking (SDN) [4], [5] is an middlebox).
emerging networking paradigm that gives hope to change An important consequence of the SDN principles is the
the limitations of current network infrastructures. First, it separation of concerns introduced between the definition
breaks the vertical integration by separating the network’s of network policies, their implementation in switching
control logic (the control plane) from the underlying rout- hardware, and the forwarding of traffic. This separation is
ers and switches that forward the traffic (the data plane). key to the desired flexibility, breaking the network control
Second, with the separation of the control and data planes, problem into tractable pieces, and making it easier to
network switches become simple forwarding devices and create and introduce new abstractions in networking, sim-
the control logic is implemented in a logically centralized plifying network management and facilitating network
controller (or network operating system1), simplifying po- evolution and innovation.
licy enforcement and network (re)configuration and evol- Although SDN and OpenFlow started as academic
ution [6]. A simplified view of this architecture is shown in experiments [9], they gained significant traction in the
Fig. 1. It is important to emphasize that a logically cen- industry over the past few years. Most vendors of com-
tralized programmatic model does not postulate a physi- mercial switches now include support of the OpenFlow
cally centralized system [7]. In fact, the need to guarantee API in their equipment. The SDN momentum was strong
adequate levels of performance, scalability, and reliability enough to make Google, Facebook, Yahoo, Microsoft,
would preclude such a solution. Instead, production-level Verizon, and Deutsche Telekom fund Open Networking
SDN network designs resort to physically distributed con- Foundation (ONF) [10] with the main goal of promotion
trol planes [7], [8]. and adoption of SDN through open standards develop-
The separation of the control plane and the data plane ment. As the initial concerns with SDN scalability were
can be realized by means of a well-defined programming addressed [11]Vin particular the myth that logical cen-
interface between the switches and the SDN controller. tralization implied a physically centralized controller, an
The controller exercises direct control over the state in the issue we will return to later onVSDN ideas have matured
data plane elements via this well-defined application prog- and evolved from an academic exercise to a commercial
ramming interface (API), as depicted in Fig. 1. The most success. Google, for example, has deployed an SDN to
notable example of such an API is OpenFlow [9], [10]. An interconnect its data centers across the globe. This pro-
OpenFlow switch has one or more tables of packet- duction network has been in deployment for three years,
helping the company to improve operational efficiency
1
We will use these two terms interchangeably. and significantly reduce costs [8]. VMware’s network

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 15

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

virtualization platform, NSX [12], is another example. guages, and interfaces. They also fall short on the analysis
NSX is a commercial solution that delivers a fully func- of cross-layer issues such as scalability, security, and de-
tional network in software, provisioned independent of the pendability. A more complete overview of ongoing re-
underlying networking devices, entirely based around search efforts, challenges, and related standardization
SDN principles. As a final example, the world’s largest IT activities is also missing.
companies (from carriers and equipment manufacturers to In this paper, we present, to the best of our knowledge,
cloud providers and financial services companies) have the most comprehensive literature survey on SDN to date.
recently joined SDN consortia such as the ONF and the We organize this survey as depicted in Fig. 2. We start, in
OpenDaylight initiative [13], another indication of the the next two sections, by explaining the context, introduc-
importance of SDN from an industrial perspective. ing the motivation for SDN and explaining the main
A few recent papers have surveyed specific architectu- concepts of this new paradigm and how it differs from
ral aspects of SDN [14]–[16]. An overview of OpenFlow traditional networking. Our aim in the early part of the
and a short literature review can be found in [14] and [15]. survey is also to explain that SDN is not as novel as a
These OpenFlow-oriented surveys present a relatively technological advance. Indeed, its existence is rooted at
simplified three-layer stack composed of high-level net- the intersection of a series of ‘‘old’’ ideas, technology driv-
work services, controllers, and the controller/switch inter- ers, and current and future needs. The concepts underly-
face. In [16], Jarraya et al. go a step further by proposing a ing SDNVthe separation of the control and data planes,
taxonomy for SDN. However, similarly to the previous the flow abstraction upon which forwarding decisions are
works, the survey is limited in terms of scope, and it does made, the (logical) centralization of network control, and
not provide an in-depth treatment of fundamental aspects the ability to program the networkVare not novel by
of SDN. In essence, existing surveys lack a thorough dis- themselves [17]. However, the integration of already tested
cussion of the essential building blocks of an SDN such as concepts with recent trends in networkingVnamely the
the network operating systems (NOSs), programming lan- availability of merchant switch silicon and the huge

Fig. 2. Condensed overview of this survey on SDN.

16 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

based tools [18], used to remotely monitor and configure the

control functionality. Network policy is defined in the man-
agement plane, the control plane enforces the policy, and
the data plane executes it by forwarding data accordingly.
In traditional IP networks, the control and data planes
are tightly coupled, embedded in the same networking
devices, and the whole structure is highly decentralized.
This was considered important for the design of the Inter-
net in the early days: it seemed the best way to guarantee
network resilience, which was a crucial design goal. In
fact, this approach has been quite effective in terms of
network performance, with a rapid increase of line rate
and port densities.
Fig. 3. Layered view of networking functionality. However, the outcome is a very complex and relatively
static architecture, as has been often reported in the net-
working literature (e.g., [1]–[3], [6], and [19]). It is also
interest in feasible forms of network virtualizationVare the fundamental reason why traditional networks are rigid,
leading to this paradigm shift in networking. As a result of and complex to manage and control. These two character-
the high industry interest and the potential to change the istics are largely responsible for a vertically integrated in-
status quo of networking from multiple perspectives, a dustry where innovation is difficult.
number of standardization efforts around SDN are ongo- Network misconfigurations and related errors are ex-
ing, as we also discuss in Section III. tremely common in today’s networks. For instance, more
Section IV is the core of this survey, presenting an than 1000 configuration errors have been observed in
extensive and comprehensive analysis of the building border gateway protocol (BGP) routers [20]. From a single
blocks of an SDN infrastructure using a bottom-up, layered misconfigured device, very undesired network behavior
approach. The option for a layered approach is grounded may result (including, among others, packet losses, for-
on the fact that SDN allows thinking of networking along warding loops, setting up of unintended paths, or service
two fundamental concepts, which are common in other contract violations). Indeed, while rare, a single miscon-
disciplines of computer science: separation of concerns figured router is able to compromise the correct operation
(leveraging the concept of abstraction) and recursion. Our of the whole Internet for hours [21], [22].
layered, bottom-up approach divides the networking prob- To support network management, a small number of
lem into eight parts: 1) hardware infrastructure; 2) south- vendors offer proprietary solutions of specialized hard-
bound interfaces; 3) network virtualization (hypervisor ware, operating systems, and control programs (network
layer between the forwarding devices and the NOSs); applications). Network operators have to acquire and
4) NOSs (SDN controllers and control platforms); maintain different management solutions and the corre-
5) northbound interfaces (to offer a common programming sponding specialized teams. The capital and operational
abstraction to the upper layers, mainly the network appli- cost of building and maintaining a networking infrastruc-
cations); 6) virtualization using slicing techniques provid- ture is significant, with long return on investment cycles,
ed by special purpose libraries or programming languages which hamper innovation and addition of new features and
and compilers; 7) network programming languages; and services (for instance, access control, load balancing,
finally 8) network applications. In addition, we also look at energy efficiency, traffic engineering). To alleviate the lack
cross-layer problems such as debugging and troubleshoot- of in-path functionalities within the network, a myriad of
ing mechanisms. The discussion in Section V on ongoing specialized components and middleboxes, such as fire-
research efforts, challenges, future work, and opportuni- walls, intrusion detection systems, and deep packet inspec-
ties concludes this paper. tion engines, proliferate in current networks. A recent
survey of 57 enterprise networks shows that the number of
middleboxes is already on par with the number of routers
II . STAT US QUO IN NET WORKING in current networks [23]. Despite helping in-path func-
Computer networks can be divided in three planes of func- tionalities, the net effect of middleboxes has increased
tionality: the data, control, and management planes (see complexity of network design and its operation.
Fig. 3). The data plane corresponds to the networking de-
vices, which are responsible for (efficiently) forwarding
data. The control plane represents the protocols used to III . WHAT IS SOFTWARE-DEFINED
populate the forwarding tables of the data plane elements. NE T WORKI NG?
The management plane includes the software services, The term SDN was originally coined to represent the ideas
such as simple network management protocol (SNMP)- and work around OpenFlow at Stanford University,

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 17

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Stanford, CA, USA [24]. As originally defined, SDN refers and information technology, being already an ubiquitous
to a network architecture where the forwarding state in the feature of many computer architectures and systems [28].
data plane is managed by a remotely controlled plane de- Ideally, the forwarding abstraction should allow any
coupled from the former. The networking industry has on forwarding behavior desired by the network application
many occasions shifted from this original view of SDN by (the control program) while hiding details of the under-
referring to anything that involves software as being SDN. lying hardware. OpenFlow is one realization of such ab-
We therefore attempt, in this section, to provide a much straction, which can be seen as the equivalent to a ‘‘device
less ambiguous definition of SDN. driver’’ in an operating system.
We define an SDN as a network architecture with four The distribution abstraction should shield SDN appli-
pillars. cations from the vagaries of distributed state, making the
1) The control and data planes are decoupled. Con- distributed control problem a logically centralized one. Its
trol functionality is removed from network devices realization requires a common distribution layer, which in
that will become simple (packet) forwarding SDN resides in the NOS. This layer has two essential
elements. functions. First, it is responsible for installing the control
2) Forwarding decisions are flow based, instead of commands on the forwarding devices. Second, it collects
destination based. A flow is broadly defined by a status information about the forwarding layer (network
set of packet field values acting as a match (filter) devices and links), to offer a global network view to net-
criterion and a set of actions (instructions). In the work applications.
SDN/OpenFlow context, a flow is a sequence of The last abstraction is specification, which should al-
packets between a source and a destination. All low a network application to express the desired network
packets of a flow receive identical service policies behavior without being responsible for implementing that
at the forwarding devices [25], [26]. The flow behavior itself. This can be achieved through virtualization
abstraction allows unifying the behavior of differ- solutions, as well as network programming languages.
ent types of network devices, including routers, These approaches map the abstract configurations that the
switches, firewalls, and middleboxes [27]. Flow applications express based on a simplified, abstract model
programming enables unprecedented flexibility, of the network, into a physical configuration for the global
limited only to the capabilities of the implemen- network view exposed by the SDN controller. Fig. 4 de-
ted flow tables [9]. picts the SDN architecture, concepts, and building blocks.
3) Control logic is moved to an external entity, the As previously mentioned, the strong coupling between
so-called SDN controller or NOS. The NOS is a control and data planes has made it difficult to add new
software platform that runs on commodity server functionality to traditional networks, a fact illustrated in
technology and provides the essential resources Fig. 5. The coupling of the control and data planes (and its
and abstractions to facilitate the programming of physical embedding in the network elements) makes the
forwarding devices based on a logically central- development and deployment of new networking features
ized, abstract network view. Its purpose is there-
fore similar to that of a traditional operating system.
4) The network is programmable through software
applications running on top of the NOS that in-
teracts with the underlying data plane devices.
This is a fundamental characteristic of SDN, con-
sidered as its main value proposition.
Note that the logical centralization of the control logic,
in particular, offers several additional benefits. First, it is
simpler and less error prone to modify network policies
through high-level languages and software components,
compared with low-level device specific configurations.
Second, a control program can automatically react to
spurious changes of the network state and thus maintain
the high-level policies intact. Third, the centralization of
the control logic in a controller with global knowledge of
the network state simplifies the development of more so-
phisticated networking functions, services, and applications.
Following the SDN concept introduced in [5], an SDN
can be defined by three fundamental abstractions: for-
warding, distribution, and specification. In fact, abstrac-
tions are essential tools of research in computer science Fig. 4. SDN architecture and its fundamental abstractions.

18 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

lancing and routing applications can be combined

sequentially, with load balancing decisions having
precedence over routing policies.

A. Terminology
To identify the different elements of an SDN as un-
equivocally as possible, we now present the essential
terminology used throughout this work.

1) Forwarding Devices (FD): These are hardware- or

software-based data plane devices that perform a set of
elementary operations. The forwarding devices have well-
defined instruction sets (e.g., flow rules) used to take ac-
tions on the incoming packets (e.g., forward to specific
ports, drop, forward to the controller, rewrite some
header). These instructions are defined by southbound
interfaces (e.g., OpenFlow [9], ForCES [30], protocol-
oblivious forwarding (POF) [31]) and are installed in the
forwarding devices by the SDN controllers implementing
the southbound protocols.

2) Data Plane (DP): Forwarding devices are intercon-

Fig. 5. Traditional networking versus SDN. With SDN, management nected through wireless radio channels or wired cables.
becomes simpler and middleboxes services can be delivered as The network infrastructure comprises the interconnected
SDN controller applications.
forwarding devices, which represent the data plane.

3) Southbound Interface (SI): The instruction set of the

(e.g., routing algorithms) very difficult, since it would forwarding devices is defined by the southbound API,
imply a modification of the control plane of all network which is part of the southbound interface. Furthermore,
devicesVthrough the installation of new firmware and, in the SI also defines the communication protocol between
some cases, hardware upgrades. Hence, the new network- forwarding devices and control plane elements. This pro-
ing features are commonly introduced via expensive, spe- tocol formalizes the way the control and data plane ele-
cialized, and hard-to-configure equipment (also known as ments interact.
middleboxes) such as load balancers, intrusion detection
systems (IDSs), and firewalls, among others. These mid- 4) Control Plane (CP): Forwarding devices are prog-
dleboxes need to be placed strategically in the network, rammed by control plane elements through well-defined
making it even harder to later change the network topo- SI embodiments. The control plane can therefore be seen
logy, configuration, and functionality. as the ‘‘network brain.’’ All control logic rests in the appli-
In contrast, SDN decouples the control plane from the cations and controllers, which form the control plane.
network devices and becomes an external entity: the NOS
or SDN controller. This approach has several advantages. 5) Northbound Interface (NI): The NOS can offer an API
• It becomes easier to program these applications to application developers. This API represents a north-
since the abstractions provided by the control plat- bound interface, i.e., a common interface for developing
form and/or the network programming languages applications. Typically, a northbound interface abstracts
can be shared. the low-level instruction sets used by southbound inter-
• All applications can take advantage of the same faces to program forwarding devices.
network information (the global network view),
leading (arguably) to more consistent and effective 6) Management Plane (MP): The management plane is
policy decisions, while reusing control plane soft- the set of applications that leverage the functions offered
ware modules. by the NI to implement network control and operation
• These applications can take actions (i.e., reconfig- logic. This includes applications such as routing, fire-
ure forwarding devices) from any part of the net- walls, load balancers, monitoring, and so forth. Essen-
work. There is therefore no need to devise a precise tially, a management application defines the policies,
strategy about the location of the new functionality. which are ultimately translated to southbound-specific
• The integration of different applications becomes instructions that program the behavior of the forwarding
more straightforward [29]. For instance, load ba- devices.

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 19

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

B. Alternative and Broadening Definitions [41] proposes a management plane at the same level of the
Since its inception in 2010 [24], the original Open- control plane, i.e., it classifies solutions in two categories:
Flow-centered SDN term has seen its scope broadened control logic (with control plane southbound interfaces) and
beyond architectures with a cleanly decoupled control management logic (with management plane southbound
plane interface. The definition of SDN will likely continue interfaces). In other words, the management plane can be
to broaden, driven by the industry business-oriented views seen as a control platform that accommodates traditional
on SDNVirrespective of the decoupling of the control network management services and protocols, such as SNMP
plane. In this survey, we focus on the original, ‘‘canonical’’ [18], BGP [42], path computation element communication
SDN definition based on the aforementioned key pillars protocol (PCEP) [43], and network configuration protocol
and the concept of layered abstractions. However, for the (NETCONF) [44].
sake of completeness and clarity, we acknowledge In addition to the broadening definitions above, the
alternative SDN definitions [32], as follows. term SDN is often used to define extensible network man-
agement planes (e.g., OpenStack [45]), whitebox/bare-
1) Control Plane/Broker SDN: A networking approach metal switches with open operating systems (e.g., Cumulus
that retains existing distributed control planes but offers Linux), open-source data planes (e.g., Pica8 Xorplus [46],
new APIs that allow applications to interact (bidirection- Quagga [47]), specialized programmable hardware devices
ally) with the network. An SDN controllerVoften called (e.g., NetFPGA [48]), virtualized software-based appli-
orchestration platformVacts as a broker between the ap- ances (e.g., open platform for network functions virtualiza-
plications and the network elements. This approach tion (OPNFV) [49]), in spite of lacking a decoupled control
effectively presents control plane data to the application and data plane or common interface along its API. Hybrid
and allows a certain degree of network programmability by SDN models are further discussed in Section V-G.
means of ‘‘plug-ins’’ between the orchestrator function and
network protocols. This API-driven approach corresponds C. Standardization Activities
to a hybrid model of SDN, since it enables the broker to The standardization landscape in SDN (and SDN-
manipulate and directly interact with the control planes of related issues) is already wide and is expected to keep
devices such as routers and switches. Examples of this view evolving over time. While some of the activities are being
on SDN include recent standardization efforts at the In- carried out in standard development organizations
ternet Engineering Task Force (IETF) (see Section III-C) (SDOs), other related efforts are ongoing at industrial or
and the design philosophy behind the OpenDaylight pro- community consortia (e.g., OpenDaylight, OpenStack,
ject [13] that goes beyond the OpenFlow split control mode. OPNFV), delivering results often considered candidates
for de facto standards. These results often come in the
2) Overlay SDN: This is a networking approach where form of open source implementations that have become
the (software- or hardware-based) network edge is dyna- the common strategy toward accelerating SDN and
mically programmed to manage tunnels between hyper- related cloud and networking technologies [50]. The
visors and/or network switches, introducing an overlay reason for this fragmentation is due to SDN concepts
network. In this hybrid networking approach, the distrib- spanning different areas of IT and networking, both
uted control plane providing the underlay remains un- from a network segmentation point of view (from access
touched. The centralized control plane provides a logical to core) and from a technology perspective (from optical
overlay that utilizes the underlay as a transport network. to wireless).
This flavor of SDN follows a proactive model to install the Table 1 presents a summary of the main SDOs and
overlay tunnels. The overlay tunnels usually terminate organizations contributing to the standardization of SDN,
inside virtual switches within hypervisors or in physical as well as the main outcomes produced to date.
devices acting as gateways to the existing network. This The ONF was conceived as a member-driven organi-
approach is very popular in recent data center network zation to promote the adoption of SDN through the devel-
virtualization [33], and are based on a variety of tunneling opment of the OpenFlow protocol as an open standard to
technologies (e.g., stateless transport tunneling [34], communicate control decisions to data plane devices. The
virtualized layer 2 networks (VXLAN) [35], network vir- ONF is structured in several working groups (WGs). Some
tualization using generic routing encapsulation (NVGRE) WGs are focused on either defining extensions to the
[36], locator/ID separation protocol (LISP) [37], [38], and OpenFlow protocol in general, such as the extensibility
generic network virtualization encapsulation (GENEVE) WG, or tailored to specific technological areas. Examples
[39]) [40]. of the latter include the optical transport (OT) WG, the
Recently, other attempts to define SDN in a layered ap- wireless and mobile (W&M) WG, and the northbound in-
proach have appeared [16], [41]. From a practical perspective terfaces (NBI) WG. Other WGs center their activity in
and trying to keep backward compatibility with existing providing new protocol capabilities to enhance the pro-
network management approaches, one initiative in the IRTF tocol itself, such as the architecture WG or the forwarding
Software-Defined Networking Research Group (SDNRG) abstractions (FA) WG.

20 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Table 1 Openflow Standardization Activities

Similar to how network programmability ideas have number of activities. A related body that focuses on re-
been considered by several IETF working groups (WGs) in search aspects for the evolution of the Internet, IRTF, has
the past, the present SDN trend is also influencing a created the SDNRG. This group investigates SDN from

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 21

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

various perspectives with the goal of identifying the ap- ing innovation inside the network by allowing program-
proaches that can be defined, deployed, and used in the mability, and altogether changing the network operational
near term, as well as identifying future research challenges. model through automation and a real shift to software-
In the International Telecommunications Union’s Tele- based platforms.
communication sector (ITU–T), some study groups (SGs) Finally, the mobile networking industry 3rd Genera-
have already started to develop recommendations for tion Partnership Project consortium is studying the
SDN, and a Joint Coordination Activity on SDN (JCA- management of virtualized networks, an effort aligned
SDN) has been established to coordinate the SDN stan- with the ETSI NFV architecture and, as such, likely to
dardization work. leverage from SDN.
The Broadband Forum (BBF) is working on SDN topics
through the Service Innovation & Market Requirements D. History of SDN
(SIMR) WG. The objective of the BBF is to release recom- Albeit a fairly recent concept, SDN leverages on net-
mendations for supporting SDN in multiservice broadband working ideas with a longer history [17]. In particular, it
networks, including hybrid environments where only builds on work made on programmable networks, such as
some of the network equipment is SDN enabled. active networks [81], programmable ATM networks [82],
The Metro Ethernet Forum (MEF) is approaching SDN [83], and on proposals for control and data plane separa-
with the aim of defining service orchestration with APIs tion, such as the network control point (NCP) [84] and
for existing networks. routing control platform (BCP) [85].
At the IEEE, the 802 LAN/MAN Standards Committee In order to present a historical perspective, we sum-
has recently started some activities to standardize SDN marize in Table 2 different instances of SDN-related work
capabilities on access networks based on IEEE 802 infra- prior to SDN, splitting it into five categories. Along with
structure through the P802.1CF project, for both wired the categories we defined, the second and third columns of
and wireless technologies to embrace new control the table mention past initiatives (pre-SDN, i.e., before the
interfaces. OpenFlow-based initiatives that sprung into the SDN con-
The Optical Internetworking Forum (OIF) Carrier WG cept) and recent developments that led to the definition
released a set of requirements for transport SDN. The ini- of SDN.
tial activities have as main goal to describe the features and Data plane programmability has a long history. Active
functionalities needed to support the deployment of SDN networks [81] represent one of the early attempts on
capabilities in carrier transport networks. The Open Data building new network architectures based on this concept.
Center Alliance (ODCA) is an organization working on The main idea behind active networks is for each node to
unifying data center in the migration to cloud computing have the capability to perform computations on, or modify
environments through interoperable solutions. Through the content of, packets. To this end, active networks pro-
the documentation of usage models, specifically one for pose two distinct approaches: programmable switches and
SDN, the ODCA is defining new requirements for cloud capsules. The former does not imply changes in the existing
deployment. The Alliance for Telecommunication Industry packet or cell format. It assumes that switching devices
Solutions (ATIS) created a focus group for analyzing ope- support the downloading of programs with specific in-
rational issues and opportunities associated with the prog- structions on how to process packets. The second approach,
rammable capabilities of network infrastructure. on the other hand, suggests that packets should be replaced
At the European Telecommunication Standards Insti- by tiny programs, which are encapsulated in transmission
tute (ETSI), efforts are being devoted to network function frames and executed at each node along their path.
virtualization (NFV) through a newly defined Industry ForCES [30], OpenFlow [9], and POF [31] represent
Specification Group (ISG). NFV and SDN concepts are recent approaches for designing and deploying program-
considered complementary, sharing the goal of accelerat- mable data plane devices. In a manner different from

Table 2 Summarized Overview of the History of Programmable Networks

22 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

active networks, these new proposals rely essentially on JUNOS [114], ExtremeXOS [115], and SR OS [116]. De-
modifying forwarding devices to support flow tables, spite being more specialized NOSs, targeting network de-
which can be dynamically configured by remote entities vices such as high-performance core routers, these NOSs
through simple operations such as adding, removing, or abstract the underlying hardware to the network operator,
updating flow rules, i.e., entries on the flow tables. making it easier to control the network infrastructure as
The earliest initiatives on separating data and control well as simplifying the development and deployment of
signaling date back to the 1980s and 1990s. The NCP [84] new protocols and management applications.
is probably the first attempt to separate control and data Finally, initiatives that can be seen as ‘‘technology pull’’
plane signaling. NCPs were introduced by AT&T to im- drivers are also worth recalling. Back in the 1990s, a
prove the management and control of its telephone net- movement toward open signaling [118] began to happen.
work. This change promoted a faster pace of innovation of The main motivation was to promote the wider adoption of
the network and provided new means for improving its the ideas proposed by projects such as NCP [84] and
efficiency, by taking advantage of the global view of the Tempest [96]. The open signaling movement worked to-
network provided by NCPs. Similarly, other initiatives such ward separating the control and data signaling by propos-
as Tempest [96], ForCES [30], RCP [85], and PCE [43] ing open and programmable interfaces. Curiously, a rather
proposed the separation of the control and data planes similar movement can be observed with the recent advent
for improved management in ATM, Ethernet, BGP, of OpenFlow and SDN, with the lead of the ONF [10]. This
and multiprotocol label switching (MPLS) networks, type of movement is crucial to promote open technologies
respectively. into the market, hopefully leading equipment manufactur-
More recently, initiatives such as SANE [100], Ethane ers to support open standards and thus fostering interop-
[101], OpenFlow [9], NOX [26], and POF [31] proposed erability, competition, and innovation.
the decoupling of the control and data planes for Ethernet For a more extensive intellectual history of program-
networks. Interestingly, these recent solutions do not re- mable networks and SDN, we direct the reader to the
quire significant modifications on the forwarding devices, recent paper by Feamster et al. [17].
making them attractive not only for the networking re-
search community, but even more to the networking in-
dustry. OpenFlow-based devices [9], for instance, can IV. SOFTWARE-DEFINED NETWORKS:
easily coexist with traditional Ethernet devices, enabling a BOTTOM-UP
progressive adoption (i.e., not requiring a disruptive An SDN architecture can be depicted as a composition of
change to existing networks). different layers, as shown in Fig. 6(b). Each layer has its
Network virtualization has gained a new traction with own specific functions. While some of them are always
the advent of SDN. Nevertheless, network virtualization present in an SDN deployment, such as the southbound
also has its roots back in the 1990s. The Tempest project API, NOSs, northbound API, and network applications,
[96] is one of the first initiatives to introduce network others may be present only in particular deployments, such
virtualization, by introducing the concept of switchlets in as hypervisor- or language-based virtualization.
ATM networks. The core idea was to allow multiple Fig. 6 presents a trifold perspective of SDNs. The SDN
switchlets on top of a single ATM switch, enabling multiple layers are represented in Fig. 6(b), as explained above.
independent ATM networks to share the same physical Fig. 6(a) and (c) depicts a plane-oriented view and a sys-
resources. Similarly, MBone [102] was one of the early tem design perspective, respectively.
initiatives that targeted the creation of virtual network to- The following sections introduce each layer, following
pologies on top of legacy networks, or overlay networks. a bottom-up approach. For each layer, the core properties
This work was followed by several other projects such as and concepts are explained based on the different tech-
Planet Lab [105], GENI [107], and VINI [108]. FlowVisor nologies and solutions. Additionally, debugging and trou-
[119] is also worth mentioning as one of the first recent bleshooting techniques and tools are discussed.
initiatives to promote a hypervisor-like virtualization ar-
chitecture for network infrastructures, resembling the A. Layer I: Infrastructure
hypervisor model common for compute and storage. More An SDN infrastructure, similarly to a traditional net-
recently, Koponen et al. proposed a network virtualization work, is composed of a set of networking equipment
platform (NVP) [112] for multitenant data centers using (switches, routers, and middlebox appliances). The main
SDN as a base technology. difference resides in the fact that those traditional physical
The concept of a NOS was reborn with the introduction devices are now simple forwarding elements without em-
of OpenFlow-based NOSs, such as NOX [26], Onix [7], and bedded control or software to take autonomous decisions.
ONOS [117]. Indeed, NOSs have been in existence for The network intelligence is removed from the data plane
decades. One of the most widely known and deployed is devices to a logically centralized control system, i.e., the
the Cisco IOS [113], which was originally conceived back NOS and applications, as shown in Fig. 6(c). More impor-
in the early 1990s. Other NOSs worth mentioning are tantly, these new networks are built (conceptually) on top

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 23

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Fig. 6. Software-Defined Networks in (a) planes, (b) layers, and (c) system design architecture.

of open and standard interfaces (e.g., OpenFlow), a crucial OpenFlow is currently the most widespread design of SDN
approach for ensuring configuration and communication data plane devices. Nevertheless, other specifications of
compatibility and interoperability among different data SDN-enabled forwarding devices are being pursued,
and control plane devices. In other words, these open in- including POF [31], [120] and the negotiable datapath
terfaces enable controller entities to dynamically program models (NDMs) from the ONF Forwarding Abstractions
heterogeneous forwarding devices, something difficult in Working Group (FAWG) [121].
traditional networks, due to the large variety of proprietary Inside an OpenFlow device, a path through a sequence
and closed interfaces and the distributed nature of the of flow tables defines how packets should be handled.
control plane. When a new packet arrives, the lookup process starts in the
In an SDN/OpenFlow architecture, there are two main first table and ends either with a match in one of the tables
elements, the controllers and the forwarding devices, as of the pipeline or with a miss (when no rule is found for
shown in Fig. 7. A data plane device is a hardware or that packet). A flow rule can be defined by combining
software element specialized in packet forwarding, while a different matching fields, as illustrated in Fig. 7. If there is
controller is a software stack (the ‘‘network brain’’) run- no default rule, the packet will be discarded. However,
ning on a commodity hardware platform. An OpenFlow- the common case is to install a default rule which tells
enabled forwarding device is based on a pipeline of flow the switch to send the packet to the controller (or to the
tables where each entry of a flow table has three parts: 1) a normal non-OpenFlow pipeline of the switch). The
matching rule; 2) actions to be executed on matching priority of the rules follows the natural sequence number
packets; and 3) counters that keep statistics of matching of the tables and the row order in a flow table. Possible
packets. This high-level and simplified model derived from actions include: 1) forward the packet to outgoing port(s);

Fig. 7. OpenFlow-enabled SDN devices.

24 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Table 3 Different Match Fields, Statistics, and Capabilities Have Been Added on Each Openflow Protocol Revision. The Number of
Required (REQ) and Optional (OPT) Capabilities Has Grown Considerably

2) encapsulate it and forward it to the controller; 3) drop it; optimized TCAM memory that supports from 125 000 up
4) send it to the normal processing pipeline; and 5) send it to to 1 000 000 flow table entries [124]. This is a clear sign
the next flow table or to special tables, such as group or that the size of the flow tables is growing at a pace aiming
metering tables introduced in the latest OpenFlow protocol. to meet the needs of future SDN deployments. Networking
As detailed in Table 3, each version of the OpenFlow hardware manufacturers have produced various kinds of
specification introduced new match fields including OpenFlow-enabled devices, as is shown in Table 4. These
Ethernet, IPv4/v6, MPLS, TCP/UDP, etc. However, only devices range from equipment for small businesses (e.g.,
a subset of those matching fields are mandatory to be GbE switches) to high-class data center equipment (e.g.,
compliant to a given protocol version. Similarly, many ac- high-density switch chassis with up to 100GbE connectiv-
tions and port types are optional features. Flow match ity for edge-to-core applications, with tens of terabits per
rules can be based on almost arbitrary combinations of bits second of switching capacity).
of the different packet headers using bit masks for each Software switches are emerging as one of the most
field. Adding new matching fields has been eased with the promising solutions for data centers and virtualized net-
extensibility capabilities introduced in OpenFlow work infrastructures [147]–[149]. Examples of software-
version 1.2 through an OpenFlow Extensible Match based OpenFlow switch implementations include Switch
(OXM) based on type-length-value (TLV) structures. To Light [145], ofsoftswitch13 [141], Open vSwitch [142],
improve the overall protocol extensibility, with OpenFlow OpenFlow Reference [143], Pica8 [150], Pantou [146], and
version 1.4, TLV structures have been also added to ports, XorPlus [46]. Recent reports show that the number of vir-
tables, and queues in replacement of the hard-coded tual access ports is already larger than physical access ports
counterparts of earlier protocol versions. on data centers [149]. Network virtualization has been one
of the drivers behind this trend. Software switches such as
1) Overview of Available OpenFlow Devices: Several Open vSwitch have been used for moving network func-
OpenFlow-enabled forwarding devices are available on tions to the edge (with the core performing traditional IP
the market, both as commercial and open source products forwarding), thus enabling network virtualization [112].
(see Table 4). There are many off-the-shelf, ready to de- An interesting observation is the number of small,
ploy, OpenFlow switches and routers, among other ap- startup enterprises devoted to SDN, such as Big Switch,
pliances. Most of the switches available on the market have Pica8, Cyan, Plexxi, and NoviFlow. This seems to imply
relatively small ternary content-addressable memory that SDN is springing a more competitive and open net-
(TCAMs), with up to 8000 entries. Nonetheless, this is working market, one of its original goals. Other effects of
changing at a fast pace. Some of the latest devices released this openness triggered by SDN include the emergence of
in the market go far beyond that figure. Gigabit Ethernet so-called ‘‘bare metal switches’’ or ‘‘whitebox switches,’’
(GbE) switches for common business purposes are already where software and hardware are sold separately and the end
supporting up to 32 000 Layer 2 (L2) + Layer 3 (L3) or user is free to load an operating system of its choice [151].
64 000 L2/L3 exact match flows [122]. Enterprise class
10GbE switches are being delivered with more than B. Layer II: Southbound Interfaces
80 000 layer 2 flow entries [123]. Other switching devices Southbound interfaces (or southbound APIs) are the
using high-performance chips (e.g., EZchip NP-4) provide connecting bridges between control and forwarding

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 25

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Table 4 OpenFlow Enabled Hardware and Software Devices

elements, thus being the crucial instrument for clearly collected by the controller. Third, packet-in messages are
separating control and data plane functionality. However, sent by forwarding devices to the controller when they do
these APIs are still tightly tied to the forwarding elements not known what to do with a new incoming flow or
of the underlying physical or virtual infrastructure. because there is an explicit ‘‘send to controller’’ action in
Typically, a new switch can take two years to be ready the matched entry of the flow table. These information
for commercialization if built from scratch, with upgrade channels are the essential means to provide flow-level
cycles that can take up to nine months. The software de- information to the NOS.
velopment for a new product can take from six months to Albeit the most visible, OpenFlow is not the only
one year [152]. The initial investment is high and risky. As a available southbound interface for SDN. There are other
central component of its design, the southbound APIs API proposals such as ForCES [30], Open vSwitch
represent one of the major barriers for the introduction and Database (OVSDB) [153], POF [31], [120], OpFlex [154],
acceptance of any new networking technology. In this light, OpenState [155], revised open-flow library (ROFL) [156],
the emergence of SDN southbound API proposals such as hardware abstraction layer (HAL) [157], [158], and
OpenFlow [9] is seen as welcome by many in the industry. programmable abstraction of data path (PAD) [159].
These standards promote interoperability, allowing the de- ForCES proposes a more flexible approach to traditional
ployment of vendor-agnostic network devices. This has al- network management without changing the current archi-
ready been demonstrated by the interoperability between tecture of the network, i.e., without the need of a logically
OpenFlow-enabled equipments from different vendors. centralized external controller. The control and data
As of this writing, OpenFlow is the most widely ac- planes are separated, but can potentially be kept in the
cepted and deployed open southbound standard for SDN. same network element. However, the control part of the
It provides a common specification to implement Open- network element can be upgraded on-the-fly with third-
Flow-enabled forwarding devices, and for the commu- party firmware.
nication channel between data and control plane devices OVSDB [153] is another type of southbound API, de-
(e.g., switches and controllers). The OpenFlow protocol signed to provide advanced management capabilities for
provides three information sources for NOSs. First, event- Open vSwitches. Beyond OpenFlow’s capabilities to confi-
based messages are sent by forwarding devices to the gure the behavior of flows in a forwarding device, an Open
controller when a link or port change is triggered. Second, vSwitch offers other networking functions. For instance, it
flow statistics are generated by the forwarding devices and allows the control elements to create multiple virtual

26 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

switch instances, set quality of service (QoS) policies on HAL [157], [158] is not exactly a southbound API, but is
interfaces, attach interfaces to the switches, configure closely related. Differently from the aforementioned ap-
tunnel interfaces on OpenFlow data paths, manage queues, proaches, HAL is rather a translator that enables a south-
and collect statistics. Therefore, the OVSDB is a comple- bound API such as OpenFlow to control heterogeneous
mentary protocol to OpenFlow for Open vSwitch. hardware devices. It thus sits between the southbound API
One of the first direct competitors of OpenFlow is POF and the hardware device. Recent research experiments
[31], [120]. One of the main goals of POF is to enhance the with HAL have demonstrated the viability of SDN control
current SDN forwarding plane. With OpenFlow, switches in access networks such as Gigabit Ethernet passive optical
have to understand the protocol headers to extract the networks (GEPONs) [160] and cable networks (DOCSISs)
required bits to be matched with the flow tables entries. [161]. A similar effort to HAL is PAD [159], a proposal that
This parsing represents a significant burden for data plane goes a bit further by also working as a southbound API by
devices, in particular if we consider that OpenFlow itself. More importantly, PAD allows a more generic prog-
version 1.3 already contains more than 40 header fields. ramming of forwarding devices by enabling the control of
Besides this inherent complexity, backward compatibility data path behavior using generic byte operations, defining
issues may arise every time new header fields are included protocol headers and providing function definitions.
in or removed from the protocol. To achieve its goal, POF
proposes a generic flow instruction set (FIS) that makes C. Layer III: Network Hypervisors
the forwarding plane protocol oblivious. A forwarding Virtualization is already a consolidated technology in
element does not need to know, by itself, anything about modern computers. The fast developments of the past de-
the packet format in advance. Forwarding devices are seen cade have made virtualization of computing platforms
as white boxes with only processing and forwarding mainstream. Based on recent reports, the number of vir-
capabilities. In POF, packet parsing is a controller task that tual servers has already exceeded the number of physical
results in a sequence of generic keys and table lookup servers [162], [112].
instructions that are installed in the forwarding elements. Hypervisors enable distinct virtual machines to share
The behavior of data plane devices is, therefore, com- the same hardware resources. In a cloud infrastructure-as-
pletely under the control of the SDN controller. Similar to a-service (IaaS), each user can have its own virtual re-
a central processing unit (CPU) in a computer system, a sources, from computing to storage. This enabled new
POF switch is application and protocol agnostic. revenue and business models where users allocate re-
A recent southbound interface proposal is OpFlex sources on demand, from a shared physical infrastructure,
[154]. Contrary to OpenFlow (and similar to ForCES), one at a relatively low cost. At the same time, providers
of the ideas behind OpFlex is to distribute part of the make better use of the capacity of their installed physical
complexity of managing the network back to the forward- infrastructures, creating new revenue streams without
ing devices, with the aim of improving scalability. Similar significantly increasing their capital expenditure and
to OpenFlow, policies are logically centralized and operational expenditure (OPEX) costs. One of the
abstracted from the underlying implementation. The dif- interesting features of virtualization technologies today is
ferences between OpenFlow and OpFlex are a clear illus- the fact that virtual machines can be easily migrated from
tration of one of the important questions to be answered one physical server to another and can be created and/or
when devising a southbound interface: where to place each destroyed on demand, enabling the provisioning of elastic
piece of the overall functionality. services with flexible and easy management. Unfor-
In contrast to OpFlex and POF, OpenState [155] and tunately, virtualization has been only partially realized in
ROFL [156] do not propose a new set of instructions for practice. Despite the great advances in virtualizing com-
programming data plane devices. OpenState proposes ex- puting and storage elements, the network is still mostly
tended finite machines (stateful programming abstrac- statically configured in a box-by-box manner [33].
tions) as an extension (superset) of the OpenFlow match/ The main network requirements can be captured along
action abstraction. Finite state machines allow the imple- two dimensions: network topology and address space.
mentation of several stateful tasks inside forwarding de- Different workloads require different network topologies
vices, i.e., without augmenting the complexity or overhead and services, such as flat L2 or L3 services, or even more
of the control plane. For instance, all tasks involving only complex L4–L7 services for advanced functionality. Cur-
local state, such as media access control (MAC) learning rently, it is very difficult for a single physical topology to
operations, port knocking, or stateful edge firewalls, can be support the diverse demands of applications and services.
performed directly on the forwarding devices without any Similarly, address space is hard to change in current net-
extra control plane communication and processing delay. works. Today, virtualized workloads have to operate in the
ROFL, on the other hand, proposes an abstraction layer same address of the physical infrastructure. Therefore, it is
that hides the details of the different OpenFlow versions, hard to keep the original network configuration for a te-
thus providing a clean API for software developers, nant, virtual machines cannot migrate to arbitrary loca-
simplifying application development. tions, and the addressing scheme is fixed and hard to

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 27

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

change. For example, IPv6 cannot be used by the virtual topology, address, and control function virtualization. All
machines (VMs) of a tenant if the underlying physical these properties are necessary in multitenant environ-
forwarding devices support only IPv4. ments where virtual networks need to be managed and
To provide complete virtualization, the network should migrated according to the computing and storage virtual
provide similar properties to the computing layer [33]. The resources. Virtual network topologies have to be mapped
network infrastructure should be able to support arbitrary onto the underlying forwarding devices, with virtual
network topologies and addressing schemes. Each tenant addresses allowing tenants to completely manage their
should have the ability to configure both the computing address space without depending on the underlying net-
nodes and the network simultaneously. Host migration work elements addressing schemes.
should automatically trigger the migration of the corre- AutoSlice [179] is another SDN-based virtualization
sponding virtual network ports. One might think that long proposal. Differently from FlowVisor, it focuses on the
standing virtualization primitives such as VLANs (virtua- automation of the deployment and operation of virtual
lized L2 domain), NAT (virtualized IP address space), and SDN (vSDN) topologies with minimal mediation or arbi-
MPLS (virtualized path) are enough to provide full and tration by the substrate network operator. Additionally,
automated network virtualization. However, these tech- AutoSlice targets also scalability aspects of network hyper-
nologies are anchored on a box-by-box basis configuration, visors by optimizing resource utilization and by mitigating
i.e., there is no single unifying abstraction that can be the flow-table limitations through a precise monitoring of
leveraged to configure (or reconfigure) the network in a the flow traffic statistics. Similarly to AutoSlice, AutoV-
global manner. As a consequence, current network provi- Flow [174] also enables multidomain network virtualiza-
sioning can take months, while computing provisioning tion. However, instead of having a single third party to
takes only minutes [112], [163]–[165]. control the mapping of vSDN topologies, as is the case of
There is hope that this situation will change with SDN AutoSlice, AutoVFlow uses a multiproxy architecture that
and the availability of new tunneling techniques (e.g., allows network owners to implement flow space virtuali-
VXLAN [35] and NVGRE [36]). For instance, solutions zation in an autonomous way by exchanging information
such as FlowVisor [111], [166], [167], FlowN [168], NVP among the different domains.
[112], OpenVirteX [169], [170], IBM SDN VE [171], [172], FlowN [168], [180] is based on a slightly different
RadioVisor [173], AutoVFlow [174], eXtensible Datapath concept. Whereas FlowVisor can be compared to a full vir-
Daemon (xDPd) [175], [176], optical transport network tualization technology, FlowN is analogous to a container-
virtualization [177], and version-agnostic OpenFlow slicing based virtualization, i.e., a lightweight virtualization
mechanisms [178], have been recently proposed, eval- approach. FlowN was also primarily conceived to address
uated, and deployed in real scenarios for on-demand pro- multitenancy in the context of cloud platforms. It is de-
visioning of virtual networks. signed to be scalable and allows a unique shared controller
platform to be used for managing multiple domains in a
1) Slicing the Network: FlowVisor is one of the early cloud environment. Each tenant has full control over its
technologies to virtualize an SDN. Its basic idea is to allow virtual networks and is free to deploy any network ab-
multiple logical networks share the same OpenFlow net- straction and application on top of the controller platform.
working infrastructure. For this purpose, it provides an The compositional SDN hypervisor [181] was designed
abstraction layer that makes it easier to slice a data plane with a different set of goals. Its main objective is to allow
based on off-the-shelf OpenFlow-enabled switches, allow- the cooperative (sequential or parallel) execution of appli-
ing multiple and diverse networks to coexist. Five slicing cations developed with different programming languages
dimensions are considered in FlowVisor: bandwidth, topo- or conceived for diverse control platforms. It thus offers
logy, traffic, device CPU, and forwarding tables. Moreover, interoperability and portability in addition to the typical
each network slice supports a controller, i.e., multiple functions of network hypervisors.
controllers can coexist on top of the same physical network
infrastructure. Each controller is allowed to act only on its 2) Commercial Multitenant Network Hypervisors: None of
own network slice. In general terms, a slice is defined as a the aforementioned approaches is designed to address all
particular set of flows on the data plane. From a system challenges of multitenant data centers. For instance, te-
design perspective, FlowVisor is a transparent proxy that nants want to be able to migrate their enterprise solutions
intercepts OpenFlow messages between switches and con- to cloud providers without the need to modify the network
trollers. It partitions the link bandwidth and flow tables of configuration of their home network. Existing networking
each switch. Each slice receives a minimum data rate, and technologies and migration strategies have mostly failed to
each guest controller gets its own virtual flow table in the meet both tenant and service provider requirements. A
switches. multitenant environment should be anchored in a network
Similarly to FlowVisor, OpenVirteX [169], [170] acts as hypervisor capable of abstracting the underlaying forward-
a proxy between the NOS and the forwarding devices. ing devices and physical network topology from the te-
However, its main goal is to provide virtual SDNs through nants. Moreover, each tenant should have access to control

28 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

abstractions and manage its own virtual networks inde- memory), and provide security protection mechanisms.
pendently and isolated from other tenants. These functionalities and resources are key enablers for
With the market demand for network virtualization increased productivity, making the life of system and ap-
and the recent research on SDN showing promise as an plication developers easier. Their widespread use has sig-
enabling technology, different commercial virtualization nificantly contributed to the evolution of various
platforms based on SDN concepts have started to appear. ecosystems (e.g., programming languages) and the devel-
VMWare has proposed a network virtualization platform opment of a myriad of applications.
(NVP) [112] that provides the necessary abstractions to In contrast, networks have so far been managed and
allow the creation of independent virtual networks for configured using lower level, device-specific instruction
large-scale multitenant environments. NVP is a complete sets and mostly closed proprietary NOSs (e.g., Cisco IOS
network virtualization solution that allows the creation of and Juniper JunOS). Moreover, the idea of operating sys-
virtual networks, each with independent service model, tems abstracting device-specific characteristics and provid-
topologies, and addressing architectures over the same ing, in a transparent way, common functionalities is still
physical network. With NVP, tenants do not need to know mostly absent in networks. For instance, today designers of
anything about the underlying network topology, config- routing protocols need to deal with complicated distrib-
uration, or other specific aspects of the forwarding devices. uted algorithms when solving networking problems. Net-
NVP’s network hypervisor translates the tenants config- work practitioners have therefore been solving the same
urations and requirements into low-level instruction sets problems over and over again.
to be installed on the forwarding devices. For this purpose, SDN is promised to facilitate network management and
the platform uses a cluster of SDN controllers to mani- ease the burden of solving networking problems by means
pulate the forwarding tables of the Open vSwitches in the of the logically centralized control offered by a NOS [26].
host’s hypervisor. Forwarding decisions are therefore made As with traditional operating systems, the crucial value of a
exclusively on the network edge. After the decision is NOS is to provide abstractions, essential services, and
made, the packet is tunneled over the physical network to common APIs to developers. Generic functionality as net-
the receiving host hypervisor (the physical network sees work state and network topology information, device dis-
nothing but ordinary IP packets). covery, and distribution of network configuration can be
IBM has also recently proposed SDN VE [171], [172], provided as services of the NOS. With NOSs, to define
another commercial and enterprise-class network virtualiza- network policies a developer no longer needs to care about
tion platform. SDN VE uses OpenDaylight as one of the the low-level details of data distribution among routing
building blocks of the so-called software-defined environ- elements, for instance. Such systems can arguably create a
ments (SDEs), a trend further discussed in Section V. This new environment capable of fostering innovation at a
solution also offers a complete implementation framework faster pace by reducing the inherent complexity of creating
for network virtualization. Like NVP, it uses a host-based new network protocols and network applications.
overlay approach, achieving advanced network abstraction A NOS (or a controller) is a critical element in an SDN
that enables application-level network services in large-scale architecture as it is the key supporting piece for the control
multitenant environments. Interestingly, SDN VE 1.0 is logic (applications) to generate the network configuration
capable of supporting in one single instantiation up to 16 000 based on the policies defined by the network operator.
virtual networks and 128 000 virtual machines [171], [172]. Similar to a traditional operating system, the control plat-
To summarize, currently there are already a few net- form abstracts the lower level details of connecting and
work hypervisor proposals leveraging the advances of interacting with forwarding devices (i.e., of materializing
SDN. There are, however, still several issues to be ad- the network policies).
dressed. These include, among others, the improvement of
virtual-to-physical mapping techniques [182], the defini- 1) Architecture and Design Axes: There is a very diverse
tion of the level of detail that should be exposed at the set of controllers and control platforms with different de-
logical level, and the support for nested virtualization [29]. sign and architectural choices [7], [13], [183]–[186]. Exist-
We anticipate, however, this ecosystem to expand in the ing controllers can be categorized based on many aspects.
near future since network virtualization will most likely From an architectural point of view, one of the most rele-
play a key role in future virtualized environments, simi- vant is if they are centralized or distributed. This is one of
larly to the expansion we have been witnessing in virtual- the key design axes of SDN control platforms, so we start
ized computing. by discussing this aspect next.

D. Layer IV: Network Operating Systems/Controllers 2) Centralized Versus Distributed: A centralized control-
Traditional operating systems provide abstractions ler is a single entity that manages all forwarding devices of
(e.g., high-level programming APIs) for accessing lower the network. Naturally, it represents a single point of fail-
level devices, manage the concurrent access to the under- ure and may have scaling limitations. A single controller
lying resources (e.g., hard drive, network adapter, CPU, may not be enough to manage a network with a large

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 29

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

number of data plane elements. Centralized controllers consistency semantics, which means that data updates on
such as NOX–MT [187], Maestro [188], Beacon [186], and distinct nodes will eventually be updated on all controller
Floodlight [189] have been designed as highly concurrent nodes. This implies that there is a period of time in which
systems, to achieve the throughput required by enterprise distinct nodes may read different values (old value or new
class networks and data centers. These controllers are value) for the same property. Strong consistency, on the
based on multithreaded designs to explore the parallelism other hand, ensures that all controller nodes will read the
of multicore computer architectures. As an example, most updated property value after a write operation. De-
Beacon can deal with more than 12 million flows per sec- spite its impact on system performance, strong consistency
ond by using large size computing nodes of cloud providers offers a simpler interface to application developers. To
such as Amazon [186]. Other centralized controllers such date, only Onix, ONOS, and SMaRtLight provide this data
as Trema [190], Ryu NOS [191], Meridian [192], and consistency model.
ProgrammableFlow [133], [193] target specific environ- Another common property of distributed controllers is
ments such as data centers, cloud infrastructures, and fault tolerance. When one node fails, another neighbor
carrier grade networks. Furthermore, controllers such as node should take over the duties and devices of the failed
Rosemary [194] offer specific functionality and guarantees, node. So far, despite some controllers tolerating crash
namely security and isolation of applications. By using a failures, they do not tolerate arbitrary failures, which
container-based architecture called micro-NOS, it achieves means that any node with an abnormal behavior will not be
its primary goal of isolating applications and preventing replaced by a potentially well-behaved one.
the propagation of failures throughout the SDN stack. A single controller may be enough to manage a small
Contrary to a centralized design, a distributed NOS can network, however it represents a single point of failure.
be scaled up to meet the requirements of potentially any Similarly, independent controllers can be spread across the
environment, from small- to large-scale networks. A dis- network, each of them managing a network segment, re-
tributed controller can be a centralized cluster of nodes or ducing the impact of a single controller failure. Yet, if the
a physically distributed set of elements. While the first control plane availability is critical, a cluster of controllers can
alternative can offer high throughput for very dense data be used to achieve a higher degree of availability and/or for
centers, the latter can be more resilient to different kinds supporting more devices. Ultimately, a distributed controller
of logical and physical failures. A cloud provider that spans can improve the control plane resilience and scalability and
multiple data centers interconnected by a wide area net- reduce the impact of problems caused by network partition,
work may require a hybrid approach, with clusters of con- for instance. SDN resiliency as a whole is an open challenge
trollers inside each data center and distributed controller that will be further discussed in Section V-C.
nodes in the different sites [8].
Onix [7], HyperFlow [195], HP VAN SDN [184], 3) Dissecting SDN Controller Platforms: To provide a bet-
ONOS [117], DISCO [185], yanc [196], PANE [197], ter architectural overview and understanding the design of
SMaRt-Light [198], and Fleet [199] are examples of distri- a NOS, Table 5 summarizes some of the most relevant
buted controllers. Most distributed controllers offer weak architectural and design properties of SDN controllers and

Table 5 Architecture and Design Elements of Control Platforms

30 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Fig. 8. SDN control platforms: elements, services, and interfaces.

control platforms. We have focused on the elements, ser- 4) Core Controller Functions: The base network service
vices, and interfaces of a selection of production-level, functions are what we consider the essential functionality
well-documented controllers and control platforms. Each all controllers should provide. As an analogy, these func-
line in the table represents a component we consider tions are like base services of operating systems, such as
important in a modular and scalable control platform. We program execution, input/output (I/O) operations control,
observe a highly diversified environment, with different communications, protection, and so on. These services are
properties and components being used by distinct control used by other operating system level services and user
platforms. This is not surprising, given an environment applications. In a similar way, functions such as topology,
with many competitors willing to be at the forefront of statistics, notifications, and device management, together
SDN development. Note also that not all components are with shortest path forwarding and security mechanisms,
available on all platforms. For instance, east/westbound are essential network control functionalities that network
APIs are not required in centralized controllers such as applications may use in building its logic. For instance, the
Beacon. In fact, some platforms have very specific niche notification manager should be able to receive, process,
markets, such as telecom companies and cloud providers, and forward events (e.g., alarm notifications, security
so the requirements will be different. alarms, state changes) [55]. Security mechanisms are
Based on the analysis of the different SDN controllers another example, as they are critical components to pro-
proposed to date (both those presented in Table 5 and vide basic isolation and security enforcement between
others, such as NOX [26], Meridian [192], ForCES [30], services and applications. For instance, rules generated by
and FortNOX [201]), we extract several common elements high priority services should not be overwritten with rules
and provide a first attempt to clearly and systematically created by applications with a lower priority.
dissect an SDN control platform in Fig. 8.
There are at least three relatively well-defined layers in 5) Southbound: On the lower level of control platforms,
most of the existing control platforms: 1) the application, the southbound APIs can be seen as a layer of device driv-
orchestration, and services; 2) the core controller func- ers. They provide a common interface for the upper layers,
tions; and 3) the elements for southbound communica- while allowing a control platform to use different south-
tions. The connection at the upper level layers is based on bound APIs (e.g., OpenFlow, OVSDB, and ForCES) and
northbound interfaces such as REST APIs [202] and prog- protocol plug-ins to manage existing or new physical or
ramming languages such as FML [203], Frenetic [204], and virtual devices (e.g., SNMP, BGP, and NetConf). This is
NetCore [205]. On the lower level part of a control essential both for backward compatibility and heteroge-
platform, southbound APIs and protocol plug-ins interface neity, i.e., to allow multiple protocols and device
the forwarding elements. The core of a controller platform management connectors. Therefore, on the data plane, a
can be characterized as a combination of its base network mix of physical devices, virtual devices (e.g., Open vSwitch
service functions and the various interfaces. [109], [142], vRouter [206]) and a variety of device

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 31

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Fig. 9. Distributed controllers: east/westbound APIs.

interfaces (e.g., OpenFlow, OVSDB, of-config [207], increases the system robustness by reducing the probabil-
NetConf, and SNMP) can coexist. ity of common faults, such as software faults [210].
Most controllers support only OpenFlow as a south- Other proposals that try to define interfaces between
bound API. Still, a few of them, such as OpenDaylight, controllers include Onix data import/export functions [7],
Onix, and HP VAN SDN Controller, offer a wider range of ForCES CE–CE interface [30], [211], ForCES Intra-NE cold-
southbound APIs and/or protocol plug-ins. Onix supports standby mechanisms for high availability [212], and distrib-
both the OpenFlow and OVSDB protocols. The HP VAN uted data stores [213]. An east/westbound API requires
SDN Controller has other southbound connectors such as advanced data distribution mechanisms such as the advanced
L2 and L3 agents. message queuing protocol (AMQP) [214] used by DISCO
OpenDaylight goes a step beyond by providing a service [185], techniques for distributed concurrent and consistent
layer abstraction (SLA) that allows several southbound policy composition [215], transactional databases and DHTs
APIs and protocols to coexist in the control platform. For [216] (as used in Onix [7]), or advanced algorithms for strong
instance, its original architecture was designed to support consistency and fault tolerance [198], [213].
at least seven different protocols and plug-ins: OpenFlow, In a multidomain setup, east/westbound APIs may re-
OVSDB [153], NETCONF [44], PCEP [43], SNMP [208], quire also more specific communication protocols between
BGP [42], and LISP Flow Mapping [13]. Hence, OpenDay- SDN domain controllers [217]. Some of the essential func-
light is one of the few control platforms being conceived to tions of such protocols are to coordinate flow setup origi-
support a broader integration of technologies in a single nated by applications, exchange reachability information
control platform. to facilitate inter-SDN routing, reachability update to keep
the network state consistent, among others.
6) Eastbound and Westbound: East/westbound APIs, as Another important issue regarding east/westbound in-
illustrated in Fig. 9, are a special case of interfaces required terfaces is heterogeneity. For instance, besides communi-
by distributed controllers. Currently, each controller imple- cating with peer SDN controllers, controllers may also
ments its own east/westbound API. The functions of these need to communicate with subordinate controllers (in a
interfaces include import/export data between controllers, hierarchy of controllers) and non-SDN controllers [218],
algorithms for data consistency models, and monitoring/ as is the case of Closed-Flow [219]. To be interoperable,
notification capabilities (e.g., check if a controller is up or east/westbound interfaces thus need to accommodate dif-
notify a take over on a set of forwarding devices). ferent controller interfaces, with their specific set of ser-
Similarly to southbound and northbound interfaces, vices, and the diverse characteristics of the underlying
east/westbound APIs are essential components of distrib- infrastructure, including the diversity of technology, the
uted controllers. To identify and provide common compa- geographic span and scale of the network, and the distinc-
tibility and interoperability between different controllers, tion between WAN and LANVpotentially across adminis-
it is necessary to have standard east/westbound interfaces. trative boundaries. In those cases, different information
For instance, SDNi [209] defines common requirements to has to be exchanged between controllers, including adja-
coordinate flow setup and exchange reachability informa- cency and capability discovery, topology information (to
tion across multiple domains. In essence, such protocols the extent of the agreed contracts between administrative
can be used in an orchestrated and interoperable way to domains), billing information, among many others [218].
create more scalable and dependable distributed control Last, an ‘‘SDN compass’’ methodology [220] suggests a
platforms. Interoperability can be leveraged to increase the finer distinction between eastbound and westbound
diversity of the control platform element. Indeed, diversity horizontal interfaces, referring to westbound interfaces

32 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Table 6 Controllers Classification

as SDN-to-SDN protocols and controller APIs while istics. As can be observed, most controllers are centralized
eastbound interfaces would be used to refer to standard and multithreaded. Curiously, the northbound API is very
protocols used to communicate with legacy network con- diverse. In particular, five controllers (Onix, Floodlight,
trol planes (e.g., PCEP [43] and GMPLS [221]). MuL, Meridian, and SDN Unified Controller) pay a bit
more attention to this interface, as a statement of its im-
7) Northbound: Current controllers offer a quite broad portance. Consistency models and fault tolerance are only
variety of northbound APIs, such as ad hoc APIs, RESTful present in Onix, HyperFlow, HP VAN SDN, ONOS, and
APIs [202], multilevel programming interfaces, file SMaRtLight. Last, when it comes to the OpenFlow stan-
systems, among other more specialized APIs such as dard as southbound API, only Ryu supports its three major
NVP NBAPI [7], [112] and SDMN API [222]. Section IV-E versions (v1.0, v1.2, and v1.3).
is devoted to a more detailed discussion on the evolving To conclude, it is important to emphasize that the
layer of northbound APIs. A second kind of northbound control platform is one of the critical points for the success
interfaces are those stemming out of SDN programming of SDN [234]. One of the main issues that needs to be
languages such as Frenetic [204], Nettle [223], NetCore addressed in this respect is interoperability. This is rather
[205], Procera [224], Pyretic [225], NetKAT [226], and interesting, as it was the very first problem that south-
other query-based languages [227]. Section IV-G gives a bound APIs (such as OpenFlow) tried to solve. For
more detailed overview of the several existing program- instance, while WiFi and long-term evolution (LTE) net-
ming languages for SDN. works [235] need specialized control platforms such as
MobileFlow [222] or SoftRAN [236], data center networks
8) Wrapping Up Remarks and Platforms Comparison: have different requirements that can be met with plat-
Table 6 shows a summary of some of the existing con- forms such as Onix [7] or OpenDaylight [13]. For this
trollers with their respective architectures and character- reason, in environments where diversity of networking

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 33

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

infrastructures is a reality, coordination and cooperation plane fault tolerance, and a variety of basic building blocks
between different controllers is crucial. Standardized APIs to ease software module and application development.
for multicontroller and multidomain deployments are SFNet [249] is another example of a northbound inter-
therefore seen as an important step to achieve this goal. face. It is a high-level API that translates application re-
quirements into lower level service requests. However,
E. Layer V: Northbound Interfaces SFNet has a limited scope, targeting queries to request the
The northbound and southbound interfaces are two key congestion state of the network and services such as band-
abstractions of the SDN ecosystem. The southbound inter- width reservation and multicast.
face has already a widely accepted proposal (OpenFlow), Other proposals use different approaches to allow ap-
but a common northbound interface is still an open issue. plications to interact with controllers. The yanc control
At this moment it may still be a bit too early to define a platform [196] explores this idea by proposing a general
standard northbound interface, as use cases are still being control platform based on Linux and abstractions such as
worked out [237]. Anyway, it is to be expected a common the virtual file system (VFS). This approach simplifies the
(or a de facto) northbound interface to arise as SDN evolves. development of SDN applications as programmers are able
An abstraction that would allow network applications not to use a traditional concept (files) to communicate with
to depend on specific implementations is important to lower level devices and subsystems.
explore the full potential of SDN. Eventually, it is unlikely that a single northbound in-
The northbound interface is mostly a software eco- terface emerges as the winner, as the requirements for
system, not a hardware one, as is the case of the south- different network applications are quite different. APIs for
bound APIs. security applications are likely to be different from those
In these ecosystems, the implementation is commonly for routing or financial applications. One possible path of
the forefront driver, while standards emerge later and are evolution for northbound APIs are vertically oriented pro-
essentially driven by wide adoption [238]. Nevertheless, an posals, before any type of standardization occurs, a chal-
initial and minimal standard for northbound interfaces can lenge the ONF has started to undertake in the NBI WG in
still play an important role for the future of SDN. Discussions parallel to open-source SDN developments [50]. The ONF
about this issue have already begun [237]–[244], and a architectural work [218] includes the possibility of north-
common consensus is that northbound APIs are indeed bound APIs providing resources to enable dynamic and
important but that it is indeed too early to define a single granular control of the network resources from customer
standard right now. The experience from the development applications, eventually across different business and orga-
of different controllers will certainly be the basis for nizational boundaries.
coming up with a common application level interface. There are also other kinds of APIs, such as those pro-
Open and standard northbound interfaces are crucial to vided by the PANE controller [197]. Designed to be suit-
promote application portability and interoperability among able for the concept of participatory networking, PANE
the different control platforms. A northbound API can be allows network administrators to define module-specific
compared to the POSIX standard [245] in operating sys- quotas and access control policies on network resources.
tems, representing an abstraction that guarantees program- The controller provides an API that allows end-host appli-
ming language and controller independence. NOSIX [246] cations to dynamically and autonomously request network
is one of the first examples of an effort in this direction. It resources. For example, audio (e.g., VoIP) and video ap-
tries to define portable low-level (e.g., flow model) appli- plications can easily be modified to use the PANE API to
cation interfaces, making southbound APIs such as Open- reserve bandwidth for certain quality guarantees during
Flow look like ‘‘device drivers.’’ However, NOSIX is not the communication session. PANE includes a compiler and
exactly a general purpose northbound interface, but rather verification engine to ensure that bandwidth requests do
a higher level abstraction for southbound interfaces. In- not exceed the limits set by the administrator and to avoid
deed, it could be part of the common abstraction layer in a starvation, i.e., other applications will not be impaired by
control platform as the one described in Section IV-D. new resource requests.
Existing controllers such as Floodlight, Trema, NOX,
Onix, and OpenDaylight propose and define their own F. Layer VI: Language-Based Virtualization
northbound APIs [239], [247]. However, each of them has Two essential characteristics of virtualization solutions
its own specific definitions. Programming languages such are the capability of expressing modularity and of allowing
as Frenetic [204], Nettle [223], NetCore [205], Procera different levels of abstractions while still guaranteeing de-
[224], Pyretic [248], and NetKAT [226] also abstract the sired properties such as protection. For instance, virtua-
inner details of the controller functions and data plane lization techniques can allow different views of a single
behavior from the application developers. Moreover, as we physical infrastructure. As an example, one virtual ‘‘big
explain in Section IV-G, programming languages can pro- switch’’ could represent a combination of several underly-
vide a wide range of powerful abstractions and mechan- ing forwarding devices. This intrinsically simplifies the
isms such as application composition, transparent data task of application developers as they do not need to think

34 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

about the sequence of switches where forwarding rules ensures that properties such as isolation are enforced
have to be installed, but rather see the network as a simple among slices, i.e., no packets of slice A can traverse to
‘‘big switch.’’ Such kind of abstraction significantly simpli- slice B unless explicitly allowed.
fies the development and deployment of complex network Other solutions, such as libNetVirt [251], try to integ-
applications, such as advanced security related services. rate heterogeneous technologies for creating static net-
Pyretic [248] is an interesting example of a program- work slices. libNetVirt is a library designed to provide a
ming language that offers this type of high-level abstrac- flexible way to create and manage virtual networks in dif-
tion of network topology. It incorporates this concept of ferent computing environments. Its main idea is similar to
abstraction by introducing network objects. These objects the OpenStack Quantum project [252]. While Quantum is
consist of an abstract network topology and the sets of designed for OpenStack (cloud environments), libNetVirt
policies applied to it. Network objects simultaneously hide is a more general purpose library which can be used in
information and offer the required services. different environments. Additionally, it goes one step be-
Another form of language-based virtualization is static yond OpenStack Quantum by enabling QoS capabilities in
slicing. This a scheme where the network is sliced by a virtual networks [251]. The libNetVirt library has two
compiler, based on application layer definitions. The out- layers: a generic network interface and technology specific
put of the compiler is a monolithic control program that device drivers (e.g., VPN, MPLS, OpenFlow). On top of
has already slicing definitions and configuration com- the layers are the network applications and virtual network
mands for the network. In such a case, there is no need for descriptions. The OpenFlow driver uses a NOX controller
a hypervisor to dynamically manage the network slices. to manage the underlying infrastructure, using OpenFlow
Static slicing can be valuable for deployments with specific rule-based flow tables to create isolated virtual networks.
requirements, in particular those where higher perfor- By supporting different technologies, it can be used as a
mance and simple isolation guarantees are preferable to bridging component in heterogeneous networks.
dynamic slicing. Table 7 summarizes the hypervisor- and nonhypervisor-
One example of static slicing approach is the Splendid based virtualization technologies. As can be observed, only
isolation [250]. In this solution, the network slices are libNetVirt supports heterogeneous technologies, not re-
made of three components: 1) topology, consisting of stricting its application to OpenFlow-enabled networks.
switches, ports, and links; 2) mapping of slice-level FlowVisor, AutoSlice, and OpenVirteX allow multiple con-
switches, ports, and links on the network infrastructure; trollers, one per network slice. FlowN provides a container-
and 3) predicates on packets, where each port of the slice’s based approach where multiple applications from different
edge switches has an associated predicate. The topology is a users can coexist on a single controller. FlowVisor allows
simple graph of the sliced nodes, ports, and links. Mapping QoS provisioning guarantees by using VLAN PCP bits for
will translate the abstract topology elements into the priority queues. SDN VE and NVP also provide their own
corresponding physical ones. The predicates are used to provisioning methods for guaranteeing QoS.
indicate whether a packet is permitted to enter a specific
slice. Different applications can be associated to each slice. G. Layer VII: Programming Languages
The compiler takes the combination of slices (topology, Programming languages have been proliferating for
mapping, and predicates) and respective programs to gen- decades. Both academia and industry have evolved from
erate a global configuration for the entire network. It also low-level hardware-specific machine languages, such as

Table 7 Virtualization Solutions

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 35

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

assembly for x86 architectures, to high-level and powerful Another interesting feature that programming lan-
programming languages such as Java and Python. The ad- guage abstractions provide is the capability of creating and
vancements toward more portable and reusable code writing programs for virtual network topologies [248],
have driven a significant shift on the computer industry [250]. This concept is similar to object-oriented prog-
[254], [255]. ramming, where objects abstract both data and specific
Similarly, programmability in networks is starting to functions for application developers, making it easier to
move from low-level machine languages such as OpenFlow focus on solving a particular problem without worrying
(‘‘assembly’’) to high-level programming languages [112], about data structures and their management. For instance,
[203]–[205], [223]–[225]. Assembly-like machine lan- in an SDN context, instead of generating and installing
guages, such as OpenFlow [9] and POF [31], [120], essen- rules in each forwarding device, one can think of creating
tially mimic the behavior of forwarding devices, forcing simplified virtual network topologies that represent the
developers to spend too much time on low-level details entire network, or a subset of it. For example, the appli-
rather than on the problem solve. Raw OpenFlow programs cation developer should be able to abstract the network as
have to deal with hardware behavior details such as over- an atomic big switch, rather than a combination of sev-
lapping rules, the priority ordering of rules, and data plane eral underlying physical devices. The programming
inconsistencies that arise from in-flight packets whose flow languages or runtime systems should be responsible for
rules are under installation [204], [205], [256]. The use of generating and installing the lower level instructions
these low-level languages makes it difficult to reuse soft- required at each forwarding device to enforce the user
ware, to create modular and extensive code, and leads to a policy across the network. With such kind of abstractions,
more error-prone development process [225], [257], [258]. developing a routing application becomes a straightfor-
Abstractions provided by high-level programming ward process. Similarly, a single physical switch could be
languages can significantly help address many of the chal- represented as a set of virtual switches, each of them
lenges of these lower level instruction sets [203]–[205], belonging to a different virtual network. These two exam-
[223]–[225]. In SDNs, high-level programming languages ples of abstract network topologies would be much harder
can be designed and used to: to implement with low-level instruction sets. In contrast,
1) create higher level abstractions for simplifying the a programming language or runtime system can more
task of programming forwarding devices; easily provide abstractions for virtual network topologies,
2) enable more productive and problem-focused en- as has already been demonstrated by languages such as
vironments for network software programmers, Pyretic [248].
speeding up development and innovation;
3) promote software modularization and code reus- 1) High-Level SDN Programming Languages: High-level
ability in the network control plane; programming languages can be powerful tools as a mean
4) foster the development of network virtualization. for implementing and providing abstractions for different
Several challenges can be better addressed by program- important properties and functions of SDN such as
ming languages in SDNs. For instance, in pure OpenFlow- network-wide structures, distributed updates, modular
based SDNs, it is hard to ensure that multiple tasks of a composition, virtualization, and formal verification [29].
single application (e.g., routing, monitoring, access con- Low-level instruction sets suffer from several pro-
trol) do not interfere with each other. For example, rules blems. To address some of these challenges, higher level
generated for one task should not override the function- programming languages have been proposed, with diverse
ality of another task [204], [256]. Another example is goals, such as:
when multiple applications run on a single controller • avoiding low-level and device-specific configura-
[201], [225], [256], [259], [260]. Typically, each applica- tions and dependencies spread across the network,
tion generates rules based on its own needs and policies as happens in traditional network configuration
without further knowledge about the rules generated by approaches;
other applications. As a consequence, conflicting rules can • providing abstractions that allow different man-
be generated and installed in forwarding devices, which agement tasks to be accomplished through easy to
can create problems for network operation. Programming understand and maintain network policies;
languages and runtime systems can help to solve these • decoupling of multiple tasks (e.g., routing, access
problems that would be otherwise hard to prevent. control, traffic engineering);
Important software design techniques such as code • implementing higher level programming interfaces
modularity and reusability are very hard to achieve using to avoid low-level instruction sets;
low-level programming models [225]. Applications thus • solving forwarding rules problems, e.g., conflicting
built are monolithic and consist of building blocks that or incomplete rules that can prevent a switch event
cannot be reused in other applications. The end result is to be triggered, in an automated way;
a very time-consuming and error-prone development • addressing different race condition issues which
process. are inherent to distributed systems;

36 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Table 8 Programming Languages

• enhancing conflict-resolution techniques on envi- easily allow applications to run on different controllers
ronments with distributed decision makers; without requiring any modification. As an example, the
• providing native fault-tolerance capabilities on Pyretic language requires only a standard socket interface
data plane path setup; and a simple OpenFlow client on the target controller
• reducing the latency in the processing of new platform [225].
flows; Several programming languages have been proposed
• easing the creation of stateful applications (e.g., for SDNs, as summarized in Table 8. The great majority
stateful firewall). propose abstractions for OpenFlow-enabled networks. The
Programming languages can also provide specialized predominant programming paradigm is the declarative
abstractions to cope with other management require- one, with a single exception, Pyretic, which is an impera-
ments, such as monitoring [204], [224], [227], [261]. For tive language. Most declarative languages are functional,
instance, the runtime system of a programming language but there are instances of the logic and reactive types. The
can do all the ‘‘laundry work’’ of installing rules, polling purposeVi.e., the specific problems they intend to solveV
the counters, receiving the responses, combining the re- and the expressiveness power vary from language to lan-
sults as needed, and composing monitoring queries in guage, while the end goal is almost always the same: to
conjunction with other policies. Consequently, application provide higher level abstractions to facilitate the develop-
developers can take advantage of the simplicity and power ment of network control logic.
of higher level query instructions to easily implement mo- Programming languages such as FML [203], Nettle
nitoring modules or applications. [223], and Procera [224] are functional and reactive. Poli-
Another aspect of paramount importance is the cies and applications written in these languages are based
portability of the programming language, necessary so on reactive actions triggered by events (e.g., a new host
that developers do not need to re-implement applications connected to the network, or the current network load).
for different control platforms. The portability of a prog- Such languages allow users to declaratively express differ-
ramming language can be considered as a significant added ent network configuration rules such as access control lists
value to the control plane ecosystem. Mechanisms such as (ACLs), virtual LANs (VLANs), and many others. Rules are
decoupled back–ends could be key architectural ingredi- essentially expressed as allow-or-deny policies, which are
ents to enable platform portability. Similarly to the Java applied to the forwarding elements to ensure the desired
virtual machine, a portable northbound interface will network behavior.

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 37

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Other SDN programming languages, such as Frenetic existing systems. To achieve this goal, Merlin generates
[204], hierarchical flow tables (HFTs) [256], NetCore specific code for each type of component. Taking a policy
[205], and Pyretic [225], were designed with the simul- definition as input, Merlin’s compiler determines forward-
taneous goal of efficiently expressing packet-forwarding ing paths, transformation placement, and bandwidth
policies and dealing with overlapping rules of different allocation. The compiled outputs are sets of component-
applications, offering advanced operators for parallel and specific low-level instructions to be installed in the devices.
sequential composition of software modules. To avoid Merlin’s policy language also allows operators to delegate
overlapping conflicts, Frenetic disambiguates rules with the control of a subnetwork to tenants, while ensuring iso-
overlapping patterns by assigning different integer prior- lation. This delegated control is expressed by means of
ities, while HFT uses hierarchical policies with enhanced policies that can be further refined by each tenant owner,
conflict-resolution operators. allowing them to customize policies for their particular
See-every-packet abstractions and race-free semantics needs.
also represent interesting features provided by program- Other recent initiatives (e.g., systems programming
ming languages (such as Frenetic [204]). The former en- languages [265]) target problems such as detecting ano-
sures that all control packets will be available for analysis, malies to improve the security of network protocols (e.g.,
sooner or later, while the latter provides the mechanisms Open-Flow), and optimizing horizontal scalability for
for suppressing unimportant packets. As an example, achieving high throughput in applications running on
packets that arise from a network race condition, such as multicore architectures [263]. Nevertheless, there is still
due to a concurrent flow rule installation on switches, can scope for further investigation and development on prog-
be simply discarded by the runtime system. ramming languages. For instance, one recent research has
Advanced operators for parallel and sequential com- revealed that current policy compilers generate unneces-
position help bind (through internal workflow operators) sary (redundant) rule updates, most of which modify only
the key characteristics of programming languages such as the priority field [266].
Pyretic [225]. Parallel composition makes it possible to Most of the value of SDN will come from the network
operate multiple policies on the same set of packets, while managements applications built on top of the infrastruc-
sequential composition facilitates the definition of a se- ture. Advances in high-level programming languages are a
quential workflow of policies to be processed on a set of fundamental component to the success of a prolific SDN
packets. Sequential policy processing allows multiple application development ecosystem. To this end, efforts
modules (e.g., access control and routing) to operate in a are undergoing to shape forthcoming standard interfaces
cooperative way. By using sequential composition complex (cf. [267]) and toward the realization of integrated devel-
applications can be built out of a combination of different opment environments (e.g., NetIDE [268]) with the goal
modules (in a similar way as pipes can be used to build of fostering the development of a myriad of SDN applica-
sophisticated Unix applications). tions. We discuss these next.
Further advanced features are provided by other SDN
programming languages. FatTire [262] is an example of a H. Layer VIII: Network Applications
declarative language that heavily relies on regular expres- Network applications can be seen as the ‘‘network
sions to allow programmers to describe network paths with brains.’’ They implement the control logic that will be
fault-tolerance requirements. For instance, each flow can translated into commands to be installed in the data plane,
have its own alternative paths for dealing with failure of dictating the behavior of the forwarding devices. Take a
the primary paths. Interestingly, this feature is provided in simple application as routing as an example. The logic of
a very programmer-friendly way, with the application this application is to define the path through which packets
programmer having only to use regular expressions with will flow from point A to point B. To achieve this goal, a
special characters, such as an asterisk. In the particular routing application has to, based on the topology input,
case of FatTire, an asterisk will produce the same behavior decide on the path to use and instruct the controller to
as a traditional regular expression, but translated into install the respective forwarding rules in all forwarding
alternative traversing paths. devices on the chosen path, from A to B.
Programming languages such as FlowLog [257] and SDNs can be deployed on any traditional network en-
Flog [258] bring different features, such as model check- vironment, from home and enterprise networks to data
ing, dynamic verification, and stateful middleboxes. For centers and Internet exchange points. Such variety of en-
instance, using a programming language such as Flog, it is vironments has led to a wide array of network applications.
possible to build a stateful firewall application with only Existing network applications perform traditional func-
five lines of code [258]. tionality such as routing, load balancing, and security po-
Merlin [264] is one of the first examples of unified licy enforcement, but also explore novel approaches, such
framework for controlling different network components, as reducing power consumption. Other examples include
such as forwarding devices, middleboxes, and end-hosts. fail-over and reliability functionalities to the data plane,
An important advantage is backward compatibility with end-to-end QoS enforcement, network virtualization,

38 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

mobility management in wireless networks, among many leveraged to optimize the energy consumption of the net-
others. The variety of network applications, combined with work [274]. By using specialized optimization algorithms
real use case deployments, is expected to be one of the and diversified configuration options, it is possible to meet
major forces on fostering a broad adoption of SDN [269]. the infrastructure goals of latency, performance, and fault
Despite the wide variety of use cases, most SDN appli- tolerance, for instance, while reducing power consump-
cations can be grouped in one of five categories: traffic tion. With the use of simple techniques, such as shutting
engineering, mobility and wireless, measurement and mo- down links and devices intelligently in response to traffic
nitoring, security and dependability and data center net- load dynamics, data center operators can save up to 50% of
working. Tables 9 and 10 summarize several applications the network energy in normal traffic conditions [274].
categorized as such, stating their main purpose, controller One of the important goals of data center networks is to
where it was implemented/evaluated, and southbound avoid or mitigate the effect of network bottlenecks on the
API used. operation of the computing services offered. Linear bisec-
tion bandwidth is a technique that can be adopted for
1) Traffic Engineering: Several traffic engineering appli- traffic patterns that stress the network by exploring path
cations have been proposed, including ElasticTree [274], diversity in a data center topology. Such technique has
Hedera [276], OpenFlow-based server load balancing been proposed in an SDN setting, allowing the maximiza-
[338], Plug-n-Serve [285] and Aster*x [273], In-packet tion of aggregated network utilization with minimal sched-
Bloom filter [277], SIM–PLE [292], QNOX [287], QoS uling overhead [276]. SDN can also be used to provide a
framework [289], QoS for SDN [288], ALTO [270], fully automated system for controlling the configuration of
ViAggre SDN [293], ProCel [282], FlowQoS [275], and routers. This can be particularly useful in scenarios that
Middlepipes [27]. In addition to these, recent proposals apply virtual aggregation [342]. This technique allows net-
include optimization of rules placement [339], the use of work operators to reduce the data replicated on routing
MAC as a universal label for efficient routing in data cen- tables, which is one of the causes of routing tables’ growth
ters [340], among other techniques for flow management, [343]. A specialized routing application [293] can calculate,
fault tolerance, topology update, and traffic characteriza- divide, and configure the routing tables of the different rout-
tion [341]. The main goal of most applications is to engineer ing devices through a southbound API such as OpenFlow.
traffic with the aim of minimizing power consumption, Traffic optimization is another interesting application
maximizing aggregate network utilization, providing opti- for large-scale service providers, where dynamic scale-out
mized load balancing, and other generic traffic optimiza- is required. For instance, the dynamic and scalable provi-
tion techniques. sioning of VPNs in cloud infrastructures, using protocolols
Load balancing was one of the first applications envi- such as ALTO [272], can be simplified through an SDN-
sioned for SDN/OpenFlow. Different algorithms and tech- based approach [270]. Recent work has also shown that
niques have been proposed for this purpose [338], [273], optimizing rules placement can increase network effi-
[285]. One particular concern is the scalability of these ciency [339]. Solutions such as ProCel [282], designed for
solutions. A technique to allow this type of applications to cellular core networks, are capable of reducing the signal-
scale is to use wildcard-based rules to perform proactive ing traffic up to 70%, which represents a significant
load balancing [338]. Wildcards can be utilized for aggre- achievement.
gating client requests based on the ranges of IP prefixes, Other applications that perform routing and traffic en-
for instance, allowing the distribution and directing of gineering include application-aware networking for video
large groups of client requests without requiring controller and data streaming [344], [345] and improved QoS by
intervention for every new flow. In tandem, operation in employing multiple packet schedulers [290] and other
reactive mode may still be used when traffic bursts are techniques [279], [287], [289], [346]. As traffic engineer-
detected. The controller application needs to monitor the ing is a crucial issue in all kinds of networks, upcoming
network traffic and use some sort of threshold in the flow methods, techniques, and innovations can be expected in
counters to redistribute clients among the servers when the context of SDNs.
bottlenecks are likely to happen.
SDN load balancing also simplifies the placement of 2) Mobility and Wireless: The current distributed control
network services in the network [285]. Every time a new plane of wireless networks is suboptimal for managing the
server is installed, the load balancing service can take the limited spectrum, allocating radio resources, implement-
appropriate actions to seamlessly distribute the traffic ing handover mechanisms, managing interference, and
among the available servers, taking into consideration both performing efficient load balancing between cells. SDN-
the network load and the available computing capacity of based approaches represent an opportunity for making it
the respective servers. This simplifies network manage- easier to deploy and manage different types of wireless
ment and provides more flexibility to network operators. networks, such as WLANs and cellular networks [236],
Existing southbound interfaces can be used for actively [296], [300], [302], [347], [348]. Traditionally hard-to-
monitoring the data plane load. This information can be implement but desired features are indeed becoming a

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 39

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Table 9 Network Applications

40 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

Table 10 Network Applications

reality with the SDN-based wireless networks. These in- ware abstraction layer for decoupling the wireless protocol
clude seamless mobility through efficient handovers [300], definition from the hardware, allowing shared MAC layers
[347], [349], load balancing [236], [300], creation of on- across different protocols using commodity multicore plat-
demand virtual access points (VAPs) [297], [300], down- forms. OpenRadio can be seen as the ‘‘OpenFlow for wire-
link scheduling (e.g., an OpenFlow switch can do a rate less networks.’’ Similarly, SoftRAN [236] proposes to
shaping or time division) [297], dynamic spectrum usage rethink the radio access layer of current LTE infrastruc-
[297], enhanced intercell interference coordination [297], tures. Its main goal is to allow operators to improve and
[347], device-to-device offloading (i.e., decide when and optimize algorithms for better handovers, fine-grained
how LTE transmissions should be offloaded to users adopt- control of transmit powers, resource block allocation,
ing the device to device paradigm [296]) [350], per client among other management tasks.
and/or base station resource block allocations (i.e., time Light virtual access points (LVAPs) is another interest-
and frequency slots in LTE/orthogonal frequency-division ing way of improving the management capabilities of wire-
multiple access (OFDMA) networks, which are known as less networks, as proposed by the Odin framework [300].
resource blocks) [236], [296], [348], control and assign Differently from OpenRadio, it works with existing
transmission and power parameters in devices or in a wireless hardware and does not impose any change on
group basis (e.g., algorithms to optimize the transmission IEEE 802.11 standards. An LVAP is implemented as a
and power parameters of WLAN devices define and assign unique basic service set identification associated with a
transmission power values to each resource block, at each specific client, which means that there is a one-to-one
base station, in LTE/OFDMA networks) [236], [296], sim- mapping between LVAPs and clients. This per-client access
plified administration [236], [300], [302], easy manage- point (AP) abstraction simplifies the handling of client
ment of heterogeneous network technologies [236], [302], associations, authentication, handovers, and unified slicing
[351], interoperability between different networks [348], of both wired and wireless portions of the network. Odin
[351], shared wireless infrastructures [351], seamless sub- achieves control logic isolation between slices, since LVAPs
scriber mobility and cellular networks [347], QoS and ac- are the primitive type upon which applications make
cess control policies made feasible and easier [347], [348], control decisions, and applications do not have visibility of
and easy deployment of new applications [236], [300], [351]. LVAPs from outside their slice. This empowers infrastruc-
One of the first steps toward realizing these features in ture operators to provide services through Odin applica-
wireless networks is to provide programmable and flexible tions, such as a mobility manager, client-based load
stack layers for wireless networks [236], [352]. One of the balancer, channel selection algorithm, and wireless trou-
first examples is OpenRadio [352], which proposes a soft- bleshooting application within different network slices. For

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 41

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

instance, when a user moves from one AP to another, the Other initiatives of this second class propose a stronger
network mobility management application can automati- decoupling between basic primitives (e.g., matching and
cally and proactively act and move the client LVAP from counting) and heavier traffic analysis functions such as the
one AP to the other. In this way, a wireless client will not detection of anomaly conditions attacks [358]. A stronger
even notice that it started to use a different AP because separation favors portability and flexibility. For instance, a
there is no perceptive handoff delay, as it would be the case functionality to detect abnormal flows should not be con-
in traditional wireless networks. strained by the basic primitives or the specific hardware
Very dense heterogeneous wireless networks have also implementation. Put in another way, developers should be
been a target for SDN. These DenseNets have limitations empowered with streaming abstractions and higher level
due to constraints such as radio access network bottle- programming capabilities.
necks, control overhead, and high operational costs [296]. In that vein, some data and control plane abstractions
A dynamic two-tier SDN controller hierarchy can be have been specifically designed for measurement purposes.
adapted to address some of these constraints [296]. Local OpenSketch [311] is a special-purpose southbound API
controllers can be used to take fast and fine-grained deci- designed to provide flexibility for network measurements.
sions, while regional (or ‘‘global’’) controllers can have a For instance, by allowing multiple measurement tasks to
broader, coarser grained scope, i.e., that take slower but execute concurrently without impairing accuracy. The in-
more global decisions. In such a way, designing a single ternal design of an OpenSketch switch can be thought of as
integrated architecture that encompasses LTE (macro/pico/ a pipeline with three stages (hashing, classification, and
femto) and WiFi cells, while challenging, seems feasible. counting). Input packets first pass through a hashing func-
tion. Then, they are classified according to a matching
3) Measurement and Monitoring: Measurement and mo- rule. Finally, the match rule identifies a counting index,
nitoring solutions can be divided into two classes: first, which is used to calculate the counter location in the
applications that provide new functionality for other net- counting stage. While a TCAM with few entries is enough
working services; and second, proposals that target to im- for the classification stage, the flexible counters are stored
prove features of OpenFlow-based SDNs, such as to reduce in SRAM. This makes the OpenSketch’s operation efficient
control plane overload due to the collection of statistics. (fast matching) and cost-effective (cheaper SRAMs to store
An example of the first class of applications is improving counters).
the visibility of broadband performance [6], [353]. An Other monitoring frameworks, such as OpenSample
SDN-based broadband home connection can simplify the [309] and PayLess [313], propose different mechanisms for
addition of new functions in measurement systems such as delivering real-time, low-latency, and flexible monitoring
BISmark [353], allowing the system to react to changing capabilities to SDN without impairing the load and perfor-
conditions in the home network [6]. As an example, a home mance of the control plane. The proposed solutions take
gateway can perform reactive traffic shaping considering advantage of sampling technologies like sFlow [310] to
the current measurement results of the home network. monitor high-speed networks, and flexible collections of
The second class of solutions typically involve different loosely coupled (plug-and-play) components to provide ab-
kinds of sampling and estimation techniques to be applied, stract network views yielding high-performance and effi-
in order to reduce the burden of the control plane with cient network monitoring approaches [309], [313], [355].
respect to the collection of data plane statistics. Different
techniques have been applied to achieve this goal, such as 4) Security and Dependability: An already diverse set of
stochastic and deterministic packet sampling techniques security and dependability proposals is emerging in the
[354], traffic matrix estimation [261], fine-grained moni- context of SDNs. Most take advantage of SDN for improv-
toring of wildcard rules [355], two-stage Bloom filters [356] ing services required to secure systems and networks, such
to represent monitoring rules and provide high measure- as policy enforcement (e.g., access control, firewalling,
ment accuracy without incurring in extra memory or con- middleboxes as middlepipes [27]) [27], [100], [326], [328],
trol plane traffic overhead [304], and special monitoring [337], DoS attacks detection and mitigation [325], [336],
functions (extensions to OpenFlow) in forwarding devices random host mutation [326] (i.e., randomly and frequently
to reduce traffic and processing load on the control plane mutate the IP addresses of end-hosts to break the attackers’
[357]. Point-to-point traffic matrix estimation, in partic- assumption about static IPs, which is the common case)
ular, can help in network design and operational tasks [331], monitoring of cloud infrastructures for fine-grained
such as load balancing, anomaly detection, capacity plan- security inspections (i.e., automatically analyze and detour
ning, and network provisioning. With information on the suspected traffic to be further inspected by specialized
set of active flows in the network, routing information network security appliances, such as deep packet inspec-
(e.g., from the routing application), flow paths, and flow tion systems) [323], traffic anomaly detection [325], [336],
counters in the switches, it is possible to construct a traffic [354], fine-grained flow-based network access control
matrix using diverse aggregation levels for sources and [327], fine-grained policy enforcement for personal mobile
destinations [261]. applications [329], and so on [100], [323], [325], [326],

42 Proceedings of the IEEE | Vol. 103, No. 1, January 2015

 Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

[328], [331], [337], [354]. Others address OpenFlow-based view of SDN security issues and challenges can be found in
networks issues, such as flow rule prioritization, security Section V-F.
services composition, protection against traffic overload,
and protection against malicious administrators [199], 7) Data Center Networking: From small enterprises to
[201], [259], [322], [330]. large-scale cloud providers, most of the existing IT systems
There are essentially two approaches, one involves and services are strongly dependent on highly scalable and
using SDNs to improve network security, and another for efficient data centers. Yet, these infrastructures still pose
improving the security of the SDN itself. The focus has significant challenges regarding computing, storage, and
been, thus far, in the latter. networking. Concerning the latter, data centers should be
designed and deployed in such a way as to offer high and
5) Using SDN to Improve the Security of Current Networks: flexible cross-section bandwidth and low latency, QoS
Probably the first instance of SDN was an application for based on the application requirements, high levels of resi-
security policies enforcement [100]. An SDN allows the lience, intelligent resource utilization to reduce energy
enforcement to be done on the first entry point to the consumption and improve overall efficiency, agility in
network (e.g., the Ethernet switch to which the user is provisioning network resources, for example by means of
connected to). Alternatively, in a hybrid environment, network virtualization and orchestration with computing
security policy enforcement can be made on a wider net- and storage, and so forth [360]–[362]. Not surprisingly,
work perimeter through programmable devices (without many of these issues remain open due to the complexity
the need to migrate the entire infrastructure to OpenFlow) and inflexibility of traditional network architectures.
[328]. With either application, malicious actions are The emergence of SDN is expected to change the cur-
blocked before entering the critical regions of the network. rent state of affairs. Early research efforts have indeed
SDN has been successfully applied for other purposes, showed that data center networking can significantly be-
namely for the detection (and reaction) against distributed nefit from SDN in solving different problems such as live
denial of service (DDoS) flooding attacks [325], and active network migration [318], improved network management
security [321]. OpenFlow forwarding devices make it [317], [318], eminent failure avoidance [317], [318], rapid
easier to collect a variety of information from the network, deployment from development to production networks
in a timely manner, which is very handy for algorithms [318], troubleshooting [318], [319], optimization of net-
specialized in detecting DDoS flooding attacks. work utilization [314], [316], [317], [319], dynamic and
The capabilities offered by SDNs in increasing the ability elastic provisioning of middleboxes-as-a-service [27], and
to collect statistics data from the network and of allowing minimization of flow setup latency and reduction of con-
applications to actively program the forwarding devices, are troller operating costs [363]. SDN can also offer network-
powerful for proactive and smart security policy enforcement ing primitives for cloud applications, solutions to predict
techniques such as Active security [321]. This novel security network transfers of applications [314], [316], mechanisms
methodology proposes a novel feedback loop to improve the for fast reaction to operation problems, network-aware VM
control of defense mechanisms of a networked infrastruc- placement [315], [319], QoS support [315], [319], real-time
ture, and is centered around five core capabilities: protect, network monitoring and problem detection [316], [317],
sense, adjust, collect, and counter. In this perspective, active [319], security policy enforcement services and mechan-
security provides a centralized programming interface that isms [315], [319], and enable programmatic adaptation of
simplifies the integration of mechanisms for detecting transport protocols [314], [320].
attacks by: 1) collecting data from different sources (to SDN can help infrastructure providers to expose more
identify attacks); 2) converging to a consistent configuration networking primitives to their customers by allowing vir-
for the security appliances; and 3) enforcing counter- tual network isolation, custom addressing, and the place-
measures to block or minimize the effect of attacks. ment of middleboxes and virtual desktop cloud applications
[315], [364]. To fully explore the potential of virtual net-
6) Improving the Security of SDN Itself: There are already works in clouds, an essential feature is virtual network
some research efforts in identifying the critical security migration. Similarly to traditional virtual machine migra-
threats of SDNs and in augmenting its security and depen- tion, a virtual network may need to be migrated when its
dability [201], [259], [359]. Early approaches try to apply virtual machines move from one place to another. Integ-
simple techniques, such as classifying applications and rating live migration of virtual machines and virtual net-
using rule prioritization, to ensure that rules generated by works is one of the forefront challenges [318]. To achieve
security applications will not be overwritten by lower this goal, it is necessary to dynamically reconfigure all
priority applications [201]. Other proposals try to go a step affected networking devices (physical or virtual). This was
further by providing a framework for developing security- shown to be possible with SDN platforms, such as NVP [112].
related applications in SDNs [259]. However, there is still Another potential application of SDN in data centers is
a long way to go in the development of secure and in detecting abnormal behaviors in network operation
dependable SDN infrastructures [359]. An in-deep over- [317]. By using different behavioral models and collecting

Vol. 103, No. 1, January 2015 | Proceedings of the IEEE 43

Kreutz et al.: Software-Defined Networking: A Comprehensive Survey

the necessary information from elements involved in the Debugging and troubleshooting in networking is at a
operation of a data center (infrastructure, operators, appli- very primitive stage. In traditional networks, engineers
cations), it is possible to continuously build signatures for and developers have to use tools such as ping,
applications by passively capturing control traffic. Then, traceroute, tcpdump, nmap, netflow, and
the signature history can be used to identify differences in SNMP statistics for debugging and troubleshooting.
behavior. Every time a difference is detected, operators Debugging a complex network with such primitive tools
can reactively or proactively take corrective measures. This is very hard. Even when one considers frameworks such as
can help to isolate abnormal components and avoid further XTrace [374], Netreplay [376], and NetCheck [377], which
damage to the infrastructure. improve debugging capabilities in networks, it is still diffi-
cult to troubleshoot networking infrastructures. For in-
8) Toward SDN App Stores: As can be observed in stance, these frameworks require a huge effort in terms of
Tables 9 and 10, most SDN applications rely on NOX and network instrumentation. The additional complexity intro-
OpenFlow. NOX was the first controller available for duced by different types of devices, technologies, and
general use, making it a natural choice for most use cases vendor-specific components and features makes matters
so far. As indicated by the sheer number of security-related worse. As a consequence, these solutions may find it hard to
applications, security is probably one of the killer appli- be widely implemented and deployed in current networks.
cations for SDNs. Curiously, while most use cases rely on SDN offers some hope in this respect. The hardware-
OpenFlow, new solutions such as SoftRAN are considering agnostic software-based control capabilities and the use of
different APIs, as is the case of the Femto API [253], [303]. open standards for control communication can potentially
This diversity of applications and APIs will most probably make debugging and troubleshooting easier. The flexibility
keep growing in SDN. and programmability introduced by SDN is indeed opening
There are other kinds of network applications that do new avenues for developing better tools to debug, trouble-
not easily fit in our taxonomy, such as Avior [365], OESS shoot, verify, and test networks [378]–[385].
[366], and SDN App Store [367], [368]. Avior and OESS Early debugging tools for OpenFlow-enabled networks,
are graphical interfaces and sets of software tools that such as ndb [378], OFRewind [379], and NetSight [386],
make it easier to configure and manage controllers (e.g., make it easier to discover the source of network problems
Floodlight) and OpenFlow-enabled switches, respectively. such as faulty device firmware [378], inconsistent or non-
By leveraging their graphical functions it is possible to existing flow rules [378], [379], lack of reachability [378],
program OpenFlow enabled devices without coding in a [379], and faulty routing [378], [379]. Similarly to the
particular programming language. well-known gdb software debugger, ndb provides basic
The SDN App Store [367], [368], owned by HP, is debugging actions such as breakpoint, watch, backtrace,
probably the first SDN application market store. Custo- single step, and continue. These primitives help applica-
mers using HP’s OpenFlow controller have access to the tion developers to debug networks in a similar way to
online SDN App Store and are able to select applications to traditional software. By using ndb’s postcards (i.e., a
be dynamically downloaded and installed in the controller. unique packet identifier composed of a truncated copy of
The idea is similar to the Android Market or the Apple the packet’s header, the matching flow entry, the switch,
Store, making it easier for developers to provide new and the output port), for instance, a programmer is able to
applications and for customers to obtain them. quickly identify and isolate a buggy OpenFlow switch with
hardware or software problems. If the switch is presenting
I. Cross-Layer Issues abnormal behavior such as corrupting parts of the packet
In this section, we look at cross-layer problems such as header, by analyzing the problematic flow sequences with
debugging and troubleshooting, testing, verification, simula- a debugging tool, one can find (in a matter of few seconds)
tion, and emulation. A summary of the existing tools for where the packets of a flow are being corrupted, and take
dealing with these cross-layer issues can be found on Table 11. the necessary actions to solve the problem.
The OFRewind [379] tool works differently. The idea is
1) Debugging and Troubleshooting: Debugging and trou- to record and replay network events, in particular control
bleshooting have been important subjects in computing messages. These usually account for less than 1% of the
infrastructures, parallel and distributed systems, embed- data traffic and are responsible for 95%–99% of the bugs
ded systems, and desktop applications [369]–[375]. The [385]. This tool allows operators to perform fine-grained
two predominant strategies applied to debug and trouble- tracing of network behavior, being able to decide which
shoot are runtime debugging (e.g., gdb-like tools) and subsets of the network will be recorded and, afterwards,
post-mortem analysis (e.g., tracing, replay, and visualiza- select specific parts of the traces to be replayed. These
tion). Despite the constant evolution and the emergence of replays provide valuable information to find the root cause
new techniques to improve debugging and troubleshoot- of the network misbehavior. Likewise, NetRevert [387]
ing, there are still several open avenues and research also records the state of OpenFlow networks. However,
questions [370]. the primary goal is not to reproduce network behavior, but

44 Proceedings of the IEEE | Vol. 103, No. 1, January 2015