ICC2017: WT04-5thIEEE International Workshop on Smart Communication Protocols and Algorithms (SCPA 2017)

Including Artificial Intelligence in a Routing Protocol

Using Software Defined Networks

Sandra Sendra1,2, Albert Rego2, Jaime Lloret2, Jose Miguel Jimenez2, Oscar Romero2
1
Dept. of Signal Theory, Telematics and Communications Department (TSTC), Universidad de Granada, Granada, Spain
2
Universitat Politecnica de Valencia, Valencia, Spain

[email protected], [email protected], [email protected], [email protected],[email protected]

Abstract—Software defined network (SDN) is one of the most This rest of this paper is structured as follows. Section II
interesting research topic that is currently being investigated. presents some of the most relevant works related to SDN and
The inclusion of artificial intelligence (AI) can improve the the application of AI in routing protocols. The proposed
performance of routing protocols. Nowadays the application of architecture is discussed in Section III where the details about
AI over routing protocols is only applied to real devices, how the SDN is built are also shown. Section IV explains the
especially in wireless sensor nodes. In this paper, we present a proposed intelligent algorithm and how the AI is added to the
new proposal to implement an intelligent routing protocol in a network. Measurement results are shown in Section V. Finally,
SDN topology. The intelligent routing protocol is based on the Section VI presents the conclusion and future work.
reinforcement learning process that allows choosing the best data
transmission paths according to the best criteria and based on II. RELATED WORK
the network status.
This section presents some previous work related on SDN
Keywords—software defined networks (SDNs); artificial or the application of AI for improving the performance in
intelligence; intelligent routing; reinforcement-learning; routing. routing protocols.

I. INTRODUCTION SDN has become in one of the most interesting research

field inside the networking. N. Feamster et al. [1] explain the
Software Defined Networks (SDNs) are used in order to beginnings of SDN from the point of view of the concept of
improve the network programmability and get easier its active networking that introduced some key concepts like
management. SDN achieves this by proposing a new network programmable network and how all these concepts evolved and
architecture [1] where a central management entity, known as converted into the SDN concept by choosing the different
controller, provides some artificial intelligence (AI) in order to alternatives in its way. The SDN definition is very well
improve the network performance and take decisions related to described in [4]. B. A. A. Nunes et al., besides doing a little
quality of service (QoS) or security, among others. This is introduction about the SDN precursors, describe the SDN
implemented by separating the data plane and the control architecture. They also explain some other relevant points like
plane. SDN data transmission works by grouping packets in the OpenFlow protocol. Finally, authors detail the different
homogenous flows. The controller is in charge of defining the tools used to test this kind of networks.
action the system must perform with a flow. It can discard a
flow or route it through whatever path it wants. In strict terms, We can also find some interesting references of real
routing in SDN does not exist. The SDN philosophy aims to applications and their performance. M. Kobayashi et al. show
get a homogenous network when all the nodes are switches. their experiments with SDN and OpenFlow protocol in [5].
The controller hosts the control and the logic plane, adding the Authors show the possibilities that SDN offer to work in a real
needed to use a communication protocol e.g. OpenFlow [2]. environment and present experiences in order to get faster the
This allows to the controller to give the orders to the switches. implementation of SDN as a technology. They focus on the
On the other hand, AI is a field in constant development. fact that some big brands like Google, Microsoft or Yahoo! can
Nowadays, Neural Networks (NNs) offer several applications use SDN to provide an important programmable component to
and solutions to diverse problems [3]. Adding AI to routing cloud computing (considering also the virtualization) a wide-
problem can enhance the routing algorithms performance. used technology.
This paper combines the appliance of artificial intelligence Mininet was presented by Kaur et al. in [6] as a tool to
into the routing problem with the implementation of an develop the virtual SDN implementation avoiding the use of
adaptive routing inside SDN. This is carried out using the expensive and dedicated equipment. Mininet performance was
Mininet emulator where a SDN routing architecture is tested by J. M. Jimenez et al. [7] when it is being delivered
proposed. Finally, an intelligence routing algorithm is multimedia traffic over SDN. They compared this performance
proposed. with the obtained results in a real and traditional network. They
concluded that the Mininet emulator is a powerful tool that can

978-1-5090-1525-2/17/$31.00 ©2017 IEEE

 ICC2017: WT04-5thIEEE International Workshop on Smart Communication Protocols and Algorithms (SCPA 2017)

be used to test experiments about SDN and gather network, we have decided not to use strictly SDN routing
measurements due to the reliability in terms of consumed philosophy and implementing a distributed adaptive routing
bandwidth and delay. This work was enhanced in [8] where J. just like the one is nowadays widely used. This allows avoiding
M. Jimenez et al. increased the experiment diversity and the use of the controller and makes easier the implementation
confirmed the fact that Mininet is a reliable SDN emulator. of AI besides the advantages of using Mininet. To develop the
scenario, we use Quagga [12] routing suite as the software that
An interesting approach, that researcher are currently implements routing tasks. Quagga has also an important
investigating is the application of reinforced learning in routing advantage; it is open-source so it can be modified to add the
protocols [9]. For example, R. Desai and B. P. Patil [10] AI-based improvements to the routing protocol. However,
presented a new learning strategy method to select the best Quagga has the disadvantage that only allows having one
route in ad hoc networks. The proposal is based on the router per PC. As a result, the architecture is getting more
cooperative reinforcement learning problem. To develop it, complex and in this case for each router we need in our
authors have based their proposal in models of social insect topology, we will need a Mininet instance running Quagga.
behavior, such as swarm intelligence. They also performed the GRE tunneling protocol can be used in order to communicate
analysis and test bench of their proposal by comparing with every Mininet in the network between them. This problem
existing routing protocols. Results show that performance affects the network virtual topology which has to be split in
parameters such as packet delivery ratio as a function of packet different Mininet instances attending to the subnets where a
rate or number of nodes in network are significantly better router is located. This does not affect only to the architecture of
when reinforced learning is used. the SDN proposal but also to the physical distribution needed
J. Solanki and A. Chauhan [11] presented some to make both the measurements and the emulated topology in
modifications over the AODV protocol. Authors’ aim was to Mininet.
improve route error tolerance mechanism of this widely used Every PC used to build the SDN topology runs over the
routing protocol. These modifications were related to Ubuntu operative system and runs a Mininet instance. One of
parameters as End to End delay, Packet delivery ratio, Traffic the virtual Linux hosts emulated inside this instance is a router
overhead and Power Consumptions. The proposed scheme was than runs Quagga to perform the routing. Every emulated node
based on the fact that transmission started from closest in Mininet communicates with another one emulated in another
neighbor node when a link failed in middle of the transmission. PC.
The simulation results show the proposed scheme presents very
important reductions in delay and it improves the packet The main goal of this research is to apply AI into routing
delivery ratio comparing with the traditional routing scheme of decisions in SDN. In order to do that, a cycle topology is
AODV. Results also show an interesting improvement in the needed. This is because a router must be able to decide, at least,
routing overhead by reducing the frequency of the route between two or more different paths depending of the
discovery process. algorithm. Fig. 1 shows the topology used in these tests. The IP
address and the subnets are indicated in the figure itself. Note,
It is possible to find several proposals where reinforced that in this topology, there are four significant subnets. The
learning is applied over the existing routing protocols. first one 20.0.0.255/24 is the one where the traffic is sent from.
However, all these proposals are implemented over real In the design, the router that applies the intelligent routing
devices. As far as we know, there is no proposal over SDN and decision is the one with 20.0.0.1 IP address (R1). In the same
due to the importance that this kind of networks are acquiring, way, router connected to the 70.0.0.255/24 (R4) is the other
we think it is important to start working and implementing one that in this flow applies intelligent routing decision to send
these interesting approaches of artificial intelligence for the answers to the 20.0.0.255/24 subnet. Although the two
improving the SDN performance. remaining routers also apply routing decisions when they send
III. SDN ARCHITECTURE PROPOSAL packets to each other, in our case, they only send packets to
their immediate neighbors as a piece of the path calculated by
This section presents the SDN architecture used to provide the previous node. Hence, the important nodes where the
routing. The section is divided into two parts; firstly, we will intelligent routing algorithm takes the decisions are, in this
see a little explanation about the design decisions and how they example, R1 and R4.
affect the architecture. The second part of the section describes
the configuration and the virtual topology creation needed to As we said before, GRE tunneling is needed to
build the network. communicate the Mininet instances and then, have an operative
routing protocol. The switches in Fig. 1 work as bridges that
A. Design connect Mininet instances. Consequently, the topology is split
In order to avoid the use of expensive hardware to build the into four Mininet instances in four different PCs called PCA,
SDN scenario, we decided to use Mininet emulator. One of the PCB1, PCB2 and PCC indicated with red squares in Fig. 1.
advantages that Mininet present is that it is able to emulate
routers as Linux Host. This way of working allows us to use B. Configuration
whatever routing implementation available as a process above First of all, the emulated SDN in Mininet has to be created.
the OS. Considering the topology showed in Fig. 1, we need to create
four different virtual topologies, i.e., one for each PC. In
Then, the routing implemented in this paper is traditional Mininet, we can define a topology using a python script that
adaptive routing. In order to provide the distributed AI to the
 ICC2017: WT04-5thIEEE International Workshop on Smart Communication Protocols and Algorithms (SCPA 2017)

will be emulated. In this script, the nodes present in the V. MEASUREMENTS AND DISCUSSION
network must be defined. Every different PC has its own script. This section shows the experiments performed and the
During the second step, Quagga should be configured. obtained results. To check the performance of the algorithm in
Quagga is a routing suite programmed in C that runs the SDN, we have compared the emulated SDN topology that uses
different routing protocols using services. Quagga has a the proposal with a network with traditional routing. As is
different service for each routing protocol implemented. explained in the previous section, OSPF is the routing protocol
Quagga also has an extra service called Zebra that manages the that has been implemented and tested in its version 2.
other services. First of all, these services have to be activated in A. Equipment used
a configuration file. Then, we can set some parameters of every
activated protocol in its own configuration file. For example, in In this kind of emulations, it is interesting to know the
order to establish the OSPF areas we have to modify the equipment used in the tests. Table I shows the features of
“ospdf.conf” file. This configuration is needed for every PC switches and routers used in the real topology. The computers
running Quagga and it is different in each one, depending on used in both networks are also specified in the table.
the router that is located in that subnet.
TABLE I. EQUIPMENT USED
Once we have the virtual topology created and Quagga
Hardware/Topology Real network Mininet Network
configured, in a normal topology the next step would be
launching Mininet and, in the node R1 (the router) running CISCO Catalyst
Switches Emulated
3560 PoE 24
Quagga to execute the routing protocol. However, the GRE
tunnels has to be established before, otherwise the Mininet Routers CISCO 2621 XM Emulated
instances would not be able to establish any communication Computers
i7- 5500U 2.4 GHz i3-4160 3.60 GH
between them. In consequence, the ovs-vsctl add-port 8 GB RAM 6GB RAM
command must be launched in every PC establishing the nodes B. Intersubnetwork ping testing
in the virtual network (s2 and s3 in the PCA example showed
The first thing we have to test is the connectivity between
before) that are connected to what IP address in the real one.
the different subnets and Mininet instances to check if routing
From this point, any routing protocol implemented and is available. Thus, a ping test is showed in Fig. 4 and the
activated in Quagga, starts to work according to the its usual routing tables are showed in Fig. 5. The ping showed in Fig. 4
behaviour. is launched from PCA which IP address is 20.0.0.10. It is
called h1. This ping goes through the network using the route
IV. ADDING ARTIFICIAL INTELLIGENCE determined by the routing protocol according to the route table
In this section, we are going to discuss how AI can be showed in Fig. 5. As a conclusion, this routing architecture
added to the proposal. As was said before, Quagga is an open- proposal works and it is able to communicate different
source routing suite programmed in C. This is a great subnetworks inside a SDN and test it using Mininet.
advantage that allows modifying the routing algorithm. C. Measurements of QoS
The algorithm showed in Fig. 2 provides a reinforcement After testing that there is communication between the
learning method to establish a path between a source and a different subnetworks inside the topology, the next step is to
destination. Given a set of possible paths I and a set of network measure the QoS level that a virtualized network offer when
measurements (delay, loss rate and bandwidth) called M, the our proposed protocol is used in comparison of the results
agent calculate for every available path i a cost ci by assigning obtained when a traditional OSPF protocol is running.
different weights w1, w2 and w3 to the parameters. The data
transmission is sent by the path with less cost. After spending Fig. 6 shows the average value, in ms., of delay (Fig. 6a)
some time, the following node in the path gives to the agent a and jitter (Fig. 6b) as a function of the packet loss rate.
reward d which is calculated using network parameters about According to results, the average values of delay and jitter
the transmission through the path and a transfer function. This increase following an exponential function. As we can see,
reward is the parameter that the agent (the router) wants to when the packet loss rate is about 50%, the value of average
increase as much as possible. With that feedback from the delay is around 45 ms., while the value of average jitter is
paths, the agent adjusts again the weights trying to obtain a higher than 50 ms. These values are translated into a very bad
greater reward from a better path. This is the learning process video visualization since there are continuous cuts of images
in which the agent learns to choose the most important and the appearance of large numbers of black pixels. These
parameters to take account in the routing process. To execute measurements are important because they demonstrate that
this algorithm, the different elements present in the network packet loss rate is an important factor to take into account when
need to exchange a set of messages. Fig. 3 shows the message the traffic flow has to be forwarded through an alternative path.
exchange. D. Discussion
This algorithm can be implemented in C language adding In order to check the feasibility of our proposal, we finally,
the learning process to a NN. Because Quagga allows compare two scenarios. The first one is the SDN network
modifying the routing daemons and recompile them, this running a traditional OSPF routing protocols, where the
algorithm can be easily added into Quagga suite using OPSF or network establishes the best routes as the inverse value of
RIP daemons as a skeleton. available bandwidth.
 ICC2017: WT04-5thIEEE International Workshop on Smart Communication Protocols and Algorithms (SCPA 2017)

Fig. 1. Proposed topology.

Network Nodes SDN Controller

Random Weights Module
Routing Module
Start w1, w2 and w3 Obtain M
Caculation Statistics Request
Start Event Statistics Reply

Statistics Reply
Calculate ci for every Obtain reward d
path i Є I and establish and recalculate W Best routes Information
the path with minimum according to the Updating routing
ci feedback tables information

Fig. 2. Reinforcement learning routing algorithm Fig. 3. Message exchange diagram.

The second scenario implements the routing proposal, with the fewer peaks that reach 300 ms with the use of the conventional
inclusion of reinforcement learning routing algorithm over OSPF protocol (Fig.7a), the most peaks reach the 300 ms. The
OSPF. In the first case, the best path is through R2 because the value of jitter for our proposal (Fig. 8b) remains in most part of
available bandwidth is always 100 Mbps while path through time lower than 40 ms. while the traditional OSPF (Fig. 8a)
R3 is 10 Mbps. However, using the intelligent algorithm, the routing is bigger than this value in almost all the time.
best route is calculated through R3 because the original route Moreover, at the end of transmission, there is a significant peak
registers a 30% of packet loss and the algorithm notices that the caused by the packet losses. As we can see, although the
QoS would be limited by the packet loss rate. Thus, the alternative path calculated by our proposal presents a very
bandwidth seems not to be so important. Fig. 7 and Fig. 8 show limited capacity the results are better that the results offered by
a comparison between the QoS measurements in both the path calculated by traditional OSPF.
scenarios. Regarding the delay, our proposal (Fig.7b) presents
60 Average Delay
Time (ms)

40

20 a)
0
0 10 20 30 40 50
Packet Loss rate (%)
Fig. 4. Routing ping test working.
60 Average Jitter

40
Time (ms)

20
b)
0
0 10 20 30 40 50
Packet Loss rate (%)
Fig. 5. Route table in PCA. Fig. 6. Average delay and Average jiter in a traditional network as a
function of the packet loss rate.
 ICC2017: WT04-5thIEEE International Workshop on Smart Communication Protocols and Algorithms (SCPA 2017)

Delay (ms)
1000 OSPF a) a)
800 100 OSPF
80

Jitter (ms)
600
400 60
200 40
0 20
0 2000 4000 6000 8000 10000 0
Packets 0 2000 4000 6000 8000 10000
1000 Proposal Packets
Delay (ms)

800 b) 100 Proposal b)

Jitter (ms)
80
600 60
400 40
200 20
0
0
0 2000 4000 6000 8000 10000
0 2000 4000 6000 8000 10000 Packets
Packets
Fig. 7. Delay comparision. Fig. 8. Jitter comparision.

VI. CONCLUSION AND FUTURE WORK [2] OpenFlow: Learn more. Available at
http://archive.openflow.org/wp/learnmore/ [Last access: Dec. 15th
In this paper, we have presented a new distributed routing 2016 ]
proposal implemented over SDN. We have analyzed the way [3] V. P. Nigam, and D. Graupe, “A neural-network-based detection of
of building the SDN topology that runs routing protocol in a epilepsy”, A Journal of Progress in Neurosurgery, Neurology and
distributed way. Moreover, we have designed an intelligent Neurosciences, 2013, Vol. 26, Pp 55-60.
algorithm based on reinforcement learning to improve some [4] B. A. A. Nunes, Marc Mendonca, Xuan-Nam Nguyen, Katia
Obraczka, and Thierry Turletti, “A Survey of Software-Defined
aspects of routing. This algorithm is added to the OSPF routing Networking: Past, Present, and Future of Programmable Networks”.
protocol using Quagga which allows modifying the routing IEEE Communications Surveys & Tutorials, 2014, Vol. 16, No. 3, pp.
algorithms. The proposed routing algorithm is tested and 1617-1634.
compared with a traditional OSPF routing protocol over a SDN [5] M. Kobayashi, S. Seetharaman, G. Parulkar, G. Appenzeller, J. Little,
topology. The results show that the routing proposal works J. van Reijendam, P. Weissmann, and N. McKeownv, “Maturing of
properly and it reaches better QoS features than the traditional OpenFlow and Software Defined Networking through Deployments”,
Computer Networks. 2014, Vol. 61, Pp 151–175.
one. Our proposal clearly achieves a more stable route with less
[6] K. Kaur, J. Singh, and N.S. Ghumman, “Mininet as Software Defined,
loss rate that implies to have lower delay. In addition, the jitter Networking Testing Platform”, In proc. of the International
obtained with the proposal is significantly better than the Conference on Communication, Computing & Systems
values offered by the traditional routing. Thereby, by (ICCCS.2014), Feb 20- 21, 2014, Chennai, India.
intelligently avoiding the routes with high loss rate (in a [7] J. M Jimenez, O. Romero, A. Rego, A. Dilendra, and J. Lloret, “Study
dynamic way) a better QoS and quality of experience has been of Multimedia Delivery over Software Defined Networks”, Network
obtained. As a future work, we want to test the algorithm in Protocols and Algorithms, 2015, Vol. 7, No. 4, Pp. 37-62.
more complex scenarios and in a physical SDN [13][14]. [8] J. M. Jimenez, O. Romero, A. Rego, A. Dilendra, and J. Lloret,
“Performance Study of a Software Defined Network Emulator”, the
ACKNOWLEDGMENT Eleventh International Conference on Internet Monitoring and
Protection, (ICIMP 2016)- May 22 - 26, 2016 - Valencia, Spain.
This work has been supported by the “Ministerio de [9] H. A. A. Al-Rawi, M. A. Ng, and K.-L. A. Yau, “Application of
Economía y Competitividad”, through the “Convocatoria 2014. reinforcement learning to routing in distributed wireless networks: a
Proyectos I+D - Programa Estatal de Investigación Científica y review”, Artificial Intelligence Review, 2015. Vol. 43, No.3, Pp.381-
416.
Técnica de Excelencia” in the “Subprograma Estatal de
[10] R. Desai, and B:P. Patil, “Cooperative reinforcement learning
Generación de Conocimiento”, project TIN2014-57991-C3-1-P approach for routing in ad hoc networks”. In proc. of the 2015
by the Spanish Ministry of Economy and Competitiveness and International Conference on Pervasive Computing (IEEE ICPC 2015),
the European Regional Development Fund (Projects TIN2013- January 8-10, 2015, Pune (India). Pp. 1-5.
46223-P, and TEC2016-76795-C6-4-R), by the “Ministerio de [11] J. Solanki, and A. Chauhan, “A Reinforcement Learning Network
based Novel Adaptive Routing Algorithm for Wireless Ad-Hoc
Educación, Cultura y Deporte”, through the “Ayudas para Network”. International Journal of Science Technology &
contratos predoctorales de Formación del Profesorado Engineering., 2015, Vol. 1, No. 12, Pp. 135-142.
Universitario FPU (Convocatoria 2015)”. Grant number [12] P. Jakma, and D. Lamparter, " Introduction to the quagga routing
FPU15/06837 and the “Programa para la Formación de suite", IEEE Network, 2014, Vol. 28, No. 2, , Pp. 42-48.
Personal Investigador – (FPI-2015-S2-884)” by the [13] Q. Yan, Q. Gong and F-A. Deng, “Detection of DDoS Attacks
“Universitat Politecnica de Valencia”. Against Wireless SDN Controllers Based on the Fuzzy Synthetic
Evaluation Decision-making Model”, Ad hoc and sensor wireless
networks, 2016, Vol. 33, No.1-4. Pp. 275-299.
REFERENCES
[14] H. Yang, J. Zhang, Y. Zhao, Y. Ji, J. Han, Y. Lin, S. Qiu, Y. Lee,
[1] N. Feamster, J. Rexford, E. Zegura, “The Road to SDN: An “Time-aware Software Defined Networking for OpenFlow-based
intellectual history of programmable networks”, ACM Queue, 2013, Datacenter Optical Networks”, Network Protocols and Algorithms,
Vol. 11, No. 12, Pp. 1-20. 2014, Vol 6, No 4,Pp. 77-91.