Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Why do we need SD-WAN?

Why do we need SD-WAN?

Inefficiencies of Traditional WAN
In recent years, businesses are embracing digital transformation more rapidly than ever expected. Remote working and online
meetings are now standards. Many applications are moved to the public cloud and many services are now available over the
Internet. Companies want to reduce costs and manage their infrastructure more effectively. However, the traditional wide-area
network (WAN) was designed to connect users at remote sites to applications hosted in the company's data center. Dedicated
leased lines and MPLS circuits were used to provide secure and reliable connectivity to the DC. Although some applications are
now in public clouds and the Internet, the traffic from the remote sites must come to the DC first and then be routed to the
Public Cloud and back. This concept is visualized in figure 1.

W X P

Enterprise Applications

Remote Sites Data Center

WAN

Figure 1. Traditional WAN architecture - Centralized Connectivity

This WAN design no longer works well in a digital world where applications are out of the data center, and the users consuming
those applications are using a diverse set of mobile devices. As businesses are rapidly adopting Software-as-a-service (SaaS)
and Infrastructure-as-a-service (IaaS) models, it is pretty common to have ERP applications hosted in AWS, office applications
such as Office 365 being used over the Internet, company-specific apps hosted in the HQ data center, and 3rd party
applications hosted in another datacenter. In this scenario, the traditional WAN connectivity between the branches and the DC
is not the most effective way to connect to all applications and creates the following inefficiencies:

Costs - Increased bandwidth demands are forcing companies to upgrade their private WAN circuits, which is expensive;
Higher Latency - Moving the traffic from remote sites to the DC and then to the Cloud increases the overall round-trip
times;
Availability - Running everything through the company's data center creates a Single-point-of-failure;
Velocity - Deploying private MPLS circuits is a slow and tedious process that usually slows down the roll-out of new
remote sites.
With the adoption of Public Cloud, companies started rethinking their WAN designs. Organizations decided to equip their
remote site with Direct Internet Access links and to offload cloud-native applications directly over the Internet. Internet
availability then becomes a very important part of the branch/campus operations.

But how do you make sure all remote sites have an Internet connection in 99.99% of the time? Well, the answer is - you
purchase at least two independent Internet connections from at least two service providers. Combining this with the
emergence of 4G/5G and having in mind that commodity internet circuits offer higher capacity at significantly lower price
points , it is a natural consequence that companies started exploring ways to rely less on Private WAN and take advantage of
the Internet circuits.

W X P

Internet Cloud Enterprise

DC2
DC1
Home Users
Remote Sites

SD-WAN
Figure 2. Software-Defined WAN architecture - Decentralized Connectivity

Software-defined WAN (SD-WAN) solutions have been designed to address these challenges. SD-WAN is part of the broader
technology trend called software-defined networking (SDN). This is a new centralized approach to network management that
abstracts the underlying network infrastructure away from the services and applications that run over the network.

De-centralized Network Management

For many years, networks have been deployed and operated in a decentralized manner, meaning each device is individually
managed and operated by network administrators. Let's compare this to a centralized approach. If I take you back to the old
days of personal computers. Installing new hardware or software required users to configure the individual elements of the PC.
For example, you bought a new sound card for your PC. You install the card and then you must go to the website of the
manufacturer and download the drivers for your operating system. Then you install the drivers and resolve any incompatibility
issues. And only then you start listening to music, eventually.
 Computer

Individual
Components as a System

Figure 3. Personal Computer as a System

Now compare this to the process nowadays. You just plug the hardware component you bought and the PC handles everything
else for you. You just signify the intent to listen to music and the operating system configures all underlying components
necessary to play the music. You are using a personal computer as one system and not as a group of individual components.

And the one million dollar question is - Why can't we apply this logic to IP Networks? Why the network can't be thought of and
administered as a system instead of a collection of individual devices such as routers, switches, and firewalls?

Users configure individual

components (network devices)
in order to use the system (the
network).

System

Individual
Components

Figure 4. Using Traditional Networks

As of today, most networks in the world are still being operated and configured manually using traditional "per device"
methods. Although many of us will agree that this device per device approach has many significant drawbacks such as:
 Risk of human error - Many kinds of research have been made over the years that shows that most network outages
happen because of a configuration error (ultimately a human error).
Services velocity - Many network engineers would agree that the network is the slowest of all technology verticals when
it comes to enabling a new feature or service. Configuring each network device one by one using CLI is just not a scalable
approach. Don't get me wrong, we all love the command line, but it was not designed to make massive scale configuration
changes to multiple devices at the same time.
Analytics - In traditional networks, where devices are managed in a decentralized manner, almost no one knows the "big
picture". In many cases, different devices are operated by different teams, and a centralized collection of analytic data
and network-wide configuration sanity is a very hard task.

Benefits of SD-WAN
Many business and technical researchers agree that the next-generation networks would be deployed and operated As-a-
System and not as a collection of individual network devices. Software-Defined WAN (SD-WAN) is a centralized approach to
managing and operating large-scale WAN networks.

Single Management Plane

One of the main ideas of SD-WAN is to administrate the WAN through a single centralized management plane, and the system
itself to manage the underlying network devices. This would provide many benefits, business opportunities, and a better overall
user experience.

Users simply signify intent to use the

system (the network) and it
configures all underlying
components (network devices).

System

Individual
Components

Figure 5. Using Networks in an Intent-Based manner

Let's look at the most obvious advantages:

Automation - Operating and administrating a network as a system in a centralized manner removes most of the
complexity of large-scale networks. The whole idea of centralized management is to use Automation. It creates simplified
consistent deployment and operational models.
 Reduced cost - SD-WAN's automation allows leveraging any combination of transport services such as
Broadband Internet circuits, 4G/5G, MPLS, and any other public transport – to securely connect users to applications.
Improved uptime - Centralized management and automation could eliminate most human errors such as
misconfigurations, wrong designs, and deployments. This would significantly improve the overall uptime of the network.
More secure - Centrally managed networks can easily apply end-to-end security policy across the enterprise network
which is very hard to do using the box-to-box approach.
Better analytics - To be honest, in traditional large-scale networks, very few people (often nobody) know and could grasp
the big picture. Because of the high number of network devices and the large volume of analytics data, most of the time
the data is very hard to read and interpret quickly. Treating a network as one system enables a single management
console that presents data from multiple sources in a unified display.

Forwarding based on auxiliary information

In traditional WAN networks, routers make forwarding decisions based on a limited set of information. Traditional routing
protocols usually consider only the link bandwidth and link status. However, having multiple different WAN transport attached
to a remote site and wanting to use them in an active-active manner requires a more complex routing forwarding
process. I have read one of the best analogies of this in the Cisco book "Cisco SD-WAN Cloud scale architecture".

Consider the analogy of traveling by car. Prior to the emergence of navigation software such as Google Maps, for the travel
from New York to Boston, a paper road map was typically used to identify the best route. If there was road closure or delay
along the route, the driver would be forced to find an alternative route based on limited information. This is the way WAN
routers operate in a traditional WAN network. Each router makes its own autonomous decisions about how to route the
packets, based on a limited view of the topology around it.

Nowadays
In the old days - Drivers are aware of road blocks and construction works
- Drivers only know that different roads exist
- Traffic Jams and road accidents
- Real-time auxiliary information is not available
- Toll Taxes and much more info

Figure 6. Analogy between routing decisions and road navigation

Now compare this approach to today's road navigation with GPS. Navigation software such as Google Maps can help a driver
to avoid road closures, accidents, travel delays, and inefficient routes. This is possible because the navigation software relies
on satellites in the sky that have a real-time sophisticated view of the road network. With SD-WAN, edge routers can now rely
on the centralized control/management plane for auxiliary information on how to forward the traffic. In the same way, as the
GPS helps drivers avoid travel delays, SD-WAN helps routers avoid jitter, packet loss, and latency in the network.

Cisco's SD-WAN solutions

Cisco offers two different SD-WAN products through its acquisitions of Meraki and Viptela. Both products are full-fledged SD-
WAN solutions and have several overlapping features. However, Cisco has made it clear that Meraki and Viptela are geared
toward two different markets.
 Meraki is designed for small and mid-sized companies that want simplicity and ease of use above everything else.
Deploying the Meraki SD-WAN solution is easier than Viptela and if the organization does not have any specific niche
requirements, it would definitely be the right choice.
Viptela has more advanced features available and requires a sophisticated network design and architecture. The product
is designed for large-scale enterprise-level networks and has a high degree of customization.

Cisco Viptela VS Cisco Meraki

Enterprise-level SD-WAN solution SD-WAN solution with a basic level of
supporting complex WAN topologies customization designed for small and
with a high degree of customization medium sized organizations

Figure 7. Cisco's SD-WAN solutions

In this course, we are going to deep-dive into the Cisco Viptela SD-WAN and won't cover the Meraki product.

« Previous Lesson
Introduction to SD-WAN
Next Lesson
What is SD-WAN? »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

OMP Source Preference

Understanding OMP Routing

OMP TLOC-Action
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / What is SD-WAN?

What is SD-WAN?
Traditional WAN was designed to route traffic from remote sites to the company's data centers using private MPLS circuits.
However, the business trends have moved the applications out of the data center and into the public clouds such as Microsoft
Azure and Amazon Web Services (AWS). Nowadays, moving the users' traffic from branches to the enterprise DC and then out
to the cloud or the Internet and back is inefficient, expensive, and not scalable. Also, the rapid digital transformation of
enterprises creates new requirements for security, cloud and Internet connectivity, WAN management, and application
performance.

Cisco SD-WAN is a Wide Area Network (WAN) overlay architecture that applies the principles of Software-Defined Networking
(SDN) into the traditional WAN. It is designed to meet the needs of modern enterprise applications and the rapidly growing
security requirements.
 vManage

On-prem Cloud Multi-tenant

ANY DEPLOYMENT

Routing Analytics Cloud (IaaS) Segmentation Security

ANY SERVICE

Internet MPLS 4G/5G Satellite

ANY TRANSPORT

Campus DataCenter Industrial SOHO Cloud

ANY LOCATION

Figure 1. Cisco SD-WAN Architecture

Cisco Viptela SD-WAN solution provides the following improvements over the traditional WAN design:

Connecting any location in a fast, secure, and highly available manner using Zero-Touch Provisioning (ZTP).
Establishing a transport-independent WAN using any type of underlying transport.
Abstracting the underlying WAN infrastructure away from the services and applications that run over the network such
as WAN Routing, Segmentations, Analytics, IaaS, and Multitenancy.
Providing end-to-end security from remote sites to the Internet, Cloud, and SaaS applications.
Providing a single pane of glass (SPOG) for management, analytics, and configuration policy across the enterprise WAN.
Providing southbound REST APIs that enable enterprises to create their own unique services and meet any niche
requirements.

Figure 1 summarizes the key architectural improvements over the Traditional WAN design. Let's now look at the components of
the Cisco SD-WAN solution.
SD-WAN Components
Cisco Viptela SD-WAN solution is made up of four segregated planes - Orchestration plane, Management Plane, Control Plane,
and Data Plane. Each plane has its own functions and responsibilities and is abstracted away from the other planes. For
example, if you replace a device in the data plane, that does not affect the control/management or orchestration plane. The
same applies if you replace a controller in the Control plane or the Management Plane.

Compare this to the Tradition WAN design where each device participates in the data plane (forwarding actual packets), in the
control plane (for example running OSPF, BGP, PIM and participate in the topology formation), and in the management plane (is
actively managed via CLI).

GUI Automation
Management Plane

vAnalytics vManage
Orchestration Plane
vBond

Control Plane
vSmart
Controllers

MPLS Internet 4G/5G

vEdge Routers
Data Plane

Cloud DC Campus Branch

Figure 2. Cisco SD-WAN Components

Cisco vManage
Cisco vMange is the Management Plane of the SD-WAN system. It runs the user interface of the system and is the dashboard
network administrators interact with daily. It is responsible for collecting network telemetry data, run analytics, and alert on
events in the SD-WAN fabric. It is also the tool that admins use to create device templates, push configurations, and perform
overlay traffic engineering.

Cisco vManage can be deployed on-prem, in the public cloud, or in the Cisco cloud-hosted environment. It is significantly
resource-intensive and most customers go with the cloud options.

Cisco vBond
Cisco vBond is the Orchestration Plane of the SD-WAN system. Its job is to orchestrate the process of onboarding new
unconfigured devices to the SD-WAN fabric. It is responsible for the authentication and whitelisting of vEdge routers and
control/management information distribution.
Cisco vSmart
Cisco vSmart is the Control Plane of the SD-WAN system. vSmart controllers are the brain of the overlay fabric. They advertise
routing, policies, and security. They are positioned as hub routers in the control plane topology and all vEdge routers peer with
all vSmart controllers. For experienced network engineers, vSmart controllers are like BGP Route-reflectors or DMVPN NHRP
routers. However, it is important to understand these appliances are not part of the Data Plane and do not participate in packet
forwarding.

Cisco vEdge
Cisco vEdge devices represent the Data Plane of the SD-WAN system. They sit at the WAN edge and establish the network
fabric and join the SD-WAN overlay. If you look at the architecture shown in figure 1, everything southbound of the vEdge
routers is typically traditional networking - offices, data centers, and branches. Everything northbound of the vEdge routers is
the SD-WAN system itself. vEdge routers exchange routing information with the vSmart controllers over the Overlay
Management Protocol (OMP). If for example, we have a campus network running OSPF. At the vEdge devices, the OSPF routes
are redistributed into the SD-WAN fabric to the vSmart controllers via OMP and then the vSmart controllers populate this
routing information to other vEdge devices if it is required by the WAN topology.

Cisco vSmart

OM
P
Up

e
at
da

d
te

Control Plane
Up
P
OM

Overlay Tunnel 1
Routing Routing
Information Overlay Tunnel 2 Information

Data Plane

Campus Network Branch Network

Figure 3. Cisco SD-WAN OMP Protocol

The WAN Edge routers could be Viptela platforms or Cisco IOS-XE devices. They can be virtual or physical appliances. vEdges
are auto-configured by the system. Back in the Viptela days, this process was called Zero-Touch Provisioning (ZTP) and
nowadays with the Cisco devices, it is called Cisco Plug-and-Play (PnP). Both terms actually mean the same and are
interchangeable.

Overlay Management Protocol (OMP)

The Cisco vSmart controllers use the Overlay Management Protocol (OMP) to manage the overlay network fabric. Upon joining
the SD-WAN fabric, each vEdge router establishes one permanent secure connection to the vSmart controller via each available
transport as shown in figure 4. These connections, usually DTLS, are then used by the vEdges to exchange control plane
information to the controller such as prefixes, crypto keys, and policy information.
 Cisco vSmart
Controller

Each vEdge keeps one

permanent connection
to the vSmart controller MPLS Internet
via each available
transport

vEdge 1 vEdge 2

Campus Network
Figure 4. Cisco vEdges OMP peering

It is important to note that OMP peering is never made between the vEdge routers onsite. This is due to the separation of
control and data plane in the SD-WAN architecture.

Three types of routes are advertised with OMP:

1. OMP routes (vRouter) are prefixes at the local site that are redistributed into OMP and advertised towards the controllers.
These might be OSPF or BGP routes, or any other routing information present on the site.
2. TLOC routes (Transport locations) are the tunnel endpoints on the WAN Edge routers that connect to the transport
networks. These routes are represented by three components- the system IP address, link color, and encapsulation type.
3. Service routes are used to exchange services such as firewall, IPS, application-specific optimizations, and load-
balancers.

Let's leave the things here and continue with our exploration of the Cisco SD-WAN architecture in the next lesson.

Key Takeaways
Let me try to summarize into a short table what the Software-Defined WAN brings compared to the Traditional WAN design.


Traditional WAN Software-Defined WAN

Integration Hardware Centric Software Centric

Administration Manual box-to-box Automated

Extension Closed Programmable via REST APIs

Operations Reactive Predictive

Drivers Network Intent Business Intent

Differences between the Software-Defined WAN and Traditional WAN


Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / How Cisco SD-WAN works?

How Cisco SD-WAN works?

In the last lesson, we looked at the main architectural components of the Cisco SD-WAN solution. Now in this lesson, we are
going to try to explain how everything fits together to create an operational overlay fabric that can move user flows more
securely and intelligently than Traditional WAN.

SD-WAN Deployment
When a company decides to migrate its traditional WAN architecture to Software-Defined WAN, the thing that always comes
first is to deploy the controllers. The next step is to migrate the main data centers and hub sites and lastly the remote sites
such as campuses and branches.

Order of Deployment

Deploy Controllers Deploy Large Sites Deploy Branches

vManage vEdge vEdge

SOHO

Datacenter HQ Campus Branch

vBond vSmart Cloud

Figure 1. Cisco SD-WAN Order of Deployment

The main idea for doing it in this sequence is to have the hub sites route the traffic between the SD-WAN and non-SD-WAN
sites for the period of the migration. Of course, if it is a brand new ground-up deployment, the sequence does not matter that
much.

Controllers Deployment Options

One of the main advantages of the Software-Defined WAN is that the controllers can be deployed in the public cloud. This can
significantly reduce the CAPEX/OPEX costs and improve the overall availability and redundancy of the management
plane/control plane. Compare this model to the scenario in which you have all controllers deployed on-premises. You need to
accommodate rack space, power, cooling, physical servers, hypervisor, and virtual machines or containers. You have to
manage redundancy and backups on your own. Using the cloud options, you can consume the management/control plane as
IaaS (Infrastructure-as-a-Service) or even SaaS (Software-as-a-Service).

Cisco offers the following options to customers to choose from:

Cisco-hosted cloud - The information that I have found regarding the existing deployments shows that most customers
(above 90%) opt for this approach. This is also the vendor's recommended model because Cisco takes care of
provisioning all controllers, they handle the backup and disaster recovery. The customer is basically consuming the SD-
WAN control plane as a Software-as-a-Service (SaaS) by using the vManage to create custom configuration templates for
their device and administer the overlay fabric.
Public cloud - The customer could decide to host the controllers in the public clouds such as Azure and AWS. In this
scenario, the controllers could be managed by a service provider or by the customer.
 On-prem - Of course, the controllers can be deployed in the company's data centers or private clouds. In this scenario, the
customer is responsible for backups and disaster recoveries. This is usually the case with financial and government
institutions that must be compliant with regional regulators.

On-prem Deployment Cloud Hosted

vSmart vSmart
Controllers Controllers
vManage vBond vManage vBond

VM VM VM VM VM VM

ESXi or KVM Azure or AWS

Physical Server

Figure 2. Cisco SD-WAN Deployment Options

Once the controllers are up and running, they must establish secure connections between them. As of now, there are two
options to choose from when it comes to the underlying secure protocol - TLS which uses TCP transport, and DTLS which uses
UDP transport. By default, all controllers use the DTLS option.

If the SD-WAN is deployed in a zero-trust environment, figure 3 shows the Layer 4 information for all permanent connections
between the controllers. Note that each core on vManage and vSmart makes a permanent DTLS connection to the vBond
resulting in four connections between vManage and vBond and two connections between vSmart and vBond.

vBond

UDP UDP UDP UDP UDP UDP

12346 12346 12346 12346 12346 12346

UDP UDP UDP UDP UDP UDP

12346 12446 12546 12646 12346 12446

UDP UDP
if DTLS
12346 12346
or
TCP TCP
if TLS
Random 23456
vManage vSmart
(4 cores) (2 cores)

Figure 3. Cisco SD-WAN Control Connections

WAN Edge Routers Onboarding
Secure onboarding of the WAN edge devices is a very important part of the SD-WAN solution.

Identity and Whitelisting

The Cisco SD-WAN solution uses a whitelisting model for authenticating and trusting the vEdge devices. This means that
before a WAN edge router is allowed to join the control plane, it has to be known by all SD-WAN controllers beforehand. Each
device is uniquely identified by its Chassis ID and certificate serial number.

Controllers reachability
Once the SD-WAN controllers are deployed and have valid certificates, WAN edge routers can start the onboarding process. At
this point, the most important thing is to make sure that the vEdge appliances have reachability to all controllers via all
available transports. It sounds like an easy and straightforward step at first, but when you look more deeply into it, you can see
that there are some decisions to be made.

SD-WAN Controllers SD-WAN Controllers

Method
2

Data Center Public Cloud Public Cloud

Public IPs Public IPs

MPLS Internet MPLS Internet

vEdge vEdge
Method
1
Remote Site Remote Site

Figure 4. Control Plane Communications

The Edge device tries to establish a control connection over each provisioned transport, first initiating one to the vBond
orchestrator before attempting to connect to the other controllers. All available transports are tried one at a time, starting with
the WAN connection to the lowest interface number. Let's look at the typical scenario where a remote site has one private
MPLS circuit and one broadband Internet connection. The WAN edge device would try to connect to controllers over the
Internet and the MPLS line. But if the controllers are deployed in a public cloud or any other 3rd part cloud, would the public IP
addresses of the controllers be reachable over the MPLS circuit by default? I guess not, there is no MPLS service having all
public prefixes redistributed into it.

There are three common implementations that solve this issue:

1. The MPLS has reachability to the public cloud by being routed through a data center or regional hub that has both
transport. This method is shown on the left side of figure 4.
2. The public routable IP addresses of the controllers are redistributed into the MPLS cloud and the provider edge router
advertises them to the vEdge. This method is shown on the right side of figure 4.
3. It is possible to establish a control plane connection through the Internet connection only. The Edge would be able to join
the SD-WAN fabric but would not have a control plane redundancy, so this approach is not recommended at all.

Onboarding process
 vEdge Onboarding Options

Automated Deployment Manual Deployment

Zero-Touch
Plug-and-Play
Provisioning Bootstrap Manual
(PnP)
(ZTP)

Cisco XE Devices Viptela Devices Cisco XE Devices Viptela Devices

Cisco XE Devices

Figure 5. WAN Edge Routers Onboarding Options

Joining the overlay fabric

Let's now put everything together and follow each step of the process of joining a WAN edge device to the overlay fabric. Let's
say that for this example we are going to use a Viptela vEdge 1000.

Step 0. IP Reachability - Upon bootup, the vEdge device obtains an IP address, default gateway, and DNS information via
DHCP. If no DHCP service is available onsite, this information could be configured manually using CLI or using a
configuration template.
Step 1. Zero-Touch Provisioning - The WAN Edge router tries to reach the ZTP server by resolving the URL
ztp.viptela.com and uses HTTPS to get information about the SD-WAN vBond orchestrator along with the organization
name.
Step 2. Authentication - The WAN edge device authenticates to the orchestrator with its root-certificate and serial
number. If the authentication is successful, the vBond sends back the vManage and vSmart
controller information.
Step 3. Connection to the Management Plane - The Edge then establishes a secure connection to the vManage and
downloads the configuration using NETCONF.
Step 4. Connection to the Control Plane - If all previous steps are successful, the router establishes a secure connection
to the vSmart controllers and joins the SD-WAN overlay fabric
 PnP/ZTP service vBond vManage vSmart

PnP

Zero-Touch Autentication Netconf OMP Peering

Provisioning 1 transient 1 permanent 1 permanent
1 transient connection connection connection per
connection transport

2
4
1

vEdge
Figure 6. Cisco SD-WAN vEdge Connections

The process of jointing the SD-WAN overlay is visualized in Figure 6. Note that some secure connections are transient and
some are permanent.

SD-WAN Operation, Administration, and Management

The beauty of the Software-Defined WAN is that it is operated as-a-System rather than the traditional box-to-box approach. This
opens up a whole new world of possibilities when it comes to the Operation, Administration, and Management (OAM) of the
solution. Some of the main benefits are:

Centralized management - and operational simplicity, resulting in reduced change and deployment times.
Transport-independent overlay - Because the underlay transport is abstracted away from the overlay fabric, any
combination of transports can be used in an active/active fashion. This significantly reduces the bandwidth costs of the
company.
Sophisticated security - If you compare the traditional control plane security of OSPF and BGP to the control plane
encryption of the SD-WAN, the latter is obviously more comprehensive using certificate identity with a zero-trust security
model.
Application visibility - Real-time analysis and application visibility are a core part of the solution. This enables the
enforcement of service-level agreements (SLA) and tracking of specific performance metrics
 Any controllers deployment
- Public Cloud
- Private Cloud vManage vBond vSmart
- On-premises
vEdge vEdge Control Plane

vEdge

Any overlay topology

- Hub and Spoke
vEdge vEdge
- Full mesh
- Partial Mesh

TLOCs TLOCs
Any underlay transport T1 T1

- Broadband T2
INET
HQ SOHO
- MPLS
- 4G/5G T1
TLOCs T1
T2
MPLS TLOCs
T2

DC Campus

Figure 7. SD-WAN Established Overlay

We obviously won't be able to go deeply into any step of the process of establishing the overlay fabric. However, this is a short
summary of how Cisco SD-WAN works. We are going to deep-dive into each step of the process in the next sections of this
course.

« Previous Lesson
What is SD-WAN?
Next Lesson
Cisco SD-WAN Main Principles »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Cisco SD-WAN Main Principles

Cisco SD-WAN Main Principles

In this lesson, we will explore the main principles that the Cisco Software-defined Wide-area Network (SD-WAN) architecture
has adopted to solve the inefficiencies of the traditional WAN. The solution utilizes some well-known and time-tested network
technologies in combination with some new innovative ideas. It transforms the complex legacy WAN infrastructure into a
secure and scalable overlay fabric. Cisco SD-WAN achieves this by using the following techniques:

Separating transport from the service side of the network

Separating control, data, and management planes
Secure the Data-Plane Automatically
Managing the fabric through centralized policies
Secure zero-touch provisioning and onboarding of new devices

Separating transport from the service side of the network

The first fundamental technique that Cisco SD-WAN utilizes is the separation of the service and transport sides of the network.

When an organization with a traditional WAN architecture grows, the wide-area network becomes increasingly expensive and
complex to manage. One of the main reasons is that there is no clear separation between users, applications, switches, and
routers on the service side of the network and the WAN links and service providers’ devices on the transport side.
Subsequently, the transport routers need to know the non-transport prefixes , making it hard to influence the WAN routing
decisions without affecting the services running on top. Figure 1 illustrates a traditional WAN with no clear separation between
service and transport sides and highlights the main inefficiencies.

Legacy routers Legacy routers

INET

MPLS

No clear separation Control decisions at Security is manually

between service and every hop implemented at every
transport sides hop
Figure 1. Legacy WAN

Cisco SD-WAN uses the time-tested concept implemented in all service providers’ networks, where the MPLS cloud is just a
transport network that does not need to know any customer prefixes . The function of the MPLS network is only to transport
packets from an entry point to an exit point of the transport cloud. Similarly, Cisco SD-WAN separates the transport side of the
network into a dedicated transport segment - VPN0. The function of the transport side is to route packets from one transport
router to another. A router on the transport side needs to know only how to reach the destination router on the other side of the
transport cloud. It does not need to know about any service-side prefixes.
 vEdges vEdges

INET

MPLS

Service Side Transport Side Service Side

VPN 1 - 65527 VPN 0 VPN 1 - 65527

Figure 2. Separating transport from the service side of the network

Separating the transport from the service side of the network abstracts the wide-area network (WAN) away from the
applications running on top. This approach has many benefits such as:

Network admins can influence the routing decisions into the WAN independently of the communication between users or
applications.
The solution can insert labels into packets and assign attributes to WAN circuits for optimal policy-based routing, load
balancing, and network segmentation/slicing.​
Security can be applied to the transport side independently of the users' traffic.
Any mix of public and private WAN transports can be used in an active-active, ECMP fashion.

Separating control, data, and management planes

The second fundamental technique that Cisco SD-WAN utilizes is the separation of control, data, and management planes.

The decoupling of control and data planes is a well-accepted concept within software-defined networking. The “control plane”
is the set of protocol messages used to control the network infrastructure. It is typically represented by the in-band exchange
of topology and link-state information between network devices in the form of OSPF/BGP or any other routing protocol
updates. Traditionally each network device runs its own instance of the control plane in firmware, makes independent routing
calculations, and determines its own Routing Information Base (RIB). Topology changes require routing updates to be
propagated box-by-box across the entire network. This process quickly becomes very complex and inefficient in large
environments with many routers.

That is why, to scale, most traditional routing protocols have techniques designed to break a large topology into smaller routing
domains such as OSPF areas, BGP confederations, IS-IS levels, and so on. Additionally, techniques such as aggregation and
redistribution are often used to help further the topology scale. Most network engineers are painfully familiar with scaling
inefficiencies of traditional routing protocols.
 vManage vSmart
Mgmt-Plane Control-plane

INET

MPLS

Service Side Transport Side Service Side

VPN 1 - 65527 VPN 0 VPN 1 - 65527

Figure 3. Secure the Data-Plane Automatically

Cisco SD-WAN has taken the more modern SDN approach to the network architecture. The solution decuples the control plane
from the data plane of all WAN edge routers and implements all control functions into a centralized software controller called
vSmart. It also decouples all management functions and implements them in a separate centralized controller called
vManage. Additionally, the solution introduces another network “plane” that runs vertically along the other two planes - a
centralized orchestration plane implemented into a dedicated controller called vBond. Cisco vBond ensures that all devices
allowed to join the overlay fabric are authenticated and white-listed. It makes sure that the infrastructure can be trusted and is
well secured against rogue devices.

The centralized management approach allows us to control and operate the network as-a-system which is much more efficient
than the traditional distributed method:

Management Plane - The centralized management plane (vManage) has a complete view of the network and pushes
templates and policies across the entire environment. It creates a network-wide configuration standardization which
significantly decreases the chance for misconfigurations and human errors. In large-scale environments with 1000+
nodes, it is much more efficient to only edit a configuration template once on vManage and push it down to all devices
instead of pre-configuring 1000+ devices manually box-to-box. vManage is also responsible for the software updates of
the entire SD-WAN fabric and its integrations with other cloud services. This controller is the single-pane-of-glass for
managing, operating, and troubleshooting the entire SD-WAN fabric.
Control Plane - The centralized control plane (vSmart) knows about all nodes and available paths in the environment.
Upon a state change in the network, the controller re-calculates the centralized routing table only once and distributes it
to all nodes as a single routing update. Individual network devices do not need to perform routing calculations or
exchange any control plane information between them. This centralized control-plane approach is much more efficient
than the traditional routing, where each device recalculates every time a state changes occur in the network.
Data Plane - Because network devices do not need to store and perform complex routing calculations anymore, more
hardware resources are available for packet forwarding. The edge devices download all necessary control and
management information from the controllers and send back network telemetry for their status.

However, implementing the control and management plane into centralized controllers has many additional advantages over
the traditional decentralized architecture:

The centralized controllers do not need to lie in the data path and can be anywhere on the transport side - on-prem, in a
private cloud provider, or consumed as a SaaS from an MSP.
The centralized controllers can be virtual appliances (VMs) or software containers that use off-the-shelf compute and
storage hardware.
Scale challenges associated with distributed routing protocols such as OSPF, IS-IS, and full-mesh BGP on the transport
side of the network are eliminated. However, the SD-WAN mode of operation devices can still run these protocols with
 other non-SD appliances if necessary.
The network can achieve segmentation without the need for complex signaling protocols such as MP-BGP, VRFs, MPLS,
etc.

Secure the Data-Plane Automatically

Another essential technique that Cisco SD-WAN utilizes is the automatic establishment of IPsec tunnels over every WAN
transport. Once an edge device vEdge router joins the Cisco SD-WAN control plane, it automatically tries to form an IPsec
tunnel to every other vEdge transport interface that it knows about. This behavior results in a full mesh of IPsec tunnels
between all transport links of all vEdges that aren’t at the same location (with the same site-id). The IPSEC tunnels use pre-
shared keys for better performance. The keys are pushed to the devices and rotated regularly by the vSmart controller. It uses
the secure channel of the established DTLS connection to download the keys to all edges. Additionally, vEdge routers exchange
the encryption keys associated with these IPsec tunnels through the vSmart controller, which is a considerable improvement
compared to traditional WAN.

IPsec tunnels
T1 T5

T2 INET T6

T3 T7

T4 MPLS T8
TLOCs

Service Side Transport Side Service Side

VPN 1 - 65527 VPN 0 VPN 1 - 65527

Figure 4. Secure the Data-Plane Automatically

In traditional IPsec environments, routers handle the key exchange using the Internet Key Exchange (IKE) protocol. We won’t
get into details about how IKE works, but the point is that each router (n) generates and exchanges a unique key with every
other remote router in the network. This means that in a fully meshed network, each router has to manage n2 key exchanges
and (n-1) keys. For example, in a network with 1000 routers, each router must handle 1 000 000 key exchanges and maintain
999 keys. Cisco SD-WAN does not utilize IKE at all, instead, routers exchange keys through the centralized control plane.

The automatically established full-mesh of IPsec tunnels is referred to as the Cisco SD-WAN overlay fabric. It ensures that the
data traffic going between all locations is secured.

This approach has many benefits:

The overlay fabric of IPsec tunnels ensures that the network is not prone to attacks and exploits from the transport side.​
The fabric ensures that user and application flow transverse the ISP networks encapsulated with new headers and
encrypted. It also provides the integrity of the transmitted data.
The centralized orchestration plane (vBond) ensures that all devices allowed to join the overlay fabric are authenticated
and white-listed. The centralized orchestrator makes sure that the infrastructure is trusted and secured against rogue
devices.
The overlay can be established over any physical transport the devices get connected to. These transports are known as
an underlay network.

Managing the fabric through centralized policies

Having a separate control plane allows us to manage the solution in a centralized fashion via Policies and Templates. A
Centralized policy configured on vSmart influences how the controller routing information is advertised among the WAN edge
routers. This allows network administrators to apply network-wide routing changes without having to configure each device
manually.
 Policy

dates OM
PU
p pda
MPU tes
O

IPsec tunnels
T1 T5

T2 T1

T3 T7

T4 T8

Service Side Transport Side Service Side

VPN 1 - 65527 VPN 0 VPN 1 - 65527

Figure 5. Managing the fabric through centralized policies

This approach has many benefits:

The centralized controller has a complete view of the environment and can make routing decisions based on auxiliary
information such as SLA policies, application types, segment types, etc.
Network administrators can implement business logic from a single-pane-of-glass, achieving efficiency at scale and
minimizing the number of touchpoints for provisioning.
The WAN network can scale horizontally - adding more WAN edge routers and remote sites does not affect the rest of the
network devices because the centralized controllers handle all control plane calculations.

Secure zero-touch provisioning and onboarding of new

devices
Cisco SD-WAN offers a fully automated process for onboarding new WAN edge devices. It allows network administrators to
provision new sites with minimal effort and involvement. A new unconfigured vEdge automatically discovers the network using
either one of the following processes - Zero Touch Provisioning (ZTP) if the device runs Viptela OS or Cisco Plug and Play
(PNP) if the device runs IOS-XE. Figure 6 illustrates a high-level overview of the onboarding process.
 PnP/ZTP vBond vManage vSmart

Brand new
unconfigured
vEdge router
1 2 3 4

INET

MPLS

Service Side Transport Side

VPN 1 - 65527 VPN 0
Figure 6. Secure zero-touch provisioning and onboarding of new devices

Suppose that the WAN edge router highlighted in red is vEdge Cloud running Viptela OS. The onboarding process will be as
follow:

The WAN edge router contacts the ZTP cloud service at ztp.viptela.com. The ZTP server checks if the organization name
exists in the ZTP database and verifies the device by its serial number. It then sends back the IP address of the respective
vBond orchestrator.
The vEdge connects to vBond and authenticates with a chassis number and certificate serial number. If successful, the
vBond sends back the vSmart and vManage IP addresses.
The WAN edge router establishes a permanent control connection to vManage, and the controller pushes the necessary
configuration to the device.
The device establishes control plane connections via each available WAN transport and forms an OMP peering with the
vSmart controller. In the end, it joins the overlay fabric and starts forwarding data traffic.

Key Takeaways
The key takeaway of this chapter is that implementing the control and management plane into centralized software controllers
has many significant advantages over the traditional decentralized architecture:

The centralized controllers do not need to lie in the data path and can be anywhere on the transport side.
The controllers are software images that run into virtual machines (VMs) or containers. They can be deployed on-prem
using standard compute and storage hardware or in the public cloud.
Scale challenges associated with distributed routing protocols such as OSPF, IS-IS, and full-mesh BGP on the transport
side of the network are eliminated because the centralized control-plane controller handles all routing computations.
Network segmentation could be achieved without the need for complex signaling protocols such as MP-BGP, VRFs,
MPLS, etc.
The centralized management plane allows network teams to manage the WAN as-a-system and not as a collection of
individual network devices.
For me, the most significant advantage is that the wide-area network can now scale horizontally - adding more remote
sites and vEdge routers do not affect the rest of the network because the centralized controllers handle all control-plane
calculations. We have seen that data centers moved to clos topologies that allow for horizontal scaling of the DC fabric.
The SD-WAN allows for horizontal scaling of the WAN.
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Key features of Cisco SD-WAN

Key features of Cisco SD-WAN

This section will explore the different Cisco SD-WAN features and capabilities that make the solution different from the
traditional WAN. The section will cover the following major topics:

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds
Direct Internet Access (DIA)
RESTful APIs
Cloud OnRamp
Embedded Security

Each set of capabilities makes the Cisco SD-WAN solution position itself as a leading WAN networking provider in the cloud
age. Let's go briefly over each major topic.

Application Quality of Experience (AppQoE)

Providing excellent application Quality of Experience (AppQoE) to users anywhere and anytime.

As organizations continue to adopt a cloud-first IT strategy, providing Application Quality of Experience (AppQoE) to users has
become significantly harder and different. When an application is hosted in the organization's data center, it is far easier to
control all necessary network metrics that ensure users have an excellent experience working with a business-critical app.

However, with business-critical apps now hosted in a public cloud or consumed as a service via Internet, network teams face
an enormous challenge ensuring the AppQoE anywhere anytime.

Cisco SD-WAN has been designed with public cloud and SaaS in mind. It provides a complete set of features and capabilities
geared toward the Quality of Experience of critical business applications. You will learn more about this in the first lesson of
this section.

Interconnecting Multiple Clouds

Connecting branches and remote workers to multi-cloud apps seamlessly with unified management, visibility, and security.

Most large organizations have business-critical applications hosted in multiple public clouds nowadays. This multi-cloud
environment imposes many challenges to the organization's network and security teams such as less control over the
Application Quality of Experience and expanded threat attack surface. Cisco SD-WAN helps organizations address these
challenges.

Cisco SD-WAN's cloud-native WAN overlay fabric ensures that organizations can embrace the power of multi-cloud and
integrated security, with minimal manual configurations by network and security teams. The solution has a set of capabilities
that improve network performance, reduce operational costs, and protect business-critical data.

Direct Internet Access (DIA)

Harnessing the benefits of SaaS and Internet-native apps.

Nowadays, most customers, contractors, and employees at remote branches access Internet-based applications. The
traditional WAN architecture backhauls the Internet traffic to a data center or regional hub for security inspection resulting in
higher latency and cost.

Cisco SD-WAN DIA is a feature set that can route certain traffic from the branch directly to the Internet, thereby bypassing the
organization's data center which lowers the latency and improves the performance of the applications.

RESTful APIs
Embracing custom automation.
Developers can integrate the Cisco SD-WAN solution into their custom workflows by utilizing the restful APIs.

Cloud OnRamp
Utilizing the benefits of the public cloud.

Cisco SD-WAN Cloud OnRamp is a set of capabilities that optimize the wide-area network for better visibility, predictability, and
performance in the public cloud.

Cloud OnRamp for SaaS (software as a service) uses real-time telemetry data to steer users' traffic onto the best available
network path to Internet native applications like Microsoft 365 and Salesforce.

Cloud OnRamp for IaaS (infrastructure as a service) simplifies the deployment workflows for public clouds like AWS, Azure,
and GCP. It allows the use of application-aware routing policies that can choose the optimal network path to workloads running
in the public cloud.

Cloud OnRamp Interconnect automates the connectivity to leading Software-Defined Cloud Interconnect (SDCI) providers such
as Megaport and Equinix.

Embedded Security
Laying the foundation for cloud-delivered unified security and SASE transformation.

Nowadays, security is paramount for all organizations. As users and applications become more distributed, network and
security teams face enormous challenges to deliver reliable connectivity while maintaining top-notch security for all users and
apps.

The Cisco SD-WAN solution provides a full stack of security capabilities for both public cloud and on-prem applications. It is
designed to combine organizations' existing security setups with their SASE strategy by providing security policies that can be
deployed anytime, anywhere.

« Previous Lesson
Cisco SD-WAN Main Principles
Next Lesson
Application Quality of Experience (AppQoE) »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Application Quality of Experience (AppQoE)

Application Quality of Experience (AppQoE)

Delivering an optimal application experience has never been an easy task from a network perspective. Traditional networks
have been designed to move packets without caring too much about the applications.

As a network engineer myself, I have managed many Traditional WAN environments and know how little visibility and control
over the app flows there is . The task has got a lot harder with the introduction of the Internet-native and Cloud-native apps. In
traditional WAN deployments that are operated in a box-to-box fashion, typically there is a 3, 5, or 8 queues QoS policy applied
on the egress transport interfaces and that's about it. Is it enough though? The answer is obviously not.

The Business Need

Let's imagine that you are a network engineer in a big enterprise that has a traditional WAN architecture. One day the company
starts streaming a real-time event from a remote branch to the data center and out to the Internet. From the company's
perspective, this video stream is a business-critical application and must work as optimally as possible. Just think for a few
seconds about each of these real-world problems.

How do you make sure that if the quality of a circuit drops and packet loss appears, the stream is immediately sent over
another circuit?
What if there is congestion on one of the WAN links. How do you make sure that low-priority traffic is automatically
moved to lower bandwidth links?
What if the stream viewers are mostly on the Internet. Maybe it is a better solution to just steer the traffic directly to the
Internet from the remote branch. How do you do that with traditional WAN?

These are just a few examples of hard problems that cannot be easily solved at scale using the legacy box-to-box operational
model. In today's enterprises where everything is getting digital, there are at least several business-critical applications.
Ensuring optimal experience for these applications is a multidimensional problem that cannot be solved with a single network
tool such as QoS.

Quality of
Service
Remote (QoS) Data
Site Center

Application Application Circuits

Visibility Experience Quality

Clients Servers

Network
SLA

Figure 1. Cisco SD-WAN Tools for improving applications experience

Cisco SD-WAN solution has a set of capabilities that can improve the overall Application Quality of Experience (AppQoE) of
business-critical applications. The package of tools includes some well-known network protocols working in conjunction with
some innovative technologies to create the following collection of AppQoE features:

Bidirectional Forwarding Detection (BFD)

Quality of Service (QoS)
Forward Error Correction (FEC)
 Packet Duplication
Fragmentation Avoidance
Software-Defined Application Visibility and Control (SD-AVC)
Application-aware routing (AAR)
TCP Flow Optimization
Cloud onRamp for SaaS (We will have another lesson dedicated to this feature, so it is just briefly mentioned here)

You may have dealt with some of these features before. In this lesson, we are going to try to briefly go through most of them.

Quality of Service
Cisco SD-WAN solution creates a transport-independent overlay fabric, leveraging tunneling techniques such as GRE and IPsec
to encapsulate and encrypt traffic before it is sent over all available circuits on the WAN edge routers. By default, this traffic
encapsulation "hides" the original packet inside a new one, and therefore the end-to-end QoS marking is lost. Cisco WAN edge
routers have the ability to copy the DSCP value of the original IP packet into the outer IP header. There is also the ability to
rewire the DSCP value to match a specific class of service of the service provider's circuit. This gives the ability to map specific
applications into the correct QoS classes on the SP side.

WAN Edge Router

Service Provider
QoS classes

SP4
Copy original DSCP marking
SP3
into the outer DSCP
Copy SP2

SP1
DSCP
DSCP

DSCP

IP Packet IP Packet
Ingress
Original Packet
Egress MPLS
Interface Encapsulated Packet Interface

Figure 2. Mapping app DSCPs to service provider classes

Avoiding packet fragmentation

As we all know, the IP protocol was designed for use on a variety of links. In the wide-area networks especially we can see that
this is true. We have edge routers connected to the WAN via many different access technologies such as SDH, DSL, Ethernet,
LTE, satellite links, etc. Each one of these links may enforce a different maximum transmission unit (MTU) value. Overlay
features such as tunneling and IPsec can further lower the MTU. The IP protocol accommodates these differences by allowing
devices to break larger packets into a number of pieces that can be reassembled later on. This process is called IP packet
fragmentation and leads to inefficiencies in the packet flow by adding unnecessary latency and jitter. MTU issues in traditional
WAN are well-known to network engineers.
 Remote Site Regional Hub
SD-WAN
tunnel

Transport
1
SD-WAN fabric detects the max path MTU
2
Apps send large packets

3
MTU exceeded, fragmentation required

4
Apps reduce packet size

5
Applications traffic send with correct MTU
No fragmentation required
Clients Servers

Figure 3. Cisco SD-WAN Path MTU Discovery Process

Applications have the option to explicitly prohibit fragmentation by setting a DF (do-not-fragment) flag in the IP header. But this
relies on the success of another process called path MTU discovery (PMTU) that is used to discover the smallest end-to-end
MTU value along the traffic path. If this process fails and the DF-bit is set, the application flow would not be able to traverse the
network and reach its destination. Cisco SD-WAN proactively discovers the path MTU across the overlay fabric and participate
in the hosts' PMTU process by notifying them of the available MTU as shown in Figure 3.

Circuits Quality
One of the main advantages of SD-WAN is that it can use any available transport at any location in an active-active fashion.
This typically means that all Internet links are utilized for application traffic. However, we all know that Internet circuits do not
have guaranteed quality, and packet loss may occur at any given time. Cisco SD-WAN provides the following features that
protect business-critical apps from packet loss and allows them to work reliably over the Internet.

Forward Error Correction (FEC)

The Forwarding Error Correction (FEC) feature allows critical apps to work well over unreliable WAN links usually Internet
circuits. The mechanism behind it is borrowed from RAID arrays logic. For example, in a RAID4 array, if one disk fails it can be
replaced with a new one and the information can be reconstructed based on the metadata stored in the parity disk. The FEC
follows the same logic, for each group of four packets, one "parity packet" is inserted. At the receiver end, if one of the four
packets is lost, it can be reconstructed based on the parity metadata. It is basically a trade-off between CPU cycles and circuit
reliability . The process is visualized in figure 4.
 For every block of 4 packets, One lost packet out of the
one parity packet is inserted four can be reconstructed

... 2 1 P 4 3 2 1 4 3 P 1

P 4 3 2 1
IPsec tunnel

Overlay fabric
vEdge vEdge
Sender Receiver
Site Site
Figure 4. Cisco SD-WAN Forward Error Correction (FEC)

In summary, the FEC capability protects applications from incurring packet loss on the transient network path . The feature has
the following characteristics:

Per tunnel - It is enabled on a per tunnel basis. This gives the flexibility to be enabled only on unreliable WAN links.
Dynamically invoked - FEC can be turned on permanently or it can be dynamically invoked if the SD-WAN fabric detects a
certain amount of packet loss.
Application traffic only - The feature can only be used for application traffic and not for control plane flows such as BFD.
Only one packet out of four can be reconstructed - It cannot remedy high packet loss.

Packet Duplication
Packet duplication is another SD-WAN capability that is used to increase application reliability. When turned on, the sending
WAN edge router can transmit the same traffic flow across multiple WAN links ultimately sending at least two copies of each
packet. At the receiving side, the vEdge device can compensate for lost packets by using these multiple copies of the same
flow and discard the unnecessary duplicates.

Packets are sent over both overlay Duplicated packets are dropped
tunnels at the sending vEdge at the receiving vEdge

4 3 2 1 4 3 2 1

4 3 2 1
IPsec tunnel 1
IPsec tunnel 2

vEdge 4 3 2 1 vEdge
Sender Receiver
Site Site
Figure 5. Cisco SD-WAN Packet Duplication

In summary, the Packet Duplication capability protects against packet loss for critical applications such as Voice at the
expense of increased bandwidth consumption. The feature has the following characteristics:

Protocol agnostic - It works for any transport protocol TCP or UDP.

Works only over multiple tunnels.
 Duplicates are discarded on the receiver.

Software-Defined Application Visibility and Control (SD-AVC)

Software-Defined Application Visibility and Control (SD-AVC) is a service that uses the capabilities of Cisco WAN Edge devices
to identify, aggregate, and communicate application data in order to make decisions like prioritizing app traffic using QoS,
group applications based on business relevance, or choose different network paths based on real-time SLA statistics.

Cisco SD-WAN devices have a Deep Packet Inspection engine integrated that can go up to Layer 7 of the OSI model and
recognize thousands of applications. This is absolutely necessary in order to be able to apply policies against a particular app
or service.

Figure 6. vEdge Deep Packet Inspection

Some engineers may argue that we have always had DPI engines and app recognition features like Cisco NBAR. However, the
key point here is that updating large volumes of new application signatures across the device fleet is not feasible at all using
the legacy box-to-box configuration model. You have to operate the network As-a-System to be able to do that at scale, and that
is what SD-WAN allows us to do.

Application-aware routing (AAR)

Application-aware routing is a feature that dynamically chooses the optimal path for a business-critical application based on a
pre-defined SLA policy. These policies can be defined in two major ways:

1. A specific path is configured to be taken while the path meets the SLA. For example, an MPLS circuit is configured as
primary for VoIP traffic.
2. Any path that is compliant with the SLA can be used. For example, if the Internet circuit meets the latency, jitter, and
packet loss requirements, it can be used for Voice traffic as well.
Let's look at the example shown in figure 7. Application X has a pre-defined SLA policy - latency <= 200ms, packet loss below
3%, and jitter below 15ms. At the moment only paths 2 and 3 are meeting this SLA though. Therefore, only these paths can be
used for this app.

App X must have:

vBond vSmart vManage Latency <= 200ms
Packet Loss <= 3%
Jitter <= 15ms

Control Plane Real-time

path measurements
Path 1: Jitter 25ms
Path 2: Jitter 15ms
Path 3: Jitter 12ms
h 1
t
Pa Broadband

Path 2

4G/5G
Branch Regional Hub
Pa
th
3
MPLS
Figure 7. Cisco SD-WAN Application-aware routing

TCP Optimization
The Cisco SD-WAN TCP Optimization feature terminates TCP connections locally at the WAN edge routers and uses TCP
Selective Acknowledgment (SACK) in order to better control the TCP-Window-Size and maximize the throughput through the
WAN links. Every network engineer has seen a bandwidth consumption graph of a TCP flow. It typically has the following
pattern - steep increase, sharp fall down with 50%, steep increase again and 50% fall down again, and so on. The goal of this
optimization tool is to normalize this graph by dynamically controlling the Window Size. However, this must be used with
caution because it breaks some fundamental network principles such as the end-to-end transport layer connectivity.

TCP Optimized TCP TCP

connection connection connection

IPsec tunnel

Overlay fabric
vEdge vEdge
Users Servers
 Figure 8. Cisco SD-WAN TCP Optimization

Summary
The Application Quality of Experience (AppQoE) is a multidimensional problem that cannot be solved by a single tool. The
Cisco SD-WAN solution has introduced a set of features and capabilities that can improve the overall application experience
and optimize the network reliability for business-critical apps. Let me try to make a short summary of all AppQoE tools in the
following table.

SD-WAN Feature Description

Bidirectional Forwarding BFD is a well-known network protocol used to detect faults between two WAN edge devices connected by an
Detection (BFD) IPsec tunnel and to measure the tunnel characteristics.

Quality of Service (QoS) QoS is a well-known network tool that is used for the classification and marking of application traffic.

Forward Error Correction The FEC capability protects applications from incurring packet loss when traversing unreliable WAN links. It
(FEC) works by inserting one parity packet in every group of 4 and then using the metadata in this parity packet to
reconstruct any lost one.

Packet Duplication The Packet Duplication capability protects against packet loss for critical applications such as Voice at the
expense of increased bandwidth consumption by sending two copies of each packet via two different WAN
links.

Fragmentation Avoidance The SD-WAN overlay fabric detects the MTU value on all tunnels and helps end hosts successfully identify what
MTU value to use.

Software-Defined SD-AVC is a service that uses the DPI engine of Cisco WAN Edge devices to identify, aggregate,
Application Visibility and and communicate application data in order to make decisions like prioritizing app traffic using QoS, group
Control (SD-AVC) applications based on business relevance, or choose different network paths based on real-time SLA statistics.

Application-aware routing AAR dynamically chooses the optimal path for a business-critical application based on a pre-defined SLA policy
(AAR)

TCP Flow Optimization This a feature that terminates TCP sessions at the local WAN edge devices and aggregates them into one
optimized TCP session. The goal is to better utilize the available WAN bandwidth by controlling the TCP
windows size.

Table 1. Cisco SD-WAN features that improve the Application Quality of Experience (AppQoE)

« Previous Lesson
Key features of Cisco SD-WAN
Next Lesson
Interconnecting Multiple Clouds »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Interconnecting Multiple Clouds

Interconnecting Multiple Clouds

I think every network engineer has already seen that applications are moved to the cloud on a massive scale in recent years.
Some apps are even developed and deployed directly in the cloud. These are known as Cloud-native applications or "born in
the cloud". This industry shift has created a lot of new challenges to network architectures such as the following:

How do you ensure that apps are taking the most optimal path to the Cloud? Since most networks have multiple Internet
circuits and eventually a direct connection to the particular cloud environment, the best-path selection on a per-
application basis is a complex task.
How do you provide a flexible and secure connection to the Cloud from every remote site? Since most traditional WAN
architectures do not allow direct connections from branches to the Cloud over the Internet, the traffic is routed from
remote sites to a regional hub/datacenter and then out to the Internet. This adds additional latency and creates
bandwidth bottlenecks.

Yes, migrating apps to the public cloud presents new challenges to network engineers. However, there is another, more rare
unicorn - interconnecting multiple clouds, that creates an even harder challenge.

The Business Need

Inevitably, when multiple business-critical applications are migrated to the same public cloud, some fundamental questions
start to arise:

What if the cloud fails? - Yes, even public clouds experience large-scale outages. Many enterprises have decided to
create instances of their most critical apps in an alternative public cloud for redundancy and disaster recovery.
Vendor lock-in - Overtime, using only one provider makes the company heavily dependent on that vendor. Also as a
consumer, an enterprise may want to have the freedom to demand custom pricing and have legroom for negotiations.
This inevitably leads to a multi-cloud model.
Different geographies - Not every public cloud provider is available in any geographical area in the world. For example, at
the time of writing this article, AWS does not have an infrastructure region in Switzerland. This forces some enterprises
into the multi-cloud model.
 vmware
aws

Salesforce Google Cloud

Software Infrastructure
Office 365 as a Service as a Service Azure
SaaS IaaS

Reliable and Secure

connection

Branch / Campus / Data Center

Figure 1. Enterprise Cloud Services

Some companies end up with a multi-cloud operation just naturally. This typically happens when some departments move
workloads into one cloud provider and other departments migrate other apps to a different provider.

Cisco SD-WAN as Multi-Cloud Interconnect

Cisco SD-WAN provides the ability to extend the company's WAN to the public cloud, ultimately connecting any WAN location
to any cloud platform in a secure and automated fashion. It ensures the connectivity requirements by using enhanced routing
techniques such as application-aware routing to the cloud IaaS applications and adjusting the IPsec routes in real-time based
on the pre-defined quality metrics (packet loss, latency, and jitter).
 vmware Azure aws
Google Cloud Salesforce

Cisco
SD-WAN

Campus Data Center Remote Site Private Cloud

Figure 2. Cisco SD-WAN as Multi-Cloud Interconnect

Automated connectivity provisioning - Cisco SD-WAN extends the overlay fabric to public clouds. This provides the ability
to choose the most optimal entry point for all data centers and hub locations in real-time.
Application and network telemetry in and out of all clouds for reporting - Cisco SD-WAN provides the ability to unify
the network management by creating a single pane infrastructure that has visibility across the entire enterprise network
for more sophisticated management of network resources and services. This single-pane view can provide a unified
centralized management of all resources including physical, virtual, and cloud.
Dynamic routing, multipathing, and deterministic failover behavior using OMP - Because the enterprise overlay fabric
basically includes the virtual WAN edge routers hosted in the clouds, all network settings could be managed in a
centralized fashion using the SD-WAN control plane. This gives the ability to create custom network topologies based on
the company's needs.

« Previous Lesson
Application Quality of Experience (AppQoE)
Next Lesson
Direct Internet Access (DIA) »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Direct Internet Access (DIA)

Direct Internet Access (DIA)

The Business Need
In traditional wide-area designs, Internet traffic from remote sites is sent first to a centralized data center or hub site. Then the
traffic is pushed through the company's security stack and only then it is routed out to the Internet. The returning traffic also
traverses the security stack before it is sent back to the remote site. This is typically done because the cost of installing and
operating a security stack in every remote location is very high.

Microsoft Teams

Salesforce
Software-as-a-Service (SaaS)

Security Stack
DNS/Web layer
security

IPS/IDS

SD-WAN URL Filtering

Overlay
Firewalls
INTERNET

MPLS
vEdges vEdges
Branch
Datacenter

VPN 10 VPN 20
Employees Infra

Figure 1. Backhauling Internet traffic through a Datacenter

However, the fast adoption of Internet-based applications and teleworking created the following issues with this WAN design:

Scale - With the ever-increasing Internet demands at each remote branch and the use of SaaS apps such as Office 365
and Salesforce, backhauling the Internet traffic from every branch to the data center creates a bottleneck at the DC's
Internet circuits. In addition, the centralized security stack and the network devices at the DC must scale vertically with
the number of branches.
Apps Performance - By re-routing traffic from the branch to DC to the Internet and back, applications incur increased
latency. Depending on the underlying geography, this can result in significant performance degradation for some
business-critical apps.
Cost - Having said the above, it is obvious that the centralized hub site must scale vertically when the number of remote
sites increases. This implies higher hardware and WAN costs at the data center location.

Cisco SD-WAN allows for a better more scalable approach to Internet usage at remote sites. The feature is called Direct
Internet Access and as the name implies, it allows particular users and applications to access SaaS/IaaS services directly
through the local Internet circuits.
Cisco SD-WAN Direct Internet Access (DIA)
Cisco SD-WAN Direct Internet Access is a solution that improves the user experience for SaaS applications at remote sites by
eliminating the performance degradations related to backhauling Internet traffic to central data centers. DIA allows control of
Internet access on a per VPN basis.

Microsoft Teams

Salesforce
Software-as-a-Service (SaaS)
Security Stack
DNS/Web layer
security

IPS/IDS

URL Filtering
SD-WAN
Firewalls Overlay
INTERNET

MPLS
vEdges vEdges
Branch
Datacenter

VPN 10 VPN 20
Employees Infra

Figure 2. Cisco SD-WAN Direct Internet Access

Security
Cisco SD-WAN allows pushing the security stack directly on the WAN edge devices onsite. This reduces the need for security
appliances at every branch, by providing inbuilt security features which include DNS security, Application-aware firewall, URL
filtering, IPS/IDS, and Advanced Malware Protection (AMP).

In addition, instead of enabling the security stack at the WAN edge routers, the DIA feature allows for routing the traffic through
a cloud security provider. In this case, the traffic from a particular remote site is routed to the cloud security provider through
point-to-point IPsec tunnels. The cloud security provider then pushes the traffic through the predefined security policies and
route it out to the Internet.
 Cloud
Security Provider

Cisco SD-WAN
DIA IPsec
tunnels

Internet Office 365

Branch

VPN 10
Employees

Figure 3. DIA traffic through a cloud service provider

Guest Access
Cisco SD-WAN provides an easy and secure way to create an isolated Guests segment that is isolated from the enterprise
network and has its own security policies. Typical DIA traffic policies include:

Restricting bandwidth usage of guests users

Restricting access to certain Internet resources
Restricting access to internal enterprise resources
Protecting the network from malicious content

« Previous Lesson
Interconnecting Multiple Clouds
Next Lesson
RESTful APIs »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / RESTful APIs

RESTful APIs
Before we start with this lesson, I'd like to quickly explain some terms and concepts upfront.

What is an API? (Application Programming Interface) - API is just an interface that allows software to interact with
another software;
What is SDK? (Software Development Kit) - SDK is a set of tools used to develop software for a specific platform;
What is the difference between API and SDK? - API is purposely built to allow specific communication between
applications. SDKs allow for the creation of applications including APIs.

And the non-technical explanation - the SDK represents an entire office: all engineers, salespersons, furniture, and all office
gear. An API represents just the Internet lines that allow communication in and out of the office.

Ok, now back to Cisco SD-WAN...

The Business Need

Legacy network devices were designed to be managed and operated by humans. They were not made to communicate natively
to other software. That's why traditional networks are not extensible and automated in a native way. Nowadays, all other IT
verticals are software-driven, so businesses are pushing the networks in that direction as well.

Cisco SD-WAN has been designed with automation and extensibility in mind. Cisco vManage provides northbound RESTful
APIs that allow customers to build their own unique business logic on top of the SD-WAN solution. For example, enterprises
can integrate their existing OSS (Operational Support System) and BSS (Billing Support System) tools and consume telemetry
data, automate incident tickets creation and lifecycle, and automate the deployment of new services.
 User defined 3rd party Custom 3rd party
automation controllers NMS tools OSS, BSS

RESTful APIs
vManage
NMS

vSmart
controllers

4G/LTE MPLS INET

Data Plane

Figure 1. Cisco SD-WAN REST APIs

The northbound APIs open a new world of possibilities to network engineers as well. Many trivial operational tasks that
consume lots of time and effort in a large-scale environment can be easily automated. For example, configuration audits,
network/security audits, inventory reports, automated backup/restore, 3rd-party tools integration, and so on.

Let's take a look at the most typical use-cases.

User-defined Automation
vManage REST APIs provide endless possibilities for the automation of user-defined tasks. You can basically write software
that can interact with the Cisco SD-WAN solution without human supervision. For the most simple cases, this software would
be a python script. However, for more advanced scenarios, sophisticated Ansible playbooks can be leveraged.

3rd-Party Controllers
One of the beauties of the vManage RESTful APIs is that they can be leveraged by other domain controllers, such as Cisco DNA
Center or Cisco ACI, in order to deliver a unified single management plane across multiple technology verticals. This allows for
the emergence of a single-pane-of-glass tool for service orchestration, configuration, administration,
and troubleshooting. Such integration might finally allow for a full-scale intent-based network that can enforce business
intend across the entire infrastructure from user to applications to cloud.

Technically, the REST APIs allow for the integration of non-Cisco SD-WAN controllers as well but at the current stage,
interoperability between SD-WAN vendors is not present.

3rd-Party OSS, BSS, and SP tools

The vManage REST APIs allow Service Providers and MSPs to integrate their existing operational and billing systems
(OSS/BSS). Most typical examples of such integrations are:
 Service and data usage statistics are collected using the RESTful APIs and then feed into the service provider's billing
system;
Using the RESTful API, service providers can design custom self-servicing portals that allow users to subscribe to new
services or change the operational status of existing ones. For example, purchasing and deploying new SD-WAN security
features, network segmentation and slicing, and so on;
Automated deployment of new customers;
CI/CD pipeline integrations, automated change scheduling, and rollbacks.
SIEM (Security Information and Event Management) integrations, that can perform automated remediative actions
against security alarms and incidents.

vManage's Embedded API Library

Cisco includes very sophisticated API documentation as part of the vManage software using the following URL:

https://[vManage-IP-address]:8443/apidocs

A screenshot of the provided API library is shown in figure 2.

Figure 2. Cisco SD-WAN API Docs

The documentation is divided into a few major categories of API calls:

Certificate Management
Configuration
Device Inventory
 Real-Time Monitoring
Troubleshooting Tools

When you select a particular API class. The documentation shows its response class, required parameters, and the returned
status codes.

Table 1 below shows some typical examples of vManage API calls:

Requested Info API Call

Environment health status of device's hardware components dataservice/device/hardware/environment?deviceId=system-ip-address

A list of all cisco sd-wan devices dataservice/device

Status of the transport interfaces of a device dataservice/device/interface?deviceId=system-ip-address&port-type=transport

Status of DTLS control connection dataservice/device/control/connections?deviceId=system-ip-address

Interface statistics and packet drops dataservice/device/interface?deviceId=system-ip-address

A list of OMP peers dataservice/device/omp/peers?deviceId=system-ip-address

A list of BGP peers dataservice/device/bgp/neighbors?deviceId=system-ip-address

Examples of vManage API calls

Cisco SD-WAN Python SDK

Cisco has provided a full-fledged Python-based SDK for Cisco vManage that has tools, libraries, and documentation to simplify
the interactions with the REST API. It is intended for engineers interested in automating the administration and operation of the
SD-WAN solution using Python without any GUI interaction.

Phyton Phyton RESTful

SDK Script JSON APIs

API vManage

vSmart vSmart
controller controller

Engineer
SD-WAN Fabric
Figure 3. vManage Python SDK

There are many great things that you can do when using Cisco vManage programmatically. A few typical use-cases are:

Software integrations with other platforms;

Programatically keeping track of device status and acting upon change;
Management of policies and device templates in an automated fashion;
 Automated backup and restore;
CI/CD integrations;
Automatic querying and aggregating of device and traffic statistics.

« Previous Lesson
Direct Internet Access (DIA)
Next Lesson
Cisco SD-WAN Control Plane »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

OMP Source Preference

Understanding OMP Routing

OMP TLOC-Action

Chapter Key Takeaways

QUIZ - Control Plane Fundamentals

4. Cisco SD-WAN Data Plane

What is a TLOC?

The Overlay Fabric

TLOC Color and Carrier

Tunnel Groups

TLOCs and NAT

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Cisco SD-WAN Control Plane

Cisco SD-WAN Control Plane

This section will continue our deep-dive into the different Cisco SD-WAN operations by exploring the control plane. The chapter
is broken into five distinct sections:

Underlay vs. Overlay routing

Control Connections
vSmart
Overlay Management Protocol
OMP Best-path Algorithm

The first lesson will examine the difference between the underlay and the overlay routing. Then we will discuss the process of
establishing control connections to each SD-WAN controller. The third section of this chapter will quickly introduce the vSmart
controller. Then, we will explore the Overlay Management Protocol (OMP) and see how vSmart uses OMP to facilitate the
control-plane function across the overlay fabric. In the last section, we will deep-dive into the OMP best-path algorithm, and
then we will go through the following practical examples:

Lab#1: OMP Best-path Algorithm - Send Path Limit

Lab#2: OMP Best-path Algorithm - Send Backup Paths
Lab#3: OMP Best-path Algorithm - vSmart Source Preference
Lab#4: Reducing control-plane traffic on LTE WAN links

« Previous Lesson
RESTful APIs
Next Lesson
Underlay vs Overlay Routing »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Underlay vs Overlay Routing

Underlay vs Overlay Routing

Why do we need the overlay?
Traditional routing protocols build per-prefix routing tables with each routing entry pointing to a next-hop IP address. This
typically means that each packet is forwarded hop-by-hop across the network according to the routing table of each individual
router along the path to the destination. This hop-by-hop routing behavior has many inefficiencies such as:

Network segmentation and network slicing are hard to achieve at a large scale:
Transporting segmentation tags hop-by-hop across the network requires complex control plane interactions
between VRFs, MPLS, and MP-BGP.
Network slicing and multi-tenancy are practically unachievable.
Scaling is hard. Horizontal scaling is even harder to achieve. Equal-cost multipathing (ECMP) over multiple types of WAN
transports and multiple different routing protocols at a large scale is practically impossible.
Service-chaining does not efficiently scale because it requires manual configuration on multiple devices.
Multicast does not natively traverse public transports such as the Internet.

IPsec tunnel (Overlay)

IP IP IP IP IP IP IP

IP IP (Underlay) IP IP
T1 T1 T2 ESP IP T2
Service Service
Side Side
vEdge-1 vEdge-2

Figure 1. Why do we need the overlay?

The Cisco SD-WAN overlay fabric solves most of these inefficiencies by changing the traditional routing concept of a next-hop
IP address with a next-hop TLOC . As vEdges build overlay tunnels between their WAN tunnel endpoints (TLOCs), they
advertise the site-local networks as reachable via their local TLOCs. Using this technique, each destination prefix in the routing
tables points to a remote tunnel endpoint (TLOC). When a packet is then forwarded according to the routing table, it is sent
through an overlay tunnel to a remote TLOC, as shown in figure 1 above. The packet is encapsulated with new outer headers
where the source and destination addresses are replaced with the tunnel endpoints. This way, the intermediate WAN networks
between both vEdge routers do not need to know anything about the original packet’s source and destination IP. Additionally,
the packet is encrypted with IPsec and labeled with a VPN tag that tells the remote router which network segment (VPN) this
traffic belongs to. Compared to the traditional VRF/MPLS/MP-BGP approach to network segmentation, this is a considerable
simplification and efficiency improvement. We will explore the overlay routing in great detail in the coming sections.

Cisco SD-WAN Underlay vs Overlay

Cisco SD-WAN architecture is divided into two very distinct parts: the underlay network and the overlay fabric.

The Underlay
The underlay network represents the hardware infrastructure - all network devices that connect to the available WAN
transports and local site networks. The router interfaces that connect the WAN transport networks are always configured in
VPN0 (the Transport VPN). The attachment points that connect to the transports are called TLOCs (colored in figure 2). TLOCs
play a very important role in abstracting the underlay network away from the overlay fabric and the applications. The main and
only function of the underlay network is to provide IP reachability between TLOCs .
 Overlay Network

IP
se c
IP

c
se
T1 T3
VPN0 0.0.0.0/0 INET 0.0.0.0/0 VPN0
(Transport) (Transport)
T2
0.0.0.0/0
MPLS 0.0.0.0/0
T4
Underlay Network

VPN1 VPN2 VPN1 VPN2

Local Networks Local Networks

Figure 2. Cisco SD-WAN Underlay vs Overlay Routing

A vEdge router must have at least one interface configured in the transport VPN 0 to establish control plane tunnels to the SD-
WAN controllers and join the overlay fabric. Each interface that connects to the WAN must have an IP address, color, and
encapsulation type configured. These parameters are then advertised to the controllers via OMP as part of the TLOC route
advertisements.

Most commonly, a default route is defined via each WAN interface, as illustrated in figure 2. Therefore, if a vEdge router has five
WAN connections, we can configure five default routes 0.0.0.0/0 via each WAN link. This is a common point of confusion for
network engineers. How does a router choose which default route to use at any time?

In Cisco SD-WAN, when multiple default routes exist, the one that is chosen depends on the local TLOC that will be used. When
the overlay routing decides to use a particular IPsec tunnel, the underlay routing uses the default route with a next-hop IP
address in the same subnet as the tunnel source IP address. If we look at figure 3, for example, when vEgde1 decides to
forward traffic over the orange IPsec tunnel, it uses the default route that points to a next-hop IP address in the same subnet as
the source interface IP (ge0/0).

Remote
vEdges

INET MPLS
14.3.2.1/30 10.1.1.1/30

0.0.0.0/0 0.0.0.0/0

14.3.2.2/30 Ge0/0 Ge0/1 10.1.1.2/30

VPN 0

vEdge-1
System IP: 1.1.1.1
Site-ID: 10
 Figure 3. Multiple Default Routes in VPN 0

Additionally, all Cisco SD-WAN devices have a gateway tracking feature that is enabled by default and can’t be stopped or
modified. Each device probes using ARP the next-hop IP of each underlay static route every 10 seconds. If the device receives
an ARP response, it maintains the static route in the VPN0’s routing table. If the device misses ten consecutive ARP responses
for a next-hop IP, the device removes the static route that points to this IP from the routing table. The device periodically probes
the next-hop address, and as soon as it receives an ARP response again, the static route is installed back in the routing table.

The Overlay
Cisco's SD-WAN Overlay network is made of IPsec tunnels that traverse from site to site using the underlay network forming
the so-called SD-WAN Fabric. Each overlay tunnel is formed between two TLOCs . The routing within the overlay is governed by
the Overlay Management Protocol (OMP), a control-plane protocol very similar to BGP. The OMP protocol runs over secure
DTLS or TLS connections between the WAN edge routers and the vSmart controllers. The process is very similar to the BGP
operation, the vSmart controller acts as a BGP route reflector (RR), it receives, modifies, and re-advertises routes from the
vEdge routers, but never participate in the data-plane (in the packet forwarding).

Network Segmentation
Abstracting the packet forwarding away from the network and application logic opens a world of possibilities. This allows for
the use of VPNs that divide the overlay network into different network segments. Essentially, segmentation is done at the WAN
edge routers, and the segmentation information is carried as a VPN label in the packets. However, the underlay network
(Transport VPN0), that connects the WAN edge routers to the WAN transport, is completely unaware of the network segments
(VPNs). Only the overlay knows about the VPNs; the underlay network follows the standard IP routing.

VPN 10 Hub-and-Spoke

VPN 20 Custom

VPN 30 Custom

VPN 40 Custom

vEdge vEdge vEdge

vEdge vEdge

INET MPLS
VPN 0 - Transport
Figure 4. Different Overlay Topologies per VPN

Key Takeaways
Let's try to summarize the difference between Cisco SD-WAN's Underlay vs Overlay in one table:
 Cisco SD-WAN Underlay Overlay

Description The underlay network represents the hardware infrastructure - all The overlay network represents the IPsec/GRE
network devices that connect to the available WAN transports and local tunnels that are built between the underlay
site networks. TLOCs.

Function To provide IP reachability between TLOCs. To provide IP reachability between sites and
offer segmentation, security, and flexibility.

Packet Packets traverse over the WAN following the standard IP routing Packets are forwarded between overlay nodes
Forwarding principles. Next-hop is an IP address. over IPsec tunnels. Next-hop is a TLOC of a
remote peer.

Packet Control Hardware oriented. Software oriented.

Packet Delivery Responsible for delivery of packets. Abstracted away from the delivery of packets.

Control-Plane Standard control-plane protocols such as OSPF, IS-IS, BGP, and static Cisco's Overlay Management Protocol (OMP)
Protocol routing.

Multipathing Achieving Equal-cost Multi-pathing (ECMP) over multiple different types Support for scalable multi-path forwarding
(ECMP) of WAN transports is associated with overhead and complexity. Very over multiple virtual IPsec/GRE tunnels.
hard to achieve at scale.

Deployment Deployment times are long. Design changes typically require hardware Ability to rapidly deploy new functions at scale.
time changes and manual activities. Design changes in the overlay are done in a
centralized fashion.

Multitenancy Requires a complex control plane (MPLS, MP-BGP) to propagate the Natively supports Multitanency and has the
VRFs across the network. Large-scale implementations are associated ability to manage overlapping IP addresses
with configuration overhead and complexity. between multiple tenants.

Scalability Less scalable due to legacy technology limitations. Designed to provide horizontal scalability,
security, and flexibility.

« Previous Lesson
Cisco SD-WAN Control Plane
Next Lesson
Control Connections »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Control Connections

Control Connections
We have seen multiple times by now that the Cisco SD-WAN architecture decouples the control and management-plane
functions away from WAN edge routers. The solution implements these functions into centralized software-based controllers
that oversee the entire network environment. Therefore, from the perspective of a vEdge router, the control and management
plane functions are remote services that are communicated to the device. That is why each vEdge router must establish and
maintain secure control connections to each Cisco SD-WAN controller, as is visualized in figure 1.

vSmart
vBond vManage

Control Plane
Orchestra on
Plane Mgmt Plane

Regional
Data Plane WAN Data Plane
Campus
Hub vEdge vEdge

Transient
Data Plane Connection
Permanent
Data vEdge Connections
center

Figure 1. Cisco SD-WAN Control Connections Overview

The SD-WAN controllers are software entities that run as virtual machines or containers, hosted either on-prem or in a public
cloud provider. Each controller plays a different role in the solution and interacts with the vEdge routers through various
protocols. For example, vSmart uses the Cisco Overlay Management Protocol (OMP) to communicate routing, TLOC, and
service information among vEdges. vManage uses NETCONF to push configurations to devices, SNMP to collect data, and
ICMP echo/reply to detect liveliness.

To meet today’s high standards for availability and security, each SD-WAN controller interacts with the WAN edge devices in a
secure, resilient, and timely manner. However, when multiple control protocols are running in a network, ensuring that each one
is compliant with the latest security policies is a complex task that quickly gets out of hand. For example, let’s say we have a
traditional WAN environment running BGP, OSPF, EIGRP, SNMP, and PIM. How do we make sure that each protocol is secured?
Each one implements key exchanges differently, supports different encryption and authentication algorithms, and is configured
differently. That is why Cisco SD-WAN has taken a different, more scalable approach when it comes to securing the control
plane protocols. Every WAN edge router establishes one or more secured control tunnels to each SD-WAN controller. The
tunnels are secured using a standard transport security protocol (DTLS or TLS).
 vEdge Controller
DTLS/TLS control tunnel

Ge0/0 SNMP OMP NETCONF OMP eth0

Figure 2. Control plane protocols run inside a DTLS/TLS control connection

Then each control plane protocol, such as OMP, NETCONF, SNMP, etc., goes through one of the DTLS/TLS secured control
tunnels, as is visualized in figure 2. This technique ensures that the native security of these protocols (OMP, SNMP, NETCONF,
etc.) is no longer a concern to the overall security because they are communicated through secured tunnels that are encrypted
and authenticated.

DTLS vs TLS
Cisco SD-WAN supports two transport layer security protocols that vEdges use for control-plane tunnels: DTLS (Datagram
Transport Layer Security) defined in RFC 6347 and TLS (Transport Layer Security) defined in RFC 5246. Both protocols are very
similar, and from a high level, they both serve the same goal - to provide end-to-end transport security between a router and an
SD-WAN controller. The main difference is that DTLS uses UDP and TLS runs over TCP. This means that they handle packet
loss differently. TLS primarily relies on TCP for packet delivery. On the other hand, DTLS implements its own sequence
numbers, fragment offsets, and retransmissions because UDP does not guarantee reliable delivery of packets. In the end, TLS
is slightly more secure and reliable but slower. DTLS is marginally faster and more efficient but less secure. However, the
differences are so slight that they are pretty much the same in practice.

By default, Cisco SD-WAN utilizes DTLS for all control plane communications because it is faster than TLS, and latency is
essential when devices are managed remotely. However, the solution allows us to change the protocol to TLS easily.

Control Connections to vBond

We will explore the various control connections that a WAN edge router establishes, starting with the vBond controller
representing the orchestration plane.

Once we power on a vEdge router for the first time in an unconfigured state, it tries to find the IP address of vBond. The vBond
orchestrator is the controller that glues everything together - that’s why it is called the “orchestrator.” vBond validates the
identity of vEdges, tells them whether they sit behind NAT in the underlay, how many vSmart controllers oversee the domain,
and most importantly - what are the IP addresses of vManage and vSmart.
In simple words, an unconfigured vEdge router needs to contact vBond to receive the necessary information required to join the
SD-WAN domain. However, how could a brand new unconfigured router know the IP address of the vBond controller for a
particular organization? Well, there are a few different deployment options that routers can utilize to discover the orchestrator’s
IP:

ZTP/PnP - A vEdge router can automatically discover the vBond IP address over the Internet using a technique called
Cisco PnP (Plug-and-Play) for IOS-XE devices and ZTP (Zero-touch provisioning) for Viptela SD-WAN devices. At a very
high level, the router simply makes a query to devicehelper.cisco.com or ztp.viptela.com over the Internet, presents its
serial/chassis number and certificate, and asks for the vBond IP address associated with its serial number.
Bootstrap configuration - As an alternative option, a vEdge router can load a configuration file from a USB stick. The
config file includes the vBond IP address, organization name, and everything else required so that the vEdge can
successfully join the overlay fabric.
Manually - And of course, the time-proven method that always works - a network administrator can configure everything
through the CLI of the router, including the vBond IP address and the organization name.

We will go through each deployment option in detail in the SD-WAN Deployment chapter of this course. For now, let’s see what
happens after a vEdge router finds the vBond orchestrator’s IP address.

The vEdge router tries to establish a DTLS control-plane tunnel to the vBond IP address over each available transport interface.
In the example shown in figure 3, vEdge-1 will try to form a DTLS control connection to the orchestrator over the Internet and
over the MPLS cloud. It will authenticate itself over each control tunnel and find whether it sits behind a NAT device along any
of the paths. In the end, vBond will send back a list with the IP addresses and ports of all other SD-WAN controllers (vSmart
and vManage). Both DTLS control connections are then terminated because the vEdge router does not need to keep
permanent control connections to vBond once it receives all necessary information.
 vBond vEdges form one transient
Controller connection to vBond over each
local color to authenticate, get
vBond has only one ge0/0 the controllers list and detect NAT
control-interface

T1
INET T5
T2 MPLS T6
vEdge-1 vEdge-3
1.1.1.1 T3 T4 3.3.3.3

vEdge-2 public-internet
2.2.2.2
Data Plane mpls

Figure 3. Control connections to vBond

At this time, the vBond orchestrator will inform vManage and vSmart to expect a control connection request from that
authenticated WAN Edge router.

Control Connections to vManage

Once a vEdge router receives the list with the IP addresses of all SD-WAN controllers from vBond, it initiates a single,
persistent DTLS control connection to vManage over only one transport . The vEdge tries every transport network starting with
the interface having the lowest port number, and it keeps the first successfully established connection permanently.

If we look at figure 4, all vEdges have two transport connections, one to the Internet and one to the MPLS cloud. However, DTLS
tunnel to vManage is established only via the Internet. If a vEdge lost connectivity to vManage over this DTLS connection, it
would form a new one through the other transport interface connected to the MPLS cloud.

vManage vEdges form and keep only

Controller one permanent control
connection to vManage (the
vManage has only Eth0
first successfully established)
one control-interface

T1
INET T5
T2 MPLS T6
vEdge-1 vEdge-3
1.1.1.1 T3 T4 3.3.3.3
vEdge-2
public-internet
Data Plane 2.2.2.2
mpls

Figure 4. Control connections to vManage

When a vEdge router establishes a control connection to vManage, the controller provisions its configuration based on the
device template attached to that vEdge.
Control Connections to vSmart
Following the successful DTLS tunnel to vManage, the vEdge router initiates a persistent control connection to the vSmart
controller over each transport interface . You can see in the example shown in figure 5 that all vEdges establish a DTLS tunnel
to vSmart through the Internet and another one through the MPLS cloud at the same time. Technically, a single control
connection to the controller is sufficient for a vEdge to receive control plane information. Still, vEdges keep one permanent
DTLS tunnel per transport interface for control-plane resiliency.

vSmart
Controller
vEdges form and keep a
vSmart has only one Eth0 permanent control connection to
control-interface vSmart over each local color

T1
INET T5
T2 MPLS T6
vEdge-1 vEdge-3
1.1.1.1 T3 T4 3.3.3.3
vEdge-2 public-internet
2.2.2.2
Data Plane mpls

Figure 5. Control connections to vSmart

When a vEdge router establishes at least one DTLS control connection to vSmart, it starts a single OMP peering session. The
router then advertises its local networks, TLOCs, and service routes to the controller via OMP. Upon receiving the information,
the controller recalculates the centralized RIB (routing information base) based on the defined policies and redistributes the
routes to all other WAN Edge devices using OMP updates.

The following output shows an example where a vEdge router has three DTLS control connections to the same vSmart
controller over each local color.

vEdge-1# show control connections

PEER PEER
PEER PEER PEER PEER PRIV PEER PUB
TYPE PROT SYSTEM IP PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE
-------------------------------------------------------------------------------
vsmart dtls 1.1.1.30 10.1.1.30 12346 10.1.1.30 12346 mpls up
vsmart dtls 1.1.1.30 10.1.1.30 12346 10.1.1.30 12346 biz-internet up
vsmart dtls 1.1.1.30 10.1.1.30 12346 10.1.1.30 12346 lte up
vbond dtls 0.0.0.0 10.1.1.10 12346 10.1.1.10 12346 mpls up
vbond dtls 0.0.0.0 10.1.1.10 12346 10.1.1.10 12346 public-internet up
vbond dtls 0.0.0.0 10.1.1.10 12346 10.1.1.10 12346 lte up
vmanage dtls 1.1.1.20 10.1.1.20 12546 10.1.1.20 12546 biz-internet up

Although there are three DTLS connections to vSmart, the router has established only one OMP peering session with the
controller, as shown in the output below:

vEdge-1# show omp peers

R -> routes received
I -> routes installed
S -> routes sent
DOMAIN OVERLAY SITE
PEER TYPE ID ID ID STATE UPTIME R/I/S
-------------------------------------------------------------------------------
1.1.1.30 vsmart 1 1 1 up 0:05:08:57 94/56/4
Notice that the IP endpoints of the OMP peering session are the system-IP addresses of the vEdge and the controller, as
illustrated in figure 6 below.

The OMP peering is Any of the DTLS control

established between connections could be
the system IPs used for transport

system-IP system-IP
DTLS tunnel
T1
T2 DTLS tunnel eth0
T3
vEdge DTLS tunnel vSmart

Figure 6. OMP peering session

The OMP peering session works the same way as a BGP peering session between loopback addresses. Although multiple IP
paths exist between the BGP peers, the BGP session itself is only one and can use each available IP path at any given moment,
according to the routing table. Similarly, an OMP session is established between the system-IP addresses and can use each
available DTLS control connection for transport between the vEdge and the controller.

Control-connections overview
When a vEdge router establishes the control plane connections to every SD-WAN controller and receives the route information
from vSmart, it joins the overlay fabric by establishing IPsec/GRE tunnels to all remote TLOCs across all local WAN transports.

The data plane IPsec tunnels should not get confused with the DTLS control plane tunnels to the SD-WAN controllers. It is also
important to note that the concept of Transport Locators (TLOCs) does not apply to controllers. All SD-WAN controllers have a
single ordinary L3 interface that terminates DTLS/TLS connections, and this interface does not have TLOC configuration (color,
encapsulation type, tunnel-group, etc.).

vManage vSmart vBond Other

vEdges

Eth0 Eth0 ge0/0 T3

Only one permanent One transient DTLS

DTLS connection to INET MPLS connection to vBond
vManage over each transport

T1 T2
One permanent DTLS Data-plane IPsec
connection to vSmart tunnels between vEdges
over each transport vEdge-1 Shouldn’t be confused
with the DTLS/TLS
1.1.1.1
control tunnels

Figure 7. Overview of all overlay connections

Figure 7 summarizes all control plane connections that a WAN edge router establishes:

One transient DTLS control connection to the vBond orchestrator over each connected WAN transports only during the
onboarding process.
One persistent DTLS control connection to vManage over a single WAN transport.
One persistent DTLS/TLS control connections to vSmart over each connected WAN transports;
IPsec tunnels to all known remote TLOCs with different site-ids over each available WAN transports. A BFD session is
automatically started over each IPsec tunnel and can not be disabled.

To check the control plane tunnels of a vEdge, we use the command show control connections as shown below.

vEdge-1# show control connections

#some columns are omitted for clarity
PEER PEER
PEER PEER PEER PEER PRIV PEER PUB
TYPE PROT SYSTEM IP PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE
--------------------------------------------------------------------------------
vsmart dtls 1.1.1.30 10.1.1.30 12346 10.1.1.30 12346 mpls up
vsmart dtls 1.1.1.30 10.1.1.30 12346 10.1.1.30 12346 biz-internet up
vbond dtls 0.0.0.0 10.1.1.10 12346 10.1.1.10 12346 mpls up
vbond dtls 0.0.0.0 10.1.1.10 12346 10.1.1.10 12346 biz-internet up
vmanage dtls 1.1.1.20 10.1.1.20 12546 10.1.1.20 12546 biz-internet up

« Previous Lesson
Underlay vs Overlay Routing
Next Lesson
OMP Overview »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

OMP Source Preference

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / OMP Overview

OMP Overview
What is OMP?
The Cisco Overlay Management Protocol (OMP) is an all-in-one TCP-based protocol, similar to BGP, that establishes and
maintains the SD-WAN control plane. OMP runs between the vEdge routers and the vSmart controllers and between the
controllers themselves. The protocol is responsible for:

Distribution of Transport Locators (TLOCs) among network sites in the sd-wan domain.
Distribution of service-side reachability information.
Distribution of service-chaining information.
Distribution of data plane security parameters, VPN labels, and crypto keys.
Distribution of data and application-aware routing (AAR) policies.

Upon joining the overlay fabric, vEdges automatically initiate OMP peering to the vSmart controllers. A key point here is that
the two endpoints of this OMP peering are the System-IPs of the vEdge router and the controller, similar to a BGP peering
between a route-reflector and a RR client established between loopback interfaces. Therefore, the OMP peering is not strictly
bound to any of the DTLS control connections to vSmart. Similar to a BGP peering between loopbacks, the OMP peering would
use all available control connections in the same way the BGP peering would use all available IP paths between the loopbacks.

All control-plane
information goes through vSmart
vSmart via OMP controller

P OM
P M P
M O
O
O
M
P

Data
Center
vEdge vEdge

MPLS
INET
Branch
OSPF
vEdge vEdge
IPsec Tunnel

vEdges never exchange control

plane information directly in-band
through the data-plane
Figure 1. Cisco SD-WAN Control plane exchange via OMP

OMP addresses the challenges that traditional IGP protocols face when scale. Since OMP operates in the overlay, the notion of
routing peers is different from a traditional WAN environment. From a logical point of view, the overlay fabric consists of a few
vSmart controllers and many vEdge routers. The routers peer only with the vSmart controllers. vEdges don’t form any control-
plane relationship among themselves over the Cisco SD-WAN fabric. This is a significant difference from the routing peering
model in traditional IGPs. vEdge routers don’t need to form routing adjacencies between themselves and do not have to
respond to an excessive number of routing updates. All of this makes OMP more manageable, secure, and more efficient
compared to traditional routing, especially in large-scale deployments with hundreds of spokes.
OMP is the key ingredient that allows the WAN to scale horizontally. The vSmart controller makes all routing computations for
the overlay and OMP propagates this information across the fabric. Therefore, adding new vEdge routers into the SD-WAN
domain does not affect the performance of the existing WAN edge routers because individual routers do not make routing
calculations for the entire overlay fabric.

OMP Peering
OMP is enabled by default on all Cisco SD-WAN edge devices. When vEdges go through the Zero-Touch Provisioning process,
they learn about the addresses of all available vSmart controllers and automatically initiate secure connections to them. By
default, these connections are authenticated and encrypted via the Datagram Transport Layer Security (DTLS) protocol.
Depending on the number of available transports, each vEdge router will try to establish a secure control connection via every
TLOC. However, as shown in figure 2, the OMP peering uses the System-IPs, and only one peering session is established
between one WAN Edge device and one vSmart controller even if there are multiple DTLS connections to the same controller.

The OMP peering is Any of the DTLS control

established between connections could be
the system IPs used for transport

system-IP system-IP
DTLS tunnel
T1
T2 DTLS tunnel eth0
T3
vEdge DTLS tunnel vSmart

Figure 2. Cisco SD-WAN OMP Peering

You can see in the following output that the WAN edge device has two DTLS control connections initiated to the vSmart
controller with IP address 1.1.0.3. One connection through the MPLS transport and another one through the Internet.

vEdge-1# show control connections

PEER
PEER PEER PEER SITE PEER PUB
TYPE PROT SYSTEM IP ID PUBLIC IP PORT LOCAL COLOR STATE
-----------------------------------------------------------------------
vsmart dtls 1.1.0.3 1 1.1.1.70 12346 mpls up
vsmart dtls 1.1.0.3 1 1.1.1.70 12346 public-internet up
vbond dtls - 0 1.1.1.60 12346 mpls up
vbond dtls - 0 1.1.1.60 12346 public-internet up
vmanage dtls 1.1.0.1 1 1.1.1.50 12346 mpls up

However, if we check how many omp peering sessions to the controller there are, you can see that there is only one.

vEdge-1# show omp peers

R -> routes received
I -> routes installed
S -> routes sent

DOMAIN OVERLAY SITE

PEER TYPE ID ID ID STATE UPTIME R/I/S
-------------------------------------------------------------
1.1.0.3 vsmart 1 1 1 up 0:14:25:18 4/1/8

Another important thing to know is that these DTLS control plane tunnels are used by other protocols as well. For example,
besides OMP, NETCONF and SNMP will also be transported via these secure connections. By utilizing these encrypted DTLS
tunnels, we no longer need to be concerned about the native security of protocols like SNMP, NTP, etc.
 vEdge Controller
DTLS/TLS control tunnel

Ge0/0 SNMP OMP NETCONF OMP eth0

Figure 3. Cisco SD-WAN DTLS Connection

In a typical production deployment, there are at least two or three controllers for redundancy purposes. When we have multiple
vSmarts, they also establish OMP peering between them in a full-mesh manner as shown in figure 4.

OMP peering between vSmarts vSmart

OMP peering between vEdges controller
and vSmart

vSmart vSmart
controller controller

vEdge vEdge
router router
Figure 4. OMP Peering with multiple controllers

OMP Route Advertisements

Cisco vEdge routers collect routes they learn from directly connected networks, static and dynamic routing protocols that run
in the site-local environment. These routes are then advertised to all OMP peers (to controllers) along with the corresponding
TLOC next-hops. The routes that represent reachability information are referred to as OMP routes or just vRoutes (to
distinguish them from traditional IP routes). However, vEdges also advertise to vSmart all locally attached services that are
running in the site-local network. Services include load balancers, firewalls, IDS (Intrusion Detection Systems) and could also
be customer-defined ones.

The vSmart controllers learn the topology of the overlay fabric and all available network services through these OMP route
advertisements coming from vEdges.
 OM
te
a vSmart

P
OMP routes d
p

Up
TLOC routes U
P

da
Service routes M Control Plane

te
O
Encryption keys
vEdge
TLOCs TLOCs
Services T1 IPsec Tunnel 1 T3
(FW, IPS, Routing
etc.) T2 IPsec Tunnel 2 T4 Information
Connected Static Data Plane Dynamic
routing (OSPF)

Campus Network Branch Network

Figure 5. Cisco SD-WAN Overlay Management Protocol

As it is visualized in figure 5, vEdge routers advertise three types of routes via the Overlay Management Protocol (OMP) to the
vSmart controllers:

OMP routes: OMP Routes, also referred to as vRoutes, are prefixes learned at the local site via connected interfaces,
static routes, and dynamic routing protocols (such as OSPF, EIGRP, and BGP) running on the service side of the vEdge.
These prefixes are redistributed into OMP and advertised to the vSmart controller so that they can be carried across the
overlay fabric to all other WAN edge nodes. OMP routes resolve their next-hop to a TLOC. An OMP route is installed in the
forwarding table only if the next-hop TLOC is known and there is a BFD session in UP state associated with that TLOC;
TLOC routes advertise Transport Locators of the connected WAN transports, along with additional attributes such as
public and private IP addresses, color, TLOC preference, site ID, weight, tags, and encryption keys.
Service routes advertise embedded network services such as firewalls and IPS that are connected to the vEdge local-site
network.

OMP Routes
Every Cisco vEdge router will advertise prefixes learned at the local site as OMP routes (vRoutes) to the vSmart controllers.
OMP can import reachability information from traditional protocols such as EIGRP, OSPF, and BGP and can also advertise
connected and static routes. The OMP route updates are very similar to traditional routing updates. However, the main
difference is that the next-hop of a vRoute points to a TLOC route. In BGP, a route is invalid if the next-hop IP address is not
resolved in the routing table. In OMP, if a vRoute has a next-hop TLOC that cannot be resolved in the TLOC route table, it is
marked as Invalid, similarly to BGP.
 vSmart RIB
1.1.1.0/24 via T1
1.1.1.0/24 via T2
2.2.2.0/24 via T3
2.2.2.0/24 via T4
OMP Update OMP Update
1.1.1.0/24 via T1 2.2.2.0/24 via T3
1.1.1.0/24 via T2 DT 2.2.2.0/24 via T4
nel LS
n tu
tu nn
LS
el
DT

BFD
T1 IPsec Tunnel 1 T3
vEdge1 VPN VPN
vEdge2
0 0
BFD
T2 IPsec Tunnel 2 T4
VPN1 VPN1
Directly Overlay Dynamic Routing
connected Fabric (OSPF, etc.)
1.1.1.0/24 2.2.2.0/24
Figure 6. vRoutes

In figure 6 for example, vEdge-1 is directly connected to subnet 1.1.1.0/24 in VPN1 and has two transport interfaces marked
with colors biz-internet and mpls (TLOCs T1 and T2). The router will advertise to the controller that 1.1.1.0/24 is reachable in
VPN1 via TLOCs T1 and T2. This advertisement is carried in an OMP UPDATE message as a vRoute. On the other side, vEdge-2
imports the reachability information learned via OSPF in VPN1 and advertises that subnet 2.2.2.0/24 is reachable in VPN1 via
TLOCs T3 and T4.

vRoutes consists of a lot of attributes in addition to the reachability information. Let’s list each attribute with a short
description:

VPN: Every OMP route is associated with a VPN, and every Cisco SD-WAN device keeps a separate routing table for each
VPN. This allows for the use of overlapping subnet ranges, provided they are in different VPNs. In the example above,
both subnets 1.1.1.0/24 and 2.2.2.0/24 are associated with VPN1 and will only be reachable in that network segment;
Originator: This is the System-IP of the router, from which the route was originally learned from. In our example for prefix
1.1.1.0/24, this value would be the system-IP of vEdge-1;
TLOC: This is the next-hop identifier of the OMP route. Note in the example that vEdge-1 advertises two vroutes for prefix
1.1.1.0/24, one via tloc T1 and one via tloc T2. This tells the vSmart controller and, subsequently, all remote WAN edge
routers that to reach subnet 1.1.1.0/24, they must have an active overlay tunnel to either tloc T1 or T2. Active means that
the BFD status associated with that tunnel must be in UP state;
Site ID: The site-id plays a similar role to a BGP AS number. It is primarily used for loop prevention. All sites should have a
unique site ID, and all devices at the same location should have the same site-id. In the example above, the site-id for
prefix 1.1.1.0/24 would be 1;
Origin-Protocol: This is the original protocol from which the vEdge router has learned the routing information. It may be a
connected interface, static route, or any existing dynamic routing protocols such as OSPF, EIGRP, or BGP. In the example
for prefix 1.1.1.0/24 via T1, the origin-proto will be Connected and OSPF-IA for 2.2.2.0/24.
Origin-Metric: OMP includes the original metric value alongside the origin protocol. These values are then used in the
best-path algorithm when OMP calculates the most optimal routes toward destinations. In the example for prefix
1.1.1.0/24 via T1, the origin-metric will be 0.
Preference: This attribute is also referred to as OMP preference or vRoute preference, so it is not confused with the TLOC
preference attribute in the TLOC routes. The OMP Preference is used for influencing the OMP best-path selection for a
given vroute. Higher is better. In the example above, if we’d like to manipulate the overlay routing in such a way so that the
 traffic towards 1.1.1.0/24 always comes through the MPLS cloud, we can just set a higher OMP Preference value to
vroute 1.1.1.0/24 via T2. The OMP preference operates very similarly to local_peference in BGP;
Tag: This is similar to the route tags in traditional routing. Once a value is set, it is a transitive attribute that can be acted
upon via policy.

A real example of a single vRoute can be seen in the output below:

vEdge-1# show omp route 172.16.1.0/24

---------------------------------------------------
omp route entries for vpn 1 route 172.16.1.0/24
---------------------------------------------------
RECEIVED FROM:
peer 1.1.1.30
path-id 5
label 1004
status C,I,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
originator 3.3.3.3
type installed
tloc 3.3.3.3, mpls, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 2
preference not set
tag not set
origin-proto connected
origin-metric 0
as-path not set
community not set
unknown-attr-len not set

OMP, as most traditional routing protocols, only advertises the best vroute or vroutes if there are multiple equal-cost ones.
Logically, invalid vroutes (having unresolved TLOC next-hop) are not advertised, even if they have the best attributes.

TLOC Routes
A TLOC route represents a WAN link that serves as a tunnel endpoint and is uniquely identified by {System-IP, Color,
Encapsulation}. Note that the System IP address is used instead of the interface IP address as an identifier for a TLOC route.
That’s because the interface IP can change at any given moment. Using the fixed System-IP ensures that the TLOC can be
uniquely identified at all times irrespective of any interface IP changes . This is very important because an OMP route (vRoute)
has a next-hop pointing to a TLOC. This separation of information allows TLOC routes to be updated with new parameters
without having to invalidate the dependent vRoutes. If a vEdge router has multiple transport interfaces connected to different
WAN providers, as shown in figure 7, a TLOC route is created and advertised for each WAN interface.
 RIB-TLOC Routes

vSmart T1, T2, T3, T4,

controller T5, T6, etc... If no policy applied,
advertises all TLOCs
to all vEdges
TLOC Route TLOC Route TLOC Route
T1, T2 T5, T6 T3, T4

T1 MPLS T3

T2 T4
INET
vEdge-1 T5 T6 vEdge-2
1.1.1.1 2.2.2.2

vEdge-3
Data Plane 3.3.3.3

Figure 7. TLOC Routes

Controlling the TLOC route advertisements is essential when customizing the overlay topology. vEdge routers attempt to form
IPsec tunnels to all remote TLOCs they receive from vSmart via all local TLOCs. If no policy is applied on vSmart, all vEdges
know about all TLOCs, resulting in a full mesh overlay (assuming that there is full IP reachability between all TLOCs). Let’s look
at figure 7. If we want to make sure that vEdge-1 would never form an overlay tunnel to vEdge-3, we apply a policy on vSmart
that filters TLOCs T5 and T6 in the TLOC route advertisements towards vEdge-1. In the end, vEdge-1 wouldn’t know about
TLOCs T5 and T6 and would never establish a data plane connection with vEdge-3.

A TLOC route advertisement contains the following attributes:

Private IPv4/IPv6 addresses and ports: These are the IP addresses configured or assigned via DHCP on the vEdge’s WAN
interface.
Public IPv4/IPv6 address/port: If the vEdge sits behind a NAT device, the outside NATed IP addresses and ports are
included in the TLOC route advertisements. If the router does not sit behind NAT, the public and private addresses and
ports are the same;
Color: The color is a logical abstraction used to identify a specific WAN interface on a WAN edge router. If no color is
explicitly configured under an interface, it is marked with the default color - “default”;
Encapsulation type: The encapsulation could be either GRE or IPsec. To successfully form a tunnel, the encapsulation
type of a TLOC must match with the remote TLOC’s encapsulation. In a typical production deployment, the encap will
always be IPsec for security reasons. However, when one is studying or testing features, it is a good practice to use GRE
encapsulation so that everything going through the overlay tunnels is cleartext and can be inspected with Wireshark.
Preference: This attribute is also referred to as a TLOC Preference, so it does not get confused with OMP Preference. It is
used in the OMP best-path algorithm when comparing multiple vroutes for the same destination. Higher is better, and the
default is 0;
Site ID: This attribute identifies the originating site for this TLOC route. WAN edge routers will never attempt to form an
overlay tunnel to a remote TLOC that has the same site-id.
Tag: An user-defined value that can be acted upon in a control policy;
Weight: Weight is a parameter used to achieve unequal traffic distribution across multiple local TLOCs with equal
preferences. A higher weight value sends more traffic to the local TLOC. For example, if TLOC-A is configured with a
weight of 20 and TLOC B with a weight of 1, then approximately 20 flows are sent out by TLOC A for every flow sent out
by TLOC-B.

You can see a real example of a TLOC route in the output below:
 vEdge-1# show omp tlocs
---------------------------------------------------
tloc entries for 1.2.3.4
public-internet
ipsec
---------------------------------------------------
RECEIVED FROM:
peer 1.1.1.30
status C,I,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
attribute-type installed
encap-key not set
encap-proto 0
encap-spi 261
encap-auth sha1-hmac,ah-sha1-hmac
encap-encrypt aes256
public-ip 24.5.4.1
public-port 12366
private-ip 192.168.1.2
private-port 12366
public-ip ::
public-port 0
private-ip ::
private-port 0
bfd-status up
domain-id not set
site-id 14
overlay-id not set
preference 0
tag not set
stale not set
weight 1
version 3
gen-id 0x80000007
carrier default
restrict 0
on-demand 0
groups [ 0 ]
bandwidth 0
qos-group default-group
border not set
unknown-attr-len not set

Service routes
A service route represents a network service such as a firewall or a load balancer connected to a WAN Edge router. Network
services are often deployed in one or several centralized locations, for example, in a data center or regional hub site. The
network must be able to reroute traffic from any remote location in the overlay through these services and then route the traffic
back to its original destination. This is called service chaining and is done using service routes.
 RIB-Service Routes

Service ID: 1(FW)

vSmart VPN ID: 50
Controller Originator: 3.3.3.3
TLOC: T3 vSmart does NOT
advertise service
OMP Update routes out to vEdges
Advertise attached FW services

vEdge-3 Firewall
Hub 5.5.5.5
3.3.3.3 VPN 50
T3

T1 T2
vEdge-1 vEdge-2
Spoke Spoke

Figure 8. Service Routes

Let’s look at figure 8, for example. vEdge-3 connects to a firewall (5.5.5.5) that provides an FW service to the overlay network. A
key point here is that the device providing the network service must be layer-2 adjacent to the vEdge router (there must be no
layer-3 nodes in between). Additionally, a network service is always associated with a VPN.

Configuring a network service on a vEdge router is as straightforward as one command line. Under the VPN configuration
hierarchy, we define the service type and the service IP address as shown in the output below:

vEdge-3
vpn 50
service FW address 5.5.5.5
!

Once we commit the configuration, vEdge-3 will advertise the FW service to the vSmart controller using a service route via the
overlay management protocol.

A service route contains the following attributes:

VPN ID: The VPN that this service applies to, in our example, would be 50;
Service ID: The service-id defines the type of service that is being advertised. There are 7 pre-defines values:
FW maps to svc-id 1;
IDS maps to svc-id 2;
IDP maps to svc-id 3;
Custom Services: The last four values are used for customer defined services:
netsvc1 maps to svc-id 4;
netsvc2 maps to svc-id 5;
netsvc3 maps to svc-id 6;
netsvc4 maps to svc-id 7;
Originator ID: The System-IP address of the vEdge that originates the service route;
TLOC: The TLOC (Transport Locator) where the service is located.

The key point here and the major difference between the Cisco SD-WAN service chaining and the Traditional WAN is that no
configuration is required on any remote WAN edge routers. If we look at the example shown in figure 8, this means that no
configuration needs to be applied on vEdge-1 and vEdge-2. The service chaining is completely done at the vSmart controller
using a policy that is then propagated to all remote sites that must redirect traffic through the service. In contrast, in the
traditional WAN, each node along the service chaining path must be manually provisioned to redirect traffic through the
network service.
 RIB-Service Routes
Service ID: 1(FW)
vSmart VPN ID: 50
controller Originator: 3.3.3.3 vSmart uses the
TLOC: T3 service route in a
policy
Policy

vEdge Firewall
Hub 5.5.5.5
3.3.3.3 VPN 50 The Policy
T3 redirects the
traffic from Site 2
vEdge-1 destined to Site 1
T1 T2
Site 1
The policy through the
vEdge-2 applied to firewall service
Site 2 site-2
VPN50 VPN50

Figure 9. Service Routes Example

At a high level, the steps to enable service chaining are as follows:

One or multiple WAN edge routers advertise a network service to the vSmart controller using an OMP service route. In
figure 9, vEdge-3 advertises the firewall service via a service route;
A policy that redirects the traffic from remote sites through the FW service is then defined on vSmart. Once processed by
the FW, the traffic is forwarded to its final destination.

We can check the network services on a vEdge router using the show omp services command, as shown in the output
below:

vEdge-3# show omp services

ADDRESS PATH
FAMILY VPN SERVICE ORIGINATOR FROM PEER ID LABEL STATUS
-------------------------------------------------------------------------------
ipv4 50 FW 3.3.3.3 10.1.1.30 67 1004 C,I,R

« Previous Lesson
Control Connections
Next Lesson
OMP Best-Path Selection »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / OMP Best-Path Selection

OMP Best-Path Selection

OMP Route Advertisements
In the Cisco SD-WAN solution, WAN Edge routers advertise their local networks to the Cisco vSmart controllers using the
Overlay Management Protocol (OMP). In a typical production deployment, most local networks are attached to two or more
vEdge devices for redundancy and most networks get advertised from multiple devices. Additionally, each subnet is advertised
as reachable via each Transport Locator (TLOC) that the WAN edge router has.

The vSmart controller

performs the OMP Best-Path
Algorithm
1.2.3.0/24 via T1
1.2.3.0/24 via T2
1.2.3.0/24 via T3
1.2.3.0/24 via T4
1.2.3.0/24 via T5
1.2.3.0/24 via T6
vSmart

Each vEdge advertises to vSmart that

By default, only the best 4
the subnet 1.2.3.0/24 is reachable via
routes (according to the OMP
all its TLOCs.
Best-Path Algorithm) are
advertised out.
T1 T2 T3 T4 T5 T6

vEdges

vEdges

1.2.3.0/24
Figure 1. OMP Routing Advertisements

If we look at the example shown in figure 1, three vEdge routers are connected to subnet 1.2.3.0/24. The first one advertises to
the vSmart controller that 1.2.3.0/24 is reachable via TLOC T1 and also that subnet 1.2.3.0/24 is reachable via TLOC T2. In the
same manner, the other two WAN edge routers advertise that subnet 1.2.3.0/24 is reachable via TLOCs 3,4,5, and 6. In the end,
the vSmart controller has six OMP routes for prefix 1.2.3.0/24. By default, vSmart is configured to advertise only four paths for
a given prefix. Therefore, it must compare all available routes for this prefix and select the best four that will be sent out to all
WAN edge routers. This is done using the Overlay Management Protocol Best-Path Algorithm.

Note that the number of routes per prefix that vSmart advertises can be changed using the following configuration:

send-path-limit (1-16)

It is also possible to configure the controller to send backup paths using the following configuration:
 send-backup-paths

OMP Best-Path Algorithm

vSmart controllers and vEdge routers perform the Best-Path Selection when they have multiple routes for the same prefix.
Figure 2 shows the Best-Path algorithm that Cisco SD-WAN devices go through.

Route State
Prefer ACTIVE routes over STALE routes
OMP
Route Resolvability Best-Path
Next-hop TLOC must be reachable
Selection
Admin Distance (vEdge only) vSmart/vEdge
Prefer OMP routes with lowest AD
Inbound
Control Policy
Route Preference
Prefer OMP routes with highest route preference
Incoming
OMP update
TLOC Preference (vEdge only)
Prefer OMP routes with highest TLOC preference

Origin
Prefer OMP routes with best origin (1.Type, 2.Metric)
The routes are equal-cost

Routes are installed in Tiebreaker (vSmart only)

the OMP table, sorted Prefer vEdge sourced routes over vSmart sourced
The tiebreakers are used
to sort the OMP routes Tiebreaker
in descending order Prefer OMP routes with lowest origin System-IP
(from best to worst)
Tiebreaker Outbound
Prefer routes with lowest TLOC private address Control Policy

send-backup-paths

send-path-limit

Outgoing
OMP update

Figure 2. OMP Best-Path Selection Algorithm

Let's look at each step of the OMP Best-Path Selection in more detail:

1. Prefer ACTIVE routes over STALE routes. A route is ACTIVE when there is an OMP session in UP state with the peer that
sent out the route. A route is STALE when the OMP session with the peer that sent out the route is in GRACEFUL RESTART
mode;
2. Select routes that are Valid. Ignore invalid routes. A route must have a next-hop TLOC that is known and reachable.
3. Prefer routes with lower administrative distance (AD) (on vEdge only). AD is a locally-significant value on each router and
depends on the OS. Different platforms may have different AD values for different protocols. For example, OMP has AD of
250 on vEdges and 251 on cEdges. Additionally, network admins can define floating static routes with various ADs for the
same prefix. AD is only compared when the same WAN edge router receives the same site-local prefix from multiple
routing protocols. AD is not a parameter in OMP, is not advertised, and does not influence vSmart.
4. Prefer routes with a higher route preference value. By default, all omp routes have 0 preference. This is typically the most
often used value when we need to do traffic engineering.
5. Prefer routes with a higher TLOC preference value (on vEdge only). TLOC preference is a parameter in TLOC routes. And
TLOC routes are not bound to VPN-id. Therefore, changing the TLOC preference affects vEdges path selection for all
VPNs.
6. Compare the origin type, and select the first match in the following order:
1. Connected
 2. Static
3. EIGRP summary
4. EBGP
5. OSFP intra-area
6. OSPF inter-area
7. IS-IS level 1
8. EIGRP external
9. OSPF external
10. IS-IS level 2
11. IBGP
12. Unknown
7. Compare the origin metric - If the origin type of the routes is the same, select the routes that have the lower origin metric.
8. Tiebreaker - Prefer vEdge sourced routes over vSmart sourced. (on vSmart only)
9. Tiebreaker - If the origin types are equal, select the routes that have the lowest router-id (System-IP).
10. Tiebreaker - If the router IDs are the same, prefer the routes with the lowest private TLOC IP address.

ECMP - To be considered equal, omp routes must be valid and equal-cost up to step 8 (green and blue steps in figure 2). When
there are more equal-cost routes than the send-path-limit value, the controller sorts the best ones based on the tiebreakers in
descending order and advertises as many as the send-path-limit. This is visualized in figure 1.

Note that Cisco vEdge routers install a route in their forwarding table (FIB) only if the TLOC to which it points is active. Active
TLOCs are ones that have a BFD session in UP state associated with them. When for whatever reason a BFD session becomes
down(inactive), the Cisco vSmart/vEdge devices remove all routes that point to that TLOC from their forwarding table.

Best-Path Selection Examples

Let's look at some basic but important examples:

A vSmart controller receives a route to 1.2.3.0/24 from a Cisco WAN Edge router with an origin code of eBGP. The
controller also receives the same route from another vSmart Controller, also with an origin code of eBGP. Assuming all
other properties are equal, the best-path algorithm would choose the route that came from the Cisco vEdge device.
A Cisco vSmart Controller learns the same route, 10.10.10.0/24, from two Cisco vEdge devices on the same site. If all
other parameters are the same, both routes are chosen and advertised to other peers. By default, up to four equal-cost
routes are selected and advertised.
A Cisco vSmart controller receives eight OMP routes for prefix 172.16.1.0/24. The send-path-limit value is the default one
- 4. Six of them are chosen as best based on the OMP best path algorithm. They have the flag C set as you can see on the
output below. The reason the other two have not been chosen as best can be seen in the loss-reason, lost-to-peer, and
lost-to-path-id columns.
 vSmart# show omp routes 172.16.1.0/24 detail | t
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
LOST
TO
PATH PATH ATTRIBUTE
VPN PREFIX FROM PEER ID LABEL STATUS LOSS REASON LOST TO PEER ID TYPE
TLOC IP COLOR ENCAP PROTOCOL METRIC DOMAIN ID SITE ID ORIGINATOR
------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------
1 172.16.1.0/24 1.1.1.3 66 1004 C,R - - - installed
1.1.1.3 mpls ipsec connected 0 - 2 1.1.1.3 -
1.1.1.3 69 1004 C,R - - - installed
1.1.1.3 public-internet ipsec connected 0 - 2 1.1.1.3 -
1.1.1.4 66 1004 R origin-protocol 14.1.1.1 69 installed
1.1.1.4 mpls ipsec OSPF-intra-area 11 - 3 1.1.1.4 -
1.1.1.4 69 1004 R tloc-id 1.1.1.4 66 installed
1.1.1.4 public-internet ipsec OSPF-intra-area 11 - 3 1.1.1.4 -
11.1.1.1 34 1004 C,R - - - installed
11.1.1.1 mpls gre connected 0 - 11 11.1.1.1 -
11.1.1.1 66 1004 C,R - - - installed
11.1.1.1 mpls ipsec connected 0 - 11 11.1.1.1 -
11.1.1.1 69 1004 C,R - - - installed
11.1.1.1 public-internet ipsec connected 0 - 11 11.1.1.1 -
14.1.1.1 69 1004 C,R - - - installed
14.1.1.1 public-internet ipsec connected 0 - 14 14.1.1.1 -

So there are six equal-cost routes (flagged with C) to 172.16.1.0/24, but the send-path-limit value is set to 4. Therefore, the
controller sorts them based on the lowest originator System-IP and if there is a tie, based on the lowest TLOC private IP
address, and advertises out the best four.

OMP Graceful Restart

While studying how Cisco SD-WAN works, have you ever wondered what happens with the overlay fabric when the SD-WAN
Control Plane becomes unavailable? Well, there is a feature called OMP Graceful Restart that allows the data plane to continue
functioning and forwarding traffic even if the control plane suddenly goes down or becomes unavailable. WAN edge devices do
this by using the last known routing information that they received from the vSmart controllers. At the same time, vEdges
actively try to re-establish a control-plane connection to the vSmart controllers. When the controllers are back up and
reachable, DTLS control connections are re-established, and the vEdge routers then receive updated and refreshed network
information from the vSmart controllers.
 Figure 3. OMP Graceful Restart

Cisco vEdge and vSmart devices cache the OMP information that they learn from peers. The cached information includes OMP,
TLOC, and SERVICE routes, IPsec SA parameters, and the centralized data policies in place. When a WAN edge device loses its
OMP peering to the vSmart controller, the device continues forwarding data traffic using the cached OMP information. The
Edge device also periodically checks whether the vSmart controller has come up. When it does come back up and the control
plane peering is re-established, the device flashes its local cache and refreshes the control plane information from the vSmart
controller. This same technique is valid in the opposite scenario when a vSmart controller no longer detects the presence
of Cisco vEdge devices. It then uses its local cache until the WAN edge device becomes reachable again.

Manipulating the Best-Path Selection Process

Similar to the well-known process of manipulating the BGP best-path selection process with route-maps, in Cisco SD-WAN we
have the ability to manipulate the route selection locally on WAN edge devices. This is typically done using a Local device
template. We have done that many times in traditional networking. The more interesting ability that SD-WAN allows us to do is
to manipulate the routing information that goes in and out of the controller's routing table. This allows for a per-prefix network-
wide routing manipulation, similarly to modifying route properties on a BGP Route-Reflector.
 Routing Table

vSmart

Inbound Centralized Outbound Centralized

Policy Policy
Modifies the routing information Modifies the routing information
before in enters the routing after the best-path selection
table of the controller has taken place

vEdge vEdge

Figure 4. Manipulating the Cisco SD-WAN best-path selection process

There are two configuration options that allow for that:

Using Inbound Centralized Policy - With an inbound centralized policy, we can change the origin code or the TLOC preference
of a particular prefix before it goes into the routing table of the vSmart controller. This will then influence the best-path
selection and lead to a different output from the best-path algorithm. Remember that the best routes are then advertised
downstream to all WAN edge routers. Therefore, any manipulation of the controller's routing table will change the control-plane
information across the whole overlay fabric.

Using Outbound Centralized Policy - With an outbound centralized policy, we typically modify the routing information that is
sent to a particular set of WAN edge devices in order to influence their own best-path selection algorithm.

« Previous Lesson
OMP Overview
Next Lesson
OMP Send Path Limit »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / OMP Send Path Limit

OMP Send Path Limit

In this lesson, we will explore an OMP configuration parameter called send-path-limit. Along the way, we will discuss in detail
how the OMP best-path algorithm works and what are the key points in the selection process that are often overlooked.

What is OMP Send Path Limit?

OMP Send-Path-Limit is a configuration parameter that defines the maximum number of equal-cost vroutes that a vSmart
controller or a vEdge router advertises to its OMP peers. By default, the omp send-path-limit is set to 4, which means that if a
vSmart controller has ten equal-cost best paths to a destination, it will only advertise four routes to vEdges.

Changing the send-path-limit value is as simple as applying one configuration line under the omp configuration hierarchy, as
shown in the output below:

vEdge/vSmart# conf t
Entering configuration mode terminal
vEdge/vSmart(config)#omp send-path-limit (1-16)
vEdge/vSmart(config)# commit and-quit
Commit complete.
vEdge/vSmart#

OMP Send-Path-Limit Configuration Example

The initial state
To demonstrate how the send-path-limit parameter works in the context of the OMP Best-Path algorithm, we are going to use
the network devices shown in figure 1 below. We have three WAN edge routers - vEdges 1, 2, and 3 that are directly connected
to network 10.1.1.0/24 in VPN 100 and one vSmart controller that oversees the sd-wan domain. We have three more vEdge
routers (vEdges 4, 5, and 6) that will be referred to as "the other" vEdges.
 All routes have the same:
- Admin Distance
- OMP Preference vSmart
Which routes
- TLOC Preference
will the vSmart
- Origin-type
controller
- Origin-metric
advertise out?
Update
OMP
Other vEdges
10.1.1.0/24 via T1 10.1.1.0/24 via T3 10.1.1.0/24 via T5
10.1.1.0/24 via T2 10.1.1.0/24 via T4 10.1.1.0/24 via T6
Which routes
T1 T2 T3 T4 T5 T6 will vEdges
39.3.0.1 10.10.0.1 39.3.0.2 10.10.0.2 39.3.0.3 10.10.0.3 install in their
FIB tables?

vEdge-1 vEdge-2 vEdge-3

1.1.1.1 2.2.2.2 3.3.3.3

Directly Connected 10.1.1.0/24 in VPN 100

Figure 1. Best-Path Algorithm Questions

There are two WAN transports that are not illustrated in the topology - an Internet cloud and an MPLS cloud. vEdge routers 1, 2,
and 3 have the following transport attachments:

vEdge-1 has two local TLOCs:

T11 with IP address 39.3.0.1 and marked with the biz-internet color;
T12 with IP address 10.10.0.1 and marked with the mpls color;
vEdge-2 has two local TLOCs:
T21 with IP address 39.3.0.2 and marked with the biz-internet color;
T22 with IP address 10.10.0.2 and marked with the mpls color;
vEdge-3 has two local TLOCs:
T31 with IP address 39.3.0.3 and marked with the biz-internet color;
T32 with IP address 10.10.0.3 and marked with the mpls color.

Each vEdge router advertises two routes for subnet 10.1.1.0/24 in VPN 100 to the vSmart controller - one via its biz-interface
TLOC and one via its mpls TLOC. Therefore, the vSmart controller receives six OMP routes for 10.1.1.0/24. All routes have
identical values for AD (250), OMP Preference (0), TLOC preference (0), Origin-type (Connected), and Origin-Metric (1). Given
this information, try to answer the following question, having in mind that everything else is by default and there is no policy
applied on vSmart:

Which routes will the vSmart controller choose as best for prefix 10.1.1.0/24?
Which routes will the vSmart controller advertise to the other vEdge routers in the SD-WAN domain?
Which ones will the controller advertise to the other vEdges if we change the default send-path-limit value to 5?
Which routes for 10.1.1.0/24 will the other vEdges install in their forwarding table (FIB)?

The answers
At first, answering the questions seem very straightforward. However, many engineers overlook a key part of the OMP best-
path selection process. The Smart controller uses the steps in the best-path algorithm to select the best routes to a
destination. This is pretty simple and well understood. However, the thing that many don't realize is that after the best paths are
selected, the vSmart controller uses the tiebreakers to sort the routes in descending order (from best to worst)! The controller
then stores and keeps the best paths in its VPN routing table sorted in descending order. Then the send-path-limit comes into
the picture. Suppose that the parameter is set to X - this tells vSmart to "take the first X routes and advertise them out".
Ok, having said that, let's go ahead and sort the routes shown in figure 1. As we have said, all routes have the same AD
(250), OMP Preference (0), TLOC preference (0), Origin-type (Connected), and Origin-Metric (1) which means that according to
the best-path algorithm, they are all best equal-cost routes to destination 10.1.1.0/24. However, the controller must sort routes
before inserting them into the VPN 100 routing table. As of Cisco SD-WAN version 20.7.1, there are 3 tiebreakers:

1. Tiebreaker (Source Preference) - Prefer vEdge sourced routes over vSmart sourced.
2. Tiebreaker (System IP) - Prefer routes that have lower System-IP.
3. Tiebreaker (Private TLOC IP) - For routes coming from the same vEdge, prefer the ones with a lower private TLOC IP
address.

Let's use these tiebreakers and sort the routes manually. From the perspective of the vSmart controller, all six routes come
from vEdges. Therefore, according to tiebreaker 1, they are all equal. Then using tiebreaker 2, we can work out that the routes
via vEdge-1 would be better than the ones via vEdge-2 because vEdge-1 has a lower system-IP than vEdge-2. Logically, the
routes via vEdge-3 would be worst (vEdge1(1.1.1.1) < vEdge2(2.2.2.2) < vEdge3(3.3.3.3)). Ok, but which one of the two routes
of vEdge-1 to put at the top? For this, we must also use tiebreaker 3. We can work out that route "10.1.1.0/24 via T2" is better
than "10.1.1.0/24 via T1" because T2(10.10.0.1) has a lower private TLOC IP address than T1(39.3.01). Applying this logic to all
routes, we will end up with the order shown in figure 2 below:

vRoutes sorted
All routes have the same: from best to worst
- Admin Distance send-path-limit
vSmart 10.1.1.0/24 via T2
- OMP Preference (4 by default)
10.1.1.0/24 via T1
- TLOC Preference 10.1.1.0/24 via T4
- Origin-type 10.1.1.0/24 via T3
- Origin-metric
10.1.1.0/24 via T6
10.1.1.0/24 via T5
Update
OMP
Other vEdges
10.1.1.0/24 via T1 10.1.1.0/24 via T3 10.1.1.0/24 via T5
10.1.1.0/24 via T2 10.1.1.0/24 via T4 10.1.1.0/24 via T6

T1 T2 T3 T4 T5 T6
39.3.0.1 10.10.0.1 39.3.0.2 10.10.0.2 39.3.0.3 10.10.0.3

vEdge-1 vEdge-2 vEdge-3

1.1.1.1 2.2.2.2 3.3.3.3

Directly Connected 10.1.1.0/24

Figure 2. Send Path Limit

We can verify that this is the correct order by checking the routing table for VPN 100 on vSmart. You can see that vEdge1's
mpls route is at the top, vEdge1's biz-internet route is second, and so on.
 vSmart# show omp routes vpn 100 | t
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved

PATH ATTRIBUTE
VPN PREFIX FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR
ENCAP PREFERENCE
------------------------------------------------------------------------------------------------------------------
--------------------
100 10.1.1.0/24 1.1.1.1 66 1011 C,R installed 1.1.1.1 mpls
ipsec -
1.1.1.1 68 1011 C,R installed 1.1.1.1 biz-internet
ipsec -
2.2.2.2 66 1018 C,R installed 2.2.2.2 mpls
ipsec -
2.2.2.2 68 1018 C,R installed 2.2.2.2 biz-internet
ipsec -
3.3.3.3 66 1009 C,R installed 3.3.3.3 mpls
ipsec -
3.3.3.3 68 1009 C,R installed 3.3.3.3 biz-internet
ipsec -

Ok, after the best routes are selected and sorted, the vSmart controller takes the first 4 route (by default) and advertise them to
the overlay fabric. This can be verified on any other vEdge router.

vEdge-6# sh omp route vpn 100 | t

Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved

PATH ATTRIBUTE
VPN PREFIX FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR
ENCAP PREFERENCE
------------------------------------------------------------------------------------------------------------------
--------------------
100 10.1.1.0/24 1.1.1.30 71 1011 C,I,R installed 1.1.1.1 mpls
ipsec -
1.1.1.30 72 1011 C,I,R installed 1.1.1.1 biz-internet
ipsec -
1.1.1.30 87 1018 C,I,R installed 2.2.2.2 mpls
ipsec -
1.1.1.30 88 1018 C,I,R installed 2.2.2.2 biz-internet
ipsec -
What is the OMP ecmp-limit?
Once we have understood what the send-path limit parameter does, let's go ahead and change it to a non-default value, for
example, let's make it 5.

As expected, the vSmart controller will advertise the first five best routes. This can easily be verified on any of the other
vEdges. However, notice another important thing. Even though vEdge6 now receives five routes for 10.1.1.0/24 in VPN100, the
router only installs the first four in the routing table. (see the flag I)

vEdge-6# sh omp route vpn 100 | t

Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved

PATH ATTRIBUTE
VPN PREFIX FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR
ENCAP PREFERENCE
------------------------------------------------------------------------------------------------------------------
--------------------
100 10.1.1.0/24 1.1.1.30 71 1011 C,I,R installed 1.1.1.1 mpls
ipsec -
1.1.1.30 72 1011 C,I,R installed 1.1.1.1 biz-internet
ipsec -
1.1.1.30 87 1018 C,I,R installed 2.2.2.2 mpls
ipsec -
1.1.1.30 88 1018 C,I,R installed 2.2.2.2 biz-internet
ipsec -
1.1.1.30 96 1009 C,R installed 3.3.3.3 mpls
ipsec -

We can verify this by checking the routing table for VPN 100.

vEdge-6# sh ip route vpn 100 10.1.1.0/24

Codes Proto-sub-type:
IA -> ospf-intra-area, IE -> ospf-inter-area,
E1 -> ospf-external1, E2 -> ospf-external2,
N1 -> ospf-nssa-external1, N2 -> ospf-nssa-external2,
e -> bgp-external, i -> bgp-internal
Codes Status flags:
F -> fib, S -> selected, I -> inactive,
B -> blackhole, R -> recursive, L -> import

PROTOCOL NEXTHOP NEXTHOP NEXTHOP

VPN PREFIX PROTOCOL SUB TYPE IF NAME ADDR VPN TLOC IP COLOR
ENCAP STATUS
------------------------------------------------------------------------------------------------------------------
---------------------------
100 10.1.1.0/24 omp - - - - 1.1.1.1 mpls
ipsec F,S
100 10.1.1.0/24 omp - - - - 1.1.1.1 biz-
internet ipsec F,S
100 10.1.1.0/24 omp - - - - 2.2.2.2 mpls
ipsec F,S
100 10.1.1.0/24 omp - - - - 2.2.2.2 biz-
internet ipsec F,S
What actually happened is illustrated in figure 3 below. Even though vEdge6 now receives five routes for 10.1.1.0/24 in
VPN100, the number of routes that get installed in the routing tables is subject to another OMP parameter called ecmp-limit.
By default, this parameter is also set to 4, which means that whatever number of best routes a vEdge may have, it will only
install the best four in the routing table.

vRoutes sorted
from best to worst Only the best N routes
vSmart 10.1.1.0/24 via T2 are advertised out.
10.1.1.0/24 via T1 N = send_path_limit
10.1.1.0/24 via T4
10.1.1.0/24 via T3
10.1.1.0/24 via T6 N=5
ate 10.1.1.0/24 via T5 Z=4
Upd
OMP

10.1.1.0/24 via T1 10.1.1.0/24 via T3 10.1.1.0/24 via T5 All other vEdges

10.1.1.0/24 via T2 10.1.1.0/24 via T4 10.1.1.0/24 via T6

T1 T2 T3 T4 T5 T6 10.1.1.0/24 via T2
10.1.1.0/24 via T1
10.1.1.0/24 via T4
10.1.1.0/24 via T3
vEdge-1 vEdge-2 vEdge-3 Only the best Z routes
1.1.1.1 2.2.2.2 3.3.3.3
are installed in the
forwarding table (FIB)
Directly Connected 10.1.1.0/24 Z = ecmp-limit
Figure 3. ECMP Limit

Let's change the ecmp-limit parameter and set it to the non-default value of 5.

vEdge-6# conf t
Entering configuration mode terminal
vEdge-6(config)# omp ecmp-limit 5
vEdge-6(config)# commit and-quit
Commit complete.

Now if we check the routing table, we can see that vEdge6 installs all five routes it receives.

vEdge-6# sh ip route vpn 100 10.1.1.0/24 | t

ADDRESS PATH PROTOCOL NEXTHOP NEXTHOP

NEXTHOP
VPN FAMILY PREFIX ID PROTOCOL SUB TYPE METRIC IFNAME ADDR TLOC IP COLOR ENCAP VPN
STATUS
------------------------------------------------------------------------------------------------------------------
------------
100 ipv4 10.1.1.0/24 0 omp - 0 - - 1.1.1.1 mpls ipsec -
F,S
100 ipv4 10.1.1.0/24 1 omp - 0 - - 1.1.1.1 biz-internet ipsec -
F,S
100 ipv4 10.1.1.0/24 2 omp - 0 - - 2.2.2.2 mpls ipsec -
F,S
100 ipv4 10.1.1.0/24 3 omp - 0 - - 2.2.2.2 biz-internet ipsec -
F,S
100 ipv4 10.1.1.0/24 4 omp - 0 - - 3.3.3.3 mpls ipsec -
F,S
Key takeaways
The OMP best-path algorithm does not only select the best routes but also sorts them in descending order (from best to
worst).
The Smart controller always inserts and keeps all routes sorted, with the best route at the top.
The send-path-limit parameter defines the maximum number of best-paths that can be advertised.
The ecmp-limit parameter defines the maximum number of best paths that can be installed in the routing table.
Bonus - The controller-send-path-limit defines the maximum number of best-paths that a vSmart controller can advertise
to another vSmart controller.

« Previous Lesson
OMP Best-Path Selection
Next Lesson
OMP Send Backup Paths »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

OMP Source Preference

Understanding OMP Routing

OMP TLOC-Action

Chapter Key Takeaways

QUIZ - Control Plane Fundamentals

4. Cisco SD-WAN Data Plane

What is a TLOC?
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / OMP Send Backup Paths

OMP Send Backup Paths

In this lesson, we will explore a common problem that occurs in scenarios where a dual-homed, dual-transport site advertises
routes with different origin metrics or origin types. The site will typically be a data center or a regional hub with a site-local IGP
such as BGP or OSPF running in the environment. We will see how the Overlay Management Protocol (OMP) best-path
selection causes a complete loss of reachability to a remote site in case of a link failure, even if backup paths exist.

What is OMP Send-Backup-Paths?

By default, when a Cisco vSmart controller receives multiple OMP routes to the same destination, it runs them through the
OMP best-path algorithm and selects the best ones to that destination subnet. Then the vSmart controller advertises only the
best routes out to the rest of the overlay fabric. This behavior is pretty well-known to network engineers. BGP route reflectors
work pretty much the same way when receiving and re-advertising routes to route-reflector clients.

The Cisco SD-WAN solution provides a configuration option that tells the vSmart controller to advertise the first set of non-best
routes to vEdge routers. The configuration is as simple as adding one configuration line on the controller, as shown in the
output below.

vSmart# config
Entering configuration mode terminal
vSmart(config)# omp send-backup-paths
vSmart(config-omp)# commit
Commit complete.
vSmart#

The key point to notice here is what "the first set of non-best routes" mean? Suppose there are 2 equal-cost best routes to a
destination and name them ''routes-X". Let's say that there are another 2 equal-cost non-best routes to the same destination
and name them "routes-Y". By default, vSmart will only advertise routes-X. When we enable the omp send-backup-path option,
the controller will advertise routes-Y alongside routes-X. However, if there are other routes that are worst than routes-Y, they will
not be sent out by the vSmart controller, because only the first set of non-best routes are advertised!

When is OMP Send-Backup-Paths used?

Let's now see a real-world example of when we would like to enable the vSmart controller to send non-best paths.

The initial state

We have three WAN edge routers - vEdges 1, 2, and 3. Routers 1 and 2 are located in a data center with site-id 1. Router 3 is
located at a remote branch with site-id 3. The initial state of this lab example is illustrated in figure 1 below.
 Routes via vEdge-1 are best
NORMAL SCENARIO due to lower origin metric
vSmart
10.1.1.0/24 via T11
10.1.1.0/24 via T12
10.1.1.0/24 via T21
10.1.1.0/24 via T11 10.1.1.0/24 via T22
10.1.1.0/24 via T12
Origin Metric 50
T11
OSPF metric 50 IPsec
vEdge1 tunn
el
T12

10.1.1.0/24
MPLS T31 vEdge3

Site-local 10.1.1.0/24 via T11

network T21 10.1.1.0/24 via T12
OSPF metric 90
vEdge2 INET
T22 The route via T12 is
10.1.1.0/24 via T21 Invalid, because
10.1.1.0/24 via T22 vEdge-3 does not have
Origin Metric 90
a BFD session to T12
Figure 1. Normal Scenario

The routers have the following transport attachments:

vEdge-1 has got two TLOCs - T11 marked with the mpls color and T12 marked with the biz-internet color.
vEdge-2 has also got two TLOCs - T21 marked with the mpls color and T22 marked with the biz-internet color.
vEdge-3 has only got one TLOC - T31 marked with the mpls color.

OSPF runs in the data center between vEdges 1 and 2 and the local network devices. The site-local router in the datacenter
advertises subnet 10.1.1.0/24 to vEdges 1 and 2 in VPN10:

vEdge-1 receives the 10.1.1.0/24 via OSPF with metric 50 and redistributes it into OMP with origin-metric 50
vEdge-2 receives the 10.1.1.0/24 via OSPF with metric 90 and redistributes it into OMP with origin-metric 90

As illustrated in figure 1, the vSmart controller receives four OMP routes for 10.1.1.0/24. It runs the OMP best-path algorithm
and selects the ones via vEdge-1 as best routes because they have lower origin-metric 50 than the router via vEdge-2 (metric
90).
 vSmart# show omp routes vpn 10 10.1.1.0/24 | t
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved

PATH ATTRIBUTE
FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
-----------------------------------------------------------------------------------------------------------
1.1.1.1 66 1009 C,R installed 1.1.1.1 mpls ipsec -
1.1.1.1 68 1009 C,R installed 1.1.1.1 biz-internet ipsec -
1.1.1.2 66 1016 R installed 1.1.1.2 mpls ipsec -
1.1.1.2 68 1016 R installed 1.1.1.2 biz-internet ipsec -

Subsequently, the vSmart controller advertises to vEdge-3 only the best routes - 10.1.1.0/24 via T11 and 10.1.1.0/24 via T12.
Therefore, vEdge-3 does not even know that subnet 10.1.1.0/24 is also reachable via vEdge-2!

Additionally, vEdge-3 marks route 10.1.1.0/24 via T12 as Invalid because it does not have an overlay tunnel to vEdge-1's biz-
internet TLOC.

vEdge-3# sh omp route vpn 10 10.1.1.0/24 | t

Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved

PATH ATTRIBUTE
FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
-----------------------------------------------------------------------------------------------------------
1.1.1.30 87 1009 C,I,R installed 1.1.1.1 mpls ipsec -
1.1.1.30 88 1009 Inv,U installed 1.1.1.1 biz-internet ipsec -

In the end, vEdge-3 has only one valid path to reach subnet 10.1.1.0/24 through the mpls interface of vEdge-1, even though the
subnet can be reached via vEdge2's mpls interface!

vEdge-3# sh ip route vpn 10 10.1.1.0/24 | t

ADDRESS PATH PROTOCOL NEXTHOP NEXTHOP NEXTHOP

VPN FAMILY PREFIX ID PROTOCOL SUB TYPE METRIC IFNAME ADDR TLOC IP COLOR ENCAP VPN
STATUS
------------------------------------------------------------------------------------------------------------------
-----
10 ipv4 10.1.1.0/24 0 omp - 0 - - 1.1.1.1 mpls ipsec - F,S

However, there is IP reachability between the remote site and the data center network in a normal state of the environment.
 vEdge-3# ping vpn 10 10.1.1.1
Ping in VPN 10
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_seq=1 ttl=254 time=46.7 ms
64 bytes from 10.1.1.1: icmp_seq=2 ttl=254 time=61.9 ms
64 bytes from 10.1.1.1: icmp_seq=3 ttl=254 time=41.7 ms
^C
--- 10.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 41.730/50.150/61.944/8.593 ms

The problem
Let's see what happens when we introduce a failure by shutting down the mpls TLOC on vEdge-1, as illustrated in figure 2
below.

FAILURE SCENARIO Routes via vEdge-1 are best

TLOC T11 is DOWN vSmart due to lower origin metric

10.1.1.0/24 via T12

10.1.1.0/24 via T21

10.1.1.0/24 via T11 10.1.1.0/24 via T22
10.1.1.0/24 via T12
Origin Metric 50
OSPF metric 50
T11
vEdge1
T12

10.1.1.0/24
MPLS l
T31 vEdge3
tunne
IPsec
Site-local 10.1.1.0/24 via T12
network T21
OSPF metric 90
vEdge2 INET vEdge-3 has no valid
T22 routes to 10.1.1.0/24
10.1.1.0/24 via T21
anymore!
10.1.1.0/24 via T22
Origin Metric 90

Figure 2. Failure Scenario - TLOC T11 is down

vEdge-1 no longer advertises the OMP route 10.1.1.0/24 via T11 because the mpls interface is down. Subsequently, the vSmart
controller has only one route to 10.1.1.0/24 via vEdge-1. However, the OMP best-path algorithm still chooses this route as best,
because it has a lower origin-metric (50) than the ones via vEdge-2 (90). In the end, vEdge-3 receives only the OMP route to
10.1.1.0/24 via T12 (the biz-internet TLOC of vEdge-1). However, vEdge-3 does not have an overlay tunnel to any remote biz-
internet TLOC. Therefore, the route 10.1.1.0/24 via the biz-internet TLOC of vEdge-1 is marked as Invalid and Unresolved.
 vEdge-3# show omp route vpn 10 10.1.1.0/24 | t
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved

PATH ATTRIBUTE
FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
-----------------------------------------------------------------------------------------------------------
1.1.1.30 94 1009 Inv,U installed 1.1.1.1 biz-internet ipsec -

Although vEdge-3 has an overlay tunnel to vEdge-2 and BFD session in UP state, it doesn't know that subnet 10.1.1.0/24 could
be reached via vEdge-2! If we ping 10.1.1.1, we will see that vEdge-3 has completely lost connectivity to the data center's
network.

vEdge-3# ping vpn 10 10.1.1.1

Ping in VPN 10
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
From 127.1.0.2 icmp_seq=1 Destination Net Unreachable
From 127.1.0.2 icmp_seq=2 Destination Net Unreachable
From 127.1.0.2 icmp_seq=3 Destination Net Unreachable
^C
--- 10.1.1.1 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss

The solution
One of the most effective solutions to this problem is to tell the vSmart controller to advertise also the first set of non-best
routes. In our example, these would be the routes via vEdge-2.

SOLUTION Routes via vEdge-1 are best

Send Backup Paths vSmart due to lower origin metric

10.1.1.0/24 via T12

10.1.1.0/24 via T21

10.1.1.0/24 via T11 10.1.1.0/24 via T22
10.1.1.0/24 via T12
Origin Metric 50 vSmart sends
T11 non-best paths
OSPF metric 50
vEdge1
T12

10.1.1.0/24
MPLS unnel
T31 vEdge3
ec t
IPs
Site-local 10.1.1.0/24 via T12
network T21 10.1.1.0/24 via T21
OSPF metric 90
vEdge2 INET 10.1.1.0/24 via T22
T22
10.1.1.0/24 via T21 vEdge-3 uses the backup
10.1.1.0/24 via T22 paths via vEdge-2
Origin Metric 90
 Figure 3. Solution Scenario

Configuring the feature is as simple as applying one configuration line omp send-backup-paths on the vSmart controller, as
shown in the output below:

vSmart# conf t
Entering configuration mode terminal
vSmart(config)# omp ?
Possible completions:
controller-send-path-limit Limit number of paths sent to vSmart controller
for a prefix
discard-rejected Enable/Disable storage of information rejected
by policy
graceful-restart Enable/Disable graceful restart
send-backup-paths Enable/Disable transmission of backup paths
send-path-limit Maximum number of paths sent for a prefix
shutdown Enable/disable OMP
timers Set timers
<cr>
vSmart(config)# omp send-backup-paths
vSmart(config-omp)# commit and-quit
Commit complete.

Once we configure the send-backup-paths option, we can see that the controller is now advertising the routes via vEdge-2
alongside the best routes via vEdge-1. Now if we check the omp routing table on vEdge-3, we will see that it has a valid route to
10.1.1.0/24 through the mpls TLOC of vEdge-2.

vEdge-3# show omp route vpn 10 10.1.1.0/24 | t

Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved

PATH ATTRIBUTE
FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
-----------------------------------------------------------------------------------------------------------
1.1.1.30 94 1009 Inv,U installed 1.1.1.1 biz-internet ipsec -
1.1.1.30 106 1016 C,I,R installed 1.1.1.2 mpls ipsec -
1.1.1.30 107 1016 Inv,U installed 1.1.1.2 biz-internet ipsec -

Now there is IP reachability between the remote site and the data center.

vEdge-3# ping vpn 10 10.1.1.1

Ping in VPN 10
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_seq=1 ttl=254 time=57.6 ms
64 bytes from 10.1.1.1: icmp_seq=2 ttl=254 time=48.9 ms
64 bytes from 10.1.1.1: icmp_seq=3 ttl=254 time=47.1 ms
^C
--- 10.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 47.182/51.229/57.600/4.563 ms

And the ultimate test is to check the actual data path using a traceroute from vEdge-3 to the data center network. You can that
the traffic entering the DC via vEdge-2.
 vEdge-3# traceroute vpn 10 10.1.1.1
Traceroute 10.1.1.1 in VPN 10
traceroute to 10.1.1.1 (10.1.1.1), 30 hops max, 60 byte packets
1 10.10.1.5 (10.10.1.5) 31.16 ms 41.46 ms 41.46 ms --> vEdge-2
2 10.10.1.6 (10.10.1.6) 42.91 ms * * --> The site-local router

Key Takeaways
vSmart only advertises the best routes to a destination according to the OMP best-path algorithm.
The default behavior of vSmart hides reachability information from remote peers, similarly to a BGP route-reflector.
In scenarios where there isn't full IP reachability between TLOCs, and there are remote sites with limited overlay, this
might create blackholes in failure scenarios.
The OMP send-back-paths option tells the vSmart controller to advertise the first set of non-best routes alongside the
best ones.
Notice that the overall number of advertised routes is subject to the OMP send-path-limit parameter.

« Previous Lesson
OMP Send Path Limit
Next Lesson
OMP Source Preference »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

OMP Source Preference

Understanding OMP Routing

OMP TLOC-Action

Chapter Key Takeaways

QUIZ - Control Plane Fundamentals

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / OMP Source Preference

OMP Source Preference

In this lesson, we are going to explore a typical misunderstanding of the OMP Best Path Selection process in scenarios with
multiple vSmart controllers. Along the way, we are going to talk about the order of operation between the OMP Send Path Limit
parameter and the OMP best-path algorithm.

The initial state

We are going to use a topology consisting of six vEdge cloud routers (vEdges 1 through 6) and two vSmart controllers (vSmart-
1 and vSmart-2). By default, when multiple vSmart controllers oversee the overlay fabric, vEdges establish an OMP peering
session with two of them. This is controlled by a system parameter called max-omp-sessions set to two by default. For the
purpose of this demonstration, the max-omp-sessions parameter on each vEdge is set to 1, which means that each vEdge will
only form an OMP peering session with one vSmart controller.

vSmart-1 vSmart-2
1.1.1.30 1.1.1.40

OMP Peering

ring OM
Pee PP
eer
P
OM ing

10.1.1.0/24 via 11 10.1.1.0/24 via 21 10.1.1.0/24 via 31 10.1.1.0/24 via 41

10.1.1.0/24 via 12 10.1.1.0/24 via 22 10.1.1.0/24 via 32 10.1.1.0/24 via 42

51 52 61 62

41 42 vEdge-5 vEdge-6
11 12 21 22 31 32 5.5.5.5 6.6.6.6
vEdge-1 vEdge-2 vEdge-3 vEdge-4
1.1.1.1 2.2.2.2 3.3.3.3 4.4.4.4

Directly Connected 10.1.1.0/24

Figure 1. OMP Peering with two vSmart controllers

As illustrated in figure 1 above, vEdges 1, 2, and 5 have established an OMP session only with vSmart-1(1.1.1.30) and vEdges
3,4 and 6 only with vSmart-2 (1.1.1.40), respectively. We can verify this directly on the vSmart controllers. You can see that
vSmart-1 has OMP peering to vSmart-2 and to vEdges 1, 2 and 5.
 vSmart-1# show omp peers
R -> routes received
I -> routes installed
S -> routes sent

DOMAIN OVERLAY SITE

PEER TYPE ID ID ID STATE UPTIME R/I/S
------------------------------------------------------------------------------------------
1.1.1.40 vsmart 1 1 100 up 0:02:39:30 8/0/8
1.1.1.1 vedge 1 1 1 up 0:01:59:04 4/0/8
2.2.2.2 vedge 1 1 1 up 0:01:51:41 2/0/4
5.5.5.5 vedge 1 1 5 up 0:01:59:07 2/0/8

And vSmart-2 has OMP peering to vSmart-1 and vEdges 3,4, and 6.

vSmart-2# sh omp peer

R -> routes received
I -> routes installed
S -> routes sent

DOMAIN OVERLAY SITE

PEER TYPE ID ID ID STATE UPTIME R/I/S
------------------------------------------------------------------------------------------
1.1.1.30 vsmart 1 1 100 up 0:02:39:51 8/0/8
3.3.3.3 vedge 1 1 2 up 0:01:59:28 4/0/8
4.4.4.4 vedge 1 1 2 up 0:01:59:29 4/0/8
6.6.6.6 vedge 1 1 6 up 0:01:59:28 0/0/4

There are two WAN transports that are not illustrated in the topology - an Internet cloud and an MPLS cloud. Each vEdge has
two transport attachments - one TLOC marked with the biz-internet color and an IP address from the range 39.3.0.0/16
(highlighted in orange) and one TLOC marked with the mpls color and an IP address from the range 10.10.0.0/16 (highlighted
in green).

vEdges 1,2,3, and 4 are directly connected to subnet 10.1.1.0/24 in VPN 100 and advertise the subnet to their respective
controllers. You can see that vEdges 1 and 2 are advertising the 10.1.1.0/24 subnet to vSmart-1 (1.1.1.30)

vEdge-1# show omp routes vpn 100 advertised detail | nomore | b ADVERTISED | i peer\|\ tloc |
peer 1.1.1.30
tloc 1.1.1.1, mpls, ipsec
tloc 1.1.1.1, biz-internet, ipsec

vEdge-2# show omp routes vpn 100 advertised detail | nomore | b ADVERTISED | i peer\|\ tloc |
peer 1.1.1.30
tloc 2.2.2.2, mpls, ipsec
tloc 2.2.2.2, biz-internet, ipsec

And vEdges 3 and 4 are advertising the 10.1.1.0/24 subnet to vSmart-2 (1.1.1.40)

vEdge-3# show omp routes vpn 100 advertised detail | nomore | b ADVERTISED | i peer\|\ tloc |
peer 1.1.1.40
tloc 3.3.3.3, mpls, ipsec
tloc 3.3.3.3, biz-internet, ipsec

vEdge-4# show omp routes vpn 100 advertised detail | nomore | b ADVERTISED | i peer\|\ tloc |
peer 1.1.1.40
tloc 4.4.4.4, mpls, ipsec
tloc 4.4.4.4, biz-internet, ipsec

The question
Knowing that all routes to 10.1.1.0/24 in VPN100 have identical values for AD (250), OMP Preference (0), TLOC preference (0),
Origin-type (Connected), and Origin-Metric (1), there is no policy applied on vSmart, that everything else is by default, could you
guess:

Which OMP routes to subnet 10.1.1.0/24 will vSmart-1 choose as best? What about vSmart-2?
 Which OMP routes to subnet 10.1.1.0/24 will vEdge-5 install in its VPN 100 routing table?
Which OMP routes to subnet 10.1.1.0/24 will vEdge-6 install in its VPN 100 routing table?

The common misunderstanding

In Cisco SD-WAN, vSmart controllers establish a full-mesh of OMP peering sessions between themselves and distribute all
OMP routes they learn to all other controllers. Therefore, in our topology, both controllers will exchange all OMP routes
resulting in both knowing all paths to 10.1.1.0/24 in VPN100. However, the key point here is in what order will each vSmart
controller sort the OMP routes ? We have already said that all OMP routes to 10.1.1.0/24 have identical values for AD
(250), OMP Preference (0), TLOC preference (0), Origin-type (Connected), and Origin-Metric (1). Therefore, all eight routes will
be considered equal-cost on both vSmarts. However, using the tiebreakers on each controller will yield a different result.

From the perspective of vSmart-1 - the controller receives four OMP routes to 10.1.1.0/24 from the direct OMP peering with
vEdges 1 and 2. At the same time, the controller receives another four OMP routes to the same destination from vSmart-2 (the
routes that vEdges 3 and 4 advertise to vSmart-2). Therefore, according to the first tiebreaker of the OMP best-path algorithm,
vSmart-1 will prefer the routes coming directly from vEdges 1 and 2 over the ones coming from vSmart-2.

From the perspective of vSmart-2 - the controller receives four OMP routes to 10.1.1.0/24 from the direct OMP peering with
vEdges 3 and 4. At the same time, vSmart-2 receives another four OMP routes to the same destination from vSmart-1 (the
routes that vEdges 1 and 2 advertise to vSmart-1). According to the first tiebreaker of the OMP best-path algorithm, vSmart-2
will insert the routes via vEdges 3 and 4 at the top and then the ones via vEdges 1 and 2 that it received from vSmart-1. In the
end, both controllers will know about all eight paths to 10.1.1.0/24 but each vSmart controller will insert the routes in its VPN
100 route table in a different order , as illustrated in figure 2 below.

10.1.1.0/24 via T12 10.1.1.0/24 via T32

10.1.1.0/24 via T11 10.1.1.0/24 via T31
10.1.1.0/24 via T22
send-path-limit 4 10.1.1.0/24 via T42
10.1.1.0/24 via T21 10.1.1.0/24 via T41
10.1.1.0/24 via T32 10.1.1.0/24 via T12
10.1.1.0/24 via T31 10.1.1.0/24 via T11
10.1.1.0/24 via T42 OMP Peering 10.1.1.0/24 via T22
10.1.1.0/24 via T41 10.1.1.0/24 via T21

10.1.1.0/24 via 11 10.1.1.0/24 via 21 10.1.1.0/24 via 31 10.1.1.0/24 via 41

10.1.1.0/24 via 12 10.1.1.0/24 via 22 10.1.1.0/24 via 32 10.1.1.0/24 via 42

51 52 61 62

41 42 vEdge-5 vEdge-6
11 12 21 22 31 32 5.5.5.5 6.6.6.6
vEdge-1 vEdge-2 vEdge-3 vEdge-4
1.1.1.1 2.2.2.2 3.3.3.3 4.4.4.4

Directly Connected 10.1.1.0/24

Figure 2. Each vSmart controller sorts the routes in a different order

Well, let's now see how the fact that each vSmart controller sorts the routes in a different order affects routers vEdge-5 and 6 -
Because the OMP send-path-limit parameter is set to four by default, each vSmart controller will only advertise the first four
equal-cost best routes to its OMP peers . In our example, this means that vSmart-1 advertises only the routes via vEdges 1 and
2 to vEdge-5 (5.5.5.5) as you can see in the output below:
 vSmart-1# show omp routes vpn 100 advertised detail | nomore | b ADVERTISED | i peer\|\ tloc |
...
peer 5.5.5.5
tloc 1.1.1.1, mpls, ipsec
tloc 1.1.1.1, biz-internet, ipsec
tloc 2.2.2.2, mpls, ipsec
tloc 2.2.2.2, biz-internet, ipsec

And vSmart-2 advertises only the routes via vEdges 3 and 4 to vEdge-6 (6.6.6.6) as you can see in the output below:

vSmart-2# show omp routes vpn 100 advertised detail | nomore | b ADVERTISED | i peer\|\ tloc |
...
peer 6.6.6.6
tloc 4.4.4.4, mpls, ipsec
tloc 4.4.4.4, biz-internet, ipsec
tloc 3.3.3.3, mpls, ipsec
tloc 3.3.3.3, biz-internet, ipsec

In the end, router 5 won't know that subnet 10.1.1.0/24 is reachable via vEdges 3 and 4 at all, and router 6 won't know that the
subnet is reachable via vEdges 1 and 2. Let's verify this by checking the routing tables of both routers:

vEdge-5# show ip routes vpn 100 | t

ADDRESS PATH NEXTHOP

VPN FAMILY PREFIX ID PROTOCOL TLOC IP COLOR ENCAP VPN STATUS
-----------------------------------------------------------------------------------------
100 ipv4 10.1.1.0/24 0 omp 1.1.1.1 mpls ipsec - F,S
100 ipv4 10.1.1.0/24 1 omp 1.1.1.1 biz-internet ipsec - F,S
100 ipv4 10.1.1.0/24 2 omp 2.2.2.2 mpls ipsec - F,S
100 ipv4 10.1.1.0/24 3 omp 2.2.2.2 biz-internet ipsec - F,S

You can see that vEdge5 has installed the routes via vEdges 1 and 2.

vEdge-6# sh ip route vpn 100 | t

ADDRESS PATH NEXTHOP

VPN FAMILY PREFIX ID PROTOCOL TLOC IP COLOR ENCAP VPN STATUS
-----------------------------------------------------------------------------------------
100 ipv4 10.1.1.0/24 0 omp 3.3.3.3 mpls ipsec - F,S
100 ipv4 10.1.1.0/24 1 omp 3.3.3.3 biz-internet ipsec - F,S
100 ipv4 10.1.1.0/24 2 omp 4.4.4.4 mpls ipsec - F,S
100 ipv4 10.1.1.0/24 3 omp 4.4.4.4 biz-internet ipsec - F,S

And that vEdge6 has installed the routes via vEdges 3 and 4. Of course, we must have in mind that the maximum number of
equal-cost routes that a WAN edge router would install in its routing table is controlled by the OMP ecmp-limit parameter.

The key takeaways

In this lesson, we have seen how the first tie-breaker in the OMP Best-Path algorithm works and how it interacts with the OMP
send-path-limit parameter. The question is, what does all that mean for a real-world deployment?

The default value set by Cisco for every parameter is not random. They know that 99% of the network environments would have
two vSmart controllers per region and four equal-cost routes per destination because the typical branch network is dual-
homed, dual-transport. However, in a network region with more than two vSmart controllers, vEdges may use suboptimal paths
to destinations if either one of the following design recommendations is not met:

Each vEdge has an OMP peering session with every vSmart controller in the region. The number of OMP sessions that a
vEdge router could have is controlled by a global system parameter called max-omp-sessions, set to 2 by default;
The OMP send-path-limit and the OMP ecmp-limit parameters are set according to the maximum number of equal-cost
paths that exist in the region (both values are four by default);
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Understanding OMP Routing

Understanding OMP Routing

This lesson will demonstrate the major differences between the Overlay Management Protocol (OMP) and traditional routing
protocols.

OMP Routing Overview

Let's summarize what we have learned so far about the Cisco OMP protocol:

WAN edge routers do not exchange control-plane information among themselves.

WAN edge routers send routing information only to the vSmart controller(s).
An OMP route for a given prefix points to a TLOC next hop. For example, 10.1.1.0/24 via TLOC (1.1.1.1, mpls, ipsec).
For an OMP route to be valid and installed in the routing table:
The next-hop TLOC must be resolved. For example, the router must have a TLOC route for TLOC (1.1.1.1, mpls,
ipsec).
There must be an overlay tunnel in UP state to the next-hop TLOC. For example, the router must have an active BFD
session to TLOC (1.1.1.1, mpls, ipsec).
The OMP route must be considered best.

All these statements sound very logical and easy to comprehend. However, there is a significant difference in the way OMP
operates compared to a traditional routing protocol such as EIGRP and OSPF. We will use a simple topology shown in figure 1
to demonstrate these differences.

OMP Routing in Failure Scenarios

We will examine two different failure scenarios - direct failure in the overlay and indirect failure in the overlay.

Initial Topology
The topology that we will use for this lesson consists of four WAN edge routers connected to a single transport. vEdges 1 and
2 are located at the same site, so there is no tunnel between them. Everything else is by default. There is no policy applied on
vSmart.

vSmart
No policy
10.1.3.0/24 10.1.4.0/24
vpn1 s vpn1
nel
n
l Tu
vEdge-3 ro vEdge-4
3.3.3.3 ont 4.4.4.4
C
site-id 3 site-id 4

T3 T4

ls
Tunne
c
IPse
T1 T2
vEdge-1 vEdge-2
1.1.1.1 2.2.2.2
site-id 1 site-id 1

vpn1 vpn1
10.1.1.0/24
 Figure 1. Initial Topology

EIGRP in failure scenarios

To really understand the difference between OMP and traditional routing protocols and what they mean in practice - let's
suppose we are running EIGRP as a routing protocol over the IPsec tunnels, as shown in figure 2 below.

10.1.3.0/24 10.1.4.0/24
VPN1 Routing Table Suppose all routers run VPN1 Routing Table
10.1.1.0/24 via IP1 vEdge-3 EIGRP (or any other vEdge-4 10.1.1.0/24 via IP1
10.1.1.0/24 via IP2
3.3.3.3 traditional routing 4.4.4.4
10.1.1.0/24 via IP2
site-id 3 site-id 4
10.1.4.0/24 via IP4
protocol) instead of OMP 10.1.3.0/24 via IP3
IP3 IP4
10.1.3.0/24 connected 10.1.4.0/24 connected

els
c Tunn
IPse
VPN1 Routing Table IP1 IP2 VPN1 Routing Table
10.1.3.0/24 via IP3 vEdge-1 vEdge-2 10.1.3.0/24 via IP3
1.1.1.1 2.2.2.2
10.1.4.0/24 via IP4 site-id 1 site-id 1 10.1.4.0/24 via IP4
10.1.1.0/24 connected 10.1.1.0/24 connected
10.1.1.0/24

Figure 2. Initial Topology with EIGRP routing

With EIGRP, the next hop of a route is an IP address.

Let's focus on routers vEdge-3 and vEdge-4. vEdge-3's best route to reach 10.1.4.0/24 is through the direct tunnel IP3 - IP4.
What will happen if this tunnel goes down? Will vEdge-3 lose connectivity to 10.1.4.0/24?

When the tunnel between vEdge-3 and vEdge-4 goes down, the EIGRP protocol will run the EIGRP Query Process and will ask all
other routers whether someone has a network path to 10.1.4.0/24. Then based on the Query replies, vEdge-3 will install new
routes to 10.1.4.0/24 in its routing table and will still have IP reachability to vEdge-4 prefix through vEdge-1/2. Figure 3
illustrates this example.

10.1.3.0/24 Tunnel between vEdges 10.1.4.0/24

3 and 4 goes down
EIGRP runs the Query Process
VPN1 Routing Table New routes are installed in the VPN1 Routing Table
vEdge-3 routing tables vEdge-4
10.1.1.0/24 via IP1 10.1.1.0/24 via IP1
3.3.3.3 4.4.4.4
10.1.1.0/24 via IP2 site-id 3 site-id 4 10.1.1.0/24 via IP2
10.1.4.0/24 via IP1 10.1.3.0/24 via IP1
IP3 IP4
10.1.4.0/24 via IP2 10.1.3.0/24 via IP2
10.1.3.0/24 connected 10.1.4.0/24 connected

fic
ta traf
Da
VPN1 Routing Table IP1 IP2 VPN1 Routing Table
10.1.3.0/24 via IP3 vEdge-1 vEdge-2 10.1.3.0/24 via IP3
1.1.1.1 2.2.2.2
10.1.4.0/24 via IP4 site-id 1 site-id 1 10.1.4.0/24 via IP4
10.1.1.0/24 connected 10.1.1.0/24 connected
10.1.1.0/24

Figure 3. EIGRP rerouting around failures

This is a very simplified description of the EIGRP convergence process. However, the main point is that - traditional routing
protocols recompute the routing topology on every path failure . The topology changes and the routes in the routing table
change according to the available network paths.

OMP in failure scenarios

Let's now examine two failure scenarios, but with the Cisco Overlay Management Protocol (OMP) running between vEdges and
vSmart.

Direct Failure
Every WAN edge router advertises its locally connected networks to the vSmart controller via OMP. The controller then
redistributes this reachability information to all vEdges. Next-hop of OMP routes is a TLOC. Figure 4 shows the routing tables
of all routers.

VPN1 Routing Table

10.1.1.0/24 via T1
10.1.1.0/24 via T2
10.1.3.0/24 via T3
10.1.4.0/24 via T4

vSmart
No policy
10.1.3.0/24 10.1.4.0/24
vpn1 s vpn1
VPN1 Routing Table nel VPN1 Routing Table
n
l Tu
vEdge-3 vEdge-4
10.1.1.0/24 via T1
ntro 10.1.1.0/24 via T1
3.3.3.3 Co 4.4.4.4
10.1.1.0/24 via T2 site-id 3 site-id 4 10.1.1.0/24 via T2
10.1.4.0/24 via T4 10.1.3.0/24 via T3
T3 T4
10.1.3.0/24 connected 10.1.4.0/24 connected

ls
Tunne
c
IPse
VPN1 Routing Table T1 T2 VPN1 Routing Table
10.1.3.0/24 via T3 vEdge-1 vEdge-2 10.1.3.0/24 via T3
1.1.1.1 2.2.2.2
10.1.4.0/24 via T4 site-id 1 site-id 1 10.1.4.0/24 via T4
10.1.1.0/24 connected 10.1.1.0/24 connected
vpn1 vpn1
10.1.1.0/24

Figure 4. Initial Topology with OMP routing

Let's focus again on vEdge-3 and vEdge-4. vEdge-3's best route to reach vEdge-4's local prefix 10.1.4.0/24 is over the direct
tunnel between T3 and T4. What will happen this time if this tunnel goes down? Will vEdge-3 lose connectivity to 10.1.4.0/24?

Let's find out by bringing the tunnel down.

vEdge-3# show bfd sessions | t

SYSTEM SITE DETECT TX

IP ID LOCAL COLOR COLOR STATE MULTIPLIER INTERVAL UPTIME
----------------------------------------------------------------------------------
1.1.1.1 1 biz-internet biz-internet up 7 1000 0:00:24:15
2.2.2.2 1 biz-internet biz-internet up 7 1000 0:00:24:31
4.4.4.4 4 biz-internet biz-internet down 7 1000 NA

Now if we try to ping between vEdge-3 and vEdge-4, we will see that there is no connectivity.
 vEdge-3# ping vpn 1 10.1.4.1
Ping in VPN 1
PING 10.1.4.1 (10.1.4.1) 56(84) bytes of data.
From 127.1.0.2 icmp_seq=1 Destination Net Unreachable
From 127.1.0.2 icmp_seq=2 Destination Net Unreachable
From 127.1.0.2 icmp_seq=3 Destination Net Unreachable
^C
--- 10.1.4.1 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss

Figure 5 below explains what actually happens.

VPN1 Routing Table

10.1.1.0/24 via T1
10.1.1.0/24 via T2
10.1.3.0/24 via T3
vEdge-3 doesn’t have a valid vEdge-4 doesn’t have a valid
route to vEdge-4. 10.1.4.0/24 via T4 route to vEdge-3.
No data-plane between 10.1.3.0 No data-plane between 10.1.4.0
and 10.1.4.0. and 10.1.3.0.

No policy
10.1.3.0/24 10.1.4.0/24
vpn1 s vpn1
VPN1 Routing Table nel VPN1 Routing Table
Tun
vEdge-3 l vEdge-4
10.1.1.0/24 via T1 tro 10.1.1.0/24 via T1
3.3.3.3 n 4.4.4.4
10.1.1.0/24 via T2 Co 10.1.1.0/24 via T2
site-id 3 site-id 4
10.1.4.0/24 via T4 Tunnel is 10.1.3.0/24 via T3
T3 T4
10.1.3.0/24 connected DOWN 10.1.4.0/24 connected

unnels
cT
IPse
VPN1 Routing Table T1 T2 VPN1 Routing Table
10.1.3.0/24 via T3 vEdge-1 vEdge-2 10.1.3.0/24 via T3
1.1.1.1 2.2.2.2
10.1.4.0/24 via T4 site-id 1 site-id 1 10.1.4.0/24 via T4
10.1.1.0/24 connected 10.1.1.0/24 connected
vpn1 vpn1
10.1.1.0/24

Figure 5. OMP routing and Overlay failures

When the tunnel T3- T4 goes down, vEdge-3 no longer has BFD session in UP state to 10.1.4.0/24's next-hop T4. Therefore,
vEdge-3 marks the route "10.1.4.0 via T4" as Invalid and takes it out of the VPN routing table, as you can see in the output
below.
 vEdge-3# show omp routes vpn 1 | t
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved

PATH ATTRIBUTE
VPN PREFIX FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP
--------------------------------------------------------------------------------------------------
1 10.1.1.0/24 1.1.1.30 5 1017 C,I,R installed 1.1.1.1 biz-internet ipsec
1.1.1.30 8 1007 C,I,R installed 2.2.2.2 biz-internet ipsec
1 10.1.3.0/24 0.0.0.0 68 1013 C,Red,R installed 3.3.3.3 biz-internet ipsec
1 10.1.4.0/24 1.1.1.30 1 1020 Inv,U installed 4.4.4.4 biz-internet ipsec

The same process happens on vEdge-4. In the end, both routers lose connectivity to each other's connected networks.

Indirect Failure on an Intermediate Hop

Suppose we have applied a control policy that changes the next hop of vEdge-3's route to 10.1.4.0/24 to go through vEdge-1
and the next hop of vEdge-4's route to 10.1.3.0/24 to go through vEdge-2. Basically, vEdges 1 and 2 will be intermediate hops
for the traffic between vEdges 3 and 4. Figure 6 shows this visually.

vSmart
Control-Policy
10.1.3.0/24 10.1.4.0/24
vpn1 s vpn1
VPN1 Routing Table nel VPN1 Routing Table
n
l Tu
10.1.1.0/24 via T1 vEdge-3 ro vEdge-4 10.1.1.0/24 via T1
3.3.3.3 ont 4.4.4.4
10.1.1.0/24 via T2 C 10.1.1.0/24 via T2
site-id 3 site-id 4
10.1.4.0/24 via T1 10.1.3.0/24 via T2
T3 T4
10.1.3.0/24 connected 10.1.4.0/24 connected

ls
Tunne
c
IPse
VPN1 Routing Table T1 T2 VPN1 Routing Table
10.1.3.0/24 via T3 vEdge-1 vEdge-2 10.1.3.0/24 via T3
1.1.1.1 2.2.2.2
10.1.4.0/24 via T4 site-id 1 site-id 1 10.1.4.0/24 via T4
10.1.1.0/24 connected 10.1.1.0/24 connected
vpn1 vpn1
10.1.1.0/24

Figure 6. Intermediate Hops between vEdges 3 and 4

In this case, what do you think will happen if we shut down the tunnel T1 - T4?

When the tunnel T1 - T4 goes down, vEdge-3 still has a valid route to 10.1.4.0/24 with next-hop T1 and it sends the traffic to
vEdge-1.
 vEdge-3 does have a valid
route to vEdge-4 via T1. vSmart
Control-Policy
10.1.3.0/24 10.1.4.0/24
vpn1 ls vpn1
VPN1 Routing Table nne VPN1 Routing Table
l Tu
vEdge-3 vEdge-4
10.1.1.0/24 via T1
ntro 10.1.1.0/24 via T1
3.3.3.3 Co 4.4.4.4
10.1.1.0/24 via T2 site-id 3 site-id 4 10.1.1.0/24 via T2
10.1.4.0/24 via T1 10.1.3.0/24 via T2
T3 T4
10.1.3.0/24 connected 10.1.4.0/24 connected

nel
Tun N
VPN1 Routing Table T1 DOW T2 VPN1 Routing Table
10.1.3.0/24 via T3 vEdge-1 vEdge-2 10.1.3.0/24 via T3
1.1.1.1 2.2.2.2
10.1.4.0/24 via T4 site-id 1 site-id 1 10.1.4.0/24 via T4
10.1.1.0/24 connected 10.1.1.0/24 connected
vpn1 vpn1
10.1.1.0/24
vEdge-1 doesn’t have a valid
route to vEdge-4. Traffic is
dropped.

Figure 7. OMP behavior in indirect failure

However, vEdge-1 now doesn't have an overlay tunnel to vEdge-4, so the route to 10.1.4.0/24 is no longer valid. The traffic is
dropped!

Key Takeaways
This example highlights some important differences between the traditional routing protocols and the Cisco Overlay
Management Protocol (OMP):

With OMP, there is no auto-discovery of routing peers and next-hops as in traditional routing protocols such as EIGRP
and OSPF. All nodes peer with vSmart and not between each other.
Subsequently, the routing topology never changes (because the topology is an overlay).
There is no topology recalculation when a tunnel goes down.
The OMP routes to a destination prefix never change when a tunnel goes down. They go in and out of the VPN routing
table depending on the BFD session to the next-hop tloc.
What actually changes is only the reachability to the next hops.

Okay, what does that mean in practice?

Practically speaking, this means two things:

We provide resilience in case of direct overlay failures with multiple tunnels to the same destination. For example, if we
have two tunnels between vEdge-3 and vEdge-4, when one tunnel goes down, the traffic will simply go through the other.
We must be careful when we introduce intermediate hops in the overlay traffic path. We provide resiliency in case of
indirect overlay failures with a technique called end-to-end path tracking (that we will explore in the next lesson).

« Previous Lesson
OMP Source Preference
Next Lesson
OMP TLOC-Action »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / OMP TLOC-Action

OMP TLOC-Action
In the last lesson, we saw the differences between traditional routing protocols and overlay routing with OMP. We have seen
that OMP does not reroute around failures in the overlay by default. This lesson will demonstrate a Cisco SD-WAN feature that
protects against direct and indirect overlay failures using a control-policy option called tloc-action.

Failures in the overlay

Before we begin, let's again see how OMP handles failures. If you haven't checked our previous lesson on OMP routing, it is
good to do so before continuing with this one.

Direct Failures
Figure 1 shows a basic overlay topology with three vEdges connected to a single transport. Direct failure in the topology leads
to a network outage, even though an alternative overlay path exists. For example, if tunnel T2-T3 goes down, branches lose
connectivity to each other, even though an alternative overlay path via the data center is available.

Direct Failure – Branch-2 can’t reach

Default Operation (no control-policy) Branch-3, even though an alternative
path exists via T1 (vEdge-1)

VPN Routing Table VPN Routing Table

Data
DC-1 Branch-2 via T2 Branch-2 via T2
Center
Branch-3 via T3 Branch-3 via T3

T1 T1

Branch-2 Branch-3 Branch-2 Branch-3

WAN WAN
T2 T3 T2 T3

VPN Routing Table VPN Routing Table VPN Routing Table VPN Routing Table
DC-1 via T1 DC-1 via T1 DC-1 via T1 DC-1 via T1
Branch-3 via T3 Branch-2 via T2 Branch-3 via T3 Branch-2 via T2

Figure 1. Direct Failure

This happens because overlay routing is different from traditional underlay routing:

There is no auto-discovery of routing peers and next-hops like in traditional routing protocols. All nodes peer only with
the vSmart controller and not between each other.
Subsequently, the routing topology never changes (because it is an overlay).
There is no topology recalculation when there is a failure in the overlay.
What actually changes is only the reachability to the next-hop tlocs (tracked with BFD).

Indirect Failures
Figure 2 shows an example where traffic is directed (via control policy) to an intermediate hop, which then forwards the traffic
to its final destination. In this case, an indirect failure leads to traffic blackholes. For example, if the tunnel T1-T3 goes down,
there is no way for Branch-2 to know that the data center doesn't have an overlay path to Branch-3, so it still sends the traffic
destined to Branch-3 to the DC, and the traffic gets dropped.
 Changed next-hop (topology is changed Indirect Failure – Branch-2 can’t reach
Branch-3, even though direct overlay
to hub-and-spoke with a control-policy) path exists via T3

VPN Routing Table VPN Routing Table

Data
DC-1 Branch-2 via T2 Branch-2 via T2
Center
Branch-3 via T3 Branch-3 via T3

T1 T1

Branch-2 Branch-3 Branch-2 Branch-3

WAN WAN
T2 T3 T2 T3

VPN Routing Table VPN Routing Table VPN Routing Table VPN Routing Table
DC-1 via T1 DC-1 via T1 DC-1 via T1 DC-1 via T1
Branch-3 via T1 Branch-2 via T1 Branch-3 via T1 Branch-2 via T1

Figure 2. Indirect Failure

You can clearly see the difference between overlay routing and the well-known routing protocols such as EIGRP and OSPF.
Although the topology is a triangle and an alternative path always exists in case of one failure, one tunnel down leads to a
network outage.

What is TLOC-Action?
TLOC Action is a control-policy option that inserts an intermediate hop in an OMP route to a destination prefix. The Smart
controller then tracks the overlay path between the intermediate hop and the ultimate tloc. If that path goes down, the
controller informs the WAN edge routers that received this OMP route.

sequence 11 Control-Policy
match route
site-id 3
action accept
set vSmart
tloc 1.1.1.1 blue ipsec
tloc-action [primary, backup, ecmp]
The policy inserts an
intermediate hop in the
OMP route
OMP Route
vEdge-1
10.1.3.0/24 via T1 1.1.1.1
ultimate-tloc T3 site-id 1 OMP Route
10.1.3.0/24 via T3
Intermediate
T1
Tloc

OMP Route vEdge-2 WAN vEdge-3

10.1.3.0/24 via T1 2.2.2.2
T2 T3 3.3.3.3
ultimate-tloc T3 site-id 2 site-id 3
Ultimate
The router knows that it Tloc
can reach 10.1.3.0 /24
via T1 and via T3
Source Destination
10.1.2.0/24 10.1.3.0/24
Figure 3. What is TLOC Action?

Figure 3 shows an example of how tloc-action works.

1. vEdge-3 advertises its connected network 10.1.3.0/24 with next-hop its local tloc T3. The route is "10.1.3.0/24 via T3".
 2. This route matches a control-policy sequence on vSmart. In the action portion of the sequence, the policy sets a new tloc
and sets one of the available tloc-actions. This new tloc that is inserted by the policy is the intermediate hop.
3. The controller advertises the OMP route to vEdge-2. The route has two next-hops now - intermediate tloc (inserted by the
policy) and ultimate tloc (the original route's next-hop). This tells vEdge-2 that the destination prefix 10.1.3.0/24 is
reachable via T1 and via T3.
4. Now it is up to the specified tloc action to define how the traffic to 10.1.3.0 will be forwarded. There are four tloc-actions:
1. primary
2. backup
3. ecmp
4. strict (default)

Now vEdge-2 knows that 10.1.3.0/24 is reachable via T1 and via T3. The router tracks both overlay tunnels T2-T1 and T2-T3
with BFD, but the router doesn't have a way to track the tunnel T1 - T3. However, if that tunnel goes down, vSmart will inform
vEdge-2 about the status change . From the perspective of vEdge-2, this creates a triangle in the overlay and gives a choice - in
normal circumstances, vEdge-2 can send the traffic destined to 10.1.3.0/24 via T1, it can send it via T3, it can send it via both
tlocs or it can send it via none. Well, this is basically what the tloc-action parameter defines - where to send the traffic in normal
circumstances and where to send the traffic in case of a failure. Let's look closely at each tloc-action.

TLOC-Action Primary
The "tloc-action primary" specifies that in normal circumstances, the communication between vEdge-2 and vEdge-3 goes
through the intermediate hop.

Normal Operation (TLOC Action PRIMARY) Failure (fall back to ultimate tloc)

VPN Routing Table VPN Routing Table

Data
DC-1 Branch-2 via T2 Branch-2 via T2
Center
Branch-3 via T3 Branch-3 via T3
Intermediate Intermediate
T1 T1
Tloc Tloc
Branch-2 Branch-3 Branch-2 Branch-3
WAN WAN
T2 T3 T2 T3
Ultimate Ultimate
Tloc Tloc
VPN Routing Table VPN Routing Table
DC-1 via T1 DC-1 via T1
Branch-3 via T1 Branch-3 via T1
ultimate-tloc T3--primary ultimate-tloc T3--primary

Figure 4. TLOC Action Primary

If the overlay tunnel between T1 and T3 goes down, vSmart informs vEdge-2 and the router forwards the traffic directly to T3.

This option is useful when we want to force specific traffic through a data center or a regional hub for security inspection and
at the same time keep the direct overlay path to the destination as an alternative.

TLOC-Action Backup
The "tloc-action backup" specifies that in normal circumstances, the communication between vEdge-2 and vEdge-3 goes
directly to the ultimate-tloc (the original route's next-hop).
 Normal Operation (TLOC Action BACKUP) Failure (fall back to intermediate tloc)

VPN Routing Table VPN Routing Table

Data
DC-1 Branch-2 via T2 Branch-2 via T2
Center
Branch-3 via T3 Branch-3 via T3
Intermediate Intermediate
T1 T1
Tloc Tloc
Branch-2 Branch-3 Branch-2 Branch-3
WAN WAN
T2 T3 T2 T3
Ultimate Ultimate
Tloc Tloc
VPN Routing Table VPN Routing Table
DC-1 via T1 DC-1 via T1
Branch-3 via T1 Branch-3 via T1
ultimate-tloc T3--backup ultimate-tloc T3--backup

Figure 5. TLOC Action Backup

If the overlay tunnel between T2 and T3 goes down, vEdge-2 forwards the traffic through the intermediate hop T1.

This option is useful when we want to force specific traffic through the direct overlay path to a destination and at the same
time have an alternative path via intermediate hop.

TLOC-Action ECMP
The "tloc-action ecmp" specifies that in normal circumstances, the communication between vEdge-2 and vEdge-3 is load-
balanced through the intermediate hop T1 and the direct next-hop T3.

Normal Operation (TLOC Action ECMP) Failure (failed path is just not used)

VPN Routing Table VPN Routing Table

Data
DC-1 Branch-2 via T2 Branch-2 via T2
Center
Branch-3 via T3 Branch-3 via T3
Intermediate Intermediate
T1 T1
Tloc Tloc
Branch-2 Branch-3 Branch-2 Branch-3
WAN WAN
T2 T3 T2 T3
Ultimate Ultimate
Tloc Tloc
VPN Routing Table VPN Routing Table
DC-1 via T1 DC-1 via T1
Branch-3 via T1 Branch-3 via T1
ultimate-tloc T3--ecmp ultimate-tloc T3--ecmp

TLOC Action ECMP

If the overlay tunnel between T1 and T3 goes down, vSmart informs vEdge-1 to stop sending traffic via the intermediate router.

If the direct tunnel T2-T3 goes down, vEdge-2 stops forwarding traffic over that overlay path.

TLOC-Action Strict (Default Option)

The "tloc-action strict" specifies that in normal circumstances, the communication between vEdge-2 and vEdge-3 goes through
T1, the intermediate hop.
 Normal Operation (TLOC Action STRICT) Failure (no traffic reaches final destination)

VPN Routing Table VPN Routing Table

Data
DC-1 Branch-2 via T2 Branch-2 via T2
Center
Branch-3 via T3 Branch-3 via T3
Intermediate Intermediate
T1 T1
Tloc Tloc
Branch-2 Branch-3 Branch-2 Branch-3
WAN WAN
T2 T3 T2 T3
Ultimate Ultimate
Tloc Tloc
VPN Routing Table VPN Routing Table
DC-1 via T1 DC-1 via T1
Branch-3 via T1 Branch-3 via T1
ultimate-tloc T3--strict ultimate-tloc T3--strict

TLOC Action Strict

If the overlay tunnel between T1 and T3 goes down, vEdge-2 drops the traffic.

This option is useful in cases where we want to ensure that traffic either gets inspected at the intermediate hop or gets
dropped but does not reach the final destination uninspected.

TLOC-Action Example
The output below shows vEdge-2's OMP route to 10.1.3.0/24 with tloc-action primary applied. The tloc highlighted with orange
is the intermediate hop, and the tloc highlighted with green is the original route's next hop.

---------------------------------------------------
omp route entries for vpn 1 route 10.1.3.0/24
---------------------------------------------------
RECEIVED FROM:
peer 1.1.1.30
path-id 17
label 1004
status C,I,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
originator 3.3.3.3
type installed
tloc 1.1.1.1, blue, ipsec
ultimate-tloc 3.3.3.3, blue, ipsec -- primary
domain-id not set
overlay-id 1
site-id 2
preference not set
tag not set
origin-proto connected
origin-metric 0
as-path not set
community not set
unknown-attr-len not set

Notice the following restrictions:

The intermediate tloc and the ultimate tloc must be the same color for the tloc-action to work.
The intermediate router must be enabled for service TE.
Key Takeaways
TLOC Action is a control-policy option that inserts an intermediate hop in an OMP route to a destination prefix.
The tloc-action parameter specifies how to forward traffic to the destination prefix via an intermediate hop.
There are four different tloc actions:
Primary
In normal operation, forwards the traffic via the intermediate tloc.
In case of failure, forwards the traffic via the ultimate tloc.
Backup
In normal operation, forwards the traffic via the ultimate tloc.
In case of failure, forwards the traffic via the intermediate tloc.
ECMP
In normal operation, load-balance the traffic via both overlay paths.
In case of failure on one path, forward the traffic via the other.
Strict (default option)
In normal operation, forwards the traffic via the intermediate tloc.
In case of failure, drop the traffic.

For further reading on this topic, check out our configuration examples on TLOC Action:

End-to-End Path Tracking

Preferred DC in a dual-DC design

« Previous Lesson
Understanding OMP Routing
Next Lesson
Chapter Key Takeaways »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

OMP Source Preference

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Chapter Key Takeaways

Chapter Key Takeaways

In this lesson, we will be looking at the key points in this section that you should learn and understand before continuing further
with the course. If you don't feel comfortable with any of the topics, go back and reread the lessons in the chapter.

The underlay network provides reachability between TLOCs using traditional routing mechanisms.
Multiple default routes can exist in the transport VPN 0 on a vEdge router. Which one will be used at any given moment
depends on the overlay routing.
A vEdge router establishes the following overlay connections:
One transient DTLS control connection to the vBond orchestrator over each connected WAN transports only during
the onboarding process.
One persistent DTLS control connection to vManage over a single WAN transport.
One persistent DTLS/TLS control connections to vSmart over each connected WAN transports;
IPsec tunnels to all known remote TLOCs with different site-ids over each available WAN transports. A BFD session
is automatically started over each IPsec tunnel and can not be disabled.
The concept of TLOCs and colors does not apply to controllers. An SD-WAN controller may only have one routing
interface that terminates DTLS connections.
The Overlay Management Protocol (OMP) governs the routing among vEdges.
The OMP best-path algorithm selects the best routes and sorts them in descending order (from best to worst).
The vSmart controller inserts and keeps all routes in separate VPN tables with the best routes at the top.
vRoutes:
Each vroute is associated with a VPN segment;
The next-hop attribute of a vroute is not an IP address but a TLOC route;
If the next-hop TLOC route of a vRoute is not known, the vroute is marked as Invalid.
The site-id is a loop prevention mechanism similar to the AS number in BGP.
TLOC routes:
TLOC routes are not associated with a VPN.
A TLOC route is uniquely identified by {System-IP, Color, Encapsulation}. Notice that the fixed system-IP address
instead of the interface IP. This ensures that a TLOC route can be identified at any given moment irrespective of any
interface changes.
Service routes:
vEdges advertise attached network services using OMP Service Routes.
The vSmart controllers do not re-advertise a service route.
The service route is used in centralized policies for service chaining.
The send-path-limit parameter defines the maximum number of best-paths that an SD-WAN device advertises to its OMP
peers.
The ecmp-limit parameter defines the maximum number of best paths that a vEdge router installs in its routing tables.
The controller-send-path-limit defines the maximum number of best-paths that a vSmart controller advertises to another
vSmart controller.

« Previous Lesson
OMP TLOC-Action
Next Lesson
QUIZ - Control Plane Fundamentals »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Cisco SD-WAN Data Plane

Cisco SD-WAN Data Plane

We have seen that the Cisco SD-WAN solution is broken up into four separate planes (data, control, management, and
orchestration) that play different roles and have different responsibilities in the architecture.

In this section, we will start exploring the unique functions of each network plane by deep-diving into the data plane. First, we
will explore how the Cisco SD-WAN solution builds and maintains the overlay fabric. Along the way, we will introduce multiple
new concepts such as TLOCs, TLOC Colors, and Tunnel Groups. Later in the chapter, we will see how the SD-WAN fabric
interacts with the different types of network address translation (NAT) in the underlay. We will also explore the process of
securing the data plane with IPsec and the process of key exchange.

In the configuration portion, we will look at the following lab examples that will emphasize our understanding of the data plane:

Lab#1: Connecting to the WAN using TLOC Extension

Lab#2: Controlling the topology with Restricted TLOC Colors
Lab#3: Controlling the topology with Tunnel Groups
Lab#4: Connecting to the WAN using a Loopback TLOC

At the end of the chapter, we will reiterate the essential concepts that we have seen and emphasize the key points that
everybody should takeaway.

« Previous Lesson
QUIZ - Control Plane Fundamentals
Next Lesson
What is a TLOC? »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / What is a TLOC?

What is a TLOC?
A TLOC is a Transport Locator that represents an attachment point where a Cisco WAN Edge device connects to a WAN
transport. A TLOC is uniquely identified by a tuple of three values - (System-IP address, Color, Encapsulation).

System-IP: The System-IP is the unique identifier of the WAN edge device across the SD-WAN fabric. It is similar to the
Router-ID in traditional routing protocols such as BGP. It does not need to be routable or reachable across the fabric.
Transport Color: The color is an abstraction used to identify different WAN transports such as MPLS, Internet, LTE, 5G,
etc. In scenarios where transport types are duplicated (for example two different Internet providers) and should be treated
differently from each other, the colors could be arbitrary, such as Green, Blue, Silver, Gold, etc.
Encapsulation Type: This value specifies the type of encapsulation this TLOC uses - IPsec or GRE. To successfully form a
data plane tunnel to another TLOC, both sides must use the same encryption type.

TLOC {1.1.1.1, green, ipsec} vSmart

Private IP: 10.1.2.3 9.9.9.9
Private Port: 12346 Site-id 30
Public IP: 31.1.2.3
Public Port: 12346
Preference: 0
Site-id: 10
Tag: not set
Weight: 1
TLOC Route OMP
Update

NAT
10.1.2.3 31.1.2.3
150.2.2.2

Internet
Ge0/0 Ge0/1

vEdge-1 1.1.1.1 vEdge-2

1.1.1.1 T1 green 2.2.2.2
Site-id 10 ipsec Site-id 20
Figure 1. Cisco SD-WAN Forming an IPsec tunnel - Step 1

Let's stop here and look at the example shown in figure 1. WAN edge router 1 has an interface Ge0/0 with IP address 10.1.2.3
that connects to the Internet. As we all know, this IP is private and not routable over the Internet, so on the provider side, this
private address is translated to the public address 31.1.2.3 through one-to-one NAT. On the other side, vEdge-2 has interface
Ge0/1 with a publicly routable IP address 150.2.2.2.

Now let's say that we are tasked to manually configure an overlay tunnel between WAN edge 1 and WAN edge 2. If the devices
are regular Cisco IOS routers and we are configuring a GRE tunnel with IPsec on top, we are going to specify a tunnel interface
with tunnel-source IP and tunnel-destination IP on both ends taking the NAT into account. Also, we are going to specify the
IPsec authentication and encryption parameters, the IPsec session keys, tunnel keys, and many additional settings depending
on the situation. Now imagine that vEdge-1 and vEdge-2 must establish the same tunnel but in an automatic fashion and on-
demand. Well, they will need to exchange all these parameters with each other. That is what the concept of Transport Locators
(TLOCs) is created for.
A TLOC route consists of all required information needed by a remote peer in order to establish an overlay tunnel with that
TLOC. This includes private and public IP addresses and ports, site-id, preference, weight, status, encapsulation info such as
encryption and authentication parameters, and much more. Note that the three-tuple uniquely identifies a particular TLOC but
the TLOC route itself consists of much more info. For example, the vEdge-1 connection to the Internet is uniquely identified by
the TLOC (1.1.1.1, green, ipsec), but the corresponding TLOC route that vEdge-1 advertises to the SD-WAN controller consists
of the IP address of interface Ge0/0, the public IP address after the NAT, all IPsec parameters and many more values required
by vEdge-2 in order to establish an IPsec tunnel to vEdge-1.

TLOC {2.2.2.2, green, ipsec}

Private IP: 150.2.2.2 vSmart
Private Port: 12346
Public IP: 150.2.2.2
9.9.9.9
Public Port: 12346 Site-id 30
Preference: 0
Site-id: 20
Tag: not set TLOC Route
Weight: 1

OMP
Update
NAT
10.1.2.3 31.1.2.3
150.2.2.2

Internet
Ge0/0 Ge0/1

vEdge-1 2.2.2.2 vEdge-2

1.1.1.1 green T2 2.2.2.2
Site-id 10 ipsec Site-id 20
Figure 2. Cisco SD-WAN Forming an IPsec tunnel - Step 2

Having all we have said about TLOCs in mind, let's now follow the process of forming an overlay tunnel between WAN edge-1
and WAN edge-2. When vEdge-1 is first connected to the Internet via interface Ge0/0, it gets an IP address/Mask/Default
Gateway via DHCP and it gets the transport color and encapsulation type from the controller or from a manual configuration. It
then tries to establish all control plane connections to vBond/vManage/vSmart and in the process, it detects its publicly
routable NATed address. Once it is able to communicate to the control plane via this interface, it creates a unique TLOC
identified by the system-IP 1.1.1.1 (not the interface IP), the WAN transport color - green, and the encapsulation type - IPSec.
This TLOC is then advertised to vSmart along with many other attributes that you can see in the output below.
 vEdge-1# show omp tlocs detail
---------------------------------------------------
tloc entries for 1.1.1.1
green
ipsec
---------------------------------------------------
ADVERTISED TO:
peer 9.9.9.9
Attributes:
encap-key not set
encap-proto 0
encap-spi 256
encap-auth sha1-hmac,ah-sha1-hmac
encap-encrypt aes256
public-ip 31.1.2.3
public-port 12346
private-ip 10.1.2.3
private-port 12346
public-ip ::
public-port 0
private-ip ::
private-port 0
domain-id not set
site-id 10
overlay-id not set
preference 0
tag not set
stale not set
weight 1
version 2
gen-id 0x80000011
carrier default
restrict 0
groups ( 0 )
border not set
unknown-attr-len not set
preference 0
tag not set
stale not set
weight 1
version 2
gen-id 0x80000015
carrier default
restrict 0
groups ( 0 )
border not set
unknown-attr-len not set

Once vSmart receives this TLOC route via OMP, it redistributes it to WAN edge router 2. The same process happens on the
other side where vEdge-2 advertises its WAN connection to the Internet as TLOC (2.2.2.2, green, ipsec). In the end, both WAN
edge routers have all the required information in order to form an overlay tunnel between each other.

NAT
31.1.2.3
10.1.2.3 150.2.2.2

Internet
Ge0/0 Ge0/1

vEdge-1 vEdge-2
1.1.1.1 T1 T2 2.2.2.2
Site-id 10 IPsec tunnel and a BFD Session Site-id 20
Figure 3. Cisco SD-WAN Forming an IPsec tunnel - Step 3
Now with the TLOC routes redistributed to both WAN Edge routers, they can attempt to form an overlay tunnel using IPsec and
to form a BFD session with each other.

vEdge-1 vEdge-2
1.1.1.1 T1 T2 2.2.2.2
Site-id 10 Site-id 20
IPsec tunnel and a BFD Session
Figure 4. Cisco SD-WAN Forming an IPsec tunnel - Step 4

Multiple TLOCs
At this point, it is very likely that you might have thought - "Ah I see, a TLOC is just a tunnel endpoint and one TLOC is equal to
one tunnel". If we have a WAN edge router that has two connections to two different WAN providers, for example, two Internet
circuits, it would have two TLOCs. One that represents Internet-1 and one that represents Internet-2 as it is visualized on the
example shown in figure 5. Well, this is the point where it gets a little bit more complicated.

By default, WAN Edge routers try to form an overlay tunnel to every TLOC over each available WAN transport , including TLOCs
that belong to other colors if there is IP reachability between the two transport networks. In the example shown in figure 5,
there is any-to-any reachability between the two clouds because they are basically two Internet providers. In this scenario, four
IPsec tunnels will form T1-T3, T1-T4, T2-T3, T2-T4. There is one important exception, WAN edge devices do not form overlay
tunnels to other devices in the same site ie having the same site-id.

vSmart
9.9.9.9
TLOC routes Site-id 30 TLOC routes
advertised via OMP advertised via OMP

NAT
31.1.2.3
Internet-1
T1
Ge0/0 T3
Ge0/1
10.1.2.3 150.2.2.2
NAT
Ge0/1 78.5.13.9 Ge0/2
T2 T4
vEdge-1 172.16.2.3 150.2.2.2
1.1.1.1 Internet-2 vEdge-2
Site-id 10 2.2.2.2
Site-id 20

T1 IPsec T3 T2 IPsec T3

T1 IPsec T4 T2 IPsec T4

Figure 5. Forming the overlay with over TLOCs

This is where the Cisco SD-WAN overlay magic happens. If you advertise all TLOCs to all WAN edge routers, a full-mesh overlay
fabric will be formed (assuming there is IP reachability between the underlay transports). However, if you want to have a
custom overlay topology, you just control which particular TLOCs are advertised to which particular WAN edge routers. The
logic is as follows - if vEdge-X receives TLOCs Y and Z, it will attempt to form IPsec tunnels from each own TLOC. If vEdge-X
does not receive TLOCs Y and Z, it won't ever attempt to establish overlay tunnels to these TLOCs.

Overlay routing and TLOCs

Ok, but what does that mean for the overlay routing? Transport Locators are also a key part of the overlay OMP routing. Each
omp route points to a TLOC next-hop and not to the particular overlay tunnel used to get there. By doing that, Cisco SD-WAN
abstracts the overlay routing away from underlay routing, and WAN edge devices do not have to keep any underlay information
in the overlay routing tables. For an OMP route to be considered as valid, the vEdge device must have at least one IPsec tunnel
with a BFD session in UP state to the next-hop TLOC.
 vEdge-1# show omp routes
---------------------------------------------------
omp route entries for vpn 1 route 172.16.50.0/24
---------------------------------------------------
RECEIVED FROM:
peer 9.9.9.9
path-id 10
label 1
status C,I,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
originator 2.2.2.2
type installed
tloc 2.2.2.2, green, ipsec
ultimate-tloc not set
domain-id not set
overlay-id 1
site-id 20
preference not set
tag not set
origin-proto connected
origin-metric 0
as-path not set
unknown-attr-len not set

« Previous Lesson
Cisco SD-WAN Data Plane
Next Lesson
The Overlay Fabric »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / The Overlay Fabric

The Overlay Fabric

One of the main advantages of Cisco SD-WAN is that it works over any mix of WAN transports in an automated, secure, and
flexible manner. The solution builds a fabric of overlay tunnels, either IPsec or GRE, between all WAN edge routers over any
available transport. The process of establishing the fabric is fully automated , and the traffic that goes through these tunnels is
encapsulated, encrypted, and authenticated.

T1 T5
T2
MPLS T6

T3 T7
IPsec tunnels
T4 T8

Overlay

Underlay

Campus MPLS Branch

NAT
INET
NAT

Datacenter Branch

Figure 1. Cisco SD-WAN fabric of overlay tunnels

The solution is essentially just that - a mesh of IPsec/GRE tunnels automatically built over any transport underlay, as shown in
the figure above. In a simplified language, we can summarize each SD-WAN capability as a function of the overlay fabric of
tunnels:

SD-WAN routing is simply choosing an outgoing overlay tunnel for a destination.

Controlling the topology is simply specifying between which sites vEdge routers can establish overlay tunnels.
Application-aware routing (AAR) is just choosing the best overlay tunnel for an application based on pre-defined SLA.
Public cloud integration is simply building IPsec tunnels to vEdges hosted in the cloud provider.
Middle-mile optimization is simply building overlay tunnels to a Colocation or Software-defined Cloud Interconnect
(SDCI) provider.
Cloud-delivered security is simply establishing overlay tunnels to a SASE provider such as Cisco Umbrella.
Securing the forwarding plane is simply encrypting and authenticating the data traffic with the overlay tunnels using
IPsec.

You can see that everything in the Cisco SD-WAN solution revolves around the overlay fabric that is established between the
WAN edge devices. Therefore, it is essential to really understand how vEdges form and maintain the fabric and how we can
manipulate and customize the process. So let’s deep-dive into the data plane.
The Default Fabric Behaviour
Suppose we have a WAN edge router with two connections to two different WAN providers - Internet and MPLS. The router will
have two TLOCs - one that represents the Internet link and one that represents the MPLS circuit. By default, the SD-WAN
solution behaves like the following (visualized in the figure below):

vSmart inserts the

TLOCs in its TLOC table

TLOC table
vSmart
Controller
T1, T2, T3, T4,
T5, T6, ...

vSmart advertises all

OMP Update TLOCs to all vEdges.
OMP Update T1, T2, T3, T4, T5, T6 ...
TLOCs T1, T2 In the end, all vEdges
know about all TLOCs
TLOC table

MPLS T1, T2, T3, T4, T5, T6, ...

T

vEdge-1 T1
Site-id 1 T2
T TLOC table
T T1, T2, T3, T4, T5, T6, ...

INET T
TLOC table
T1, T2, T3, T4, T5, T6, ...

Figure 2. Default SD-WAN behaviour

A vEdge router advertises its TLOCs to the vSmart controller.

The vSmart controller stores all received TLOCs in a TLOC table. If no policy is applied, the controller advertises all TLOCs
to all vEdge routers.
Every vEdge router attempts to form an overlay tunnel with every TLOC in its TLOC table over each connected WAN
transport. The only exception is that a vEdge does not try to create a tunnel with TLOCs that have the same site-id or the
same system-IP.

Let’s look at the example shown in figure 2 above. If vEdge-1 receives two TLOC routes (let’s name them T3 and T4) from
vSmart, it immediately attempts to form a tunnel to each of the received TLOCs (T3 and T4) over each of its own TLOCs (T1
and T2). Therefore, this will result in the following tunnels:

1. T1< -- >T3
2. T2< -- >T3
3. T1< -- >T4
4. T2< -- >T4

You can see that one TLOC, for example, T1, does not represent one tunnel - a TLOC is a tunnel endpoint for all remote TLOCs .
The default behavior results in a full mesh of IPsec tunnels between all vEdge routers with different site-ids. Let's visualize that
with the example shown in the figure below:
 Underlay Transport Overlay Fabric
vEdge-2 vEdge-3 vEdge-2 vEdge-3
Site-id 2 Site-id 3 Site-id 2 Site-id 3

full-mesh of
T2 T3 overlay T2 T3
tunnels

vEdge-1 vEdge-4 vEdge-1 vEdge-4

Site-id 1 Site-id 4 Site-id 1 Site-id 4

T1 T4
T1
Internet T4

T6 T5 T6 T5

vEdge-6 vEdge-5 vEdge-6 vEdge-5

Site-id 6 Site-id 5 Site-id 6 Site-id 5

Figure 3. Full-mesh overlay fabric over one transport

Suppose we have six WAN edge routers connected to a single WAN transport, as shown on the left side of figure 3. Each vEdge
will advertise its TLOCs to the vSmart controller. The controller will then know about all six TLOCs (T1 through T6) and
advertise all TLOCs to all vEdges. In the end, all vEdges will know about all TLOCs. Once a vEdge receives a TLOC route, it
automatically attempts to form a tunnel to that TLOC over each of its own TLOCs (over each connected WAN transport).
Therefore, as shown on the right side of figure 3, the vEdges will build a full-mesh overlay fabric of n*(n-1)/2 IPsec tunnels.
Notice that each TLOC is an endpoint for all remote TLOCs.

In a real-world deployment, it is more likely to encounter a design where each vEdge router is connected to at least two WAN
transports for resiliency, as shown in the figure below, for example.
 Underlay Overlay Fabric
(no connectivity btw clouds) (partial mesh)
vEdge1 vEdge2 vEdge1
Site-id 1 Site-id 2 Site-id 1
T11 T12

T11 T12 T21 T22

T21 T41

INET MPLS T22 T42

vEdge2
Site-id 2 vEdge4
Site-id 4

T31 T32 T41 T42

Combinations
T31 T32
T IPsec T

vEdge3 vEdge4 vEdge3

Site-id 3 Site-id 4 Site-id 3 T IPsec T

Figure 4. Partial-mesh overlay fabric over two transports

When we have a design with multiple WAN transport clouds, such as a mix of Internet and MPLS circuits, one of the primary
considerations is whether there is IP reachability between the WAN clouds. Figure 4 shows an example where there isn’t IP
connectivity between the MPLS cloud and the Internet. In this case, again, all vEdges advertise their TLOCs to vSmart. The
controller redistributes them to all other vEdges resulting in all routers knowing all TLOCs. Then each vEdge router attempts to
form a tunnel to all known TLOCs over each connected WAN transports. For example, vEdge-1 will attempt to form a tunnel to
every TLOC (T21, T22, T31, T32, T41, T42) over its orange TLOC T11 and over its green TLOC T12. However, because there is no
IP reachability between the WAN clouds when a vEdge tries to connect to an MPLS tunnel endpoint (green TLOC) from its
Internet tunnel endpoint (orange TLOC), it won’t be able to form a tunnel because there is no IP connectivity between these two
TLOCs (orange TLOC can’t reach green TLOC). Therefore, this underlay will result in a mesh of tunnels between the Internet
TLOCs and another mesh of tunnels between the MPLS TLOCs as illustrated on the right side.

The figure below visualizes the same example, with the only difference being that there is IP connectivity between the Internet
and the MPLS clouds. In this case, once all vEdges learn all TLOCs, they will be able to form a tunnel to each TLOC (green or
orange) over every WAN transport. In the end, this will result in a full mesh of overlay tunnels between all routers over every
transport network.
 Underlay Overlay Fabric
(full IP reachability) (full-mesh of overlay tunnels)

vEdge1 vEdge2 vEdge1

Site-id 1 Site-id 2 Site-id 1
T11 T12

T11 T12 T21 T22

T21 T41

INET MPLS T22 T42

vEdge2 vEdge4
Site-id 2 Site-id 4

T31 T32 T41 T42 Combinations

T IPsec T
T31 T32
T IPsec T

vEdge3 vEdge4 vEdge3

Site-id 3 Site-id 4 Site-id 3 T IPsec T

Figure 5. Full-mesh overlay fabric over two transports

Before we move ahead and introduce a few new concepts, let’s stop here and summarize what we have learned so far:

A Transport Locator (TLOC) is a unique identifier of a WAN transport interface that serves as a tunnel endpoint to all
remote TLOCs.
A TLOC is a tunnel endpoint for all remote TLOCs that have IP reachability to that TLOC.
All vEdges advertise their TLOCs to the vSmart controller in the form of TLOC routes.
A TLOC route includes all information that a remote vEdge needs to form a tunnel to that TLOC.
By default, with no policy applied, the vSmart controller advertises all TLOCs to all vEdges.
A vEdge attempts to form an overlay tunnel to all TLOCs it knows about over each of its own TLOCs (over each WAN
transport).
If there is full IP reachability in the underlay, the default behavior results in a full-mesh of overlay tunnels between all
vEdges at different locations (having different site-ids).

In a real-world deployment with a mix of public and private transports such as Internet and MPLS, the WAN clouds most likely
would not be inter-connected. Therefore, MPLS TLOCs would not have IP reachability to Internet TLOCs and vice versa.
However, vEdges will still constantly attempt to form a tunnel to all TLOCs over all available transports, unnecessarily wasting
bandwidth and resources on all devices. Even if there is an interconnection between the two WAN clouds, we would want to
have a mechanism that controls the establishment of tunnels between WAN transports. Additionally, public WAN transports,
such as the Internet, utilizes NAT when packets enter the cloud. In contrast, private WAN transports such as MPLS use NAT
when packets leave the MPLS cloud toward other public clouds. Therefore, vEdges need a mechanism that tells them when to
use the remote TLOC's public or private IP address when attempting to form a tunnel.

Cisco SD-WAN introduced a new innovative concept of TLOC Colors that solves these two inefficiencies and provides network
administrators with more granular control over the tunnel establishment process .

« Previous Lesson
What is a TLOC?
Next Lesson
TLOC Color and Carrier »
1. Introduction to SD-WAN
Why do we need SD-WAN?
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / TLOC Color and Carrier

TLOC Color and Carrier

What is TLOC Color?
TLOC Color is a logical abstraction used to identify specific WAN transport that connects to a WAN Edge device. The color is a
statically defined keyword that distinguishes a particular WAN transport as either public or private and is globally
significant across the Cisco SD-WAN fabric. If you think about it, there is no automatic way for a vEdge router to understand
which interface is connected to which transport cloud. If we look at the example in figure 1, vEdge-1 has three interfaces
connected to three different providers. From the perspective of vEdge-1, the only way to distinguish which interface is
connected to which cloud is through the concept of colors that would be externally defined by the controller or locally via CLI.

WAN
lic
Ge0/0
Pub
INET
Color: “biz-internet”
vEdge-1 Private WAN
Ge0/1
1.1.1.1
Site-id 1
Color: “mpls”
Ge0/2
MPLS
Color: “lte” Pub
lic W
AN

LTE
Figure 1. What is TLOC Color?

The TLOC color is configured per interface under the transport vpn0/ interface / tunnel-interface settings as in is shown below:

vpn 0
interface ge0/0
ip address 10.1.1.43/24
tunnel-interface
encapsulation ipsec
color mpls

As of now, there are 22 pre-defined color keywords that are summarized in Table 1 below. They are divided into two main
categories - public and private colors. The public colors are designed to distinguish connections to public networks such as the
Internet where typically the attachment interface has an RFC1918 address that is later translated to a publicly routable address
via NAT. On the other hand, private colors are intended for use on connections to clouds where NAT is not utilized. On WAN
Edge routers, each Transport Locator is associated with a private-public IP address pair. The TLOC color dictates whether the
private or public IP address will be used when attempting to form a data plane tunnel to a remote TLOC.

Public Colors Private Colors

public-internet mpls

biz-internet metro-ethernet

3g private1
 Public Colors Private Colors

lte private2

blue private3

green private4

red private5

bronze private6

silver

gold

custom1

custom2

custom3

Table 1. Cisco SD-WAN TLOC Colors

Communication Between Colors

During the authentication process with the vBond orchestrator, WAN edge devices learn whether they sit behind a NAT
device and what is their NATed address and port. This is done using the STUN protocol and the process is explained in further
detail in our lesson about TLOCs and NAT. In the end, each TLOC contains a pair of private/public addresses and ports. If there
is no NAT, both the private and public addresses are the same, if there is a NAT device along the path, the private address
represents the native interface IP and the public address represents the post-NAT address. When two Cisco SD-WAN devices
attempt to form an overlay tunnel between each other, they look at the colors at both ends in order to decide which IP address
to use.

If the TLOC color at both ends is a Public one, the WAN edge devices attempt to form the data plane tunnel using their public IP
addresses.

Public Color < -- > Public Color

Private Public Public Private

IP IP IP IP
NAT
Internet NAT

vpn 0 vpn 0
interface ge0/0
tunnel-interface
IPsec
IPsec
interface ge0/0
tunnel-interface
color biz-internet color biz-internet
carrier default carrier default

Figure 2. Overlay tunnel between public colors

Even if only one of the colors is public, the WAN edge devices will also attempt to form the data plane tunnel using the public IP
addresses.
 Public Color < -- > Private Color

Private Public Public Private

IP IP IP IP
NAT
Internet NAT

vpn 0 vpn 0
interface ge0/0
tunnel-interface
IPsec
IPsec
interface ge0/0
tunnel-interface
color biz-internet color mpls
carrier default carrier default

Figure 3. Overlay tunnel between public and private colors

However, If the TLOC color at both ends is a Private one, the WAN edge devices attempt to form the data plane tunnel using
their private IP addresses.

Private Color < -- > Private Color

Private Public Public Private

IP IP IP IP
NAT
Internet NAT

IPsec
vpn 0 vpn 0
interface ge0/0 interface ge0/0
tunnel-interface tunnel-interface
color mpls color mpls
carrier default carrier default

Figure 4. Overlay tunnel between private colors

This is particularly important in cases where WAN edge devices communicate directly with their native address over a private
cloud such as MPLS but at the same time, they access the control plane through the same cloud via Network Address
Translation. That is why data plane tunnels between TLOCs marked with private colors are formed using the private IP
addresses as is demonstrated in figure 5.
 vManage vSmart vBond vEdge-2

Control Plane
Private IP

Public
IP

Internet MPLS
The private IP is NATed
to a publicly routable IP
before passing through
the Internet Private IP

vEdge-1
Figure 5. Why private clouds use private IPs?

TLOC Carrier
However, specific scenarios might occur where using the public IP addresses between private colors is the desired behavior.
An example would be having two MPLS clouds that are interconnected using NAT. For such cases, there is a particular TLOC
attribute called carrier that changes this behavior - if the carrier setting is the same in the local and remote TLOCs, the WAN
edge device attempts to form a tunnel using the private IP address, and if the carrier setting is different, then the WAN edge
device attempts to form a tunnel using the public IP address. The diagram below visualizes this:
 NAT NAT
IP IP
MPLS MPLS
(Carrier1) (Carrier2)
Overlay tunnels are built
between the NATed IP
Private IP addresses Private IP

vpn 0 vpn 0
interface ge0/0 interface ge0/0
tunnel-interface tunnel-interface
color mpls color mpls
carrier carrier1 carrier carrier2
vEdge-1 vEdge-2
Figure 6. Overlay tunnels when using the Carrier settings

TLOC Color Restrict

By default, WAN edge routers try to form overlay tunnels to every received TLOC from a different site using every available
color. This is usually the desired outcome in scenarios where we have two Internet connections from two different providers.
Although we typically mark them with different colors in order to treat them separately as shown in figure 7, we would like to
have a full mesh of tunnels because there is IP reachability between the clouds.

vSmart
9.9.9.9
TLOC routes Site-id 30 TLOC routes
advertised via OMP advertised via OMP

NAT
31.1.2.3
Internet-1
T1
Ge0/0 T3
Ge0/1
10.1.2.3 150.2.2.2
NAT
Ge0/1 78.5.13.9 Ge0/2
T2 T4
vEdge-1 172.16.2.3 150.2.2.2
1.1.1.1 Internet-2 vEdge-2
Site-id 10 2.2.2.2
Site-id 20

T1 IPsec T3 T2 IPsec T3

T1 IPsec T4 T2 IPsec T4

Figure 7. Default overlay fabric without Restrict keyword

However, this behavior might not be desirable in scenarios where we have one private transport alongside an Internet cloud, as
it could lead to inefficient routing—such as WAN edge routers trying to build tunnels through the MPLS cloud to Internet TLOCs.
Even though the IP reachability between the clouds may exist, the tunnels might be established over paths that are inefficient
or unintended. This behavior can be changed with the restrict keyword or by using tunnel groups.
 vpn 0
interface ge0/0
ip dhcp-client
tunnel-interface
encapsulation ipsec
color mpls restrict

When a TLOC is marked as restricted, a WAN edge route router will attempt to establish a data plane tunnel to a remote TLOC
only via WAN connections marked with the same color . This behavior is demonstrated in figure 8. vEdge-1 will never try to
establish an IPsec tunnel from T1 to T4 because TLOC1 and TLOC4 are not marked with the same color.

NAT
Internet-1
T1
Ge0/0 T3
Ge0/1

Ge0/1 Ge0/2
T2 T4
vEdge-1
1.1.1.1 MPLS vEdge-2
Site-id 10 2.2.2.2
Site-id 20
T1 IPsec T3 T2 IPsec T4

Figure 8. Overlay fabric using Restrict keyword

The restrict attribute can be verified using the following show command
 vEdge-1# show omp tlocs
---------------------------------------------------
tloc entries for 1.1.1.1
mpls
ipsec
---------------------------------------------------
RECEIVED FROM:
peer 0.0.0.0
status C,Red,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
attribute-type installed
encap-key not set
encap-proto 0
encap-spi 256
encap-auth sha1-hmac,ah-sha1-hmac
encap-encrypt aes256
public-ip 60.1.1.1
public-port 12366
private-ip 192.168.1.2
private-port 12366
public-ip ::
public-port 0
private-ip ::
private-port 0
bfd-status up
domain-id not set
site-id 60
overlay-id not set
preference 0
tag not set
stale not set
weight 1
version 2
gen-id 0x80000013
carrier default
restrict 1
groups [ 0 ]
border not set
unknown-attr-len not set

Another option to achieve the same goal of restricting the data plane connectivity between the same colors is by using tunnel
groups. Only tunnels with matching tunnel groups will form a data plane connection (regardless of the color).

vpn 0
interface ge0/0
ip dhcp-client
tunnel-interface
encapsulation ipsec
group 199

« Previous Lesson
The Overlay Fabric
Next Lesson
Tunnel Groups »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Tunnel Groups

Tunnel Groups
TLOC Color vs Tunnel Group
By default in Cisco SD-WAN, vEdges will try to build a full-mesh overlay by establishing tunnels to all other TLOCs, regardless of
their color. This behavior was explained in detail in our lesson for TLOC colors. For scenarios, where a full-mesh is not the
desired overlay topology, there is an option called restrict, that allows only tunnels to TLOCs marked with the same color.
Typically, this feature is configured on transports marked with private colors because a private cloud usually does not have
reachability to public ones such as the Internet. However, the TLOC color-restrict option is not flexible enough because of the
following limitation - only one interface can be marked with a particular color per WAN edge router.

IMPORTANT TOPIC A WAN edge router can't have multiple interfaces marked with the same color, because it breaks the
uniqueness of the TLOC route! A TLOC is uniquely represented by a three-tuple (System-IP, Color, Encap). The system IP
makes the route unique to a particular WAN edge device that has this system-IP address and the color makes the route
unique to a particular interface on this exact WAN edge router.

If we look at the example shown in figure 1, vEdge-2 has one connection to the MPLS cloud that is marked with the mpls color.
Therefore, different private color has to be assigned to the second interface (metro-ethernet).

Use Case 1: Grouping different interfaces to the same transport

If we want to have two overlay tunnels to vEdge-2 over the same MPLS transport, we can not use the restrict option on the
mpls color. But then, if the private colors do not have the restrict option configured, they will try to establish tunnels to all other
public colors that exist.

vEdge-1 metro-ethernet vEdge-2

Tunnel Group 1
mpls
Tun Group 1
MPLS mpls
Tunnel Group 1

Figure 1. Multiple interfaces to the same transport (no restrict option used)

The tunnel-group feature is designed to give more flexibility and granular control over the overlay tunnel establishments
irrespective of the TLOC color. It works by assigning a tunnel group ID under a tunnel. Once the group-ID is configured under
the TLOC, it obeys the following rules:

TLOCs can only establish tunnels with remote TLOCs with the same tunnel-group IDs irrespective of the TLOC color.
TLOCs with any tunnel-group ID will also form tunnels with TLOCs that have no tunnel-group IDs assigned.
If the restrict-option is configured in conjunction with the tunnel-group option, then TLOCs will only form an overlay tunnel
to remote TLOCs having the same tunnel-group ID and TLOC color

So if we go back to the example shown in figure 1, all interfaces attached to the MPLS cloud can be configured with the
same tunnel-group 1 without the restrict feature. In this way, vEdge-1 will form an overlay tunnel to both interfaces of vEdge-2
but at the same time, tunnels to other public colors/tunnel-groups will not be attempted.

Use Case 2: Grouping different colors

Another typical use case that is illustrated in figure 2 is when a remote site (vEdge-3) uses different colors compared to the
rest of the sites. In a typical real-world deployment, you would like to configure the private colors to only form tunnels over the
private MPLS cloud and the same for the public colors of the Internet. This exact setup is only possible using tunnel-groups. By
assigning all private colors to one tunnel-group (for example 2) and assigning all public colors with different tunnel group (for
example 1), we will prevent the forming of overlay tunnels between the public and private transports while still allowing
different private colors to form tunnels (which would not be possible if we use the restrict-option).

vEdge-1 vEdge-2
mpls mpls
Tun Group 2 Tun Group 2

biz-internet MPLS biz-internet

Tun Group 1 Tun Group 1

INET
Combinations Combinations
public-internet metro-ethernet between private
between public
colors Tunnel Group 1 Tunnel Group 2 colors

vEdge-3

Figure 2. Multiple colors combined in two tunnel groups (no restrict option used)

Use Case 3: Grouping different meshes

Another typical use-case would be if we like to achieve groupings of meshed tunnels as it is illustrated in figure 3. All interfaces
in the left tunnel-mesh are configured with group-id 10 and all interfaces in the right tunnel-mesh are assigned a group-id of 20.
However, the key point of this example is that the hub routers don't have tunnel-group IDs configured on their interfaces , so
they will form overlay tunnels with all other tunnel-group IDs.

Hub Site
No Tunnel Group

ISP-1 ISP-2 ISP-3

Tunnel Tunnel
Group 10 Group 20

Figure 3. Grouping different meshes (no restrict option used)

Configuring Tunnel-Groups
Configuring this feature is very straight-forward. You just enter in the tunnel configuration mode of a particular interface and
enter a group value as is shown in the example below.

vpn 0
interface ge0/0
ip address 80.1.1.1/30
ipv6 address 2001:AB44:15F:A332::1/64
tunnel-interface
encapsulation ipsec
group 10
color public-internet
allow-service all
!
no shutdown
!

The tunnel group is advertised as an attribute in the TLOC route. The possible values for tunnel groups are between 0 and
4294967295.

vEdge-5# show omp tlocs

---------------------------------------------------
tloc entries for 80.80.80.80
public-internet
ipsec
---------------------------------------------------
RECEIVED FROM:
peer 0.0.0.0
status C,Red,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
attribute-type installed
encap-key not set
encap-proto 0
encap-spi 258
encap-auth sha1-hmac,ah-sha1-hmac
encap-encrypt aes256
public-ip 80.1.1.1
public-port 12346
private-ip 80.1.1.1
private-port 12346
public-ip 2001:ab44:15f:a332::1
public-port 12346
private-ip 2001:ab44:15f:a332::1
private-port 12346
bfd-status up
domain-id not set
site-id 80
overlay-id not set
preference 0
tag not set
stale not set
weight 1
version 2
gen-id 0x80000013
carrier default
restrict 0
groups ( 10 )
border not set
unknown-attr-len not set
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / TLOCs and NAT

TLOCs and NAT

NAT Detection
Cisco SD-WAN solution is designed to run over any kind of WAN transport that is available to the WAN edge devices including
all different public networks such as Broadband, 4G/5G, LTE, Business Internet, and so on. This implies that the overlay fabric
should be able to form through all flavors of Network Address Translations that these public networks utilize. In practice, any
Cisco SD-WAN device may be unknowingly sitting behind one or more NAT devices. In order to discover the public IP
addresses/ports allocated by NAT, Cisco SD-WAN devices use the Session Traversal Utilities for NAT (STUN) protocol defined
in RFC5389.

1 SRC-IP DST-IP 2 SRC-IP DST-IP

10.1.1.1 1.2.3.4 53.6.1.1 1.2.3.4
STUN Request STUN Request
„What is my IP“ „What is my IP“ 3
10.1.1.1 NAT
53.6.1.1 1.2.3.4
INET
“My IP address vBond
DST-IP SRC-IP DST-IP SRC-IP
6 is NATed to 10.1.1.1 1.2.3.4 53.6.1.1 1.2.3.4
53.6.1.1” STUN Response 5 STUN Response
4
“Your SRC-IP is 53.6.1.1” “Your SRC-IP is 53.6.1.1”
Figure 1. Session Traversal Utilities for NAT (STUN) Operations

STUN is a client-server protocol that uses a request/response transaction in which a client sends a request to a server, and the
server returns a response. As the request (called STUN Binding Request) passes through a NAT, the NAT will modify the source
IP address/port of the packet. Therefore, the STUN server will receive the request with the public IP address/port created by
the closest NAT device. The STUN server then copies the public address into an XOR-MAPPED- ADDRESS attribute in the STUN
Binding response and sends it back to the client. Going back through the NAT, the public address/port in the IP header will be
un-NATted back to the private ones, but the public address copy in the body of the STUN response will remain untouched. In
this way, the client can learn its IP address allocated by the outermost NAT with respect to the STUN server.

As it is shown in Figure 1, all Cisco SD-WAN devices have an embedded STUN client and the vBond orchestrator acts as a
STUN Server. When the initial control communication to vBond takes place, the SD-WAN device performs the STUN operations
and discovers its public IP address and port. Once determined, this information is then advertised as part of the TLOC routes to
the vSmart controllers and then re-advertised to all other SD-WAN devices.

NAT Types
In a typical production SD-WAN deployment, we would probably have many remote sites connected via many different Internet
connections to a centralized data center or a regional hub. In most regions in the world, Internet providers will always use some
type of private-public address translation due to a shortage of public IPv4 addresses. Let's look at the NAT classifications
according to the STUN protocol and how they can affect whether sites can form connections and communicate directly with
each other or not.

Full-Cone NAT
A full-cone is one where all packets from the same internal IP address are mapped to the same NAT IP address. This type of
address translation is also known as One-to-One.

Additionally, external hosts can send packets to the internal host, by sending packets to the mapped NAT IP address.
 Full-Cone NAT Src X:8001 After
Cisco Static 1-to-1 NAT Dst B:80 NAT

Src A:8001 Before t Port 80

ke
Dst B:80 NAT l Pa
c IP-B
itia Port 81
In

NAT
IP-A Port 8001
IP-X
Port 80
Full-Co
ne IP-C
Port 81

Figure 2. Full-Cone NAT

Restricted-Cone NAT
A Restricted-Cone network address translation is also known as Address-Restricted-Cone. It is a network translation
technique where all packets from the same internal IP address are mapped to the same NAT IP address. The difference to a
Full-Cone is that an external host can send packets to the internal host only if the internal host had previously sent a packet to
the IP address of the external destination. It is important to note that once the NAT mapping state is created, the external
destination can communicate back to the internal host on any port.

Src X:8001 After

Restricted-Cone NAT Dst B:80 NAT
Cisco Dynamic NAT

Src A:8001 Before t Port 80

ke
Dst B:80 NAT l Pa
c IP-B
nitia Port 81
I

NAT
IP-A Port 8001
IP-X
Port 80
Restr IP-C
icted
-Con Port 81
e

Figure 3. Restricted-Cone NAT

Port-Restricted-Cone NAT
A Port-Restricted-Cone is similar to the Restricted-Cone address translation, but the restriction includes also port numbers. The
difference is that an external destination can send back packets to the internal host only if the internal host had previously sent
a packet to this destination on this exact port number. In a typical Cisco IOS/IOS-XE or Cisco ASA configuration, this feature is
known as Port Address Translation (PAT).
 Src X:8001 After
Port-Restricted-Cone Dst B:80 NAT
Cisco Dynamic NAT

Src A:8001 Before t Port 80

ke
Dst B:80 NAT l Pa
c IP-B
nitia Port 81
I

NAT
IP-A Port 8001
IP-X
Port 80
Port-R
estrict IP-C
ed-Co Port 81
ne

Figure 4. Port-Restricted-Cone NAT

Symmetric
Symmetric NAT is also known as Port Address Translation (PAT) and is the most restrictive of all other types. It is a network
translation technique where all requests from the same internal IP address and port to a specific destination IP address and
port, are mapped to a unique NAT IP address and NAT port. Furthermore, only the external destination that received a packet
can send packets back to the internal host. In a typical Cisco IOS/IOS-XE or Cisco ASA configuration, this feature is known as
Port Address Translation (PAT) with port-randomization.

Src X:31644* After

Symmetric NAT Dst B:80 NAT
Cisco Dynamic PAT

Src A:8001 Before et Port 80

k
Dst B:80 NAT l Pa
c IP-B
itia Port 81
In

NAT
IP-A Port 8001
IP-X
Port 80
IP-C
Port 81

Figure 5. Symmetric-NAT

Best-Practices
Although Cisco SD-WAN supports several types of Network Address Translations, to create a full mesh overlay fabric, at least
one side of the WAN Edge tunnels is recommended to be able to initiate a connection inbound to the second WAN Edge. This
means that at least one side of the tunnel is recommended to have a public IP address or to be behind a Full-Cone (1-to-1). It is
also strongly recommended to configure full-cone, or one-to-one address translation at the data centers or regional hub sites
so that, regardless of what NAT type is running at the remote sites (restricted-cone, port-restricted cone, or symmetric ), they
can send traffic to the hubs without issues.

vEdge-1 vEdge-2 IPsec tunnel can form GRE tunnel can form

No-NAT (Public IP) No-NAT (Public IP) YES YES

 vEdge-1 vEdge-2 IPsec tunnel can form GRE tunnel can form

No-NAT (Public IP) Symmetric YES NO

Full Cone (One-to-one) Full Cone (One-to-one) YES YES

Full Cone (One-to-one) Restricted-Cone YES NO

Full Cone (One-to-one) Symmetric YES NO

Restricted-Cone Restricted-Cone YES NO

Symmetric Restricted-Cone NO NO

Symmetric Symmetric NO NO

NAT combinations between WAN Edge routers

Symmetric address translation configured at the transport attached to one vEdge requires a full-cone or a public IP on the other
vEdge to establish a direct IPsec tunnel between them. Sites that cannot connect directly should be set up in a hub-and-spoke
topology so they can reach each other through a regional hub site or data center.

IMPORTANT Note that for overlay tunnels configured to use GRE encapsulation instead of IPsec, only public IP
addressing or one-to-one address translation is supported. Any type of Network Address Translation with port
overloading is not supported since GRE packets lack an L4 header.

TLOC Routes
Once every WAN edge router discovers its private-public translated address and port, it advertises them to the vSmart
controller via OMP using the OMP TLOC routes. The vSmart controller then re-advertises this information across the overlay
fabric.

Port-Restricted-Cone Port-Restricted-Cone
NAT NAT

vEdge-1 vEdge-2
60.1.1.1 70.1.1.1

192.168.1.2 INET 172.16.1.2

IPsec tunnel forms

GRE tunnel does not form

Figure 6. An example of two WAN edge router connected through Port-Restricted-Cone

Lastly, let's see an example of two WAN edge devices connected through a Port-Restricted-Cone. As you can verify in the
combination table, they are able to form an IPsec encapsulation tunnel between themselves but if we change the
encapsulation type to GRE - the data plane tunnel does not come up. Let's quickly verify that.

These are both TLOCs when the enc is set to ipsec.

---------------------------------------------------
tloc entries for 60.60.60.60
lte
ipsec
---------------------------------------------------
RECEIVED FROM:
peer 1.1.0.3
status C,I,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
attribute-type installed
encap-key not set
encap-proto 0
encap-spi 256
encap-auth sha1-hmac,ah-sha1-hmac
encap-encrypt aes256
public-ip 60.1.1.1
public-port 12346
private-ip 192.168.1.2
private-port 12346
public-ip ::
public-port 0
private-ip ::
private-port 0
bfd-status up
domain-id not set
site-id 60
overlay-id not set
preference 0
tag not set
stale not set
weight 1
version 2
gen-id 0x80000010
carrier default
restrict 0
groups ( 0 )
border not set
unknown-attr-len not set
 ---------------------------------------------------
tloc entries for 70.70.70.70
lte
ipsec
---------------------------------------------------
RECEIVED FROM:
peer 1.1.0.3
status C,I,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
attribute-type installed
encap-key not set
encap-proto 0
encap-spi 256
encap-auth sha1-hmac,ah-sha1-hmac
encap-encrypt aes256
public-ip 70.1.1.1
public-port 12426
private-ip 172.16.1.2
private-port 12426
public-ip ::
public-port 0
private-ip ::
private-port 0
bfd-status up
domain-id not set
site-id 70
overlay-id not set
preference 0
tag not set
stale not set
weight 1
version 2
gen-id 0x80000014
carrier default
restrict 1
groups ( 0 )
border not set
unknown-attr-len not set

You can clearly see that the BFD session is UP which means that the tunnel is up and running and data plane traffic is able to
go through back and forth. Now let's change the encapsulation type to GRE and see what will happen.

vEdge-3(config)# vpn 0
vEdge-3(config-vpn-0)# interface ge0/0
vEdge-3(config-interface-ge0/0)# tunnel-interface
vEdge-3(config-tunnel-interface)# encapsulation ?
Possible completions:
gre ipsec
vEdge-3(config-tunnel-interface)# encapsulation gre
vEdge-3(config-tunnel-interface)# commit
Commit complete.

vEdge-4(config)# vpn 0
vEdge-4(config-vpn-0)# interface ge0/0
vEdge-4(config-interface-ge0/0)# tunnel-interface
vEdge-4(config-tunnel-interface)# encapsulation gre
vEdge-4(config-tunnel-interface)# commit
Commit complete.

Now if we check the status of the BFD sessions we can clearly see that the GRE tunnel is down.
 vEdge-4# show bfd sessions
SOURCE REMOTE DST PUBLIC
SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP ENCAP UPTIME
------------------------------------------------------------------------------------
50.50.50.50 50 up mpls mpls 10.70.1.1 10.50.1.1 ipsec 0:01:51
60.60.60.60 60 down lte lte 172.16.1.2 60.1.1.1 gre NA

« Previous Lesson
Tunnel Groups
Next Lesson
Data Plane Encryption »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

OMP Source Preference

Understanding OMP Routing

OMP TLOC-Action

Chapter Key Takeaways

QUIZ - Control Plane Fundamentals

4. Cisco SD-WAN Data Plane

What is a TLOC?

The Overlay Fabric

TLOC Color and Carrier

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Data Plane Encryption

Data Plane Encryption

Most overlay solutions these days encrypt and authenticate data plane traffic using IPsec and Cisco SD-WAN is no different.
Although there is one major difference that Cisco SD-WAN utilizes in order to scale better and more efficiently. Most traditional
IPsec environments use Internet Key Exchange (IKE) to handle the key exchange between IPsec peers. However, IKE creates
scalability issues in full-meshed environments with thousands of spokes because each spoke is required to manage n^2 key
exchanges and n-1 different keys.

Cisco SD-WAN was designed to overcome these scaling limitations by not utilizing IKE at all but instead implementing the key
exchange within the control plane as shown in Figure 1. This is possible because vEdges identity is established during the
provisioning process with the vBond orchestrator.

vSmart

Local OMP OMP Local

Encr-Key-1 Update Update Encr-Key-3

En
Encr-Key-2 p ted wi cryp Encr-Key-4
n cry ey-3 th
ke ted
E
hk
MPLS
it y-
w 1

Received T1 T3 Received
Encr-Key-3 T2 T4 Encr-Key-1

Encr-Key-4
vEdge-1 Encr
with ypted
INET r yp
Enc ey-2
ted vEdge-2 Encr-Key-2
key k
-4 with

Figure 1. Data Plane Encryption Overview

The main idea is that WAN edge routers can leverage the existing encrypted control connections to the vSmart controller and
advertise their keys to the controller via OMP. The controller then redistributes them as OMP updates to all other peers so the
exchange is completely done through the SD-WAN control plane.

Let's look at the example shown in Figure 2. vEdge-1 generates an AES-256-bit key for each connected WAN transport. In the
example, there is only one transport so there is only one generated key - encr-key-1 . However, three symmetric keys will be
generated if we have three WAN providers. Once the encr-key-1 is generated, vEdge-1 advertises it in an OMP update to
vSmart, along with the corresponding TLOC T1. This route advertisement is then re-advertised to the rest of the overlay fabric.
vEdge-2 and vEdge-3 will then use this information to build their IPsec tunnels to vEdge-1 and encrypt the data plane traffic
with the received AES-256 key.
 vSmart
controller
En
cr
-K
ey

Enc
-1

1
y-

r-K
Ke
cr-

ey-
En

1
vEdge-1 generates T2
an AES-256 key sec
IP
and advertises it
to vSmart vEdge-2
T1

IP
se
vEdge-1 c
T3 vEdge-3
Overlay fabric
Cisco SD-WAN Key Exchange

Essentially, this keys exchange model removes the burden of individual negotiations between WAN edge devices that using IKE
would have brought. In addition to that, each key lifetime is 24 hours and each WAN edge router will regenerate its keys every
12 hours in order to provide enhanced encryption and authentication. This means that two keys (old and new) are present at
any one time. The renegotiation of keys does not affect existing traffic, as it happens in parallel with the existing ones and the
old key is still held for another 12 hours so any traffic is accepted using either one.

If we summarize everything we have said up to this point - the Cisco SD-WAN solution exchanges keys between WAN Edges
and vSmart controllers and uses symmetric keys in an asymmetric fashion. This means the following:

The same key is used for encryption and decryption of data plane traffic.
WAN edge routers use their remote peer’s key to encrypt the data rather than their own when sending traffic over the
tunnel.

vEdge-1 Traffic encrypted and decrypted with key-2 vEdge-2

T1 IPsec tunnel T2

Traffic encrypted and decrypted with key-1

Generated Locally Generated Locally
Encr-key-1 Encr-key-2
Received from vSmart Received from vSmart
Encr-key-2 Encr-key-1

Figure 3. Traffic encryption with Symmetric Keys

Let's look at the example shown in figure 3 where two WAN edge devices are going to communicate over a secure overlay
tunnel. Encryption and decryption will occur using the following process:

vEdge-1 generates an AES-256 key called encr-key-1 and vEdge-2 generates one called encr-key-2 .
Both routers advertise these via OMP to the controller and it distributes them across the overlay.
When vEdge-1 sends data to vEdge-2, it will encrypt the data using vEdge-2’s key.
When vEdge-2 receives the data, it will use its key for the decryption of that data.
When vEdge-2 sends data to vEdge-1, it will encrypt the data using vEdge-1’s key.
When vEdge-1 receives the data, it will use its key for the decryption of that data.

Additional security with Pairwise

The IPsec Pairwise keys feature provides additional security by ensuring that multiple vEdge devices across the fabric do not
use the same session key for encryption and decryption. The feature functions by generating a pair of IPsec session keys (one
encryption and one decryption key) for each pair of local - remote TLOCs. This is visualized in the example shown in figure 4
where there is a fabric with three WAN edge devices. The security improvement comes from the fact that encryption and
decryption between vEdge-1 and vEdge-2 will use a session key that is unique to that pair of TLOCs. The green tunnel between
WAN Edge 1 and WAN Edge 3 will also use a different session key pair.

Local Received
BA AB

Local Received
AB BA

ey AB
key BA with k
with y pted vEdge-B
ed cr
crypt En
En

vEdge-A Enc Local Received

ryp
ted CA AC
wit
hk
ey
CA

Enc
ryp
ted
wit
Local Received hk
ey
AC
AC CA

vEdge-C
Figure 4. Encryption with Pairwise Keys

Therefore, the process of encryption and decryption of data when using the IPsec Pairwise keys feature will be as follow:

Each WAN Edge will generate a key for each pair of local-remote TLOC. The session key will then be advertised to the
vSmart via OMP.
The vSmart controller will redistribute the key to the respective peers.
When WAN edge A sends data to WAN edge B, the IPsec session key BA will be used. In the reverse scenario, WAN Edge
B will use the IPsec session key AB.
When vEdge-A sends data to vEdge-C, key CA will be used. In the reverse direction, vEdge-C will send traffic using AC.
Another very important thing to note is that the IPsec Pairwise feature is backward compatible with devices that don’t support
pairwise keys. The feature is disabled by default on the Cisco SD-WAN device and can be enabled via templates.

« Previous Lesson
TLOCs and NAT
Next Lesson
VPN Segmentation »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

OMP Source Preference

Understanding OMP Routing

OMP TLOC-Action

Chapter Key Takeaways

QUIZ - Control Plane Fundamentals

4. Cisco SD-WAN Data Plane

What is a TLOC?

The Overlay Fabric

TLOC Color and Carrier

Tunnel Groups

TLOCs and NAT

Data Plane Encryption

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / VPN Segmentation

VPN Segmentation
Traffic isolation is a key part of the security strategy of any company these days. Network segmentation has many use cases
within the business such as:

A company wants to keep traffic from different business verticals separate.

An organization wants to keep guest users separate from authenticated users.
An enterprise wants to grant access to partners and suppliers only to some portions of the network.
A company needs to enforce regulatory compliance such as ISO27001 or the Payment Card Industry (PCI) security
standards.

In traditional networking, the most rudimentary forms of network segmentation are VLANs (virtual LANs) at layer 2, and VRFs
(virtual routing and forwarding) at layer 3. However, these technologies are limited in scope because they are implemented in
a locally significant fashion (on a single device per interface) or require complex control plane interactions to work (for
example MPLS-based BGP L3VPNs).

Cisco SD-WAN uses a simpler but more scalable approach to network segmentation using two very efficient techniques:

Enforcing segmentation at the edges on WAN edge routers.

Carrying the segmentation information in the data plane packets by inserting VPN labels that identify the network
segment.

Cisco SD-WAN Pre-defined VPNs

By default, the Cisco SD-WAN solution has two pre-defined VPNs that cannot be deleted or modified: Transport VPN 0 and
Management VPN 512.

WAN Edge device

Segmentation Paradigm
Controllers

Connected

Service
Ge0/2 Ge0/0 INET
VPN 5
Dynamic Transport
routing
VPN 0
Service Ge0/1 MPLS
Ge0/3
VPN 10

Management
Management
VPN 512 Network

Eth0

OOB
 Figure 1. Cisco WAN Edge Segmentation

Transport VPN 0
VPN 0 is the pre-defined Transport VPN of the Cisco SD-WAN solution. It cannot be deleted nor modified. The purpose of this
VPN is to enforce a separation between the WAN transport networks (the underlay) and network services (the
overlay). Therefore, all WAN links such as Internet and MPLS circuits are kept in VPN 0 as is visualized in figure 2.

vSmart

INET MPLS

10.1.1.1/30 14.3.2.1/30

Default Default
10.1.1.2/30 Ge0/0 Ge0/1 14.3.2.2/30
Route Route
VPN 0

vEdge-1
System IP: 1.1.1.1
Site-ID: 10

Figure 2. Cisco SD-WAN Transport VPN 0

Additionally, WAN edge routers must have at least one interface configured in VPN0 in order to establish the control plane
tunnels to the vSmart controllers. Each interface that is connected to the WAN must have an IP address, color, and
encapsulation type configured. These parameters are then advertised to the controllers via OMP as part of the TLOC route
advertisements. Typically, a default route is defined via each WAN interface. Note that in Transport VPN0, multiple default
routes can exist. In the underlay routing, the actual default route that is chosen depends on the overlay tunnel that is going to
be used. When the overlay routing decides to use a particular IPsec tunnel, the underlay routing uses the default route that has
a next-hop IP address in the same subnet as the tunnel source IP address.
 !
vpn 0
interface ge0/0
ip address 10.1.1.2/30
tunnel-interface
encapsulation ipsec
color public-internet
!
interface ge0/1
ip address 14.3.2.2/30
tunnel-interface
encapsulation ipsec
color mpls
!
ip route 0.0.0.0/0 10.1.1.1
ip route 0.0.0.0/0 14.3.2.1
!

Let's elaborate on that with the example shown in figure 2. When vEdge-1 forwards packets through IPsec tunnels sourced
from interface ge0/0, in the underlay it will use the default route with next-hop 10.1.1.1, because the next-hop address is in the
same subnet as the IP address of ge0/0 (10.1.1.2/30).

Management VPN 512

By default in Cisco SD-WAN, VPN 512 is configured for out-of-band management. It is enabled and ready-to-go out of the box.

Service VPNs
When we want to create an isolated network domain and isolate the data traffic in this domain from the other user networks
onsite, we create a new service VPN on the vEdge routers. This VPN is specified by a number different from 0 (Transport) and
512 (Management). Once the network segment is created, you associate interfaces, enable routing protocols, and other
network services such as VRRP and QoS within this VPN. One important thing to point out is that interfaces associated with
user segments must not be connected to WAN transports.

VPN VPN
5 5
VPN5
VPN IPsec VPN
10 VPN10 10
tunnel
VPN33
VPN VPN
33 vEdge1 vEdge2 33

Figure 3. Cisco SD-WAN VPNs

Secure Segmentation
Each interface on a Cisco WAN edge device must be configured in a particular VPN. All prefixes learned via interfaces or
routing instances in a VPN are kept in a separate routing table. When this network information is advertised to the vSmart
controller, each prefix is associated with a VPN-ID. In addition, the vSmart controller maintains the VPN context of each prefix.
 vSmart
controller

vEdge3 VPN
5
VPN VPN 0
VPN
5
VPN 0 10
VPN INET
10 VPN
vEdge1 5
VPN 0
VPN
10
vEdge2

IP UDP ESP VPN Data

20 4 36 4 ...

Figure 4. Cisco SD-WAN VPN Lables

This mechanism is very similar to traditional layer 3 isolation with VRFs (Virtual Routing and Forwarding). Therefore, separate
route tables provide network segmentation on a single device. However, the question is how to populate this isolated routing
information across the overlay domain. At the packet forwarding level, WAN edge routers will insert a new VPN label field in
each IP packet. This label will identify the network segments that the packets belong to. The process is visualized in figure 4.
When we configure a new VPN on a WAN edge router, it will associate a label to it. The WAN Edge device will then advertise
this label along with the VPN-ID to the vSmart controller via OMP. The controller itself will then redistribute this VPN-ID
mapping to the other WAN Edges device in the network. The remote WAN Edges in the network will then use these labels to
send traffic for the appropriate VPNs, similarly to the label switching in MPLS.

Arbitrary VPN Topologies

Because the control plane is completely separated from the data forwarding plane, Cisco SD-WAN allows us to define different
topologies per VPN. Typical topology types are full-mesh, partial-mesh, hub-and-spoke, and point-to-point. By default, if a
specific topology is not defined, all VPNs will build a full mesh. Custom segment topologies can be created by using
centralized control policies that filter TLOCs or modify next-hop TLOC attributes for specific OMP routes.
 Full-mesh Hub-and-spoke

VPN 1 VPN 2

Point-to-point
Custom-mesh

VPN 3 VPN 4

Figure 5. Cisco SD-WAN Arbitrary Topologies

At this point, you might be wondering why custom topologies are important. Well, they are important because some
applications may benefit from going via the shortest possible path, e.g VoIP works best in full-mesh topologies. On the other
hand, some segments of the network might benefit from controlled connectivity topology such as hub-and-spoke.

« Previous Lesson
Data Plane Encryption
Next Lesson
Understanding VPNs and LABELs »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Understanding VPNs and LABELs

Understanding VPNs and LABELs

In this lesson, we will explore the concept of labels in Cisco SD-WAN. Labels are an essential part of the solution and are used
for segmentation, route leaking, and service chaining. Most Cisco SD-WAN courses do not focus enough on the function of
labels, and subsequently, many network engineers don't really understand where they fit into the solution.

VPNs and Labels

VPNs divide the Cisco SD-WAN network into different isolated layer 3 segments. Segmentation is an essential networking
function these days as organizations want to keep different business verticals separate for security and compliance reasons.
Additionally, security teams want to keep guests separate from employees separate from contractors. Application teams want
to separate the front-end from the back-end from the database layer, and so on.

What is a VPN?
In the context of Cisco SD-WAN, a VPN is just a separate routing table on a node. All routes belonging to the same VPN are
kept in the same routing table. All other routes, part of another VPN, are kept in another routing table, and so on. This technique
provides the Layer 3 isolation between the VPN segments on a single node.

Okay, an interesting question then arises - if a router has multiple routing tables, how does it know in which routing table to
make an IP lookup for each incoming traffic flow?

Overlay
fabric
On the transport-side, TLOCs
Ge0/0 Ge0/1 aren’t associated with VPNs.
TLOC1 TLOC2 They are always in VPN0.

VPN 20 VPN 30

Routing table Routing table

Ge0/2 Ge0/3 On the service-side, an interface is

VPN 20 VPN 30 always associated with a VPN
in a 1-to-1 fashion.

Figure 1. Cisco SD-WAN VPNs Overview

On the service side of a WAN edge router, this is fairly simple - each service-side interface is associated with a VPN-ID in a one-
to-one fashion. For example, interface ge0/2 is in VPN20, and ge0/3 is in VPN30. This tells the router - "When you receive traffic
on interface ge0/2, make an IP lookup in VPN20's routing table. When you receive routing information from a routing peer on
interface ge0/2, insert it in the VPN20's routing table." The layer 3 separation is pretty straightforward here.

What about the transport side, though? TLOCs aren't associated with VPNs. When a WAN edge router receives data traffic on
an overlay IPsec tunnel, how does it know in which routing table to make an IP lookup? Well, this is where the LABEL comes
into the picture.
What is a label in Cisco SD-WAN?
When a network administrator configures a new VPN ID on a WAN edge router, the router automatically associates the VPN ID
with a LABEL. If we take figure 2 for example, when we configure VPN 50 on vEdge-1, the router automatically maps the VPN-
ID 50 to the next available LABEL number - for example - 1016. Then it advertises this VPN-to-LABEL mapping to vSmart via
OMP. The Cisco vSmart controller maintains all VPN-ID to ROUTER-ID to LABEL mappings of all WAN edge routers.

vSmart maintains the

VPN-ROUTER-LABEL
associations for vEdges

OMP VPN ID ORIGINATOR LABEL OMP

Advertisement 10 1.1.1.1 1014 Advertisement
10 4.4.4.4 1013
30 1.1.1.1 1015
30 4.4.4.4 1014
VPN ID LABEL 50 1.1.1.1 1016 VPN ID LABEL
10 1014 10 1013
50 4.4.4.4 1015
30 1015 30 1014
50 1016 50 1015

Labels are locally significant values.

For example:
- vEdge-1 associates 1014 with VPN10
- vEdge-4 associates 1014 with VPN30
vEdge-1 vEdge-4
1.1.1.1 4.4.4.4
Figure 2. VPN to Label associations

We can check the VPN-to-LABEL mapping on a vEdge router using the show omp services service vpn command, as shown
in the output below. You can see that vEdge-1 has associated VPN 10 with LABEL 1014, VPN 30 with label 1015 and VPN 50
with LABEL 1016.

vEdge-1# show omp services service vpn

ADDRESS PATH
FAMILY VPN SERVICE ORIGINATOR FROM PEER ID LABEL STATUS
--------------------------------------------------------------------------------
ipv4 10 VPN 1.1.1.1 0.0.0.0 66 1014 C,Red,R
0.0.0.0 68 1014 C,Red,R
30 VPN 1.1.1.1 0.0.0.0 66 1015 C,Red,R
0.0.0.0 68 1015 C,Red,R
50 VPN 1.1.1.1 0.0.0.0 66 1016 C,Red,R
0.0.0.0 68 1016 C,Red,R

On the vSmart controller, we check the VPN-to-ROUTER-to-LABEL mapping of all WAN edge routers in the topology. Currently,
for this example, only two vEdge routers are powered on (1.1.1.1 and 4.4.4.4), so you can see their label association on vSmart
using the show omp services service vpn command as shown below.
 vSmart# show omp services service vpn

ADDRESS PATH
FAMILY VPN SERVICE ORIGINATOR FROM PEER ID LABEL STATUS
------------------------------------------------------------------------
ipv4 10 VPN 1.1.1.1 1.1.1.1 66 1014 C,I,R
1.1.1.1 68 1014 C,I,R
10 VPN 4.4.4.4 4.4.4.4 66 1013 C,I,R
4.4.4.4 68 1013 C,I,R
30 VPN 1.1.1.1 1.1.1.1 66 1015 C,I,R
1.1.1.1 68 1015 C,I,R
30 VPN 4.4.4.4 4.4.4.4 66 1014 C,I,R
4.4.4.4 68 1014 C,I,R
50 VPN 1.1.1.1 1.1.1.1 66 1016 C,I,R
1.1.1.1 68 1016 C,I,R
50 VPN 4.4.4.4 4.4.4.4 66 1015 C,I,R
4.4.4.4 68 1015 C,I,R

Notice an important aspect of labels - a label is a locally significant value on an individual WAN edge router. For example,
notice that vEdge-1 associates LABEL 1014 with VPN-10 , and vEdge-4 associates LABEL 1014 with VPN-30 . There is no
global LABEL assignment for the entire network by the centralized control plane. Further down in the lesson, we will talk about
why a label isn't a globally significant value.

Labels in the Control-plane

Once a new VPN is created on a vEdge, the router automatically associates the VPN-ID with a locally significant LABEL. Then
when a service-side interface connects to the site-local network, all routes learned via this interface are placed in the
respective VPN routing table, and subsequently, advertised to vSmart with the LABEL id that represents this VPN on the router.
If we take figure 3 for example, vEdge-1 associates VPN-10 with LABEL-1014. Then all site-local routes that are learned via
interfaces in VPN10 are advertised with LABEL 1014 to vSmart via OMP.

vSmart
vEdges advertises their
site-local routes with the
respective VPN label OMP Update

10.1.10.0/24 via T1, LABEL 1014 10.4.10.0/24 via T4, LABEL 1013
10.1.10.0/24 via T2, LABEL 1014 10.4.10.0/24 via T5, LABEL 1013

T1 T2 T4 T5

VPN ID LABEL VPN ID LABEL

vEdge-1 10 1014 10 1013 vEdge-4
1.1.1.1 30 1015 30 1014 4.4.4.4
50 1016 50 1015
VPN 10 VPN 10

10.1.10.0/24 10.4.10.0/24
Figure 3. Labels in the Control-plane

The CLI output below shows how vEdge-1 advertises the VPN10's prefix 10.1.10.0/24 to the vSmart controller. Notice that it
includes the LABEL-1014 that is associated with the respective VPN-ID 10.
 vEdge-1# show omp routes advertised detail
---------------------------------------------------
omp route entries for vpn 10 route 10.1.10.0/24
---------------------------------------------------
ADVERTISED TO:
peer 1.1.1.30
Attributes:
originator 1.1.1.1
label 1014
path-id 66
tloc 1.1.1.1, mpls, ipsec
ultimate-tloc not set
domain-id not set
site-id 1
overlay-id 1
preference not set
tag not set
origin-proto connected
origin-metric 0
as-path not set
community not set
unknown-attr-len not set
Attributes:
originator 1.1.1.1
label 1014
path-id 68
tloc 1.1.1.1, biz-internet, ipsec
ultimate-tloc not set
domain-id not set
site-id 1
overlay-id 1
preference not set
tag not set
origin-proto connected
origin-metric 0
as-path not set
community not set
unknown-attr-len not set

The label in the vRoute tells the vEdge routers that receive this route - "If you want to send me traffic to VPN 10 prefix
10.1.10.0/24, insert label 1014 in the data packets so that I can demultiplex them to the correct VPN routing table."

Every OMP route (vRoute) in Cisco SD-WAN has a LABEL-ID that points to a next-hop router's VPN table.

Labels in the Data-plane

We have seen that each OMP route (vRoute) in the control plane has a LABEL ID that points to the VPN routing table. When
data traffic matches an OMP route, the router sends it to the next-hop tloc and inserts the label found in the route itself into the
packets.
 The receiving router uses the label
to make an IP lookup in the correct
VPN routing table
VPN ID LABEL VPN ID LABEL
10 1014 LABEL 10 1013
IP UDP ESP 1013 Payload
30 1015 30 1014
50 1016 50 1015

T1 pckt pckt pckt T4

vEdge-1 vEdge-4
1.1.1.1 T2 pckt pckt pckt T5 4.4.4.4

VPN 10 VPN 10
LABEL 10.1.10.0/24 via T2
IP UDP ESP 1014 Payload LABEL 1014
10.1.10.0/24 Traffic matches
OMP route pckt
The receiving router uses the label to
pckt
make an IP lookup in the correct VPN
routing table pckt
Figure 4. Labels in the Data-plane

Figure 4 shows an example of this process. Packets coming in from the service-side of vEdge-4 match an OMP route that
points to vEdge-1's TLOC T2 and label 1014. vEdge-4 sends the data traffic to the next-hop tloc and inserts label 1014.

When vEdge-1 receives the data packets on TLOC T2, it first looks at the label value. The label 1014 tells the router to make an
IP lookup into VPN 10's routing table. Then it matches the traffic to the directly connected subnet 10.1.10.0/24 and switches
the packets in the respective interface.

Notice an important concept - vEdge-4 doesn't know the vEdge-1's VPN-to-LABEL mappings (vpn-label mappings are locally
significant to a single node), it finds the correct label in the OMP route that matches the data packets.

Why aren't labels globally significant?

Engineers often ask why are LABELs locally significant values and why each individual vEdge router keeps independent VPN-to-
LABEL mapping? Why doesn't Cisco SD-WAN just use a globally significant LABAL ID for each VPN? Wouldn't it be easier to
have label 1014 associated with VPN 10 on all vEdges? Then when we see label 1014 we will immediately know that this is
VPN10 traffic, right? Looks pretty logical and straightforward at first, but it isn't.

Having locally significant LABEL-to-VPN mappings on each individual router makes the segmentation process stateless from
vEdges' perspective. Figure 5 shows an example - when we configure a new VPN 60 on vEdge-1, the router associates VPN-60
with LABEL-1060 and advertise it to vSmart via OMP. Since every WAN edge router keeps only its own local VPN-to-LABEL
mapping, all other vEdges don't need to be updated with this new VPN60-to-LABEL1014 mapping on vEdge-1.
 The VPN-to-LABEL
mapping is advertised
to vSmart

vSmart

All other vEdges don’t need to

A new VPN-60 be updated with the new VPN-
associated with
P
OM to-LABEL mapping
LABEL-1060

Horizontal scaling – adding routers/vpns doesn’t affect existing nodes

vEdge-1
1.1.1.1

Figure 5. Why aren't labels globally significant?

Having locally significant LABEL-to-VPN mappings allows the SD-WAN fabric to scale horizontally - adding more WAN edge
routers and more VPNs doesn't affect the existing vEdges.

Labels used for Service Chaining

Labels play another very important role in Cisco SD-WAN. They are used to inject a network service into the path that specific
users/applications take through the network. In this lesson, we will not go into great detail about the whole process of service
chaining but will see how LABELs play a major role in gluing everything together. If you want to learn more about service
chaining, you could check this lesson.

Service Chaining in Control Plane

The process of service chaining in the control plane can be broken down into three steps:

1. vEdge-1 advertises its attached firewall service to vSmart via OMP. The service is associated with locally significant label
1017.
2. vSmart knows that an FW service is available at vEdge-1 with next-hop tlocs T1, T2, and label 1017. The Cisco vSmart
controller uses the FW service in an outbound control policy as a next-hop for routers matched by site-list X.
3. When advertising the OMP routes, the vSmart controller replaces the original next-hop of VPN10's vroutes with tlocs T1,
T2, and label 1017.
 sequence 11
match route Control Policy
vpn-id 10 applied outbound
action accept to site-list X
set service FW

OMP
All routes in VPN10 have
next-hops T1,T2 and
vSmart label 1017
OMP Service Route
FW service associated with label 1014

T1 T2
VPN ID LABEL
10 1014
vEdges
30 1015 vEdges
vEdge-1
50 1016 vEdges
1.1.1.1
SERVICE LABEL
VPN 10 FW 1017 VPN 10

vEdges matched
Attached
FW Service by site-list X

Figure 6. Service Chaining in Control Plane

Now let's see what happens in the data plane when the routers matched by site-list X send data traffic to the destination in
VPN10.

Service Chaining in Data Plane

This step is key to the whole service chaining process - when a WAN edge router receives a packet, the packet's label
determines the lookup context. If the label points to a VPN service, the router performs an IP lookup on the destination IP
found in the header. If the label points to a network service, the router does not do IP lookup but instead directly switches the
traffic to the service IP address .

The router doesn’t make an IP

lookup but directly switches the
traffic to the attached FW service

VPN ID LABEL
10 1014 T1 T4
30 1015 vEdge-1 vEdge-4
50 1016 1.1.1.1 T2 pckt pckt pckt T5 4.4.4.4
SERVICE LABEL
FW 1017 VPN 10 VPN 10
LABEL
Attached IP UDP ESP Payload
1017
FW Service

The label determines the

lookup context to be either
VPN/RIB or VPN/Service

Figure 7. Service Chaining in Data Plane

In the example shown in figure 7 above, the vSmart controller will change the next-hop tlocs and the LABEL of all omp routes
sent to routers matched by site-list X. Subsequently, when the routers send traffic in VPN10 they insert label 1017. When the
traffic reaches vEdge-1 in the data plane, vEdge-1 looks at the incoming label first. The label matches its attached FW service,
and the router switches the traffic directly to the FW service's next-hop address without making an IP lookup in VPN10's routing
table.

Labels used for VPN Traffic Leaking

Labels play another significant role in the process of data traffic leaking between VPNs. Organizations often need to leak traffic
between different VPN segments. A common use case is when an organization wants to provide a shared service to clients
connected to different VPNs while at the same time making sure that the clients cannot access each other.

VPN Traffic Leaking in Control-Plane

Traffic leaking between VPNs can be achieved via a Centralised Control Policy that leaks OMP routes between VPN routing
tables and a Centralised Data Policy that directly forces the data traffic to a next-hop in a different VPN. In this lesson, we will
not get into great detail about the route leaking process but instead, focus on the LABEL's role.

Figure 8 shows a basic example of VPN traffic leaking via Centralised Data Policies on vSmart. There are two data policies as
follows:

The policy on the left side matches the service-side traffic in VPN 10 and forces it to next-hop tloc (4.4.4.4 mpls) in VPN
30.
The policy on the right side matches the service-side traffic in VPN 30 and forces it to next-hop tloc (1.1.1.1 mpls) in VPN
10.

Data Policy Data Policy

applied from-service applied from-service
to site-list X to site-list Y
data-policy VPN10-VPN30 data-policy VPN30-VPN10
vpn-list VPN10 vpn-list VPN30
sequence 1 sequence 1
match match
! !
action accept action accept
set set
vpn 30 vpn 10
tloc 4.4.4.4 color mpls vSmart tloc 1.1.1.1 color mpls

1.1.1.1 4.4.4.4
T1 mpls mpls T4

VPN ID LABEL VPN ID LABEL

vEdge-1 10 1014 10 1013 vEdge-4
1.1.1.1 30 1015 30 1014 4.4.4.4
50 1016 50 1015
VPN 10 VPN 30

vEdges matched vEdges matched

by site-list X by site-list Y
Figure 8. VPN Leaking in Control-Plane

The question here is how this VPN traffic leaking happens in the data plane. Think about this - How does vEdge-4 know the
VPN10's label of vEdge-1 in order to insert the correct label value in packets? vEdge-1's vpn-to-label mappings are only known
to vSmart because they are locally significant.

VPN Leaking in Data-Plane

When vSmart advertises the centralized data policy to vEdge-4, it sends the vpn-label along with the policy config because
there is no way for vEdge-4 to know what is the VPN-to-LABEL mapping of vEdge-1 (all mapping are locally significant and are
only known by vSmart)
 vEdge-4# show policy from-vsmart
from-vsmart data-policy VPN30-VPN10
direction from-service
vpn-list VPN30
sequence 1
action accept
set
vpn 10
vpn-label 1014
tloc 1.1.1.1
tloc color mpls
default-action drop
from-vsmart lists vpn-list VPN30
vpn 30

Now that vEdge-4 knows the correct label, it just inserts the label value in data packets and sends them to the enforced next-
hop tloc, as illustrated in figure 9 before.

VPN ID LABEL VPN ID LABEL

10 1014 10 1013
30 1015 30 1014
50 1016 50 1015

T1 T4
vEdge-1 vEdge-4
1.1.1.1 T2 pckt pckt pckt T5 4.4.4.4

VPN 10 VPN 30
LABEL
IP UDP ESP Payload
1014

The traffic originated in VPN30 is

tagged with vEdge-1's VPN10's label.

Figure 9. VPN Leaking in Data-Plane

Key Takeaways
A label is 20 bits long, giving a range of 0-1048575. Some lower values are reserved and not used.
A label has local significance on a single node.
One WAN edge router maps its VPNs to LABELs independently.
There is no global LABEL assignment for the entire network.
Each WAN edge router advertises its VPN-to-LABEL mappings to vSmart via OMP.
vSmart maintains all VPN-ID to ROUTER-ID to LABEL mappings of all WAN edge routers in the SD-WAN domain.
Every OMP route (vRoute) in Cisco SD-WAN has a LABEL-ID that points to the next-hop router's VPN table.
The label determines the lookup context on the receiving vEdge router to be either VPN/RIB or VPN/Service.

« Previous Lesson
VPN Segmentation
Next Lesson
LAB 1 - Connecting to the WAN using TLOC »
Extension

1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / LAB 1 - Connecting to the WAN using TLOC Extension

LAB 1 - Connecting to the WAN using TLOC

Extension
In real-world deployments, a typical use case is a dual-homed branch with each vEdge router directly connected to a single
WAN transport only. We can see such an example shown in figure 1 below, where vEdge-1 is connected only to the Internet
while vEdge-2 is connected only to the MPLS. Having only one WAN transport per-router seriously reduces the network
resiliency and creates traffic inefficiencies in the overlay fabric. If vEdge-1's ge0/0 interface or link goes down, the router will
lose connectivity to the SD-WAN overlay fabric.

IPsec tunnels IPsec tunnels

over INET over MPLS

INET MPLS

vEdge-1 vEdge-2
1.1.1.1 2.2.2.2

Site-local networks

Figure 1. Dual-home site with each router connected to a single transport

One solution is to connect both routers to both WAN clouds. However, this might not be the most cost-efficient way because of
the contracts with service providers. For example, in many parts of the world, service providers provision and charge per
access port, which means that connecting vEdge-1 to the MPLS cloud would require the organization to pay extra fees for a
second port/circuit.

What is a TLOC Extension?

TLOC extension is a Cisco SD-WAN feature that allows a vEdge router to communicate over the adjacent vEdge's WAN
transports through a TLOC extension interface. Figure 2 shows an example where vEdge-1 is directly connected to the Internet
and uses the feature to connect to the MPLS transport via vEdge-2. In the end, vEdge-1 has overlay tunnels built over both WAN
clouds even though it is directly attached only to one of the WAN transports. On the opposite side, vEdge-2 is attached to the
MPLS cloud and uses the TLOC extension feature to connect to the Internet via vEdge-1.
 IPsec tunnels over INET IPsec tunnels over MPLS

INET MPLS
Transport side
(vpn 0)
vEdge-1 vEdge-2
1.1.1.1 2.2.2.2

Site-local networks
Figure 2. What is TLOC Extension?

The TLOC extension feature is set up per interface and provides transparent connectivity from one interface (called a TLOC
extension interface) to a particular WAN transport. Let's look at vEdge-1, for example. It is unaware that the green tunnel to the
MPLS cloud passes through another vEdge router because vEdge-2 extends their directly connected interface transparently to
the MPLS transport.

TLOC Extension Types

We can extend a vEdge router to the opposite transport in multiple ways. The simplest, most instructive use-case we will see in
this example is directly connecting the vEdge routers using two separate interfaces, as illustrated in figure 2 above. However, if
we want to be a little bit more efficient port-wise, we can use a physical connection between the routers configured as a dot1Q
trunk and then use two separate directly connected VLANs. We can also configure TLOC extension through the site-local LAN,
typically over the distribution switches. In this case, the routers must be L2-adjacent and in the same subnet.

IPsec tunnels IPsec tunnels IPsec tunnels IPsec tunnels IPsec tunnels IPsec tunnels
over INET over MPLS over INET over MPLS over INET over MPLS

INET MPLS INET MPLS INET MPLS

dot1Q

L2 switching L3 routing
Figure 3. TLOC Extension Types

We can go even further and configure TLOC extension over the site-local L3 routing in some exceptional circumstances. In this
case, the vEdge routers are separated by an L3 device and their extension links, which could be dedicated interfaces or dot1q
subinterfaces, are in different subnets. The L3 TLOC-extension is implemented using GRE tunnels and is typically not
recommended unless necessary. Figure 3 illustrates the various deployment types.

Configuring TLOC Extension

We always configure a TLOC extension interface in the transport VPN 0. The configuration is pretty simple - we assign an IP
address and specify the WAN interface to which the TLOC Extension is bound. Let's look at figure 4, for example. To extend
vEdge-2 to the Internet, we connect it to vEdge-1's extension interface ge0/7, which is bound to the WAN interface connected
to the Internet - ge0/0. On the opposite side, to extend vEdge-1 to the MPLS cloud, we link it to vEdge-2's extension interface
ge0/7, which is bound to the MPLS cloud through ge0/1. Then we configure static default routes in VPN0 on each WAN Edge
router, pointing to the adjacent vEdge's extension interface IP as a next-hop, and that's it.

Routing Considerations
You can see that the TLOC extension config is pretty straightforward. However, we must always consider the reverse IP
reachability from the transport networks back to the TLOC extension interfaces. Otherwise, the overlay tunnels and BFD
sessions won't come up.

IPsec tunnels over INET IPsec tunnels over MPLS

INET MPLS
NAT Advertise
Subnet-A Subnet-B
Ge0/0 Ge0/1

Ge0/7 Subnet-A Ge0/0

TLOC
Extension 0.0.0.0/0

TLOC
Ge0/1 Subnet-B Ge0/7 Extension
0.0.0.0/0
vEdge-1 vEdge-2
Figure 4. TLOC Extension using separate directly connected interfaces

For example, to ensure that packets can be routed from the MPLS cloud back to the TLOC extension interface, we must
configure vEdge-2 to advertise subnet-B to the MPLS provider over the PE-CE routing protocol. On the other hand, in a typical
production deployment, subnet-A would always be from the private IP address space. Therefore, we must enable NAT on
vEdge-1's interface to the Internet so that subnet-A can be translated to a publicly routable address before being sent out.
Otherwise, the packets would be dropped at the Internet edge.

Let's now go through a configuration example and see how we implement the TLOC-Extention feature.

Initial State
Before we begin with the actual configuration, let's quickly check the initial state of the network. We have vEdge-1 connected
only to the Internet and vEdge-2 connected to the MPLS, as shown in the following lab diagram. Let's verify that using CLI.
 Controllers 10.1.X.0/24
Site-id 100 VPN-X VPN-X
vEdge-1 vEdge-2
Data Center
1.1.1.1 1.1.1.2
(Hub)
Site-id 1 Site-id 1
aws ge0/0 ge0/1

INET MPLS
NAT
Colors: 39.3.0.0/16 10.10.0.0/16
biz-internet
mpls
ge0/0 ge0/1 ge0/0 ge0/1 ge0/0 ge0/1 ge0/0 ge0/1
vEdge-3 vEdge-4 vEdge-5 vEdge-6
3.3.3.3 4.4.4.4 5.5.5.5 6.6.6.6
Site-id 3 Site-id 4 Site-id 5 Site-id 6
VPN-X VPN-X VPN-X VPN-X

10.3.X.0/24 10.4.X.0/24 10.5.X.0/24 10.6.X.0/24

Figure 5. Lab topology

If we check the interfaces on vEdge-1 using the show interface command, we will see that vEdge-1 has only one TLOC
configured on ge0/0. Notice the column PORT TYPE. If an interface is listed as "transport," that means the interface has a
TLOC configuration. The interfaces listed as port type transport are the local TLOCs of a vEdge router. They are not regular L3
interfaces that we are used to in traditional networking, though. A transport interface acts as a tunnel endpoint and will only
accept data traffic destined for itself coming from remote DTLS/TLS/IPsec peers. It won't work as a transit interface and won't
forward traffic coming from the underlay destined for other destinations, even though the router may have routing entries for
the destination subnet. For example, vEdge-1's ge0/0 interface has an IP address 39.3.0.1/24 and is configured as a local
TLOC. It will only accept traffic from the underlay that is destined to IP address 39.3.0.1.

vEdge-1# sh int | t
#some rows and columns are omitted for clarity
IF IF
AF ADMIN OPER
VPN INTERFACE TYPE IP ADDRESS STATUS STATUS PORT TYPE
--------------------------------------------------------------------
0 ge0/0 ipv4 39.3.0.1/24 Up Up transport
0 ge0/1 ipv4 - Down Down service
0 ge0/6 ipv4 - Down Down service
0 ge0/7 ipv4 - Down Down service
0 system ipv4 1.1.1.1/32 Up Up loopback
2 ge0/2 ipv4 10.1.2.1/24 Up Up service
3 ge0/3 ipv4 10.1.3.1/24 Up Up service
4 ge0/4 ipv4 10.1.4.1/24 Up Up service
5 ge0/5 ipv4 10.1.5.1/24 Up Up service
512 eth0 ipv4 - Down Down service

If we check the running configuration of the transport interface ge0/0 that is connected to the Internet, we can see that it has
an IP address and tunnel-interface configuration. Under the tunnel-interface configuration (highlighted in orange), we configure
the TLOC parameters such as color and encapsulation type. This interface is marked with the "biz-internet" color and
configured with encapsulation type "ipsec." Therefore, this interface is a local tunnel endpoint (TLOC) that can be uniquely
identified with the tuple {1.1.1.1, biz-internet, ipsec}. Notice that the tuple that identifies this TLOC uses the system-ip address
1.1.1.1 and not the interface IP address 39.3.0.1. This is a common mistake that engineers make in the beginning.

vEdge-1# sh run vpn 0 int ge0/0

#some rows and columns are omitted for clarity
vpn 0
interface ge0/0
description WAN-INET
ip address 39.3.0.1/24
!
tunnel-interface
encapsulation ipsec
color biz-internet
allow-service all
!
no shutdown
!

Ok, so vEdge-1 has only one local TLOC on the interface connected to the Internet. The easiest way to check how many overlay
tunnels are established to a local TLOC is by looking at the BFD sessions. Remember that once an overlay tunnel is
established, BFD is automatically started on top and cannot be disabled. Therefore, each BFD session represents one overlay
tunnel.

If we look at the output of show bfd sessions on vEdge-1, we can see that it has tunnels to all other vEdges with the
exception of vEdge-2 because vEdge-2 is at the same location (has the same site-id 1). Notice that all tunnels are established
over the local color "biz-internet". That is because vEdge-1 does not have a connection to other WAN networks and hence does
not have other local TLOCs that can form tunnels.

Also, notice that vEdge-1 has formed overlay tunnels to all colors. Recall that a vEdge attempts to form an overlay tunnel to all
remote TLOCs it has received (from vSmart) over each of its own TLOCs (over each WAN transport).
For example, the highlighted BFD session is to TLOC {3.3.3.3, mpls, ipsec} over the local colors biz-internet. This is the BFD
session that runs on the tunnel between vEdge-1's biz-internet interface and vEdge-3's mpls interface. The second BFD session
is between vEdge-1's biz-internet interface and vEdge-3's biz-internet interface, and so on.

vEdge-1# show bfd sessions | t

#some columns are omitted for clarity

SYSTEM SITE
SRC IP DST IP PROTO IP ID LOCAL COLOR COLOR STATE
-----------------------------------------------------------------------------
39.3.0.1 10.10.0.3 ipsec 3.3.3.3 3 biz-internet mpls up
39.3.0.1 39.3.0.3 ipsec 3.3.3.3 3 biz-internet biz-internet up
39.3.0.1 10.10.0.4 ipsec 4.4.4.4 4 biz-internet mpls up
39.3.0.1 39.3.0.4 ipsec 4.4.4.4 4 biz-internet biz-internet up
39.3.0.1 10.10.0.5 ipsec 5.5.5.5 5 biz-internet mpls up
39.3.0.1 39.3.0.5 ipsec 5.5.5.5 5 biz-internet biz-internet up
39.3.0.1 10.10.0.6 ipsec 6.6.6.6 6 biz-internet mpls up
39.3.0.1 39.3.0.6 ipsec 6.6.6.6 6 biz-internet biz-internet up

If we quickly check the initial state of vEdge-2, we will see that it has only one transport interface, ge0/1.
 vEdge-2# sh int | t
#some columns are omitted for clarity
IF IF
AF ADMIN OPER
VPN INTERFACE TYPE IP ADDRESS STATUS STATUS PORT TYPE
--------------------------------------------------------------------
0 ge0/0 ipv4 - Down Down service
0 ge0/1 ipv4 10.10.0.2/24 Up Up transport
0 ge0/6 ipv4 - Down Down service
0 ge0/7 ipv4 - Down Down service
0 system ipv4 1.1.1.2/32 Up Up loopback
2 ge0/2 ipv4 10.1.2.2/24 Up Up service
3 ge0/3 ipv4 10.1.3.2/24 Up Up service
4 ge0/4 ipv4 10.1.4.2/24 Up Up service
5 ge0/5 ipv4 10.1.5.2/24 Up Up service
512 eth0 ipv4 - Down Down service

We can see that the transport interface is marked with the "mpls" color, which means that we have a local TLOC {2.2.2.2, mpls,
ipsec}.

vEdge-1# sh run vpn 0 int ge0/1

#some columns are omitted for clarity
vpn 0
interface ge0/1
description WAN-MPLS
ip address 10.10.0.2/24
tunnel-interface
encapsulation ipsec
color mpls
allow-service all
!
no shutdown
!

If we check the BFD sessions, we will see that it has established an overlay tunnel to all vEdges with the exception of vEdge-1
because it has the same site-id. Notice that vEdge-2 has only one local TLOC marked with the "mpls" color and has formed a
tunnel to all remote colors over the mpls interface.

vEdge-2# show bfd sessions | t

#some columns are omitted for clarity

SYSTEM SITE
SRC IP DST IP PROTO IP ID LOCAL COLOR COLOR STATE
-----------------------------------------------------------------------------
10.10.0.2 10.10.0.3 ipsec 3.3.3.3 3 mpls mpls up
10.10.0.2 39.3.0.3 ipsec 3.3.3.3 3 mpls biz-internet up
10.10.0.2 10.10.0.4 ipsec 4.4.4.4 4 mpls mpls up
10.10.0.2 39.3.0.4 ipsec 4.4.4.4 4 mpls biz-internet up
10.10.0.2 10.10.0.5 ipsec 5.5.5.5 5 mpls mpls up
10.10.0.2 39.3.0.5 ipsec 5.5.5.5 5 mpls biz-internet up
10.10.0.2 10.10.0.6 ipsec 6.6.6.6 6 mpls mpls up
10.10.0.2 39.3.0.6 ipsec 6.6.6.6 6 mpls biz-internet up

Extending vEdge-1 to the MPLS

Let's first extend vEdge-1 to the MPLS cloud. For this example, we will use the private subnet 10.10.1.0/30 between vEdge-1's
Ge0/1 and vEdge-2's ge0/7 as illustrated in figure 6 below.
 IPsec tunnels IPsec tunnels
over INET over MPLS

INET MPLS
Adver se
10.10.1.0/30
Ge0/0 Ge0/1
via eBGP
TLOC
Extension
vEdge-1 vEdge-2
Ge0/1 10.10.1.0/30 Ge0/7

interface ge0/1 interface ge0/7

ip address 10.10.1.1/30 ip address 10.10.1.2/30
tunnel-interface tloc-extension ge0/1
encapsulation ipsec !
color mpls restrict
!
ip route 0.0.0.0/0 10.10.1.2
!
Figure 6. Extending vEdge-1 to the MPLS

To have IP reachability in the direction from the MPLS cloud back to the TLOC extension interface, we will just add static
routing on the MPLS router. However, in real-world deployments, the subnet 10.10.1.0/30 would most likely be advertised to the
provider's edge router via BGP.

It is important to understand that in order to extend vEdge-1 to the MPLS cloud, the actual TLOC extension happens on the
adjacent router . From the perspective of vEdge-1, Ge0/1 is connected to the MPLS provider in the same way as it would have
been with a direct attachment link to the provider itself. Therefore, there is nothing special in the configuration on the vEdge-1,
as you can see below. The transport interface is configured with IP address, color, and encapsulation type, and a default route
to the MPLS cloud is specified. That's about it.
Configure this on vEdge-1
vpn 0
interface ge0/1
ip address 10.10.1.1/30
tunnel-interface
encapsulation ipsec
color mpls
!
no shutdown
!
ip route 0.0.0.0/0 10.10.1.2
!
As we said, the magic happens on the adjacent router that has a transport interface connected to the WAN cloud we are
extending. On vEdge-2, we will configure interface ge0/7 as TLOC extension interface bound to ge0/1 (the interface connected
to the MPLS cloud). The configuration is pretty simple; we just configure an IP address and specify the transport interface that
we are extending to.

Configure this on vEdge-2

vpn 0
interface ge0/7
ip address 10.10.1.2/30
tloc-extension ge0/1
no shutdown
!

The last piece is to ensure that there is connectivity in the reverse direction from the MPLS cloud back to the TLOC interface
subnet. In our example, we will simply configure a static route on the MPLS router for subnet 10.10.1.0/30 that points to vEdge-
2’s mpls interface.

MPLS# conf t
Enter configuration commands, one per line. End with CNTL/Z.
MPLS(config)# ip route 10.10.1.0 255.255.255.252 10.10.0.2

Now, vEdge-1 should have two transport interfaces with local TLOC configuration and should be able to form overlay tunnels
over both WAN clouds.

vEdge-1# sh int | t
#some columns are omitted for clarity
IF IF
AF ADMIN OPER
VPN INTERFACE TYPE IP ADDRESS STATUS STATUS PORT TYPE
---------------------------------------------------------------------
0 eth9 ipv4 - Down Down service
0 ge0/0 ipv4 39.3.0.1/24 Up Up transport
0 ge0/1 ipv4 10.10.1.1/30 Up Up transport
0 ge0/6 ipv4 10.0.0.12/24 Up Up service
0 ge0/7 ipv4 192.168.1.1/24 Up Up service
0 system ipv4 1.1.1.1/32 Up Up loopback
2 ge0/2 ipv4 10.1.2.1/24 Up Up service
3 ge0/3 ipv4 10.1.3.1/24 Up Up service
4 ge0/4 ipv4 10.1.4.1/24 Up Up service
5 ge0/5 ipv4 10.1.5.1/24 Up Up service
512 eth0 ipv4 - Down Down service

The ultimate verification is to check that there are BFD sessions to all remote TLOCs over each local color - biz-internet and the
mpls color which is extended via vEdge-2. You can see in green that we now have tunnels to all remote TLOCs over the mpls
transport.
 vEdge-1# show bfd sessions | t
#some columns are omitted for clarity
SYSTEM SITE
SRC IP DST IP PROTO IP ID LOCAL COLOR COLOR STATE
-----------------------------------------------------------------------------
10.10.1.1 10.10.0.3 ipsec 3.3.3.3 3 mpls mpls up
10.10.1.1 39.3.0.3 ipsec 3.3.3.3 3 mpls biz-internet up
39.3.0.1 10.10.0.3 ipsec 3.3.3.3 3 biz-internet mpls up
39.3.0.1 39.3.0.3 ipsec 3.3.3.3 3 biz-internet biz-internet up
10.10.1.1 10.10.0.4 ipsec 4.4.4.4 4 mpls mpls up
10.10.1.1 39.3.0.4 ipsec 4.4.4.4 4 mpls biz-internet up
39.3.0.1 10.10.0.4 ipsec 4.4.4.4 4 biz-internet mpls up
39.3.0.1 39.3.0.4 ipsec 4.4.4.4 4 biz-internet biz-internet up
10.10.1.1 10.10.0.5 ipsec 5.5.5.5 5 mpls mpls up
10.10.1.1 39.3.0.5 ipsec 5.5.5.5 5 mpls biz-internet up
39.3.0.1 10.10.0.5 ipsec 5.5.5.5 5 biz-internet mpls up
39.3.0.1 39.3.0.5 ipsec 5.5.5.5 5 biz-internet biz-internet up
10.10.1.1 10.10.0.6 ipsec 6.6.6.6 6 mpls mpls up
10.10.1.1 39.3.0.6 ipsec 6.6.6.6 6 mpls biz-internet up
39.3.0.1 10.10.0.6 ipsec 6.6.6.6 6 biz-internet mpls up
39.3.0.1 39.3.0.6 ipsec 6.6.6.6 6 biz-internet biz-internet up

Extending vEdge-2 to the Internet

Let's now extend vEdge-2 to the Internet provider. The configuration is simpler than the MPLS one because we do not need to
advertise the interconnecting subnet 192.168.1.0/24 to the Internet provider. We simply need to translate the TLOC extension
subnet to a publicly routable IP address using NAT.

IPsec tunnels IPsec tunnels

over INET over MPLS

INET MPLS
NAT
192.168.1.0/30
Ge0/0 Ge0/1
TLOC
Extension

vEdge-1 Ge0/7 Subnet-A Ge0/0 vEdge-2

vpn 0 vpn 0
interface ge0/0 interface ge0/0
nat ip address 192.168.1.2/30
! interface ge0/7 tunnel-interface
ip address 192.168.1.1/30 encapsulation ipsec
tloc-extension ge0/0 color biz-internet
! !
ip route 0.0.0.0/0 192.168.1.1
!
Figure 7. Extending vEdge-2 to the Internet
All required configuration is shown in figure 7 above. Once the config is applied, we can see that vEdge-2’s ge0/0 is now a
transport interface and acts as a local TLOC marked with the “biz-internet” color.

vEdge-2# sh int | t
#some columns are omitted for clarity
IF IF
AF ADMIN OPER
VPN INTERFACE TYPE IP ADDRESS STATUS STATUS PORT TYPE
--------------------------------------------------------------------
0 eth9 ipv4 - Down Down service
0 ge0/0 ipv4 192.168.1.2/24 Up Up transport
0 ge0/1 ipv4 10.10.0.2/24 Up Up transport
0 ge0/6 ipv4 - Down Down service
0 ge0/7 ipv4 10.10.1.2/30 Up Up service
0 system ipv4 1.1.1.2/32 Up Up loopback
2 ge0/2 ipv4 10.1.2.2/24 Up Up service
3 ge0/3 ipv4 10.1.3.2/24 Up Up service
4 ge0/4 ipv4 10.1.4.2/24 Up Up service
5 ge0/5 ipv4 10.1.5.2/24 Up Up service
512 eth0 ipv4 - Down Down service

Notice in the TLOC route that the private IP address of vEdge-2’s biz-internet interface is translated to the public IP address of
vEdge-1’s internet interface.

vEdge-2# show omp tlocs ip 1.1.1.2 color biz-internet encap ipsec

#some rows and columns are omitted for clarity
---------------------------------------------------
tloc entries for 1.1.1.2
biz-internet
ipsec
---------------------------------------------------
RECEIVED FROM:
peer 0.0.0.0
status C,Red,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
encap-key not set
encap-proto 0
encap-spi 266
encap-auth sha1-hmac,ah-sha1-hmac
encap-encrypt aes256
public-ip 39.3.0.1
public-port 12406
private-ip 192.168.1.2
private-port 12406
bfd-status up
domain-id not set
site-id 1
...

If we now check the BFD sessions on vEdge-2, we will see highlighted in orange that there are tunnels to all remote TLOCs over
the local biz-internet.
 vEdge-2# show bfd sessions | t
#some columns are omitted for clarity

SYSTEM SITE
SRC IP DST IP PROTO IP ID LOCAL COLOR COLOR STATE
-----------------------------------------------------------------------------
10.10.0.2 10.10.0.3 ipsec 3.3.3.3 3 mpls mpls up
10.10.0.2 39.3.0.3 ipsec 3.3.3.3 3 mpls biz-internet up
192.168.1.2 10.10.0.3 ipsec 3.3.3.3 3 biz-internet mpls up
192.168.1.2 39.3.0.3 ipsec 3.3.3.3 3 biz-internet biz-internet up
10.10.0.2 10.10.0.4 ipsec 4.4.4.4 4 mpls mpls up
10.10.0.2 39.3.0.4 ipsec 4.4.4.4 4 mpls biz-internet up
192.168.1.2 10.10.0.4 ipsec 4.4.4.4 4 biz-internet mpls up
192.168.1.2 39.3.0.4 ipsec 4.4.4.4 4 biz-internet biz-internet up
10.10.0.2 10.10.0.5 ipsec 5.5.5.5 5 mpls mpls up
10.10.0.2 39.3.0.5 ipsec 5.5.5.5 5 mpls biz-internet up
192.168.1.2 10.10.0.5 ipsec 5.5.5.5 5 biz-internet mpls up
192.168.1.2 39.3.0.5 ipsec 5.5.5.5 5 biz-internet biz-internet up
10.10.0.2 10.10.0.6 ipsec 6.6.6.6 6 mpls mpls up
10.10.0.2 39.3.0.6 ipsec 6.6.6.6 6 mpls biz-internet up
192.168.1.2 10.10.0.6 ipsec 6.6.6.6 6 biz-internet mpls up
192.168.1.2 39.3.0.6 ipsec 6.6.6.6 6 biz-internet biz-internet up

Now both vEdge routers at the data center have connections to both WAN providers. In case of a WAN link failure on either of
the routers, it would not lose connectivity to the sd-wan overlay fabric.

« Previous Lesson
Understanding VPNs and LABELs
Next Lesson
LAB 2 - Controlling the topology with »
restricted TLOC Colors

1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive

/ LAB 2 - Controlling the topology with restricted TLOC Colors

LAB 2 - Controlling the topology with restricted

TLOC Colors
In the previous lab lesson, we have seen how we can connect a vEdge router to the WAN transport attached to the adjacent
vEdge router using a feature called TLOC Extension. After we have extended vEdge-1 and vEdge-2 to both WAN providers, all
routers in the topology now have two local TLOCs each - one marked with “biz-internet” color and another one marked with the
“mpls” color.

In this lab lesson, the Internet and MPLS clouds are interconnected and there is full any-to-any IP reachability in the underlay.
Therefore, the default behavior of vEdges results in a full mesh of overlay tunnels between all TLOCs as illustrated in figure 1
below.

10.1.X.0/24
Controllers VPN-X VPN-X
Site-id 100
vEdge-1 vEdge-2
Data Center
1.1.1.1 1.1.1.2
(Hub)
Site-id 1 Site-id 1
aws T11 T12 T21 T22

vEdge-3 T31 T61 vEdge-6

3.3.3.3 6.6.6.6
Site-id 3 T32 T62 Site-id 6

Colors:
biz-internet
mpls T41 T51
T42 T52
Combinations: vEdge-4 vEdge-5
4.4.4.4 5.5.5.5
Site-id 4 Site-id 5
Figure 1. A full-mesh of overlay tunnels

Recall that vEdges attempt to form a tunnel to every remote TLOC over each local color. For example, vEdge-1 attempts to form
an overlay tunnel to every remote TLOC over its biz-internet interface (T11) and then again over its mpls interface (T12). This
results in three different types of tunnels based on the combination of colors at both ends:

biz-internet <-> biz-internet

biz-internet <-> mpls
 mpls <-> mpls

In real-world deployments, it is far more likely that an MPLS cloud is not reachable from the Internet and there is no any-to-any
reachability between all available WAN transports in the underlay. In such cases, we typically want to configure the interfaces
connected to a WAN cloud that is not reachable from the outside, with an additional TLOC Color parameter called Restrict.

What is a TLOC Color Restrict

By default, WAN edge routers try to form overlay tunnels to every received TLOC from a different site using every available
local color. That is usually the desired behavior in scenarios with multiple Internet connections from various ISPs because the
Internet provides any-to-any reachability irrespective of the particular ISP. However, this behavior might not be desirable in
scenarios where we have private transports alongside the Internet. For example, an MPLS cloud is typically not reachable
through the Internet. Therefore, we would like to stop the establishment of overlay tunnels between the Internet and MPLS
TLOCs. Even though IP reachability between the clouds may exist, the tunnels might be established over paths that are
inefficient or unintended.

Tunnel combinations without

TLOC Restrict

T1 ISP-1 T3

T2 T4
vEdge-1 ISP-2 vEdge-2

Tunnel combinations with

TLOC Restrict

T1 INET T3

T2 T4
vEdge-1 MPLS vEdge-2

Figure 2. TLOC Color Restrict

Cisco SD-WAN allows us to restrict attempts to establish tunnels to TLOCs with different colors (TLOCs from different
transports) using the ‘restrict’ keyword under the TLOC color configuration, as shown in the output below.

vpn 0
interface ge0/0
ip address 10.1.1.43/24
tunnel-interface
encapsulation ipsec
color mpls restrict
!

When a TLOC is marked as restricted, a WAN edge route router will attempt to establish an overlay tunnel to a remote TLOC
only via its transport interfaces marked with the same color. For example, when a vEdge router receives a TLOC marked with
the color “mpls” - it will only attempt to form a tunnel over its interfaces marked with the “mpls” color (if any). This behavior is
demonstrated in the second part of figure 2. vEdge-1 will never try to establish an IPsec tunnel from T1 to T4 or T2 to T3
because the TLOCs are not marked with the same color.
In real-world deployments, private colors such as MPLS are typically configured as restricted, while public colors such as biz-
internet are generally unrestricted.

Configuring TLOC Color Restrict

Configuring a TLOC color as restricted is as simple as adding one additional parameter, “restrict,” alongside the color keyword.
In our lab topology, let’s log into every vEdge router and configure the mpls color as restricted.

Configure this on all vEdges

vEdge-1# conf t
vEdge-1(config)# vpn 0 int ge0/1
vEdge-1(config-interface-ge0/1)# tunnel-interface
vEdge-1(config-tunnel-interface)# color mpls restrict
vEdge-1(config-tunnel-interface)# commit and-quit
Commit complete.

Once the configuration is committed, we can see that now there are only two combinations of tunnels:

biz-internet<->biz-internet
mpls<->mpls

Notice that there aren’t any biz-internet <--> mpls tunnels because the mpls color is now restricted, and vEdges would not form
a tunnel to an MPLS TLOC over their local biz-internet TLOCs.

vEdge-1# show bfd sessions | t

#some columns are omitted for clarity
SYSTEM SITE
SRC IP DST IP PROTO IP ID LOCAL COLOR COLOR STATE
-----------------------------------------------------------------------------
10.10.1.1 10.10.0.3 ipsec 3.3.3.3 3 mpls mpls up
39.3.0.1 39.3.0.3 ipsec 3.3.3.3 3 biz-internet biz-internet up
10.10.1.1 10.10.0.4 ipsec 4.4.4.4 4 mpls mpls up
39.3.0.1 39.3.0.4 ipsec 4.4.4.4 4 biz-internet biz-internet up
10.10.1.1 10.10.0.5 ipsec 5.5.5.5 5 mpls mpls up
39.3.0.1 39.3.0.5 ipsec 5.5.5.5 5 biz-internet biz-internet up
10.10.1.1 10.10.0.6 ipsec 6.6.6.6 6 mpls mpls up
39.3.0.1 39.3.0.6 ipsec 6.6.6.6 6 biz-internet biz-internet up

Now compare the overlay topology of tunnels shown in figure 3 below with the full-mesh that the vEdges had formed initially.
 10.1.X.0/24
Controllers VPN-X VPN-X
Site-id 100
vEdge-1 vEdge-2
Data Center
1.1.1.1 1.1.1.2
(Hub)
Site-id 1 Site-id 1
aws T11 T12 T21 T22

vEdge-3 T31 T61 vEdge-6

3.3.3.3 6.6.6.6
Site-id 3 T32 T62 Site-id 6

Colors:
biz-internet
mpls T41 T51
T42 T52
Combinations: vEdge-4 vEdge-5
4.4.4.4 5.5.5.5
Site-id 4 Site-id 5
Figure 3. A partial-mesh of overlay tunnels

In the common section for Centralised Data Policies, we are going to see that there are other methods that we can use to
control the overlay fabric topology. However, the TLOC color is one of the simplest and most efficient ways to forbid the cross
transport tunnels between WAN clouds.

« Previous Lesson
LAB 1 - Connecting to the WAN using TLOC
Extension
Next Lesson
LAB 3 - Controlling the topology with Tunnel
Groups
»
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / LAB 3 - Controlling the topology with Tunnel Groups

LAB 3 - Controlling the topology with Tunnel

Groups
In the previous lab example, we saw how to control the overlay topology using restricted TLOC colors. In this example, we will
see how to use tunnel-groups in conjunction with the restrict parameter to break up the overlay fabric into two separate
meshes of tunnels.

In real-world deployments, it is pretty standard for organizations to have a group of branches in close geographical proximity,
such as stores on the east coast and another faraway group, for example, on the west coast.

We will make a mesh of overlay tunnels between vEdges 1,2,3 and 4 (highlighted in yellow) and another mesh of tunnels
between vEdges 1,2,5,6 (highlighted in purple). To achieve that, we will configure the TLOCs of vEdge-3 and 4 with the tunnel-
group id 1. On the opposite side, we will configure the TLOCs of vEdge-5 and 6 with the tunnel-group id 2.

10.1.X.0/24
Controllers VPN-X VPN-X
Site-id 100
vEdge-1 vEdge-2
Data Center
1.1.1.1 1.1.1.2
(Hub)
Site-id 1 Site-id 1
aws T11 T12 T21 T22

vEdge-3 T31 T61 vEdge-6

3.3.3.3 6.6.6.6
Site-id 3 T32 T62 Site-id 6

An overlay An overlay
mesh between mesh between
T41 T51
vEdges-1,2,3,4 T42 T52 vEdges-1,2,5,6
vEdge-4 vEdge-5
4.4.4.4 5.5.5.5
Site-id 4 Site-id 5
Tunnel-Group 1 Tunnel-Group 2
Figure 1. Partial meshes

Recall that a local TLOC will only form an overlay tunnel to remote TLOCs that have the same tunnel-group-id or no tunnel-
group-id. Tunnel groups work in conjunction with the color restrict option. If a color is restricted, a local TLOC with that color
will only form a tunnel to remote TLOCs with the same tunnel-group ID and the same TLOC color.
Therefore, to achieve the topology shown in figure 1, we must configure the local TLOCs of vEdge-3 and 4 with tunnel-group id
1 and the ones of vEdge-5 and 6 with group id 2. The key point here is that the local TLOCs of vEdges 1 and 2 must be
configured with no group id so they can make tunnels to all tunnel-group ids.

Configuring tunnel-groups on vEdges 3,4,5 and 6

Configuring tunnel groups is as simple as adding one parameter to the local TLOC configuration on a transport interface in vpn
0. Just go ahead and configure the local TLOCs of vEdges 3 and 4 with group id 1 as shown in the output below.

Configure this on vEdge-3 and vEdge-4

vpn 0
interface ge0/0
tunnel-interface
group 1
!
interface ge0/1
tunnel-interface
group 1
!

When we apply and commit this configuration, nothing will change yet because all other TLOCs in the environment will still
have no group-id configured. (any group-id forms a tunnel to no group-id). However, let’s go ahead and configure the local
TLOCs of vEdges 5 and 6 with a different group-id 2.

Configure this on vEdge-5 and vEdge-6

vpn 0
interface ge0/0
tunnel-interface
group 2
!
interface ge0/1
tunnel-interface
group 2
!

Once we apply this configuration, we can see that vEdges 3 and 4 no longer form overlay tunnels to vEdges 5 and 6 because
their TLOCs have different tunnel-group id.

Let’s verify the topology by checking the BFD sessions on vEdge-3 and vEdge-6.

vEdge-3# show bfd sessions | t

#some columns are omitted for clarity
SYSTEM SITE
SRC IP DST IP PROTO IP ID LOCAL COLOR COLOR STATE
-----------------------------------------------------------------------------
10.10.0.3 10.10.1.1 ipsec 1.1.1.1 1 mpls mpls up
192.168.1.3 39.3.0.1 ipsec 1.1.1.1 1 biz-internet biz-internet up
10.10.0.3 10.10.0.2 ipsec 1.1.1.2 1 mpls mpls up
192.168.1.3 39.3.0.1 ipsec 1.1.1.2 1 biz-internet biz-internet up
10.10.0.3 10.10.0.4 ipsec 4.4.4.4 4 mpls mpls up
192.168.1.3 39.3.0.4 ipsec 4.4.4.4 4 biz-internet biz-internet up

You can see that vEdge-3 has overlay tunnels to vEdges 1 and 2 (highlighted in green) and vEdge-4 (highlighted in yellow). As
expected, there aren’t any tunnels to vEdges 5 and 6.
 vEdge-6# show bfd sessions | t
#some columns are omitted for clarity
SYSTEM SITE
SRC IP DST IP PROTO IP ID LOCAL COLOR COLOR STATE
----------------------------------------------------------------------------
10.10.0.6 10.10.1.1 ipsec 1.1.1.1 1 mpls mpls up
39.3.0.6 39.3.0.1 ipsec 1.1.1.1 1 biz-internet biz-internet up
10.10.0.6 10.10.0.2 ipsec 1.1.1.2 1 mpls mpls up
39.3.0.6 39.3.0.1 ipsec 1.1.1.2 1 biz-internet biz-internet up
10.10.0.6 10.10.0.5 ipsec 5.5.5.5 5 mpls mpls up
39.3.0.6 39.3.0.5 ipsec 5.5.5.5 5 biz-internet biz-internet up

The same could be verified on vEdge-6. There are tunnels to the data center and to vEdge-5. As expected, there aren’t any
tunnels to vEdges 3 and 4. The topology is now as intended - there are two separate meshes of overlay tunnels. Notice that the
mpls restrict option still applies. There aren’t any tunnels between the biz-internet and mpls colors because the mpls color is
restricted.

If we want to check the tunnel-group ID of a particular TLOC (highlighted in purple), we can do that using the show omp
tlocs command on any vEdge. In the same output we can check whether a color is restricted (highlighted in red) .

vEdge-6# show omp tlocs ip 5.5.5.5 color mpls

#some columns are omitted for clarity
---------------------------------------------------
tloc entries for 5.5.5.5
mpls
ipsec
---------------------------------------------------
RECEIVED FROM:
peer 1.1.1.30
status C,I,R
Attributes:
...
carrier default
restrict 1
on-demand 0
groups ( 2 )
bandwidth 0
...

« Previous Lesson
LAB 2 - Controlling the topology with
restricted TLOC Colors
Next Lesson
LAB 4 - Connecting to the WAN using a
Loopback TLOC
»
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive

/ LAB 4 - Connecting to the WAN using a Loopback TLOC

LAB 4 - Connecting to the WAN using a Loopback

TLOC
We have seen earlier in this chapter that we cannot assign a particular TLOC color to more than one interface per vEdge
because the color uniquely identifies a single WAN link . Therefore, in scenarios where a vEdge has multiple interfaces
connected to the same WAN provider, we must use different colors on each interface. However, this can overcomplicate the
overlay topology with restricted colors and tunnel groups. That is why the Cisco SD-WAN solution allows us to configure a
loopback interface as a local TLOC instead of a physical interface.

Loopback TLOCs
Using a loopback interface as a local TLOC is a technique that allows a vEdge router to have multiple physical interfaces
attached to the same WAN provider and utilize ECMP across them. The loopback interface serves as a tunnel endpoint and is
marked with a single TLOC color. The following figure visualizes this use case:

vpn 0
interface loopback1
ip address 10.10.4.4/24 The MPLS cloud must be able
tunnel-interface to reach the loopback’s IP
encapsulation ipsec address via both transport
color mpls interfaces

c or
T41 sta ti
uting
Loopback ic ro
MPLS
am
Interface dyn
Lo1 ge0/6
T
ge0/7
T42 T
vEdge-4 A remote
4.4.4.4
INET vEdge

Figure 1. Loopback TLOC over two physical interfaces

Depending on the specific scenario, there are two modes for a loopback TLOC - standard mode and bind mode.

Standard mode
We use the standard mode in scenarios where a vEdge router has multiple physical interfaces to the same WAN provider.
Figure 2 shows a typical use-case where vEdge-4 has two interfaces connected to the same MPLS cloud - ge0/6 with IP
address 10.10.2.1/30 and ge0/7 with addresses 10.10.2.5/30. This scenario often happens when the service provider has two
CE routers onside, and we want to have extra resiliency in the underlay by connecting each vEdge to both CEs.
 Overlay tunnels to
remote TLOCs
T41
Loopback
Interface
Lo1
10.10.2.0/30
ge0/6

MPLS
static routing
ge0/7
10.10.2.4/30 Remote
vEdge-4 TLOCs
4.4.4.4
Figure 2. Connecting to the MPLS cloud via two physical interfaces

In standard mode, traffic coming from the underlay can reach the loopback IP address via each of the physical links. The
loopback interface is not strictly bound to any of the physical interfaces and can load-balance between them based on the
ECMP hash. From a configuration standpoint, there is no tunnel configuration on the physical links but instead, the local TLOC
configuration is applied on the loopback interface, as is highlighted in the output below:

!
vpn 0
interface ge0/6
ip address 10.10.2.1/30
no shutdown
!
interface ge0/7
ip address 10.10.2.5/30
no shutdown
!
interface loopback1
ip address 10.10.0.4/32
tunnel-interface
encapsulation ipsec
color mpls
allowed services all
!
ip route 0.0.0.0/0 10.10.2.2
ip route 0.0.0.0/0 10.10.2.6
!

The key point here is to make sure that the service provider network has IP reachability back to the loopback IP address. In our
example, for simplicity, we will just add two static routes on the MPLS router for vEdge-4’s loo1 IP address via each physical
interface.

MPLS# conf t
Enter configuration commands, one per line. End with CNTL/Z.
MPLS(config)# ip route 10.10.0.4 255.255.255.255 Ethernet0/0 10.10.2.1
MPLS(config)# ip route 10.10.0.4 255.255.255.255 Ethernet0/2 10.10.2.5

However, in real-world deployments, the vEdge router will typically advertise the loopback subnet to the CE routers through a
dynamic routing protocol such as BGP or OSPF.

In standard mode, there is something essential to keep in mind. Because the loopback interface is not strictly bound to any of
the physical WAN links, the link status of the overlay tunnels can be seen differently on different remote ends . That could
happen because the BFD probes measuring the tunnels’ liveliness and performance metrics to different remote TLOCs would
be hashed over different physical links. If one of the CE routers onside experiences an outage or performance degradation,
some overlay tunnels to the mpls color may degrade while others work normally.
Bind mode
In bind mode, the loopback interface is strictly bound to a single physical link. Any traffic coming from the underlay destined
for the physical connection is transparently carried over to the loopback interface and vice-versa.
A typical use case when we configure a Loopback TLOC in bind mode is when the IP address of the physical WAN link cannot
be used for a tunnel endpoint. That could happen when the last mile provider assigns an IP address to the WAN link, and this IP
is filtered within the MPLS cloud or in another transit WAN along the way. In such cases, we apply the local TLOC configuration
under a loopback interface with a different IP address range. Then we bound the loopback interface to the physical interface as
visualized in figure 3 below:

vpn 0
interface loopback1
ip address 10.10.0.4/32
tunnel-interface
encapsulation ipsec
color public-internet
bind ge0/0
T41

MPLS
Loopback
Interface
y T
Ge0/0 an ing
out
r
T42 T
vEdge-4 Remote
4.4.4.4
INET vEdge

Figure 3. Loopback TLOC bound to a physical interface

The loopback configuration is pretty straightforward. We just configure an IP address to the physical WAN interface and ensure
that the loopback IP address is reachable from the WAN provider. Then we apply the local TLOC configuration to the loopback
interface and bind it to the physical link using the command bind [interface name] as shown in figure 3 above.

« Previous Lesson
LAB 3 - Controlling the topology with Tunnel
Next Lesson
QUIZ - Data Plane Fundamentals »
Groups

1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Cisco SD-WAN Deployment

Cisco SD-WAN Deployment

This section will begin our discussion on the Cisco SD-WAN design and deployment topics.

A sequence of steps needs to be performed to have a fully functional Cisco SD-WAN overlay fabric. Figure 1 illustrates a
deployment workflow we will use as an example to explain all required steps.

1 2 3 4
Configure,
Deployment Deploying SD- Upload authorized
upgrade and tune
Planning WAN controllers WAN edge list
controllers

5 6 7 8
Configure feature Configure
Bring up WAN Configure
and device Centralized
edge routers Localized Policy
templates Policies

Figure 1. Cisco SD-WAN Deployment Overview

1. Deployment Planning - The first step in every Cisco SD-WAN deployment is always the planning phase. The better the
planning the easier the deployment . For large-scale deployments, it is crucial to design a good site-list, system-IP, and
hostname conventions. If it is a brownfield deployment, it is critical to put together a detailed migration plan that includes
WAN edge placement, any firewall ports that need to be opened, and code versions.
2. Deploying SD-WAN Controllers - The second step is always to deploy the Cisco SD-WAN controllers. vBond, vSmart, and
vManage should have valid certificates and should be able to authenticate each other successfully. In the case of Cisco
CloudOps deployment, this step would be already covered by Cisco.
3. Configuring, updating, and tuning controllers - In Cisco SD-WAN, controllers are the hearth and brain of the solution.
Therefore, before starting with the WAN edge deployments, we must verify that all controllers are fully operational, are
running on the currently recommended software versions, and follow all configuration best practices.
4. Uploading authorized WAN edge list - Once controllers are up and running, a network administrator must upload the
authorized WAN edge list containing all chassis and serial numbers for all routers authorized to join the SD-WAN overlay.
The file can be uploaded manually, or alternatively, vManage can be synched with the organization's Cisco Smart Account.
5. Configuring feature and device templates - Once the authorized WAN edge list is present on vManage, each router must
be attached to a device template with supplied variables to the necessary parameters (such as IP addresses, port
numbers, etc.). Then when the router joins the overlay fabric, vManage pushes the full configuration to the devices.
Generally, the order of WAN edge deployments is as follows:
1. Deploy data centers and regional hubs first.
2. Deploy branches and Soho second.
3. Deploy IoT devices.
6. Configuring Centralised Policies - At this point, the organization's topology, traffic engineering, and services requirements
must be enforced via Centralised Control policies. The policies are configured on vManage, which pushes them to
the vSmart controllers in the domain.
7. Bring up WAN edge routers - Ideally, when all prior steps are successfully executed, devices would be able to authenticate
and join the SD-WAN overlay. This could be accomplished either via Cisco PnP/ZTP (Plug and Play/Zero-touch
Provisioning) process or via bootstrap configuration. During the onboarding process, depending on the attached device
template, vManage may perform a software upgrade of the device.
 8. Configuring Localized Policies - Any localized policies that are specific for individual WAN edge routers are configured
and attached last.

Notice that the deployment steps may not be exactly in this order. The process is fairly flexible, with the following exceptions:

Deployment planning should always come first and is the most important step of all.
Logically, Cisco SD-WAN controllers should be deployed before WAN edge routers. When upgrading, vManage should be
upgraded first, then vSmart, and lastly vBond.
The authorized WAN edge list must be uploaded to vManage before any vEdge routers can successfully join the SD-WAN
overlay.
Device templates must be attached to WAN edge routers on vManage before successful onboarding them via the
PnP/ZTP process.

« Previous Lesson
QUIZ - Data Plane Fundamentals
Next Lesson
Cisco SD-WAN Certificates Explained »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

OMP Source Preference

Understanding OMP Routing

OMP TLOC-Action

Chapter Key Takeaways

QUIZ - Control Plane Fundamentals

4. Cisco SD-WAN Data Plane

What is a TLOC?
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Cisco SD-WAN Certificates Explained

Cisco SD-WAN Certificates Explained

In this lesson, we are going to start discussing the SD-WAN topic of Certificates. I have found out that many network engineers
do not really understand the basic concepts of online trust, PKI, Certification Authorities, and Certificates. That is why I've
decided to make this elementary, high-level lesson that explains each term in plain simple language with lots of funny
examples.

What is Trust?
Let's start with the basics.

In general terms, trust is confidence in or reliance on another person or entity. In networking, trust is to be able to verify whether
the software and the hardware that incorporate your network infrastructure are uncompromised, genuine, and functioning as
intended.

D
ct Tr ire
e
ir ust us ct
D r t
T

Third-party
Direct Trust Trust

If one person trusts another If two individuals trust each

person, it is direct trust other, because each trusts a
third person, it is third-party
trust
Figure 1. What is Trust?

There are two general types of trust, as illustrated in figure 1 above:

If one person trusts another person, this is Direct trust.

If two individuals trust each other, because each trusts a third person or entity, this is Third-Party trust. For example, if
you and a colleague that you don't personally know both unquestioningly trust your employer, you trust each other even
though you've never met.

Because of the vastness of the Internet and the billions of connected devices, websites, and all kinds of other stuff, the direct
trust model simply can't scale that much and is not feasible. That is why, in online communications and the Internet, we use
the Third-party trust model . An entity on the Internet (device, website, service, etc.) trusts another one because each one
trusts a third-party entity called a Certificate Authority (CA).
In Cisco SD-WAN, devices trust each other because they trust a common Root CA, as shown on the left in figure 2 below. This
is typically referred to as the chain-of-trust.

The private key

The devices trust each other, Root CA never leaves
because each trusts the same root Certificate the Root CA
Certification Authority (CA)
Root CA Root CA
Distinguished Name Private Key
Root CA
Root CA Public Key

Root CA Signature
Self-Sign

D ru
us ct

ir st
T
Tr ire
t

ec
D

t
Root CA
Third-party Certificate
Trust that includes the
Root CA public key

Controller Router Controller Router

Figure 2. Cisco SD-WAN Trust via Certificates

A device knows which Root CAs it trusts by having an installed root certificate that identifies each root certificate authority
(CA), as shown on the right side of figure 2. The root certificate includes the public key of the CA. You will see later in the
lesson why this is important.

What is Certificate Authority (CA)?

A certificate authority (CA), generally referred to as the Root CA, is an organization that verifies the identities of entities (such
as devices, websites, emails, etc) and ties them to cryptographic keys through the assignment of digital certificates.

Root Certificates
The CA issues root certificates that are installed on devices and identify the root certificate authority (CA). The key point here is
that the root certificates are always self-signed. Therefore, nobody can verify them directly . Root certificates are generally
made trustable by other means different than a certificate, such as regulations, public scrutiny, secure distribution, and
hardware root of trust modules integrated into devices during manufacturing. For example, Microsoft and Apple distribute their
operating systems (Windows and MAC OS) with pre-installed root certificates of all well-known CAs. Cisco ships all SD-WAN
devices with pre-installed root-ca-certificates as you can see in the output below.

vEdge-1# show certificate root-ca-cert | in Issuer

Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc.
Issuer: C=US, O=Symantec Corporation, CN=Symantec Trust Services Private SHA256 Root CA
Issuer: C=US, O=Symantec Corporation, CN=Symantec Trust Services Private SHA256 Root CA
Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc.
Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc.
Issuer: C=US, O=VeriSign, Inc., OU=Class 3 Public Primary Certification Authority
Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
Issuer: C=US, O=VeriSign, Inc., OU=VeriSign Trust Network, OU=(c) 2006 VeriSign, Inc.
Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA
Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root CA
Issuer: OU=Arcturus, O=Cisco, CN=Internal Customer Root CA

In use cases where an organization wants to use its own Enterprise CA, the network administrators install an additional root
certificate on all devices.
End-Client Certificates (as a passport)
The client certificate sometimes referred to as an end-entity or subscriber certificate, is like a person's passport. When a
person wants to prove his identity to someone, it shows his passport. In the same way, when a device wants to prove its
identity to a remote device, it sends its client certificate during the SSL/TLS handshake . The following table makes a funny
comparison between a client certificate and a passport.

A person's passport An End-client Certificate

Issued By a trusted entity - the government By a trusted entity - a Root Certificate Authority (CA)

Apply The person submits a new passport request including its The device generates and submits a Certification Signing
unique citizen number to the government institution. Request(CSR) that includes its public key to the Root CA.

Validates The person's identity to others that trust this country's The client's identity to others that trust this Root CA
passports

Validity It has a few years validity period and has to be renewed It has a few years validity period and has to be renewed

A client certificate as a passport

Imagine that you are at an airport somewhere in a random country in the world. When you want to prove your identity to the
border control, you show your passport. The border control will verify your identity if they accept your country's passports and
have a tool that can check whether your passport is genuine. However, if you show your passport to someone that doesn't
accept your country's passports, it won't verify your identity.

Similar to the example above, when a client initiates an SSL/TLS connection to a server, the client presents its certificate in
order to prove its identity. If the server has the public key of the issuer of the client's certificate, it will verify the certificate's
signature and trust the client. The example is illustrated in figure 3 below.

We accept your country’s

Here is my passport that passports and can verify
proves my identity your identity

Passport with
embedded data chip

verify

Here is my client certificate I have your issuer’s public

that proves my identity key and can verify your
identity

Client Certificate
issued by VeriSign
VeriSign root
SSL/TLS certificate with
handshake public key

verify

Figure 3. Client Certificate as a Passport example

Another important aspect of a client certificate is that it cannot issue additional certificates. It is basically the final link in the
chain-of-trust.

Certificate Signing Request (CSR)

Ok, at this point you may be wondering how a client gets a certificate.
Generally, a digital certificate applicant, such as a Cisco SD-WAN device, generates a key pair consisting of a public and a
private key, along with a certificate signing request (CSR). The CSR is basically a text file that includes the client's identity
information such as organization name, country, state, email address, domain name, and so on, along with the client's public
key. However, unlike the public key, the client's private key is never shown to Root CA (or anyone else) is kept secure at all
cost. The key generation and the CSR are typically done on the client where the certificate will be later on installed.

After the client generates the CSR request, it sends it to a CA (step 1). The root CA double-checks the information and signs the
certificate with its private key (step 2). Then it sends the certificate back to the client as shown in step 3 of figure 4 below. The
certificate is now digitally signed and each device that has a root-ca-certificate from this CA would be able to verify the
signature.

Certificate of vSmart
Root CA
Certificate signing
Sending the CSR to request (CSR)
the Root CA 1
Organization Name
Country
City Root CA Name
Certificate signing State
... Public Key Root CA
request (CSR)
of vSmart Private Key
Organization Name Issuer’s (CA)
Country Distinguished Name
City 2
State Public Key Digital Signature of Sign
... of vSmart Root CA
3
Issues a signed certificate
vSmart

Public Key Private Key

Figure 4. Issuing a Certificate

Let's now see how the verification process happens.

Verifying Identity via Certificates

When the client initiates an SSL/TLS secure connection to a remote server, it presents its signed certificate. When the server
receives the client's certificate, it checks the Issuer's distinguished name. If the server has the corresponding issuer's root-ca-
certificate installed, it can cryptographically verify the client certificate's signature via the Root CA’s public key . Furthermore,
the server can use the certificate's public key to confirm that content was sent by a client in possession of the corresponding
private key and that the information's integrity (the information has not been altered after the signing). If the server does not
have the issuer's root-ca-certificate, it won't be able to verify the identity of this client.

Figure 5 shows an example of this process where a vSmart controller (a client) initiates a secure SSL/TLS connection to a
vManage controller (a server).
 vSmart is now trusted.
vSmart’s public key can
be used for encryption.
2
vSmart vManage 3
vManage validates
vSmart the certificate
IP Certificate
vSmart
Distinguished Name
1
Root CA
vSmart Public Key
vSmart provides Certificate
signed certificate to Issuer’s (CA) Root CA
vManage Reference
Distinguished Name Distinguished Name
Root CA Signature Verify Root CA Public Key

Root CA Signature

Figure 5. Cisco vManage validates vSmart

Notice that in Cisco SD-WAN, devices authenticate mutually in both directions. For example, when forming a DTLS control
connection between a vSmart and a vManage controller, the vSmart proves its identity to vManage, as shown in figure 5 above,
and then vManage proves its identity to vSmart as shown in figure 6 below.

vManage is now trusted.

vManage’s public key can
be used for encryption.
3 2 vManage
vSmart
vSmart validates
the certificate vManage
Certificate IP
vManage
Distinguished Name 1
Root CA
Certificate vManage Public Key vManage provides
signed certificate to
Root CA Issuer’s (CA)
Reference vSmart
Distinguished Name Distinguished Name

Root CA Public Key Verify Root CA Signature

Root CA Signature

Figure 6. Cisco vSmart validates vManage

At this point, you must have a general idea behind the online trust model using third-party trust via certificates. However, I have
intentionally simplified something important - In the real world, there is typically one more link in the chain-of-trust between the
end client and the CA and that is the Intermediate CA. Let's see why.

Root CA Security is Paramount

As you have seen so far, the security of the root CA's private keys is uttermost important. If a private key gets compromised
and falls into bad hands, the entire chain-of-trust gets compromised which could lead to large-scale security breaches.

That is why the Root CAs store their private keys within data centers with state-of-the-art physical and IT security. This might
seem extreme but is absolutely necessary to protect the authenticity of the issued root certificates.
One important mitigation measure is the introduction of the CA hierarchy that involves three tiers - Root Certificate Authority →
Intermediate Certificate Authorities → End-Client Certificates.
Intermediate CAs
While the root certificate is sufficient to implement the online trust, in reality, almost all well-known CAs use intermediate
certificates as it adds an additional layer of security and helps scale. In case of a security attack, only the intermediate
certificate needs to be revoked instead of revoking the root certificate and all of its associated certificates that it has been
used to sign.

Intermediate
CAs End-client certificates
issued by
Intermediate CA 1
1
Root CA
End-client certificates
2 issued by
Intermediate CA 2

3 End-client certificates
issued by
Intermediate CA 3

Figure 7. Intermediate CAs

Some more reasons why intermediate certificates are used are listed down below

Intermediate certificates help control the number of root certificates in use and help mitigate security risks and fraud. As
more and more users implement SSL sites, the number of root CAs will also increase. But having too many root CAs can
lead to serious security implications, which intermediate certificates aim to resolve. They provide a means for the root
CAs to delegate some of the certificate issuing responsibilities to intermediate CAs, providing intermediate certificates
that will substitute for a root certificate.
Intermediate certificates can be replicated in high numbers without compromising the security framework and helping
establish the Chain of trust.
They help with the scalable implementation of the SSL network.

The chain of trust

When the Root CA hierarchy consists of more than one CA layer, the chain-of-trust changes, as shown in figure 8 below. The
end clients now send their certificate signing requests (CSR) to an Intermediate CA which signs it and issues the client
certificate.
 Client
Certificate
Client
Distinguished Name
Intermediate
Client Public Key Certificate
Issuer’s (CA) Issuer’s
Distinguished Name
Refer Distinguished Name
Root
Issuer’s Signature Verify Issuer’s Public Key Certificate
Root CA Root CA
Sign Distinguished Name
Refer Distinguished Name

Root CA Signature Verify Root CA Public Key

Root CA Signature
Issuer’s Private Key
Sign

Root CA Private Key

Self-sign
Figure 8. Three layers chain-of-trust hierarchy

Now a remote server that receives such a client certificate needs to have both the Intermediate and the root certificate
installed in order to verify the client certificate. However, Cisco vManage accepts only one root certificate, so what do we do in
this case?

In such cases, we download/export the root CA and intermediate CA certificates and place both of them into a root chain.
Then we install the root chain on vManage!

Undoubtedly, managing Cisco SD-WAN certificates and understanding all their technicalities is challenging. Well, I hope not
anymore!

« Previous Lesson
Cisco SD-WAN Deployment
Next Lesson
Controllers Identity and Whitelisting »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing
Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Controllers Identity and Whitelisting

Controllers Identity and Whitelisting

Cisco SD-WAN is designed with the highest level of security in mind. Part of the security stack of the solution is the
authentication and authorization of components. Basically, before establishing a control-plane connection, each component
must authenticate to each other using their signed certificate, which is validated against the specified chain of trust. The idea
is visualized in figure 1 below.

Controller
Certificate

Root Root
Certificate Certificate

vManage vSmart
Figuer 1. Cisco SD-WAN Establishing Trust

Controllers Identity
Cisco SD-WAN Controllers can not be brought into operation unless their identity is validated by an established chain of trust.
This identity validation process is intended to ensure that only trusted devices can join the SD-WAN solution while still
retaining flexibility. Each controller must have a root certificate installed and a controller certificate installed and signed by a
trusted CA (Certification Authority). Depending on the deployment method, the chain of trust may be defined in the following
manner:

In cloud and managed deployments, the root certificate chains might be pre-loaded or automatically installed.
In on-prem deployments, the root certificate is retrieved from an Enterprise CA that can be OpenSSL, Cisco ISE, AWS
Certificate Manager, or another private CA option.

For the controller certificates, a CSR (Certificate Signing Request) is generated for every controller through the vManage GUI.
Then each certificate request is submitted either automatically or manually to the certification authority. After the certification
is signed, it is downloaded and installed on the respective controller. There are several different methods to obtain and install a
controller certificate:

1. Automated certificate deployment using Cisco PKI: This is the recommended method according to Cisco's documentation .
A network administrator generates a CSR for each SD-WAN controller. The CSR is then automatically sent to Cisco's Public Key
Infrastructure. When the CSRs are signed, vManage automatically downloads each signed certificate and installs it on the
respective SD-WAN controller. Note that the root certificate that defines the chain of trust is pre-loaded by default on each SD-
WAN controller. This method requires vManage version 19.1 or higher.
 Cisco
Server

Certificates
retrieved vBond
CSRs Sent 2 3

Certs
1 4
installed
Generate
CSRs
Admin vManage

vSmart
Figure 2. Automated Certificate signing using Cisco PKI

2. Automated certificate deployment using 3rd-party CA: A network administrator generates a CSR for each SD-WAN
controller. The CSR is then automatically sent to the Symantec/Digicert PKI. The difference here is that a Cisco
TAC case needs to be submitted in order to complete the certificate signing process. Once the certs are signed, vManage
automatically downloads each one and deploys it on the respective SD-WAN controller. Note that again the root certificate that
defines the chain of trust is pre-loaded by default on each device.

CSRs
Cisco TAC Server
Signed
4

Certificates
Open TAC retrieved vBond
3
case CSRs Sent 2 5

Certs
1 6
installed
Generate
CSRs
Admin vManage

vSmart
 Figure 3. Automated Certificate signing using 3rd-party CA

3. Manual Cisco PKI certificate signing: This option is similar to the automated certificate deployment using Cisco PKI with the
only difference that the CSRs are downloaded locally and manually submitted for signing. After the certs are signed, they
are downloaded locally and upload manually to vManage. Once that is completed, vManage deploys each cert on the
respective controller. Note that again the root certificate that defines the chain of trust is pre-loaded by default on each device.

Cisco
Server

CSR
4
Signed

Download
Manually Certificates vBond
Submit 3 5
CSRs
Generate
CSRs
1 Certs
7
installed
2
Download
CSRs
Admin vManage
6
Upload
certs
vSmart
Figure 4. Manual Certificate signing using Cisco PKI

4. Manual third-party certificate signing through Symantec/Digicert: This option is similar to the automated
certificate deployment using 3rd-party CA with the only difference that the CSRs are downloaded locally and manually
submitted for signing. A Cisco TAC case needs to be opened in order to complete the certificate signing process. After the
certs are signed, they are downloaded locally and upload manually to vManage. Once that is completed, vManage deploys
each cert on the respective controller. Note that again the root certificate that defines the chain of trust is pre-loaded by default
on each device.
 Cisco TAC CSRs Signed Server

Manually
Submit CSRs
Open TAC
4 3
case
6 Download
Certificates vBond

Generate
CSRs
1
Certs
2 8 installed
Download
Admin CSRs
7
Upload vManage
certs

vSmart
Figure 5. Manual Certificate signing using 3rd-party CA

5. Enterprise Root Certificate Authority (CA): Of course, Cisco SD-WAN provides the flexibility to enterprises to use their
private PKI infrastructure and sign controller certs with their own CA. This method requires the most manual steps comparing
to the other available options. The first one is to install the Enterprise CA root certificate on vManage, vManage then
automatically distributes this root cert to the other configured controllers. This process is visualized with the black lines in
figure 6. A network administrator then generates the CSRs and makes a request for each controller to the Enterprise Root
CA. This process is visualized with the blue lines in the figure below. Once the certificates are signed, they are manually
uploaded to vManage. vManage will then deploy each certificate on the respective controller. This is visualized with the green
lines.
 Enterprise
CA

CSR
7
Signed

Download
Certificates vBond
Get Root 1 6 8
Certificate Root Cert
3
installed
Upload Root
Cert
2
Generate Certs
10
CSR installed
4
Download
Admin CSR vManage
5
Upload Certs
9
vSmart
Figure 6. Manual Certificate signing using Enterprise CA

Controller Whitelisting
Cisco SD-WAN employs a whitelisting approach when it comes to adding new controllers into the solution domain. When a
network administrator configures a new control device through the vManage GUI, it is automatically added to an authorized list
that includes certificate serial numbers of all controllers. This list is then distributed by vManage to all other control-plane
devices within the domain. Before establishing control-plane tunnels with each other, the controllers always check whether the
remote node's parameters against the authorized whitelist. This technique prevents rouge unauthorized controllers from
joining the solution and pushing unapproved configs. However, note that vBond is not checked against the authorized list, but
all devices are explicitly configured with the vBond IP address so it basically achieves the same goal.

vBond

vManage
Admin

vSmart vSmart
All Cisco SD-WAN controllers
are defined explicitly
Controller Hostname System IP Site-ID
vSmart
vBond vBond1 5.5.5.1 20
vManage vManage1 5.5.5.2 20
vSmart vSmart1 5.5.5.3 30
vSmart vSmart2 5.5.5.4 40

Figure 7. Controllers Whitelisting

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / WAN Edge Deployment

WAN Edge Deployment

In this lesson, we are going to go through the different WAN edge onboarding options that Cisco SD-WAN provides.

vEdge devices could be physical appliances or virtual instances. Both types can be onboarded using an automated deployment
process, such as Zero Touch Provisioning (ZTP) for Viptela devices and Cisco Plug-and-Play for IOS XE devices. However, there
are two available options in case that automated deployment is not possible - manual deployment using the CLI or bootstrap
configuration that can be loaded via USB stick.

WAN Edge Deployment Options

Automated Provisioning Manual Bootstrap

PnP ZTP
via CLI via USB
(Plug-and-Play) (Zero-touch-provisioning)

Cisco IOS-XE Viptela vEdge devices All device IOS-XE

Figure 1. vEdge Deployment Options

It is important to make sure that the following statements are true before a WAN edge device can be onboarded:

All Cisco SD-WAN Controllers (vManage, vBond, and vSmart) should be deployed and operational with valid certificates
installed.
The Edge device should have IP reachability to all controllers .

Automated Deployments
The automated WAN edge deployment is the recommended method for adding new nodes to the Cisco SD-WAN fabric. It is
enabled by default on all vEdge devices and provides a true zero-touch experience. In essence, this automated onboarding
just discovers the vBond IP address dynamically using one of the following processes:

On all Cisco IOS-XE devices, the process is called Cisco Plug-and-Play (PnP). It basically resolves the
hostname devicehelper.cisco.com and asks what is the vBond IP for my organization-name.
On all Viptela vEdge appliances, the process is called Zero-Touch provisioning (ZTP). It resolves the
hostname vtp.viptela.com and gets the vBond IP for the given organization-name.
 vEdge vManage vBond ZTP DNS DHCP

ZTP DHCP

OFF -> ON
Obtain IP
1
ztp.viptela.com
2
Get vBond
address
3
Authenticate
Controllers list
4
Authenticate
5
Join Fabric
6 Overlay
Fabric

Figure 2. ZTP / PnP Process Sequence

A high-level overview of the steps involved during the Zero-touch Provisioning (ZTP) / Cisco Plug-and-Play (PnP) deployment
process is listed below:

0. The WAN edge device is powered up.

1. The vEdge attempts to assign an IP address to its transport interface in VPN0. If it receives IP settings
(address/mask/gateway/DNS) via DHCP or Auto-IP, it continues to step 2, otherwise, the automatic deployment does not
continue
2. The router tries to resolve the URL ztp.viptela.com (for Viptela vEdge devices) or devicehelper.cisco.com ( for Cisco IOS-
XE device).
3. The device contacts the PnP/ZTP server. The server verifies the vEdge router and sends back the IP address of the
respective vBond Orchestrator for this Organization-name.
4. The vEdge establishes a transient connection to the vBond orchestrator. Note that at this point in the automated
deployment process, the WAN edge router does not have a system-IP configured, so the connection is established with a
NULL system IP address . The Edge authenticates to vBond with a chassis number and serial number. The vBond then
sends back the IP address/port of the other SD-WAN controllers as visualized in figure 3.
5. The WAN edge node then establishes a connection to vManage and gets its System-IP address. Then it repeats the
process but this time with the correct System-IP (not NULL):
1. It re-establishes a connection to the vBond using its system IP.
2. It re-establishes a connection to vManage using its system IP.
3. vManage then pushes the full configuration to the WAN edge routers.
6. The router establishes a connection to vSmart and joins the overlay fabric .
 vBond vSmart1 vSmart2 vManage

Private IP

Public IP

Permanent sessions
with vBond
Controllers List:
vSmart1: Private/Public IP, System IP
vSmart2: Private/Public IP, System IP
vManage: Private/Public IP, System IP

vEdge

Figure 3. vBond sends back the SD-WAN controllers list

Bootstrap Deployment
In use-cases where the WAN edge router is not able to obtain a dynamic IP address (no DHCP), or cannot reach the PnP/ZTP
public URL addresses, or the device is deployed in an air-gapped environment, there is an alternative onboarding option called
Bootstrap deployment. It is only available on the Cisco IOS-XE device though ! It requires a device template configuration to be
configured and attached to the WAN Edge device in the vManage GUI. The configuration file can then be uploaded to the
device’s internal flash memory or by using a bootable USB stick, which can be plugged into the device at bootup. It is important
to note that, the config file must have a specific filename for the device to load it - ciscosdwan.cfg (exception is ASR1002-x,
where the file must be named ciscosdwan_cloud_init.cfg)
 vEdge vManage vBond Bootstrap Config

CFG

OFF -> ON PnP process looks for the

ciscosdwan.cfg file in flash
or bootable usb
1
The vEdge learns
the vBond address
and Organization-name
2
Authenticate to vBond
Get Controllers list
4
Authenticate
5
Join Fabric
6 Overlay
Fabric

Figure 4. Bootstrap Onboarding Process Sequence

The bootstrap onboarding process sequence is listed below:

1. At bootup, a WAN Edge router searches its boot flash memory for a configuration file with a specific filename based on
the platform. If the file is not present, the PnP process continues to search for the file on all connected USB sticks. If it
manages to find the file, it loads the config, otherwise, the process does not continue further
2. If the config file is successfully loaded, the WAN Edge router learns the vBond IP address and organization name and
establishes a secure connection to the vBond orchestrator. The Bond sends back the controllers-list.
3. The WAN Edge router then establishes secure connections to vManage and vSmart, downloads its configuration using
NETCONF over SSH (TCP 830) from vManage, and joins the SD-WAN overlay fabric.

Manual Deployment
The manual onboarding option is something that Network Engineers are pretty familiar with. The WAN edge device is basically
configured via the console port or using the KVM/ESXi virtual console connection if the device is a virtual one.
 vEdge vManage vBond CONSOLE

>_

OFF -> ON Net admin configures:

- System IP
- Site-ID
- Org-name
- vBond Address
- Transport VPN 0
1
Authenticate to vBond
Get Controllers list
4
Authenticate
5
Join Fabric
6 Overlay
Fabric
Figure 5. Manual Onboarding Process Sequence

Тhe minimum configuration that is required to successfully onboard a WAN edge router is as follow:

System-IP, Site-id, Organization-name, vBond IP address.

VPN 0 interface with IP address/mask, default route, and tunnel interface.

WAN Edge Authorized List

If you have been going through the lesson very closely, you should have noted that the automatic authentication of vEdges can
only occur if the vBond/vSmart knows the serial and chassis numbers of the WAN edge routers . The SD-WAN controllers learn
this information through a document called vEdge authorized list. This provisioning file can be downloaded from the Cisco
Software Central > Plug and Play Connect portal and then uploaded to vManage, which then, sends the list to all vSmart and
vBond controllers.
 vEdge
Authorized
List
Network
Admin

vBond vSmart1 vSmart2 vManage

Private IP

Public IP

Permanent sessions with vBond

· Organization Name
· Serial Number
vEdge
· Root Certificate
· Device Certificate

Figure 6. vEdge authorized serial numbers list

This process will be covered in more detail in the lab lessons about vEdge onboarding.

WAN edge deployment behind a Firewall

At the beginning of this lesson, I have written that it is mandatory to have IP reachability from the vEdge to all controllers in
order to onboard a device. In reality, the connectivity can be restricted only to the following protocols/ports in case that the
solution uses the default DTLS encapsulation.

vEdge / vBond vSmart / vManage DNS NTP ICMP

UDP 12346 - 12445 UDP 12346 - 13065 UDP 53 UDP 123 Echo / Reply

Firewall
Layer

vEdge

Figure 7. WAN Edge Required FW Ports

Note that, for the certificate authentication to succeed, the time between the WAN Edge routers and the SD-WAN controllers
should be synced. That is why NTP should be allowed through the firewall.

« Previous Lesson
Controllers Identity and Whitelisting
Next Lesson
Last Resort Circuit »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

OMP Source Preference

Understanding OMP Routing

OMP TLOC-Action

Chapter Key Takeaways

QUIZ - Control Plane Fundamentals

4. Cisco SD-WAN Data Plane

What is a TLOC?

The Overlay Fabric

TLOC Color and Carrier

Tunnel Groups

TLOCs and NAT

Data Plane Encryption

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Last Resort Circuit

Last Resort Circuit

The Business Need
Cisco SD-WAN's device portfolio includes WAN edge routers that support WAN connections over 3G/4G LTE. This is a great
option in remote areas where Internet circuits are expensive or not available

However, 3G/4G LTE is not a service provider leased line and is not designed for communicating a large amount of data at a
constant rate 24/7. In many parts of the world, an LTE SIM card comes with a data limit that only allows for a certain volume of
data to be sent over the LTE line per month. After the data limit is exhausted, either the radio link speed is greatly decreased or
there are additional charges for provisioning additional data.

Therefore, in many real-world deployments, where we have a remote site connected to two WAN transports, one of which is
LTE, we would generally like to use the LTE radio link only in case the other transport goes down.

4G/LTE

T3

IPsec + BFD

T1 T2
INET
IPsec + BFD
vEdge-1 vEdge-3
Figure 1. Why do we need a Last Resort Circuit

One way of offloading the traffic from the LTE link is by configuring a higher TLOC preference and higher WEIGHT to the primary
WAN transport. This will make sure that in normal circumstances, most of the traffic will pass through the 'primary' tunnel.
However, this is not an optimal solution, because even the IPsec tunnel to the LTE TLOC is generating constant traffic. There is
a BFD session that exchanges keepalives every second (as shown in figure 1) and there are DTLS control connections via
which the vEgde constantly pings the controllers (as shown in figure 2)

These control/overlay connections will still consume a lot of data, even though application traffic does not go over this TLOC.
 4G/LTE

T3
DTLS

T1 T2
INET
DTLS
vEdge-1 vEdge-3
Figure 2. Control Connections over 4G/LTE

Let's verify that on vEdge-3 using the CLI. You can see that there is a BFD session that is UP and the TX interval time is 1
second. Therefore, each second there will be at least two BFD probes to this TLOC (one originated by vEdge-1 and one by
vEdge-3). But what if there are multiple WAN edge routers and there are many BFD sessions? Depending on the 4G LTE plan,
this may not be very efficient and consume a lot of data unnecessarily.

vEdge-3# show bfd sessions | tab

SRC DST SYSTEM SITE LOCAL DETECT TX

SRC IP DST IP PROTO PORT PORT IP ID COLOR COLOR STATE MULTIPLIER INTERVAL
UPTIME TRANSITIONS
------------------------------------------------------------------------------------------------------------------
------------------------
10.10.0.1 10.10.1.51 ipsec 12386 12366 15.1.1.1 15 mpls mpls up 7 1000
0:01:35:14 0
39.3.0.2 39.3.0.1 ipsec 12366 12346 15.1.1.1 15 lte public-internet up 7 1000
0:02:30:41 0

We can also verify that there are control connections over this orange TLOC. Therefore the WAN edge router is constantly
pinging the controllers to make sure they are reachable. This may consume additional data as well.

vEdge-3# show control connections | tab

#some columns are omitted for clarity

LOCAL LOCAL
PEER SITE PRIVATE PRIVATE PUBLIC SYSTEM LOCAL REMOTE PRIVATE PRIVATE
BEHIND
INSTANCE TYPE ID IP PORT PUBLIC IP PORT IP COLOR COLOR IP PORT
STATE UPTIME V ORG NAME PROXY
------------------------------------------------------------------------------------------------------------------
------------------------------------------
0 vsmart 1 10.10.0.1 12386 10.1.1.30 12346 1.1.1.30 mpls default 10.1.1.30 12346 up
0:02:02:45 networkacademy-io No
0 vsmart 1 39.3.0.2 12366 10.1.1.30 12346 1.1.1.30 lte default 10.1.1.30 12346 up
0:02:03:46 networkacademy-io No
0 vbond 0 10.10.0.1 12386 10.1.1.10 12346 0.0.0.0 mpls mpls 10.1.1.10 12346 up
0:02:02:12 networkacademy-io -
0 vbond 0 39.3.0.2 12366 10.1.1.10 12346 0.0.0.0 lte lte 10.1.1.10 12346 up
0:02:02:36 networkacademy-io -
0 vmanage 1 39.3.0.2 12366 10.1.1.20 12546 1.1.1.20 lte default 10.1.1.20 12546 up
0:01:23:45 networkacademy-io No
A better solution - Last Resort Circuit
A better solution to this problem would be to form an IPsec tunnel over this 4G TLOC only in case that the primary WAN
transport goes down. Well, Cisco SD-WAN provides such an option in the solution. It is called Last Resort Circuit and is very
straightforward and easy to set up.

Last Resort A tunnel between T1–T3 forms only

Circuit in case that T1–T2 goes down

4G/LTE

T3
Last-resort
circuit

T1 T2
INET
IPsec + BFD
vEdge-1 vEdge-3
Figure 3. Cisco SD-WAN Last Resort Circuit

The idea is visualized in figures 3 and 4. We would like to advertise the LTE TLOC to the vEdges but only form a tunnel when the
primary IPsec tunnel goes down.

4G/LTE

T3
Last-resort
circuit

T1 T2
INET
DTLS
vEdge-1 vEdge-3
Figure 4. Cisco SD-WAN Last Resort Circuit Control Connections

The same logic applies to the control connections as well. We would like to form a control connection and OMP peering over
the LTE TLOC only in case of primary link failure.

Last Resort Circuit Configuration

Let's first check the initial configuration of both TLOCs of vEdge-3. There is nothing out of the ordinary.
 !
vpn 0
interface ge0/0
ip dhcp-client
ipv6 dhcp-client
tunnel-interface
encapsulation ipsec
color lte
allow-service all
!
no shutdown
!
interface ge0/1
ip dhcp-client
tunnel-interface
encapsulation ipsec
color mpls restrict
allow-service all
!
no shutdown
!
!

To enable the Last Resort Feature on the 4G transport, we just configure the command Last Resort Circuit under the tunnel-
interface configuration as follows:

!
vpn 0
interface ge0/0
ip dhcp-client
ipv6 dhcp-client
tunnel-interface
encapsulation ipsec
color lte
last-resort-circuit
allow-service all
!
no shutdown
!

Now let's verify whether there is still a tunnel via the 4G TLOC.

You can see that there is no tunnel over the LTE connection.

vEdge-3# show bfd sessions | tab

SRC DST SYSTEM SITE LOCAL DETECT TX

SRC IP DST IP PROTO PORT PORT IP ID COLOR COLOR STATE MULTIPLIER INTERVAL UPTIME
TRANSITIONS
------------------------------------------------------------------------------------------------------------------
--------------
10.10.0.1 10.10.1.51 ipsec 12386 12366 15.1.1.1 15 mpls mpls up 7 1000 0:00:06:47
1

There aren't control connections as well.

 vEdge-3# show control connections | tab
#some columns are omitted for clarity

LOCAL LOCAL
PEER SITE PRIVATE PRIVATE PUBLIC SYSTEM LOCAL REMOTE PRIVATE PRIVATE
BEHIND
INSTANCE TYPE ID IP PORT PUBLIC IP PORT IP COLOR COLOR IP PORT STATE
UPTIME V ORG NAME PROXY
------------------------------------------------------------------------------------------------------------------
-----------------------------------------
0 vsmart 1 10.10.0.1 12386 10.1.1.30 12346 1.1.1.30 mpls default 10.1.1.30 12346 up
0:00:02:50 networkacademy-io No
0 vbond 0 10.10.0.1 12386 10.1.1.10 12346 0.0.0.0 mpls mpls 10.1.1.10 12346 up
0:00:02:51 networkacademy-io -
0 vmanage 1 10.10.0.1 12386 10.1.1.20 12546 1.1.1.20 mpls default 10.1.1.20 12546 up
0:00:02:34 networkacademy-io No

That is how simple it is to set up the Last Resort Circuit feature in Cisco SD-WAN. Now let's verify that the feature will work
when the primary transport is down.

Verification
To verify that the feature is working, we are going to shut down the primary tunnel and see whether an IPsec overlay will form
over the 4G.

4G/LTE

T3
Last-resort
circuit

T1 T2
INET
IPsec + BFD
vEdge-1 vEdge-3
Figure 5. How does the Last Resort Circuit work

We shut down the interface marked with the mpls color as it is shown in figure 5:

!
interface ge0/1
tunnel-interface
encapsulation ipsec
color mpls restrict
!
shutdown
!
!

Now if we check the BFD sessions, we can see that a session over the 4G/LTE TLOC has just come up.
 vEdge-3# show bfd sessions | tab

SRC DST SYSTEM SITE LOCAL DETECT TX

SRC IP DST IP PROTO PORT PORT IP ID COLOR COLOR STATE MULTIPLIER INTERVAL
UPTIME TRANSITIONS
------------------------------------------------------------------------------------------------------------------
---------------------
39.3.0.1 39.3.0.2 ipsec 12366 12366 15.1.1.1 15 lte public-internet up 7 1000
0:00:00:29 0

You can see that CIsco SD-WAN Last Resort Circuit is a very useful and flexible feature that can be easily deployed at remote
sites that use data-constrained WAN transports.

« Previous Lesson
WAN Edge Deployment
Next Lesson
Connection with legacy (non-sdwan) »
branches

1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Application Quality of Experience (AppQoE)

Interconnecting Multiple Clouds

Direct Internet Access (DIA)

RESTful APIs

3. Cisco SD-WAN Control Plane

Underlay vs Overlay Routing

Control Connections

OMP Overview

OMP Best-Path Selection

OMP Send Path Limit

OMP Send Backup Paths

OMP Source Preference

Understanding OMP Routing

OMP TLOC-Action

Chapter Key Takeaways

QUIZ - Control Plane Fundamentals

4. Cisco SD-WAN Data Plane

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / Connection with legacy (non-sdwan) branches

Connection with legacy (non-sdwan) branches

When an organization migrates its existing WAN network to SD-WAN, installing SD-WAN equipment at some sites is not always
possible. In such cases, migrated SD-WAN sites should be able to communicate with legacy branches over the underlay MPLS
network. Figure 1 illustrates this graphically.

Data Center /
Regional Hub

SD
ab WAN
l
no e to site
n-m co ss
igr m mu houl
ate nic d b
d b at e
ran e to
ch
Overlay es
INET fabric MPLS

SDWAN Non
site SDWAN
site

Figure 1. SDWAN sites communicating with non-migrated sites

An important design consideration here is whether the SDWAN branches need to communicate with the legacy sites directly
over the MPLS site or they should pass a centralized location such as a data center.
Both use cases are feasible options. However, in this lesson, we will show how to enable communication to legacy branches
via a regional data center.

The problem with user traffic in the underlay

In Cisco SD-WAN, the WAN interfaces and the LAN interfaces of a vEdge router belong to different VPNs. WAN interfaces are
always in the transport VPN0 which represents the underlay. On the other hand, LAN interfaces are always placed in a service
VPN. Therefore, data traffic cannot be easily routed between the site-local network and the underlay, because LAN and WAN
interfaces are isolated in different VPN segments.

Local breakouts can be used to force data traffic received on a service-side VPN to be forwarded to the underlay (VPN0).
However, it is challenging to forcibly transfer data traffic received in the underlay to a service-side VPN, as shown in figure 2
below. (Strictly speaking, it could be possible using NAT and Port Forwarding in VPN0)
 We can easily leak
traffic from VPN5
into the underlay

VPN 0

VPN 5 VPN 5
(ge0/5)
(ge0/0)
INET
VPN 0
(ge0/1)
MPLS
It is difficult to
transfer underlay
traffic into a VPN

Figure 2. Leaking traffic into the underlay

When facing this design task, many engineers start thinking that if we connect an interface belonging to Transport VPN 0 on
the LAN side, we can easily enable the communication between the service-side VPN and the underlay. However, it is not that
simple. According to the Cisco SD-WAN specification, in the transport VPN 0, packets received on a transport interface (TLOC-
enabled interface) cannot be forwarded to other interfaces . Transport interfaces only serve as tunnel endpoints for the overlay
fabric. Additionally, packets coming on other interfaces cannot be forwarded to a TLOC-enabled interface. Figure 3 illustrates
this graphically.

IPsec Tunnel
LAN VPN 5 VPN 0
(ge0/5) (ge0/0)
INET
IPsec Tunnel
VPN 0 VPN 0
(ge0/6) (ge0/1)
MPLS
Figure 3. Traffic to/from the underlay

The solution to this problem is to enable the local TLOC on a loopback interface instead of on a physical one, as shown in
figure 4 below. This allows the router to forward data traffic between the site-local network connected in VPN 0 and the WAN
underlay.
 IPsec Tunnel
LAN VPN 5 VPN 0
(ge0/5) (ge0/0)
INET
VPN 0 VPN 0
(ge0/6) (ge0/1)
IPsec Tunnel
IPsec Tunnel
MPLS
VPN 0
(Lo1)

Figure 4. Using Loopback as a local TLOC

You can learn about the different ways of configuring a loopback interfaces as a local TLOC in the following lesson.

Configuration Example
Now that we have covered the key point in this design challenge, let's go ahead and show an example in practice.

Initial Topology
Figure 5 shows the initial topology that we are going to use in this lab lesson. vEdge1 and 2 represent the data center, while
vEdges 3-6 are remote branches. The overlay topology is hub-and-spoke and the mpls color is restricted. Therefore, each
branch has one Internet and one MPLS tunnel to the hubs.

10.1.5.0/24 Core
VLAN5
Switch(es)
Controllers
VPN5 VPN5
Site-id 100 ge0/5 ge0/5
vEdge-1 vEdge-2
Data Center
1.1.1.1 1.1.1.2
(Hub)
Site-id 1 Site-id 1
aws ge0/0 ge0/1 ge0/0 ge0/1

INET MPLS
NAT
Colors: 39.3.0.0/16 10.10.0.0/16
10.10.3.0
biz-internet /30
mpls
Eth0/0
ge0/0 ge0/1 ge0/0 ge0/1 ge0/0 ge0/1 ge0/0 ge0/1
vEdge-3 vEdge-4 vEdge-5 vEdge-6
3.3.3.3 4.4.4.4 5.5.5.5 6.6.6.6
Site-id 3 Site-id 4 Site-id 5 Site-id 6 10.10.30.0/24
VPN5 VPN5 VPN5 VPN5
Legacy
10.3.5.0/24 10.4.5.0/24 10.5.5.0/24 10.6.5.0/24 Branch

Figure 5. Initial Topology

The goal of the configuration example would be to enable communication between VPN5 and the legacy branch.
Configuration Steps
Enabling the communication between sdwan branches and the legacy branch requires the following steps at a minimum:

1. Migrating the local MPLS TLOC to a loopback interface on both hub vEdges. It is currently configured on a physical
interface ge0/1.
2. On the core switches to advertise the legacy branches' subnets to VPN5. Any routing protocol can be used, including
static routing in VPN5 on the vEdges themselves. In this example, we will use static routing.
3. On the core switches to advertise the VPN5's IP range to the transport VPN0. In principle, any routing protocol can be
used. However, BGP is most commonly used so we will use BGP in the example as well.

Configuring Loopback TLOC

The first thing that we are going to do in order to enable communication to the legacy branches is to change the local TLOC
configuration on the mpls interfaces from the physical interface to a loopback interface. Currently, the local MPLS tloc of
vEdge-2 is configured as follows:

#vEdge-2
!
interface ge0/1
description WAN-MPLS
ip address 10.10.0.2/24
tunnel-interface
encapsulation ipsec
color mpls restrict
allow-service dhcp
allow-service dns
allow-service icmp
!
no shutdown
!

The local mpls tloc of the other vEdge router is configured the same way with just a different IP address.

Let's go ahead and migrate the local tloc configuration to a loopback interface. First, we will need to delete the tunnel-interface
configuration of the physical interface ge0/1 and configure it on a new loopback interface, as shown in the following output.
#on vEdge-2
!
interface ge0/1
no description WAN-MPLS
no tunnel-interface
!
interface loopback1
description WAN-MPLS
ip address 10.10.10.1/32
tunnel-interface
encapsulation ipsec
color mpls restrict
bind ge0/1
!
no shutdown

Now, we can see that the loopback is now a transport interface. Highlighted with orange is the local Internet tloc, and
highlighted with green is now the local MPLS tloc.
 vEdge-2# sh int | t

IF IF IF
TCP
AF ADMIN OPER TRACKER ENCAP
SPEED MSS RX TX
VPN INTERFACE TYPE IP ADDRESS STATUS STATUS STATUS TYPE PORT TYPE MTU HWADDR
MBPS DUPLEX ADJUST UPTIME PACKETS PACKETS
------------------------------------------------------------------------------------------------------------------
---------------------------------------------------
0 eth9 ipv4 - Down Down NA null service 1500 50:00:00:07:00:09 -
- 1416 - 0 0
0 ge0/0 ipv4 192.168.1.2/24 Up Up NA null transport 1500 50:00:00:07:00:01
1000 full 1416 0:00:21:22 11636 13888
0 ge0/1 ipv4 10.10.0.2/24 Up Up NA null service 1500 50:00:00:07:00:02
1000 full 1416 0:00:21:22 39949 45894
0 ge0/6 ipv4 10.10.2.2/24 Up Up NA null service 1500 50:00:00:07:00:07
1000 full 1416 0:00:21:22 93 78
0 ge0/7 ipv4 10.10.1.2/30 Up Up NA null service 1500 50:00:00:07:00:08
1000 full 1416 0:00:21:22 6149 10302
0 system ipv4 1.1.1.2/32 Up Up NA null loopback 1500 00:00:00:00:00:00
1000 full 1416 0:00:21:51 0 0
0 loopback1 ipv4 10.10.10.1/32 Up Up NA null transport 1500 00:00:00:00:00:00
1000 full 1416 0:00:21:40 0 0
3 ge0/3 ipv4 10.1.3.2/24 Up Up NA null service 1500 50:00:00:07:00:04
1000 full 1416 0:00:21:21 0 0
4 ge0/4 ipv4 10.1.4.2/24 Up Up NA null service 1500 50:00:00:07:00:05
1000 full 1416 0:00:21:21 0 0
5 ge0/5 ipv4 10.1.5.3/24 Up Up NA null service 1500 50:00:00:07:00:06
1000 full 1416 0:00:21:22 24 1307
6 ge0/2 ipv4 10.1.6.3/24 Up Up NA null service 1500 50:00:00:07:00:03
1000 full 1416 0:00:21:22 0 1279

The critical point now is that the loopback IP address should be reachable from the MPLS cloud; otherwise, bfd sessions to
remote branches won't come up. Therefore, we will need to advertise the loopback's IP address to the service provider. Since
we already had BGP peering with the SP, we only need to include a network command for the loopback IP, as shown below.

vpn 0
router
bgp 65001
address-family ipv4-unicast
network 10.10.10.1/32
!

Advertising VPN5's IP range to VPN0

Now that the physical interface connected to the MPLS service provider is not a local tloc, we need to focus on the routing
part. Figure 6 visualizes what we want to achieve. We want the core switches to route the traffic between VPN5 and the
underlay.
 Core switches route traffic
between VPN5 and VPN0.
They must have routing to both
VLAN5 VLAN1 VPN5's IP range and the legacy
branch’s IP range.
VPN5 VPN0 VPN5 VPN0
ge0/5 ge0/6 ge0/5 ge0/6

vEdge-1 Loo1 vEdge-2 Loo1

Data Center
ge0/0 ge0/1 ge0/0 ge0/1 (Hub)

INET MPLS
39.3.0.0/16 10.10.0.0/16
10.10.3.0
/30
ge0/0 ge0/1
Eth0/0

VPN5 VPN5 VPN5 VPN5 10.10.30.0/24

Legacy
SD-WAN Branches Branch

Figure 6. Routing traffic between VPN5 and the underlay

We first need to advertise the VPN5's IP ranges to the hub's VPN0 and the service provider. For this, we are going to configure
new interfaces in VPN0 on both vEdge 1 and 2 (ge0/6) and enable iBGP peering with the core switches:

#on vEdge-2:
vpn 0
interface ge0/6
ip address 10.10.2.2/24
no shutdown
!
router
bgp 65001
!
neighbor 10.10.2.254
no shutdown
remote-as 65001
!
!

Then on the data center core switches, we configure static routes for the sdwan branches and advertise them via iBGP in VPN0
to the hub routers.
 DC-CORE-SW# sh run | in ip route
ip route 10.3.5.0 255.255.255.0 10.1.5.1
ip route 10.4.5.0 255.255.255.0 10.1.5.1
ip route 10.5.5.0 255.255.255.0 10.1.5.1
ip route 10.6.5.0 255.255.255.0 10.1.5.1
!
DC-CORE-SW# sh run | s bgp
router bgp 65001
bgp log-neighbor-changes
network 10.3.5.0 mask 255.255.255.0
network 10.4.5.0 mask 255.255.255.0
network 10.5.5.0 mask 255.255.255.0
network 10.6.5.0 mask 255.255.255.0
redistribute connected
neighbor 10.10.2.2 remote-as 65001

Both vEdges 1 and 2 should now have routing for the sdwan branches in the transport VPN. Subsequently, the vEdges should
advertise the subnets to the MPLS service provider.

vEdge-2# sh ip route vpn 0 bgp

Codes Proto-sub-type:
IA -> ospf-intra-area, IE -> ospf-inter-area,
E1 -> ospf-external1, E2 -> ospf-external2,
N1 -> ospf-nssa-external1, N2 -> ospf-nssa-external2,
e -> bgp-external, i -> bgp-internal
Codes Status flags:
F -> fib, S -> selected, I -> inactive,
B -> blackhole, R -> recursive, L -> import

PROTOCOL NEXTHOP NEXTHOP NEXTHOP

VPN PREFIX PROTOCOL SUB TYPE IF NAME ADDR VPN TLOC IP COLOR
ENCAP STATUS
------------------------------------------------------------------------------------------------------------------
---------------------------
0 10.1.5.0/24 bgp i ge0/6 10.10.2.254 - - -
- F,S
0 10.3.5.0/24 bgp i - 10.1.5.1 - - -
- F,S,R
0 10.4.5.0/24 bgp i - 10.1.5.1 - - -
- F,S,R
0 10.5.5.0/24 bgp i - 10.1.5.1 - - -
- -
0 10.6.5.0/24 bgp i - 10.1.5.1 - - -
- F,S,R
0 10.10.30.0/24 bgp - ge0/1 10.10.0.254 - - -
- -

The last thing we need to do is to configure a static route on vEdge 1 and 2's VPN5 for the legacy branch's IP range and
advertise it via OMP to the sdwan branches. The static route will point to the core switches.

vpn 5
ip route 10.10.30.0/24 10.1.5.254
!
omp
advertise static
!

Figure 7 illustrates the routing information exchange that we have configured.

 Core
VLAN5 VLAN1 SWs
Static routes to
sdwan branches
Advertises the static

iBGP
routes via iBGP to

Stati
Static routes to
legacy branches VPN0.

Hub
VPN5 VPN0
vEdges

ic
Advertises the SDWAN
at

eBGP
St branches' IP ranges to
e

the MPLS provider

is
rt
ve
Ad

Overlay MPLS eB
Advertises legacy
ranges to the
GP
MPLS provider

VPN5 VPN5 VPN5 VPN5 10.10.30.0/24

Legacy
SD-WAN Branches Branch

Figure 7. Routing exchange between sdwan and legacy branches

Let's now go ahead and verify whether we now have IP reachability between the sdwan and the legacy branches.

Verification
If we perform a ping from any of the sdwan branches (for example vEdge-5) to the legacy branch, we will see that the ping is
successful.

vEdge-5# ping 10.10.30.1

Ping in VPN 0
PING 10.10.30.1 (10.10.30.1) 56(84) bytes of data.
64 bytes from 10.10.30.1: icmp_seq=1 ttl=254 time=16.6 ms
64 bytes from 10.10.30.1: icmp_seq=2 ttl=254 time=31.1 ms
64 bytes from 10.10.30.1: icmp_seq=3 ttl=254 time=7.78 ms
64 bytes from 10.10.30.1: icmp_seq=4 ttl=254 time=16.4 ms
64 bytes from 10.10.30.1: icmp_seq=5 ttl=254 time=19.4 ms
^C
--- 10.10.30.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 7.785/18.289/31.123/7.520 ms

To confirm the traffic path, let's perform a traceroute.

vEdge-5# traceroute vpn 5 10.10.30.1

Traceroute 10.10.30.1 in VPN 5
traceroute to 10.10.30.1 (10.10.30.1), 30 hops max, 60 byte packets
1 10.1.5.2 (10.1.5.2) 165.168 ms 173.491 ms 184.091 ms -->> vEdge-1 VPN5
2 10.1.5.254 (10.1.5.254) 186.506 ms 186.493 ms 186.478 ms -->> Core Switch
3 10.10.2.2 (10.10.2.2) 205.467 ms 207.155 ms 230.499 ms -->> vEdge-2 VPN0
4 10.10.0.254 (10.10.0.254) 231.957 ms 251.416 ms 252.944 ms -->> MPLS Cloud
5 10.10.3.2 (10.10.3.2) 252.977 ms * * -->> Legacy Branch
Figure 8 illustrates the traffic path of the traceroute above. You can that the routing between VPN5 and the underlay happens
on the data center core switches.

Core switches route traffic

VLAN5 VLAN1 between VPN5 and VPN0.

VPN5 VPN0 VPN5 VPN0

ge0/5 ge0/6 ge0/5 ge0/6

vEdge-1 Loo1 vEdge-2 Loo1

Data Center
ge0/0 ge0/1 ge0/0 ge0/1 (Hub)

ge0/0 ge0/1
Eth0/0

VPN5 VPN5 VPN5 VPN5 10.10.30.0/24

Legacy
SD-WAN Branches Branch

Figure 8. Traffic Path between VPN5 and Legacy Branches

Key takeaways
The key takeaways of this lesson are the following:

Transport interfaces (ones with tloc configuration) do not forward traffic. They only serve as tunnel endpoints for the
overlay IPsec/GRE tunnels.
We can route traffic between a service VPN and the underlay using external routing devices such as data center core
switches.
To route traffic between VPN0's interfaces connected to the LAN and to the WAN on a vEdge, we need to migrate the local
TLOC configuration to loopback interfaces. Then it becomes a matter of regular IP routing between the sdwan and legacy
branches.

« Previous Lesson
Last Resort Circuit
Next Lesson
TLOC Extension »
1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?

How Cisco SD-WAN works?

Cisco SD-WAN Main Principles

2. Key features of Cisco SD-WAN

Cisco CCIE Enterprise Infrastructure Practical Exam Learning Path / Cisco SD-WAN Deep-Dive / TLOC Extension

TLOC Extension
In real-world deployments, there will always be cases where the WAN edge routers cannot be directly connected to each
available WAN transport as is shown in figure 1. In the example on the left, only one WAN edge device is connected to a single
transport. This significantly reduces the overall availability and creates inefficiencies in the overlay fabric. TLOC-extension
feature is designed to overcome this problem by extending the WAN transports to both SD-WAN routers without requiring
direct attachment to both service provider clouds.

INET MPLS INET MPLS

DSL Leased Line

Switch Switch

Local Networks Local Networks

Figure 1. WAN Edges with a Single WAN Trasnport

Of course, the SD-WAN routers can connect to each WAN through front-facing switches as shown on the right but this is not a
recommended approach because it adds additional costs and results in having another device to operate.

What is TLOC Extension?

TLOC extension is a feature that allows a WAN Edge router to communicate over the WAN transport connected to
the adjacent WAN Edge router through a TLOC-extension interface . In the example shown in figure 2, vEdge-1 is directly
connected to the Internet and also uses the TLOC extension feature to connect to the MPLS transport via vEdge-2. In the end,
vEdge-1 has an overlay tunnel built over both WAN clouds even though it is directly attached only to one of the WAN transports.
On the opposite side, vEdge-2 is attached to the MPLS cloud and uses the TLOC extension feature to connect to the Internet via
WAN Edge 1.
 INET MPLS

Directly Connected

vEdge-1 vEdge-2

Local Networks
Figure 2. TLOC Extension with Directly Connected vEdges

The feature is set up in a per-interface manner and provides transparent connectivity from one interface (called a TLOC
extension interface) to a particular WAN transport. If we take vEdge-1 for example, it is unaware that the blue tunnel to the
MPLS cloud passes through another WAN edge device because vEdge-2 extends their directly connected interface
transparently to the MPLS transport.

TLOC Extension Types

Cisco SD-WAN allows for multiple ways of implementing TLOC extensions on WAN edge routers. The most typical and
straightforward way of extending the transports is by using directly connected interfaces between the vEdges. However,
depending on the local site design, the SD-WAN routers can be connected through Layer 2 switches as is illustrated in figure
3. L2 TLOC-extensions describe extensions between vEdges that are Layer2-adjacent are situated in the same broadcast
domain/the same subnet.
 INET MPLS

vEdge-1 vEdge-2

L2 Switching
Figure 3. TLOC Extension via L2

The SD-WAN routers can also be connected at Layer 3 via any sort of IP routing. L3 TLOC-extensions describe extensions
between two vEdges that are separated by an L3 device and are situated in different IP subnets. L3 TLOC extensions are
implemented using GRE tunnels.
 INET MPLS

vEdge-1 vEdge-2

L3 Routing
Figure 4. TLOC Extension via L3

I'd also like to point out that TLOC-extensions can be configured using physical interfaces such as Ge0/0, Eth0, etc., and also
using L3 subinterfaces.

Notable Limitations
At this point, it is a good time to point out some notable limitation of this feature:

As you might already know, TLOCs are only supported on routing interfaces. Well, the same applies to TLOC-extension
interfaces as well. They are only supported on L3 routed interfaces. Switchports and SVIs cannot be configured as
WAN/Overlay ports and can only be used within the service VPNs.
LTE can not be used as a TLOC extension interface between WAN Edge routers.
L3 TLOC-extension isn't supported on Viptela vEdge routers but only on SD-WAN devices running IOS-XE.
The feature would not work on transport ports that are bound to the loopback tunnels.

Configuring TLOC Extensions

Routing Considerations
A TLOC extension interface is always configured in VPN 0 and has an IP address assigned. Then the WAN interface to which it
is bound is specified. For example, the vEdge-1's extension interface is ge0/4 and is bound to the Internet through ge0/0 and
vEdge-2's extension interface is ge0/5 and is bound to the MPLS cloud through ge0/1. Then static default routes are
configured in VPN0 on each WAN Edge router, pointing to the adjacent vEdge's IP as a next-hop.

However, some IP reachability considerations must be done in order for the overlay tunnels and BFD sessions to come up with
remote peers over the TLOC extension interfaces. For example, to reach the MPLS cloud, vEdge-1 must be configured with a
default route pointing to Ge0/5 of vEdge-2. Another very important part is the reverse IP reachability. To make sure that
packets can be routed back to the TLOC extension interface, WAN Edge 2 must advertise subnet B to the MPLS provider. This
can be done using any dynamic routing protocol that the service provider is willing to run. In a typical production deployment,
some sort of inbound route policy is applied to deny all incoming dynamic routes from the provider since there is a static route
pointing to the MPLS cloud.
 INET MPLS
NAT Advertise
T11 Subnet A Subnet B T22
T21
Ge0/0 Ge0/1

Ge0/4 Subnet A Ge0/4

Transport TLOC static 0.0.0.0/0 Transport
Extension
VPN 0 VPN 0
Ge0/5 Subnet B Ge0/5 TLOC
static 0.0.0.0/0 Extension

T12

Service Service
VPNs 1-511 VPNs 1-511

WAN Edge router 1 WAN Edge router 2

Figure 5.TLOC Extension via L3

In the case of WAN edge router 2, to reach the Internet cloud, it must be configured with a default route pointing to vEdge-1’s
ge0/4 IP address. In a typical production deployment, subnet-A would be from the RFC1918 address space and NAT must be
utilized on WAN edge-1 towards the INET cloud in order to ensure that traffic can be routed back from the Internet to vEdge-
2 over the extension interface.

Let's go through a configuration example and see how this feature is implemented.

Extending vEdge-1 to the MPLS

Let's first extend router-1 to the MPLS cloud. For this example, we are going to use the private subnet 10.51.1.0/30 between
Ge0/5 interfaces of both routers. This prefix 10.51.1.0/30 must also be advertised to the MPLS provider in order to have IP
reachability in the other direction. For this purpose, we are going to utilize BGP.
 INET MPLS
Advertise
T11
10.51.1.0/30 10.50.2.1
Ge0/0
via BGP Ge0/1

vEdge-1 vEdge-2
Ge0/5 10.51.1.0/30 Ge0/5

vpn 0 vpn 0
interface ge0/5 T12 interface ge0/5
ip address 10.51.1.1/30 ip address 10.51.1.2/30
tunnel-interface tloc-extension ge0/1
encapsulation ipsec no shutdown
color mpls restrict !
!
ip route 0.0.0.0/0 10.51.1.2

Configuring vEdge-1

It is important to understand that in order to extend vEdge-1 to the MPLS cloud, the actual TLOC extension happens on the
adjacent router. From the perspective of router-1, Ge0/5 is connected to the MPLS provider in the same way as it would have
been with a direct attachment link. Therefore, there is nothing special in the configuration on the vEdge-1 as you can see below.
The interface is configured with IP address, color, and encapsulation type and a default route is specified. That's about it.

!
vpn 0
!
interface ge0/5
ip address 10.51.1.1/30
tunnel-interface
encapsulation ipsec
color mpls restrict
allow-service all
!
no shutdown
!
ip route 0.0.0.0/0 10.51.1.2
ip route 0.0.0.0/0 50.1.1.2
!

The "magic" happens on the neighboring router-2. In there, we should configure two things. The first one is to tell interface
Ge0/5 that it extends the MPLS cloud attached to Ge0/1. The other important thing is to advertise the subnet 10.51.1.0/30
toward the service provider using BGP.
 !
vpn 0
!
interface ge0/1
ip address 10.50.2.1/30
tunnel-interface
encapsulation ipsec
color mpls restrict
allow-service all
!
no shutdown
!
interface ge0/5
ip address 10.51.1.2/30
tloc-extension ge0/1
no shutdown
!
ip route 0.0.0.0/0 10.50.2.2
!
router
bgp 64551
address-family ipv4-unicast
network 10.51.1.0/30
!
neighbor 10.50.2.2
no shutdown
remote-as 65051
address-family ipv4-unicast
!
!
!

Once both steps are configured, we can see that vEdge-1 has a valid TLOC marked with the mpls color that has a BFD session
in UP state even though the router is not directly connected to the MPLS cloud.
 vEdge-1# show omp tlocs
---------------------------------------------------
tloc entries for 50.50.50.50
mpls
ipsec
---------------------------------------------------
RECEIVED FROM:
peer 0.0.0.0
status C,Red,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
attribute-type installed
encap-key not set
encap-proto 0
encap-spi 256
encap-auth sha1-hmac,ah-sha1-hmac
encap-encrypt aes256
public-ip 10.51.1.1
public-port 12406
private-ip 10.51.1.1
private-port 12406
public-ip ::
public-port 0
private-ip ::
private-port 0
bfd-status up
domain-id not set
site-id 50
overlay-id not set
preference 0
tag not set
stale not set
weight 1
version 2
gen-id 0x80000019
carrier default
restrict 1
groups [ 0 ]
border not set
unknown-attr-len not set

Extending vEdge-2 to the Internet

Let's now extend WAN edge 2 to the Internet provider. The configuration is simpler than the MPLS one because we do not need
to advertise the interconnecting subnet 192.168.51.0/30 to the Internet provider, but only to NAT it when leaving the transport
interface toward the provider.
 INET MPLS
NAT
192.168.51.2 T22
toward INET
Ge0/0 Ge0/1

vEdge-1 vEdge-2
Ge0/4 192.168.51.0/30 Ge0/4

vpn 0 vpn 0
nat interface ge0/4 T21
! ip address 192.168.51.2/30
interface ge0/4 tunnel-interface
ip address 192.168.51.1/30 encapsulation ipsec
tloc-extension ge0/0 color public-internet
no shutdown !
! ip route 0.0.0.0/0 192.168.51.1

Configuring vEdge-2

All required configuration is shown in the diagram above. Once the config is applied, we can see that vEdge-2 has a valid TLOC
marked with the public-internet color. Note that in the TLOC route, we can see the private-public IP address pair.
 vEdge-2# show omp tlocs
---------------------------------------------------
tloc entries for 50.50.50.51
public-internet
ipsec
---------------------------------------------------
RECEIVED FROM:
peer 0.0.0.0
status C,Red,R
loss-reason not set
lost-to-peer not set
lost-to-path-id not set
Attributes:
attribute-type installed
encap-key not set
encap-proto 0
encap-spi 256
encap-auth sha1-hmac,ah-sha1-hmac
encap-encrypt aes256
public-ip 50.1.1.1
public-port 12366
private-ip 192.168.51.2
private-port 12366
public-ip ::
public-port 0
private-ip ::
private-port 0
bfd-status up
domain-id not set
site-id 50
overlay-id not set
preference 0
tag not set
stale not set
weight 1
version 2
gen-id 0x80000014
carrier default
restrict 0
groups [ 0 ]
border not set
unknown-attr-len not set

If we now check whether there is a BFD session over that color, you can see that there is one is UP state sourced from the
TLOC extension interface 192.168.51.2 to a public IP address 80.1.1.1.

vEdge-2# show bfd sessions

SOURCE TLOC REMOTE TLOC DST PUBLIC
SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP
-------------------------------------------------------------------------------------
80.80.80.80 80 up mpls mpls 10.50.2.1 10.80.1.1
80.80.80.80 80 up public-internet public-internet 192.168.51.2 80.1.1.1

« Previous Lesson
Connection with legacy (non-sdwan)
Next Lesson
Cisco SD-WAN Management Plane »
branches

1. Introduction to SD-WAN
Why do we need SD-WAN?

What is SD-WAN?