VXLAN BGP EVPN: Technology Building Blocks

VXLAN BGP EVPN: TECHNOLOGY
BUILDING BLOCKS.
UNDERLAY / OVERLAY / IP FABRIC /VXLAN / EVPN /MULTI-TENANT
Jide Akintola
jide@xpresspath.net
27/02/2019

AGENDA
Ø VxLAN Overview and Configuration
Ø EVPN Overview and Configuration
Ø Underlay Configuration Walk through
Ø Overlay Configuration Walk through
Ø EVPN VxLAN Service Configuration Walk through
Ø Sample Legacy Device Migration to VxLAN BGP EVPN

DATA CENTER TECHNOLOGY
Sample Vendors’ Supported Options
L2 + STP + L3 + RVI
MC-LAG
QFabric
Virtual Chassis Fabric
CLOS: 3 / 5 -Stage
VXLAN + EPVN Fabric
Traditional Ethernet Fabric IP Fabric
VCP /VCP+ ACI
Virtual Chassis

VXLAN ACRONYMS
Ø VXLAN - Virtual eXtensible Local Area Network
Ø VNI - VXLAN Network Identifier (or VXLAN Segment ID)
Ø VXLAN Segment - VXLAN Layer 2 overlay network over which VMs
communicate.
Ø VTEP - VXLAN Tunnel End Point. An entity that originates and/or
terminates VXLAN tunnels.
Ø VXLAN Gateway - an entity that forwards traffic between VXLANs.

VXLAN OVERVIEW
Ø VxLAN is a Layer 2 overlay scheme over an existing Layer 3
network infrastructure.
Ø An overlay network is used to carry the MAC traffic from the
individual VMs/host in an encapsulated format over a logical
stateless "tunnel".
Ø With VxLAN overlay network, the original packet is encapsulated on
the ingress device with an outer header before being forwarded to
the egress device. All intermediate devices simply forward the
encapsulated packet based on the outer header and are not aware
of the original packet payload. At the egress device, the
encapsulated packet header is removed and the original packet is
forwarded based on the inner payload.

VXLAN OVERVIEW
Ø Each overlay is termed a VXLAN segment. Only VMs/hosts within
the same VXLAN segment can communicate with each other.
Ø Each VXLAN segment is identified through a 24-bit segment ID,
termed the "VXLAN Network Identifier (VNI)". This allows up to 16
Million VXLAN segments to coexist within the same administrative
domain.
Ø The VNI is in an outer header that encapsulates the inner MAC
frame originated by the VM/host, hence providing traffic isolation
while allowing for overlapping MAC addresses across different VNI.
Ø The underlay network on the contrary, is a transport network that
provides network reachability between the ingress and egress
overlay devices.

VXLAN NETWORK OVERLAY
Underlay Network
Overlay Tunnels

VXLAN OVERVIEW – UDP WHY?
Ø VXLAN uses UDP encapsulation to take advantage of the load
balancing in the network.
Ø The UDP source port can be set to the hash of inner packet fields and
the UDP destination port is set to the 4789
Ø Setting the UDP source port as packet hash allows for load balancing
of the packets using 5-tuples.
Ø The existing IP network infrastructure supports this and no changes are
required to support VXLAN in the network

VXLAN VTEP PEER DISCOVERY
Ø The vanilla implementation of VxLAN has no mechanism for VTEP peer
auto-discovery but rather relies on manual definition of those Vxlan
overlay edge devices as part of the device configuration. EVPN is used
to address this shortcoming.

VXLAN END-HOST DEVICES DISCOVERY
Ø Similar to VPLS, the original implementation of VxLAN relies on the
data plane flood and learn (F&L) discovery scheme.
Ø To however address the scalability concern of F&L discovery
scheme, other controller-less control plane discovery scheme such
as BGP EVPN and OVSDB have been defined. It is also worth
noting that other SDN controller-based discovery scheme such as
Cisco APIC or Juniper Contrail can also be used.

VXLAN AND MULTICAST TRAFFIC
Ø The original VxLAN implementation mandated the underlay network
to support native IP multicast for forwarding BUM (broadcast,
unknown unicast & multicast) traffic.
Ø Layer 2 VNI is mapped to an IP multicast group address, VTEP then
sends out PIM Join/Prune message expressing interest in the
multicast traffic. Network does the replication.
Ø Newer software from all vendors now support Ingress Replication
(IR) or Head-End Replication (HER), eliminating the need for the
underlay to support native IP multicast.
Ø With HER, the ingress router builds a flood list which basically
specifies all remote VTEPs to replicate the BUM traffic to.

VXLAN PACKET HEADER AND ENCAPSULATION
OUTER
MAC
OUTER
IP
OUTER
UDP
VXLAN
Header
F
C
S
Original L2 Frame
Reserved
VXLAN Network Identiﬁer (VNI) Reserved
R R R R I R R R
Flag
The I flag is set to 1 for
a valid VNI. R flag are
reserved and must be
set to 0.
50 Bytes (14+20+8+8) of additional overhead added.

VXLAN – SAMPLE CONFIGURATION ARISTA
!
hostname aris-lf1
!
vlan 20
name vla20
!
interface Ethernet3
switchport trunk allowed vlan
10,20,30,40
switchport mode trunk
!
interface Loopback0
ip address 10.1.1.3/32
!
interface Vxlan1
vxlan source-interface Loopback0
vxlan udp-port 4789
vxlan vlan 20 vni 10020
vxlan vlan 20 flood vtep 10.1.1.4
!
!
hostname aris-lf2
!
vlan 20
name vla20
!
interface Ethernet5
switchport trunk allowed vlan 20
!
interface Loopback0
ip address 10.1.1.4/32
!
interface Vxlan1
vxlan udp-port 4789
vxlan vlan 20 flood vtep 10.1.1.3
!
Leaf 1 Leaf 2

!
hostname CLIENT1
!
!
vlan 20
name vla20
!
interface Ethernet1
switchport trunk allowed vlan 10,20
!
!
interface Vlan20
ip address 20.20.20.1/24
!
ip routing
!
!
hostname CLIENT3
!
!
vlan 20
name vla20
!
!
interface Ethernet2
switchport trunk allowed vlan 20,30
!
!
interface Vlan20
ip address 20.20.20.3/24
!
ip routing
!
VXLAN – SAMPLE CONFIGURATION ARISTA

VXLAN – SAMPLE OUTPUTS
aris-lf1#sh ip route 10.1.1.4
VRF: default
Codes: C - connected, S - static, K - kernel,
O - OSPF, IA - OSPF inter area, E1 -
OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF
NSSA external type 1,
N2 - OSPF NSSA external type2, B I -
iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS
level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O -
OSPF Summary,
NG - Nexthop Group Static Route, V -
VXLAN Control Service,
DH - Dhcp client installed default route
O 10.1.1.4/32 [110/30] via 10.10.10.4,
Ethernet2
aris-lf1#
aris-lf2#sh ip route 10.1.1.3
VRF: default
O - OSPF, IA - OSPF inter area, E1 -
OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF
NSSA external type 1,
N2 - OSPF NSSA external type2, B I -
iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS
level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O -
OSPF Summary,
NG - Nexthop Group Static Route, V -
VXLAN Control Service,
O 10.1.1.3/32 [110/30] via 10.10.10.8,
Ethernet2
aris-lf2#

CLIENT3#ping 20.20.20.1
PING 20.20.20.1 (20.20.20.1) 72(100)
bytes of data.
80 bytes from 20.20.20.1: icmp_seq=1
ttl=64 time=212 ms
ttl=64 time=216 ms
ttl=64 time=228 ms
ttl=64 time=248 ms
ttl=64 time=244 ms
--- 20.20.20.1 ping statistics ---
5 packets transmitted, 5 received, 0%
packet loss, time 864ms
rtt min/avg/max/mdev =
212.013/229.614/248.016/14.451 ms,
pipe 2, ipg/ewma 216.013/221.817 ms
CLIENT3#
CLIENT1#ping 20.20.20.3
PING 20.20.20.3 (20.20.20.3) 72(100)
bytes of data.
ttl=64 time=200 ms
ttl=64 time=220 ms
ttl=64 time=248 ms
ttl=64 time=260 ms
ttl=64 time=268 ms
--- 20.20.20.3 ping statistics ---
5 packets transmitted, 5 received, 0%
packet loss, time 824ms
rtt min/avg/max/mdev =
200.013/239.215/268.017/25.476 ms,
pipe 2, ipg/ewma 206.013/221.345 ms
CLIENT1#

aris-lf1#sh vxlan vtep
Remote VTEPS for Vxlan1:
10.1.1.4
Total number of remote VTEPS: 1
aris-lf1#
aris-lf1#sh vxlan address-table
Vxlan Mac Address Table
------------------------------------------------------
----------------
VLAN Mac Address Type Prt
VTEP Moves Last Move
---- ----------- ---- --- ---- -----
---------
20 5000.00d7.ee0b DYNAMIC Vx1
10.1.1.4 1 0:01:01 ago
Total Remote Mac Addresses for this
criterion: 1
aris-lf1#
aris-lf2#sh vxlan vtep
Remote VTEPS for Vxlan1:
10.1.1.3
Total number of remote VTEPS: 1
aris-lf2#
aris-lf2#sh vxlan address-table
------------------------------------------------------
----------------
VLAN Mac Address Type Prt
VTEP Moves Last Move
---- ----------- ---- --- ---- -----
---------
20 5000.00af.d3f6 DYNAMIC Vx1
10.1.1.3 1 0:02:02 ago
Total Remote Mac Addresses for this
criterion: 1
aris-lf2#

EVPN ACRONYMS
Ø EVPN - Ethernet VPN.
Ø EVI - EVPN Instance. An EVPN instance spanning the
Provider Edge (PE) devices participating in that EVPN.
Ø MAC-VRF - A Virtual Routing and Forwarding table for
Media Access Control (MAC) addresses on a PE.
Ø IP-VRF - A Virtual Routing and Forwarding table for
Internet Protocol (IP) addresses on a PE.
Ø DF – Designated Forwarder.

EVPN ACRONYMS
Ø ES - Ethernet Segment. When a customer site (device or
network) is connected to one or more PEs via a set of Ethernet
links, then that set of links is referred to as an ’Ethernet
segment’.
Ø VTEP - VXLAN Tunnel End Point. An entity that originates
and/or terminates VXLAN tunnels.
Ø NVE - Network Virtualization Edges (same as a PE/VTEP).
Ø NVGRE - Network Virtualization using Generic Routing
Encapsulation.

EVPN OVERVIEW
Ø While the VxLAN draft defines an extensible data plane for
virtual networks, a control plane was never not specified.
The implication of this is that, the vanilla VxLAN
implementation relies on the data plane flood and learn
(F&L) approach, leading to scalability concern.
Ø EVPN was develop to address the above limitation in
VxLAN.
Ø EVPN technology is also used within the data center to
offer multi-tenancy.

EVPN OVERVIEW
Ø In an EVPN, MAC learning between PEs occurs not in the
data plane as was the case in VPLS but rather in the control
plane. Data plane MAC learning in EVPN is limited to PE-CE
link only.
Ø Data plane learning requires the flooding of unknown unicast
and Address Resolution Protocol (ARP) frames, whereas,
the control plane learning eliminates flooding.
Ø Moreover, control plane information is distributed with MP-
BGP which allows for auto-discovery of PE devices
participating in a given EVPN instance.

ADVANTAGES OF EVPN
ØImproved network efficiency and Scalability
ØReduced unknown-unicast flooding due to control-plane MAC
learning.
ØMulti-path traffic over multiple spine switches.
ØMulti-path traffic to active / active dual-homed server.
ØDistributed layer-3 gateway.
ØVery scalable MP-BGP-based control plane.
ØImproved Network Convergence
ØFaster re-convergence when link to dual-homed server fails (mass-
withdrawal).

EVPN DATA PLANE OPTIONS – IP / MPLS
Ø The following data-plane encapsulation are defined and supported
with EVPN
Value Name
8 VXLAN Encapsulation
9 NVGRE Encapsulation
10 MPLS Encapsulation
11 MPLS in GRE Encapsulation
12 VXLAN GPE Encapsulation

EVPN DATA PLANE ENCAP– IP / MPLS
Transport Label Service Label PayloadMPLS
Outer IP Header VXLAN VNID PayloadVXLAN
Ø Both VXLAN and NVGRE are examples of technologies that provide
a data plane encapsulation which is used to transport a packet over
native IP infrastructure.
Outer IP Header NVGRE VSID PayloadNVGRE

EVPN-VXLAN
Ø Multiprotocol Border Gateway Protocol Ethernet Virtual Private
Network (MP-BGP EVPN) is used as the control plane for VXLAN.
Ø It provides VTEP peer discovery and end-host reachability information
distribution.
Ø It allows more scalable VXLAN overlay network designs suitable for
private and public clouds.
Ø The MP-BGP EVPN control plane introduces a set of features that
reduces or eliminates traffic flooding in the overlay network and
enables optimal forwarding for both west-east and south-north traffic.

ØThe current EVPN service model otherwise known as the deployment
scenarios specifies different ways of how VLAN-to-VNI Mapping can be
achieved. The following three service models are defined:
1. VLAN-Based Service Interface
2. VLAN Bundle Service Interface / Port-Based Service Interface
3. VLAN-Aware Bundle Service Interface
ØMost vendors however, only support option 1 and 3 from the list above
though.
EVPN SERVICE MODEL – VLAN-TO-VNI MAPPING

EVPN SERVICE MODEL – VLAN-to-VNI MAPPING
• VLAN-Based Service Interface:
Ø Has a one-to-one mapping between a VLAN ID (VID) on the
interface and a MAC-VRF. Also, the EVPN instance consists of only
a single broadcast domain.
• VLAN Bundle Service Interface:
ØHas a many-to-one mapping between VLANs and a MAC-VRF, and
the MAC-VRF consists of a single bridge table. Also, the EVPN
instance corresponds to multiple broadcast domains.

EVPN SERVICE MODEL – VLAN-to-VNI MAPPING
• VLAN-Aware Bundle Service Interface:
Ø Here EVPN instance consists of multiple broadcast domains with
each VLAN having its own bridge table.

EVPN SERVICES MODEL SUMMARY
Attribute VLAN-Based Service VLAN Bundle Service VLAN Aware Service
VLAN to EVPN Instance Ratio 1:1 N:1 N:1
Route Target VLAN VRF VRF
Service Label VLAN VRF VLAN
VLAN Normalization Yes No Yes
Overlapping MAC Addresses Yes No Yes

EVPN ROUTE TYPES
Route Type Description Usage
1 Ethernet Auto-Discovery PE Discovery and Mass
Withdraw
2 MAC Advertisement MAC Advertisement
3 Multicast Route BUM Flooding
4 Ethernet Segment Route ES Discovery and DF
Election
5 IP Prefix Route IP Route Advertisement
ØThere is no change to the encoding of the original EVPN routes to support
VXLAN or NVGRE data-plane encapsulation.
ØIn order to indicate which type of data-plane encapsulation is to be used,
the BGP encapsulation extended community is included with all EVPN
routes to signify which data-plane encapsulation is in used.

EVPN ROUTE TYPES FORMAT – TYPE 1
ØAn Ethernet Tag ID is a 32-bit field containing either a 12-bit or 24-bit
identifier that identifies a particular broadcast domain for instance, a
VLAN in an EVPN instance.
Route Distinguisher (RD) (8 octets)
Ethernet Segment Identiﬁer (10 octets)
Ethernet Tag ID (4 octets)
MPLS Label / VNI (3 octets)
Ø Also known as Ethernet Auto-
Discovery Route (Ethernet A-D per
ESI and Ethernet A-D per EVI)
Ø Used for remote VTEP auto-
discovery.
Ø Used for advertising split-horizon
label
Ø Also provides for fast convergence
through mass withdrawal

MAC Address Length (1 octet)
IP Address (0, 4, or 16 octets)
MPLS Label 2 (0 or 3 octets)
MAC Address (6 octets)
IP Address Length (1 octet)
MPLS Label 1 / VNI Field (3 octets)
Ø Also known as MAC/IP
Advertisement Route
Ø Used to provides end-host
reachability information.

Originating Router's IP Address (4 or 16 octets)
Ø Also known as Inclusive
Multicast Ethernet Tag (IMET)
Route
Ø Used to create the distribution
list for ingress replication.
Ø Used to set up paths for BUM
traffic per VLAN per EVI basis.
Ø Used to discover the multicast
tunnels among the endpoints
associated with a given EVI.
This route is tagged with the
PMSI Tunnel attribute, which
is used to encode the type of
multicast tunnel to be used
The following PMSI Tunnel attribute types are supported for
VXLAN/NVGRE encapsulation:
Ø PIM-SSM Tree
Ø PIM-SM Tree
Ø BIDIR-PIM Tree
Ø Ingress Replication

Ø Also known as Ethernet
Segment Route
Ø Used for Ethernet Segment
auto-discovery by allowing
VNE with the same ESI to
discover each other.
Ø It also allows for designated
forwarder (DF) election.
Originating Router's IP Address (4 or 16 octets)

Ø Also known as IP Prefix Route
Ø Used to decouple IP Prefix
from MAC/IP route to provide
IP prefix advertisement.
IP Preﬁx Length (1 octet)
Ethernet Segment ID (ESI) (10 octets)
MPLS label / VNI (3 octets)
IP Preﬁx (4 or 16 octets)
Gateway IP Address (4 or 16 octets)

DESIGNATED FORWARDER (DF)
Ø The designated forwarder (DF) is the NVE / PE router responsible for sending broadcast, unknown
unicast and multicast (BUM) traffic to multi-homed CE on a particular Ethernet Segment (ES) within
a given VLAN.
Ø The original DF election process elects a DF per <ES, EVI> and uses the following election
algorithm: each PE that is multi-homed to a given Ethernet Segment builds an ordered list of the IP
addresses of all the PE nodes connected to the Ethernet segment including itself in a numerical
ascending order starting from zero. Each IP address in the list is then assigned an ordinal number
based on its position in the list. The ordinal number starts from zero with value zero assigned to the
PE that has the least IP address. Then given a total of N PEs multi-homed to the same Ethernet
segment, the PE's with the ordinal number “o” is the DF if (VLAN-ID mod N == o) where
VLAN-ID is the “dividend” and N is the “divisor” and “mod” is Modulo and “o” is the “remainder” in
the formula.
Ø To ensure that the service is evenly carved, the above original DF election algorithm however
assumes Ethernet tag are uniformly distributed between odd and even VLAN/Ethernet Tag values.
Hence for cases where this uniformity does not exist, such as if all VLAN ID are odd numbers or all
VLAN ID are even numbers then no DF load balancing happens and one of the PE never gets
elected at all.

DESIGNATED FORWARDER
Ø Example assuming we have two PEs (PE0 and PE1) connected to the same Ethernet Segment,
meaning N=2, then assume again that all the VLAN IDs are even as follows, 4, 34, 44, 88; In this
case applying the DF default election algorithm PE's with the ordinal number “o” is the DF if
(VLAN-ID mod N == o) ==> (4 mod 2 == 0; 34 mod 2 == 0; 44 mod 2 == 0; 88 mod 2 == 0). As can
be seen PE0 would always be elected as the DF for all these even VLAN IDs with the default DF
algorithm, hence defeating the service carving notion.
Ø The proposed updated DF election process is defined in “draft-ietf-bess-evpn-df-election-framework-
09” and it elects a DF per <ES, BD> as oppose to the default DF election method of <ES, EVI> .
Ø The new DF election algorithm is based on Highest Random Weight (HRW) Algorithm that allows for
fair load distribution, avoidance of needless service disruption, redundancy and fast access.

EVPN ROUTE EXTENDED COMMUNITY– MAC MOBILITY
Ø Advertised along with MAC/IP
advertisement routes
Ø The sequence number is used
to ensure that PEs retain the
correct MAC/IP Advertisement
route when multiple updates
occur for the same MAC
address.
Ø PE increments sequence
number.
Ø PE with highest sequence
number wins.
Ø If a tie occurs, highest router-id
flushes its cache
Type Sub-Type
Sequence Number
Flags (1 Octect) Reserved

EVPN ROUTE EXTENDED COMMUNITY– ES-IMPORT RT
Ø Transitive Route Target
extended community carried
with the Ethernet Segment
route ES Type 4 route.
Ø Enables all the PEs connected
to the same multi-homed site to
import the Ethernet Segment
routes. Hence limiting the
scope of the ES route to the
multi-homed segment.
Type Sub-Type
ES-Import Cont'd
ES-Import

EVPN ROUTE EXTENDED COMMUNITY– ESI LABEL
Ø Advertised with the Ethernet
Auto-discovery routes
Ø Used for split-horizon filtering
in multi-home sites and used to
encode the split-horizon label.
Ø It is also used to indicate
whether an ES segment is
operating in Single-Active, or
All-Active redundancy mode.
Type Sub-Type
ESI Label
Flags (1 Octect) Reserved
Reserved

• Split Horizon Operation
Ø In EVPN with MPLS encapsulation setup, an MPLS label is used for split-horizon filtering to
support all-active multi-homing where an ingress NVE adds an ESI label corresponding to
the site of origin when encapsulating the packet.
Ø The egress NVE checks the ESI label when attempting to forward a multi-destination frame
out an interface, and if the label corresponds to the same site identifier (ESI) associated with
that interface, the packet gets dropped. This prevents the occurrence of forwarding loops on
that segment.
Ø With VXLAN or NVGRE encapsulation however, there is no concept of labels, hence every
NVE tracks the IP address associated with the other NVE with which it has shared multi-
homed ESs.
Ø When the NVE receives a multi-destination frame from the overlay network, it examines the
source IP address in the tunnel header and filters out the frame on all local interfaces
connected to ESs that are shared with the ingress NVE.

• Split Horizon Operation
Ø It is also worth noting that with VXLAN or NVGRE encapsulation, the ingress VNE is "Locally
Biased", meaning that the ingress NVE performs replication locally to all directly attached
Ethernet segments regardless of the DF election state for all flooded traffic ingress from the
access interfaces.

EVPN MASS WITHDRAW – FAST CONVERGENCE
All-Active Mode
LAG
Ø PE withdraws the set of Ethernet A-D per ES
routes. This triggers all PEs that receive the
withdrawal to update their next-hop
adjacencies for all MAC addresses associated
with the Ethernet segment in question.
Ø PE then withdraws all MAC addresses
associated with the Ethernet Segment (ES) L3 L4
L5
L6L2
L1

EVPN MAC ALIASING
MAC learned
MAC not learned
Ø Aliasing improves load-balancing by
allowing remote VNEs to continue to
load-balance traffic evenly though they
have only received a single MAC/IP
from a single ingress VNE.
Ø Aliasing is define as the ability of a PE to signal
that it has reachability to an EVPN instance on
a given ES even when it has learned no MAC
addresses from that EVI/ES.
Ø Aliasing uses the Ethernet A-D per EVI type 1 routes
Ø A remote PE that receives a MAC/IP Advertisement
route with a non-reserved ESI would consider the
advertised MAC address to be reachable via all PEs
that have advertised reachability to that MAC
address EVI/ES via the Ethernet A-D per EVI route.
L4
L3
L2
L1

DISTRIBUTED ANYCAST GATEWAY
Server1
S1 S2
L2L1 L3
.1/24
.1/24
ØGateway is closer to the
end-hosts reducing the
failure domain.
ØEliminate traffic hair
pinning and unnecessary
traffic backhauling to
centralized gateway.
ØUses Anycast Gateway
MAC (AGM) address to
prevent traffic block-holed
resulting from MAC
mobility.
.1/24
L3
Server2

INTEGRATED ROUTING AND BRIDGING (IRB)
• Two different operations are specified for IRB with VXLAN BGP EVPN deployment depending on the
number of operations carried out on both the ingress and egress NVE.
Ø Asymmetric IRB
Ø Symmetric IRB
• Asymmetric IRB performs two operations on the ingress and one operation on the egress device
hence the name. It follows the bridge-route-bridge approach, bridging and routing operations are
performed on the ingress NVE followed by bridging to the respective destination through the Layer-2
VNI (L2VNI) on the egress NVE. This means that the device hosting the first-hop gateway function is
required to have all possible destination MAC/IP binding information resulting in scaling concern.
• Symmetric IRB on the other hand uses a bridge-route-route-bridge approach, meaning both ingress
and egress device perform the same number of operations (route-bridge) in this case. Routed traffic
from ingress to egress is forwarded via a transit segment, defined on a per-VRF basis and termed the
Layer-3 VNI or L3VNI. This means that only MAC/IP bindings associated with locally attached End-
Points are required on the device hosting the first-hop gateway function, making this a more scalable
approach.

VXLAN BGP EVPN FABRIC RECOMMENDATION
Spine 1 Spine 2
Leaf 2 Leaf 3 Leaf 4Leaf 1
AS101 AS101
AS201 AS202 AS203 AS204
Ø Simple design, suitable for most enterprise. Unless
traffic engineering (TE) is required intra and inter DC,
in which case Segment Routing can be considered.
Ø Underlay eBGP bound to /31 physical interfaces.
Ø BGP ASN per switch pair.
Ø Export loopback prefixes for the overlay EVPN
session.
Ø No IGP required, single protocol to manage. Unless
TE / Segment Routing is required and used.
Ø /31 interface addresses can be re-use across multiple
data centers, meaning new DC can be turn up very
quickly.
Ø Ethernet OAM – Link Fault Management (LFM).
Leaf 1 Leaf 2 Leaf 3 Leaf 4

AS per router
AS 65000 AS 65000
AS 65100 AS 65101 AS 65102 AS 65103
Easy
Configuration
Templating
/31 per link
EBGP
• Multipath for ECMP
• Export loopbacks
VXLAN BGP EVPN FABRIC RECOMMENDATION
Ethernet OAM -LFM

VXLAN BGP EVPN– TEST TOPOLOGY

VXLAN BGP EVPN UNDERLAY – CONFIG ARISTA
service routing protocols model multi-agent
router bgp 65000
neighbor LEAF-PEERS-UNDERLAY peer-group
neighbor LEAF-PEERS-UNDERLAY description
"LEAF NEIBOURS UNDERLAY”
neighbor LEAF-PEERS-UNDERLAY allowas-in 2
neighbor LEAF-PEERS-UNDERLAY send-community
standard extended
neighbor 10.10.10.0 peer-group LEAF-PEERS-
UNDERLAY
neighbor 10.10.10.0 remote-as 65001
UNDERLAY
UNDERLAY
UNDERLAY
redistribute connected route-map ADV-
LOOPBACK
Spine 5 Spine 6
router bgp 65000
neighbor LEAF-PEERS-UNDERLAY peer-group
neighbor LEAF-PEERS-UNDERLAY description
"LEAF NEIBOURS UNDERLAY”
neighbor LEAF-PEERS-UNDERLAY allowas-in 2
neighbor LEAF-PEERS-UNDERLAY send-community
standard extended
UNDERLAY
UNDERLAY
UNDERLAY
UNDERLAY
redistribute connected route-map ADV-
LOOPBACK

router bgp 65001
neighbor MLAG-IBGP peer-group
neighbor MLAG-IBGP remote-as 65001
neighbor MLAG-IBGP next-hop-self
neighbor MLAG-IBGP weight 0
neighbor MLAG-IBGP description "MLAG PEER
UNDERLAY"
neighbor MLAG-IBGP send-community
neighbor SPINE-PEERS-UNDERLAY peer-group
neighbor SPINE-PEERS-UNDERLAY remote-as 65000
neighbor SPINE-PEERS-UNDERLAY weight 100
neighbor SPINE-PEERS-UNDERLAY description "SPINE
NEIBOURS UNDERLAY"
neighbor SPINE-PEERS-UNDERLAY allowas-in 2
neighbor SPINE-PEERS-UNDERLAY route-map ADV-
LOOPBACK out
neighbor SPINE-PEERS-UNDERLAY send-community
standard extended
neighbor 10.0.0.2 peer-group MLAG-IBGP
neighbor 10.10.10.1 peer-group SPINE-PEERS-UNDERLAY
redistribute connected route-map ADV-LOOPBACK
Leaf 9 Leaf 10
router bgp 65001
neighbor MLAG-IBGP peer-group
neighbor MLAG-IBGP remote-as 65001
neighbor MLAG-IBGP next-hop-self
neighbor MLAG-IBGP weight 0
neighbor MLAG-IBGP description "MLAG PEER
UNDERLAY"
neighbor MLAG-IBGP send-community
neighbor SPINE-PEERS-UNDERLAY peer-group
neighbor SPINE-PEERS-UNDERLAY remote-as 65000
neighbor SPINE-PEERS-UNDERLAY weight 100
neighbor SPINE-PEERS-UNDERLAY description "SPINE
NEIBOURS UNDERLAY"
neighbor SPINE-PEERS-UNDERLAY allowas-in 2
neighbor SPINE-PEERS-UNDERLAY route-map ADV-
LOOPBACK out
neighbor SPINE-PEERS-UNDERLAY send-community
standard extended
neighbor 10.0.0.1 peer-group MLAG-IBGP
redistribute connected route-map ADV-LOOPBACK
VXLAN BGP EVPN UNDERLAY – CONFIG ARISTA

DC1-LF9#sh ip bgp summary
BGP summary information for VRF default
Router identifier 192.168.255.3, local AS number 65001
Neighbor Status Codes: m - Under maintenance
Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc
10.0.0.2 4 65001 143 142 0 0 01:37:36 Estab 9 9
10.10.10.1 4 65000 194 194 0 0 01:54:42 Estab 8 8
10.10.10.3 4 65000 292 286 0 0 02:14:34 Estab 8 8
DC1-LF9#
Leaf9
VXLAN BGP EVPN UNDERLAY – OUTPUTS
Spine 5
DCI-SP1#sh ip bgp summary
10.10.10.0 4 65001 115 120 0 0 01:35:27 Estab 3 3
10.10.10.4 4 65001 77 83 0 0 01:01:14 Estab 2 2
10.10.10.8 4 65002 51 58 0 0 00:41:07 Estab 3 3
10.10.10.12 4 65002 57 61 0 0 00:26:52 Estab 3 3
DCI-SP1#

DC1-LF9#sh ip route bgp
VRF: default
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
NG - Nexthop Group Static Route, V - VXLAN Control Service,
B E 172.16.255.10/32 [20/0] via 10.10.10.1, Ethernet1
via 10.10.10.3, Ethernet2
B E 172.168.254.11/32 [20/0] via 10.10.10.1, Ethernet1
B E 172.168.255.11/32 [20/0] via 10.10.10.1, Ethernet1
B E 172.168.255.12/32 [20/0] via 10.10.10.1, Ethernet1
B E 192.168.255.1/32 [20/0] via 10.10.10.1, Ethernet1
B E 192.168.255.2/32 [20/0] via 10.10.10.3, Ethernet2
B E 192.168.255.4/32 [20/0] via 10.10.10.1, Ethernet1
B E 192.168.255.5/32 [20/0] via 10.10.10.1, Ethernet1
B E 192.168.255.6/32 [20/0] via 10.10.10.1, Ethernet1
DC1-LF9#
Leaf9
VXLAN BGP EVPN UNDERLAY – OUTPUTS

router bgp 65000
neighbor LEAF-PEERS-OVERLAY peer-group
neighbor LEAF-PEERS-OVERLAY next-hop-unchanged
neighbor LEAF-PEERS-OVERLAY update-source
Loopback0
neighbor LEAF-PEERS-OVERLAY description "LEAF
NEIBOURS OVERLAY"
neighbor LEAF-PEERS-OVERLAY ebgp-multihop 3
neighbor LEAF-PEERS-OVERLAY send-community
standard extended
neighbor 192.168.255.3 peer-group LEAF-PEERS-OVERLAY
!
address-family evpn
neighbor LEAF-PEERS-OVERLAY activate
!
address-family ipv4
no neighbor LEAF-PEERS-OVERLAY activate
!
Spine 5 Spine 6
VXLAN BGP EVPN OVERLAY– CONFIG ARISTA
router bgp 65000
neighbor LEAF-PEERS-OVERLAY peer-group
neighbor LEAF-PEERS-OVERLAY next-hop-unchanged
neighbor LEAF-PEERS-OVERLAY update-source
Loopback0
neighbor LEAF-PEERS-OVERLAY description "LEAF
NEIBOURS OVERLAY"
neighbor LEAF-PEERS-OVERLAY ebgp-multihop 3
neighbor LEAF-PEERS-OVERLAY send-community
standard extended
!
address-family evpn
neighbor LEAF-PEERS-OVERLAY activate
!
address-family ipv4
no neighbor LEAF-PEERS-OVERLAY activate
!

router bgp 65001
neighbor SPINE-PEERS-OVERLAY peer-group
neighbor SPINE-PEERS-OVERLAY remote-as 65000
neighbor SPINE-PEERS-OVERLAY update-source
Loopback0
neighbor SPINE-PEERS-OVERLAY description
"SPINE NEIBOURS OVERLAY"
neighbor SPINE-PEERS-OVERLAY ebgp-multihop 3
neighbor SPINE-PEERS-OVERLAY send-community
standard extended
neighbor 192.168.255.1 peer-group SPINE-PEERS-
OVERLAY
OVERLAY
!
address-family evpn
neighbor SPINE-PEERS-OVERLAY activate
!
address-family ipv4
no neighbor SPINE-PEERS-OVERLAY activate
!
Leaf 9 Leaf 10
VXLAN BGP EVPN OVERLAY– CONFIG ARISTA
router bgp 65001
neighbor SPINE-PEERS-OVERLAY peer-group
neighbor SPINE-PEERS-OVERLAY remote-as 65000
neighbor SPINE-PEERS-OVERLAY update-source
Loopback0
neighbor SPINE-PEERS-OVERLAY description
"SPINE NEIBOURS OVERLAY"
neighbor SPINE-PEERS-OVERLAY ebgp-multihop 3
neighbor SPINE-PEERS-OVERLAY send-community
standard extended
OVERLAY
OVERLAY
!
address-family evpn
neighbor SPINE-PEERS-OVERLAY activate
!
address-family ipv4
no neighbor SPINE-PEERS-OVERLAY activate
!

DC1-LF9#show bgp evpn summary
192.168.255.1 4 65000 193 188 0 0 02:16:01 Estab 6 6
192.168.255.2 4 65000 201 210 0 0 02:27:31 Estab 6 6
DC1-LF9#
Leaf9
VXLAN BGP EVPN OVERLAY– OUTPUTS
Spine 5
DCI-SP1#show bgp evpn summary
192.168.255.3 4 65001 171 189 0 0 02:16:10 Estab 2 2
192.168.255.4 4 65001 133 144 0 0 01:41:53 Estab 4 4
192.168.255.5 4 65002 112 119 0 0 01:21:26 Estab 3 3
192.168.255.6 4 65002 92 101 0 0 01:06:48 Estab 3 3
DCI-SP1#

DC1-LF9#show bgp evpn
BGP routing table information for VRF default
Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
Network Next Hop Metric LocPref Weight Path
* >Ec RD: 192.168.255.5:10727 imet 172.168.255.11
172.168.255.11 - 100 0 65000 65002 i
* ec RD: 192.168.255.5:10727 imet 172.168.255.11
172.168.255.11 - 100 0 65000 65002 i
* >Ec RD: 192.168.255.6:10727 imet 172.168.255.12
172.168.255.12 - 100 0 65000 65002 i
* ec RD: 192.168.255.6:10727 imet 172.168.255.12
172.168.255.12 - 100 0 65000 65002 i
* > RD: 192.168.255.3:10030 ip-prefix 10.10.10.10/32
- - 100 0 6400 i
* > RD: 192.168.255.3:10030 ip-prefix 172.16.16.0/24
- - - 0 i
* RD: 192.168.255.3:10030 ip-prefix 172.16.16.0/24
- - 100 0 6400 i
* >Ec RD: 192.168.255.5:10727 ip-prefix 172.168.72.0/24
172.168.255.11 - 100 0 65000 65002 i
* ec RD: 192.168.255.5:10727 ip-prefix 172.168.72.0/24
172.168.255.11 - 100 0 65000 65002 i
* >Ec RD: 192.168.255.6:10727 ip-prefix 172.168.72.0/24
172.168.255.12 - 100 0 65000 65002 i
* ec RD: 192.168.255.6:10727 ip-prefix 172.168.72.0/24
172.168.255.12 - 100 0 65000 65002 i
Leaf9
VXLAN BGP EVPN OVERLAY– OUTPUTS

!
interface Vlan28
vrf forwarding TEST-VRF-VLAN28
ip address virtual 192.168.20.254/24
!
ip virtual-router mac-address 00:00:00:00:00:aa
!
ip routing vrf TEST-VRF-VLAN28
!
router bgp 65001
!
vlan 28
rd 192.168.255.3:10028
route-target both 10028:10028
redistribute learned
!
vrf TEST-VRF-VLAN28
rd 192.168.255.3:12828
route-target import 12828:12828
route-target export 12828:12828
redistribute connected
!
interface Vxlan1
vxlan udp-port 4789
vxlan vrf TEST-VRF-VLAN28 vni 12828
!
Leaf 9 Leaf 10
VXLAN BGP EVPN OVERLAY SERVICE– CONFIG ARISTA
!
interface Vlan28
!
!
!
router bgp 65001
!
vlan 28
rd 192.168.255.4:10028
!
vrf TEST-VRF-VLAN28
rd 192.168.255.4:12828
!
interface Vxlan1
vxlan udp-port 4789
!

!
interface Vlan28
!
!
!
router bgp 65002
!
vlan 28
rd 192.168.255.5:10028
!
vrf TEST-VRF-VLAN28
rd 192.168.255.5:12828
!
interface Vxlan1
vxlan udp-port 4789
!
Leaf 11 Leaf 12
VXLAN BGP EVPN OVERLAY SERVICE– CONFIG ARISTA
!
interface Vlan28
!
!
!
router bgp 65002
!
vlan 28
rd 192.168.255.6:10028
!
vrf TEST-VRF-VLAN28
rd 192.168.255.6:12828
!
interface Vxlan1
vxlan udp-port 4789
!

DC1-LF11#sh bgp evpn route-type mac-ip
BGP routing table information for VRF default
Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e
- ECMP
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local
Nexthop
Network Next Hop Metric LocPref Weight Path
* > RD: 192.168.255.5:10028 mac-ip 5000.00c6.6396
- - - 0 i
* > RD: 192.168.255.5:10028 mac-ip 5000.00c6.6396 192.168.20.28
- - - 0 i
* >Ec RD: 192.168.255.3:10028 mac-ip 5000.00c6.c8d3
172.16.255.9 - 100 0 65000 65001 i
* ec RD: 192.168.255.3:10028 mac-ip 5000.00c6.c8d3
172.16.255.9 - 100 0 65000 65001 i
* >Ec RD: 192.168.255.4:10028 mac-ip 5000.00c6.c8d3
172.16.255.10 - 100 0 65000 65001 i
* ec RD: 192.168.255.4:10028 mac-ip 5000.00c6.c8d3
172.16.255.10 - 100 0 65000 65001 i
* >Ec RD: 192.168.255.3:10028 mac-ip 5000.00c6.c8d3 192.168.20.27
172.16.255.9 - 100 0 65000 65001 i
* ec RD: 192.168.255.3:10028 mac-ip 5000.00c6.c8d3 192.168.20.27
172.16.255.9 - 100 0 65000 65001 i
* >Ec RD: 192.168.255.4:10028 mac-ip 5000.00c6.c8d3 192.168.20.27
172.16.255.10 - 100 0 65000 65001 i
* ec RD: 192.168.255.4:10028 mac-ip 5000.00c6.c8d3 192.168.20.27
172.16.255.10 - 100 0 65000 65001 i
DC1-LF11#
Leaf 11
VXLAN BGP EVPN OVERLAY SERVICE– OUTPUTS

DC1-LF11#sh vxlan address-table vlan 28
----------------------------------------------------------------------
VLAN Mac Address Type Prt VTEP Moves Last
Move
---- ----------- ---- --- ---- ----- ---------
28 5000.00c6.c8d3 EVPN Vx1 172.16.255.9 2
0:03:29 ago
Total Remote Mac Addresses for this criterion: 1
Leaf 11
DC1-LF11#sh vxlan address-table evpn vlan 28
----------------------------------------------------------------------
VLAN Mac Address Type Prt VTEP Moves Last
Move
---- ----------- ---- --- ---- ----- ---------
28 5000.00c6.c8d3 EVPN Vx1 172.16.255.9 2
0:05:09 ago
Total Remote Mac Addresses for this criterion: 1
DC1-LF11#

DC1-LF11#sh ip route vrf TEST-VRF-VLAN28
VRF: TEST-VRF-VLAN28
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type2, B I - iBGP, B E - eBGP,
R - RIP, I L1 - IS-IS level 1, I L2 - IS-IS level 2,
O3 - OSPFv3, A B - BGP Aggregate, A O - OSPF Summary,
NG - Nexthop Group Static Route, V - VXLAN Control Service,
Gateway of last resort is not set
B E 10.10.10.10/32 [200/0] via VTEP 172.16.255.10 VNI 13030 router-
mac 50:00:00:d7:ee:0b
via VTEP 172.16.255.9 VNI 13030 router-mac
50:00:00:6b:2e:70
B E 172.16.16.0/24 [200/0] via VTEP 172.16.255.10 VNI 13030 router-
mac 50:00:00:d7:ee:0b
via VTEP 172.16.255.9 VNI 13030 router-mac
50:00:00:6b:2e:70
C 192.168.20.0/24 is directly connected, Vlan28
DC1-LF11#
Leaf 11

ØUse case Juniper Qfabric
SAMPLE MIGRATION OF LEGACY PLATFORM TO VXLAN EVPN

L2 CONNECTIVITY INTRA DC- MIGRATION
Ø Dedicated Border Leaf BLF
connects the Qfabric NNG
device via a layer 2 trunk
interface.
Ø Per customer L2 domain is
stretched to the BLF and
terminates in per customer
MAC VRF.
Ø Connectivity between BLF
and leafs LF1…N is via
VXLAN EVPN.
DCI1 DCI2
BLF
LF1…N
SP1 SP2
Servers and Other L2/L3
Devices
Backbone
trunk
Qfabric

L3 CONENNECTIVTY INTRA/ INTER-DC - MIGRATION
Ø Separate BGP/EVPN Session needed between
DCI and the Border Leaf to learn remove EVPN
routes.
Ø Per customer SVI /IP VRF eBGP session between
Qfabric and BLF. To allow for smooth
decommission of the Qfabric, also per tenant
VRF eBGP session between DCI VRF and BLF is
needed.
Ø Both DCI and Qfabric advertise a single default
route into the per customer IP VRF on the BLF.
BGP attributes can then be manipulated to
control exit traffic path from BLF.
Ø Specific routes are advertised from the per
customer IP VRF on the BLF back to the Qfabric
and DCI. BGP attributes can be manipulated to
determine the traffic flow.
Ø Connectivity between BLF and leafs LF1…N and
new remote inter-DC is via VXLAN EVPN.
DCI1 DCI2
BLF
LF1…N
SP1 SP2
Devices
Backbone
trunk
Qfabric
L3 link -runs 802.1Q from DCI VRF/
eBGP/PIM

TOPOLOGY POST DC MIGRATION
DCI1 DCI2
BLF
LF1…N
SP1 SP2
Devices
Backbone
L3 link -runs 802.1Q from DCI VRF/
eBGP/PIM

ØCisco Press – Building Data Centers with VXLAN BGP EVPN by Lukas
Krattiger, Shyam Kapadia, David Jansen.
ØO’Reilly – Juniper QFX1000 Series A Compressive Guide to Building Next-
Generation Data Centers by Douglas Richard Hanks,Jr.
Øhttp://eve-ng.net/
Øhttps://tools.ietf.org/html/rfc7348
Øhttps://tools.ietf.org/html/rfc7432
Øhttps://datatracker.ietf.org/doc/draft-ietf-bess-evpn-df-election-
framework/?include_text=1
Øhttps://www.arista.com/en
Øhttps://www.microsoft.com/en-us/research/wp-
content/uploads/2017/02/HRW98.pdf
Øhttps://www.juniper.net/documentation/en_US/junos/topics/concept/qfabric
-overview.html
REFERENCES

VXLAN BGP EVPN: Technology Building Blocks

More Related Content

What's hot (20)

Similar to VXLAN BGP EVPN: Technology Building Blocks (20)

More from APNIC (20)

Recently uploaded (20)

VXLAN BGP EVPN: Technology Building Blocks