SlideShare a Scribd company logo

Brkdcn 2035 multi-x

Mason Mei, profile picture
Mason Mei

The document discusses VXLAN BGP EVPN based multi-pod, multi-fabric, and multi-site network architectures. It provides an overview of the evolution from single-pod/fabric deployments to more complex multi-site designs. Key concepts covered include hierarchical overlay domains, isolated underlay domains, and border gateways used to interconnect sites while maintaining underlay isolation.

Internet
1 of 141
Downloaded 107 times
1
2
3
4
Most read
5
6
7
8
9
10
11
12
Most read
13
14
15
16
17
18
19
20
21
22
23
Most read
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
Brkdcn 2035 multi-x
VXLAN BGP EVPN based Multi-Pod, Multi- Fabric, Multi-Site Max Ardica – Principal Engineer Lukas Krattiger – Principal Engineer BRKDCN-2035
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Cisco Spark Questions? Use Cisco Spark to chat with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click “Join the Discussion” 3. Install Spark or go directly to the space 4. Enter messages/questions in the space How cs.co/ciscolivebot#BRKDCN-2035Cisco Spark spaces will be available until July 3, 2017.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Who Are the Presenters? Max Ardica Principal Engineer - INSBU Lukas Krattiger Principal Engineer – INSBU @ccie21921 BRKDCN-2035 4
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Session Objectives At the end of the session, the participants should be able to: Articulate the different deployment options to interconnect VXLAN EVPN Networks (Multi-Pod vs. Multi-Fabric vs. Multi- Site) Understand the functionalities and specific design considerations associated to the new VXLAN Multi-Site architecture Initial assumption: The audience already has a good knowledge of the VXLAN EVPN technology and its use to deploy modern Data Center Fabrics BRKDCN-2035 5
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Session Reference • BRKDCN-2304 • L4-L7 Service Integration in Multi-Tenant VXLAN EVPN Data Center Fabrics • BRKDCN-3378 • Building DataCenter Networks with VXLAN BGP-EVPN • Wednesday, Jun 28, 1:30 pm • BRKDCN-2125 • Overlay Management and Visibility with VXLAN • Thursday, Jun 29, 10:30 am • BRKDCN-2342 • Programmable Fabric Automation and Management with DCNM 10 • Thursday, Jun 29, 1:00 p.m. 6BRKDCN-2035
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Agenda Introduction VXLAN EVPN Interconnect Evolution • Multi-Pod • Multi-Fabric • Multi-Site VXLAN EVPN Multi-Site Deep Dive • Walkthrough • Control- and Data-Plane • Deployment Considerations Conclusions and Q&A
Introduction
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Layer 2 Domain Elasticity Local LAN Fabric Extended LAN fabric VN-link notifications IP Mobility Optimal Ingress and Egress Routing VM-awareness VXLAN, DFA, ACI, VN-link Storage Elasticity SAN Extensions Network Service Localization Any service anywhere OTV OTV OTV OTV Fabric Consolidation Unified Fabric & I/O Device Virtualization Segmentation Data Center Interconnect – DCI Model Connecting Virtualized Data Centers Multi-tenancy/Segmentation Segment-IDs in VXLAN, LISP, FabricPath, and OTV Storage Solutions & Partners: FCIP, I/O Acceleration EMC, NetApp BRKDCN-2035 9
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Back Then Yet Another Encapsulation Flood & Learn (Multicast-based) Data-Plane only Yesterday VXLAN for the Data Center – Intra-DC Control-Plane Active VTEP Discovery Multicast and Unicast Now! VXLAN for DCI – Inter-DC DCI Ready ARP/ND caching/suppress Multi-Homing Failure Domain Isolation Loop Protection VXLAN Evolves as the Control Plane Evolves! BRKDCN-2035 10
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Back Then VXLAN for Interconnecting Networks BRKDCN-2035 11
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Inter-X Connectivity • Single Fabric with End- to-End Encapsulation • Build Hierarchy in the Underlay – Flatten it in the Overlay Multi-Pod Overlay VTE P VTE P VTE P VTE P Bar em eta l Bar em eta l Fabric #2 Overlay VTE P VTE P Bar em eta l Bar em eta l VTE P VTE P Fabric #1EVPN Control- Plane Domain 1 EVPN Control- Plane Domain 2 Single Data-Plane – End-to-End BGP EVPN Overlay VTE P VTE P VTE P VTE P Bar em etal Bar em etal Fabric #2 Overlay VTE P VTE P Bar em etal Bar em etal VTE P VTE P Fabric #1EVPN Control-Plane Domain 1 EVPN Control-Plane Domain 2 Data-Plane Domain 1 Data-Plane Domain 2 DCI Data-Plane • Multiple Fabrics – Normalized through Ethernet • Multiple Fabrics Interconnect using DCI (Layer 2 and Layer 3) Multi-Fabric Data-Plane Domain 1 Data-Plane Domain 2 DCI Data-Plane Overlay VTE P VTE P VTE P VTE P Bar em etal Bar em etal Fabric #2 Overlay VTE P VTE P Bar em etal Bar em etal VTE P VTE P Fabric #1EVPN Control-Plane Domain 1 EVPN Control-Plane Domain 2 BGP EVPN • Multiple Fabrics with Integrated DCI • Integrated DCI – Scaling within and between Fabrics • The Happy Place Multi-Site BRKDCN-2035 12
VXLAN EVPN Interconnect Evolution
Multi-Pod
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public VXLAN EVPN – Single Pod / Single Fabric SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod 1 VTEP VTEP External Network BRKDCN-2035 15
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public VXLAN EVPN – Multi-Pod SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod n VTEP VTEP Underlay Extension BRKDCN-2035 16
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Single Overlay Domain – End-to-End Encapsulation Single Overlay Control-Plane Domain – End-to-End EVPN Updates Single Underlay Domain End-to-End Single Replication Domain for BUM Single VNI Administrative Domain Multi-Pod Characteristics – ”The Single” Building Underlay Hierarchies – Non Hierarchical Overlay BRKDCN-2035 17
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Pod – End-to-End Encapsulation SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod n VTEP VTEP Underlay Extension Overlay Baremetal Baremetal Unicast VTEP 10.1.1.1 VTEP 10.2.2.7 BRKDCN-2035 18
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Pod – BUM Replication SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod 2 VTEP VTEP Underlay Extension Overlay Baremetal BUM BRKDCN-2035 19
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Single Overlay Domain – End-to-End Encapsulation • Scaling the VXLAN EVPN Network Single Overlay Control-Plane Domain – End-to-End EVPN Updates • Overlay Control-Plane Update Propagation Single Underlay Domain End-to-End • Network must be extended in Underlay (VTEP to VTEP reachability) Single Replication Domain for BUM • One BUM flooding domain through out all connected Pods Multi-Pod Challenges – ”The Single” BRKDCN-2035 20
Multi-Fabric
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public VXLAN EVPN – Multi-Fabric SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric 2 VTEP VTEP Underlay No Extension L2 DCI L2 DCI L3 DCI L3 DCI L2 DCI L2 DCI BRKDCN-2035 22
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public • Separate Overlay Domains –Independent L2 and L3 DCI (complexity) • Separate Overlay Control-Plane Domains – Manual Configuration • Separate Underlay Domains - Isolated • Separate Replication Domains for BUM – Independent BUM transport/DCI • Dedicated Border Leaf – no local End-Point Attachment Multi-Fabric Characteristics – ”The Separate” Underlay Isolation – Separate DC Interconnection BRKDCN-2035 23
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric n VTEP VTEP Underlay No Extension L2 DCI L2 DCI L3 DCI L3 DCI L2 DCI L2 DCI Multi-Fabric – End-to-End Encapsulation Overlay Site 1 Overlay Site n L2 DCI Baremetal Baremetal Unicast VLAN Hand-Off VRF-Lite Hand-Off BRKDCN-2035 24
VXLAN EVPN Multi-Site
Multi-Site Walkthrough
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public VXLAN EVPN – Multi-Site SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP No Underlay Extension BGW BGW BGW BGW BRKDCN-2035 27
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multiple Overlay Domains – Interconnected & Controlled Multiple Overlay Control-Plane Domains – Interconnected & Controlled Multiple Underlay Domains - Isolated Multiple Replication Domains for BUM – Interconnected & Controlled Multiple VNI Administrative Domains – Phase 2 Multi-Site Characteristics – ”The Multiple” Underlay Isolation – Overlay Hierarchies BRKDCN-2035 28
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site – Hierarchical Overlay Domains SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Site 1 Overlay Site n Overlay Multi-Site Baremetal Baremetal Unicast BGW BGW BGW BGW BRKDCN-2035 29
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site – Underlay Isolation SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP No Underlay Extension VTEP 10.1.1.1 Border (VIP) 10.1.1.111 Border (VIP) 10.2.2.222 Site 1 Underlay Routing Table Leaf: 10.1.1.1 10.1.1.2 10.1.1.3 10.1.1.4 10.1.1.5 10.1.1.6 10.1.1.7 Border: 10.1.1.101 10.1.1.102 10.1.1.111 VTEP 10.2.2.7 Site n Underlay Routing Table Leaf: 10.2.2.1 10.2.2.2 10.2.2.3 10.2.2.4 10.2.2.5 10.2.2.6 10.2.2.7 Border: 10.2.2.101 10.2.2.102 10.2.2.222 BGW BGW BGW BGW Border (PIP) 10.1.1.101 Border (PIP) 10.1.1.102 Border (PIP) 10.2.2.101 Border (PIP) 10.2.2.102 BRKDCN-2035 30
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Inter Site Network Multi-Site – Inter Site Network SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP VTEP 10.1.1.1 Border (VIP) 10.2.2.222 Border (VIP) 10.1.1.111 Inter-Site Network Routing Table Border Site1: 10.1.1.101 10.1.1.102 10.1.1.111 Border Site2: 10.2.2.101 10.2.2.102 10.2.2.222 VTEP 10.2.2.7 BGW BGW BGW BGW Border (PIP) 10.1.1.101 Border (PIP) 10.1.1.102 Border (PIP) 10.2.2.101 Border (PIP) 10.2.2.102 BRKDCN-2035 31
Border Gateways Deployment Considerations
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Border Gateways Deployment Considerations Site 1 VTEP BGW VTEP BGW VTEP BGW VTEP BGW Site 1 VTEP BGW VTEP BGW Border Gateways used for two main functions: 1. Interconnecting each site to the Inter-Site network (for East-West traffic flows) 2. Connecting each site to the external Layer 3 domain (for North-South traffic flows) May also be used to connect End-Points and/or network service nodes (FWs, ADCs) Two deployment models supported: 1. Anycast Border Gateways 2. VPC Border Gateways Anycast Border Gateways VPC Border Gateways BRKDCN-2035 33
Anycast Border Gateways
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Anycast Border Gateway (1) Anycast Border Gateway Up to 4 Border Gateways Border Gateway • Deploying at Leaf – 7.0(3)I7(1) • Deploying at Spine – 7.0(3)I7(2) VTEP BGW VTEP BGW VTEP BGW VTEP BGW BRKDCN-2035 35
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Anycast Border Gateway (2) Anycast Border Gateway Common Virtual IP (VIP) across BGW • VIP is used for Intra- and Inter-Site Communication • VIP for communication between the Border Gateways in different Sites • VIP for communication between Border Gateway and Leaf within a Site Individual Primary IP (PIP) per BGW • Used for Broadcast, Unknown Unicast and Multicast (BUM) replication • PIP for communication with Single-Homed End-Points (routed only), intra- and inter-Site VTEP BGW VTEP BGW VTEP BGW VTEP BGW Border VIP 10.1.1.111 Border VIP 10.1.1.111 PIP-BGW1 10.1.1.101 PIP-BGW2 10.1.1.102 PIP-BGW3 10.1.1.103 PIP-BGW4 10.1.1.104 BRKDCN-2035 36
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Anycast Border Gateway (3) Anycast Border Gateway Per-VNI Designated Forwarder (DF) election • Each BGW can serve as DF for a single or a set of Layer-2 VNI • DF election and assignment is automatic Using BGP EVPN Route Type 4 for DF election • Operator Managed Assignment (Type: 00) • Six Octet Site Identifier (System MAC: 00:00:00:00:00:01) • Multi-Site Discriminator (Ethernet-Segment: 00:00:07) • Originators IP Address (PIP): 10.1.1.101 • Layer-2 VNI: 30010 VTEP BGW VTEP BGW VTEP BGW VTEP BGW Spine RR Spine RR BGP EVPN Type: 00 System MAC: 00:00:00:00:00:01 Ethernet Segment: 00:00:074 IP: 10.1.1.101 VNI: 30010 DF 30010 DF 30099 DF 30012 DF 30011 BRKDCN-2035 37
VPC Border Gateways
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 VPC Border Gateway (1) VPC Border Gateway 2 Border Gateways Border Gateway • Using a Leaf – 7.0(3)I7(2) VTEP BGW VTEP BGW BRKDCN-2035 39
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 VPC Border Gateway (2) VPC Border Gateway Common Virtual IP (VIP) across BGW • VIP is used for Intra- and Inter-Site Communication • VIP for communication between the Border Gateways in different Sites • VIP for communication between Border Gateway and Leaf within a Site Individual Primary IP (PIP) per BGW • Used for Broadcast, Unknown Unicast and Multicast (BUM) replication • PIP for communication with Single-Homed End-Points, intra- and inter-Site VTEP BGW VTEP BGW Border VIP 10.1.1.111 Border VIP 10.1.1.111 PIP-BGW1 10.1.1.101 PIP-BGW2 10.1.1.102 BRKDCN-2035 40
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 VPC Border Gateway (3) VPC Border Gateway VPC-based Designated Forwarder Election Per-Site Designated Forwarder (DF) election • Using same approach as in VPC • Best Path to Rendezvous-Point or VPC Primary Node VTEP BGW VTEP BGW DF BRKDCN-2035 41
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 VPC Border Gateway (4) VPC Border Gateway Single- or Dual-Homed End-Points • Services Appliance (i.e. Firewall, ADC etc.) • Physical or Virtual Servers Advertised and Reachable through Virtual IP Address (VIP) • Intra-Site: Leaf nodes use VIP to reach End- Points connected to Border Gateways • Inter-Site: Remote Border Gateways use VIP to reach End-Points connected to Border • Traffic potentially traverses VPC Peer-Link VTEP BGW VTEP BGW VTEP Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111 2 0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001 10.1.1.111 Border VIP 10.1.1.111 Border VIP 10.1.1.111 ADC 0000.3010.1102 192.168.10.102 ADC ADC 0000.3010.1101 192.168.10.101 ADC BRKDCN-2035 42
Multi-Site Control Plane
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Control Plane Deployment Considerations Both MP-eBGP or MP-iBGP peering supported intra-Site between leaf nodes Only MP-eBGP EVPN sessions supported inter-Sites mandates that each site is part of a separate AS Full mesh of MP-eBGP EVPN adjacencies only currently supported across sites • Recommended to deploy a couple of Route-Servers in the Inter-Site network when 3 or more sites are deployed • Route-Servers only perform control plane functions (“eBGP Route-Reflectors”) • Need to ensure that Route-Servers offer support for Route Type 4 EVPN routes, required for DF election BRKDCN-2035 44
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (L3Core) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR iBGP-EVPN iBGP-EVPN DC Core (Layer-3 Unicast) BRKDCN-2035 45
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (L3Core) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR iBGP-EVPN iBGP-EVPN DC Core (Layer-3 Unicast) RS Route Server (eBGP ”Route Reflector”) RS BRKDCN-2035 46
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (L3Core) DC Core (Layer-3 Unicast) RS Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR iBGP-EVPN iBGP-EVPN RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 47
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (L3Core, no RS) DC Core (Layer-3 Unicast) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR iBGP-EVPN iBGP-EVPN RS – Route Server (eBGP ”Route Reflector”) eBGP-EVPN BRKDCN-2035 48
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane DC Core (Layer-3 Unicast) RS Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR iBGP-EVPN Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 VRF Tenant1 L3VNI: 50001 Route-Target: 65501:50001 VRF Tenant1 L3VNI: 50001 Route-Target: 65502:50001 L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1) L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) iBGP-EVPN VIP1 10.1.1.111 VIP2 10.2.2.222 RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 49
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (Site1) DC Core (Layer-3 Unicast) RS Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 VRF Tenant1 L3VNI: 50001 Route-Target: 65502:50001 L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1) L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VRF Tenant1 L3VNI: 50001 Route-Target: 65501:50001 VIP1 10.1.1.111 VIP2 10.2.2.222 Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3010.1101/48 30010, 65501:30010 192.168.10.101/32 50001, 65501:50001 10.1.1.1 2 0000.3020.2101/48 30020, 65501:30020 192.168.20.101/32 50001, 65501:50001 10.1.1.111 2 0000.3010.1102/48 30010, 65501:30010 192.168.10.102/32 50001, 65501:50001 10.1.1.111 BRKDCN-2035 50
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (Site2) DC Core (Layer-3 Unicast) RS Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 VRF Tenant1 L3VNI: 50001 Route-Target: 65502:50001 L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1) L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VRF Tenant1 L3VNI: 50001 Route-Target: 65502:50001 VIP1 10.1.1.111 VIP2 10.2.2.222 Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3010.1101/48 30010, 65502:30010 192.168.10.101/32 50001, 65502:50001 10.2.2.222 2 0000.3020.2101/48 30020, 65502:30020 192.168.20.101/32 50001, 65502:50001 10.2.2.1 2 0000.3010.1102/48 30010, 65502:30010 192.168.10.102/32 50001, 65502:50001 10.2.2.3 BRKDCN-2035 51
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (DCI) DC Core (Layer-3 Unicast) RS RS – Route Server (eBGP ”Route Reflector”) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR VRF Tenant1 L3VNI: 50001 Route-Target: 65501:50001 VRF Tenant1 L3VNI: 50001 Route-Target: 65502:50001 VIP1 10.1.1.111 VIP2 10.2.2.222 L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1) L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111 2 0000.3020.2101/48 30020, 65599:30020 192.168.20.101/32 50001, 65599:50001 10.2.2.222 2 0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001 10.2.2.222 BRKDCN-2035 52
Multi-Site – Selective Advertisements
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site – Selective Advertisements The Multi-Site architecture provides granular control on how Layer-2 and Layer-3 communication is extended across sites Layer-2 and/or Layer-3 VNIs configured on the Border Gateways (BGW) control the Control-Plane advertisement towards DCI Enhances the overall scalability of the solution • Scale up the total number of End-Points supported across sites BRKDCN-2035 54
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Selective Advertisements (DCI) DC Core (Layer-3 Unicast) RS Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR RS – Route Server (eBGP ”Route Reflector”) Only prefixes of VRF ”Tenant1” and L2VNI 30010 are advertised from Site1 towards DCI. In this example this is Host1. All prefixes of VRF ”Tenant2” and L2VNI 30020 are not advertised from Site2 towards DCI. These prefixes are not seen within the DCI VRF Tenant1 L3VNI: 50001 Route-Target: 65501:50001 VIP1 10.1.1.111 VIP2 10.2.2.222Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111 Host1 0000.3010.1101 192.168.10.101 L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) L2VNI: 30020 (VLAN 20) L3VNI: 50002 (Tenant2) Host2 0000.3020.2101 192.168.20.101 BRKDCN-2035 55
Multi-Site Data Plane
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Data Plane DC Core (Layer-3 Unicast) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 Intra-site VXLAN Data Plane Inter-site VXLAN Data Plane De-capsulation and Re-encapsulation on BGW De-capsulation and Re-encapsulation on BGW BRKDCN-2035 57
Multi-Site Packet Walk (BUM)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (BUM) – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255 Bridge DF 30010 DF 30010 Host 1 sends a L2 BUM frame 1 2 Leaf10 replicates traffic intra-Site BRKDCN-2035 59
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (DF & Split Horizon) – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255 Bridge DF 30010 DF 30010 BUM Forward Drop due to Split-Horizon rule Drop due to Designated Forwarder (DF) rule BRKDCN-2035 60
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (BUM) – DCI VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge DF 30010 DF 30010 SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255 BGW-VIP1 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255 BGW-VIP1 BGW12 30010 H1-MAC ALL-F H1-IP ALL-255 3 BGW11 replicates traffic inter- Sites toward BGW nodes BUM Forward BRKDCN-2035 61
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (DF & Split Horizon) – DCI VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge DF 30010 DF 30010 SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255 BGW-VIP1 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255 BGW-VIP1 BGW12 30010 H1-MAC ALL-F H1-IP ALL-255 BUM Forward BUM Forward Drop due to Split-Horizon rule Drop due to Designated Forwarder (DF) rule BRKDCN-2035 62
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (BUM) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge DF 30010 DF 30010 SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255 4 BGW22 replicates traffic intra-Site BUM Forward BRKDCN-2035 63
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (DF & Split Horizon) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge DF 30010 DF 30010 SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255 BUM Forward Drop due to Split-Horizon rule Drop due to Designated Forwarder (DF) rule BRKDCN-2035 64
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (BUM) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge DF 30010 DF 30010 Leaf20 sends traffic to local Host 2 5 BRKDCN-2035 65
Multi-Site Packet Walk (Bridging)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 1 to Host 2) – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP Bridge Host 1 sends traffic destined to remote Host 2 1 2 Leaf10 performs L2 lookup and encapsulates toward local BGW VIP1 address VIP2VIP1 BRKDCN-2035 67
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 1 to Host 2) – DCI VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 BGW-VIP2 30010 H1-MAC H2-MAC H1-IP H2-IP Bridge 3 BGW11 performs L2 lookup and encapsulates toward remote BGW VIP2 address VIP2VIP1 BRKDCN-2035 68
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 1 to Host 2) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 L20 30010 H1-MAC H2-MAC H1-IP H2-IP 4 BGW22 performs L2 lookup and encapsulates toward destination L20 node Leaf20 bridges traffic to local Host 2 5 VIP2VIP1 BRKDCN-2035 69
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 2 to Host 1) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge SIP DIP VXLAN SMAC DMAC SIP DIP Payload L20 BGW-VIP2 30010 H2-MAC H1-MAC H2-IP H1-IP Host 2 replies to remote Host 1 6 7 Leaf20 performs L2 lookup and encapsulates toward local BGW VIP2 address VIP2VIP1 BRKDCN-2035 70
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 2 to Host 1) – DCI VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 BGW-VIP1 30010 H2-MAC H1-MAC H2-IP H1-IP 8 BGW21 performs L2 lookup and encapsulates toward remote BGW VIP1 address VIP2VIP1 BRKDCN-2035 71
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 2 to Host 1) – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 L10 30010 H2-MAC H1-MAC H2-IP H1-IP 9 BGW12 performs L2 lookup and encapsulates toward destination L10 node Leaf10 bridges traffic toward Host 1 10 VIP2VIP1 BRKDCN-2035 72
Multi-Site Packet Walk (Routing)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-3 (Host 1 to Host 3) – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 3 0000.3010.1102 192.168.20.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 50001 L10-MAC BGW-VMAC1 H1-IP H3-IP Route Host 1 sends a data packet to the remote Host 3 1 2 Leaf10 performs a L3 lookup and encapsulates toward local BGW VIP1 address VIP2 VMAC2 VIP1 VMAC1 BRKDCN-2035 74
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-3 (Host 1 to Host 3) – DCI VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 3 0000.3010.1102 192.168.20.102 Route SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 BGW-VIP2 50001 BGW-VMAC1 BGW-VMAC2 H1-IP H3-IP 3 BGW11 performs a L3 lookup and encapsulates toward remote BGW VIP2 address VIP2 VMAC2 VIP1 VMAC1 BRKDCN-2035 75
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-3 (Host 1 to Host 3) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 3 0000.3010.1102 192.168.20.102 Route SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 L20 50001 BGW-VMAC1 L20-MAC H1-IP H3-IP 4 BGW21 performs a L3 lookup and encapsulates toward destination L20 node Leaf20 routes traffic to local Host 3 5 VIP2 VMAC2 VIP1 VMAC1 BRKDCN-2035 76
Multi-Site and Failure Detection on BGW
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Steady State Traffic – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP VIP2VIP1 BRKDCN-2035 78
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DCI Link Failure BGW12 – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP On DCI Link Failure (i.e. BGW12) • Virtual IP (VIP) on BGW is disabled • BGW will stop participating in DF election • BGW acts like a Leaf (Layer-3 only) • Traffic towards others Sites is served by remaining BGWs (i.e. BGW11) BGW12 gets isolated from the DCI Core Network Intra-site VXLAN traffic re- routing VIP2VIP1 BRKDCN-2035 79
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Link Failure BGW12 – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP VIP2VIP1 BRKDCN-2035 80
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Link Failure BGW12 – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP VIP2VIP1 BGW12 gets isolated from the Spine nodes Intra-site VXLAN traffic re- routing On Fabric Link Failure (i.e. BGW12) • Virtual IP (VIP) on BGW is disabled • Primary IP (PIP) on BGW is disabled • BGW will stop participating in the Overlay BRKDCN-2035 81
Multi-Site Setup Walkthrough
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – Enable Border Gateway Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 BGW2 Multi-Site Commands are marked in red Various options do exist but the recommended design choices are: • Fabric Internal IGP Underlay, iBGP Overlay • DCI (primary choice) eBGP Underlay, eBGP Overlay Route Server for DCI Overlay peerings DC Core for reachability across n Sites • DCI (alternative option) Any Routing Protocol Underlay, eBGP Overlay Full-Mesh for DCI Overlay peerings Back-to-Back Site Reachability (physical, full- mesh) BRKDCN-2035 83
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – Enable Border Gateway Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 BGW2 feature nv overlay nv overlay evpn feature bgp feature interface-vlan feature vn-segment-vlan-based evpn multisite border-gateway BGW2BGW1 BRKDCN-2035 84
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – BGW 1 Loopback & VTEP Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 interface loopback1 description PIP VTEP ip address 10.1.1.101/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface loopback100 description VIP Multi-Site 1 ip address 10.1.1.111/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface loopback0 description RID ip address 10.10.10.101/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode BGW1 BRKDCN-2035 85
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – BGW 2 Loopback & VTEP Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP …. BGW2 interface loopback1 description PIP VTEP ip address 10.1.1.102/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface loopback100 description VIP Multi-Site 1 ip address 10.1.1.111/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface loopback0 description RID ip address 10.10.10.102/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode BGW2 BRKDCN-2035 86
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – Fabric Link Tracking BGW 1 Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 interface Ethernet1/53 description TO-SPINE1 ip address 10.0.1.1/30 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode evpn multisite fabric-tracking interface Ethernet1/54 description TO-SPINE2 ip address 10.0.2.1/30 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode evpn multisite fabric-tracking BGW1 Allows to bring down the PIP/VIP loopback interfaces when the BGW is isolated from the spines BRKDCN-2035 87
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – Fabric Link Tracking BGW 2 Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP …. BGW2 interface Ethernet1/53 description TO-SPINE1 ip address 10.0.1.5/30 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode evpn multisite fabric-tracking interface Ethernet1/54 description TO-SPINE2 ip address 10.0.2.5/30 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode evpn multisite fabric-tracking BGW2 BRKDCN-2035 88
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 Setup – Multi-Site Underlay Interface Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 BGW2 interface Ethernet1/1 description TO-DC-CORE1 ip address 10.111.111.1/30 tag 12345 evpn multisite dci-tracking interface Ethernet1/2 description TO-DC-CORE2 ip address 10.111.222.1/30 tag 12345 evpn multisite dci-tracking BGW1 DCI interface Ethernet1/1 description TO-DC-CORE1 ip address 10.222.111.1/30 tag 12345 evpn multisite dci-tracking interface Ethernet1/2 description TO-DC-CORE2 ip address 10.222.222.1/30 tag 12345 evpn multisite dci-tracking BGW2 Allows to bring down the PIP/VIP loopback interfaces when the BGW is isolated from the DC core BRKDCN-2035 89
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 BGW 1 Setup – Multi-Site Overlay Peering Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 DCI RS router bgp 65501 router-id 10.10.10.101 address-family ipv4 unicast redistribute direct route-map REDIST-LOCAL neighbor 10.111.111.2 remote-as 65599 update-source ethernet1/1 address-family ipv4 unicast neighbor 10.111.222.2 remote-as 65599 update-source ethernet1/2 address-family ipv4 unicast neighbor 10.99.99.201 remote-as 65599 update-source loopback0 ebgp-multihop 5 peer-type fabric-external address-family l2vpn evpn rewrite-evpn-rt-asn send-community send-community both BGW1 RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 90
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 BGW 2 Setup – Multi-Site Overlay Peering Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP …. DCI RS router bgp 65501 router-id 10.10.10.102 address-family ipv4 unicast redistribute direct route-map REDIST-LOCAL neighbor 10.222.111.2 remote-as 65599 update-source ethernet1/1 address-family ipv4 unicast neighbor 10.222.222.2 remote-as 65599 update-source ethernet1/2 address-family ipv4 unicast neighbor 10.99.99.201 remote-as 65599 update-source loopback0 ebgp-multihop 5 peer-type fabric-external address-family l2vpn evpn rewrite-evpn-rt-asn send-community send-community both BGW1 RS – Route Server (eBGP ”Route Reflector”) BGW2 BRKDCN-2035 91
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 Setup – Multi-Site Overlay Peering Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 DCI RS RS – Route Server (eBGP ”Route Reflector”) BGW2 peer-type fabric-external • Enables Next-Hop Rewrite for Multi-Site • Defines Site External BGP neighbors for EVPN exchange rewrite-evpn-rt-asn • Rewrites Route-Target Auto information to simplify MAC-VRF and IP-VRF configuration • Normalizes outgoing Route-Targets AS number to match remote AS number • Uses BGP configured Neighbors Remote AS BRKDCN-2035 92
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Site 1 Setup – Multi-Site Overlay Peering DC Core (Layer-3 Unicast) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 peer-type fabric-external peer-type fabric-external BGP Update: MAC: 0000.3010.1101 (L2VNI 30001) IP: 192.168.20.101 (L3VNI 50001) NH: 10.2.2.222 RMAC: BGW-VMAC2 BGP Update: MAC: 0000.3010.1101 (L2VNI 30001) IP: 192.168.20.101 (L3VNI 50001) NH: 10.1.1.111 RMAC: BGW-VMAC1 Rewrite Next-Hop IP and Next- Hop MAC (RMAC) based on Neighbor Site BGW BGP Update: MAC: 0000.3010.1101 (L2VNI 30001) IP: 192.168.20.101 (L3VNI 50001) NH: 10.1.1.1 RMAC: Leaf1 Rewrite Next-Hop IP and Next- Hop MAC (RMAC) based on Neighbor Site BGW BRKDCN-2035 93
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Site 1 Setup – Multi-Site Overlay Peering DC Core (Layer-3 Unicast) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host2 0000.3020.2101 192.168.20.101 rewrite-evpn-rt-asn rewrite-evpn-rt-asn BGP Update: Remote AS: 65502 VNI: 50001 Route-Target: 65502:50001 Rewrite Route-Target based on BGP Neighbors Remote ASN BGP Update: Remote AS : 65501 VNI: 50001 Route-Target: 65501:50001 BGP Update: Remote AS: 65502 VNI: 50001 Route-Target: 65502:50001 BRKDCN-2035 94
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 Setup – Anycast BGW VTEP Configuration Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 DCI RS – Route Server (eBGP ”Route Reflector”) interface nve1 no shutdown host-reachability protocol bgp multisite ethernet-segment 7 system-mac 0000.0000.0001 source-interface loopback1 multisite border-gateway interface loopback100 member vni 30010 multisite ingress-replication mcast-group 239.1.1.1 member vni 30011-30020 mcast-group 239.1.1.2 member vni 50001 associate-vrf BGW1 BGW2 BGW2 BRKDCN-2035 95
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 Setup – Anycast BGW VTEP Configuration Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 DCI BGW2 multisite ethernet-segment • Defines the discriminator for Sites in a common Domain system-mac • Defines the Multi-Site Site-Id (6 octets hex) multisite border-gateway interface loopback# • Defines the Loopback Interface used for the Border Gateway Virtual IP Address (VIP) multisite ingress-replication • Per-VNI knob for extending Layer-2 VNI • Defines the Multi-Site BUM Replication method BRKDCN-2035 96
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 Setup – Multi-Site Overlay Traffic Policy Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 DCI BGW2 • BUM Traffic Policing • Limits Broadcast, Unknown Unicast and Layer-2 Multicast Traffic across Multi-Site • Level 0 = No B/U/M Forwarding • Level 100 = All B/U/M Forwarding Forwarding • Enforced on Encapsulation towards remote Sites evpn storm-control broadcast level 10 evpn storm-control unicast level 10 evpn storm-control multicast level 10 BGW1 BGW2 BRKDCN-2035 97
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Setup – Multi-Site Overlay Traffic Policy SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Multi-Site Overlay Site 1 Overlay Site n Baremetal BUM BGW BGW BGW BGW BRKDCN-2035 98
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Setup – Multi-Site Overlay Traffic Policy SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Multi-Site Overlay Site 1 Overlay Site n Baremetal BUM Storm Control Broadcast 0-100% Unknown Unicast 0-100% Multicast 0-100% Storm Control Broadcast 0-100% Unknown Unicast 0-100% Multicast 0-100% BGW BGW BGW BGW BRKDCN-2035 99
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Setup – Multi-Site BUM Replication Modes SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Multi-Site Overlay Site 1 Overlay Site n Multicast Multicast Ingress Replication BGW BGW BGW BGW BRKDCN-2035 100
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Setup – Multi-Site BUM Replication Modes SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Multi-Site Overlay Site 1 Overlay Site n Ingress Replication Ingress Replication Ingress Replication BGW BGW BGW BGW BRKDCN-2035 101
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Setup – Multi-Site BUM Replication Modes SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Multi-Site Overlay Site 1 Overlay Site n Ingress Replication Multicast Ingress Replication BGW BGW BGW BGW BRKDCN-2035 102
Connectivity to the External Layer 3 Domain
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Connectivity to the External Layer 3 Domain The BGW nodes can be used to provide Layer-3 external connectivity to each site Different connectivity models are supported • VRF-Lite peering with an external pair of WAN Edge routers • MP-BGP EVPN peering with the external WAN Edge routers (GOLF) • Dedicated or shared pair of WAN Edge routers across sites External Layer-3 network may be different from the DCI network used for inter-site communication BRKDCN-2035 104
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Border Gateway and VRF-Lite Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 VRF-CVRF-BVRF-ASeparate routing peering for each VRF (IGP or eBGP) Dedicated interface (logical or physical) for each VRF BRKDCN-2035 105
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Border Gateway and GOLF Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 VRF-CVRF-BVRF-A Single MP-BGP EVPN instance to exchange routes for all VRFs VXLAN Data Plane between BGW and WAN Edge Router BRKDCN-2035 106
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Fabric DCI Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host2 0000.3020.2101 192.168.20.101 Host3 0000.3010.1102 192.168.10.102 Multi-Site – Shared Internet/WAN Gateways Internet/WAN BorderPE BorderPE Inter-Site VXLAN Communication between Border Gateways BRKDCN-2035 107
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site – Per Site Internet/WAN Gateway DC Core (Layer-3 Unicast) MPLS L3VPN Fabric DCI Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host2 0000.3020.2101 192.168.20.101 Host3 0000.3010.1102 192.168.10.102 Internet/WAN BorderPE BorderPE BorderPE BorderPE Inter-Site VXLAN Communication between Border Gateways BRKDCN-2035 108
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public MPLS L3VPN Fabric DCI Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host2 0000.3020.2101 192.168.20.101 Host3 0000.3010.1102 192.168.10.102 Internet/WAN BorderPE BorderPE BorderPE BorderPE Multi-Site – Consolidated WAN and DCI NetworkPerform simple routing for inter-site flows, VXLAN (or VRF-Lite) to MPLS VPN hand-off for north-south communication Inter-Site VXLAN Communication between Border Gateways BRKDCN-2035 109
Ingress and Egress Traffic Optimization
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP BGW WAN Active FWActive FW The stretching of Layer-2 domains across separate sites may lead to the creation of asymmetric traffic paths Deploying independent stateful services (like FWs) across sites would result in traffic drops In this case it is required to ensure the symmetry of ingress and egress communication paths Ingress and Egress Traffic Optimization The Issue of Extending Layer 2 Domains BGW BGW BGW DC Core (Layer-3 Unicast) BRKDCN-2035 111
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Guarantee routing symmetry with the outside of the Data Center • Egress Always prefer the local BGW • Ingress Steer traffic to the specific destination End-Point’s location Maintain optimal routing over the dedicated DCI network (if existing) for Server-to-Server traffic • The DC fabric must discriminate between DC and WAN destinations If required provide a fallback path via DCI for WAN isolation situations Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP BGW BGW BGW BGW WAN Ingress and Egress Traffic Optimization Maintaining Traffic Symmetry over Optimal Paths BRKDCN-2035 112
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 Multi-Site – Egress Path Optimization 172.16.1.10 172.16.1.0/24 Border-PEs 1-2 172.16.1.0/24 Border-PEs 3-4 172.16.1.0/24 VIP1 172.16.1.0/24 VIP2 eBGP-EVPN Less preferred advertisement of 172.16.1.0 because of longer AS-Path BRKDCN-2035 113
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 Multi-Site – Egress Path Optimization 172.16.1.10 eBGP-EVPN Optimized Egress Traffic Path Optimized Egress Traffic Path BRKDCN-2035 114
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 Multi-Site – Egress Path Optimization WAN Isolation Scenario 172.16.1.10 172.16.1.0/24 Border-PEs 3-4 172.16.1.0/24 VIP1 172.16.1.0/24 VIP2 eBGP-EVPN WAN Isolation Scenario 172.16.1.0/24 VIP2 BRKDCN-2035 115
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 Multi-Site – Ingress Path Optimization 192.168.10.0/24 BGW 1-2 192.168.10.101/32 BGW-1-2 192.168.10.0/24 BGW 3-4 192.168.10.102/32 BGW 3-4 192.168.10.101/32 Leaf1 192.168.10.102/32 -> Leaf3 eBGP-EVPN Host routes advertised across sites but NOT re- advertised toward the local Border-PEs 192.168.10.0/24 Border-PE 1-4 192.168.10.101/32 Border-PE 1-2 192.168.10.102/32 Border-PE 3-4Host routes advertisement in the WAN Deploying LISP on the Border-PEs is a viable alternative to host routes advertisement Filter out host routes received from remote sites. Only announce local host route information BRKDCN-2035 116
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 Multi-Site – Ingress Path Optimization eBGP-EVPN 192.168.10.0/24 Border-PE 1-4 192.168.10.101/32 Border-PE 1-2 192.168.10.102/32 Border-PE 3-4 Optimized Ingress Traffic Path Optimized Ingress Traffic Path BRKDCN-2035 117
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 eBGP-EVPN 192.168.10.0/24 Border-PE 3-4 192.168.10.101/32 Border-PE 1-2 192.168.10.102/32 Border-PE 3-4 Multi-Site – Ingress Path Optimization WAN Isolation Scenario WAN Isolation Scenario BRKDCN-2035 118
Network Services Integration
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Network Services Integration Couple of different options where to connect network services: 1. Service Leaf nodes: recommended to connect devices used for east- west communication 2. Border Gateway Nodes: used to connect network services for north- south traffic flows Depending on the specifics of the Multi-Site deployment, the following deployment models would be possible: • Active/Standby Service Nodes pair connected to different sites • Active/Active cluster of Service Nodes deployed across sites • Independent Active/Standby Service nodes pairs deployed in separate sites BRKDCN-2035 120
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Network Services Integration Active/Standby Pair Deployed across Sites Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP BGW WAN Standby FWActive FW BGW BGW BGW DC Core (Layer-3 Unicast) Requirement to extend Layer 2 communication between Active/Standby nodes for keep-alives and state information exchange Perimeter service nodes connected to VPC Border Gateways Ingress and egress traffic always traversing the Active node in Site 1 No issues related to the creation of asymmetric traffic paths East-West flows must be hair-pinned to the active FW connected to the Service leaf nodes in Site 1 • Need to properly dimension bandwidth in the DC Core to accommodate for this extra traffic Active FW Standby FW Baremetal BaremetalBaremetal North-South traffic flows North-South traffic flows East-West traffic flows BRKDCN-2035 121
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Network Services Integration Active/Active Cluster of Service Nodes Deployed across Sites Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP BGW WAN Active/Active FW Cluster BGW BGW BGW DC Core (Layer-3 Unicast) Requirement to extend Layer 2 communication between Active/Active nodes for intra-cluster communication and traffic redirection Perimeter service nodes connected to VPC Border Gateways Asymmetric traffic issues taken care by native intra-cluster traffic redirection Option to deploy ingress/egress optimization technique to avoid inter-site traffic hair-pinning Service node cluster integration not supported at FCS and planned for a future SW release Active/Active FW Cluster Logical Intra- Cluster Link (ICL) Baremetal North-South traffic flows BRKDCN-2035 122
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Network Services Integration Independent Active/Standby Pair Deployed in Separate Sites Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP BGW WAN Active/Standby FW Active/Standby FW BGW BGW BGW DC Core (Layer-3 Unicast) Mandates the deployment of Ingress/Egress traffic optimization to avoid creation of asymmetric traffic path for north-south communication Active/Standby nodes can use direct links to sync state Perimeter service nodes can connected to Anycast Border Gateways Active/Standby pair (or cluster) still required for service nodes used for east-west traffic flows Baremetal Baremetal Active FW Standby FW North-South traffic flows North-South traffic flows BRKDCN-2035 123
Legacy Site Integration
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site and Legacy Site Integration Extend Layer-2 and Layer-3 connectivity between sites • Coexistence and/or application migration use cases Proposed approach is to deploy a pair of ‘remote’ VPC Border Gateways in the legacy site • Offers native Multi-Site functionalities (BUM containment, etc) to the legacy site SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Site VTEP VTEP Legacy Site VTEP VTEP BGW BGW BGW BGW Pair of VPC Border Gateways BRKDCN-2035 125
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site and Legacy Site Integration Layer-2 Connectivity with the ‘Remote’ BGW Legacy Site VTEP VTEP BGW BGW Legacy Aggregation Layer devices support MLAG Single logical link to extend VLANs toward the Greenfield VXLAN EVPN site VLANs mapped to L2VNIs on the Border Gateways Legacy Site VTEP VTEP BGW BGW Legacy Aggregation Layer devices do not support MLAG Single port-channel from each aggregation layer device VLANs mapped to L2VNIs on the Border Gateways Recommended to move the STP root to the BGW devices BRKDCN-2035 126
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site and Legacy Site Integration Layer-2 Control Plane Exchange across Sites Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Site VTEP VTEP Legacy Site VTEP VTEP BGW BGW BGW BGW VIP1 10.1.1.111 VIP2 10.2.2.222 MAC NH 0000.3010.1101 Leaf1 0000.3010.1102 VIP2 MAC NH 0000.3010.1101 VIP1 0000.3010.1102 Po1 Po1 All End-Points in the legacy site are learned as directly connected to the BGW eBGP-EVPN BRKDCN-2035 127
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Integration between Anycast Gateway and legacy default gateway (HSRP, VRRP, etc.) not initially supported with VXLAN Multi-Site First option is to keep on the legacy network the active default gateway for the stretched IP subnets SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Site VTEP VTEP Legacy Site VTEP VTEP BGW BGW BGW BGW Default Gateway deployed on the legacy aggregation devices Multi-Site and Legacy Site Integration Default Gateway Deployment – Option 1 L3 L2 Greenfield VXLAN EVPN Fabric only offers L2 services for the stretched IP subnets BRKDCN-2035 128
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Recommended approach is to migrate the default gateway from the legacy aggregation devices to the Border Gateways (VXLAN EVPN Anycast Gateway) Optimize routing between End-Points deployed across sites SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Site VTEP VTEP Legacy Site VTEP VTEP BGW BGW BGW BGW Default Gateway migrated to the Border Gateways (VXLAN EVPN Anycast Gateway) Multi-Site and Legacy Site Integration Default Gateway Deployment – Option 2 L3 L2 Greenfield VXLAN EVPN Fabric offers L2 and L3 services for the stretched IP subnets Legacy infrastructure offers only L2 services Distributed Anycast Gateway function L3 L2 BRKDCN-2035 129
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site and Legacy Site Integration Layer-3 Control Plane Exchange across Sites Baremetal Host 1 0000.3010.1101 192.168.10.101 SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Site VTEP VTEP Legacy Site VTEP VTEP BGW BGW BGW BGW VIP1 10.1.1.111 VIP2 10.2.2.222 Po1 All End-Points in the legacy site are learned as directly connected to the BGW eBGP-EVPN L3 L2 Baremetal Host 3 0000.3010.1102 192.168.20.101 IP NH 192.168.10.101 Leaf1 192.168.20.101 VIP1 IP L3VNI 192.168.10.101 VIP1 192.168.20.101 Po1 BRKDCN-2035 130
Migration to Multi- Site
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Migration to Multi-Site Use Cases 1. Site addition: need to connect a Greenfield VXLAN EVPN Fabric to an existing VXLAN EVPN Fabric built with 1st generation Nexus 9000 2. Migrating a VXLAN Multi-Pod Fabric to Multi-Site 3. Migrating a VXLAN Multi-Fabric design to Multi-Site BRKDCN-2035 132
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Migration to Multi-Site Site Addition SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Existing VXLAN EVPN Fabric VTEP VTEP BGW BGW SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Fabric VTEP VTEP BGW BGW SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Existing VXLAN EVPN Fabric Step 1: add a pair of Border Gateways to the existing VXLAN EVPN Fabric, running the proper SW release supporting Multi-Site Note: no requirement to change the HW/SW version on existing leaf nodes Step 2: connect the BGW to the inter-site network and establish control plane peering with the BGW in the Greenfield Fabric Step 3: configure on the BGW the L2VNIs and L3VNIs to be extended eBGP-EVPN BRKDCN-2035 133
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Migration to Multi-Site Multi-Fabric to Multi-Site SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 2 VTEP VTEP No Underlay Extension BGW BGW BGW BGW Step 1: add a pair of Border Gateways to each Pod (if needed) and connect them to the spines and to the inter-site network Step 2: upgrade the SW on both Fabrics BGW to be able to support Multi-Site Step 3: establish control plane adjacencies across sites Step 4: disconnect the previously used DCI and extend Layer-2 and Layer-3 across Multi-Site SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric 2 VTEP VTEP BRKDCN-2035 134
Conclusion
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multiple Overlay Domains – Interconnected & Controlled • Scaling and Segregating VXLAN EVPN Networks Multiple Overlay Control-Plane Domains – Interconnected & Controlled • Limited Overlay Control-Plane Update Propagation Multiple Underlay Domains - Isolated • Isolated Underlay Domains – No need for Extension Multiple Replication Domains for BUM – Interconnected & Controlled • Individual BUM flooding domain with Traffic control Multi-Site Advantages – ”The Multiple” BRKDCN-2035 136
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public • New IETF Draft for Multi-Site Design • Multi-site EVPN based VXLAN using Border Gateways • https://tools.ietf.org/html/draft-sharma-multi-site-evpn VXLAN EVPN – Multi-Site BRKDCN-2035 137
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public • Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card. • Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us. Complete Your Online Session Evaluation Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Continue Your Education • Demos in the Cisco campus • Walk-in Self-Paced Labs • Lunch & Learn • Meet the Engineer 1:1 meetings • Related sessions BRKDCN-2035 139
Thank you
Brkdcn 2035 multi-x

More Related Content

PDF
Building DataCenter networks with VXLAN BGP-EVPN
Cisco Canada
 
PDF
EVPN Introduction
Bangladesh Network Operators Group
 
PDF
MPLS L3 VPN Deployment
APNIC
 
PPTX
Scaleway Approach to VXLAN EVPN Fabric
Scaleway
 
PPTX
Operationalizing VRF in the Data Center
Cumulus Networks
 
PPTX
Vxlan control plane and routing
Wilfredzeng
 
PDF
IOS Cisco - Cheat sheets
Alejandro Marin
 
PDF
Chapter 8 - IP Subnetting, Troubleshooting and Introduction to NAT 9e
adpeer
 
Building DataCenter networks with VXLAN BGP-EVPN
Cisco Canada
 
EVPN Introduction
Bangladesh Network Operators Group
 
MPLS L3 VPN Deployment
APNIC
 
Scaleway Approach to VXLAN EVPN Fabric
Scaleway
 
Operationalizing VRF in the Data Center
Cumulus Networks
 
Vxlan control plane and routing
Wilfredzeng
 
IOS Cisco - Cheat sheets
Alejandro Marin
 
Chapter 8 - IP Subnetting, Troubleshooting and Introduction to NAT 9e
adpeer
 

What's hot (20)

DOCX
Ccna command
Siddhartha Rajbhatt
 
PPTX
Cisco Live Milan 2015 - BGP advance
Bertrand Duvivier
 
PDF
Cisco router-commands
Robin Rohit
 
PPTX
Junos vs ios Troubleshooting comands
sandeep kumar
 
PPTX
Introduction to vxlan
Mohammed Umair
 
PDF
Segment Routing Technology Deep Dive and Advanced Use Cases
Cisco Canada
 
PPTX
Vxlan deep dive session rev0.5 final
KwonSun Bae
 
PDF
Cisco Live! :: Introduction to Segment Routing :: BRKRST-2124 | Las Vegas 2017
Bruno Teixeira
 
PDF
Designing Multi-tenant Data Centers Using EVPN
Anas
 
PDF
Segment Routing Advanced Use Cases - Cisco Live 2016 USA
Jose Liste
 
PPT
Nat pat
CYBERINTELLIGENTS
 
PPT
Mpls L3_vpn
Reza Farahani
 
PDF
Ccnp workbook network bulls
Swapnil Kapate
 
PPT
Routing and OSPF
arpit
 
PPTX
Ethernet VPN (EVPN) EVerything Provider Needs
CSUC - Consorci de Serveis Universitaris de Catalunya
 
PDF
VRF (virtual routing and forwarding)
Netwax Lab
 
PDF
MPLS Presentation
Unni Kannan VijayaKumar
 
PDF
CCNAv5 - S2: Chapter3 Vlans
Vuz Dở Hơi
 
PDF
Implementing cisco mpls
Matiullah Jamil
 
PPTX
EVPN-Presentation.pptx
VimalMallick
 
Ccna command
Siddhartha Rajbhatt
 
Cisco Live Milan 2015 - BGP advance
Bertrand Duvivier
 
Cisco router-commands
Robin Rohit
 
Junos vs ios Troubleshooting comands
sandeep kumar
 
Introduction to vxlan
Mohammed Umair
 
Segment Routing Technology Deep Dive and Advanced Use Cases
Cisco Canada
 
Vxlan deep dive session rev0.5 final
KwonSun Bae
 
Cisco Live! :: Introduction to Segment Routing :: BRKRST-2124 | Las Vegas 2017
Bruno Teixeira
 
Designing Multi-tenant Data Centers Using EVPN
Anas
 
Segment Routing Advanced Use Cases - Cisco Live 2016 USA
Jose Liste
 
Nat pat
CYBERINTELLIGENTS
 
Mpls L3_vpn
Reza Farahani
 
Ccnp workbook network bulls
Swapnil Kapate
 
Routing and OSPF
arpit
 
Ethernet VPN (EVPN) EVerything Provider Needs
CSUC - Consorci de Serveis Universitaris de Catalunya
 
VRF (virtual routing and forwarding)
Netwax Lab
 
MPLS Presentation
Unni Kannan VijayaKumar
 
CCNAv5 - S2: Chapter3 Vlans
Vuz Dở Hơi
 
Implementing cisco mpls
Matiullah Jamil
 
EVPN-Presentation.pptx
VimalMallick
 
Ad

Similar to Brkdcn 2035 multi-x (20)

PDF
Sharing session of cisco BRKDCN 1.6.4.5.pdf
pintudunia
 
PDF
CISCO DCNM.pdf
JulianBelisario1
 
PDF
NetBox as the Source of Truth for Cisco NSO Configurations
Hank Preston
 
PDF
Cisco SDN/NVF Innovations (SDN NVF Day ITB 2016)
SDNRG ITB
 
PDF
BRKDCN-2670 Day2 operations for Datacenter VxLAN EVPN fabrics.pdf
HarryH11
 
PPTX
The Data Center Network Evolution
Cisco Canada
 
PDF
BRKCRS-2110.pdf
Asif Qureshi
 
PPTX
Presentation on CISCO ACI basic fundamental
mdmahmuduli427
 
PDF
1-Infrastructure as Code for NXOS and NDFC with Ansible.pdf
AntonioIsipJr1
 
PDF
Network Function Virtualisation (NFV) BoF
APNIC
 
PDF
BRKDCN-2613.pdf Cisco AI DC NVIDIA presentation
demidovs1
 
PDF
Integration and Interoperation of existing Nexus networks into an ACI Archite...
Cisco Canada
 
PDF
Open v ran
Rajasa Pramudya Wardhana
 
PDF
MPLS EVPN...............................
AntonioIsipJr1
 
PDF
EVPN: Migration from Legacy to Modern Architecture
6fm7gg7hxs
 
PDF
BRKACI-1003 ACI Brownfield Migration - Real World Experiences and Best Practi...
fcesargonca
 
PDF
Cisco at v mworld 2015 theater presentation brfarnha
ldangelo0772
 
PDF
VXLAN Design and Deployment.pdf
NelAlv1
 
PDF
CiscoACI-BRKACI-2608presentationUploaded.pdf
LeeHiu1
 
PDF
BRKSP-2551 - Introduction to Segment Routing.pdf
fcesargonca
 
Sharing session of cisco BRKDCN 1.6.4.5.pdf
pintudunia
 
CISCO DCNM.pdf
JulianBelisario1
 
NetBox as the Source of Truth for Cisco NSO Configurations
Hank Preston
 
Cisco SDN/NVF Innovations (SDN NVF Day ITB 2016)
SDNRG ITB
 
BRKDCN-2670 Day2 operations for Datacenter VxLAN EVPN fabrics.pdf
HarryH11
 
The Data Center Network Evolution
Cisco Canada
 
BRKCRS-2110.pdf
Asif Qureshi
 
Presentation on CISCO ACI basic fundamental
mdmahmuduli427
 
1-Infrastructure as Code for NXOS and NDFC with Ansible.pdf
AntonioIsipJr1
 
Network Function Virtualisation (NFV) BoF
APNIC
 
BRKDCN-2613.pdf Cisco AI DC NVIDIA presentation
demidovs1
 
Integration and Interoperation of existing Nexus networks into an ACI Archite...
Cisco Canada
 
Open v ran
Rajasa Pramudya Wardhana
 
MPLS EVPN...............................
AntonioIsipJr1
 
EVPN: Migration from Legacy to Modern Architecture
6fm7gg7hxs
 
BRKACI-1003 ACI Brownfield Migration - Real World Experiences and Best Practi...
fcesargonca
 
Cisco at v mworld 2015 theater presentation brfarnha
ldangelo0772
 
VXLAN Design and Deployment.pdf
NelAlv1
 
CiscoACI-BRKACI-2608presentationUploaded.pdf
LeeHiu1
 
BRKSP-2551 - Introduction to Segment Routing.pdf
fcesargonca
 
Ad

More from Mason Mei (20)

PDF
Ovn vancouver
Mason Mei
 
PDF
11 zhuai chuanpu h3 c
Mason Mei
 
PDF
10 2013 sdn summit ch reviewed-new
Mason Mei
 
PDF
08 sdn system intelligence short public beijing sdn conference - 130828
Mason Mei
 
PDF
07 tang xiongyan
Mason Mei
 
PDF
06 duan xiaodong
Mason Mei
 
PDF
05 zhao huiling
Mason Mei
 
PDF
04 hou ziqiang
Mason Mei
 
PDF
03 jiang lintao
Mason Mei
 
PDF
02 china sdn conf ron keynote
Mason Mei
 
PDF
01 dan chinese-chinese sdn china 2013- dan's keynote draft aug 14 2013
Mason Mei
 
PDF
New idc architecture
Mason Mei
 
PPTX
H3 cswitch2015
Mason Mei
 
PDF
201507131408448146
Mason Mei
 
PDF
16 vxlan配置指导-整本手册
Mason Mei
 
PDF
Atf 3 q15-8 - introducing macro-segementation
Mason Mei
 
PDF
Atf 3 q15-9 - summary and close
Mason Mei
 
PDF
Atf 3 q15-7 - delivering cloud scale workflow automation control and visibili...
Mason Mei
 
PDF
Atf 3 q15-4 - scaling the the software driven cloud network
Mason Mei
 
PDF
Atf 3 q15-6 - solutions for scaling the cloud computing network infrastructure
Mason Mei
 
Ovn vancouver
Mason Mei
 
11 zhuai chuanpu h3 c
Mason Mei
 
10 2013 sdn summit ch reviewed-new
Mason Mei
 
08 sdn system intelligence short public beijing sdn conference - 130828
Mason Mei
 
07 tang xiongyan
Mason Mei
 
06 duan xiaodong
Mason Mei
 
05 zhao huiling
Mason Mei
 
04 hou ziqiang
Mason Mei
 
03 jiang lintao
Mason Mei
 
02 china sdn conf ron keynote
Mason Mei
 
01 dan chinese-chinese sdn china 2013- dan's keynote draft aug 14 2013
Mason Mei
 
New idc architecture
Mason Mei
 
H3 cswitch2015
Mason Mei
 
201507131408448146
Mason Mei
 
16 vxlan配置指导-整本手册
Mason Mei
 
Atf 3 q15-8 - introducing macro-segementation
Mason Mei
 
Atf 3 q15-9 - summary and close
Mason Mei
 
Atf 3 q15-7 - delivering cloud scale workflow automation control and visibili...
Mason Mei
 
Atf 3 q15-4 - scaling the the software driven cloud network
Mason Mei
 
Atf 3 q15-6 - solutions for scaling the cloud computing network infrastructure
Mason Mei
 

Recently uploaded (20)

PDF
Exploring VPS Hosting Trends for SMBs in 2025
Liquid Web
 
PPTX
Power Point - Lesson 3_2.pptx grad school presentation
JoyPPD
 
PPT
250152213-Excitation-SystemWERRT (1).ppt
woldemariamworku2
 
PDF
Introduction to the IoT system, how the IoT system works
TinBinh
 
PPTX
E -tech empowerment technologies PowerPoint
kylebalatero12
 
PDF
The New Creative Director: How AI Tools for Social Media Content Creation Are...
SageTitans
 
PPTX
Database Information System - Management Information System
faizhossain3
 
PDF
SASE Traffic Flow - ZTNA Connector-1.pdf
mackwell7777
 
PPTX
June-4-Sermon-Powerpoint.pptx USE THIS FOR YOUR MOTIVATION
KuyaRogie
 
DOCX
Unit-3 cyber security network security of internet system
shivangisharma636228
 
PDF
Smart Home Technology for Health Monitoring (www.kiu.ac.ug)
publication11
 
PPTX
Internet___Basics___Styled_ presentation
poonam62133
 
PDF
mera desh ae watn.(a source of motivation and patriotism to the youth of the ...
DtMalhonSharma
 
PPT
Ethics in Information System - Management Information System
faizhossain3
 
PDF
Vigrab.top – Online Tool for Downloading and Converting Social Media Videos a...
Vigrab Top
 
PPTX
Module 1 - Cyber Law and Ethics 101.pptx
anantyakhrisna1
 
PPTX
Digital Literacy And Online Safety on internet
riksa rifqi
 
PDF
Sims 4 Historia para lo sims 4 para jugar
lolpopy34
 
PPTX
SAP Ariba Sourcing PPT for learning material
NKKumar16
 
PPTX
INTERNET------BASICS-------UPDATED PPT PRESENTATION
poonam62133
 
Exploring VPS Hosting Trends for SMBs in 2025
Liquid Web
 
Power Point - Lesson 3_2.pptx grad school presentation
JoyPPD
 
250152213-Excitation-SystemWERRT (1).ppt
woldemariamworku2
 
Introduction to the IoT system, how the IoT system works
TinBinh
 
E -tech empowerment technologies PowerPoint
kylebalatero12
 
The New Creative Director: How AI Tools for Social Media Content Creation Are...
SageTitans
 
Database Information System - Management Information System
faizhossain3
 
SASE Traffic Flow - ZTNA Connector-1.pdf
mackwell7777
 
June-4-Sermon-Powerpoint.pptx USE THIS FOR YOUR MOTIVATION
KuyaRogie
 
Unit-3 cyber security network security of internet system
shivangisharma636228
 
Smart Home Technology for Health Monitoring (www.kiu.ac.ug)
publication11
 
Internet___Basics___Styled_ presentation
poonam62133
 
mera desh ae watn.(a source of motivation and patriotism to the youth of the ...
DtMalhonSharma
 
Ethics in Information System - Management Information System
faizhossain3
 
Vigrab.top – Online Tool for Downloading and Converting Social Media Videos a...
Vigrab Top
 
Module 1 - Cyber Law and Ethics 101.pptx
anantyakhrisna1
 
Digital Literacy And Online Safety on internet
riksa rifqi
 
Sims 4 Historia para lo sims 4 para jugar
lolpopy34
 
SAP Ariba Sourcing PPT for learning material
NKKumar16
 
INTERNET------BASICS-------UPDATED PPT PRESENTATION
poonam62133
 

Brkdcn 2035 multi-x

  • 2. VXLAN BGP EVPN based Multi-Pod, Multi- Fabric, Multi-Site Max Ardica – Principal Engineer Lukas Krattiger – Principal Engineer BRKDCN-2035
  • 3. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Cisco Spark Questions? Use Cisco Spark to chat with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click “Join the Discussion” 3. Install Spark or go directly to the space 4. Enter messages/questions in the space How cs.co/ciscolivebot#BRKDCN-2035Cisco Spark spaces will be available until July 3, 2017.
  • 4. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Who Are the Presenters? Max Ardica Principal Engineer - INSBU Lukas Krattiger Principal Engineer – INSBU @ccie21921 BRKDCN-2035 4
  • 5. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Session Objectives At the end of the session, the participants should be able to: Articulate the different deployment options to interconnect VXLAN EVPN Networks (Multi-Pod vs. Multi-Fabric vs. Multi- Site) Understand the functionalities and specific design considerations associated to the new VXLAN Multi-Site architecture Initial assumption: The audience already has a good knowledge of the VXLAN EVPN technology and its use to deploy modern Data Center Fabrics BRKDCN-2035 5
  • 6. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Session Reference • BRKDCN-2304 • L4-L7 Service Integration in Multi-Tenant VXLAN EVPN Data Center Fabrics • BRKDCN-3378 • Building DataCenter Networks with VXLAN BGP-EVPN • Wednesday, Jun 28, 1:30 pm • BRKDCN-2125 • Overlay Management and Visibility with VXLAN • Thursday, Jun 29, 10:30 am • BRKDCN-2342 • Programmable Fabric Automation and Management with DCNM 10 • Thursday, Jun 29, 1:00 p.m. 6BRKDCN-2035
  • 7. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Agenda Introduction VXLAN EVPN Interconnect Evolution • Multi-Pod • Multi-Fabric • Multi-Site VXLAN EVPN Multi-Site Deep Dive • Walkthrough • Control- and Data-Plane • Deployment Considerations Conclusions and Q&A
  • 9. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Layer 2 Domain Elasticity Local LAN Fabric Extended LAN fabric VN-link notifications IP Mobility Optimal Ingress and Egress Routing VM-awareness VXLAN, DFA, ACI, VN-link Storage Elasticity SAN Extensions Network Service Localization Any service anywhere OTV OTV OTV OTV Fabric Consolidation Unified Fabric & I/O Device Virtualization Segmentation Data Center Interconnect – DCI Model Connecting Virtualized Data Centers Multi-tenancy/Segmentation Segment-IDs in VXLAN, LISP, FabricPath, and OTV Storage Solutions & Partners: FCIP, I/O Acceleration EMC, NetApp BRKDCN-2035 9
  • 10. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Back Then Yet Another Encapsulation Flood & Learn (Multicast-based) Data-Plane only Yesterday VXLAN for the Data Center – Intra-DC Control-Plane Active VTEP Discovery Multicast and Unicast Now! VXLAN for DCI – Inter-DC DCI Ready ARP/ND caching/suppress Multi-Homing Failure Domain Isolation Loop Protection VXLAN Evolves as the Control Plane Evolves! BRKDCN-2035 10
  • 11. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Back Then VXLAN for Interconnecting Networks BRKDCN-2035 11
  • 12. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Inter-X Connectivity • Single Fabric with End- to-End Encapsulation • Build Hierarchy in the Underlay – Flatten it in the Overlay Multi-Pod Overlay VTE P VTE P VTE P VTE P Bar em eta l Bar em eta l Fabric #2 Overlay VTE P VTE P Bar em eta l Bar em eta l VTE P VTE P Fabric #1EVPN Control- Plane Domain 1 EVPN Control- Plane Domain 2 Single Data-Plane – End-to-End BGP EVPN Overlay VTE P VTE P VTE P VTE P Bar em etal Bar em etal Fabric #2 Overlay VTE P VTE P Bar em etal Bar em etal VTE P VTE P Fabric #1EVPN Control-Plane Domain 1 EVPN Control-Plane Domain 2 Data-Plane Domain 1 Data-Plane Domain 2 DCI Data-Plane • Multiple Fabrics – Normalized through Ethernet • Multiple Fabrics Interconnect using DCI (Layer 2 and Layer 3) Multi-Fabric Data-Plane Domain 1 Data-Plane Domain 2 DCI Data-Plane Overlay VTE P VTE P VTE P VTE P Bar em etal Bar em etal Fabric #2 Overlay VTE P VTE P Bar em etal Bar em etal VTE P VTE P Fabric #1EVPN Control-Plane Domain 1 EVPN Control-Plane Domain 2 BGP EVPN • Multiple Fabrics with Integrated DCI • Integrated DCI – Scaling within and between Fabrics • The Happy Place Multi-Site BRKDCN-2035 12
  • 15. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public VXLAN EVPN – Single Pod / Single Fabric SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod 1 VTEP VTEP External Network BRKDCN-2035 15
  • 16. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public VXLAN EVPN – Multi-Pod SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod n VTEP VTEP Underlay Extension BRKDCN-2035 16
  • 17. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Single Overlay Domain – End-to-End Encapsulation Single Overlay Control-Plane Domain – End-to-End EVPN Updates Single Underlay Domain End-to-End Single Replication Domain for BUM Single VNI Administrative Domain Multi-Pod Characteristics – ”The Single” Building Underlay Hierarchies – Non Hierarchical Overlay BRKDCN-2035 17
  • 18. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Pod – End-to-End Encapsulation SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod n VTEP VTEP Underlay Extension Overlay Baremetal Baremetal Unicast VTEP 10.1.1.1 VTEP 10.2.2.7 BRKDCN-2035 18
  • 19. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Pod – BUM Replication SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Pod 2 VTEP VTEP Underlay Extension Overlay Baremetal BUM BRKDCN-2035 19
  • 20. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Single Overlay Domain – End-to-End Encapsulation • Scaling the VXLAN EVPN Network Single Overlay Control-Plane Domain – End-to-End EVPN Updates • Overlay Control-Plane Update Propagation Single Underlay Domain End-to-End • Network must be extended in Underlay (VTEP to VTEP reachability) Single Replication Domain for BUM • One BUM flooding domain through out all connected Pods Multi-Pod Challenges – ”The Single” BRKDCN-2035 20
  • 22. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public VXLAN EVPN – Multi-Fabric SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric 2 VTEP VTEP Underlay No Extension L2 DCI L2 DCI L3 DCI L3 DCI L2 DCI L2 DCI BRKDCN-2035 22
  • 23. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public • Separate Overlay Domains –Independent L2 and L3 DCI (complexity) • Separate Overlay Control-Plane Domains – Manual Configuration • Separate Underlay Domains - Isolated • Separate Replication Domains for BUM – Independent BUM transport/DCI • Dedicated Border Leaf – no local End-Point Attachment Multi-Fabric Characteristics – ”The Separate” Underlay Isolation – Separate DC Interconnection BRKDCN-2035 23
  • 24. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric n VTEP VTEP Underlay No Extension L2 DCI L2 DCI L3 DCI L3 DCI L2 DCI L2 DCI Multi-Fabric – End-to-End Encapsulation Overlay Site 1 Overlay Site n L2 DCI Baremetal Baremetal Unicast VLAN Hand-Off VRF-Lite Hand-Off BRKDCN-2035 24
  • 27. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public VXLAN EVPN – Multi-Site SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP No Underlay Extension BGW BGW BGW BGW BRKDCN-2035 27
  • 28. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multiple Overlay Domains – Interconnected & Controlled Multiple Overlay Control-Plane Domains – Interconnected & Controlled Multiple Underlay Domains - Isolated Multiple Replication Domains for BUM – Interconnected & Controlled Multiple VNI Administrative Domains – Phase 2 Multi-Site Characteristics – ”The Multiple” Underlay Isolation – Overlay Hierarchies BRKDCN-2035 28
  • 29. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site – Hierarchical Overlay Domains SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Site 1 Overlay Site n Overlay Multi-Site Baremetal Baremetal Unicast BGW BGW BGW BGW BRKDCN-2035 29
  • 30. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site – Underlay Isolation SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP No Underlay Extension VTEP 10.1.1.1 Border (VIP) 10.1.1.111 Border (VIP) 10.2.2.222 Site 1 Underlay Routing Table Leaf: 10.1.1.1 10.1.1.2 10.1.1.3 10.1.1.4 10.1.1.5 10.1.1.6 10.1.1.7 Border: 10.1.1.101 10.1.1.102 10.1.1.111 VTEP 10.2.2.7 Site n Underlay Routing Table Leaf: 10.2.2.1 10.2.2.2 10.2.2.3 10.2.2.4 10.2.2.5 10.2.2.6 10.2.2.7 Border: 10.2.2.101 10.2.2.102 10.2.2.222 BGW BGW BGW BGW Border (PIP) 10.1.1.101 Border (PIP) 10.1.1.102 Border (PIP) 10.2.2.101 Border (PIP) 10.2.2.102 BRKDCN-2035 30
  • 31. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Inter Site Network Multi-Site – Inter Site Network SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP VTEP 10.1.1.1 Border (VIP) 10.2.2.222 Border (VIP) 10.1.1.111 Inter-Site Network Routing Table Border Site1: 10.1.1.101 10.1.1.102 10.1.1.111 Border Site2: 10.2.2.101 10.2.2.102 10.2.2.222 VTEP 10.2.2.7 BGW BGW BGW BGW Border (PIP) 10.1.1.101 Border (PIP) 10.1.1.102 Border (PIP) 10.2.2.101 Border (PIP) 10.2.2.102 BRKDCN-2035 31
  • 33. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Border Gateways Deployment Considerations Site 1 VTEP BGW VTEP BGW VTEP BGW VTEP BGW Site 1 VTEP BGW VTEP BGW Border Gateways used for two main functions: 1. Interconnecting each site to the Inter-Site network (for East-West traffic flows) 2. Connecting each site to the external Layer 3 domain (for North-South traffic flows) May also be used to connect End-Points and/or network service nodes (FWs, ADCs) Two deployment models supported: 1. Anycast Border Gateways 2. VPC Border Gateways Anycast Border Gateways VPC Border Gateways BRKDCN-2035 33
  • 35. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Anycast Border Gateway (1) Anycast Border Gateway Up to 4 Border Gateways Border Gateway • Deploying at Leaf – 7.0(3)I7(1) • Deploying at Spine – 7.0(3)I7(2) VTEP BGW VTEP BGW VTEP BGW VTEP BGW BRKDCN-2035 35
  • 36. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Anycast Border Gateway (2) Anycast Border Gateway Common Virtual IP (VIP) across BGW • VIP is used for Intra- and Inter-Site Communication • VIP for communication between the Border Gateways in different Sites • VIP for communication between Border Gateway and Leaf within a Site Individual Primary IP (PIP) per BGW • Used for Broadcast, Unknown Unicast and Multicast (BUM) replication • PIP for communication with Single-Homed End-Points (routed only), intra- and inter-Site VTEP BGW VTEP BGW VTEP BGW VTEP BGW Border VIP 10.1.1.111 Border VIP 10.1.1.111 PIP-BGW1 10.1.1.101 PIP-BGW2 10.1.1.102 PIP-BGW3 10.1.1.103 PIP-BGW4 10.1.1.104 BRKDCN-2035 36
  • 37. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Anycast Border Gateway (3) Anycast Border Gateway Per-VNI Designated Forwarder (DF) election • Each BGW can serve as DF for a single or a set of Layer-2 VNI • DF election and assignment is automatic Using BGP EVPN Route Type 4 for DF election • Operator Managed Assignment (Type: 00) • Six Octet Site Identifier (System MAC: 00:00:00:00:00:01) • Multi-Site Discriminator (Ethernet-Segment: 00:00:07) • Originators IP Address (PIP): 10.1.1.101 • Layer-2 VNI: 30010 VTEP BGW VTEP BGW VTEP BGW VTEP BGW Spine RR Spine RR BGP EVPN Type: 00 System MAC: 00:00:00:00:00:01 Ethernet Segment: 00:00:074 IP: 10.1.1.101 VNI: 30010 DF 30010 DF 30099 DF 30012 DF 30011 BRKDCN-2035 37
  • 39. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 VPC Border Gateway (1) VPC Border Gateway 2 Border Gateways Border Gateway • Using a Leaf – 7.0(3)I7(2) VTEP BGW VTEP BGW BRKDCN-2035 39
  • 40. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 VPC Border Gateway (2) VPC Border Gateway Common Virtual IP (VIP) across BGW • VIP is used for Intra- and Inter-Site Communication • VIP for communication between the Border Gateways in different Sites • VIP for communication between Border Gateway and Leaf within a Site Individual Primary IP (PIP) per BGW • Used for Broadcast, Unknown Unicast and Multicast (BUM) replication • PIP for communication with Single-Homed End-Points, intra- and inter-Site VTEP BGW VTEP BGW Border VIP 10.1.1.111 Border VIP 10.1.1.111 PIP-BGW1 10.1.1.101 PIP-BGW2 10.1.1.102 BRKDCN-2035 40
  • 41. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 VPC Border Gateway (3) VPC Border Gateway VPC-based Designated Forwarder Election Per-Site Designated Forwarder (DF) election • Using same approach as in VPC • Best Path to Rendezvous-Point or VPC Primary Node VTEP BGW VTEP BGW DF BRKDCN-2035 41
  • 42. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 VPC Border Gateway (4) VPC Border Gateway Single- or Dual-Homed End-Points • Services Appliance (i.e. Firewall, ADC etc.) • Physical or Virtual Servers Advertised and Reachable through Virtual IP Address (VIP) • Intra-Site: Leaf nodes use VIP to reach End- Points connected to Border Gateways • Inter-Site: Remote Border Gateways use VIP to reach End-Points connected to Border • Traffic potentially traverses VPC Peer-Link VTEP BGW VTEP BGW VTEP Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111 2 0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001 10.1.1.111 Border VIP 10.1.1.111 Border VIP 10.1.1.111 ADC 0000.3010.1102 192.168.10.102 ADC ADC 0000.3010.1101 192.168.10.101 ADC BRKDCN-2035 42
  • 44. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Control Plane Deployment Considerations Both MP-eBGP or MP-iBGP peering supported intra-Site between leaf nodes Only MP-eBGP EVPN sessions supported inter-Sites mandates that each site is part of a separate AS Full mesh of MP-eBGP EVPN adjacencies only currently supported across sites • Recommended to deploy a couple of Route-Servers in the Inter-Site network when 3 or more sites are deployed • Route-Servers only perform control plane functions (“eBGP Route-Reflectors”) • Need to ensure that Route-Servers offer support for Route Type 4 EVPN routes, required for DF election BRKDCN-2035 44
  • 45. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (L3Core) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR iBGP-EVPN iBGP-EVPN DC Core (Layer-3 Unicast) BRKDCN-2035 45
  • 46. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (L3Core) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR iBGP-EVPN iBGP-EVPN DC Core (Layer-3 Unicast) RS Route Server (eBGP ”Route Reflector”) RS BRKDCN-2035 46
  • 47. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (L3Core) DC Core (Layer-3 Unicast) RS Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR iBGP-EVPN iBGP-EVPN RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 47
  • 48. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (L3Core, no RS) DC Core (Layer-3 Unicast) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR iBGP-EVPN iBGP-EVPN RS – Route Server (eBGP ”Route Reflector”) eBGP-EVPN BRKDCN-2035 48
  • 49. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane DC Core (Layer-3 Unicast) RS Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR iBGP-EVPN Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 VRF Tenant1 L3VNI: 50001 Route-Target: 65501:50001 VRF Tenant1 L3VNI: 50001 Route-Target: 65502:50001 L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1) L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) iBGP-EVPN VIP1 10.1.1.111 VIP2 10.2.2.222 RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 49
  • 50. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (Site1) DC Core (Layer-3 Unicast) RS Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 VRF Tenant1 L3VNI: 50001 Route-Target: 65502:50001 L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1) L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VRF Tenant1 L3VNI: 50001 Route-Target: 65501:50001 VIP1 10.1.1.111 VIP2 10.2.2.222 Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3010.1101/48 30010, 65501:30010 192.168.10.101/32 50001, 65501:50001 10.1.1.1 2 0000.3020.2101/48 30020, 65501:30020 192.168.20.101/32 50001, 65501:50001 10.1.1.111 2 0000.3010.1102/48 30010, 65501:30010 192.168.10.102/32 50001, 65501:50001 10.1.1.111 BRKDCN-2035 50
  • 51. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (Site2) DC Core (Layer-3 Unicast) RS Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 VRF Tenant1 L3VNI: 50001 Route-Target: 65502:50001 L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1) L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) VRF Tenant1 L3VNI: 50001 Route-Target: 65502:50001 VIP1 10.1.1.111 VIP2 10.2.2.222 Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3010.1101/48 30010, 65502:30010 192.168.10.101/32 50001, 65502:50001 10.2.2.222 2 0000.3020.2101/48 30020, 65502:30020 192.168.20.101/32 50001, 65502:50001 10.2.2.1 2 0000.3010.1102/48 30010, 65502:30010 192.168.10.102/32 50001, 65502:50001 10.2.2.3 BRKDCN-2035 51
  • 52. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Control-Plane (DCI) DC Core (Layer-3 Unicast) RS RS – Route Server (eBGP ”Route Reflector”) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR VRF Tenant1 L3VNI: 50001 Route-Target: 65501:50001 VRF Tenant1 L3VNI: 50001 Route-Target: 65502:50001 VIP1 10.1.1.111 VIP2 10.2.2.222 L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) L2VNI: 30020 (VLAN 20) L3VNI: 50001 (Tenant1) L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111 2 0000.3020.2101/48 30020, 65599:30020 192.168.20.101/32 50001, 65599:50001 10.2.2.222 2 0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001 10.2.2.222 BRKDCN-2035 52
  • 54. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site – Selective Advertisements The Multi-Site architecture provides granular control on how Layer-2 and Layer-3 communication is extended across sites Layer-2 and/or Layer-3 VNIs configured on the Border Gateways (BGW) control the Control-Plane advertisement towards DCI Enhances the overall scalability of the solution • Scale up the total number of End-Points supported across sites BRKDCN-2035 54
  • 55. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Selective Advertisements (DCI) DC Core (Layer-3 Unicast) RS Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP ….BGW BGW BGW BGW RR RR RS – Route Server (eBGP ”Route Reflector”) Only prefixes of VRF ”Tenant1” and L2VNI 30010 are advertised from Site1 towards DCI. In this example this is Host1. All prefixes of VRF ”Tenant2” and L2VNI 30020 are not advertised from Site2 towards DCI. These prefixes are not seen within the DCI VRF Tenant1 L3VNI: 50001 Route-Target: 65501:50001 VIP1 10.1.1.111 VIP2 10.2.2.222Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111 Host1 0000.3010.1101 192.168.10.101 L2VNI: 30010 (VLAN 10) L3VNI: 50001 (Tenant1) L2VNI: 30020 (VLAN 20) L3VNI: 50002 (Tenant2) Host2 0000.3020.2101 192.168.20.101 BRKDCN-2035 55
  • 57. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Overlay Data Plane DC Core (Layer-3 Unicast) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 Intra-site VXLAN Data Plane Inter-site VXLAN Data Plane De-capsulation and Re-encapsulation on BGW De-capsulation and Re-encapsulation on BGW BRKDCN-2035 57
  • 59. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (BUM) – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255 Bridge DF 30010 DF 30010 Host 1 sends a L2 BUM frame 1 2 Leaf10 replicates traffic intra-Site BRKDCN-2035 59
  • 60. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (DF & Split Horizon) – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255 Bridge DF 30010 DF 30010 BUM Forward Drop due to Split-Horizon rule Drop due to Designated Forwarder (DF) rule BRKDCN-2035 60
  • 61. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (BUM) – DCI VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge DF 30010 DF 30010 SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255 BGW-VIP1 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255 BGW-VIP1 BGW12 30010 H1-MAC ALL-F H1-IP ALL-255 3 BGW11 replicates traffic inter- Sites toward BGW nodes BUM Forward BRKDCN-2035 61
  • 62. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (DF & Split Horizon) – DCI VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge DF 30010 DF 30010 SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255 BGW-VIP1 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255 BGW-VIP1 BGW12 30010 H1-MAC ALL-F H1-IP ALL-255 BUM Forward BUM Forward Drop due to Split-Horizon rule Drop due to Designated Forwarder (DF) rule BRKDCN-2035 62
  • 63. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (BUM) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge DF 30010 DF 30010 SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255 4 BGW22 replicates traffic intra-Site BUM Forward BRKDCN-2035 63
  • 64. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (DF & Split Horizon) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge DF 30010 DF 30010 SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255 BUM Forward Drop due to Split-Horizon rule Drop due to Designated Forwarder (DF) rule BRKDCN-2035 64
  • 65. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (BUM) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge DF 30010 DF 30010 Leaf20 sends traffic to local Host 2 5 BRKDCN-2035 65
  • 67. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 1 to Host 2) – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP Bridge Host 1 sends traffic destined to remote Host 2 1 2 Leaf10 performs L2 lookup and encapsulates toward local BGW VIP1 address VIP2VIP1 BRKDCN-2035 67
  • 68. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 1 to Host 2) – DCI VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 BGW-VIP2 30010 H1-MAC H2-MAC H1-IP H2-IP Bridge 3 BGW11 performs L2 lookup and encapsulates toward remote BGW VIP2 address VIP2VIP1 BRKDCN-2035 68
  • 69. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 1 to Host 2) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 L20 30010 H1-MAC H2-MAC H1-IP H2-IP 4 BGW22 performs L2 lookup and encapsulates toward destination L20 node Leaf20 bridges traffic to local Host 2 5 VIP2VIP1 BRKDCN-2035 69
  • 70. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 2 to Host 1) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge SIP DIP VXLAN SMAC DMAC SIP DIP Payload L20 BGW-VIP2 30010 H2-MAC H1-MAC H2-IP H1-IP Host 2 replies to remote Host 1 6 7 Leaf20 performs L2 lookup and encapsulates toward local BGW VIP2 address VIP2VIP1 BRKDCN-2035 70
  • 71. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 2 to Host 1) – DCI VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 BGW-VIP1 30010 H2-MAC H1-MAC H2-IP H1-IP 8 BGW21 performs L2 lookup and encapsulates toward remote BGW VIP1 address VIP2VIP1 BRKDCN-2035 71
  • 72. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-2 (Host 2 to Host 1) – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 Bridge SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 L10 30010 H2-MAC H1-MAC H2-IP H1-IP 9 BGW12 performs L2 lookup and encapsulates toward destination L10 node Leaf10 bridges traffic toward Host 1 10 VIP2VIP1 BRKDCN-2035 72
  • 74. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-3 (Host 1 to Host 3) – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 3 0000.3010.1102 192.168.20.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 50001 L10-MAC BGW-VMAC1 H1-IP H3-IP Route Host 1 sends a data packet to the remote Host 3 1 2 Leaf10 performs a L3 lookup and encapsulates toward local BGW VIP1 address VIP2 VMAC2 VIP1 VMAC1 BRKDCN-2035 74
  • 75. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-3 (Host 1 to Host 3) – DCI VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 3 0000.3010.1102 192.168.20.102 Route SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 BGW-VIP2 50001 BGW-VMAC1 BGW-VMAC2 H1-IP H3-IP 3 BGW11 performs a L3 lookup and encapsulates toward remote BGW VIP2 address VIP2 VMAC2 VIP1 VMAC1 BRKDCN-2035 75
  • 76. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Packet Walk – Layer-3 (Host 1 to Host 3) – Site2 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 3 0000.3010.1102 192.168.20.102 Route SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 L20 50001 BGW-VMAC1 L20-MAC H1-IP H3-IP 4 BGW21 performs a L3 lookup and encapsulates toward destination L20 node Leaf20 routes traffic to local Host 3 5 VIP2 VMAC2 VIP1 VMAC1 BRKDCN-2035 76
  • 78. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Steady State Traffic – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP VIP2VIP1 BRKDCN-2035 78
  • 79. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DCI Link Failure BGW12 – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP On DCI Link Failure (i.e. BGW12) • Virtual IP (VIP) on BGW is disabled • BGW will stop participating in DF election • BGW acts like a Leaf (Layer-3 only) • Traffic towards others Sites is served by remaining BGWs (i.e. BGW11) BGW12 gets isolated from the DCI Core Network Intra-site VXLAN traffic re- routing VIP2VIP1 BRKDCN-2035 79
  • 80. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Link Failure BGW12 – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP VIP2VIP1 BRKDCN-2035 80
  • 81. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Link Failure BGW12 – Site1 VXLAN EVPN Site2 VTEP Leaf20 VTEP BGW21 VTEP BGW22 VXLAN EVPN Site1 VTEP Leaf10 VTEP BGW11 VTEP BGW12 VXLAN EVPN DCI Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP VIP2VIP1 BGW12 gets isolated from the Spine nodes Intra-site VXLAN traffic re- routing On Fabric Link Failure (i.e. BGW12) • Virtual IP (VIP) on BGW is disabled • Primary IP (PIP) on BGW is disabled • BGW will stop participating in the Overlay BRKDCN-2035 81
  • 83. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – Enable Border Gateway Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 BGW2 Multi-Site Commands are marked in red Various options do exist but the recommended design choices are: • Fabric Internal IGP Underlay, iBGP Overlay • DCI (primary choice) eBGP Underlay, eBGP Overlay Route Server for DCI Overlay peerings DC Core for reachability across n Sites • DCI (alternative option) Any Routing Protocol Underlay, eBGP Overlay Full-Mesh for DCI Overlay peerings Back-to-Back Site Reachability (physical, full- mesh) BRKDCN-2035 83
  • 84. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – Enable Border Gateway Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 BGW2 feature nv overlay nv overlay evpn feature bgp feature interface-vlan feature vn-segment-vlan-based evpn multisite border-gateway BGW2BGW1 BRKDCN-2035 84
  • 85. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – BGW 1 Loopback & VTEP Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 interface loopback1 description PIP VTEP ip address 10.1.1.101/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface loopback100 description VIP Multi-Site 1 ip address 10.1.1.111/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface loopback0 description RID ip address 10.10.10.101/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode BGW1 BRKDCN-2035 85
  • 86. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – BGW 2 Loopback & VTEP Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP …. BGW2 interface loopback1 description PIP VTEP ip address 10.1.1.102/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface loopback100 description VIP Multi-Site 1 ip address 10.1.1.111/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode interface loopback0 description RID ip address 10.10.10.102/32 tag 12345 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode BGW2 BRKDCN-2035 86
  • 87. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – Fabric Link Tracking BGW 1 Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 interface Ethernet1/53 description TO-SPINE1 ip address 10.0.1.1/30 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode evpn multisite fabric-tracking interface Ethernet1/54 description TO-SPINE2 ip address 10.0.2.1/30 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode evpn multisite fabric-tracking BGW1 Allows to bring down the PIP/VIP loopback interfaces when the BGW is isolated from the spines BRKDCN-2035 87
  • 88. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric Site 1 Setup – Fabric Link Tracking BGW 2 Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP …. BGW2 interface Ethernet1/53 description TO-SPINE1 ip address 10.0.1.5/30 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode evpn multisite fabric-tracking interface Ethernet1/54 description TO-SPINE2 ip address 10.0.2.5/30 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode evpn multisite fabric-tracking BGW2 BRKDCN-2035 88
  • 89. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 Setup – Multi-Site Underlay Interface Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 BGW2 interface Ethernet1/1 description TO-DC-CORE1 ip address 10.111.111.1/30 tag 12345 evpn multisite dci-tracking interface Ethernet1/2 description TO-DC-CORE2 ip address 10.111.222.1/30 tag 12345 evpn multisite dci-tracking BGW1 DCI interface Ethernet1/1 description TO-DC-CORE1 ip address 10.222.111.1/30 tag 12345 evpn multisite dci-tracking interface Ethernet1/2 description TO-DC-CORE2 ip address 10.222.222.1/30 tag 12345 evpn multisite dci-tracking BGW2 Allows to bring down the PIP/VIP loopback interfaces when the BGW is isolated from the DC core BRKDCN-2035 89
  • 90. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 BGW 1 Setup – Multi-Site Overlay Peering Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 DCI RS router bgp 65501 router-id 10.10.10.101 address-family ipv4 unicast redistribute direct route-map REDIST-LOCAL neighbor 10.111.111.2 remote-as 65599 update-source ethernet1/1 address-family ipv4 unicast neighbor 10.111.222.2 remote-as 65599 update-source ethernet1/2 address-family ipv4 unicast neighbor 10.99.99.201 remote-as 65599 update-source loopback0 ebgp-multihop 5 peer-type fabric-external address-family l2vpn evpn rewrite-evpn-rt-asn send-community send-community both BGW1 RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 90
  • 91. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 BGW 2 Setup – Multi-Site Overlay Peering Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP …. DCI RS router bgp 65501 router-id 10.10.10.102 address-family ipv4 unicast redistribute direct route-map REDIST-LOCAL neighbor 10.222.111.2 remote-as 65599 update-source ethernet1/1 address-family ipv4 unicast neighbor 10.222.222.2 remote-as 65599 update-source ethernet1/2 address-family ipv4 unicast neighbor 10.99.99.201 remote-as 65599 update-source loopback0 ebgp-multihop 5 peer-type fabric-external address-family l2vpn evpn rewrite-evpn-rt-asn send-community send-community both BGW1 RS – Route Server (eBGP ”Route Reflector”) BGW2 BRKDCN-2035 91
  • 92. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 Setup – Multi-Site Overlay Peering Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 DCI RS RS – Route Server (eBGP ”Route Reflector”) BGW2 peer-type fabric-external • Enables Next-Hop Rewrite for Multi-Site • Defines Site External BGP neighbors for EVPN exchange rewrite-evpn-rt-asn • Rewrites Route-Target Auto information to simplify MAC-VRF and IP-VRF configuration • Normalizes outgoing Route-Targets AS number to match remote AS number • Uses BGP configured Neighbors Remote AS BRKDCN-2035 92
  • 93. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Site 1 Setup – Multi-Site Overlay Peering DC Core (Layer-3 Unicast) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 peer-type fabric-external peer-type fabric-external BGP Update: MAC: 0000.3010.1101 (L2VNI 30001) IP: 192.168.20.101 (L3VNI 50001) NH: 10.2.2.222 RMAC: BGW-VMAC2 BGP Update: MAC: 0000.3010.1101 (L2VNI 30001) IP: 192.168.20.101 (L3VNI 50001) NH: 10.1.1.111 RMAC: BGW-VMAC1 Rewrite Next-Hop IP and Next- Hop MAC (RMAC) based on Neighbor Site BGW BGP Update: MAC: 0000.3010.1101 (L2VNI 30001) IP: 192.168.20.101 (L3VNI 50001) NH: 10.1.1.1 RMAC: Leaf1 Rewrite Next-Hop IP and Next- Hop MAC (RMAC) based on Neighbor Site BGW BRKDCN-2035 93
  • 94. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Site 1 Setup – Multi-Site Overlay Peering DC Core (Layer-3 Unicast) Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host2 0000.3020.2101 192.168.20.101 rewrite-evpn-rt-asn rewrite-evpn-rt-asn BGP Update: Remote AS: 65502 VNI: 50001 Route-Target: 65502:50001 Rewrite Route-Target based on BGP Neighbors Remote ASN BGP Update: Remote AS : 65501 VNI: 50001 Route-Target: 65501:50001 BGP Update: Remote AS: 65502 VNI: 50001 Route-Target: 65502:50001 BRKDCN-2035 94
  • 95. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 Setup – Anycast BGW VTEP Configuration Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 DCI RS – Route Server (eBGP ”Route Reflector”) interface nve1 no shutdown host-reachability protocol bgp multisite ethernet-segment 7 system-mac 0000.0000.0001 source-interface loopback1 multisite border-gateway interface loopback100 member vni 30010 multisite ingress-replication mcast-group 239.1.1.1 member vni 30011-30020 mcast-group 239.1.1.2 member vni 50001 associate-vrf BGW1 BGW2 BGW2 BRKDCN-2035 95
  • 96. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 Setup – Anycast BGW VTEP Configuration Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 DCI BGW2 multisite ethernet-segment • Defines the discriminator for Sites in a common Domain system-mac • Defines the Multi-Site Site-Id (6 octets hex) multisite border-gateway interface loopback# • Defines the Loopback Interface used for the Border Gateway Virtual IP Address (VIP) multisite ingress-replication • Per-VNI knob for extending Layer-2 VNI • Defines the Multi-Site BUM Replication method BRKDCN-2035 96
  • 97. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Fabric Site 1 Setup – Multi-Site Overlay Traffic Policy Spine Spine VTEP VTEP VTEP VTEP VTEP VTEP ….BGW1 DCI BGW2 • BUM Traffic Policing • Limits Broadcast, Unknown Unicast and Layer-2 Multicast Traffic across Multi-Site • Level 0 = No B/U/M Forwarding • Level 100 = All B/U/M Forwarding Forwarding • Enforced on Encapsulation towards remote Sites evpn storm-control broadcast level 10 evpn storm-control unicast level 10 evpn storm-control multicast level 10 BGW1 BGW2 BRKDCN-2035 97
  • 98. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Setup – Multi-Site Overlay Traffic Policy SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Multi-Site Overlay Site 1 Overlay Site n Baremetal BUM BGW BGW BGW BGW BRKDCN-2035 98
  • 99. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Setup – Multi-Site Overlay Traffic Policy SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Multi-Site Overlay Site 1 Overlay Site n Baremetal BUM Storm Control Broadcast 0-100% Unknown Unicast 0-100% Multicast 0-100% Storm Control Broadcast 0-100% Unknown Unicast 0-100% Multicast 0-100% BGW BGW BGW BGW BRKDCN-2035 99
  • 100. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Setup – Multi-Site BUM Replication Modes SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Multi-Site Overlay Site 1 Overlay Site n Multicast Multicast Ingress Replication BGW BGW BGW BGW BRKDCN-2035 100
  • 101. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Setup – Multi-Site BUM Replication Modes SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Multi-Site Overlay Site 1 Overlay Site n Ingress Replication Ingress Replication Ingress Replication BGW BGW BGW BGW BRKDCN-2035 101
  • 102. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Site 1 Setup – Multi-Site BUM Replication Modes SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site n VTEP VTEP Overlay Multi-Site Overlay Site 1 Overlay Site n Ingress Replication Multicast Ingress Replication BGW BGW BGW BGW BRKDCN-2035 102
  • 103. Connectivity to the External Layer 3 Domain
  • 104. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Connectivity to the External Layer 3 Domain The BGW nodes can be used to provide Layer-3 external connectivity to each site Different connectivity models are supported • VRF-Lite peering with an external pair of WAN Edge routers • MP-BGP EVPN peering with the external WAN Edge routers (GOLF) • Dedicated or shared pair of WAN Edge routers across sites External Layer-3 network may be different from the DCI network used for inter-site communication BRKDCN-2035 104
  • 105. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Border Gateway and VRF-Lite Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 VRF-CVRF-BVRF-ASeparate routing peering for each VRF (IGP or eBGP) Dedicated interface (logical or physical) for each VRF BRKDCN-2035 105
  • 106. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Fabric DCI Multi-Site – Border Gateway and GOLF Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 Host2 0000.3020.2101 192.168.20.101 VRF-CVRF-BVRF-A Single MP-BGP EVPN instance to exchange routes for all VRFs VXLAN Data Plane between BGW and WAN Edge Router BRKDCN-2035 106
  • 107. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Fabric DCI Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host2 0000.3020.2101 192.168.20.101 Host3 0000.3010.1102 192.168.10.102 Multi-Site – Shared Internet/WAN Gateways Internet/WAN BorderPE BorderPE Inter-Site VXLAN Communication between Border Gateways BRKDCN-2035 107
  • 108. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site – Per Site Internet/WAN Gateway DC Core (Layer-3 Unicast) MPLS L3VPN Fabric DCI Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host2 0000.3020.2101 192.168.20.101 Host3 0000.3010.1102 192.168.10.102 Internet/WAN BorderPE BorderPE BorderPE BorderPE Inter-Site VXLAN Communication between Border Gateways BRKDCN-2035 108
  • 109. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public MPLS L3VPN Fabric DCI Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP …. Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP …. VIP1 10.1.1.111 VIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host2 0000.3020.2101 192.168.20.101 Host3 0000.3010.1102 192.168.10.102 Internet/WAN BorderPE BorderPE BorderPE BorderPE Multi-Site – Consolidated WAN and DCI NetworkPerform simple routing for inter-site flows, VXLAN (or VRF-Lite) to MPLS VPN hand-off for north-south communication Inter-Site VXLAN Communication between Border Gateways BRKDCN-2035 109
  • 110. Ingress and Egress Traffic Optimization
  • 111. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP BGW WAN Active FWActive FW The stretching of Layer-2 domains across separate sites may lead to the creation of asymmetric traffic paths Deploying independent stateful services (like FWs) across sites would result in traffic drops In this case it is required to ensure the symmetry of ingress and egress communication paths Ingress and Egress Traffic Optimization The Issue of Extending Layer 2 Domains BGW BGW BGW DC Core (Layer-3 Unicast) BRKDCN-2035 111
  • 112. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) Guarantee routing symmetry with the outside of the Data Center • Egress Always prefer the local BGW • Ingress Steer traffic to the specific destination End-Point’s location Maintain optimal routing over the dedicated DCI network (if existing) for Server-to-Server traffic • The DC fabric must discriminate between DC and WAN destinations If required provide a fallback path via DCI for WAN isolation situations Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP BGW BGW BGW BGW WAN Ingress and Egress Traffic Optimization Maintaining Traffic Symmetry over Optimal Paths BRKDCN-2035 112
  • 113. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 Multi-Site – Egress Path Optimization 172.16.1.10 172.16.1.0/24 Border-PEs 1-2 172.16.1.0/24 Border-PEs 3-4 172.16.1.0/24 VIP1 172.16.1.0/24 VIP2 eBGP-EVPN Less preferred advertisement of 172.16.1.0 because of longer AS-Path BRKDCN-2035 113
  • 114. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 Multi-Site – Egress Path Optimization 172.16.1.10 eBGP-EVPN Optimized Egress Traffic Path Optimized Egress Traffic Path BRKDCN-2035 114
  • 115. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 Multi-Site – Egress Path Optimization WAN Isolation Scenario 172.16.1.10 172.16.1.0/24 Border-PEs 3-4 172.16.1.0/24 VIP1 172.16.1.0/24 VIP2 eBGP-EVPN WAN Isolation Scenario 172.16.1.0/24 VIP2 BRKDCN-2035 115
  • 116. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 Multi-Site – Ingress Path Optimization 192.168.10.0/24 BGW 1-2 192.168.10.101/32 BGW-1-2 192.168.10.0/24 BGW 3-4 192.168.10.102/32 BGW 3-4 192.168.10.101/32 Leaf1 192.168.10.102/32 -> Leaf3 eBGP-EVPN Host routes advertised across sites but NOT re- advertised toward the local Border-PEs 192.168.10.0/24 Border-PE 1-4 192.168.10.101/32 Border-PE 1-2 192.168.10.102/32 Border-PE 3-4Host routes advertisement in the WAN Deploying LISP on the Border-PEs is a viable alternative to host routes advertisement Filter out host routes received from remote sites. Only announce local host route information BRKDCN-2035 116
  • 117. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 Multi-Site – Ingress Path Optimization eBGP-EVPN 192.168.10.0/24 Border-PE 1-4 192.168.10.101/32 Border-PE 1-2 192.168.10.102/32 Border-PE 3-4 Optimized Ingress Traffic Path Optimized Ingress Traffic Path BRKDCN-2035 117
  • 118. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public DC Core (Layer-3 Unicast) MPLS L3VPN Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEPVIP2 10.2.2.222 BGW BGW BGW BGW Host1 0000.3010.1101 192.168.10.101 Host3 0000.3010.1102 192.168.10.102 BorderPE BorderPE BorderPE BorderPE VIP1 10.1.1.111 eBGP-EVPN 192.168.10.0/24 Border-PE 3-4 192.168.10.101/32 Border-PE 1-2 192.168.10.102/32 Border-PE 3-4 Multi-Site – Ingress Path Optimization WAN Isolation Scenario WAN Isolation Scenario BRKDCN-2035 118
  • 120. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Network Services Integration Couple of different options where to connect network services: 1. Service Leaf nodes: recommended to connect devices used for east- west communication 2. Border Gateway Nodes: used to connect network services for north- south traffic flows Depending on the specifics of the Multi-Site deployment, the following deployment models would be possible: • Active/Standby Service Nodes pair connected to different sites • Active/Active cluster of Service Nodes deployed across sites • Independent Active/Standby Service nodes pairs deployed in separate sites BRKDCN-2035 120
  • 121. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Network Services Integration Active/Standby Pair Deployed across Sites Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP BGW WAN Standby FWActive FW BGW BGW BGW DC Core (Layer-3 Unicast) Requirement to extend Layer 2 communication between Active/Standby nodes for keep-alives and state information exchange Perimeter service nodes connected to VPC Border Gateways Ingress and egress traffic always traversing the Active node in Site 1 No issues related to the creation of asymmetric traffic paths East-West flows must be hair-pinned to the active FW connected to the Service leaf nodes in Site 1 • Need to properly dimension bandwidth in the DC Core to accommodate for this extra traffic Active FW Standby FW Baremetal BaremetalBaremetal North-South traffic flows North-South traffic flows East-West traffic flows BRKDCN-2035 121
  • 122. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Network Services Integration Active/Active Cluster of Service Nodes Deployed across Sites Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP BGW WAN Active/Active FW Cluster BGW BGW BGW DC Core (Layer-3 Unicast) Requirement to extend Layer 2 communication between Active/Active nodes for intra-cluster communication and traffic redirection Perimeter service nodes connected to VPC Border Gateways Asymmetric traffic issues taken care by native intra-cluster traffic redirection Option to deploy ingress/egress optimization technique to avoid inter-site traffic hair-pinning Service node cluster integration not supported at FCS and planned for a future SW release Active/Active FW Cluster Logical Intra- Cluster Link (ICL) Baremetal North-South traffic flows BRKDCN-2035 122
  • 123. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Network Services Integration Independent Active/Standby Pair Deployed in Separate Sites Spine Spine VXLAN EVPN Site1 VTEP VTEP VTEP VTEP VTEP VTEP Spine Spine VXLAN EVPN Site2 VTEP VTEP VTEP VTEP VTEP VTEP BGW WAN Active/Standby FW Active/Standby FW BGW BGW BGW DC Core (Layer-3 Unicast) Mandates the deployment of Ingress/Egress traffic optimization to avoid creation of asymmetric traffic path for north-south communication Active/Standby nodes can use direct links to sync state Perimeter service nodes can connected to Anycast Border Gateways Active/Standby pair (or cluster) still required for service nodes used for east-west traffic flows Baremetal Baremetal Active FW Standby FW North-South traffic flows North-South traffic flows BRKDCN-2035 123
  • 125. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site and Legacy Site Integration Extend Layer-2 and Layer-3 connectivity between sites • Coexistence and/or application migration use cases Proposed approach is to deploy a pair of ‘remote’ VPC Border Gateways in the legacy site • Offers native Multi-Site functionalities (BUM containment, etc) to the legacy site SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Site VTEP VTEP Legacy Site VTEP VTEP BGW BGW BGW BGW Pair of VPC Border Gateways BRKDCN-2035 125
  • 126. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site and Legacy Site Integration Layer-2 Connectivity with the ‘Remote’ BGW Legacy Site VTEP VTEP BGW BGW Legacy Aggregation Layer devices support MLAG Single logical link to extend VLANs toward the Greenfield VXLAN EVPN site VLANs mapped to L2VNIs on the Border Gateways Legacy Site VTEP VTEP BGW BGW Legacy Aggregation Layer devices do not support MLAG Single port-channel from each aggregation layer device VLANs mapped to L2VNIs on the Border Gateways Recommended to move the STP root to the BGW devices BRKDCN-2035 126
  • 127. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site and Legacy Site Integration Layer-2 Control Plane Exchange across Sites Baremetal Host 1 0000.3010.1101 192.168.10.101 Baremetal Host 2 0000.3010.1102 192.168.10.102 SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Site VTEP VTEP Legacy Site VTEP VTEP BGW BGW BGW BGW VIP1 10.1.1.111 VIP2 10.2.2.222 MAC NH 0000.3010.1101 Leaf1 0000.3010.1102 VIP2 MAC NH 0000.3010.1101 VIP1 0000.3010.1102 Po1 Po1 All End-Points in the legacy site are learned as directly connected to the BGW eBGP-EVPN BRKDCN-2035 127
  • 128. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Integration between Anycast Gateway and legacy default gateway (HSRP, VRRP, etc.) not initially supported with VXLAN Multi-Site First option is to keep on the legacy network the active default gateway for the stretched IP subnets SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Site VTEP VTEP Legacy Site VTEP VTEP BGW BGW BGW BGW Default Gateway deployed on the legacy aggregation devices Multi-Site and Legacy Site Integration Default Gateway Deployment – Option 1 L3 L2 Greenfield VXLAN EVPN Fabric only offers L2 services for the stretched IP subnets BRKDCN-2035 128
  • 129. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Recommended approach is to migrate the default gateway from the legacy aggregation devices to the Border Gateways (VXLAN EVPN Anycast Gateway) Optimize routing between End-Points deployed across sites SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Site VTEP VTEP Legacy Site VTEP VTEP BGW BGW BGW BGW Default Gateway migrated to the Border Gateways (VXLAN EVPN Anycast Gateway) Multi-Site and Legacy Site Integration Default Gateway Deployment – Option 2 L3 L2 Greenfield VXLAN EVPN Fabric offers L2 and L3 services for the stretched IP subnets Legacy infrastructure offers only L2 services Distributed Anycast Gateway function L3 L2 BRKDCN-2035 129
  • 130. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multi-Site and Legacy Site Integration Layer-3 Control Plane Exchange across Sites Baremetal Host 1 0000.3010.1101 192.168.10.101 SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Site VTEP VTEP Legacy Site VTEP VTEP BGW BGW BGW BGW VIP1 10.1.1.111 VIP2 10.2.2.222 Po1 All End-Points in the legacy site are learned as directly connected to the BGW eBGP-EVPN L3 L2 Baremetal Host 3 0000.3010.1102 192.168.20.101 IP NH 192.168.10.101 Leaf1 192.168.20.101 VIP1 IP L3VNI 192.168.10.101 VIP1 192.168.20.101 Po1 BRKDCN-2035 130
  • 132. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Migration to Multi-Site Use Cases 1. Site addition: need to connect a Greenfield VXLAN EVPN Fabric to an existing VXLAN EVPN Fabric built with 1st generation Nexus 9000 2. Migrating a VXLAN Multi-Pod Fabric to Multi-Site 3. Migrating a VXLAN Multi-Fabric design to Multi-Site BRKDCN-2035 132
  • 133. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Migration to Multi-Site Site Addition SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Existing VXLAN EVPN Fabric VTEP VTEP BGW BGW SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Greenfield Fabric VTEP VTEP BGW BGW SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Existing VXLAN EVPN Fabric Step 1: add a pair of Border Gateways to the existing VXLAN EVPN Fabric, running the proper SW release supporting Multi-Site Note: no requirement to change the HW/SW version on existing leaf nodes Step 2: connect the BGW to the inter-site network and establish control plane peering with the BGW in the Greenfield Fabric Step 3: configure on the BGW the L2VNIs and L3VNIs to be extended eBGP-EVPN BRKDCN-2035 133
  • 134. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Migration to Multi-Site Multi-Fabric to Multi-Site SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Site 2 VTEP VTEP No Underlay Extension BGW BGW BGW BGW Step 1: add a pair of Border Gateways to each Pod (if needed) and connect them to the spines and to the inter-site network Step 2: upgrade the SW on both Fabrics BGW to be able to support Multi-Site Step 3: establish control plane adjacencies across sites Step 4: disconnect the previously used DCI and extend Layer-2 and Layer-3 across Multi-Site SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric 1 VTEP VTEP SpineSpine Spine Spine VTEP VTEPVTEP VTEPVTEP VTEP VTEP Fabric 2 VTEP VTEP BRKDCN-2035 134
  • 136. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Multiple Overlay Domains – Interconnected & Controlled • Scaling and Segregating VXLAN EVPN Networks Multiple Overlay Control-Plane Domains – Interconnected & Controlled • Limited Overlay Control-Plane Update Propagation Multiple Underlay Domains - Isolated • Isolated Underlay Domains – No need for Extension Multiple Replication Domains for BUM – Interconnected & Controlled • Individual BUM flooding domain with Traffic control Multi-Site Advantages – ”The Multiple” BRKDCN-2035 136
  • 137. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public • New IETF Draft for Multi-Site Design • Multi-site EVPN based VXLAN using Border Gateways • https://tools.ietf.org/html/draft-sharma-multi-site-evpn VXLAN EVPN – Multi-Site BRKDCN-2035 137
  • 138. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public • Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card. • Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us. Complete Your Online Session Evaluation Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.
  • 139. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Continue Your Education • Demos in the Cisco campus • Walk-in Self-Paced Labs • Lunch & Learn • Meet the Engineer 1:1 meetings • Related sessions BRKDCN-2035 139