100% found this document useful (2 votes)

780 views

Ccie DC Full Scale Labs PDF

CCIE Data Center Full-Scale Labs are self-paced, self-paced study. All devices used in this scenario will be pre-configured for you. Do not modify or remove this initial configuration.

Uploaded by

naveedrana

Available Formats

Download as PDF, TXT or read online on Scribd

100% found this document useful (2 votes)

780 views

Ccie DC Full Scale Labs PDF

CCIE Data Center Full-Scale Labs are self-paced, self-paced study. All devices used in this scenario will be pre-configured for you. Do not modify or remove this initial configuration.

Uploaded by

naveedrana

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 185

CCIE Data Center Full-Scale Labs - Bootcamp

Members - CCIE Data Center Full-Scale Lab 1

CCIE DC Full-Scale Lab 1 Tasks
This workbook is not yet compatible with current DC racks for
self-paced study.
Introduction
1. Data Center Infrastructure
2. Data Center Storage Networking
3. Unified Computing
4. Data Center Virtualization

Introduction

All devices used in this scenario, with the exception of the UCS and Nexus 7K, will
be pre-configured for you with a basic initial configuration before starting. Do not
modify or remove this initial configuration, such as pre-configured MGMT0 IP
addresses, pre-configured VRFs, pre-configured routing, etc. These initial configs are
required to successfully complete this scenario.
NX-OS device logins are admin with the password Cciedc01. The UCS
Management VM's login is Administrator with the password cisco. Do not modify
the admin role on any platform, change the console speed, configure AAA, or make
any other configuration changes that would potentially lock you out of the CLI
interface. Rack rental tokens will not be refunded in cases where configuration errors
on your part cause you or the automation system to be locked out of the devices.
Pre-configured MGMT0 addresses for this scenario are as follows:
N5K1 192.168.101.51/24
N5K2 192.168.101.52/24
MDS1 192.168.101.61/24
MDS2 192.168.101.62/24
N7K1 192.168.101.71/24
Any references to "Y" in this scenario refer to the last octet of the MGMT0 interface.

1. Data Center Infrastructure

1.1 UCS Initialization
Connect to UCS Fabric Interconnect A's CLI and use the following options for the
initial configuration dialog:
Enforce strong passwords: yes
Admin password: Cciedc01
Cluster: yes
Switch fabric: A
System name: UCS-FI
MGMT0 IP address: 192.168.101.201
Netmask: 255.255.255.0
Default gateway: 192.168.101.1
Cluster IP address: 192.168.101.200
Configure UCS FI B to join the cluster and use the IP address 192.168.101.202/24.
Enable both Telnet and SSH access to the Fabric Interconnects.

Score: 3 Points

1.2 Nexus 7K VDC Initialization

Create three VDCs on N7K1 as follows:
VDC 2 named N7K2
VDC 3 named N7K3
VDC 4 named N7K4
Do not inherit the VDC hostname from the default VDC's hostname.
Allocate the interfaces to these VDCs according to the diagram. Any unneeded
interfaces should be assigned to VDC 0.
Connect to these VDCs from the console and configure the admin user with the
password Cciedc01.
Configure the MGMT0 IP addresses of the VDCs as follows:
VDC 2: 192.168.101.72/24
VDC 3: 192.168.101.73/24
VDC 4: 192.168.101.74/24
Enable both telnet and SSH access to all VDCs.
Score: 5 Points

1.3 Initial IP Addressing

Configure the higher-numbered M1 port in the diagram between N7K1 and N7K3 as
a native layer 3 routed interface using the addresses 10.71.73.Y/24.
Configure the M1 ports between N7K2 and N7K4 as layer 3 Port-Channel10. Use
LACP for the Port-Channel, and the addresses 10.72.74.Y/24.
Configure N7K3 and N7K4's links to the Data Center Interconnect as layer 2 access
edge ports in VLANs 1050 and 1051, respectively. Configure interfaces VLAN 1050
and 1051 on N7K3 and N7K4, respectively, with addresses 10.50.73.0/31 and
10.51.74.0/31.
Score: 3 Points

1.4 Layer 3 Routing

Configure N7K1 and N7K2 to default to N7K3 and N7K4, respectively.

Configure N7K3 and N7K4 to peer BGP with the DCI provider. The provider uses
BGP AS 100, whereas N7K3 and N7K4 have been allocated BGP ASes 65001 and
65002, respectively. The DCI provider also requires MD5 authentication using the
password DCIPROVIDER.
Do not modify any DCI-related configuration on N5K1 or 3750G.
When complete, N7K1 and N7K2 should have IP reachability to each other over the
DCI.
Score: 5 Points

1.5 FabricPath
N5K1 and N7K4 should form Port-Channel20 using LACP on the links connecting
them according to the diagram.
Configure FabricPath on the port channel as well as the link connecting N7K4 and
N5K2 according to the diagram.
Create VLANs 200299 as FabricPath VLANs on these switches.
Authenticate all FabricPath IS-IS adjacencies using an MD5 hash of the password
FPAUTH.
Score: 6 Points

1.6 vPC+
Configure UCS-FI-A to form Port-Channel201 up to N5K1 and N5K2 using the links
in the diagram.
Configure UCS-FI-B to form Port-Channel202 up to N5K1 and N5K2 using the links
in the diagram.
From N5K1 and N5K2's perspective, these links should be vPC 201 and 202.
vPC 201 and 202 should be 802.1Q trunk links, STP edge ports, and only allow
VLANs 200299.
Use the vPC Domain ID 500 and the FabricPath Switch-ID 501.
Score: 6 Points

1.7 FabricPath Traffic Engineering

Ensure that N7K4 can use both N5K1 and N5K2 to reach their southbound Classical

Ethernet peers in VLANs 200299.

Score: 5 Points

1.8 Spanning-Tree Protocol Optimization

Modify N5K1 and N5K2's Classical Ethernet configuration so that they run the
minimum number of spanning-tree instances necessary to deliver traffic from the
northbound FabricPath domain into the southbound UCS domain.
Any new switches that are attached to the Classical Ethernet domain of N5K1 or
N5K2 that have a non-zero STP priority should not be able to be elected the STP
root bridge.
Score: 6 Points

1.9 Fabric Extenders

N7K3 has two links to each N2K1 and N2K2, which are then used to dual-home to
the UCS C200 server. Configure N7K3 to pair with N2K1 and N2K2 as FEX 131 and
132, respectively. Use Port-Channel 131 and 132, respectively.
Score: 5 Points

1.10 OTV
Configure OTV on N7K1 and N7K2 to bridge VLANs 200299 over the Data Center
Interconnect.
N7K1 should use the Site VLAN and Identifier 3001, and N7K2 should use the Site
VLAN and Identifier 3002.
Trunk the minimum number of necessary VLANs between N7K1 and N7K3, and
N7K2 and N7K4.
N7K3 and N7K4 should use PIM Sparse Mode for multicast routing with the DCI, and
use the RP address 10.0.0.51, which is hosted by the provider.
Multicast Control Plane traffic for the OTV should be tunneled over the DCI using the
group 224.71.72.0.
Multicast Data Plane traffic originating from N7K1 should use the group range
232.71.71.0/24.
Multicast Data Plane traffic originating from N7K2 should use the group range

232.72.72.0/24.
Authenticate the IS-IS adjacency between N7K1 and N7K2 using an MD5 hash of
the password OTVAUTH.
Create Interface VLAN 200 on N7K3 and N7K4 with the IP addresses
192.168.200.Y/24.
When complete, N7K3 and N7K4 should be able to ping each other over the DCI
through the OTV tunnel, as well as the VMKernel interfaces of the ESXi instances on
UCS Blades 1 and 2, and the C200 server. The ESXi addresses are
192.168.200.101, 192.168.200.102, and 192.168.200.104, respectively.
Score: 7 Points

2. Data Center Storage Networking

2.1 Fibre Channel Initialization
Configure N5K1, N5K2, UCS-FI-A, and UCS-FI-B's Unified Ports in Fibre Channel
mode as shown in the diagram.
N5K1's links to MDS1 and N5K2's links to MDS2 should be configured as PortChannel101 and 102, respectively. The port channels should use dynamic
negotiation and be configured as Trunking Expansion ports.
N5K1's links to UCS-FI-A and N5K2's links to UCS-FI-B should be configured as PortChannel 103 and 104, respectively. The port channels should use dynamic
negotiation and be configured as non-trunking Fabric ports on the N5K1 and N5K2
sides.
Score: 5 Points

2.2 VSANs and Trunking

The SAN A side of the UCS blade servers will use VSAN 103, and the SAN B side
will use VSAN 104. Internal to UCS, these should map to VLANs 1103 and 1104,
respectively.
UCS-FI-A's Port-Channel103 to N5K1 and UCS-FI-B's Port-Channe104 to N5K2
should be non-trunking NP ports in VSANs 103 and 104, respectively.
N5K1's Port-Channel101 to MDS1 and N5K2's Port-Channel102 to MDS2 should be
TE ports that only forward VSANs 103 and 104, respectively.

MDS1 and MDS2's link to the SAN should be F ports in VSANs 103 and 104,
respectively.
Score: 6 Points

2.3 Fibre Channel Zoning

Configure Enhanced Zoning and Enhanced Device Aliases on both the SAN A and
SAN B sides of the UCS blade server.
Device Aliases in SAN A should be configured as follows:
Alias "FC-SAN-A" pWWN 21:00:00:1b:32:04:5e:dc
Alias "BLADE1-SAN-A" pWWN 20:00:00:cc:1e:dc:01:0a
Alias "BLADE2-SAN-A" pWWN 20:00:00:cc:1e:dc:02:0a
Device Aliases in SAN B should be configured as follows:
Alias "FC-SAN-B" pWWN 21:01:00:1b:32:24:5e:dc
Alias "BLADE1-SAN-B" pWWN 20:00:00:cc:1e:dc:01:0b
Alias "BLADE2-SAN-B" pWWN 20:00:00:cc:1e:dc:02:0b
Configure Zoning for SAN A so that both blades can reach "FC-SAN-A" on the A side.
Configure Zoning for SAN B so that both blades can reach "FC-SAN-B" on the B side.
Use the minimum amount of zones necessary to accomplish this.
Score: 5 Points

2.4 iSCSI Virtual Target

The UCS C200 is preconfigured to mount its VMware ESXi Datastores via iSCSI.
Configure the network as follows to allow for this.
The C200 uses VLAN 202 and the initiator IP address 192.168.202.104/24 for iSCSI,
and has the target address configured as 192.168.202.61.
The 3750G is preconfigured with VLAN 202 trunking toward N7K3, and an access
VLAN 202 assignment toward MDS1.
Configure N7K3 so that it trunks only VLAN 202 traffic received from the C200 server
toward MDS1.
Configure MDS1 so that the C200 server is assigned the pWWN
20:00:00:cc:1e:dc:03:0a.
Target LUNs reachable via MDS1's link in VSAN 103 to the FC SAN should be
represented with the IQN "iqn.1987-05.com.cisco:05.mds1.0101.01234567890abcde".

Ensure that the C200 is the only initiator that can use this target.
Do not add any additional zones to accomplish this.
Score: 6 Points

3. Unified Computing
3.1 Address Pools
Configure default pools in the Root ORG on UCS as follows:
UUIDs 0000-000000000001 - 0000-000000000080
MAC Addresses 00:CC:1E:DC:00:01 00:CC:1E:DC:00:FF
nWWNs 20:01:00:CC:1E:DC:01:01 - 20:01:00:CC:1E:DC:01:FF
Management IPs 192.168.101.210 - 192.168.101.219 (GW 192.168.101.1)
Score: 5 Points

3.2 UCS Service Profile Templates

Create a Service Profile Initial Template that will be used for Blades 1 and 2 called
PROFILE.
UUIDs, MAC Addresses, nWWNs, and Management IPs should be pulled from the
previously created default pools.
For SAN connectivity, there should be two vHBAs, fc0 on SAN A using VSAN 103,
and fc1 on SAN B using VSAN 104.
For LAN connectivity, create five vNICs as follows:
vNIC0 named VMKernelA on Fabric A in VLAN 200
vNIC1 named VMKernelB on Fabric B in VLAN 200
vNIC2 named vMotion on Fabric B in VLAN 201
vNIC3 named VMGuestsA on Fabric A with VLANs 202 - 210
vNIC4 named VMGuestsB on Fabric B with VLANs 202 - 210
Ensure that if FI-B loses upstream connectivity that the vMotion NIC does not lose
reachability to the rest of the network.
If a change in this service profile in the future requires re-association to apply the
change, ensure that the administrator is notified before the blade is automatically
rebooted.

Score: 6 Points

3.3 Service Profiles

Create two Service Profiles from the previously created template called PROFILE1
and PROFILE2 for Blade 1 and Blade 2, respectively.
PROFILE1 should be customized as follows:
Assign vHBA FC0 the pWNN 20:00:00:cc:1e:dc:01:0a.
Assign vHBA FC1 the pWNN 20:00:00:cc:1e:dc:01:0b.
Boot to LUN 0 on the SAN target 21:00:00:1b:32:24:5e:dc via FC0 as the
primary, and then to LUN 0 on the SAN target 21:01:00:1b:32:24:5e:dc via
FC1 if booting via FC0 fails.
PROFILE2 should be customized as follows:
Assign vHBA FC0 the pWNN 20:00:00:cc:1e:dc:02:0a.
Assign vHBA FC1 the pWNN 20:00:00:cc:1e:dc:02:0b.
Boot to LUN 0 on the SAN target 21:01:00:1b:32:24:5e:dc via FC1 as the
primary, and then to LUN 0 on the SAN target 21:00:00:1b:32:24:5e:dc via
FC0 if booting via FC1 fails.
Associate PROFILE1 to Blade 1 and PROFILE2 to Blade 2. If successful, the blades
should boot their ESXi instances from the SAN.
Score: 6 Points

4. Data Center Virtualization

4.1 Nexus 1000v
Nexus 1000v VSMs are pre-installed on the ESXi instances for Blade 1 and Blade 2.
The VSM's MGMT0 IP address is 192.168.200.200, and it has a login of admin with
the password Cciedc01.
Modify the existing N1Kv configuration so that the VEM on Blade 1's ESXi host
(192.168.200.101) appears as module 10.
The VEM on Blade 2's ESXi host (192.168.200.102) should appear as module 20.
The C200's ESXi host (192.168.200.104) should dynamically choose any available
VEM slot.

Score: 5 Points

4.2 Private VLANs

Virtual Machines (VMs) Win2k8-www-1 through 6 are preconfigured with IP
addresses 192.168.255.1 through 6, and they have a pre-defined port-group on the
Nexus 1000v. These VMs can be reached through the VMware Console of the
vSphere Client and have the username/password combination Administrator/
Cciedc01.
Create Interface VLAN 204 on N7K3 with the IP address 192.168.255.73/24.
Configure Private-VLANs in such a way that all VMs can ping N7K3's VLAN 204
interface, but cannot ping each other.
Do not make changes to any other devices besides the Nexus 1000v and N7K3 to
accomplish this, including the vCenter server.
Score: 5 Points

CCIE Data Center Full-Scale Labs - Bootcamp

Members - CCIE Data Center Full-Scale Lab 1
CCIE DC Full-Scale Lab 1 Solutions
1. Data Center Infrastructure
2. Data Center Storage Networking
3. Unified Computing
4. Data Center Virtualization

1. Data Center Infrastructure

1.1 UCS Initialization
Configuration
UCS-FI-A:
Enter the configuration method. (console/gui) ?console
Enter the setup mode; setup newly or restore from backup. (setup/restore) ?setup
You have chosen to setup a new Fabric interconnect. Continue? (y/n):y
Enforce strong password? (y/n) [y]:y
Enter the password for "admin":Cciedc01
Confirm the password for "admin":Cciedc01
Is this Fabric interconnect part of a cluster(select 'no' for standalone)? (yes/no) [n]:yes
Enter the switch fabric (A/B) []:A
Enter the system name:UCS-FI
Physical Switch Mgmt0 IPv4 address :192.168.101.201
Physical Switch Mgmt0 IPv4 netmask :255.255.255.0
IPv4 address of the default gateway :192.168.101.1
Cluster IPv4 address :192.168.101.200

Configure the DNS Server IPv4 address? (yes/no) [n]:

Configure the default domain name? (yes/no) [n]:

Following configurations will be applied:

Switch Fabric=A

System Name=UCS-FI
Enforced Strong Password=yes
Physical Switch Mgmt0 IP Address=192.168.101.201
Physical Switch Mgmt0 IP Netmask=255.255.255.0
Default Gateway=192.168.101.1

Cluster Enabled=yes
Cluster IP Address=192.168.101.200
NOTE: Cluster IP will be configured only after both Fabric Interconnects are initialized
Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no):yes
Applying configuration. Please wait.

Configuration file - Ok
UCS-FI-B:
Enter the configuration method. (console/gui) ?console

Installer has detected the presence of a peer Fabric interconnect. This Fabric interconnect will be added to the c
y
Enter the admin password of the peer Fabric interconnect:Cciedc01
Connecting to peer Fabric interconnect... done
Retrieving config from peer Fabric interconnect... done
Peer Fabric interconnect Mgmt0 IP Address: 192.168.101.201
Peer Fabric interconnect Mgmt0 IP Netmask: 255.255.255.0
Cluster IP address

: 192.168.101.200

Physical Switch Mgmt0 IPv4 address :192.168.101.202

Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no):yes

Applying configuration. Please wait.

Configuration file - Ok

Like Nexus, UCS allows SSH access by default. Telnet can be enabled from the
UCSM GUI, or from the CLI as follows.
UCS-FI-A#scope system
UCS-FI-A /system #scope services
UCS-FI-A /system/services #enable telnet-server
UCS-FI-A /system/services* #commit-buffer
UCS-FI-A /system/services #end
UCS-FI-A#exit

1.2 Nexus 7K VDC Initialization

Configuration
First remove all interfaces from the default VDC by allowing only F2 ports. This will
force all M1 and F1 ports to be allocated to VDC 0:
N7K1#config t
N7K1(config)#feature telnet
N7K1(config)#vdc N7K1
N7K1(config-vdc)#limit-resource module-type f2
This will cause all ports of unallowed types to be removed from this vdc. Continue (y/n)? [yes]yes
N7K1(config-vdc)#show vdc membership

vdc_id: 0 vdc_name: Unallocated interfaces:

Ethernet1/1

Ethernet1/2

Ethernet1/3

Ethernet1/4

Ethernet1/5

Ethernet1/6

Ethernet1/7

Ethernet1/8

Ethernet1/9

Ethernet1/10

Ethernet1/11

Ethernet1/12

Ethernet1/13

Ethernet1/14

Ethernet1/15

Ethernet1/16

Ethernet1/17

Ethernet1/18

Ethernet1/19

Ethernet1/20

Ethernet1/21

Ethernet1/22

Ethernet1/23

Ethernet1/24

Ethernet1/25

Ethernet1/26

Ethernet1/27

Ethernet1/28

Ethernet1/29

Ethernet1/30

Ethernet1/31

Ethernet1/32

Ethernet2/1

Ethernet2/2

Ethernet2/3

Ethernet2/4

Ethernet2/5

Ethernet2/6

Ethernet2/7

Ethernet2/8

Ethernet2/9

Ethernet2/10

Ethernet2/11

Ethernet2/12

Ethernet2/13

Ethernet2/14

Ethernet2/15

Ethernet2/16

Ethernet2/17

Ethernet2/18

Ethernet2/19

Ethernet2/20

Ethernet2/21

Ethernet2/22

Ethernet2/23

Ethernet2/24

Ethernet2/25

Ethernet2/26

Ethernet2/27

Ethernet2/28

Ethernet2/29

Ethernet2/30

Ethernet2/31

Ethernet2/32

vdc_id: 1 vdc_name: N7K1 interfaces:

Now change the default VDC back to allow both M1 and F1 ports, create the other

VDCs, and allocate the needed ports.

N7K1(config)#no vdc combined-hostname
N7K1(config)#vdc N7K1
N7K1(config-vdc)#limit-resource module-type m1 f1 m1xl
This will cause all ports of unallowed types to be removed from this vdc. Continue (y/n)? [yes]yes
N7K1(config-vdc)#allocate interface Ethernet1/1-8

Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the
yes
N7K1(config-vdc)#vdc N7K2 id 2
Note:

Creating VDC, one moment please ...

N7K1 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 2 has come online N7K1(config-vdc)#

allocate interface Ethernet1/25-32

Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the
yes
N7K1(config-vdc)#allocate interface Ethernet2/3-4

Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the
yes
N7K1(config-vdc)#vdc N7K3 id 3
Note:

Creating VDC, one moment please ...

N7K1 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 3 has come online N7K1(config-vdc)#

allocate interface Ethernet1/9-16

Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the
yes
N7K1(config-vdc)#allocate interface Ethernet2/21-24

Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the
yes
N7K1(config-vdc)#vdc N7K4 id 4
Note:

Creating VDC, one moment please ...

N7K1 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 4 has come online N7K1(config-vdc)#

allocate interface Ethernet1/17-24

Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the
yes
N7K1(config-vdc)#allocate interface Ethernet2/5-8,Ethernet2/13-14,Ethernet2/19-20

Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the
yes
N7K1(config-vdc)#end

Now "switchto" the VDCs to configure the admin password as well as the MGMT0 IP
address.
N7K1#switchto vdc N7K2

---- System Admin Account Setup ----

Do you want to enforce secure password standard (yes/no) [y]:y

Enter the password for "admin":Cciedc01
Confirm the password for "admin":Cciedc01

---- Basic System Configuration Dialog VDC: 2 ----

This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.

Please register Cisco Nexus7000 Family devices promptly with your

supplier. Failure to register may affect response times for initial
service calls. Nexus7000 devices must be registered to receive
entitled support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime

to skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): n

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php N7K2#config t
Enter configuration commands, one per line.

End with CNTL/Z. N7K2(config)#feature telnet

N7K2(config)#interface mgmt0
N7K2(config-if)#ip address 192.168.101.72/24
N7K2(config-if)#end
N7K2#switchback
N7K1#switchto vdc N7K3

---- System Admin Account Setup ----

Do you want to enforce secure password standard (yes/no) [y]:y

Enter the password for "admin":Cciedc01
Confirm the password for "admin":Cciedc01

---- Basic System Configuration Dialog VDC: 3 ----

This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.

Please register Cisco Nexus7000 Family devices promptly with your

supplier. Failure to register may affect response times for initial
service calls. Nexus7000 devices must be registered to receive
entitled support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime

to skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): n

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php N7K3#conf t
Enter configuration commands, one per line.

End with CNTL/Z. N7K3(config)#feature telnet

N7K3(config)#int mgmt0
N7K3(config-if)#ip address 192.168.101.73/24
N7K3(config-if)#end
N7K3#switchback

N7K1#switchto vdc N7K4

---- System Admin Account Setup ----

Do you want to enforce secure password standard (yes/no) [y]:y

Enter the password for "admin":Cciedc01
Confirm the password for "admin":Cciedc01

---- Basic System Configuration Dialog VDC: 4 ----

This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management

of the system.

Please register Cisco Nexus7000 Family devices promptly with your

supplier. Failure to register may affect response times for initial
service calls. Nexus7000 devices must be registered to receive
entitled support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime

to skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): n

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php N7K4#config t
Enter configuration commands, one per line.

End with CNTL/Z. N7K4(config)#feature telnet

N7K4(config)#interface mgmt 0
N7K4(config-if)#ip address 192.168.101.74/24
N7K4(config-if)#end
N7K4#switchback
N7K1#copy running-config startup-config vdc-all

[#######

17%

[############

29%

[######################

53%

[############################

69%

[#####################################

90%

[########################################] 100%

Verification
N7K1#show vdc membership
vdc_id: 0 vdc_name: Unallocated interfaces:

Ethernet2/1

Ethernet2/2

Ethernet2/9

Ethernet2/10

Ethernet2/11

Ethernet2/12

Ethernet2/15

Ethernet2/16

Ethernet2/17

Ethernet2/18

Ethernet2/25

Ethernet2/26

Ethernet2/27

Ethernet2/28

Ethernet2/29

Ethernet2/30

Ethernet2/31

Ethernet2/32

vdc_id: 1 vdc_name: N7K1 interfaces:

Ethernet1/1

Ethernet1/2

Ethernet1/3

Ethernet1/4

Ethernet1/5

Ethernet1/6

Ethernet1/7

Ethernet1/8

vdc_id: 2 vdc_name: N7K2 interfaces:

Ethernet1/25

Ethernet1/26

Ethernet1/27

Ethernet1/28

Ethernet1/29

Ethernet1/30

Ethernet1/31

Ethernet1/32

Ethernet2/3

Ethernet2/4

vdc_id: 3 vdc_name: N7K3 interfaces:

Ethernet1/9

Ethernet1/10

Ethernet1/11

Ethernet1/12

Ethernet1/13

Ethernet1/14

Ethernet1/15

Ethernet1/16

Ethernet2/21

Ethernet2/22

Ethernet2/23

Ethernet2/24

vdc_id: 4 vdc_name: N7K4 interfaces:

Ethernet1/17

Ethernet1/18

Ethernet1/19

Ethernet1/20

Ethernet1/21

Ethernet1/22

Ethernet1/23

Ethernet1/24

Ethernet2/5

Ethernet2/6

Ethernet2/7

Ethernet2/8

Ethernet2/13

Ethernet2/14

Ethernet2/19

Ethernet2/20

Some interfaces not listed on the diagram must still be allocated to

VDCs 1 - 4 due to the port-group boundaries. Port-groupings can be
verified as shown below.

N7K1#show interface capabilities | include "Ethernet|Group"

Ethernet1/1 Port Group Members:

1,3,5,7

Ethernet1/2 Port Group Members:

2,4,6,8

Ethernet1/3
Port Group Members:

1,3,5,7

Ethernet1/4
Port Group Members:

2,4,6,8

Ethernet1/5
Port Group Members:

1,3,5,7

Ethernet1/6
Port Group Members:

2,4,6,8

Ethernet1/7
Port Group Members:

1,3,5,7

Ethernet1/8
Port Group Members:

2,4,6,8

1.3 Initial IP Addressing

Configuration
N7K1:
interface Ethernet1/2
ip address 10.71.73.71/24
no shutdown
N7K2:
feature lacp
!
interface Ethernet1/25
channel-group 10 mode active
no shutdown
!
interface Ethernet1/26
channel-group 10 mode active
no shutdown
!
interface port-channel10
ip address 10.72.74.72/24
N7K3:
feature interface-vlan
!
vlan 1050
!
interface Ethernet1/10
ip address 10.71.73.73/24
no shutdown
!
interface Ethernet2/21
switchport access vlan 1050
spanning-tree port type edge
no shutdown
!

interface Vlan1050
no shutdown
ip address 10.50.73.0/31
N7K4:

feature interface-vlan
!
feature lacp
!
vlan 1051
!
interface Ethernet1/17
channel-group 10 mode active
no shutdown
!
interface Ethernet1/18
channel-group 10 mode active
no shutdown
!
interface port-channel10
ip address 10.72.74.74/24
!
interface Ethernet2/5
switchport access vlan 1051
spanning-tree port type edge
no shutdown
!
interface Vlan1051
no shutdown
ip address 10.51.74.0/31

Verification
N7K2#show port-channel summary
Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------10

Po10(RU)

Eth

LACP

Eth1/25(P)

Eth1/26(P)

N7K2#show ip route direct

IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
10.72.74.0/24
, ubest/mbest: 1/0, attached *via 10.72.74.72, Po10
, [0/0], 21:47:09, direct
N7K2#ping 10.72.74.72
PING 10.72.74.72 (10.72.74.72): 56 data bytes
64 bytes from 10.72.74.72: icmp_seq=0 ttl=255 time=0.597 ms
64 bytes from 10.72.74.72: icmp_seq=1 ttl=255 time=0.295 ms
64 bytes from 10.72.74.72: icmp_seq=2 ttl=255 time=0.539 ms
64 bytes from 10.72.74.72: icmp_seq=3 ttl=255 time=0.345 ms
64 bytes from 10.72.74.72: icmp_seq=4 ttl=255 time=0.336 ms

--- 10.72.74.72 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 0.295/0.422/0.597 ms

1.4 Layer 3 Routing

Configuration
N7K1:
ip route 0.0.0.0/0 10.71.73.73
N7K2:
ip route 0.0.0.0/0 10.72.74.74
N7K3:
feature bgp
!
router bgp 65001
address-family ipv4 unicast
network 10.71.73.0/24
neighbor 10.50.73.1
remote-as 100
password 0 DCIPROVIDER
address-family ipv4 unicast
N7K4:

feature bgp
!
router bgp 65002

log-neighbor-changes
address-family ipv4 unicast
network 10.72.74.0/24
neighbor 10.51.74.1
remote-as 100
password 0 DCIPROVIDER
address-family ipv4 unicast

Verification
N7K3#show ip bgp neighbors

BGP neighbor is 10.50.73.1,

remote AS 100, ebgp link,

Peer index 1

BGP version 4, remote router ID 10.0.0.50

BGP state = Established, up for 21:47:57
Peer is directly attached, interface Vlan1050 TCP MD5 authentication is enabled

N7K3#show bgp ipv4 unicast summary

BGP summary information for VRF default, address family IPv4 Unicast
BGP router identifier 10.71.73.73, local AS number 65001
BGP table version is 8, IPv4 Unicast config peers 1, capable peers 1
4 network entries and 4 paths using 496 bytes of memory
BGP attribute entries [4/512], BGP AS path entries [2/16]
BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor

10.50.73.1

AS MsgRcvd MsgSent
100

1301

1310

TblVer
8

InQ OutQ Up/Down

State/PfxRcd

0 21:45:52 3

N7K3#show bgp ipv4 unicast

BGP routing table information for VRF default, address family IPv4 Unicast
BGP table version is 8, local router ID is 10.71.73.73
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath

Network

Next Hop

*>e10.0.0.50/32

10.50.73.1

*>e10.0.0.51/32

10.50.73.1

*>l10.71.73.0/24

0.0.0.0

10.50.73.1
N7K3#show ip route bgp
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop

Metric

LocPrf

Weight Path

0 100 i
0 100 i
100

32768 i *>e10.72.74.0/24

0 100 65002 i

'[x/y]' denotes [preference/metric]

'%' in via output denotes VRF

10.0.0.50/32, ubest/mbest: 1/0

*via 10.50.73.1, [20/0], 21:46:03, bgp-65001, external, tag 100
10.0.0.51/32, ubest/mbest: 1/0
*via 10.50.73.1, [20/0], 21:46:03, bgp-65001, external, tag 100 10.72.74.0/24
, ubest/mbest: 1/0 *via 10.50.73.1, [20/0], 21:45:14, bgp-65001
, external, tag 100
N7K2#ping 10.71.73.71
PING 10.71.73.71 (10.71.73.71): 56 data bytes
64 bytes from 10.71.73.71: icmp_seq=0 ttl=250 time=1.343 ms
64 bytes from 10.71.73.71: icmp_seq=1 ttl=250 time=0.741 ms
64 bytes from 10.71.73.71: icmp_seq=2 ttl=250 time=0.822 ms
64 bytes from 10.71.73.71: icmp_seq=3 ttl=250 time=0.85 ms
64 bytes from 10.71.73.71: icmp_seq=4 ttl=250 time=0.844 ms

--- 10.71.73.71 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 0.741/0.92/1.343 ms

1.5 FabricPath
Configuration
N5K1:
install feature-set fabricpath
feature-set fabricpath
feature lacp
!
vlan 200-299
mode fabricpath
!
key chain FABRICPATH
key 1
key-string 0 FPAUTH
!
interface port-channel20
switchport
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FABRICPATH
!
interface Ethernet1/4

switchport mode fabricpath

channel-group 20 mode active
no shutdown
!
interface Ethernet1/5
switchport mode fabricpath
channel-group 20 mode active
no shutdown
N5K2:
install feature-set fabricpath
feature-set fabricpath
!
vlan 200-299
mode fabricpath
!
key chain FABRICPATH
key 1
key-string 0 FPAUTH
!
interface Ethernet1/3
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FABRICPATH
no shutdown
N7K1:
install feature-set fabricpath
N7K4:

feature-set fabricpath
!
vlan 200-299
mode fabricpath
!
key chain FABRICPATH
key 1
key-string 0 FPAUTH
!
interface port-channel20
switchport
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FABRICPATH
!
interface Ethernet2/6
switchport mode fabricpath
fabricpath isis authentication-type md5

fabricpath isis authentication key-chain FABRICPATH

no shutdown
!
interface Ethernet2/7
switchport mode fabricpath
channel-group 20 mode active
no shutdown
!
interface Ethernet2/13
switchport mode fabricpath
channel-group 20 mode active
no shutdown

Verification
N7K4#show port-channel summary
Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------10

Po10(RU)

Eth

LACP

Eth1/17(P)

Eth1/18(P)

Po20(SU)

Eth

LACP

Eth2/7(P)

Eth2/13(P)

N7K4#show fabricpath isis adjacency

Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID

SNPA

Level

State

Hold Time

Interface

N5K1

N/A

00:00:24

port-channel20

N5K2

N/A

00:00:29

Ethernet2/6

N7K4#show fabricpath isis interface port-channel 20

Fabricpath IS-IS domain: default
Interface: port-channel20
Status: protocol-up/link-up/admin-up
Index: 0x0002, Local Circuit ID: 0x01, Circuit Type: L1 Authentication type MD5
Authentication keychain is FABRICPATH
Authentication check specified

Extended Local Circuit ID: 0x16000013, P2P Circuit ID: 0000.0000.0000.00

Retx interval: 5, Retx throttle interval: 66 ms
LSP interval: 33 ms, MTU: 1500

P2P Adjs: 1, AdjsUp: 1, Priority 64

Hello Interval: 10, Multi: 3, Next IIH: 00:00:04
Level

Adjs

AdjsUp

Metric

CSNP

Next CSNP

Last LSP ID

00:00:55

ffff.ffff.ffff.ff-ff

Topologies enabled:
Topology Metric

MetricConfig Forwarding

1.6 vPC+
Configuration
N5K1:
feature vpc
!
vpc domain 500
peer-keepalive destination 192.168.101.52
fabricpath switch-id 501
!
interface Ethernet1/1
switchport mode fabricpath
channel-group 500 mode active
no shutdown
!
interface Ethernet1/2
switchport mode fabricpath
channel-group 500 mode active
no shutdown
!
interface Ethernet1/8
switchport mode trunk
switchport trunk allowed vlan 200-299
channel-group 201 mode active
no shutdown
!
interface Ethernet1/9
switchport mode trunk
switchport trunk allowed vlan 200-299
channel-group 202 mode active
no shutdown
!
interface port-channel500
switchport mode fabricpath
vpc peer-link

fabricpath isis authentication-type md5

fabricpath isis authentication key-chain FABRICPATH
!
interface port-channel201
switchport mode trunk
switchport trunk allowed vlan 200-299
spanning-tree port type edge trunk
vpc 201
!
interface port-channel202
switchport mode trunk
switchport trunk allowed vlan 200-299
spanning-tree port type edge trunk
vpc 202
N5K2:

feature vpc
feature lacp
!
vpc domain 500
peer-keepalive destination 192.168.101.51
fabricpath switch-id 501
!
interface Ethernet1/1
switchport mode fabricpath
channel-group 500 mode active
no shutdown
!
interface Ethernet1/2
switchport mode fabricpath
channel-group 500 mode active
no shutdown
!
interface Ethernet1/7
switchport mode trunk
switchport trunk allowed vlan 200-299
channel-group 201 mode active
no shutdown
!
interface Ethernet1/10
switchport mode trunk
switchport trunk allowed vlan 200-299
channel-group 202 mode active
no shutdown
!
interface port-channel500

switchport mode fabricpath

vpc peer-link
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FABRICPATH
!
interface port-channel201
switchport mode trunk
switchport trunk allowed vlan 200-299
spanning-tree port type edge trunk
vpc 201
!
interface port-channel202
switchport mode trunk
switchport trunk allowed vlan 200-299
spanning-tree port type edge trunk
vpc 202

Connect to the UCSM using the credentials that you previously configured. Next,
under the Fabric Interconnects on the Equipment tab, configure the Ethernet links
connecting northbound to the N5Ks in the diagram as Uplink Ports.

Now under the LAN tab, create and enable Port-Channels 201 and 202 on FI-A and
FI-B respectively.

Verification
N5K1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id
vPC+ switch id

: 500
: 501

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive

vPC fabricpath status

: peer is reachable through fabricpath

Configuration consistency status: success

Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: secondary

Number of vPCs configured

: 2

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

vPC Peer-link status

--------------------------------------------------------------------id

Port

Status Active vlans

----

------ -------------------------------------------------- 1

Po500

vPC status
--------------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans vPC+ Attrib

----------

------ ----------- ------

------------ -----------

201

Po201

200-299

202

Po202

success
up

success

success
success

200-299

DF: Partial
DF: Partial

UCS-FI-A:
UCS-FI-A#connect nxos
UCS-FI-A(nxos)#show run interface ethernet 1/4 - 5

interface Ethernet1/4
description U: Uplink
pinning border
switchport mode trunk
switchport trunk allowed vlan 1,200-299
channel-group 201 mode active
no shutdown

interface Ethernet1/5
description U: Uplink
pinning border
switchport mode trunk
switchport trunk allowed vlan 1,200-299
channel-group 201 mode active
no shutdown
UCS-FI-A(nxos)#show port-channel summary
Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
--------------------------------------------------------------------------------

200-299

Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------201

Po201(SU)

Eth

LACP

Eth1/4(P)

Eth1/5(P)

1.7 FabricPath Traffic Engineering

Configuration
N7K4:

interface port-channel20
fabricpath isis metric 40

Verification
N5K1 and N5K2 share the emulated FabricPath Switch-ID 501 for the vPC+, as
shown below:
N7K4#show fabricpath switch-id

FABRICPATH SWITCH-ID TABLE

Legend: '*' - this system
=========================================================================
SWITCH-ID

SYSTEM-ID

FLAGS

STATE

STATIC

EMULATED

----------+----------------+------------+-----------+-------------------501

547f.ee79.137c
501

*645

Primary

547f.ee7a.4d7c

Confirmed

Primary

Confirmed

Yes
No

Yes

64a0.e742.8dc4

Primary

Confirmed

1207

547f.ee79.137c

Primary

Confirmed

3550

547f.ee7a.4d7c

Primary

Confirmed

Total Switch-ids: 5

The port channel between N7K4 and N5K1 has an IS-IS metric of 20, whereas the
single 10GigE link from N7K4 to N5K2 has an IS-IS metric of 40. This means that
the shortest path from N7K4 to Switch-ID 501 (the vPC+ pair) is only via N5K1.
N7K4#show fabricpath route

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id

'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/645/0, number of next-hops: 0

via ---- , [60/0], 0 day/s 22:19:30, local 1/501/0, number of next-hops: 1
via Po20, [115/20]
, 0 day/s 20:30:58, isis_fabricpath-default
1/1207/0, number of next-hops: 2
via Po20, [115/40], 0 day/s 20:30:58, isis_fabricpath-default
via Eth2/6, [115/40], 0 day/s 22:19:16, isis_fabricpath-default
1/3550/0, number of next-hops: 1
via Po20, [115/20], 0 day/s 22:16:06, isis_fabricpath-default

To allow for Equal Cost Multipath (ECMP), the port channel to N5K1 and the single
link to N5K2 must have equal costs. This can be configured either by raising the
cost of the port channel or by lowering the cost of the link to N5K2.
N7K4#config t
Enter configuration commands, one per line.

End with CNTL/Z. N7K4(config)#interface port-channel20

N7K4(config-if)#fabricpath isis metric 40

N7K4(config-if)# end

Now Switch-ID 501 is reachable via both N5K1 and N5K2 with a metric of 40.
N7K4#show fabricpath route

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/645/0, number of next-hops: 0

via ---- , [60/0], 0 day/s 22:19:58, local 1/501/0, number of next-hops: 2

via Po20, [115/40]

, 0 day/s 20:31:26, isis_fabricpath-default via Eth2/6, [115/40]
, 0 day/s 00:00:06, isis_fabricpath-default
1/1207/0, number of next-hops: 1
via Eth2/6, [115/40], 0 day/s 22:19:44, isis_fabricpath-default
1/3550/0, number of next-hops: 1
via Po20, [115/40], 0 day/s 22:16:34, isis_fabricpath-default

1.8 Spanning-Tree Protocol Optimization

Configuration
N5K1:
spanning-tree mode mst
spanning-tree mst 0 priority 0
spanning-tree mst configuration
name MST0
revision 1
N5K2:

spanning-tree mode mst

spanning-tree mst 0 priority 0
spanning-tree mst configuration
name MST0
revision 1

Verification
In the below output, we can see that both N5K1 and N5K2 have collapsed all of their
STP instances into the single default MST0 instance. Additionally, both switches in
the vPC+ pair should always appear as the root of the Spanning-Tree, and share
the Bridge-ID c84c.75fa.6000. Note that Spanning-Tree only forwards southbound
toward the Classical Ethernet domain, and not northbound toward the FabricPath
domain.
N5K1#show spanning-tree mst 0
##### MST0 vlans mapped:

1-4094

Bridge

address c84c.75fa.6000

Root

this switch for the CIST

priority

(0 sysid 0)

Regional Root this switch

Operational

hello time 2 , forward delay 15, max age 20, txholdcount 6

Configured

hello time 2 , forward delay 15, max age 20, max hops

Interface

Role Sts Cost

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po201

Desg FWD 200

128.4296 (vPC) Edge P2p

Po202

Desg FWD 200

128.4297 (vPC) Edge P2p

N5K2#show spanning-tree mst 0

##### MST0 vlans mapped:

1-4094

Bridge

address c84c.75fa.6000

Root

this switch for the CIST

priority

(0 sysid 0)

Regional Root this switch

Operational

hello time 2 , forward delay 15, max age 20, txholdcount 6

Configured

hello time 2 , forward delay 15, max age 20, max hops

Interface

Role Sts Cost

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po201

Desg FWD 200

128.4296 (vPC) Edge P2p

Po202

Desg FWD 200

128.4297 (vPC) Edge P2p

Eth1/11

Desg FWD 20000

128.139

1.9 Fabric Extenders

N7K1:
install feature-set fex
N7K3:

feature-set fex
!
interface port-channel131
switchport
switchport mode fex-fabric
fex associate 131
!
interface port-channel132
switchport
switchport mode fex-fabric
fex associate 132
!
interface Ethernet1/13
switchport
switchport mode fex-fabric
fex associate 131
channel-group 131
no shutdown

P2p Bound(PVST)

!
interface Ethernet1/14
switchport
switchport mode fex-fabric
fex associate 131
channel-group 131
no shutdown
!
interface Ethernet1/15
switchport
switchport mode fex-fabric
fex associate 132
channel-group 132
no shutdown
!
interface Ethernet1/16
switchport
switchport mode fex-fabric
fex associate 132
channel-group 132
no shutdown
!
interface Ethernet131/1/1
switchport
switchport mode trunk
switchport trunk allowed vlan 200-299
no shutdown
!
interface Ethernet132/1/1
switchport
switchport mode trunk
switchport trunk allowed vlan 200-299
no shutdown

Verification
N7K3#show fex
FEX
Number

FEX

Description

State

FEX
Model

Serial

-----------------------------------------------------------------------131

FEX0131

Online

N2K-C2232PP-10GE

FOC17100NHX 132

N2K-C2232PP-10GE

FOC17100NHU

FEX0132

Online

N7K3#show fex detail

FEX: 131 Description: FEX0131

state: Online

FEX version: 6.0(2) [Switch version: 6.0(2)]

FEX Interim version: 6.0(2.9)
Switch Interim version: 6.0(2)
Extender Model: N2K-C2232PP-10GE,

Extender Serial: FOC17100NHX

Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:00:42, Num Macs: 64
Module Sw Gen: 12594
pinning-mode: static

[Switch Sw Gen: 21]

Max-links: 1

Fabric port for control traffic: Eth1/14

Fabric interface state: Po131 - Interface Up. State: Active
Eth1/13 - Interface Up. State: Active
Eth1/14 - Interface Up. State: Active
Fex Port

State

Fabric Port Eth131/1/1

FEX: 132 Description: FEX0132

Po131

state: Online

FEX version: 6.0(2) [Switch version: 6.0(2)]

FEX Interim version: 6.0(2.9)
Switch Interim version: 6.0(2)
Extender Model: N2K-C2232PP-10GE,

Extender Serial: FOC17100NHU

Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:02:02, Num Macs: 64
Module Sw Gen: 12594
pinning-mode: static

[Switch Sw Gen: 21]

Max-links: 1

Fabric port for control traffic: Eth1/15

Fabric interface state: Po132 - Interface Up. State: Active
Eth1/15 - Interface Up. State: Active
Eth1/16 - Interface Up. State: Active
Fex Port

State

Fabric Port Eth132/1/1

Po132

1.10 OTV
Configuration
The OTV Site VLAN is in decimal, but the OTV Site Identifier is in hex,
which means that a decimal to hex conversion is required.

N7K1:
feature otv
!

vlan 200-299,3001
!
otv site-vlan 3001
otv site-identifier 0xbb9
!
spanning-tree vlan 3001 priority 0
!
key chain OTV
key 1
key-string 0 OTVAUTH
!
interface Overlay1
otv isis authentication-type md5
otv isis authentication key-chain OTV
otv join-interface Ethernet1/2
otv control-group 224.71.72.0
otv data-group 232.71.71.0/24
otv extend-vlan 200-299
no shutdown
!
interface Ethernet1/1
switchport
switchport mode trunk
switchport trunk allowed vlan 200-299,3001
no shutdown
!
interface Ethernet1/2
ip igmp version 3
N7K2:
feature otv
!
vlan 200-299,3002
!
otv site-vlan 3002
otv site-identifier 0xbba
!
key chain OTV
key 1
key-string 0 OTVAUTH
!
interface port-channel10
ip igmp version 3
!
interface Overlay1
otv isis authentication-type md5
otv isis authentication key-chain OTV

otv join-interface port-channel10

otv control-group 224.71.72.0
otv data-group 232.72.72.0/24
otv extend-vlan 200-299
no shutdown
!
interface Ethernet2/3
switchport mode trunk
switchport trunk allowed vlan 200-299,3002
no shutdown
N7K3:
feature pim
!
vlan 200-299,3001
!
interface Vlan200
no shutdown
ip address 192.168.200.73/24
!
interface Vlan1050
ip pim sparse-mode
!
interface Ethernet1/9
switchport
switchport mode trunk
switchport trunk allowed vlan 200-299,3001
no shutdown
!
interface Ethernet1/10
ip pim sparse-mode
ip igmp version 3
!
ip pim rp-address 10.0.0.51 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
N7K4:

feature pim
!
vlan 3002
!
spanning-tree vlan 200-299 priority 0
!
interface Vlan200
no shutdown
ip address 192.168.200.74/24
!

interface Vlan1051
ip pim sparse-mode
!
interface port-channel10
ip pim sparse-mode
ip igmp version 3
!
interface Ethernet2/19
switchport mode trunk
switchport trunk allowed vlan 200-299,3002
no shutdown
!
ip pim rp-address 10.0.0.51 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8

Verification
To establish the OTV tunnel, the AEDs must have multicast reachability to each
other with the control group. The first step in verification, then, is to ensure that the
tree for the control multicast group is built in the DCI core. Both N7K3 and N7K4
should see the (S,G) entries for the control group 224.71.72.0.
N7K3#show ip mroute
IP Multicast Routing Table for VRF "default"

(*, 224.71.72.0/32), uptime: 00:11:06, igmp ip pim

Incoming interface: Vlan1050, RPF nbr: 10.50.73.1
Outgoing interface list: (count: 1)
Ethernet1/10, uptime: 00:11:06, igmp
(10.71.73.71/32, 224.71.72.0/32)
, uptime: 00:12:45, ip pim mrib

Incoming interface: Ethernet1/10

, RPF nbr: 10.71.73.71 Outgoing interface list

: (count: 2)
Ethernet1/10, uptime: 00:11:06, mrib, (RPF) Vlan1050
, uptime: 00:12:34, pim
(10.72.74.72/32, 224.71.72.0/32)
, uptime: 00:11:03, ip mrib pim

Incoming interface: Vlan1050

, RPF nbr: 10.50.73.1 Outgoing interface list

: (count: 1) Ethernet1/10
, uptime: 00:11:03, mrib

(*, 232.0.0.0/8), uptime: 00:12:54, pim ip

Incoming interface: Null, RPF nbr: 0.0.0.0
Outgoing interface list: (count: 0)

N7K4#show ip mroute

IP Multicast Routing Table for VRF "default"

(*, 224.71.72.0/32), uptime: 00:13:47, igmp ip pim

Incoming interface: Vlan1051, RPF nbr: 10.51.74.1
Outgoing interface list: (count: 1)
port-channel10, uptime: 00:13:47, igmp
(10.71.73.71/32, 224.71.72.0/32)
, uptime: 00:13:39, ip mrib pim

Incoming interface: Vlan1051

, RPF nbr: 10.51.74.1 Outgoing interface list

: (count: 1) port-channel10
, uptime: 00:13:39, mrib
(10.72.74.72/32, 224.71.72.0/32)
, uptime: 00:13:44, ip mrib pim

Incoming interface: port-channel10

, RPF nbr: 10.72.74.72 Outgoing interface list

: (count: 2) Vlan1051
, uptime: 00:12:18, pim
port-channel10, uptime: 00:13:44, mrib, (RPF)

(*, 232.0.0.0/8), uptime: 00:13:53, pim ip

Incoming interface: Null, RPF nbr: 0.0.0.0
Outgoing interface list: (count: 0)

Ensure that the Site VLAN is up on both AEDs.

N7K1#show otv

OTV Overlay Information

Site Identifier 0000.0000.0bb9

Overlay interface Overlay1

VPN name

: Overlay1

VPN state

: UP

Extended vlans

: 200-299 (Total:100)

Control group

: 224.71.72.0

Data group range(s) : 232.71.71.0/24

Join interface(s)

: Eth1/2 (10.71.73.71)

AED-Capable

: Yes

Capability

: Multicast-Reachable

N7K2#show otv

OTV Overlay Information

Site Identifier 0000.0000.0bba

Site vlan

: 3001 (up)

Overlay interface Overlay1

VPN name

: Overlay1

VPN state

: UP

Extended vlans

: 200-299 (Total:100)

Control group

: 224.71.72.0

Data group range(s) : 232.72.72.0/24

Join interface(s)

: Po10 (10.72.74.72)

AED-Capable

: Yes

Capability

: Multicast-Reachable

Site vlan

: 3002 (up)

Now the AEDs should be able to form an IS-IS adjacency over the OTV tunnel.
N7K1#show otv isis adjacency
OTV-IS-IS process: default VPN: Overlay1
OTV-IS-IS adjacency database:
System ID

SNPA

Level

State

Hold Time

Interface Site-ID

N7K2

64a0.e742.8dc2

00:00:08

Overlay1 0000.0000.0bba

Verify that MD5 authentication for IS-IS is enabled on the Overlay1 interface.

N7K1#show otv isis interface overlay 1

OTV-IS-IS process: default VPN: Overlay1

Overlay1, Interface status: protocol-up/link-up/admin-up
IP address: none
IPv6 address: none
IPv6 link-local address: none
Index: 0x0001, Local Circuit ID: 0x01, Circuit Type: L1
Level1
Adjacency server (local/remote) : disabled / none
Adjacency server capability : multicast Authentication type is MD5
Authentication keychain is OTV
Authentication check specified

LSP interval: 33 ms, MTU: 1400

Level

Metric

CSNP

Level
1

Adjs

AdjsUp Pri

Next CSNP

Hello

Inactive

Multi

Next IIH

00:00:03

Circuit ID

Since

N7K2.01

00:15:55

N7K3 and N7K4 should now be able to reach each other's VLAN 200 interfaces,
and the OTV AEDs should learn the routes to these MAC addresses.
N7K4#show interface vlan 200 | include ddress
Hardware is EtherSVI, address is 64a0.e742.8dc4
Internet Address is 192.168.200.74/24
N7K3#ping 192.168.200.74
PING 192.168.200.74 (192.168.200.74): 56 data bytes
64 bytes from 192.168.200.74: icmp_seq=0 ttl=254 time=1.256 ms
64 bytes from 192.168.200.74: icmp_seq=1 ttl=254 time=0.938 ms
64 bytes from 192.168.200.74: icmp_seq=2 ttl=254 time=0.859 ms
64 bytes from 192.168.200.74: icmp_seq=3 ttl=254 time=0.924 ms
64 bytes from 192.168.200.74: icmp_seq=4 ttl=254 time=0.852 ms

--- 192.168.200.74 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.852/0.965/1.256 ms
N7K1#show otv route

OTV Unicast MAC Routing Table For Overlay1

VLAN MAC-Address

Metric

Uptime

Owner

Next-hop(s)

---- --------------

------

--------

---------

-----------

00:18:25

overlay

N7K2

200 000c.29bb.9b82

200 64a0.e742.8dc3
200 64a0.e742.8dc4
200 d48c.b5bd.460c

1
42
1

00:18:15
00:18:14
00:18:23

site

Ethernet1/1

overlay
site

N7K2
Ethernet1/1

N7K2#show otv route

OTV Unicast MAC Routing Table For Overlay1

VLAN MAC-Address

Metric

Uptime

Owner

Next-hop(s)

---- --------------

------

--------

---------

-----------

200 000c.29bb.9b82

00:19:03

site

Ethernet2/3

200 64a0.e742.8dc3

00:18:24

overlay

N7K1

200 64a0.e742.8dc4

200 d48c.b5bd.460c

00:18:24

00:18:32

site

Ethernet2/3

overlay

N7K1

Multicast tunneling can be verified by joining a multicast group on one of the

switches and then sending ICMP pings from the remote OTV site. If successful, a
new OTV multicast tunnel should form using the OTV multicast data groups.
N7K3#config t
Enter configuration commands, one per line.

End with CNTL/Z. N7K3(config)#interface vlan 200

N7K3(config-if)#ip pim sparse-mode

N7K3(config-if)#ip igmp join-group 224.1.1.1
N7K4#ping multicast 224.1.1.1 interface vlan 200
PING 224.1.1.1 (224.1.1.1): 56 data bytes
64 bytes from 192.168.200.73: icmp_seq=0 ttl=254 time=1.566 ms
64 bytes from 192.168.200.73: icmp_seq=1 ttl=254 time=1.02 ms
64 bytes from 192.168.200.73: icmp_seq=2 ttl=254 time=1.318 ms
64 bytes from 192.168.200.73: icmp_seq=3 ttl=254 time=1.042 ms
64 bytes from 192.168.200.73: icmp_seq=4 ttl=254 time=1.139 ms

--- 224.1.1.1 ping multicast statistics --5 packets transmitted, From member 192.168.200.73: 5 packets received, 0.00% packet loss
--- in total, 1 group member responded --N7K3#show ip mroute 232.72.72.0
IP Multicast Routing Table for VRF "default"
(10.72.74.72/32, 232.72.72.0/32)
, uptime: 00:02:44, igmp ip pim

Incoming interface: Vlan1050

, RPF nbr: 10.50.73.1 Outgoing interface list

: (count: 1) Ethernet1/10
, uptime: 00:02:44, igmp

2. Data Center Storage Networking

2.1 Fibre Channel Initialization
Configuration
N5K1:
feature fcoe
feature npiv
feature fport-channel-trunk
!
slot 1
port 28-32 type fc
!
interface fc1/28
channel-group 101
no shutdown
!
interface fc1/29
channel-group 101
no shutdown
!
interface fc1/30
switchport mode F
switchport trunk mode off
channel-group 103
no shutdown
!
interface fc1/31
switchport mode F
switchport trunk mode off
channel-group 103
no shutdown
!
interface san-port-channel 101
channel mode active
!
interface san-port-channel 103
channel mode active
switchport mode F
switchport trunk mode off
N5K2:

feature fcoe
feature npiv
feature fport-channel-trunk
!
slot 1
port 28-32 type fc
!
interface fc1/28
channel-group 102
no shutdown
!
interface fc1/29
channel-group 102
no shutdown
!
interface fc1/30
switchport mode F
switchport trunk mode off
channel-group 104
no shutdown
!
interface fc1/31
switchport mode F
switchport trunk mode off
channel-group 104
no shutdown
!
interface san-port-channel 102
channel mode active
!
interface san-port-channel 104
channel mode active
switchport mode F
switchport trunk mode off
MDS1:
interface fc1/3
channel-group 101
no shutdown
!
interface fc1/4
channel-group 101
no shutdown
!
interface port-channel 101
channel mode active
MDS2:

interface fc1/3
channel-group 102
no shutdown
!
interface fc1/4
channel-group 102
no shutdown
!
interface port-channel 102
channel mode active

In UCSM, go to the Equipment tab, and then, under the Fabric Interconnects, go to
Configure Unified Ports. Just like on the 5Ks, changing the port type from Ethernet
to Fibre Channel requires a reboot, so to save time, start with FI-B first, and then
configure FI-A.

When the FIs have rebooted, go to the SAN tab and configure FC uplinks on FI-A
and FI-B as SAN-Port-Channels 103 and 104, respectively. Remember to enable
the port channels when created, because like on the 5Ks, they are in the shutdown
state when created.

Verification
Changing Unified Port types between Ethernet and Fibre Channel requires a reload
of the Nexus 5000 or the UCS Fabric Interconnect on which the change was made.
N5K2#config t
Enter configuration commands, one per line.

End with CNTL/Z. N5K2(config)#feature fcoe

FC license checked out successfully

fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
Enabled FCoE QoS policies successfully N5K2(config)#feature npiv
N5K2(config)# ! N5K2(config)#slot 1
N5K2(config-slot)# port 28-32 type fc
N5K2(config-slot)#end
N5K2#copy running-config startup-config

[########################################] 100%
Copy complete, now saving to disk (please wait)...
N5K2# reload
WARNING: This command will reboot the system Do you want to continue? (y/n) [n]y

Shutdown Ports..
writing reset reason 9,

When the SAN port channels are configured, you may need to flap the links for the
port channels to come up, as shown below.
N5K2#show san-port-channel database
san-port-channel 102
Last membership update is successful
2 ports in total, 2 ports up

First operational port is fc1/28

Age of the port-channel is 0d:00h:10m:14s Ports:
fc1/29

fc1/28

[up] *

fc1/30

[down]

[up]

san-port-channel 104
Last membership update is successful
2 ports in total, 0 ports up
Age of the port-channel is 0d:00h:10m:14s Ports:
fc1/31

[down]

N5K2#conf t
Enter configuration commands, one per line.

End with CNTL/Z. N5K2(config)#int san-port-channel 104

N5K2(config-if)#shut
N5K2 %$ VDC-1 %$ %PORT-5-IF_DOWN_ADMIN_DOWN: %$VSAN 1%$ Interface san-port-channel 104 is down
(Administratively down)

N5K2(config-if)#no shut

N5K2(config-if)#end

N5K2 %$ VDC-1 %$ %PORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: %$VSAN 1%$ Interface san-port-channel 104 is down (No op
N5K2 %$ VDC-1 %$ Apr

6 20:48:00 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x

N5K2 %$ VDC-1 %$ Apr

6 20:48:00 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x

N5K2 %$ VDC-1 %$ %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on console0

N5K2 %$ VDC-1 %$ Apr

6 20:48:10 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x

N5K2 %$ VDC-1 %$ Apr

6 20:48:20 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x

N5K2 %$ VDC-1 %$ %PORT-5-IF_UP: %$VSAN 1%$ Interface san-port-channel 104 is up in mode F

N5K2 %$ VDC-1 %$ Apr

6 20:48:30 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x

N5K2#show san-port-channel database

san-port-channel 102
Last membership update is successful
2 ports in total, 2 ports up
First operational port is fc1/28
Age of the port-channel is 0d:00h:11m:15s Ports:
fc1/29

fc1/28

[up] *

fc1/30

[up]

san-port-channel 104
Last membership update is successful
2 ports in total, 2 ports up
First operational port is fc1/31
Age of the port-channel is 0d:00h:11m:15s Ports:
fc1/31

[up] *

On the UCS side, the SAN port channels are configured in Proxy Node Port (NP)
mode, for Node Port Virtualizer (NPV), or in other words, Fibre Channel End Host
Mode.

UCS-FI-A#connect nxos
UCS-FI-A(nxos)#show run interface fc1/31 - 32

interface fc1/31 switchport mode NP

channel-group 103 force
no shutdown

interface fc1/32 switchport mode NP

channel-group 103 force
no shutdown
UCS-FI-A(nxos)#show run interface san-port-channel 103

interface san-port-channel 103

channel mode active switchport mode NP
UCS-FI-A(nxos)#show san-port-channel database
san-port-channel 103
Last membership update is successful
2 ports in total, 2 ports up
First operational port is fc1/31
Age of the port-channel is 0d:00h:11m:44s Ports:
fc1/32

[up]

2.2 VSANs & Trunking

Configuration
N5K1:
vsan database
vsan 103
vsan 103 interface san-port-channel 103
!
interface san-port-channel 101
switchport trunk allowed vsan 103
N5K2:
vsan database
vsan 104
vsan 104 interface san-port-channel 104
!
interface san-port-channel 102
switchport trunk allowed vsan 104
MDS1:
vsan database
vsan 103

fc1/31

[up] *

vsan 103 interface fc1/7

!
interface fc1/7
no shutdown
!
interface port-channel 101
switchport trunk allowed vsan 103
MDS2:
vsan database
vsan 104
vsan 104 interface fc1/7
!
interface fc1/7
no shutdown
!
interface port-channel 102
switchport trunk allowed vsan 104
UCS-FI-A:
UCS-FI-A#connect nxos
UCS-FI-A(nxos)#show run | section "vsan database"
vsan database
vsan 103

UCS-FI-A(nxos)#show run | section "vlan 1104"

vlan 1103
fcoe vsan 103
name fcoe-vsan-1103

UCS-FI-B:
UCS-FI-B#connect nxos
UCS-FI-B(nxos)#show run | section "vsan database"
vsan database
vsan 104

UCS-FI-B(nxos)#show run | section "vlan 1104"

vlan 1104
fcoe vsan 104
name fcoe-vsan-1104

In UCSM, browse to the SAN tab, and then, under SAN Cloud, right-click VSANs to
create new VSANs. Ensure that VSAN 103 is on the Fabric A side and VSAN 104 is
on the Fabric B side.

To assign the VSANs to the SAN-Port-Channels, go back to the SAN tab, and under
SAN Cloud, right-click the appropriate FC interface and click Show Navigator. SANPort-Channel 103 should be in VSAN 103, and Port-Channel 104 should be in
VSAN 104.

Verification
When all the VSANs are created and assigned, check N5K1 and N5K2 to ensure

that the UCS FIs have performed a Fabric Login (FLOGI) on the SAN port channel
interfaces.
N5K1#show flogi database vsan 103
-------------------------------------------------------------------------------INTERFACE

VSAN

FCID

PORT NAME

NODE NAME

-------------------------------------------------------------------------------San-po103

103

0xbc0000

24:67:00:2a:6a:15:66:80 20:67:00:2a:6a:15:66:81

Total number of flogi = 1.

N5K2#show flogi database vsan 104
-------------------------------------------------------------------------------INTERFACE

VSAN

FCID

PORT NAME

NODE NAME

-------------------------------------------------------------------------------San-po104

104

0x6e0000

24:68:00:2a:6a:15:05:00 20:68:00:2a:6a:15:05:01

Total number of flogi = 1.

On MDS1 and MDS2, ensure that the Fibre Channel SAN has performed FLOGI.
MDS1#show flogi database vsan 103
-------------------------------------------------------------------------------INTERFACE

VSAN

FCID

PORT NAME

NODE NAME

-------------------------------------------------------------------------------fc1/7

103

0x0d0000

21:00:00:1b:32:04:5e:dc 20:00:00:1b:32:04:5e:dc

Total number of flogi = 1.

MDS2#show flogi database vsan 104
-------------------------------------------------------------------------------INTERFACE

VSAN

FCID

PORT NAME

NODE NAME

-------------------------------------------------------------------------------fc1/7

104

0xaa0000

21:01:00:1b:32:24:5e:dc 20:01:00:1b:32:24:5e:dc

Total number of flogi = 1.

Both N5K1 and MDS1 on the SAN A side and N5K2 and MDS2 on the SAN B side
should agree on the Fibre Channel Name Service (FCNS) database. This verifies
that both the initiators and targets are logged in and have been assigned Fibre
Channel Identifiers (FCIDs) and that VSAN trunking in the fabric is end to end.

N5K1#show fcns database vsan 103

VSAN 103:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

-------------------------------------------------------------------------0x0d0000

21:00:00:1b:32:04:5e:dc (Qlogic)

0xbc0000

24:67:00:2a:6a:15:66:80 (Cisco)

npv

Total number of entries = 2

N5K2#show fcns database vsan 104

VSAN 104:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

-------------------------------------------------------------------------0x6e0000

24:68:00:2a:6a:15:05:00 (Cisco)

0xaa0000

21:01:00:1b:32:24:5e:dc (Qlogic)

npv

Total number of entries = 2

zoneset name VSAN_103_ZONESET vsan 103

member VSAN_103_ZONE
!
zoneset activate name VSAN_103_ZONESET vsan 103
zone commit vsan 103
N5K2:

device-alias mode enhanced

device-alias database
device-alias name FC-SAN-B pwwn 21:01:00:1b:32:24:5e:dc
device-alias name BLADE1-SAN-B pwwn 20:00:00:cc:1e:dc:01:0b
device-alias name BLADE2-SAN-B pwwn 20:00:00:cc:1e:dc:02:0b
!
device-alias commit
!
zone mode enhanced vsan 104
!
zone name VSAN_104_ZONE vsan 104
member device-alias FC-SAN-B
member device-alias BLADE1-SAN-B
member device-alias BLADE2-SAN-B
!
zoneset name VSAN_104_ZONESET vsan 104
member VSAN_104_ZONE
!
zoneset activate name VSAN_104_ZONESET vsan 104
zone commit vsan 104

Verification
Devices on the SAN A side should agree on the Device Alias database and zoneset
for VSAN 103.
MDS1#show device-alias status
Fabric Distribution: Enabled
Database:- Device Aliases 3 Mode: Enhanced
Checksum: 0x252e3d5059933b2826cabfe0ee148
MDS1#show device-alias database
device-alias name FC-SAN-A pwwn 21:00:00:1b:32:04:5e:dc
device-alias name BLADE1-SAN-A pwwn 20:00:00:cc:1e:dc:01:0a
device-alias name BLADE2-SAN-A pwwn 20:00:00:cc:1e:dc:02:0a

Total number of entries = 3

MDS1#show zone status vsan 103

VSAN: 103 default-zone: deny distribute: active only Interop: default mode: enhanced
merge-control: allow
session: none
hard-zoning: enabled broadcast: enabled
Default zone:
qos: none broadcast: disabled ronly: disabled
Full Zoning Database :
DB size: 224 bytes
Zonesets:1

Zones:1 Aliases: 0 Attribute-groups: 1 Active Zoning Database

:
DB size: 148 bytes Name: VSAN_103_ZONESET

Zonesets:1

Zones:1

Status: Activation completed at 20:55:21 UTC May 26 2013

MDS1 learned the zoning configuration applied on N5K1, but it does not yet see an
FCID for the UCS blades. This is because we haven't configured the service profiles
for the blades, which means they're not yet logged in to the fabric. When the SP
association is complete, we should see the FCIDs of the blades get dynamically
assigned, as well as the pWWNs we manually configure on them logged in to the
fabric.
MDS1#show zoneset active vsan 103
zoneset name VSAN_103_ZONESET vsan 103
zone name VSAN_103_ZONE vsan 103
* fcid 0x0d0000 [device-alias FC-SAN-A]
device-alias BLADE1-SAN-A
device-alias BLADE2-SAN-A

2.4 iSCSI Virtual Target

Configuration
N7K3:
interface Ethernet2/23
switchport mode trunk
switchport trunk allowed vlan 202
no shutdown
MDS1:

device-alias database
device-alias name UCS-C200-SAN-A pwwn 20:00:00:cc:1e:dc:03:0a
!

device-alias commit
!
feature iscsi
iscsi enable module 1
!
vsan database
vsan 103 interface iscsi1/1
!
iscsi virtual-target name iqn.1987-05.com.cisco:05.mds1.01-01.01234567890abcde
pWWN 21:00:00:1b:32:04:5e:dc
initiator ip address 192.168.202.104 permit
!
iscsi initiator ip-address 192.168.202.104
static pWWN 20:00:00:cc:1e:dc:03:0a
!
zone name VSAN_103_ZONE vsan 103
member device-alias UCS-C200-SAN-A
!
zoneset activate name VSAN_103_ZONESET vsan 103
zone commit vsan 103
!
interface GigabitEthernet1/1
ip address 192.168.202.61 255.255.255.0
no shutdown
!
interface iscsi1/1
no shutdown

Verification
When the iSCSI configuration is complete, MDS1 should see the UCS C200 server
log in as an iSCSI Initiator. The nWWN can be dynamic, but because zoning and
LUN Masking on the SAN is done based on the pWWN, this needs to be manually
assigned to the iSCSI Initiator.
MDS1#show iscsi initiator
iSCSI Node name is 192.168.202.104
iSCSI Initiator name: iqn.1998-01.com.vmware:localhost-7463f71b
iSCSI alias name:
Configured node (iSCSI)
Node WWN is 21:01:00:0d:ec:4a:21:02 (dynamic)
Member of vsans: 103
Number of Virtual n_ports: 1 Virtual Port WWN is 20:00:00:cc:1e:dc:03:0a (configured)

Interface iSCSI 1/1, Portal group tag: 0x3000

VSAN ID 103, FCID 0x0d0100

From the iSCSI Initiator's point of view, the MDS is an iSCSI Target. Note that only
the C200's IP address is allowed to use this target.
MDS1#show iscsi virtual-target
target: iqn.1987-05.com.cisco:05.mds1.01-01.01234567890abcde

* Port WWN 21:00:00:1b:32:04:5e:dc

Configured node (iSCSI)
No. of initiators permitted: 1

initiator 192.168.202.104/32 is permitted

All initiator permit is disabled

Trespass support is

disabled

Revert to primary support is

disabled

MDS1 should see the C200 server registered to the fabric in the FLOGI database.
MDS1#show flogi database

-------------------------------------------------------------------------------INTERFACE

VSAN

FCID

PORT NAME

NODE NAME

-------------------------------------------------------------------------------fc1/7

103

0x0d0000

21:00:00:1b:32:04:5e:dc 20:00:00:1b:32:04:5e:dc

[FC-SAN-A] iscsi1/1

103

0x0d0100

20:00:00:cc:1e:dc:03:0a

21:01:00:0d:ec:4a:21:02 [UCS-C200-SAN-A]

Total number of flogi = 2.

Adding the C200's pWWN to the already defined zone for VSAN 103 will allow it
access to the LUNs that the SAN is presenting for this initiator.
MDS1#show zoneset active
zoneset name VSAN_103_ZONESET vsan 103
zone name VSAN_103_ZONE vsan 103
* fcid 0x0d0000 [device-alias FC-SAN-A]
device-alias BLADE1-SAN-A
device-alias BLADE2-SAN-A * fcid 0x0d0100 [device-alias UCS-C200-SAN-A]

The final verification for this task is to ensure that the ESXi instance has actually

mounted the iSCSI LUNs. To check this, go to the vSphere client, select the C200
host on the left, click the Configuration tab, and then click Storage Adapters.
Under the iSCSI Software Adapter, you should see the LUNs appear as shown
below.

3. Unified Computing
3.1 Address Pools
UUID Pools in UCSM are configured under the Servers tab, Pools, then UUID
Suffix Pools, as shown below.

MAC Address Pools are under the LAN tab, Pools, then MAC Pools.

Node World Wide Name Pools are under the SAN tab, Pools, then WWNN Pools.

Management IP Address Pools are under the Admin tab, Communication

Management, then Management IP Pool. Note that the default gateway here is
arbitrary, because the task did not ask for a specific value, but it is still a required
field.

3.2 UCS Service Profile Templates

Create a new Service Profile Template under the Servers tab, then Service Profile
Templates. The task requires that this be an Initial Template and get its addresses
from the default pools that were created in the previous task.

Under Storage, ensure that the vHBAs are assigned to VSANs 103 and 104 on
Fabric A and Fabric B, respectively.

For vNICs, use the Expert option, and add the five new vNICs according to the task
requirements. The VLANs needed are created in this step to save time, but could
also be configured as a separate step under the LAN Cloud.

Ensure that the vMotion vNIC has Fabric Failover enabled according to the task
requirements.

The vNICs for the VMGuests are trunks that carry the rest of the VLANs.

The Maintenance Policy is where we define that the administrator must

acknowledge a change that would cause the blade to reboot.

The Operational Policies define where the Management IP addresses of the Service
Profiles come from.

3.3 Service Profiles

To assign the service profiles, we must first enable the southbound links from the

FIs to the Blade Chassis. To do so, configure them as Server ports under the
Fabric Interconnects on the Equipment tab.

Create two copies of the Service Profile Template previously created.

Before we customize the boot options for the individual service profiles, a QoS
policy is created that will apply to the vHBAs. Note that this is just for clarity of the
configuration, so that we know for certain that the vHBAs are being assigned to a nodrop QoS policy.

Modify the vHBAs to have the appropriate pWWNs according to the task. Note that
if these values are incorrect, the blades will fail to boot from the SAN, because the
LUN masking on the SAN only allows specific initiating pWWNs to access their
LUNs.

We need to create a Boot Policy that tells the blade which SAN target it needs to
boot to.

Again, ensure 100% accuracy, because an incorrect pWWN value will cause the
blade to be unable to boot.

Repeat the above steps, but now for the backup boot target.

Don't forget to actually assign the Boot Policy to the service profile after it is
successfully created.

Repeat the above steps for the second service profile that will be assigned to blade
2.

Finally, associate the service profiles to the blades.

When the blades begin to boot, you can track their progress by connecting to their
KVMs. When the blades are fully booted, you should see the console screen for the
ESXi instances, as shown below.

4. Data Center Virtualization

4.1 Nexus 1000v

Configuration
First we need to determine which UUIDs were dynamically assigned to the blades,
and which VEMs they are currently inserted as. The below output shows us the
module number (VEM number), the UUID, and the IP address.
N1Kv#show module
Mod

Ports

Module-Type

Model

Status

---

-----

--------------------------------

------------------

------------

Virtual Supervisor Module

Nexus1000V

active *

Virtual Supervisor Module

Nexus1000V

ha-standby

248

Virtual Ethernet Module

248

Virtual Ethernet Module

248

Virtual Ethernet Module

Mod

---

------------------

------------------------------------------------

4.2(1)SV2(1.1)

0.0

4.2(1)SV2(1.1)

0.0

4.2(1)SV2(1.1)

VMware ESXi 5.1.0 Releasebuild-799733 (3.1)

4.2(1)SV2(1.1)

VMware ESXi 5.1.0 Releasebuild-799733 (3.1)

4.2(1)SV2(1.1)

VMware ESXi 5.1.0 Releasebuild-799733 (3.1)

Mod

MAC-Address(es)

Serial-Num

---

--------------------------------------

----------

00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8

02-00-0c-00-04-00 to 02-00-0c-00-04-80

02-00-0c-00-05-00 to 02-00-0c-00-05-80

02-00-0c-00-06-00 to 02-00-0c-00-06-80

Mod

Server-IP

Server-UUID

Server-Name

---

---------------

------------------------------------

--------------------

192.168.200.200

192.168.200.102

6220349f-9cc4-e211-0000-00000000007f

NA 5

192.168.200.104

d57068d1-dfa4-11e1-a49d-30f70d088146

NA 6

192.168.200.101

6220349f-9cc4-e211-0000-00000000006f

* this terminal session

The VEM number is part of the running config, as shown below.

N1Kv#sh run | b vem

vem 3
host vmware id 2e2baff2-03bd-e211-0000-00000000007f
vem 4
host vmware id 6220349f-9cc4-e211-0000-00000000007f
vem 5
host vmware id d57068d1-dfa4-11e1-a49d-30f70d088146
vem 6
host vmware id 6220349f-9cc4-e211-0000-00000000006f
vem 7
host vmware id 2e2baff2-03bd-e211-0000-00000000006f

Power cycle blade 2, which will cause its VEM to be removed. Then we can delete
its current VEM number and move it to module 20.

N1Kv#
N1Kv %PLATFORM-2-MOD_REMOVE: Module 2 removed (Serial number T5056BB0E4A)
N1Kv %VEM_MGR-2-VEM_MGR_REMOVE_NO_HB: Removing VEM 4 (heartbeats lost) N1Kv %VEM_MGR-2-MOD_OFFLINE:
Module 4 is offline
N1Kv#config t
Enter configuration commands, one per line.

End with CNTL/Z. N1Kv(config)#no vem 4

N1Kv(config)#vem 20
N1Kv(config-vem-slot)#host vmware id 6220349f-9cc4-e211-0000-00000000007f
N1Kv(config-vem-slot)#end
N1Kv#copy run start

[########################################] 100%

N1Kv#
N1Kv %VEM_MGR-2-VEM_MGR_DETECTED: Host 192.168.200.102 detected as module 20
N1Kv %VEM_MGR-2-MOD_ONLINE: Module 20 is online

N1Kv %PLATFORM-2-MOD_DETECT: Module 2 detected (Serial number :unavailable) Module-Type Virtual Supervisor Module Mo

Finally, power cycle blade 1, which will cause its VEM to be removed. Because the
primary VSM also runs on this blade, you will have to disconnect and reconnect
your terminal session. Before blade 1 fully reboots, delete its current VEM number
and move it to module 10.
N1Kv#conf t
Enter configuration commands, one per line.

End with CNTL/Z. N1Kv(config)#no vem 6

N1Kv(config)#vem 10
N1Kv(config-vem-slot)#host vmware id 6220349f-9cc4-e211-0000-00000000006f
N1Kv(config-vem-slot)#end
N1Kv %VEM_MGR-2-VEM_MGR_DETECTED: Host 192.168.200.101 detected as module 10
N1Kv %VIM-5-IF_ATTACHED: Interface Vethernet1 is attached to vmk0 on port 1 of module 10 with dvport id 1216
N1Kv %VEM_MGR-2-MOD_ONLINE: Module 10 is online
N1Kv#show module
Mod

Ports

Module-Type

Model

Status

---

-----

--------------------------------

------------------

------------

Virtual Supervisor Module

Nexus1000V

ha-standby

Virtual Supervisor Module

Nexus1000V

active *

248

Virtual Ethernet Module

248

Virtual Ethernet Module

248

Virtual Ethernet Module

Mod

---

------------------

------------------------------------------------

4.2(1)SV2(1.1)

0.0

4.2(1)SV2(1.1)

0.0

4.2(1)SV2(1.1)

VMware ESXi 5.1.0 Releasebuild-799733 (3.1)

4.2(1)SV2(1.1)

VMware ESXi 5.1.0 Releasebuild-799733 (3.1)

4.2(1)SV2(1.1)

VMware ESXi 5.1.0 Releasebuild-799733 (3.1)

Mod

MAC-Address(es)

Serial-Num

---

--------------------------------------

----------

00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8

02-00-0c-00-05-00 to 02-00-0c-00-05-80

02-00-0c-00-0a-00 to 02-00-0c-00-0a-80

02-00-0c-00-14-00 to 02-00-0c-00-14-80

Mod

Server-IP

Server-UUID

Server-Name

---

---------------

------------------------------------

--------------------

192.168.200.200

192.168.200.104

d57068d1-dfa4-11e1-a49d-30f70d088146

192.168.200.104

192.168.200.101

6220349f-9cc4-e211-0000-00000000006f

192.168.200.101

192.168.200.102

6220349f-9cc4-e211-0000-00000000007f

192.168.200.102

4.2 Private VLANs

N7K3:
feature private-vlan
!
vlan 203
private-vlan isolated
vlan 204
private-vlan primary
private-vlan association 203
!
interface Vlan204
no shutdown
private-vlan mapping 203
ip address 192.168.255.73/24
N1Kv:

vlan 203
private-vlan isolated
vlan 204
private-vlan primary
private-vlan association 203
!
port-profile type ethernet VMGuests_UPLINK
switchport mode private-vlan trunk promiscuous
switchport private-vlan mapping trunk 204 203
!
port-profile type vethernet VLAN203
switchport mode private-vlan host
switchport private-vlan host-association 204 203

CCIE Data Center Full-Scale Labs - Bootcamp

Members - CCIE Data Center Full-Scale Lab 2
CCIE DC Full-Scale Lab 2 Tasks
Introduction
1. Data Center Infrastructure
2. Data Center Storage Networking
3. Unified Computing
4. Data Center Virtualization

Introduction
General Lab Guidelines
You may not use any links that may physically be present but not specifically pictured
and labeled in this topology.
Name and number all VLANs, port channels, SAN port channels, service profiles,
templates, and so on exactly as described in this lab. Failure to do so will result in
missed points for that task.
You may not change any passwords on any devices unless explicitly directed to do
so.
You may not change any management IP addresses or default routes on any
devices or VDCs unless explicitly directed to do so (you may add them if they do not
exist, but you may not change existing).
You may not disable telnet on any device. Telnet must work properly on all devices
and VDCs.
You may not log on to the 3750G switch for this particular lab. It is fully functional and
pre-configured for you.

1. Data Center Infrastructure

1.1 VLANs
Do not create any unnecessary VLANs on any switch.
Create VLANs 120, 125, 130, 135, 140, 200, 201, 710, and 711 on N7K1.
Create VLANs 120, 125, 130, 135, 140, 200, 201, 720, and 721 on N7K2.
Create VLANs 120, 125, 130, 135, 200, 201, and 140 on N7K3.
Create VLANs 120, 125, 130, 135, 200, 201 and 140 on N7K4
Create VLANs 120, 125, 130, 135, 200, and 201 on N5K1 and N5K2.
Name VLANs on every device they appear on according to Table 1.

Table 1

VLAN

Name

120

VM-DATA1

125

VM-DATA2

130

VM-DATA3

135

VM-DATA4

140

OTV-SITE

150

BACKUP

200

DCI-ESXI

201

DCI-VMOTION

710

DC1-ISP-1

VLAN

Name

711

DC1-ISP-2

720

DC2-ISP-1

721

DC2-ISP-2

1.2 DCI L3 Routing

Configure an L3 link over N7K1 e2/29 with the IP address and subnet mask of
10.71.71.0 255.255.255.254.
Use VLAN 710 to accomplish this.
This L3 link must belong to VRF "DC1".
Configure the L3 link to form an OSPF adjacency in area 0.0.0.5.
Use a router id of 10.71.71.71 for the OSPF process.
The OSPF process should be named "DC1".
Ensure that e2/29 will only ever run at a rate of 1Gbps.
Configure an L3 link over N7K1 e2/31 with the IP address and subnet mask of
10.71.71.2 255.255.255.254.
Use VLAN 711 to accomplish this.
This L3 link must belong to VRF "DC1".
Configure the L3 link to form an OSPF adjacency in area 0.0.0.5.
Use a router id of 10.71.71.71 for the OSPF process.
The OSPF process should be named "DC1".
Ensure that e2/29 will only ever run at a rate of 1Gbps.
Configure an L3 link over N7K2 e2/21 with the IP address and subnet mask of
10.72.72.0 255.255.255.254.
Use VLAN 720 to accomplish this.
This L3 link must belong to VRF "DC2".
Configure the L3 link to form an OSPF adjacency in area 0.0.0.3.
Use a router id of 10.72.72.72 for the OSPF process.
The OSPF process should be named "DC2".
Ensure that e2/29 will only ever run at a rate of 1Gbps.
Configure an L3 link over N7K2 e2/23 with the IP address and subnet mask of
10.72.72.2 255.255.255.254.
Use VLAN 721 to accomplish this.

This L3 link must belong to VRF "DC2".

Configure the L3 link to form an OSPF adjacency in area 0.0.0.3.
Use a router id of 10.72.72.72 for the OSPF process.
The OSPF process should be named "DC2".
Ensure that e2/29 will only ever run at a rate of 1Gbps.
These four ports should all immediately go into a forwarding state when brought up
and should go into an errDisabled state if they receive any STP BPDUs.
Do not modify any configuration on the 3750G switch for this or any other task in this
lab.
Ensure OSPF converges with whatever means necessary

1.3 L2 Trunking and L3 Routed Interfaces

Configure trunking between N7K1 e2/1 and N7K3 e2/9.
Allow only previously created VLANs 120-140 and 200-201 over this link.
Ensure that N7K1 is the root for all STP instances.
Configure an L3 routed interface between N7K1 e1/1 using the IP address
10.13.13.0/31 and N7K3 e1/9 using the IP address 10.13.13.1/31.
Ensure that this L3 link can participate in the OSPF process and route over
the DCI.
Configure trunking between N7K2 e2/11 and N7K4 e2/20.
Only allow only previously created VLANs 120-140 and 200-201 over this
link.
Configure an L3 routed interface between N7K2 e1/17 using the IP address
10.24.24.0/31 and N7K4 e1/25 using the IP address 10.24.24.1/31.
Ensure that this L3 link can participate in the OSPF process and route over
the DCI.

1.4 Port Channels

Assuming that more links will be added later, with the desire for minimal traffic
disruption, configure the following:
Configure trunking on port channel 215 from N7K1 to UCS FI-A, and ensure
that the same port channel number is used later from the UCS side.
Configure trunking on port channel 218 from N7K1 to UCS FI-B, and ensure
that the same port channel number is used later from the UCS side.
Ensure that both of these port channels transition immediately to a state of

forwarding traffic.
Ensure that the N7K1 is the primary device in LACP negotiation.
Ensure that the hashing algorithm takes L3 and L4 for both source and
destination into account.
Trunk only previously created VLANs 120-135 and 200-201 southbound
from N7K1 to both FIs.

1.5 HSRP
Using information from Table 2, configure SVIs on N7K1 and N7K2 for all VLANs that
are present on that switch.
Assume that a second Nexus 7000 will be added to each Data Center, and with that
in mind, go ahead and provision HSRP for all SVIs at both sites, as follows:
Use the newest version of HSRP supported.
Make HSRP group numbers correspond with their respective VLAN/SVI
numbers.
Use the virtual IP address of .254 for SVIs on both switches.
Use the host IP address of .251 for each current SVIs on N7K1.
(.250 will be used in the future for the other HSRP member at DC1).
Use the host IP address of .252 for each current SVIs on N7K2.
(.253 will be used in the future for the other HSRP member at DC1).
These current SVIs will be the primary HSRP group member even after the
other N7K is put into service at each DC; ensure that these SVIs have a
higher preference for being the Active forwarder assuming the others come
online with defaults.
Have the SVIs for VLAN 200 use the fastest possible hello and hold timers.

Table 2

VLAN

IP Subnet / Mask

VRF

120

192.168.120.0 255.255.255.0

default

125

192.168.125.0 255.255.255.0

default

VLAN

IP Subnet / Mask

VRF

130

192.168.130.0 255.255.255.0

default

135

192.168.135.0 255.255.255.0

default

200

192.168.200.0 255.255.255.0

default

201

192.168.201.0 255.255.255.0

default

1.6 vPC
Configure vPC between N5K1 and N5K2 with the Domain ID 12.
Configure the peer-link with an LACP trunking over ports e1/1-2 on Port Channel 512
between N5K1 and N5K2 according to the diagram.
Ensure that any vPC numbers correspond with their designated port channel
numbers, as listed in the tasks that follow.
You are not permitted to create any additional links that are not explicitly pictured in
the diagram.
Ensure that N5K1 is the root for all STP instances; however, you may not configure
any spanning tree priority or root commands globally or at the interface level on
N5K1.
Ensure that N5K1 holds the primary role for the vPC domain.
Ensure that N5K1 always decides which links are active in any port channel.
Synchronize all ARP tables.
Ensure that if our SAN was an EMC VPLEX or VMAX using IP technologies, vPC
would not cause any problems with forwarding frames.

1.7 Port Channels, FEX, and vPC

Configure trunking on Port Channel 100 from N7K2 to N5K1 and N5K2 according to
the diagram, and ensure that the pair of N5Ks are the only ones initiating any port
channel protocol negotiation.
Configure FEX 113 using trunking on Port Channel 113 from N5K1 and N5K2
according to the diagram.
Configure FEX 123 using trunking on Port Channel 123 from N5K1 and N5K2

according to the diagram.

1.8 Mgmt VM Access

Configure a 1Gbps access link in VLAN 200 to the Management VM on N5K1 e1/11.
Ensure that traffic forwards immediately and goes into an errDisable state if it
receives any STP packets.

1.9 Access Trunking

Configure trunking on both ports individually coming from SVR1 up to N5K1 e113/1/1
and N5K2 e123/1/1 according to the diagram.
For now, trunk only previously created VLANs 120-135 and 200-201 (there may be
additional VLANs needed later).

1.10 OTV
Extend only previously created VLANs 120-135 and 200-201 between Data Centers
using OTV.
Use the OTV site VLAN of 140 on both sides of the DCI.
You may use whatever site identifiers you prefer.
The ISP supports SSM and ASM, and for ASM it provides a PIM RP of 10.10.10.25;
use this as your only RP.
OTV should be authenticated using a hashed value from the word "DCIOTV".
Any of the SVIs on N7K1 or N7K2 for the VLANs that are extended across the DCI
should be able to ping each other.
Prevent HSRP groups at DC1 from becoming active/standby members of the same
HSRP group numbers at DC2, and vice-versa.
Prevent any device ARPing at either DC from getting the virtual MAC address of the
HSRP group from the 7K at the opposite side of the DCI.
When finished, both N7K1 and N7K2 should be able to ping the actual host IP
address of the SVI at the opposite data center traversing the overlay.
Each N7K1 and N7K2 should also be able to ping the virtual IP address of .254,
which should keep traffic local to the site from which the ping originates.

2. Data Center Storage Networking

2.1 VSANs and FCoE VLANs

Create VSAN 10 on MDS1, MDS2, N5K1, and N5K2.
Create VSAN 20 only on MDS1, MDS2, and N5K2.
Create VLAN 10 to carry FCoE traffic for VSAN 10 on N5K1 and N5K2.
Create VLAN 20 to carry FCoE traffic for 20 respectively only on N5K2.

2.2 UCS SAN Connectivity

Configure FC links on MDS1 as pictured in the diagram ready for both UCS FIs.
Do not use any port channeling or trunking.
Configure links coming from FI-A to MDS1 to use VSAN 10.
Configure links coming from FI-B to MDS1 to use VSAN 20.

2.3 E Port Trunking

N5K1 should be configured as an E trunk to N5K2 and should trunk only VSAN 10
over SAN Port Channel 256 using interfaces fc1/26 and fc1/27.
Configure a trunk between N5K2 fc1/28 and MDS2 fc1/3 that trunks only VSANs 10
and 20.
N5K2 fc1/32 should provide connectivity to the SAN array for VSAN 10.
MDS2 fc1/7 should provide connectivity to the SAN array for VSAN 20.

2.4 Cisco C200 P81E (VIC) CNA FLOGIs

Configure FCoE for Svr1 so that it logs in to VSAN 10 over FEX 113.
Configure FCoE for Svr1 so that it logs in to VSAN 20 over FEX 123.
Svr1 is set up to FLOGI to both fabrics.

2.5 FCIP
Configure FCIP between MDS1 and MDS2 on interfaces G1/1 and G1/2 on each
switch.
Use the IP address of 12.12.12.1/30 on MDS1 G1/1 and 12.12.12.2/30 on MDS2
G1/1 over FCIP Profile 10 and interface FCIP 10 on both sides.
Use the IP address of 12.12.12.5/30 on MDS1 G1/2 and 12.12.12.6/30 on MDS2

G1/2 over FCIP Profile 20 and interface FCIP 20 on both sides.

The 3750G switch is already configured properly; do not connect to it at all.
Configure SAN Port Channel 50 over both of these links and trunk only VSAN 10 and
VSAN 20 over it.
Optimize FCIP on MDS1 and MDS2 to account for optimum TCP window scaling
based on the approximate actual RTT (within 20% variance is allowed).
Allow FCIP to monitor the congestion window and increase the burst size to the
maximum allowed.
Ensure that there is no fragmentation of FCIP packets over the link.

2.6 Zoning
Ensure that MDS1 appears to the fabric as domain 0x61 for VSAN 10 and 20.
Ensure that MDS2 appears to the fabric as domain 0x62 for VSAN 10 and 20.
Ensure that N5K2 appears to the fabric as domain 0x52 for VSAN 10 and 20.
Ensure that N5K1 appears to the fabric as domain 0x51 for VSAN 10 and 20.
Zone according to the following information.
You may only make zoning changes for both Fabric A and Fabric B from MDS1.
According to information given in Table 3:
Zone so that "ESXi1", "ESXi2", and "ESXi3" all have access to their FCTARGET-SAN-x for the appropriate Fabrics (fc0's to Fabric A; fc1's to Fabric
B).
Fabric A uses VSAN 10.
Fabric B uses VSAN 20.
Zoning for Fabric A should use the zone name "ZONE-A".
Zoning for Fabric B should use the zone name "ZONE-B".
The zoneset for Fabric A should be named "ZoneSet_VSAN10".
The zoneset for Fabric B should be named "ZoneSet_VSAN20".
Aliases must be created according to Table 3 and must be used in the zoning
configuration.
Many pWWN's are the same below. They are sorted first by FC-4
Type and then by Fabric.

Table 3

Fabric

pWWN

LUN

20:aa:00:25:b5:01:01:01

FC-4
Type

Description

Alias

N/A

ESXi1 vHBA
"fc0"

ESXi1A-fc0

Init

20:aa:00:25:b5:01:01:02

N/A

ESXi2 vHBA
"fc0"

ESXi2A-fc0

Init

20:00:d4:8c:b5:bd:46:0e

N/A

ESXi3 vHBA
"fc0"

ESXi3A-fc0

Init

20:bb:00:25:b5:01:01:01

N/A

ESXi1 vHBA
"fc1"

ESXi1B-fc1

Init

20:bb:00:25:b5:01:01:02

N/A

ESXi2 vHBA
"fc1"

ESXi2B-fc1

Init

20:00:d4:8c:b5:bd:46:0f

N/A

ESXi3 vHBA
"fc1"

ESXi3B-fc1

Init

ESXi1 Boot
Volume

FCTARGETSAN-A

Target

ESXi2 Boot
Volume

FCTARGETSAN-A

Target

FC_Datastore
1

FCTARGETSAN-A

Target

FC_Datastore
2

FCTARGETSAN-A

Target

21:03:00:1b:32:64:5e:dc

Fabric

pWWN

21:01:00:1b:32:24:5e:dc

Alias

FC-4
Type

ESXi1 Boot
Volume

FCTARGETSAN-B

Target

ESXi2 Boot
Volume

FCTARGETSAN-B

Target

FC_Datastore
1

FCTARGETSAN-B

Target

FC_Datastore
2

FCTARGETSAN-B

Target

LUN

Description

3. Unified Computing
3.1 UCS Initialization
Initialize both UCS Fabric Interconnects (FIs).
Fabric Interconnect A should use the IP address of 192.168.101.201/24.
Fabric Interconnect B should use the IP address of 192.168.101.202/24.
Both Fabric Interconnects should use a VIP of 192.168.101.200.

3.2 SAN Uplinks and VSANs

Disable all confirmation messages for creation and deletion of objects.
Configure individual FC uplinks as instructed earlier in the Storage Networking
section and according to the diagram.
Do not use any port channeling or trunking.
Create VSAN 10 and name it "VSAN10" and ensure that it uses VLAN 10 for FCoE.
Create VSAN 20 and name it "VSAN20" and ensure that it uses VLAN 20 for FCoE.
Configure links coming from FI-A to MDS1 to use VSAN 10.

Configure links coming from FI-B to MDS1 to use VSAN 20.

Disable any unused FC ports according to diagram.

3.3 LAN Uplinks and VLANs

Configure port channels for all links from FIs to IOM/FEXs in UCS chassis according
to the diagram.
Configure a port channel from each FI to N7K1 according to the diagram, and use
the same port channel number as previously instructed from the N7K side.
Create VLANs 120-135 and 200-201 and VLAN 150 from Table 1, with correct
names on both UCS FIs (only the ones in the table).
Only allow the BACKUP VLAN to traverse the 1Gbps ports designated in the
diagram toward the 3750G switch, and ensure that it is in an UP state.

3.4 Disk Policies

Create a hard disk policy named "MAXRAID" that specifies a method that both
mirrors and then stripes local disks.
Ensure that if any service policy ever uses this policy and tries to associate with a
blade whose hard drives are not already provisioned with this RAID method, the
association will fail.
Do not associate this policy with any service profiles.

3.5 Pools
Create a UUID pool called "Global-UUIDs" and allocate suffixes from the range of
0001-000000000101 to 0001-00000000010f.
Create a MAC address pool called "Global-MACs" ranging from 00:25:b5:0a:0a:01 to
00:25:b5:0a:0a:11.
Create an nWWN pool called "Global-nWWNs" ranging from 20:ff:00:25:b5:01:01:01
to 20:ff:00:25:b5:01:01:11.
Create a Management IP address pool ranging from 192.168.101.210 to
192.168.101.219 with the default gateway of 192.168.101.1.

3.6 Service Profiles

Configure a service profile named "ESXi1" with the following values.

Anything changed in this service profile template should never affect any service
profiles instantiated from it.
UUIDs should be dynamically allocated from the Global-UUIDs pool.
2 vHBAs should be created with the following information:
Name them "fc0" and "fc1".
"fc0" must be assigned the initiator pWWN of 20:aa:00:25:b5:01:01:01.
"fc1" must be assigned the initiator pWWN of 20:bb:00:25:b5:01:01:01.
Both vHBAs must be able to dynamically obtain nWWNs from the GlobalnWWNs pool.
Neither of these vHBAs should be allowed to re-attempt FLOGIs more than
3 times.
Configure a specific boot policy to boot from SAN with the following information:
"fc0" should attempt first to boot from Fabric A using the pWWN for "ESXi1
Boot Volume" in Table 3.
"fc1" should attempt first to boot from Fabric B using the pWWN for "ESXi1
Boot Volume" in Table 3.
5 vNICs should be created with the following information:
Name them "eth0", "eth1", "eth2", "eth3", and "eth4".
"eth0" and "eth3" should only be allowed to ever use Fabric A.
"eth1" and "eth4" should only be allowed to ever use Fabric B.
"eth2" primarily uses Fabric A, but should automatically use Fabric B if all
uplinks on FI-A are down.
MAC addresses should must be allocated dynamically from the GlobalMACs pool.
All VLANs should be allowed on all vNICs except for VLAN 1 and VLAN
150; these should not be allowed on any vNICs.
All hosts will explicitly tag their VLAN IDs.
Any changes to the service profile requiring a reboot should force the administrator to
manually allow it.
Any service profile created from this template should not automatically associate with
any blades in the chassis.
Only allow this service profile to ever associate with blades that have a Palo
mezzanine adapter.
Do not allow blade to automatically boot after this service profile is associated.
Ensure that when booting, the KVM console viewer can see the FC disk that
attaches directly after the FC drivers load.
Configure the management IP addresses to be dynamically assigned from the global
pool.

Manually associate this profile with blade 1 and boot the blade.

3.7 Cloning Service Profiles

Create a clone of the previous service profile and call it "ESXi2".
Change what is necessary for the vHBAs to be set up as follows:
"fc0" must be assigned the initiator pWWN of 20:aa:00:25:b5:01:01:02.
"fc1" must be assigned the initiator pWWN of 20:bb:00:25:b5:01:01:02.
Ensure that this service profile always uses links fc1/30 on Fabric A and
fc1/28 Fabric B for its SAN traffic.
Manually associate this profile with blade 2 and boot the blade.

3.8 Traffic Monitoring

Measure traffic in a policy called "Over_3Gbps" on vNIC "eth2" in Service Profile
"ESXi1", and raise an informational alert if the traffic received by the vNIC rises
above 3Gbps.
Do not change the collection interval for any device in the system.

4. Data Center Virtualization

4.1 VSM and VEM Connectivity
Ensure reachability to both VSMs that are running on both UCS blades.
Ensure that the VEMs running on both UCS blades insert into the Nexus1000v
chassis properly.
Ensure that service profile ESX1 shows up as VEM 4 and service profile ESX2
shows up as VEM 5.
Do not worry about the UCS C200 VEM for this lab.

4.2 N1Kv QoS

Ensure that all traffic coming from vNIC "eth2" on both blades is marked with CoS 4
only by the use of Nexus1000v, and that the UCS trusts that marking.
You are not permitted to attach any policy directly to that interface.

CCIE Data Center Full-Scale Labs - Bootcamp

Members - CCIE Data Center Full-Scale Lab 2
CCIE DC Full-Scale Lab 2 Solutions
Introduction
1. Data Center Infrastructure
2. Data Center Storage Networking
3. Unified Computing
4. Data Center Virtualization

Introduction
General Lab Guidelines
You may not use any links that may physically be present but are not specifically
pictured and labeled in this topology.
Name and number all VLANs, port channels, SAN port channels, service profiles,
templates, and so on exactly as described in this lab. Failure to do so will result in
missed points for that task.
You may not change any passwords on any devices unless explicity directed to do
so.
You may not change any management IP addresses or default routes on any
devices or VDCs unless explicitly directed to do so (you may add them if they do not
exist, but you may not change existing).
You may not disable telnet on any device. Telnet must work properly on all devices
and VDCs.
You may not log on to the 3750G switch for this particular lab. It is fully functional and
pre-configured for you.

1. Data Center Infrastructure

1.1 VLANs
Configuration
This task doesn't seem like it would be a difficult one, and it obviously isn't hard to
create VLANs. What can be difficult is getting them all in exactly as instructed. Other
than VLAN 1, if you entered additional VLANs beyond what was asked of you, or if
you simply mis-typed a name, you would have missed the points for this task.
N7K1:

vlan 120
name VM-DATA1
vlan 125
name VM-DATA2
vlan 130
name VM-DATA3
vlan 135
name VM-DATA4
vlan 140
name OTV-SITE
vlan 200
name DCI-ESXI
vlan 201
name DCI-VMOTION
vlan 710
name DC1-ISP-1
vlan 711
name DC1-ISP-2

N7K2:

vlan 120
name VM-DATA1
vlan 125
name VM-DATA2
vlan 130
name VM-DATA3
vlan 135
name VM-DATA4
vlan 140
name OTV-SITE
vlan 200

name DCI-ESXI
vlan 201
name DCI-VMOTION
vlan 710
name DC1-ISP-1
vlan 711
name DC1-ISP-2

N7K3:

vlan 120
name VM-DATA1
vlan 125
name VM-DATA2
vlan 130
name VM-DATA3
vlan 135
name VM-DATA4
vlan 140
name OTV-SITE
vlan 200
name DCI-ESXI
vlan 201
name DCI-VMOTION
N7K4:

vlan 120
name VM-DATA1
vlan 125
name VM-DATA2
vlan 130
name VM-DATA3
vlan 135
name VM-DATA4
vlan 140
name OTV-SITE
vlan 200
name DCI-ESXI
vlan 201
name DCI-VMOTION

N5K1:

vlan 120
name VM-DATA1
vlan 125

name VM-DATA2
vlan 130
name VM-DATA3
vlan 135
name VM-DATA4
vlan 200
name DCI-ESXI
vlan 201
name DCI-VMOTION
N5K2:

vlan 120
name VM-DATA1
vlan 125
name VM-DATA2
vlan 130
name VM-DATA3
vlan 135
name VM-DATA4
vlan 200
name DCI-ESXI
vlan 201
name DCI-VMOTION

1.2 DCI L3 Routing

Configuration
Before we even start with the configuration instructed, you may notice that not all
ports are allocated to their proper VDC, which was a part of our inherent
troubleshooting. When you are moving ports, it is recommended to do so one at a
time, because if you move a sequential range of ports, it may move more than you
intended and upset other port groups. Remember that on M1 modules, port groups
are groups of 4 odd and groups of 4 even ports, so moving a sequential range can
certainly have adverse effects.
N7K1:

feature ospf
feature interface-vlan

vdc N7K1 id 1
allocate interface Ethernet1/1-8,Ethernet1/18,Ethernet1/20,Ethernet1/22,Ethernet1/24

allocate interface Ethernet2/1-2,Ethernet2/7-8,Ethernet2/17-18,Ethernet2/27-32

vdc N7K2 id 2
allocate interface Ethernet1/17,Ethernet1/19,Ethernet1/21,Ethernet1/23
allocate interface Ethernet2/3-6,Ethernet2/11-12,Ethernet2/21-24
vdc N7K3 id 3
allocate interface Ethernet1/9-16
allocate interface Ethernet2/9-10,Ethernet2/25-26
vdc N7K4 id 4
allocate interface Ethernet1/25-32
allocate interface Ethernet2/13-16,Ethernet2/19-20

system jumbomtu 9000

vrf context DC1

interface Vlan710
no shutdown
vrf member DC1
ip address 10.71.71.0/31
ip ospf mtu-ignore
ip router ospf DC1 area 0.0.0.5
ip pim sparse-mode

interface Vlan711
no shutdown
vrf member DC1
ip address 10.71.71.2/31
ip ospf mtu-ignore
ip router ospf DC1 area 0.0.0.5
ip pim sparse-mode

interface Ethernet2/29
switchport access vlan 710
spanning-tree port type edge
spanning-tree bpduguard enable
speed 1000
mtu 9000
no shutdown

interface Ethernet2/31
switchport access vlan 711
spanning-tree port type edge
spanning-tree bpduguard enable
speed 1000
mtu 9000
no shutdown

router ospf DC1

vrf DC1
router-id 10.71.71.71
N7K2:

feature ospf
feature interface-vlan

system jumbomtu 9000

vrf context DC2

interface Vlan720
no shutdown
vrf member DC2
ip address 10.72.72.0/31
ip ospf mtu-ignore
ip router ospf DC2 area 0.0.0.3
ip pim sparse-mode

interface Vlan721
no shutdown
vrf member DC2
ip address 10.72.72.2/31
ip ospf mtu-ignore
ip router ospf DC2 area 0.0.0.3
ip pim sparse-mode

interface Ethernet2/21
switchport access vlan 720
spanning-tree port type edge
spanning-tree bpdufilter enable
speed 1000
mtu 9000
no shutdown

interface Ethernet2/23
switchport access vlan 721
spanning-tree port type edge
spanning-tree bpdufilter enable
speed 1000
mtu 9000
no shutdown

router ospf DC2

vrf DC2
router-id 10.72.72.72

Verification
N7K1(config)#sh ip ospf neighbors vrf DC1
OSPF Process ID DC1 VRF DC1
Total number of neighbors: 2
Neighbor ID
10.10.10.25
10.10.10.25

Pri State
1 FULL/DR
1 FULL/DR

Up Time

Address

01:27:26 10.71.71.3
01:27:20 10.71.71.1

Interface
Vlan711
Vlan710

N7K1(config)#sh ip route vrf DC1

IP Route Table for VRF "DC1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.0.0.25/32, ubest/mbest: 2/0

*via 10.71.71.1, Vlan710, [110/41], 01:28:28, ospf-DC1, inter
*via 10.71.71.3, Vlan711, [110/41], 01:28:28, ospf-DC1, inter
10.10.10.25/32, ubest/mbest: 2/0
*via 10.71.71.1, [1/0], 01:28:34, static
*via 10.71.71.3, [1/0], 01:28:34, static
10.71.71.0/31, ubest/mbest: 1/0, attached
*via 10.71.71.0, Vlan710, [0/0], 01:28:35, direct
10.71.71.0/32, ubest/mbest: 1/0, attached
*via 10.71.71.0, Vlan710, [0/0], 01:28:35, local
10.71.71.2/31, ubest/mbest: 1/0, attached
*via 10.71.71.2, Vlan711, [0/0], 01:28:35, direct
10.71.71.2/32, ubest/mbest: 1/0, attached
*via 10.71.71.2, Vlan711, [0/0], 01:28:35, local
10.72.72.0/31, ubest/mbest: 2/0
*via 10.71.71.1, Vlan710, [110/41], 01:27:47, ospf-DC1, inter
*via 10.71.71.3, Vlan711, [110/41], 01:27:47, ospf-DC1, inter
10.72.72.2/31, ubest/mbest: 2/0
*via 10.71.71.1, Vlan710, [110/41], 01:27:37, ospf-DC1, inter
*via 10.71.71.3, Vlan711, [110/41], 01:27:37, ospf-DC1, inter
N7K1(config)#

N7K2(config)#sh ip ospf neighbors vrf DC2

OSPF Process ID DC2 VRF DC2
Total number of neighbors: 2

Neighbor ID
10.10.10.25
10.10.10.25

Pri State
1 FULL/BDR
1 FULL/BDR

Up Time

Address

01:28:17 10.72.72.3
01:28:18 10.72.72.1

Interface
Vlan721
Vlan720

N7K2(config)#sh ip route vrf DC2

IP Route Table for VRF "DC2"

'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.0.0.25/32, ubest/mbest: 2/0

*via 10.72.72.1, Vlan720, [110/41], 01:28:22, ospf-DC2, inter
*via 10.72.72.3, Vlan721, [110/41], 01:28:22, ospf-DC2, inter
10.10.10.25/32, ubest/mbest: 2/0
*via 10.72.72.1, [1/0], 01:28:32, static
*via 10.72.72.3, [1/0], 01:28:32, static
10.71.71.0/31, ubest/mbest: 2/0
*via 10.72.72.1, Vlan720, [110/41], 01:28:22, ospf-DC2, inter
*via 10.72.72.3, Vlan721, [110/41], 01:28:22, ospf-DC2, inter
10.71.71.2/31, ubest/mbest: 2/0
*via 10.72.72.1, Vlan720, [110/41], 01:28:22, ospf-DC2, inter
*via 10.72.72.3, Vlan721, [110/41], 01:28:22, ospf-DC2, inter
10.72.72.0/31, ubest/mbest: 1/0, attached
*via 10.72.72.0, Vlan720, [0/0], 01:28:32, direct
10.72.72.0/32, ubest/mbest: 1/0, attached
*via 10.72.72.0, Vlan720, [0/0], 01:28:32, local
10.72.72.2/31, ubest/mbest: 1/0, attached
*via 10.72.72.2, Vlan721, [0/0], 01:28:32, direct
10.72.72.2/32, ubest/mbest: 1/0, attached
*via 10.72.72.2, Vlan721, [0/0], 01:28:32, local
N7K2(config)#

1.3 L2 Trunking and L3 Routed Interfaces

Configuration
To ensure that the L3 interfaces in our aggregation layer VDCs have the ability to
route, we must include them in their site's respective VRFs. We don't neccessarily
need to set these specific VRFs up in the OTV layer if we don't want to (we weren't
instructed to), so we will leave them in the default VRF.
N7K1:

spanning-tree vlan 1-3967 priority 8192

interface Ethernet1/1
vrf member DC1
ip address 10.13.13.0/31
ip router ospf DC1 area 0.0.0.5
no shutdown

interface Ethernet2/1
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,140,200-201
no shutdown
N7K3:

interface Ethernet1/9
ip address 10.13.13.1/31
no shutdown

interface Ethernet2/9
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,140,200-201
no shutdown
N7K2:

interface Ethernet1/17
vrf member DC2
ip address 10.24.24.0/31
ip router ospf DC2 area 0.0.0.3
no shutdown

interface Ethernet2/11
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,140,200-201
no shutdown

N7K4:

interface Ethernet1/25
ip address 10.24.24.1/31
no shutdown

interface Ethernet2/20
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,140,200-201

no shutdown

Verification
N7K1(config)#sh ip route vrf DC1
IP Route Table for VRF "DC1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.0.0.25/32, ubest/mbest: 2/0

*via 10.71.71.1, Vlan710, [110/41], 01:28:28, ospf-DC1, inter
*via 10.71.71.3, Vlan711, [110/41], 01:28:28, ospf-DC1, inter
10.10.10.25/32, ubest/mbest: 2/0
*via 10.71.71.1, [1/0], 01:28:34, static
*via 10.71.71.3, [1/0], 01:28:34, static
10.13.13.0/31, ubest/mbest: 1/0, attached
*via 10.13.13.0, Eth1/1, [0/0], 01:27:19, direct
10.13.13.0/32, ubest/mbest: 1/0, attached
*via 10.13.13.0, Eth1/1, [0/0], 01:27:19, local
10.24.24.0/31, ubest/mbest: 2/0
*via 10.71.71.1, Vlan710, [110/45], 01:26:37, ospf-DC1, inter
*via 10.71.71.3, Vlan711, [110/45], 01:26:37, ospf-DC1, inter
10.71.71.0/31, ubest/mbest: 1/0, attached
*via 10.71.71.0, Vlan710, [0/0], 01:28:35, direct
10.71.71.0/32, ubest/mbest: 1/0, attached
*via 10.71.71.0, Vlan710, [0/0], 01:28:35, local
10.71.71.2/31, ubest/mbest: 1/0, attached
*via 10.71.71.2, Vlan711, [0/0], 01:28:35, direct
10.71.71.2/32, ubest/mbest: 1/0, attached
*via 10.71.71.2, Vlan711, [0/0], 01:28:35, local
10.72.72.0/31, ubest/mbest: 2/0
*via 10.71.71.1, Vlan710, [110/41], 01:27:47, ospf-DC1, inter
*via 10.71.71.3, Vlan711, [110/41], 01:27:47, ospf-DC1, inter
10.72.72.2/31, ubest/mbest: 2/0
*via 10.71.71.1, Vlan710, [110/41], 01:27:37, ospf-DC1, inter
*via 10.71.71.3, Vlan711, [110/41], 01:27:37, ospf-DC1, inter
N7K1(config)#

N7K2(config)#sh ip route vrf DC2

IP Route Table for VRF "DC2"
'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.0.0.25/32, ubest/mbest: 2/0

10.24.24.0/31, ubest/mbest: 1/0, attached

*via 10.24.24.0, Eth1/17, [0/0], 01:27:23, direct
10.24.24.0/32, ubest/mbest: 1/0, attached
*via 10.24.24.0, Eth1/17, [0/0], 01:27:23, local
10.71.71.0/31, ubest/mbest: 2/0
*via 10.72.72.1, Vlan720, [110/41], 01:28:22, ospf-DC2, inter
*via 10.72.72.3, Vlan721, [110/41], 01:28:22, ospf-DC2, inter
10.71.71.2/31, ubest/mbest: 2/0
*via 10.72.72.1, Vlan720, [110/41], 01:28:22, ospf-DC2, inter
*via 10.72.72.3, Vlan721, [110/41], 01:28:22, ospf-DC2, inter
10.72.72.0/31, ubest/mbest: 1/0, attached
*via 10.72.72.0, Vlan720, [0/0], 01:28:32, direct
10.72.72.0/32, ubest/mbest: 1/0, attached
*via 10.72.72.0, Vlan720, [0/0], 01:28:32, local
10.72.72.2/31, ubest/mbest: 1/0, attached
*via 10.72.72.2, Vlan721, [0/0], 01:28:32, direct
10.72.72.2/32, ubest/mbest: 1/0, attached
*via 10.72.72.2, Vlan721, [0/0], 01:28:32, local
N7K2(config)#

1.4 Port Channels

Configuration
N7K1:

lacp system-priority 1
port-channel load-balance src-dst ip-l4port

interface port-channel215
switchport
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
spanning-tree port type edge trunk

interface port-channel218
switchport
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
spanning-tree port type edge trunk

interface Ethernet1/22
switchport
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
channel-group 215 mode active
no shutdown

interface Ethernet1/24
switchport
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
channel-group 218 mode active
no shutdown

Verification

These won't show up until you complete the UCS side, but once you have, they should show:

N7K1(config)# sh port-channel summary

Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------215

Po215(SU)

Eth

LACP

N7K1(config)#

1.5 HSRP
Configuration
N7K1:

interface Vlan120
no shutdown
ip address 192.168.120.251/24
hsrp version 2
hsrp 120
priority 110
preempt
ip 192.168.120.254

interface Vlan125
no shutdown
ip address 192.168.125.251/24
hsrp version 2
hsrp 125
priority 110
preempt
ip 192.168.125.254

interface Vlan130
no shutdown

Eth1/22(P)

218

Po218(SU)

Eth

LACP

Eth1/24(P)

ip address 192.168.130.251/24
hsrp version 2
hsrp 130
priority 110
preempt
ip 192.168.130.254

interface Vlan135
no shutdown
ip address 192.168.135.251/24
hsrp version 2
hsrp 135
priority 110
preempt
ip 192.168.135.254

interface Vlan200
no shutdown
ip address 192.168.200.251/24
hsrp version 2
hsrp 200
priority 110
preempt
timers msec 250 msec 750
ip 192.168.200.254

interface Vlan201
no shutdown
ip address 192.168.201.251/24
hsrp version 2
hsrp 201
priority 110
preempt
ip 192.168.201.254

N7K2:

interface Vlan120
no shutdown
ip address 192.168.120.252/24
hsrp version 2
hsrp 120
priority 110
preempt
ip 192.168.120.254

interface Vlan125
no shutdown
ip address 192.168.125.252/24
hsrp version 2
hsrp 125
priority 110
preempt
ip 192.168.125.254

interface Vlan130
no shutdown
ip address 192.168.130.252/24
hsrp version 2
hsrp 130
priority 110
preempt
ip 192.168.130.254

interface Vlan135
no shutdown
ip address 192.168.135.252/24
hsrp version 2
hsrp 135
priority 110
preempt
ip 192.168.135.254

interface Vlan200
no shutdown
ip address 192.168.200.252/24
hsrp version 2
hsrp 200
priority 110
preempt
timers msec 250 msec 750
ip 192.168.200.254

interface Vlan201
no shutdown
ip address 192.168.201.252/24
hsrp version 2
hsrp 201
priority 110
preempt

ip 192.168.201.254

Verification
These HSRP group are shown after task 1.10 has been completed. At
that time, even though the vlans are extended, because both the
HSRP Hellos and the Virtual MAC addresses are both blocked by
ACLs, each side responds to ARP requests locally and also considers
itself the Active router.

N7K1(config)# sh hsrp
Vlan120 - Group 120 (HSRP-V2) (IPv4)
Local state is Active, priority 100 (Cfged 100), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 0.634000 sec(s)
Virtual IP address is 192.168.120.254 (Cfged) Active router is local
Standby router is unknown
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f078 (Default MAC)
2 state changes, last state change 01:34:10
IP redundancy name is hsrp-Vlan120-120 (default)

Vlan125 - Group 125 (HSRP-V2) (IPv4)

Local state is Active, priority 100 (Cfged 100), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 0.634000 sec(s)
Virtual IP address is 192.168.125.254 (Cfged) Active router is local
Standby router is unknown
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f07d (Default MAC)
2 state changes, last state change 01:34:10
IP redundancy name is hsrp-Vlan125-125 (default)

Vlan130 - Group 130 (HSRP-V2) (IPv4)

Local state is Active, priority 100 (Cfged 100), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 0.634000 sec(s)
Virtual IP address is 192.168.130.254 (Cfged) Active router is local
Standby router is unknown
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f082 (Default MAC)

2 state changes, last state change 01:34:10

IP redundancy name is hsrp-Vlan130-130 (default)

Vlan135 - Group 135 (HSRP-V2) (IPv4)

Local state is Active, priority 100 (Cfged 100), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 0.633000 sec(s)
Virtual IP address is 192.168.135.254 (Cfged) Active router is local
Standby router is unknown
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f087 (Default MAC)
2 state changes, last state change 01:34:10
IP redundancy name is hsrp-Vlan135-135 (default)

Vlan200 - Group 200 (HSRP-V2) (IPv4)

Local state is Active, priority 100 (Cfged 100), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 250 msec, holdtime 750 msec
Next hello sent in 0.163000 sec(s)
Virtual IP address is 192.168.200.254 (Cfged) Active router is local
Standby router is unknown
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f0c8 (Default MAC)
2 state changes, last state change 01:34:30
IP redundancy name is hsrp-Vlan200-200 (default)

Vlan201 - Group 201 (HSRP-V2) (IPv4)

Local state is Active, priority 100 (Cfged 100), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 250 msec, holdtime 750 msec
Next hello sent in 0.163000 sec(s)
Virtual IP address is 192.168.201.254 (Cfged) Active router is local
Standby router is unknown
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f0c9 (Default MAC)
2 state changes, last state change 01:34:30
IP redundancy name is hsrp-Vlan201-201 (default)

N7K1(config)#

N7K2(config)# sh hsrp
Vlan120 - Group 120 (HSRP-V2) (IPv4)
Local state is Active, priority 100 (Cfged 100), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 100

Hellotime 3 sec, holdtime 10 sec

Next hello sent in 0.188000 sec(s)
Virtual IP address is 192.168.120.254 (Cfged) Active router is local
Standby router is unknown
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f078 (Default MAC)
2 state changes, last state change 01:39:53
IP redundancy name is hsrp-Vlan120-120 (default)

Vlan125 - Group 125 (HSRP-V2) (IPv4)

Local state is Active, priority 100 (Cfged 100), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 0.188000 sec(s)
Virtual IP address is 192.168.125.254 (Cfged) Active router is local
Standby router is unknown
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f07d (Default MAC)
2 state changes, last state change 01:39:53
IP redundancy name is hsrp-Vlan125-125 (default)

Vlan130 - Group 130 (HSRP-V2) (IPv4)

Local state is Active, priority 100 (Cfged 100), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 0.188000 sec(s)
Virtual IP address is 192.168.130.254 (Cfged) Active router is local
Standby router is unknown
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f082 (Default MAC)
2 state changes, last state change 01:39:53
IP redundancy name is hsrp-Vlan130-130 (default)

Vlan135 - Group 135 (HSRP-V2) (IPv4)

Local state is Active, priority 100 (Cfged 100), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 0.188000 sec(s)
Virtual IP address is 192.168.135.254 (Cfged) Active router is local
Standby router is unknown
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f087 (Default MAC)
2 state changes, last state change 01:39:53
IP redundancy name is hsrp-Vlan135-135 (default)

Vlan200 - Group 200 (HSRP-V2) (IPv4)

Local state is Active, priority 100 (Cfged 100), may preempt

Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 250 msec, holdtime 750 msec
Next hello sent in 0.188000 sec(s)
Virtual IP address is 192.168.200.254 (Cfged) Active router is local
Standby router is unknown
Authentication text "cisco"
Virtual mac address is 0000.0c9f.f0c8 (Default MAC)
2 state changes, last state change 01:40:12
IP redundancy name is hsrp-Vlan200-200 (default)

Vlan201 - Group 201 (HSRP-V2) (IPv4)

Local state is Active, priority 100 (Cfged 100), may preempt
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 250 msec, holdtime 750 msec
Next hello sent in 0.188000 sec(s)
Virtual IP address is 192.168.201.254 (Cfged) Active router is local

Standby router is unknown

Authentication text "cisco"
Virtual mac address is 0000.0c9f.f0c9 (Default MAC)
2 state changes, last state change 01:40:12
IP redundancy name is hsrp-Vlan201-201 (default)

1.6 vPC
Configuration
N7K2:

spanning-tree vlan 1-3967 priority 57344

N5K1:

feature lacp
feature vpc

lacp system-priority 1

vpc domain 12
role priority 1
system-priority 1
peer-keepalive destination 192.168.101.52 source 192.168.101.51
peer-gateway

ip arp synchronize

interface port-channel512
switchport mode trunk
spanning-tree port type network
speed 10000
vpc peer-link

interface Ethernet1/1
switchport mode trunk
channel-group 512 mode active

interface Ethernet1/2
switchport mode trunk
channel-group 512 mode active
N5K2:

feature lacp
feature vpc

spanning-tree vlan 1-3967 priority 61440

lacp system-priority 1

vpc domain 12
system-priority 1
peer-keepalive destination 192.168.101.51 source 192.168.101.52
peer-gateway
ip arp synchronize

interface port-channel512
switchport mode trunk
spanning-tree port type network
speed 10000
vpc peer-link

interface Ethernet1/1
switchport mode trunk
channel-group 512 mode active

interface Ethernet1/2
switchport mode trunk

channel-group 512 mode active

Verification
N5K1(config)# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 12

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive Configuration consistency status: success

Per-vlan consistency status

: success

Type-2 consistency status

: success vPC role

Number of vPCs configured

: 67 Peer Gateway

Peer gateway excluded VLANs

: -

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

: primary
: Enabled

vPC Peer-link status

--------------------------------------------------------------------id

Port

Status Active vlans

----

------ --------------------------------------------------

Po512

1,120,125,130,135,200-201

N5K2(config)# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 12

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive

Configuration consistency status: success

Per-vlan consistency status

: success

Type-2 consistency status

: success vPC role

Number of vPCs configured

: 67

Peer Gateway

: Enabled

Peer gateway excluded VLANs

: -

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

vPC Peer-link status

---------------------------------------------------------------------

: secondary

Port

Status Active vlans

----

------ --------------------------------------------------

Po512

1,120,125,130,135,200-201

This spanning-tree show command is after the next task has been performed.
Remember the whole topology when thinking about where the STP domain will
reach. Only one VLAN is shown, although all have same result.
N5K1(config)# sh spanning-tree vlan 120

VLAN0120
Spanning tree enabled protocol rstp
Root ID

32888

Address

547f.ee7a.4d7c This bridge is the root

Hello Time

Priority

32888

Address

547f.ee7a.4d7c

Hello Time

Bridge ID

Priority

Interface

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 120)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po100

Desg FWD 1

128.4195 (vPC) P2p

Po512

Desg FWD 1

128.4607 (vPC peer-link) Network P2p

Eth113/1/1

Desg FWD 1

128.1

(vPC) Edge P2p

Eth123/1/1

Desg FWD 1

128.1

(vPC) Edge P2p

N5K1(config)#

N5K2(config)# sh spanning-tree vlan 120

VLAN0120
Spanning tree enabled protocol rstp
Root ID

Bridge ID

Interface

Priority

32888

Address

547f.ee7a.4d7c

Cost

Port

4607 (port-channel512)

Hello Time

Priority

61560

Address

547f.ee79.137c

Hello Time

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 61440 sys-id-ext 120)

Max Age 20 sec

Prio.Nbr Type

Forward Delay 15 sec

---------------- ---- --- --------- -------- -------------------------------Po100

128.4195 (vPC) P2p Po512

Desg FWD 1

Root FWD

128.4607 (vPC peer-link) Network P2p

Eth113/1/1

Desg FWD 1

128.2689 (vPC) Edge P2p

Eth123/1/1

Desg FWD 1

128.3969 (vPC) Edge P2p

N5K2(config)#

N7K2(config)# sh spanning-tree vlan 120

VLAN0120
Spanning tree enabled protocol rstp
Root ID

32888

Address

547f.ee7a.4d7c

Cost

Port

4195 (port-channel100)

Hello Time

Priority

57464

Address

64a0.e742.8dc2

Hello Time

Bridge ID

Priority

Interface

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 57344 sys-id-ext 120)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------- Po100

128.4195 P2p

Eth2/11

Desg FWD 2

128.267

P2p

N7K2(config)#

1.7 Port Channels, FEX, and vPC

Configuration
N7K2:

interface port-channel100
switchport
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201

interface Ethernet2/5
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201

Root FWD

channel-group 100 mode passive

no shutdown

interface Ethernet2/6
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
channel-group 100 mode passive
no shutdown

N5K1:

fex 113
pinning max-links 1
description "FEX0113"
fex 123
pinning max-links 1
description "FEX0123"

slot 113
provision model N2K-C2232P
slot 123
provision model N2K-C2232P

interface port-channel100
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
speed 10000
vpc 100

interface port-channel113
switchport mode fex-fabric
fex associate 113
vpc 113

interface port-channel123
switchport mode fex-fabric
fex associate 123
vpc 123

interface port-channel1550
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
speed 10000

interface Ethernet1/3
switchport mode trunk

switchport trunk allowed vlan 120,125,130,135,200-201

channel-group 100 mode active

interface Ethernet1/12
switchport mode fex-fabric
fex associate 113
channel-group 113

interface Ethernet1/13
shutdown

interface Ethernet1/14
switchport mode fex-fabric
fex associate 123
channel-group 123

interface Ethernet1/15
shutdown
N5K2:

fex 113
pinning max-links 1
description "FEX0113"
fex 123
pinning max-links 1
description "FEX0123"

slot 113
provision model N2K-C2232P
slot 123
provision model N2K-C2232P

interface port-channel100
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
speed 10000
vpc 100

interface port-channel113
switchport mode fex-fabric
fex associate 113
vpc 113

interface port-channel123
switchport mode fex-fabric

fex associate 123

vpc 123

interface port-channel1550
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
speed 10000

interface Ethernet1/3
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
channel-group 100 mode active

interface Ethernet1/12
switchport mode fex-fabric
fex associate 113
channel-group 113

interface Ethernet1/13
shutdown

interface Ethernet1/14
switchport mode fex-fabric
fex associate 123
channel-group 123

interface Ethernet1/15
shutdown

Verification
N5K1(config)# sh fex
FEX
Number

FEX

Description

State

FEX
Model

Serial

-----------------------------------------------------------------------113

FEX0113

Online

N2K-C2232PP-10GE

SSI165204YC

123

FEX0123

Online

N2K-C2232PP-10GE

SSI16510AWF

N5K1(config)# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 12

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive

Configuration consistency status: success

Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: primary

Number of vPCs configured

: 67

Peer Gateway

: Enabled

Peer gateway excluded VLANs

: -

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

vPC Peer-link status

--------------------------------------------------------------------id

Port

Status Active vlans

----

------ --------------------------------------------------

Po512

1,120,125,130,135,200-201

vPC status
---------------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

------ ----------- ------ ----------- -------------------------- ----------100

Po100

success

1,10,120,12
5,130,135,2
00-201

113

Po113

success

123

Po123

success

N5K2(config)# sh fex
FEX
Number

FEX

Description

State

FEX
Model

Serial

-----------------------------------------------------------------------113

FEX0113

Online

N2K-C2232PP-10GE

SSI165204YC

123

FEX0123

Online

N2K-C2232PP-10GE

SSI16510AWF

N5K2(config)# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 12

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive

Configuration consistency status: success

Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: secondary

Number of vPCs configured

: 67

Peer Gateway

: Enabled

Peer gateway excluded VLANs

: -

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

vPC Peer-link status

--------------------------------------------------------------------id

Port

Status Active vlans

----

------ --------------------------------------------------

Po512

1,120,125,130,135,200-201

vPC status
---------------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

------ ----------- ------ ----------- -------------------------- ----------100

Po100

success

1,10,20,120
,125,130,13
5,200-201

113

Po113

success

123

Po123

success

N5K1(config)# sh port-channel summary

Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------100

Po100(SU)

Eth

LACP

Eth1/3(P)

113

Po113(SU)

Eth

NONE

Eth1/12(P) 123

512

Po512(SU)

Eth

LACP

Eth1/1(P)

Po123(SU)

Eth

NONE

Eth1/2(P)

N5K1(config)#

N5K2(config)# sh port-channel summary

Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Eth1/14(P)

Channel
-------------------------------------------------------------------------------100

Po100(SU)

Eth

LACP

Eth1/3(P)

113

Po113(SU)

Eth

NONE

Eth1/12(P) 123

512

Po512(SU)

Eth

LACP

Eth1/1(P)

Po123(SU)

Eth

NONE

Eth1/2(P)

N5K2(config)#

1.8 Mgmt VM Access

Configuration
N5K1:

interface Ethernet1/11
switchport access vlan 200
spanning-tree port type edge
spanning-tree bpduguard enable
speed 1000

Verification
N5K1(config)# sh int e1/11 | in up|1000 Ethernet1/11 is up
Hardware: 1000/10000 Ethernet, address: 547f.ee7a.4d32 (bia 547f.ee7a.4d32)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec

full-duplex, 1000 Mb/s

, media type is 10G

0 jumbo packets

0 storm suppression bytes

N5K1(config)#

N5K1(config)# sh spanning-tree interface e1/11

Vlan

Role Sts Cost

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------VLAN0200

Desg FWD 4

128.139

1.9 Access Trunking

Edge P2p

Eth1/14(P)

Configuration
N5K1:

interface Ethernet113/1/1
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
spanning-tree port type edge trunk

interface Ethernet123/1/1
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
spanning-tree port type edge trunk
N5K2:

interface Ethernet113/1/1
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
spanning-tree port type edge trunk

interface Ethernet123/1/1
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,200-201
spanning-tree port type edge trunk

1.10 OTV
Configuration:
The L2 inside and L3 join interfaces have already been configured from both the
Aggregation and OTV layers, but we are displaying them here again just to bring
everything necessary into contextual view, and adding the necessary PIM and IGMP
configuration where necessary. You may notice that the ISP RP address of
10.10.10.25 doesn't appear in our RIB; this is because OSPF didn't advertise it to
us. We weren't instructed not to use static routes, so those will certainly suffice here
to point us toward it. We can only hope it has a route back, and when we confiure
redundant static routes out each of our links, we ping it and see that it, in fact, does.
Note that we must do this under our VRF, as well as set up PIM there.
N7K1:

vrf context DC1

ip route 10.10.10.25/32 10.71.71.1
ip route 10.10.10.25/32 10.71.71.3
ip pim rp-address 10.10.10.25 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8

interface Ethernet1/1
vrf member DC1
ip address 10.13.13.0/31
ip router ospf DC1 area 0.0.0.5
ip pim sparse-mode
ip igmp version 3
no shutdown

interface Ethernet2/1
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,140,200-201
no shutdown
N7K3:

feature otv

ip access-list ALL_IPs
10 permit ip any any
ip access-list HSRP_IP
10 permit udp any 224.0.0.2/32 eq 1985
20 permit udp any 224.0.0.102/32 eq 1985
vlan access-map HSRP_Localization 10
match ip address HSRP_IP
action drop
vlan access-map HSRP_Localization 20
match ip address ALL_IPs
action forward
vlan filter HSRP_Localization vlan-list 120,125,130,135,200-201

mac-list OTV_HSRP_VMAC_deny seq 10 deny 0000.0c07.ac00 ffff.ffff.ff00

mac-list OTV_HSRP_VMAC_deny seq 20 deny 0000.0c9f.f000 ffff.ffff.f000
mac-list OTV_HSRP_VMAC_deny seq 30 permit 0000.0000.0000 0000.0000.0000

route-map OTV_HSRP_filter permit 10

match mac-list OTV_HSRP_VMAC_deny

key chain OTV

key 0
key-string DCIOTV

otv site-vlan 140

otv site-identifier 0x1

interface Overlay1
otv isis authentication-type md5
otv isis authentication key-chain OTV
otv join-interface Ethernet1/9
otv control-group 224.1.1.1
otv data-group 232.1.1.0/24
otv extend-vlan 120, 125, 130, 135, 200-201
no otv suppress-arp-nd
no shutdown

interface Ethernet1/9
ip address 10.13.13.1/31
ip igmp version 3
no shutdown

interface Ethernet2/9
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,140,200-201
no shutdown

otv-isis default
vpn Overlay1
redistribute filter route-map OTV_HSRP_filter

ip route 0.0.0.0/0 10.13.13.0

N7K2:

vrf context DC2

ip route 10.10.10.25/32 10.72.72.1
ip route 10.10.10.25/32 10.72.72.3
ip pim rp-address 10.10.10.25 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8

interface Ethernet1/17
vrf member DC2
ip address 10.24.24.0/31
ip router ospf DC2 area 0.0.0.3
ip pim sparse-mode
ip igmp version 3
no shutdown

interface Ethernet2/11
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,140,200-201
no shutdown

N7K4:

feature otv

mac-list OTV_HSRP_VMAC_deny seq 10 deny 0000.0c07.ac00 ffff.ffff.ff00

mac-list OTV_HSRP_VMAC_deny seq 20 deny 0000.0c9f.f000 ffff.ffff.f000
mac-list OTV_HSRP_VMAC_deny seq 30 permit 0000.0000.0000 0000.0000.0000

route-map OTV_HSRP_filter permit 10

match mac-list OTV_HSRP_VMAC_deny

key chain OTV

key 0
key-string DCIOTV

otv site-vlan 140

otv site-identifier 0x2

interface Overlay1
otv isis authentication-type md5
otv isis authentication key-chain OTV
otv join-interface Ethernet1/25
otv control-group 224.1.1.1
otv data-group 232.1.1.0/24
otv extend-vlan 120, 125, 130, 135, 200-201
no otv suppress-arp-nd
no shutdown

interface Ethernet1/25
ip address 10.24.24.1/31
ip igmp version 3
no shutdown

interface Ethernet2/20
switchport mode trunk
switchport trunk allowed vlan 120,125,130,135,140,200-201
no shutdown

otv-isis default
vpn Overlay1
redistribute filter route-map OTV_HSRP_filter

ip route 0.0.0.0/0 10.24.24.0

Verification
First, let's look at general OTV information from both sides of the tunnel.
N7K3(config)#sh otv

OTV Overlay Information

Site Identifier 0000.0000.0001

Overlay interface Overlay1

VPN name

: Overlay1

VPN state

: UP

Extended vlans

: 120 125 130 135 200-201 (Total:6)

Control group

: 224.1.1.1

Data group range(s) : 232.1.1.0/24

Join interface(s)

: Eth1/9 (10.13.13.1)

Site vlan

: 140 (up)

AED-Capable

: Yes

Capability

: Multicast-Reachable N7K3(config)#sh otv adjacency

Overlay Adjacency database

Overlay-Interface Overlay1

Hostname

System-ID

N7K4

64a0.e742.8dc4 10.24.24.1

Dest Addr

OTV Extended VLANs and Edge Device State Information (* - AED)

Up Time

State

13:18:15

UP N7K3(config)#sh otv vlan

VLAN

Auth. Edge Device

Vlan State

Overlay

----

-----------------------------------

----------

-------

120*

N7K3

active

Overlay1

125*

N7K3

active

Overlay1

130*

N7K3

active

Overlay1

135*

N7K3

active

Overlay1

200*

N7K3

active

Overlay1

201*

N7K3

active

Overlay1

N7K3(config)#

N7K4(config)#sh otv

OTV Overlay Information

Site Identifier 0000.0000.0002

Overlay interface Overlay1

VPN name

: Overlay1

VPN state

: UP

Extended vlans

: 120 125 130 135 200-201 (Total:6)

Control group

: 224.1.1.1

Data group range(s) : 232.1.1.0/24

Join interface(s)

: Eth1/25 (10.24.24.1)

Site vlan

: 140 (up)

AED-Capable

: Yes

Capability

: Multicast-Reachable N7K4(config)#sh otv adjacency

Overlay Adjacency database

Overlay-Interface Overlay1

Hostname

System-ID

Dest Addr

N7K3

64a0.e742.8dc3 10.13.13.1

Up Time

State

13:35:44

UP N7K4(config)#sh otv vlan

OTV Extended VLANs and Edge Device State Information (* - AED)

VLAN

Auth. Edge Device

Vlan State

Overlay

----

-----------------------------------

----------

-------

120*

N7K4

active

Overlay1

125*

N7K4

active

Overlay1

130*

N7K4

active

Overlay1

135*

N7K4

active

Overlay1

200*

N7K4

active

Overlay1

201*

N7K4

active

Overlay1

N7K4(config)#

We will ping the SVI for VLAN 200 on N7K2 in DC2, but first let's see what its MAC
and IP are.
N7K2(config)# sh int vlan200 | in ddress
Hardware is EtherSVI, address is

64a0.e742.8dc2

Internet Address is 192.168.200.252/24

N7K2(config)#

Let's look at the OTV routing table before we ping this particular IP; notice that the
MAC is not in the table.
N7K3(config)# sh otv route

OTV Unicast MAC Routing Table For Overlay1

VLAN MAC-Address

Metric

Uptime

Owner

Next-hop(s)

---- --------------

------

--------

---------

-----------

200 000c.29bb.9b82

13:18:17

overlay

N7K4

200 0025.b50a.0a06

13:18:16

site

Ethernet2/9

200 0025.b50a.0a0b

13:18:12

site

Ethernet2/9

200 d48c.b5bd.460c

13:18:17

overlay

N7K4

N7K3(config)#

Now we try to ping 192.168.200.252 and see that after a while, the OTV tunnel
routes traffic for the newly learned MAC address of 64a0.e742.8dc2.

N7K1(config)# ping 192.168.200.252

PING 192.168.200.252 (192.168.200.252): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
64 bytes from 192.168.200.252: icmp_seq=3 ttl=254 time=1.52 ms
64 bytes from 192.168.200.252: icmp_seq=4 ttl=254 time=1.003 ms

--- 192.168.200.252 ping statistics --5 packets transmitted, 2 packets received, 60.00% packet loss
round-trip min/avg/max = 1.003/1.261/1.52 ms
N7K1(config)#

Verify that route was added. We also see our local address (64a0.e742.8dc1) as
now known by our internal interface.
N7K3(config)# sh otv route

OTV Unicast MAC Routing Table For Overlay1

VLAN MAC-Address

Metric

Uptime

Owner

Next-hop(s)

---- --------------

------

--------

---------

-----------

200 000c.29bb.9b82