Cisco Nexus Switches Part 1: A

Brief Overview

A brief overview of Cisco Nexus switches.

Modern data centers arguably require a purpose-built switch to meet the unique
needs of today’s data center network traffic. LAN switches had to evolve to
accommodate new bandwidth, fault tolerance, latency and centralized management
requirements. As a result of this technical evolution, the Cisco Nexus family of
switches was born. In 2008 the first models were introduced, and soon after they
took the network world by storm. Today these are probably the most common
switches in server racks. For the next couple weeks I’ll be diving into a several part
series introducing the Cisco Nexus family and breaking down step-by-step how to
configure and deploy specifically the 5500 series switches.

To keep my posts relevant to someone new to Nexus switches, I’ll be focusing

mainly on high level concepts and general configuration. I won’t be looking into
architectures such as FlexPod or explaining technologies such as ACI, fabric path,
or OTV. There are several narratives being told in the industry about the benefits
of one design over another or one technology over another, but for the next few
posts I’ll be writing for an audience that’s learning about this technology for the
first time.

First, let’s examine the defining attributes of Nexus switches.

An important benefit of these switches is their flexibility to accommodate ethernet,

FCoE or fibre channel, but probably the most important benefit is the Nexus’s
ability to switch frames really fast. That, along with virtual port channels (vPC),
virtual device contexts (vDC), and near lossless transmission over fiber channel,
makes Nexus series switches almost as cool as Chris Cornell in 1994.
There are a variety of Nexus switch models currently on the market including the
1000, 2200, 3000, 4000, 5500, 6000, 7000 and 9000 series. Though the design and
configuration among these is pretty similar, there are some significant differences
to note. The 3000 doesn’t support fabric extenders, for example, and offers only a
few 10 Gbps SFP+ ports while the rest are RJ45. The 1000 is a virtual switch
extending the access layer right into the hypervisor, the 2200 series is unique in
that it isn’t a standalone switch at all, the 4000 is an IBM blade center switch, and
the 9000 can be deployed in standalone or ACI mode.

But for now let’s focus on the 7k and 5k.

The Nexus 7000 series switches are chassis switches normally used at the data
center core providing multiple high bandwidth linecards, dual supervisors, several
power supplies, crossbar fabric modules for an insane backplane and all the
awesome features expected in a top-of-the-line core switch. The Nexus 7009 and
7010 are the most common, with nine and ten slots, respectively. The Nexus 7000
series uses a Unified Ports architecture providing SFP+ slots that can be configured
for either 1/2/4/8-Gbps Fibre Channel or 1/10-Gbps Ethernet SFP+ modules. This
modular approach gives it a similar feel to the to the 4500 and 6500 series chassis
switches.

The Nexus 5500 series switches can be used at the data center core of smaller
environments, but they’re also commonly used as end-of-row switches aggregating
uplinks from smaller top-of-rack switches such as other 5000s, 3000s, or most
commonly 2200s. The 5500 series comes in 1U (5548) and 2U (5596) form
factors. The 5548 provides 32 ports built-in and has a single, 16-port expansion
slot for a total maximum of 48 ports. The 5596 provides 48 ports built-in and has
three 16-port expansion slots for a maximum of 96 ports. The Nexus 5548UP and
5596UP provide the ability to configure SPF+ slots for either 1/2/4/8-Gbps Fibre
Channel or 1/10-Gbps Ethernet SFP+ modules. The ‘UP’ in the model name stands
for Unified Ports which signifies that particular model’s ability to accommodate
Ethernet or Fibre Channel.

I’ll mention the Nexus 2200 series switch here as well because it’s often deployed
along with the 5k switches. A Nexus 2200 series switch is more commonly known
as a fabric extender, or FEX, because it’s not configurable as a standalone switch.
It is what it’s name suggests: an extension of an upstream switch in a sort of
deconstructed switch fabric design. The switch is normally installed top-of-rack for
server access and then homed into a 5548, 5596 or 6001 end-of-row switch or pair
of switches. This allows the 5500/7000 architecture to scale easily with a
centralized management capability because all FEX configuration is done on the
upstream switch. Taken as a whole, the entire system is the Nexus switch fabric
deconstructed throughout multiple server racks or an entire data center.
One of the more powerful features of the Nexus operating system, NX-OS, is the
virtual port channel, or vPC. NX-OS vPCs allow two or more links to be connected
between a device such as a core switch and two Nexus switches. To the connected
device, the Nexus switches appear as a single switch. This is similar but not the
same as VSS used with some Cisco Catalyst switches. vPCs allow layer 2 link
redundancy but retain Nexus switch independence in the control plane.

– A simple image of a fully redundant design –

I’ve been in many data centers in which there was no redundancy at the server
access layer though redundancy on several levels at the core. I’ve also seen designs
in which there was access layer redundancy, top of rack redundancy but then no
core fault tolerance. The above drawing is a simplified image of an ideal scenario
in which servers use LACP to create port channels to each access layer FEX, and
then each FEX uses port channel vPCs to dual home to disparate core switches also
joined by a port channel. The cost associated with eliminating every conceivable
point of failure is tremendous, but this basic design using what in 2015 is common
technology provides a much more resilient application delivery medium.

The remaining posts in this series will be focused on configuring these switches in
both a single-homed and dual-homed topology. I’ll introduce additional concepts
when they’re relevant, but my main goal is to provide a clear guide for getting
these awesome switches up and running with solid configurations. Get some more
coffee, microwave a hot pocket, and follow along.

Cisco Nexus Switches Part 2:

Basic Configuration

This week’s post will cover basic information gathering and configuration of Cisco
Nexus switches. I’ll be using the 5500 series as my example and covering the
basics without getting into features such as fibre channel, VSANs and that sort of
thing.

But before hacking away at the CLI, the first step should be to gather all the
relevant information needed for a successful deployment. The goal is to use
whatever awesome new gear you’re installing to meet some technical and business
need, so getting it right the first time is paramount. Set up a spreadsheet, set
something up in Google Docs, start an Evernote account, or just use a checklist
with blanks to fill out. Really, the key is to gather the information you need early
on so you can configure and deploy the equipment correctly, efficiently, and
professionally.

When deploying Cisco Nexus switches, you’ll need some specific information
about the network already in place, and you’ll need to make some configuration
decisions ahead of time. You should spend some time running the hardware,
recording serial numbers and meeting with your team or with the customer. Here is
just an overview of some things to do and consider.

1. Start off by unboxing the new gear and powering everything up. Let the
new switches run for a few days just so you know you don’t have any
DOA devices to RMA. I try to do this whether the switches are for
internal use or for a customer. If you’re on site with a customer, you may
not be able to do run them at all before racking them, but the key is
letting them run for a while at least before putting them into production.

2. Make a list of the following preliminary information:

 Hostname for each device

 Management IP addresses, subnet mask and default gateway

 Local user accounts

 Features to enable such as vPC, FCoE, DHCP, FEX, VTP,

LACP, etc

 Role of switch (end-of-row, top-of-rack, core)

 All VLANs needed on the Nexus switches

 Rack location, type of cage nuts to use

 vPC number(s) (just a unique identifier you’ll need to set up

vPCs later on)

 Uplink trunk ports to data center/LAN core

  DHCP relay information

 Any VLAN interfaces that will used on the switches for your
design

 A list of all the devices that will connect to the Nexus switches

3. Check that you have the correct power cables for the PDUs, correct SFPs
(1/10 Gbps ethernet, 8 Gbps fibre channel) and appropriate storage
connectivity.

4. Identify the hot and cold aisles and plan to install the switches
accordingly. Default airflow on the 5500 series is front-to-back, for
example, the back being where all the ports are located. Airflow on the
switches can be ordered in either direction, so this is an important thing
to check.
I like to gather the Nexus specific information before getting into mounting
hardware or configuring anything at all. In my experience, sitting down with the
team or with the customer before doing anything whatsoever is the best way to
ensure a smooth project. Below is a simplified version of a spreadsheet I’ve used to
gather relevant information. It’s a variation of something I used when working for
a Cisco partner a few years ago and should be part of a larger spreadsheet in which
you should capture DNS and RADIUS server addresses, SmartNet contract
numbers, serial numbers, asset tag information, rack and data center location, and
all that sort of thing. You can download the spreadsheet here.
 

Now let’s get into the initial configuration wizard.

1. Power up the new Nexus switch and connect to the console port using a
serial cable. The switch will take several minutes to boot.

2. The initial configuration wizard starts automatically. Use the information

you worked out with your team or with the customer to complete the
wizard. These settings can be changed later. The Nexus 7000 series
initial configuration is almost the same, but it will prompt you for
additional information about the default virtual device context.
Next we’ll upgrade the firmware.
Upgrading the firmware requires a reboot, so make sure to do this before moving
forward with any significant configuration and of course before putting the switch
into production. There are several methods for moving files around, but I prefer
using a USB stick because it’s fast, straightforward, and reliable.

1. Download the latest recommended firmware code version for your

specific switch from Cisco’s download page website (you’ll need to log
in) and save it to your USB stick.

2. Insert the USB stick into the USB port of the switch and run the
following commands:
 

After the firmware is upgraded, we can start the configuration.

I prefer to start with the more basic elements of the configuration and move to the
more complex. Every deployment is different, so keep in mind that there is more
than one way to accomplish some of these tasks. Typically Nexus 5500 series
switches are configured in pairs, so make sure to repeat the below configuration on
the second switch. Also, make sure you’re saving your config periodically as you
go.

1. Now configure basic Spanning Tree.

NEXUS5K-A#config term
NEXUS5K-A(config)#spanning-tree  port type network default
NEXUS5K-A(config)#spanning-tree  port type edge bpduguard default
 

2. Now enable all the features you’ll need for this implementation. Below is just an
example of common features. It’s typically best practice not to enable features you
don’t need.

NEXUS5K-A(config)#feature lacp
NEXUS5K-A(config)#feature fex
NEXUS5K-A(config)#feature interface-vlan
NEXUS5K-A(config)#feature vpc
NEXUS5K-A(config)#feature lldp
 
3. Typical IP storage traffic requires the switch to accommodate jumbo frames, but
by default the switch is configured to process 1500 byte ethernet frames. Configure
a QoS policy to accommodate 9000 byte ethernet frames.

NEXUS5K-A(config)#policy-map  type network-qos jumbo

NEXUS5K-A(config-pmap)#class  type network-qos class-default
NEXUS5K-A(config-pmap-nq-c)#mtu  9216system  qos
NEXUS5K-A(config-pmap-nq-c)#system  qos
NEXUS5K-A(config-sys-qos)#service-policy type network-qos jumbo
NEXUS5K-A(config-sys-qos)#end
 

4. Next configure the VLANs needed for this deployment. In a large network with
a lot of VLANs I’ve used VTP in client mode to quickly get all the VLANs onto
the switch, but generally I don’t recommend doing that. If you choose to use VTP,
you’ll need to enable the feature and make sure you configure VTP in client mode.
Afterward you can disable the protocol and the feature.

NEXUS5K-A#config term
NEXUS5K-A(config)#vlan 10
NEXUS5K-A(config-vlan)#name iSCSI
NEXUS5K-A(config)#vlan 20
NEXUS5K-A(config-vlan)#name vMOTION
NEXUS5K-A(config)#vlan 30
NEXUS5K-A(config-vlan)#name VM_MANAGEMENT
NEXUS5K-A(config)#vlan 40
NEXUS5K-A(config-vlan)#name NFS
NEXUS5K-A(config-if)#exit
 
5.  Now configure the virtual port channel (vPC). Configuring a vPC requires a
peer link, vPC domain ID, and the appropriate interface configuration. The
example below has two 10 Gbps ports in a port channel, though I typically
configure four ports if I know they will be available. The channel-group mode
must be active in order to utilize LACP.
NEXUS5K-A(config)#vpc domain 10
NEXUS5K-A(config-vpc)#peer-keepalive destination [IP address of switch B]
source [IP address of switch A]
NEXUS5K-A(config-vpc)#interface  e1/5-6
NEXUS5K-A(config-if)#channel-group  10 mode active
NEXUS5K-A(config-if)#interface  po 10
NEXUS5K-A(config-if)#description  vpc peer link
NEXUS5K-A(config-if)#switchport  mode trunk
NEXUS5K-A(config-if)#switchport  trunk allowed vlan 1, vlan 10, vlan 20,
[include additional necessary vlans]
NEXUS5K-A(config-if)#spanning-tree  port type network
NEXUS5K-A(config-if)#vpc peer link
NEXUS5K-A(config-if)#no shut
NEXUS5K-A(config-if)#exit
 

The NX-OS operating system chooses the primary and secondary switch priorities
automatically, but the role priority command can be used to manually configure
which is which. The lower priority value sets the switch as primary. You can also
add the delay restore [time in seconds] command to manually control how long it
takes before the vPC comes back up on the peer switch after a reload. There are a
variety of other commands you can use to control more precisely the behavior of
the vPC, but for this exercise I’ve kept the configuration simple.
 
6. Configure the uplink trunk ports to the core switch. The upstream switch will
likely be the data center core (Nexus 7009/7010) or the LAN core. The config
below is for a Nexus 7k upstream switch.

NEXUS5K-A(config)#interface e1/1-2
NEXUS5K-A(config-if)#description  TRUNK_TO_CORE
NEXUS5K-A(config-if)#switchport
NEXUS5K-A(config-if)#switchport  mode trunk
NEXUS5K-A(config-if)#spanning-tree  port type network
NEXUS5K-A(config-if)#end
 

Notice above that in order to configure a range of ports on a Nexus switch it isn’t
necessary to use the interface range command you may be used to from
configuring Catalyst switches. Also note the interface command spanning-tree
port type network. This is extremely important to use on interfaces connecting to
other Nexus switches. When connecting to an IP storage controller use the
interface command spanning-tree port type edge trunk. This command is used
when connecting to end hosts that carry multiple VLANs. When connecting to
non-Nexus switches such as a Catalyst 6500 series switch use the spanning-tree
port type normal command. If you have redundant core switches, you should use
a vPC for the uplink(s).
 

7. Configure the access ports.

NEXUS5K-A#config t
NEXUS5K-A(config)#interface e1/15
NEXUS5K-A(config-if)#description  UCS-FI-A Port e1/15
NEXUS5K-A(config-if)#switchport
NEXUS5K-A(config-if)#switchport  mode access
NEXUS5K-A(config-if)#switchport  access vlan 200
NEXUS5K-A(config-if)#end
 

8. Configure the fabric extenders. Each FEX will have a unique identifier which
will also end up being the prefix on the interface number. In the example below,
the first FEX is assigned the identifier 101, so the interfaces will appear as 101/1/1.
A new vPC also needs to be created for each FEX which means each Nexus
5548/5596 will have two additional vPCs configured: one for each FEX. The
example below is for one. Use a port channel to each FEX so you have link
redundancy as well as switch redundancy.

NEXUS5K-A#conf t
NEXUS5K-A(config)#interface e1/10-11
NEXUS5K-A(config-if)#switchport  mode fex-fabric
NEXUS5K-A(config-if)#fex associate 101
NEXUS5K-A(config-if)#channel-group  101
NEXUS5K-A(config-if)#no shutdown
NEXUS5K-A(config-if)#interface  po 101
NEXUS5K-A(config-if)#switchport  mode fex-fabric
NEXUS5K-A(config-if)#fex associate 101
NEXUS5K-A(config-if)#vpc 101
NEXUS5K-A(config-if)#description  DUAL_HOMED_NX2248
NEXUS5K-A(config-if)#end
NEXUS5K-A#copy run start
 

 
You’ll still need to fine tune your configuration including configuring your vty
lines, SNMP, VRFs, RADIUS servers, and whatever features and optimizations
you prefer to use. You may also want to employ a function called configuration
synchronization (config-sync). Also, I don’t typically like to route on Nexus 5k
switches so that they can focus on doing what they do best: switching frames super
fast at layer 2. You can take a look at a basic configuration used in
production here.
This basic overview of configuring Cisco Nexus switches is meant to get you
started. There are a variety of additional features and nuances that you can look
into on Cisco’s website and all over the internet. For now, stay tuned for my next
posts on Nexus switching, network engineering and professional development.