Advanced Storage Area

Network Design

Edward Mazurek
Technical Lead Data Center Storage Networking
[email protected]
@TheRealEdMaz

BRKSAN-2883
Agenda

• Introduction
• Technology Overview
• Design Principles
• Storage Fabric Design Considerations
• Data Center SAN Topologies
• Intelligent SAN Services
• Q&A

3
Introduction

6
An Era of Massive Data Growth
Creating New Business Imperatives for IT

10X Increase in Data Produced (From 4.4T GB to 44T GB)

32B IoT Devices (Will be Connected to Internet)

By 2020
40% of Data Will Be “Touched” by Cloud

85% of Data for Which Enterprises Will Have

Liability and Responsibility
IDC April 2014: The Digital Universe of Opportunities: Rich Data and Increasing Value of Internet of Things

7
Evolution of Storage Networking….
Enterprise Apps: OLTP, VDI, etc. Big Data, Scale-Out NAS Cloud Storage (Object)

Compute Nodes

REST API
Fabric

Fabric

Block and/or File Arrays

Multi-Protocol (FC, FICON, FCIP, FCoE, NAS, iSCSI, HTTP)

Performance (16G FC, 10GE, 40GE, 100GE)
Scale (Tens of Thousands P/V Devices, Billions of Objects)
8

Operational Simplicity (Automation, Self-Service Provisioning)

 Enterprise Flash Drives = More IO
Significantly More IO/s per Drive at Much Lower Response Time
• Drive performance hasn’t
110
changed since 2003 (15K
100
SATA drives drives)
90 (8 drives)
• Supports new application
80 100% Random Read Miss 8KB performance requirements
Response Time Msec

70 One Drive per DA Processor - 8 processors

60
• Price/performance making
50
15K rpm drives SSD more affordable
(8 drives)
40 • Solid state drives dramatically
30 Enterprise Flash increase IOPS that a given
Drives (8 drives) array can support
20

10 • Increased IO directly translates

0 to increased throughput
0 5000 10000 15000 20000 25000 30000 35000 40000 45000
IOPs
9
Technology Overview

15
 Fibre Channel – Foundations
Based on SCSI

• Foundational protocol, forms the basis of

Host Disk
(Initiator) (Target) an I/O transaction
SCSI READ Operation • Communications are based upon Point to
SCSI I/O Channel
Point
• Storage is accessed at a block-level via
SCSI
Host Disk
(Initiator) (Target) • High-performance interconnect providing
SCSI WRITE Operation
high I/O throughput
SCSI I/O Channel
• The foundation for all block-based storage
connectivity
• Mature - SCSI-1 developed in 1986
16
Fibre Channel - Communications
Point-to-point oriented Transmitter Receiver

• Facilitated through device login N_port N_port

Receiver Transmitter

N_Port-to-N_Port connection
Host Disk
• Logical node connection point (Initiator) (Target)

Flow controlled
• Buffer-to-buffer credits and end-to-end basis Transmitter Receiver
N_port

Acknowledged Receiver Transmitter

• For certain classes of traffic, none for others Host SAN

(Initiator) (Switch)

Multiple connections allowed per device

17
 Fibre Channel Addressing
Dual Port HBA Every Fibre Channel port and node has two
10:00:00:00:c9:6e:a8:16 64 bit hard-coded addresses called World
10:00:00:00:c9:6e:a8:17
Wide Names (WWN)
50:0a:09:83:9d:53:43:54 • NWWN(node) uniquely identify devices
• PWWN(port) uniquely identify each port in a
device
• Allocated to manufacturer by IEEE
Host Switch Disk

phx2-9513# show int fc 1/1 • Coded into each device when manufactured
fc1/1 is up
Hardware is Fibre Channel, SFP is short wave laser Switch Name Server maps PWWN to FCID
Port WWN is 20:01:00:05:9b:29:e8:80
4 bits 12 bits 24 bits 24 bits
N-port or IEEE Organizational Unique ID
0002 Locally Assigned Identifier
F_port Identifier (OUI)
Format Identifier Port Identifier Assigned to each vendor Vendor-Unique Assignment
18
 Port Initialization – FLOGI and PLOGIGIs/PLOGIs Target
Step 1: Fabric Login (FLOGI)
• Determines the presence or absence of a Fabric 3
FC Fabric
• Exchanges Service Parameters with the Fabric
• Switch identifies the WWN in the service parameters
of the accept frame and assigns a Fibre Channel ID
(FCID)
• Initializes the buffer-to-buffer credits E_Port

Step 2: Port Login (PLOGI) F_Port

• Required between nodes that want to communicate 1
• Similar to FLOGI – Transports a PLOGI frame to the N_Port
designation node port
• In P2P topology (no fabric present), initializes buffer- 2
HBA
to-buffer credits

Initiator
19
FC_ID Address Model
• FC_ID address models help speed up FC routing
• Switches assign FC_ID addresses to N_Ports
• Some addresses are reserved for fabric services
• Private loop devices only understand 8-bit address (0x0000xx)
• FL_Port can provide proxy service for public address translation
• Maximum switch domains = 239 (based on standard)
8 Bits 8 Bits 8 Bits
Switch
Switch Topology Model Area Device
Domain
Private Loop Device Arbitrated Loop
00 00 Physical Address (AL_PA)
Address Model
Public Loop Device Switch Arbitrated Loop
Area Physical Address (AL_PA)
Address Model Domain

20
FSPF
Fabric Shortest Path First
• Provides routing services within any FC fabric
• Supports multipath routing
• Bases path status on a link state protocol similar to OSPF
• Routes hop by hop, based only on the domain ID
• Runs on E ports or TE ports and provides a loop free topology
• Runs on a per VSAN basis. Connectivity in a given VSAN in a fabric is guaranteed only for the switches
configured in that VSAN.
• Uses a topology database to keep track of the state of the links on all switches in the fabric and
associates a cost with each link
• Fibre Channel standard ANSI T11 FC-SW2

21
 FSPF
phx2-5548-3# show fsp database vsan 12

FSPF Link State Database for VSAN 12 Domain 0x02(2)

LSR Type = 1
Advertising domain ID = 0x02(2)
LSR Age = 1400 16G
Number of links = 4 Port-Channel
NbrDomainId IfIndex NbrIfIndex Link Type Cost
-----------------------------------------------------------------------------
0x01(1) 0x00010101 0x00010003 1 125
0x01(1) 0x00010100 0x00010002 1 125
0x03(3) 0x00040000 0x00040000 1 62 16G
0x03(3) 0x00040001 0x00040001 1 125

FSPF Link State Database for VSAN 12 Domain 0x03(3) 8G

LSR Type = 1
Advertising domain ID = 0x03(3)
LSR Age
Number of links
= 1486
= 2
5548 9148-2
NbrDomainId IfIndex NbrIfIndex Link Type Cost 1/1
----------------------------------------------------------------------------- 2/13
0x02(2) 0x00040000 0x00040000 1 62 2/14 1 1/2
0x02(2) 0x00040001 0x00040001 1 125
phx2-5548-3# 2/15 1/3
D2 2/16 2 1/4 D3
22
Fibre Channel FC-2 Hierarchy
• Multiple exchanges are initiated between initiators (hosts) and targets (disks)
• Each exchange consists of one or more bidirectional sequences
• Each sequence consists of one or more frames
• For the SCSI3 ULP, each exchange maps to a SCSI command
OX_ID and
RX_ID Exchange

SEQ_ID Sequence Sequence Sequence

SEQ_CNT Frame Frame Frame

Frame Fields
ULP Information Unit
23
What Is FCoE?
It’s Fibre Channel
From a Fibre Channel standpoint it’s
• FC connectivity over a new type of cable called… Ethernet
From an Ethernet standpoints it’s
• Yet another ULP (Upper Layer Protocol) to be transported

FC-4 ULP Mapping FC-4 ULP Mapping

FC-3 Generic Services FC-3 Generic Services
FC-2 Framing & Flow Control FC-2 Framing & Flow Control
FC-1 Encoding FCoE Logical End Point
FC-0 Physical Interface Ethernet Media Access Control
Ethernet Physical Layer
24
 Standards for FCoE
FCoE is fully defined in FC-BB-5 standard
FCoE works alongside additional technologies to make I/O Consolidation a reality

FCoE
IEEE 802.1
T11 DCB
FC on
FC on Other
other
Network
network
Media
PFC ETS DCBX
media

Lossless Priority Configuration

Ethernet Grouping Verification

FC-BB-5 802.1Qbb 802.1Qaz 802.1Qaz

Technically stable October, 2008

Standard Completed in June 2009
Sponsor Ballot July 2010 Sponsor Ballot October 2010 Sponsor Ballot October
25 2010

Status Published Fall 2011 Published Fall 2011 Published Fall 2011
Published in May, 2010 25
 • VLAN Tag enables 8 priorities
FCoE Flow Control for Ethernet traffic
IEEE 802.1Qbb Priority Flow Control
3.3ms • PFC enables Flow Control on a
Per-Priority basis using
Resume
PAUSE frames (IEEE 802.1p)
• Receiving device/switch sends
Pause frame when receiving
buffer passes threshold
• Two types of pause frames
• Quanta = 65535 = 3.3ms
• Quanta = 0 = Immediate resume

• Distance support is determined

FCoE
by how much buffer is
available to absorb data in
Ethernet Wire flight after Pause frame sent
26
ETS: Enhanced Transmission Selection
IEEE 802.1Qaz
• Allows you to create priority groups
• Can guarantee bandwidth
• Can assign bandwidth percentages
to groups
• Not all priorities need to be used or
in groups

80%20% 80% 20%

FCoE

Ethernet Wire
27
FCoE Is Really Two Different Protocols
FIP (FCoE Initialization Protocol) FCoE Itself
• It is the control plane protocol • Is the data plane protocol
• It is used to discover the FC entities • It is used to carry most of the FC
connected to an Ethernet cloud frames and all the SCSI traffic
• It is also used to login to and logout • Ethertype 0x8906
from the FC fabric
• Uses unique BIA on CNA for MAC
• Ethertype 0x8914
The Two Protocols Have
• Two different Ethertypes
• Two different frame formats
• Both are defined in FC-BB-5
28
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html
FPMA - Fabric Provided MAC Address
Fibre Channel over Ethernet Addressing Scheme Domain ID

 FPMA assigned for each FCID FC Fabric Domain

ID 11

 FPMA composed of a FC-MAP and FCID

FCID
 FC-MAP – Mapped Address Prefix Domain
ID 10
11.00.01

 the upper 24 bits of the FPMA FCID

 FCID is the lower 24 bits of the FPMA 10.00.01

 FCoE forwarding decisions still made based on Fibre Channel

FSPF and the FCID within the FPMA FCID Addressing
FC-MAP FC-ID
(0E-FC-xx) 10.00.01

FC-MAP FC-ID
FPMA (0E-FC-xx) 10.00.01

29
What is an FCoE Switch?
• FCF (Fibre Channel Forwarder) accepts a Fibre Channel frame
encapsulated in an Ethernet packet and forwards that packet
over a VLAN across an Ethernet network to a remote FCoE end FCoE
device Attached
Storage
• FCF is a logical FC switch inside an FCoE switch
• Fibre Channel login happens at the FCF Nexus
• Contains an FCF-MAC address
FCF
• Consumes a Domain ID

• FCoE encapsulation/decapsulation happens within the FCF

Nexus FCoE MDS
• NPV devices are not FCF’s and do not have domains FCF

FC

30
FCoE is Operationally Identical
• Supports both FC and FCoE
• FCoE is treated exactly the same as FC
• After zoning device perform registration and then performs discovery

phx2-9513# show fcns database vsan 42

VSAN 42:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0xac0600 N 50:0a:09:83:8d:53:43:54 (NetApp) scsi-fcp:target
Which 0xac0700 N 50:0a:09:84:9d:53:43:54 (NetApp) scsi-fcp:target
are 0xac0c00 N 20:41:54:7f:ee:07:9c:00 (Cisco) npv
FCoE 0xac1800 N 10:00:00:00:c9:6e:b7:f0 scsi-fcp:init fc-gs
0xef0000 N 20:01:a0:36:9f:0d:eb:25 scsi-fcp:init fc-gs
hosts?

33
After Link Is Up, Accessing Storage
FIP and FCoE Login Process
Target
Step 1: FIP Discovery Process FC or FCoE
• Enables FCoE adapters to discover which VLAN to
transmit & receive FCoE frames
Fabric
• Enables FCoE adapters and FCoE switches to discover
other FCoE capable devices
• Occurs over Lossless Ethernet
E_Ports or
Step 2: FIP Login Process VE_Port
• Similar to existing Fibre Channel Login (FLOGI) process
– Sent to upstream FCF VF_Port
• FCF assigns the host a FCID and FPMA to be used for
FCoE forwarding VN_Port
• Returns the FCID and the Fabric Provided MAC Address
(FPMA) to the ENode
FC-MAC CNA

ENode FIP Discovery

34
SCSI is the foundation for all
Operating System

SCSI SCSI SCSI SCSI SCSI

FCP FCP FCP iSCSI

FCP FCP FCP

FCIP

TCP TCP
TCP
FC
F Co
oEE IP IP
Lossless
Ethernet
Ethernet Ethernet
Physical Wire

35
 Connectivity Types
FC FCoE

N F F N VN VF VF VN

Target Switch Initiator Target Switch Initiator

E E
F NP VE VE VF VNP
TE TE

Switch Switch Blade Server Switch Switch Blade Server

Chassis Chassis
36
Fibre Channel Port Types
Summary

Fibre Channel Switch

Fabric NPV
E_Port E_Port F_Port NP_Port
Switch Switch

Fabric TE_Port NPV

TE_Port VF_Port VNP_Port
Switch Switch

Fabric VE_Port VE_Port End

F_Port N_Port
Switch Node
End
G_Port VF_Port VN_Port Node

37
The Story of Interface Speeds
Clocking Encoding Data Rate • Comparing speeds is more
Protocol
Gbps Data/Sent Gbps MB/s complex than just the
8G FC 8.500 8b/10b 6.8 850 “apparent” speed
• Data throughput is based on
10G FC 10.51875 64b/66b 10.2 1,275
both the interface clocking
10G FCoE 10.3125 64b/66b 10.0 1,250 (how fast the interface
transmits) and how efficient
16G FC 14.025 64b/66b 13.6 1,700 the interface transmits (how
much encoding overhead)
32G FC 28.050 64b/66b 27.2 3,400

40G FCoE 41.250 64b/66b 40.0 5,000

38

38
Design Principles

39
VSANs
Introduced in 2002
• A Virtual SAN (VSAN) Provides a Method to Allocate Ports within a
Physical Fabric and Create Virtual Fabrics
• Analogous to VLANs in Ethernet Per Port Allocation
• Virtual fabrics created from larger cost-effective redundant physical
fabric
• Reduces wasted ports of a SAN island approach
• Fabric events are isolated per VSAN which gives further isolation
for High Availability
• FC Features can be configured on a per VSAN basis.
• ANSI T.11 committee and is now part of Fibre Channel standards
as Virtual Fabrics

40
 • Assign ports to VSANs
VSAN • Logically separate fabrics
• Hardware enforced
• Prevents fabric disruptions
• RSCN sent within fabric only

• Each fabric service (zone server, name server,

login server, etc.) operates independently in
each VSAN
• Each VSAN is configured and managed
independently

phx2-9513# show fspf vsan 43 vsan database

FSPF routing for VSAN 43 vsan 2 interface fc1/1
FSPF routing administration status is enabled vsan 2 interface fc1/2
FSPF routing operational status is UP vsan 4 interface fc1/8
It is an intra-domain router vsan 4 interface fc1/9
Autonomous region is 0
MinLsArrival = 1000 msec , MinLsInterval = 2000 msec phx2-9513# show zoneset active vsan 43
Local Domain is 0xe6(230) zoneset name UCS-Fabric-B vsan 43
Number of LSRs = 3, Total Checksum = 0x00012848 zone name UCS-B-VMware-Netapp vsan 43 41
Zoning & VSANs
1. Assign physical ports to VSANs
VSAN 2

Disk2
2. Configure zones within each VSAN
Disk3 • A zone consists of multiple zone members
Zone A Host1 Disk1
Zone C
Zone B 3. Assign zones to zoneset
Disk4 Host2
• Each VSAN has its own zoneset
Zoneset 1
4. Activate zoneset in VSAN
VSAN 3
• Members in a zone can access each other;
Zone A
Host4 members in different zones cannot access
Zone B
Host3 Disk5 each other
Disk6
Zoneset 1
• Devices can belong to more than one zone
42
 Zoning examples
• Non-zoned devices are members of zone name AS01_NetApp vsan 42
the default zone member pwwn 20:03:00:25:b5:0a:00:06
member pwwn 50:0a:09:84:9d:53:43:54
• A physical fabric can have a maximum
of 16,000 zones (9700-only network)
device-alias name AS01
• Attributes can include pWWN, FC pwwn 20:03:00:25:b5:0a:00:06
alias, FCID, FWWN, Switch Interface device-alias name NTAP
member pwwn 50:0a:09:84:9d:53:43:54
fc x/y, Symbolic node name, Device zone name AS01_NetApp vsan 42
alias member device-alias AS01
member device-alias NTAP

43
The Trouble with sizable Zoning
All Zone Members are Created Equal

 Standard zoning model just

Number of ACLs
has “members” 10,000

Number of ACL Entries

 Any member can talk to any 8,000
other member
6,000
 Recommendation: 1-1 zoning
4,000
 Each pair consumes an ACL
entry in TCAM 2,000
 Result: n*(n-1) entries 0

100
60
10
20
30
40
50
70
80
90
Number of Members
44
Smart Zoning
Operation Today – 1:1Operation
Zoning Today – Many
Operation
- Many Smart Zoning
8xI
Zones Cmds ACLs Zones Cmds ACLs Zones Cmds ACLs
Create Create Create
4xT
32 96 64 1 13 132 1 13 64
zones(s) zones(s) zones(s)
Add an +4 +12 Add +8
an +1 Add+24an +1 +8
initiator initiator initiator
Add a +8 +24 Add a
+16 +1 Add a
+24 +1 +16
target target target

• Feature added in NX-OS 5.2(6)

• Allows storage admins to create larger zones while still keeping premise of single initiator & single target

• Dramatic reduction SAN administrative time for zoning

• Utility to convert existing zone or zoneset to Smart Zoning

45
How to enable Smart Zoning
New Zone Existing Zone

46
Zoning Best Practices
• zone mode enhanced
• Acquires lock on all switches while zoning changes are underway
• Enables full zoneset distribution

• zone confirm-commit
• Causes zoning changes to be displayed during zone commit
• zoneset overwrite-control – New in NX-OS 6.2(13)
• Prevents a different zoneset than the currently activated zoneset from being
inadvertently activated

Note: Above setting are per-VSAN

48
IVR - Inter-VSAN Routing
• Enables devices in different VSANs to
VSAN 2
communicate
Disk2

Zone A
Disk3
Disk1
• Allows selective routing between specific
Host1
Zone C members of two or
Zone B
Disk4 Host2 more VSANs
• Traffic flow between selective devices
Zoneset 1

• Resource sharing, i.e., tape libraries and

VSAN 3
disks
Host4
Zone A • IVR Zoneset
Zone B
Disk5
Host3 • A collection of IVR zones that must be activated
Disk6
to be operational
Zoneset 1
49
Forward Error Correction - FEC
• Allows for the correction of some errors in frames
9710-2# show interface fc1/8
• Almost zero latency penalty
fc1/8 is trunking

• Can prevent SCSI timeouts and aborts …

Port mode is TE
• Applies to MDS 9700 FC and MDS 9396S Port vsan is 1
Speed is 16 Gbps
• Applies to 16G fixed speed FC ISLs only Rate mode is dedicated

switchport speed 16000 Transmit B2B Credit is 500

Receive B2B Credit is 500
• Configured via: B2B State Change Number is 14
Receive data field Size is 2112
switchport fec tts Beacon is turned off
admin fec state is up
• No reason not to use it! oper fec state is up
Trunk vsans (admin allowed and active) (1-2,20,237)
50
Trunking & Port Channels
Trunking
Single-link ISL or PortChannel ISL can be configured to
become EISL – (TE_Port)
Trunk

VSAN1 VSAN1 Traffic engineering with pruning VSANs on/off the trunk
VSAN2 VSAN2
Efficient use of ISL bandwidth
VSAN3 VSAN3

TE Port TE Port Up to 16 links can be combined into a PortChannel

increasing the aggregate bandwidth by distributing traffic
granularly among all functional links in the channel
Port Channel
Port
TE Port
Load balances across multiple links and maintains optimum
TE Port Channel
bandwidth utilization. Load balancing is based on the
source ID, destination ID, and exchange ID

If one link fails, traffic previously carried on this link is

switched to the remaining links. To the upper protocol, the
link is still there, although the bandwidth is diminished. The
E Port E Port
routing tables are not affected by link failure
51
 N-Port Virtualization
Scaling Fabrics with Stability
• N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a “switch” to act like a server/HBA performing multiple fabric
logins through a single physical link

• Physical servers connect to the NPV switch and login to the upstream NPIV core switch

• No local switching is done on an FC switch in NPV mode

• FC edge switch in NPV mode does not take up a domain ID phx2-9513 (config)# feature npiv
• Helps to alleviate domain ID exhaustion in large fabrics

NPV Switch FC NPIV

Blade Server Core Switch
F-Port
Server1 F-Port
FC1/1 N_Port_ID 1
Server2
FC1/2 N_Port_ID 2 NP-Port

Server3
FC1/3 F_Port
N_Port_ID 3
52
N-Port
Comparison Between NPIV and NPV
NPIV (N-Port ID Virtualization) NPV (N-Port Virtualizer)
•Used by HBA and FC •Used by FC (MDS 9124, 9148,
9148S, etc.), FCOE switches
switches (Nexus 5K), blade switches and
•Enables multiple logins on a Cisco UCS Fabric InterConnects
single interface (UCS6100)
•Allows SAN to control and •Aggregate multiple physical/logical
monitor virtual machines logins to the core switch
(VMs) •Addresses the explosion of number
of FC switches
•Used for VMWare, MS Virtual
Server and Linux Xen •Used for server consolidation
applications
applications

53
 NPV Uplink Selection
NPV supports automatic selection of NP uplinks. When a server interface is brought up, the NP uplink
interface with the minimum load is selected from the available NP uplinks in the same VSAN as the
server interface.

When a new NP uplink interface becomes operational, the existing load is not redistributed automatically
to include the newly available uplink. Server interfaces that become operational after the NP uplink can
select the new NP uplink.

Manual method with NPV Traffic-Maps associates one or more NP uplink interfaces with a server
interface.

Note: Use of parallel NPV links will pin traffic to one NPV link. Use of SAN Portchannels with NPV actual
traffic will be load balanced.
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli_rel_4_0_1a/CLIConfigurationGuide/npv.html#wp1534672

54
NPV Uplink Selection – UCS Example
• NPV uplink selection can be automatic or manual
• With UCS autoselection, the vHBAs will be uniformly assigned to the available
uplinks depending on the number of logins on each uplink
Cisco UCS FC NPIV
Blade Server NPV Switch NP-Port Core Switch
F-Port

FC1/1
FC1/2 F_Port
FC1/3
FC1/4 F_Port
FC1/5
FC1/6

58
Uplink Port Failure
• Failure of an uplink moves pinned hosts from failed port to up port(s)
• Path selection is the same as when new hosts join NPV switch and pathing
decision is made
2 devices re-login
Cisco UCS
Blade Server
FC NPIV
NPV Switch NP-Port
Core Switch
F-Port

FC1/1
FC1/2 F_Port
FC1/3 Port is Down

FC1/4 F_Port
FC1/5
FC1/6

59
Uplink Port Recovery
• No automatic redistribution of hosts to recovered NP port

Cisco UCS
Blade Server FC NPIV
NPV Switch NP-Port
Core Switch
F-Port

FC1/1
FC1/2 F_Port
FC1/3 Port is Up

FC1/4 F_Port
FC1/5
FC1/6

60
New F-Port Attached Host
• New host entering fabric is automatically pinned to recovered NP_Port
• Previously pinned hosts are still not automatically redistributed
Cisco UCS
Blade Server
FC NPIV
NPV Switch NP-Port
Core Switch
F-Port

FC1/1
FC1/2 F_Port
FC1/3
FC1/4 F_Port
FC1/5
FC1/6

61
New NP_Port & New F-Port Attached Host
• NPV continues to distribute new hosts joining fabric

Cisco UCS
Blade Server
FC NPIV
NPV Switch NP-Port
Core Switch
F-Port

FC1/1
FC1/2 F_Port
FC1/3
FC1/4 F_Port
FC1/5
FC1/6 F_Port
New Port Added
62
Auto-Load-Balance

npv_switch(config)# npv auto-load-balance disruptive

This is Disruptive

Disruptive load balance works independent of automatic selection of interfaces and a configured traffic map of external
interfaces. This feature forces reinitialization of the server interfaces to achieve load balance when this feature is
enabled and whenever a new external interface comes up. To avoid flapping the server interfaces too often, enable this
feature once and then disable it whenever the needed load balance is achieved.

If disruptive load balance is not enabled, you need to manually flap the server interface to move some of the load to a
new external interface.

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/6_2/configuration/guides/interf
aces/nx-os/cli_interfaces/npv.html#pgfId-1072790

63
F-Port Port Channel and F-Port Trunking
Enhanced Blade Switch Resiliency
F-Port Port Channel F-Port Port Channel w/ NPV

Core Bundle multiple ports in to 1 logical link

F-Port Port
Channel Director Storage
Any port, any module
Blade System

Blade N  High-Availability (HA)

Blade 2
Blade Servers are transparent if a cable, port, or line
Blade 1 cards fails
 Traffic Management
N-Ports F-Ports
Higher aggregate bandwidth
Hardware-based load balancing
F-Port Trunking
F-Port Trunking w/ NPV
Core
Director Partition F-Port to carry traffic for multiple VSANs
F-Port
Trunking
 Extend VSAN benefits to Blade
Blade System

VSAN1
Blade N Servers
Blade 2 VSAN2
Separate management domains
Blade 1 VSAN3
Separate fault isolation domains
N-Port F-Port Differentiated services: QoS, Security
64
Port Channeling & Trunking - Configuration
phx2-5548-3# show run interface san-port-channel 1

interface san-port-channel 1
switchport trunk allowed vsan 1
switchport trunk allowed vsan add 12

phx2-5548-3# show run interface fc 2/13-14

interface fc2/13
channel-group 1 force Nexus MDS
no shutdown 5548 9148
fc2/13 fc1/1
interface fc2/14 fc2/14 1 fc1/2
channel-group 1 force
no shutdown
D2 D3

67
Port Channeling & Trunking - Configuration
phx2-9148-2# show run interface port-channel 1

interface port-channel1
switchport trunk allowed vsan 1
switchport trunk allowed vsan add 12

phx2-9148-2# show run interface fc1/1-2

interface fc1/1
channel-group 1 force Nexus MDS
no shutdown 5548 9148
fc2/13 fc1/1
interface fc1/2 fc2/14 1 fc1/2
channel-group 1 force
no shutdown
D2 D3

68
Port Channel – Nexus switch config
phx2-5548-3# show run int san-port-channel 3 Nexus
5548
interface san-port-channel 3
channel mode active
switchport mode F
switchport trunk allowed vsan 1
switchport trunk allowed vsan add 12 D2
phx2-5548-3# show run int fc 2/9-10 fc2/10
fc2/9
interface fc2/9 3
switchport mode F
channel-group 3 force
fc2/1 fc2/2
no shutdown

interface fc2/10
switchport mode F Fabric
channel-group 3 force
no shutdown Interconnect
71
Port Channel – FI Config
5548

D2

fc2/9 fc2/10
3

fc2/1 fc2/2

Fabric
Interconnect
72
FLOGI – Before Port Channel
phx2-5548-3# show flogi database 5548
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
fc2/9 12 0x020000 20:41:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01
fc2/9 12 0x020001 20:02:00:25:b5:0b:00:02 20:02:00:25:b5:00:00:02
fc2/9 12 0x020002 20:02:00:25:b5:0b:00:04 20:02:00:25:b5:00:00:04
D2
fc2/9 12 0x020003 20:02:00:25:b5:0b:00:01 20:02:00:25:b5:00:00:01
fc2/10 12 0x020020 20:42:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01 fc2/9 fc2/10
fc2/10 12 0x020021 20:02:00:25:b5:0b:00:03 20:02:00:25:b5:00:00:03
fc2/10 12 0x020022 20:02:00:25:b5:0b:00:00 20:02:00:25:b5:00:00:00

Total number of flogi = 7

fc2/1 fc2/2
phx2-5548-3#

Fabric
Interconnect
73
FLOGI- After port channel
phx2-5548-3# show flogi database 5548
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
San-po3 12 0x020040 24:0c:00:0d:ec:fd:9e:00 20:0c:00:0d:ec:fd:9e:01
San-po3 12 0x020001 20:02:00:25:b5:0b:00:02 20:02:00:25:b5:00:00:02
San-po3 12 0x020002 20:02:00:25:b5:0b:00:04 20:02:00:25:b5:00:00:04 D2
San-po3 12 0x020003 20:02:00:25:b5:0b:00:01 20:02:00:25:b5:00:00:01
San-po3 12 0x020021 20:02:00:25:b5:0b:00:03 20:02:00:25:b5:00:00:03 2/9 2/10
San-po3 12 0x020022 20:02:00:25:b5:0b:00:00 20:02:00:25:b5:00:00:00

Total number of flogi = 6

phx2-5548-3# 2/1 2/2

Fabric
Interconnect
74
Port-channel design considerations
• All types of switches
• Name port-channels the same on both sides
• Common port allocation in both fabrics
• ISL speeds should be >= edge device speeds
• Maximum 16 members per port-channel allowed
• Multiple port-channels to same adjacent switch should be equal cost
• Member of VSAN 1 + trunk other VSANs
• Check TCAM usage:
• show system internal acl tcam-usage

75
port-channel design considerations
• Director class
• Split port-channel members across multiple line cards
• When possible use same port on each LC:
• Ex. fc1/5, fc2/5, fc3/5, fc4/5, etc.
• If multiple members per linecard distribute across port-groups
• show port-resources module x

76
Port-channel design considerations
• Fabric switches
• Ensure enough credits for distance
• Can “rob” buffers from other ports in port-group that are “out-of-service”
• Split port-channel member across different forwarding engines to distribute
ACLTCAM
• For F port-channels to NPV switches (like UCS FIs)
• Each device’s zoning ACLTCAM programming will be repeated on each PC member
• For E port-channels using IVR
• Each host/target session that gets translated will take up ACLTCAM on each member
• Use following table:
• Ex. On a 9148S a six member port-channel could be allocated across the 3 fwd engines as
follows:
• fc1/1, fc1/2, fc1/17, fc1/18, fc1/33 and fc1/34

• Consider MDS 9396S for larger scale deployments

77
F port-channel design considerations
Ports are allocated to fwd-engines according the following table:
Fwd Zoning Region Bottom Region
Switch/Module Port Range(s) Fwd-Eng Number
Engines Entries Entries
MDS 9148 3 fc1/25-36 & fc1/45-48 1 2852 407
fc1/5-12 & fc1/37-44 2 2852 407
1-4 & 13-24 3 2852 407
MDS 9250i 4 fc1/5-12 & eth1/1-8 1 2852 407
fc1/1-4 & fc1/13-20 &
2 2852 407
fc1/37-40
fc1/21-36 3 2852 407
ips1/1-2 4 2852 407
MDS 9148S 3 1-16 1 2852 407
17-32 2 2852 407
33-48 3 2852 407

78
F port-channel design considerations
Fwd Zoning Region Bottom Region
Switch/Module Port Range(s) Fwd-Eng Number
Engines Entries Entries
MDS 9396S 12 1-8 0 49136 19664
9-16 1 49136 19664
17-24 2 49136 19664
25-32 3 49136 19664
33-40 4 49136 19664
41-48 5 49136 19664
49-56 6 49136 19664
57-64 7 49136 19664
65-72 8 49136 19664
73-80 9 49136 19664
81-88 10 49136 19664
89-96 11 49136 19664

79
F port-channel design considerations
Fwd Zoning Region Bottom Region
Switch/Module Port Range(s) Fwd-Eng Number
Engines Entries Entries
DS-X9248-48K9 1 1-48 0 27168 2680
DS-X9248-96K9 2 1-24 0 27168 2680
25-48 1 27168 2680
DS-X9224-96K9 2 1-12 0 27168 2680
13-24 1 27168 2680
DS-X9232-256K9 4 1-8 0 49136 19664
9-16 1 49136 19664
17-24 2 49136 19664
25-32 3 49136 19664
DS-X9248-256K9 4 1-12 0 49136 19664
13-24 1 49136 19664
25-36 2 49136 19664
37-48 3 49136 19664

80
F port-channel design considerations
Fwd Zoning Region Bottom Region
Switch/Module Port Range(s) Fwd-Eng Number
Engines Entries Entries
DS-X9448-768K9 6 1-8 0 49136 19664
9-16 1 49136 19664
17-24 2 49136 19664
25-32 3 49136 19664
33-40 4 49136 19664
41-48 5 49136 19664

81
Internal CRC handling
• New feature to handle frames internally corrupted due to bad HW
• Frames that are received corrupted are dropped at the ingress port
• These frames are not included in this feature
• In rare cases frames can get corrupted internally due to bad hardware
• These are then dropped
• Sometimes difficult to detect

• New feature detects the condition and isolates hardware

• 5 possible stages where frames can get corrupted

82
Internal CRC handling
• Stages of Internal CRC Detection and Isolation

The five possible stages at which

internal CRC errors may occur in a
switch:
1. Ingress buffer of a module
2. Ingress crossbar of a module
3. Crossbar of a fabric module
4. Egress crossbar of a module
5. Egress buffer of a module
83
Internal CRC handling
• The modules that support this functionality are:
• Cisco MDS 9700 48-Port 16-Gbps Fibre Channel Switching Module
• Cisco MDS 9700 48-Port 10-Gbps Fibre Channel over Ethernet Switching Module
• Cisco MDS 9700 Fabric Module 1
• Cisco MDS 9700 Supervisors

• Enabled via the following configuration command:

• hardware fabric crc threshold 1-100
• When detected failing module is powered down
• New in NX-OS 6.2(13)

84
Device-alias
• device-alias(DA) is a way of naming PWWNs
• DAs are distributed on a fabric basis via CFS
• device-alias database is independent of VSANs
• If a device is moved from one VSAN to another no DA changes are needed
• device-alias can run in two modes:
• Basic – device-alias names can be used but PWWNs are substituted in config
• Enhanced – device-alias names exist in configuration natively – Allows rename without
zoneset re-activations
• device-alias are used in zoning, IVR zoning and port-security
• copy running-config startup-config fabric after making changes!

85
Device-alias
• device-alias confirm-commit
• Displays the changes and prompts for confirmation

MDS9710-2(config)# device-alias confirm-commit enable

MDS9710-2(config)# device-alias database
MDS9710-2(config-device-alias-db)# device-alias name edm pwwn 1000000011111111
MDS9710-2(config-device-alias-db)# device-alias commit
The following device-alias changes are about to be committed
+ device-alias name edm pwwn 10:00:00:00:11:11:11:11
Do you want to continue? (y/n) [n]

86
Device-alias
• Note: To prevent problems the same device-alias is only allowed once per
commit.
• Example:
MDS9148s-1(config)# device-alias database
MDS9148s-1(config-device-alias-db)# device-alias name test pwwn 1122334455667788
MDS9148s-1(config-device-alias-db)# device-alias rename test test1
Command rejected. Device-alias reused in current session :test
Please use 'show device-alias session rejected' to display the rejected set of commands and for the
device-alias best-practices recommendation.

87
 Cisco Prime Data Center Network Manager
Feature Support and User Interface
VMpath Analysis provides VM connectivity to
network and storage across Unified Compute
and Unified Fabric
• Visibility past physical access (switch)
layer
• Standard & Custom Reports
• On Nexus and MDS platforms
• Dynamic Topology Views
• Rule-based event filtering and
forwarding
• Threshold Alerting
• Integration via vCenter API

88
SAN Design Security Challenges
SAN design security is often overlooked as an area of concern
• Application integrity and security is addressed, but not back-end storage network carrying actual data
• SAN extension solutions now push SANs outside datacenter boundaries

Not all compromises are intentional

FC
• Accidental breaches can still have the same consequences

SAN design security is only one part of complete data center solution
• Host access security—one-time passwords, auditing, VPNs
• Storage security—data-at-rest encryption, LUN security
Privilege Escalation/ Theft
External DOS
or Other Unintended Privilege
Unauthorized Data
Intrusion Connections Tampering
Application
Tampering (Internal)
(Trojans, etc.)

SAN

LAN 89
 SAN Security
Device/SAN
Secure management access Management
Security Via SSH,
• Role-based access control
SFTP, SNMPv3, and
• CLI, SNMP, and web access User Roles RADIUS or
TACACS+ or LDAP
Secure management protocols Server for
Authentication
• SSH, SFTP, and SNMPv3

Secure switch control protocols

• TrustSec SAN
• FC-SP (DH-CHAP) Protocol
Security
(FC-SP)
AAA - RADIUS, TACACS+ and LDAP
iSCSI-
• User, switch and iSCSI host authentication VSANs Attached
Provide Servers
Fabric Binding Secure Hardware-Based
• Prevent unauthorized switches from joining the Isolation Zoning Via Port and
fabric WWN

Shared Physical Storage 90

Slow Drain
Slow Drain Device Detection and Congestion Avoidance

• Devices can impart slowness in a fabric

• Feature of the fabric that’ll expose that device for remediation
• BRKSAN-3446 SAN Congestion! Understanding, Troubleshooting, Mitigating in
a Cisco Fabric
White paper (2013) http://www.cisco.com/en/US/prod/collateral/ps4159/ps6409/ps12970/white_paper_c11-729444.pdf

91
Storage Fabric Topology
Considerations

92
The Importance of “Architecture”

SAN designs traditionally robust: Latency

dual fabrics, data loss is not • Initiator to target
tolerated • Slow drain

Must manage ratios • Performance under load: does my

• Fan in/out fabric perform the same

• ISL oversubscription Application independence
• Virtualized storage IO streams • Consistent fabric performance
(NPIV attached devices, server regardless of changes to SCSI profile
RDM, LPARs, etc.) • Number of frames
• Frame size
• Queue depth
• Speed or throughput

93
 SAN Major Design Factors High
Performance
Port density Crossbar
• How many now, how many later?
2
• Topology to accommodate port
requirements Large Port
QoS, Count
Congestion Directors
Network performance
Control,
• What is acceptable? Unavoidable? Reduce FSPF 3
Routes 1
Traffic management
8 8 8 8 8 8 8 8 8 8 8 8
• Preferential routing or resource allocation

Fault isolation
• Consolidation while maintaining isolation

Management
• Secure, simplified management
4
Failure of One Device Has
No Impact on Others 94

94
Scalability—Port Density
Topology Requirements
Considerations
• Number of ports for end devices
Large Port
• How many ports are needed now? Count
Directors
• What is the expected life of the
SAN?

• How many will be needed in

the future?
8 8 8 8 8 8 8 8 8 8 8 8
• Hierarchical SAN design

Best Practice
• Design to cater for future requirements

• Doesn’t imply “build it all now,” but means “cater for it” and
avoids costly retrofits tomorrow

95
Scalability—Port Density – MDS Switch selection
• MDS 9148S – 48 ports 16G FC
• MDS 9250i – 40 ports 16G FC + 8 port 10G FCoE + 2 FCIP ports
• MDS 9396S – 96 ports 16G FC
• MDS 9706 – Up to 192 ports 16G FC and/or 10G FCoE and/or 40G FCoE
• MDS 9710 – Up to 384ports 16G FC and/or 10G FCoE and/or 40G FCoE
• MDS 9718 – Up to 768 ports 16G FC and/or 10G FCoE and/or 40G FCoE
• All MDS 97xx chassis are 32G ready!
• All 16G MDS platforms are full line rate

96
Scalability—Port Density – Nexus Switch selection
• Nexus 55xx – Up to 96 ports 10G FCoE and/or 8G FC ports
• Nexus 5672UP – Up to 48 10G FCoE and/or 16 8G FC ports
• Nexus 5672UP-16G – Up to 48 10G FCoE and/or 24 16G FC ports
• Nexus5624Q – 12 ports 40G or 48 ports 10G FCoE
• Nexus5648Q – 24 ports 40G or 96 ports 10G FCoE
• Nexus5696Q – Up to 32 ports 100G / 96 ports 40G / 384 ports 10G FCoE or 60
8G FC
• Nexus 56128P – Up to 96 10G FCoE and/or 48 8G FC ports
• All Nexus platforms are full line rate

97
Traffic Management
Do different apps/servers have different
performance requirements?
• Should bandwidth be
reserved for specific applications? QoS,
Congestion
• Is preferential treatment/ Control,
QoS necessary? Reduce FSPF
Routes
8 8 8 8 8 8 8 8 8 8 8 8
Given two alternate paths for traffic
between data centers, should traffic
use one path in preference to the other?
• Preferential routes

98
 Network Performance
Oversubscription Design Considerations
All SAN Designs Have Some Degree of
Oversubscription
• Without oversubscription, SANs would Tape Oversubscription
be too costly Disk Oversubscription
Disk do not sustain wire-rate I/O Need to sustain close to
with ‘realistic’ I/O mixtures maximum data rate
• Oversubscription is introduced at LTO-6 Native Transfer
Vendors may recommend a 6:1 to
multiple points as high as 20:1 host to disk Rate ~ 160 MBps
fan-out ratio
• Switches are rarely the bottleneck Highly application dependent
in SAN implementations

• Device capabilities (peak and sustained) 8 8 8 8 8 Port Channels

8
must be considered along with network Help Reduce
oversubscription Oversubscription
While Maintaining
HA Requirements
• Must consider oversubscription during a
network failure event ISL Oversubscription
Two-tier design
ratio less than fan-out ratio Host Oversubscription
Largest variance observed at this level. DB
servers close to line rate, others highly
oversubscribed
16Gb line cards non-oversubscribed

99
Fault Isolation
Consolidation of Storage
• Single Fabric = Increased Storage Utilization +
Reduced Administration Overhead
Major Drawback
• Faults Are No Longer Isolated Physical SAN Islands Are
Virtualized onto Common
• Technologies such as VSANs enable consolidation SAN Infrastructure
and scalability while maintaining security and
stability
• VSANs constrain fault impacts
Fabric
• Faults in one virtual fabric (VSAN) are contained #3
and do not impact other virtual fabrics Fabric
#1 Fabric
#2

100
Data Center SAN
Topologies

101
 Denser Server Cabinets
What are the implications?
Uplinks change from 40 GE servers Vertical Horizontal
to 4x 10G servers Cabling Cabling EoR X-Connect

ToR Main X-Connect

DC Infrastructure Changes
Denser: cabinets, cross-connects cable runs
From 42U to ~58U Horizontal Cabling: from 10G, through 40G to 100G – longer distances
Vertical Cable: match appropriate server connectivity choice
Is SAN EoR economical now? 102
Structured Cabling
Supporting new EoR & ToR designs

• Pricing advantage for manufactured cabling

systems
• Removes guessing game of how many strands to
pull per cabinet
• Growth at 6 or 12 LC ports per cassette
• Fiber-only cable plant designs possible

103
Core-Edge
Highly Scalable Network Design

• Traditional SAN design for growing

SANs
• High density directors in core and fabric
switches, directors or blade switches on
edge
• Predictable performance
• Scalable growth up to core and ISL
capacity
• Evolves to support EoR & ToR
End of Row Top of Rack Blade Server
• MDS 9718 as core
105
Large Edge-Core-Edge/End-of-Row Design “A”
Large Edge/Core/Edge Fabric
(2496 End Device Ports per Fabric) 240 Storage ports at 16Gb Shown,
• Traditional Edge-Core-Edge design Is ideal for (optional 480 @ 8Gb without
Repeat
changing bandwidth ratios)
very large centralized services and consistent
host-disk performance regardless of location for “B”
Fabric
• Full line rate ports, no fabric oversubscription
• 8Gb or 16Gb hosts and targets
• Services consolidated in the core 240 ISLs from storage edge
120
to core @ 16Gb

MDS 9710
• Easy expansion
Ports Deployed 3456 per fabric 6,912 total

Used Ports 5,760 total @ 16Gb 240 ISLs from host

6,240 total @ 8Gb edge to core @ 16Gb
24
Storage Ports 480 total @ 16Gb, or
960 total @ 8Gb

Host Ports 3360 total 1680 hosts @ 8Gb

or 16Gb
ISL ports 960 total

Host ISL Oversubscription 7:1 @ 16Gb

End to End 7:1 @ 16Gb storage

Oversubscription 7:1 @ 8Gb storage

106
Very Large Edge-Core/End-of-Row Design
“A” Fabric Shown,
Very Large Edge/Core/Edge 576(288 per switch) Repeat for “B” Fabric
(6144 End Device Ports per Fabric) Storage ports at 16Gb
MDS 9718
• Traditional Core-Edge design Is ideal for very
large centralized services and consistent host-
disk performance regardless of location
• Full line rate ports, no fabric oversubscription
• 16Gb hosts and targets
• Services consolidated in the core
• Easy expansion
768(48 per switch) 24
Ports Deployed 12,288 ISLs from host edge to
core @ 16Gb
Used Ports 10,368 @ 16Gb

Storage Ports 1152 @ 16Gb MDS 9710

Host Ports 8064
4032 (252 per
ISL ports 768
switch) hosts @ 8Gb
Host ISL Oversubscription 7:1 @ 16Gb or 16Gb

End to End Oversubscription 7:1 @ 16Gb storage

107
 SAN Top of Rack – MDS 9148S
SAN Top of Rack
(5,376 Usable Ports)
352 Storage ports at 16Gb
• Ideal for centralized services while reducing cabling MDS 9710
requirements
• Consistent host/target performance regardless of
location in rack
• 8Gb hosts & 16Gb targets
• Easy edge expansion A B
• Massive cabling infrastructure avoided as compared
to EoR designs 4 ISLs from each
edge to core @ 16Gb
• Additional efficiencies with in rack IO convergence
MDS 9148S
Ports Deployed 5,376
4,224 hosts @ 16Gb
Used Ports 5,344

Storage Ports 352 @ 16Gb

Host Ports 4,224

Host ISL Oversubscription 12:1 @ 16Gb

48 Racks
End to End Oversubscription 12:1 @ 16G hosts
44 Dual-attached
servers per rack
108
Rack
Top-of-Rack Design - Blade Centers
SAN Top of Rack – Blade Centers
(1,920 Usable Ports per Fabric)
96 Storage ports
• Ideal for centralized services at 16Gb MDS 9710
• Consistent host/target performance regardless of
location in blade enclosure or rack
• 8Gb hosts & 16Gb targets
• Need to manage more SAN Edge switches/Blade Switches
• NPV attachment reduces fabric complexity A B

• Assumes little east-west SAN traffic 8 ISLs from each

edge to core @ 8Gb
• Add blade server ISLs to reduce fabric oversubscription Blade Center
Ports Deployed 1,920

Used Ports 192 @ 16Gb 960 hosts @ 8Gb

1056 @ 8Gb

Storage Ports 192 @ 16Gb, or

192 @ 8Gb

Host Ports 2304

Host ISL Oversubscription 4:1 @ 8G 12 Racks, 72 chassis

96 Dual-attached blade
servers per rack
End to End Oversubscription 6:1 @ 16Gb Storage
12:1 @ 8Gb Storage 109
Rack
 Medium Scale Dual Fabric
Collapsed Core Design
“A” Fabric Shown,
Medium Scale Dual Fabric Repeat for “B” Fabric
(768 Usable Ports per Fabric) 96 Storage ports at 16Gb
MDS 9710
• Ideal for centralized services
• Consistent host/target performance regardless of
location
• 8Gb or 16Gb hosts & targets (if they exist)
• Relatively easy edge expansion to Core/Edge
• EoR design
• Supports blade centers connectivity
Ports Deployed 768

Used Ports 768@ 16Gb

Storage Ports 96 @ 16Gb 672 hosts @ 16Gb

Host Ports 672 @ 16Gb

Host ISL Oversubscription N/A

End to End Oversubscription 7:1 @ 16Gb

110
POD SAN Design
POD SAN Design
Ideal for centralized services
36-48 Storage
• Consistent host/target performance regardless of ports at 16Gb
location in blade enclosure or rack
• 10/16Gb hosts & 16Gb targets
• Need to manage more SAN Edge switches/Blade MDS 9396S MDS 9396S
Switches
• NPV attachment reduces fabric complexity A B
• Add blade server ISLs to reduce fabric
6 ISLs from each edge 8 ISLs from each edge
oversubscription to core @ 16Gb to core @ 8Gb

MDS 9148S UCS FI 6248UP

6 Racks, 252 chassis 6 Racks, 288 blades

42 Dual-attached servers 48 Dual-attached blade
per rack servers per rack

111
252 hosts @ 16Gb or 288 hosts @ 10Gb
FI 6332-16UP, FI 6332 UCS SAN Design
FI 6332-16UP Use Case FI 6332 Use Case

40G 40G
Nexus Nexus
7K/9K 7K/9K
16G FC 40G FCoE

FI 6332-16UP FI 6332
40G 40G 40G 40G
Storage Storage
MDS Array MDS Array
UCS UCS 9700 UCS UCS 9700
B-Series C-Series B-Series C-Series
B200 C220 B200 C220
B260 C240 B260 C240
B460 C460 B460 C460
and and
IOM 2304 IOM 2304

40G 40G
16G FC 40G FCoE

112
Intelligent SAN Services

113
Enhancing SAN Design with Services
Extend Fabrics
• FCIP
• Extended Buffer to Buffer credits
• Encrypt the pipe

SAN Services extend the effective distance for remote

applications
• SAN IO acceleration
• Write acceleration SAN Extension with FCIP
• Tape acceleration

Enhance array replication

requirements

Reduces WAN-induced latency

Improves application performance

over distance
Data Migration with
DMM IO Acceleration
Data Migration

Fabric is aware of all data frames from initiator to target 114

SAN Extension with FCIP
Fibre Channel over IP

• Encapsulation of Fibre Channel frames into IP packets and tunneling through

an existing TCP/IP network infrastructure, in order to connect geographically
distant islands
• Write Acceleration to improve throughput and latency
• Hardware-based compression
• Hardware-based IPSec encryption
Array to Array
Replication

FCIP Tunnel TE Port

115
 FC Redirect - How IOA Works
Replication Replication
Starts Starts

Initiator to Flow redirected to Flow accelerated and

target IOA Engine sent towards normal
routing path

IOA IOA

MAN/WAN

IOA IOA

IOA= I/O Accelerator

Initiator Target Initiator Target

Virtual Initiator Virtual Target 116

Data Acceleration
A fabric service to accelerate I/O between SAN devices

• Accelerate SCSI I/O

Over both Fibre Channel (FC) and Fibre Channel over IP (FCIP) links
For both Write Acceleration (WA) and Tape Acceleration (TA)

• I/O Acceleration Node platforms: MSM-18/4, SSN-16, MDS-9222i, MDS-9250i

• Uses FC Redirect

IOA IOA

MAN/WAN

IOA IOA

IOA= I/O Accelerator

117
IOA FCIP Tape Backup
Large Health Insurance Firm

MDS IOA Results

92% throughput
FCIP increase

 Highly resilient– Clustering of IOA engines allows for load balancing and failover
 Improved Scalability- Scale without increasing management overhead
 Significant reutilization of existing infrastructure- All chassis and common
equipment re-utilized
 Flat VSAN topology- Simple capacity and availability planning

118
SAN Extension – FC over long distance
BB_Credits and Distance
~1 km per Frame
2 Gbps FC

~0.5 km per Frame

4 Gbps FC

8 Gbps FC ~0.25 km per Frame

16 Gbps FC ~0.125 km per Frame

16 Km
phx2-9513(config)# feature fcrxbbcredit extended • BB_Credits are used to ensure enough FC frames in flight
phx2-9513(config)# interface 1/1
phx2-9513(config-if)# switchport fcrxbbcredit extended 1000 • A full (2112 byte) FC frame is approx 1 km long @ 2 Gbps,
phx2-9513# show interface 1/1 ½ km long @ 4 Gbps ¼ km long at 8 Gbps
fc1/1 is up
….. • As distance increases, the number of available BB_Credits
Transmit B2B Credit is 128 need to increase as well
Receive B2B Credit is 1000 • Insufficient BB_Credits will throttle performance - no data 119
will be transmitted until R_RDY is returned
 SAN Extension – FCoE over long distance
FCoE Flow Control
For long distance FCoE, receiving switch Ingress Buffer must be large enough to absorb all
packets in flight from the time the Pause frame is sent to the to time the Pause Frame is
received
Buffer Threshold
• A 10GE, 50 km link can hold ~300 frames
• That means 600+ frames could be either in flight or will be transmitted by the time the receiver
detects buffer congestion and sends a Pause frame to the time the Pause frame is received and the
sender stops transmitting

Egress Buffer Frame

Latency Buffer Frame

Frame Frame Pause threshold

Frame Frame
Frame
Frame Ingress Buffer
Frame

Frame Frame Frame Frame Frame

Pause

Latency Buffer turning is platform specific

120
Data Mobility
Application Servers
• Migrates data between storage arrays for
• Technology refreshes
• Workload balancing
• Storage consolidation

• DMM offers
• Online migration of heterogeneous arrays
Data Mobility Manager
• Simultaneous migration of multiple LUNs
Application Data • Unequal size LUN migration
I/O Migration
• Rate adjusted migration
• Verification of migrated data
• Dual fabric support
• CLI and wizard-based management with Cisco Fabric Manager
• Not metered on no. of terabytes migrated or no. of arrays
Old New • Requires no SAN reconfiguration or rewiring
Array Array
• Uses FC Redirect 121
SAN Extension - CWDM
Course Wavelength Division Multiplexing
TX RX
Transmission
TX RX
TX Optical fiber pair RX
TX RX
Optical OADM Optical
transmitters receivers
• 8 channels WDM using 20nm spacing
• Colored CWDM SFPs used in FC switch
• Optical multiplexing done in OADM
• Passive device
122
SAN Extension - DWDM
Dense Wavelength Division Multiplexing
TX Transmission RX
TX RX
Optical Splitter Protection
TX RX
Optical fiber pair
TX RX
Optical DWDM devices Optical
transmitters receivers

• DWDM systems use optical devices to combine the output of several

optical transmitters
• Higher density technology compared with CWDM, <1nm spacing
123
 Dense vs Coarse (DWDM vs CWDM)
DWDM CWDM
Application Long Haul Metro
Amplifiers Typically EDFAs Almost Never
# Channels Up to 80 Up to 8
Channel Spacing 0.4 nm 20nm
Distance Up to 3000km Up to 80km
Spectrum 1530nm to 1560nm 1270nm to 1610nm
Filter Technology Intelligent Passive

Site 1 Site 2 Site 1 Site 2

MDS ONS

Array
124
DWDM CWDM
Summary
Drivers in DC are forcing change Many design options
• 10G convergence & server virtualization • Optimized for performance

• It's not just about FCP anymore. FCoE, NFS, iSCSI are • Some for management
being adopted • Others for cable plant optimization
Proper SAN design is holistic in the approach
• Performance, Scale, Management attributes all play critical roles

• Not all security issues are external

• Fault isolation goes beyond SAN A/B separation

• Consider performance under load

• Design for SAN services

125
 Additional Relevant Sessions
Storage Networking – Cisco Live Berlin

• BRKSAN-3446 - SAN Congestion! Understanding, Troubleshooting, Mitigating in a Cisco Fabric

• Friday 9AM

126
Call to Action
• Visit the World of Solutions for:
• Multiprotocol Storage Networking booth
• See the MDS 9718, Nexus 5672UP, 2348UPQ, and MDS 40G FCoE blade
• Data Center Switching Whisper Suite
• Strategy & Roadmap (Product portfolio includes: Cisco Nexus 2K, 5K, 6K, 7K, and MDS products).
• Technical Solution Clinics
• Meet the Engineer
• Available Tuesday and Thursday

128
Complete Your Online Session Evaluation
• Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.

• All surveys can be completed via

the Cisco Live Mobile App or the
Communication Stations

129
Thank you

130